🎉 DevOps Interview Prep Bundle is live — 1000+ Q&A across 20 topicsGet it →
All Articles
awskubernetestroubleshootingsecurity

AWS IRSA Permission Denied in Kubernetes — Fix

Your Kubernetes pod can't access AWS services even though IRSA is configured. Here's every reason IRSA fails and exactly how to debug and fix each one.

DevOpsBoysJun 7, 20263 min read
Share:Tweet

IRSA (IAM Roles for Service Accounts) lets Kubernetes pods assume AWS IAM roles without static credentials. When it breaks, you get cryptic AccessDenied errors that are hard to trace.

Here's every failure mode and its fix.

Quick Diagnosis

bash
# Check if pod has the AWS token mounted
kubectl exec -it <pod-name> -- env | grep AWS
# Should see: AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE
 
# Test AWS access from inside the pod
kubectl exec -it <pod-name> -- aws sts get-caller-identity
# If this fails → IRSA is broken
# If this works but S3/etc fails → IAM policy issue
 
# Check service account annotation
kubectl get serviceaccount <sa-name> -n <namespace> -o yaml
# Should have: eks.amazonaws.com/role-arn annotation

Case 1: Service Account Missing IAM Role Annotation

The most common cause. No annotation = no role assumption.

bash
# Check annotation
kubectl get serviceaccount my-app-sa -n my-app -o yaml
# Missing if output doesn't show:
# annotations:
#   eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/my-role
 
# Fix: add annotation
kubectl annotate serviceaccount my-app-sa \
  -n my-app \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789:role/my-eks-role

Or in your deployment YAML:

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: my-app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-eks-role

Case 2: IAM Role Trust Policy Wrong

The IAM role must explicitly trust the EKS OIDC provider and the specific service account.

bash
# Get your cluster's OIDC provider URL
aws eks describe-cluster \
  --name my-cluster \
  --query "cluster.identity.oidc.issuer" \
  --output text
# Output: https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E
 
# Check if OIDC provider is registered in IAM
aws iam list-open-id-connect-providers

Correct trust policy:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:my-app:my-app-sa",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Common mistakes in the trust policy:

  • Wrong namespace in sub field
  • Wrong service account name in sub field
  • Missing :aud condition
  • OIDC provider URL has trailing slash or wrong format

Case 3: OIDC Provider Not Registered

bash
# Associate OIDC provider with cluster (one-time setup)
eksctl utils associate-iam-oidc-provider \
  --cluster my-cluster \
  --region us-east-1 \
  --approve
 
# Or via AWS CLI:
aws iam create-open-id-connect-provider \
  --url $(aws eks describe-cluster --name my-cluster \
          --query "cluster.identity.oidc.issuer" --output text) \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list <thumbprint>

Case 4: Pod Not Using the Correct Service Account

bash
# Check which SA the pod is using
kubectl get pod <pod-name> -n my-app -o yaml | grep serviceAccount
 
# If wrong SA or default SA:
# Fix in deployment spec:
spec:
  serviceAccountName: my-app-sa   # Must match the annotated SA

Case 5: IAM Policy Missing Required Permissions

IRSA can assume the role but the role's policy doesn't allow the action.

bash
# Check what the role can do
aws iam list-attached-role-policies --role-name my-eks-role
aws iam get-role-policy --role-name my-eks-role --policy-name inline-policy
 
# Test the exact call that's failing with simulation
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789:role/my-eks-role \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::my-bucket/*

Case 6: Token Expiry Issues

IRSA tokens expire every 24 hours by default. If your pod has been running a long time and suddenly gets AccessDenied, token rotation might be the issue.

bash
# Check token expiry in pod
kubectl exec -it <pod-name> -- cat $AWS_WEB_IDENTITY_TOKEN_FILE | \
  python3 -c "import sys,json,base64; p=sys.stdin.read().split('.')[1]; print(json.loads(base64.b64decode(p+'==')))"
# Look at 'exp' field
 
# AWS SDKs refresh tokens automatically — if you're using raw token:
# Use AWS SDK (boto3, AWS SDK for Go, etc.) which handles refresh

Debug Checklist

1. kubectl exec into pod → aws sts get-caller-identity
   ✅ Works → IAM policy issue (wrong permissions)
   ❌ Fails → IRSA setup issue

2. Check service account annotation exists
3. Check IAM role trust policy has correct OIDC + namespace + SA name
4. Check OIDC provider is registered in IAM
5. Check pod spec uses the annotated service account
6. Check IAM role policy allows the specific action on the specific resource

Learn AWS IAM and EKS security deeply at KodeKloud.

Free Tools
Free Tool

K8s Resource Calculator

Calculate CPU & memory requests/limits for your pods. Never get OOMKilled again.

Free Tool

CI/CD Pipeline Generator

Generate GitHub Actions, GitLab CI, or Jenkins pipelines for your stack instantly.

🔧

Today I Fixed

Short real fixes from production — posted daily

Browse fixes
Back to all articles
Share:Tweet
Newsletter

Stay ahead of the curve

Get the latest DevOps, Kubernetes, AWS, and AI/ML guides delivered straight to your inbox. No spam — just practical engineering content.

Related Articles

Comments