Unauthorized to Perform AWS sts:AssumeRoleWithWebIdentity – Error 403

1. Overview

In this tutorial, we’ll explore the standard AWS error: “Unauthorized to perform sts – 403”. This error often occurs when working with AWS services like Amazon Elastic Kubernetes Service (EKS), where applications must assume IAM roles using web identity federation.

We’ll first explain the reasons behind this error and walk through detailed steps to diagnose and resolve it using the AWS Command Line Interface (CLI).

By the end of this article, we’ll understand how to properly configure AWS Identity and Access Management (IAM) roles and policies to allow STS operations and ensure secure and seamless access to AWS resources.

2. Understanding the Error

When we encounter the “Unauthorized to perform sts– 403” error, it indicates that the AWS Security Token Service (STS) has denied our request to assume a role using web identity federation. This typically happens in scenarios where an application or service is trying to assume an IAM role by exchanging a web identity token. An example of this is JSON Web Tokens (JWT) from an identity provider like AWS Cognito, Google, or the OpenID Connect (OIDC) provider associated with an EKS cluster.

The HTTP status code 403 Forbidden signifies that the client made a valid request, but the server refuses the action due to a lack of necessary permissions. Specifically, the identity attempting the operation doesn’t have the required permissions to perform STS.

Our understanding of this error is crucial because it directly affects our application’s ability to authenticate and access AWS resources securely. Misconfigurations can lead to security vulnerabilities or service disruptions.

3. Common Causes

Several issues can lead to this error, and identifying the root cause is essential for a proper fix:

• Incorrect IAM Role Trust Policy: The assumed IAM role might not trust the identity provider (IdP) or the AWS service making the request. The trust relationship defines which entities can assume the role
• Misconfigured Kubernetes Service Account: In EKS environments, the Kubernetes Service Account might not be adequately annotated with the IAM role ARN, preventing pods from assuming the desired role
• Expired or Invalid Tokens: The web identity token used for authentication may be invalid, expired, or not correctly signed, leading to authentication failures
• Lack of Permissions: The IAM user or role making the request doesn’t have permission to call STS due to missing policies or explicit deny statements
• Incorrect Identity Provider Configuration: The OIDC provider might not be correctly set up in IAM, or there may be a mismatch in the issuer URL and client IDs

Let’s now try to find out how to diagnose this issue and how to fix it.

4. Diagnosing the Issue

To effectively resolve the error, we have to find the root cause of the problem using the AWS CLI and Kubernetes commands.

4.1. Verifying the IAM Role Trust Policy

First, we should check the trust policy of the IAM role we’re trying to assume. The trust policy specifies which entities (users, roles, services, or federated identities) can assume the role.

We can retrieve the trust policy using the following command:

aws iam get-role --role-name <ROLE_NAME> --query 'Role.AssumeRolePolicyDocument' --output json

In the example above, we need to replace <ROLE_NAME> with the name of our IAM role.

We then can review the output to ensure that the Principal element includes the correct federated identity provider and that the Action includes STS.

4.2. Checking the Identity Provider

The IAM role must trust the correct identity provider for the web identity federation. In the context of EKS, this is the OIDC provider associated with our cluster.

We can list all OIDC providers in our AWS account:

aws iam list-open-id-connect-providers

The output will show ARNs of the form arn:aws:iam::<ACCOUNT_ID>/<OIDC_PROVIDER_URL>.

We aim to check that the OIDC provider ARN matches the one specified in the IAM role’s trust policy. Any mismatch will prevent successful role assumption.

4.3. Inspecting the Kubernetes Service Account

In EKS, pod applications assume IAM roles based on their Service Account, which must be annotated with the IAM role ARN.

We can describe the Service Account to check its annotations:

kubectl describe sa <SERVICE_ACCOUNT_NAME> -n <NAMESPACE>

We should look for the annotation eks.amazonaws.com/role-arn. It should contain the correct IAM role ARN. If the annotation is missing or incorrect, pods using this Service Account won’t be able to assume the role.

4.4. Validating the Web Identity Token

If possible, we should validate the web identity token used in the request. We need to ensure that it’s appropriately signed by the identity provider, not expired, and contains the correct audience (aud) claim.

While we can’t directly inspect tokens from AWS services, ensuring that the token source (for example, Service Account tokens in EKS) is correctly configured helps eliminate token-related issues.

5. Resolving the Error

After identifying potential causes, let’s resolve the error step by step.

5.1. Updating the IAM Role Trust Policy

Let’s first focus on updating the IAM Role Trust Policy to ensure that the IAM role’s trust policy allows the correct OIDC provider to assume the role and that the conditions match our application’s identity.

First, let’s find the OIDC provider URL for our EKS cluster:

aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text

The output will be similar to https://oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>.

Next, we can list OIDC providers to find the matching ARN:

aws iam list-open-id-connect-providers --query "OpenIDConnectProviderList[?contains(Arn, '<OIDC_ID>')].Arn" --output text

We need to be sure that the OIDC provider exists in IAM by modifying the trust policy.

We should create or update the trust policy to include the correct Principal and Condition. Let’s see an example based on the trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
        }
      }
    }
  ]
}

We need to replace placeholders with our specific values:

• <ACCOUNT_ID>: Our AWS account ID
• <REGION>: The AWS region of our cluster
• <OIDC_ID>: The unique ID of our OIDC provider
• <NAMESPACE>: The Kubernetes namespace
• <SERVICE_ACCOUNT_NAME>: The name of our Service Account

Once updated, we can update the IAM role with the new trust policy:

aws iam update-assume-role-policy --role-name <ROLE_NAME> --policy-document file://trust-policy.json

This command replaces the existing trust policy with the one we specified.

5.2. Ensuring Correct Permissions

The IAM role or user attempting to assume the role must have permission to call STS. This is particularly important if our application or service runs under a specific IAM identity.

Let’s first check the permissions by listing the inline and attached policies for the IAM role or user:

# For a role
aws iam list-attached-role-policies --role-name <ROLE_NAME>
aws iam list-role-policies --role-name <ROLE_NAME>

# For a user
aws iam list-attached-user-policies --user-name <USER_NAME>
aws iam list-user-policies --user-name <USER_NAME>

If necessary, we can create or update an IAM policy to include the required action. Here’s an example policy.json file that we can upload to our AWS account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
    }
  ]
}

Now, let’s attach the policy to the IAM role or user:

# For a role
aws iam put-role-policy --role-name <ROLE_NAME>
  --policy-name AllowAssumeRoleWithWebIdentity --policy-document file://policy.json

# For a user
aws iam put-user-policy --user-name <USER_NAME>
  --policy-name AllowAssumeRoleWithWebIdentity --policy-document file://policy.json

5.3. Updating the Kubernetes Service Account

As discussed above, we should check that the Service Account in Kubernetes is correctly annotated with the IAM role ARN.

We can use the following command to annotate the Service Account:

kubectl annotate serviceaccount <SERVICE_ACCOUNT_NAME>
  -n <NAMESPACE> eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME> --overwrite

The –overwrite flag verifies that any existing annotation is updated.

We can then confirm that the annotation is in place:

kubectl describe serviceaccount <SERVICE_ACCOUNT_NAME> -n <NAMESPACE>

We need to check for the Annotations section to see the eks.amazonaws.com/role-arn key with the correct value.

5.4. Refreshing Tokens

If tokens are expired or invalid, we may need to refresh them to ensure successful authentication.

In EKS, Service Account tokens are projected into pods at creation time. If we’ve updated the Service Account or IAM role, we need to delete the pods so they can be recreated with the new tokens:

kubectl delete pod <POD_NAME> -n <NAMESPACE>

Alternatively, we can delete all pods using the Service Account:

kubectl delete pods -n <NAMESPACE> -l app=<YOUR_APP_LABEL>

The pods will restart, and the new token and permissions will be applied.

6. Additional Tips

We can consider the following tips to further aid in diagnosing and resolving the error.

6.1. Enable AWS CLI Debugging

To get more detailed error messages and request/response data, we can enable debugging in the AWS CLI:

aws <SERVICE> <OPERATION> --debug

For example:

aws sts assume-role-with-web-identity --role-arn arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>
  --role-session-name test-session --web-identity-token <TOKEN> --debug

This provides a verbose output to help us identify issues like incorrect parameters or permissions.

6.2. Use IAM Policy Simulator

AWS provides an IAM Policy Simulator to test and troubleshoot IAM policies:

• We can access the IAM Policy Simulator in the AWS Management Console
• Let’s select the user or role and input the action STS
• We can simulate to see if the action is allowed or denied and understand which policies affect the decision.

This tool helps us identify which policy statements are blocking or allowing access.

6.3. Verify AWS CLI Configuration

Another approach is to verify the AWS CLI config:

aws configure

We can check the current identity to ensure we’re operating under the expected user or role:

aws sts get-caller-identity

This command returns the AWS account ID, user ID, and ARN associated with the used credentials.

6.4. Review CloudTrail Logs

AWS CloudTrail records all API calls made in our AWS account. Reviewing CloudTrail logs can provide insights into failed AssumeRoleWithWebIdentity attempts:

• Let’s navigate to the CloudTrail console
• Let’s search for events related to AssumeRoleWithWebIdentity
• We can examine the event details to identify issues like incorrect role ARN or insufficient permissions

7. Conclusion

In this article, we’ve tackled the “Unauthorized to perform sts– 403” error in AWS. We’ve learned that this error often stems from misconfigured IAM roles, trust policies, or Kubernetes Service Accounts in EKS environments. We can resolve this error by carefully updating the IAM role’s trust policy to include the correct identity provider and conditions.

Understanding the role of web identity federation and how AWS STS operates is crucial for securely granting temporary credentials to applications and services. Properly configuring IAM roles and policies not only resolves the error but also enhances the security posture of our AWS environment by following the principle of least privilege.

By methodically diagnosing the issue and applying the solutions outlined, we can verify that our applications have the necessary access without compromising security. Regularly reviewing IAM policies and staying informed about AWS best practices will help us prevent similar issues in the future.