eks iam roles for services account not working

Question

I'm trying my hand on iam roles for services account to secure the autoscaller. But I seem to be missing something. Little precision I'm using terraform to create the cluster.

I followed these documentation:

• https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
• https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

So I've created a role other than the one for the nodes and applied the policy for the autoscaller to this new role. This part is basic, no issue there.

I also activated the openid provider in terraform:

resource "aws_iam_openid_connect_provider" "example" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.eks.identity.0.oidc.0.issuer
}

No issue the cluster is creating itself with no issue.

No I added the annotation to service account for the autoscalling:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ID:role/terraform-eks-autoscaller
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
  name: cluster-autoscaler
  namespace: kube-system

My problem is that it does not seems to works and the pod is still trying to use the new IAM role but still using the node role:

Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: arn:aws:sts::ID:assumed-role/terraform-eks-node/i-ID is not authorized to perform: autoscaling:DescribeTags

Does someone know what step I'm missing here?

Thanks in advance for the help ;)

Demi

Answer 1

So answer is very simple. Your OIDC provider configuration is missing the thumbprint. It is essential for Iam to work correctly. Normally if you create OIDC provider in AWS console that thumbprint gets populated automatically, however it is not the case when you do it through terraform.

I have been caught by this as well so I have written a blog about this that you can find here: https://medium.com/@marcincuber/amazon-eks-with-oidc-provider-iam-roles-for-kubernetes-services-accounts-59015d15cb0c

To solve your issue simply add the following:

9E99A48A9960B14926BB7F3B02E22DA2B0AB7280

The above is the hashed root CA that doesn’t change for another 10+ years and it is the same across all regions. How to acquire it, you can read the blog I added link to above.

Additionally, ensure to use the latest autoscaler version which is matching the version of your kubernetes. Also, try adding security context with fsGroup: 65534. That is the current workaround to make the OIDC work properly for some apps.

Demetricedemetris

I tried to add your thumbprint but I still have the same error :) I have the impression that it's something from the cluster autoscaller... – Demi

Yes I figured that one, I got the last version from the AWS doc but it's not up to date... Now I have the error of OIDC but It says that it doesn't match the thumbprint... I will do some more testing – Demi

In that case, generate one following my notes in the blog that should lead you to fully working solution. – Demetricedemetris

Ok, I got it, I copied the OICD from the blog and that one doesn't work, the one you posted here is ok though – Demi

The one in the blog is an example. I shall probably update it with the correct one. Thanks for confirming that everything works as expected. – Demetricedemetris

@Demetricedemetris I followed all the steps and created the oidc provider manually which had thumbprint too. It fals stating "Failed to create AWS Manager: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials". I followed each step for deploying. I am stilll facing the issue. It could be great if you can help me out. – Desist

Answer 2

i use terraform to create the cluster too, i checked i have the thumbprint,

but i for this err:

Failed to regenerate ASG cache: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid

i resolve it by re-enter the thumbprint manually (upper case), it works, i find that the terraform module default value is lower case, not sure is it the issue

Lebel