K8s Pods Failure : error while running "VolumeBinding" prebind plugin for pod "app": Failed to bind volumes: timed out waiting for the condition

Question

I am trying to use Persistent Volumes in AWS EKS.

I followed all the steps mentioned from the below link

https://aws.amazon.com/premiumsupport/knowledge-center/eks-persistent-storage/

When I deployed sample pods to check PVCs are created fine, I am getting below error.

Warning  FailedScheduling  2m6s (x265 over 44h)  default-scheduler  error while running "VolumeBinding" prebind plugin for pod "app": Failed to bind volumes: timed out waiting for the condition

Checked further persistent volume claims

kubectl describe pvc
  Warning  ProvisioningFailed    62s (x7 over 3m5s)    ebs.csi.aws.com_ebs-csi-controller-7cb88dcffc-vkvdk_2ae45c3c-b4f2-4200-a5d4-cfdb35a5f0db  failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded

From looking at the logs of ebs csi controller

I0721 10:44:10.049138       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"avengers", Name:"ebs-claim", UID:"88a9684c-1be7-412f-8616-b662eff692df", APIVersion:"v1", ResourceVersion:"19320908", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = Internal desc = WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-2.amazonaws.com/id/XXXXXXXXXXXX34FDCBXXXXX
        status code: 400, request id: 90b430ba-6999-41d8-ac15-13f5ec73def2
I0721 10:44:45.207103       1 controller.go:1332] provision "default/ebs-claim" class "ebs-sc": started
I0721 10:44:45.207406       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3922ab97-483a-486a-bbbd-57875e506021", APIVersion:"v1", ResourceVersion:"19875393", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-claim"
I0721 10:44:55.207594       1 controller.go:1106] Temporary error received, adding PVC 3922ab97-483a-486a-bbbd-57875e51 to claims in progress
W0721 10:44:55.207639       1 controller.go:958] Retrying syncing claim "3922ab97-483a-486a-bbbd-57875e506021", failure 9
E0721 10:44:55.207664       1 controller.go:981] error syncing claim "3922ab97-483a-486a-bbbd-57875e506021": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0721 10:44:55.207859       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3922ab97-483a-486a-bbbd-57875e506021", APIVersion:"v1", ResourceVersion:"19875393", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context

What did I miss here ?

Any help will be appreciated.

Gelation

Answer 1

From the error log, pick the id for search and i.e. XXXXXXXXXXXX34FDCBXXXXX

caused by: InvalidIdentityToken: No OpenIDConnect provider 
found in your account for 
https://oidc.eks.us-east-2.amazonaws.com/id
/XXXXXXXXXXXX34FDCBXXXXX
status code: 400, request id: 90b430ba-6999-41d8-ac15-13f5ec73def2

Search in OpenID connect providers list

IAM OIDC (OpenID Connect) Identity provider didn’t exist for current EKS Cluster.

aws iam list-open-id-connect-providers | grep XXXXXXXXXXXX34FDCBXXXXX

returned nothing.

Enable OIDC IAM

Enabled OIDC IAM in existing EKS Cluster:

$ eksctl utils associate-iam-oidc-provider --cluster=eks-lab --region us-east-2 --approve

Deleted everything and deployed again and now it works fine.

References.

Gelation

Answer 2

For me, I had to install the AWS EBS CSI driver/add on to make Volume Claims work. don't know why it's not the default on EKS

Mechanist

Answer 3

Prometheus-server pod remains in pending state, you can inspect the pvc which also happens to be in "Waiting" mode. Need to install, Amazon EBS CSI driver as EKS add-on https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html. (Requires IAM role with required permissions before installing add on)

This step is needed in order to access EBS volumes by EKS. It's a bummer that its not natively provided by EKS

Arching