"services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope"

Question

I've been following the walkthrough to create an AWS ALB Ingress Controller for my app which is also deployed at an EKS cluster.
Everything seems okay , similar answers with the walkthrough but when it comes to the setting up of an external DNS I get the error :

kubectl logs -f $(kubectl get po | egrep -o 'external-dns[A-Za-z0-9-]+')

time="2020-02-20T16:21:57Z" level=error msg="services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope" time="2020-02-20T16:22:58Z" level=error msg="services is forbidden: User \"system:serviceaccount:tick:external-dns\" cannot list resource \"services\" in API group \"\" at the cluster scope"

every one minute . I made sure that all the permissions are the needed ones so it should not be because of that.

I tried the solutions from here , but nothing helped and I couldn't find any other solutions.

What does this error practically means? What should I do to fix it?

UPDATE EDIT my external-dns configuration looks like:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::*my*account*id*:role/EKSRole
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: tick
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  selector:
    matchLabels:
      app: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::*my*account*id*:role/EKSRole
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.9
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=external-dns-test.my-org.com   #external-dns-test.my-org.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=my-identifier
      securityContext:
        fsGroup: 65534

Meltage

Answer 1

Your error suggests that service account with name external-dns in tick namespace can't perform certain actions. In this case it is list services. To solve this you can apply the following:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: tick
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns-role
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns-role
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: tick

Note, that first rule inside ClusterRole is granting correct permissions to list services in "" apiGroup, which is solving the error you have reported in your question.

Uni

Thanks , the only thing that was different was the field namespace which was default and changed it to tick . Now I need to figure out how to access my app through the service – Meltage

@FloraBiletsiou I just updated my answer. In you ServiceAccount definition, you are also missing a namespace. If you don't specify it, the service account usually gets created in kube-system or default namespace. – Uni

the ServiceAccount was already at my tick namespace because I set it to be the default one, but I changed anyway – Meltage

I am currently struggling the same issue, or at least the same error, despite the command: kubectl auth can-i list service/poc-service --as system:serviceaccount:poc:poc-sa returns yes. I tried assigning cluster-admin to poc-sa (with kubectl create clusterrolebinding poc-admin --clusterrole cluster-admin --serviceaccount=poc:poc-sa) and that seems to get rid of the error. But that's a really bad solution, given the security risk involved. – Heldentenor

Answer 2

I think you are using Amazon EKS with IAM Roles for Service Accounts therefore following this should work. The walkthrough you were following, in the external DNS section Step 2 yaml file doesn't have your

annotations:
    # Substitute your account ID and IAM service role name below.
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT-ID:role/IAM-SERVICE-ROLE-NAME

This error practically means your serviceaccount (external-dns) doesn't have permissions to access the kubernetes api server

Tonga

Thank you for your answer, I actually added this part to my external-dns ServiceAccount configuration ,but since I am a beginner at AWS I am not sure if I have the correct IAM-SERVICE-ROLE-NAME. my configuration looks like : annotations: eks.amazonaws.com/role-arn:arn:aws:iam::**myaccountid**:role/EKSRole I am not sure how to find the IAM-SERVICE-ROLE-NAME – Meltage

Answer 3

For those who reached this question due to the "at the cluster scope" postfix at the error output.

If you're:

1. Using a namespaced role.
2. Working directly with a kubernetes client library (or with the rest API) and executing actions like (but not limited just to .Pods) :

clientSet.CoreV1().Pods("namespace").List/Patch/Etc

Then if you're not passing a namespace (keeping it with empty string like some examples suggests) then you will see the same error because the role is namespaced but the code is trying the execute a cluster level operation.

Microorganism