Error from server (NotFound): podmetrics.metrics.k8s.io "mem-example/memory-demo" not found

Question

I am following this tutorial: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/

I have created the memory pod demo and I am trying to get the metrics from the pod but it is not working.

I installed the metrics server by cloning: https://github.com/kubernetes-incubator/metrics-server

And then running this command from top level:

kubectl create -f deploy/1.8+/

I am using kubernetes version 1.10.11.

The pod is definitely created:

λ kubectl get pod memory-demo --namespace=mem-example
NAME          READY     STATUS    RESTARTS   AGE
memory-demo   1/1       Running   0          6m

But the metics command does not work and gives an error:

λ kubectl top pod memory-demo --namespace=mem-example
Error from server (NotFound): podmetrics.metrics.k8s.io "mem-example/memory-demo" not found

What did I do wrong?

Idaliaidalina

Answer 1

There are some patches to be done to metrics server deployment to get the metrics working.

Follow the below steps

kubectl delete -f deploy/1.8+/

wait till the metrics server gets undeployed

run the below command
kubectl create -f https://raw.githubusercontent.com/epasham/docker-repo/master/k8s/metrics-server.yaml

master $ kubectl get po -n kube-system
NAME                              READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-6zg78          1/1       Running   0          2h
coredns-78fcdf6894-gk4sb          1/1       Running   0          2h
etcd-master                       1/1       Running   0          2h
kube-apiserver-master             1/1       Running   0          2h
kube-controller-manager-master    1/1       Running   0          2h
kube-proxy-f5z9p                  1/1       Running   0          2h
kube-proxy-ghbvn                  1/1       Running   0          2h
kube-scheduler-master             1/1       Running   0          2h
metrics-server-85c54d44c8-rmvxh   2/2       Running   0          1m
weave-net-4j7cl                   2/2       Running   1          2h
weave-net-82fzn                   2/2       Running   1          2h

master $ kubectl top pod -n kube-system
NAME                              CPU(cores)   MEMORY(bytes)
coredns-78fcdf6894-6zg78          2m           11Mi
coredns-78fcdf6894-gk4sb          2m           9Mi
etcd-master                       14m          90Mi
kube-apiserver-master             24m          425Mi
kube-controller-manager-master    26m          62Mi
kube-proxy-f5z9p                  2m           19Mi
kube-proxy-ghbvn                  3m           17Mi
kube-scheduler-master             8m           14Mi
metrics-server-85c54d44c8-rmvxh   1m           19Mi
weave-net-4j7cl                   2m           59Mi
weave-net-82fzn                   1m           60Mi

Check and verify the below lines in metrics server deployment manifest.

        command:
        - /metrics-server
        - --metric-resolution=30s
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-insecure-tls

Auriga

That doesn't look like an official repo. I am wary of installing random code. Do you have a reference to the patches you are talking about? – Idaliaidalina

It was taken from official report and patches were added on top of that – Auriga

updated the changes that you need to apply to metrics server. check my answer above – Auriga

Answer 2

On Minikube, I had to wait for 20-25 minutes after enabling the metrics-server addon. I was getting the same error for 20-25 minutes but later I could see the output without attempting for any solution.

Gerontocracy

Me too. It's even in the documentation: kubectl top pod --help says: Due to the metrics pipeline delay, they may be unavailable for a few minutes since pod creation. – Twigg

Answer 3

I faced the similar issue of

Error from server (NotFound): podmetrics.metrics.k8s.io "default/apple-app" not found

I followed two steps and I was able to resolve the issue.

1. Download the latest customized components.yaml, which is their official file used for easy deployment.

2. Update the change

          # - /metrics-server
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP

to the command section of the deployment specification. I have commented the first line because it is the entrypoint of the image used by kubernetes metrics-server.

$ docker image inspect k8s.gcr.io/metrics-server-amd64:v0.3.6 -f {{.ContainerConfig.Entrypoint}}
[/metrics-server]

Even If you use it or not, it doesn't matter.

Note: You have to wait for few seconds for it to properly work.

After this running the top command will work for you.

$ kubectl top pod apple-app
NAME        CPU(cores)   MEMORY(bytes)   
apple-app   1m           3Mi             

Pleochroism

Answer 4

I know this is an old thread may be someone will find this answer useful.

1. You have to checkout the following repo:

   https://github.com/kubernetes-incubator/metrics-server

   Go to the root of the repo and checkout release-0.3.2.

2. Remove default metrics server by:

   kubectl delete -f deploy/1.8+/
   

3. Download the container yaml

   wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
   

4. Edit the container.yaml by adding the following lines to the argument section. You will see these two lines there

   args:
         - --kubelet-preferred-address-types=InternalIP
         - --kubelet-insecure-tls=true
   

   There is only one args parameter in that file.

Deploy your pod/deployment and you should be able to do:

kubectl top pod <pod-name>

Tremaine