kubectl top nodes shows error: metrics not available yet

Question

I'm using Kubernetes 1.7 and running hepaster. When I run

kubectl top nodes --heapster-namespace=kube-system

it shows me

error: metrics not available yet

I also tried this

kubectl top nodes --heapster-namespace=kube-system --heapster-service=heapster --heapster-scheme=http --heapster-port=12213

Where the heapster is running then it shows following error.

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:12213)

Any clue for tackling the error?

Marconigraph

It's working now. The heapster was not configured properly. – Marconigraph

I have the same problem too. How did you solve it? – Shive

Is your heapster running properly? Are you able to access it via terminal command curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/? – Marconigraph

When I run that command it returns this results: { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "model \"metrics\" is forbidden: User \"system:anonymous\" cannot get model at the cluster scope", "reason": "Forbidden", "details": { "name": "metrics", "kind": "model" }, "code": 403 } – Shive

Do you see any error in the log of heapster container? – Marconigraph

Answer 1

It means that heapster is not properly configured.

You need to make sure that heapster is running on kube-system namespace, and check if the /healthz endpoint is ok:

$ export HEAPSTER_POD=$(kubectl get po -l k8s-app=heapster -n kube-system -o jsonpath='{.items[*].metadata.name}')
$ export HEAPSTER_SERVICE=$(kubectl get service/heapster --namespace=kube-system -o jsonpath="{.spec.clusterIP}")
$ curl -L "http://${HEAPSTER_SERVICE}/healthz"
ok

Then, you can check if the metrics API is available:

$ curl -L "http://${HEAPSTER_SERVICE}/api/v1/model/metrics/"
[
  "cpu/usage_rate",
  "memory/usage",
  "cpu/request",
  "cpu/limit",
  "memory/request",
  "memory/limit"
 ]

If it's not returning as above, take a look at container logs for errors:

$ kubectl logs -n kube-system ${HEAPSTER_POD} --all-containers

---

Although, keep in mind that Heapster is a deprecated project and you may have problems when running it in recent Kubernetes versions.

See Heapster Deprecation Timeline:

| Kubernetes Release  | Action              | Policy/Support                                                                   |
|---------------------|---------------------|----------------------------------------------------------------------------------|
| Kubernetes 1.11     | Initial Deprecation | No new features or sinks are added.  Bugfixes may be made.                       |
| Kubernetes 1.12     | Setup Removal       | The optional to install Heapster via the Kubernetes setup script is removed.     |
| Kubernetes 1.13     | Removal             | No new bugfixes will be made.  Move to kubernetes-retired organization.          |

Since Kubernetes v1.10, the kubectl top relies on metrics-server by default.

CHANGELOG-1.10.md:

• Support metrics API in kubectl top commands. (#56206, @brancz)

This PR implements support for the kubectl top commands to use the metrics-server as an aggregated API, instead of requesting the metrics from heapster directly. If the metrics.k8s.io API is not served by the apiserver, then this still falls back to the previous behavior.

It's better to use a kubectl version v1.10 or above, as it fetches the metrics from metrics-server.

However, beware of kubectl Version Skew Policy:

kubectl is supported within one minor version (older or newer) of kube-apiserver

Check your kube-apiserver version before choosing your kubectl version.

Croaky