How to get the resource usage of a pod in Kubernetes?

Question

How can we get the real resource usage (not resource requests) of each pod on Kubernetes by command line? Heapster is deprecated. Meanwhile, Metrics-server still does not support kubectl top pod.

1. Heapster -

   I deployed Heapster using the following command

   $ heapster/deploy/kube.sh start
   kubectl get pods --all-namespaces
   NAMESPACE     NAME                                                           READY     STATUS    RESTARTS   AGE
   kube-system   calico-node-hlcbl                                              2/2       Running   0          39m
   kube-system   calico-node-m8jl2                                              2/2       Running   0          35m
   kube-system   coredns-78fcdf6894-bl94w                                       1/1       Running   0          39m
   kube-system   coredns-78fcdf6894-fwx95                                       1/1       Running   0          39m
   kube-system   etcd-ctl.kube.yarnrm-pg0.utah.cloudlab.us                      1/1       Running   0          39m
   kube-system   heapster-84c9bc48c4-qzt8x                                      1/1       Running   0          15s
   kube-system   kube-apiserver-ctl.kube.yarnrm-pg0.utah.cloudlab.us            1/1       Running   0          39m
   kube-system   kube-controller-manager-ctl.kube.yarnrm-pg0.utah.cloudlab.us   1/1       Running   0          38m
   kube-system   kube-proxy-nj9f8                                               1/1       Running   0          35m
   kube-system   kube-proxy-zvr2b                                               1/1       Running   0          39m
   kube-system   kube-scheduler-ctl.kube.yarnrm-pg0.utah.cloudlab.us            1/1       Running   0          39m
   kube-system   monitoring-grafana-555545f477-jldmz                            1/1       Running   0          15s
   kube-system   monitoring-influxdb-848b9b66f6-k2k4f                           1/1       Running   0          15s
   

   When I used kubectl top, I encountered the following errors.

   $ kubectl top pods
   Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)
   $ kubectl top nodes
   Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)
   

2. metrics-server:

   metrics-server has not supported kubectl top Resource Metrics API

If anyone already solved the same problem, please help me. Thanks.

Specialistic

Try about docker stats, this is useful for a single node. – Fatherless

Answer 1

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

It sounds like the heapster deployment just forgot to install the Service for heapster; I would expect this would get you past that error, but unknown whether it would actually cause kubectl top pods to start to work:

kubectl create -f /dev/stdin <<SVC
apiVersion: v1
kind: Service
metadata:
  name: heapster
  namespace: kube-system
spec:
  selector:
    whatever-label: is-on-heapster-pods
  ports:
  - name: http
    port: 80
    targetPort: whatever-is-heapster-is-listening-on
SVC

Salomone

I have the same question. When I ran the command, It would show The Service "heapster" is invalid: spec.ports[0].targetPort: Invalid value: "whatever-is-heapster-is-listening-on": must be no more than 15 characters – Pauperism

heh, in that case, I'm glad I picked a completely illegal example value for my example; I don't know what the label is on your heapster Pods, nor do I know what port those Pods are listening on. I guess I should have just omitted values entirely, and left comments instead – Salomone

Thanks @MatthewLDaniel. I did not encounter that problem anymore but got another one with latest Kuberntes and latest Heapster. kubectl top pod --namespace=kube-system W0813 13:48:08.426536 27831 top_pod.go:263] Metrics not available for pod kube-system/calico-node-8d5d4, age: 38m1.426522624s error: Metrics not available for pod kube-system/calico-node-8d5d4, age: 38m1.426522624s. In this case, is "calico" causing the issue? if so, which one can be used instead of calico? – Specialistic

In this case, is "calico" causing the issue? No, I have strong reason to believe that since heapster is deprecated, that it is not collecting those metrics anymore. You can, of course, poke around in your influx to find out if it has any actual data, and just isn't being surfaced by kubectl top or whether -- as they have promised -- they are moving to metrics-server and don't care about whether heapster works or not. – Salomone

I also believe that Kubernetes is going to use metrics server that causes "metrics are not available". Thanks Matthew. – Specialistic