Kubernetes HPA: Unable to get metrics for resource memory: no metrics returned from resource metrics API

Question

I'm just starting with Kubernetes, and I'm working with HPA (HorizontalPodAutoscaler):

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: find-complementary-account-info-1
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: find-complementary-account-info-1
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageUtilization: 50

But when I try to see the metrics on the console, it doesn't show me:

$ kubectl get hpa
NAME                                REFERENCE                                      TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
find-complementary-account-info-1   Deployment/find-complementary-account-info-1   <unknown>/50%   2         5         2          143m

My manifest is:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: find-complementary-account-info-1
  labels:
    app: find-complementary-account-info-1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: find-complementary-account-info-1
  template:
    metadata:
      labels:
        app: find-complementary-account-info-1
    spec:
      containers:
      - name: find-complementary-account-info-1
        image: find-complementary-account-info-1:latest
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "350Mi"
          requests:
            memory: "300Mi"
        ports:
        - containerPort: 8083
        env:
        - name: URL_CONNECTION_BD
          value: jdbc:oracle:thin:@10.161.6.15:1531/DEFAULTSRV.WORLD
        - name: USERNAME_CONNECTION_BD
          valueFrom:
            secretKeyRef:
              name: credentials-bd-pers
              key: user_pers
        - name: PASSWORD_CONNECTION_BD
          valueFrom:
            secretKeyRef:
              name: credentials-bd-pers
              key: password_pers
apiVersion: v1
kind: Service
metadata:
  name: svc-find-complementary-account-info-1
  labels:
    app: find-complementary-account-info-1
  namespace: default
spec:
  selector:
    app: find-complementary-account-info-1
  type: LoadBalancer
  ports:
   -
    protocol: TCP
    port: 8083
    targetPort: 8083
    nodePort: 30025
  externalIPs:
    - 10.161.174.68

Show me the <unknown> tag

Also, if I perform a kubectl describe hpa, it throws me the following:

$ kubectl describe hpa find-complementary-account-info-1
Name:                                                     find-complementary-account-info-1
Namespace:                                                default
Labels:                                                   <none>
Annotations:                                              <none>
CreationTimestamp:                                        Thu, 29 Oct 2020 13:57:58 -0400
Reference:                                                Deployment/find-complementary-account-info-1
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                             2
Max replicas:                                             5
Deployment pods:                                          2 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource memory: no metrics returned from resource metrics API
Events:
  Type     Reason                   Age                     From                       Message
  ----     ------                   ----                    ----                       -------
  Warning  FailedGetResourceMetric  4m49s (x551 over 144m)  horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics API

I am not working in the cloud, I am working in an on-premise environment configured with Bare-Metal

metrics-server is installed:

$ kubectl get deployment metrics-server -n kube-system
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           177m

What's missing? Could someone give me a hand?

$ kubectl describe pod metrics-server -n kube-system
Name:         metrics-server-5f4b6b9889-6pbv8
Namespace:    kube-system
Priority:     0
Node:         tmp224/10.164.21.169
Start Time:   Thu, 29 Oct 2020 13:27:19 -0400
Labels:       k8s-app=metrics-server
              pod-template-hash=5f4b6b9889
Annotations:  cni.projectcalico.org/podIP: 10.244.119.140/32
              cni.projectcalico.org/podIPs: 10.244.119.140/32
Status:       Running
IP:           10.244.119.140
IPs:
  IP:           10.244.119.140
Controlled By:  ReplicaSet/metrics-server-5f4b6b9889
Containers:
  metrics-server:
    Container ID:  docker://f71d26dc2c8e787ae9551faad66f9588a950bf0a6d0d5cb90ff11ceb219e9b37
    Image:         k8s.gcr.io/metrics-server-amd64:v0.3.6
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
    Port:          4443/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
    State:          Running
      Started:      Thu, 29 Oct 2020 13:27:51 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-4mn92 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  metrics-server-token-4mn92:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-4mn92
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/arch=amd64
                 kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

Disquieting

Is your metrics-server running correctly? Is it returning any values when you execute, for example: kubectl top pod? – Hypothermia

$ kubectl top nodes error: metrics not available yet – Disquieting

It is not returning anything. How can I configure a metrics server correctly? – Disquieting

Could you share output of kubectl describe pod <metrics-server-pod-name> -n kube-system? – Hypothermia

Reday. Add output the comando kubectl describe pod metrics-server -n kube-system – Disquieting

For I had't installed the MetricServer. Following this doc helped me out. How do I set up Metrics Server and Horizontal Pod Autoscaler on an Amazon EKS cluster? – Lilongwe

Answer 1

The error unable to get metrics for resource memory: no metrics returned from resource metrics API is not caused by faulty HPA but by metrics-server that is not able to scrape any metrics.

Most of the time this error is caused by missing commands in metrics-server deployment. It can be done by adding following arguments in the deployment.

  --kubelet-insecure-tls
  --kubelet-preferred-address-types=InternalIP,ExternalIP

Hypothermia

It is so, this was missing. Thank you so much – Disquieting

Answer 2

Since you are using apiVersion: autoscaling/v2beta2 , the correct way to specify scaling policy based on resources would be,

type: Resource
resource:
  name: cpu
  target:
    type: Utilization
    averageUtilization: 60

You can also specify resource metrics in terms of direct values, instead of as percentages of the requested value, by using a target.type of AverageValue instead of Utilization, and setting the corresponding target.averageValue field instead of the target.averageUtilization

And if you want to try out with v2beta1, you can try the below piece of yml in metrics,

 type: Resource
 resource:
    name: memory
    targetAverageUtilization: 60

Nauseate

Answer 3

The problem is at:

target: type: AverageValue averageUtilization: 50

You specified the type "AverageValue" but with "averageUtilization". You should either change type: to "Utilization" or value field with "averageValue".

Riparian

Not sure why was this downvoted. To me it was the actual solution. RKE UI sets averageUtilization and renders it into failure. Changing it manually to averageValue fixed it. – Wyattwyche