How to retrieve kubernetes metrics via client-go and golang

Question

I want to access metrics from kubernetes with golang. Something like cpu and memory per node as well as the same for pods and/or namespaces.

I am kind of lost here because the documentation is not as clear as it could be.

I have learned that there is heapster (which is deprecated according to the github repo). There is also metric server and a rest api.

Where can I find some examples to get started? I do not want to install another app, package or service in kubernetes. I'd like to get the information as native as possible. What is the preferred way to access these information with client-go and golang?

Genitals

Something like: #49193008 ? – Mons

Answer 1

There's a much better API for this: https://github.com/kubernetes/metrics. Using this, you don't have to create the data structs or handle row byte slices.

import (
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  metricsv "k8s.io/metrics/pkg/client/clientset/versioned"
  ...
)

...
clientset, err := metricsv.NewForConfig(config)
podMetricsList, err := clientset.MetricsV1beta1().PodMetricses("").List(metav1.ListOptions{})

Apology

Now, .List() takes a context.Context argument: clientset.MetricsV1beta1().PodMetricses("").List(context.TODO(), metav1.ListOptions{}) – Coitus

Answer 2

Here's an example of using the REST API to query node metrics and return a []byte in JSON format. Replace "nodes" with "pods" to get pod/container metrics.

data, err := clientset.RESTClient().Get().AbsPath("apis/metrics.k8s.io/v1beta1/nodes").DoRaw()

Sukiyaki

Answer 3

As explained in the question, the documentations are not clear for a beginner. Even go-client examples retrieve the data, I wanted to get Type support.

As it explained by above answer, you can get the data in []byte in JSON format. This is how I did it.

package main

import (
    "encoding/json"
    "fmt"
    "time"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
)

// PodMetricsList : PodMetricsList
type PodMetricsList struct {
    Kind       string `json:"kind"`
    APIVersion string `json:"apiVersion"`
    Metadata   struct {
        SelfLink string `json:"selfLink"`
    } `json:"metadata"`
    Items []struct {
        Metadata struct {
            Name              string    `json:"name"`
            Namespace         string    `json:"namespace"`
            SelfLink          string    `json:"selfLink"`
            CreationTimestamp time.Time `json:"creationTimestamp"`
        } `json:"metadata"`
        Timestamp  time.Time `json:"timestamp"`
        Window     string    `json:"window"`
        Containers []struct {
            Name  string `json:"name"`
            Usage struct {
                CPU    string `json:"cpu"`
                Memory string `json:"memory"`
            } `json:"usage"`
        } `json:"containers"`
    } `json:"items"`
}

func getMetrics(clientset *kubernetes.Clientset, pods *PodMetricsList) error {
    data, err := clientset.RESTClient().Get().AbsPath("apis/metrics.k8s.io/v1beta1/pods").DoRaw()
    if err != nil {
        return err
    }
    err = json.Unmarshal(data, &pods)
    return err
}

func main() {
    // creates the in-cluster config
    // https://github.com/kubernetes/client-go/tree/master/examples#configuration
    config, err := rest.InClusterConfig()
    if err != nil {
        panic(err.Error())
    }
    // creates the clientset
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err.Error())
    }
    var pods PodMetricsList
    err = getMetrics(clientset, &pods)
    if err != nil {
        panic(err.Error())
    }
    for _, m := range pods.Items {
        fmt.Println(m.Metadata.Name, m.Metadata.Namespace, m.Timestamp.String())
    }
}

Install following Go packages: go get -u k8s.io/client-go/kubernetes k8s.io/client-go/rest

You can use following endpoints to retrieve the data as you want;

• Nodes: apis/metrics.k8s.io/v1beta1/nodes
• Pods: apis/metrics.k8s.io/v1beta1/pods
• Pods of default namespace: apis/metrics.k8s.io/v1beta1/namespaces/default/pods
• Specific Pod: /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/<POD-NAME>

NOTE: You may need to change the Type before json.Unmarshal. You can define the Type only for the field which you are interested with.

Inion

@mR.aTA can you provide more details? Also make sure your client has permission to access the metrics. – Inion

This is the link to the dump of the config Also, the command kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods works well. How can I make sure that my client has enough permission? It panics at data, err := clientset.RESTClient().Get().AbsPath("apis/metrics.k8s.io/v1beta1/pods").DoRaw() and the err is unknown I also have these two in my kubelet confi: authenticationTokenWebhook: true and authorizationMode: Webhook. – Suspensive

I've updated my authenticatioinMode:AlwaysAllow so the example here to get the list of nodes works fine, but the code for getting the metrics returns the unknown error. – Suspensive

Make sure you have installed metrics-server. Create a cluster role for default name space with kubectl create clusterrolebinding default-cluster-admin --clusterrole=cluster-admin --serviceaccount=default:default – Inion

Thanks @gihanchanuka! However, I didn't use cluster-admin role, but I wrote a rbca clusterrol/clusterrolbinding with custom permissions and it works now. Thanks again. – Suspensive

Answer 4

From the latest version of client-go you have to add a Context to DoRaw(). For example use context.TODO() importing the context library.

Rasure