How to configure prometheus kubernetes_sd_configs to specify a specific host port?

Question

I am running node exporter on GKE on a different port (11100) and configuring prometheus.yml to use kubernetes_sd_configs. However, the service discovery seems to be returning the node IP with Kubelet port (10250) <node-ip>:10250/metrics. I can’t seem to find a way to specify which port to use. Any ideas?

 - job_name: gke-nodes
      kubernetes_sd_configs:
      - role: node

Also, node-exporter is running correctly in the port 11100. I validated it by executing curl in the internal node IP <node-ip>:11100/metrics and it works like a charm

---

Here is my Node exporter definition

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter-ds
  namespace: monitoring
  labels:
    app: node-exporter
    belongsTo: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      serviceAccountName: monitoring-sa
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
      containers:
        - name: node-exporter
          image: prom/node-exporter:v0.18.1
          args:
            - "--web.listen-address=0.0.0.0:11100"
            - "--path.procfs=/proc_host"
            - "--path.sysfs=/host_sys"
          ports:
            - containerPort: 11100
              hostPort: 11100
          volumeMounts:
          - name: sys
            readOnly: true
            mountPath: /host_sys
          - name: proc
            readOnly: true
            mountPath: /proc_host
          imagePullPolicy: IfNotPresent
      hostNetwork: true
      hostPID: true

Grille

How did you setup the prometheus/node exporter? – Nightgown

I am going to update it in the question so I can show you my node exporter definition – Grille

Could you please share the error messages and logs from prometheus/node exporter? – Nightgown

Answer 1

Just in case someone is facing the same issue, this configuration will add the k8s nodes IPs with a different port than Kubelet:

  - job_name: 'gke-nodes'
    kubernetes_sd_configs: 
    - role: node
    relabel_configs:
      - source_labels: [__address__]
        action: replace
        regex: ([^:]+):.*
        replacement: $1:11100 # port you want to use
        target_label: __address__

The default behavior of the kubernetes_sd_config is to add as targets the IPs of the nodes with the port of Kubelet, this is specified in the documentation:

The node role discovers one target per cluster node with the address defaulting to the Kubelet's HTTP port

However, you can use relabel_config to override the __address__ label and replace the Kubelet port with any port that need.

Humidity