Logs of pods missing from /var/logs/ in Kubernetes

Question

I have a cluster that has numerous services running as pods from which I want to pull logs with fluentd. All services show logs when doing kubectl logs service. However, some logs don't show up in those folders:

• /var/log
• /var/log/containers
• /var/log/pods

although the other containers are there. The containers that ARE there are created as a Cronjob, or as a Helm chart, like a MongoDB installation.

The containers that aren't logging are created by me with a Deployment file like so:

kind: Deployment
metadata:
    namespace: {{.Values.global.namespace | quote}}
    name: {{.Values.serviceName}}-deployment
spec:
    replicas: {{.Values.replicaCount}}
    selector:
        matchLabels:
            app: {{.Values.serviceName}}
    template:
        metadata:
            labels:
                app: {{.Values.serviceName}}
            annotations:
                releaseTime: {{ dateInZone "2006-01-02 15:04:05Z" (now) "UTC"| quote }}
        spec:
            containers:
                  - name: {{.Values.serviceName}}
                    # local: use skaffold, dev: use passed tag, test: use released version
                    image: {{ .Values.image }}
                        {{- if (eq .Values.global.env "dev") }}:{{ .Values.imageConfig.tag}}{{ end }}
                    imagePullPolicy: {{ .Values.global.imagePullPolicy }}
                    envFrom:
                        - configMapRef:
                            name: {{.Values.serviceName}}-config
                    {{- if .Values.resources }}
                    resources:
                        {{- if .Values.resources.requests }}
                        requests:
                            memory: {{.Values.resources.requests.memory}}
                            cpu: {{.Values.resources.requests.cpu}}
                        {{- end }}
                        {{- if .Values.resources.limits }}
                        limits:
                            memory: {{.Values.resources.limits.memory}}
                            cpu: {{.Values.resources.limits.cpu}}
                        {{- end }}
                    {{- end }}
            imagePullSecrets:
                - name: {{ .Values.global.imagePullSecret }}  
            restartPolicy: {{ .Values.global.restartPolicy }}
{{- end }}

and a Dockerfile CMD like so: CMD ["node", "./bin/www"]

One assumption might be that the CMD doesn't pipe to STDOUT, but why would the logs show up in kubectl logs then?

Yabber

Answer 1

This is how I would proceed to find out where a container is logging:

1. Identify the node on which the Pod is running with:

   kubectl get pod pod-name -owide
   

2. SSH on that node, you can check which logging driver is being used by the node with:

   docker info | grep -i logging
   

   if the output is json-file, then the logs are being written to file as expected. If there is something different, then it may depends on what the driver do (there are many drivers, they could write to journald for example, or other options)

3. If the logging driver writes to file, you can check the current output for a specific Pod by knowing the container id of that Pod, to do so, on a control-plane node:

   kubectl get pod pod-name -ojsonpath='{.status.containerStatuses[0].containerID}'
   

   (if there are more containers in the same pod, the index to use may vary, depending on which container you want to inspect)

4. With the id extracted, which will be something like docker://f834508490bd2b248a2bbc1efc4c395d0b8086aac4b6ff03b3cc8fd16d10ce2c, you can inspect the container with docker, on the node on which the container is running. Just remove the docker:// part from the id, SSH again on the node you identified before, then do a:

   docker inspect container-id | grep -i logpath
   

Which should output where the container is actively writing its logs to file.

---

In my case, the particular container I tried this procedure on, is currently logging into:

/var/lib/docker/containers/289271086d977dc4e2e0b80cc28a7a6aca32c888b7ea5e1b5f24b28f7601ff63/289271086d977dc4e2e0b80cc28a7a6aca32c888b7ea5e1b5f24b28f7601ff63-json.log

Evelunn