Kubernetes: Redirecting pod STDOUT, STDERR and pod termination log to a PVC file

Question

I am trying to get STDOUT and STDERR of a pod written to the file at PVC mount location.

Following is the template content of my deployment:

    "template": {
      "metadata": {
        "name": "python-stdout-app",
        "creationTimestamp": null,
        "labels": {
          "k8s-app": "python-stdout-app"
        }
      },
      "spec": {
        "volumes": [
          {
            "name": "task-pv-volume",
            "persistentVolumeClaim": {
              "claimName": "task-pv-claim"
            }
          }
        ],
        "containers": [
          {
            "name": "python-stdout-app",
            "image": "trideep/demo-stdout-app",
            "resources": {},
            "volumeMounts": [
              {
                "name": "task-pv-volume",
                "mountPath": "/usr/share"
              }
            ],
            "terminationMessagePath": "/usr/share/1fed8c03-bc30-4889-952e-46f4c19b6ac1.log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always",
            "securityContext": {
              "privileged": false
            }
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "schedulerName": "default-scheduler"
      }
    }

I can see the file being written while inside the pod. But not seeing the output on mounted host location.

Following is the execution command

python demo_stdout.py >> /usr/share/1fed8c03-bc30-4889-952e-46f4c19b6ac1.log 2>&1

One thing which I did is the output file and the "terminationMessagePath" is the same as i want the pod termination footprint and the stdout/stderr in the same file.

Dockerfile is as below:

FROM python:2.7.15-alpine3.9

WORKDIR /usr/src/app

COPY . .

CMD ["sh", "-c", "tail -f /dev/null"]

Tried the following:

python demo_stdout.py >> /usr/share/test.log 2>&1

The above produces the log in PVC. But need to get the pod termination log in the same file.

Can someone help me with this?

Makassar

please, could you show your dockerfile? – Volley

I have added the dockerfile, please have a look. – Makassar

I think you need to put in your CMD your python app. – Volley

@Volley If I put the python app in CMD the pod will terminate after completion of the python app, instead i want the pod to be running and pumping streaming logs to persistent volume claim. – Makassar

that is the idea, the container need to terminated when the application will finish the execution, if you want to keep running the application you need to manage this in the application code, doing this CMD ["sh", "-c", "tail -f /dev/null"] you are only running tail command inside the container – Volley

This is just a demo code, I will use the same for a rest-service, so, in that case, the pod will be running. So, while the pod is running, I need to get the std-out/err in a file located at an persistent volume claim, shared between pods, so that the logs can be pushed to a logging backend. Also, if a pod gets terminated, I need the pod termination log in the same file so that it can be tracked to a particular deployment. I have tried the same with python flask instead of the standalone python app. – Makassar

Did you try the other policy option. Could it be cause termination logs are empty? FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. – Vedi

@Makassar Did you solve the problem ? – Byssinosis

Answer 1

tee command reads the standard input and writes it to both the standard output and one or more files##

• your cmd should be modified to , so that your output of demo_stdout.py will be logged to /usr/share/test.log and also to stdin and stdout:

python demo_stdout.py | tee /usr/share/test.log

Oppugnant