Log rotation on logs consuming disk space in Google Cloud Kubernetes pod

Question

We have a pod in a Google Cloud Platform Kubernetes cluster writing JsonFormatted to StdOut. This is picked up by Stackdriver out of box. However, we see the disk usage of the pod just growing and growing, and we can't understand how to set a max size on the Deployment for log rotation.

Documentation on Google Cloud and Kubernetes is unclear on this.

This is just the last hour:

Leontina

@chisva Just hit the same issue. Single binary in a scratch image, no file system logging, only stdout. Did you figure it out? Did the disk chart ever jump back or level out? – Etzel

Answer 1

Are you sure that disk usage of the pod is high because of the logs? If the application writes logs to stdout, it doesn't use any disk space inside the pod. All logs are usually stored in a log file on the node’s filesystem and can be managed by the node logrotate process.

Perhaps application uses pod's disk space for something else, like temp files or debug information?

Here is the part of documentation related to log rotation:

Logging at the node level:

Everything a containerized application writes to stdout and stderr is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in json format.

An important consideration in node-level logging is implementing log rotation, so that logs don’t consume all available storage on the node.

Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour.

You can also set up a container runtime to rotate application’s logs automatically, e.g. by using Docker’s log-opt.

In the kube-up.sh script, the latter approach is used for COS image on GCP, and the former approach is used in any other environment. In both cases, by default rotation is configured to take place when log file exceeds 10MB.

As an example, you can find detailed information about how kube-up.sh sets up logging for COS image on GCP in the corresponding script.

Here is a part of the script related to logrotate:

# Installs logrotate configuration files
function setup-logrotate() {
  mkdir -p /etc/logrotate.d/
  # Configure log rotation for all logs in /var/log, which is where k8s services
  # are configured to write their log files. Whenever logrotate is ran, this
  # config will:
  # * rotate the log file if its size is > 100Mb OR if one day has elapsed
  # * save rotated logs into a gzipped timestamped backup
  # * log file timestamp (controlled by 'dateformat') includes seconds too. This
  #   ensures that logrotate can generate unique logfiles during each rotation
  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
  #   day).
  # * keep only 5 old (rotated) logs, and will discard older logs.
  cat > /etc/logrotate.d/allvarlogs <<EOF
/var/log/*.log {
    rotate ${LOGROTATE_FILES_MAX_COUNT:-5}
    copytruncate
    missingok
    notifempty
    compress
    maxsize ${LOGROTATE_MAX_SIZE:-100M}
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
EOF

}

Lions

This does not appear to be entirely correct. Log rotation is done using Docker's max-size and max-file options. See github.com/kubernetes/kubernetes/pull/40634. – Alnico

i can confirm that the above is correct for GKE and COS work node image – Jamila