Is there a way to specify file size limit for docker logs on Google Container Optimized OS?

Question

I had a very long running (and verbose) container job on Google's Container-Optimized OS (COS) that eventually generated enough logs to fill the disk.

To my knowledge there isn't a way to rotate / limit log file size while using COS.

Inspecting the running container, it appears that it writes an ever-growing file to /var/lib/docker/containers/ (mounted on the stateful partition) and that HostConfig.LogConfig.Config is empty.

I ended up having to SSH in and manually delete the multi-gigabyte log file to make the VM operational again.

I read through https://cloud.google.com/compute/docs/containers/configuring-options-to-run-containers and as far as I can tell there isn't a way to (say) pass --log-opt max-size=XX as per the Docker documentation: https://docs.docker.com/config/containers/logging/json-file/

Is there some way to pass that flag? Failing that, are there recommendations on how to rotate logs / limit log size / avoid hitting this problem?

Azral

Answer 1

Currently there isn't any way to pass that flag directly to your container. That being said there is a workaround that might work for you.

It requires you to set the flag globally in /etc/docker/daemon.json It will then apply to all containers started on that VM. You can do that with a startup script, by using this fragment:

cat <<EOF > /etc/docker/daemon.json
{
  "live-restore": true,
  "storage-driver": "overlay2",
  "log-opts": {
    "max-size": "10m"
  }
}
EOF
systemctl restart docker

See https://cloud.google.com/compute/docs/startupscript#providing_startup_script_contents_directly for instructions on startup scripts. Please refer to docker's documentation on the "log-opts" flag: https://docs.docker.com/config/containers/logging/json-file/#usage

Bots

Answer 2

You can limit the logs file size for each service in docker-compose.yml.

Just need to add these lines:

services:
  app:
    container_name: app
    image: node
    restart: always
    volumes:
      - ./app:/home/node/app
    working_dir: /home/node/app
    ports:
      - 3000:3000
    networks:
      - main
    command: "npm start" 
    logging:
      driver: "json-file"
      options:
        max-file: "5"   # number of files or file count
        max-size: "10m" # file size

  db:
    ...
    logging:
      driver: "json-file"
      options:
        max-file: "3"   # number of files or file count
        max-size: "10m" # file size

Detrital

Google's Container-Optimized OS (COS) doesn't run docker-compose – Nimmons

Reference for these parameters: docs.docker.com/config/containers/logging/json-file – Mariahmariam

Answer 3

Container logs by default are not configured to be rotated or limited to a max size by docker. A log file can grow so large that it fills up the disk space if the container runs for long enough and generate enough logs.

See Docker Logging Documentation for more information on what can be set.

To set log limits for containers on a host --log-opt can be configured with max-size and max-file so that a containers logs are rolled over when they reach a max limit and only a certain number of files are saved before being discarded.

# cat /etc/sysconfig/docker 
  OPTIONS='--insecure-registry=172.30.0.0/16 --selinux-enabled --log-opt max size=50m --log-opt max-file=5'

Restart docker service for the changes to take effect.

# systemctl restart docker 

If there is already a log file already present on the host that needs to be removed, run the following to clear its contents and reduce its size.

# cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log

OR

# cat /dev/null >  $(docker inspect --format='{{.LogPath}}'  CONTAINER_ID)

Hope this helps!

Brendin

Adding the --log-opt options in /etc/sysconfig/docker file, log rotation working for new containers, but existing containers its not working . wondering any update option to exisiting containers ,without recreating the containers ? – Weese