containerized nginx log rotation with logrotate

Question

Nginx doesn't have native log rotation, so an external tool, such as logrotate, is required. Nginx presents a challenge in that the logs have to be reopened post rotation. You can send a USR1 signal to it if the pid is available in /var/run.

But when running in a docker container, the pid file is missing in /var/run (and the pid actually belongs to the host, since it is technically a host process).

If you don't reopen the logs, nginx doesn't log anything at all, though it continues to function otherwise as web server, reverse proxy, etc.

Gunrunning

Why would you want to use logrotate if the the information is available through docker logs? – Cantal

This rotation is for process-native logs, not docker logs - web access and error logs. – Gunrunning

Answer 1

You can get the process id from the Pid attribute using docker inspect and use kill -USR1 {pid} to have nginx reopen the logs.

Here's the /etc/logrotate.d/nginx file I created:

/var/log/nginx/access.log
{
    size 2M
    rotate 10
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        docker inspect -f '{{ .State.Pid }}' nginx | xargs kill -USR1
    endscript
}

Gunrunning

docker inspect -f '{{ .State.Pid }}' will do the same. Most commands have a --format option that takes a go template. – Pinhole

kill: sending signal to 27707 failed: Operation not permitted I got this when run docker inspect -f '{{ .State.Pid }}' nginx | xargs kill -USR1 – Abhenry

If you are running the containers in a swarm services docker stack deploy -c docker-compose.yml (single node in my case) then the command in the answer does not work. You will have to use docker service ps -f "desired-state=running" --format "{{.ID}}" web_nginx | xargs docker inspect -f '{{ .Status.ContainerStatus.PID}}' note that service name will be <stack>_<container> format. – Unruly

at <.State.Pid>: map has no entry for key "State" :( – Safelight

Answer 2

If you want to run logrotate in a dedicated container (e.g to rotate both nginx logs and Rails' file log) rather than on the host machine, here's how I did it. The trickiest part by far was as above, getting the reload signals to nginx, Rails, etc so that they would create and log to fresh logfiles post-rotation.

Summary:

• put all the logs on a single shared volume
• export docker socket to the logrotate container
• build a logrotate image with logrotate, cron, curl, and jq
• build logrotate.conf with postrotate calls using docker exec API as detailed below
• schedule logrotate using cron in the container

The hard part:

To get nginx (/etcetera) to reload thus connect to fresh log files, I sent exec commands to the other containers using Docker's API via socket. It expects a POST with the command in JSON format, to which it responds with an exec instance ID. You then need to explicitly run that instance.

An example postrotate section from my logrotate.conf file:

postrotate
    exec_id=`curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"cmd": ["nginx", "-s", "reopen"]}' \
      http:/v1.41/containers/hofg_nginx_1/exec  \
    | jq -r '.Id'`
    curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"Detach": true}' \
      http:/v1.41/exec/"$exec_id"/start
endscript

Commentary on the hard part:

  exec_id=`curl -X POST --unix-socket /var/run/docker.sock \

This is the first of two calls to curl, saving the result into a variable to use in the second. Also don't forget to (insecurely) mount the socket into the container, '/var/run/docker.sock:/var/run/docker.sock'

  -H "Content-Type: application/json" \
  -d '{"cmd": ["nginx", "-s", "reopen"]}' \

Docker's API docs say the command can be a string or array of strings, but it only worked for me as an array of strings. I used the nginx command line tool, but something like 'kill -SIGUSR1 $(cat /var/run/nginx.pid)' would probably work too.

  http:/v1.41/containers/hofg_nginx_1/exec  \

I hard-coded the container name, if you're dealing with something more complicated you're probably also using a fancier logging service

  | jq -r '.Id'`

The response is JSON-formatted, I used jq to extract the id (excuse me, 'Id') to use next.

  curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"Detach": true}' \

The Detach: true is probably not necessary, just a placeholder for POST data that was handy while debugging

  http:/v1.41/exec/"$exec_id"/start

Making use of the exec instance ID returned by the first curl to actually run the command.

I'm sure it will evolve (say with error handling), but this should be a good starting point.

Forcier