Logrotate - nginx logs not rotating inside docker container

Question

I've a docker container running nginx which is writing logs to /var/log/nginx Logrotate is installed in the docker container and the logrotate config file for nginx is set up correctly. Still, the logs are not being automatically rotated by logrotate. Manually forcing log rotate to rotate the logs via logrotate -f /path/to/conf-file works as expected.

My conclusion is that something is not triggering the cron to fire but I can't find the reason.

Here's the Dockerfile for the docker container running nginx:

FROM nginx:1.11

# Remove sym links from nginx image
RUN rm /var/log/nginx/access.log
RUN rm /var/log/nginx/error.log

# Install logrotate
RUN apt-get update && apt-get -y install logrotate

# Copy MyApp nginx config
COPY config/nginx.conf /etc/nginx/nginx.conf

#Copy logrotate nginx configuration
COPY config/logrotate.d/nginx /etc/logrotate.d/

And the docker-compose file:

version: '2'
services:
  nginx:
    build: ./build
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./auth:/etc/nginx/auth
      - ./certs:/etc/nginx/certs
      - ./conf:/etc/nginx/conf
      - ./sites-enabled:/etc/nginx/sites-enabled
      - ./web:/etc/nginx/web
      - nginx_logs:/var/log/nginx
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "1"

volumes:
  nginx_logs:

networks:
  default:
    external:
      name: my-network

Here's the content of: /etc/logrotate.d/nginx

/var/log/nginx/*.log {
        daily
        dateext
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
        endscript
}

Content of /etc/cron.daily/logrotate

#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

Content of /etc/logrotate.conf

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

Can someone point me in the right direction to why nginx logs are not being automatically rotated by logrotate?

EDIT

I can trace the root cause of this problem to the cron service not being run on the container. A possible solution is to find a way to make the container run both nginx and cron service at the same time.

Arterial

Answer 1

As stated on the edit on my question the problem was that CMD from nginx:1.11 was only starting the nginx process. A work around is to place the following command on my Dockerfile

CMD service cron start && nginx -g 'daemon off;'

This will start nginx as nginx:1.11 starts it and well as start the cron service.

The Dockerfile would look something like:

FROM nginx:1.11

# Remove sym links from nginx image
RUN rm /var/log/nginx/access.log
RUN rm /var/log/nginx/error.log

# Install logrotate
RUN apt-get update && apt-get -y install logrotate

# Copy MyApp nginx config
COPY config/nginx.conf /etc/nginx/nginx.conf

#Copy logrotate nginx configuration
COPY config/logrotate.d/nginx /etc/logrotate.d/

# Start nginx and cron as a service
CMD service cron start && nginx -g 'daemon off;'

Arterial

Answer 2

i found a method from this link, its core idea is use logrotate outside, and conf is as below:

$ sudo vi /etc/logrotate.d/test
/home/test/logs/*log {
    rotate 90
    missingok
    ifempty
    sharedscripts
    compress
    postrotate
        /usr/bin/docker exec nginx-test /bin/sh -c '/usr/sbin/nginx -s reopen > /dev/null 2>/dev/null'
    endscript
}

Dye

Better yet, do this in postrotate: if test $(docker ps -q --filter "name=nginx"); then docker exec nginx sh -c 'kill -USR1 $(cat /var/run/nginx.pid)'; fi – Replace

I think it is better to use "docker inspect" much quicker and doesn't require any special privileges, see this answer: https://mcmap.net/q/451924/-containerized-nginx-log-rotation-with-logrotate – Gyroscope

On behalf of rzlvmp: Postrotate script is 100% not working, because of '-it' parameters wrong here. Logrotate don't has logs, so finding the problem really may take long time. – Accentuate

copytruncate is your friend. This allows you to completely ignore postrotate. There is of course a down-side: if the log file receives an entry between the copy and truncate, you'll lose that entry. – Pediatrician

Answer 3

My answer is close to @touchstone, but I wanted to add a coherent answer with some explanation as debugging the issue without this knowledge is going to take a lot of time.

So, this configuration will work:

$ sudo vi /etc/logrotate.d/test
/home/test/logs/*log {
    rotate 90
    missingok
    ifempty
    sharedscripts
    compress
    postrotate
        /usr/bin/docker exec nginx bash -c 'kill -USR1 $(cat /var/run/nginx.pid)'
    endscript
}

What you need to be aware of:

• Always use absolute paths for all paths. This goes for scripts, executables, and paths inside scripts, etc.
  • bash does not need an absolute path, as this is referencing the bash command inside the docker container and not on the host machine.
• Do not use the -it parameters for docker exec

If either of above points are wrong you can experience that running logrotate manually (sudo logrotate --force /etc/logrotate.d/nginx) will pass and work as expected. However, the cron task will silently fail. Specifically you will see the log files have been rotated, but not compressed. Also NGINX will not write to the log files. Since logrotate by default has no logging of its own actions when running cron tasks, you have no way of seeing what the issue was.

• The command docker exec nginx bash -c 'kill -USR1 $(cat /var/run/nginx.pid)' is based of off the official NGINX documentation. NGINX will re-open its logs in response to the USR1 signal.. Additionally, you need the PID of the process, which can be found in /var/run/nginx.pid.

I hope this is useful to someone, as I was not using absolute paths and used the -it parameters for docker exec and it worked when running logrotate manually, but not automatically.

Fellmonger

Answer 4

docker exec --user root `docker container ls --filter "name=nginx" --format "{{.Names}}"` nginx -s reopen

Beneath

Please add some explanation to your answer by editing it, such that others can learn from it – Tm