How do I rotate my Rails 5 logs?

Question

I'm running Rails 5 on Ubuntu 14.04. Is there a way to rotate my logs without relying on the Linux logrotate system? I have this set up ...

myuser@myapp:~$ cat /etc/logrotate.d/myapp
/home/rails/myapp/log/*.log {
  daily
  missingok
  rotate 2
  compress
  delaycompress
  notifempty
  copytruncate
}

but yet my logs never rotate. Behold how bloated they are ...

myuser@myapp:~$ ls -al /home/rails/myapp/log/
total 3958356
drwxr-xr-x  2 rails rails       4096 Jul  3 22:31 .
drwxr-xr-x 15 rails rails       4096 Sep 21 17:21 ..
-rw-rw-r--  1 rails rails          0 Jun 22 10:22 development.log
-rw-rw-r--  1 rails rails      14960 Jun  1 22:39 development.log.1
-rw-rw-r--  1 rails rails          0 Oct 22  2016 .keep
-rw-r--r--  1 rails rails  198362787 Oct 31 16:28 production.log
-rw-r--r--  1 rails rails    8615654 Jul  3 22:31 production.log.1
-rw-r--r--  1 rails rails  640621243 Jun 29 13:16 production.log.2.gz
-rw-rw-r--  1 rails rails 2856792698 Oct 31 17:12 sidekiq.log
-rw-rw-r--  1 rails rails  348853619 Jul  3 22:31 sidekiq.log.1
-rw-rw-r--  1 rails rails          0 Jul  3 22:31 test.log
-rw-rw-r--  1 rails rails      54246 Jul  3 22:31 test.log.1

Is there another way to get the logs rotated or is there a way to fix the configuration I have included?

Edit: Here's the cron script that's set

myuser@myapp:~$ cat /etc/cron.daily/logrotate
#!/bin/sh

# Clean non existent log file entries from status file
cd /var/lib/logrotate
test -e status || touch status
head -1 status > status.clean
sed 's/"//g' status | while read logfile date
do
    [ -e "$logfile" ] && echo "\"$logfile\" $date"
done >> status.clean
mv status.clean status

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

Edit: Per the comment I tried adding this to my config/environment/production.rb file ...

config.logger = ActiveSupport::Logger.new(config.paths['log'].first, 1, 50 * 1024 * 1024)

but the logs get ever bigger without being rotated.

Perretta

How often is the cron job run? What's the command that runs the cron job? If it's just a script in /etc/cron.daily (probably etc/cron.daily/logrotate), what are its contents? What are the contents of the global logrotate conf file (probably at /etc/logrotate.conf)? – Chaconne

Do I have to explicitly tell logrotate to run? I thought it was one of those Linux utilities that runs automatically. – Perretta

It's run by a cron job. On Ubuntu, it's typically /etc/cron.daily/logrotate and it's already there, so changing the confs should start getting things rotated. – Chaconne

I added the script as an edit to my question. I"m not sure how to change it though to get things to run. – Perretta

Is there a reason you don't want to use the environment file in your config directory ala https://mcmap.net/q/135263/-ruby-on-rails-production-log-rotation ? – Gehman

You also have to add /etc/logrotate.conf to the question, as the settings in there are combined with the ones in your myapp conf file. – Chaconne

@SWoo, using the logic in the question you referenced, what happens to the logs once it exceeds the size set? Is it deleted, archived? – Perretta

What I have seen in production servers is that the log files are rotated when the size is exceeded, up to the max number of files to rotate. Oldest file is deleted. You can set this on your development server on localhost and set the size really small to play with it to see if it does what you want. – Gehman

What's the output of logrotate -d /etc/logrotate.conf? And what does /var/lib/logrotate/status look like? – Chaconne

@Perretta you might want to take a look at this post #4884391 – Dahna

I'm already using logrotate. I posted my config and results in teh question. – Perretta

Answer 1

By following the 12-factor methodologies (Treat logs as event streams) you could "should" delegate this task to the supervisor.

For example, by using immortal it will do all the rotation process for you not depending on the operating system.

A basic configuration file (run.yml) may look like:

cmd: bundle exec unicorn -c unicorn.rb
cwd: /arena/app-1
env:
    DEBUG: 1
    ENVIRONMENT: production
log:
    file: /var/log/app-1.log
    age: 86400 # seconds
    num: 7     # int
    size: 1    # MegaBytes
    timestamp: true # will add timesamp to log

In case you would like to split the logs stderr & stdout this could be used:

cmd: bundle exec unicorn -c unicorn.rb
cwd: /arena/app-1
env:
    DEBUG: 1
    ENVIRONMENT: production
log:
  file: /var/log/app.log
  age: 86400 # seconds
  num: 7     # int
  size: 1    # MegaBytes
stderr:
  file: /var/log/app-error.log
  age: 86400 # seconds
  num: 7     # int
  size: 1    # MegaBytes
  timestamp: true # will add timesamp to log

As a side note, from the 12-factor site:

Twelve-factor app processes should never daemonize or write PID files. Instead, rely on the operating system’s process manager to manage output streams, respond to crashed processes, and handle user-initiated restarts and shutdowns.

[Disclaimer: I'm the author of immortal], and one of the ideas behind was indeed to cover the automation of applications at scale without need to worry about filling up the disk when logs where not properly rotated besides being available to deploy (start/restart) in the easiest possible way by just modifying a run.yml file without requiring root privileges.

Varion

Where on my file system does run.yml get installed? – Perretta

@Perretta check immortaldir you could add your files in /etc/immortal/ this will keep them persistent across reboots, or if just want to test you could just run immortal -c your.yml – Varion

@Perretta Probably also this could help immortal.run/post/systemd It should work for Ubuntu 14 and also new versions >=16 – Varion

I'm having a lot of trouble installing immortal following their directions. When I run "git clone [email protected]:immortal/immortal.git $HOME/go/src/github.com/immortal/immortal", I get a "Permission denied (publickey)." error. When I download the immortal zip and run make, I get a "make: execvp: go: Permission denied" error. Why couldn't they just make a yum install like everyone else?! – Perretta

@Perretta try to download a precompiled version probably is easier github.com/immortal/immortal/releases, any help creating a yum file will be more than welcome, in any case, please feel free to create any issue/question here: github.com/immortal/immortal/issues – Varion

@Perretta try downloading github.com/immortal/immortal/releases/download/0.17.0/… and try with sudo dpkg -i immortal_0.17.0_amd64.deb – Varion

Cool, thx, I think that "dpkg" command worked in as far as I got no errors and it said "Setting up immortal (0.17.0) ...". However, when I went to create that run.yml file, I didnt' see a "/etc/immortal/" directory. Is taht something I need to create on my own or is there another setup step? – Perretta

@Perretta please check this immortal.run/post/how-to-install should guide you to setup immortal on your system, in case need anything else please let me know. – Varion

Thanks. Two more questions I couldn't figure out from teh docs -- in the run.yml file, what is supposed to go here -- "cwd: /arena/app-1"? Also, I would liek to rotate both my production.log and sidekiq.log files, but the "file:" seems to only allow for a single file. How can I incorporate multiple files here? – Perretta

cwd is for change directory, so before you run the process it changes to that directory, for sidekiq you may create a sidekiq.yml you can have as many *.yml files just be aware that your applications should write to stdout or stderr – Varion

Oh ok so I shoudl be "cwd"'ing to the directory from where I Ican run "bundle exec unicorn -c unicorn.rb"? Btw, why do I need to run this command? – Perretta

@Perretta you don't need that option is just an example, please check the project site immortal.run for more info and give a try to the multiple options should be pretty straightforward, in case that you found an issue please use this: github.com/immortal/immortal/issues – Varion

I don't need which option -- the "cwd" or the "bundle exec unicorn" option? – Perretta

Let us continue this discussion in chat. – Varion

Answer 2

You can use ruby Logger class and shift_age method to select frequency of rotation: daily, weekly or monthly.

shift_age: Number of old log files to keep, or frequency of rotation (daily, weekly or monthly). Default value is 0, which disables log file rotation.

https://ruby-doc.org/stdlib-2.4.0/libdoc/logger/rdoc/Logger.html#method-c-new

To use it in Rails you can insert a line like this in your config/application.rb:

config.logger = ActiveSupport::Logger.new("log/#{Rails.env}.log", shift_age = 'daily')

Note: today's log will not have any date, only tomorrow (if daily rotation) the old log will be 'rotated' and date appended to filename.

Bairam

Answer 3

Recommending using STDOUT for app logging is a terrible idea. The Twelve-factor app is highly focused for container-based deployment when UNIX daemons are not welcomed, you should not buy this just as-is.

Standard output has no structure, it's just a plain-text interface. While it makes much sense for UNIX multi-process communication, it's just awful for logging. The only proper logging interface has always been either syslog or system journal. Our Rails app takes advantage of both and you can configure it either with STDOUT (for dev environments), syslog or journald for structured logging.

This way you can embed things like request, session or correlation id along with all messages. Logs can be sent over the wire to logging servers easily. Log file rotation is no problem at all if you use this approach. And finally, you can integrate with central logging solutions like ElasticSearch (or ELK stack) for further data analysis.

Indeed it's a little bit of work and unfortunately Rails developers are very opinionated about this - they use super-simple and not flexible Ruby logger, they also send input parameters to INFO level log and refuse to make it a configurable option (some APIs have huge inputs in our app which floods log). But it is what it is - you can override this in your app and give Rails a proper logging stack. Our solution:

• https://github.com/TwP/logging
• https://github.com/lzap/logging-journald (plugin for logging gem)

Lemuel