How to redirect output of systemd service to a file

Question

I am trying to redirect output of a systemd service to a file but it doesn't seem to work:

[Unit]
Description=customprocess
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=/var/log1.log
StandardError=/var/log2.log
Restart=always

[Install]
WantedBy=multi-user.target

Please correct my approach.

Worthington

I’m voting to close this question because From the tag: systemd questions should be for programming questions using systemd or its libraries. Questions about configuring the daemon (including writing unit files) are better directed to Unix & Linux: unix.stackexchange.com. Please delete this. – Wiencke

I would agree with Rob if the question was new, but it's been around for 7 years and has several highly upvoted answers; it's too late to delete it now imo. – Dorotheadorothee

@GuntramBlohm: Actually Rob's wrong. I found this today trying to find out if I had to write log redirect code or not. – Bogeyman

Answer 1

I think there's a more elegant way to solve the problem: send the stdout/stderr to syslog with an identifier and instruct your syslog manager to split its output by program name.

Use the following properties in your systemd service unit file:

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=<your program identifier> # without any quote

Then, assuming your distribution is using rsyslog to manage syslogs, create a file in /etc/rsyslog.d/<new_file>.conf with the following content:

if $programname == '<your program identifier>' then /path/to/log/file.log
& stop

Now make the log file writable by syslog:

# ls -alth /var/log/syslog 
-rw-r----- 1 syslog adm 439K Mar  5 19:35 /var/log/syslog
# chown syslog:adm /path/to/log/file.log

Restart rsyslog (sudo systemctl restart rsyslog) and enjoy! Your program stdout/stderr will still be available through journalctl (sudo journalctl -u <your program identifier>) but they will also be available in your file of choice.

Source via archive.org

Evers

Not working for me on Ubuntu 16.04. journalctl -u still works but nothing gets sent to the specified file. – Teresaterese

This works great on Debian stretch, however it complains that ~ is deprecated and stop should be used instead. Also note that the second line can be shortened to & stop if the two come after each other. – Guernica

With systemd 236 or newer you can also write directly to a file using StandardOutput=file:/some/path github.com/systemd/systemd/pull/7198 – Torruella

@Torruella FYI i wrote an answer that is similar to your comment. I wrote my answer before I saw your comment. – Paisley

error during parsing file /etc/rsyslog.d/project.conf, on or before line 1: syntax error on token '==' [v8.32.0 try rsyslog.com/e/2207 ] – Lagerkvist

Source is no longer available. – Luca

I got this working by changing /etc/rsyslog.d/<newfile>.conf contents to: :programname, isequal, "<your program identifier>" /var/log/somelog.log Here's documentation on the rsyslog filters: rsyslog.com/doc/v8-stable/configuration/filters.html And here's docs on the properties like programname: rsyslog.com/doc/master/configuration/properties.html – Bevus

I had problem using this configuration until I found that rsyslog has its own user syslog and it has to have write access to the logs location. So use chown accordingly. Hope this helps somebody. – Hester

on ubuntu 16.04 this logs to /path/to/log/file.log and /var/log/syslog – Amhara

Is it possible to separate stdout from stderr using this method? – Crosstie

@Hester Thanks! This was the clincher for me on Xenial. – Eyrie

Ubuntu uses ReinerScript in its default configuration files. So changing contents of /etc/rsyslog.d/<new_file>.conf to if $programname == 'program_identifier' then { action (type="omfile" file="/var/log/my_program.log") } worked out of the box without changing permissions of /var/log/my_program.log – Ascent

Thanks, Test it on Centos 7.5 - Logs are appeared both on journalctl -u SERVICE and /var/log/FILE.log – Fourteenth

The following rsyslog config worked for me: :programname, isequal, "xxx" /var/log/xxx/xxx.log – Arrowworm

the answer may provide a useful alternative solution, but it doesn't describe why the OP's approach doesn't work. I personally would like to know, as I've tried his approach myself, and it didn't work for me either. But why? – Barbette

@GregWoods at the time of writing, the StandardOutput setting simply didn't take a file descriptor. Now there's a way to do it by providing a "file:<path>" value. Just read the manual: man7.org/linux/man-pages/man5/systemd.exec.5.html – Evers

syslog and process name are defaults in systyemd for stdout and stderr, so you do not need to clutter your config at all. – Operational

Step 2 talks about /path/to/log/file.log then step 3 talks about chown that file - but it doesnt exist as no part of the steps create it – Conciseness

more complete answer: unix.stackexchange.com/a/479669 – Marabout

Note that StandardOutput, StandardError, and SyslogIdentifier go in the [Service] section of the .service file. – Assistance

N.b. this answer suggests a different strategy instead of directly answering the question, which other people have done. The submitter here may be happier with this suggestion, but there are plenty of situations where it is more appropriate to directly write to specified files without more dependencies, and people searching for that won't get helped by this answer. – Abdel

The StandardOutput=syslog doesn't wotk anymore: "Standard output type syslog is obsolete, automatically updating to journal" – Demimonde

As of 2020 or so this answer does not work. Need to pipe through logger to make it work now. – Bogeyman

Answer 2

I would suggest adding stdout and stderr file in systemd service file itself.

Referring : https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=

As you have configured it should not like:

StandardOutput=/home/user/log1.log
StandardError=/home/user/log2.log

It should be:

StandardOutput=file:/home/user/log1.log
StandardError=file:/home/user/log2.log

This works when you don't want to restart the service again and again.

This will create a new file and does not append to the existing file.

Use Instead:

StandardOutput=append:/home/user/log1.log
StandardError=append:/home/user/log2.log

NOTE: Make sure you create the directory already. I guess it does not support to create a directory. This feature was introduced in systemd 240, but backported to earlier versions on e.g. RHEL8.

Gnathonic

I think, I made it more direct\easy to understand. – Gnathonic

For me the file: route works on the first load of the service, but on subsequent restarts it no longer writes to the file. I tried append: from the docs and that didn't work at all. – Detrital

Note that the docs make clear that file: writes to the start of the file every time, and does not truncate... further, append: seems to be a new addition (i.e. not present in the man systemd.exec page on Ubuntu 18.04). – Directorial

append: has been introduced in systemd version 240. To work around file: placing new log output at the beginning of the target file, something like this may help: ExecStartPre=/bin/bash -c 'mv /var/log/my/logs.log /var/log/my/$$(date +%%T)-logs.log'. Keeps a clean log and emulates log rotation effect in a way – Whitton

For anybody curious, Ubuntu 20.04 currently has systemd version 245, while Ubuntu 18.04 has 237. So unfortunately you have to have 20.04 to use append:. – Tarahtaran

truncate: was added in version 248. It will truncate before opening. – Saguaro

append: gives error Failed to parse output specifier, ignoring:\home\path\log.log – Cockfight

@SiddharthDas Make sure you have the correct path defined. Also to use append: you have to have Ubuntu 20.04 – Gnathonic

Answer 3

If you have a newer distro with a newer systemd (systemd version 236 or newer), you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME.

---

Long story:

In newer versions of systemd there is a relatively new option (the github request is from 2016 ish and the enhancement is merged/closed 2017 ish) where you can set the values of StandardOutput or StandardError to file:YOUR_ABSPATH_FILENAME. The file:path option is documented in the most recent systemd.exec man page.

This new feature is relatively new and so is not available for older distros like centos-7 (or any centos before that).

Paisley

Not working in ubuntu 1604 in 2018-03-20. The systemd version in ubuntu 1604 is only 229. – Leadership

Thanks, you said very clear. I just can not believe the systemd in ubuntu 1604 can not redirect output to a file just by config.I have to use the sh way to solve this problem. – Leadership

@bronzeman the feature request wasn't closed until 2017, while Ubuntu 16.04 came out in 2016. In a given major release of Ubuntu (e.g. 16.04, 16.10, 17.04, etc.), Ubuntu maintains ABI compatibility in its core system packages. So they won't upgrade systemd (or the Linux kernel, or glibc, or anything) unless it maintains the same ABI as when the Ubuntu version was first released. – Landsknecht

FWIW: I've searched a bit but this feature doesn't appear to have provisions for log rotation, such as function to reopen the log file, with one having to use the likes of copytruncate in logrotate. – Administer

The problem is, that file is always created as root:root, regardless of the unit's user and group... – Bar

i'm using Ubuntu 18.04 and its systemd version is 237 but is does not work. The file is created but it is empty! – Purgative

Answer 4

You possibly get this error:

Failed to parse output specifier, ignoring: /var/log1.log

From the systemd.exec(5) man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket.

The systemd.exec(5) man page explains other options related to logging. See also the systemd.service(5) and systemd.unit(5) man pages.

Or maybe you can try things like this (all on one line):

ExecStart=/bin/sh -c '/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server 2>&1 > /var/log.log' 

Nakitanalani

Among the options, logging into the systemd journal is recommend. You view just the logs for your process in the journal by using journalctl -u your-unit-name. – Tesch

To specify a file, there's another cleaner option, as indicated by the documentation: The fd option connects the output stream to a single file descriptor provided by a socket unit. A custom named file descriptor can be specified as part of this option, after a ":" (e.g. "fd:foobar"). – Deciduous

Awsome answer, it's solved my problem. I just want to extend, because currently if the service restart overwrite the old logs, have to replace this part: 2>&1 > /var/log.log to this: 2>&1 >> /var/log.log. Thank you – Apocynaceous

Frankly, calling shell with a command string in ExecStart sounds like the Really Wrong Way to do it. – Itinerant

The '/bin/sh -c' looks wrong but it works. It also works with 'Restart=always' . I also suggest you add 'StandardOutput=null' to the config to avoid some rubbish log go into /var/syslog – Leadership

The '/bin/sh -c' doesn't work for me, I think due to strange quoting rules in systemd. See e.g. https://mcmap.net/q/101884/-systemd-string-escaping . Does anyone have this working? – Ideational

"/bin/sh" is a great workaround, but you MUST use "exec", otherwise the service will not restart properly as SIGTERM will not get passed to the child process. See veithen.io/2014/11/16/sigterm-propagation.html – Ideational

'/bin/sh -c' has one undesired consequence with not working "systemctl stop service" command. It just kills sh process and not touches the service – Conspecific

journalctl -u is ridiculous and has numerous shortcomings. It needs redesigned with actual workflows and simple commands in mind. Every QA team member hates it (same with systemctrl not giving output). File output wasn't thought of from the start, which is just criminal; and, a lot of enterprise users still don't have it. This was a mistake and it hasn't been improved. This is one case where companies took over and left users struggling. – Utile

To redirect stderr correctly, you must use >/var/log.log 2>&1, not the other way round. – Megaphone

Answer 5

If for a some reason can't use rsyslog, this will do: ExecStart=/bin/bash -ce "exec /usr/local/bin/binary1 agent -config-dir /etc/sample.d/server >> /var/log/agent.log 2>&1"

Betthezul

What does the -e option of bash do? – Nation

The only solution that worked in my case for not losing the log file at every service restart as append is not available in my current systemd version unfortunately. – Smasher

@Lamp, I doubt you still need this but I was also curious about the -e option in bash. I found this: https://mcmap.net/q/101885/-bash-e-flag-and-39-amp-amp-39-list-command – Oversweet

Answer 6

Short answer:

StandardOutput=file:/var/log1.log
StandardError=file:/var/log2.log

If you don't want the files to be cleared every time the service is run, use append instead:

StandardOutput=append:/var/log1.log
StandardError=append:/var/log2.log

Midrash

Duplicate of this answer, with less details – Jenkins

Super helpful. I used this technique to restore logging to /var/log/tomcatX/catalina.out for Ubuntu 18.04 and 20.04. – Botch

Wrong about "file". Read the docs about what that does. It's "truncate" mode that actually does: ...cleared every time the service is run Also, there is no reason to specify StandardError as you do here, since default behavior is for the error stream to inherit the stream set up by StandardOutput. – Abdel

It does log trancating based on size automatically with this. Is there a way to change that? I don't seem to see anyone talking about that. – Abduction

Answer 7

We are using Centos7, spring boot application with systemd. I was running java as below. and setting StandardOutput to file was not working for me.

ExecStart=/bin/java -jar xxx.jar  -Xmx512-Xms32M

Below workaround solution working without setting StandardOutput. running java through sh as below.

ExecStart=/bin/sh -c 'exec /bin/java -jar xxx.jar -Xmx512M -Xms32M >> /data/logs/xxx.log 2>&1'

Bassoon

-1 for defining jvm parameters in wrong order. -Xmx512M must be defined before -jar . Also what you experience is expected. Systemd does not invoke services using shell – Tonytonya

@SamiKorhonen, I added my comments after testing this is working. Even I was thinking about order of -Xmx512M is smilar to you. Please test before adding blind comments. – Bassoon

Answer 8

Assume logs are already put to stdout/stderr, and have systemd unit's log in /var/log/syslog

journalctl -u unitxxx.service

Jun 30 13:51:46 host unitxxx[1437]: time="2018-06-30T11:51:46Z" level=info msg="127.0.0.1
Jun 30 15:02:15 host unitxxx[1437]: time="2018-06-30T13:02:15Z" level=info msg="127.0.0.1
Jun 30 15:33:02 host unitxxx[1437]: time="2018-06-30T13:33:02Z" level=info msg="127.0.0.1
Jun 30 15:56:31 host unitxxx[1437]: time="2018-06-30T13:56:31Z" level=info msg="127.0.0.1

Config rsyslog (System Logging Service)

# Create directory for log file
mkdir /var/log/unitxxx

# Then add config file /etc/rsyslog.d/unitxxx.conf

if $programname == 'unitxxx' then /var/log/unitxxx/unitxxx.log
& stop

Restart rsyslog

systemctl restart rsyslog.service

Isma

Answer 9

Make your service file call a shell script instead of running the app directly. This way you have extra control. For example, you can make output files like those in /var/log/

Make a shell script like /opt/myapp/myapp.sh

#!/bin/sh
/usr/sbin/logrotate --force /opt/myapp/myapp.conf --state /opt/myapp/state.tmp
logger "[myapp] Run" # send a marker to syslog
myapp > /opt/myapp/myapp.log 2>&1 &

And your service file myapp.service contains:

...
[Service]
Type=forking
ExecStart=/bin/sh -c /opt/myapp/myapp.sh
...

A sample of log config file /opt/myapp/myapp.conf

/opt/myapp/myapp.log {
    daily
    rotate 20
    missingok
    compress
}

Then you will get myapp.log, and zipped myapp.log.1.gz ... for each time the service was started, and previous zipped.

Casilde

Answer 10

• In my case 2>&1(stdout and stderr file descriptor symbol) had to be placed correctly,then log redirection worked as I expected

[Unit]
Description=events-server

[Service]
User=manjunath
Type=simple
ExecStart=/bin/bash -c '/opt/events-server/bin/start.sh my-conf   2>&1 >> /var/log/events-server/events.log'

[Install]
WantedBy=multi-user.target

Kalk