Truncating a file while it's being used (Linux)

Question

I have a process that's writing a lot of data to stdout, which I'm redirecting to a log file. I'd like to limit the size of the file by occasionally copying the current file to a new name and truncating it.

My usual techniques of truncating a file, like

cp /dev/null file

don't work, presumably because the process is using it.

Is there some way I can truncate the file? Or delete it and somehow associate the process' stdout with a new file?

FWIW, it's a third party product that I can't modify to change its logging model.

EDIT redirecting over the file seems to have the same issue as the copy above - the file returns to its previous size next time it's written to:

ls -l sample.log ; echo > sample.log ; ls -l sample.log ; sleep 10 ; ls -l sample.log
-rw-rw-r-- 1 user group 1291999 Jun 11  2009 sample.log
-rw-rw-r-- 1 user group 1 Jun 11  2009 sample.log
-rw-rw-r-- 1 user group 1292311 Jun 11  2009 sample.log

Hearsh

Answer 1

Take a look at the utility split(1), part of GNU Coreutils.

Lackadaisical

Excellent - thanks. Looks like the best option for what I want to do. – Hearsh

It seems odd that this is the chosen solution. It does nothing to reduce the size of the file; all it does is give you copies of the sections of the file as well as the original. – Pilferage

Yeah, this does what I need, rather than what I asked. I'll redirect via split, rather than directly to the log file, which will limit the size of the individual log files. – Hearsh

for those who are scratching their head that what does this mean: $your_command | split --bytes 100M - prefix – Global

Downvoted. Needs more of an answer, with details, to be SO quality. – Entrust

Answer 2

As of coreutils 7.0, there is a truncate command.

Pachyderm

Is that a command-line option? Or a C function? The man pages seem to imply the latter. Not that I'm averse to writing a wrapper, just for my understanding. – Hearsh

There has always (for some value of always) been a C function truncate(). As of coreutils 7.0, there is also a program truncate, which presumably wraps the C function. – Pachyderm

Thank you, it helped. I've been trying to clear a tomcat log without stopping it and it works. – Warty

Does it really work for java apps? For me it does not! Even if truncate -s 0 GC.log for a few seconds shows a "truncated" file, after this time (I'm guessing - when the app writes more) - the file gets back to it's original size. – Brendis

Answer 3

The interesting thing about those regrown files is that the first 128 KB or so will be all zeroes after you truncate the file by copying /dev/null over it. This happens because the file is truncated to zero length, but the file descriptor in the application still points immediately after its last write. When it writes again, the file system treats the start of the file as all zero bytes - without actually writing the zeroes to disk.

Ideally, you should ask the vendor of the application to open the log file with the O_APPEND flag. This means that after you truncate the file, the next write will implicitly seek to the end of the file (meaning back to offset zero) and then write the new information.

---

This code rigs standard output so it is in O_APPEND mode and then invokes the command given by its arguments (rather like nice runs a command after adjusting its nice-level, or nohup runs a command after fixing things so it ignores SIGHUP).

#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>

static char *arg0 = "<unknown>";

static void error(const char *fmt, ...)
{
    va_list args;
    int errnum = errno;
    fprintf(stderr, "%s: ", arg0);
    va_start(args, fmt);
    vfprintf(stderr, fmt, args);
    va_end(args);
    if (errnum != 0)
        fprintf(stderr, " (%d: %s)", errnum, strerror(errnum));
    putc('\n', stderr);
    fflush(0);
    exit(1);
}

int main(int argc, char **argv)
{
    int attr;
    arg0 = argv[0];

    if (argc < 2)
        error("Usage: %s cmd [arg ...]", arg0);
    if ((attr = fcntl(1, F_GETFL, &attr)) < 0)
        error("fcntl(F_GETFL) failed");
    attr |= O_APPEND;
    if (fcntl(1, F_SETFL, attr) != 0)
        error("fcntl(F_SETFL) failed");
    execvp(argv[1], &argv[1]);
    error("failed to exec %s", argv[1]);
    return(1);
}

My testing of it was somewhat casual, but just barely enough to persuade me that it worked.

---

Simpler alternative

Billy notes in his answer that '>>' is the append operator - and indeed, on Solaris 10, bash (version 3.00.16(1)) does use the O_APPEND flag - thereby making the code above unnecessary, as shown ('Black JL:' is my prompt on this machine):

Black JL: truss -o bash.truss bash -c "echo Hi >> x3.29"
Black JL: grep open bash.truss
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
open("/usr/lib/libcurses.so.1", O_RDONLY)       = 3
open("/usr/lib/libsocket.so.1", O_RDONLY)       = 3
open("/usr/lib/libnsl.so.1", O_RDONLY)          = 3
open("/usr/lib/libdl.so.1", O_RDONLY)           = 3
open("/usr/lib/libc.so.1", O_RDONLY)            = 3
open("/platform/SUNW,Ultra-4/lib/libc_psr.so.1", O_RDONLY) = 3
open64("/dev/tty", O_RDWR|O_NONBLOCK)           = 3
stat64("/usr/openssl/v0.9.8e/bin/bash", 0xFFBFF2A8) Err#2 ENOENT
open64("x3.29", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3
Black JL:

Use append redirection rather than the wrapper ('cantrip') code above. This just goes to show that when you use one particular technique for other (valid) purposes, adapting it to yet another is not necessarily the simplest mechanism - even though it works.

Pilferage

The log file's being created by me, by redirecting stdout. Any way to do the equivalent of O_APPEND on a file created by a simple redirect? – Hearsh

@Hobo: succinctly - no. Or, at least, very unlikely. If you were able to start the program under a debugger, you might be able to make a fcntl() system call to add the O_APPEND attribute to standard output, but otherwise, you are stuck, I think. Well, let's think...you could write a wrapper around the other application...that does the fcntl() - and maybe the redirect too - and then runs the 3rd Party Program. Give me a while to look at that properly... – Pilferage

Wow. Cheers for taking the time to write that. I think it's more than I'd like to do at this stage - I think I'll stick with split - but I really appreciate you taking the time. Just a pity I can't vote more than once for your answer ;-) – Hearsh

@Hearsh - it wasn't a problem; it was mostly recycling previously used fragments (though adjusting the flags with the two calls to fcntl() was new). You can see it took me 25 minutes to assemble - not long. – Pilferage

@Hobo: Just redirect the standard output with >> instead of >, as noted in @Billy's answer. – Linlithgow

Thank you for this answer. I got into that issue when rotating with copy/truncate a log file and did not understand at first why the file was growing back to its original logical size. – Brigand

Answer 4

Redirect the output using >> instead of >. This will allow you to truncate the file without the file growing back to its original size. Also, don't forget to redirect STDERR (2>&1).

So the end result would be: myprogram >> myprogram.log 2>&1 &

Forewent

Answer 5

Try > file.

---

Update regarding the comments: it works nicely for me:

robert@rm:~> echo "content" > test-file
robert@rm:~> cat test-file 
content
robert@rm:~> > test-file
robert@rm:~> cat test-file 

Alrzc

Redirecting over the file doesn't seem to work - it has the same problem as the "cp /dev/null file" I mentioned in the question. It seems to work, but the next time the file's written to it reverts to its previous size. On my flavour of linux that particular command returns "Invalid null command."; not sure if you were suggesting "[something] > file", rather than just "> file". – Hearsh

I mean exactly '> file', perhaps it's a bash-specific command. The file reverting is quite strage though. – Alrzc

I've seen this behaviour too. It almost seems like the file pointer is being changed to a new empty file. But the running process has a handle on the old file. When it writes it changes the file pointer back to the old file. – Complicity

Answer 6

I had a similar issue on redhat v6, echo > file or > file was causing apache and tomcat to go faulty as log files would become inaccessible to them.

And the fix was strange

echo " " > file

would clean the file and not cause any problem.

Atalya

Worked for me as well. You saved my day – Parlando

Answer 7

In Linux (actually all unicies) files are created when they are opened and deleted when nothing holds a reference to them. In this case the program that opened it and the directory it was opened 'in' hold references to the file. When the cp program wants to write to the file it gets a reference to it from the directory, writes a length of zero into the metadata stored in the directory (this is a slight simplification) and gives up the handle. Then the original program, still holding the original file handle, writes some more data to the file and saves what it thinks the length should be.

even if you where to delete the file from the directory the program would continue to write data to it (and use up disc space) even though no other program would have any way of referencing it.

in short once the program has a reference (handle) to a file nothing you do is going to change that.

there are in theory ways of modifying the programs behavior by setting LD_LIBRARY_PATH to include a program that intercepts all the file access system calls. I recall seeing something like this somewhere though cant recall the name.

Underslung

You probably need LD_PRELOAD rather than LD_LIBRARY_PATH. I think you're on the right track with your explanation, though the details are a little woolly - see my explanation (which can probably have the same accusation lobbed at it). – Pilferage

Thanks for clarifying why it's happening. I figured it was something like that, but good to have it confirmed. I think intercepting system calls is more than I'd like to do in this case, but thanks for another alternative. – Hearsh

Answer 8

as the file is being used, if you try to nullify it or something like that, sometimes it might "confuse" the app that's writing into the log file and it might not log anything after that.

What I'd try ot do is to set up a kind of proxy/filter for that log, instead of redirecting to file, redirect to a process or something that would get input and write to a rolling file.

Maybe it can be done by script otherwise you could write a simple app for that ( java or something else ). The impact on app performance should be quite small, but you'll have to run some tests.

Btw, your app, is it a stand-alone, web app, ... ? Maybe there are other options to be investigated.

Edit: there's also an Append Redirection Operator >> that I've personally never used, but it might not lock the file.

Bunton

Cheers. Yeah, I think that's the approach I'm going to take. I'll use Michiel's suggestion of using split. I hadn't realised it worked on stdin as well as files (which I shoudl have). – Hearsh

Answer 9

I downloaded and compiled the latest coreutils so I could have truncate available.

Ran ./configure and make, but did not run make install.

All the compiled utilities appear in the "src" folder.

I ran

[path]/src/truncate -s 1024000 textfileineedtotruncate.log

on a 1.7 GB log file.

It did not change the size listed when using ls -l, but it did free up all the disk space - which is what I really needed to do before /var filled up and killed the process.

Thanks for the tip on "truncate"!

Filariasis

Answer 10

Did you check the behavior of any signals like SIGHUP to the third party product, to see if it will start logging a fresh file? You would move the old file to a permanent name, first.

kill -HUP [process-id]

And then it would start writing out again.

Alternatively (as Billy suggested) maybe redirecting the output from the application to a logging program like multilog or the one that is commonly used with Apache, known as cronolog. Then you'll have more fine grained control of where everything goes before it is written to that initial file descriptor (file), which is really all it is.

Keto

I don't think I want to restart the process (it's pretty slow to come up, and that'd affect our users), but cheers for teaching me about kill -HUP - i wasn't aware of it. – Hearsh

@Hobo: Just to clarify, for daemons which recognize HUP, 'kill -HUP' tells them to reread their configuration and/or close and reopen any files they're using. It does not restart the process. (If the daemon doesn't trap the HUP, then, yeah, it'll die normally...) – Haldane

Answer 11

instead of redirecting it to a file you could pipe it to a program that automatically rotates the file by closing it, moving it and opening a new one every time it gets too big.

Underslung

Answer 12

@Hobo use freopen(), it reuses stream to either open the file specified by filename or to change its access mode. If a new filename is specified, the function first attempts to close any file already associated with stream (third parameter) and disassociates it. Then, independently of whether that stream was successfully closed or not, freopen opens the file specified by filename and associates it with the stream just as fopen would do using the specified mode.

if a thirdparty binary is generating logs we need to write a wrapper which will rotate the logs, and thirdparty will run in proxyrun thread as below.

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <unistd.h>
#include <string.h>

using namespace std;

extern "C" void * proxyrun(void * pArg){
   static int lsiLineNum = 0;
   while(1) 
   {
     printf("\nLOGGER: %d",++lsiLineNum);
     fflush(stdout);
   }
  return NULL;
}


int main(int argc, char **argv)
{
  pthread_t lThdId;
  if(0 != pthread_create(&lThdId, NULL, proxyrun, NULL))
  {
    return 1;
  }

  char lpcFileName[256] = {0,};

  static int x = 0;

  while(1)
  {
    printf("\n<<<MAIN SLEEP>>>");
    fflush(stdout);
    sprintf(lpcFileName, "/home/yogesh/C++TestPrograms/std.txt%d",++x);
    freopen(lpcFileName,"w",stdout);
    sleep(10);
  }

  return 0;
}

Duckett

Answer 13

I had a similar problem and was unable to do a "tail -f" on the output of a script that was run from cron:

    * * * * * my_script >> /var/log/my_script.log 2>&1

I fixed it by changing the stderr redirect:

    * * * * * my_script >> /var/log/my_script.log 2>/var/log/my_script.err

Acephalous

Answer 14

There are (at least) four solutions to your "foreign software writes endless log files" problem:

1. If you can restart/reload your foreign software without affecting the service provided by it, then use a script to move the logfile regularly to a different name (say: mv logfile logfile.old), then restart/reload the code, so that it starts writing a new log file with the usual name. In many cases, you don't need to write such a script yourself even, as the system utility logrotate provides enough options, like when to rotate, how to name the rotated files, whether to compress them, and so on.

2. If your foreign code has opened the logfile with the O_APPEND flag AND if you can live with loosing the contents of the logfile at regular intervals, then you can write a script, that truncates the size of the logfile to zero at regular intervals. Use echo -n "" >logfile or truncate --size=0 logfile for that.

   If the foreign code does not use O_APPEND, these methods will still succeed in releasing the disk space of the old log file, but the log file will be continued to be written at the old position, resulting in a (possibly huge) unallocated space at the beginning of the logfile. If you open such a sparse file with an editor, you will see zeros at the front. tail -N logfile will work, though, to see the last N lines of it. Use cp --sparse=always and tar --sparse to handle such files efficiently.

3. If you don't want to loose all of the logfile at the time of the truncate, you can use fallocate --collapse-range --length LEN logfile to only remove LEN bytes from it. Use a script to calculate LEN as the current file size minus how much you want to keep, and then round LEN down to a multiple of the file system's block size. Unfortunately, this only works on certain file systems on linux.

   When your foreign application doesn't use O_APPEND, it will still continue to write at the old file position. As the old tail of the log file is moved to the front (without physically moving blocks on the disk, only the logical allocation of the blocks is altered), the hole will now be between the old and new data.

   If you have to deal with a sparse file, anyways, you can also use fallocate --punch-hole --length LEN logfile. That will zero out the first LEN bytes of the logfile and deallocate blocks, that have become fully zero. This is also supported on a larger subset of the linux file systems as fallocate --collapse-range.

4. Finally, you can make logfile a named pipe, that is read by a script, that (for example) reads its input line by line and writes it to a daily rotating log file. That script could even scan its input for certain error conditions and use e-mail or an instant messenger to alert you directly. This is the solution with the most overhead, though. Your logdata forwarding script will eat extra CPU cycles and if it crashes, logdata will be either lost, or the main application will exit, too, because of errors writing the logfile.

Unshaped