Keeping log files under a certain size

Question

I have an application that is running on a stand-alone panel PC in a kiosk (C#/WPF). It performs some typical logging operations to a text file. The PC has some limited amount of disk space to store these logs as they grow.

What I need to do is be able to specify the maximum size that a log file is allowed to be. If, when attempting to write to the log, the max size is exceeded, new data will be written to the end of the log and the oldest data will be purged from the beginning.

Getting the file size is no problem, but are there any typical file manipulation techniques to keep a file under a certain size?

Inoperative

Answer 1

One technique to handle this is to have two log files which are half the maximum size each. You simply rotate between the two as you reach the max size of each file. Rotating to a file causes it to be overwritten with a new file.

A logging framework such as log4net has this functionality built in.

Pollinize

Answer 2

Try using Log4Net

http://www.codeproject.com/KB/aspnet/log4net.aspx

Compressor

Answer 3

There's no easy way to strip the data from the beginning of file. So you have several options:

1. Keep the log in several smaller log files and delete the oldest "chunks" if the total size of all log files exceeds your limit. This is similar to what you want to do, but on different level
2. Rename the log file to "log.date" and start a new log. Similar to (1) but not an option if you have limited disk space.
3. IF you have enough RAM and your log size is relatively small to fit in memory, you can do the following: map the whole file into memory using Memory-mapped file, then perform move operation by taking the data from the middle of the file and moving them to the beginning. Then truncate the file. This is the only way to easily strip the data from the beginning of the log file without creating a copy of it.

Aerospace

Great summary of viable options. – Aisne

Answer 4

Linux os: check out logrotate - http://www.cyberciti.biz/faq/how-do-i-rotate-log-files/

Windows os: try googling windows logrotate. for example: http://blog.arithm.com/2008/02/07/windows-log-file-rotation/

Ejaculate

Answer 5

I wanted a simple solution as well, but I didn't want to add another dependency so I made a simple method. This has everything you need other than the part of compressing the old file to a zip, which you can find here: Create zip file in memory from bytes (text with arbitrary encoding)

static int iMaxLogLength = 2000; // Probably should be bigger, say 200,000
static int KeepLines = 5; // minimum of how much of the old log to leave

public static void ManageLogs(string strFileName)
{
    try
    {
        FileInfo fi = new FileInfo(strFileName);
        if (fi.Length > iMaxLogLength) // if the log file length is already too long
        {
            int TotalLines = 0;
                var file = File.ReadAllLines(strFileName);
                var LineArray = file.ToList();
                var AmountToCull = (int)(LineArray.Count - KeepLines);
                var trimmed = LineArray.Skip(AmountToCull).ToList();
                File.WriteAllLines(strFileName, trimmed);
                string archiveName = strFileName + "-" + DateTime.Now.ToString("MM-dd-yyyy") + ".zip";
                File.WriteAllBytes(archiveName, Compression.Zip(string.Join("\n", file)));
        }

    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed to write to logfile : " + ex.Message);
    }
}

I have this as part of the initialization / reinitialization section of my application, so it gets run a few times a day.

ErrorLogging.ManageLogs("Application.log");

Chanachance

This code seems sound. But the numbers need to be much larger, like your comment. At 5, your basically emptying the file. – Evictee

Answer 6

I wouldn't use this for a file meant to be over say 1 Meg and it's not terribly efficient, but it works good if you need to solve a pesky problem of when you need a log file that you can't conveniently maintain. Make sure the log file exists before you use this though... or you could add code for it as well as checking the location exists, etc.

// This is how to call it
private void buttonLog_Click(object sender, EventArgs e)
{
    c_Log.writeToFile(textBoxMessages.Text, "../../log.log", 1);
}


public static class c_Log
{
    static int iMaxLogLength = 15000; // Probably should be bigger, say 200,000
    static int iTrimmedLogLength = -1000; // minimum of how much of the old log to leave

    static public void writeToFile(string strNewLogMessage, string strFile, int iLogLevel)
    {
        try
        {
            FileInfo fi = new FileInfo(strFile);

            Byte[] bytesSavedFromEndOfOldLog = null;

            if (fi.Length > iMaxLogLength) // if the log file length is already too long
            {
                using (BinaryReader br = new BinaryReader(File.Open(strFile, FileMode.Open)))
                {
                    // Seek to our required position of what you want saved.
                    br.BaseStream.Seek(iTrimmedLogLength, SeekOrigin.End);

                    // Read what you want to save and hang onto it.
                    bytesSavedFromEndOfOldLog = br.ReadBytes((-1 * iTrimmedLogLength));
                }
            }

            byte[] newLine = System.Text.ASCIIEncoding.ASCII.GetBytes(Environment.NewLine);

            FileStream fs = null;
            // If the log file is less than the max length, just open it at the end to write there
            if (fi.Length < iMaxLogLength) 
                fs = new FileStream(strFile, FileMode.Append, FileAccess.Write, FileShare.Read);
            else // If the log file is more than the max length, just open it empty
                fs = new FileStream(strFile, FileMode.Create, FileAccess.Write, FileShare.Read);

            using (fs)
            {
                // If you are trimming the file length, write what you saved. 
                if (bytesSavedFromEndOfOldLog != null)
                {
                    Byte[] lineBreak = Encoding.ASCII.GetBytes("### " + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " *** *** *** Old Log Start Position *** *** *** *** ###");
                    fs.Write(newLine, 0, newLine.Length);
                    fs.Write(newLine, 0, newLine.Length);
                    fs.Write(lineBreak, 0, lineBreak.Length);
                    fs.Write(newLine, 0, newLine.Length);
                    fs.Write(bytesSavedFromEndOfOldLog, 0, bytesSavedFromEndOfOldLog.Length);
                    fs.Write(newLine, 0, newLine.Length);
                }
                Byte[] sendBytes = Encoding.ASCII.GetBytes(strNewLogMessage);
                // Append your last log message. 
                fs.Write(sendBytes, 0, sendBytes.Length);
                fs.Write(newLine, 0, newLine.Length);
            }
        }
        catch (Exception ex)
        {
            ; // Nothing to do...
              //writeEvent("writeToFile() Failed to write to logfile : " + ex.Message + "...", 5);
        }
    }
}

Mussorgsky