Secure / Encrypt log4j files

Question

I have a problem ; security requirement i have java swing app that have logging files generated with log4j for support issues in case of tracking a bug.

I have to ecrypt/cypher/secure the files so the client cant open them and see them (at least not as human readable way) and at the same time when support tech team take these files they will know how to read (decrypt) them .

I did a lot of searches and i tried my best option i found which is build custom appender by extending SkeletonAppender .

Now know that i have log4j working great as below configuration, but i created new class to encrypt it but i cant get it work even with simple setup it dose not create the file , so i can continue in the ecnryption part.

Any help , links are good.

Working...version

<appender name="cache" class="com.MyAppender">  
            <param name="Threshold" value="ALL" />
            <param name="ImmediateFlush" value="true" />  
            <param name="File" value="${home}/logs/cache.log"/> 
            <param name="Append" value="true"/>
            <param name="Threshold" value="ALL" />
            <param name="Encoding" value="UTF-8" />

            <layout class="org.apache.log4j.EnhancedPatternLayout">
            <param name="ConversionPattern" value="%-5p %d{MMM-dd-yyyy HH:mm:ss,SSS} %c{1} - %m%n" />
        </layout>
    </appender>

Not Working...version

   <appender name="cache" class="com.MyAppender">   
            <param name="Threshold" value="ALL" />
            <param name="ImmediateFlush" value="true" />  
            <param name="File" value="${home}/logs/cache.log"/> 
            <param name="Append" value="true"/>
            <param name="Threshold" value="ALL" />
            <param name="Encoding" value="UTF-8" />

            <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
                 <param name="FileNamePattern"
            value="${home}/logs/cache.%d{yyyy-MM-dd-HH}.gz" />
                 <param name="ActiveFileName" value="${home}/logs/cache.log" />
             </rollingPolicy> 

            <layout class="org.apache.log4j.EnhancedPatternLayout">
            <param name="ConversionPattern"
                value="%-5p %d{MMM-dd-yyyy HH:mm:ss,SSS} %c{1} - %m%n" />
        </layout>
    </appender>

The simple class test

http://www.javaworld.com/article/2075817/core-java/customized-appender-extending-org-apache-log4j-fileappender.html

package com.MyAppender;

import org.apache.log4j.spi.LoggingEvent;

public class MyAppender extends org.apache.log4j.RollingFileAppender {

    private String file;
    private boolean initialized = false;
    private String baseFileName = null;

    // private static final Log log = LogFactory.getLog(MyAppender.class);

    /**
     * 
     * write to ActivityLog
     * 
     * @param event
     *            logging event invoked.
     * 
     */
    @Override
    protected void subAppend(LoggingEvent event) {
        if (!initialized) {
            createNewFile();
        }
        synchronized (this) {
            super.subAppend(event);
        }
    }

    /**
     * 
     * create a new ActivityLog File
     * 
     */
    public void createNewFile() {
        try {
            baseFileName = file;
            super.setFile(baseFileName);
            super.activateOptions();
            initialized = true;
        } catch (Exception e) {
            // log.error("*#*Error in configuration of log4j params,unable to create ActivityLog file");
        }
    }

    /**
     * 
     * invokes File Appender's activateOptions() which controls the creation of
     * log files.
     * 
     */
    @Override
    public void activateOptions() {
        super.setFile(file);
        super.activateOptions();
    }

    /**
     * 
     * Close and rename the current ActivityLog file and reset counter and
     * timestamp.
     * 
     */
    public void rollOver() {
        closeFile();
        initialized = false;
    }

    @Override
    public void setFile(String file) {
        this.file = file;
    }

}

Then i plan to implement the code in Cipher OutputStream

Sirmons

i used the code in javaworld.com/article/2075817/core-java/… and its working but without the rollingPolicy tag in the log4j.xml how to include it *i will update my code * – Sirmons

Rather than mess with an appender, would you consider encrypting the data written into the log, rather than the log itself. In other words, it is critical that everything in the log be encrypted, or is it OK to encrypt only the sensitive data and leave non-sensitive data (eg timestamps etc) in plain text? – County

Thanks @County good idea but look at #630255 it has security breach do you any other good ideas – Sirmons

I forgot to mention its SWING appliaciotn – Sirmons

I think you should implement an Encoder which decorates the OutputStream with your CypherOutputStream: logback.qos.ch/manual/encoders.html – Rationale

Answer 1

A possible workaround to the problem is to write the logs to an embedded database that supports encryption, e.g. H2 natively supports encryption and SQLite has open source encryption extensions - this way you can just use the JDBCAppender and let the database take care of encryption without having to worry about a custom appender.

---

From this question, SQLite config would look something like

<appender name="jdbcAppender" class="org.apache.log4j.jdbc.JDBCAppender">
    <param name="URL" value="jdbc:sqlite:D:/download/mapLogic/sf_log.db" />
    <param name="user" value="" />
    <param name="password" value="" />
    <param name="driver" value="org.sqlite.JDBC" />
    <param name="sql"
        value="INSERT INTO Log(Message,Priority,Logger,Date) VALUES ('%m','%p','%c','%d{ABSOLUTE}')" />
</appender>

where your log table looks like

CREATE TABLE Log (
    LogId        INTEGER PRIMARY KEY,
    Date         DATETIME NOT NULL,
    Level        VARCHAR(50) NOT NULL,
    Logger       VARCHAR(255) NOT NULL,
    Message      TEXT DEFAULT NULL
);

Documentation on the JDBCAppender can be found here

---

There's an official encryption extension for SQLite as well as at least one third party open source extension; I've never had to encrypt SQLite, but if I had to do so then I'd go with the official extension unless I ran into problems with it.

---

If you're running this on the client, then ideally you'll be able to have the program phone home at boot time to get the database encryption key so that the key never exists on the client's disk drive (ignoring the possibility that it goes to the swap file) - the client could still use a debugger or whatever to try to get the key out of memory, but presumably they're not interested enough in decrypting the logs to go to that amount of trouble. If you've got to store the key on the client side then you can at a minimum obfuscate it by hashing it several times before using it, e.g. hard-code the base_key in the program, then at boot time you create actual_key by running base_key through SHA512 (or whatever) several times; the client could still figure out what you're doing by using a debugger, but again they hopefully won't want to go to the trouble.

Ching

thanks for the suggestion but , I forgot to mention its SWING appliaciotn – Sirmons

i dont think its applicable in my case its java swing app , i dont want to add the database just for encryption ... or should i add sql lite at client machine , because i really care about preformance. – Sirmons

@Sirmons With an embedded database you generally handle all operations within your host application - you don't need to install a separate database server, i.e. it would be appropriate/feasible to use an encrypted embedded database on the client's machine. (For example, Firefox uses embedded SQLite to manage cookies/history/etc - it doesn't need to use a separate SQLite server to do this.) – Merrill

you know jdbc with embedded sql looks promosing if you enhance the answer with some links.. to optimize the answer so other people may use too iwill accept the answer thanks – Sirmons

@Sirmons I've expanded my answer – Merrill

Answer 2

Option 1: Use a Custom SocketAppender

As an alternative to Zim-Zam's answer about using a JDBC-capable appender (remember to enable secure transport as well, by the way, if you go down this route), you could also look into using a SocketAppender and roll out your own encryption method.

Option 2: Use Flume and a FlumeAppender

Refer to the log4j documentation on appenders and have a look at using a FlumeAppender, which supports event encryption:

A sample FlumeAppender configuration that is configured with a primary and a secondary agent, compresses the body, formats the body using the RFC5424Layout, and persists encrypted events to disk. This sample "compresses the body, formats the body using the RFC5424Layout, and persists encrypted events to disk:"

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
  <Appenders>
    <Flume name="eventLogger" compress="true" type="persistent" dataDir="./logData">
      <Agent host="192.168.10.101" port="8800"/>
      <RFC5424Layout enterpriseNumber="18060" includeMDC="true" appName="MyApp"/>
      <Property name="keyProvider">MySecretProvider</Property>
    </Flume>
  </Appenders>
  <Loggers>
    <Root level="error">
      <AppenderRef ref="eventLogger"/>
    </Root>
  </Loggers>
</Configuration>

---

Interesting Reads

This does not answer your question directly, but is fairly interesting as well: Creating an encrypted log file

Warlike

Answer 3

I think you are looking something like this. Though I won't suggest to use it. If you are shipping key with the code, there are plenty de-compilers to decompile jar/class files and get "key". Otherwise you should use PKI and all..? I haven't look into that option here.

extend RollingFileAppender class

public class EncryptedRollingFileAppender extends RollingFileAppender

since RollingFileAppender class not final class so you can do.

overwrite method

public synchronized void setFile(String fileName, boolean append,
        boolean bufferedIO, int bufferSize) throws IOException

something like this.

        LogLog.debug("setFile called: " + fileName + ", " + append);

    // It does not make sense to have immediate flush and bufferedIO.
    if (bufferedIO) {
        setImmediateFlush(false);
    }

    reset();
    OutputStream ostream = null;
    try {
        //
        // attempt to create file
        //
        // ostream = new FileOutputStream(fileName, append);
        ostream = this.createEncryptedOutputStream(fileName, append);
    } catch (FileNotFoundException ex) {
        //
        // if parent directory does not exist then
        // attempt to create it and try to create file
        // see bug 9150
        //
        String parentName = new File(fileName).getParent();
        if (parentName != null) {
            File parentDir = new File(parentName);
            if (!parentDir.exists() && parentDir.mkdirs()) {
                // ostream = new FileOutputStream(fileName, append);
                try {
                    ostream = this.createEncryptedOutputStream(fileName, append);
                } catch (Exception e) {
                    e.printStackTrace();
                } 
            } else {
                throw ex;
            }
        } else {
            throw ex;
        }
    } catch (Exception e) {
        throw new FileNotFoundException();
    }
    Writer fw = createWriter(ostream);
    if (bufferedIO) {
        fw = new BufferedWriter(fw, bufferSize);
    }
    this.setQWForFiles(fw);
    this.fileName = fileName;
    this.fileAppend = append;
    this.bufferedIO = bufferedIO;
    this.bufferSize = bufferSize;
    writeHeader();
    LogLog.debug("setFile ended");

    if (append) {
        File f = new File(fileName);
        ((CountingQuietWriter) qw).setCount(f.length());
    }

most of the code is copied from base class.

encryption private method should look something like this

    private OutputStream createEncryptedOutputStream(String filename, boolean append) throws FileNotFoundException, 
                                                                            NoSuchAlgorithmException, 
                                                                            NoSuchPaddingException, 
                                                                            InvalidKeyException, 
                                                                            InvalidAlgorithmParameterException {
    CipherOutputStream cstream = null;

    try {
        byte[] keyBytes = "1234123412341234".getBytes();  //example
        final byte[] ivBytes = new byte[] { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 
                 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f }; //example

        final SecretKey key = new SecretKeySpec(keyBytes, "AES");
        final IvParameterSpec IV = new IvParameterSpec(ivBytes);
        final Cipher cipher = Cipher.getInstance("AES/CFB8/NoPadding");
        cipher.init(Cipher.ENCRYPT_MODE, key, IV);

        cstream = new CipherOutputStream(new FileOutputStream(filename, append), cipher);
    } catch (FileNotFoundException e) {
        throw e;
    }
    return (cstream);
}

You won't loose any capabilities of RollingFileAppender. You can do similar coding @ other appenders as well.

Disclaimer :- Please don't these keys @ production, if you are using it. Again, you are shipping keys with your jar. A smart hacker can hack things easily.Fine tune of code to be done since it is 'logging'. I haven't tested this code.

Colza