Log4J2 dynamic appender doesn't work with maven-shade-plugin

Question

I add an appender programmatically. It was working until I added maven-shade-plugin. I wonder what makes my appender fail.

The appender works ✅ or not ❌ in these scenarios:

1. ✅ From IDE (IntelliJ IDEA)
2. ❌ With the shade (Uber/fat) jar <-- now works ✅ see answer
3. ✅ With separate jars (app jar, log4j jars)
4. ❌ With the app jar, and the log4j jars unzipped into a folder
5. ✅ With the app jar, and the log4j folder re-zipped

Sample project

• https://github.com/fmaylinch/log4j2-thread-context
• Here's the point where I add the custom appender: ThreadContextFileLogHandler.java#L79
• When the appender fails, it's because it doesn't receive the events here: ThreadContextFileLogHandler.java#L156

Reproduce scenarios with the sample project above

mvn clean compile
mkdir -p local/log4j-jars
unzip $HOME/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar -d local/log4j-jars
unzip -o $HOME/.m2/repository/org/apache/logging/log4j/log4j-core/2.17.2/log4j-core-2.17.2.jar -d local/log4j-jars
cd local/log4j-jars
zip -r ../log4j-jars.zip .
cd ../..

# Scenario 2 ❌ uses fat jar
java -cp "target/log4j-test-1.0-SNAPSHOT.jar" org.example.Main

# Scenario 3 ✅ uses separate jars
java -cp "target/original-log4j-test-1.0-SNAPSHOT.jar:$HOME/.m2/repository/org/apache/logging/log4j/log4j-core/2.17.2/log4j-core-2.17.2.jar:$HOME/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.jar" org.example.Main

# Scenario 4 ❌ uses log4j files unzipped
java -cp "target/original-log4j-test-1.0-SNAPSHOT.jar:local/log4j-jars" org.example.Main

# Scenario 5 ✅ uses log4j files re-zipped
java -cp "target/original-log4j-test-1.0-SNAPSHOT.jar:local/log4j-jars.zip" org.example.Main

Extra notes

In the scenario 5, I have noticed that I can remove some files in META-INF, but for my appender to work, I need to keep the following:

• META-INF
  • org (contains Log4j2Plugins.dat)
  • services (without this, the app even crashes)
  • versions
  • MANIFES.MF

Related questions

• log4j2 JsonTemplateLayout not working with maven shade plugin - this seems to be a problem with Log4j2Plugins.dat, but in my project there's only one file, in log4j2-core.
• is there any diff between run a java program with jar or with a package unpacked? - according to my problem, there IS a difference (maybe the META-INF folder is not processed like in a jar)

Masto

Does this answer your question? failing to load log4j2 while running fatjar – Kermes

@PiotrP.Karwasz thanks so much for your comment. It didn't solve my problem but it leaded me to some useful information. I will post the solution I found. – Masto

Answer 1

The problem with maven-shade-plugin is that it breaks the manifest of the original jars and overwrites important resources. I find the spring-boot-maven-plugin much more useful and it can be also used by applications that don't use Spring at all.

The maven-shade-plugin in the context of Log4j requires a minimal configuration as in this question:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <dependencies>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-transform-maven-shade-plugin-extensions</artifactId>
      <version>0.1.0</version>
    </dependency>
  </dependencies>
  <configuration>
    <transformers>
      <transformer implementation="org.apache.logging.log4j.maven.plugins.shade.transformer.Log4j2PluginCacheFileTransformer"/>
      <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
      <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
        <manifestEntries>
          <Multi-Release>true</Multi-Release>
        </manifestEntries>
      </transformer>
    </transformers>
  </configuration>
</plugin>

This configuration takes care of:

• merging Log4j2Plugins.dat files. Without Log4j2PluginCacheFileTransformer you can not use additional component libraries except log4j-api and log4j-core,
• merging service files. Without ServiceResourceTransformer you'll lose additional component like property sources,
• marking the JAR as multi-release jar: some classes used to gather caller's information were replaced in JDK 11. Therefore some Log4j classes have two versions (JDK 8 and JDK 9+). If you don't mark the JAR as multi-release, it will not work on JDK 11+.

Edit: All these problems with the maven-shade-plugin sum up to one: every time two jars have a file with the same name, it must be somehow merged.

That is why I prefer the spring-boot-maven-plugin: instead of breaking multiple jars and adding their files into a single archive, it adds the original jars to the archive. The exact structure of the resulting jar is described in executable Jar format.

The usage is straightforward: just add the repackage goal to your build and remove maven-shade-plugin.

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <version>2.7.10</version>
    <executions>
        <execution>
            <goals>
                <goal>repackage</goal>
            </goals>
        </execution>
    </executions>
</plugin>

This will effectively add the small spring-boot-loader to your application. Version 2.x of the library requires Java 8, while version 3.x requires Java 17.

Kermes

Could you add an example of spring-boot-maven-plugin? Does it have the problem with Log4j2Plugins.dat. Is it anyway necessary to use Log4j2PluginCacheFileTransformer? – Masto

Answer 2

After more investigation, I added a new commit to the sample project log4j2-thread-context with the fixes, so now it works fine.

I also added another fix related to Log4j2Plugins.dat (see failing to load log4j2 while running fatjar), that I didn't mention in my question.

In summary:

• I add the required Multi-Release entry to MANIFEST.MF using ManifestResourceTransformer.
• I copy the correct Log4j2Plugins.dat using IncludeResourceTransformer.

Maybe someone can explain about the Multi-Release entry.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.4.1</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <filters>
            <!-- Ignore these files, we will write the right one below -->
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>**/Log4j2Plugins.dat</exclude>
                </excludes>
            </filter>
        </filters>
        <transformers>
            <!--
              Copies the right Log4j2Plugins.dat file.
              You need a copy of the right Log4j2Plugins.dat file from log4j2-core into src/main/resources.
              You can get that file with these commands:
                log4j2_version=2.17.2
                rm src/main/resources/Log4j2Plugins-*.dat
                unzip -j $HOME/.m2/repository/org/apache/logging/log4j/log4j-core/${log4j2_version}/log4j-core-${log4j2_version}.jar META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat -d src/main/resources
                mv src/main/resources/Log4j2Plugins.dat src/main/resources/Log4j2Plugins-${log4j2_version}.dat
            -->
            <transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
                <resource>META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat</resource>
                <file>src/main/resources/Log4j2Plugins-${log4j2.version}.dat</file>
            </transformer>

            <!-- Adds the Multi-Release entry in MANIFEST.MF -->
            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                <mainClass>org.example.Main</mainClass>
                <manifestEntries>
                    <Multi-Release>true</Multi-Release>
                </manifestEntries>
            </transformer>
        </transformers>
    </configuration>
</plugin>

About Log4j2Plugins.dat, an alternative to IncludeResourceTransformer is filtering all the versions except the one in log4j-core. For example:

<filter>
    <artifact>io.github.technologize:*</artifact>
    <excludes>
        <exclude>**/Log4j2Plugins.dat</exclude>
    </excludes>
</filter>
<!-- filter other artifacts if needed -->

I don't like either solution because:

• If you add a copy of the file, you need to update it in case you update Log4j2. You may use this 3rd party plugin, but then you rely on a 3rd party plugin.
• If you add filters, you need to modify them in case you modify the dependencies (e.g. maybe you add a new dependency that contains another version of this dat file).

Masto

The Log4j2Plugins.dat files need to be merged. Removing a copy of Log4j2Plugins.dat prevents Log4j from discovering the plugins contained in that jar. You might as well remove the jar itself. – Kermes

@PiotrP.Karwasz thanks so much for your answer. My solution works too, but maybe because I am not using classes from other Log4j2Plugins.dat files. I will keep that in mind. – Masto

I am sure it works to some extent, but it prevents you from using all the Log4j 2.x plugins on the classpath. BTW: Eduard's transformer has been donated to Log4j project a while ago (cf. PR #2), we just didn't have the time to publish it yet. We should be able to do it soon. – Kermes