Upgrade Apache Solr 8.10.1 log4j 2.14.1 version to 2.15 to address critical vulnerability

Question

Recently a critical log4j vulnerability was discovered.

I want to upgrade the log4j as used by my current Solr instance, so I checked here. However, I don't see a log4j.properties file in "/server/resources/" folder. All I see there is:

• jetty-logging.properties
• log4j2.xml
• log4j2-console.xml

None of these files contain a version. So to upgrade, is it safe to download the latest version of log4j and overwrite the existing jars in folder "\solr-8.10.1\server\lib\ext", or what are the recommended steps to upgrade?

Bouse

the files containing the version should be in your directories server/lib/ext and contrib/prometheus-exporter/lib and these - all JARs - need to be replaced. – Querist

Official info is now available: solr.apache.org/… – Frighten

Answer 1

The link you're pointing to is for an older version of Solr (6.6 instead of 8.10.1). The correct version is https://solr.apache.org/guide/8_10/configuring-logging.html where it mentions using log4j 2.

The file log4j2.xml (and even `log4j.properties for that matter) configure the logging itself, not the version of log4j. So updating that file is irrelevant.

Here's what the project page recommends:

2021-12-10, Apache Solr affected by Apache Log4J CVE-2021-44228

...

Description: Apache Solr releases prior to 8.11.1 were using a bundled version of the Apache Log4J library vulnerable to RCE. For full impact and additional detail consult the Log4J security page.

...

Mitigation: Any of the following are enough to prevent this vulnerability for Solr servers:

• Upgrade to Solr 8.11.1 or greater (when available), which will include an updated version of the log4j2 dependency.
• Manually update the version of log4j2 on your runtime classpath and restart your Solr application.
• (Linux/MacOS) Edit your solr.in.sh file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"
• (Windows) Edit your solr.in.cmd file to include: set SOLR_OPTS=%SOLR_OPTS% -Dlog4j2.formatMsgNoLookups=true
• Follow any of the other mitgations listed at https://logging.apache.org/log4j/2.x/security.html

What you're proposing (overwrite the existing jars in folder "\solr-8.10.1\server\lib\ext") seems like the second approach, so it should probably work fine. Just make sure this is the correct place that contains the log4j dependency.

Mikaelamikal

Thanks. I removed the old jars and replaced with newer ones. Is there a way to validate everything is safe now? – Bouse

@Bouse As long as no old jar is present in your installation and only the 2.15 new version, it should be fine. This tool can also help, but haven't tried it. – Mikaelamikal

@Bouse you can put a TCP listener (e.g. ncat) on a specific port and submit a query with the offending string to validate that Solr is not making the connection as it logs. hectorcorrea.com/blog/log4shell-and-solr/88 – Madonnamadora

Followed the instructions here. But after restart I keep getting SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.. Any suggestions on what else I can do to fix it? – Manhunt

Ok solved it by adding slf4j-simple-*.jar as well in the classpath – Manhunt

Answer 2

Replace all the jars with their corresponding 2.15 version in lib/ext and restart solr. Seem to work.

1. log4j-api-2.11.0.jar
2. log4j-core-2.11.0.jar
3. log4j-1.2-api-2.11.0.jar
4. log4j-slf4j-impl-2.11.0.jar

Skulduggery

did you just replace them without renaming the file or did you rename all 2.15 to 2.11? – Querist

From where you get all the specific jars? – Grossman

@Grossman see logging.apache.org/log4j/2.x/download.html – Querist

Will it effect any dependency with solr? – Niels

Thanks for sharing this answers. It saves my time !! – Autotomize

Answer 3

I updated all of my log4j jars from 2.11 to 2.15 in my /opt/solr-7.4.0/server/lib/ext folder and I am not seeing any issues. Your approach seems to work.

Hellenist

did you just replace them without renaming the file or did you rename all 2.15 to 2.11? – Querist

I deleted all files with the suffix of 2.11 and added files with suffix 2.15. I did not rename any files. I just added and deleted. – Hellenist

Answer 4

With the new CVE SOLR_OPTS is not enough. Upgrading to 2.17 seems like the best option. Here's what worked for us.

sudo find / -name "*log4j*"
cd /app/solr/solr-7.7.0/server/lib/ext/
sudo curl https://dlcdn.apache.org/logging/log4j/2.17.0/apache-log4j-2.17.0-bin.zip --output apache-log4j-2.17.0-bin.zip
sudo unzip apache-log4j-2.17.0-bin.zip
sudo rm log4j-*-2.11*
sudo cp apache-log4j-2.17.0-bin/{log4j-1.2-api-2.17.0.jar,log4j-api-2.17.0.jar,log4j-core-2.17.0.jar,log4j-slf4j-impl-2.17.0.jar} ./
sudo chown solr:solrgrp *
sudo chmod 755 *
sudo service solr restart

cd /app/solr/solr-7.7.0/contrib/prometheus-exporter/lib/
sudo rm log4j-*-2.11*
sudo cp /app/solr/solr-7.7.0/server/lib/ext/apache-log4j-2.17.0-bin/{log4j-api-2.17.0.jar,log4j-core-2.17.0.jar,log4j-slf4j-impl-2.17.0.jar} ./
sudo chown solr:solrgrp *
sudo chmod 755 *
sudo service solr restart

Penzance

Answer 5

For a Solr version 8.6.4 I did now an update to log4j2 2.16 by simply exchanging the 5 log4j*.jar files under "Solr\server\lib\ext". In addition I also had to update slf4j-api.jar to version 1.7.24. This seems to work out of the box.

download links:

• https://logging.apache.org/log4j/2.x/download.html
• https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.9/

-> updated lib files

Contemplate

Answer 6

Mitigation: Any of the following are enough to prevent this vulnerability for Solr servers:

• Upgrade to Solr 8.11.1 or greater (when available), which will include an updated version (>= 2.17.0) of the Log4J dependency.

• If you are using Solr's official docker image, it has already been mitigated in all versions listed as supported on Docker Hub: https://hub.docker.com/_/solr. You may need to re-pull the image.

• Manually update the version of Log4J on your runtime classpath and restart your Solr application.

• (Linux/MacOS) Edit your solr.in.sh file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"

• (Windows) Edit your solr.in.cmd file to include: set SOLR_OPTS=%SOLR_OPTS% -Dlog4j2.formatMsgNoLookups=true

• Follow any of the other mitgations listed at https://logging.apache.org/log4j/2.x/security.html

Refering to: https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228

In my case the following jars were affected and replaced by the latest package directly from Apache. Use something like the log4j detector to identify affected jars.

• %SOLRINST%\solr\contrib\prometheus-exporter\lib\log4j-api-2.11.2.jar
• %SOLRINST%\solr\contrib\prometheus-exporter\lib\log4j-core-2.11.2.jar
• %SOLRINST%\solr\contrib\prometheus-exporter\lib\log4j-slf4j-impl-2.11.2.jar
• %SOLRINST%\solr\server\lib\ext\log4j-1.2-api-2.11.2.jar
• %SOLRINST%\solr\server\lib\ext\log4j-api-2.11.2.jar
• %SOLRINST%\solr\server\lib\ext\log4j-core-2.11.2.jar
• %SOLRINST%\solr\server\lib\ext\log4j-slf4j-impl-2.11.2.jar
• %SOLRINST%\solr\server\lib\ext\log4j-web-2.11.2.jar

Frighten

Watch out! Although the SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" will mitigate the specific vulnerability, there are still other attacks possible ( infoq.com/news/2021/12/log4j-zero-day-vulnerability ). You should upgrade Log4j to at least 2.17.1. – Doubleteam

Answer 7

If you are running Solr as a docker, what you can do is mount the same log4j-core version that uses in Solr image, outside after removing JndiLookup.class from the jar.

Let me go through the steps that I have used.

• Find the log4j-core version that Solr image is using. You can do it by executing following command in your Solr running host machine or after go inside your Solr container

  find / -name "log4j-core-*.jar"

• There you will get two paths. According to Solr security news page, we can neglect the prometheus-exporter path. What will remain is server/lib/ext/ path

• Copy that particular .jar file or download the same version from the Internet. I have tested this with log4j-core-2.11.2.jar

• Copy that jar file in to your Solr container running host machine and check it's vulnerability status using this log4j-detector.

• It's nothing complicated, to check that vulnerability using above mentioned tool. What you need to do is download that log4j-detector-2021.12.16.jar file to an accessible location and run the command against your log4j-core jar file.

  java -jar log4j-detector-2021.12.16.jar 8.4.1-ext > hits_8.4.1_ext.txt

Then you will get an output like below by saying it is vulnerable.

/ext/log4j-core-2.11.2.jar contains Log4J-2.x >= 2.10.0 VULNERABLE :-(

• Now let's remove the JndiLookup.class from that log4j-core jar file

  zip -q -d 8.4.1-ext/ext/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

• Then, check the vulnerability again against the same log4j-core jar file and you will be able to see something like below.

/ext/log4j-core-2.11.2.jar contains Log4J-2.x <= 2.0-beta8 POTENTIALLY_SAFE :-| (or did you already remove JndiLookup.class?)

That means JndiLookup.class removal has done successfully.

• Let's mount that log4j-core jar file using using docker-compose.yaml

  volumes:

  - ./log4j-core-2.11.2.jar:/opt/solr-8.4.1/server/lib/ext/log4j-core-2.11.2.jar

• Now, restart your Solr docker container.

---

My personal preference is updating property with SOLR_OPTS with environment variables, since it is nice and clean.

Loritalorn