Add jar to general Kafka Connect classpath in the Confluent Docker

Question

I am using the confluent docker for Kafka connect v4.1.1 and would like to add a jar with specific log4j appenders.
With the classpath isolation by connector, I am not sure where to place that jar in the confluent docker, as it is used by the parent kafka connect itself and not the connectors.

Thanks in advance for the help!

Garnish

What happens when you just pick any directory for the plugin.path? – Rainey

Then I get a class not found – Garnish

Answer 1

If you want to add a jar that will be used by the Connect framework and is not specific to any of the connectors, you have the following options:

1. Add the jar to the classpath with:

   -e CLASSPATH=/path/to/custom/library.jar

2. Copy the jar directly to the directory that includes the Kafka Connect framework. In Confluent docker images this is currently:

   /usr/share/java/kafka

You need to make sure this jar is not in conflict with any of the dependencies of Kafka Connect.

Boole

I remember trying no.2 but without success, will see if I did anything wrong and also give no.1 a shot. Thanks! – Garnish

-e CLASSPATH=/path/to/custom/library.jar goes where? – Ribbon

Answer 2

For me, it didn't work on /usr/share/java/ and for what I've seen, the Kafka Connect Docker uses a different path in Standalone than in Distributed mode.

Running something like ps aux | grep java in your running instance will show you the paths it's loading, you'll see something like: java -cp /usr/share/java/cp-base-new/*

In Standalone it's using /usr/share/java/cp-base-new/

In Distributed mode it uses more paths, one of them is /etc/kafka-connect/jars/

Caste