Kafka Connect failed to start

Question

I installed kafka confluent oss 4.0 on a fresh linux centos 7 but kafka connect failed to start.

Steps to reproduce :

 - Install Oracle JDK 8
 - Copy confluent-4.0.0 folder on opt/confluent-4.0.0
 - Run /opt/confluent-4.0.0/confluent start

Result :

Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
\Kafka Connect failed to start
connect is [DOWN]

Error Log (connect.stderr) :

Exception in thread "main" java.lang.NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:54)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.newInstance(Class.java:412)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:279)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:260)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)
Caused by: java.lang.ClassNotFoundException: io.confluent.connect.storage.StorageSinkConnectorConfig
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 22 more

Additional informations :

Java version :

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode

Centos version :

centos-release-7-4.1708.el7.centos.x86_64

[Edit : 30/11/2017] Editing plugin.path variables in every properties files didn't fix the problem.

List of files containing 'plugin.path' variable :

./etc/schema-registry/connect-avro-distributed.properties:84:plugin.path=/opt/confluent-4.0.0/share/java
./etc/schema-registry/connect-avro-standalone.properties:51:plugin.path=/opt/confluent-4.0.0/share/java
./etc/kafka/connect-distributed.properties:95:plugin.path=/opt/confluent-4.0.0/share/java
./etc/kafka/connect-standalone.properties:50:plugin.path=/opt/confluent-4.0.0/share/java

Jocund

Answer 1

With Confluent 4.0.0, classloading isolation with plugin.path is enabled by default for Kafka Connect.

When you install Confluent Platform from deb or rpm packages the default location of your plugin.path is known beforehand.

However, when you download and extract the zip or tar.gz archive of Confluent Platform somewhere in your filesystem, it's set to:

plugin.path=share/java

This is a relative path, because when you download Confluent Platform as an archive (zip or tar.gz), the location where you extract the archive is not known (in your example above it's /opt/confluent-4.0.0/).

The CLI or Connect's bin scripts will be able to guess this location if you run it from the directory where you extracted Confluent platform:

For instance, in the example above:

cd /opt/confluent-4.0.0

./bin/confluent start

In order for you to be able to start Connect from any directory within your filesystem, given that the bin directory for Confluent Platform is in your PATH, you will need to set the property plugin.path to the absolute path location of your plugins:

To use Confluent CLI edit:

etc/schema-registry/connect-avro-distributed.properties

and set your plugin.path appropriately (here: plugin.path=/opt/confluent-4.0.0/share/java)

For the regular bin scripts edit:

./etc/kafka/connect-distributed.properties

and

./etc/kafka/connect-standalone.properties

and set your plugin.path as above (again, in your example: plugin.path=/opt/confluent-4.0.0/share/java).

Olvan

Thanks for your time and your answer. I edited all properties files within ./etc ,destroyed the confluent instance and restarted everything with ./bin/confluent start but nothing change, all services are UP but connect is still down with the same error. – Jocund

List all files with plugin.path : ./etc/schema-registry/connect-avro-distributed.properties:84:plugin.path=/opt/confluent-4.0.0/share/java ./etc/schema-registry/connect-avro-standalone.properties:51:plugin.path=/opt/confluent-4.0.0/share/java ./etc/kafka/connect-distributed.properties:95:plugin.path=/opt/confluent-4.0.0/share/java ./etc/kafka/connect-standalone.properties:50:plugin.path=/opt/confluent-4.0.0/share/java – Jocund

Is there a symbolic link from share/java/kafka-connect-s3/storage-common to ../kafka-connect-storage-common that is valid? Similar link should be present in kafka-connect-hdfs. – Olvan

Answer 2

Given that you used the tar installation (rather than the Docker image approach). Long story short, you need to be inside of the Confluent distribution, in my example, confluent-6.1.0
As shown in the screenshot, when you run the command confluent local services start in the root directory, Connect failed. And anything after that failure (e.g. ksqlDB, Control Center etc.) didn't even get a chance to start. When you run the same command inside of confluent-6.1.0, everything worked out.

Cacodyl