I am unable to integrate security to Kafka 1.0 with ssl enabled. Here are changes to my server.properties
security.inter.broker.protocol=SSL
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9094
Here is my advertised listeners
advertised.listeners=PLAINTEXT://EXTERNAL_IP:9092,SSL://EXTERNAL_IP:9094
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=PASSWORD
ssl.client.auth=required
ssl.keystore.type=JKS
ssl.truststore.type=JKS
Other Configurations include
broker.id=1
advertised.host.name=EXTERNAL_IP_ADDRESS
host.name=0.0.0.0
num.network.threads=3
num.io.threads=8
auto.create.topics.enable=false
min.insync.replicas=2
log.dirs=/kafka1,/kafka2
num.partitions=10
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2
log.retention.hours=24
log.retention.bytes=200073741824
zookeeper.connect=BROKER1_INTERNAL_IP:2181,BROKER2_INTERNAL_IP:2181,BROKER3_INTERNAL_IP:2181
security.inter.broker.protocol=SSL
I did the same on my 3 brokers and always only two brokers are getting started and the third brokers is throwing many "Could Not Established" messages. As an example, broker-1 and broker-3 appears to get started
[2018-04-12 13:50:00,406] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)
[2018-04-12 13:49:57,942] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)
But Server 2 is throwing these
[2018-04-12 13:58:34,247] WARN [Controller id=2, targetBrokerId=1] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-12 13:58:34,254] WARN [Controller id=2, targetBrokerId=3] Connection to node 3 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-12 13:58:34,349] WARN [Controller id=2, targetBrokerId=2] Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
If I remove security.inter.broker.protocol=SSL everything works. But there is no security.
Can anyone please guide me in resolving this ?
I remove PLAINTEXT and now I am getting this error in one of 3 brokers
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:435)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:301)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:255)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:79)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:460)
at org.apache.kafka.common.network.Selector.poll(Selector.java:398)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:91)
at kafka.server.ReplicaFetcherThread.fetchEpochsFromLeader(ReplicaFetcherThread.scala:312)
at kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:130)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:389)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:469)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:328)
... 11 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 26 more