kafka failed authentication due to: SSL handshake failed

Question

I have to add encryption and authentication with SSL in kafka.

This is what I have done:

1. Generate certificate for each broker kafka:

keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey

1. Create CA. The generated CA is a public-private key pair and certificate used to sign other certificates. A CA is responsible for signing certificates.

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

2. Sign all brokers certificates with the generated CA Export the certificate from the keystore:

   keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file

   Sign it with the CA:

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}

3. Import both the certificate of the CA and the signed certificate into the keystore:

   keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert

   keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed

4. Import CA to client truststore and broker/server truststore:

   keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

5. Add these line in the configuration server.properties: listeners=PLAINTEXT://localhost:9092, SSL://localhost:9192 ssl.client.auth=required ssl.keystore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.keystore.jks ssl.keystore.password=blablabla ssl.key.password=blablabla ssl.truststore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.truststore.jks ssl.truststore.password=blablabla security.inter.broker.protocol=SSL

The problem is that when I start kafka, then I get this error:

[2019-02-26 19:03:59,783] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2019-02-26 19:04:00,011] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-02-26 19:04:00,178] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-02-26 19:04:00,319] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9192) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

Why?

EDIT: server.properties:

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092

listeners=PLAINTEXT://localhost:9092, SSL://localhost:9192

ssl.client.auth=required
ssl.keystore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.keystore.jks
ssl.keystore.password=onailime
ssl.key.password=onailime
ssl.truststore.location=/home/xrobot/kafka_2.12-2.1.0/certificate/server.truststore.jks
ssl.truststore.password=onailime

security.inter.broker.protocol=SSL

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/home/xrobot/kafka_2.12-2.1.0/data/kafka

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

zookeeper.properties:

# the directory where the snapshot is stored.
dataDir=/home/xrobot/kafka_2.12-2.1.0/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0

Langill

can you share the client configuration too? – Historiographer

@MickaelMaison yes, just done :) – Langill

I think you should remove plaintext listener or put it in another port. – Episode

@Langill I see server.properties and zookeeper.properties but not client properties – Historiographer

@MickaelMaison I don't have a client.properties file in my config folder. Do I have to create it? – Langill

@Langill did you solve the issue ? What did you do to solve the problem ? – Necaise

same issue here, finally did you manage to solve the issue? – Forerun

Answer 1

probably your hostname and your certificate don't match. add this line to your server.properties file.

ssl.endpoint.identification.algorithm=

From Kafka version 2.0.0 onwards, hostname verification of servers is enabled by default for client connections as well as inter-broker connections. by adding this line, you assign an empty string for ssl.endpoint.identification.algorithm.

Viridi

There are different options to disable the hostname validation. You can choose to only disable it for a specific client. See details here cwiki.apache.org/confluence/display/KAFKA/… – Hagfish

This worked for me, for my case hostname and certificate are different. This saved my day! – Paxton

Worked for me too. This property SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG was set to null before the upgrade. And post upgrade, it started failing as ConcurrentHashmap doesn't allow nulls as keys or values. Debug the code to find where exactly the Null Pointer is originating, and then this comment helped fix it. Thank you! – Sanchez

Answer 2

This is an old thread but I can share some lessons learnt the hard way: Authentication failure can happen for a number of reasons. It's necessary to understand what failed in the SSL handshake. A pcap with the SSL handshake messages will definitely help.

If this is is for a client connecting to a broker. In server.properties you have:

ssl.client.auth=required

It should be

ssl.client.auth=none 

if the clients are not authenticating to the server. In the question there is no step described in which the clients are created their own key/certificate.

Also, just for testing purposes, in the client you can configure:

enable.ssl.certificate.verification=false

This property in false makes the client not validate the server's certificate with the CA. It's useful when the SSL hanshake error is due to server's certificate not validated.

Guanaco

Answer 3

This problem can be caused by a connection problem or firewall setting.

You can test access with:
openssl s_client -debug -connect servername:port -tls1_2

Answer should be "Verify return code: 0 (ok)

If not you have no access.

Valleau

Answer 4

I got this error because I had not updated my certificates. Make sure that you have updated certificates and are referring those.

Gudrin

Maybe add more information on how someone would check for expired certificate(s) – Illmannered

In my case I had also neglected to restart the correct service, which should be done after updating the JKS. – Catabolite

Answer 5

Kafka connect initially did not start in my case because of the ssl handshake error. I can sort it with setting the property ssl.endpoint.identification.algorithm=

Thereafter i got similar error when I create a source connector.

[2022-09-07 15:35:18,817] ERROR [Producer clientId=connector-producer-SourceConnector-0] Connection to node -1 (kafkaserver:9092) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2022-09-07 15:35:18,817]

WARN [Producer clientId=connector-producer-SourceConnector-0] Bootstrap broker kafkaserver:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

the solution for this adding identification algorithm parameter for both consumer and producer.

ssl.endpoint.identification.algorithm=
producer.ssl.endpoint.identification.algorithm=
consumer.ssl.endpoint.identification.algorithm=

Eyetooth

Answer 6

Yes this could be happening because of domain name mismatch, but just to add if you have two-way authentication enabled

ssl.client.auth=required

you will have to disable the below flag in both places, broker(server.properties) as well as in you client(producer or consumer) configs

ssl.endpoint.identification.algorithm=""

If your client is java based you can do the following:

props.put(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "");

Skyway

Answer 7

I see a lot of responses here trying to guess how to solve a SSL handshake problem that is not clear. The logs only show

failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

The problem is that we don't know the reason for SSL handshake failure. I created a tool helps you determine what is the real error by looking at the network handshake exchange:

https://github.com/rodolk/dlltagent_tools

Then you can decide whether that was due to an expired certificate, not accepted self-signed certificate, unknown CA or any other of the many possible errors. And after hat you can choose one of all the helpful responses. The tool is prepared for Kubernetes but it can also be used with docker without kubernetes.

Guanaco

Answer 8

In my case, after seeing similar errors, I had to add the host name to the listeners value in my server.properties.

Before:

listeners=SSL://:9093

After:

listeners=SSL://my-host-name:9093

This host name should be the same as the CN-value in your broker certificate. This is the same value you should have for the host.name property.

Ln

The IP in this parameter is the bind address. – Catabolite