AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Question

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ?

I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication" page of the AWS MSK and found that to be very confusing.

Can anyone help me with steps on this ? I am just trying to deploy this application which uses the AWS MSK (3 brokers).

Thank you.

Epigeal

Answer 1

Short answer: Your kafka client will need this in the configuration:

# security settings
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
ssl.truststore.password=
ssl.endpoint.identification.algorithm=

That is if you use the same JVM truststore from the tutorial, and no password. The ssl.endpoint.identification.algorithm turns off the host name verification.

Long answer: I wondered the same thing after going through the tutorial, wondering why the JVM truststore magically works when connecting to MSK. The explanation is this:

If you take a peek at what certificates this truststore imported

keytool --list -v -keystore /tmp/kafka.client.truststore.jks | grep Owner

One of them is Starfield Services Root Certificate Authority, when Amazon purchased the company, the CA became one of Amazon's (see all of them here https://www.amazontrust.com/repository/). Since JVM truststore trusts this CA, it also trusts anything signed by the CA, and the MSK cluster is one of them.

If you would prefer to generate your own truststore, download one of the Amazon's certificate and import

keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file {downloaded-cert} -storepass {your-password}

Thanks, Yanan

Tiphani

Answer 2

I was looking for the same thing so posting as an answer might help someone else. Here is official documentation that can help.

On Ubuntu machine follow working for me, also you do not need to import/create just used the existing cacerts

cp /home/ubuntu/.sdkman/candidates/java/8.0.232.hs-adpt/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks

create a text file named client.properties with the following contents.

security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks

and we are good to consume message

./kafka-console-consumer.sh --bootstrap-server  mykafka.kafka.us-west-2.amazonaws.com:9094  --consumer.config /home/ubuntu/client.properties --topic demo --from-beginning

Fogbow