How can I access my AWS MSK managed kafka queue from my local machine and EC2 instances in other regions

Question

I'm setting up a managed kafka queue on AWS MSK. I can't seem to get the security to work when connecting from a local machine and I can't work out if I can use security groups from one region to another.

I've gone through the information on setting up the security groups on the main documentation here. I still can't seem to connect to the broker though. I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm currently using kafka scripts from my local machine in the following way:

bin/kafka-console-producer.sh --broker-list "my-broker-ip:9092" --topic "some-topic"

but keep getting the result

[2019-01-28 12:06:13,278] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've associated my local IP with all the ports on the security group that I set up with my VPC and associated with the kafka queue but it doesn't seem to have helped. I also don't see how I can associate my boxes with the kafka queue as they are in different regions. Is this possible?

I'm expecting to be able to connect my local producer code the kafka queue and observe the output rather than have it constantly reject the connection.

Macaco

Answer 1

As far as I know, you have to access your MSK cluster from a client machine on EC2, and cannot do so from a local machine.

Etymologize

Is this really a limitation? What if I have a producer in my EC2, but my consumers are anywhere outside the AWS environment, maybe an on premise hosted VM or another cloud instance – Algolagnia

hi Robbin, what about SQS or RabbitMQ. Can you publish/subscribes from client on your laptop? Or VM outside AWS? – Rollie

I tried a few things including add inbound/outbound rules (port 9092/9094 on the security group), AWS client VPN, and turning off firewall on my laptop, nothing worked. #68018283 – Rollie

Answer 2

As @Robin mentioned, you cannot access MSK directly from a local machine using kafka client or kafka stream. Because the broker url, zookeeper connection string are private ip's of the msk cluster vpc/subnet. To access through kafka client, you need to launch ec2 instance in the same vpc of MsK and execute kafka client(producer/consumer) to access msk cluster.

But you can set up kafka Rest Proxy framework open-sourced by Confluent to acess the MSK cluster from the outside world via rest api. This framework is not a full-fledgeddged kafka client and it doesn't allow all kafka client operations, but you can do some operations on the cluster like: fetching metadata of the cluster, fetching topic information, producing and consuming messages, etc.

I have answered in detail this scenario along with a few other questions related to MSK, refer-

Amazon Managed Streaming for Kafka- MSK features and performance

Distiller

Answer 3

As @Kuntal-G mentioned, the broker URLs etc. are private IPs and cannot be accessed from outside the VPC. However, you can assign ("public") Elastic IPs to the brokers and adjust the security groups to allow traffic to/from the Zookeeper and Kafka ports as explained in more detail here:

https://www.repetitive.it/aws-msk-how-to-expose-the-cluster-on-the-public-network/

Alternative solutions are also mentioned in the official AWS documentation (e.g. using AWS Transit Gateway or REST proxies):

https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html

Cockoftherock

Answer 4

One solution I can think which we have is using a transit gateway. That way the VPC where AWS MSK resides and the on-premises network where your laptop is present can be interconnected and can be accessed seamlessly.

Also you can take advantage of VPC peering.

Please refer to the below documentation for details.

https://docs.aws.amazon.com/msk/latest/developerguide/client-access.html

Details about Transit gateway

https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html

Integument

Answer 5

You can access your brokers from anywhere by setting up public access to your brokers. From that page:

For security reasons, you can't turn on public access while creating an MSK cluster. However, you can update an existing cluster to make it publicly accessible. You can also create a new cluster and then update it to make it publicly accessible.

The page goes on to list other restrictions on public access, but as long as you adhere to those, it is possible.

Gaiety