Connect to MongoDB atlas within AWS VPC using AWS MSK connector

Question

I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services.

I created an AWS MSK cluster within the VPC and created an AWS MSK connect using mongodb Kafka connector. I am getting the following error (Exception opening socket, connect timed out)

To test whether it has to do with VPC, I gave the public access to MongoDB by whitelisting 0.0.0.0, and then the connector is working fine. So I believe it has something to do with the VPC.

I tried multiple times with AWS MSK with VPC with public subnets, NAT gateway, etc, and it is still not working. Am getting the same error. I believe the AWS MSK connecter uses the same VPC configuration as AWS MSK since there's no separate configuration for AWS MSK Connect.

Ganja

Have you checked mongo db's security group, it has to allow MSK connect security group for mongo's port? if mongo is not on AWS then you have to NAT gateway in a subnet where you have internet gateway enabled for mongo's IP. And then configure route table of your MSK subnet to use NAT gateway for mongo's IP – Herpetology

Is the VPC peered? Or is the mongodb atlas cluster public? I use AWS private link to connect my AWS VPC to MongoDb atlas. Also you need to create an user and give individual access to all databases that you want to connect as source via the connector. – Andean

Did you solve this? What was it? – Farmland

No haven't solved it yet, I had to reprioritize some other work, I will get back to this after some time. I will update here if I make it work – Ganja

Answer 1

The answer is to create a Private Endpoint in Atlas then a VPC endpoint in AWS.

There is a section in this guide that explains how to set up the private endpoint: https://www.mongodb.com/developer/languages/java/integrating-mongodb-amazon-apache-kafka/?bpass=true#create-mongodb-atlas-cluster

For me even though we had a peering connection set up between AWS and Atlas with all of the correct routes etc, we still had to go via this private endpoint option, it's a limitation of the MSK I believe

Iquique