How AWS MSK and Confluent Schema Registry and Confluent Kafka connect recommended to use together?

Question

We are planning to use AWS MSK service for Managed Kafka and Schema Registry and Kafka Connect services from Confluent together to run our connectors (Elasticsearch Sink Connector). We have planned to run Schema Registry and Connectors in EC2.

As per the Confluent team, They could not officially support Confluent Schema Registry and Kafka Connect if we use MSK for Kafka.

So, Anyone can share their experience? like if Anybuddy has used a combination of MSK and Confluent services together in the production environment?

Is there any risk in using this kind of combination?

Is it recommended or not to use this combination?

How is Confluent community support if we will face any issue with Connectors?

Any other suggestions, comments, or alternatives?

We already have a Confluent Corporate Platform license but We want to have managed Kafka service that's why we have chosen AWS MKS as it's very cost-effective than Confluent Cloud as per our analysis?

Kindly please share your thoughts and Thanks in advance.

Thanks

Poppas

I would be very surprised if MSK is actually cheaper (for what you get) than Confluent Cloud, given that you get more services and support with Confluent Cloud – Mady

Answer 1

Objectively answering your question this is something doable but it depends where is your major pain.

From the licensing perspective there is nothing that forces you to have a Confluent subscription just to use Kafka Connect or Schema Registry, as they are based on the Apache License 2.0 and Confluent Community License respectively.

From the technical perspective you can run both Kafka Connect and Schema Registry on EC2 and; as long they are running in the same VPC that the MSK cluster they will work flawlessly.

From the cost perspective you will have to evaluate how much it costs to have Kafka Connect and Schema Registry being managed by you and/or your team. Think not only about the install and setup phase but the manage and evolve phase as well. The software might not have any cost but the effort to operate these components can be translated into cost.

How is Confluent community support if we will face any issue with Connectors?

The Kafka community is usually very helpful whether if you ask for help in the Apache Kafka users group or the community that Confluent owns in Slack. Of course, it is all about best effort and you can't rely on them to get support. It may take several days until some good Samaritan decide to help you. Which also translates to cost: how much costs being down and/or waiting for a resolution?

I am no longer a Confluent employee and therefore I won't even try to convince you to buy from them. But you should evaluate this component of cost and check if using Confluent Cloud wouldn't provide you a more cost effective solution since it includes a managed version of Kafka, Kafka Connect, and Schema Registry. In my experience, the managed Kafka on Confluent Cloud is not that costly and the managed Schema Registry is "free", but using a managed connector can be very costly and it can be worse depending of the number of tasks that you configure in the managed connector. This is the only gotcha that you ought to watch out.

Footfall

Answer 2

AWS MSK now supports fully managed free schema registry service that easily integrates with Kafka and other AWS services like Kinesis, Glue etc. It's much easier to get started with it.

Stannite

AWS MSK schema registry offers limited capabilities compared to Confluent's one (they don't support Protobuf, for example). – Namedropper

As for April 2022 - protobuf is supported – Ypsilanti

In my case a software that we use (Apache Druid) does not support AWS Glue Registry integration for schema lookup. The option was to keep confluent schema registry running on the side with AWS MSK. It technically works but your solution won't be bullet proof supported by AWS and you'l loose valuable tooling such as the Control Center for troubleshooting schema related issues, looking at decoded avro messages, etc. – Eleph

Pretty sure that GLUE doesn't support KSQLdb also, unless someone can answer my open question...#73324149 – Irvingirwin

@Stannite worse part of a Glue is it forces IAM authentication. How am I going to authenticate for the producer or consumer that are not a part of AWS eco system. AWS team need some shakedown. – Numerator