Use Avro in KafkaConnect without Confluent Schema Registry

Question

We have vanilla apache Kafka setup in current infrastructure and we started logging some data that we want to process using Kafka Connect. Currently we use Avro for our message format, but there's no Schema Registry in our infrastructure. In future, we plan to replace current stack with Confluent and use Schema Registry and Connect, but for some time we need to deploy only Connect for that.

Is it possible to configure Connect sinks somehow so they use explicit avsc files or schema without connecting to Schema Registry and without using Confluent format with magic bytes and schema ID?

Englis

Answer 1

Yes, it is possible using the registryless-avro-converter on Github.

Follow the build instructions there, add a JAR to your plugin.path folder as other connectors are loaded, then setup like so

key.converter=me.frmr.kafka.connect.RegistrylessAvroConverter
key.converter.schema.path=/path/to/schema/file.avsc
value.converter=me.frmr.kafka.connect.RegistrylessAvroConverter
value.converter.schema.path=/path/to/schema/file.avsc

Note that this will require you to store/maintain/sync the schema files on all Connect workers, however

---

Alternatively, you can setup the Schema Registry with your vanilla Kafka - No reason to do some "Confluent migration" since the registry doesn't require any infrastructure changes other than your Serializer & Deserializer configs.

Anselma

Is there anychance that schema can be fed as a string or something rather than from file using path – Dossier

I've never used that code. You'd have to post an issue in that repo or look at the code – Anselma

Answer 2

Yes, but you're going to have to implement your own Converter to handle the vanilla avro records. You'd then specify the avsc file's location, or a service that could provide the Schema in the connector configuration using the [value|key].converter namespace.

For instance, the Confluent AvroConverter requires a property telling it where the Schema Registry is:

value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry:8081

So you could copy the Confluent AvroConverter but maybe provide a path to the avsc file?

Antiphonal