Kafka: disable create topic from Java

Question

Using Kafka with the Java lib, I want to disable the automatic creation of topic (if it doesn't already exist).

Some sites says I should put auto.create.topics.enable to false, but this is not recognized in Java.

15:11:56.962 [main] WARN  o.a.k.c.consumer.ConsumerConfig -  The configuration 'auto.create.topics.enable' was supplied but isn't a known config.

Currently I put it in my docker-compose as environment variable:

KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

This works, but I'd like to manag it from Java by user.

Is this possible?

Icebound

As it is a broker level config you need to put into server.properties to make it work – Caddaric

Answer 1

It is Broker level config. You can't let user manage this from the java client programs.

As auto.create.topics.enable property is read-only broker configuration, which requires the kafka services to be restarted. Hence it is not possible to handle it from client side.

You can read the configurations here with the dynamic update mode : https://kafka.apache.org/documentation/#brokerconfigs

Culicid

Answer 2

Spring has the spring.cloud.stream.kafka.binder.autoCreateTopics property that defaults to true and works independently of the Broker property auto.create.topics.enable. I'm guessing it's only effective if your Broker config allows clients to create topics. It's probably safer to block that permission than to rely on clients to behave.

https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html

Unceasing

Answer 3

Starting with version 2.3.0 Kafka has allow.auto.create.topics config option available for consumers along with auto.create.topics.enable config option for brokers. Interactions of these options is described in details in the corresponding KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation

TLDR: if you have auto.create.topics.enable set to true for your brokers and you can't or don't want to change that, you can set allow.auto.create.topics to false for your consumers and this will prevent Kafka from automatically creating new topics with default settings when you try to subscribe to the non-existent topic for the first time.

Bobstay