How to create a list of topics in Apache Kafka using single command

Question

As of now I am creating a topic one by one by using below command.

sh ./bin/kafka-topics --create --zookeeper localhost:2181  --topic sdelivery --replication-factor 1 --partitions 1

Since I have 200+ topics to be created. Is there any way to create a list of topic with a single command?

I am using 0.10.2 version of Apache Kafka.

Supersensual

Answer 1

This seems like more of a unix/bash question than a Kafka one: the xargs utility is specifically designed to run a command repeatedly from a list of arguments. In your specific case you could use:

cat topic_list.txt | \
  xargs -I % -L1 sh ./bin/kafka-topics \
    --create --zookeeper localhost:2181 \
    --topic % \
    --replication-factor 1 \
    --partitions 1

If you want to do a "dry run" and see what commands will be executed you can replace the sh with echo sh.

Alternatively you can just make sure that your config files have default topic settings of --replication-factor 1 and --partitions 1 and just allow those topics to be automatically created the first time you send a message to them.

Topfull

The answer specifies -I % -L1, but manual says it's not correct combination The options --max-lines (-L, -l), --replace (-I, -i) and --max-args (-n) are mutually exclusive.. – Lunette

Answer 2

You could use Terraform or Kubernetes Operators which will not only help you create topics (with one command), but also manage them later if you did need to delete or modify their configs.

But without custom solutions, and only for purpose of batch-creation, you can use awk for this

Create a file

$ cat /tmp/topics.txt
test1:1:1
test2:1:2

Then use AWK system function to execute the kafka-topics script, and parse the file

awk -F':' '{ system("./bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic=" $1 " --replication-factor=" $2 --partitions=" $3 " ) }' /tmp/topics.txt
Created topic "test1".
Created topic "test2".

And we can see the topics are created

$ ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
test1
test2

Note: Hundreds of topics this quickly might overload Zookeeper, so might help to add a call to "; sleep 10" at the end.

As of Kafka 3.0, the Zookeeper flag is replaced by bootstrap-servers.

Magically