How can I delete a topic in Apache Kafka? [duplicate]

Question

I need to delete a topic in Kafka 0.8.2.2.3. I have used the below command for deleting the topic:

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic DummyTopic

The command executed successfully, but when I run a command to list the topics, I could see that the topic is still there and it shows marked for deletion.

bin/kafka-topics.sh --list --zookeeper localhost:2181
DummyTopic - marked for deletion

And when I create the topic DummyTopic, it outputs the exception. The topic already exists, and below is the stack trace:

Error while executing topic command Topic "DummyTopic" already exists.
kafka.common.TopicExistsException: Topic "DummyTopic" already exists.
    at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:248)
    at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:233)
    at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:92)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

How can I delete this topic?

Organotherapy

Apache Kafka never deletes a topic marked for deletion if that topic has producers still producing to it, or consumers still consuming from it, or messages left hanging out in the queue. One way to try to try to force it is to restart Kafka. Or if that doesn't work, go in under the hood and Delete the directory that is your topic name under /var/local/kafka/data and then restart Kafka then re-issue the delete command. I wish Apache Kafka had a "nuke the bleeping topic" option so the developer can issue the command: "Really, totes for real this time nuke the bleeping topic no sass please". – Allx

bin/kafka-topics.sh –delete –zookeeper localhost:2181 –topic <topic-name> – Spoof

Option zookeeper is deprecated, use --bootstrap-server instead. https://mcmap.net/q/182805/-zookeeper-is-not-a-recognized-option-when-executing-kafka-console-consumer-sh – Carrillo

This worked for me (within docker): bin/kafka-topics --delete --topic DummyTopic --bootstrap-server localhost:29092 – Fascist

Answer 1

Deletion of a topic has been supported since 0.8.2.x version. You have to enable topic deletion (setting delete.topic.enable to true) on all brokers first.

Note: Ever since 1.0.x, the functionality being stable, delete.topic.enable is by default true.

Follow this step by step process for manual deletion of topics

1. Stop Kafka server
2. Delete the topic directory, on each broker (as defined in the logs.dirs and log.dir properties) with rm -rf command
3. Connect to Zookeeper instance: zookeeper-shell.sh host:port
4. From within the Zookeeper instance:
   1. List the topics using: ls /brokers/topics
   2. Remove the topic folder from ZooKeeper using: rmr /brokers/topics/yourtopic
   3. Exit the Zookeeper instance (Ctrl+C)
5. Restart Kafka server
6. Confirm if it was deleted or not by using this command kafka-topics.sh --list --zookeeper host:port

Gaiter

ok...thanks...I will try restarting the kafka cluster..:) – Organotherapy

Link is broken! – Oloughlin

Where is the "/brokers/topics" directory? cant seem to find that folder – Beriberi

Use zookeeper-shell.sh to get to zookeeper and ls /brokers/topics – Forgetful

If I check the docs on bin/kafka-topics.sh I see a --delete option. But using it does not work. Is it still in beta? – Roane

NOTE: If u use --delete from kafka-topics.sh ---> make sure there is no CONSUMER running ---> and you set 'delete.topic.enable:true' in server.properties file on all kafka brokers. – Humour

kafka.admin.TopicCommand shows how to do this programmatically. – Zenia

how to delete all topics not just one? – Barb

i cant believe its so difficult to delete a topic – Tavares

@ErikK Is there a better way? – Selfregulating

rmr command is deprecated. you can use deleteall instead of rmr – Schall

I wouldn't recommend this rm -rf approach for prod clusters. The proper way is to enable delete topic in server.properties. Use --delete Wait for the offsets to expire naturally on these topics. The zookeeper shell uninstall works too but causes controller node to change. – Tessietessier

windows users can run zookeeper-shell.bat <zk address>, for example: zookeeper-shell.bat localhost:2821. btw, there is deleteall command – Bayless

I visit this answer every couple of weeks. Sometimes multiple times in a week. – Schizomycete

Don't delete the folders while kafka is still running! (an easy concept to grasp which I am learning currently the hard way how to deal with) – Tangelatangelo

Hmm.. now we just need to use the command mentioned in the query bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic DummyTopic and voila it's done. – Kettle

Note one issue which caught me out was that our kafka cluster is configured to auto create topics = true. I had to change the value of auto.create.topics.enable to false in conf/server.properties without this the topic may be automatically created when data arrives and with the default partition/replicas setup. – Dulcimer