Reproducing UnknownTopicOrPartitionException: This server does not host this topic-partition

Question

We have encountered few exception on production environment:

UnknownTopicOrPartitionException: This server does not host this topic-partition

As per my analysis, one possible workaround for this issue is increasing no of retries since this is a retriable exception.

I am facing some difficulties which reproducing this issue locally. I tried bringing down broker while producing but it is failing with TimeoutException.

I am looking for suggestions to reproduce this issue.

Magnetics

This usually happens when partition is electing the new leader. – Merchantman

And as per my understanding that happens when the current leader goes down. Right? So by that logic when I am producing messages and I brings down current broker then I should be able to reproduce this exception? – Magnetics

What's your version of Kafka? Besides, I don't think it's easy to reproduce this issue. – Merchantman

My kafka version is 0.10.0.1. Increasing the no of retries should reduce the possibility of this exception right? – Magnetics

LeaderNotAvailableException might be more possible to be observed on the producer side. UnknownTopicOrPartitionException is more likely thrown by ReplicaFetcherThread. – Merchantman

I have the same issue, all nodes are up , still it thinks other guy is down. I checked, network, hostname etc..seems like restart only solves it for short period of time. – Desma

I saw this log in the broker after killing all brokers as well as zookeeper, but leave the consumers running. Then wipe all zookeeper and kafka logs and restart zookeeper and the brokers. Will see these errors as well as timeouts on the consumer side like: TimeoutException: Timeout of 60000ms expired before the position for partition mylocaltopic could be determined. – Barbee

Answer 1

If you get this error log during topic creation process, there is an open issue for this:

KAFKA-6221 ReplicaFetcherThread throws UnknownTopicOrPartitionException on topic creation

at some point of time during batch creating topics, it's likely that UpdateMetadata requests got processed later than FetchRequest, therefore metadata cache was not updated on a timely basis.

issue was about log messages that have no impact on cluster health.

Berkelium

Answer 2

For me, it was because I had a '.' and an '_' in my topic's name

Ankus

I had the same problem and my topic names contained _ (underscores), removing them fixed my issue. Though my environment was different which can help to reproduce the issue, so I will provide the complete answer. – Warfeld

Answer 3

Make sure your topics are created and accessible.

This happened to me when I was (re)building the project, and just forgot to create topics. Kafka was trying to access topics that don't exist.

Tocharian

Answer 4

I created a Spark Structured Streaming application using pyspark. I was not running any cluster of spark, just launching pyspark application using submit-job script. In my application I am using Kafka streaming endpoints (for both reader + writer). Given are three scenarios.

NOTE: All of my configurations, topic names, partition counts everything was same in all three of these scenarios.

1. pyspark application is running on host machine. Kafka connectors were connecting just fine and my application was running smooth, no issue at all

2. I have created container of pyspark application and when I run my spark application in the container I was seeing this UnknownTopicOrPartitionException error.

3. I have created a containerized version of standalone cluster of pyspark with 2 worker nodes and submitted the job to the master. My job is same pyspark application. I was still seeing this UnknownTopicOrPartitionException error.

What fixed the issue: My topic names contained underscores (_), removing all underscores fixed the issue. my_cool_topic_name --> mycooltopicname

What I have tried: I created new topics, changed partition count, changed different configuration settings of spark structured streaming kafka configuration settings, nothing worked.

My Kafka broker is hosted on DigitalOcean with Version 3.5. I was using Kafka Client for the same version (i.e spark-sql-kafka-0-10_2.12:3.5.0).

Warfeld

Answer 5

Enable the auto creation of topics in your kafka server, the name of the config is:

kafka.auto_create_topics_enable

Terrific