I am using python-kafka to listen to a kafka topic and use that the records. I want to keep it polling infinitely without any exit. This is my code below: def test(): consumer = KafkaConsumer('abc...

Using Kafka with the Java lib, I want to disable the automatic creation of topic (if it doesn't already exist). Some sites says I should put auto.create.topics.enable to false, but this is not rec...

I am using Apache Kafka for messaging. I have implemented the producer and consumer in Java. How can we get the number of messages in a particular topic?

Assuming that I have a number of topics with the same prefix, e.g: giorgos-topic1 giorgos-topic2 giorgos-topic3 ... The command used for deleting a single topic (say giorgos-topic1) is the followi...

I am learning Apache Kafka and I do not understand how to make kafka-topics.sh work with configured SASL_PLAINTEXT authentication on the server. This is a server.properties content: security.protoc...

There is a kafka topic with 16 partitions With a given consumer group name, we are currently launching single consumer to read from the topic. Does single consumer read from partition 0(only) of ...

I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker. As per the text I understood, a segment will not become eli...

suppose I have a kafka topic with say about 10 partitions, I understand that every consumer group should have 10 consumers reading from the topic at any given time to achieve maximum paralellism. ...

Log compacted topics are not supposed to keep duplicates against the same key. But in our case, when a new value with the same key is sent, the previous one isn't deleted. What could be the issue? ...

Can I delete only ONE message from a topic if I know the topic, the offset and the partition? And if not is there any alternative?

In many articles, I've read that compacted Kafka topics can be used as a database. However, when looking at the Kafka API, I cannot find methods that allow me to query a topic for a value based on ...

I would like to implement in Apache Flink the following scenario: Given a Kafka topic having 4 partitions, I would like to process the intra-partition data independently in Flink using different l...

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I...

Are kafka leaders partitions themselves or are they brokers? My initial understanding was that they were partitions which acted as read/write agents which then deffered their value to ISRs. Howeve...

I've written a test class for kafka stream application as per https://kafka.apache.org/24/documentation/streams/developer-guide/testing.html , the code for which is import com.EventSerde; import or...

I am learning Kafka and trying to create a topic for my recent search application. The data being pushed to kafka topics is assumed be a high number. My kafka cluster have 3 brokers and there are...

I have Kafka setup on my local machine and have started the zookeeper and a single broker server. Now i have a single topic with following description: ~/Documents/backups/kafka_2.12-2.2.0/data/k...

I hava kafka cluster having 3 brokers and a couple of topics with each having 5 partitions. Now i want to set the replication factor for the partitions. What is the maximum replication factor whic...

There are documents such as the ones below stating that Kafka topic-level configuration changes can be taken place while Kafka cluster is still running. https://docs.confluent.io/current/kafka/dy...

I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing: ...

I have a 3 member kafka-cluster setup, the __consumer_offsets topic has 50 partitions. The following is the result of describe command on: root@kafka-cluster-0:~# kafka-topics.sh --zookeeper loca...

I am new to Kafka and trying to create a new topic on my local machine. I am following this link. Here are the steps which i followed: Start zookeeper bin/zookeeper-server-start.sh config/zook...