kafka-python Questions
1
We use schema registry for storing schemas, and messages are serialised to avro and pushed to kafka topics.
Wanted to know, when reading data from consumer, how to find the schema id, for which t...
Wellbeloved asked 6/5, 2020 at 9:14
5
I am using Kafka 0.8.1 and Kafka python-0.9.0. In my setup, I have 2 kafka brokers setup. When I run my kafka consumer, I can see it retrieving messages from the queue and keeping track of offsets ...
Septavalent asked 9/7, 2014 at 18:46
1
Solved
I'm trying to consume messages from a Kafka topic. I'm using a wrapper around confluent_kafka consumer. I need to check if connection is established before I start consuming messages.
I read that t...
Exhaust asked 15/4, 2020 at 10:50
4
Solved
I'm beginner to kafka client in python, i need some help to describe the topics using the client.
I was able to list all my kafka topics using the following code:-
consumer = kafka.KafkaConsumer(...
Coelho asked 22/5, 2019 at 12:52
0
I am using confluent-kafka-python and see that it hangs infinitely when I try to connect to a broker which is down. I can't seem to apply any time out settings I found in the docs:
from confluent_...
Oliana asked 20/3, 2020 at 11:28
0
I am facing an error with the library AIOKafka in Python (versions at the end). Basically, I am receiving a failed heartbeat message and then the commit of the offsets cannot be performed. This is ...
Untinged asked 31/1, 2020 at 16:1
2
Solved
I found this where I reset my LAG with the kafka-consumer-groups.sh tool How to change start offset for topic? but I am needing to reset it within the application. I found this example, but it does...
Balefire asked 25/4, 2018 at 18:8
1
I am pretty new to confluent_kafka but I've gained some experience with kafka-python. What I am trying to do is changing the offset where to start consuming messages. This why I'd like to build a c...
Chateau asked 22/4, 2018 at 16:45
2
Solved
I am a fairly new in Python and starting with Kafka. I have a requirement where I need to send and consume json messages. For this I am using kafka-python to communicate with Kafka.
#Producer.py
...
Sapp asked 5/4, 2019 at 11:38
3
Solved
I am trying to send a very simple JSON object through Kafka and read it out the other side using Python and kafka-python. However, I keep seeing the following error:
2017-04-07 10:28:52,030.30.999...
Inkerman asked 7/4, 2017 at 16:21
2
Solved
I have a Kafka10 cluster with SASL_SSL (Authentication( JAAS ) and Authorization) enabled.
Able to connect thru SASL using the Java client with the below props.
ssl.keystore.location="client_keys...
Velour asked 23/3, 2017 at 21:15
1
Solved
I'm trying to build an application with kafka-python where a consumer reads data from a range of topics. It is extremely important that the consumer never reads the same message twice, but also nev...
Proposal asked 11/8, 2018 at 11:11
3
I am trying to pass data from kafka to spark streaming.
This is what I've done till now:
Installed both kafka and spark
Started zookeeper with default properties config
Started kafka server with...
Reluctant asked 18/1, 2016 at 12:12
7
kafka-python (1.0.0) throws error while connecting to the broker.
At the same time /usr/bin/kafka-console-producer and /usr/bin/kafka-console-consumer work fine.
Python application used to work we...
Monro asked 28/2, 2016 at 22:43
2
Solved
I have a simple JSON object like the following
d = { 'tag ': 'blah',
'name' : 'sam',
'score':
{'row1': 100,
'row2': 200
}
}
The following is my python code which is sending messages to Ka...
Kellner asked 5/8, 2015 at 4:20
4
Solved
i am using kafka-python to consume messages from a kafka queue (kafka version 0.10.2.0). In particular i am using KafkaConsumer type.
If the consumer stops and after a while it is restarted i would...
Rheotropism asked 5/4, 2017 at 16:57
1
Solved
I would like to set a callback to be fired if a produced records fail. Initially, I would just like to log the failed record.
The Confluent Kafka python library provides a mechanism for adding a c...
Thimble asked 24/9, 2017 at 8:27
3
Solved
I have an application for downloading specific web-content, from a stream of URL's generated from 1 Kafka-producer. I've created a topic with 5 partitions and there are 5 kafka-consumers. However t...
Afrikah asked 29/8, 2017 at 13:15
3
From the AvroProducer example in the confluent-kafka-python repo, it appears that the key/value schema are loaded from files. That is, from this code:
from confluent_kafka import avro
from conflu...
Habakkuk asked 28/4, 2017 at 0:55
1
Solved
I am fairly new to kafka so forgive me if this question is trivial. I have a very simple setup for purposes of timing tests as follows:
Machine A -> writes to topic 1 (Broker) -> Machine B reads f...
Pro asked 18/2, 2017 at 4:35
1
Solved
I have been using the python-kaka module to consume from a kafka broker. I want to consume from the same topic with 'x' number of partitions in parallel. The documentation has this :
# Use multip...
Essieessinger asked 24/5, 2016 at 14:59
1
Solved
I'm trying to write a simple pyspark job, which would receive data from a kafka broker topic, did some transformation on that data, and put the transformed data on a different kafka broker topic.
...
Pensive asked 20/5, 2016 at 2:37
1
If I set Kafka config param at Producer as:
1. retries = 3
2. max.in.flight.requests.per.connection = 5
then its likely that Messages within one partition may not be in send_order.
Does Kafka t...
Magellan asked 18/4, 2016 at 11:1
1
I am using the below code to read messages from a topic. I am facing two issues.
Whenever i start consumer, it is reading all the messages in the queue?
How do read only the unread messages?
from...
Gallican asked 9/1, 2016 at 7:8
1
I am using Python Kafka topic.
Is there any provision producer that can update a message in a queue in Kafka and append it to the top of queue again?
According to spec of Kafka, it doesn't seems ...
Phone asked 10/9, 2015 at 17:40
© 2022 - 2024 — McMap. All rights reserved.