I'm running Spark version 2.3.0.2.6.5.1175-1 with Python 3. 6.8 on Ambari. While submitting the application I get the following logs in stderr 22/06/15 12:29:31 INFO StateStoreCoordinatorRef: Regi...

I'm having problem understanding how to connect Kafka and PySpark. I have kafka installation on Windows 10 with topic nicely streaming data. I've installed pyspark which runs properly-I'm able to c...

I understand that using Kafka's own offset tracking instead of other methods (like checkpointing) is problematic for streaming jobs. However I just want to run a Spark batch job every day, reading ...

I want to use Spark Structured Streaming to read from a secure kafka. This means that I will need to force a specific group.id. However, as is stated in the documentation this is not possible. Stil...

Based on the introduction in Spark 3.0, https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html. It should be possible to set "kafka.group.id" to track the offset...

Kafka with spark-streaming throws an error: from pyspark.streaming.kafka import KafkaUtils ImportError: No module named kafka I have already setup a kafka broker and a working spark environment wi...

I was going through the Spark structured streaming - Kafka integration guide here. It is told at this link that enable.auto.commit: Kafka source doesn’t commit any offset. So how do I manual...