spark-streaming-kafka Questions
1
Environment: Spark 2.3.0, Scala 2.11.12, Kafka (Whatever the latest version is)
I have a secure Kafka system, to which I'm trying to connect my Spark Streaming Consumer. Below is my build.sbt file...
Chandler asked 10/6, 2019 at 11:12
2
I would like to know about the unit testing side of Spark Structured Streaming. My scenario is, I am getting data from Kafka and I am consuming it using Spark Structured Streaming and applying some...
Fillmore asked 4/7, 2019 at 20:47
8
Solved
I am using Spark Streaming to process data between two Kafka queues but I can not seem to find a good way to write on Kafka from Spark. I have tried this:
input.foreachRDD(rdd =>
rdd.foreachPa...
Trickle asked 23/7, 2015 at 14:39
1
Solved
I would like run 2 spark structured streaming jobs in the same emr cluster to consumer the same kafka topic. Both jobs are in the running status. However, only one job can get the kafka data. My co...
Cyclades asked 1/8, 2020 at 8:53
2
Solved
I have a Spark Streaming application and a Kafka Streams application running side by side, for benchmarking purposes. Both consume from the same input topic and write to different targets databases...
Hageman asked 23/6, 2020 at 18:8
3
Solved
Unable to send avro format message to Kafka topic from spark streaming application. Very less information is available online about avro spark streaming example code. "to_avro" method doesn't requi...
Fryer asked 26/12, 2019 at 10:37
6
Solved
I have creating a simple Kafka Producer & Consumer.I am using kafka_2.11-0.9.0.0. Here is my Producer code.
public class KafkaProducerTest {
public static String topicName = "test-topic-2";
pu...
Complacency asked 21/5, 2016 at 12:46
1
I am working on Kafka streaming and trying to integrate it with Apache Spark. However, while running I am getting into issues. I am getting the below error.
This is the command I am using.
df_TR ...
Coadjutor asked 6/11, 2019 at 4:53
2
Solved
I've set up Spark Structured Streaming (Spark 2.3.2) to read from Kafka (2.0.0). I'm unable to consume from the beginning of the topic if messages entered the topic before Spark streaming job is st...
Severson asked 19/6, 2019 at 2:31
1
Solved
I am using Spark Structured Streaming to consume events from Kafka and upload them to S3.
Checkpoints are committed on S3:
DataFrameWriter<Row> writer = input.writeStream()
.format("orc")
...
Crain asked 21/3, 2019 at 14:17
2
Solved
I'm trying to run a spark stream from a kafka queue containing Avro messages.
As per https://spark.apache.org/docs/latest/sql-data-sources-avro.html I should be able to use from_avro to convert c...
Tendril asked 6/3, 2019 at 15:25
1
Why and when one would choose to use Spark streaming with Kafka?
Suppose I have a system getting thousand messages per seconds through Kafka. I need to apply some real time analytics on these mess...
Cyrillic asked 23/7, 2017 at 8:11
2
Solved
I'm having some issues while trying to read from kafka with spark streaming.
My code is:
val sparkConf = new SparkConf().setMaster("local[2]").setAppName("KafkaIngestor")
val ssc = new Streaming...
Leatherback asked 7/12, 2015 at 23:43
1
We have a Spark Streaming application, it reads data from a Kafka queue in receiver and does some transformation and output to HDFS. The batch interval is 1min, we have already tuned the backpressu...
Khichabia asked 15/4, 2016 at 7:57
1
Solved
I'm trying to consume a kafka 0.8 topic using spark-streaming2.0.0, i'm trying to identify the required dependencies i have tried using these dependencies in my build.sbt file
libraryDependencies...
Hohenlinden asked 15/9, 2016 at 17:26
2
Solved
I just copied the spark streaming wodcount python code, and use spark-submit to run the wordcount python code in Spark cluster, but it shows the following errors:
py4j.protocol.Py4JJavaError: An ...
Crabtree asked 7/4, 2015 at 6:28
1
© 2022 - 2024 — McMap. All rights reserved.