Spark Streaming: long queued/active batches

Question

could anyone please point out what's the cause of this active batches hanging there for many weeks and never being processed? Thanks a lot.

My guess is not enough executors, and more workers/executors will solve the problem? Or Spark assign priority on different batches within its task scheduler?

But the situation here is, very recent batches (end of June) got processed successfully, but batches in May still being queued.

I just checked my Spark setting, scheduler policy is FIFO

spark.scheduler.mode    FIFO

Manamanacle

Answer 1

It turns out that the master node is the bottleneck.

Master node is short of memory, and then maybe scheduler cannot process fast enough.

Solution: change master node to be a more powerful EC2 instance

Manamanacle

I was running a VM with 1 VCPU this was causing the backlog – Saluki

Answer 2

in spark-submit set --diver-memory and --executor-memory to appropriate values base on your events count such that 10000M

Paigepaik