How does Spark's StreamingLinearRegressionWithSGD work?

Question

I am working on StreamingLinearRegressionWithSGD which has two methods trainOn and predictOn. This class has a model object that is updated as training data arrives in the stream specified in trainOn argument.

Simultaneously It give prediction using same model.

I want to know that how the model weights are updated and synchronized across workers/executors.

Any link or reference will be helpful. Thanks.

Roundworm

Answer 1

There is no magic here. StreamingLinearAlgorithm keeps a mutable reference to the current GeneralizedLinearModel.

trainOn uses DStream.foreachRDD to train a new model on each batch, and then updates the model. Similarly predictOn uses DStream.map to predict with the current version of the model.

Since Spark will serialize closures for each stage there is no need for any additional synchronization. Spark will use the current value of the model each time it computes the closure.

Effectively it equivalent to running a loop on the driver with interleaving run and predict.

Stylopodium

Thank u for answer, I dug up the code and found that in GradientDescent weights are being broadcasted. Is this the step where synchronization of weights happen. Each executor may compute different weight locally what is happening in that case. – Roundworm