Pyspark: Error executing Jupyter command while running a file using spark-submit

Question

I am able to run pyspark and run a script on Jupyter notebook. But when I try to run the file from terminal using spark-submit, getting this error:

Error executing Jupyter command file path [Errno 2] No such file or directory

Can anyone help me what's wrong with my config?

I'm using Python 2.7 and Spark 1.6

Wallaby

Answer 1

These problem occurs because you have set jupyter to run pyspark script . Now you should unset PYSPARK_DRIVER_PYTHON variable command for the same is $ unset PYSPARK_DRIVER_PYTHON then try spark-submit again ./bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost 9999.

Absorbing

I was using the following in spark-env.sh export PYSPARK_PYTHON=python3.7 # export PYSPARK_DRIVER_PYTHON=jupyter # export PYSPARK_DRIVER_PYTHON_OPTS="notebook" If I want to run Jupyter will have to set the driver and run separately pyspark command. Thanks not spark-submit command. – Geraint

Answer 2

@Shubham's solution resolved my problem: "Jupyter command jupyter-spark_simple.py not found."

unset PYSPARK_DRIVER_PYTHON

Rattoon