pyspark: The system cannot find the path specified

Question

I just installed pyspark 2.2.0 using conda (using python v3.6 on windows 7 64bit, java v1.8)

$conda install pyspark

It downloaded and seemed to install correctly with no errors. Now when I run pyspark on the command line, it just tells me "The system cannot find the path specified."

$pyspark
The system cannot find the path specified.
The system cannot find the path specified.

I tried including the pyspark path directory in my PATH environment variables, but that still didn't seem to work, but maybe I am giving the wrong path? Can anyone please advise. Does the Java path need to be specified in PATH environment variables or something? Thanks

Conspiracy

Answer 1

PySpark from PyPi (i.e. installed with pip or conda) does not contain the full PySpark functionality; it is only intended for use with a Spark installation in an already existing cluster, in which case you might want to avoid downloading the whole Spark distribution. From the docs:

The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.

If you intend to work in the PySpark shell, I suggest you download Spark as said above (PySpark is an essential component of it).

Luminesce

Thanks - still having trouble in getting it up and running though, even after downloading from the page. Is it necessary to change PATH variables in the .bashrc as outlined here? – Conspiracy

@killerT2333 In general, yes; you'll find literally dozens of articles in the web for this. And if you found the answer helpful, kindly accept it (upvotes are also welcome) - thanks – Luminesce

Answer 2

It seems to be a Java path problem.

I had the same issue and had exactly the same response, and JAVA_HOME path was not set.

Check this page and see Moustafa Mahmoud's answer. I only had to take care of suggestion 1. (JAVA_HOME variable).

Also see link as well.

Julijulia

Answer 3

In my case, the problem was caused by double path. remove spark path from environment.

pip uninstall pyspark
pip install pyspark

Nephrolith

Answer 4

export PYSPARK_PYTHON=python3.5

This worked for me when I was having PATH issues. Hope it helps. If not, check out your config files.

Dumpcart

Answer 5

Please check the compatibility as well , It seems that your spark version is not compatible with the Java Version for example spark-2.4.3 is compatible with java-8.0(jdk-8u51-.../jdk1.8.0_51).

Espouse

Answer 6

I was getting the error from pycharm. it was fix with JAVA_HOME setup with environmental variable Step 1 Variable name : JAVA_HOME variable path : java installtion Dir without bin Step 2 define the bin in Path %JAVA_HOME%\bin Step 2 Restart your system Problem will resovle.

Bugbear

As it is currently written, your answer is hard to follow. Please consider formatting your answer so that your code is fenced insider codeblocks. Spacing between each step will also help to make it easier to read. – Portillo