The system cannot find the path specified error while running pyspark

Question

I just downloaded spark-2.3.0-bin-hadoop2.7.tgz. After downloading I followed the steps mentioned here pyspark installation for windows 10.I used the comment bin\pyspark to run the spark & got error message

The system cannot find the path specified

Attached is the screen shot of error message

Attached is the screen shot of my spark bin folder

Screen shot of my path variable looks like

I have python 3.6 & Java "1.8.0_151" in my windows 10 system Can you suggest me how to resolve this issue?

Garrow

Please check this answer https://mcmap.net/q/584709/-spark-shell-the-system-cannot-find-the-path-specified – Corrientes

I had a similar issue that turned out to be as a result of a JDK upgrade, after the upgrade my JAVA_HOME pointed to the wrong place. Since I had forgotten about the upgrade I had this issue. I used the excellent Procmon from sysinternals to debug the issue as to the eye everything looked good when it wasn't. – Amboina

Answer 1

Actually, the problem was with the JAVA_HOME environment variable path. The JAVA_HOME path was set to .../jdk/bin previously,

I stripped the last /bin part for JAVA_HOME while keeping it (/jdk/bin) in system or environment path variable (%path%) did the trick.

Oraorabel

This is also the cause of the same problem for me. It's strange that the java path has been set with java/bin predefined. – Boys

My problem was that I had the wrong version name in my JAVA_HOME. Changed it to "C:\Program Files\Java\jdk1.8.0_281" and it works. – Electroshock

Amazing it worked. – Versicular

Answer 2

My problem was that the JAVA_HOME was pointing to JRE folder instead of JDK. Make sure that you take care of that

Behest

Actually for me it was exactly opposite and it works now. – Corney

Answer 3

Worked hours and hours on this. My problem was with Java 10 installation. I uninstalled it and installed Java 8, and now Pyspark works.

Ottillia

Answer 4

For those who use Windows and still trying, what solved to me was reinstalling Python (3.9) as a local user (c:\Users\<user>\AppData\Local\Programs\Python) and defined both env variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to c:\Users\<user>\AppData\Local\Programs\Python\python.exe

Mcginty

Could you elaborate on how you found out you had to set these variables up? – Laborer

@JorgeEstebanMendoza Sorry for the late response. It was just random googling. – Mcginty

Answer 5

Fixing problems installing Pyspark (Windows)

Incorrect JAVA_HOME path

> pyspark  
The system cannot find the path specified.

Open System Environment variables:

rundll32 sysdm.cpl,EditEnvironmentVariables

Set JAVA_HOME: System Variables > New:

Variable Name: JAVA_HOME
Variable Value: C:\Program Files\Java\jdk1.8.0_261

Also, check that SPARK_HOME and HADOOP_HOME are correctly set, e.g.:

SPARK_HOME=C:\Spark\spark-3.2.0-bin-hadoop3.2
HADOOP_HOME=C:\Spark\spark-3.2.0-bin-hadoop3.2

Important: Double-check the following

1. The path exists
2. The path does not contain the bin folder

Incorrect Java version

> pyspark
WARN SparkContext: Another SparkContext is being constructed 
UserWarning: Failed to initialize Spark session.
java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$

Ensure that JAVA_HOME is set to Java 8 (jdk1.8.0)

winutils not installed

> pyspark
WARN Shell: Did not find winutils.exe
java.io.FileNotFoundException: Could not locate Hadoop executable

Download winutils.exe and copy it to your spark home bin folder

 curl -OutFile C:\Spark\spark-3.2.0-bin-hadoop3.2\bin\winutils.exe -Uri https://github.com/steveloughran/winutils/raw/master/hadoop-3.0.0/bin/winutils.exe

Cisterna

Answer 6

Switching SPARK_HOME to C:\spark\spark-2.3.0-bin-hadoop2.7 and changing PATH to include %SPARK_HOME%\bin did the trick for me.

Originally my SPARK_HOME was set to C:\spark\spark-2.3.0-bin-hadoop2.7\bin and PATH was referencing it as %SPARK_HOME%.

Running a spark command directly in my SPARK_HOME dir worked but only once. After that initial success I then noticed your same error and that echo %SPARK_HOME% was showing C:\spark\spark-2.3.0-bin-hadoop2.7\bin\.. I thought perhaps spark-shell2.cmd had edited it in attempts to get itself working, which led me here.

Stasny

Answer 7

Most likely you forgot to define the Windows environment variables such that the Spark bin directory is in your PATH environment variable.

Define the following environment variables using the usual methods for Windows.

First define an environment variable called SPARK_HOME to be C:\spark\spark-2.3.0-bin-hadoop2.7

Then either add %SPARK_HOME%\bin to your existing PATH environment variable, or if none exists (unlikely) define PATH to be %SPARK_HOME%\bin

If there is no typo specifying the PATH, echo %PATH% should give you the fully resolved path to the Spark bin directory i.e. it should look like

C:\spark\spark-2.3.0-bin-hadoop2.7\bin;

If PATH is correct, you should be able to type pyspark in any directory and it should run.

If this does not resolve the issue perhaps the issue is as specified in pyspark: The system cannot find the path specified in which case this question is a duplicate.

Bluestone

No, I did that also restarted my command prompt after setting path variable & then use pyspark comment. I have added my path variable screen shot in first post – Garrow

Answer 8

Update: in my case it came down to wrong path for JAVA, I got it to work...

I'm having the same problem. I initially installed Spark through pip, and pyspark ran successfully. Then I started messing with Anaconda updates and it never worked again. Any help will be appreciated...

I'm assuming PATH is installed correctly for the original author. A way to check that is to run spark-class from command prompt. With correct PATH it will return Usage: spark-class <class> [<args>] when ran from an arbitrary location. The error from pyspark comes from a string of .cmd files that I traced to the last lines in spark-class2.cmd

This maybe silly, but altering the last block of code shown below changes the error message you get from pyspark from "The system cannot find the path specified" to "The syntax of the command is incorrect". Removing this whole block makes pyspark do nothing.

rem The launcher library prints the command to be executed in a single line suitable for being
rem executed by the batch interpreter. So read all the output of the launcher into a variable.
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
"%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" org.apache.spark.launcher.Main 
%* > %LAUNCHER_OUTPUT%
for /f "tokens=*" %%i in (%LAUNCHER_OUTPUT%) do (
  set SPARK_CMD=%%i
)
del %LAUNCHER_OUTPUT%
%SPARK_CMD%

I removed "del %LAUNCHER_OUTPUT%" and saw that the text file generated remains empty. Turns out "%RUNNER%" failed to find correct directory with java.exe because I messed up the PATH to Java (not Spark).

Slowly

Can you please let me know how to set spark-class?I'm getting error message as 'spark-class' is not recognized as an internal or external command, operable program or batch file. – Garrow

Yes now I'm getting Usage: spark-class <class> [<args>] by submitting the command spark-class.But still getting The system cannot find the path specified. after submitting pyspark command – Garrow

Try to remove 'del %LAUNCHER_OUTPUT%' from bin\spark-class2.cmd and look for the temp file in C:\Users\~\AppData\Local\Temp It should be called 'spark-class-launcher-output-####.txt' and say something like 'PYSPARK_SUBMIT_ARGS="--name" "PySparkShell" "pyspark-shell" && python' What folder did you install your JDK into? – Slowly

I'm seeing my spark-class-launcher-output-20430 in C:\Users\~\AppData\Local\Temp is empty.It's showing 0 KB file size.JDK is installed under C:\Program Files. Removed del %LAUNCHER_OUTPUT% from bin\spark-class2.cmd.Kindly suggest what to do. – Garrow

Some component of the installation doesn't like a space in the file path (Program Files). Try reinstalling Java into C:\Java or something like this – Slowly

Answer 9

if you use anaconda for window. The below command can save your time

conda install -c conda-forge pyspark

After that restart anaconda and start "jupyter notebook"

Ancestress

Answer 10

I know this is an old post, but I am adding my finding in case it helps anyone.

The issue is mainly due to the line source "${SPARK_HOME}"/bin/load-spark-env.sh in pyspark file. As you can see it's not expecting 'bin' in SPARK_HOME. All I had to do was remove 'bin' from my SPARK_HOME environment variable and it worked (C:\spark\spark-3.0.1-bin-hadoop2.7\bin to C:\spark\spark-3.0.1-bin-hadoop2.7\).

The error on Windows Command Prompt made it appear like it wasn't recognizing 'pyspark', while the real issue was with it not able to find the file 'load-spark-env.sh.'

Playacting