Pyspark Azure Blob Storage - Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found

I'm trying to read a CSV file on Azure Blob Storage with pyspark from a Jupyter Notebook, but I'm facing the following error:

Py4JJavaError: An error occurred while calling o34.csv. : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2667) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:747) at scala.collection.immutable.List.map(List.scala:293) at org.apache.spark.sql.execution.datasources.DataSource$.checkAndGlobPathIfNecessary(DataSource.scala:745) at org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:577) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:408) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245) at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:571) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665) ... 29 more

Here are the steps I followed: I have a kubernetes cluster available.

I installed a HELM chart JupyterHub which seems to work correctly, I installed Pyspark there.

I installed a HELM Chart (Bitnami) to set up a Spark cluster.

I was able to connect to my Spark cluster via pyspark from a Jupyter notebook :

from pyspark.sql import SparkSession
spark = SparkSession.builder.master("spark://spark-master-svc:7077").getOrCreate()
spark.sparkContext

I can execute some commands in remote Spark without any issue.

I tried to read a csv file located on a Blob Storage, but I get the error message that I pasted above

SECRET_ACCESS_KEY = "***"
STORAGE_NAME = "***"
file_path = "wasb://***@***.blob.core.windows.net/***.csv"

fs_acc_key = "fs.azure.account.key." + STORAGE_NAME + ".blob.core.windows.net"
spark.conf.set(fs_acc_key, SECRET_ACCESS_KEY)

df_csv = spark.read.csv(
    path=file_path,
    sep='|',
    inferSchema=True,
    header=True
)

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found

After some research, I saw that it was necessary to install multiple jars (at least hadoop-azure and azure-storage), so I did it in a Dockerfile, as mentionned in Bitnami Documentation :

# https://github.com/bitnami/bitnami-docker-spark/blob/master/3/debian-10/Dockerfile
FROM bitnami/spark:3.2.0-debian-10-r73

USER root

### ADDITIONAL JARS
# https://github.com/bitnami/bitnami-docker-spark#installing-additional-jars
RUN curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.3.1/hadoop-azure-3.3.1.jar --output /opt/bitnami/spark/jars/hadoop-azure-3.3.1.jar &&\
    curl https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar --output /opt/bitnami/spark/jars/azure-storage-8.6.6.jar &&\
    curl https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util/11.0.7/jetty-util-11.0.7.jar --output /opt/bitnami/spark/jars/jetty-util-11.0.7.jar &&\
    curl https://repo1.maven.org/maven2/org/apache/hadoop/thirdparty/hadoop-shaded-guava/1.1.1/hadoop-shaded-guava-1.1.1.jar --output /opt/bitnami/spark/jars/hadoop-shaded-guava-1.1.1.jar &&\
    curl https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar --output /opt/bitnami/spark/jars/httpclient-4.5.13.jar &&\
    curl https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.1/jackson-databind-2.13.1.jars --output /opt/bitnami/spark/jars/jackson-databind-2.13.1.jars &&\
    curl https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.13.1/jackson-core-2.13.1.jar --output /opt/bitnami/spark/jars/jackson-core-2.13.1.jar &&\
    curl https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-util-ajax/11.0.7/jetty-util-ajax-11.0.7.jar --output /opt/bitnami/spark/jars/jetty-util-ajax-11.0.7.jar &&\
    curl https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl/2.2.0.Final/wildfly-openssl-2.2.0.Final.jar --output /opt/bitnami/spark/jars/wildfly-openssl-2.2.0.Final.jar &&\
    curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/3.3.1/hadoop-common-3.3.1.jar --output /opt/bitnami/spark/jars/hadoop-common-3.3.1.jar &&\
    curl https://repo1.maven.org/maven2/com/microsoft/azure/azure-keyvault-core/1.2.6/azure-keyvault-core-1.2.6.jar --output /opt/bitnami/spark/jars/azure-keyvault-core-1.2.6.jar

USER 1001

I redeployed my Spark cluster, the jars are present in the expected folder

However, I still get the same error:

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found

I have tried a lot of configurations found on stackoverflow but still get the same result.

spark = SparkSession.builder.master("spark://spark-master-svc:7077") \
            .config("spark.jars.packages", "org.apache.hadoop:hadoop-azure-3.3.1,com.microsoft.azure:azure-storage:8.6.6").getOrCreate()

spark = SparkSession.builder.master("spark://spark-master-svc:7077") \
            .config("spark.jars.packages", "org.apache.hadoop:hadoop-azure-3.3.1").getOrCreate()

spark.sparkContext._conf.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
spark.sparkContext._conf.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
spark.sparkContext._conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")

No matter what configuration I try, when I try to read the CSV file I get the same error message.

I don't really know what to try anymore, there are certainly things that escape me.

I hope someone here can help me?

Recommended topics

Hot tags