Login/Register
How to run Spark SQL Thrift Server in local mode and connect to Delta using JDBC
Asked Answered
Solved apache-sparkapache-spark-sqldelta-lakespark-thriftserver
S

3

5

I'd like connect to Delta using JDBC and would like to run the Spark Thrift Server (STS) in local mode to kick the tyres.

I start STS using the following command:

$SPARK_HOME/sbin/start-thriftserver.sh \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --packages 'io.delta:delta-core_2.12:1.0.0'

I get the following error when trying to connect using beeline:

Beeline version 2.3.7 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000

Connecting to jdbc:hive2://localhost:10000
Enter username for jdbc:hive2://localhost:10000:
Enter password for jdbc:hive2://localhost:10000: 

Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog (state=08S01,code=0)

Looking through the thrift server logs, I didn't see any obvious errors and the delta jar file was loaded without any errors.

Spark Command: /Users/sandbox/.sdkman/candidates/java/8.0.282.j9-adpt/bin/java -cp /Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/conf/:/Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --packages io.delta:delta-core_2.12:1.0.0 spark-internal
========================================
:: loading settings :: url = jar:file:/Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/sandbox/.ivy2/cache
The jars for the packages stored in: /Users/sandbox/.ivy2/jars
io.delta#delta-core_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-8f65b233-668d-46ea-98d6-0da629b12d1e;1.0
        confs: [default]
        found io.delta#delta-core_2.12;1.0.0 in central
        found org.antlr#antlr4;4.7 in central
        found org.antlr#antlr4-runtime;4.7 in central
        found org.antlr#antlr-runtime;3.5.2 in central
        found org.antlr#ST4;4.0.8 in central
        found org.abego.treelayout#org.abego.treelayout.core;1.0.3 in central
        found org.glassfish#javax.json;1.0.4 in central
        found com.ibm.icu#icu4j;58.2 in central
:: resolution report :: resolve 385ms :: artifacts dl 13ms
        :: modules in use:
        com.ibm.icu#icu4j;58.2 from central in [default]
        io.delta#delta-core_2.12;1.0.0 from central in [default]
        org.abego.treelayout#org.abego.treelayout.core;1.0.3 from central in [default]
        org.antlr#ST4;4.0.8 from central in [default]
        org.antlr#antlr-runtime;3.5.2 from central in [default]
        org.antlr#antlr4;4.7 from central in [default]
        org.antlr#antlr4-runtime;4.7 from central in [default]
        org.glassfish#javax.json;1.0.4 from central in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-8f65b233-668d-46ea-98d6-0da629b12d1e
        confs: [default]
        0 artifacts copied, 8 already retrieved (0kB/13ms)
21/11/06 07:28:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/06 07:28:30 INFO HiveThriftServer2: Started daemon with process name: 75591@MacBook-Pro
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for TERM
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for HUP
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for INT
21/11/06 07:28:30 INFO HiveThriftServer2: Starting SparkContext
21/11/06 07:28:31 INFO HiveConf: Found configuration file null
21/11/06 07:28:31 INFO SparkContext: Running Spark version 3.1.2
21/11/06 07:28:31 INFO ResourceUtils: ==============================================================
21/11/06 07:28:31 INFO ResourceUtils: No custom resources configured for spark.driver.
21/11/06 07:28:31 INFO ResourceUtils: ==============================================================
21/11/06 07:28:31 INFO SparkContext: Submitted application: Thrift JDBC/ODBC Server
21/11/06 07:28:31 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
21/11/06 07:28:31 INFO ResourceProfile: Limiting resource is cpu
21/11/06 07:28:31 INFO ResourceProfileManager: Added ResourceProfile id: 0
...
21/11/06 07:28:31 INFO Utils: Successfully started service 'sparkDriver' on port 55507.
...
21/11/06 07:28:32 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/11/06 07:28:32 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://macbook-pro:4040
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar at spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar at spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar at spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar at spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar at spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar at spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar at spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar at spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar at file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar at file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar at file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar at file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:32 INFO Executor: Starting executor ID driver on host macbook-pro
21/11/06 07:28:32 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:32 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO TransportClientFactory: Successfully created connection to MacBook-Pro/192.168.1.207:55507 after 53 ms (0 ms spent in bootstraps)
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7179925225828426556.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7179925225828426556.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp6953292115063484853.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp6953292115063484853.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7481622034847054241.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7481622034847054241.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp8137443686268894230.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp8137443686268894230.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7474775280611839356.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7474775280611839356.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7899823112197045994.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7899823112197045994.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp1438945171356535536.tmp
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp836117221049857229.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp836117221049857229.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar to class loader
21/11/06 07:28:33 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55509.
21/11/06 07:28:33 INFO NettyBlockTransferService: Server created on macbook-pro:55509
...
21/11/06 07:28:33 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/sandbox/projects/delta/spark-warehouse/').
21/11/06 07:28:33 INFO SharedState: Warehouse path is 'file:/Users/sandbox/projects/delta/spark-warehouse/'.
21/11/06 07:28:34 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
21/11/06 07:28:35 INFO HiveConf: Found configuration file null
21/11/06 07:28:35 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda
21/11/06 07:28:35 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda
21/11/06 07:28:35 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda/_tmp_space.db
21/11/06 07:28:35 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/sandbox/projects/delta/spark-warehouse/
21/11/06 07:28:36 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/11/06 07:28:36 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/11/06 07:28:36 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
21/11/06 07:28:36 INFO ObjectStore: ObjectStore, initialize called
21/11/06 07:28:36 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/11/06 07:28:36 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/11/06 07:28:37 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/11/06 07:28:39 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/11/06 07:28:39 INFO ObjectStore: Initialized ObjectStore
21/11/06 07:28:39 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/11/06 07:28:39 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore [email protected]
21/11/06 07:28:39 INFO HiveMetaStore: Added admin role in metastore
21/11/06 07:28:39 INFO HiveMetaStore: Added public role in metastore
21/11/06 07:28:39 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/11/06 07:28:39 INFO HiveMetaStore: 0: get_all_functions
21/11/06 07:28:39 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_all_functions
21/11/06 07:28:39 INFO HiveMetaStore: 0: get_database: default
21/11/06 07:28:39 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_database: default      
21/11/06 07:28:39 INFO HiveUtils: Initializing execution hive, version 2.3.7
21/11/06 07:28:39 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d
21/11/06 07:28:39 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d
21/11/06 07:28:39 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d/_tmp_space.db
21/11/06 07:28:39 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/sandbox/projects/delta/spark-warehouse/
21/11/06 07:28:40 INFO SessionManager: Operation log root directory is created: /var/folders/qp/g1z81/T/sandbox/operation_logs
...
21/11/06 07:28:40 INFO AbstractService: Service:OperationManager is inited.
21/11/06 07:28:40 INFO AbstractService: Service:SessionManager is inited.
21/11/06 07:28:40 INFO AbstractService: Service: CLIService is inited.
21/11/06 07:28:40 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
21/11/06 07:28:40 INFO AbstractService: Service: HiveServer2 is inited.
21/11/06 07:28:40 INFO AbstractService: Service:OperationManager is started.
21/11/06 07:28:40 INFO AbstractService: Service:SessionManager is started.
21/11/06 07:28:40 INFO AbstractService: Service: CLIService is started.
21/11/06 07:28:40 INFO AbstractService: Service:ThriftBinaryCLIService is started.
21/11/06 07:28:40 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads
21/11/06 07:28:40 INFO AbstractService: Service:HiveServer2 is started.
21/11/06 07:28:40 INFO HiveThriftServer2: HiveThriftServer2 started
...
21/11/06 07:29:48 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/11/06 07:29:48 INFO ObjectStore: Initialized ObjectStore
21/11/06 07:29:48 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/11/06 07:29:48 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore [email protected]
21/11/06 07:29:48 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/11/06 07:29:49 INFO HiveMetaStore: Added admin role in metastore
21/11/06 07:29:49 INFO HiveMetaStore: Added public role in metastore
21/11/06 07:29:49 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/11/06 07:29:49 INFO HiveMetaStore: 0: get_all_functions
21/11/06 07:29:49 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_all_functions  
21/11/06 07:29:49 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
21/11/06 07:29:49 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
21/11/06 07:29:49 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d/_tmp_space.db
21/11/06 07:29:49 INFO HiveSessionImpl: Operation log session directory is created: /var/folders/qp/g1z81/T/sandbox/operation_logs/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.821/11/06 07:29:50 INFO SessionState: Deleted directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d on fs with scheme file
21/11/06 07:29:50 INFO SessionState: Deleted directory: /var/folders/qp/g1z81/T/sandbox/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d on fs with scheme file
21/11/06 07:29:50 INFO HiveMetaStore: 0: Cleaning up thread local RawStore...
21/11/06 07:29:50 INFO audit: ugi=anonymous     ip=unknown-ip-addr      cmd=Cleaning up thread local RawStore...
21/11/06 07:29:50 INFO HiveMetaStore: 0: Done cleaning up thread local RawStore
21/11/06 07:29:50 INFO audit: ugi=anonymous     ip=unknown-ip-addr      cmd=Done cleaning up thread local RawStore     
21/11/06 07:29:50 WARN ThriftCLIService: Error opening session:
org.apache.hive.service.cli.HiveSQLException: Failed to open new session: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog
        at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:85)
        at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:204)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:371)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:243)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog
        at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:66)
        at org.apache.spark.sql.connector.catalog.CatalogManager.loadV2SessionCatalog(CatalogManager.scala:66)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$2(CatalogManager.scala:85)
        at org.apache.spark.sql.connector.catalog.CatalogManager$$Lambda$1357/0x0000000000000000.apply(Unknown Source)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$1(CatalogManager.scala:85)
        at org.apache.spark.sql.connector.catalog.CatalogManager$$Lambda$1356/0x0000000000000000.apply(Unknown Source)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.sql.connector.catalog.CatalogManager.v2SessionCatalog(CatalogManager.scala:84)
        at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:50)
        at org.apache.spark.sql.connector.catalog.CatalogManager.currentCatalog(CatalogManager.scala:117)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog(LookupCatalog.scala:35)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog$(LookupCatalog.scala:35)
...

I'm using Java 8 with Spark 3.1.2 with pre-built for Hadoop 3.2 and later and Delta 1.0.0, running on Mac OSX

What could be the problem?

Sprue answered 6/11, 2021 at 8:8 Comment(2)
interesting, it looks like it was broken in Spark 3.1... Spark 3.0 works just fine with that commands (I've tested it before: #66750974) – Unsay
this specific commit: github.com/apache/spark/commit/…, although it could be more a problem with classloader or something like, because I see that delta jar was added – Unsay
I
3

Once you can copy io.delta:delta-core_2.12:1.0.0 JAR file to $SPARK_HOME/lib and restart, this error goes away.

Intermixture answered 8/1, 2022 at 6:42 Comment(2)
I had the same error and with this change, I'm able to connect through Beeline. But now I'm getting this error 22/01/08 12:09:33 ERROR SparkExecuteStatementOperation: Error executing query with c0e4419a-5ed0-473e-8ac8-d80437bf7581, currentState RUNNING, org.apache.spark.sql.AnalysisException: STORED AS with file format 'delta' is invalid. at org.apache.spark.sql.errors.QueryCompilationErrors$.invalidFileFormatForStoredAsError(QueryCompilationErrors.scala:537) – Intermixture
Now I'm able to create a table with DELTA CREATE TABLE order_items (identity INT, name string, date DATE) USING DELTA PARTITIONED BY (identity) LOCATION '/tmp/delta/order_items'; – Intermixture
S
3

For 1.2.0 and after you need to add delta-core and delta-storage jar to spark jars. This is how I solved the issus:

wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/2.2.0/delta-core_2.12-2.2.0.jar &&\
mv delta-core_2.12-2.2.0.jar opt/spark/jars/ &&\
wget https://repo1.maven.org/maven2/io/delta/delta-storage/2.2.0/delta-storage-2.2.0.jar &&\
mv delta-storage-2.2.0.jar opt/spark/jars/
Seasickness answered 28/2, 2023 at 20:4 Comment(2)
Thanks for the solution. It has worked for me after either choosing root as working directory or prepending the target of the mv operation with /. – Alena
This was the solution for me as well. Spark 3.4.0, Delta Lake 2.4.0. – Mor
S
0

For Spark 3.5 you need delta-spark instead of delta-core:

  1. download the jar in https://mvnrepository.com/artifact/io.delta/delta-spark_2.12/3.0.0 and copy it to the spark-3.5.0-bin-hadoop3/jar folder.

  2. start the thrift server via

start-thriftserver.sh \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --packages 'io.delta:delta-spark_2.12:3.0.0'

3. start beeline 4. initialise the connection with!connect jdbc:hive2://localhost:10000`

Soporific answered 21/11, 2023 at 9:14 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.