How to enable the spark SQL with %sql Magic string on Hive in pyspark using jupyter notebook

Question

How to enable the %sql Magic string on jupyter notebook and how to use %sql magic string on a cell with the below line of code.

spark.sql('select * from test').show()

Metanephros

Answer 1

Try

%%sparksql
select * from test

Before trying install

pip install sparksql-magic

Refer: https://github.com/cryeo/sparksql-magic

Metanephros

Answer 2

You don't need the %sql magic string to work with Spark SQL. You need to first create a Spark DataFrame as described in the SparkSession API docs, like by using df = createDataFrame(data). Then you would create a global view, calling df.createOrReplaceTempView("test"). Then your above query would work.

Maser

How to use the above query with %sql magic string in jupyter notebook, How to enable the spark SQL via %sql, Working like on databricks – Metanephros

Answer 3

before using %%sparksql (in case you've installed sparksql-magic), you need to load the extension using below command in a separate cell.

 %load_ext sparksql_magic

Botticelli

Answer 4

Try

%%sql
select * from test

Link https://github.com/jupyter-incubator/sparkmagic

Pussy

sparkmagic can be installed but listing the magic strings on notebook %sql is not found, please see the above image. – Metanephros