Why does importing SparkSession in spark-shell fail with "object SparkSession is not a member of package org.apache.spark.sql"?

Question

I use Spark 1.6.0 on my VM, Cloudera machine.

I'm trying to enter some data into Hive table from Spark shell. To do that, I am trying to use SparkSession. But the below import is not working.

scala> import org.apache.spark.sql.SparkSession
<console>:33: error: object SparkSession is not a member of package org.apache.spark.sql
         import org.apache.spark.sql.SparkSession

And without that, I cannot execute this statement:

val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()
<console>:33: error: not found: value SparkSession
         val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()

Can anyone tell me what mistake am I doing here ?

Chromomere

Well, are you 100% sure you have Spark2? – Drynurse

Which version of spark do you use? Have you ensured you could open up the spark shell without any errors? – Dorolice

@PraveenKumarKrishnaiyer There are no errors when I open the spark shell. – Chromomere

Answer 1

SparkSession is available as of Spark 2.0 so you should be using SQLContext instead (or upgrade your Spark to the latest and greatest 2.1.1).

Quoting Spark 1.6.0's Starting Point: SQLContext:

The entry point into all functionality in Spark SQL is the SQLContext class, or one of its descendants.

In addition to the basic SQLContext, you can also create a HiveContext, which provides a superset of the functionality provided by the basic SQLContext.

Tiffanitiffanie