I am loading some data into Spark with a wrapper function: def load_data( filename ): df = sqlContext.read.format("com.databricks.spark.csv")\ .option("delimiter", "\t")\ .option("header", "f...

I have a pandas data frame my_df, and my_df.dtypes gives us: ts int64 fieldA object fieldB object fieldC object fieldD object fieldE object dtype: object Then I am trying to convert the pandas d...

This is my dataframe I'm trying to drop the duplicate columns with same name using index: df = spark.createDataFrame([(1,2,3,4,5)],['c','b','a','a','b']) df.show() Output: +---+---+---+---+---+...

Using Apache Spark (or pyspark) I can read/load a text file into a spark dataframe and load that dataframe into a sql db, as follows: df = spark.read.csv("MyFilePath/MyDataFile.txt", sep=...

When I create a DataFrame from a JSON file in Spark SQL, how can I tell if a given column exists before calling .select Example JSON schema: { "a": { "b": 1, "c": 2 } } This is what I want ...

I have a text file on HDFS and I want to convert it to a Data Frame in Spark. I am using the Spark Context to load the file and then try to generate individual columns from that file. val myFile...

I run a query on Databricks: DROP TABLE IF EXISTS dublicates_hotels; CREATE TABLE IF NOT EXISTS dublicates_hotels ... I'm trying to understand why I receive the following error: Error in SQL stat...

val columnName=Seq("col1","col2",....."coln"); Is there a way to do dataframe.select operation to get dataframe containing only the column names specified . I know I can do dataframe.select("col...

I am trying to anonymize/hash a nested column, but haven't been successful. The schema looks something like this: -- abc: struct (nullable = true) | |-- xyz: struct (nullable = true) | | |-- abc123...

I try to do very simple - update a value of a nested column;however, I cannot figure out how Environment: Apache Spark 2.4.5 Databricks 6.4 Python 3.7 dataDF = [ (('Jon','','Smith'),'1580-01-06'...

How can I create a Spark DataFrame in Scala with 100 rows and 3 columns that have random integer values in range (1, 100)? I know how to create a DataFrame manually, but I cannot automate it: val...

There are several similar-yet-different concepts in Spark-land surrounding how work gets farmed out to different nodes and executed concurrently. Specifically, there is: The Spark Driver node (sp...

I am trying to execute a simple mysql query using Apache Spark and create a data frame. But for some reasons spark appends 'WHERE 1=0' at the end of the query which I want to execute and throws an ...

The data looks like this - +-----------+-----------+-----------------------------+ | id| point| data| +-----------------------------------------------------+ | abc| 6|{"key1":"124", "key2": "345"...

I want to read json or xml file in pyspark.lf my file is split in multiple line in rdd= sc.textFile(json or xml) Input { " employees": [ { "firstName":"John", &qu...

I have another question that is related to the split function. I am new to Spark/Scala. below is the sample data frame - +-------------------+---------+ | VALUES|Delimiter| +-------------------+--...

I have a Dataframe that I am trying to flatten. As part of the process, I want to explode it, so if I have a column of arrays, each value of the array will be used to create a separate row. For ins...

I use a sqlContext.read.parquet function in PySpark to read the parquet files everyday. The data has a timestamp column. They changed the timestamp field from 2019-08-26T00:00:13.600+0000 to 2019-0...

How can I replicate this code to get the dataframe size in pyspark? scala> val df = spark.range(10) scala> print(spark.sessionState.executePlan(df.queryExecution.logical).optimizedPlan.stats...

I am trying to find a reliable way to compute the size (in bytes) of a Spark dataframe programmatically. The reason is that I would like to have a method to compute an "optimal" number of partiti...

I have a Spark DataFrame that has one column that has lots of zeros and very few ones (only 0.01% of ones). I'd like to take a random subsample but a stratified one - so that it keeps the ratio o...

I am trying to split the Dataset into different Datasets based on Manufacturer column contents. It is very slow Please suggest a way to improve the code, so that it can execute faster and reduce th...

So as I know in Spark Dataframe, that for multiple columns can have the same name as shown in below dataframe snapshot: [ Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}),...

I want to select all columns in a table except StudentAddress and hence I wrote following query: select `(StudentAddress)?+.+` from student; It gives following error in Squirrel Sql client. org....

I want to delete data from a delta file in databricks. Im using these commands Ex: PR=spark.read.format('delta').options(header=True).load('/mnt/landing/Base_Tables/EventHistory/') PR.write.format(...