I am trying to use spark cassandra connector to fetch values from the table. The load() yields the IncompatibleClassChangeError. My development environment is defined as follows, Intellij : 2019.3....

I'm trying to make year and month columns from a column named logtimestamp (of type TimeStampType) in spark. The data source is cassandra. I am using sparkshell to perform these steps, here is the ...

This error has been the hardest to trace. I am not sure what is going on. I am running a Spark cluster on my location machine. so the entire spark cluster is under one host which is 127.0.0.1 and I...

There are many tables in cassandra database, which contain column titled user_id. The values user_id are referred to user stored in table users. As some users are deleted, I would like to delete or...

I'm always getting the following error.Can somebody help me please? Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging at java.lang.ClassLoader.defineClass1(Nativ...

I have a Cassandra table that for simplicity looks something like: key: text jsonData: text blobData: blob I can create a basic data frame for this using spark and the spark-cassandra-connector ...

So I have a Python Stream-sourced DataFrame df that has all the data I want to place into a Cassandra table with the spark-cassandra-connector. I've tried doing this in two ways: df.write \ .form...

I am trying to read a parquet file which is present in AWS S3 and getting the below error. 17/12/19 11:27:40 DEBUG DAGScheduler: ShuffleMapTask finished on 0 17/12/19 11:27:40 DEBUG DAGScheduler:...

I'm trying to perform a large number of operations on a dataframe from a cassandra table and then save it in another table. One of these operations is as follows : val leadWindow = Window.partitio...

I'm using Scala Version 2.10.5 Cassandra 3.0 and Spark 1.6. I want to insert data into cassandra so I tried Out basic Example scala> val collection = sc.parallelize(Seq(("cat", 30), ("fox", 40)...

I am creating one Spark-Cassandra App (Spark 1.6.0 & spark-cassandra-connector 1.6.0-M1), in which i am asking multiple users to enter their Cassandra properties like Host, Username, Password, ...

I have large Cassandra table. I want to load only 50 rows from Cassandra. Following code val ds = sparkSession.read .format("org.apache.spark.sql.cassandra") .options(Map("table" -> s"$Aggreg...

I have a problem when i use spark streaming to read from Cassandra. https://github.com/datastax/spark-cassandra-connector/blob/master/doc/8_streaming.md#reading-from-cassandra-from-the-streamingco...

First of all, I am not using the DSE Cassandra. I am building this on my own and using Microsoft Azure to host the servers. I have a 2-node Cassandra cluster, I've managed to set up Spark on a si...

I'm trying to use the spark-cassandra-connector via spark-shell on dataproc, however I am unable to connect to my cluster. It appears that there is a version mismatch since the classpath is includi...

I have these configuration set for Spark but every time either I am reading from or writing to Cassandra table I am getting ioException .setMaster(sparkIp) .set("spark.cassandra.connection.host"...

I have the following piece of code in Spark: rdd .map(processFunction(_)) .saveToCassandra("keyspace", "tableName") Where def processFunction(src: String): Seq[Any] = src match { case "a" =...

I have the following code I invoke spark-shell as follows ./spark-shell --conf spark.cassandra.connection.host=170.99.99.134 --executor-memory 15G --executor-cores 12 --conf spark.cassandra.input...

When trying to save data to Cassandra(in Scala), I get the following exception: java.lang.ClassCastException: com.datastax.driver.core.DefaultResultSetFuture cannot be cast to com.google.commo...

I am new to Spark and Cassandra. On trying to submit a spark job, I am getting an error while connecting to Cassandra. Details: Versions: Spark : 1.3.1 (build for hadoop 2.6 or later : spark-1.3...

The exact Exception is as follows com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [varchar <-> java.math.BigDecimal] These are the version...

I have a streaming process which reads data from Kafka, processes it using Spark and writes data to Cassandra. This will run on cluster which will have 3 - 5 nodes. My plan is to deploy spark, kaf...

How do I collect these metrics on a console (Spark Shell or Spark submit job) right after the task or job is done. We are using Spark to load data from Mysql to Cassandra and it is quite huge (ex:...

I'm facing an issue with spark cassandra connector on scala while updating a table in my keyspace Here is my piece of code val query = "UPDATE " + COLUMN_FAMILY_UNIQUE_TRAFFIC + DATA_SET_DEVICE +...

I had a issue while loading data into Spark as dataframe from cassandra table which contains Tuple type. My system specification is given below. Spark : 1.6.0 Spark Cassandra Connector : 1.6.0-M...