saveToCassandra with spark-cassandra connector throws java.lang.ClassCastException

Question

When trying to save data to Cassandra(in Scala), I get the following exception:

java.lang.ClassCastException: com.datastax.driver.core.DefaultResultSetFuture cannot be cast to com.google.common.util.concurrent.ListenableFuture

Please note that I do not get this error every time, but it comes up randomly once in a while which makes it more dangerous in production.

I am using YARN and I have shaded com.google.** to avoid the Guava symbol clash.

Here's the code snippet:

rdd.saveToCassandra(keyspace,"movie_attributes", SomeColumns("movie_id","movie_title","genre"))

Any help would be much appreciated.

UPDATE Adding details from the pom file as requested:

<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.10</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector-java_2.10</artifactId>
    <version>1.5.0</version>
</dependency>

**Shading guava**

<relocation> <!-- Conflicts between Cassandra Java driver and YARN -->
    <pattern>com.google</pattern>
    <shadedPattern>oryx.com.google</shadedPattern>
    <includes>
         <include>com.google.common.**</include>
    </includes>
 </relocation>

Spark version: 1.5.2 Cassandra version: 2.2.3

Torry

Can you share the Cassandra version, connector version, and spark version you are using, as well as how you shaded the library? – Yesseniayester

@Christophe, I have updated the question to reflect component version numbers. – Torry

I have the same problem. It occurrs in a unit test. When running the test inside IntelliJ, it works, but sbt myProject/test fails. – Stromboli

Answer 1

Almost everyone who works on C* and Spark has seen these type of errors. The root cause is explained here.

C* driver depends on a relatively new version of guava while Spark depends on an older guava. To solve this before connector 1.6.2, you need to explicitly embed C* driver and guava with your application.

Since 1.6.2 and 2.0.0-M3, by default connector ships with the correct C* driver and guava shaded. So you should be OK with just connector artifact included in your project.

Things get tricky if your Spark application uses other libraries that depend on C* driver. Then you will have to manually include un-shaded version of connector, correct C* driver and shaded guava and deploy a fat jar. You essentially make your own connector package. In this case, you can't use --package to launch Spark cluster anymore.

tl;dr

use connector 1.6.2/2.0.0-M3 or above. 99% you should be OK.

Ellora