Issue with Hector API and Cassandra database: Undocumented exception

Question

Whenever I use any Hector API function to access my Cassandra database, I get an exception:

me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.

My server does have Cassandra database running in the background.

I read up on the exception and it is effectively undocumented. It appears that the exception is due to connection issues.

How do I fix it?

Stavro

Answer 1

You will get that error if the Hector client can't connect to Cassandra. There can be a number of reasons for this and things to try:

• Make sure the connection properties in your code (ip/host/port) are configured correctly.
• Make sure you can connect to it with cassandra-cli remotely -- it could be a networking issue.
• Try posting your connection code here -- perhaps there's a problem there.

Clea

Answer 2

I get this error randomly due to issues with network connectivity, but retrying several times usually fixes it. Here's the code I use to retry Hector API functions:

/** An interface where inside the execute() method I call Hector */
public interface Retriable<T> {
    T execute();
}

/**
 * Executes operation and retries N times in case of an exception
 * @param retriable
 * @param maxRetries
 * @param <T>
 * @return
 */
public static <T> T executeWithRetry(Retriable<T> retriable, int maxRetries) {
    T result;
    int retries = 0;
    long sleepSec = 1;
    // retry in case of an exception:
    while (true) {
        try {
            result = retriable.execute();
            break;
        } catch (Exception e) {
            if (retries == maxRetries) {
                LOG.error("Exception occurred. Reached max retries.", e);
                throw e;
            }
            retries++;
            LOG.error(String.format("Exception occurred. Retrying in %d seconds - #%d", sleepSec, retries), e);
            try {
                Thread.sleep(sleepSec * 1000);
                // increase sleepSec exponentially:
                sleepSec *= 2;
            } catch (InterruptedException e1) {
                e1.printStackTrace();
            }
        }
    }
    return result;
}

And an example on how to use it:

    ColumnFamilyResult<String, String> columns = executeWithRetry(new Retriable<ColumnFamilyResult<String, String>>() {
        @Override
        public ColumnFamilyResult<String, String> execute() {
            return template.queryColumns(row.getKey());
        }
    });

Clintclintock

Answer 3

I was getting the same error with cassandra-unit 2.0.2.1 but Downgrading the version to 2.0.2.0 solved the problem. That's strange but I used 2.0.2.0 for now with some additional sbt dependencies

         "com.datastax.cassandra" % "cassandra-driver-core" % "2.0.1",
         "org.cassandraunit" % "cassandra-unit" % "2.0.2.0" withSources() withJavadoc()

Phineas