Cassandra server throws java.lang.AssertionError: DecoratedKey(...) != DecoratedKey

Question

I'm currently experimenting around with Cassandra.

On the client-side (with Hector) I look up a few keys like this:

ColumnFamilyResult<String, String> result = template.queryColumns(Arrays.asList("key1","key2","key3"));

Most of the time it seems to work. But other times I get a timeout exception on the client:

Caused by: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
    at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:100)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.sliceInternal(ThriftColumnFamilyTemplate.java:88)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate.doExecuteSlice(ThriftColumnFamilyTemplate.java:46)
    at me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.queryColumns(ColumnFamilyTemplate.java:113)
    at info.gamlor.experiments.Cassandra.readObjectByKey(ComplexCassandra.java:255)

Caused by: TimedOutException()
    at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7772)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
    at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
    at me.prettyprint.cassandra.service.template.ThriftColumnFamilyTemplate$1.execute(ThriftColumnFamilyTemplate.java:95)

And on the server this exception shows up:

ERROR 11:33:55,312 Exception in thread Thread[ReadStage:91,5,main]
java.lang.AssertionError: DecoratedKey(4948402862350542345439897754126541659, 6932) != DecoratedKey(132475956107784875457507977471906551877, 726f6f74) in C:\tem
p\cassandra\lib\cassandra\data\CassandraPolepos\ComplexObjects\CassandraPolepos-ComplexObjects-hd-2-Data.db
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:58)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1128)
        at org.apache.cassandra.db.Table.getRow(Table.java:378)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Sometimes the key-values in the DecoratedKey(...) part takes up pages.

Anyone a hint what I'm doing wrong. Or how to investigate this issue.

Thanks.

Kristakristal

That looks like a bug that was fixed a really long time ago: issues.apache.org/jira/browse/CASSANDRA-866 Can you reproduce with a modern version of Cassandra? (1.0.10 or 1.1.2) – Witching

Yes, I can reproduce it with Cassandra 1.1.2, on a Windows box. I'll try to come up with a minimalistic example, which creates this issue. – Kristakristal

Yes, same here with with newest Cassandra and Hector 1.0-5. ERROR [ReplicateOnWriteStage:537] 2012-08-16 13:10:15,673 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ReplicateOnWriteStage:537,5,main] – Superficial

@Gamlor, did you ever solve this? I'm seeing the same problem. – Abert

@JasonAxelson No...never solved it. For me it only occurred in a test-suite, where I dropped and recreated the keyspace for test runs. The work around for that suite was to create new, with a unique name, namespace for the test runs. That works for me, but not a real solution. Really should write a nice test case for this...damn I'm lazy. – Kristakristal

Answer 1

This is either https://issues.apache.org/jira/browse/CASSANDRA-4687 or https://issues.apache.org/jira/browse/CASSANDRA-5202 You can run the command "nodetool invalidatekeycache" on all of the server showing the error. If it keeps coming back you may want to disable key cache. If it is 5202 it should only happen after dropping and recreating a column family using the same name.

Bye

Answer 2

To me it looks like it was a bug.

With the later version (1.1.5) the error disappeared. At least for me.

Kristakristal

Sadly, I'm still seeing this in 1.1.8 on linux. Restarting the cassandra service removes the error... (put here for googling purposes) – Behave

Answer 3

This looks like https://issues.apache.org/jira/browse/CASSANDRA-4687, which is not yet fixed. A workaround is to disable the key cache.

Chappelka