Cassandra / Hector Timeouts = What to do?

Question

i would be thankfull if a sophisticated user could name all possible solutions (best practices) how to fix Hector Client Timeouts like this:

Caused by: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()


Caused by: TimedOutException()
    at org.apache.cassandra.thrift.Cassandra$multiget_slice_result.read(Cassandra.java:9628)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_multiget_slice(Cassandra.java:636)
    at org.apache.cassandra.thrift.Cassandra$Client.multiget_slice(Cassandra.java:608)
    at me.prettyprint.cassandra.service.KeyspaceServiceImpl$10.execute(KeyspaceServiceImpl.java:388)
    ... 21 more

HECTOR:

Taken from the Hector Documentation :https://github.com/rantav/hector/wiki/User-Guide

I found the following related to timeouts:

1.) cassandraThriftSocketTimeout

CASSANDRA:

1.) rpc_timeout_in_ms: 10000 (in cassandra.yaml)

What other Settings are available related to timeouts both on Hector and on Cassandra side? I have time! So I simply want to wait longer! But I have not found the settings therefor to wait longer.

Thanks Markus

Finger

For future reference, were you using an Ordered or Random Partitioning? Since I see you were using multiget_slice , then I assume no use of Secondary Indexes too? – Belindabelisarius

Answer 1

From the cassandra.thrift API in the Apache Cassandra source tree regarding TimeoutException:

"RPC timeout was exceeded. either a node failed mid-operation, or load was too high, or the requested op was too large."

In short you were asking for too much data. What sort of query were you sending? Can you post a code snippet of such?

Graupel

hello zznate. Thanks very much for your answer!!! What I am trying to do is querying and sending batches of 100 Records/Rows: Each record has about 15 Columns and the total size in memory of each record is about 1KB. This is a ridicolous low amout of data. The only thing is, it runs on my Windows Dev Environment and Windows does alot of paging, so it seems then, Cassandra just is busy. But looking at the Cassandra "logs"(windows console), the Garbage Collection is in the range of millisecons... I have not seen any information about compactions in the logs(?)... I will try further... – Finger

Keep an eye on IO and memory usage in combination with 'nodetool cfstats' (paying particular attention to "Read Latency" and "Write Latency") and that should give you some more information on where to find the bottle neck. If it does seem like excessive paging, try turning down the heap size and memtable thresholds as well. – Graupel