OperationTimedOut: errors={}, last_host=127.0.0.1

Question

I am using a single node Cassandra and I intend to run some queries in order to check the response time. In some queries, after 10s of execution occurs to me the following error:

OperationTimedOut: errors = {}, last_host = 127.0.0.1

So I ran the following command:

sudo gedit /usr/bin/cqlsh.py

And changed cqlsh.py file:

# cqlsh should run correctly when run out of a Cassandra source tree,
# out of an unpacked Cassandra tarball, and after a proper package install.
cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
if os.path.isdir(cqlshlibdir):
    sys.path.insert(0, cqlshlibdir)

from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
                                 RED, FormattedValue, colorme)

from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
                                 DEFAULT_TIMESTAMP_FORMAT, DateTimeFormat,
                                 format_by_type, format_value_utype,
                                 formatter_for)

from cqlshlib.tracing import print_trace, print_trace_session
from cqlshlib.util import get_file_encoding_bomsize, trim_if_present

DEFAULT_HOST = '127.0.0.1'
DEFAULT_PORT = 9042
DEFAULT_CQLVER = '3.3.1'
DEFAULT_PROTOCOL_VERSION = 4
DEFAULT_CONNECT_TIMEOUT_SECONDS = 240

DEFAULT_FLOAT_PRECISION = 5
DEFAULT_MAX_TRACE_WAIT = 300

However, when I try to run the query again, cql return the same error after 10s:

OperationTimedOut: errors = {}, last_host = 127.0.0.1

What I have to do so that the query has no answer timeout?

Trivet

Answer 1

Are you executing these queries in cqlsh?

If so, you are hitting the client request timeout (not the connect timeout, nor the server-side read request timeout).

You can change the default timeout by setting one in ~/.cassandra/cqlshrc:

[connection]
client_timeout = 20
# Can also be set to None to disable:
# client_timeout = None

See https://issues.apache.org/jira/browse/CASSANDRA-7516 for more detail.

I see from another comment you are already aware of paging. This will be the best approach because it does not require you to marshal the entire result set in memory at the data and app tiers.

Honegger

which is the directory that I should change default timeout in ~/.cassandra/cqlshrc? I cannot find ... – Trivet

I do not intend to use the "paging" because I'm doing a query performance comparison between RDBMS and Cassandra. I intend to get the full result of the query and check the response time... With "paging" option I think it is not possible... – Trivet

You can either put it at the path specified (~/.cassandra/cqlshrc), or specify an alternate path on the cqlsh command line: github.com/apache/cassandra/blob/cassandra-2.2.3/bin/… – Honegger

Answer 2

The latest version of cassandra allows you to specify cqlsh timeout when you use it, instead of having to edit your cqlshrc file.

cqlsh --request-timeout <your-timeout>

Zayin

Answer 3

You'll see a handful of responses telling you how to raise the various timeouts, but the real answer is that you almost never want to raise those timeouts, because if you have a real data set, you will kill your server (or drop requests/mutations) with lots of long-running queries. You are better off using paging and more short-running queries than huge, long running queries.

Shrimp

I needed the timeout raise for interactive inspection while benchmarking. So the "real answer" is in the eye of the beholder ... ;-) – Pernod

Answer 4

You have to change the read_request_timeout_in_ms parameter in the cassandra.yaml file. And then restart Cassandra.

Gladdie

I have changed the [read_request_timeout] for '5000000' and the error continues. I'm working with one node, then I thought to change the 'cqlsh.py' file should solve the problem but not... – Trivet

The cassandra.yaml file you have to edit is the one of the node not the one of the cqlsh client. Is it what you did ? Did you restart the node ? – Gladdie

I'm just doing some tests in a family of columns with 10 million records. If I run the query 'select * from tail_by_time', after 10 seconds occurs to me that error. If I 'Paging ON' the column family, the query runs in groups of 100 rows. But I want the full results to check response time... – Trivet