Strange exception in /var/log/cassandra/system.log

Question

Unexpected errors in the Cassandra logs, haven't been able to trace down the underlaying cause yet. What component utilise Netty, or is this problem well known? (couldn't find any info)

INFO  [SharedPool-Worker-1] 2016-05-18 13:47:41,004  Message.java:532 - Unexpected exception during request; channel = [id: 0xe93fe01e, /40.68.XX.XXX:50818 :> /10.1.XX.X:9042]
io.netty.channel.unix.Errors$NativeIoException: readAddress() failed: Connection timed out
        at io.netty.channel.unix.Errors.newIOException(Errors.java:105) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.unix.Errors.ioResult(Errors.java:121) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.unix.FileDescriptor.readAddress(FileDescriptor.java:134) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:239) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:822) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:348) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.34.Final.jar:4.0.34.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

Hanni

Does /40.68.XX.XXX:50818 :> /10.1.XX.X:9042 have a route? Can you ping the 10.1.XX.X address using the interface that is assigned 40.68.XX.XXX (ping -I <pub_interface> 10.1.XX.X)? (hint: 10/8 is private and 40.68/16 is public, so if that public interface does not have a VPN connection or similar to the 10. address, that's not going to work) – Corrinacorrine

Answer 1

If you are using Gocql driver for Cassandra and getting this please see if adding the keepAlive setting to Connection configuration would solve this. The read timeout was happening and this configuration helped us

type ConnConfig
type ConnConfig struct {
    ProtoVersion   int
    CQLVersion     string
    Timeout        time.Duration
    ConnectTimeout time.Duration
    Compressor     Compressor
    Authenticator  Authenticator
    Keepalive      time.Duration
    // contains filtered or unexported fields
}

https://godoc.org/github.com/gocql/gocql

Thrombophlebitis

Answer 2

Seems like the newer NETTY version is more verbose but in any case it is a harmless INFO message - you may change your logback config if you rather not see it.

Northwestwards