Programmatically flush data to cassandra every time before cassandra shut down

Question

I am using embedded Cassandra. When I shut down and restart my Cassandra service data is lost. I think decent data are not properly flushed into the disk. So I tried using nodetool to flush data manually and check if data are available. But nodetool doesn't seem to work properly for embedded Cassandra service. I get the following error:

c:\vijay\cassandra\bin>nodetool -host 192.168.2.86 -p 7199 drain

Starting NodeTool

Failed to connect to '192.168.2.86:7199': Connection refused: connect

I tried setting jmx properties still I am getting error. I added following lines to my code:

System.setProperty("com.sun.management.jmxremote", "true");
System.setProperty("com.sun.management.jmxremote.port", "7197");
System.setProperty("com.sun.management.jmxremote.authenticate", "false");
System.setProperty("com.sun.management.jmxremote.ssl", "false");
System.setProperty("java.rmi.server.hostname", "my ip");

So, is there any way to manually flush data to Cassandra without using nodetool?

Edit 1:
After hours of trying I am now able to run nodetool (instead of adding jmx configurations to the code I added to Eclipse debug configurations and it worked). I ran drain command now the data is properly flushed to the disk. So now my question is: why isn't data properly flushed? Every time when I restart Cassandra service recent changes are gone.

Hammered

Data shouldn't be flushed until the memtable has reached a certain size. Before that time the persistance is guaranteed by the commit-log which is flushed at the time of write acknoweldgement. – Attar

That's where the problem is.. In my case, i guess commit logs are not properly flushed.. when i flush the data using nodetool before shutting the service down, it works good, datas are available after server restart.. So is there any way to programatically flush data every time when server goes down?? – Hammered

I think you should try to figure out why the commit log is not flushing. It should be constantly writing and flushing unless you disabled fsync or fsync doesn't behave properly on your system. Once this is fixed you will not lose data on restart and you won't have to manually flush. – Attar

see wiki.apache.org/cassandra/Durability, log is flushed periodically by default – Stature

Answer 1

Commitlogs are not properly flushed in cassandra versions 1.1.0 to 1.1.4 This is a open issue. Please refer the following jira ticket.

Commitlog not replayed after restart

Hammered

Answer 2

How are you stopping and starting the Cassandra server? The call to stopServer on the Cassandra daemon should be flushing any outstanding writes to the commit log. The thread will continue to do some processing even after the method returns, so if you're killing the JVM after stopServer() then you might be preventing the data from being written.

Bathos

Ya i tried a proper shutdown, even then problem persists.. apparently it is due to this open ticket issues.apache.org/jira/browse/CASSANDRA-4782 .. I upgraded my cassandra to higher version, which solved the issue for me.. No other go..! – Hammered

Answer 3

Most likely because of your commitlog settings. If you are using periodic mode, try reducing the sync window or your use batch. Batch will make writes slightly slower.

https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L231

Hawsepiece

I have tried all possible changes in cassandra.yaml related to this issue .. no good :( – Hammered

Answer 4

It may be that a flush is issued but is failing. Flushing should be recorded in the logs. Verify that you are not flushing. If you are flushing and it is failing, there should be some indication of why flush failed.

Papuan