Cassandra cleanup on several servers at once

Question

We have a big Cassandra cluster 18 Servers (on one server near 5T data )

http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html - We have added a new nodes following this documentation .

After we have added new servers, we began the process of cleaning data (nodetool cleanup)

In the documentation advise: After all new nodes are running, run nodetool cleanup on each of the previously existing nodes to remove the keys no longer belonging to those nodes. Wait for cleanup to complete on one node before doing the next)

But cleanup for one server takes near 2 - 3 days in our case. My question is can I start cleaning at once on multiple servers, 2 or 3 ...

Or it may lead to data loss ?

Some more info .

We use cassandra 2.0.13 with vnodes . Also We keep files in blons in cassandra .

Replication factor = 3

Theophilus

Answer 1

Cleanup doesn't involve any other nodes so it is safe to run in parallel. However, you may want to run on one at once to reduce the performance impact since cleanup may use lots of disk I/O.

Yale

For the record, Richard is right, but in Cassandra 2.1 and higher, there's another concern - cleanup is run in parallel across all sstables in a CF, using all compactors. Compaction will fall behind on that node, and depending on ingestion volume, you may have hundreds or thousands of sstables on disk, which can impact read performance significantly. For that reason, running cleanup on one node at a time may help maintain low latency on reads during cleanup. – Solarium