What options are there to speed up a full repair in Cassandra?

Question

I have a Cassandra datacenter which I'd like to run a full repair on. The datacenter is used for analytics/batch processing and I'm willing to sacrifice latencies to speed up a full repair (nodetool repair). Writes to the datacenter is moderate.

What are my options to make the full repair faster? Some ideas:

• Increase streamthroughput?
• I guess I could disable autocompation and decrase compactionthroughput temporarily. Not sure I'd want to that, though...

Additional information:

• I'm running SSDs but haven't spent any time adjusting cassandra.yaml for this.

Revisory

The fastest way to run repair is not to use vnodes and use incremental repair. All of the other tuning you can do won't come close to the speed improvements you'll see from those two options. – Flipper

+1. You could run 8 tokens, but you have to manage them yourself. That gives a decent balance of vnode benefit but with a lot less operational hassle. – Wayside

Answer 1

Full repairs are run sequentially by default. The state and differences of the nodes' datasets are stored in binary trees. Recreating these is the main factor here. According to this datastax blog entry, "Every time a repair is carried out, the tree has to be calculated, each node that is involved in the repair has to construct its merkle tree from all the sstables it stores making the calculation very expensive."

The only way I see to significantly increase the speed of a full repair is to run it in parallel or repair subrange by subrange. Your tag implies that you run Cassandra 2.0.

1) Parallel full repair

 nodetool repair -par, or --parallel, means carry out a parallel repair.

According to the nodetool documentation for Cassandra 2.0

Unlike sequential repair (described above), parallel repair constructs the Merkle tables for all nodes at the same time. Therefore, no snapshots are required (or generated). Use a parallel repair to complete the repair quickly or when you have operational downtime that allows the resources to be completely consumed during the repair.

2) Subrange repair nodetool accepts start and end token parameters like so

 nodetool repair -st (start token) -et (end token) $keyspace $columnfamily

For simplicity sake, check out this python script that calculates tokens for you and executes the range repairs: https://github.com/BrianGallew/cassandra_range_repair

Let me point out two alternative options:

A) Jeff Jirsa pointed to incremental repairs.

These are available starting with Cassandra 2.1. You will need to perform certain migration steps before you can use nodetool like this:

nodetool repair -inc, or --incremental means do an incremental repair.

B) OpsCenter Repair Service

For the couple of clusters at my company itembase.com, we use the repair service in DataStax OpsCenter which is executing and managing small range repairs as a service.

Accent