Cassandra compaction tasks number keep growing

I`m using Cassandra dsc 2.1.5 with 3 nodes, and the following table description:

cqlsh> DESCRIBE TABLE mykeyspace.mytable;

CREATE TABLE mykeyspace.mytable (
a text,
b text,
c timestamp,
d timestamp,
e  text,
PRIMARY KEY ((a, b), c)
) WITH CLUSTERING ORDER BY (c ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';

The first and the third nodes are working fine (all have the same cassandra.yaml) but the second node started to have more and more pending compaction tasks that I can see with nodetool compactionstats -H command. The situation is so bad that my spark jobs get stuck and work only when I completely shut down the second node.

I have around 130G free on the second node...

Also, here a cfstatss state:

> nodetool cfstatss mykeyspace.mytable
Keyspace: mykeyspace
    Read Count: 0
    Read Latency: NaN ms.
    Write Count: 54316                                                                                                                         [1/1863]        Write Latency: 0.1877597945356801 ms.
    Pending Flushes: 0
            Table: mytable
            SSTable count: 1249
            Space used (live): 1125634027755
            Space used (total): 1125634027755
            Space used by snapshots (total): 0
            Off heap memory used (total): 1202327957
            SSTable Compression Ratio: 0.11699340657338655
            Number of keys (estimate): 34300801
            Memtable cell count: 758856
            Memtable data size: 351011415
            Memtable off heap memory used: 0
            Memtable switch count: 10
            Local read count: 0
            Local read latency: NaN ms
            Local write count: 54319
            Local write latency: 0.188 ms
            Pending flushes: 0
            Bloom filter false positives: 0
            Bloom filter false ratio: 0.00000
            Bloom filter space used: 48230904
            Bloom filter off heap memory used: 48220912
            Index summary off heap memory used: 11161093
            Compression metadata off heap memory used: 1142945952
            Compacted partition minimum bytes: 925
            Compacted partition maximum bytes: 52066354
            Compacted partition mean bytes: 299014
            Average live cells per slice (last five minutes): 0.0
            Maximum live cells per slice (last five minutes): 0.0
            Average tombstones per slice (last five minutes): 0.0
            Maximum tombstones per slice (last five minutes): 0.0

What can be the problem?

Recommended topics

Hot tags