How to read the cassandra nodetool histograms percentile and other columns?

Question

How to read the cassandra nodetool histograms percentile and other coulmns?

Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)                  
50%             1.00             14.24           4055.27               149                 2
75%            35.00             17.08          17436.92               149                 2
95%            35.00             24.60          74975.55               642                 2
98%            86.00             35.43         129557.75               770                 2
99%           103.00             51.01         186563.16               770                 2
Min             0.00              2.76             51.01               104                 2
Max           124.00       36904729.27       12359319.16               924                 2

Trammel

Answer 1

They show the distribution of the metrics. For example, in your data the write latency for 95% of the requests were 24.60 microseconds or less. 95% of the partitions are 642 bytes or less with 2 cells. The SStables column is how many sstables are touched on a read, so 95% or read requests are looking at 35 sstables (this is fairly high).

Sandal

Thank you! Does it make sense to say 99% of read requests have touched 103 SStables and 95% of read requests have touched 35 SStables? – Trammel

yeah, with a -c option (in some 2.0 versions, later versions remove it) you can see the actual histogram values. You probably have had just a few requests that are really bad, touching over 100 sstables. – Sandal

when I say 99% of read requests have touched 103 how is that very few? I think I missing a major point. – Trammel

thats not very few, thats a lot. 103 is bad. but since its only at 99th percentile it means that youve had very few requests that have taken that many – Sandal

I see. so is it appropriate to say 1% of read requests have touched 103 SStables? – Trammel

Kind of, you can say the worst 1% touched 103 or more sstables. The bottom 99% touched 103 or less. You know the max is 124 sstables so theres between 103 and 124 sstables for the worst 1% of queries. – Sandal

This percentile thing is not very intuitive because you seem to be ok with saying "worst 1% touched 103 or more sstables" and "99% of read requests have touched 103" while they both sound contradictory – Trammel

"99% of read requests have touched 103 or less". check out en.wikipedia.org/wiki/Percentile which may describe it better. "A percentile (or a centile) is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall." – Sandal

Lets say I assume that is true but 99% of read requests are not very few! But you second comment says "You probably have had just a few requests that are really bad, touching over 100 sstables". That is what I dont seem to understand – Trammel