I need to select a compression algorithm when configuring "well-known application".
Also, as part of my day job, my company is developing distributed application that deal with a fair amount of data. We've been looking into compressing data to try to reduce network bandwidth, but we are hitting a wall on what algorithm to use. There's too many choices.
How do I decide between LZ4 and Snappy?
TL;DR The answer is always LZ4.
First, let's discuss what they have in common
They are both algorithms that are designed to operate at "wire" speed (order of 1 GB/s per core) when compressing and decompressing.
The main use case is to apply compression before writing data to disk or to network (that usually operate nowhere near GB/s). Compress data to reduce IO, it's transparent since the compression algorithm is so fast -faster than reading/writing from the medium-.
Both algorithms appeared in early 2010s and can be considered relatively recent. It takes a good decade for newer technologies to gain adoption and for optimized stable libraries to emerge in all popular languages.
They are both widely usable now and have good libraries available (I am writing this in 2021), however that wasn't the case a few years ago .
They are both compressing at a similar speed and a similar compression ratio (except decompression speed where LZ4 is much faster).
For historical reference there is a third algorithm called LZO that plays in the same league, it's much older (paper from 1996) and not widely used.
Second, let's discuss the differences.
While they are both extremely fast, LZ4 is (slightly) faster and stronger, hence it should be preferred.
In particular when it comes to decompression speed, LZ4 is multiple times faster.
LZ algorithms are generally extremely fast at decompression (they can operate in constant time), that's one of the reasons they are popular. LZ4 was constructed to fully take advantage of that property and saturate CPU/memory bandwidth.
In addition, LZ4 is tunable, the compression level can be finely tuned from 1 to 16, which allows to have stronger compression if you have CPU to spare. It would be great if all software supporting LZ4 should would expose the compression level as a setting, but not all do.
"faster is better" of course, yet, you may be tempted to ask if it really matters at these kind of speed? Do we care to do 1 GB/s or 2 GB/s per core?
The answer is yes, because the effect is noticeable and on-the-wire compression should keep up with the hardware it's running on, including NVMe SSD (750+ MB/s) and local network (1.25+ GB/s).
For client-server applications where the server will be receiving and decompressing many streams from many clients, the cost of decompression can add up real quick. One practical example is distributed queues like Kafka that have to decompress/recompress data on the fly, adapting to whatever formats the many clients can send/receive.
Another major use case is databases, where data can be compressed before being stored on disk. A well-known example is ElasticSearch, data is compressed with LZ4 out-of-the-box (internally data is immutable/append-only which works very well with compression and logs), when you run a query on the last month of logs it could be decompressing Terabytes of data on the fly (1 GB/s doesn't sound that quick anymore ;) )
Third, compatibility and availability of libraries
Last but not least, you will need to find some libraries to support whichever compression you intend to use.
Or, if we are talking about tuning a third party application/database, you will need to see what algorithms can be configured.
As of 2021 when I am writing this answer, there are mature libraries available in all popular languages for LZ4 (and snappy (and ZSTD)).
If you're the guy developing software that could benefit from wire-speed compression, you should use LZ4. If you are looking for a stronger compression -albeit slower- you can look into ZSTD instead. Forget about snappy.
One exception may be for some Java software, that has may support snappy but not lz4.
A Bit of History and Software Archeology
There is one edge case around java software. Snappy had an optimized java implementation for much longer, notably driven by Kafka. There's a good chance you ended up on this post because you were looking into tuning Kafka compression.
Kafka settled on snappy compression early-on and required all kafka clients (in all languages) to support snappy. It drove snappy adoption and further optimization.
If you see old comparisons that puts snappy ahead, for example this extensive Kafka benchmark from CloudFlare from 2018. The reason it does is because the article is old and LZ4 wasn't equally supported/optimized at the time (CloudFlare couldn't use lz4 in the end because not all clients supported it at the time).
It's hard work to retrofit more compression algorithms into an existing systems. LZ4 (and ZSTD) should be supported now but your mileage may vary. You may need to upgrade your cluster and upgrade your client libraries. You may find that some client libraries don't support it. The difference between snappy vs lz4 being thin, it's not worth the hassle to tune if you've got either working fine.
On a side node. If you run across multiple datacenters and find yourself heavily limited by the network, you should have a look into ZSTD, that has a much stronger compression (can reduce network traffic by 2 or 3).
LZ4 is mature and widely usable now, it wasn't as much pre-2020 (same with snappy outside of java). Many software are seeing noticeable performance improvements by adopting LZ4 and then further improvements as the libraries are deeply optimized.
© 2022 - 2024 — McMap. All rights reserved.
