lzmpo is an experimental compressor built around the following idea: when looking for a match to, say, string "abc123456", the parser will find many places where data starts with "abcX", such that X is not "1", and very few places where data starts with "abc123".
It builds hashchains (linking positions with the same hash of the data) for several lengths of the hashed substring (called prefix) which cover the entire file. Due to this, better matches can be found quicker by searching along the hashchain built for the greatest prefix length. Another upside is that compression ratio gets better as the size of the input file grows, outperforming zstd on enwik9. Additionally, decompression speed stays at about 70-80% that of zstd, and memory usage is about 1.5*N bytes where N is the size of the file.
However, the greatest downside is memory usage during compression, as each hashchain requires 4*N bytes to be stored in memory. Also, compression time (on max levels) can get far beyond zstd's in the negative way.
- lzmpo operates on the entire file, which, when paired with the next ideas, allows it to reach good ratios. My calculations show that a substantial portion of the generated matches (35%) are at distances >10% of the file size, and about 15% of all matches are at distances >50% of the file size (tested on enwik8).
- lzmpo builds multiple hashchains for substrings for different lengths. When parser is looking for a match, it uses the chain that corresponds to the greatest length, which is more likely to result in a longer match.
- lzmpo splits data into blocks which are then calcualted by threads.
- Within each block, true optimal parsing is run: the parser collects all possible
matches of length up to 256 for each position withing the block, and then
proceedes to find the optimal coverage of the block with matches. My calculations
show that limiting match length by 256 is not hurting the performance, as 99% of
generated matches are of length <50 (when used with
-9level). - Cost of a match is estimated in two ways: in the first pass, it is a simple heuristic. On subsequent passes, results of previous pass are used, and cost of a token is roughly its entropy according to the stream of tokens generated by the previous pass.
- There are many entropy encoders supported, with the best one being Turbo-Range-Coder, but the second best one and the fastest one (in terms of decompression) being rans_static order0 avx2.
To explain how compression works, let's consider running the following command:
lzmpo -e -i enwik9 -o compressed --pl 5,8,16 --mm 100,50,20 -T16 -k256 -a5 -b20 --fse
The first thing that happens is enwik9 being mmaped into the memory view. After
this, an encoder class will be instantiated, and the mmaped file will be passed to
it in order to load the bytes into the compressor. Only 1e9 - 32 bytes are
considered for parsing, as the last 32 bytes are encoded as literals. This is done in
order to remove slow branches from the decoder.
After the file has been loaded, the encoder will start building the hash chains. At
first, it will calculate hashes for data of size 5 (first prefix length in the --pl
argument). To do this, 16 threads (as specified by -T16) will be created to process
16 (= number of threads) sequential equally sized blocks of data. Each thread will
calculate hashes for each position in its portion of the data, and store them for the
next step.
After hashes have been calculated, a single thread calculates the hash chain by
maintaining a hash table (let's call it head) such that when processing position
i we have head[i] = most recent position j < i such that data at that position has the same hash as data at i. After we set head[i], we update chains[0][i] = head[i], which means "in hash chain correspending to the prefix length 5 (which is
at position 0 in the --pl list) data at head[i] has the same hash as at i".
There are actually 2 types of hash tables that can be used here. First is
__gnu_pbds::gp_hash_table<uint32_t, uint32_t> which would be used if we specified
-b0. However, since we are dealing with -b20, a different structure is used. It
has an array of 2^20 integers, each corresponding to a 20-bit-long hash. Basically,
we truncate 32-bit hashes to just 20 bits, which saves space and increases speed, but
decreases precision as more collisions will occur. Note: if we specified -b32, it
would have the same effect as -b0 in terms of collisions, but would generally be
faster.
The previous 2 steps will also be repeated for prefix lengths 8 and 16, resulting in total of 3 hash chains built.
Now the encoder will begin the actual parsing of the file. The file is split into 256
(as specified by -k256) blocks, where each block will be processed by a separate of
the 16 threads. This is done to improve performance, as in the worst case only 1
threat will be left processing 1e9 / 256 bytes instead of 1e9 / 16. Basically, we
spend as much time utilizing all 16 threads as possible with higher -k values.
Each block will calculate a separate stream of tokens, and those tokens will not be able to cross the boundaries of the blocks. In the worst case scenario, we will not see 255 great (of length 256 which is the max length of a match) matches crossing those boundaries, replacing them with literals, which is at most 32KiB wasted with literals instead of long matches. After all the tokens have been calculated, they will be merged into a single stream.
Let's focus on a single block. DP is used to evaluate the best possible path (made of tokens) from the start to the end of the block. We decide which path is better by evaluating its cost, which is the sum of costs of all tokens that form that path. Cost of a token is evaluated differently for the first pass and the subsequent passes. For the first pass, cost is simply a guess. During the subsequent runs, cost of a token is calculated using the stream of tokens generated by the previous pass, which leads to much better compression ratio, but requires more time and each pass is a complete run of the parser.
Progressing through the block, at each position the encoder calculates a sheet: an
array which stores closest matches of length i for i = 1..255. To find such
matches, an optimized function is run which will use the chain for prefix length 16
and will find all matches of length 1..255 by jumping along the chain at most 20
times (element of --mm which corresponds to the last prefix length) . Note that
this iteration is very likely to result in a long match (>16 bytes). Next, same will
be done using the chain for prefix length 8, but this time the length of a match will
be limited to the range 1..15 and will jump only 50 times as the previous iteration
would have found all longer matches (well, almost, but generally this is true). Same
goes for prefix length 5.
After the matches for each length (the sheet) have been collected, they are
normalized (i.e. if a match of length 100 exists at distance 100, and a match of
length 99 exists at distance 100000, we can just take the match of length 100, ignore
the last byte, and it will be better since distance is smaller). Next, assuming
encoder is at position i, it will try to improve the path to i + len for each
len=5..255 using sheet[len]. During testing I discovered that minimal match len 5
works best with long files.
After DP has been calculated, backtracking is run and a stream of tokens is created. The thread deallocates sheet, DP arrays, and everything else that was using in that block. After this, the thread can take another block and repeat.
Once all blocks have been computed, the finalized stream of tokens is created and
validated. Next, depending on the number of passes (-a), if the current pass is not
the final one, tokens are used to improve the estimator of cost of a token.
Basic estimator is a simple guessing model. Like in zstd, tokens will be later split
into: controls (match, cache hit 0, cache hit 1, cache hit 2, literal), literals,
length and distances. Distances are split into distance group and extra bits.
Basically, distances are logarithmically binned, so that lower bins require fewer
extra bits. Cost of a token returned by the basic estimator is: 1 for controll, 7 for
literal, log2(length) + 1 if a match is in the distance cache, log2(length) + 1 + log2(distance group) + extra distance bits.
However, if we already completed the first pass, we can estimate this better. The estimator that is used for subsequent passes calculates entropy of a token based on what tokens were calculated previously. This is done for controls, literals, distances that are in cache, distances that are binned, length. Each pass recalculates entropy based on the results of the previous pass.
The last step of compression is passing tokens through some entropy encoder. lzmpo supports multiple ones:
--turborc- Turbo-Range-Coder, RC, gives best ratio and worst performance.--turboans- Turbo-Range-Coder, ANS, gives almost best ratio and good performance.--fse- finitestateentropy, FSE, good ratio and faster.--huf- finitestateentropy, HUF, good ratio and even faster.--memcpy- memcpy.....rans_static0- rans_static, r32x16b_avx2 order0, great ratio and best performance, decompression about 70-80% of zstd.rans_static- rans_static, r32x16b_avx2 order1, bad ratio probably due to poor precision.
Each stream (controls, literals, distanc groups, distance extra bits, lengths) is encoded and written to the output file.
Decompression is highly efficient as almost no memory is used.For copying, efficient simd routine is used.Branches are minimized.
During compression, memory usage is about 4*N*M + 8*N/K*T + 4*N/K*T + 4*N/K*T + 12*N/K*T = 4*N*M + 28*N/K*T bytes where N is the input file size, M is the number
of prefix lengths, K is the number of blocks, T is the number of threads.
Additionally, if -b is set to 0 < X <= 32, exactly 4*2^X bytes will be used by
the hash table.Otherwise, if its set to 0, it will use at most as many bytes as when
-b is set to log2(N)
These were created purely on my intuition and some prior testing.
-0Same as-T<supported threads> -k256 --pl 5 --mm 5 -a1-1Same as-T<supported threads> -k256 --pl 5,6,8 --mm 20,10,5 -a1-2Same as-T<supported threads> -k256 --pl 5,6,8 --mm 100,50,20 -a1-3Same as-T<supported threads> -k256 --pl 5,6,8,12 --mm 200,100,70,50 -a1-4Same as-T<supported threads> -k256 --pl 5,6,8,12,16 --mm 200,100,70,50,40 -a2-5Same as-T<supported threads> -k256 --pl 5,6,8,12,16,20 --mm 300,150,90,70,50,40 -a2-6Same as-T<supported threads> -k256 --pl 5,6,8,12,16,20 --mm 400,250,150,100,80,60 -a2-7Same as-T<supported threads> -k256 --pl 5,6,8,12,16,20,24 --mm 500,300,200,150,100,70,40 -a3-8Same as-T<supported threads> -k256 --pl 5,6,8,12,16,20,24 --mm 1000,800,600,400,200,100,100 -a4-9Same as-T<supported threads> -k256 --pl 5,6,8,12,16,20,24 --mm 5000,3000,2000,1500,800,500,300 -a4
Parameters such as -b are not set by any level. Also, if no level is specified, you
can set all parameters to custom values.
Tested compressors: zstd -8, zstd -22, xz -5, xz -9, brotli -q9, brotli -q11
These results were obtained on AMD Ryzen 7 260 (w/ AVX512) with 32GB DDR5 5600MT/s
RAM + 128GB swap space on ssd. lzmpo -9rc is the same as lzmpo -9 but with
--turborc turned on for maximal ratio.
All compressors were tested with 16 threads (except brotli). Decompression is single-threaded in all 4 compressors. Decompression was executed 5 times and the results are average of all those 5 runs.
File: enwik9
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
* lzmpo -9rc 20.07113% 0.18 51312.9 78.93 1556.8
1 lzmpo -9 20.40550% 0.18 51312.9 495.27 1559.9
2 lzmpo -8 20.42281% 0.27 51312.9 496.24 1560.8
3 lzmpo -7 20.46996% 0.43 51312.9 499.47 1561.8
4 lzmpo -6 20.53418% 0.76 47498.2 507.15 1562.3
5 lzmpo -5 20.55473% 0.86 47498.2 477.80 1563.1
6 lzmpo -4 20.60857% 1.44 32880.9 500.05 1565.2
7 lzmpo -3 20.99710% 2.66 29066.2 485.98 1578.7
8 zstd -22 21.37212% 1.76 8669.9 638.34 215.7
9 xz -9 21.56742% 5.36 5653.8 466.15 1528.2
10 lzmpo -2 22.20778% 5.13 20040.4 465.42 1632.5
11 brotli -q11 22.33457% 0.58 247.0 321.60 22.5
12 lzmpo -1 23.20203% 8.16 20119.5 454.98 1677.4
13 xz -5 23.68271% 19.21 3729.1 1047.92 1137.3
14 zstd -18 23.95552% 11.59 1698.5 969.48 151.4
15 brotli -q9 25.17618% 3.68 128.6 349.06 22.8
16 lzmpo -0 26.13254% 16.88 12858.7 435.32 1827.6
File: enwik8
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 24.86525% 1.09 1137.7 108.53 199.7
* lzmpo -9rc 25.04109% 0.46 6563.8 55.71 188.8
2 zstd -22 25.27247% 1.05 7258.7 474.99 182.9
3 lzmpo -9 25.48685% 0.47 6563.8 453.68 189.1
4 lzmpo -8 25.49408% 0.58 6563.8 462.17 189.1
5 lzmpo -7 25.52564% 0.88 6563.8 443.51 189.1
6 lzmpo -6 25.58249% 1.36 6182.3 439.21 189.1
7 lzmpo -5 25.58834% 1.47 6182.3 466.65 189.1
8 lzmpo -4 25.60556% 1.84 5800.9 451.11 189.2
9 brotli -q11 25.74200% 0.56 235.9 268.97 22.4
10 lzmpo -3 25.92686% 3.36 3783.8 443.78 190.0
11 lzmpo -2 26.71511% 5.60 3266.3 436.21 193.7
12 xz -5 26.80511% 6.99 860.8 431.80 193.8
13 zstd -18 27.33398% 5.09 907.4 825.89 151.4
14 lzmpo -1 27.55053% 7.68 2986.3 422.60 197.6
15 brotli -q9 28.88779% 3.08 128.6 313.92 22.8
16 lzmpo -0 30.34589% 16.88 2195.6 403.04 212.4
File: dickens
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 brotli -q11 27.74385% 0.78 215.7 420.55 21.9
2 xz -9 27.77167% 2.27 1137.7 165.05 93.1
3 xz -5 27.77688% 2.34 221.7 164.87 37.1
* lzmpo -9rc 27.91625% 1.02 1569.1 54.22 34.8
4 zstd -22 27.95274% 2.36 1240.9 664.02 151.4
5 zstd -18 28.10278% 2.56 631.4 646.20 151.4
6 lzmpo -9 28.27396% 1.14 1633.2 596.65 34.7
7 lzmpo -8 28.27417% 1.17 2179.2 595.10 34.7
8 lzmpo -7 28.30567% 1.45 1569.2 582.55 34.7
9 lzmpo -6 28.39041% 2.07 1594.3 583.08 34.7
10 lzmpo -5 28.39252% 2.13 1914.3 563.62 34.7
11 lzmpo -4 28.39501% 2.69 2119.4 593.78 34.7
12 lzmpo -3 28.71115% 4.99 1400.6 554.50 34.7
13 lzmpo -2 29.10215% 8.29 1567.6 556.83 34.9
14 lzmpo -1 29.81209% 9.98 2153.6 525.51 35.3
15 brotli -q9 30.45964% 7.69 107.8 433.79 22.1
16 lzmpo -0 32.94043% 21.84 1776.9 446.66 37.1
File: mozilla
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 26.11098% 3.27 1137.7 129.78 142.3
2 xz -5 26.64614% 7.82 647.7 238.36 118.6
3 brotli -q11 27.08343% 0.57 254.5 334.57 27.4
4 zstd -22 29.11749% 3.08 3929.6 603.84 151.4
5 zstd -18 29.54596% 7.09 762.0 862.91 151.4
* lzmpo -9rc 29.62109% 0.77 3454.1 31.87 115.8
6 brotli -q9 31.21380% 3.50 130.4 414.21 27.7
7 lzmpo -9 32.14679% 0.84 3375.9 448.85 116.9
8 lzmpo -8 32.15191% 1.15 3375.9 447.43 116.9
9 lzmpo -7 32.15808% 1.56 3375.9 445.01 116.9
10 lzmpo -6 32.17096% 2.14 3180.5 442.57 116.9
11 lzmpo -5 32.17752% 2.24 3180.5 443.40 116.9
12 lzmpo -4 32.19387% 2.74 2987.3 447.43 117.0
13 lzmpo -3 32.44183% 4.50 2545.9 434.69 117.4
14 lzmpo -2 32.57378% 6.17 2592.5 443.72 117.7
15 lzmpo -1 32.84073% 6.59 2232.3 426.60 118.4
16 lzmpo -0 33.53696% 13.61 2281.7 400.14 120.2
File: mr
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 27.58400% 3.75 1137.7 114.65 92.8
2 xz -5 27.60157% 3.99 221.7 114.86 36.8
3 brotli -q11 28.31471% 0.74 228.7 309.95 22.1
* lzmpo -9rc 30.50191% 1.33 1918.6 43.21 35.2
4 zstd -22 31.11561% 2.84 1240.9 644.51 151.4
5 zstd -18 31.15615% 4.54 631.4 625.84 151.4
6 lzmpo -8 31.71621% 1.52 1562.6 457.55 35.2
7 lzmpo -9 31.71649% 1.31 1562.6 445.09 35.2
8 lzmpo -7 31.79118% 1.90 1966.6 454.18 35.2
9 brotli -q9 31.86285% 8.60 105.6 371.08 22.5
10 lzmpo -6 31.96821% 2.57 1524.6 441.58 35.2
11 lzmpo -5 31.96948% 2.66 2010.6 437.01 35.2
12 lzmpo -4 31.97559% 3.12 2044.5 421.75 35.2
13 lzmpo -3 32.69999% 5.23 2054.5 415.96 35.3
14 lzmpo -2 32.78727% 7.81 2086.4 408.45 35.4
15 lzmpo -1 33.02831% 8.50 1934.4 404.98 35.6
16 lzmpo -0 34.10947% 15.25 1334.4 373.72 36.6
File: nci
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 brotli -q11 4.52939% 0.89 237.8 959.52 21.9
* lzmpo -9rc 4.66588% 0.67 2839.8 203.77 53.3
2 zstd -22 4.70767% 1.70 2137.1 1026.09 151.4
3 lzmpo -9 4.91105% 0.64 2683.8 987.64 53.3
4 lzmpo -8 4.96652% 1.13 2259.8 940.12 53.4
5 lzmpo -7 5.04959% 2.35 2817.8 995.46 53.4
6 lzmpo -6 5.12005% 3.53 2597.8 1015.41 53.5
7 lzmpo -5 5.17166% 4.08 2131.8 996.97 53.5
8 xz -9 5.18245% 6.22 1137.7 391.72 114.3
9 lzmpo -4 5.22107% 4.82 2491.8 998.11 53.6
10 zstd -18 5.47214% 6.24 631.4 1706.02 151.4
11 lzmpo -3 5.50529% 9.25 2503.8 1046.60 53.7
12 brotli -q9 5.72669% 28.95 111.2 995.34 22.2
13 lzmpo -2 6.00754% 14.23 1729.8 960.26 54.4
14 xz -5 6.04851% 15.55 434.7 543.80 74.6
15 lzmpo -1 6.75339% 16.97 1857.8 925.25 55.6
16 lzmpo -0 7.24651% 23.77 1962.3 878.23 56.3
File: ooffice
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 39.44649% 3.81 1137.7 92.02 88.8
2 xz -5 39.47998% 3.95 221.7 88.83 32.8
3 brotli -q11 40.29222% 0.59 127.2 209.69 14.8
4 zstd -22 42.18425% 4.07 696.6 448.97 151.4
5 zstd -18 42.19883% 4.53 631.4 478.86 151.4
* lzmpo -9rc 43.21520% 1.27 1901.7 22.66 30.5
6 brotli -q9 44.98998% 7.07 78.8 255.49 14.8
7 lzmpo -9 46.29252% 1.47 1431.2 304.00 30.6
8 lzmpo -8 46.29412% 1.50 1501.7 307.15 30.6
9 lzmpo -7 46.29843% 1.65 2109.7 303.27 30.6
10 lzmpo -6 46.29911% 2.11 2002.2 293.70 30.6
11 lzmpo -5 46.30062% 2.08 1570.2 307.49 30.6
12 lzmpo -4 46.31032% 2.50 1390.8 307.28 30.6
13 lzmpo -3 46.52035% 3.63 1927.3 313.86 30.6
14 lzmpo -2 46.59383% 4.90 1975.8 293.38 30.6
15 lzmpo -1 46.75818% 4.90 1343.8 283.59 30.6
16 lzmpo -0 47.47015% 11.57 1296.9 288.15 30.9
File: osdb
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 brotli -q11 27.92352% 0.86 232.2 305.87 21.9
2 xz -9 28.25704% 3.49 1137.7 133.86 93.0
3 xz -5 28.46443% 3.98 221.7 133.94 37.0
* lzmpo -9rc 29.44729% 1.07 2100.0 33.47 34.5
4 zstd -22 30.73787% 3.43 1240.9 667.48 151.4
5 lzmpo -9 30.75857% 1.19 2310.0 583.86 34.5
6 lzmpo -8 30.77433% 1.31 1810.0 564.53 34.5
7 lzmpo -7 30.81774% 1.65 2080.0 528.91 34.5
8 brotli -q9 30.82076% 7.89 100.7 406.86 21.7
9 zstd -18 30.86533% 3.97 631.4 711.82 151.4
10 lzmpo -6 31.02043% 2.31 1923.5 523.54 34.6
11 lzmpo -5 31.10672% 2.42 2053.5 574.19 34.6
12 lzmpo -4 31.34260% 2.90 1941.0 558.81 34.7
13 lzmpo -3 31.58555% 4.71 1842.6 554.05 34.8
14 lzmpo -2 32.14327% 7.08 1408.2 509.17 35.0
15 lzmpo -1 32.84479% 7.77 1464.1 521.52 35.3
16 lzmpo -0 34.00215% 14.90 1331.2 518.82 36.0
File: reymont
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 19.87505% 2.77 1137.7 191.29 88.2
2 brotli -q11 20.10136% 0.87 126.8 495.19 13.9
3 xz -5 20.23563% 3.17 221.7 182.17 32.3
* lzmpo -9rc 20.28809% 1.03 1400.8 64.25 26.6
4 zstd -22 20.34829% 2.67 696.6 590.64 151.4
5 zstd -18 20.37525% 2.96 631.4 669.90 151.4
6 lzmpo -9 20.94448% 1.17 1852.4 553.69 26.5
7 lzmpo -8 20.95954% 1.35 1400.8 611.46 25.2
8 lzmpo -7 21.00962% 1.82 1438.4 593.27 25.2
9 lzmpo -6 21.09699% 2.74 2203.2 602.20 25.2
10 lzmpo -5 21.11283% 2.87 1413.2 592.18 26.5
11 lzmpo -4 21.16771% 3.76 1387.9 629.39 26.5
12 lzmpo -3 21.48078% 6.83 1362.6 543.94 26.5
13 lzmpo -2 22.29259% 12.00 1344.6 546.04 26.8
14 brotli -q9 22.94188% 12.69 78.0 488.33 14.3
15 lzmpo -1 23.43677% 14.38 1286.7 550.62 27.2
16 lzmpo -0 26.28156% 23.79 1280.5 491.01 28.2
File: samba
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 17.41902% 5.19 1137.7 202.11 104.9
2 brotli -q11 17.43159% 0.98 248.8 474.82 23.2
* lzmpo -9rc 17.54700% 1.41 2424.6 52.30 49.7
3 zstd -22 17.89463% 2.53 2137.1 827.29 151.4
4 xz -5 17.94783% 6.63 221.7 214.11 49.0
5 zstd -18 18.18862% 5.97 631.4 1052.78 151.4
6 brotli -q9 19.23054% 11.00 117.6 607.70 23.0
7 lzmpo -9 19.83059% 1.44 2350.6 584.84 50.0
8 lzmpo -8 19.83495% 1.59 2198.6 570.17 50.0
9 lzmpo -7 19.85309% 1.96 1934.8 577.85 50.0
10 lzmpo -6 19.87844% 2.66 1852.4 569.54 50.0
11 lzmpo -5 19.88331% 2.77 2132.2 544.14 50.0
12 lzmpo -4 19.89004% 3.36 2129.7 581.05 50.0
13 lzmpo -3 20.03339% 5.94 1655.4 572.96 50.1
14 lzmpo -2 20.18512% 8.37 1573.0 651.92 50.2
15 lzmpo -1 20.47884% 8.96 1756.9 607.93 50.5
16 lzmpo -0 21.27220% 16.16 2012.0 588.57 51.3
File: sao
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -5 60.86655% 2.61 221.7 64.55 35.8
2 xz -9 60.88133% 2.70 1137.7 65.03 91.8
3 brotli -q11 63.23948% 0.59 137.9 174.00 14.1
* lzmpo -9rc 68.63543% 1.09 2062.2 13.01 37.9
4 zstd -22 68.95497% 3.45 696.6 517.56 151.4
5 zstd -18 69.00397% 3.36 631.4 518.71 151.4
6 brotli -q9 71.77238% 3.09 70.3 187.05 18.2
7 lzmpo -8 71.88409% 1.18 2134.2 271.27 38.0
8 lzmpo -9 71.88409% 1.18 1623.2 269.88 38.0
9 lzmpo -7 71.88914% 1.30 1948.2 283.04 38.0
10 lzmpo -6 71.92702% 1.63 1595.5 297.38 38.0
11 lzmpo -4 71.92715% 1.88 2206.9 269.09 38.0
12 lzmpo -5 71.92738% 1.62 1595.5 271.82 38.0
13 lzmpo -3 72.34587% 2.64 2087.2 270.30 38.0
14 lzmpo -2 72.38193% 3.34 1903.5 261.51 38.0
15 lzmpo -1 72.48236% 3.46 1663.5 275.44 38.1
16 lzmpo -0 73.39322% 7.66 1472.2 269.28 38.4
File: webster
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -9 20.22706% 1.56 1137.7 142.79 128.2
* lzmpo -9rc 20.30834% 0.56 2925.5 68.83 81.6
2 brotli -q11 20.33005% 0.67 231.7 359.02 21.9
3 zstd -22 20.33745% 1.42 3929.6 551.43 151.4
4 lzmpo -9 20.45392% 0.57 3043.5 554.01 81.5
5 lzmpo -8 20.48366% 0.82 3069.5 551.66 81.6
6 lzmpo -7 20.53826% 1.27 2927.5 534.36 81.6
7 lzmpo -6 20.61077% 1.90 2370.2 549.36 81.6
8 lzmpo -5 20.63484% 2.10 2871.3 534.34 81.7
9 lzmpo -4 20.70508% 2.55 2721.2 550.57 81.8
10 lzmpo -3 20.98722% 4.78 2785.0 534.95 82.0
11 zstd -18 21.24366% 2.58 762.0 900.20 151.4
12 lzmpo -2 21.55403% 7.97 2364.9 548.45 83.0
13 xz -5 21.72527% 3.60 434.7 246.04 88.8
14 lzmpo -1 22.41083% 10.26 2542.9 527.68 84.5
15 brotli -q9 22.96711% 4.77 123.3 372.35 22.5
16 lzmpo -0 24.67653% 19.74 2034.6 504.16 89.2
File: x-ray
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 xz -5 52.94261% 2.66 221.7 59.14 37.0
2 xz -9 52.98264% 2.66 1137.7 59.08 93.0
3 brotli -q11 55.25868% 0.54 232.7 166.77 22.5
* lzmpo -9rc 60.39444% 1.14 1597.5 19.40 40.6
4 zstd -22 60.53437% 2.54 1240.9 421.12 151.4
5 zstd -18 60.62243% 2.90 631.4 422.45 151.4
6 lzmpo -8 62.77015% 1.25 1896.3 274.32 40.7
7 lzmpo -9 62.77015% 1.23 1808.3 278.28 40.7
8 lzmpo -7 62.78223% 1.29 1826.3 274.48 40.7
9 lzmpo -4 62.84614% 1.95 1487.6 283.52 40.6
10 lzmpo -5 62.84614% 1.58 1976.0 285.31 40.6
11 lzmpo -6 62.84614% 1.57 1896.0 276.84 40.6
12 brotli -q9 63.06708% 3.07 106.9 191.55 29.8
13 lzmpo -1 63.87261% 3.87 1877.0 260.79 40.6
14 lzmpo -2 63.87261% 3.84 1793.0 264.29 40.6
15 lzmpo -3 63.87262% 2.72 1417.4 276.37 40.6
16 lzmpo -0 64.05983% 10.05 1320.5 248.72 40.8
File: xml
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
Rank Compressor Ratio C.Speed MB/s C.Mem MB D.Speed MB/s D.Mem MB
──── ─────────── ───────── ──────────── ──────── ──────────── ────────
1 brotli -q11 8.05507% 1.14 104.6 561.12 14.1
* lzmpo -9rc 8.37810% 1.83 1475.4 114.97 21.4
2 zstd -22 8.44977% 2.36 696.6 760.23 151.4
3 xz -9 8.47978% 7.63 1137.7 282.28 86.2
4 zstd -18 8.70256% 7.31 631.4 765.11 151.4
5 brotli -q9 8.97192% 34.12 69.7 539.61 14.5
6 xz -5 9.06422% 10.50 221.7 292.68 30.2
7 lzmpo -9 9.22769% 1.87 1347.4 697.77 22.3
8 lzmpo -8 9.25854% 2.10 1807.4 628.47 22.3
9 lzmpo -7 9.30367% 2.78 1731.4 653.23 22.4
10 lzmpo -6 9.32497% 3.93 2055.1 664.68 22.4
11 lzmpo -5 9.34759% 4.06 2111.1 671.20 22.4
12 lzmpo -4 9.38342% 4.69 1782.7 747.75 22.4
13 lzmpo -3 9.55929% 8.33 1840.7 695.51 22.4
14 lzmpo -2 9.76901% 10.44 1694.3 703.72 22.6
15 lzmpo -1 10.33596% 12.09 1934.3 666.51 22.7
16 lzmpo -0 10.85320% 19.79 1272.0 674.54 22.8
- a5hash for fast hashing
- Turbo-Range-Coder for
--turborcand--turboans - finitestateentropy for
--fseand--huf - rans_static for
--rans_static0and--rans_static1 - CLI11 for CLI
- misacpy which was modified by its author and me to fit lzmpo
Pull all dependencies into ./third_party and run make. It should build without
issues. To build an executable with minimal size, run make MODE=size. For speed,
set mode to speed, and for compatibility with modern systems set to release.