Add ALP Encoding blog - #195
Conversation
|
@alamb still need to add some charts, but the text is ready for review. cc. @devanbenz |
|
Amazing -- thank you @sdf-jkl |
|
BTW there is a vote thread to add ALP here: https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt 
|
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @sdf-jkl -- this is a great start. I am sorry it has taken so long to get you feedback on this.
I have left quite a few comments. Let me know if they make sense.
My core feedback is that I recommend we reframe this piece to focus and explain what was added to Parquet, rather than ALP more broadly. I am thinking that this blog will likely the first page that someone will find when they search for ALP in parquet (or that an agent will find / summarize for them). Ideally it has all the info about when to choose ALP (what data / systems are good candidates) and then a technical overview of how it works.
Finally, I would love to be a coauthor on this post with you if you are willing
100% 🔥 |
|
Ok great -- thank you. I plan to focus on getting the code / PR ready in rust first and then come back to this one. If you haven't had a chance to incorporate the feedback by then, no worries, I can take a direct pass; That is probably sometime next week (hopefully) |
This comment was marked as outdated.
This comment was marked as outdated.
|
This is a branch with the blog bench - https://github.com/sdf-jkl/arrow-rs/tree/alp-benchmarks Example output on my machine: DetailsBenchmark environment
Parquet compression results
GB/s is decimal billions of uncompressed input bytes processed per second; higher is better. Compressed size includes Parquet data-page headers but excludes the file footer. Speed processes every value in pages of up to 131072 values and excludes file I/O. PLAIN + ZSTD includes both stages: PLAIN encoding plus ZSTD compression, and ZSTD decompression plus PLAIN decoding. Short pages are repeated for timing stability and normalized to one page. Random accessTime to decode 100 deterministic, uniformly distributed rows from
PLAIN and ALP reset the page decoder, skip to the selected row, and decode one value. PLAIN + ZSTD additionally decompresses the complete target page for every independent lookup. Encoded pages are already in memory; file I/O and page lookup are excluded. 30 datasets. Arithmetic mean: PLAIN 64.01, PLAIN + ZSTD 22.75, ALP 24.27 bits/value. |
|
the output looks great |
|
My hope is in the next day or two to take a pass on this doc and try to incorporate feedback |
|
I'll address the existing feedback myself, would be down for another review 😃 |
|
I'll address the existing feedback myself, would be down for another review 😃 As in you plan to address it and update this PR? Also I think @prtkgaur was working on a blog post as well and he may wish to help out. |
I meant I'll address your existing comment on this PR.
🔥 |
|
I am working on some diagrams and charts for this post |
|
Lmk if you're working on the rest of your suggestions too. I was working on the prose earlier and don't want our work to conflict. |
I was not going to do any pose editing likewise because I didn't want our work to conflict I'll work on diagrams and charts instead and then when you have something you can push it up -- that way we won't conlflict |
This comment was marked as outdated.
This comment was marked as outdated.
|
Here are some charts https://docs.google.com/spreadsheets/d/1ZSo8Rx3bPzCGkRn4VNivivZzuiIVUfBFy7QNVXdLfhM/edit 
I haven't really dug into the numbers yet or tried yet to reproduce the results (aka is gov26 really so much better than ALP, or is there something that ) 
But I think this gives some ideas For the actual post, I think we can pick a few representative data sets that reflect the pattern |
|
I also wanted to add byte stream split + zstd to the bench as it's another way to store floats |
|
Sorry for taking a while. I can post changes more frequently so we're on the same page. |
Sure -- I also want to be careful not to treat this as an academic paper (really explore the ins/outs). I think we need enough data to back up the key points, but not a full on treatment. Comparing to other parquet encoding implementations is a good idea |
|
Here is the result of running the benchmark on a gcp mchine Details
Benchmark environment
Parquet compression results
GB/s is decimal billions of uncompressed input bytes processed per second; higher is better. Compressed size includes Parquet data-page headers but excludes the file footer. Speed processes every value in pages of up to 131072 values and excludes file I/O. PLAIN + ZSTD includes both stages: PLAIN encoding plus ZSTD compression, and ZSTD decompression plus PLAIN decoding. Short pages are repeated for timing stability and normalized to one page. Random accessTime to decode 100 deterministic, uniformly distributed rows from
PLAIN and ALP reset the page decoder, skip to the selected row, and decode one value. PLAIN + ZSTD additionally decompresses the complete target page for every independent lookup. Encoded pages are already in memory; file I/O and page lookup are excluded. 30 datasets. Arithmetic mean: PLAIN 64.01, PLAIN + ZSTD 22.75, ALP 24.27 bits/value. |
|
Thanks again to all for the great work. Question+discussion to this: #195 (comment). I see this comment, but I'm not sure if there's a plan to keep such text in this blog. The meta-question from me is whether we should treat this blog as a positioning. By positioning, I mean showing that Parquet with ALP is just as good as the newer file formats (I won't name-drop here). The positioning should give the reader a clear takeaway on Parquet: future Parquet with ALP is the file format to go with.
|
👍
I don't think it is appropriate for Parquet to try and position itself against other formats. In my mind:
I would love to hear other opinions as well |
|
Update here: Once we merge these two PRs Parquet 2.14 will be available. Then I will refresh this PR (and probably update the adoption section a bit) and get it pubished |
|
Ok, now we have a few more PRs to publish 2.14 format changes to the website. Once those are in, I will polish up the links and get this one out for a final round of review |
|
I am adding some final touchups here;
I will re-update the preview site and update the screen shots and then send it out to the mailing list one more time. I hope to merge early next week (though we still need a committer to approve) |
|
We can rerun ALP encoding benchmarks now that bit-packing is SIMD on arrow-rs 🤓 |
We can indeed -- though I think the numbers in the current post stand for themselves already quite well. It would be interesting to see how much the bit packing improvements (apache/arrow-rs#10432 for anyone else who is curious) improve things |
|
BTW sorry for obsessing over this blog -- I have spent way more time than I would like to admit on it, but I am quite proud of the result |
|
|
||
| ## Why ALP? | ||
|
|
||
| Encoding floating-point data is a complicated engineering problem due to the nature of floating-point values. They do not exactly represent most real values. This leads to rounding errors that prevent using existing lightweight encodings like Delta and Frame of Reference (FOR). |
There was a problem hiding this comment.
Most lines wrap around 80-100 characters. This one doesn't.
| <p/> | ||
| </div> | ||
|
|
||
| Note that these numbers are for the pre-release Rust implementation of ALP, and |
There was a problem hiding this comment.
would be add a link the the performance numbers that we got for c++ impl too?
https://docs.google.com/spreadsheets/d/1NmCg0WZKeZUc6vNXXD8M3GIyNqF_H3goj6mVbT8at7A/?
(to show cross language performance improvement?)
There was a problem hiding this comment.
Maybe exclude PLAIN from the table's conditional formatting? RN it looks like everything (including ALP) is bad compared to it. Instead we want to show that ALP performs better than every other encoding.
There was a problem hiding this comment.
I think this is a comment for @prtkgaur for the spreadsheet. I agree that it was somewhat hard to fully grok the numbers there quickly without study
vinooganesh
left a comment
vinooganesh
left a comment
There was a problem hiding this comment.
lgtm!(from my non-work account)
|
I plan to publish this tomorrow Sep 22, 2026 |
|
|
||
| <div class="row g-3 td-max-width-on-larger-screens"> | ||
| <div class="col-12 col-md-6"> | ||
| <img src="/blog/alp/avg_compression_ratio.png" alt="Average compression ratio benchmark" class="img-fluid"> |
There was a problem hiding this comment.
There's no point in comparing the compression ratio between different machines. The algorithm stays the same
There was a problem hiding this comment.
that is technically true. Though it currently has a nice symmetry -- if we changed it then one of the charts will be visually different than the others which might cause confusion 🤔
| <p/> | ||
| </div> | ||
|
|
||
| Note that these numbers are for the pre-release Rust implementation of ALP, and |
There was a problem hiding this comment.
No longer pre-release numbers
There was a problem hiding this comment.
I didn't regenerate the numbers with the released version 🤔 Though I don't think we made any changes, I would want to regenerate the numbers if we claimed it was with the released version
| <p/> | ||
| </div> | ||
|
|
||
| Note that these numbers are for the pre-release Rust implementation of ALP, and |
There was a problem hiding this comment.
Maybe exclude PLAIN from the table's conditional formatting? RN it looks like everything (including ALP) is bad compared to it. Instead we want to show that ALP performs better than every other encoding.
| value = encoded × 10<sup>f</sup> × 10<sup>-e</sup> | ||
| </pre> | ||
|
|
||
| This calculation uses floating-point arithmetic, which rounds to the nearest |
There was a problem hiding this comment.
This is the first mention of floating-point arithmetic in the blog. There's another link to IEEE 754, but we might add a link to a wiki page explaining the range of representable values?
There was a problem hiding this comment.
I feel like I would like to assume readers of this post will have a basic familiarity with floating point (or can look it up themselves if they have questions). If they don't understand floating point is not exact I think there is a lot of this blog that wont make sense.
Co-authored-by: Kosta Tarasov <33369833+sdf-jkl@users.noreply.github.com>
|
Thank you everyone for your comments and reviews. This has been a long time in the making |
8 participants
Rendered Preview: https://alamb.github.io/parquet-site/blog/2026/09/18/alp-adaptive-lossless-floating-point-encoding-in-apache-parquet/