Repository navigation
feat(ipc): Skip unchanged dictionaries in ArrowIpcWriterWriteArrayViewWithDictionaries - #979
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #979 +/- ##
==========================================
+ Coverage 78.52% 78.56% +0.04%
==========================================
Files 106 106
Lines 17195 17334 +139
Branches 2064 2108 +44
==========================================
+ Hits 13502 13619 +117
- Misses 2431 2432 +1
- Partials 1262 1283 +21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3aa0b89 to
57c4810
Compare
…wWithDictionaries ArrowIpcWriterWriteArrayViewWithDictionaries() wrote every dictionary before every RecordBatch: its input is borrowed, so it couldn't compare a dictionary with the previous batch's by address as ArrowIpcWriterWriteArrayStream() does. The writer now keeps a copy of each dictionary it writes and compares the next batch's values with it, writing only dictionaries that changed (and those whose values contain a nested dictionary that was written). In a file, batches whose dictionaries are unchanged can now be written this way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
57c4810 to
8f8e296
Compare
paleolimbot
left a comment
There was a problem hiding this comment.
Sorry for taking a while to circle back here.
This is great! We may want the ability to disable this (maybe ArrowIpcWriterSetDeduplicateDictionariesByValue()) just in case there is an issue with it for some reason.
A few comments inline, happy to do another round whenever you have the time to circle back.
| /// each dictionary it writes to compare with the next batch's (so the comparison costs | ||
| /// a copy of each dictionary per batch); values that can't be copied (e.g., a dense | ||
| /// union) are written every time. A changed dictionary is written in full, replacing |
There was a problem hiding this comment.
While I think it will be a while until somebody is up to the task of dictionary encoding a dense union, remind me why it can't be copied?
| ArrowErrorCode result = ArrowArrayStartAppending(out); | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayReserve(out, values_view->length); | ||
| } | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayAppendStorageFromArrayView(out, values_view, error); | ||
| } | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayFinishBuildingDefault(out, error); | ||
| } |
There was a problem hiding this comment.
Also elsewhere in this PR, but we typically handle this by nesting the pieces that can fail (but whose failure requires cleanup) in an inner function, so we only check the result and clean up once.
| ArrowErrorCode result = ArrowIpcWriterArrayViewInitLike(&lhs_view, values_view); | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowIpcWriterArrayViewInitLike(&rhs_view, values_view); | ||
| } | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayViewSetArray(&lhs_view, lhs, error); | ||
| } | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayViewSetArray(&rhs_view, rhs, error); | ||
| } | ||
| if (result == NANOARROW_OK) { | ||
| result = ArrowArrayViewCompare(&lhs_view, &rhs_view, NANOARROW_COMPARE_IDENTICAL, out, | ||
| NULL); | ||
| } |
| // Initialize out as an array view with the same types as src | ||
| static ArrowErrorCode ArrowIpcWriterArrayViewInitLike(struct ArrowArrayView* out, | ||
| const struct ArrowArrayView* src) { |
There was a problem hiding this comment.
If we don't have a ArrowArrayViewInitFromArrayView(), I think this would be a good name for it and a better place for it would be nanoarrow.h / array.c so that future internals reuse it. I thought we had an internal one already but that may have been a different combination.
| // Copy the values of a dictionary into out, in a form that only depends on the values: | ||
| // zero offset, buffers sized to the values, and cleared bitmap padding. A dictionary | ||
| // nested in the values is not copied: it is written (and compared) separately. | ||
| static ArrowErrorCode ArrowIpcWriterCopyDictionaryValues( |
There was a problem hiding this comment.
You can have claude double check this, but Codex flagged for me that null storage slots aren't necessarily equal but may be considered equal by ArrowArrayViewCompare(). I believe we can either fix that in ArrowArrayViewCompare, fix that in AppendStorage (by writing canonical null slots), or flag that as a limitation of this function in a comment. In this case, I think that would be rare and wouldn't result in a correctness issue (just an extra emitted dictionary).
Rebased onto current
main; prerequisites #928 and #968 have merged. This PR now contains only the dictionary value deduplication change.ArrowIpcWriterWriteArrayViewWithDictionaries()(#968) writes every dictionary before every RecordBatch. Its input is borrowed, so it can't do whatArrowIpcWriterWriteArrayStream()does (#928): keep the previous array alive and compare dictionaries by buffer address. A producer that writes batches one at a time with the same dictionary therefore sends that dictionary again with every batch. In a file, where a dictionary ID may be written only once, it can't write a second batch with dictionaries at all.The writer now keeps a copy of each dictionary this function writes, keyed by dictionary ID, and compares the next batch's values with it:
ArrowArrayViewCompare(..., NANOARROW_COMPARE_IDENTICAL, ...)). This is the common case, e.g. when the producer builds each batch's dictionary the same way, and it needs no copy.ArrowArrayAppendStorageFromArrayView()doesn't support dense unions) are written every time, as before.ArrowIpcWriterWriteDictionaryBatch()drops the saved copy for its ID, so a dictionary written that way is never skipped by mistake, andArrowIpcWriterWriteSchema()drops all of them.ArrowIpcWriterWriteArrayStream()is unchanged. The function's documentation now describes this, including the cost: a copy of each dictionary that is written.Cost: 1000 batches of 2048 rows,
dictionary<int32, string>with 10,000 values, written to a memory buffer (Release, Linux aarch64; best of 3):A dictionary that changes on every batch costs a copy per batch. The follow-up, #980, writes a dictionary that only grew as a delta, which brings that row to 346 ms and 9.2 MB.
Tests (
writer_test.cc):WriteArrayViewWithDictionariesSkipsUnchanged: separately built arrays with dictionaries[foo, bar], [foo, bar], [foo, baz], [foo, baz]write two DictionaryBatches. Read back with nanoarrow and Arrow C++.WriteArrayViewWithDictionariesComparesValues: anint64dictionary[1, null, 3], then the same values as a slice of a longer array with an unknown null count and padding bits set, write one DictionaryBatch.WriteArrayViewWithDictionariesNested: when only a nested dictionary changes, both it and the dictionary that refers to it are written (IDs1, 0, 1, 0over three batches).WriteArrayViewWithDictionariesAfterWriteDictionaryBatch: a dictionary written withArrowIpcWriterWriteDictionaryBatch()in between isn't mistaken for the saved copy.WriteArrayViewWithDictionariesUncopyableValues: a dense union dictionary is written for every batch.WriteArrayViewWithDictionariesInFilenow writes a second, unchanged batch to the file, checks that a changed dictionary returns ENOTSUP, and reads the file back with Arrow C++.Against #968's
writer.c, the first three new tests and the updated file test fail.ctestpasses (Linux aarch64, Debug, 878 tests withNANOARROW_BUILD_TESTS_WITH_ARROW), and 403 tests under ASan/UBSan with clang. The writer tests are clean under valgrind. The changed files are formatted with clang-format 16.0.6, the pre-commit version.This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer.
🤖 Generated with Claude Code
Validation after rebasing (October 26): macOS arm64 Debug, 426 CTest cases, no failures; 16 compression cases skipped because compression is disabled in this build.