Repository navigation
chore: Update to arrow/parquet 60.1.0
#26867
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
5a12062
chore: Update to arrow 60.1.0
alamb 19e5457
trim comments
alamb e17a2c4
cleaning up
alamb 2d6fe15
Update sqllogictest expectations for arrow-rs 60.1.0 behavior changes
alamb 5051604
Move behavior-change explanations out of slt comments
alamb 26d9ba0
Merge branch 'main' into alamb/arrow_60.1.0
alamb bee542f
Merge remote-tracking branch 'apache/main' into alamb/arrow_60.1.0
alamb 601a217
simplify test
alamb dedf179
Merge branch 'main' into alamb/arrow_60.1.0
Jefffrey aaa4740
fix datafusion-testing submodule
Jefffrey 4e77671
clarify versions
alamb e7ddd45
Merge remote-tracking branch 'apache/main' into alamb/arrow_60.1.0
alamb File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -819,8 +819,8 @@ fn gc_array(array: &ArrayRef) -> Result<(ArrayRef, bool)> { | |
| /// The IPC writer writes every data buffer of an array in full, once per | ||
| /// entry, and `gc()` copies the bytes of every view separately. Neither | ||
| /// handles views that share bytes, as the views of a dictionary-encoded | ||
| /// Parquet column do: after `concat` or `interleave` the dictionary is often | ||
| /// listed once per input batch, and `gc()` copies it once per row | ||
| /// Parquet column do: after `concat` the dictionary is listed once per input | ||
| /// batch, and `gc()` copies it once per row | ||
| /// (<https://github.com/apache/datafusion/issues/23564>). | ||
| /// | ||
| /// Repeated buffer entries are merged first. `gc()` then runs only if it | ||
|
|
@@ -1985,15 +1985,15 @@ mod tests { | |
| // After appending each batch, spilled_rows and spilled_bytes should increase incrementally, | ||
| // while spill_file_count remains 1 (since we're writing to the same file) | ||
| in_progress_file.append_batch(&batch1)?; | ||
| verify_metrics(&in_progress_file, 1, 440, 3)?; | ||
| verify_metrics(&in_progress_file, 1, 424, 3)?; | ||
|
|
||
| in_progress_file.append_batch(&batch2)?; | ||
| verify_metrics(&in_progress_file, 1, 704, 6)?; | ||
| verify_metrics(&in_progress_file, 1, 672, 6)?; | ||
|
|
||
| let completed_file = in_progress_file.finish()?; | ||
| assert!(completed_file.is_some()); | ||
| assert!(completed_file.unwrap().path().unwrap().exists()); | ||
| verify_metrics(&in_progress_file, 1, 712, 6)?; | ||
| verify_metrics(&in_progress_file, 1, 680, 6)?; | ||
| // Double finish produce error | ||
| let result = in_progress_file.finish(); | ||
| assert!(result.is_err()); | ||
|
|
@@ -2128,7 +2128,7 @@ mod tests { | |
|
|
||
| // Metrics should be updated immediately (at least schema and first batch) | ||
| let bytes_after_batch1 = metrics.spilled_bytes.value(); | ||
| assert_eq!(bytes_after_batch1, 440); | ||
| assert_eq!(bytes_after_batch1, 424); | ||
| assert_eq!(metrics.spill_file_count.value(), 1); | ||
|
|
||
| // Check global progress | ||
|
|
@@ -2493,16 +2493,16 @@ mod tests { | |
| } | ||
|
|
||
| /// Arrays that each hold views into one shared dictionary buffer, as the | ||
| /// Parquet reader produces for a dictionary-encoded column, combined the | ||
| /// way a sort combines its input batches. Each output row cycles through | ||
| /// Parquet reader produces for a dictionary-encoded column, combined with | ||
| /// `concat` as a sort combines its buffered batches. The rows cycle through | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. interleave no longer duplicates the buffers |
||
| /// the dictionary (<https://github.com/apache/datafusion/issues/23564>). | ||
| fn shared_dictionary_views( | ||
| dictionary_len: u32, | ||
| inputs: u32, | ||
| rows_per_input: u32, | ||
| ) -> Result<(StringViewArray, StringViewArray)> { | ||
| use arrow::array::{AsArray, UInt32Array}; | ||
| use arrow::compute::{interleave, take}; | ||
| use arrow::compute::{concat, take}; | ||
|
|
||
| let dictionary = dictionary_array(dictionary_len)?; | ||
| let arrays = (0..inputs) | ||
|
|
@@ -2515,17 +2515,14 @@ mod tests { | |
| }) | ||
| .collect::<std::result::Result<Vec<_>, _>>()?; | ||
| let arrays: Vec<&dyn Array> = arrays.iter().map(|a| a.as_ref()).collect(); | ||
| let indices: Vec<(usize, usize)> = (0..rows_per_input as usize) | ||
| .flat_map(|row| (0..inputs as usize).map(move |input| (input, row))) | ||
| .collect(); | ||
| let merged = interleave(&arrays, &indices)?; | ||
| let merged = concat(&arrays)?; | ||
| Ok((dictionary, merged.as_string_view().clone())) | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_gc_keeps_shared_dictionary_buffer() -> Result<()> { | ||
| let (dictionary, merged) = shared_dictionary_views(1000, 8, 1024)?; | ||
| // `interleave` lists the dictionary once per input | ||
| // `concat` lists the dictionary once per input | ||
| assert_eq!(merged.data_buffers().len(), 8); | ||
| // `gc()` would copy 8192 values of 63 bytes, 8x the dictionary | ||
| assert!(merged.total_buffer_bytes_used() > data_buffers_size(&dictionary)); | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Due to
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
size is smaller 🎉