Repository navigation
Fix dict-encoded BYTE_ARRAY decimal decode in Parquet reader - #25349
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
No actionable comments were generated in the recent review. Summary
Merge Risk:
|
| auto footer = | ||
| cudf::io::read_parquet_footers(cudf::io::make_datasources(cudf::io::source_info{ | ||
| cudf::host_span<char const>{buffer.data(), buffer.size()}})) | ||
| .front(); | ||
| auto& schema = footer.schema[1]; | ||
| schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL; | ||
| schema.decimal_scale = scale; | ||
| schema.decimal_precision = precision; | ||
| schema.logical_type = cudf::io::parquet::LogicalType{ | ||
| cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}}; | ||
|
|
||
| // Swap the trailing footer for the annotated one, keeping the data pages | ||
| uint32_t footer_len{}; | ||
| std::memcpy(&footer_len, buffer.data() + buffer.size() - 8, sizeof(footer_len)); | ||
| auto const new_footer = parquet_delta_test::serialize_footer(footer); | ||
| buffer.resize(buffer.size() - 8 - footer_len); | ||
| buffer.insert(buffer.end(), new_footer.begin(), new_footer.end()); | ||
| footer_len = new_footer.size(); | ||
| buffer.insert(buffer.end(), | ||
| reinterpret_cast<char const*>(&footer_len), | ||
| reinterpret_cast<char const*>(&footer_len + 1)); | ||
| buffer.insert(buffer.end(), {'P', 'A', 'R', '1'}); |
There was a problem hiding this comment.
Once #25339 merges, this can use the shared edit_footer helper from parquet_common.hpp:
| auto footer = | |
| cudf::io::read_parquet_footers(cudf::io::make_datasources(cudf::io::source_info{ | |
| cudf::host_span<char const>{buffer.data(), buffer.size()}})) | |
| .front(); | |
| auto& schema = footer.schema[1]; | |
| schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL; | |
| schema.decimal_scale = scale; | |
| schema.decimal_precision = precision; | |
| schema.logical_type = cudf::io::parquet::LogicalType{ | |
| cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}}; | |
| // Swap the trailing footer for the annotated one, keeping the data pages | |
| uint32_t footer_len{}; | |
| std::memcpy(&footer_len, buffer.data() + buffer.size() - 8, sizeof(footer_len)); | |
| auto const new_footer = parquet_delta_test::serialize_footer(footer); | |
| buffer.resize(buffer.size() - 8 - footer_len); | |
| buffer.insert(buffer.end(), new_footer.begin(), new_footer.end()); | |
| footer_len = new_footer.size(); | |
| buffer.insert(buffer.end(), | |
| reinterpret_cast<char const*>(&footer_len), | |
| reinterpret_cast<char const*>(&footer_len + 1)); | |
| buffer.insert(buffer.end(), {'P', 'A', 'R', '1'}); | |
| edit_footer(buffer, [&](cudf::io::parquet::FileMetaData& footer) { | |
| auto& schema = footer.schema[1]; | |
| schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL; | |
| schema.decimal_scale = scale; | |
| schema.decimal_precision = precision; | |
| schema.logical_type = cudf::io::parquet::LogicalType{ | |
| cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}}; | |
| }); |
shrshi
left a comment
There was a problem hiding this comment.
Non-blocking nits only. I think the PR description needs to be updated. The description says the test covers decimal32/64/128 "with and without dictionary encoding" but only the dictionary-encoded test case is present.
|
/merge |
Description
This PR fixes an illegal memory access when reading dictionary-encoded
BYTE_ARRAYdecimal columns with the Parquet reader.is_string_chunkexcluded all decimals, so the string dictionary index was never built for these chunks andgpuGetStringDataread the raw dictionary page asstring_index_pairs. The index is now built for allBYTE_ARRAYchunks (FLBA decimals still read the raw dictionary), and page setup uses it whenever it exists. Also adds a typed regression test for decimal32/64/128 with and without dictionary encoding.Checklist