Skip to content

Fix dict-encoded BYTE_ARRAY decimal decode in Parquet reader - #25349

Merged
rapids-bot[bot] merged 5 commits into
NVIDIA:mainfrom
mhaseeb123:fix-parquet-byte-array-decimal-dict
Nov 17, 2026
Merged

rapids-bot[bot] merged 5 commits into
NVIDIA:mainfrom
mhaseeb123:fix-parquet-byte-array-decimal-dict

Conversation

@mhaseeb123

Copy link
Copy Markdown
Contributor

Description

This PR fixes an illegal memory access when reading dictionary-encoded BYTE_ARRAY decimal columns with the Parquet reader. is_string_chunk excluded all decimals, so the string dictionary index was never built for these chunks and gpuGetStringData read the raw dictionary page as string_index_pairs. The index is now built for all BYTE_ARRAY chunks (FLBA decimals still read the raw dictionary), and page setup uses it whenever it exists. Also adds a typed regression test for decimal32/64/128 with and without dictionary encoding.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the libcudf Affects libcudf (C++/CUDA) code. label Nov 16, 2026
@mhaseeb123 mhaseeb123 changed the title Fix reading dictionary-encoded BYTE_ARRAY decimals in Parquet reader Fix decoding dict-encoded BYTE_ARRAY decimals in Parquet reader Nov 17, 2026
@mhaseeb123
mhaseeb123 marked this pull request as ready for review November 17, 2026 00:21
@mhaseeb123
mhaseeb123 requested a review from a team as a code owner November 17, 2026 00:21
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Fixed reading dictionary-encoded Parquet decimal values stored as byte arrays. Supported fixed-point types now return the expected values when these columns use dictionary encoding.
    • Byte-array decimal columns are handled consistently across supported decimal precisions, including when values are represented through dictionary indexes.

The Parquet reader now assigns string dictionary indexes to BYTE_ARRAY decimal chunks and uses a non-null str_dict_index during dictionary decoding. A typed test covers dictionary-encoded decimal values across fixed-point types.

Changes: BYTE_ARRAY Decimal Dictionary Decoding

Layer / File(s) Summary
Decimal classification, dictionary decoding, and test coverage
cpp/src/io/parquet/reader_impl_preprocess_utils.cuh, cpp/src/io/parquet/page_decode.cuh, cpp/tests/io/parquet_reader_test.cpp
needs_str_dict_index now includes BYTE_ARRAY decimals. Dictionary-index counts and pointer assignment use this predicate. Dictionary decoding selects str_dict_index when it is non-null. The typed test writes dictionary-encoded binary values with DECIMAL footer annotations, reads the file, and compares the fixed-point values.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: vyasr

Merge Risk: Minimal · up to 51e80

This fixes an illegal memory access when reading dictionary-encoded BYTE_ARRAY decimal Parquet columns, and adds regression tests for the three decimal widths. No actionable merge-blocking risk remains.

Security architecture risk: Minimal · up to 51e80

This is a localized reader correction. Dictionary allocation and decoding use consistent representation selection, while existing ownership and error checks remain in place. No introduced or worsened architecture-level security risk was established.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a fix for decoding dictionary-encoded BYTE_ARRAY decimal columns in the Parquet reader. The raw summary shows the dictionary index now covers all BYTE_ARRAY chunks, in…
Description check Passed The description states the bug (an illegal memory access when reading dictionary-encoded BYTE_ARRAY decimal columns), its cause (is_string_chunk excluded decimals), the fix (string dictionary index bu…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a fix for decoding dictionary-encoded BYTE_ARRAY decimal columns in the Parquet reader. The raw summary shows the dictionary index now covers all BYTE_ARRAY chunks, including decimals, and the new regression test covers this path. The title is concise and specific, so it passes.

Full details: Description check

Explanation

The description states the bug (an illegal memory access when reading dictionary-encoded BYTE_ARRAY decimal columns), its cause (is_string_chunk excluded decimals), the fix (string dictionary index built for all BYTE_ARRAY chunks and used in page setup), and the added typed regression test. These match the raw summary of changes in page_decode.cuh, reader_impl_preprocess_utils.cuh, and parquet_reader_test.cpp, so it passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: NVIDIA/cudf
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: aa81fef
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 02f5306e-7150-4330-bc5c-965544424337
Commits

Reviewing files that changed from the base of the PR and between 937a2dc and 51e80d9.

Files selected for processing: 1

  • cpp/tests/io/parquet_reader_test.cpp

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.

@vyasr vyasr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mhaseeb123 mhaseeb123 added bug Something isn't working 4 - Needs Review Waiting for reviewer to review or respond non-breaking Non-breaking change labels Nov 17, 2026
Comment on lines +1830 to +1851
auto footer =
cudf::io::read_parquet_footers(cudf::io::make_datasources(cudf::io::source_info{
cudf::host_span<char const>{buffer.data(), buffer.size()}}))
.front();
auto& schema = footer.schema[1];
schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL;
schema.decimal_scale = scale;
schema.decimal_precision = precision;
schema.logical_type = cudf::io::parquet::LogicalType{
cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}};

// Swap the trailing footer for the annotated one, keeping the data pages
uint32_t footer_len{};
std::memcpy(&footer_len, buffer.data() + buffer.size() - 8, sizeof(footer_len));
auto const new_footer = parquet_delta_test::serialize_footer(footer);
buffer.resize(buffer.size() - 8 - footer_len);
buffer.insert(buffer.end(), new_footer.begin(), new_footer.end());
footer_len = new_footer.size();
buffer.insert(buffer.end(),
reinterpret_cast<char const*>(&footer_len),
reinterpret_cast<char const*>(&footer_len + 1));
buffer.insert(buffer.end(), {'P', 'A', 'R', '1'});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #25339 merges, this can use the shared edit_footer helper from parquet_common.hpp:

Suggested change
auto footer =
cudf::io::read_parquet_footers(cudf::io::make_datasources(cudf::io::source_info{
cudf::host_span<char const>{buffer.data(), buffer.size()}}))
.front();
auto& schema = footer.schema[1];
schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL;
schema.decimal_scale = scale;
schema.decimal_precision = precision;
schema.logical_type = cudf::io::parquet::LogicalType{
cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}};
// Swap the trailing footer for the annotated one, keeping the data pages
uint32_t footer_len{};
std::memcpy(&footer_len, buffer.data() + buffer.size() - 8, sizeof(footer_len));
auto const new_footer = parquet_delta_test::serialize_footer(footer);
buffer.resize(buffer.size() - 8 - footer_len);
buffer.insert(buffer.end(), new_footer.begin(), new_footer.end());
footer_len = new_footer.size();
buffer.insert(buffer.end(),
reinterpret_cast<char const*>(&footer_len),
reinterpret_cast<char const*>(&footer_len + 1));
buffer.insert(buffer.end(), {'P', 'A', 'R', '1'});
edit_footer(buffer, [&](cudf::io::parquet::FileMetaData& footer) {
auto& schema = footer.schema[1];
schema.converted_type = cudf::io::parquet::ConvertedType::DECIMAL;
schema.decimal_scale = scale;
schema.decimal_precision = precision;
schema.logical_type = cudf::io::parquet::LogicalType{
cudf::io::parquet::DecimalType{.scale = scale, .precision = precision}};
});

@mhaseeb123 mhaseeb123 changed the title Fix decoding dict-encoded BYTE_ARRAY decimals in Parquet reader Fix dict-encoded BYTE_ARRAY decimal decode in Parquet reader Nov 17, 2026

@shrshi shrshi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nits only. I think the PR description needs to be updated. The description says the test covers decimal32/64/128 "with and without dictionary encoding" but only the dictionary-encoded test case is present.

Comment thread cpp/tests/io/parquet_reader_test.cpp Resolved
@mhaseeb123 mhaseeb123 added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs Review Waiting for reviewer to review or respond labels Nov 17, 2026
@mhaseeb123

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit ecd0aa7 into NVIDIA:main Nov 17, 2026
194 of 196 checks passed
@mhaseeb123
mhaseeb123 deleted the fix-parquet-byte-array-decimal-dict branch November 17, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to Merge Testing and reviews complete, ready to merge bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants