Skip to content

Ignore unsupported Parquet logical/physical type combinations - #25338

Draft
mhaseeb123 wants to merge 2 commits into
NVIDIA:mainfrom
mhaseeb123:claude/libcudf-logical-physical-types-9ac4e0
Draft

mhaseeb123 wants to merge 2 commits into
NVIDIA:mainfrom
mhaseeb123:claude/libcudf-logical-physical-types-9ac4e0

Conversation

@mhaseeb123

Copy link
Copy Markdown
Contributor

Description

Implements the fallback for unrecognized logical/physical type combinations from parquet-format#606.

This PR drops the logical annotation on a leaf column whose logical type is unrecognized or invalid for its physical type (e.g. INTEGER(32) on INT64, DECIMAL on BOOLEAN), and reads it as plain physical data. Min/max statistics of such columns are ignored in row-group and page-index filtering so they cannot prune using an ordering we don't understand. The check lives in metadata::sanitize_schema() so both the regular and hybrid scan readers get it.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…ir statistics

Signed-off-by: Muhammad Haseeb <14217455+mhaseeb123@users.noreply.github.com>
@mhaseeb123 mhaseeb123 added libcudf Affects libcudf (C++/CUDA) code. improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Nov 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Muhammad Haseeb <14217455+mhaseeb123@users.noreply.github.com>
Comment on lines +267 to +292
pq::FileMetaData fmd;
read_footer(cudf::io::datasource::create(filepath), &fmd);

// Make the statistics claim every value is 0, so a reader that trusts them prunes `a > 50`
std::vector<uint8_t> zero(sizeof(int64_t), 0);
auto& stats = fmd.row_groups[0].columns[0].meta_data.statistics;
stats.min = stats.max = stats.min_value = stats.max_value = zero;
if (logical_type.has_value()) { fmd.schema[1].logical_type = *logical_type; }

std::vector<uint8_t> footer;
pq::detail::CompactProtocolWriter{&footer}.write(fmd);

std::ifstream in(filepath, std::ios::binary);
std::vector<char> bytes((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
uint32_t old_len;
std::memcpy(&old_len, bytes.data() + bytes.size() - 8, sizeof(old_len));
bytes.resize(bytes.size() - 8 - old_len);
bytes.insert(bytes.end(), footer.begin(), footer.end());
auto const new_len = static_cast<uint32_t>(footer.size());
bytes.insert(bytes.end(),
reinterpret_cast<char const*>(&new_len),
reinterpret_cast<char const*>(&new_len) + sizeof(new_len));
bytes.insert(bytes.end(), {'P', 'A', 'R', '1'});

auto const patched = temp_env->get_temp_filepath(name + ".parquet");
std::ofstream(patched, std::ios::binary).write(bytes.data(), bytes.size());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #25339 merges, this can use the shared edit_footer helper from parquet_common.hpp:

Suggested change
pq::FileMetaData fmd;
read_footer(cudf::io::datasource::create(filepath), &fmd);
// Make the statistics claim every value is 0, so a reader that trusts them prunes `a > 50`
std::vector<uint8_t> zero(sizeof(int64_t), 0);
auto& stats = fmd.row_groups[0].columns[0].meta_data.statistics;
stats.min = stats.max = stats.min_value = stats.max_value = zero;
if (logical_type.has_value()) { fmd.schema[1].logical_type = *logical_type; }
std::vector<uint8_t> footer;
pq::detail::CompactProtocolWriter{&footer}.write(fmd);
std::ifstream in(filepath, std::ios::binary);
std::vector<char> bytes((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
uint32_t old_len;
std::memcpy(&old_len, bytes.data() + bytes.size() - 8, sizeof(old_len));
bytes.resize(bytes.size() - 8 - old_len);
bytes.insert(bytes.end(), footer.begin(), footer.end());
auto const new_len = static_cast<uint32_t>(footer.size());
bytes.insert(bytes.end(),
reinterpret_cast<char const*>(&new_len),
reinterpret_cast<char const*>(&new_len) + sizeof(new_len));
bytes.insert(bytes.end(), {'P', 'A', 'R', '1'});
auto const patched = temp_env->get_temp_filepath(name + ".parquet");
std::ofstream(patched, std::ios::binary).write(bytes.data(), bytes.size());
std::ifstream in(filepath, std::ios::binary);
std::vector<char> bytes((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
edit_footer(bytes, [&](pq::FileMetaData& fmd) {
// Make the statistics claim every value is 0, so a reader that trusts them prunes `a > 50`
std::vector<uint8_t> zero(sizeof(int64_t), 0);
auto& stats = fmd.row_groups[0].columns[0].meta_data.statistics;
stats.min = stats.max = stats.min_value = stats.max_value = zero;
if (logical_type.has_value()) { fmd.schema[1].logical_type = *logical_type; }
});
auto const patched = temp_env->get_temp_filepath(name + ".parquet");
std::ofstream(patched, std::ios::binary).write(bytes.data(), bytes.size());

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant