Skip to content

Parquet: Validate geospatial projection parameters - #17578

Merged
szehon-ho merged 6 commits into
apache:mainfrom
manuzhang:codex/validate-parquet-geospatial-types
Oct 21, 2026
Merged

szehon-ho merged 6 commits into
apache:mainfrom
manuzhang:codex/validate-parquet-geospatial-types

Conversation

@manuzhang

Copy link
Copy Markdown
Member

Summary

  • validate Parquet geometry and geography logical-type parameters against the projected Iceberg type
  • reuse the existing Parquet-to-Iceberg primitive conversion so omitted defaults and case-insensitive CRS matching remain consistent
  • add focused coverage for matching parameters, geometry and geography CRS mismatches, and geography edge-algorithm mismatches

Context

Parquet geospatial values carry their CRS and edge interpolation algorithm in the logical-type annotation rather than in the WKB payload. PruneColumns previously ignored these parameters, so a file could be interpreted using incompatible parameters from the projected Iceberg schema.

This is the Java-side counterpart discovered while reviewing apache/iceberg-cpp#880.

Testing

  • ./gradlew :iceberg-parquet:test --tests org.apache.iceberg.parquet.TestPruneColumns
  • ./gradlew :iceberg-parquet:spotlessJavaCheck
  • git diff --check

AI Disclosure

  • Model: GPT-5
  • Platform/Tool: Codex
  • Human Oversight: User requested the fix; this draft is awaiting human review.
  • Prompt Summary: Compare the Iceberg C++ geospatial reader behavior with Java, confirm whether Java has the same issue, and create a Java-side fix PR.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Validates Parquet geospatial annotations against projected Iceberg types.

Changes:

  • Reuses primitive-type conversion for validation.
  • Rejects CRS and edge-algorithm mismatches.
  • Adds focused geospatial projection tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
PruneColumns.java Validates projected geospatial parameters.
MessageTypeToType.java Extracts reusable primitive conversion.
TestPruneColumns.java Tests matching and mismatched parameters.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@manuzhang
manuzhang requested a balanced review from Copilot August 9, 2026 15:31
@manuzhang
manuzhang marked this pull request as ready for review August 9, 2026 15:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java:165

  • This validates only top-level primitives in the ID-less fallback path. When a selected top-level field is a struct, list, or map, the entire Parquet group is retained unchanged, so any projected nested geometry/geography still bypasses the new CRS/edge-algorithm check and can be read with incompatible parameters. Please traverse each matched group positionally against the projected type and validate its nested primitives as well; add a nested ID-less mismatch test to cover this path.
        Types.NestedField expectedField = expectedSchema.findField(ordinal);
        if (type.isPrimitive() && expectedField.type().isPrimitiveType()) {
          PruneColumns.validatePrimitive(
              expectedField.type().asPrimitiveType(), type.asPrimitiveType());

@manuzhang

Copy link
Copy Markdown
Member Author

@huan233usc Please take a look, thanks!

@anoopj anoopj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change LGTM. Just one comment in the test.

@manuzhang

Copy link
Copy Markdown
Member Author

@szehon-ho @singhpk234 Could you please take a look? Thanks!

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation itself is sound, CI is clean, and there's no backward-compat risk. Leaving a few comments for things that could be worth resolving before merge. Thank you @manuzhang for working on this! And thank you @anoopj and @huan233usc for reviewing the changes!

int ordinal = 1;
for (Type type : fileSchema.getFields()) {
if (selectedIds.contains(ordinal)) {
Types.NestedField expectedField = expectedSchema.findField(ordinal);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested types in the fallback path - ParquetSchemaUtil.java:162-171 (pruneColumnsFallback) validates only top-level primitives: the type.isPrimitive() && expectedField.type().isPrimitiveType() guard skips any top-level struct/list/map that contains a nested geometry/geography field, so CRS/algorithm go unchecked on the id-less path (the id-based PruneColumns visitor reaches nested types fine via recursive descent). Either recurse positionally against the projected type, or add an explicit comment documenting the top-level-only scope.

@uros-b

uros-b commented Sep 9, 2026

Copy link
Copy Markdown
Member

And definitely cc @szehon-ho here for further review, given the context in GeoSpatial.

@manuzhang manuzhang removed the parquet label Sep 12, 2026
@manuzhang
manuzhang force-pushed the codex/validate-parquet-geospatial-types branch from fab9056 to 758a973 Compare September 12, 2026 05:12
@manuzhang
manuzhang requested a review from szehon-ho September 15, 2026 04:09
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Oct 20, 2026
@szehon-ho

szehon-ho commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

sorry missed it. can you rebase? also cc @huan233usc . I suppose it will be a bit of behave change (validation) but it seems fine to me

@github-actions github-actions Bot removed the stale label Oct 21, 2026
Validate geospatial parameters recursively for ID-less nested fields and include the field path in incompatible projection errors.

Generated-by: Codex
Match projected nested fields to ID-less Parquet fields only when their names establish alignment, avoiding shifted validation for compressed struct projections.

Generated-by: Codex
@manuzhang
manuzhang force-pushed the codex/validate-parquet-geospatial-types branch from ebf2451 to 3239fc6 Compare October 21, 2026 14:28
@szehon-ho
szehon-ho merged commit 7a74178 into apache:main Oct 21, 2026
40 checks passed
@szehon-ho

Copy link
Copy Markdown
Member

Merged, thanks @manuzhang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants