Repository navigation
Parquet: Validate geospatial projection parameters - #17578
Conversation
There was a problem hiding this comment.
Pull request overview
Validates Parquet geospatial annotations against projected Iceberg types.
Changes:
- Reuses primitive-type conversion for validation.
- Rejects CRS and edge-algorithm mismatches.
- Adds focused geospatial projection tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
PruneColumns.java |
Validates projected geospatial parameters. |
MessageTypeToType.java |
Extracts reusable primitive conversion. |
TestPruneColumns.java |
Tests matching and mismatched parameters. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java:165
- This validates only top-level primitives in the ID-less fallback path. When a selected top-level field is a struct, list, or map, the entire Parquet group is retained unchanged, so any projected nested geometry/geography still bypasses the new CRS/edge-algorithm check and can be read with incompatible parameters. Please traverse each matched group positionally against the projected type and validate its nested primitives as well; add a nested ID-less mismatch test to cover this path.
Types.NestedField expectedField = expectedSchema.findField(ordinal);
if (type.isPrimitive() && expectedField.type().isPrimitiveType()) {
PruneColumns.validatePrimitive(
expectedField.type().asPrimitiveType(), type.asPrimitiveType());
|
@huan233usc Please take a look, thanks! |
anoopj
left a comment
There was a problem hiding this comment.
Code change LGTM. Just one comment in the test.
|
@szehon-ho @singhpk234 Could you please take a look? Thanks! |
uros-b
left a comment
There was a problem hiding this comment.
The validation itself is sound, CI is clean, and there's no backward-compat risk. Leaving a few comments for things that could be worth resolving before merge. Thank you @manuzhang for working on this! And thank you @anoopj and @huan233usc for reviewing the changes!
| int ordinal = 1; | ||
| for (Type type : fileSchema.getFields()) { | ||
| if (selectedIds.contains(ordinal)) { | ||
| Types.NestedField expectedField = expectedSchema.findField(ordinal); |
There was a problem hiding this comment.
Nested types in the fallback path - ParquetSchemaUtil.java:162-171 (pruneColumnsFallback) validates only top-level primitives: the type.isPrimitive() && expectedField.type().isPrimitiveType() guard skips any top-level struct/list/map that contains a nested geometry/geography field, so CRS/algorithm go unchecked on the id-less path (the id-based PruneColumns visitor reaches nested types fine via recursive descent). Either recurse positionally against the projected type, or add an explicit comment documenting the top-level-only scope.
|
And definitely cc @szehon-ho here for further review, given the context in GeoSpatial. |
fab9056 to
758a973
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
sorry missed it. can you rebase? also cc @huan233usc . I suppose it will be a bit of behave change (validation) but it seems fine to me |
Generated-by: Codex
Generated-by: Codex
Generated-by: Codex
Generated-by: Codex
Validate geospatial parameters recursively for ID-less nested fields and include the field path in incompatible projection errors. Generated-by: Codex
Match projected nested fields to ID-less Parquet fields only when their names establish alignment, avoiding shifted validation for compressed struct projections. Generated-by: Codex
ebf2451 to
3239fc6
Compare
|
Merged, thanks @manuzhang |
Summary
Context
Parquet geospatial values carry their CRS and edge interpolation algorithm in the logical-type annotation rather than in the WKB payload.
PruneColumnspreviously ignored these parameters, so a file could be interpreted using incompatible parameters from the projected Iceberg schema.This is the Java-side counterpart discovered while reviewing apache/iceberg-cpp#880.
Testing
./gradlew :iceberg-parquet:test --tests org.apache.iceberg.parquet.TestPruneColumns./gradlew :iceberg-parquet:spotlessJavaCheckgit diff --checkAI Disclosure