Repository navigation
fix(arrow): filter absent nullable columns in ordered comparisons - #3508
mattfaltyn wants to merge 4 commits into
Conversation
| })) | ||
| } | ||
|
|
||
| fn missing_field_is_null(&self, reference: &BoundReference) -> bool { |
There was a problem hiding this comment.
not_in and not_starts_with still return build_always_true() for a missing nullable column. On this branch b NOT IN ('x', 'y') and b NOT STARTS WITH 'x' give 3 rows when b is missing from the file and 0 when it's stored as NULL. Same mismatch as <, so maybe worth covering here, or a follow-up?
There was a problem hiding this comment.
Thanks for catching this! Both predicates now treat an absent nullable column like a stored NULL. I added Parquet regression coverage, and the local Rust and Python suites passed.
There was a problem hiding this comment.
Re-ran my probe on 90649d6. NOT IN and NOT STARTS WITH now give 0 rows for the missing column, same as the stored-NULL file. NOT (b < 'm'), IS NULL and IS NOT NULL match too. LGTM.
Which issue does this PR close?
Closes #3355.
What changes are included in this PR?
<and<=keep every row when an optional column is absent from a Parquet file, even though the same column stored as NULL matches no rows. Treat the absent column as NULL when it has no initial default or non-null identity partition value.Required-field validation and existing default/partition-constant handling are preserved. The broader evaluation of non-null missing values remains in #3302; this fix can land independently.
Are these changes tested?
Local validation passed: 7 focused tests; 2,513 workspace nextest tests (1 pre-existing ignored test); 96 doctests (19 ignored examples); and 41 Python tests (1 expected failure). Workspace build, Clippy, documentation, formatting, TOML, dependency/license checks, MSRV, no-default-feature build, and all 10 public API baselines passed. API comparisons used a path-safe equivalent of the Make recipe for a checkout path containing a space. Live HuggingFace tests were gated off without credentials.
All 25 CI checks passed. The audit reproduction passes all 32 comparison configurations, including pruning switches and projected/filter-only columns; 16 failed before the fix.
AI Disclosure
Codex assisted with investigation, implementation, tests, and this description. Review should focus on preserving required-field errors and default/identity-partition behavior while changing only the known-NULL comparison fallback. Non-null missing-value filtering and null-partition/default precedence remain outside this fix.