Skip to content

refactor: convert filters to Iceberg predicates with Option - #117

Open
NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/predicate-conversion
Open

NoahKusaba wants to merge 1 commit into
apache:mainfrom
NoahKusaba:refactor/predicate-conversion

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A. Refactor; no behavior change.

Rationale for this change

read/predicate.rs translates a query's WHERE filters into an Iceberg Predicate, which Iceberg uses to skip data files that can't hold a matching row. DataFusion applies the original filters again after the scan. So a filter that isn't translated only means more files get read. A filter translated wrongly can skip matching rows, and they silently disappear from the result. So this code has two jobs:

  1. Give up on anything it can't translate. This is the most common outcome.
  2. Never produce a predicate that excludes a matching row.

The current code doesn't make either job easy to see:

  • Giving up is spelled out by hand. Rust expresses "this may fail" with Option and ?. Here it's a custom enum with a NotTransformed variant that appears 25 times. Most match arms exist only to return it, which buries the arms that do the translating.
  • One type holds two different kinds of thing. The same enum holds finished predicates and the raw pieces they're built from (columns and literals). Every caller has to check which kind it got. For example, a bare boolean column b comes back as a column rather than a predicate. In b AND foo > 1 the AND logic treats it as untranslatable and drops it. That is safe, but it's easy to miss when reading the code.
  • The "never exclude a matching row" rule is spread out. AND may drop a side, OR may not, and NOT may only negate an exact translation. These rules are enforced by variant names scattered across several functions. Nothing in the types states them.
  • It reimplements DataFusion. Rewriting 1 < foo as foo > 1 uses a hand-written operator table ending in unreachable!. DataFusion already provides Operator::swap().

Upcoming work will add to this file: LIKE ESCAPE, decimal literals, and boolean columns inside AND/OR. Each of those is a chance to break rule 2, so the structure should make the rules obvious before the file grows.

What changes are included in this PR?

  • Conversion returns Option and uses ?. Columns and literals come from column(expr) and literal(expr), so only predicates are combined.
  • Converted::{Exact, Implied} replaces Predicate/PartialPredicate; the AND, OR and NOT soundness rules are unchanged.
  • literal op column uses DataFusion's Operator::swap(), removing reverse_predicate_operator.
  • Uses iceberg's Reference builders (is_null, is_in, starts_with, ...) instead of building expressions by hand.
  • Marks three follow-ups with TODO: a boolean column as an AND/OR operand, LIKE ESCAPE, and Decimal128 literals.

Are there any user-facing changes?

No. convert_filters_to_predicate returns the same predicates.

Are these changes tested?

Yes. Adds unit tests for behavior the refactor touches that wasn't covered: every reversed comparison (1 <= foo, ...), a bare boolean column inside AND and OR, LIKE ... ESCAPE, and comparisons that aren't column-to-literal.

They pass on main too. I also compared the old and new conversion on about 40,000 generated filters; all results were identical.

Filter conversion signalled failure through NotTransformed variants on
two ad-hoc enums, and one enum mixed columns and literals with finished
predicates. Return Option and use `?`, take columns and literals from
`column()` and `literal()`, and track exact versus implied predicates in
`Converted`. Swap `literal op column` with `Operator::swap()` instead of a
hand-written table with an `unreachable!`.

The conversion is unchanged. Adds tests for the cases the refactor
touches that weren't covered, and TODOs for three follow-ups.
@NoahKusaba
NoahKusaba marked this pull request as ready for review November 18, 2026 16:01

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Perseus could you review this one? as (I think) you are more or less familiar with iceberg->df expression conversion

@Perseus

Perseus commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

@Perseus could you review this one? as (I think) you are more or less familiar with iceberg->df expression conversion

Yes, happy to have a look later today. I took a cursory glance just now and a lot of it seems similar to the physical predicate converter that I've been working on here . I removed a bunch of changes in my last commit to simplify the PR, but I had more-or-less the same structure for exact vs inexact filters.

@NoahKusaba

Copy link
Copy Markdown
Contributor Author

@Perseus could you review this one? as (I think) you are more or less familiar with iceberg->df expression conversion

Yes, happy to have a look later today. I took a cursory glance just now and a lot of it seems similar to the physical predicate converter that I've been working on here . I removed a bunch of changes in my last commit to simplify the PR, but I had more-or-less the same structure for exact vs inexact filters.

Wasn't aware of your work in this area 😄 , really excited to see a resolution on apache/iceberg-rust#3675

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants