Repository navigation
refactor: convert filters to Iceberg predicates with Option - #117
NoahKusaba wants to merge 1 commit into
Conversation
Filter conversion signalled failure through NotTransformed variants on two ad-hoc enums, and one enum mixed columns and literals with finished predicates. Return Option and use `?`, take columns and literals from `column()` and `literal()`, and track exact versus implied predicates in `Converted`. Swap `literal op column` with `Operator::swap()` instead of a hand-written table with an `unreachable!`. The conversion is unchanged. Adds tests for the cases the refactor touches that weren't covered, and TODOs for three follow-ups.
Yes, happy to have a look later today. I took a cursory glance just now and a lot of it seems similar to the physical predicate converter that I've been working on here . I removed a bunch of changes in my last commit to simplify the PR, but I had more-or-less the same structure for exact vs inexact filters. |
Wasn't aware of your work in this area 😄 , really excited to see a resolution on apache/iceberg-rust#3675 |
Which issue does this PR close?
N/A. Refactor; no behavior change.
Rationale for this change
read/predicate.rstranslates a query'sWHEREfilters into an IcebergPredicate, which Iceberg uses to skip data files that can't hold a matching row. DataFusion applies the original filters again after the scan. So a filter that isn't translated only means more files get read. A filter translated wrongly can skip matching rows, and they silently disappear from the result. So this code has two jobs:The current code doesn't make either job easy to see:
Optionand?. Here it's a custom enum with aNotTransformedvariant that appears 25 times. Most match arms exist only to return it, which buries the arms that do the translating.bcomes back as a column rather than a predicate. Inb AND foo > 1the AND logic treats it as untranslatable and drops it. That is safe, but it's easy to miss when reading the code.1 < fooasfoo > 1uses a hand-written operator table ending inunreachable!. DataFusion already providesOperator::swap().Upcoming work will add to this file: LIKE
ESCAPE, decimal literals, and boolean columns inside AND/OR. Each of those is a chance to break rule 2, so the structure should make the rules obvious before the file grows.What changes are included in this PR?
Optionand uses?. Columns and literals come fromcolumn(expr)andliteral(expr), so only predicates are combined.Converted::{Exact, Implied}replacesPredicate/PartialPredicate; the AND, OR and NOT soundness rules are unchanged.literal op columnuses DataFusion'sOperator::swap(), removingreverse_predicate_operator.Referencebuilders (is_null,is_in,starts_with, ...) instead of building expressions by hand.TODO: a boolean column as an AND/OR operand, LIKEESCAPE, andDecimal128literals.Are there any user-facing changes?
No.
convert_filters_to_predicatereturns the same predicates.Are these changes tested?
Yes. Adds unit tests for behavior the refactor touches that wasn't covered: every reversed comparison (
1 <= foo, ...), a bare boolean column inside AND and OR,LIKE ... ESCAPE, and comparisons that aren't column-to-literal.They pass on
maintoo. I also compared the old and new conversion on about 40,000 generated filters; all results were identical.