Skip to content

Use internal values for temporal initial-defaults - #4435

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/initial-default-temporal-filter
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/initial-default-temporal-filter

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: when a data file lacks a date, time, timestamp or timestamptz column, = and IN miss its non-null initial-default, while != matches it. _ColumnNameTranslator compares a Python temporal value with a bound literal in days or microseconds:

# v3 table: rows 1-3 were written before `d` was added with initial-default 2020-01-01
tbl.scan(row_filter=EqualTo("d", date(2020, 1, 1))).to_arrow()  # 0 rows, expected 3

Fix: add a private NestedField accessor that returns a temporal initial-default in the literal's form (the conversion add_column uses), and use it in the translator, the Avro default reader and Record defaults. Partition-projected values are already in that form and keep precedence.

Java: SchemaParser and SchemaUpdate supply typed defaults to row-group filtering; GenericReader normalizes reader output through InternalRecordWrapper before Evaluator. This PR normalizes defaults used in internal records.

  • NestedField.initial_default stays a Python object; internal default consumers share the conversion.

Are these changes tested?

Yes, filter tests in tests/expressions/test_visitors.py and tests/io/test_pyarrow.py, plus Avro and Record default tests in tests/avro/test_resolver.py and tests/test_typedef.py. All fail on main (wrong filter result, or a Python date where an int is expected) and pass with this change.

Are there any user-facing changes?

Yes, this fixes incorrect temporal-default filter results and range errors. Avro now returns defaulted temporal fields as integer days or microseconds, matching stored fields.

Please add the changelog label.

This pull request and its description were written by Isaac.

A file that lacks a date, time, timestamp or timestamptz column with a
non-null initial-default compared the Python value with bound literals,
which hold days or microseconds.
Equality and IN dropped matching rows, != kept them, and range predicates raised TypeError.
Add a private NestedField accessor for the internal form, used by the
translator, the Avro default reader and Record defaults.
initial_default is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@kevinjqliu, you reviewed the initial-default projection in #1644. @Erigara, you worked on projected-field filtering in #2029. Could you take a look when you have a chance?

Comment thread pyiceberg/types.py
Comment on lines +435 to +442
@property
def _initial_default_value(self) -> object:
"""Return the initial-default as bound literals hold it: days or microseconds for temporal types."""
if self.initial_default is not None and isinstance(self.field_type, (DateType, TimeType, TimestampType, TimestamptzType)):
from pyiceberg.expressions.literals import literal

return literal(self.initial_default).to(self.field_type).value
return self.initial_default

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it make more sense to do this when we create the type? We know the type then right away.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants