Repository navigation
Use internal values for temporal initial-defaults - #4435
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
A file that lacks a date, time, timestamp or timestamptz column with a non-null initial-default compared the Python value with bound literals, which hold days or microseconds. Equality and IN dropped matching rows, != kept them, and range predicates raised TypeError. Add a private NestedField accessor for the internal form, used by the translator, the Avro default reader and Record defaults. initial_default is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@kevinjqliu, you reviewed the initial-default projection in #1644. @Erigara, you worked on projected-field filtering in #2029. Could you take a look when you have a chance? |
Fokko
reviewed
Nov 21, 2026
Comment on lines
+435
to
+442
| @property | ||
| def _initial_default_value(self) -> object: | ||
| """Return the initial-default as bound literals hold it: days or microseconds for temporal types.""" | ||
| if self.initial_default is not None and isinstance(self.field_type, (DateType, TimeType, TimestampType, TimestamptzType)): | ||
| from pyiceberg.expressions.literals import literal | ||
|
|
||
| return literal(self.initial_default).to(self.field_type).value | ||
| return self.initial_default |
Contributor
There was a problem hiding this comment.
Wouldn't it make more sense to do this when we create the type? We know the type then right away.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Bug: when a data file lacks a date, time, timestamp or timestamptz column,
=andINmiss its non-nullinitial-default, while!=matches it._ColumnNameTranslatorcompares a Python temporal value with a bound literal in days or microseconds:Fix: add a private
NestedFieldaccessor that returns a temporal initial-default in the literal's form (the conversionadd_columnuses), and use it in the translator, the Avro default reader andRecorddefaults. Partition-projected values are already in that form and keep precedence.Java:
SchemaParserandSchemaUpdatesupply typed defaults to row-group filtering;GenericReadernormalizes reader output throughInternalRecordWrapperbeforeEvaluator. This PR normalizes defaults used in internal records.NestedField.initial_defaultstays a Python object; internal default consumers share the conversion.Are these changes tested?
Yes, filter tests in
tests/expressions/test_visitors.pyandtests/io/test_pyarrow.py, plus Avro andRecorddefault tests intests/avro/test_resolver.pyandtests/test_typedef.py. All fail on main (wrong filter result, or a Python date where an int is expected) and pass with this change.Are there any user-facing changes?
Yes, this fixes incorrect temporal-default filter results and range errors. Avro now returns defaulted temporal fields as integer days or microseconds, matching stored fields.
Please add the
changeloglabel.This pull request and its description were written by Isaac.