Repository navigation
fix: keep last_updated_ms from preceding the latest metadata-log entry - #4273
Conversation
| if metadata_location: | ||
| new_metadata = _update_table_metadata_log(new_metadata, metadata_location, base_metadata.last_updated_ms) | ||
| if base_metadata.last_updated_ms == new_metadata.last_updated_ms: | ||
| if new_metadata.last_updated_ms <= base_metadata.last_updated_ms: |
There was a problem hiding this comment.
I compared this against Java's TableMetadata.Builder (core/src/main/java/org/apache/iceberg/TableMetadata.java). Java never inherits the snapshot timestamp into last-updated-ms at all: addSnapshot() does not touch it, and build() stamps System.currentTimeMillis() (lines 1559-1561). PyIceberg inherits the snapshot timestamp in AddSnapshotUpdate because snapshots are staged eagerly (in Java the snapshot is produced at commit time), which is the root cause of #4189. This change restores the invariant Java enforces at lines 394-403 (last-updated-ms within one minute after the latest metadata-log entry), so from Java's perspective the output is accepted again.
| new_metadata = update_table_metadata( | ||
| table_v2.metadata, (AddSnapshotUpdate(snapshot=stale_snapshot),), False, table_v2.metadata_location | ||
| ) | ||
| assert new_metadata.last_updated_ms > new_metadata.metadata_log[-1].timestamp_ms |
There was a problem hiding this comment.
Nit: nice regression test. The issue's repro also covers the SetSnapshotRefUpdate path (the snapshot log gets stamped with the stale snapshot timestamp). I probed that separately: Java tolerates it (snapshot-log entries are validated with the same one-minute tolerance, and the stamping actually matches Java's timeOfChange behavior for added snapshots), so this is fine as is. Just mentioning in case you want a variant of this test with a SetSnapshotRefUpdate to main for completeness.
Closes #4189
Rationale for this change
last_updated_msto its own timestamp. If another commit lands before the transaction commits, the new metadata ends up older than its latestmetadata-logentry, and Java rejects it once the gap exceeds a minute.update_table_metadatanow bumpslast_updated_msto the current time whenever it doesn't come after the base metadata's, not only when they're equal.Are these changes tested?
test_init.py— a snapshot older than the base metadata no longer setslast_updated_msbefore the latest metadata-log entry.test_update_metadata_with_multiple_updatesused a snapshot timestamp older than its base metadata, so it now uses a later one.Are there any user-facing changes?