Repository navigation
fix(table): load full snapshot history before expiration - #2601
laskoviymishka merged 29 commits into
Conversation
badalprasadsingh
left a comment
There was a problem hiding this comment.
all minor comments @iceberg-kitten
| synctest.Test(t, func(t *testing.T) { | ||
| fs, inner, tbl := newExpireReplayTable(t, nil) | ||
| tbl, first := appendOnBranch(t, tbl, MainBranch, "first") | ||
| tbl, _ = appendOnBranch(t, tbl, MainBranch, "head") |
There was a problem hiding this comment.
drop the assignment: this is where the lint is failing as tbl is never read after this line
| }) | ||
| } | ||
|
|
||
| func TestExpireReplayRejectsRefsOnlyHistoryAfterRetry(t *testing.T) { |
There was a problem hiding this comment.
a second Commit after a failed one also refreshes, but no test covers that path
can you please add this test case: with CommitNumRetriesKey: "0": the first Commit fails with "ErrCommitFailed", the second returns "ErrTransactionUnusable", and the catalog still has the tag and ids[0]
| return err | ||
| } | ||
| if t.tbl.snapshotHistoryRefsOnly { | ||
| return errSnapshotHistoryIncompleteForExpiration |
There was a problem hiding this comment.
callers cant' match it with error.Is
better: do this and check it with "require.ErrorIs" in the test
fmt.Errorf("%w: %w", ErrInvalidOperation, errSnapshotHistoryIncompleteForExpiration)| - **Commit rejections now match `table.ErrCommitRejected`.** SQL, Glue, Hive and Hadoop failures while preparing metadata, plus REST commit responses 400, 401, 403 and 404, leave transactions committable (after a REST 404, the next `Commit` fails when it reloads the table). Existing `errors.Is` matches keep working, but SQL, Glue, Hive and Hadoop rejection messages now start with `commit rejected: `, so code that matches error text may need updating. | ||
| - **Float comparisons against literals beyond the float32 range follow IEEE 754.** On a float32 column, `x < 1e100` no longer returns NaN, `+Inf` or NULL rows, and `NOT(x > 1e100)` no longer returns NULL rows. An infinite literal now binds to a float32 infinity, so `x = +Inf` matches `+Inf` rows. Java still folds these comparisons to always true or false, see [apache/iceberg#18700](https://github.com/apache/iceberg/issues/18700). | ||
|
|
||
| - **Snapshot expiration requires complete snapshot history.** Staging `Transaction.ExpireSnapshots` fails when REST loaded the table with `snapshot-loading-mode=refs`. A commit retry can also return `table.ErrTransactionUnusable` if its refresh omits history before expiration replay. Reload with `snapshot-loading-mode=all` and start a new transaction. |
There was a problem hiding this comment.
there is a "ExpireSnapshots" doc and the website that don't mention this limit yet
better: add a line to the doc, to api.md "Expiring snapshots" and to "cli.md" expire-snapshots, like the existing refs-mode notes for fast-forward and the branch commands. this note could also name the iceberg expire-snapshots command.
badalprasadsingh
left a comment
There was a problem hiding this comment.
one minor comment @iceberg-kitten
| iceberg expire-snapshots --older-than 7d --retain-last 5 my_db.events | ||
| ``` | ||
|
|
||
| For REST catalogs, the `iceberg expire-snapshots` command requires complete snapshot history. Configure `snapshot-loading-mode=all` before running it; `snapshot-loading-mode=refs` omits history needed to enforce retention and causes expiration to fail. |
There was a problem hiding this comment.
make it:
iceberg expire-snapshots fails when the REST server sets snapshot-loading-mode=refs; the CLI cannot override it.cli can't set snapshot-loading-mode: there is no flag, and .iceberg-go.yaml silently ignores the key, so refs mode only comes from the server's config.
the branch note at lin314 has the same gap.
Signed-off-by: Vu <europeantimlapses@gmail.com>
zeroshade
left a comment
There was a problem hiding this comment.
LGTM. ExpireSnapshots now fails on refs-only tables before staging anything, and the retry/resume paths stop before replaying an expiry against refs-only history. TestExpireReplayRejectsRefsOnlyHistoryAfterRetry shows why: the walk back from main stops early, so the tag-expiry path would drop s1 even though it's within main's retain-last 5. Tests pass under -race locally.
@badalprasadsingh's threads all look addressed in the current head (wrapped sentinel + ErrorIs, resume test, api.md/cli.md notes, CLI wording). I'll leave it to them to resolve those threads. One docs nit inline.
This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.
More on how Apache Iceberg Go handles contributions: CONTRIBUTING.md.
| - **Commit rejections now match `table.ErrCommitRejected`.** SQL, Glue, Hive and Hadoop failures while preparing metadata, plus REST commit responses 400, 401, 403 and 404, leave transactions committable (after a REST 404, the next `Commit` fails when it reloads the table). Existing `errors.Is` matches keep working, but SQL, Glue, Hive and Hadoop rejection messages now start with `commit rejected: `, so code that matches error text may need updating. | ||
| - **Float comparisons against literals beyond the float32 range follow IEEE 754.** On a float32 column, `x < 1e100` no longer returns NaN, `+Inf` or NULL rows, and `NOT(x > 1e100)` no longer returns NULL rows. An infinite literal now binds to a float32 infinity, so `x = +Inf` matches `+Inf` rows. Java still folds these comparisons to always true or false, see [apache/iceberg#18700](https://github.com/apache/iceberg/issues/18700). | ||
|
|
||
| - **Snapshot expiration requires complete snapshot history.** Staging `Transaction.ExpireSnapshots` fails when REST loaded the table with `snapshot-loading-mode=refs`. A commit retry can also return `table.ErrTransactionUnusable` if its refresh omits history before expiration replay. Reload with `snapshot-loading-mode=all` and start a new transaction. |
There was a problem hiding this comment.
Nit: the blank line before this entry splits it off from the rest of the list. It would read better next to "Snapshot expiration now tolerates concurrent ref changes". Also, the four other entries added in 46f97e3 (RowDelta Commit once, SQL nested namespaces, Directory IO context, RowDelta DV conflict) are copies of lines already on main. Merging main instead would keep them out of this diff.
badalprasadsingh
left a comment
There was a problem hiding this comment.
approving, +1 to zeroshade's release note comment though
nssalian
left a comment
There was a problem hiding this comment.
Thanks @iceberg-kitten for the PR. I re-ran the new tests with each guard removed: dropping the staging check (transaction.go:838) fails TestExpireSnapshotsRejectsRefsOnlySnapshotHistory and TestFailedRefsOnlyExpirationLeavesAppendUsable, and dropping the retry check (table.go:829) fails TestExpireReplayRejectsRefsOnlyHistoryAfterRetry and ...OnResume, while the append control stays green. Two small asks inline (api.md wording, one missing test) and a non-blocking question.
| If a concurrent writer forces a commit retry or a second `Commit` attempt | ||
| to reload refs-only history, the expiration cannot be replayed safely. | ||
| `Transaction.Commit` then returns `table.ErrTransactionUnusable` and | ||
| does not resubmit the expiration; it leaves the catalog's existing snapshots |
There was a problem hiding this comment.
For library users snapshot-loading-mode is a client catalog property that wins over server defaults (catalog/rest/rest.go:1301-1306), and reloading through the same catalog stays in refs mode (rest.go:2027). The note at api.md:684 already says "Load with snapshot-loading-mode=all instead". Suggest: "Create the REST catalog with snapshot-loading-mode=all (or omit it), reload, and start a new transaction. If the server forces refs through config overrides, expiration is not available through that catalog." Same wording as suggested on #2599.
| // removals after a concurrent commit. Check the recorded operations, not the | ||
| // length of staged updates, before replaying against incomplete history. | ||
| func (r *stagedReplay) hasExpirationRecords() bool { | ||
| return r != nil && len(r.records) > 0 |
There was a problem hiding this comment.
The comment above is the reason for checking records, but nothing pins it: with this returning len(r.walkedRefs) > 0 (only expiries that produced updates) all of ./table/ still passes. A test that stages an ExpireSnapshots that plans nothing plus an append, then retries into refs-only history and expects ErrTransactionUnusable, would help cover that gap.
|
|
||
| func TestExpireReplayRejectsRefsOnlyHistoryOnResume(t *testing.T) { | ||
| synctest.Test(t, func(t *testing.T) { | ||
| fs, inner, tbl := newExpireReplayTable(t, iceberg.Properties{CommitNumRetriesKey: "0"}) |
There was a problem hiding this comment.
minor: the setup through the expired-tag assertion is the same 14 lines as ...AfterRetry, and the tail assertions match too. A stage := func(t *testing.T, numRetries string) closure like TestExpireReplayKeepsExpiryThatReplannedToNothing (:580) would let both run as subtests.
| require.ErrorIs(t, err, errSnapshotHistoryIncompleteForExpiration) | ||
| staged, err := tx.StagedTable() | ||
| require.NoError(t, err) | ||
| require.Empty(t, staged.Metadata().Snapshots(), "rejected expiration must not stage updates") |
There was a problem hiding this comment.
nit: the fixture starts with no snapshots, so this can't fail. Comparing the transaction's staged updates and requirements before and after the call would check what the message says.
| if err := validateGCEnabledForSnapshotExpiration(meta.props); err != nil { | ||
| return err | ||
| } | ||
| if t.tbl.snapshotHistoryRefsOnly { |
There was a problem hiding this comment.
not blocking: a table returned by Commit keeps the refs-only flag (table.go:1029 copies it) even though REST commit responses are parsed with the full snapshot list (rest.go:1892-1894), so ExpireSnapshots on the committed table is rejected too. Failing closed is fine for 0.8.0 or do we want to clean it up here. CC: @zeroshade @laskoviymishka
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
laskoviymishka
left a comment
There was a problem hiding this comment.
Rejecting here is already safer than main, which plans expiration over whatever partial snapshot set refs mode returned.
Two things should change before this merges:
1/ The first is loading full history on demand instead of rejecting, which is the thread on the ExpireSnapshots guard.
2/ The second is the ErrInvalidOperation wrap with require.ErrorIs that's already asked for on the staging error; +1 to that one. The message rewording is a nit.
| ) | ||
|
|
||
| var errSnapshotHistoryIncompleteForExpiration = errors.New( | ||
| "cannot expire snapshots with incomplete history under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all", |
There was a problem hiding this comment.
Could the message say the history may be incomplete? Both guards check the load mode, not what came back (rest.go), so a refs-mode table whose snapshots are all ref heads gets this error too. Cherry-pick only rejects on !complete (cherry_pick.go).
| "cannot expire snapshots with incomplete history under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all", | |
| "cannot expire snapshots: history may be incomplete under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all", |
| if err := validateGCEnabledForSnapshotExpiration(meta.props); err != nil { | ||
| return err | ||
| } | ||
| if t.tbl.snapshotHistoryRefsOnly { |
There was a problem hiding this comment.
Could we load the full history here instead of rejecting? In refs mode Java's RESTSessionCatalog sets a snapshots supplier that reloads with SnapshotMode.ALL (RESTSessionCatalog.java). As I read it, a table from a refs-mode LoadTable can't be expired with this guard. loadTableWithMode is unexported (rest.go), so the only way out is a second catalog with snapshot-loading-mode=all.
…y-history-20261115 Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
…y-history-20261115
laskoviymishka
left a comment
There was a problem hiding this comment.
I think this is good to land
What
Why
Implementation
ErrInvalidOperationif staging cannot load full history for that revision.