Skip to content

fix(table): load full snapshot history before expiration - #2601

Merged
laskoviymishka merged 29 commits into
apache:mainfrom
iceberg-kitten:fix/expire-refs-only-history-20261115
Nov 19, 2026
Merged

laskoviymishka merged 29 commits into
apache:mainfrom
iceberg-kitten:fix/expire-refs-only-history-20261115

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Load full snapshot history when expiring snapshots from REST refs mode.

Why

Implementation

  • Request all snapshots for staging, commit retries, and CLI previews.
  • Keep staged updates and verify the metadata revision before planning.
  • Return ErrInvalidOperation if staging cannot load full history for that revision.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all minor comments @iceberg-kitten

Comment thread table/expire_replay_test.go Outdated
synctest.Test(t, func(t *testing.T) {
fs, inner, tbl := newExpireReplayTable(t, nil)
tbl, first := appendOnBranch(t, tbl, MainBranch, "first")
tbl, _ = appendOnBranch(t, tbl, MainBranch, "head")

@badalprasadsingh badalprasadsingh Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop the assignment: this is where the lint is failing as tbl is never read after this line

Comment thread table/expire_replay_test.go Outdated
})
}

func TestExpireReplayRejectsRefsOnlyHistoryAfterRetry(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a second Commit after a failed one also refreshes, but no test covers that path

can you please add this test case: with CommitNumRetriesKey: "0": the first Commit fails with "ErrCommitFailed", the second returns "ErrTransactionUnusable", and the catalog still has the tag and ids[0]

Comment thread table/transaction.go Outdated
return err
}
if t.tbl.snapshotHistoryRefsOnly {
return errSnapshotHistoryIncompleteForExpiration

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

callers cant' match it with error.Is

better: do this and check it with "require.ErrorIs" in the test

fmt.Errorf("%w: %w", ErrInvalidOperation, errSnapshotHistoryIncompleteForExpiration)

Comment thread website/src/releases.md Outdated
- **Commit rejections now match `table.ErrCommitRejected`.** SQL, Glue, Hive and Hadoop failures while preparing metadata, plus REST commit responses 400, 401, 403 and 404, leave transactions committable (after a REST 404, the next `Commit` fails when it reloads the table). Existing `errors.Is` matches keep working, but SQL, Glue, Hive and Hadoop rejection messages now start with `commit rejected: `, so code that matches error text may need updating.
- **Float comparisons against literals beyond the float32 range follow IEEE 754.** On a float32 column, `x < 1e100` no longer returns NaN, `+Inf` or NULL rows, and `NOT(x > 1e100)` no longer returns NULL rows. An infinite literal now binds to a float32 infinity, so `x = +Inf` matches `+Inf` rows. Java still folds these comparisons to always true or false, see [apache/iceberg#18700](https://github.com/apache/iceberg/issues/18700).

- **Snapshot expiration requires complete snapshot history.** Staging `Transaction.ExpireSnapshots` fails when REST loaded the table with `snapshot-loading-mode=refs`. A commit retry can also return `table.ErrTransactionUnusable` if its refresh omits history before expiration replay. Reload with `snapshot-loading-mode=all` and start a new transaction.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a "ExpireSnapshots" doc and the website that don't mention this limit yet

better: add a line to the doc, to api.md "Expiring snapshots" and to "cli.md" expire-snapshots, like the existing refs-mode notes for fast-forward and the branch commands. this note could also name the iceberg expire-snapshots command.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor comment @iceberg-kitten

Comment thread website/src/cli.md Outdated
iceberg expire-snapshots --older-than 7d --retain-last 5 my_db.events
```

For REST catalogs, the `iceberg expire-snapshots` command requires complete snapshot history. Configure `snapshot-loading-mode=all` before running it; `snapshot-loading-mode=refs` omits history needed to enforce retention and causes expiration to fail.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it:

iceberg expire-snapshots fails when the REST server sets snapshot-loading-mode=refs; the CLI cannot override it.

cli can't set snapshot-loading-mode: there is no flag, and .iceberg-go.yaml silently ignores the key, so refs mode only comes from the server's config.

the branch note at lin314 has the same gap.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. ExpireSnapshots now fails on refs-only tables before staging anything, and the retry/resume paths stop before replaying an expiry against refs-only history. TestExpireReplayRejectsRefsOnlyHistoryAfterRetry shows why: the walk back from main stops early, so the tag-expiry path would drop s1 even though it's within main's retain-last 5. Tests pass under -race locally.

@badalprasadsingh's threads all look addressed in the current head (wrapped sentinel + ErrorIs, resume test, api.md/cli.md notes, CLI wording). I'll leave it to them to resolve those threads. One docs nit inline.


This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Iceberg Go handles contributions: CONTRIBUTING.md.

Comment thread website/src/releases.md Outdated
- **Commit rejections now match `table.ErrCommitRejected`.** SQL, Glue, Hive and Hadoop failures while preparing metadata, plus REST commit responses 400, 401, 403 and 404, leave transactions committable (after a REST 404, the next `Commit` fails when it reloads the table). Existing `errors.Is` matches keep working, but SQL, Glue, Hive and Hadoop rejection messages now start with `commit rejected: `, so code that matches error text may need updating.
- **Float comparisons against literals beyond the float32 range follow IEEE 754.** On a float32 column, `x < 1e100` no longer returns NaN, `+Inf` or NULL rows, and `NOT(x > 1e100)` no longer returns NULL rows. An infinite literal now binds to a float32 infinity, so `x = +Inf` matches `+Inf` rows. Java still folds these comparisons to always true or false, see [apache/iceberg#18700](https://github.com/apache/iceberg/issues/18700).

- **Snapshot expiration requires complete snapshot history.** Staging `Transaction.ExpireSnapshots` fails when REST loaded the table with `snapshot-loading-mode=refs`. A commit retry can also return `table.ErrTransactionUnusable` if its refresh omits history before expiration replay. Reload with `snapshot-loading-mode=all` and start a new transaction.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the blank line before this entry splits it off from the rest of the list. It would read better next to "Snapshot expiration now tolerates concurrent ref changes". Also, the four other entries added in 46f97e3 (RowDelta Commit once, SQL nested namespaces, Directory IO context, RowDelta DV conflict) are copies of lines already on main. Merging main instead would keep them out of this diff.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving, +1 to zeroshade's release note comment though

@nssalian nssalian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @iceberg-kitten for the PR. I re-ran the new tests with each guard removed: dropping the staging check (transaction.go:838) fails TestExpireSnapshotsRejectsRefsOnlySnapshotHistory and TestFailedRefsOnlyExpirationLeavesAppendUsable, and dropping the retry check (table.go:829) fails TestExpireReplayRejectsRefsOnlyHistoryAfterRetry and ...OnResume, while the append control stays green. Two small asks inline (api.md wording, one missing test) and a non-blocking question.

Comment thread website/src/api.md Outdated
If a concurrent writer forces a commit retry or a second `Commit` attempt
to reload refs-only history, the expiration cannot be replayed safely.
`Transaction.Commit` then returns `table.ErrTransactionUnusable` and
does not resubmit the expiration; it leaves the catalog's existing snapshots

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For library users snapshot-loading-mode is a client catalog property that wins over server defaults (catalog/rest/rest.go:1301-1306), and reloading through the same catalog stays in refs mode (rest.go:2027). The note at api.md:684 already says "Load with snapshot-loading-mode=all instead". Suggest: "Create the REST catalog with snapshot-loading-mode=all (or omit it), reload, and start a new transaction. If the server forces refs through config overrides, expiration is not available through that catalog." Same wording as suggested on #2599.

Comment thread table/staged_replay.go
// removals after a concurrent commit. Check the recorded operations, not the
// length of staged updates, before replaying against incomplete history.
func (r *stagedReplay) hasExpirationRecords() bool {
return r != nil && len(r.records) > 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above is the reason for checking records, but nothing pins it: with this returning len(r.walkedRefs) > 0 (only expiries that produced updates) all of ./table/ still passes. A test that stages an ExpireSnapshots that plans nothing plus an append, then retries into refs-only history and expects ErrTransactionUnusable, would help cover that gap.

Comment thread table/expire_replay_test.go Outdated

func TestExpireReplayRejectsRefsOnlyHistoryOnResume(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
fs, inner, tbl := newExpireReplayTable(t, iceberg.Properties{CommitNumRetriesKey: "0"})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the setup through the expired-tag assertion is the same 14 lines as ...AfterRetry, and the tail assertions match too. A stage := func(t *testing.T, numRetries string) closure like TestExpireReplayKeepsExpiryThatReplannedToNothing (:580) would let both run as subtests.

Comment thread table/expire_replay_test.go Outdated
require.ErrorIs(t, err, errSnapshotHistoryIncompleteForExpiration)
staged, err := tx.StagedTable()
require.NoError(t, err)
require.Empty(t, staged.Metadata().Snapshots(), "rejected expiration must not stage updates")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the fixture starts with no snapshots, so this can't fail. Comparing the transaction's staged updates and requirements before and after the call would check what the message says.

Comment thread table/transaction.go Outdated
if err := validateGCEnabledForSnapshotExpiration(meta.props); err != nil {
return err
}
if t.tbl.snapshotHistoryRefsOnly {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not blocking: a table returned by Commit keeps the refs-only flag (table.go:1029 copies it) even though REST commit responses are parsed with the full snapshot list (rest.go:1892-1894), so ExpireSnapshots on the committed table is rejected too. Failing closed is fine for 0.8.0 or do we want to clean it up here. CC: @zeroshade @laskoviymishka

Minh Vu added 2 commits November 18, 2026 06:06
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rejecting here is already safer than main, which plans expiration over whatever partial snapshot set refs mode returned.

Two things should change before this merges:
1/ The first is loading full history on demand instead of rejecting, which is the thread on the ExpireSnapshots guard.
2/ The second is the ErrInvalidOperation wrap with require.ErrorIs that's already asked for on the staging error; +1 to that one. The message rewording is a nit.

Comment thread table/expire_replay.go Outdated
)

var errSnapshotHistoryIncompleteForExpiration = errors.New(
"cannot expire snapshots with incomplete history under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the message say the history may be incomplete? Both guards check the load mode, not what came back (rest.go), so a refs-mode table whose snapshots are all ref heads gets this error too. Cherry-pick only rejects on !complete (cherry_pick.go).

Suggested change
"cannot expire snapshots with incomplete history under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all",
"cannot expire snapshots: history may be incomplete under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all",

Comment thread table/transaction.go Outdated
if err := validateGCEnabledForSnapshotExpiration(meta.props); err != nil {
return err
}
if t.tbl.snapshotHistoryRefsOnly {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we load the full history here instead of rejecting? In refs mode Java's RESTSessionCatalog sets a snapshots supplier that reloads with SnapshotMode.ALL (RESTSessionCatalog.java). As I read it, a table from a refs-mode LoadTable can't be expired with this guard. loadTableWithMode is unexported (rest.go), so the only way out is a second catalog with snapshot-loading-mode=all.

Minh Vu added 3 commits November 18, 2026 19:42
…y-history-20261115

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@iceberg-kitten iceberg-kitten changed the title fix(table): reject expiry with refs-only snapshot history fix(table): load full snapshot history before expiration Nov 18, 2026

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good to land

@laskoviymishka
laskoviymishka merged commit f7dc20d into apache:main Nov 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants