Skip to content

fix(maintenance): require full snapshot history for orphan cleanup - #2599

Open
iceberg-kitten wants to merge 8 commits into
apache:mainfrom
iceberg-kitten:fix/maintenance-refs-only-history-20261115
Open

iceberg-kitten wants to merge 8 commits into
apache:mainfrom
iceberg-kitten:fix/maintenance-refs-only-history-20261115

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What

Require full snapshot history for orphan cleanup. This covers planning, dry runs, execution of saved plans, and the refreshes around one-shot cleanup.

Why

REST refs mode can omit snapshots that still reference live files. Cleanup needs the complete history to determine which files are safe to remove.

Implementation

  • Return an error wrapping table.ErrInvalidOperation when a table was loaded with refs-only history or a refresh returns refs-only history.
  • Check refreshed history even when the table UUID and metadata location still match a saved plan.
  • Document how to reload with snapshot-loading-mode=all and how server overrides affect catalog configuration.
  • Keep PurgeFiles available for table drops.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few blocking comments @iceberg-kitten

Comment thread table/maintenance/orphan_cleanup.go Outdated

func validateCompleteSnapshotHistory(tbl *table.Table) error {
if tbl.SnapshotHistoryLimitedToRefs() {
return fmt.Errorf("table snapshot history is incomplete under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ran the lint locally, and its failing here: fmt.Errorf with no arguments, also, no blank lines before many "return err"

better: wrap the sentinel, like the refs-only cherry-pick check, so callers can use "errors.Is", and assert it with "require.ErrorIs"

return fmt.Errorf("%w: table snapshot history is incomplete under REST snapshot-loading-mode=refs; reload with snapshot-loading-mode=all", table.ErrInvalidOperation)

Comment thread table/maintenance/orphan_cleanup.go Outdated
// concurrently and is bounded to defaultPurgeMaxConcurrency operations because
// it is typically I/O-bound.
func PurgeFiles(ctx context.Context, tbl *table.Table) error {
if err := validateCompleteSnapshotHistory(tbl); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"PurgeFiles" only deletes. with the default gc.enabled=true it walks the table location, so refs-only history can only miss files outside it -> catalogs call it after the drop, when the table can't be reloaded as the hint says.

better:drop this check, its getReferencedFiles copy and the release-note mention.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the guard itself looks right: it runs before tbl.FS() in all three orphan-cleanup entry points, and both new rejection tests fail if the check is disabled. CI is green. The changes requested are mostly docs and description:

  1. website/src/api.md already has a ### Orphan file cleanup section; this adds a second one (see inline).
  2. The api.md remediation text is misleading for library users (see inline).
  3. The PR description says "Block orphan cleanup and PurgeFiles", but the code and docs deliberately allow PurgeFiles. Please update it, since it becomes the squash commit message.
  4. Commit ab02732 copies four unrelated release-note entries from main into releases.md (RowDelta single Commit, SQL nested namespaces, Directory IO context, RowDelta deletion-vector conflict). They merge cleanly, but please rebase onto main so the diff only contains this PR's entry.

Non-blocking:

  • TestPlanOrphanFilesRejectsRefsOnlyWithLiveHistoricalData builds an httptest REST server and IO registry inside the maintenance package. The REST-specific part (that LoadTable sets SnapshotHistoryLimitedToRefs() for refs vs all) would fit better in catalog/rest tests, leaving the maintenance test to use table.WithSnapshotHistoryLimitedToRefs directly.
  • Java's REST catalog lazily loads the full snapshot list in refs mode, so orphan cleanup works there. Rejecting here is the conservative choice; worth a sentence in the description.
  • Possible follow-up: ExpireSnapshots with post-commit deletion has no refs-only guard either. Expiring a tag head on a refs-only table may delete files still referenced by an omitted but valid snapshot.

Comment thread website/src/api.md Outdated

Pass `table.WithPostCommit(true)` to delete the unreferenced data and metadata files after the commit lands. The `iceberg expire-snapshots` CLI command wraps the same operation - see [CLI](./cli.md).

### Orphan cleanup

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This page already has ### Orphan file cleanup under ## Maintenance (around line 773). Could the plan/execute description and the safety paragraph move into that section instead of adding a second orphan-cleanup heading?

Comment thread website/src/api.md Outdated
`snapshot-loading-mode=refs`, including dry runs. Refs-only metadata may
omit snapshots whose files are still live, so cleanup fails with
`table.ErrInvalidOperation` (check with `errors.Is`) before touching
the filesystem. Configure the REST server to load

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For library users snapshot-loading-mode is a client catalog property (e.g. rest.WithAdditionalProps), not only a server setting. The server can force it via /v1/config overrides, which win over client props. Suggest something like: "Create the REST catalog with snapshot-loading-mode=all (or omit it); if the server forces refs through config overrides, orphan cleanup is not available through that catalog."

Comment thread table/maintenance/orphan_cleanup.go Outdated
if cfg.invalidPlanOption != nil {
return OrphanCleanupResult{}, cfg.invalidPlanOption
}
if err := validateCompleteSnapshotHistory(tbl); err != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: here the refs-only check runs before validateOrphanCleanupGC, while DeleteOrphanFiles/PlanOrphanFiles check GC first (refs-only is checked inside planOrphanFiles). A table with both conditions returns a different error depending on the entry point. Consider using the same order everywhere.

@nssalian nssalian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran TestPlanOrphanFilesRejectsRefsOnlySnapshotHistory with orphan_cleanup.go swapped back to the merge-base: plan, delete and execute all fail, and they pass on this head. git merge-tree against main and against #2601 (the ExpireSnapshots follow-up mentioned above) is clean, so the two v0.8.0 blockers can land in either order. Nothing blocking from me beyond the doc and description items already raised.

Comment thread website/src/releases.md Outdated
- **`RowDelta.Commit` can be called once per builder.** A second `Commit` on the same `RowDelta` now returns an error wrapping `table.ErrInvalidOperation` and stages nothing, even if the first call failed, as `RewriteFiles.Commit` does. Previously, calling `Commit` again after a failed first call could succeed, and calling it again after a successful one staged the builder's files a second time. Create a new builder with `Transaction.NewRowDelta` for each commit.
- **Data-file replacements now reject concurrent deletes.** Commits staged through `ReplaceDataFiles`, `ReplaceDataFilesByPath`, `ReplaceFiles`, or rewrite operations can return `table.ErrConflictingDeleteFiles` when a concurrent position delete targets a replaced file or any concurrent equality delete is added. Compaction on tables with streaming equality deletes may see additional conflicts.
- **Orphan cleanup now rejects `gc.enabled=false`.** `DeleteOrphanFiles`, `PlanOrphanFiles`, `ExecuteOrphanCleanup`, and dry runs now return an error before filesystem access. The `iceberg clean-orphan-files` command is also affected.
- **Orphan cleanup requires complete snapshot history.** `DeleteOrphanFiles`, `PlanOrphanFiles`, and `ExecuteOrphanCleanup` now return an error when the table was loaded with REST `snapshot-loading-mode=refs`. Reload with `snapshot-loading-mode=all` before orphan cleanup. `maintenance.PurgeFiles` is unaffected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the gc.enabled entry right above names dry runs and iceberg clean-orphan-files. Both hit this rejection too (the CLI calls PlanOrphanFiles before its dry-run branch in cmd/iceberg/clean_orphan_files.go:50), so worth naming them here as well.

}

meta, err := table.NewMetadata(iceberg.NewSchema(0), nil,
table.UnsortedSortOrder, location, iceberg.Properties{"gc.enabled": tc.gcEnabled})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table.GCEnabledKey instead of the "gc.enabled" literal, like the other tests in this package.

@iceberg-kitten
iceberg-kitten force-pushed the fix/maintenance-refs-only-history-20261115 branch from 8f92ec3 to 1681697 Compare November 18, 2026 06:14
Minh Vu added 3 commits November 18, 2026 13:17
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the failing ci job is a lint timeout issue so a re-run should fix ci

otherwise this is approved from my side :)

…s-only-history-20261115

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@laskoviymishka
laskoviymishka self-requested a review November 18, 2026 20:05
Minh Vu added 3 commits November 19, 2026 14:29
@iceberg-kitten iceberg-kitten changed the title fix(maintenance): reject refs-only snapshot history fix(maintenance): require full snapshot history for orphan cleanup Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants