Repository navigation
Rebase polynya patches onto v0.10.1; rewrites keep summary totals - #3
Merged
Merged
Conversation
Two related changes that let downstream crates author their own TransactionActions and drive schema evolution without going through the private TableCommit::builder. 1. Flip TransactionAction (and BoxedTransactionAction) from pub(crate) to pub. The blanket ApplyTransactionAction impl already accepts any T: TransactionAction, so external implementors can add custom actions to a Transaction without further changes. 2. Add ReplaceSchemaAction. Callers compute the desired Schema and hand it to set_schema(); on commit the action emits AddSchema + SetCurrentSchema(-1) so TableMetadataBuilder allocates the new schema id. Three requirements (UuidMatch, CurrentSchemaIdMatch, LastAssignedFieldIdMatch) make concurrent schema/column changes fail loudly. Upstream's UpdateSchemaAction (0.10) evolves a schema column by column and can't rename, promote or make a column optional, so this stays alongside it. Also add a Transaction::replace_schema() convenience method matching the existing fast_append / update_location / update_statistics shape.
…ction
FastAppend previously rejected any DataFile whose content_type was not
Data, blocking commits that include equality- or position-delete files.
This change accepts mixed input and routes by content_type into
separate manifests at commit time, matching the Iceberg spec
requirement that one manifest must hold a single content type.
- FastAppendAction.add_data_files() now routes incoming files into
added_data_files (Data) or added_delete_files (EqualityDeletes /
PositionDeletes). The old "Only data content type is allowed" check
is gone.
- SnapshotProducer carries added_delete_files alongside added_data_files
and writes them to a separate delete manifest via build_v{2,3}_deletes.
- validate_added_data_files renamed to validate_added_files and is now
generic over a slice — the partition-spec / partition-value rules
apply uniformly to data and delete files.
- Snapshot summary collector iterates both vecs so file/record counts
reflect everything added in the snapshot.
- New test test_fast_append_routes_equality_deletes_into_separate_manifest
exercises a mixed Data + EqualityDeletes commit through MemoryCatalog
and asserts both manifests appear in the manifest list with the right
ManifestContentType.
Adds a new transaction action that produces snapshots with operation =
Replace: drops a set of existing data/delete files and adds new ones in
one atomic commit. Wired as `Transaction::rewrite_files()`.
Manifests from the prior snapshot are walked and classified per-manifest:
- No removals: carry forward unchanged.
- All entries removed: drop manifest from new manifest list.
- Partial removal: rewrite manifest with survivors marked Existing
(preserving original snapshot_id + sequence_number).
Added files go into a new ManifestStatus::Added manifest, routed by
content_type() into separate Data / Deletes manifests as the spec
requires.
To support partial-manifest rewrite, SnapshotProduceOperation::existing_manifest
now takes &mut SnapshotProducer so impls can call write_existing_manifest_for
(new helper) inline. FastAppendOperation's signature was updated to match;
its body is unchanged.
snapshot_summary::update_snapshot_summaries now accepts Operation::Replace
alongside Append/Overwrite/Delete.
Tests:
- full-manifest drop + swap
- partial rewrite preserves survivors as Existing
- mixed data + delete manifests carry forward correctly
- no-removal path is just a Replace-tagged append
- empty-input action errors (matches SnapshotProducer's contract)
Mirrors Java/PyIceberg's RewriteFiles semantics but keeps the implementation
narrow — we don't emit Deleted entries for audit-trail purposes, since the
new manifest list simply doesn't reference the dropped files. Can revisit
when expiration tooling needs them.
Plumbs the REST loadTable response config through Table::properties() so downstream consumers can extract vended S3 credentials without duplicating the catalog request. Previous behavior: response.config was merged into FileIO and discarded. The new field is empty for catalogs that don't surface table-scoped config (file, glue, memory) — it's an additive change with no behavioral impact on existing callers. Used by pg2iceberg-rust to build per-table object stores for the vended-credentials S3 router (Polaris, Snowflake, Tabular).
AWS-native Iceberg REST endpoints (S3 Tables, Glue) require every request to be SigV4-signed; the REST client previously only supported bearer/OAuth, so requests returned 403 MissingAuthenticationToken. Add an opt-in SigV4Signer in the REST HttpClient, enabled per-catalog via the `rest.sigv4-enabled` property (with `rest.signing-region` / `rest.signing-name`, falling back to `s3.region`). Credentials load lazily from the standard AWS chain (env, ECS/EKS task role, EC2 IMDS, shared config) and are cached for the process. Signing runs last in execute(), over the final headers + body, so it is inert unless enabled.
The snapshot summary's `total-*` counts are carried from the previous snapshot plus this one's `added-*` and minus its `deleted-*` / `removed-*`. A rewrite recorded no removed files, so after a compaction `total-records`, `total-data-files` and the delete totals still counted everything it replaced. Engines read these: ClickHouse 26.2 answered `count()` from `total-records`. The rewrite now looks up the live files it removes before the snapshot is produced (the summary is built before the manifests are written) and the producer subtracts them, under each file's own partition spec. The same pass carries the existing check that every removed file is still live, still on every commit attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fork's patches, replayed onto upstream v0.10.1.
polynya-patches-v0.10.1is upstream's v0.10.1 commit, so this PR's diff is exactly the fork's changes.polynya-patches, the 0.9.0-based branch pg2iceberg's lockfile still pins, is untouched.Upstream since 0.9.0 brings, among others, apache#2391. Every snapshot summary's
total-*had restarted from zero, because the previous summary was looked up through the uncommitted snapshot's id. pg2iceberg's smoke test showed ClickHouse'scount()returning 6 for a 3098-row table.Patches
TransactionAction+ReplaceSchemaActionUpdateSchemaAction(add_column/delete_column), so the fork's whole-schema action is renamedReplaceSchemaAction(Transaction::replace_schema()). Upstream's can't rename, promote or make a column optional, which pg2iceberg needs.FastAppendActionTable::manifest_list_reader()(upstream droppedSnapshot::load_manifest_list).RewriteFilesActionkey_metadata(upstream removed it fromSnapshotProducer::newandFastAppendAction), andmanifest_list_reader().Table::properties()(vended credentials)table.rsgainedruntimeand an encryption manager upstream.aws-*dependencies, missing on the old branch, is now in that commit.manifest_list_reader().style: rustfmt the fork's patchesRewrites keep the summary's running totals
RewriteFilesActionrecorded no removed files. Once apache#2391 makes the totals accumulate, every compaction would have added its outputs tototal-records/total-data-fileswithout subtracting its inputs, and lefttotal-delete-files/total-equality-deletescounting retired deletes.SnapshotProducersubtracts them, each under its own partition spec.Tests
iceberglib: 1392 passed.iceberg-catalog-restlib: 44 passed.rewrite_summary_keeps_running_totals: two appends, one with an equality delete, then a rewrite of a data file that retires the delete.total-records30 → 29,total-data-files2,total-delete-files0,deleted-records10. It fails without the patch (mutation check).cargo fmt --checkandcargo clippyfor both crates are clean.Next
A pg2iceberg PR that points the dependency at this branch and migrates to the 0.10 API. For example,
TableBuildernow requires aruntime, andupdate_schema().set_schema()becomesreplace_schema().set_schema(). A DST invariant will check summary totals against the live files on the prod backend.🤖 Generated with Claude Code