Skip to content

Rebase polynya patches onto v0.10.1; rewrites keep summary totals - #3

Merged
hasyimibhar merged 8 commits into
polynya-patches-v0.10.1from
rebase-v0.10.1
Oct 6, 2026
Merged

hasyimibhar merged 8 commits into
polynya-patches-v0.10.1from
rebase-v0.10.1

Conversation

@hasyimibhar

Copy link
Copy Markdown

The fork's patches, replayed onto upstream v0.10.1. polynya-patches-v0.10.1 is upstream's v0.10.1 commit, so this PR's diff is exactly the fork's changes. polynya-patches, the 0.9.0-based branch pg2iceberg's lockfile still pins, is untouched.

Upstream since 0.9.0 brings, among others, apache#2391. Every snapshot summary's total-* had restarted from zero, because the previous summary was looked up through the uncommitted snapshot's id. pg2iceberg's smoke test showed ClickHouse's count() returning 6 for a 3098-row table.

Patches

Commit Change on 0.10.1
Public TransactionAction + ReplaceSchemaAction Upstream now has its own UpdateSchemaAction (add_column / delete_column), so the fork's whole-schema action is renamed ReplaceSchemaAction (Transaction::replace_schema()). Upstream's can't rename, promote or make a column optional, which pg2iceberg needs.
Equality/position deletes through FastAppendAction Applied cleanly; tests ported to Table::manifest_list_reader() (upstream dropped Snapshot::load_manifest_list).
RewriteFilesAction Ported: no key_metadata (upstream removed it from SnapshotProducer::new and FastAppendAction), and manifest_list_reader().
Table::properties() (vended credentials) Re-applied by hand: table.rs gained runtime and an encryption manager upstream.
SigV4 request signing Applied cleanly. The lockfile update for its aws-* dependencies, missing on the old branch, is now in that commit.
Data sequence number + removed-file check (#2) Tests ported to manifest_list_reader().
New: rewrites keep the summary's running totals See below.
style: rustfmt the fork's patches The pinned nightly rustfmt over the fork's code only (upstream's files were already clean). It also merges a duplicated doc paragraph clippy flagged.

Rewrites keep the summary's running totals

RewriteFilesAction recorded no removed files. Once apache#2391 makes the totals accumulate, every compaction would have added its outputs to total-records / total-data-files without subtracting its inputs, and left total-delete-files / total-equality-deletes counting retired deletes.

Tests

  • iceberg lib: 1392 passed. iceberg-catalog-rest lib: 44 passed.
  • New rewrite_summary_keeps_running_totals: two appends, one with an equality delete, then a rewrite of a data file that retires the delete. total-records 30 → 29, total-data-files 2, total-delete-files 0, deleted-records 10. It fails without the patch (mutation check).
  • cargo fmt --check and cargo clippy for both crates are clean.

Next

A pg2iceberg PR that points the dependency at this branch and migrates to the 0.10 API. For example, TableBuilder now requires a runtime, and update_schema().set_schema() becomes replace_schema().set_schema(). A DST invariant will check summary totals against the live files on the prod backend.

🤖 Generated with Claude Code

hasyimibhar and others added 8 commits October 6, 2026 14:51
Two related changes that let downstream crates author their own
TransactionActions and drive schema evolution without going through the
private TableCommit::builder.

1. Flip TransactionAction (and BoxedTransactionAction) from pub(crate)
   to pub. The blanket ApplyTransactionAction impl already accepts any
   T: TransactionAction, so external implementors can add custom actions
   to a Transaction without further changes.

2. Add ReplaceSchemaAction. Callers compute the desired Schema and hand
   it to set_schema(); on commit the action emits AddSchema +
   SetCurrentSchema(-1) so TableMetadataBuilder allocates the new
   schema id. Three requirements (UuidMatch, CurrentSchemaIdMatch,
   LastAssignedFieldIdMatch) make concurrent schema/column changes
   fail loudly. Upstream's UpdateSchemaAction (0.10) evolves a schema
   column by column and can't rename, promote or make a column
   optional, so this stays alongside it.

Also add a Transaction::replace_schema() convenience method matching the
existing fast_append / update_location / update_statistics shape.
…ction

FastAppend previously rejected any DataFile whose content_type was not
Data, blocking commits that include equality- or position-delete files.
This change accepts mixed input and routes by content_type into
separate manifests at commit time, matching the Iceberg spec
requirement that one manifest must hold a single content type.

- FastAppendAction.add_data_files() now routes incoming files into
  added_data_files (Data) or added_delete_files (EqualityDeletes /
  PositionDeletes). The old "Only data content type is allowed" check
  is gone.
- SnapshotProducer carries added_delete_files alongside added_data_files
  and writes them to a separate delete manifest via build_v{2,3}_deletes.
- validate_added_data_files renamed to validate_added_files and is now
  generic over a slice — the partition-spec / partition-value rules
  apply uniformly to data and delete files.
- Snapshot summary collector iterates both vecs so file/record counts
  reflect everything added in the snapshot.
- New test test_fast_append_routes_equality_deletes_into_separate_manifest
  exercises a mixed Data + EqualityDeletes commit through MemoryCatalog
  and asserts both manifests appear in the manifest list with the right
  ManifestContentType.
Adds a new transaction action that produces snapshots with operation =
Replace: drops a set of existing data/delete files and adds new ones in
one atomic commit. Wired as `Transaction::rewrite_files()`.

Manifests from the prior snapshot are walked and classified per-manifest:
  - No removals: carry forward unchanged.
  - All entries removed: drop manifest from new manifest list.
  - Partial removal: rewrite manifest with survivors marked Existing
    (preserving original snapshot_id + sequence_number).

Added files go into a new ManifestStatus::Added manifest, routed by
content_type() into separate Data / Deletes manifests as the spec
requires.

To support partial-manifest rewrite, SnapshotProduceOperation::existing_manifest
now takes &mut SnapshotProducer so impls can call write_existing_manifest_for
(new helper) inline. FastAppendOperation's signature was updated to match;
its body is unchanged.

snapshot_summary::update_snapshot_summaries now accepts Operation::Replace
alongside Append/Overwrite/Delete.

Tests:
- full-manifest drop + swap
- partial rewrite preserves survivors as Existing
- mixed data + delete manifests carry forward correctly
- no-removal path is just a Replace-tagged append
- empty-input action errors (matches SnapshotProducer's contract)

Mirrors Java/PyIceberg's RewriteFiles semantics but keeps the implementation
narrow — we don't emit Deleted entries for audit-trail purposes, since the
new manifest list simply doesn't reference the dropped files. Can revisit
when expiration tooling needs them.
Plumbs the REST loadTable response config through Table::properties()
so downstream consumers can extract vended S3 credentials without
duplicating the catalog request. Previous behavior: response.config
was merged into FileIO and discarded.

The new field is empty for catalogs that don't surface table-scoped
config (file, glue, memory) — it's an additive change with no
behavioral impact on existing callers.

Used by pg2iceberg-rust to build per-table object stores for the
vended-credentials S3 router (Polaris, Snowflake, Tabular).
AWS-native Iceberg REST endpoints (S3 Tables, Glue) require every
request to be SigV4-signed; the REST client previously only supported
bearer/OAuth, so requests returned 403 MissingAuthenticationToken.

Add an opt-in SigV4Signer in the REST HttpClient, enabled per-catalog
via the `rest.sigv4-enabled` property (with `rest.signing-region` /
`rest.signing-name`, falling back to `s3.region`). Credentials load
lazily from the standard AWS chain (env, ECS/EKS task role, EC2 IMDS,
shared config) and are cached for the process. Signing runs last in
execute(), over the final headers + body, so it is inert unless enabled.
The snapshot summary's `total-*` counts are carried from the previous
snapshot plus this one's `added-*` and minus its `deleted-*` /
`removed-*`. A rewrite recorded no removed files, so after a compaction
`total-records`, `total-data-files` and the delete totals still counted
everything it replaced. Engines read these: ClickHouse 26.2 answered
`count()` from `total-records`.

The rewrite now looks up the live files it removes before the snapshot
is produced (the summary is built before the manifests are written) and
the producer subtracts them, under each file's own partition spec. The
same pass carries the existing check that every removed file is still
live, still on every commit attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar merged commit cc60df5 into polynya-patches-v0.10.1 Oct 6, 2026
15 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant