Skip to content

Print schemas, snapshots, summaries and metadata in a stable order - #46

Merged
mdub merged 1 commit into
mainfrom
mdub-20261111-stable-output-order
Nov 11, 2026
Merged

mdub merged 1 commit into
mainfrom
mdub-20261111-stable-output-order

Conversation

@mdub

@mdub mdub commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

bergr now prints schemas, snapshots, snapshot summaries, table metadata and namespace properties in a stable order, so running the same command twice gives the same output. Fixes #43.

iceberg-rust keeps these collections in HashMaps, which iterate in a different order in every process.

Output Order
schemas by schema ID
snapshots oldest first: by sequence number, then timestamp, then snapshot ID
snapshot summaries (snapshots, snapshot … info) operation first, then the other keys sorted
metadata object keys sorted; schemas, partition-specs, sort-orders, snapshots and the statistics sorted by ID
namespace … info properties sorted

metadata is a workaround

metadata prints iceberg-rust's own serialisation of TableMetadata, which builds those arrays with HashMap::into_values(), and bergr can't change a foreign type's Serialize. So canonical_metadata serialises to a serde_json::Value, which sorts object keys (bergr's serde_json doesn't enable preserve_order), and then sorts the arrays listed in UNORDERED_METADATA_ARRAYS by ID. Other arrays, such as schema fields, sort-order fields and the snapshot and metadata logs, are already in a meaningful order and are left alone.

Visible change: top-level keys in metadata are now alphabetical, so format-version no longer comes first.

I considered printing the metadata file itself instead, but rejected it. That would make metadata need storage access as well as catalog access, and a REST loadTable response may omit metadata-location. The right long-term fix is in iceberg-rust, after which the workaround can go.

Testing

  • Each new test uses ten entries listed in reverse, so a hash-ordered result almost never passes by chance; each failed before its fix. The metadata test also asserts that two loads of the same metadata print identically.
  • cargo fmt, cargo clippy --all-targets -- -D warnings and cargo test (67 tests) pass.
  • Against a real AWS Glue Iceberg REST catalog, two runs each of schemas, snapshots, snapshot current info and metadata (4,000 lines) on the same table give byte-identical output.

iceberg-rust keeps schemas, snapshots and other table metadata collections in
HashMaps, so bergr printed them in an order that changed from run to run.

- schemas: by schema ID
- snapshots: by sequence number, then timestamp, then ID (oldest first)
- snapshot summaries and namespace properties: keys sorted, `operation` first
- metadata: object keys sorted, and arrays built from HashMap values sorted by
  ID; a workaround until iceberg-rust serialises TableMetadata stably

Fixes #43
@mdub
mdub merged commit 4683a4a into main Nov 11, 2026
4 checks passed
@mdub
mdub deleted the mdub-20261111-stable-output-order branch November 13, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output order of schemas, snapshots and snapshot summaries varies between runs

1 participant