Repository navigation
Print schemas, snapshots, summaries and metadata in a stable order - #46
Merged
Merged
Conversation
iceberg-rust keeps schemas, snapshots and other table metadata collections in HashMaps, so bergr printed them in an order that changed from run to run. - schemas: by schema ID - snapshots: by sequence number, then timestamp, then ID (oldest first) - snapshot summaries and namespace properties: keys sorted, `operation` first - metadata: object keys sorted, and arrays built from HashMap values sorted by ID; a workaround until iceberg-rust serialises TableMetadata stably Fixes #43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bergr now prints schemas, snapshots, snapshot summaries, table metadata and namespace properties in a stable order, so running the same command twice gives the same output. Fixes #43.
iceberg-rust keeps these collections in
HashMaps, which iterate in a different order in every process.schemassnapshotssnapshots,snapshot … info)operationfirst, then the other keys sortedmetadataschemas,partition-specs,sort-orders,snapshotsand the statistics sorted by IDnamespace … infometadatais a workaroundmetadataprints iceberg-rust's own serialisation ofTableMetadata, which builds those arrays withHashMap::into_values(), and bergr can't change a foreign type'sSerialize. Socanonical_metadataserialises to aserde_json::Value, which sorts object keys (bergr's serde_json doesn't enablepreserve_order), and then sorts the arrays listed inUNORDERED_METADATA_ARRAYSby ID. Other arrays, such as schema fields, sort-order fields and the snapshot and metadata logs, are already in a meaningful order and are left alone.Visible change: top-level keys in
metadataare now alphabetical, soformat-versionno longer comes first.I considered printing the metadata file itself instead, but rejected it. That would make
metadataneed storage access as well as catalog access, and a RESTloadTableresponse may omitmetadata-location. The right long-term fix is in iceberg-rust, after which the workaround can go.Testing
metadatatest also asserts that two loads of the same metadata print identically.cargo fmt,cargo clippy --all-targets -- -D warningsandcargo test(67 tests) pass.schemas,snapshots,snapshot current infoandmetadata(4,000 lines) on the same table give byte-identical output.