Skip to content

Output order of schemas, snapshots and snapshot summaries varies between runs #43

Closed

Description

@mdub

Running the same command twice against the same table can list things in a different order. For example, on an unchanged table:

bergr glue table db.events snapshots > run-1.out
bergr glue table db.events snapshots > run-2.out
cmp run-1.out run-2.out   # differ; equal once sorted

This affects bergr from, bergr glue and bergr rest alike. I noticed it while comparing two builds' output in #42.

Cause

iceberg-rust's TableMetadata keeps schemas and snapshots in HashMaps, and schemas_iter() and snapshots() iterate over them in hash order, which is randomised per process. A snapshot summary's extra properties (Summary::additional_properties) are a HashMap too, so the keys in snapshot … info move around as well.

Proposal

Print in a stable, meaningful order:

  • schemas: by schema ID.
  • snapshots: by sequence number, then timestamp, so the history reads oldest to newest (or newest first; either way, consistent).
  • snapshot … info: summary keys sorted, e.g. by collecting into a BTreeMap before serialising.

It's worth checking metadata too, and anything else that serialises maps (partition specs, sort orders, refs, properties).

Stable output makes runs easy to diff and to test. A regression test could build metadata with several schemas and snapshots and assert the order.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions