Skip to content

Load-sensitive tests: bucket_transport::fleet_hammer_distinct_branches_all_land_without_coordinator fails repeatedly under load; fill::tests::a_wide_level_is_chunked_by_count_... runs >20 CPU-minutes locally; readers_and_writers_stay_consistent_while_compaction_runs hung once #271

Closed

Description

@tonibergholm-codento

Reported by several agents' full local runs while many cargo builds/tests ran in parallel (CI passes): (1) bucket_transport::tests::fleet_hammer_distinct_branches_all_land_without_coordinator failed in 3 consecutive full runs under load (and once in earlier runs), passes alone; it exercises the manifest CAS retry budget (16 attempts) — under load a legitimate writer may exhaust its retries, which would be a real robustness bug, not only a test problem. (2) fill::tests::a_wide_level_is_chunked_by_count_before_anything_is_split_by_size ran >20 CPU minutes locally. (3) readers_and_writers_stay_consistent_while_compaction_runs hung >30 min once (being investigated in #264). Needed: root-cause each (livelock vs starvation vs unbounded work), bound the retry/backoff behavior (jittered backoff, fair retries), give worker threads hard deadlines so tests fail with a diagnostic instead of hanging, and make test cost deterministic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions