Skip to content

test_uk_replay_memory fails about 1 run in 12: its private marker is digits #237

Open

Description

@MaxGhenis

tests/test_uk_replay_memory.py::test_measured_run_samples_the_tree_and_keeps_only_aggregate_lines fails by chance. It checks that 777, the private tail FAKE_COMPUTE prints (line 532), never reaches json.dumps(result) (line 547). That result holds measured numbers, so any of them containing 777 fails the test. No private data leaks.

Evidence

  • CI run 41137112033 failed at f6bea13 with assert '777' not in '{"exit_code...Exchequer"]}', with 3,052 other tests passing. Pull request Move publish-db out of the pull-request workflow #220 reran clean on later heads.
  • Twelve local runs of the test gave one failure with that assertion.
  • Forty direct calls to measure.measured_run gave three results containing 777. One held "_t": 99522.897772, the time.perf_counter() reading from pipeline/modal_uk_replay_memory.py:268.

Fix

Use a marker no measurement can contain, such as a non-numeric token, in both places. Keep it as the tail of the otherwise accepted line, so the test still proves the runner drops that tail. Confirm the test still fails when the tail leaks, and passes on repeated runs.

#234 changes tests/test_uk_replay_memory.py, so check it first.

Activity

  1. MaxGhenis commented on Oct 30, 2026

    @MaxGhenis
    ContributorAuthor

    A second cause feeds the first. In measured_run, {**phase, ...} copies _t into each recorded phase before phase.pop("_t") runs, so results keep the raw time.perf_counter() reading. docs/uk_replay/evidence/replay_memory.json on main has 191 lines with "_t". Dropping the field removes one source of chance matches, but byte counts and seconds can still contain 777, so the marker still needs changing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions