Skip to content

Keep git's background maintenance out of test repositories (engine-free flake) - #1266

Merged
MaxGhenis merged 3 commits into
mainfrom
fix-fixture-git-maintenance-race
Oct 13, 2026
Merged

MaxGhenis merged 3 commits into
mainfrom
fix-fixture-git-maintenance-race

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes a flake in the engine-free job. No issue tracked it; the NZ hub reported it on 2026-10-13.

The failure

microcosm main's Tests run at 74f5e61 (the #1254 merge) failed in engine-free (3.14) (run 38405652457):

packages/microcosm-frame/tests/engine_free/shared/test_axiom_graph_adapter.py:1582: in test_a_clean_checkout_at_the_declared_commit_is_accepted
    first = self._checkout_ref(root, head)
packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py:1276: in axiom_engine_ref
    tree_sha256, tree_bytes = rulespec_tree_digest(resolved_root)
packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py:1143: in rulespec_tree_digest
    content = candidate.read_bytes()
E   FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_a_clean_checkout_at_the_d0/rulespec-git/.git/objects/maintenance.lock'

The runs either side passed. It is a flake, not #1254's content. Since the checkout tests merged (#1118, 2026-10-07), 356 Tests runs have started: 193 passed, 49 failed, 108 were cancelled and the rest are still running. 43 of the failed runs have a failed engine-free job. I read the 42 whose logs GitHub still serves, and this is the only one that names maintenance.lock (the unreadable one, run 37653545314, failed in both engine-free jobs at once).

Cause (confirmed)

  1. The fixture's git commit starts a maintenance process. In git 2.55.0 (the version on the CI runner and on the host I measured on), builtin/commit.c calls run_auto_maintenance, and prepare_auto_maintenance (run-command.c) starts git maintenance run --auto --detach unless maintenance.auto is false, or maintenance.auto is unset and gc.auto is 0 or less. The test helper _git set neither. A trace2 event file shows that child in 200 of 200 commits made with the helper's options.
  2. That process holds a lock after the commit returns. maintenance_run_tasks (builtin/gc.c) takes <objects>/maintenance.lock, runs its foreground tasks, detaches, runs its background tasks, then removes the lock. Checked the instant git commit returned, the lock was still there in 22, 94 and 32 of 200 commits in three runs, and stayed for up to 34 ms.
  3. The checkout reference hashes .git, in two steps. rulespec_tree_digest lists every file under the root (rglob and is_file), then reads each. A lock that is listed and then removed raises FileNotFoundError at the read. That is the frame in the CI traceback.

So the test does not fail on what it asserts. It fails when the detached maintenance process removes its lock between the digest's listing and its read.

The fix

Tests: no test repository starts background maintenance

New test_support/git_repositories.py defines NO_BACKGROUND_MAINTENANCE: -c maintenance.auto=false -c gc.auto=0 -c maintenance.autoDetach=false. Beside it, NO_BACKGROUND_MAINTENANCE_ENV gives the same three settings as GIT_CONFIG_COUNT environment variables, for git that the code under test runs itself from os.environ. Every test that commits or fetches in a throwaway repository now uses one of them:

File How Reads .git itself?
packages/microcosm-frame/tests/engine_free/shared/test_axiom_graph_adapter.py (_git, used for every repository in the file) options yes: the flaky site
packages/microcosm-build/tests/engine_free/us/test_us_chronicle_feed.py options no
packages/microcosm-build/tests/engine_free/us/test_us_modal_stage_plan_tool.py variables in _GIT_ENV, which reach both its helper and the git fetch that _verify_branch runs no
packages/microcosm-build/tests/engine_workflow/us/test_us_annual_static_aging.py options no
test_support/microcosm_graph/graph_acceptance_burndown.py options no

Only the first can hit this failure. The code the other four exercise never reads .git as files: it asks git (status, ls-files, ls-tree, rev-parse, show, fetch). They get the same settings so that no test repository has a process still writing to it after the command returns. git fetch starts the same maintenance process as git commit (a trace2 file shows it in the branch check's fetch into its scratch clone, and none with the variables set). The remaining test-side git calls (test_route_a_driver.py, test_us_modal_stage_puf_support_base.py, test_uk_staging_integration.py, test_logbook_adoption.py) only read this repository or mock git, and are unchanged. docs/agent-guide.md now tells new tests to do the same.

Each option closes the race on its own (table below), so the three together do not depend on which one a given git honours.

Production: rulespec_tree_digest refuses by name

The same race exists on a real checkout. If axiom_engine_ref is given a git checkout root while git's background maintenance (or any other git process) is still working in it, the digest can list a lock and find it gone, or hash a lock that is then removed, which moves the reference.

  • It keeps hashing .git. That is deliberate and documented. The digest must equal the graph's directory source key (microcosm.graph.keys._directory_identity, held by test_differential_digest_equals_the_graph_directory_identity), and a checkout reference is meant to be clone-specific (test_a_checkout_reference_is_clone_specific_and_an_export_is_not). Leaving .git out would change what a checkout reference names. That is a design change, not a flake fix.
  • It does not skip a file that vanished. A tree that changes while it is hashed has no digest to name, so refusing is right, and the code already refused.
  • The one change: the refusal is now the function's ValueError, naming the file and the remedy (wait for git, or reference a git archive export), where it was an unhandled FileNotFoundError. axiom_engine_ref documents ValueError for every other refusal. No digest value changes.

The docstrings of rulespec_tree_digest and axiom_engine_ref now say this. The graph's own _directory_identity is unchanged: it still raises FileNotFoundError in the same case.

Invariants

  • For a tree nothing is writing to, rulespec_tree_digest returns what it returned before, equal to _directory_identity (existing Hypothesis differential, 60 examples; unchanged and passing).
  • No digest is returned for a tree that loses a file between the listing and its read. For every generated tree and every file in it, removing that file after the listing raises ValueError naming it, and the settled tree then hashes to the graph's identity (new Hypothesis property, 40 examples).
  • A commit made through the test helper starts no git maintenance or git gc process and leaves no maintenance.lock; nor does the branch check's fetch (two new tests; they read a trace2 event file, so they do not depend on timing).
  • The environment form sets exactly what the options set: git config --list --show-scope lists the same command-scope entries under both (new test, the consistency check docs/shared-constants.md asks for of a derived view).

New tests

Test With the matching half of the fix reverted Here
test_a_commit_in_a_test_repository_starts_no_git_maintenance fails: the trace holds git maintenance run --auto --quiet --detach passes
test_a_lock_git_removes_while_the_checkout_is_hashed_is_refused (the CI failure, reproduced without timing) fails with the CI error, FileNotFoundError on .git/objects/maintenance.lock passes
test_property_a_file_removed_after_the_listing_is_refused_by_name fails with FileNotFoundError passes
test_the_branch_check_starts_no_git_maintenance (plan tool) fails: the trace holds git maintenance run --auto … --detach passes
test_the_maintenance_variables_set_what_the_options_set (plan tool) (a consistency check of the derived view; not mutation-tested) passes

I measured the middle column in this checkout: NO_BACKGROUND_MAINTENANCE emptied for the first row, axiom.py restored from main for the next two, and NO_BACKGROUND_MAINTENANCE_ENV left out of the plan tool's _GIT_ENV for the fourth.

Before and after

One host throughout: macOS 26.6.2, 18 CPUs at a load average of 50 to 100, git 2.55.0, CPython 3.14.7 (free-threading build). Never more than 4 pytest workers in total; OMP_NUM_THREADS=4. "Before" is a separate worktree of origin/main 4b33e3c with its own environment. The host's load control paused some runs for a while and then resumed them; a pause changes timing only, and no run failed.

Measurement Before (main) After (this PR)
maintenance.lock still present when the helper's git commit returns (200 commits, three runs) 22, 94 and 32 0
rulespec_tree_digest called twice straight after the fixture's commit (3,000 repositories): a listed file was gone at the read 630 (21%), each on .git/objects/maintenance.lock 0
The same 3,000: both calls returned but the digest moved 7 0
The flaky test alone, 2,000 runs, -n 4 0 failed 0 failed
The whole test file, 30 runs at -n 2: 15 side by side with the PR's first commit 1123be8, then 15 alternating with 219e0b4 30 of 30 passed 30 of 30 passed

The exact test does not fail on this host, before or after. In that test the digest starts 134 to 242 ms after the commit returns (median 175 ms, 100 runs), because the fixture and the clean-checkout checks run eight git commands first. Here the lock lasted at most 34 ms in my runs (the first independent reviewer saw up to 149 ms with empty commits). The CI failure shows it outlived that gap on the runner at least once; I did not measure on a runner. On CI, the 193 passing runs each ran two engine-free jobs against this one failure, about 1 in 400 jobs, so 30 local runs could not be expected to show it either way. The second and third rows are the same race with that delay removed: the fixture's own git commit and the production digest, nothing mocked.

Lock observation by option, 200 commits each:

Options added to the helper Commits that started git maintenance Lock present when the commit returned
none (main) 200 32
gc.auto=0 0 0
maintenance.auto=false 0 0
maintenance.autoDetach=false 200 0
gc.autoDetach=false 200 0
all three in this PR 0 0

The second column comes from a trace2 event file per commit; the third is checked the instant git commit returns.

Review

Three commits, each reviewed independently; every review approved with no blocking findings.

Commit Reviewer Verdict What it raised, and where it went
1123be8 (the fix) GPT-6.1 Sol (Subfleet review, hard tier) APPROVE One more site: the plan tool test's _verify_branch runs its own git fetch, which starts the same maintenance process. Covered in 219e0b4.
1123be8 Opus 5.5 (separate agent) APPROVE Use maintenance.autoDetach rather than its fallback gc.autoDetach; hedge the error message for non-git roots; a guide note for new tests; changelog wording. All in 219e0b4.
219e0b4 (review follow-ups) Opus 5.5 APPROVE Wording in the changelog, guide and docstring; a skip guard. All in 3c4182f.
3c4182f (wording) Opus 5.5 APPROVE One optional changelog clause, left as is.

Left as is, on purpose:

  • The production branch check (tools/modal_us_stage_plan.py) still lets its fetch start maintenance in its scratch clone, which _verify_branch then deletes with rmtree(..., ignore_errors=True). That races only a temporary directory, and nothing has failed there. git fetch --no-auto-maintenance would close it if it ever matters.
  • microcosm.graph.keys._directory_identity has the same list-then-read loop and still raises FileNotFoundError. Digest values agree (the differential test); only the error type differs.
  • _require_clean_checkout can still raise FileNotFoundError if a git command rewrites the work tree during the check. Background maintenance cannot cause that: _checkout_files skips the top-level .git.

axiom: n/a (test infrastructure and an error type; no policy change)

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits October 13, 2026 13:24
main's Tests run at 74f5e61 failed once in engine-free (3.14):
test_a_clean_checkout_at_the_declared_commit_is_accepted raised
FileNotFoundError on .git/objects/maintenance.lock inside
rulespec_tree_digest.

git commit starts `git maintenance run --auto`, which takes that lock,
detaches and removes it after the commit has returned. The checkout
reference hashes .git: the digest lists every file and then reads each,
so it can list the lock and find it gone.

Every test helper that commits to a throwaway repository now passes
test_support.git_repositories.NO_BACKGROUND_MAINTENANCE, so no
maintenance process starts. rulespec_tree_digest still refuses a tree
that loses a file while it is hashed, now with a ValueError naming the
file instead of an unhandled FileNotFoundError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both independent reviews approved; this takes their non-blocking points.

- The modal stage plan test runs _verify_branch, whose own git fetch
  started background maintenance in its scratch clone. Git run by the
  code under test reads os.environ, so test_support.git_repositories
  now also gives the settings as GIT_CONFIG_* variables
  (NO_BACKGROUND_MAINTENANCE_ENV), and that test sets them. New tests:
  the branch check starts no maintenance (trace2), and the variables
  set exactly what the -c options set.
- Use maintenance.autoDetach=false, not its fallback gc.autoDetach.
- The digest's refusal says what to do for any root, then for a git
  checkout; the docstrings name the refusal where compiles re-hash.
- docs/agent-guide.md: tests that commit or fetch in a throwaway
  repository pass these settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wording from the second review, and one skip guard; no behaviour change.

- Changelog and guide: tests that commit or fetch apply the settings
  as -c options or, where the code under test runs git, as variables;
  a helper whose environment carries the variables needs no options;
  code that strips git's local variables drops them.
- NO_BACKGROUND_MAINTENANCE_ENV overrides GIT_CONFIG_COUNT entries,
  not every GIT_CONFIG_* variable.
- The digest's refusal says a git process "can" remove the file.
- The variables-versus-options test skips without git, like the rest
  of its section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 13, 2026 22:19
@MaxGhenis
MaxGhenis merged commit a91c753 into main Oct 13, 2026
10 checks passed
@MaxGhenis
MaxGhenis deleted the fix-fixture-git-maintenance-race branch October 13, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant