Skip to content

test(irq): replace real sleeps with a stepped fake clock (#7598) - #7619

Merged
dwu96 merged 1 commit into
mainfrom
fix/irq-test-deterministic-clock-7598
Sep 1, 2026
Merged

test(irq): replace real sleeps with a stepped fake clock (#7598)#7619
dwu96 merged 1 commit into
mainfrom
fix/irq-test-deterministic-clock-7598

Conversation

@bolichen97

Copy link
Copy Markdown
Collaborator

Problem / Motivation

test/test_irq.py::test_an_entry_joining_after_a_partial_fire_serves_its_own_floor fails intermittently on loaded CI runners (observed on PR #7300, Backend Tests (3.12, 2), as a bare AssertionError: assert False; the rerun of the same commit passed). The file measures its coalescing floor in real wall clock: _COALESCE = 0.01 with _settle() sleeping _COALESCE * 3. Two assertions require the opposite of settling -- that a floor has NOT yet expired -- and their entire budget is the gap between two back-to-back _verdict() calls. Any scheduling stall >= 10 ms there (4 xdist workers, ~19.8k items, memory-bounded box) lets the joining entry's own floor close, so it Reports instead of Skips.

Closes #7598

Why it matters

The flake reddens whichever Backend Tests shard carries the test plus the downstream Coverage Gate, taxing every PR on the repo with rerun round-trips. And the test guards a recently fixed behaviour (#7431: a coalescing entry gets its own floor, not the window's age), so it must stay -- deleting/skipping/flaky-marking it would unguard that fix.

What changed (motivation -> approach -> change)

Symptom: assertions about intervals staying SHORT race real scheduling. Root cause: the kernel reads time.time() while the test controls elapsed time only by sleeping, so the clock keeps moving between the test's own statements. Change: give the test module full ownership of the clock the kernel reads.

  • A _FakeClock (time() / advance() / reset()) is installed by an autouse fixture over the module attribute kiro_crew.irq.time -- the name the kernel's clock reads resolve -- via monkeypatch.setattr, never the stdlib module object, so the fake is scoped to the kernel and invisible to everything else in the process.
  • _settle() now advances the fake clock by _COALESCE * 3 instead of sleeping; every existing call site keeps its exact meaning at zero wall cost.
  • The three pre-aged state stamps (opened_at/alerted/coalesce_started_at) derive from the fake clock, and the one time.sleep(0.05) crossing a realert_secs=0.01 window (a kernel-measured interval) becomes a clock advance. No other real-time dependency exists in the file; test_future_timestamp_reads_as_stale_... uses the absolute literal 2**40 and is unaffected.
  • The two "has not waited" assertions are now exact BY CONSTRUCTION: a clock that moves only when the test moves it cannot be aged by a scheduling stall between two _verdict() calls. That construction -- not a reproduced flake -- is the proof of the fix (the reporter could not reproduce locally and said so; a workstation has more headroom than a loaded CI box).
  • _FakeClock.__getattr__ raises a directive error if kiro_crew.irq ever grows a clock read beyond time.time() (e.g. monotonic), so the failure names this fixture instead of reading as harness breakage.

Deliberately NOT done: no injectable-clock parameter in src/kiro_crew/irq.py. The test-side seam is sufficient, a production seam widens the blast radius, and PR #7584 (open) is concurrently editing that file -- a production change here invites a real conflict. Also not done: raising _COALESCE (only widens the window a stall must beat, keeping the assertion probabilistic while slowing the suite).

Relationship to PR #7609: opened concurrently for the same issue by a kiro-agent session under the same login, it threads a clock callable through run() in src/kiro_crew/irq.py (+9/-2 production lines overlapping the region #7584 edits) and converts 2 of the 4 real-clock uses in the test file. This PR is the zero-production-surface alternative: no irq.py change, no #7584 conflict, all 4 real-clock uses converted, and the mutation check below. One of the two should merge; maintainer's call.

Tests

All 60 tests in test/test_irq.py pass, now in ~5s wall (previously each _settle() burned 30 ms of real sleep). Full suite: 78786 passed; the 96 failures + 2 errors are host-environment issues byte-identical to a pristine-main control run at the same base (fail-set diff empty in both directions).

Mutation check (proving the converted test still guards #7431): temporarily reverting the joining-entry behaviour in src/kiro_crew/irq.py -- handing a joining entry the window's oldest opened_at instead of its own -- makes test_an_entry_joining_after_a_partial_fire_serves_its_own_floor FAIL under the fake clock with exactly "a joining entry must not inherit the window's age"; restoring the file returns it to green. A determinism conversion that still catches the guarded defect is a determinism fix, not a defanged test.

Pre-push review: GPT 5.6 Sol PASS (0 findings); Opus 5 PASS (0 blocking, 4 Low advisories -- 3 adopted: stale clock-read-count docstring, __getattr__ directive guard, shared-global invariant comment; 1 was explicitly no-action-needed).

Manual verification

N/A -- unit coverage sufficient: the change is confined to one test module and the mutation check exercises the kernel path the tests guard.

Pattern harvest

Class: a test asserting an interval stays SHORT under a real clock is a latent flake on any loaded runner -- _settle()-style helpers make the window-CLOSED direction safe, but nothing protects the window-STILL-OPEN direction. Reusable shape: install a stepped fake clock over the clock name as the module under test resolves it (monkeypatch.setattr("pkg.mod.time", fake)), convert sleeps to advances, and mutation-check that the converted test still fails when the guarded behaviour is reverted. Candidates elsewhere would grep for time.sleep adjacent to assertions that something has NOT happened; this PR deliberately stops at test_irq.py per the issue scope.

test_irq.py measured its 10ms coalescing floor in real wall clock:
two assertions that a floor has NOT yet expired had only the gap
between two _verdict() calls as budget, so any >=10ms scheduling
stall on a loaded CI runner let the floor close and the assertion
fail (observed on PR #7300, Backend Tests (3.12, 2)).

An autouse fixture now installs a fake clock over the time name as
kiro_crew.irq resolves it (never the stdlib module object), _settle()
advances that clock instead of sleeping, and the three pre-aged state
stamps derive from it. Intervals the kernel measures are now exact by
construction: the clock only moves when the test moves it, so no
stall can age a window between two calls.

Mutation-verified: reverting the #7431 behaviour (joining entry
inherits the window's age) makes
test_an_entry_joining_after_a_partial_fire_serves_its_own_floor fail
under the fake clock, so the converted test still guards the defect.

Closes #7598
@bolichen97
bolichen97 requested a review from a team as a code owner September 1, 2026 12:11
@bolichen97
bolichen97 requested a review from pepmach September 1, 2026 12:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 427d2844eadb9c259f08b4105845e7cfe97f6073 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

The seam checks out: irq.py does import time and calls only time.time() (lines 684, 761), so patching the module attribute kiro_crew.irq.time intercepts every kernel clock read, and the __getattr__ guard fails loud if irq ever grows another clock read. No real sleeps remain in the test file; the module-global _clock is re-seeded per test by the autouse fixture, and xdist workers are separate processes so the global can't cross-contaminate. The change is test-only, converts all four real-clock uses, deliberately declines a production clock seam (with a stated conflict rationale against #7584 and an explicit comparison to the competing #7609), and the mutation check confirms the test still guards the #7431 behavior. Every hunk is accounted for by the stated purpose.

Design-Verdict: PASS

Root-cause fix at the right layer: the test owns the clock the kernel reads, zero production surface, and the guarded behavior is mutation-verified.

[DESIGN-REVIEWED] 427d284

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 427d2844eadb9c259f08b4105845e7cfe97f6073 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 427d284

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 427d2844eadb9c259f08b4105845e7cfe97f6073: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 427d2844eadb9c259f08b4105845e7cfe97f6073 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 427d284

Verdict parsed from the review's SHA-scoped output markers for commit 427d2844eadb9c259f08b4105845e7cfe97f6073.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 427d2844eadb9c259f08b4105845e7cfe97f6073: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 1, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 1, 2026 13:03

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change to test/test_irq.py -- replaces real sleep calls in the interrupt-controller tests with a stepped fake clock, so no production file is touched.

@dwu96
dwu96 merged commit 51888f8 into main Sep 1, 2026
68 checks passed
@dwu96
dwu96 deleted the fix/irq-test-deterministic-clock-7598 branch September 1, 2026 13:05

@dwu96 dwu96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change to test/test_irq.py — real time.sleep() calls replaced with a module-scoped fake clock monkeypatched over kiro_crew.irq.time, removing CI-timing flakiness; no production file touched.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_irq coalesce-floor test measures a 10ms window in real wall clock

3 participants