Skip to content

fix: pin test_channel_slots reconcile tests to a frozen clock so long shards cannot decay eligibility - #9009

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/8968-channel-slots-frozen-clock
Sep 6, 2026
Merged

fix: pin test_channel_slots reconcile tests to a frozen clock so long shards cannot decay eligibility#9009
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/8968-channel-slots-frozen-clock

Conversation

@javenciu

@javenciu javenciu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backend Tests (3.12, N) fails with 10 errors in test/test_channel_slots.py (all assert 0 == 1 / assert 0 == 2) on any shard that runs longer than ~30 minutes. The file passes in seconds when run alone, so it reads as a flake — but it is deterministic in elapsed time, and shard composition reshuffles (no committed .test_durations, so pytest-split balances by count) decide which PRs get hit. Observed on run 34021088434, where the shard ran 37:27 and all 10 failed on every xdist worker.

Why it matters

Every PR is exposed: the slowest 3.12 shard already runs ~29 minutes, within one minute of the 1800s window this module silently depends on. Any shard growth or reshuffle trips it, costing contributors a spurious red CI round and re-run time. Worse, past the window the suite's == 0 assertions keep passing vacuously — everything is ineligible, so the tests stop testing what they were written to test before they fail loudly.

What changed (motivation → approach → change)

  • Symptom: reconcile tests fail once elapsed-since-import exceeds the recency window.
  • Root cause: NOW = time.time() is captured at module import and stamps sessions via _session(), while reconcile_channel_slots derives its cutoff from the live clock (cutoff = time.time() - window_minutes * 60). A session stamped modified=NOW is eligible only while time.time() - NOW < 1800.
  • Change (option 2 from the issue — freeze the production clock): a frozen_clock fixture pins time.time to the module's NOW for the three classes that drive the real reconcile pass, so stamps and cutoff read one clock and eligibility becomes pure arithmetic at any elapsed time. Applied to TestReconcilePass, TestReconcileMore, and TestFailedTranscriptReadDefers — the latter was not in the CI failure list only because its window is 60 minutes: the same landmine one slow shard further out (reproduced: shifting NOW by −4000s fails test_a_genuinely_empty_transcript_still_surfaces on main).
  • Classes that pass an explicit cutoff=NOW - 1800 (self-consistent) or that legitimately bracket production writes with live-clock reads (TestClosedAtStamp, TestCompareAndClear, TestMtimeOf) are deliberately not frozen — freezing them would change what they verify.
  • Freezing time.time also makes the previously wall-clock-coupled assertions exact: the snapshot-cutoff bracket (before <= cutoff <= after) collapses to equality, and tombstone TTL arithmetic is unchanged. asyncio scheduling uses time.monotonic, which is untouched.

Tests

  • TestReconcileClockCoherence::test_a_fresh_stamp_survives_any_shard_elapsed_time — the defect's exact firing point: a session stamped at the pass's own now surfaces even with 7200 simulated seconds elapsed since import (fails on main's shape).
  • TestReconcileClockCoherence::test_the_window_still_filters_under_a_frozen_clock — a stamp older than the window is still filtered: the freeze does not disable the recency rule, so == 0 verdicts stay meaningful (guards against the vacuous-pass failure mode).
  • TestReconcileClockCoherence::test_a_stamp_exactly_at_the_cutoff_is_eligible — boundary pin: eligibility filters strictly (modified < cutoff); a stamp exactly at the edge surfaces.
  • Fails-before / fails-after: with NOW = time.time() - 2000 (the issue's reproduction) main fails the same 10 tests byte-for-byte; with this fix the full file passes 73/73 at rest, under −2000s, and under −4000s.
  • Seam neighbors: 59 test files touching channel_slots / chat_persistence pass (5063 tests).

Manual verification

N/A — unit coverage sufficient: the defect and fix are both fully expressed in the test process (clock perturbation reproduces CI's elapsed-time condition deterministically).

Screenshots / video

Test-only change to one backend test file; zero shipped pixels, no UI surface touched.

Related Issues

Fixes #8968

Pattern harvest

Rule candidate: lint
Pattern: module-level NOW = time.time() in a test file whose subject reads the live clock — stamps decay against the code under test as shard elapsed time grows; flag module-scope wall-clock captures used as default fixture stamps.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A, test-only change
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

I confirm this contribution is made under the same terms as my prior merged contributions to this repository (#8835).

…instant

test_channel_slots.py froze NOW at module import while the reconcile pass
under test derives its recency cutoff from the live clock, giving the module
a 30-minute shelf life: any CI shard running longer aged every default stamp
out of the window, failing all 10 nonzero reconcile assertions while the
== 0 assertions kept passing vacuously (kirodotdev#8968).

A frozen_clock fixture now pins time.time to NOW for the three classes that
drive the real reconcile pass (TestReconcilePass, TestReconcileMore, and
TestFailedTranscriptReadDefers, whose 60-minute window was the same landmine
one slow shard further out). Stamps and cutoff share one clock, so
eligibility is pure arithmetic at any elapsed time. TestReconcileClockCoherence
pins the invariant: a fresh stamp survives simulated hours of shard elapsed
time, the window still filters under a frozen clock, and the boundary stays
strict (modified < cutoff).

Fixes kirodotdev#8968
@javenciu
javenciu requested a review from a team as a code owner September 6, 2026 11:30
@javenciu
javenciu requested a review from bolichen97 September 6, 2026 11:30
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 6, 2026
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 75071617b72e18cf4df81184b3080ed4cd32c4d9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7507161

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 75071617b72e18cf4df81184b3080ed4cd32c4d9 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

The fix checks out against the base tree: channel_slots.py:797 derives the cutoff from the live clock (time.time() - window_minutes * 60) while the test fixtures stamp modified=NOW captured at import — exactly the decay the PR describes. The three frozen classes are the ones that drive the real reconcile pass; the unfrozen ones either pass an explicit self-consistent cutoff=NOW - 1800, fake out reconcile_channel_slots entirely (TestImmediateDispatcherSurface), or deliberately bracket live-clock writes (TestClosedAtStamp, TestCompareAndClear, TestMtimeOf). The eligibility filter is strictly < cutoff (channel_slots.py:314), so the boundary regression pin is accurate. The new regression class freezes to NOW + 7200 to simulate the long shard directly, and includes a negative test ensuring the freeze doesn't make the recency filter vacuous. Test-only change, no shipped surface, fully reversible.

Design-Verdict: PASS

Correct root-cause fix at the right layer: pinning the one clock both stamp and cutoff read, with regression pins that simulate the failing shard directly.

[DESIGN-REVIEWED] 7507161

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 75071617b72e18cf4df81184b3080ed4cd32c4d9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7507161

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 6, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 6, 2026 12:05

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change to test/test_channel_slots.py adding a frozen_clock fixture so reconcile tests pin the same clock the production recency cutoff reads, fixing #8968 where eligibility decayed with elapsed shard time; no production file touched. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.

@iamwhatever
iamwhatever merged commit 1d43fdb into kirodotdev:main Sep 6, 2026
67 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_channel_slots.py fails deterministically on any shard running >30min: NOW is frozen at import but the cutoff uses the live clock

2 participants