Skip to content

test(ci): widen the git-probe stall budget and its stall floor (BLO-22985) - #1850

Merged
kkroo merged 1 commit into
masterfrom
staff/blo-22985-git-probe-stall-budget
Sep 16, 2026
Merged

kkroo merged 1 commit into
masterfrom
staff/blo-22985-git-probe-stall-budget

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Sep 14, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • Its server test suite runs in a 4-shard merge-queue job, where shards contend for CPU and IO
  • BLO-22985 tracks a class of test defect in that suite: hardcoded wall-clock budgets tight enough to trip under that contention, with no defect in the code under test
  • Each occurrence resets and re-stages the entire merge-queue batch behind the failing head (~36 min of fleet queue drain for the first one)
  • The sweep for this class flagged heartbeat-workspace-session.test.ts:1087 early, but it was deferred behind fix(workspaces): collect completed per-run worktrees (BLO-22984) #1252's hold on the same file; fix(workspaces): collect completed per-run worktrees (BLO-22984) #1252 has been dirty and untouched since 2026-08-25, so the deferral is stale and no open PR now touches the file
  • This pull request closes that last stated residual of the sweep
  • The benefit is one fewer wall-clock tripwire in the merge queue, with the invariant it guards made harder to fool, not easier

Linked Issues or Issue Description

Refs BLO-22985 — CI reliability: hardcoded wall-clock budgets in server tests fail under CI load.

Sibling PRs from the same sweep, all currently in the merge queue: #1792, #1806, #1826. This PR touches a file none of them touch (verified by file-set intersection against all 56 queued PRs).

What Changed

  • server/src/__tests__/heartbeat-workspace-session.test.ts — "rejects when the git checkout probe stalls past its timeout":
    • Budget expect(Date.now() - startedAt).toBeLessThan(1_500)5_000.
    • Fake git stall sleep 5sleep 20.
    • Comments stating the margin rationale on both ends.

The stall change is required, not incidental. The test proves the probe honours its 200ms deadline instead of waiting out the stalled git. With the old 5s stall, a budget of 5s would have left a probe that ignored its timeout landing at ~5.1s against a 5s budget — i.e. widening alone would have quietly stopped catching the regression, which BLO-22985's AC3 explicitly forbids. Raising the stall restores the separation.

Verification

npx vitest run server/src/__tests__/heartbeat-workspace-session.test.ts \
  -t "rejects when the git checkout probe stalls past its timeout"
case elapsed vs 5000ms budget
healthy (probe honours its 200ms deadline) 280ms passes, ~18x margin
negative control: probe timeout forced to 30000ms so it waits out the stall 20028ms failsAssertionError: expected 20028 to be less than 5000

The negative control was run by temporarily setting PAPERCLIP_STRICT_GIT_CHECKOUT_PROBE_TIMEOUT_MS = "30000" in the test and reverting it; it is empirical, not arithmetic. Test-file cap is the global testTimeout: 60_000, so the 20s stall fits with 3x headroom.

Risks

Low risk; test-only, one test. Two failure modes considered:

  • Budget too loose to catch a real regression — refuted by the negative control above: an ignored timeout is 4x over budget.
  • A genuine regression now takes 20s instead of 5s to surface. It still fails, and this file already runs serialized (--no-file-parallelism --maxWorkers=1 in test:k8s-concurrency-chaos), so the extra 15s is on a single serialized path and only on the failing branch.

No production code touched. No other budget in the file changed.

Model Used

Claude Opus 4.5 (claude-opus-4-5), extended thinking, via Claude Code with tool use.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots (n/a — test-only)
  • I have updated relevant documentation to reflect my changes (n/a — rationale is in-code)
  • I have considered and documented any risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

…2985)

heartbeat-workspace-session.test.ts asserted the strict git-checkout probe
returns in < 1500ms when the probe's own deadline is 200ms. Measured actual
is 280ms, so the budget carried ~5x margin against a merge-queue runner that
has produced 33% wall-clock overruns on this repo's timing tests.

Widening the budget alone would have broken the invariant: the fake git only
slept 5s, so a probe that ignored its timeout would have landed at ~5.1s
against a 5s budget. Raise the stall to 20s in the same change, keeping a
genuinely-hung probe 4x over the budget.

Measured: healthy 280ms / budget 5000ms / ignored-timeout 20028ms.

Refs BLO-22985

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Sep 14, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22985

1 similar comment
@allyblockcast

allyblockcast Bot commented Sep 14, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22985

@github-actions

Copy link
Copy Markdown

@ally head 60a516c has been awaiting review for 2.2h with no review on either surface (pulls/1850/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 60a516c.

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 60a516c

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

  • The synthetic stall is widened to 20 seconds while the probe timeout remains 200ms, preserving a clear negative-control separation.
  • The 5-second assertion gives the timeout-bound test meaningful runner-contention headroom without allowing the old ignored-timeout behavior to pass.
  • Environment variables and temporary filesystem state are restored in finally, keeping the test isolated from neighboring tests.

Recommended Action

  1. No Critical or Important issues found. The App-authored PR should receive this formal review as COMMENTED.

@kkroo
kkroo added this pull request to the merge queue Sep 14, 2026
Merged via the queue into master with commit c1e1594 Sep 16, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant