Skip to content

test: fix flaky toolValueText and detached-abort unit tests - #687

Merged
Rinat S (rsolmano) merged 3 commits into
mainfrom
investigate-flaky-unit-tests
Oct 8, 2026
Merged

Rinat S (rsolmano) merged 3 commits into
mainfrom
investigate-flaky-unit-tests

Conversation

@rsolmano

Copy link
Copy Markdown
Collaborator

Problem

The Unit Tests CI job failed in 15 of 280 attempts between Sep 30 and Oct 8, and 11 of those came in the last two days, so PRs and merge-queue runs keep failing for reasons unrelated to their changes. Two flaky tests caused 9 of the 15 failures:

  • toolValueText > falls back to String when JSON cannot serialize a value (extension-api, 6 failures; added Oct 6 in refactor(extensions): extract the host UI SDK and visualize web renderer #638). In Bun, JSON.stringify(cyclic, null, 2) takes time proportional to the stack size before it throws: about 300ms on macOS, about 650ms in Linux Docker, and 2.5–4.7s on CI even in passing runs, against the 5s test timeout. Without the indent argument it takes 0.4ms. There's no production impact: toolValueText only runs in the browser (V8).
  • a detached run SURVIVES a parent-turn abort (pi-subagents, 3 failures). The test aborted the parent after Bun.sleep(30) and expected the abort to land inside a 150ms fake response, while the child finished at 250ms. When the runner is busy, the abort comes late and lands on the parent's follow-up turn (triggered by the child's completion), so POST_ABORT_COMPLETION never appears and the test times out. I reproduced this locally by aborting at 244–254ms. If the abort is even later, the test passed without checking anything.

The failure logs also showed misleading git … failed errors in server tests. Those come from turbo stopping at the first failing package and killing the other test tasks mid-run (Tasks: 10 successful, 20 total).

Approach

  • Keep the toolValueText fallback covered, just without a cyclic object. Production code is untouched.
  • Replace the sleeps in the abort test with gates the test controls. The test aborts only once the parent's second response is known to be in flight, and it releases the child only after the abort. The test is deterministic and also faster: about 50ms instead of about 290ms.
  • Run turbo with --continue so one failing package doesn't hide the results of the others. Turbo still exits non-zero if any task fails.

Deliberately out of scope (diagnosed, but the fix isn't proven yet):

  • server terminal test closing a tab removes it and reports closed (2 failures): on Ubuntu, interactive bash starts child processes while reading its startup files, so an immediate close sees busy. The proposed fix has not been run on Linux yet.
  • pi-dag tests complete admitted topology remains readable beyond 64 KiB / cancel does not turn a human input gate… (4 failures): they fail because the CI runner is overloaded (about 200ms locally vs 0.7–6.8s on CI). This needs profiling rather than a longer timeout.

Changes

  • packages/extension-api/src/toolResultContent.test.ts: the fallback test uses an object whose toJSON throws, plus a BigInt, instead of a cyclic object.
  • packages/pi-subagents/src/extension.test.ts: the detached-run abort test is gated with deferred() instead of sleeps. It now also asserts that the child is still running and that SLOW_ACK never appears right after the abort.
  • package.json: test runs turbo run test --continue.

Screenshots

Not applicable — no frontend changes.

Related issues

None.

Checklist

  • Fast gates pass: bun run lint, bun run typecheck, bun run test
  • E2E suite passes for app-affecting changes (bun run e2e, or bun run e2e:full when touching agent behavior) — not applicable, only tests and the test script changed
  • Before/after screenshots are included for frontend changes, or marked not applicable
  • Relevant SPEC.md / top-level specs updated to reflect any boundary, contract, or behavior change — not applicable, no boundary, contract, or behavior change
  • I have read the Contributing guide and agree to the Code of Conduct

Testing

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

🌐 Website preview

Preview retired when this pull request closed.

@jetbrains-air jetbrains-air Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — ready to merge.

@rsolmano
Rinat S (rsolmano) added this pull request to the merge queue Oct 8, 2026
Merged via the queue into main with commit 20db2b2 Oct 8, 2026
19 checks passed
@rsolmano
Rinat S (rsolmano) deleted the investigate-flaky-unit-tests branch October 8, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant