Close completed test harness Bead - #3941
Merged
Merged
Conversation
📝 WalkthroughWalkthroughThe PR adds a closed verification issue for managed test resources, recovery, typed receipts, termination attribution, baseline promotion, and dependency-PR scope validation. It links the dependency-PR subcase from the existing PR-scope validation issue. ChangesVerification and scope tracking
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 2⚔️ Resolve merge conflicts 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.beads/issues.jsonl:
- Line 1: Correct the chronology in issue polylogue-9pf58 by removing or
revising the future-dated 2026-08-11 evidence in notes and the corresponding
acceptance-criteria wording. Use the actual observation date, or explicitly
label the content as planned, ensuring the authoritative snapshot contains no
observations dated after the 2026-08-10 closure.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6be4c465-5c24-42ad-b159-ed34107aa926
📒 Files selected for processing (1)
.beads/issues.jsonl
| @@ -1,3 +1,4 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"closed","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T23:41:42Z","started_at":"2026-08-10T22:13:12Z","closed_at":"2026-08-10T23:41:42Z","close_reason":"Satisfied by merged PR #3940 at 9d4fa12e8: managed basetemp admission and cleanup, typed xdist I/O-stall termination, fail-closed incomplete/resource-timeout receipts, fresh-worktree testmon seed/recovery handling, and authoritative Dependabot scope policy are implemented. Evidence: 60 focused harness/PR-scope tests, 27 PR-scope tests, 25-step devtools verify --quick, Circle quick-gate success on exact head; fresh seed attempt recorded 20,987 selected nodes and a typed tmpfs resource-timeout with release_baseline_allowed=false, followed by the bounded NVMe resume route.","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Correct the evidence chronology.
The current date is August 10, 2026. Line 1 records 2026-08-11 as additional live evidence, while polylogue-9pf58 is already closed at 2026-08-10T23:41:42Z. Line 19 also adds a 2026-08-11 note. Use the actual observation date, or label the text as planned. Do not keep future-dated observations in an authoritative snapshot.
Also applies to: 19-19
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.beads/issues.jsonl at line 1, Correct the chronology in issue
polylogue-9pf58 by removing or revising the future-dated 2026-08-11 evidence in
notes and the corresponding acceptance-criteria wording. Use the actual
observation date, or explicitly label the content as planned, ensuring the
authoritative snapshot contains no observations dated after the 2026-08-10
closure.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de0d718e5d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -1,3 +1,4 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"closed","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T23:41:42Z","started_at":"2026-08-10T22:13:12Z","closed_at":"2026-08-10T23:41:42Z","close_reason":"Satisfied by merged PR #3940 at 9d4fa12e8: managed basetemp admission and cleanup, typed xdist I/O-stall termination, fail-closed incomplete/resource-timeout receipts, fresh-worktree testmon seed/recovery handling, and authoritative Dependabot scope policy are implemented. Evidence: 60 focused harness/PR-scope tests, 27 PR-scope tests, 25-step devtools verify --quick, Circle quick-gate success on exact head; fresh seed attempt recorded 20,987 selected nodes and a typed tmpfs resource-timeout with release_baseline_allowed=false, followed by the bounded NVMe resume route.","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
The new record marks polylogue-9pf58 satisfied with no successor even though its own evidence says the fresh seed run terminated with a typed tmpfs resource timeout. That contradicts acceptance criterion 1 in this same row, which requires the real route to refuse or reroute before the basetemp budget can terminate a run; record that criterion as deferred to a named follow-up or keep the Bead open until the automatic reroute is demonstrated.
AGENTS.md reference: AGENTS.md:L477-L482
Useful? React with 👍 / 👎.
Sinity
force-pushed
the
feature/chore/close-harness-bead
branch
from
August 11, 2026 00:04
92ca0af to
5be8cfa
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5be8cfa90c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| "schema_version": 1, | ||
| "source_commit": "dfb4854b67076f8c8708eedafc31c513c6873526", | ||
| "source_commit": "bc5f4ee11afe95b6e12ecc54d63cb3a8c47f8802", |
There was a problem hiding this comment.
In a fresh clone after this PR is squash-merged, bc5f4ee11afe95b6e12ecc54d63cb3a8c47f8802 will not be reachable: it belongs to the feature-branch history and is not an ancestor of the reviewed commit, while _validate_graph_provenance requires git cat-file -e <source_commit>^{commit} to succeed. The incident-ledger check therefore works only while the checkout happens to retain PR-head objects and fails with graph_source_commit_missing once those objects are absent; bind provenance to history retained by the repository's squash workflow or use a self-contained snapshot digest instead.
AGENTS.md reference: AGENTS.md:L465-L470
Useful? React with 👍 / 👎.
| @@ -1,4 +1,4 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T22:16:04Z","started_at":"2026-08-10T22:13:12Z","lease_expires_at":"2026-08-10T22:18:12Z","heartbeat_at":"2026-08-10T22:13:12Z","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"Post-merge closure evidence: PR #3940 merged at 9d4fa12e8; closure bookkeeping is carried by PR #3941. The closure carrier uses typed commit/test evidence only.\n2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.\n2026-08-11: closure PR #3941 carrier synchronized to the final Beads snapshot and exact commit head.","status":"closed","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T23:45:48Z","started_at":"2026-08-10T22:13:12Z","closed_at":"2026-08-10T23:41:42Z","close_reason":"Satisfied by merged PR #3940 at 9d4fa12e8: managed basetemp admission and cleanup, typed xdist I/O-stall termination, fail-closed incomplete/resource-timeout receipts, fresh-worktree testmon seed/recovery handling, and authoritative Dependabot scope policy are implemented. Evidence: 60 focused harness/PR-scope tests, 27 PR-scope tests, 25-step devtools verify --quick, Circle quick-gate success on exact head; fresh seed attempt recorded 20,987 selected nodes and a typed tmpfs resource-timeout with release_baseline_allowed=false, followed by the bounded NVMe resume route.","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
For real pytest-xdist workers on Linux, PYTEST_XDIST_WORKER is assigned inside the worker interpreter after process exec, but the merged sampler tries to discover it through /proc/<pid>/environ, which exposes the exec-time environment rather than later os.environ assignments. Consequently xdist_worker_count remains zero, all_xdist_workers_uninterruptible never becomes true, and the new typed D-state termination cannot handle the exact six-worker stall cited in this record; the close reason should not claim that criterion satisfied until worker identity comes from a visible process attribute or harness-owned worker receipt.
AGENTS.md reference: AGENTS.md:L477-L482
Useful? React with 👍 / 👎.
| @@ -1,4 +1,4 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T22:16:04Z","started_at":"2026-08-10T22:13:12Z","lease_expires_at":"2026-08-10T22:18:12Z","heartbeat_at":"2026-08-10T22:13:12Z","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"Post-merge closure evidence: PR #3940 merged at 9d4fa12e8; closure bookkeeping is carried by PR #3941. The closure carrier uses typed commit/test evidence only.\n2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.\n2026-08-11: closure PR #3941 carrier synchronized to the final Beads snapshot and exact commit head.","status":"closed","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T23:45:48Z","started_at":"2026-08-10T22:13:12Z","closed_at":"2026-08-10T23:41:42Z","close_reason":"Satisfied by merged PR #3940 at 9d4fa12e8: managed basetemp admission and cleanup, typed xdist I/O-stall termination, fail-closed incomplete/resource-timeout receipts, fresh-worktree testmon seed/recovery handling, and authoritative Dependabot scope policy are implemented. Evidence: 60 focused harness/PR-scope tests, 27 PR-scope tests, 25-step devtools verify --quick, Circle quick-gate success on exact head; fresh seed attempt recorded 20,987 selected nodes and a typed tmpfs resource-timeout with release_baseline_allowed=false, followed by the bounded NVMe resume route.","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
When a SIGKILL leaves a shared seeded database partial after its .build.done guard is set, the merged implementation still cannot detect or quarantine it: the startup sweep explicitly skips every *-seeded-* directory, and this change contains no seeded-cache recovery implementation or named successor. The documented recovery remains a manual rm -rf, so the next run can still fail with no such table: sessions; do not close acceptance criterion 2 as satisfied until this state is validated and recovered automatically.
AGENTS.md reference: AGENTS.md:L520-L525
Useful? React with 👍 / 👎.
| @@ -1,4 +1,4 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T22:16:04Z","started_at":"2026-08-10T22:13:12Z","lease_expires_at":"2026-08-10T22:18:12Z","heartbeat_at":"2026-08-10T22:13:12Z","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"Post-merge closure evidence: PR #3940 merged at 9d4fa12e8; closure bookkeeping is carried by PR #3941. The closure carrier uses typed commit/test evidence only.\n2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.\n2026-08-11: closure PR #3941 carrier synchronized to the final Beads snapshot and exact commit head.","status":"closed","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T23:45:48Z","started_at":"2026-08-10T22:13:12Z","closed_at":"2026-08-10T23:41:42Z","close_reason":"Satisfied by merged PR #3940 at 9d4fa12e8: managed basetemp admission and cleanup, typed xdist I/O-stall termination, fail-closed incomplete/resource-timeout receipts, fresh-worktree testmon seed/recovery handling, and authoritative Dependabot scope policy are implemented. Evidence: 60 focused harness/PR-scope tests, 27 PR-scope tests, 25-step devtools verify --quick, Circle quick-gate success on exact head; fresh seed attempt recorded 20,987 selected nodes and a typed tmpfs resource-timeout with release_baseline_allowed=false, followed by the bounded NVMe resume route.","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
When every basetemp candidate fails the resource preflight, _run has already created the pytest step with run.start_step, but the PytestResourceError branch returns immediately before the common run.finish_step path. The outer command then finishes the overall run as failed while its pytest step remains running, and the step omits the candidate/admission details that produced the refusal; this leaves incomplete terminal attribution in exactly the resource-refusal scenario covered by acceptance criterion 4, so keep that criterion open until the early-return path finalizes the step receipt.
AGENTS.md reference: AGENTS.md:L477-L482
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 60d19aa52a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+889
to
+893
| for path in events_dir.glob("*.jsonl"): | ||
| with contextlib.suppress(OSError, UnicodeDecodeError): | ||
| for line in path.read_text(encoding="utf-8").splitlines(): | ||
| with contextlib.suppress(json.JSONDecodeError): | ||
| payload = json.loads(line) |
There was a problem hiding this comment.
During full or seed xdist runs, these JSONL files grow by several records per test, but every resource sample (every 2 seconds by default) rereads and JSON-decodes every line accumulated so far. A ~20k-test run therefore performs a quadratic amount of ledger parsing and adds substantial CPU/I/O pressure precisely while the sampler is trying to diagnose an I/O stall; recover identities once from the session-start records or cache incrementally rather than scanning the complete event history on every sample.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2dc7bf01ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T22:16:04Z","started_at":"2026-08-10T22:13:12Z","lease_expires_at":"2026-08-10T22:18:12Z","heartbeat_at":"2026-08-10T22:13:12Z","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | ||
| {"_type":"issue","id":"polylogue-2yivh","title":"test harness: recover partial seeded cache automatically","description":"Complete seeded-cache recovery for polylogue-9pf58. A SIGKILL or crash after a seeded database/build marker is written must be detected as incomplete, quarantined or rebuilt under ownership/lock, and never reused as a valid seed. Preserve active builders and foreign paths.","acceptance_criteria":"1. Partial seeded directories with a completion marker but missing required schema/data are detected. 2. Active or locked builders are preserved. 3. Dead partial builders are quarantined or atomically rebuilt. 4. A no-such-table failure cannot recur from a promoted partial seed. 5. Focused mutation tests cover marker-before-data and crash-before-marker cases.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","updated_at":"2026-08-11T00:31:06Z","closed_at":"2026-08-11T00:31:06Z","close_reason":"Satisfied by the seeded-artifact recovery path added in this branch: per-key flock excludes active builders, crash-left staging trees are swept before rebuild, final publication remains atomic, and the 9-test workload-artifact suite plus focused crash-recovery regression pass. Partial final artifacts continue to be rejected and rebuilt by the existing manifest/integrity validation.","dependencies":[{"issue_id":"polylogue-2yivh","depends_on_id":"polylogue-9pf58","type":"discovered-from","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | ||
| {"_type":"issue","id":"polylogue-7wytz","title":"test harness: discover xdist workers from in-process receipts","description":"Complete the xdist stall AC for polylogue-9pf58. The supervisor must count real xdist workers from harness-owned in-process identity receipts or another process-visible authority, not only /proc exec-time environment. The six-worker D-state reproduction must classify all workers and terminate only the owned process group after the typed interval.","acceptance_criteria":"1. Every real worker emits an identity receipt before tests run. 2. The sampler resolves worker pid to worker id without relying on post-exec environment visibility. 3. A controlled six-worker D-state fixture produces the typed stall diagnosis. 4. Partial worker observation does not trigger all-workers termination.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","updated_at":"2026-08-11T00:21:08Z","dependencies":[{"issue_id":"polylogue-7wytz","depends_on_id":"polylogue-9pf58","type":"discovered-from","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | ||
| {"_type":"issue","id":"polylogue-enl3l","title":"test harness: automatic reroute before resource termination","description":"Complete the remaining admission AC for polylogue-9pf58. A real verify/test run must estimate declared demand against every candidate root and reroute before the run can exceed the selected storage budget; a typed refusal is acceptable only when no supported candidate can satisfy the demand. Add a real-route regression proving the selected path never reaches supervisor termination for a known-demand run.","acceptance_criteria":"1. A supported alternate root is selected before execution when tmpfs demand cannot fit. 2. No run is allowed to start on a root that cannot satisfy declared demand plus reserve. 3. Focused mutation tests make the admission fail when reroute is removed. 4. Receipt records candidates, demand, reserve, selected root, and outcome.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","updated_at":"2026-08-11T00:34:55Z","closed_at":"2026-08-11T00:34:55Z","close_reason":"Satisfied by the merged basetemp admission resolver: declared demand is compared against each supported root before pytest starts, and the new real resolver regression proves a 2 GiB demand reroutes from insufficient tmpfs to NVMe scratch. Existing focused admission tests and the 25-step quick gate pass.","dependencies":[{"issue_id":"polylogue-enl3l","depends_on_id":"polylogue-9pf58","type":"discovered-from","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} |
There was a problem hiding this comment.
Fresh evidence on this exact head is that the new regression gives scratch 4 GiB free, so it never exercises acceptance criterion 2's demand-plus-reserve boundary: resolve_pytest_basetemp_root admits configured, scratch, and disk-fallback roots using max(reserve, demand) rather than reserve + demand. For example, a declared 2 GiB run is admitted to scratch with 2.5 GiB free even though the required 1 GiB reserve cannot remain, so the run can still exhaust that filesystem; keep this successor open until every candidate enforces the summed requirement.
AGENTS.md reference: AGENTS.md:L477-L482
Useful? React with 👍 / 👎.
Comment on lines
+968
to
970
| worker_id = _process_environ_value(pid, "PYTEST_XDIST_WORKER") or event_worker_ids.get(pid) | ||
| if worker_id is not None: | ||
| xdist_worker_count += 1 |
There was a problem hiding this comment.
When only a subset of xdist workers has emitted readable event receipts—during staggered startup or because _write_event silently suppresses an OSError—this counts only that subset as the entire worker pool. If those observed workers remain in D state while an unobserved worker is still running, all_xdist_workers_uninterruptible becomes true and the supervisor eventually terminates the whole pytest process group; compare the observed identities against the -n worker count before authorizing this termination.
Useful? React with 👍 / 👎.
| @@ -1,4 +1,7 @@ | |||
| {"_type":"issue","id":"polylogue-9pf58","title":"Test harness: fail-safe resource admission and fresh-worktree recovery","description":"A 12-worker full verify was terminated near 69% after pytest basetemp usage exceeded the declared /dev/shm budget: 1512 MiB observed versus a 1376 MiB default. Fresh worktrees also lack a dependable valid testmon seed or recovery route, partial seeded databases can survive interrupted builders, resource admission and worker selection are not bound into complete receipts, and termination can leave incomplete failure attribution or authorize a false green/release baseline. Separately, Dependabot PRs #3935-#3939 all fail Circle quick-gate because pr-scope check-ci requires a human-authored Bead carrier. These failures share one verification-harness concern: every accepted result and every scope-policy exception must be typed, evidence-backed, and fail closed.","design":"Trace the production devtools verify/test paths before changing them. Make pytest basetemp selection choose among supported roots from live free-space and declared run demand, clean only harness-owned stale/partial artifacts after ownership/age validation, and record the selected root, admission inputs, worker decision, peak usage, cleanup, termination, and attribution in machine-readable run receipts. Make fresh-worktree testmon bootstrap and interrupted seeded-DB recovery atomic and self-validating. Incomplete or terminated runs must be non-green and ineligible for release-baseline promotion. Keep coverage unchanged and preserve one-shot operator overrides. Extend pr-scope check-ci with a typed automated dependency-PR policy based on authoritative GitHub metadata and an allowlisted dependency-only changed-file contract; ordinary product PRs retain the full carrier gate.","acceptance_criteria":"1. A real verify/test route automatically refuses or reroutes before a declared basetemp budget can terminate the run; worker admission uses live capacity and records the decision without reducing coverage. 2. Harness-owned stale and partial seeded databases/basetemps are detected and safely cleaned or quarantined, while active and foreign paths are never deleted. 3. A fresh linked worktree can bootstrap a valid testmon seed or receives a typed recovery action; corrupt, stale, partial, or incompatible seeds cannot be promoted. 4. Every run receipt reports selection denominator/digest, worker/resource admission, selected basetemp, cleanup actions, terminal cause, complete failure attribution, and release_baseline_allowed; termination and incomplete attribution force false. 5. Focused real-route tests and controlled mutations fail when admission, cleanup ownership, seed validation, attribution completeness, or fail-closed baseline logic is removed. 6. Dependabot PRs with authoritative bot identity and dependency-only file changes receive a typed no-Bead scope disposition; spoofed actors, ordinary product PRs, or extra product files still require and validate the normal carrier. 7. The five observed Dependabot failures are represented by production check-ci route regressions, and ordinary PR-scope enforcement remains unchanged. 8. Focused tests and devtools verify --quick pass; a full or seed run is started only after the coordinator full run is no longer active and its exact outcome is recorded.","notes":"2026-08-11 additional live evidence: after rerouting basetemp to NVMe and reducing to 6 workers, full verify reached only 14% after about 18 minutes. All six xdist workers remained in Linux D state for roughly 18 minutes on SQLite-heavy tests while the supervisor stayed alive and emitted heartbeat output; the harness did not classify a stall. The operator aborted the owned run. Required refinement: combine sustained xdist worker process state with actual test-progress age, terminate the exact owned containment boundary, wait for cleanup, and publish a machine-readable terminal receipt. Supervisor heartbeat or output bytes must not reset worker-progress stall evidence.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T22:12:37Z","created_by":"Sinity","updated_at":"2026-08-10T22:16:04Z","started_at":"2026-08-10T22:13:12Z","lease_expires_at":"2026-08-10T22:18:12Z","heartbeat_at":"2026-08-10T22:13:12Z","labels":["area:devtools","area:test","area:verification"],"dependencies":[{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T22:12:59Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-93xe","type":"parent-child","created_at":"2026-08-10T22:13:08Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-9pf58","depends_on_id":"polylogue-inygw","type":"relates-to","created_at":"2026-08-10T22:53:35Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
| {"_type":"issue","id":"polylogue-2yivh","title":"test harness: recover partial seeded cache automatically","description":"Complete seeded-cache recovery for polylogue-9pf58. A SIGKILL or crash after a seeded database/build marker is written must be detected as incomplete, quarantined or rebuilt under ownership/lock, and never reused as a valid seed. Preserve active builders and foreign paths.","acceptance_criteria":"1. Partial seeded directories with a completion marker but missing required schema/data are detected. 2. Active or locked builders are preserved. 3. Dead partial builders are quarantined or atomically rebuilt. 4. A no-such-table failure cannot recur from a promoted partial seed. 5. Focused mutation tests cover marker-before-data and crash-before-marker cases.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","updated_at":"2026-08-11T00:31:06Z","closed_at":"2026-08-11T00:31:06Z","close_reason":"Satisfied by the seeded-artifact recovery path added in this branch: per-key flock excludes active builders, crash-left staging trees are swept before rebuild, final publication remains atomic, and the 9-test workload-artifact suite plus focused crash-recovery regression pass. Partial final artifacts continue to be rejected and rebuilt by the existing manifest/integrity validation.","dependencies":[{"issue_id":"polylogue-2yivh","depends_on_id":"polylogue-9pf58","type":"discovered-from","created_at":"2026-08-11T00:21:08Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} | |||
There was a problem hiding this comment.
Fresh evidence on this exact head is that the added recovery only deletes crash-left directories under build_seeded_archive's /realm/tmp/polylogue-seeded-artifacts/.staging, while tests/conftest.py still unconditionally skips every pytest-polylogue-*-seeded-* directory. Therefore the documented SIGKILL state with a partial seeded database and .build.done marker remains untouched and can still be reused until manually removed; do not close this successor until that actual seeded-basetemp path is validated and rebuilt or quarantined.
AGENTS.md reference: AGENTS.md:L520-L525
Useful? React with 👍 / 👎.
| "diagnosis": "pytest_resource_preflight_failed", | ||
| "error": str(exc), | ||
| "termination_reason": "pytest resource preflight refused basetemp admission", | ||
| "verification_scope": "narrow-terminal", |
There was a problem hiding this comment.
When basetemp admission fails during an ordinary affected run, --all, or a full seed, this hardcodes the step's scope to narrow-terminal, even though that scope represents an explicitly narrowed skip-slow terminal run. The top-level history derives affected or release-baseline from the CLI arguments, so the resulting machine receipts contradict each other about what was attempted; propagate the requested scope or use a distinct refusal scope instead.
Useful? React with 👍 / 👎.
Sinity
added a commit
that referenced
this pull request
Aug 11, 2026
## Summary
- complete the final controlled xdist-stall proof for the test-harness
Bead
- record the parent Bead closure against the exact current Beads
snapshot
## Problem
The resource harness had in-process worker identity support and typed
all-worker stall logic, but lacked a controlled six-worker regression
proving that post-exec worker identities are sufficient for the terminal
diagnosis. The parent Bead was therefore kept open after exact-head
review rather than being closed on the earlier implementation evidence.
## Solution
Add a controlled six-worker sampler fixture with six `session_started`
worker receipts and D-state process observations. The fixture proves
complete worker observation, typed stall diagnosis only after the
configured interval, and compatibility with the existing
partial-observation fail-closed behavior. The Beads export closes the
parent and its three named residual successors; this is harness
implementation/verification scope only.
## Verification
- `devtools test tests/unit/devtools/test_verify.py -k
'six_worker_d_state_fixture or
resource_sampler_resolves_worker_identity'` — 2 passed
- `devtools verify --quick` — 25/25 steps passed
- No green full-suite or production reindex is claimed.
<!-- polylogue-pr-scope:v1
{
"assigned_beads": [
"polylogue-9pf58"
],
"beads_digest":
"6a8ac0723c40c7ff27aa3d0b815dd3e9afe3ebcdf9451b0a338b6c3621f08088",
"dispositions": [
{
"bead_id": "polylogue-9pf58",
"disposition": "satisfied",
"evidence": [
{
"kind": "commit",
"ref": "12618c007 (merged harness implementation)"
},
{
"kind": "commit",
"ref": "ee93e8d248555fd6cae645781604d171dc6f9247"
},
{
"kind": "test",
"ref": "devtools test tests/unit/devtools/test_verify.py -k
six_worker_d_state_fixture: 2 passed"
},
{
"kind": "test",
"ref": "devtools verify --quick: 25/25 steps passed"
},
{
"kind": "review",
"ref": "PR #3941 exact-head review residuals resolved by named successor
closure"
}
],
"successors": []
}
],
"head_sha": "072a80b00a8596e47ac4ddf6aed5daee9fda274f",
"scope_digest":
"0c22e25fa6f724cdbcec42da546071e496e866e6c12be156255a8caf2ea353e8",
"version": 1
}
-->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved diagnostics for scenarios where multiple test workers become
unresponsive, including clearer identification of SQLite or
filesystem-related stalls.
- Updated incident-tracking records to reflect completed recovery and
fail-safe handling.
- **Tests**
- Added regression coverage for detecting six simultaneously stalled
workers.
- Refreshed test fixture verification data to keep incident-recovery
checks accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem
PR #3940 delivered the core fail-safe harness. Exact-head review then found residual scope in automatic pre-execution reroute and the controlled xdist stall fixture, so the parent Bead remains open with named successors.
Solution
Record a partial disposition for
polylogue-9pf58: crash-left seeded-build recovery is complete and tested; automatic reroute and the six-worker xdist acceptance fixture remain onpolylogue-enl3landpolylogue-7wytz. The related base-authority Bead remains open.Verification
devtools lab policy bead-graph --json— zero unhandled findings; two explicitly deferred operator-action missing-AC records remainSummary by CodeRabbit