perf(harness): bound the load_sensitive lane and fix its startup race

CONTRIBUTING.md

@@ -310,8 +310,8 @@ complete correctness corpus. The pytest step covers unit, property, fuzz, and
 integration tests while excluding the separately operated `tests/benchmarks`
 performance surface. It uses
 `--testmon-forceselect` for incremental selection, with one parallel `not
-load_sensitive` lane and one serial `load_sensitive` lane over the same native
-environment. `tui` is a category marker and remains parallel unless a test is
+load_sensitive` lane and one bounded-concurrency `load_sensitive` lane over the
+same native environment. `tui` is a category marker and remains parallel unless a test is
 also explicitly `load_sensitive`. Use
 There is no manual seed or repair command, and no separate full-corpus flag:
 every plain run is scoped to the complete corpus, executing what changed and

TESTING.md

@@ -84,10 +84,22 @@ concurrency when memory pressure is elevated.
 
 Every native run has exactly two semantic lanes over one environment and one
 database: a parallel lane for tests not marked `load_sensitive`, followed by a
-serial lane for the load-sensitive set. Ordinary test failures in the parallel
-lane do not suppress the serial lane. Typed collection, containment, resource,
+bounded lane for the load-sensitive set. Ordinary test failures in the parallel
+lane do not suppress the bounded lane. Typed collection, containment, resource,
 or timeout failures do.
 
+The bounded lane is not strictly serial. `load_sensitive` marks a test whose
+wall-clock deadlines the *parallel* lane's full worker count starves, which
+bounds how much concurrency the lane may use — it does not establish that the
+members contend with each other. Measured on the 7-test daemon-resilience
+corpus (2026-08-19): 71.95s at one process, 35.08s at four workers, and a hard
+cliff above that (5 workers 107.76s with one starved SIGTERM deadline, 7 workers
+95.20s with two). The lane therefore runs at
+`devtools.verify.SERIAL_LANE_MAX_WORKERS` (4) under `--dist=loadgroup`, with
+members assigned to `xdist_group` bins packed longest-first so the makespan is
+bounded by the largest bin rather than by the order xdist happens to dispatch
+in. Raising the cap requires repeating that measurement.
+
 The lane boundary is evidence-based. On 2026-08-13 the complete correctness
 corpus collected 20,447 nodes in 35.06s; only 17 were `load_sensitive`, while
 16 were tagged `tui` with no overlap. A managed serial run retained 93.09s of

devtools/pytest_supervisor.py

@@ -911,20 +911,82 @@ def _restore_owner_write(root: Path) -> None:
             candidate.chmod(candidate.stat().st_mode | stat.S_IWUSR)
 
 
+def force_rmtree(path: Path, *, ignore_errors: bool = False) -> None:
+    """Remove a harness-owned tree, repairing the permissions the harness revoked.
+
+    The shared entry point for every reclaimer, so the repair above has exactly
+    one implementation. `verify_runs.cleanup_managed_pytest_basetemp` and the
+    stale-basetemp sweep both need it: the read-only artifact trees that caused
+    polylogue-b9yw7 defeat a plain rmtree wherever it is called from, and the
+    supervisor's exit pass was only one of those places.
+
+    Repair is the recovery path, not the common one -- an ordinary tree is
+    removed by the first call and never walked twice -- and it only ever runs
+    against a tree the caller has already established is harness-owned.
+    """
+    try:
+        shutil.rmtree(path)
+        return
+    except FileNotFoundError:
+        return
+    except OSError:
+        pass
+    _restore_owner_write(path)
+    try:
+        shutil.rmtree(path)
+    except FileNotFoundError:
+        return
+    except OSError:
+        if not ignore_errors:
+            raise
+
+
 def cleanup_managed_tmpfs_path(path: Path | None) -> bool:
     """Remove only a harness-owned direct child of the system tmpfs."""
-    if path is None or not path.name.startswith("pytest-polylogue-"):
-        return False
+    return describe_managed_tmpfs_cleanup(path)[0]
+
+
+def describe_managed_tmpfs_cleanup(path: Path | None) -> tuple[bool, str, list[str]]:
+    """Reclaim a harness-owned tmpfs basetemp and say what happened.
+
+    Returns ``(complete, reason, residual)``. The reason and residual sample
+    exist because a bare ``False`` here is load-bearing and opaque: it becomes
+    the run's ``cleanup.complete``, which `_release_baseline_allowed` requires
+    to be True, so an otherwise-green complete-corpus run silently loses release
+    authority over a scratch directory that failed to unlink. Observed
+    2026-08-19 (run 20260819T003921Z, exit 0, no signals, controller group
+    quiescent) with nothing in the receipt to say why (polylogue-b9yw7).
+
+    The cause found for that receipt was a permission bit, not a race:
+    published seeded-archive artifacts chmod their DIRECTORIES non-writable, and
+    entries cannot be unlinked from a directory without its write bit, so the
+    tree was simply unremovable. `_restore_owner_write` above repairs exactly
+    that. The residual sample stays because it is what makes the NEXT such
+    failure legible in one receipt rather than a night of bisecting: it names
+    the surviving paths, which is how this one was ultimately identified.
+    """
+    if path is None:
+        return False, "no managed tmpfs path for this run", []
+    if not path.name.startswith("pytest-polylogue-"):
+        return False, f"not a harness-owned basetemp name: {path.name}", []
     try:
         if path.parent.resolve() != Path("/dev/shm").resolve():
-            return False
-    except OSError:
-        return False
-    shutil.rmtree(path, ignore_errors=True)
-    if path.exists():
-        _restore_owner_write(path)
-        shutil.rmtree(path, ignore_errors=True)
-    return not path.exists()
+            return False, f"not a direct child of /dev/shm: {path.parent}", []
+    except OSError as exc:
+        return False, f"could not resolve parent: {exc}", []
+    if not path.exists():
+        return True, "already reclaimed before this pass", []
+    force_rmtree(path, ignore_errors=True)
+    if not path.exists():
+        return True, "reclaimed by this pass", []
+    residual: list[str] = []
+    with contextlib.suppress(OSError):
+        for index, entry in enumerate(path.rglob("*")):
+            if index >= 20:
+                residual.append("... (truncated)")
+                break
+            residual.append(str(entry.relative_to(path)))
+    return False, "tree survived rmtree", residual
 
 
 def main(argv: Sequence[str] | None = None) -> int:
@@ -945,14 +1007,23 @@ def main(argv: Sequence[str] | None = None) -> int:
     finally:
         receipt = read_receipt(args.receipt)
         receipt_quiescent = receipt is not None and receipt.get("controller_group_alive") is False
-        cleanup_complete = cleanup_managed_tmpfs_path(args.cleanup_path) if receipt_quiescent else False
+        if receipt_quiescent:
+            cleanup_complete, cleanup_reason, cleanup_residual = describe_managed_tmpfs_cleanup(args.cleanup_path)
+        else:
+            cleanup_complete, cleanup_reason, cleanup_residual = (
+                False,
+                "controller process group still alive at supervisor exit",
+                [],
+            )
         if args.cleanup_path is not None:
             with contextlib.suppress(OSError):
                 update_receipt(
                     args.receipt,
                     {
                         "tmpfs_cleanup_path": str(args.cleanup_path),
                         "tmpfs_cleanup_complete": cleanup_complete,
+                        "tmpfs_cleanup_reason": cleanup_reason,
+                        "tmpfs_cleanup_residual": cleanup_residual,
                     },
                 )
 

devtools/verify.py

@@ -41,7 +41,7 @@
 from dataclasses import dataclass
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any
+from typing import Any, Final
 
 from devtools.checkout_guard import (
     CheckoutImportMismatchError,
@@ -119,6 +119,7 @@
     pytest_tmpfs_budget_exceeded,
     pytest_tmpfs_budget_kb,
     start_checkout_mutation_monitor,
+    sweep_stale_managed_basetemps,
     utc_now,
     worktree_fingerprint,
     xdist_uninterruptible_stall_reason,
@@ -2109,6 +2110,7 @@ def _run_step(
     runtime_policy = None
     pytest_concurrency = 0
     basetemp_cleanup: Path | None = None
+    swept: list[Path] = []
     if is_pytest or has_managed_pytest_child:
         try:
             if has_managed_pytest_child:
@@ -2120,6 +2122,13 @@ def _run_step(
                 env = force_managed_pytest_scratch(env)
             if is_pytest:
                 pytest_concurrency = _pytest_command_concurrency(cmd, env=env)
+            # Reclaim prior runs' leaked tmpfs trees BEFORE admission, not only
+            # at exit. Cleanup used to be purely trailing, so a tree that failed
+            # to unlink stayed resident until someone noticed -- 16 of them and
+            # two blocked merges on 2026-08-19 (polylogue-b9yw7). Sweeping here
+            # also feeds the decision immediately below: the space it returns is
+            # headroom the basetemp admission policy can then admit against.
+            swept = sweep_stale_managed_basetemps()
             env, runtime_policy = apply_managed_pytest_runtime_policy(
                 env,
                 worker_count=pytest_concurrency,
@@ -2241,6 +2250,7 @@ def _run_step(
         metadata["postmortem_path"] = str(CURRENT_POSTMORTEM_PATH)
         metadata["containment_path"] = str(PYTEST_CONTAINMENT_PATH)
         metadata["basetemp_cleanup"] = str(basetemp_cleanup) if basetemp_cleanup is not None else None
+        metadata["basetemp_swept"] = [str(path) for path in swept]
         junit_paths = [
             str(path) for path in _pytest_artifact_paths(cmd) if path.suffix == ".xml" or path.name.endswith(".xml")
         ]
@@ -2597,6 +2607,7 @@ def _native_pytest_steps(
     testmon_mode: str,
     testmon_environment: str,
     parallel_worker_args: Sequence[str],
+    serial_worker_args: Sequence[str],
 ) -> list[tuple[str, list[str]]]:
     pytest_cmd = [
         sys.executable,
@@ -2655,8 +2666,7 @@ def _serial_report_arg(arg: str) -> str:
             *native_args,
             "-p",
             "no:randomly",
-            "-n",
-            "0",
+            *serial_worker_args,
         ]
     )
     return [
@@ -2674,10 +2684,15 @@ def _native_pytest_command_is_closed_world(label: str, cmd: Sequence[str]) -> bo
     worker_request = pytest_command_worker_request(cmd)
     if len(environment_args) != 1 or worker_request is None or not worker_request.isdigit():
         return False
+    # Only the labelled lane's command is compared below, so reconstructing both
+    # lanes from the observed worker request is exact for the lane under test and
+    # irrelevant for its sibling.
+    observed_worker_args = ("--dist=loadgroup", "-n", worker_request)
     expected_steps = _native_pytest_steps(
         testmon_mode=match.group(2),
         testmon_environment=environment_args[0].removeprefix("--testmon-env="),
-        parallel_worker_args=("--dist=loadgroup", "-n", worker_request),
+        parallel_worker_args=observed_worker_args,
+        serial_worker_args=observed_worker_args,
     )
     expected = dict(expected_steps).get(label)
     return expected is not None and list(cmd) == expected
@@ -2755,6 +2770,7 @@ def build_verify_steps(
                 testmon_mode=testmon_mode,
                 testmon_environment=testmon_environment,
                 parallel_worker_args=_pytest_worker_args(),
+                serial_worker_args=_pytest_worker_args(maximum=SERIAL_LANE_MAX_WORKERS),
             )
         )
 
@@ -2865,6 +2881,27 @@ def _atomic_write_json(path: Path, payload: Mapping[str, Any]) -> None:
     temporary.replace(path)
 
 
+#: Worker ceiling for the `load_sensitive` lane.
+#:
+#: The lane exists because these tests drive real daemon subprocesses under
+#: wall-clock deadlines, and the parallel lane's full worker count starves
+#: interpreter startup badly enough to flake them. It does NOT follow that the
+#: lane must be strictly serial: measured on the 7-test corpus (2026-08-19,
+#: tests/integration/test_daemon_resilience.py, same host and containment),
+#:
+#:   -n 0  71.95s  green
+#:   -n 4  35.08s  green
+#:   -n 5 107.76s  1 failed (SIGTERM deadline starved)
+#:   -n 7  95.20s  2 failed (both SIGTERM tests, 90s subprocess timeout blown)
+#:
+#: so 4 is the last concurrency the deadline-sensitive members survive, with a
+#: clear cliff immediately above it. Raising this needs the same measurement,
+#: repeated, not an assumption that a faster host has more headroom -- the
+#: failures are starvation of an idle-scheduled containment slice, not a
+#: shortage of cores.
+SERIAL_LANE_MAX_WORKERS: Final = 4
+
+
 def _pytest_worker_args(*, maximum: int | None = None) -> list[str]:
     """Return the managed worker count, optionally capped for a bounded lane."""
     workers = adaptive_pytest_worker_count(os.environ)

devtools/verify_runs.py

@@ -30,6 +30,7 @@
 import watchfiles
 
 from devtools.cloud_sentinels import CLOUD_SENTINELS, running_in_cloud_sandbox
+from devtools.pytest_supervisor import force_rmtree
 from devtools.testmon_bootstrap import canonical_test_nodeid
 from polylogue.core.metrics import read_cgroup_memory_headroom_bytes
 
@@ -1237,6 +1238,12 @@ def aggregate_pytest_statistics(
             "complete": True
             if isinstance(parent_cleanup, str) and parent_cleanup
             else containment.get("tmpfs_cleanup_complete"),
+            # Carried so an incomplete cleanup explains itself. `complete` gates
+            # release-baseline authority, and until 2026-08-19 a False here left
+            # no evidence of which reclaimer failed or what survived
+            # (polylogue-b9yw7).
+            "reason": containment.get("tmpfs_cleanup_reason"),
+            "residual": containment.get("tmpfs_cleanup_residual"),
             "termination_reason": containment.get("termination_reason"),
             "escalated_to_sigkill": containment.get("escalated_to_sigkill"),
             "exit_code": containment.get("exit_code", (step_result or {}).get("exit")),
@@ -2807,7 +2814,10 @@ def cleanup_managed_pytest_basetemp(*, root: Path, run_id: str, env: dict[str, s
             return None
         with contextlib.suppress(OSError):
             if basetemp.exists():
-                shutil.rmtree(basetemp)
+                # Permission-repairing: a published seeded-archive cache under
+                # this tree is deliberately read-only, directories included, and
+                # a plain rmtree cannot unlink through it (polylogue-b9yw7).
+                force_rmtree(basetemp)
                 if not basetemp.exists():
                     clear_managed_pytest_basetemp_claim(basetemp)
                     return basetemp
@@ -2817,6 +2827,49 @@ def cleanup_managed_pytest_basetemp(*, root: Path, run_id: str, env: dict[str, s
     return None
 
 
+def sweep_stale_managed_basetemps(*, limit: int = 64) -> list[Path]:
+    """Reclaim harness-owned tmpfs basetemps left behind by finished runs.
+
+    Cleanup used to be purely trailing: each run removed its own tree at exit,
+    so anything that failed to unlink stayed until someone noticed. On
+    2026-08-19 that was 16 leaked trees and two blocked merges, because the
+    removal itself could not succeed (see `force_rmtree`). Repairing removal
+    without also reclaiming the existing debt would leave those trees resident
+    on a 16 GiB tmpfs indefinitely, so the harness now sweeps at start as well
+    as at exit -- eager and self-healing rather than trailing only.
+
+    Ownership is the same test the exit path applies, and it is conservative in
+    the same direction: a tree is reclaimed only when its claim lock is free
+    (nothing else is using it) AND its recorded owner is positively dead. An
+    unknown or live owner is left alone. The shared `-seeded-` corpus cache is
+    never a candidate; it is a deliberate cross-run artifact, not run debt.
+    """
+    reclaimed: list[Path] = []
+    shm = Path("/dev/shm")
+    try:
+        candidates = sorted(shm.glob("pytest-polylogue-*"))
+    except OSError:
+        return reclaimed
+    for candidate in candidates[:limit]:
+        if not candidate.is_dir() or "-seeded-" in candidate.name:
+            continue
+        claim_lock = _try_acquire_pytest_basetemp_claim_lock(candidate)
+        if claim_lock is None:
+            continue
+        try:
+            if managed_pytest_basetemp_owner_alive(candidate) is not False:
+                continue
+            with contextlib.suppress(OSError):
+                force_rmtree(candidate)
+                if not candidate.exists():
+                    clear_managed_pytest_basetemp_claim(candidate)
+                    reclaimed.append(candidate)
+        finally:
+            with contextlib.suppress(OSError):
+                claim_lock.close()
+    return reclaimed
+
+
 def _pytest_event_worker_ids(events_dir: Path | None) -> dict[int, str]:
     """Recover xdist worker identities emitted after process exec.
 

docs/plans/oracle-integrity-baseline.json

@@ -2,7 +2,7 @@
   "entries": [
     {
       "code": "ambient_path_call",
-      "detail": "line 136: resolves an ambient location via Path.home()",
+      "detail": "line 165: resolves an ambient location via Path.home()",
       "path": "tests/integration/test_daemon_resilience.py"
     },
     {
@@ -82,7 +82,7 @@
     },
     {
       "code": "ambient_path_literal",
-      "detail": "line 1456: reads ambient path '~/.codex'",
+      "detail": "line 1464: reads ambient path '~/.codex'",
       "path": "tests/unit/daemon/test_web_reader.py"
     },
     {