Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
d35c3cd
perf: record durable pytest run statistics
Sinity Aug 12, 2026
c54028a
perf: reclaim pytest temp trees eagerly
Sinity Aug 12, 2026
20a660b
docs: describe eager pytest temp cleanup
Sinity Aug 12, 2026
744db17
perf: measure fixture setup without duplicate scans
Sinity Aug 12, 2026
2256a91
fix(test): keep pytest run evidence durable
Sinity Aug 12, 2026
c0f5978
fix(test): unify cross-worktree run history
Sinity Aug 12, 2026
ad6704f
fix(test): serialize durable history appends
Sinity Aug 12, 2026
23e4b2f
fix(test): retain compact statistics across cleanup
Sinity Aug 12, 2026
e3dd927
fix(test): repair verify run evidence review
Sinity Aug 12, 2026
cf67f44
fix(test): retain terminal verify evidence
Sinity Aug 12, 2026
c95cd8b
fix(test): finalize verify run evidence
Sinity Aug 12, 2026
782b0ef
fix(test): preserve complete verify-run evidence
Sinity Aug 12, 2026
107ce54
fix(test): refuse unsafe explicit tmpfs basetemps
Sinity Aug 12, 2026
2c835ed
fix(test): close tmpfs basetemp ownership gaps
Sinity Aug 12, 2026
d2d601d
fix: preserve pytest basetemp ownership evidence
Sinity Aug 12, 2026
5e7efa3
fix(test): preserve pytest ownership evidence
Sinity Aug 13, 2026
044acb1
fix(test): auto-reroute configured tmpfs roots
Sinity Aug 13, 2026
17f8b20
fix(test): preserve exact pytest run evidence
Sinity Aug 13, 2026
ff3066e
fix(test): retain failed pytest admission evidence
Sinity Aug 13, 2026
0697f57
fix(test): preserve exact pytest outcome evidence
Sinity Aug 13, 2026
066769c
test(devtools): isolate verification anchor regression
Sinity Aug 13, 2026
b0eaa5b
fix(test): bind terminal evidence to exact checkout state
Sinity Aug 13, 2026
f0fcd1c
fix(test): make focused xdist runs isolation-safe
Sinity Aug 13, 2026
8640677
fix(test): preserve canonical completed-run evidence
Sinity Aug 13, 2026
dcee6b7
fix(test): reject unverifiable cached gate evidence
Sinity Aug 13, 2026
258306c
fix(devtools): recognize empty pytest reports
Sinity Aug 13, 2026
2f89c21
fix(test): isolate nested testmon state
Sinity Aug 13, 2026
3b87b21
fix(verify): anchor nested report paths
Sinity Aug 13, 2026
867d57b
fix(devtools): parse streamed merge receipts
Sinity Aug 13, 2026
e134526
fix(devtools): bind merge gate to run artifact
Sinity Aug 13, 2026
d38940f
fix(devtools): expose current run evidence path
Sinity Aug 13, 2026
c5a3e0e
fix(devtools): bind merge evidence to invocation
Sinity Aug 13, 2026
d536ab3
fix(devtools): preserve exact pytest evidence
Sinity Aug 13, 2026
ceca178
fix(devtools): bind merge scope to checkout root
Sinity Aug 13, 2026
b1f1eb4
test(devtools): release direct basetemp claims
Sinity Aug 13, 2026
7dac899
fix(devtools): reject mixed-checkout test evidence
Sinity Aug 13, 2026
cc8b29a
fix(test): preserve nested pytest authority
Sinity Aug 13, 2026
81a9b0e
fix(test): preserve nested pytest evidence authority
Sinity Aug 13, 2026
aab4b5e
fix: harden nested verification harness evidence
Sinity Aug 13, 2026
a6deeb0
fix: use portable checkout mutation watcher
Sinity Aug 13, 2026
524ffe0
fix: reject lossy checkout mutation watchers
Sinity Aug 13, 2026
3a8dbe1
fix(devtools): prune ignored watcher trees
Sinity Aug 13, 2026
95e0f55
fix(devtools): bind mutation watch to Git authority
Sinity Aug 13, 2026
f591340
test(devtools): type partial enumeration probe
Sinity Aug 13, 2026
f57c739
fix(devtools): discard unstable verification authority
Sinity Aug 13, 2026
99e2219
test(devtools): isolate fingerprint failure receipts
Sinity Aug 13, 2026
34e9c08
fix(devtools): pin verification Git authority
Sinity Aug 13, 2026
995f7d1
fix(devtools): close Git authority races
Sinity Aug 13, 2026
c0f0c3d
fix(devtools): cover hidden Git authority
Sinity Aug 13, 2026
2579dc6
fix(devtools): close verification monitor gaps
Sinity Aug 13, 2026
c66671d
fix(devtools): bind verification recovery evidence
Sinity Aug 13, 2026
97bf6c7
fix(devtools): bind terminal verification evidence
Sinity Aug 13, 2026
51a32ef
fix(devtools): persist checkout interruption evidence
Sinity Aug 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,18 +125,23 @@ stale per-run dirs from every known root (`/dev/shm`, `/realm/tmp/polylogue-pyte
`/tmp/polylogue-pytest`, plus any explicit configured root) — never based on
age alone: each managed basetemp carries a PID plus process-start identity,
and a directory whose exact owner process is still alive is never removed
regardless of age. An owner that cannot be confirmed dead (no marker) gets a
multi-hour grace period rather than the normal ~30-minute one. The sweeper
regardless of age. A tree without a valid managed claim or whose owner cannot
be confirmed dead is never removed. The thirty-minute age threshold applies
only after a positive managed claim identifies a dead owner. The sweeper
restores owner-write permission only after a tree is adjudicated stale, so
published read-only fixture copies cannot leak tmpfs indefinitely. Shared
`pytest-polylogue-*-seeded-*` caches are never touched by the sweep — they
are shared, reused, and built once behind their own `.build.done` guard.

Managed verification refuses to start below 1 GiB available memory instead of
falling back to the pathological disk lane. Passing-test roots are reclaimed at
teardown; the external supervisor and parent runner independently remove the
whole run root on completion or termination, with startup stale-root cleanup as
recovery after an uncatchable process kill or reboot.
falling back to the pathological disk lane. Every per-test `tmp_path` tree is
reclaimed in fixture teardown, including failures and interruptions; node
failure evidence remains in the managed event, longrepr, summary, and resource
receipts. The controller removes only the exact basetemp it created. An
explicit `--basetemp` is retained for targeted filesystem diagnosis. The
external supervisor and parent runner independently remove the whole run root
on completion or termination, with startup stale-root cleanup as recovery
after an uncatchable process kill or reboot.

An affected run that selects zero tests is accepted only when no executable,
test, dependency, or harness path changed. A zero selection after such a change
Expand All @@ -163,6 +168,9 @@ and a postmortem diagnosis. The latest run is mirrored to
- `.cache/verify/current-pytest-resources.jsonl`
- `.cache/verify/current-pytest-postmortem.json`
- `.cache/verify/current-pytest-containment.json`
- `.cache/verify/current-pytest-statistics.json` — derived phase
distributions, worker count, storage, resource peaks, and cleanup outcome;
the same file is retained under each run's `steps/*/statistics.json`.
- `.cache/verify/current-pytest-output.log`

The devtools process drains pytest output, prints periodic heartbeat lines, and
Expand Down Expand Up @@ -195,9 +203,13 @@ IDs by default (`POLYLOGUE_PYTEST_SELECTION_NODEID_LIMIT`, default 500) so
broad collection does not retain or write unbounded node-id lists in controller
or worker processes.

`devtools workspace tasks recent` shows the run id, diagnosis, and peak pytest
RSS when the current run metadata is available. `devtools workspace tasks stats
--resources` aggregates recorded pytest memory peaks over time.
The detailed artifacts above are checkout-local and disposable. Each `devtools
verify` or `devtools test` invocation automatically appends its compact run
summary to `$XDG_STATE_HOME/polylogue/devtools/verify-history.jsonl` (or the
corresponding `~/.local/state` path), shared across linked worktrees without a
separate recording command. `devtools verify --history` prints the recent
cross-worktree runs. Setup, call, and teardown timings come only from pytest
reports in the event stream.

`devtools test` uses the same pytest progress plugin and process supervisor for
focused selections. During or after a run, inspect
Expand Down
66 changes: 59 additions & 7 deletions devtools/evidence_dashboard.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
- pytest health from ``.cache/verify/last-pytest.json``;
- coverage from ``.coverage`` / ``coverage.xml`` when present;
- benchmark/SLO catalog rows and their required-artifact coverage;
- static gate status from ``.cache/verify-history.jsonl``;
- static gate status from the shared XDG verify history;
- witness lifecycle counts;
- mutation/benchmark campaign freshness.

Expand All @@ -28,12 +28,13 @@
from typing import Any

from devtools import repo_root as _get_root
from devtools.verify_runs import VERIFY_HISTORY_PATH, git_dirty, git_head
from devtools.verify_runs import worktree_fingerprint as _worktree_fingerprint

ROOT = _get_root()

# Artifact paths (relative to repo root).
PYTEST_REPORT_REL = Path(".cache/verify/last-pytest.json")
VERIFY_HISTORY_REL = Path(".cache/verify-history.jsonl")
LAST_VERIFY_RESULT_REL = Path(".cache/last-verify-result.json")
COVERAGE_DATA_REL = Path(".coverage")
COVERAGE_XML_REL = Path("coverage.xml")
Expand Down Expand Up @@ -224,9 +225,37 @@ def _benchmark_slo(root: Path, *, now: datetime) -> dict[str, Any]:
)


def _static_evidence_is_bound(
entry: dict[str, Any],
*,
checkout_root: str,
checkout_head: str | None,
worktree_fingerprint: str,
) -> bool:
"""Accept only evidence tied to the exact checkout contents being viewed."""
steps = entry.get("steps")
stability_failed = isinstance(steps, list) and any(
isinstance(step, dict) and step.get("name") == "checkout stability" and step.get("exit") != 0 for step in steps
)
return (
not stability_failed
and entry.get("diagnosis") not in {"checkout_changed_during_verification", "checkout_fingerprint_unavailable"}
and entry.get("checkout_root") == checkout_root
and entry.get("git_head") == checkout_head
and entry.get("worktree_fingerprint") == worktree_fingerprint
Comment thread
Sinity marked this conversation as resolved.
and entry.get("final_worktree_fingerprint") == worktree_fingerprint
)
Comment on lines +240 to +247

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject watcher-invalidated runs in the evidence dashboard

When the mutation watcher observes an edit that is restored before the final sample, both fingerprints equal the current checkout even though verification exits 125 with a checkout stability failure. This predicate therefore accepts that rejected history entry and _static_gates() presents its successful steps as current green evidence. Fresh evidence beyond the prior final-fingerprint repair is that transient watcher failures deliberately preserve equal endpoint fingerprints; reject entries carrying a checkout-stability diagnosis or failed stability step rather than relying only on fingerprint equality.

AGENTS.md reference: AGENTS.md:L402-L409

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at 99e2219: static evidence rejects a failed checkout-stability step or either checkout-instability diagnosis even if endpoint fingerprints match.



def _static_gates(root: Path, *, now: datetime) -> dict[str, Any]:
history_path = root / VERIFY_HISTORY_REL
history_path = VERIFY_HISTORY_PATH
Comment thread
Sinity marked this conversation as resolved.
last_result_path = root / LAST_VERIFY_RESULT_REL
checkout_root = str(root.resolve())
checkout_head = git_head(root)
checkout_dirty = git_dirty(root)
fingerprint = None if checkout_dirty or checkout_head is None else _worktree_fingerprint(root)
worktree_fingerprint = None if fingerprint == "unavailable" else fingerprint
identity_available = checkout_head is not None and worktree_fingerprint is not None

# Prefer last-verify-result.json (the most recent run) then walk back through
# history to find the last status for each gate.
Expand All @@ -236,7 +265,16 @@ def _static_gates(root: Path, *, now: datetime) -> dict[str, Any]:
try:
data = json.loads(last_result_path.read_text())
result = data.get("result") if isinstance(data, dict) else None
if isinstance(result, dict):
if (
isinstance(result, dict)
and worktree_fingerprint is not None
and _static_evidence_is_bound(
result,
checkout_root=checkout_root,
checkout_head=checkout_head,
worktree_fingerprint=worktree_fingerprint,
)
):
for step in result.get("steps", []):
if isinstance(step, dict) and isinstance(step.get("name"), str):
last_steps[step["name"]] = step
Expand All @@ -263,6 +301,13 @@ def _static_gates(root: Path, *, now: datetime) -> dict[str, Any]:
# appearance in history.
if history_entries:
for entry in reversed(history_entries):
if worktree_fingerprint is None or not _static_evidence_is_bound(
entry,
checkout_root=checkout_root,
checkout_head=checkout_head,
worktree_fingerprint=worktree_fingerprint,
):
continue
steps = entry.get("steps", [])
for step in steps:
if not isinstance(step, dict):
Expand All @@ -275,7 +320,14 @@ def _static_gates(root: Path, *, now: datetime) -> dict[str, Any]:
for gate_name in _STATIC_GATE_NAMES:
step = last_steps.get(gate_name)
if step is None:
gates.append({"name": gate_name, "available": False, "reason": "no run observed in cached history"})
reason = (
"checkout has uncommitted changes"
if checkout_dirty
else "checkout Git identity is unavailable"
if not identity_available
else "no bound run observed in cached history"
)
gates.append({"name": gate_name, "available": False, "reason": reason})
continue
exit_code = step.get("exit", -1)
gates.append(
Expand All @@ -290,8 +342,8 @@ def _static_gates(root: Path, *, now: datetime) -> dict[str, Any]:
)
failing = [g for g in gates if g.get("status") == "fail"]
return {
"available": last_result_path.exists() or history_path.exists(),
"history_path": str(VERIFY_HISTORY_REL),
"available": bool(last_steps) and not checkout_dirty and identity_available,
"history_path": str(history_path),
"last_result_path": str(LAST_VERIFY_RESULT_REL),
"total_gates_tracked": len(_STATIC_GATE_NAMES),
"gates_with_status": sum(1 for g in gates if g.get("available")),
Expand Down
52 changes: 37 additions & 15 deletions devtools/merge_boundary.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
import sys
import tempfile
import time
import uuid
from collections.abc import Iterator, Mapping
from pathlib import Path
from typing import Any
Expand Down Expand Up @@ -608,7 +609,13 @@ def cmd_merge(
return 1

head_sha = info["headRefOid"]
scope = merge_gate._scope_verdict(pr, info, head_sha=head_sha)
checkout_root = merge_gate._repository_root()
scope = merge_gate._scope_verdict(
pr,
info,
head_sha=head_sha,
checkout_root=checkout_root,
)

if not scope.ok:
print(f"REFUSING to merge PR #{pr}: invalid structured pr-scope carrier:", file=sys.stderr)
Expand Down Expand Up @@ -666,7 +673,12 @@ def cmd_merge(
file=sys.stderr,
)
return 1
final_scope = merge_gate._scope_verdict(pr, final_info, head_sha=head_sha)
final_scope = merge_gate._scope_verdict(
pr,
final_info,
head_sha=head_sha,
checkout_root=checkout_root,
)
initial_attestation = pr_scope.attestation_payload(
scope, head_sha=head_sha, base_sha=merge_gate._base_sha(info)
).get("attestation_digest")
Expand Down Expand Up @@ -798,20 +810,30 @@ def cmd_record_full_verify(
if execution_root is not None:
argv = ["direnv", "exec", str(execution_root), *argv]
started = verification_started_at
try:
result = subprocess.run(argv, capture_output=True, text=True, cwd=cwd)
except OSError as exc:
print(f"REFUSING: could not run {command!r}: {exc}", file=sys.stderr)
return 2
checkout_root = Path(cwd) if cwd is not None else Path.cwd()
invocation_id = uuid.uuid4().hex
with tempfile.TemporaryDirectory(prefix="polylogue-terminal-verify-") as temp_dir:
receipt_path = Path(temp_dir) / "run.json"
env = dict(os.environ)
env[merge_gate.VERIFICATION_INVOCATION_ID_ENV] = invocation_id
env[merge_gate.VERIFICATION_RECEIPT_PATH_ENV] = str(receipt_path)
try:
result = subprocess.run(argv, capture_output=True, text=True, cwd=checkout_root, env=env)
except OSError as exc:
print(f"REFUSING: could not run {command!r}: {exc}", file=sys.stderr)
return 2
receipt = merge_gate._invocation_receipt(
path=receipt_path,
invocation_id=invocation_id,
head_sha=target_sha,
command_exit=result.returncode,
checkout_root=checkout_root,
)
duration_s = round(time.time() - started, 2)
release_allowed = merge_gate._release_baseline_permission(result.stdout)
verification_scope = merge_gate._verification_scope(result.stdout)
terminal_authorization = merge_gate._terminal_authorization(result.stdout)
try:
structured = json.loads(result.stdout)
except (TypeError, json.JSONDecodeError):
structured = None
verified_head = structured.get("git_head") if isinstance(structured, dict) else None
release_allowed = merge_gate._release_baseline_permission(receipt)
verification_scope = merge_gate._verification_scope(receipt)
terminal_authorization = merge_gate._terminal_authorization(receipt)
verified_head = receipt.get("git_head") if isinstance(receipt, dict) else None
accepted = (
result.returncode == 0
and release_allowed is True
Expand Down
Loading