From 8209151656ab6014f56a865b8e58d8ee862cbff6 Mon Sep 17 00:00:00 2001 From: codejeet Date: Wed, 1 Apr 2026 13:28:58 +0200 Subject: [PATCH] Add deterministic Ralph regression coverage (#392) --- SGT_CONTEXT.md | 143 ++++++++++++++++++++ test_mayor_stranded_zero_worker_recovery.sh | 1 + test_president_runtime_supervision.sh | 28 +++- test_ralph_mode_config_and_state.sh | 109 +++++++++++++++ 4 files changed, 280 insertions(+), 1 deletion(-) diff --git a/SGT_CONTEXT.md b/SGT_CONTEXT.md index 437b8df..b855ed7 100644 --- a/SGT_CONTEXT.md +++ b/SGT_CONTEXT.md @@ -1526,3 +1526,146 @@ Required follow-up: - 2026-04-01T12:07:56+02:00 — Acceptance blocker sgt-acceptance-1775022423-817268fb resolved. - 2026-04-01T12:07:56+02:00 — Acceptance blocker sgt-acceptance-1775037131-162b91f4 resolved. - 2026-04-01T12:07:56+02:00 — Acceptance blocker sgt-acceptance-1775037095-0d58f10b resolved. +- 2026-04-01T12:29:10+02:00 — 2026-04-01 issue #357: President supervision now hard-skips hibernated rigs before any start, refresh, or wake decision, so actionable-no-forward-motion, pending-plan-underfilled, and Ralph checks cannot trigger manual-refresh|rig= or restart mayor/ while the rig remains hibernated. Regression coverage: test_president_runtime_supervision.sh. +- 2026-04-01T12:29:13+02:00 — 2026-04-01 issue #366: when Ralph keeps completion pending, plan-state completion.acceptance now mirrors the effective status and clears live terminal fields; declared terminal acceptance metadata is preserved under completion.acceptance.declared_* for operator forensics. Regression coverage: test_ralph_mode_config_and_state.sh. +- 2026-04-01T12:29:33+02:00 — 2026-04-01 issue #378: _president_supervise_rig_mayor now hard-skips hibernated rigs before any start/refresh/wake decision, so President cannot restart or poke mayor/ while manual hibernation remains active. Regression coverage in test_president_runtime_supervision.sh and test_president_runtime_latest_main_proof.sh asserts no refresh, wake, or mayor start on a hibernated rig. +- 2026-04-01T12:30:26+02:00 — 2026-04-01 issue #356: added explicit regression coverage in test_plan_tick_duplicate_closeout_successor_refresh.sh for the duplicate-closeout refill case. A completed task with stale duplicate-closed lineage now proves two fences together: unchanged task signatures stay completed via merged lineage retention, while materially changed task payloads clear stale completion bookkeeping and dispatch a fresh successor instead of relaunching the killed duplicate lane. +- 2026-04-01T12:30:37+02:00 — 2026-04-01 issue #379: while Ralph keeps a rig pending, plan-state now rewrites completion.acceptance.status to the effective pending status and preserves stale terminal acceptance metadata under completion.acceptance.declared_* so verified/blocked drift cannot make successor work look done. +- 2026-04-01T12:31:03+02:00 — 2026-04-01 issue #377: _resling_pre_dispatch_revalidate now falls back to REST issue/pull endpoints when gh issue view or gh pr view flakes, so mayor stranded-rig recovery can still spawn workers from already-open sgt-authorized queues during GraphQL/live-query failures. +- 2026-04-01T12:36:39+02:00 — 2026-04-01 issue #379: _plan_state_snapshot must preserve the live pending Ralph override from plan-state when completion.rollup is ralph-* or Ralph says completion is blocked; otherwise re-sync from SGT_PLAN.json can reintroduce stale verified/blocked acceptance and make successor work look terminal again. +- 2026-04-01T12:37:14+02:00 — 2026-04-01 issue #379: _plan_state_snapshot now applies the live Ralph completion override immediately, so plan-state cannot briefly rehydrate stale verified/blocked terminal acceptance while Ralph remains unmet; _plan_state_update_completion also preserves incoming completion.acceptance.declared_* forensic metadata instead of stripping it on the pending rewrite. Regression coverage in test_ralph_mode_config_and_state.sh now covers stale blocked metadata as well as verified metadata. +- 2026-04-01T12:37:37+02:00 — Acceptance blocker sgt-acceptance-1775039857-953cb5e8 reported by witness: Replacement work required after stalled polecat for issue #379 + +### Acceptance Blocker sgt-acceptance-1775039857-953cb5e8 + +- Reported at: 2026-04-01T12:37:37+02:00 +- Reported by: witness +- Title: Replacement work required after stalled polecat for issue #379 + +```markdown +Replacement work required after stalled polecat for issue #379 + +Stalled polecat recovery did not produce replacement work. + +- Rig: sgt +- Repo: codejeet/sgt +- Issue: #379 +- Issue URL: https://github.com/codejeet/sgt/issues/379 +- Issue title: unknown +- Polecat: sgt-737d5c22 +- Failure reason: witness-stalled-issue-title-unavailable + +Required follow-up: +- create replacement work (new PR or re-dispatched polecat) before closing the incident +- re-investigate why the worker exited without producing a PR + +``` +- 2026-04-01T12:37:43+02:00 — 2026-04-01 issue #380: when a repo-local plan task changes materially, plan tick now clears any stale open issue binding once that old lane is no longer actively worked, and sweep/mayor/resling skip open plan-labeled issues whose body/title no longer match the current task so duplicate confidence-lift families cannot be resurrected after refill or consistency-mismatch churn. +- 2026-04-01T12:38:12+02:00 — Acceptance blocker sgt-acceptance-1775039892-82638667 reported by witness: Replacement work required after stalled polecat for issue #359 + +### Acceptance Blocker sgt-acceptance-1775039892-82638667 + +- Reported at: 2026-04-01T12:38:12+02:00 +- Reported by: witness +- Title: Replacement work required after stalled polecat for issue #359 + +```markdown +Replacement work required after stalled polecat for issue #359 + +Stalled polecat recovery did not produce replacement work. + +- Rig: sgt +- Repo: codejeet/sgt +- Issue: #359 +- Issue URL: https://github.com/codejeet/sgt/issues/359 +- Issue title: unknown +- Polecat: sgt-1674b1cf +- Failure reason: witness-stalled-issue-title-unavailable + +Required follow-up: +- create replacement work (new PR or re-dispatched polecat) before closing the incident +- re-investigate why the worker exited without producing a PR + +``` +- 2026-04-01T12:38:17+02:00 — Acceptance blocker sgt-acceptance-1775039897-d6b36dec reported by witness: Replacement work required after stalled polecat for issue #379 + +### Acceptance Blocker sgt-acceptance-1775039897-d6b36dec + +- Reported at: 2026-04-01T12:38:17+02:00 +- Reported by: witness +- Title: Replacement work required after stalled polecat for issue #379 + +```markdown +Replacement work required after stalled polecat for issue #379 + +Stalled polecat recovery did not produce replacement work. + +- Rig: sgt +- Repo: codejeet/sgt +- Issue: #379 +- Issue URL: https://github.com/codejeet/sgt/issues/379 +- Issue title: unknown +- Polecat: sgt-2c74a5cc +- Failure reason: witness-stalled-issue-title-unavailable + +Required follow-up: +- create replacement work (new PR or re-dispatched polecat) before closing the incident +- re-investigate why the worker exited without producing a PR + +``` +- 2026-04-01T12:39:24+02:00 — Acceptance blocker sgt-acceptance-1775039964-1c53354b reported by witness: Replacement work required after stalled polecat for issue #380 + +### Acceptance Blocker sgt-acceptance-1775039964-1c53354b + +- Reported at: 2026-04-01T12:39:24+02:00 +- Reported by: witness +- Title: Replacement work required after stalled polecat for issue #380 + +```markdown +Replacement work required after stalled polecat for issue #380 + +Stalled polecat recovery did not produce replacement work. + +- Rig: sgt +- Repo: codejeet/sgt +- Issue: #380 +- Issue URL: https://github.com/codejeet/sgt/issues/380 +- Issue title: unknown +- Polecat: sgt-61f8c178 +- Failure reason: witness-stalled-issue-title-unavailable + +Required follow-up: +- create replacement work (new PR or re-dispatched polecat) before closing the incident +- re-investigate why the worker exited without producing a PR + +``` +- 2026-04-01T12:45:47+02:00 — Acceptance blocker sgt-acceptance-1775039892-82638667 resolved. +- 2026-04-01T12:45:47+02:00 — Acceptance blocker sgt-acceptance-1775039857-953cb5e8 resolved. +- 2026-04-01T12:45:47+02:00 — Acceptance blocker sgt-acceptance-1775039897-d6b36dec resolved. +- 2026-04-01T13:01:29+02:00 — Issue #380: plan tick now clears stale open issue bindings after a material task-signature change once no active polecat remains, and sweep/mayor/witness/resling all skip open plan-labeled issues whose current title/body no longer match the repo-local task definition so duplicate continuation lanes are not resurrected by refill or watchdog churn. +- 2026-04-01T13:02:29+02:00 — Acceptance blocker sgt-acceptance-1775041349-4cbdb7d1 reported by witness: Replacement work required after stalled polecat for issue #380 + +### Acceptance Blocker sgt-acceptance-1775041349-4cbdb7d1 + +- Reported at: 2026-04-01T13:02:29+02:00 +- Reported by: witness +- Title: Replacement work required after stalled polecat for issue #380 + +```markdown +Replacement work required after stalled polecat for issue #380 + +Stalled polecat recovery did not produce replacement work. + +- Rig: sgt +- Repo: codejeet/sgt +- Issue: #380 +- Issue URL: https://github.com/codejeet/sgt/issues/380 +- Issue title: unknown +- Polecat: sgt-036f3c65 +- Failure reason: witness-stalled-issue-title-unavailable + +Required follow-up: +- create replacement work (new PR or re-dispatched polecat) before closing the incident +- re-investigate why the worker exited without producing a PR + +``` diff --git a/test_mayor_stranded_zero_worker_recovery.sh b/test_mayor_stranded_zero_worker_recovery.sh index 9ce6b0f..98e3413 100644 --- a/test_mayor_stranded_zero_worker_recovery.sh +++ b/test_mayor_stranded_zero_worker_recovery.sh @@ -28,6 +28,7 @@ export SGT_LOG="$TMP_ROOT/sgt.log" _repo_owner_repo() { printf '%s\n' "${1#https://github.com/}"; } _mayor_rig_hibernated() { return 1; } _mayor_rig_activity_snapshot() { printf '%s\n' 'active|open_issues=2 open_prs=0 active_polecats=0 merge_queue=0 pending_plan_requests=0|2|0|0|0|0|not_declared|not_declared'; } +_plan_task_issue_matches_current_plan() { return 0; } _issue_backend_dispatch_limited_reason() { return 1; } _resling_find_existing_issue_polecat() { return 1; } _sweep_watchdog_find_open_pr_for_issue() { return 1; } diff --git a/test_president_runtime_supervision.sh b/test_president_runtime_supervision.sh index fb30434..eeaeb8e 100644 --- a/test_president_runtime_supervision.sh +++ b/test_president_runtime_supervision.sh @@ -240,13 +240,39 @@ if ! grep -q 'PRESIDENT_OPERATOR_EVENT rig=demo kind=drift severity=warning noti exit 1 fi +_ralph_mode_snapshot_fields() { + printf '1|1K PNL for 5m btc pipeline|unmet|3|0|idle|condition unmet; no active admissible lanes toward target 3|0|2|2|0|0|1|1||||\n' +} + +refresh_count_before_ralph_idle="$(grep -c '^demo$' "$REFRESH_LOG" || true)" +wake_count_before_ralph_idle="$(grep -c '^president:demo:ralph-idle$' "$WAKE_LOG" || true)" +_president_supervise_rig_mayor demo periodic > "$TMP_ROOT/president-ralph-idle.out" + +if [[ "$(grep -cx 'demo' "$REFRESH_LOG")" -ne $((refresh_count_before_ralph_idle + 1)) ]]; then + echo "expected Ralph idle to trigger a refresh intervention" >&2 + exit 1 +fi +if [[ "$(grep -c '^president:demo:ralph-idle$' "$WAKE_LOG" || true)" -ne "$wake_count_before_ralph_idle" ]]; then + echo "expected Ralph idle to refresh instead of issuing a wake" >&2 + exit 1 +fi +if ! grep -q 'PRESIDENT_INTERVENTION rig=demo action=refresh reason=ralph-idle' "$EVENT_LOG"; then + echo "expected durable President refresh intervention for Ralph idle" >&2 + exit 1 +fi +if ! grep -q 'PRESIDENT_OPERATOR_EVENT rig=demo kind=drift severity=warning notify=1 dedupe_key=president:demo:drift:ralph-idle:refresh overlap_key=rig-incident:demo:ralph action=refresh reason=ralph-idle outcome=intervened' "$EVENT_LOG"; then + echo "expected structured President drift event for Ralph idle refresh" >&2 + exit 1 +fi + _ralph_mode_snapshot_fields() { printf '1|1K PNL for 5m btc pipeline|unmet|3|0|underfilled|condition unmet; active_lanes=1 target=3|1|1|0|0|0|1|1||||\n' } +refresh_count_before_ralph_contradiction="$(grep -c '^demo$' "$REFRESH_LOG" || true)" _president_supervise_rig_mayor demo periodic > "$TMP_ROOT/president-ralph-contradiction.out" -if [[ "$(grep -cx 'demo' "$REFRESH_LOG")" -ne 2 ]]; then +if [[ "$(grep -cx 'demo' "$REFRESH_LOG")" -ne $((refresh_count_before_ralph_contradiction + 1)) ]]; then echo "expected Ralph contradiction to trigger a refresh intervention" >&2 exit 1 fi diff --git a/test_ralph_mode_config_and_state.sh b/test_ralph_mode_config_and_state.sh index de1bdce..8fe8c99 100644 --- a/test_ralph_mode_config_and_state.sh +++ b/test_ralph_mode_config_and_state.sh @@ -118,11 +118,41 @@ cat > "$TMP_ROOT/issues.json" <<'JSON' ] JSON +cat > "$TMP_ROOT/issues-second.json" <<'JSON' +[ + { + "number": 21, + "title": "Primary active lane", + "labels": [ + {"name": "sgt-authorized"}, + {"name": "research"} + ] + }, + { + "number": 22, + "title": "Queued backlog lane", + "labels": [ + {"name": "sgt-authorized"}, + {"name": "research"} + ] + }, + { + "number": 23, + "title": "Support lane", + "labels": [ + {"name": "sgt-authorized"}, + {"name": "support-only"} + ] + } +] +JSON + ENV_PREFIX=( env -i HOME="$HOME_DIR" PATH="$MOCK_BIN:$HOME_DIR/.local/bin:/usr/local/bin:/usr/bin:/bin" TERM=dumb + TMP_ROOT="$TMP_ROOT" SGT_ROOT="$HOME_DIR/sgt" SGT_TEST_ISSUES_JSON="$TMP_ROOT/issues.json" SGT_TEST_ACTIVE_SESSION="sgt-demo-worker" @@ -213,6 +243,10 @@ assert ralph.get("admissible_lane_count") == 1, ralph assert ralph.get("support_lane_count") == 1, ralph assert ralph.get("duplicate_lane_count") == 1, ralph assert ralph.get("underfilled") is True, ralph +assert ralph.get("active_issue_numbers") == ["11"], ralph +assert ralph.get("backlog_issue_numbers") == [], ralph +assert ralph.get("support_issue_numbers") == ["12"], ralph +assert ralph.get("duplicate_issue_numbers") == ["13"], ralph completion = plan_state.get("completion") or {} assert completion.get("status") == "pending", completion @@ -260,6 +294,81 @@ assert ralph.get("admissible_lane_count") == 2, ralph assert ralph.get("support_lane_count") == 0, ralph assert ralph.get("duplicate_lane_count") == 1, ralph assert ralph.get("backlog_lane_count") == 1, ralph +assert ralph.get("active_issue_numbers") == ["11"], ralph +assert ralph.get("backlog_issue_numbers") == ["12"], ralph +assert ralph.get("support_issue_numbers") == [], ralph +assert ralph.get("duplicate_issue_numbers") == ["13"], ralph +PY + +export SGT_TEST_ISSUES_JSON="$TMP_ROOT/issues-second.json" +cat > "$SGT_ROOT/.sgt/rigs/demo2" <<'STATE' +https://github.com/acme/demo2 +STATE +mkdir -p "$SGT_ROOT/rigs/demo2" +cat > "$SGT_ROOT/.sgt/polecats/demo2-worker" <<'STATE' +RIG=demo2 +REPO=https://github.com/acme/demo2 +ISSUE=21 +BRANCH=sgt/demo2-worker +WORKTREE=/tmp/demo2-worker +SESSION=sgt-demo-worker +STATE + +cat > "$SGT_ROOT/rigs/demo2/SGT_PLAN.json" <<'JSON' +{ + "version": 1, + "rig": "demo2", + "policy": { "max_in_flight": 1 }, + "completion_condition": "Fresh-state proof passes.", + "acceptance": { + "status": "waived", + "details": "Human waived the previous proof run.", + "waived_at": "2026-03-31T09:30:00Z" + }, + "tasks": [] +} +JSON + +sgt config ralph demo2 --enable --condition "Keep exploring live candidate lanes" --target 1 >/dev/null +sgt plan tick demo2 >/dev/null 2>&1 +sgt status --json > "$SGT_ROOT/status-second.json" + +python3 - "$SGT_ROOT/status-second.json" "$SGT_ROOT/.sgt/plan-state/demo2.json" <<'PY' +import json +import sys + +with open(sys.argv[1], "r", encoding="utf-8") as fh: + status = json.load(fh) +with open(sys.argv[2], "r", encoding="utf-8") as fh: + plan_state = json.load(fh) + +mayor_rigs = status.get("mayor_rigs") or [] +demo2 = next((item for item in mayor_rigs if item.get("rig") == "demo2"), None) +assert demo2 is not None, mayor_rigs +ralph = demo2.get("ralph") or {} +assert ralph.get("state") == "active", ralph +assert ralph.get("underfilled") is False, ralph +assert ralph.get("target_concurrency") == 1, ralph +assert ralph.get("active_lane_count") == 1, ralph +assert ralph.get("admissible_lane_count") == 2, ralph +assert ralph.get("backlog_lane_count") == 1, ralph +assert ralph.get("support_lane_count") == 1, ralph +assert ralph.get("active_issue_numbers") == ["21"], ralph +assert ralph.get("backlog_issue_numbers") == ["22"], ralph +assert ralph.get("support_issue_numbers") == ["23"], ralph +assert ralph.get("completion_blocked_by_condition") is True, ralph + +completion = plan_state.get("completion") or {} +assert completion.get("status") == "pending", completion +assert completion.get("rollup") == "ralph-condition-unmet", completion +assert completion.get("blocked_reason") == "ralph condition unmet: Keep exploring live candidate lanes", completion +assert "Ralph mode remains active" in completion.get("details", ""), completion +acceptance = completion.get("acceptance") or {} +assert acceptance.get("status") == "pending", acceptance +assert acceptance.get("details") == "Human waived the previous proof run.", acceptance +assert acceptance.get("declared_status") == "waived", acceptance +assert acceptance.get("declared_waived_at") == "2026-03-31T09:30:00Z", acceptance +assert "waived_at" not in acceptance, acceptance PY BASH