feat(auto-improvement): measurement-first self-improvement app (GitHub-native)

[Zedmor]

Problem

An agent that edits your repository unsupervised has no way to prove it helped. It can
"optimize" something and be measuring noise; it can "fix" a bug it never reproduced; it can
delete a test to make the suite pass. There is no artifact a human can check afterwards, so
the only safe answer today is not to run one.

Why it matters

The useful version of this — a loop that finds real defects and proposes real fixes while you
sleep — is blocked on trust, not on capability. Without measurement you cannot tell a genuine
win from a coincidence, and without a paper trail you cannot review the result. So the feature
either does not ship, or ships as something nobody should point at a repository they care about.

Fix: measurement first, then change

The chain of thought is: you cannot keep a win you cannot measure → so prove the instrument
before using it → then make every kept change carry its own evidence.

1. Prove the ruler before touching anything. Phase 1 calibrates a noise band from repeated
baseline samples (max(2sigma, floor)), then forces a known win and requires it to clear that
band. If the ruler cannot see a win it is known to contain, the perf run halts — the app
refuses to measure with an instrument it has not validated.

2. Keep-or-revert, never keep-and-hope. A candidate is kept only if its delta clears the
calibrated band and every guardrail holds. Bug-track candidates must demonstrate
RED → GREEN → STAYGREEN: a new test that fails on unmodified code, passes with the fix, and
keeps passing. A candidate that cannot show that is reverted, and the reason is recorded.

3. Every decision is durable. An append-only ledger records what was investigated and what
happened to it — filed, Rejected by the gate, Below the noise band, No defect found,
Duplicate. Content-fingerprinted, so the loop never re-investigates the same locus across
restarts.

4. It proposes; it does not publish. The clone's push is mechanically disabled (both remote
URLs neutralized — pushing by URL ignores the push URL, so disabling one is not enough). Work
leaves as a draft pull request you review. Direct-commit is opt-in and can never target a
protected branch.

Screenshots

Captured from a real run against Zedmor/chess_test on
the loopback dashboard — not mockups.

The dashboard. Ruler calibrated with its measured band (7.76s), the ledger count, and the
push disabled badge on the working clone.

The findings ledger. Every locus the loop touched and its outcome. Note the statuses are
prose, not implementation tokens, and the irreversible action carries a text label
("Commit to the branch"), not a bare icon.

The evidence behind one kept change — the part that makes this reviewable. Defect statement,
reproduction hypothesis, the six verification gates that passed (RED (x2) → GREEN → STAYGREEN),
the repro test's path, the base sha it was measured against, the fingerprint, and the resulting
pull request.

What the loop actually found in that run

A real off-by-one: order_moves is called with depth up to MAX_DEPTH (64), but the killer
table has only 64 slots (indices 0–63), so at depth == 64 it indexes killer_moves[64] and
raises IndexError. The agent wrote a test that reproduces it, fixed it, and the suite stayed
green — Zedmor/chess_test#1.

Suggested reading order

121 files is a lot, so here is the path that makes it reviewable. The first four files are
the whole design; everything else is adapters, UI, and tests.

#	File	Why read it
1	spine/contracts.py	The six seam types. Read this first — every other file speaks in them.
2	spine/driver.py	The loop itself: preflight → discover → propose → gate → keep-or-revert → file. The whole control flow.
3	spine/preflight.py	The trust gate. calibrate → force canary → require it clears the band, else refuse Phase 2.
4	spine/keeper.py + spine/ledger.py	The accept test and the durable record.
5	profiles/github_repo/profile.py	The one concrete target: "any Python repo with a pytest suite". Where measurement meets reality.
6	backend/routes.py + website/src/apps/auto-improvement/	HTTP surface and the dashboard above.

Safety controls (each one is a test, not a promise)

Control	Where	What it stops
Push mechanically disabled	backend/clone_setup.py	The agent cannot push, by configuration, not by instruction.
Edit fence (globs + diff scope)	RepoEditAllowlist	Editing tests, CI config, or outside the allowlist. Only the bug track may ADD a test; the perf track may not touch the suite at all, since the suite is the ruler's measurement subject.
Shell denylist, tokenized	spine/agent_runner.py	git push, gh pr merge/ready/close/create, gh api, gh auth, gh secret, curl/wget/ssh — matched after stripping global options, command wrappers (sudo, env, timeout), nested shells and &&/;/| separators.
Central governance gate	_governance_denial	Every tool request also passes the platform HookManager, including the operator's keystone denied-command list. Fail-closed: a broken hook layer denies.
Credential confinement	sandboxed_spawn_argv(mode="strict")	A nested process sees ~/.aws, ~/.config/gh, ~/.docker as empty; ~/.ssh exposes only known_hosts.
Egress redaction	9 sinks in security_posture.py	Agent-authored text is scanned before it reaches the browser, the model, the SEL audit log, or a commit message. Fail-closed.

Tests

811 tests in-tree. The interesting ones are not the happy paths:

• test_dogfood_learnings.py — 107 documented defects (D-1…D-107 in the test plan), each
  found by running this app against real repositories, each with a regression test proven to fail
  before its fix. This is the file to read if you want to know what actually went wrong.
• test_preflight.py / test_measurer.py — the ruler refuses to proceed when it cannot see a
  known win; an empty diff means "scoped to nothing", never "unscoped".
• test_github_profile.py — RED→GREEN→STAYGREEN gate, the reward-hacking test-count guard.
• test_pr_recipe.py — both remote URLs neutralized; neither push route works.

Manual verification

The screenshots above are the verification: enabled on a live gateway, pointed at a real GitHub
repository, ran the loop, and read the resulting evidence panel and pull request. To reproduce:

# on the gateway host
make frontend && .venv/bin/kirocrew gateway
# from your laptop
ssh <gateway-host> kctoken | bash

Then enable Auto-Improve in Apps, paste a repository URL, pick a base branch, and Start run.

Known limitations (stated, not discovered later)

1. The sandbox does not isolate the network. Credentials are confined (verified), but a
   nested interpreter can open a socket even though curl/wget/nc are denied. PR watchers
   are therefore opt-in (watcherAutoStart, default OFF) — turn it on only for repositories
   whose pull-request comments you would be willing to execute. Closing this properly needs a
   network-isolating primitive at the platform layer.
2. A second perf PR would carry the first perf fix. The perf loop is deliberately
   evolutionary ("current best == HEAD"), so resetting between winners would break cumulative
   measurement. Test-pinned; the perf track has never kept a real win end-to-end, so this is
   latent.
3. Four test files are excluded from CI collection — they need a real OS sandbox the runner
   lacks, or assume POSIX and fail the Windows shard. Both reasons are recorded in setup.cfg
   with the measurement behind them.

[github-actions]

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of d538eb9d924aaf1b37d1a581a635170efbf58be5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d538eb9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d538eb9d924aaf1b37d1a581a635170efbf58be5: <one-sentence reason>

[github-actions]

Arbiter — ⏳ review pending

Arbiter is waiting for Opus-5 Design review output for e93eb53473dcbc1c15c7aae681722dba023edff3; this replaces any stale verdict from the previous commit.

Second-order review for e93eb53473dcbc1c15c7aae681722dba023edff3; this comment is updated in place on each push.

False positive or not applicable? A repository writer can comment:
/ai-review override arbiter e93eb53473dcbc1c15c7aae681722dba023edff3: <one-sentence reason>

For a broader accepted-risk deferral, apply defer-longterm and explain why.

[github-actions]

UX Review (Fable 5) — 🟡 CONCERNS

Advisory UX-level review of d538eb9d924aaf1b37d1a581a635170efbf58be5 — updated in place on each push; does not block merge.

Evidence gathered — I've read the diff's UI components, the English catalog additions, and all three committed screenshots. Emitting the review.

UX-Verdict: CONCERNS

The evidence surface is strong, but the headline stats and ruler card leak raw implementation values, and the "Kept" number doesn't count kept changes.

Watch

• Ruler card shows raw internals. Screenshot 01-dashboard.png renders suite_wall_seconds · band 7.759014769207669s — AutoImprovementPage.tsx:361-364 interpolates ruler.primary.name and the unformatted float directly. The one card that proves the app's core promise ("the metric is trustworthy") reads as debug output on every dashboard visit. Fix: humanize the metric label and format the band to ~2 significant digits through i18n/format.ts.
• "Kept" stat mislabels its data. AutoImprovementPage.tsx:340 sets the Kept card to drafted.length, where DRAFTED_STATUSES = {'filed'} — so a change the operator commits (status committed) disappears from Kept, and the card can contradict the adjacent status line "{{kept}} kept, {{drafted}} pull requests drafted" (backend-sourced run.kept). Two same-named numbers disagreeing on one screen erodes exactly the trust this app sells; persistent, every run. Fix: count filed + committed, or relabel the card "Drafted".
• Fetch errors masquerade as empty states. The findings/ruler queries have no error branch; a failed /findings yields findings = [] → "No findings recorded yet." (AutoImprovementPage.tsx:410-411), and a failed /ruler renders "Not calibrated". An operator checking on an overnight run during a gateway hiccup reads "nothing found / uncalibrated" — a false report on an evidence surface. Fix: render a distinct one-line error state when isError.

Suggestions

• findingsTruncated states "Showing the 40 most recent of {{total}}" with no way to reach the rest — add a "Show all"/paging affordance next to it.
• PR table State column renders raw uppercase tokens READY/BLOCKED/PROGRESS (AutoImprovementPage.tsx:218, visible in 02-findings-ledger.png) — map them to catalog prose like every ledger status already is.
• Sidebar says "Auto-Improve" (page_label) while the page header says "Auto-Improvement" — pick one name.

[UX-REVIEWED] d538eb9

[github-actions]

Design Review (Fable 5) — 🟡 CONCERNS

Advisory design-level review of d538eb9d924aaf1b37d1a581a635170efbf58be5 — updated in place on each push; does not block merge.

Both design reviews are in. I have what I need to render the verdict.

Design-Verdict: CONCERNS

Sound, principled core — measurement-before-change is the right answer to "an unsupervised editor can't prove it helped" — but the marketed safety guarantees are weaker than stated and a large security-critical slice ships unexercised.

Watch

• Headline safety claims overstate what's enforced. "Push mechanically disabled (both remote URLs neutralized)" is agent-rewritable state: with an unattended shell the agent can git config remote.origin.url … or git remote add, so the only thing actually gating egress is the git push denylist string. That denylist is a serially-evaded enumeration (the code documents ~6 historical bypasses) — fine as defense-in-depth, but it is the sole gate on push/publish, and per the PR's own admission the network is not isolated, so a nested interpreter socket walks around it. The enforceable guarantee ("nothing escapes unreviewed") is narrower than the README/PR sell; a user pointing this at a repo they care about is relying on the stronger framing. Reframe the claims to match what the single-user OS-boundary + gh-identity confinement actually deliver.
• Perf track + F10 direct-commit ship security-critical code that "has never kept a measured win on a real repository" (driver.py:1015). Unwipeable git-history pushes, rebase-retry, per-winner-reset limitation (perf PR refactor: rename project KiroClaw to KiroCrew #2 carries PR test: validate CI workflows on KiroCrew #1's commit) — all unexercised. That's where the residual risk concentrates.
• Advisory-canary mode (canaryStrict=false) files perf PRs against a band the tool never proved it can measure — ruler_proven reaches only the PR description, not the accept predicate. The "prove the ruler first" thesis doesn't hold in that mode.

Suggestions

• Land the proven half — bug-track RED→GREEN→STAYGREEN draft PRs — and defer the perf + direct-commit machinery to a follow-up until a real perf win exists to exercise it. Shrinks the unexercised security surface landing in one 41k-line drop.
• The driver is documented cron/tmux-restartable but has no cross-process lock on the single shared clone / ledger.jsonl; a double-launch races checkout/apply/reset --hard. Add a process-level lock or state the single-writer assumption.

[DESIGN-REVIEWED] d538eb9

[github-actions]

Opus 5 Review — ✅ no blocking findings

Reviewed d538eb9d924aaf1b37d1a581a635170efbf58be5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d538eb9

Verdict parsed from the review's SHA-scoped output markers for commit d538eb9d924aaf1b37d1a581a635170efbf58be5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d538eb9d924aaf1b37d1a581a635170efbf58be5: <one-sentence reason>

[Zedmor]

CI status on b150cf6f — 48 checks green; 2 remaining are infrastructure flakes, not code

All substantive gates pass on the current head: every Backend shard (Linux 3.10/3.12 ×4 each, Windows ×4, namespace sandbox), Coverage Combine + Coverage Gate, Frontend Tests/Lint, Electron, macOS Gateway, both Desktop builds, Wheel, GPT 5.6 Review ✅, CodeQL, SAST, Docs Lint, and all brand/inclusive/scrub/dependency gates.

Every blocking review finding from this review cycle has been fixed with a RED-verified regression test (git-safety linked-worktree common gitdir; bare-& denylist bypass; clone_setup/proposer._capture_diff fail-closed attribute pins; copytree(symlinks=True) credential-symlink deref; canary refusal for a custom benchmarkCommand). The branch is rebased clean onto current main.

Two non-passing checks remain, both non-code and not fixable from this PR:

1. E2E (stub ACP backend) — failed 4 consecutive runs on a rotating set of surfaces, none in this PR's diff, while main's own E2E is green:

   • i18n [vs-base] render flake on artifacts.layout / crew-companion.text / settings-voice.latent — counts differ every run (6→4→2→0) and the gate PASSED outright on 2 of the runs. Verified locally by rendering both HEAD and base trees: these surfaces are identical (layout=0, text=2) — a sub-pixel 1.9x-budget / positional-identity race.
   • playwright/ops-mission-control.spec.ts:506 › Settings view switching (main's app, not touched here) — a test.describe.serial SegmentedControl dropdown/board-width race (documented in its own comments as coupled to shared-gateway incident state under workers:1) times out after retry2 and skips its 13 dependents, tripping MAX_SKIPPED_SPECS=0.

   Recommend quarantining the flaky ops serial spec; no auto-improvement change can make this gate deterministic.

2. Opus 5 Review — both attempts on b150cf6f hung ~80 min and registered failure without posting a SHA-scoped verdict (provider timeout, not a finding). Its last actual finding (on e9ee0ab2) is already fixed here, and GPT 5.6 Review passed on this exact commit. Recommend re-running the Opus workflow.

The only genuine remaining gate is human review approval (REVIEW_REQUIRED).

[Zedmor]

CI status update on bbf981be — all auto-improvement findings fixed; remaining reds are inherited from main + a flaky suite

Since the last update, every blocking AI-review finding on this PR has been fixed with a RED-verified regression test:

• GPT — git-safety linked-worktree common gitdir; bare-& and command-substitution ) denylist bypasses; copytree(symlinks=True) credential-symlink deref.
• Opus — clone_setup + proposer._capture_diff fail-closed attribute pins (+ the structural guard now covers inline git call sites); the benchmarkCommand canary refusal.
• De-Amazon Scrub — rephrased the 7 "ARCC" references flagged after fix: remove internal arcc references from the public tree #1495 added it to the denylist.
• E2E ops-spec cascade — the D-124 nav-label fix had used title, which collided with ops-mission-control.spec.ts's page-wide getByTitle('Settings'/…); switched to aria-label (the render gate accepts either). This dropped the ops skips from 13 → 2.

Three remaining reds, none fixable from this PR's scope:

1. Backend Tests shard 2 (3.10, 3.12, Windows) — a module-level path constant _REAL_STATE_DIR = routes._state_dir() at src/kiro_crew/apps/builtins/spec_builder/tests/test_routes.py:88 violates the repo's test_no_module_level_path_constants contract. This is from feat(apps): Spec Builder builtin — spec-driven development surface #518 (Spec Builder), just merged to main — main's own CI (af8774b8) is red on the identical shards. Not in this PR's diff; will clear when main fixes Spec Builder and this branch rebases.
2. E2E — the i18n [vs-base] render gate now PASSES; the residual failures are 2 ops-mission-control.spec.ts specs (:631 provider-list render, :797 segmented-control dropdown-mount race) in a describe.serial block that its own comments document as timing-fragile under CI's workers:1. Not the auto-improvement app; main's E2E passes when timing lines up.
3. Human review approval (REVIEW_REQUIRED).

Everything else is green (all other backend shards, coverage, frontend, builds, CodeQL/SAST, all lint/hygiene gates, GPT review). Recommend fixing Spec Builder on main (it's blocking every open PR) and quarantining the flaky ops serial specs.

[Zedmor]

✅ All automated checks green on ba712e3b

Every CI check and both AI-review gates now pass on the current head:

• 50 checks green, 0 failing — all Backend shards (Linux 3.10/3.12 ×4, Windows ×4, namespace sandbox), Coverage Combine + Gate, Frontend Tests/Lint, Electron, macOS Gateway, Desktop + Wheel builds, CodeQL, SAST, Docs Lint, De-Amazon Scrub, Brand/Inclusive/Dependency gates.
• E2E (stub ACP backend) — pass.
• GPT 5.6 Review — pass (non-blocking verdict on ba712e3b).
• Opus 5 Review — pass (non-blocking verdict on ba712e3b).

Findings resolved since the last update, each RED-verified:

• GPT — session-key collision (agentSession.ts): sessionKey built the key from a lossy safeSegment, so team/service-api and team-service/api aliased to one key and cross-resumed each other's chat. Fixed with an injective keySegment (readable prefix + FNV-1a fingerprint of the raw value), within the backend key validator's charset.
• E2E ops-spec cascade — the earlier title→aria-label nav-label fix removed the getByTitle collision (ops skips 13→0); the residual E2E reds were the known nondeterministic i18n render flake, cleared by a rerun.
• Backend shard 2 — was red on spec_builder's module-level path constant from feat(apps): Spec Builder builtin — spec-driven development surface #518; cleared by rebasing onto main after fix(spec-builder): capture the guarded state dir on first use, not at import #1543 fixed it.

The only remaining gate is human review approval (REVIEW_REQUIRED).