Skip to content

feat: add sqlite run coordinator shadow - #5279

Open
kyleseaman wants to merge 1 commit into
feat/run-coordinator-boundariesfrom
feat/run-coordinator-shadow
Open

feat: add sqlite run coordinator shadow#5279
kyleseaman wants to merge 1 commit into
feat/run-coordinator-boundariesfrom
feat/run-coordinator-shadow

Conversation

@kyleseaman

@kyleseaman kyleseaman commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Stacked change: PR 3 of 7

Stack: #5277#5278#5279#5280#5281#5282#5283
Base: #5278
Next: #5280

Problem / Motivation

The coordinator contract needs a durable implementation and production parity evidence before it can become authoritative.

Why it matters

An explicitly injectable, fail-contained shadow adapter exposes semantic mismatches without placing working subagent execution at risk. The production seam remains disabled until durable command authority and recovery coverage land in later stack layers.

What changed (motivation → approach → change)

  • Adds a SQLite coordinator with transactional migrations, WAL durability, strict file-link checks, and owner-only storage.
  • Anchors coordinator storage outside agent-writable prefixes, derives linked override identity from the validated raw spelling, creates and owner-tightens the stable anchor before Linux constructs its mask, denies both lexical and canonical ledger and anchor aliases in macOS Seatbelt, masks canonical and relocated homes from every supported OS sandbox path, rejects any existing anchor that does not match the currently resolved ledger, including after a linked home becomes an ordinary directory, and refuses unconfined execution after a linked data home is retargeted away from the persisted ledger.
  • Validates known database-file ACLs inside the mutation transaction before commit, so a fail-loud permission error rolls the transition back instead of reporting failure after durable state changed.
  • Routes coordinator I/O through a dedicated bounded executor.
  • Adds a shadow coordinator that mirrors mutations, records parity mismatches, and cannot change the legacy result.
  • Bounds shadow submission so an unavailable injected coordinator cannot stall already accepted legacy execution in this parity-only layer.
  • Leaves the production coordinator disabled in this pre-authority layer; explicit injection exercises parity without retaining plaintext task payloads by default.
  • Makes shared-runtime integration tests wait on completion state instead of a fixed delay, keeping slow Windows CI deterministic.

Tests

  • test/test_run_coordinator_sqlite.py
  • test/test_run_coordinator_shadow.py
  • test/test_executors.py
  • Coordinator wiring, bounded shadow submission, opt-in staging, fail-closed linked-home replacement, lexical and canonical ledger masking, retargeted-home fail-closed behavior, and security tests

Manual verification

N/A — persistence, migration, security, timeout, and parity behavior are covered by automated tests.

Related Issues

No linked issue: this stack implements the locally reviewed durable run coordinator RFC.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 790e8b10500261d15971afef62bd12e5a140d0d9 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/sandbox.py:1114 -- "runs on the event loop" contradicts wrap_argv’s event-loop guard -> Fix: state that async callers offload this check.
[GPT-REVIEWED] 790e8b1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 790e8b10500261d15971afef62bd12e5a140d0d9: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 23, 2026
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 790e8b10500261d15971afef62bd12e5a140d0d9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound staged migration, but the fail-closed sandbox costs land on users now while the ledger they protect ships unwired.

Watch

  • Availability before benefit. _run_coordinator_uses_custom_home() makes any non-default KIROCREW_HOME refuse Windows/macOS internal-sandbox delegation ("Refusing to start the agent rather than expose durable coordinator state"), yet in production SubagentManager._coordinator defaults to None and nothing constructs SQLiteRunCoordinator — relocated-home Windows operators lose every kiro-cli spawn on upgrade to protect a directory nothing writes this release. Note the collision: --approval yolo requires a non-default KIROCREW_HOME, so Windows yolo hosts hit this unconditionally. The pre-seed-window rationale justifies the sensitive-floor and hidden-dir entries now, but the delegation refusal could gate on the coordinator actually being enabled, or ship with the wiring PR.
  • No recovery path for legitimate retargeting. Once anchored, a genuinely intended data-home link move makes run_coordinator_anchor_matches_current_home() false permanently; _refuse_retargeted_run_coordinator_unconfined then denies every off/no-backend spawn and neither the error message nor the docs name an operator remedy for migrating the ledger to the new target.

Suggestions

  • Split the black-baseline reformat of security.py into its own commit per AGENTS.md — ~100 formatting hunks currently bury the two substantive security-floor lines in the file that most needs a clean audit diff.

[DESIGN-REVIEWED] 790e8b1

@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-shadow branch from 5a37ff3 to 3cee769 Compare August 23, 2026 12:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 23, 2026
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 790e8b10500261d15971afef62bd12e5a140d0d9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All evidence gathered. Here is the review.

First-Principles-Verdict: CONCERNS

Two unrelated frontend fixes and a half-done reformat ride along undeclared in a security-heavy coordinator PR whose declared items are all RFC-derived.

What this change ships

Intent: give the run-coordinator contract a durable, injectable SQLite implementation plus parity evidence, without touching legacy execution authority. ADDITION (stack PR 3/7 of the recorded rfc-durable-run-coordinator).

  1. Durable SQLite coordinator store, injectable only — justified (RFC-recorded stack layer)
  2. Primary-preserving shadow adapter recording parity mismatches — justified
  3. Manager mirrors accepted runs into an injected coordinator; default coordinator now None — justified, declared
  4. Dedicated two-worker coordinator_executor bulkhead — justified (1 consumer, named starvation harm)
  5. Commands persist payload_json; schema v2 migration — justified
  6. Persistent anchor dir ~/.kirocrew.run-coordinator pinning canonical ledger identity — justified (agent-vs-ceiling boundary)
  7. Ledger on the sensitive floor and hidden in every OS sandbox tier, with pinned ancestors — justified (keystone invariant)
  8. Relocated/linked ledger fails Kiro-delegation and unconfined spawns closed — justified; replaces a fallback the nesting-EPERM invariant made broken anyway
  9. Theme-layer unmount/audio-stop on remote-Crew switch + iframe height-reporter load re-post — rides along, undeclared
  10. Darwin pass_fds test, ~150 lines of pure reformat in baseline-exempt security.py, test_session_sharing.py reformat — rides along, undeclared

Watch

  • Items 9–10 appear nowhere in the description ("What changed" lists only coordinator/sandbox/test-timing work). They are real fixes, but a reviewer of "sqlite run coordinator shadow" cannot know website/src/components/ThemeExperienceLayer.tsx and widgetSrcdoc.ts behavior ships here.
  • on_mismatch on ShadowRunCoordinator has zero non-test consumers (grepped on_mismatch across src/: only shadow.py defines it; all constructions are in tests). The logger.warning beside it already records the same fields.

Subtractions

  • Drop the pure-reformat hunks in src/kiro_crew/security.py (e.g. the _DATA_CONSUMER_PROGRAMS re-wrap): the file is in .github/black-baseline.txt, which AGENTS.md says to format only in its own commit with a baseline prune — this diff does neither, leaving a half-formatted baseline file.
  • Drop the on_mismatch constructor parameter from ShadowRunCoordinator (0 production consumers); reintroduce it in the stack layer that wires a metrics sink.
  • Defer the website/ and test_acp_client.py riders to their own PRs.

[FIRST-PRINCIPLES-REVIEWED] 790e8b1

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 790e8b10500261d15971afef62bd12e5a140d0d9 — this comment is updated in place on each push.

Review details

No findings.

FINDING — src/kiro_crew/subagent.py:2409 — parity check ("run.task", run.task == raw_task) compares the coordinator's stored value (which is the redacted/truncated stored_task, since SubmitRun(task=stored_task)RunRecord.task = request.task) against the unredacted raw_task, so any task carrying a credential, a UTF-8 surrogate, or >1000 chars falsely emits run coordinator legacy mismatch at boundary=submit fields=run.task — precisely the sensitive tasks the shadow exists to validate → Fix: compare run.task == stored_task.

[OPUS-REVIEWED] 790e8b1

Verdict parsed from the review's SHA-scoped output markers for commit 790e8b10500261d15971afef62bd12e5a140d0d9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 790e8b10500261d15971afef62bd12e5a140d0d9: <one-sentence reason>

@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-shadow branch from 3cee769 to 8ac7451 Compare August 23, 2026 13:57
@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-shadow branch from 8ac7451 to c1bf100 Compare August 23, 2026 14:54
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Aug 23, 2026
@iamwhatever
iamwhatever force-pushed the feat/run-coordinator-shadow branch from c1bf100 to 777f0a4 Compare August 23, 2026 17:33
@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-shadow branch from 777f0a4 to c1bf100 Compare August 23, 2026 17:36
@kyleseaman
kyleseaman force-pushed the feat/run-coordinator-shadow branch from c1bf100 to fed76a5 Compare August 23, 2026 17:47
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Aug 25, 2026
bolichen97
bolichen97 previously approved these changes Aug 26, 2026
bolichen97
bolichen97 previously approved these changes Aug 27, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

bolichen97
bolichen97 previously approved these changes Aug 28, 2026
bolichen97
bolichen97 previously approved these changes Aug 28, 2026
@kyleseaman

Copy link
Copy Markdown
Collaborator Author

span=84e8c2158bc5 — Fixed in 7f415e3a42f1de2aaf668cd05774c0a454683e43. The final owner-only ACL validation now runs before commit inside the mutation transaction, so a fail-loud ACL error rolls the transition back instead of reporting failure after durable state changed. A regression injects the final ACL failure and proves a fresh coordinator cannot observe the run.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=fa7ab943e399

Relocated coordinator storage escapes the OS sandbox.

Evidence: coordinator storage now resolves through the trusted anchor in run_coordinator_anchor.py; sandbox masking covers canonical, linked, and relocated homes, with focused dynamic-path and security regressions passing.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=81362a8802e7

    A persisted canonical ledger drops out of the hook-layer sensitive paths
    Fixed in 2509ca095: every route that would execute without an OS sandbox now compares the persisted canonical ledger with the ledger under the currently resolved KIROCREW_HOME and fails closed when a retarget makes them differ. Regression coverage exercises both explicit sandbox=off and the no-backend opt-in, while sandboxed access and durable restart behavior remain intact.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=81362a8802e7

    Symlinked overrides bypass durable ledger anchoring
    Fixed in 6c3c930d6: canonical coordinator path selection now preserves the validated raw KIROCREW_HOME spelling instead of starting from data_home()’s resolved target, so a linked override creates and reuses its durable external anchor across a retarget and gateway restart. The regression uses the real config resolver and proves the original ledger remains canonical.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: SQLite run coordinator + shadow parity

Reviewed origin/feat/run-coordinator-boundaries...origin/feat/run-coordinator-shadow (22 files, +2085/−143) at b915711c1. Findings verified by executing the real SQLiteRunCoordinator, ShadowRunCoordinator and _shadow_submit_accepted_run — including cross-process runs, injected mid-transaction failures, and measured latency curves.

Two framing problems worth stating before the list. First, the shadow's job is to collect parity data without risking the primary, and it can destroy the primary's committed decision in three separate ways. Second, the whole apparatus has no production caller: grep -rn 'SQLiteRunCoordinator|ShadowRunCoordinator' src/ outside run_coordinator/ returns nothing, slack/gateway.py:7875 builds SubagentManager with 16 kwargs and no coordinator=, and the RFC's mandated three-value rollout mode (legacy/shadow/coordinator, §5.9, with "rollback… is supported") exists nowhere — grep -rni 'coordinator_mode|rollout_mode' src/kiro_crew/config/ config-baseline.json is empty. So the feature is simultaneously un-gated and unreachable: it collects zero parity data, while any future caller or merge that passes coordinator= turns on durable SQLite writes on the spawn hot path with no config gate and no kill switch.

Blocking

1. run_coordinator/shadow.py:117except Exception does not contain BaseException, so a cancelled shadow leg discards the primary's already-committed decision — and at the _run_impl seam that lands in except asyncio.CancelledError and respawns an accepted run whose _run_inner never started. shutdown_maintenance_executor() (atexit-registered ×10, exported, called by tests) runs _coordinator_pool.shutdown(wait=False, cancel_futures=True), so a queued _offload future makes wrap_future raise CancelledError in a task that was never cancelled. It passes through _mirror's except Exception and subagent.py:2293's, and reaches subagent_manager/run.py:330. There info.tool_count == 0 and _cancel_retry_used is False (the shadow await precedes _run_inner), so the side-effect gate passes and _schedule_cancel_recovery(info) fires. Executed: _run_inner ever awaited: 0 / _schedule_cancel_recovery called (duplicate respawn): 1, and independently caller task was NEVER cancelled, yet await raised CancelledError / primary already APPLIED the submit: True. KeyboardInterrupt, SystemExit and GeneratorExit escape identically. This directly contradicts the spec's "an unhealthy shadow cannot fail an accepted legacy run"; the PR's only containment test injects RuntimeError.

2. shadow.py:120fields = _mismatch_fields(primary, shadow) sits OUTSIDE the try that guards the shadow call, so any failure while comparing a non-conforming shadow value propagates and destroys the primary's committed result. Five executed crash classes, every one with primary row committed = True (the mutation happened, only the decision was lost): an object whose __eq__ raises → ValueError; a dataclass whose field read raises → OSError; a dataclass with different fields → OSError; a nested list of depth 2000 → RecursionError (collect bounds itself on field count via _MAX_MISMATCH_FIELDS=8 but never on depth); a dict with non-comparable keys → TypeError from sorted(left.keys() | right.keys()). An on_mismatch observer raising BaseException escapes the same way (:130 is also except Exception). This is precisely the threat model subagent.py:2351 claims to handle: "Keep this phase primary-preserving even when an integration violates it."

3. sqlite.py:120 — migration v2's ALTER TABLE commands ADD COLUMN payload_json is not idempotent and _schema_version() returns 0 for a DB that HAS the physical schema, so any lost or corrupt metadata value permanently bricks the store on every call — including read-only get_run — with no recovery path. Executed against a healthy v2 DB, twice each so the brick is permanent rather than transient: schema_version='1'OperationalError: duplicate column name: payload_json; row DELETED → same; metadata table DROPPED → same; schema_version='-1' → same. Second mechanism at :227 (return int(row[0]) if row is not None else 0): 'corrupt', '', '2.9' each raise an unhandled ValueError: invalid literal for int() out of _connect forever (the except BaseException only closes and re-raises), while '02' and '2 ' are silently accepted so the downgrade guard's notion of "newer" is whatever int() parses. Violates the RFC's own exit criterion "schema creation and upgrade are crash-safe and idempotent" — and note migration atomicity does hold (an injected bad statement rolls back cleanly, verified), so it is replay that is broken.

4. subagent.py:2308raw_task = info._raw_task or info.task is the first site in the repo to write the deliberately UNREDACTED subagent prompt to disk, and it lands in two columns of a store nothing ever prunes. info.task is redact_credentials(redact_exfiltration_urls(task)[0])[0]; info._raw_task is documented at subagent.py:1309 as "unredacted task for kiro-cli execution prompt" and previously had exactly one reader (run.py:746, in-memory prompt build, never persisted). Executed: a task containing AKIAIOSFODNN7EXAMPLE + an AWS secret key + an exfil URL is stored verbatim in both runs.task and commands.payload_json, and all three needles are present in the raw coordinator.db bytes — while the redacted form the rest of the codebase persists is deploy using [REDACTED: credential] / [REDACTED: credential]. Every other persistence/emit site uses the redacted copy (admission.py:763, run.py:847, terminal.py:115, monitoring.py:745, all SEL metadata). grep -rn 'DELETE|prune|retention|vacuum' src/kiro_crew/run_coordinator/ finds only the three full-table wipes inside _save_memory, so unlike result.txt (which has agent.subagent_result_ttl_secs and a reaper) the plaintext accumulates for the life of the install.

5. sqlite.py:150_database_path checks only the leaf and its immediate parent, then resolve()s through any ancestor link, so with the spec-supported symlinked data home the physical DB sits at a path is_sensitive_path does not match. Executed: with ~/.kiro/crew~/elsewhere/crew, is_sensitive_path('~/.kiro/crew/run-coordinator/coordinator.db') is True but is_sensitive_path('~/elsewhere/crew/…') is False, and is_sensitive_bash_command("sqlite3 ~/elsewhere/crew/run-coordinator/coordinator.db 'select task from runs'")blocked=False (the logical spelling is blocked). So the unredacted task payloads of finding 4 are readable through an ungated spelling. Also executed with a tmp-dir ancestor symlink: submit -> applied, DB physically created inside the attacker-controlled target, no refusal. platform_compat.first_linked_ancestor exists for exactly this ("a caller that checks only the path it was handed still resolves through a linked PARENT"; on Windows an ancestor junction to \\host\share turns the first is_dir() into an authenticated SMB call) and is already paired with is_link_or_junction at clone_setup.py:141 and themes.py:242 — it is not used here. Related: is_link_or_junction cannot see hardlinks; executed, a hardlinked coordinator.db and a hardlinked -wal are both accepted and SQLite writes plaintext task text through them ('SECOND-SECRET-TASK' readable through hardlink: True, 32,992 bytes of WAL frames in the attacker inode).

6. sqlite.py:173 — the -journal sidecar that _database_files explicitly enumerates is the one file _secure_existing_database_files can never secure, and PRAGMA journal_mode=WAL's result is discarded. Executed over 60 submits with umask 022: coordinator.db-journal ['0o644'], journal EVER 0o600? False (db, -wal and -shm are all 0o600). The rollback journal is created by the WAL header write, lives at umask perms for its whole lifetime, and is deleted before any chmod pass sees it — so 0o644 pre-images of every task payload hit disk. Separately executed: with a foreign connection holding a read transaction on a not-yet-WAL DB, PRAGMA journal_mode=WAL raised: OperationalError database is locked immediately, well inside the 5s busy timeout — the code's own comment says this pragma "does not reliably honor busy_timeout", yet _JOURNAL_MODE_LOCK is a threading.Lock that serialises only threads in one interpreter, the return value is never inspected, and nothing retries. Both contradict the spec added in this commit ("The directory and database/known sidecars are tightened owner-only"; "Concurrent initialization serializes the WAL-mode transition").

7. sqlite.py:341 — a task containing a lone surrogate is accepted by the in-memory oracle but makes every SQLite mutation raise UnicodeEncodeError, which the shadow swallows, so the run is silently absent from the durable store forever. Executed end-to-end through the real manager entry point: json.loads (of an MCP/ACP tool argument containing a \udXXX escape) gives info._raw_task = 'summarise \ud83d the thread' — a legal Python str, not UTF-8 encodable. MemoryRunCoordinator.submit → applied; _save_memory's INSERT INTO runs binding run.taskUnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d'_invoke rolls back → _mirror's except Exception swallows it → primary APPLIED returned, legacy run proceeds. Result: clean-run durable row -> PRESENT / surrogate-run durable row -> MISSING, and every exact retry fails identically. commands.payload_json is unaffected because json.dumps defaults to ensure_ascii=True, so the hash column is writable while runs.task is not. The contract suite is now parametrised over memory and sqlite yet covers no non-encodable text (NUL bytes round-trip fine, so surrogates are the specific gap).

Should fix

8. sqlite.py:413 — every operation loads all three tables into a throwaway MemoryRunCoordinator, DELETEs and re-INSERTs every row, and runs a whole-database PRAGMA quick_check — under BEGIN IMMEDIATE even for the read-only get_run — so cost is O(total lifetime runs) and nothing ever prunes. Measured single-submit latency: 2.5–5.2 ms at N=0 · 17–19 ms at N=1000 · 90–107 ms at N=5000 · 391–1500 ms at N=20000 (139 MiB rewritten under synchronous=FULL for one new row) · 1025 ms at N=50000. Write amplification at N=5000/10 KB tasks: 20,006 row mutations and 52,245,499 bytes re-serialised to persist 2 useful rows; tracemalloc peak 55.1 MB per call, ×2 workers. get_run (persist=False) still does all of it: 6.07 ms @ N=1 → 43 ms @ N=5000 → 479 ms @ N=50000, vs 0.005 ms for SELECT * FROM runs WHERE run_id=? (up to 16,114×). And with one other writer holding the lock: plain WAL read 0.4 ms OK vs coordinator get_run after 5.20 s -> OperationalError: database is locked — the exact property WAL was enabled for. Consequence: submit crosses _SHADOW_SUBMIT_TIMEOUT_SECS = 1.0 with zero contention around 15k–50k lifetime runs, after which every spawn stalls a full second at run.py:319 and the shadow silently records nothing. The five declared indexes and the version OCC column are never used — grep 'WHERE' over sqlite.py hits only the two metadata lookups.

9. subagent.py:2289 — the 1.0 s asyncio.wait_for bounds only the caller, not the offloaded SQLite work, so under contention started writes commit AFTER the caller logged "shadow submission failed" while queued writes are cancelled and never mirrored — both emitting the identical warning. Executed with an external writer holding the lock and 6 concurrent shadow submits on the 2-worker pool: all six callers released at exactly 1.001 s with TimeoutError, pool queue still 4 deep, and after the lock cleared rows actually persisted: ['warm']6 of 6 mirrors lost. A second run split the other way: committed anyway AFTER the caller reported failure: ['run-1','run-2'] / silently DROPPED, never mirrored: ['run-3','run-4','run-5'], the late commits landing at t=10.3 s. Measured capacity: 18.3 submits/s at N=5000/64 B (first timeout at concurrent spawn #19; burst 16 → 3 timeouts); at 10 KB tasks 2.4/s (burst 4 → 2 timeouts, burst 16 → 14). So the parity dataset this PR exists to collect is non-deterministically partial in both directions and the log cannot distinguish them. executors.py already has the two-phase queued-vs-claimed discipline (run_in_cron_pool) that would bound this; the coordinator path does not use it. The PR's own test patches the timeout to 0, so it proves nothing about the real budget.

10. sqlite.py:333runs and outbox are written with column-less positional INSERTs (18 and 14 placeholders) while commands names its 13 columns, so the next ADD COLUMN migration — the only shape SQLite's ALTER supports, and exactly what this PR's own v2 migration does — wedges every write while reads keep working. Executed: append (3, ("ALTER TABLE runs ADD COLUMN cancel_reason TEXT NOT NULL DEFAULT ''",))v2 submit: applied → migration commits → v3 submit FAILS: OperationalError table runs has 19 columns but 18 values were suppliedread still works: Trueretry FAILS identically. Because _shadow_submit_accepted_run swallows everything, the only symptom is one warning per spawn while the store looks alive and accepts nothing. The inconsistency is inside one function: commands was given an explicit column list precisely because v2 appended a column to it, and the lesson was not applied to the other two tables. The same construct also means reordering RunRecord's frozen-dataclass fields silently writes values into the wrong columns — the tables are not STRICT, so a TEXT stored in lease_expires_at REAL raises nothing.

11. shadow.py:29 — the parity oracle is structurally blind in four ways at once, so a shadow with broken leases, a dropped mutation, or a wrong terminal instant compares CLEAN while the delivery path reports a permanent false mismatch — and there is no counter, so the migration gate cannot be read at all. (a) _VOLATILE_FIELDS excludes lease_expires_at/claim_expires_at/terminal_at, which are caller-supplied (memory.py:173/305/346/372), not clock-derived — executed: shadow lease 30 → 1e9 (never expires) = CLEAN, terminal_at None → 999999 = CLEAN, while version/attempt/lease_epoch/owner_id/outcome/error/result_path are caught. (b) The REJECTED early-return at :108 skips a decision class that does mutate: submit(accepted=False) writes a TERMINAL run + a REJECTED command and then returns REJECTED — executed: primary: runs=['run-A'] commands=['cmd-A'] / shadow: runs=[] commands=[] / mismatches reported = [], and a resubmit then gets primary=identity_conflict vs shadow=applied, still silent. (c) event_id masking hides a guaranteed outbox-identity divergence (two independent uuid4s) and then every claim_outbox/mark_delivered reports decision,reason,value forever, because the primary's fence id does not exist in the shadow. (d) _mirror has no timeout on any of the 10 boundaries (grep -c 'wait_for|timeout' shadow.py = 0) — a 10 s shadow returns the primary's already-computed decision after 10.008 s; only submit has a call-site bound. RFC §8 requires "shadow parity mismatch by field class" as a metric; grep -rn 'metric|counter|gauge' src/kiro_crew/run_coordinator/ finds nothing and on_mismatch has no production caller.

12. subagent.py:2311 — the "canonical execution payload" omits approval_mode, requested_model and resolved_model, so two submissions differing only in whether the run gets blanket tool auto-approval produce an identical payload_hash and are accepted as an idempotent replay instead of an idempotency conflict. approval_mode='auto' grants blanket tool auto-approval in _run_inner_impl and emits a subagent.approval_mode_auto_policy SEL record. Executed against the real _shadow_submit_accepted_run: approval_mode='' payload_hash=c7fc09689ce53e75 and approval_mode='auto' payload_hash=c7fc09689ce53e75identical hash for different approval policy. memory.submit's only conflict test is payload_hash != request.payload_hash, so a resubmit under the same key that flips the approval policy returns unchanged/idempotent_replay and the durable command keeps the first request's semantics. This also breaks the purpose the spec states for storing the payload ("a claimed command can be reconstructed after restart"): a restart replay drops the approval policy and re-resolves the model unpinned. Nothing anywhere validates sha256(payload_json) == payload_hash, and SubmitRun.payload_json defaults to "" while payload_hash is mandatory, so a hash-with-no-payload row is representable — the PR's own test_run_coordinator_shadow.py::_request constructs exactly that.

13. memory.py:333renew() never checks observed_state, so the fencing lease on an already-TERMINAL run can be extended to any future timestamp, and SQLite persists it — permanently disabling claim_commands' expired-lease reclaim arm for that run. After complete() the run is TERMINAL but owner_id, lease_epoch and lease_expires_at are left intact. The old owner's heartbeat — which has no reason to know it lost, since finding 1 of the #5277 review shows its completion was reported as a clean replay — calls renew('run-1', RunFence('run-1','gw',1), until=9999). All four guards pass. Executed on both adapters: renew(TERMINAL run) -> True lease now: 9999.0 observed: terminal, durable under SQLite. claim_commands' only recovery arm for a wedged command is status is CLAIMED and run.lease_expires_at <= now, which can now never fire, so a fenced-out worker holds a live renewable fence on a finished run indefinitely. Every other transition gates on _STARTABLE_STATES/_COMPLETABLE_STATES; renew is the exception.

14. memory.py:271complete()'s outbox-replay short-circuit returns BEFORE _validate_transition, so it is the only boundary that never checks the fence, lease epoch, lease expiry or expected_version — and it hands the caller the terminal OutboxEvent including its payload_json. Executed on both adapters: complete(expired lease + wrong version) -> unchanged completion_replay | leaked payload: {"summary":"SECRET RESULT"}. Every other transition validates the fence first. SQLite now makes the terminal row survive restarts, so the window is unbounded rather than process-local, and once PRs 5–7 wire terminal delivery two owners both get a success-shaped answer plus the delivery event. (Same root cause as #5277's finding 1; flagged here because this PR is what makes it durable.)

15. sqlite.py:198_prepare_path re-mkdirs and re-chmods the ledger directory on every transaction, and _secure_existing_database_files runs twice per call. Measured per coordinator transaction: 1 mkdir + 7 chmod + 10 lstat + 9 stat. A keyed spawn is 4 transactions, so ~4 mkdir + 28 chmod + ~76 stat/lstat syscalls per spawn on the data-home mount, for a path whose identity is already memoised in _resolved_path and in run_coordinator_anchor._anchor_cache. _invoke also spins up and tears down a brand-new asyncio event loop per transaction (asyncio.run(...) inside a worker thread) just to drive an uncontended in-memory lock — measured 0.174 ms/call, ~0.7 ms of pure loop churn per keyed spawn plus two fds created and destroyed. Cheaper: harden the path once per resolved path behind the same _path_resolution_lock that already caches it (exactly as prime_voice_runtime_sandbox_paths does for the voice runtime), drop the duplicate pre-commit securing, and drive the coroutine directly.

Below the cap (all verified)

_SCHEMA_V1 ships the RFC's nullable owner/lease columns as NOT NULL with ""/0.0 sentinels, so "unowned" is inexpressible in SQL · complete()'s replay-equality omits terminal_at, silently keeping the first value · run_id = uuid4().hex[:8] (32 bits, uniqueness contracted only among live agents) becomes a permanent PRIMARY KEY in a never-pruned table · claim_commands/complete raise bare KeyError on a partially-corrupt store · run_coordinator/__init__.py eagerly imports .sqlite, adding 15.4 ms and executors/platform_compat/ctypes/windows_acl to every consumer of the pure-stdlib contract · _claim_commands sorts and scans the entire command table even when a single command_id is requested (0.069 ms at 201 → 1.706 ms at 10,001, ~25× for identical work), and renew/complete materialise tuple(self._commands.items()) to touch one run's rows · three test defects: test_default_manager_does_not_retain_shadow_runs only re-asserts _coordinator is None, test_vanished_sqlite_sidecar_does_not_break_concurrent_preflight has no assert and contains latent unbounded recursion, and test_existing_database_files_are_restricted_before_sqlite_opens monkeypatches process-global sqlite3.connect with an asserting wrapper · conventions: the RFC's last-audited/audited-at/implementation-prs front matter is left stale against docs/request-for-change/README.md:160-162, and ~25 hand-applied (not black-produced) reformat hunks in security.py plus ~91 in test_session_sharing.py are bundled into a feature commit against AGENTS.md:358-361 · cleanup: 149 lines of hand-written row↔dataclass mapping reaching into five MemoryRunCoordinator privates with no snapshot/restore API; a 4th open-coded sidecar list that already disagrees with three siblings; a 10th verbatim lazy-pool copy with atexit.register per pool; a duplicate parity comparator in subagent.py; _refuse_newer_schema duplicated with the non-atomic outer copy dead; 10 cast() forwarders; -journal/attempt dead columns.

Note for CI: scripts/check_black_formatting.py fails on test/test_run_coordinator_sqlite.py, which this PR introduces and which is not in .github/black-baseline.txt. That makes the gate red for every PR from here up the stack — #5283's review hit the same failure independently.

Verified clean, so nobody needs to re-check

Migration atomicity genuinely rolls back (tables after failed migration: [], schema_version: ('2',)); the pre-commit ACL failure rolls the transition back (runs table: ['r1'] after injecting an OSError at :424); cross-process serialisation holds (3 processes × 40 submits → 120/120 rows, no clobber); journal_mode=wal, synchronous=FULL, foreign_keys=1, busy_timeout=5000 are all live inside the transaction and the DELETE/INSERT ordering never violates the FK or either UNIQUE; db/-wal/-shm are 0o600 on POSIX; the newer-schema refusal precedes sidecar creation; all 89 touched security.py regexes are byte-identical (only the additive run-coordinator alternations differ) and _data_consumer_exempt matches over 1404 enumerated inputs; the leaf does not over-block (run-coordinator-backup, run-coordinatorX, subagents/run-1/result.txt all fine); coordinator=None causes no None-deref; the new await at run.py:319 is on the correct side of every lifecycle anchor; all 63 coordinator tests pass on 3.12 and 3.10; semgrep/isort/mypy/docs-lint clean; jscpd is website-only; the inline wokeignore is the documented pattern; CHANGELOG absence is correct.


Execution-verified AI-assisted review (Claude Code) across 11 angles plus a gap sweep, run against a local checkout at b915711c1; the working tree is unmodified and nothing was posted elsewhere. Findings name the input that reproduces them — please push back where one misreads intent.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=fa7ab943e399

    An absent anchor directory is not sandboxed.

    Fixed in 5f0994615: the Linux launcher now creates, rejects links around, and fail-loud owner-tightens the stable anchor before constructing its existence-guarded bind masks. The regression failed on the reviewed head and passes on the fixed head; the affected coordinator, sandbox, and delivery suite passes with 197 tests and 2 valid Linux-only skips, together with Black, subprocess, docs, boundary, flake8, and Linux mypy gates.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=32695e9db201

    Launcher tests mutate the operator’s home.

    Launcher construction is now patched by an autouse fixture to use the repository test floor’s per-test isolation directory. A regression asserts that the emitted launcher masks that isolated anchor. The focused three-test proof passed and the real persistent anchor mtime remained unchanged; the full affected run-coordinator/security/delivery slice passed with 198 tests and 2 platform skips.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=fa7ab943e399

    canonical-home alias bypasses anchor protection.

    The macOS Seatbelt profile now emits read, write, hardlink, and literal-write denies for both the lexical stable-anchor path and its canonical realpath. The regression models a symlinked-home alias and requires both spellings; all 41 Seatbelt-profile tests pass.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

Addressed both current-head automated review findings in 5e8863c4f:

  • Existing anchor records are now consulted before the ordinary-directory fast path, so replacing a selected linked home with a directory cannot abandon the durable coordinator ledger.
  • macOS Seatbelt now denies both the configured lexical and persisted canonical ledger spellings, including read, write, hardlink, and rename-sensitive ancestor guards.

Verification: the two regressions passed red-to-green; 409 coordinator tests passed with one valid platform skip; all 41 Seatbelt profile tests passed; Black, subprocess encoding, flake8, Linux mypy, docs-lint, agent-sdk boundary, harness parity, and brand gates passed.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

Disposition for the current-head GPT finding on 5e8863c: fixed in 4d5e174. Existing deterministic anchor records are now accepted only when they match the coordinator path currently resolved from the configured data home; pre-seeded records and linked-home replacement fail closed. Regression coverage exercises both restart retargeting and link-to-directory replacement. Focused verification: 409 coordinator tests passed with 1 valid skip; 41 Seatbelt profile tests passed; formatting, lint, Linux mypy, docs, agent-SDK boundary, harness-parity, and brand gates passed.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=fa7ab943e399

    macOS remediation disables the only sandbox in off mode.

    Fixed in fb75d7056: when Kiro Crew sandbox mode is off, the macOS conflict remedy now requires setting agent.sandbox to "auto" before disabling Kiro CLI’s internal sandbox. The focused regression failed on the reviewed head and passes on the fix; 478 top-layer tests pass with one valid skip, together with Black, subprocess encoding, flake8, Linux mypy, docs, agent-SDK boundary, harness-parity, and brand gates.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=81362a8802e7

    Retargeted linked homes bypass the unconfined-execution refusal

    Fixed in 37822119a. The anchor path helper now uses the live _valid_override_home() result instead of comparing the current symlink target with memoized data_home(); invalid live overrides fail the unconfined proof closed. A regression primes the config memo, retargets the selected link without clearing that memo, and proves the persisted ledger no longer compares with itself. The focused anchor and unconfined-sandbox checks pass.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author
  • Fixed — span=81362a8802e7

    Rejected overrides prevent unconfined agent startup

    Rejected KIROCREW_HOME values now compare the coordinator anchor against the live default-home fallback instead of refusing unconfined startup outright. A regression test covers a filesystem-root override, while the existing retarget test still proves that a live link retarget cannot reuse memoized home state.

@kyleseaman

Copy link
Copy Markdown
Collaborator Author

Bolin review disposition for the current stack

All fifteen numbered items from the September 2 review were rechecked.

  1. Fixed.
  2. Fixed.
  3. Fixed.
  4. Fixed.
  5. Fixed.
  6. Fixed.
  7. Fixed.
  8. Fixed in the recovery layer: legacy imports are batched so a sweep does not persist one full snapshot per folder. The general SQLite adapter remains a bounded, serialized full-snapshot implementation for this stack.
  9. Rebutted: bounded wait_for cancellation is intentional fail-containment for a parity-only shadow; the legacy result remains authoritative.
  10. Fixed.
  11. Fixed.
  12. Fixed.
  13. Fixed.
  14. Fixed.
  15. Rebutted as a security invariant: path preparation is deliberately repeated at mutation boundaries so link and anchor state is revalidated immediately before access.

The fixes include transactional rollback, parity containment, migration and enum hydration coverage, redaction and size bounds, symlink and stable-anchor protection, journaling and ACL hardening, explicit UTF-8 handling, and named inserts.

Current submitted head: 790e8b1.

@bolichen97

Copy link
Copy Markdown
Collaborator

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • PR #5277 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5277: MERGE_DISCUSSION. Extends this PR's port with a second implementation; strictly additive and stacked behind it. Files: src/kiro_crew/run_coordinator/models.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • This is a large PR (~3400 added lines, 30 files) introducing a new subsystem (sqlite run coordinator shadow), which is beyond what the pipeline will safely auto-drive.
  • It is stacked change 3 of 7 with base refactor: extract subagent lifecycle boundaries #5278, not main -- the pipeline rebases onto main, which is the wrong base for a stacked PR, so it cannot mechanically resolve the conflict without reworking the stack.
  • Every CI lane is red (Backend Tests 3.12 shards 2/3, Backend Tests Windows, Coverage Gate, Backend Lint & Type Check, Build Desktop, Frontend Tests), which on a new subsystem this size needs your triage of which are real vs stack/base drift.

Please land or rebase the earlier PRs in the stack (#5277/#5278) and resolve the conflict against the correct base, then the pipeline will re-assess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants