feat(conductor): ship the conductor agent + acceptance-evaluator skill as built-ins - #5652
Conversation
9baebec to
4c8c9f9
Compare
699b17a to
979b4f4
Compare
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @iamwhatever overrides the GPT 5.6 finding for This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound assembly of merged mechanisms, but per-cycle approval prompts undercut the unattended-coordination value the PR is built to deliver. Watch
[DESIGN-REVIEWED] d884a06 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of All claims verified against the repo. Summary of the counts I ran:
First-Principles-Verdict: CONCERNS The evaluator ships as a shell script the conductor must get approved every patrol cycle, when the repo's own MCP-first rule provides the promptless, path-stable spelling. What this change shipsIntent: let one agent own a goal too big for one session — decompose, dispatch sessions, verify, re-plan. ADDITION.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] d884a06 |
4c8c9f9 to
7316ef5
Compare
7316ef5 to
cd445b4
Compare
979b4f4 to
79ee449
Compare
cd445b4 to
06b4403
Compare
|
Head is now
|
|
Head is now
|
|
Advisory
|
|
Advisory
|
79ee449 to
22e2b38
Compare
22e2b38 to
fcd0cde
Compare
5276ebb to
e637b0d
Compare
|
Head is now
|
|
/ai-review override gpt e637b0d: maintainer accepts both as residuals — the planted-gh path needs pre-existing PATH write access and every evaluator call is human-approved, and the file-existence probe yields a boolean whose whole class is removed by moving the evaluator into kirocrew-core (tracked in #5926). |
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
e637b0d to
d884a06
Compare
|
/ai-review override gpt d884a06: maintainer accepts both accept_eval.py findings as residuals — the planted-gh path needs pre-existing PATH write access and every evaluator call is human-approved, and the file-existence probe yields a boolean whose whole class is removed by moving the evaluator into kirocrew-core (tracked in #5926). |
|
Head is now
|
Human judgment recorded@iamwhatever marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
Verdict moved BLOCK → CONCERNS on this head; the
|
|
Re-verdicted on this head; both items are advisory
|
Problem / Motivation
Kiro Crew can automate work inside one session, but a goal too large for one session — "triage N issues and publish fix PRs", "take this feature from design to merged PRs" — still requires the user to decompose it by hand, open a session per piece, paste context into each, poll them all, and decide every next step. The pieces to automate this already exist (#2435 session control, #5198 session ledger, #5650
session_send) but nothing assembles them into a usable role.Why it matters
With dozens of concurrent sessions the marginal cost of supervision grows linearly: every open workstream is a tab the user must remember, re-read, and re-decide. The pain is no longer running parallel work — it is being the only coordinator of it.
What changed (motivation → approach → change)
Goal: one agent that owns a long-horizon goal and does only the four things a work item cannot do for itself — decompose, dispatch, verify, decide the next round. Approach: assemble the merged mechanisms rather than build new ones, and put the charter in the spec wherever a prompt sentence would otherwise be the only thing enforcing it.
Why not the dynamic-workflow engine.
src/kiro_crew/workflows/is the nearest existing mechanism — it already "orchestrates several Kiro Crew agents through ordered phases" — so it is worth saying why this is not a second spelling of it. A workflow is an authored script executed to completion: its phases are fixed before it starts, its agents are sub-agents rather than sessions the user can open, and there is no point at which a human re-plans it mid-run. The conductor is the opposite on all three: it dispatches top-level, steerable sessions the user can read and redirect while they run, it treatshuman_approvalas a first-class acceptance kind, and re-planning between rounds is the normal path rather than a failure. A goal whose next round genuinely depends on what the last round produced cannot be a workflow script, because the script would have to be written before that information exists.1.
kirocrew-conductoragent —_install_conductor_agent()insrc/kiro_crew/agent.py, following the_install_research_agentpattern: generated at install time frombuild_agent_config(), so MCP invocations resolve to the machine's real paths (a static spec cannot ship). What the spec enforces structurally:fs_write. A task needing a write is a work item for a child session, never the conductor's own work.fs_write, and notcodeeither — governance mapscodetofilesystem.writebecause it "writes files and can shell out" (platform/governance.pyBUILTIN_TOOL_SCOPES), and it sits inWITHHELD_FROM_AUTO_APPROVEfor the same reason. That is what makes "never does a work item's work itself" a property of the spec rather than of the prompt, and it is pinned bytest_mounts_no_tool_that_can_write_a_fileso the claim cannot outlive the tool list.kirocrew-core+kirocrew-dashboard; inherited servers the charter has no use for are dropped. The builtin set is narrowed on the same rule — an unused grant is surface the charter cannot account for:web_fetchstays because the skill's worked example reads an issue list, whileweb_search,grepandglobare not mounted (fs_readcovers every read the charter describes).tool_searchis the one apparent exception and is load-bearing rather than unused: with MCP Tool Search active the session-control verbs are deferred, so without it the conductor cannot reachsession_create/chat_folder_*/monitor_startat all. The prompt names that use explicitly.@kirocrew-dashboardnorexecute_bashis inallowedTools. Auto-approve is the one path that never reaches the PreToolUse gate, so both stay off it and every session-control call — and every evaluator run — passes throughhooks.on_tool_callwhere the deny floor and governance ceiling apply. Granting the opt-in dashboard set in this spec is the explicit per-agent assignment that set requires.allowedToolsis built through_may_auto_approve()(may_skip_gate_now), the entry point whose whole purpose is that a new writer cannot reopen the bypass by restating a literal list. A ceiling with an opinion about@kirocrew-corenow leaves it mounted but not pre-approved.permissionsblock is derived from that filtered list, viaallowed_tools_to_permissions(), instead of restating a blanketkirocrew-core/*allow. The rules come out identical on an ungoverned host, a laterallowedToolsedit carries through, and a ceiling that strips a grant strips its KAS rule with it.{"rules": []}when nothing qualifies, because the key's mere presence is what makes KAS load the spec.kirocrew-dashboardentry carries the managed-server metadata every entrybuild_agent_configemits gets: the conditional"type": "registry"marker and theKIROCREW_HOMEenvpin. Without the marker a registry-mode client silently drops the entry and the conductor's session-control tools never launch; without the pin the shim reads the default data home while the gateway runs under an override. Both helpers return empty on a default install, so the emitted spec is unchanged there.CONDUCTOR_AGENT_FILENAMEis registered inOWNED_KIRO_AGENT_FILES(src/kiro_crew/agent_files.py), the tuple whose docstring says every managed spec registers there. Absent from it, three consumers (the Playwright convergence sweep,doctor's dead-path repair, connection minting) classify Kiro Crew's own spec as a third-party file, so a dead resolved command path is reported as unfixable instead of repaired.2.
goal-conductorbuilt-in skill (wheel-shipped undersrc/kiro_crew/builtin_skills/) carrying the operating procedure: the three work-item tests (independent / assertable / long-running), the conductor-vs-work-item boundary rule, dispatch steps (folder →session_create→session_sendseed → ledger record), a patrol loop overmonitor_start, two-phase acceptance for values that only exist after an item starts (PR numbers), ownership-signal respect during triage (claimedlabels), and four stop conditions including an explicit needs-human exit.goal-conductor, notconductor. That name is owned by the generated delegation skill (conductor_skill.generate_conductor_skill), and two existing paths delete<skills>/conductor/SKILL.mdwhenagent.conductor_skillis false — the default:cli_setup.pyon every setup run and the dashboard config handler on toggle-off. Sharing the name would letkirocrew setuperase the packaged skill on a stock install, and quarantine the user's delegation skill when the flag is on.item-<n>entry in the ledger'sartifactsmap — acceptance spec, session key, round, status, and read cursor — and rotates finished items out. The ledger is the only store that survives compaction and judging an item from its transcript is forbidden, so a spec held only in context is a spec patrol cannot rebuild. The encoding fits the field's real bounds (32 entries, key ≤128 chars, value ≤2000 chars), which are also what keeps it to pointers-plus-spec rather than prose.~/.kiro/crew/skills/...— aKIROCREW_HOMEoverride moves the skills root, so the hardcoded form would fail every call before patrol ran.3.
scripts/accept_eval.py(stdlib-only, Python 3.8+), the deterministic half of patrol: reads acceptance specs on stdin, emitspass|fail|pending|refused|error. Acceptance is never the model reading a child transcript — it is this script's verdict. Per-check timeout, evidence tail-capped,gh pr checks' pending exit code (8) mapped topendingrather thanfail, decode pinned to UTF-8 witherrors="replace"so a check's output cannot turn its verdict into a crash.subprocess. The evaluator accepts no command, argv array, or shell string from a spec. Every argv it runs is built here from a fixed template out of narrowly-typed fields, sopr_checksbecomesgh pr checks <n>and nothing else executes._SELF_BUILT_COMMANDS({"gh"}) is the internal fail-closed assertion that this stayed true — it is not a spec-facing allowlist.cmdkind behind an allowlist, and three review rounds established that no constraint onargvfixes the shape: bare interpreters on the list letpython -c <payload>run anything; a basename-only check let/tmp/gitexecute an arbitrary binary under an allowlisted name; and then plaingit reset --hard— allowlisted, and destructive. The third is decisive, because this script is invoked as an approved wrapper: Kiro Crew's denied-command floor reads theexecute_bashstring, which sayspython3 accept_eval.py, while the real argv arrives on stdin and runs withshell=False. A generic executor there removes the deny floor for whatever it accepts, so the bypass was the wrapper rather than the list.kindwhose handler builds its own argv (apytestkind taking test paths, annpm_scriptkind taking a script name) — never a kind that accepts a command. The script's docstring states this as the supported path so the next contributor does not re-open the generic form.cmdis still recognised and answered with arefusedverdict naming what to use instead, so a conductor carrying an older skill gets guidance rather than "unknown kind".pr_checkson the PR carrying the work, which covers it transitively because CI runs the suite. The skill's own worked example is already issue → PR → checks green.try: a non-object entry ({"items": [1, {...}]}) raised on.getbefore the handler, aborting the run and discarding every sibling verdict. It now yields a positionalerrorverdict.Tests
test/test_conductor_agent.py— 22 tests. Installer: identity + charter;fs_writeabsent whileexecute_bash/@kirocrew-dashboardare reachable but not pre-approved; no tool the charter never names (web_searchabsent,web_fetchpresent); MCP narrowing; the dashboard entry carryingtype/envunder registry mode + home override and staying minimal without them; a ceiling-governed ref keeping its mount and losing its grant;permissionsderived from the filtered list (collapsing to{"rules": []}when the grant is withheld); the withheld grant emitting onemcp_auto_approve_withheldSEL event, no event when nothing is withheld, and a raising audit sink still letting the spec land;CONDUCTOR_AGENT_FILENAMEinOWNED_KIRO_AGENT_FILES; the packaged skill not colliding with the delegation skill's name; the ledger item encoding documented as a string map.Evaluator — the invariant gets its own class:
pr_checksbuilds its own argv (asserted exactly, with_runcaptured rather than executed);_runrefuses a command it did not build; a source ratchet fails if any handler ever reads anargv/command/shellspec field, which is the regression a behavioural test cannot cover;pr_checksrejects a non-integerprincludingbool. Behaviour: verdict vocabulary across the remaining kinds, the removedcmdkind refused with guidance namingpr_checks, a non-object item not aborting the run, malformed stdin as a clean exit 2.Full run: 1400 passed / 3 skipped across the affected modules (agent, agent_files, skills, doctor dead-path, connections mint, session control, ledger, MCP dashboard, spawn roster). Gates green: black, isort, flake8, mypy, subprocess-encoding, brand-name, changelog-history, focus-cue, harness-parity, vendor-manifest, loop-bound-locks.
accept_eval.pyneeds nosecurity_postureregistration (reads local state and prints; no redactor or egress calls).Manual verification
After install,
~/.kiro/agents/kirocrew-conductor.jsonexists with resolved command paths. Open a session on the conductor agent and hand it a multi-item goal: it should present a plan before dispatching, seed children viasession_send, evaluate acceptance through the bundled script (one approval prompt per evaluator call — by design), and surface non-assertable items instead of dispatching them. A logic dry-run against 10 real repo issues was performed and its two findings (two-phase acceptance,claimed-label exclusion) are folded into the skill. Note the runtime preconditions:agent.session_controlmust betrue, and strict session identity must resolve for the reflexive tools.Why no screenshot: backend-only — a generated agent spec, a wheel-shipped skill, a stdlib script and tests. No frontend path is touched and nothing renders differently.
Related Issues
no linked issue: this assembles already-merged mechanisms (#2435, #5198, #5650) into a new role; there is no filed issue it resolves. Those references are context, not closing trailers.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)