Repository navigation
[feature] supervised-run usability: watch, gating clarity, planner selection, per-issue delivery, verify and attempt records #50
Description
Summary
Two live runs on 0.3.0 found gaps in how an origin agent supervises a run, beyond the bugs filed as #46, #47 and #48. Run 1 was smota/meshloop #44 and #45: 2 nodes, Claude Sonnet workers, delivered as PR #49. Run 2 was a sprint for smota/holoself #31–#34: 12 nodes, Claude and Codex workers, 3 graph versions. Each gap below cost a human round-trip, hand-written tooling or a wrong assumption. This issue tracks them together. Split out any item that grows.
Not covered here: prompt delivery and wrapper scripts (#27), spend-aware routing (#17), and the status, liveness and output bugs (#46, #47, #48).
Version / OS
meshloop 0.3.0, origin/main @ 2769f08. Windows 11 native, Claude Code 2.1.292, codex-cli 0.160.1.
Findings and proposals
1. No way to wait for a run except polling inspect (high)
In both runs the origin agent wrote its own polling loop. It ran inspect --json every N seconds, diffed node states with jq, and treated changes as events. Every orchestrator rewrites this loop, and it depends on status and inspect being correct, which #46 and #47 show they aren't yet.
Proposal: meshloop watch --graph <id> [--config] [--json] prints one NDJSON line per node or plan state change. It exits 0 when the run needs a human (AwaitingReview, awaiting_acceptance) or reaches a terminal state, and exits non-zero on failure. The exit reason goes in the last line. The skills (meshloop-run, meshloop-status) should point to it.
2. Acceptance gating isn't clear from tier or status (high)
- Run 2's operator expected only Tier3 nodes to stop for
accept. In fact every node stops atAwaitingReview, and the tier only selects the harness. The operator found this out by reading the code after node 1 (Tier2) stopped. - The plan used "HUMAN GATE: wait for PR merge" in node descriptions. Meshloop doesn't enforce prose, so a dependent node could start from
mainwithout the merged change.
Proposal:
- Document the rule in
docs/start.mdand themeshloop-review-planskill: every node waits for acceptance; tier routes harnesses only. statusandinspectshow awaiting_forreason on each blocked node (acceptance,dependency <id>,retry budget).- Optional, needs a decision: a node-level
gate(human(default) |verify), so read-only or low-risk nodes can advance on a passingverify_command. This would be a change to ADR 0009, so it belongs in a proposal.
3. The planner needs a harness labelled top (medium)
Run 1 wanted a low-cost Sonnet-only setup. plan failed with MidTier does not fit task Tier3 until Sonnet was given model_tier = "top". As a result, the plan record says "top" for a mid-tier model, which makes the provenance and any later cost analysis wrong.
Proposal: a config key planner_harness = "<id>" (or planner_tier) that picks the planner directly. Without it, fall back to the current rule. When no harness fits, the error names the rule and the key that overrides it.
4. Delivering a multi-issue graph as one PR per issue is manual (medium)
integrate merges every accepted node into one branch. Run 2 had four issues, and AGENTS-style policy wants one PR per issue. The plan was to cherry-pick each issue's commits from the integrate branch onto fresh branches by hand.
Proposal: an optional deliverable tag on each node (e.g. "issue-32"), plus integrate --deliverable <tag> --into <branch>. That merges only that group's nodes, plus their accepted dependencies, onto the target branch. status groups nodes by deliverable.
5. An empty verify_command passes every node without a word (medium)
Run 1's config had verify_command = []. In node 2, the worker added tests that didn't compile: a non-Sync test double was passed into a Sync harness interface. The node passed "verification" anyway and reached AwaitingReview. Nothing in doctor, run, status or inspect said no verification had run. The origin agent found the problem only after integrating, when it ran cargo test itself.
Proposal:
doctor --configandrunwarn whenverify_commandis empty.- An empty command records a
verify: skippedevidence row instead of nothing, so the attempt doesn't look as if it passed a check. inspectshows the verify command and its exit status next to the diff, so the reviewer sees what was and wasn't checked before runningaccept.
6. Attempts don't record duration or model (medium)
Working out what run 1 cost (about $4.50 estimated, about $5.70 if everything had run on Opus) meant parsing each worker's harness transcript. Meshloop records HarnessExited but not wall time or the resolved model_ref in a form inspect shows.
Proposal: store model_ref, started_at, ended_at and the exit status on each attempt, and show them in inspect --task. Add a run summary to status: attempts per node, total worker wall time, retries used. Token usage can stay out of scope, or be read from harness JSON output when present, which ties in with #17.
7. Supervised operation needs the user to run the start commands (low, docs)
Under Claude Code auto mode, the permission classifier blocks the origin agent from running meshloop plan and run, because they start unattended writing agents. That is reasonable. But the skills don't mention it, so the agent tried, failed, then handed the user commands to paste. One of those had a <scratchpad> placeholder that PowerShell parsed as a redirect.
Proposal:
- The
meshloop-planandmeshloop-runskills say up front that launching workers needs either an explicit permission rule the user adds formeshloop planandmeshloop run, or the user running the command. When the user runs it, the skill gives a ready-to-paste command with absolute, quoted paths and no placeholders. - Read-only commands (
status,inspect,watch,doctor) stay agent-runnable. doctor --jsoncould report whether an origin session was detected, since run 2 had to pass--origin-harnessand--origin-sessionby hand.
8. Unknown flags are dropped silently (low)
args.rs builds Command::Status from --graph only, so status --config x --db y parses fine and the two flags are ignored. That is the parser half of #46. It's a general problem: a mistyped or unsupported flag should be an error.
Proposal: each subcommand rejects flags it doesn't define, with an error that names the flag and the subcommand.
Suggested order
1 and 2 first: they decide whether an agent can supervise a run without guessing. Then 3, 5 and 8, which are small. Then 4 and 6.
Acceptance criteria
- Each item above is either done, split into its own issue, or explicitly declined with a reason recorded here.
-
watch: fixture test where a run with one node reachingAwaitingReviewmakeswatchexit 0 with that reason as its last line. - Unknown flag test:
status --bogusexits non-zero and names the flag. -
docs/start.mdstates the acceptance-gating rule and how to choose the planner harness.
🤖 Generated with Claude Code
Activity
smota commentedon Oct 7, 2026
I've split each item into its own issue. They are independent: each can be picked up and merged alone. The table shows where they touch each other.
watchstatusstate, so needs #46 and #47 to be correct. Showswaiting_forfrom #52 if it exists.waiting_for+ docs on gatingstatusandinspectoutput, same as #57.TaskNodefield, as #55 does.run_loop.rs:248, router).TaskNodefield, as #53 does. The planner contract from #45 picks it up automatically.verify_commandpasses nodesargs.rs).Corrections to the original text, found while splitting:
verify_commandskips them without a word (run_loop.rs:1301).attemptstable already storesmodel_ref,started_at,ended_atandoutcome. They just aren't shown, so [feature] inspect: expose per-attempt model, timing, outcome and evidence #57 is display work with no schema change.Suggested order: first #46 and #47 (correct state), plus #56 (a small bug). Then #51, #52, #57 and #59. Then #54, #55 and #58. #53 waits on an ADR decision.
This issue now only tracks those nine and closes when they are all closed or declined.
smota commentedon Oct 8, 2026
Delivered in #69 (merged as
811a871): #51, #52, #57, #59 closed. Earlier: #46, #47, #56 closed via #63. Still open from this tracker: #53 (proposal, needs ADR), #54, #55, #58.smota commentedon Nov 1, 2026
Item 1 landed on main in #103.
meshloop watch --json prints one NDJSON object per node change and per plan-state change. A completed graph or a human gate exits 0. failed, cancelled, or a settled blocked graph exits 1, and the last line names the reason. A running node beside a blocked one keeps the watch open.
Items 2–5 of this epic are not done. This comment does not close #50.
smota commentedon Nov 1, 2026
Item 2 landed on main in #105.
Operator docs now say every node stops at AwaitingReview after deterministic checks pass. The tier selects the harness and model; it does not select which nodes wait for meshloop accept. A HUMAN GATE sentence in a plan is not enforced. Dependent nodes start after meshloop accept and the meshloop resume merge. meshloop integrate lands that result on the target branch and is a separate step.
Items 3–5 are not done. This comment does not close #50.
smota commentedon Nov 1, 2026
Item 8 is already on main. status --bogus exits non-zero and names the flag. Tests: unknown_flag_exits_non_zero_with_message, unknown_flag_with_json_reports_ok_false, and unknown_flag_is_rejected_on_every_subcommand.
This comment does not close #50. Items 3, 4, 5, 6, and 7 are still open.
smota commentedon Nov 1, 2026
Item 5 landed on main in #107.
An empty verify_command still lets the node reach AwaitingReview. The attempt stores a verify evidence row with exit code 0 and summary "skipped: empty verify_command". inspect shows that row. doctor and run warn that the key is empty. A non-empty command that exits non-zero still fails the node.
Items 3, 4, 6, and 7 are still open. This comment does not close #50.
smota commentedon Nov 1, 2026
Item 3 is already on main. [planner] harness picks the planner directly and reports that harness's real model tier. Without it, planning keeps the tier rule. When no harness fits, the error is planner-no-candidate and names [planner] harness. Tests: plan_uses_the_configured_planner_harness_and_reports_its_real_tier and plan_with_only_a_mid_tier_harness_names_the_planner_override.
Items 4, 6, and 7 are still open. This comment does not close #50.
smota commentedon Nov 2, 2026
Items 4, 6, and 7 are already on main. They were split out and closed:
integrate --deliverable <tag> --into <branch>, landed in feat: planner selection, per-deliverable integrate, worker-launch skills #70. Status and inspect copydeliverableonto each node.model_ref,started_at, andended_at.inspect --taskshows them.status --jsonincludes attempt count, retries, and worker wall time.meshloop-planandmeshloop-runsay the user runs plan/run or adds a permission rule, and they give one paste-safe command with quoted paths.Items 1, 2, 3, 5, and 8 are also on main (#103, #105, #54, #107, #59). The optional node-level verify gate stays in #53, which is still a proposal and is not accepted here.
The epic acceptance criteria are met by those landed items and that split. Closing #50 does not close #53.
smota commentedon Nov 2, 2026
Closing the epic. The optional verify-gate decision remains #53 and is not accepted.