Repository navigation
feat(desktop): build several tasks from one capture, and measure each task's pass rate within a budget - #9216
Conversation
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
[Medium impact] The PR extends REA's desktop-build workflow from capture through paid task measurement. Not ready to merge: sandbox cleanup and the stale export assertions remain unresolved, and the new recovery issues must also be fixed. Findings
Fix with agent prompt### Issue 1
src/desktop/AgentEnvSolveHelper.ts:192-201
When a solve remains stuck after `STOP_MS`, `supervisor.stop()` kills its process group, but that does not remove its Docker computer. This path retains only failed process cleanup. It never keeps the sandbox identity or an owner that can remove the computer, so `closeAll()` cannot retry its teardown after the helper is killed.
Share the sandbox cleanup ownership used by `ComputerHelper.release()`, including fallback removal and retry. `AGENTS.md` requires retaining ownership when cleanup fails.
### Issue 2
src/application/desktop/DesktopGzipDocument.test.ts:301-303
This new assertion expects `normalized_result.task`, but `exportDesktopApplication()` returns only `tasks`. The successful gzip archive case in `DesktopArchiveTask.test.ts` has the same stale expectation. Both assertions fail when the exports succeed, breaking the desktop test suites.
Update both to expect `tasks: [{ failing_on_seed: ... }]`.
### Issue 3
src/application/desktop/DesktopBuildSolves.ts:349-359
If closing a task, README or difficulty file fails, `readHostFile` throws `ArtifactFileCleanupFailure`, which is not an `AnalysisError`. This catch wraps it without copying its cleanup facts or keeping its retry lease in `resources`. The caller sees an ordinary measurement error, and closing REA cannot retry the handle.
Retain the lease and bind it to the returned error, including during cancellation. `AGENTS.md` requires keeping ownership when cleanup fails; this requirement must be satisfied before merging.
### Issue 4
src/application/Setup.ts:537-538
Cancellation during `installSkill` discards its returned result before setup reports it. A failure with rollback details or recovery backup paths becomes a bare cancellation error. `configureDetectedClients` similarly loses its recorded client results when cancellation arrives after a write.
Keep completed actions and their failure details on the cancellation error before unwinding. `AGENTS.md` requires preserving local evidence, partial facts and meaningful failure reasons; this requirement must be satisfied before merging.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryAdds several tasks per desktop capture and budget-aware measurements of each task’s pass rate.
Previous-thread context:
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Capture one app] --> B[Seed and task end checkpoints]
B --> C[Check each task at seed and its end]
C --> D[Check optional near miss]
D --> E[Export tasks and verifiers]
E --> F[Witness each task on a fresh computer]
F --> G[Measure fresh solves]
G --> H{Band decided or budget stop?}
H -->|No| G
H -->|Yes| I[difficulty.json and README]
Reviews (11) · Last reviewed commit: "Merge remote-tracking branch 'live-5600/..." · Reviewed by Greptile |
…ain) into morluto#9216 The stack's seed check views and root-array counts cover this branch's own versions, and its split controller argument retires the 96 KiB task refusal; the several-tasks test now expects the stack's export schema probe first.
Several tasks pass --task-end to the build computer's export, so a runtime packed before it now fails at the start of the capture with the repack fix, as a runtime without synthetic seeds already did, rather than at export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xfmt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The records describe grew past the 150-line lint limit with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound = The desktop entry parser accepts `Exec = notes %F` and an indented `TryExec =notes`, so the install script now matches those keys the same way instead of leaving the original program in place. Adds the feature's tasks.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A solve that returned an outcome dropped the runner's error, so a run that failed to deploy, solve or verify was retried with no reason recorded. The record, the measure result and difficulty.json now carry it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dy, and keep a failed action's client setting - The Hopper install gets setup's signal: Linux already honoured it, and the macOS installer now stops its downloads and stops before mounting or copying. - runSetup checks the signal after the final doctor and close, so a cancellation there raises instead of returning ready. - A failed pack or agentenv install no longer writes its archive or runner to client configs, so the setting the client already had stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leanup A cancellation during the final doctor or host close carried only the applied actions, dropping each client's result and backup path and the doctor report. It now carries the whole result as setup_result, as doctor's cancellation carries its report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .agents/skills/reverse-engineer-anything/references/desktop-build.md # docs/desktop-application-capture.md # docs/desktop-application-export.md # docs/desktop-application-state.md # docs/desktop-application-tasks.md # docs/desktop-build-computer.md # docs/desktop-build-witness.md # docs/specs/desktop-app-environments/build-witness/spec.md # docs/specs/desktop-app-environments/build-witness/verification.md # src/application/desktop/DesktopApplicationCapture.ts # src/application/desktop/DesktopApplicationExport.test.ts # src/application/desktop/DesktopApplicationExport.ts # src/application/desktop/DesktopArchiveTask.test.ts # src/application/desktop/DesktopBuildBundle.ts # src/application/desktop/DesktopBuildCapture.ts # src/application/desktop/DesktopBuildComputers.ts # src/application/desktop/DesktopBuildExport.ts # src/application/desktop/DesktopBuildOutputRead.test.ts # src/application/desktop/DesktopBuildSummary.ts # src/application/desktop/DesktopBuildWitness.test.ts # src/application/desktop/DesktopExportFiles.ts # src/application/desktop/DesktopExportLaunch.ts # src/application/desktop/DesktopGzipDocument.test.ts # src/application/desktop/DesktopTaskDraft.ts # src/application/desktop/DesktopTaskDraftNotes.test.ts # src/application/desktop/DesktopTaskExport.test.ts # src/application/desktop/DesktopTaskExport.ts # src/composition/capabilityServices.ts # src/contracts/desktopBuildToolContracts.ts # src/desktop/DesktopApplicationSession.test.ts # src/domain/desktop/appExport.test.ts # src/domain/desktop/appExport.ts # src/domain/desktop/appExportPython.ts # src/domain/desktop/appExportUnowned.test.ts # src/domain/desktop/appExportUnowned.ts # src/domain/desktop/appTaskChecks.fixture.ts # src/domain/desktop/appTaskChecks.ts # src/domain/desktop/appTaskDraft.fixture.ts # src/domain/desktop/appTaskDraft.ts # src/domain/desktop/appTaskDraftAdded.test.ts # src/domain/desktop/appTaskDraftAdded.ts # src/domain/desktop/appTaskDraftFileNotes.test.ts # src/domain/desktop/appTaskDraftFileNotes.ts # src/domain/desktop/appTaskDraftGuards.test.ts # src/domain/desktop/appTaskDraftGuards.ts # src/domain/desktop/appTaskDraftLineGuards.ts # src/domain/desktop/appTaskDraftNamed.ts # src/domain/desktop/appTaskDraftNoise.test.ts # src/domain/desktop/appTaskDraftNoise.ts # src/domain/desktop/appTaskDraftNotes.ts # src/domain/desktop/appTaskDraftPositional.test.ts # src/domain/desktop/appTaskDraftScalars.ts # src/domain/desktop/appTaskDraftText.ts # src/domain/desktop/appTaskDraftTextAdded.test.ts # src/domain/desktop/appTaskDraftValues.test.ts # src/domain/desktop/appTaskDraftValues.ts # src/domain/desktop/appTaskPython.ts # src/domain/desktop/attributeRank.ts # src/domain/desktop/desktopCapture.test.ts # src/domain/desktop/desktopCapture.ts # src/domain/desktop/positionalKeys.ts # src/domain/desktop/runtimeFiles.ts # src/domain/desktop/seedDocument.test.ts # src/domain/desktop/seedScrub.ts # src/domain/desktop/selectorCandidates.ts # src/domain/desktop/stateChanges.ts # src/domain/desktop/stateFormats.ts # src/domain/desktop/stateRoles.test.ts # src/domain/desktop/stateRoles.ts # src/domain/desktop/viewPointer.test.ts # src/domain/desktop/viewPointer.ts # src/server/registerDesktopBuildTools.ts # tests/boundary/mcp/buildCaptureDraftMcp.test.ts # tests/boundary/mcp/buildCaptureMcp.test.ts # tests/boundary/mcp/buildCaptureRetryMcp.test.ts # tests/boundary/mcp/buildCaptureStatePathsMcp.test.ts # tests/boundary/mcp/buildComputerMcp.test.ts # tests/boundary/providers/desktopExportPython.test.ts # tests/fixtures/buildCaptureFakes.ts # tests/fixtures/buildComputerMcp.ts # tests/fixtures/fakeBuildComputerRea.ts
Desktop bundles, setup's --install-agentenv and the doctor's exact-version check now target 0.9.1464, which keeps an exported app's data/get call, holds the computer fixture above boot dialogs, leaves an unresolved setting's build unverified rather than failed, and mounts lock packages instead of copying them into the image. The web environment lane stays on 0.9.1431. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tenv helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0.9.1482 agentenv-framework 0.9.1482 lists the local sandboxes under a root (local_sandbox_ids) and rebuilds one killed while it wrote its port map for teardown. The solve cleanup helper now terminates each sandbox agentenv lists, keeping its Docker and host-process checks, and reports anything else left under the root. Desktop bundles, setup's --install-agentenv and the doctor's version check target 0.9.1482. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s empty A solve's sandboxes are all local under its private root, and agentenv now keeps a sandbox's folder until its teardown is confirmed. So when the helper ends without a report and REA's cleanup empties that root, the cleanup is verified: a cancelled solve ends as a cancellation and a solve past its deadline as a failed run, rather than as cleanup the measure cannot vouch for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.9.1485 keeps a local sandbox's folder until its teardown is confirmed, which the solve cleanup's verified result relies on. (0.9.1484 failed to publish.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as a refused request A helper killed from outside or stopped at its deadline left no report, so the measure refused with invalid input and told the caller to fix its model endpoint. A helper a signal ended now gives a failed solve of unknown cost, which the measure records and stops on; one that exits on its own without a report still ends the measure as a solve that could not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main now holds the downloads, readers and screen judge, and secrets features
beside which this branch was built. Every feature from both sides stays, and
the desktop pin stays at 0.9.1485:
- Several tasks: readers run at each task's own end and near miss; a screen
judge compares the seed with the task's own end and checks file changes
from the previous task's end; withheld-file checks apply to every task.
- The export, capture and summary paths take both several task files and
withheld secrets; the build tools register measure_build_task beside
record_build_consent.
- Main's tests that exported one task now pass tasks: [{ task }].
# Conflicts: # scripts/verify-windows-native.mjs # src/ghidra/GhidraTransport.ts # src/windows/WindowsPrivateRuntime.test.ts # tests/boundary/providers/desktopTaskPython.test.ts
Current merge and cleanup status (2026-11-10)
Merged main at
55b6d2ca20in488ad82838, retaining the multi-task behavior and main's newer typed alias handling.db6f06b991reconciles obsolete cleanup recovery text and records current upstream verification.The original missing-sandbox-identity concern is addressed for the pinned agentenv-framework 0.9.1485. Upstream #555 provides discovery under each solve's private root; #561 reports incomplete teardown while retaining recovery folders. Upstream #546 is closed. The published pinned wheel was inspected.
Fresh qualification used the published provider and REA cleanup helper with real Docker containers, networks, and marked Linux processes: SIGKILL during allocation; SIGKILL with active resources; and a forced removal refusal followed by retry. All cleaned up; the refusal retained its folder/container until retry; an unrelated container stayed running. See the verification note and linked JSON record. This was an isolated Linux arm64 process namespace, not a paid model/full
run_bundlesolve or native macOS process qualification. Existing conservative checks remain.Local checks passed after the merge: fast CI static gate; 211 focused merge tests; 22 cleanup/report/cancellation tests; Python producer tests;
docs:checkanddocs:build. The previous full GitHub run was green at57b3a3a050; CI for the new head is pending. Human ownership declarations below are unchanged.Documentation follow-up
b2fb4e720f: the qualification JSON is repository evidence, not a published VitePress asset. Its link now follows the repository-source convention. The exactnpm run verify:docs:routesCI command passed locally: 396 HTML pages and 2,517 distinct internal targets/anchors. All other selected lanes passed atdb6f06b991; the new full CI run is pending.Fixes #9676.
Summary
A desktop environment can now hold several tasks from one capture. Each task is checked at its own end checkpoint and, optionally, against a near miss.
A new MCP tool,
measure_build_task, runs a bundle's task in fresh computers. It runs 3 solves, plus 2 more unless all passed or all failed. It stops before a solve that would pass the budget. It records the band, cost, turns and failures per check in the bundle'sdifficulty.json.Problem and expected behavior
Every bundle held one task, and every task was one edit with the value in the prompt. Sonnet 5.5 solved 21 of 21 such solves across 7 bundles, at $0.17 to $1.07 each. Such tasks show little about a model.
Getting harder tasks needed three things REA lacked:
Solves used to be shell loops of
agent-env runoutside REA. Those loops had no budget cap, no adaptive stop and no failure count per check.Change and scope
Several tasks per capture (
f9e5b2db7):end-<task name>once it is saved.near-<task name>.near-<task name>checkpoint exists, at least one check must fail on it, and on what its views actually show.stop_build_capturereturnslater_drafts: one draft perend-<name>, drafted from the end before it, so it shows only that task's change.check_build_taskneeded no change for this.Measuring a task (
6946454c7):measure_build_tasktakesbundle,task,model,agent(defaultclaude-code) andbudget_usd.bridge/agentenv/solve.pywith the agentenv runner's Python. It callsrun_bundleonce, withsandbox="local".too_hard.too_easy.in_band, and 4 of 5 istoo_easy.difficulty.jsonat the bundle's root and a "Measured difficulty" README section. It holds:k(the number of solves that finished) and the passes;DesktopBuildOwnedWorknow tracks the work that closing REA stops, shared by computer starts and measures.Skill and docs:
several-tasksandtask-difficulty, each have a verification note.Not included:
Contract and boundary impact
admitDesktopTasks), the build tools' contracts, and a new agentenv solve helper.export-desktop-application --taskis repeatable, with a new--task-end NAME=CHECKPOINT. The result'staskbecomestasks, a list.export_build_environmenttakestasks: [{task, end_checkpoint}]instead oftask, at most 8 tasks. It also reportstasks. This tool is new in the unmerged stack.stop_build_capturegainslater_drafts, andmark_build_checkpointdocumentsend-<name>andnear-<name>.measure_build_task.bridge/agentenv/solve.pytargets agentenv-framework 0.9.1464'srun_bundle. It runs on Linux on amd64 or arm64, as build computers do.tasks, and its parameters gaintask_ends.difficulty.jsonand a README section. agentenv ignores both.docs/public/product-catalog.json), package, or installation impact: the generated MCP catalog gainsmeasure_build_task. No committed catalog file changes.Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.
Evidence and regression coverage
DesktopSeveralTasks.test.ts: exports two tasks from one capture, each at its own end, with a near miss. It also covers six refusals:=;All 7 failed on the base.
buildSeveralTasksMcp.test.ts: the later draft, the export of two tasks with--task-end, and the refusal of an end the capture never took. Both tests failed on the base.taskDifficulty.test.ts: the band, the next-solve decision, the budget stop, unfinished solves, the record and the README section. All 13 failed on the base.buildMeasureMcp.test.ts, run against a stand-in agentenv:All 4 failed on the base.
Existing tests moved from
tasktotasks.Several tasks:
agent-env run --dry-runplans the three tasks. No solves were spent.Measure, with Sonnet 5.5:
rename-categoryset-foundation-yearadd-asimov-bookapply-correctionsAbout
apply-corrections:The result, plainly: no task reached the band.
difficulty.jsonrecord and README line for each measured task.set-foundation-year: anthropic/claude-sonnet-5-5 passed 3 of 3, too easy, $1.89 in all."apply-correctionswas rendered into an earlier pilot bundle, not exported from a capture, and wasn't witnessed live.end-<name>taken beforeend, which gets no later draft.For evidence-bearing changes:
Validation performed
Follow-up
af6791e3bc: completed solve reports survive failed or cancelled final cleanup and recording refusal; preflight file failures return typed results with repair guidance. Controlled helper/measurement suites passed (10 tests), including cleanup retry, cancellation, EIO and error projection; typecheck and diff checks passed on macOS. No real paid agentenv/model solve or failing filesystem was exercised.npx oxfmt --check .: passed.oxlint on the changed files: no errors.
toolEffects.tsgrew by 9 lines, andDesktopBuildComputers.tsshrank from 529 to 525.desktopBuildTaskSchemas.ts, so the contracts file stays under the limit.npm run typecheck: no errors outside the API-contract files. Those fail only on a dependency missing locally.vitest over
src/domain/desktop,src/application/desktop,tests/boundary/providers,tests/boundary/mcp/build*,src/contracts, the CLI coverage test and the advertised-pattern test:DatabaseSync.limits, and they pass on the VM. The others need local IDA, Kuna, Ghidra or adb.npx knip: the same 2 unresolved worker imports as the base.npm run verify:module-boundaries: passed.Not run:
docs:checkandmcp-catalog:check.Compatibility, safety, and release
export_build_environment'staskbecametasks. That tool exists only in the unmerged stack.taskbecametasks.Review follow-up (e482e9e)
JSON task drafts now suggest count checks and guards for top-level arrays. Focused draft and generated Python evaluator tests passed (23 tests), and typecheck passed. The dependent stack and other outstanding review findings remain separate.
Merge with the stack's latest (dfebbad) and a runtime check (80a0d25)
eea99c7c7), so the merge keeps the stack's code. This PR keeps its draft-level guard test and its Python root-count test, which the stack lacks.tasks: [{ task }].export-desktop-application --schemaat the start of every capture. The several-tasks MCP test now expects that call first.80a0d2505): that probe now also requirestaskEnd. A build runtime packed before several tasks is refused at the start of the capture, with the repack fix, instead of failing at export. The new test failed before the change.Maintainer follow-up
Each solve now owns a private local sandbox root. After a helper stops, REA retries teardown for sandbox folders still present and retains failed cleanup for retry. The 0.9.1464 provider can remove a folder after an unverified teardown: with a Docker command that refuses removal, its provider termination returned successfully and
remove_local_work_dirremoved the folder in a controlled local probe. That was the verification gap on 0.9.1464; it is superseded by the private-root workflow and verified teardown in pinned 0.9.1485. Native macOS host-process verification remains limited. The export targets 0.9.1464.Current validation: five Python cleanup regressions; the solve helper, root ownership and setup/doctor source tests; the measurement and multi-task MCP boundaries;
check:fast,format:check,docs:checkand package inclusion passed. With the real 0.9.1464 provider, a Linux container using Docker Desktop removed an owned Docker container and a marked Linux process, and a macOS run removed the container while retaining its folder because host-process cleanup could not be verified. This was not a fullrun_bundleSIGKILL or paid model solve.Integrated the owning selector-format and setup process-lifetime fixes together with the corrected seed/export parents. JSON records retain JSON semantics through matching and task evaluation; setup retains cancellation and cleanup ownership through finalization. Multi-task and pass-rate workflows are validated with controlled source/MCP fixtures. Task hashing follows the shared runtime-resource limit and reports capacity distinctly. Staged results use the shared writer and retain retry ownership; affordable solves are not stopped by floating-point rounding. Known returned sandbox identities retain targeted cleanup ownership, while killed helpers without an identity report unknown cleanup. The former upstream identity blocker is addressed for the pinned local provider; see the current cleanup verification below. Additional checks passed:
npm run docs:check.At head
abdb2d6e4b, the affected tests below completed with 163 passed (163);npm run check:fast,npm run format:checkandgit diff --checkpassed. This includes the updated parent stack and main through25d3c7237f. Checks use source/domain and controlled boundary fixtures; no real desktop provider qualification was performed. Current-head CI and review remain required before auto-merge.Upstream sandbox-identity blocker: scaleapi/agentenv-framework#546 . The issue documents source inspection of published 0.9.1431 and 0.9.1464; it does not claim a real Docker leak reproduction. The original local cleanup gap is addressed by upstream #555/#561 and pinned 0.9.1485; see the current verification below. Auto-merge has not been enabled.
agentenv 0.9.1485 cleanup (a48513b..e586f18)
solve_cleanup.pyterminates each sandbox agentenv lists under the solve's private root withlocal_sandbox_ids()(feat(local): a supervisor can terminate the sandboxes of a run it killed scaleapi/agentenv-framework#555), rather than parsing folder names. The Docker and host-process checks stay. A test of a sandbox id the old pattern rejected failed first.98b0a25b2): the solve's sandboxes are all local under that root. With fix(local): retain sandbox folders after incomplete teardown scaleapi/agentenv-framework#561 in 0.9.1485, agentenv keeps a sandbox's folder until its teardown is confirmed. So when the helper ends without a report and REA's cleanup empties the root, cleanup is verified: a cancelled solve ends as a cancellation, not ascleanup_incomplete.e586f183f): a helper ended by a signal, whether killed from outside or stopped at its deadline, now counts as afailedsolve with unknown cost, which the measure records and stops on. Before, the measure refused withinvalid_inputand advice to fix the model endpoint. A helper that exits on its own without a report still ends the measure as a solve that could not run.--install-agentenvand the doctor target 0.9.1485, since 0.9.1484 failed to publish.a48513bacaddssolve_cleanup.pyto the shipped-helpers test.A Gnote bundle held all four witness stages.
In each case below every container, network and the private root were gone within 5 s, and no measure reported
cleanup_incomplete:Both cancels returned
cancelledwith cleanup complete.The SIGKILL runs returned the refusal that
e586f183freplaces. That commit is covered by unit tests, not by the VM.An earlier run on 0.9.1482 showed a KeePassXC bundle exported on 0.9.1431 still deploys.
Merge with main (996ea01)
Main now holds the sibling features this branch was built beside: downloads (#9215), readers and the screen judge (#9217), and secrets (#9225). Every feature from both sides stays, and the desktop pin stays at 0.9.1485.
measure_build_taskbesiderecord_build_consent.tasks: [{ task }].8dfc47b107: the duplicate ADB schema declaration, desktop-secret composite keys, and unused code reported by Knip.CI repairs
Merged main's
8dfc47b107fixes, then addressed failures exposed by the full and canceled test-shard logs:/./in the selection.Executed locally with Node 24.22.1 and npm 11.16.0:
check:fast,check:ci:fast,docs:check,docs:build, andtest:pythonpassed. Focused validation passed: 154 model acceptance/boundary tests, 79 reproduced boundary cases, 193 related lifecycle/provider tests, 103 desktop Python/install boundary tests, 90 export service/domain tests, 25 reader/equivalence tests, and 20 several-task/measurement/verifier tests. The settings-alias and model-acceptance failures were reproduced locally before repair.Earlier GitHub runs passed the full source suite, static checks, build, documentation, Python producers, and native Windows conformance. Desktop/IDA simulations do not establish real-provider behavior. The full CI run for
57b3a3a050passed. A new run is required after the current main merge.Author readiness
Human owner: @earakely-scale
Before merge
type(scope): outcome.🤖 Generated with Claude Code