Skip to content

feat(desktop): build several tasks from one capture, and measure each task's pass rate within a budget - #9216

Merged
morluto merged 795 commits into
morluto:mainfrom
earakely-scale:feat/desktop-several-tasks
Nov 10, 2026
Merged

morluto merged 795 commits into
morluto:mainfrom
earakely-scale:feat/desktop-several-tasks

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Current merge and cleanup status (2026-11-10)

Merged main at 55b6d2ca20 in 488ad82838, retaining the multi-task behavior and main's newer typed alias handling. db6f06b991 reconciles obsolete cleanup recovery text and records current upstream verification.

The original missing-sandbox-identity concern is addressed for the pinned agentenv-framework 0.9.1485. Upstream #555 provides discovery under each solve's private root; #561 reports incomplete teardown while retaining recovery folders. Upstream #546 is closed. The published pinned wheel was inspected.

Fresh qualification used the published provider and REA cleanup helper with real Docker containers, networks, and marked Linux processes: SIGKILL during allocation; SIGKILL with active resources; and a forced removal refusal followed by retry. All cleaned up; the refusal retained its folder/container until retry; an unrelated container stayed running. See the verification note and linked JSON record. This was an isolated Linux arm64 process namespace, not a paid model/full run_bundle solve or native macOS process qualification. Existing conservative checks remain.

Local checks passed after the merge: fast CI static gate; 211 focused merge tests; 22 cleanup/report/cancellation tests; Python producer tests; docs:check and docs:build. The previous full GitHub run was green at 57b3a3a050; CI for the new head is pending. Human ownership declarations below are unchanged.

Documentation follow-up b2fb4e720f: the qualification JSON is repository evidence, not a published VitePress asset. Its link now follows the repository-source convention. The exact npm run verify:docs:routes CI command passed locally: 396 HTML pages and 2,517 distinct internal targets/anchors. All other selected lanes passed at db6f06b991; the new full CI run is pending.

Fixes #9676.

Summary

A desktop environment can now hold several tasks from one capture. Each task is checked at its own end checkpoint and, optionally, against a near miss.

A new MCP tool, measure_build_task, runs a bundle's task in fresh computers. It runs 3 solves, plus 2 more unless all passed or all failed. It stops before a solve that would pass the budget. It records the band, cost, turns and failures per check in the bundle's difficulty.json.

Problem and expected behavior

Every bundle held one task, and every task was one edit with the value in the prompt. Sonnet 5.5 solved 21 of 21 such solves across 7 bundles, at $0.17 to $1.07 each. Such tasks show little about a model.

Getting harder tasks needed three things REA lacked:

  • several tasks per environment, each still proven sound;
  • a way to measure how often a model solves a task, within a budget;
  • a record of that measure that ships with the bundle.

Solves used to be shell loops of agent-env run outside REA. Those loops had no budget cap, no adaptive stop and no failure count per check.

Change and scope

Several tasks per capture (f9e5b2db7):

  • The building agent makes each later task's change on top of the last, and marks end-<task name> once it is saved.
  • It may then make a near miss of any task and mark near-<task name>.
  • The export checks each task against the seed and against its own end.
  • When a near-<task name> checkpoint exists, at least one check must fail on it, and on what its views actually show.
  • Refusals name the task and the checkpoint.
  • The bundle gets every task's steps and verifier, one README section per task, and one eval that lists them all.
  • stop_build_capture returns later_drafts: one draft per end-<name>, drafted from the end before it, so it shows only that task's change.
  • Each task is witnessed on its own fresh computer. check_build_task needed no change for this.

Measuring a task (6946454c7):

  • measure_build_task takes bundle, task, model, agent (default claude-code) and budget_usd.
  • Each solve runs bridge/agentenv/solve.py with the agentenv runner's Python. It calls run_bundle once, with sandbox="local".
  • The solver uses the model endpoint agentenv reads from the environment REA was started in. REA holds no model client and never reads the key.
  • Band rules:
    • 0 of 3 is too_hard.
    • 3 of 3 is too_easy.
    • Otherwise 2 more solves run: 1 to 3 of 5 is in_band, and 4 of 5 is too_easy.
  • A solve that did not finish runs again, at most twice, and is not counted.
  • Before each solve, the measure stops if the largest cost so far would take the total past the budget.
  • The record goes into difficulty.json at the bundle's root and a "Measured difficulty" README section. It holds:
    • the model, agent and band;
    • k (the number of solves that finished) and the passes;
    • the cost, and each solve's turns;
    • failures per check;
    • a SHA-256 of the task's steps and verifier;
    • REA's and agentenv's versions.
  • DesktopBuildOwnedWork now tracks the work that closing REA stops, shared by computer starts and measures.

Skill and docs:

  • The building guide covers several tasks, near misses, and a witness per task.
  • It gains "Measure" and "Harder tasks" sections. They say which hardening patterns have already failed to bite.
  • The tasks, export, build-witness and build-computer guides are updated.
  • Two new specs, several-tasks and task-difficulty, each have a verification note.

Not included:

  • Parallel solves. Solves run one at a time.
  • Resuming a measure. A measure stopped by the budget starts over when run again.
  • Reset inside a capture. Later tasks build on earlier ones, and their checks should not read what a later task changes.

Contract and boundary impact

  • Semantic owner and earliest changed stage: the desktop task export (admitDesktopTasks), the build tools' contracts, and a new agentenv solve helper.
  • Contribution scope fit and linked maintainer decision: same approved desktop-environments work as feat(desktop): capture a Linux desktop application in use #8874.
  • CLI and MCP/tool-catalog contract:
    • export-desktop-application --task is repeatable, with a new --task-end NAME=CHECKPOINT. The result's task becomes tasks, a list.
    • export_build_environment takes tasks: [{task, end_checkpoint}] instead of task, at most 8 tasks. It also reports tasks. This tool is new in the unmerged stack.
    • stop_build_capture gains later_drafts, and mark_build_checkpoint documents end-<name> and near-<name>.
    • New MCP-only tool measure_build_task.
  • Provider, bridge, target-format, or platform compatibility: the new bridge/agentenv/solve.py targets agentenv-framework 0.9.1464's run_bundle. It runs on Linux on amd64 or arm64, as build computers do.
  • Evidence, artifact, provenance, or reconstruction contract:
    • The export Evidence's result now has tasks, and its parameters gain task_ends.
    • Measured bundles gain difficulty.json and a README section. agentenv ignores both.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable):
    • Each solve runs in a fresh local computer that its agentenv run tears down.
    • Cancelling a measure, or closing REA, stops the running solve.
    • The measure writes only into the bundle folder it was given.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: the generated MCP catalog gains measure_build_task. No committed catalog file changes.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated:
    • DesktopSeveralTasks.test.ts: exports two tasks from one capture, each at its own end, with a near miss. It also covers six refusals:

      • a task checked at an end before its change;
      • a near miss that passes;
      • two tasks with the same name;
      • an end for a task that wasn't given;
      • an end without =;
      • a checkpoint the capture never took.

      All 7 failed on the base.

    • buildSeveralTasksMcp.test.ts: the later draft, the export of two tasks with --task-end, and the refusal of an end the capture never took. Both tests failed on the base.

    • taskDifficulty.test.ts: the band, the next-solve decision, the budget stop, unfinished solves, the record and the README section. All 13 failed on the base.

    • buildMeasureMcp.test.ts, run against a stand-in agentenv:

      • 2 of 5 is in the band, and the bundle records it with the task's digest;
      • 3 of 3 stops after re-running an unfinished solve;
      • the budget stops before a third solve;
      • a missing task is refused before any solve.

      All 4 failed on the base.

    • Existing tests moved from task to tasks.

  • Base reproduction or other evidence, all on an arm64 Ubuntu 24.04 VM with real TreeLine 3.1.5:
    • Several tasks:

      • REA's tools captured TreeLine once, with three tasks: rename a category, change Foundation's year, and add a book under Asimov.
      • The capture also holds three near misses.
      • The two later drafts each showed only their own change.
      • The export kept all three tasks, each failing at its near miss.
      • Each task was witnessed on a fresh computer. All 4 stages held for all three tasks.
      • agent-env run --dry-run plans the three tasks. No solves were spent.
    • Measure, with Sonnet 5.5:

      Task Budget Passes Band Turns Cost
      rename-category $2.50 2 of 2 undecided (budget stop) 16, 20 $2.02
      set-foundation-year $2.50 3 of 3 too easy 17, 18, 22 $1.89
      add-asimov-book $2.50 3 of 3 too easy 17, 17, 17 $1.64
      apply-corrections $9 3 of 3 too easy 106, 61, 74 $5.14
    • About apply-corrections:

      • The solver opens a second document in TreeLine and applies its ten corrections across a 15-book list.
      • One correction is "the same year as The Quiet Glacier's", which the solver must look up.
      • Its checks fail on the seed and pass when the corrections are applied. They fail all four near misses offline.
      • All three solves passed.
    • The result, plainly: no task reached the band.

      • The tool works end to end: the band rules, the budget stop and the records all behaved as specified.
      • The hardening tried so far doesn't bite on Sonnet 5.5. That covers one more dialog, a conditional move of 7 records, and 10 corrections read from a second document.
      • An earlier pilot of the first two of those, outside the tool, passed 6 of 6.
  • User-visible CLI/MCP output (if applicable): the difficulty.json record and README line for each measured task.
    • Example README line: "set-foundation-year: anthropic/claude-sonnet-5-5 passed 3 of 3, too easy, $1.89 in all."
  • Remaining proof gaps:
    • No task in the band yet, and "mistakes that don't show once saved" is untried. The $4.31 left of the $15 budget couldn't cover 3 more solves.
    • Only TreeLine, and only Sonnet 5.5.
    • apply-corrections was rendered into an earlier pilot bundle, not exported from a capture, and wasn't witnessed live.
    • Not tested:
      • a near miss whose only failures come from a view REA couldn't read, or read only in part;
      • an end-<name> taken before end, which gets no later draft.
    • Cost:
      • The cost is Claude Code's own estimate.
      • Nothing caps the first solve's cost.
      • A solve that can't run at all records nothing.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • Follow-up af6791e3bc: completed solve reports survive failed or cancelled final cleanup and recording refusal; preflight file failures return typed results with repair guidance. Controlled helper/measurement suites passed (10 tests), including cleanup retry, cancellation, EIO and error projection; typecheck and diff checks passed on macOS. No real paid agentenv/model solve or failing filesystem was exercised.

  • npx oxfmt --check .: passed.

  • oxlint on the changed files: no errors.

    • Two existing size warnings changed: toolEffects.ts grew by 9 lines, and DesktopBuildComputers.ts shrank from 529 to 525.
    • The draft and task schemas moved to desktopBuildTaskSchemas.ts, so the contracts file stays under the limit.
  • npm run typecheck: no errors outside the API-contract files. Those fail only on a dependency missing locally.

  • vitest over src/domain/desktop, src/application/desktop, tests/boundary/providers, tests/boundary/mcp/build*, src/contracts, the CLI coverage test and the advertised-pattern test:

    • 3,505 passed.
    • 41 failed, in 10 provider files. The same 41 fail on the base commit.
    • Of those, 12 need a newer Node's DatabaseSync.limits, and they pass on the VM. The others need local IDA, Kuna, Ghidra or adb.
  • npx knip: the same 2 unresolved worker imports as the base.

  • npm run verify:module-boundaries: passed.

  • Not run: docs:check and mcp-catalog:check.

Compatibility, safety, and release

  • Breaking changes or migration steps:
    • export_build_environment's task became tasks. That tool exists only in the unmerged stack.
    • The CLI export result's task became tasks.
  • Real Hopper/Ghidra, browser, or OS coverage: Linux arm64 with real TreeLine. Unit and boundary tests ran on macOS.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review:
    • The model key stays in the environment REA was started in. REA passes that environment to agentenv without reading it.
    • Solves run in local agentenv computers that are torn down after each run.
    • The helper is an owned process group that cancelling or closing REA stops.
    • Validation spend: $10.69 of a $15 budget.

Review follow-up (e482e9e)

JSON task drafts now suggest count checks and guards for top-level arrays. Focused draft and generated Python evaluator tests passed (23 tests), and typecheck passed. The dependent stack and other outstanding review findings remain separate.

Merge with the stack's latest (dfebbad) and a runtime check (80a0d25)

  • Root-array counts: the stack now has its own version of the e482e9e fix (feat(desktop): name a record by a nested field or a child's attribute #9185's eea99c7c7), so the merge keeps the stack's code. This PR keeps its draft-level guard test and its Python root-count test, which the stack lacks.
  • Seed check views: the stack also has its own version of this PR's shipped-seed check views. The merge keeps the stack's code, plus this PR's integration test "checks the seed the bundle ships, without the app's log it leaves out".
  • Task size: the stack splits the controller's JSON argument, so the 96 KiB task refusal is gone, here and in the several-tasks spec. Its "exports a task larger than the old 96 KiB refusal" test now uses tasks: [{ task }].
  • Schema probe: the stack probes export-desktop-application --schema at the start of every capture. The several-tasks MCP test now expects that call first.
  • Runtime check (80a0d2505): that probe now also requires taskEnd. A build runtime packed before several tasks is refused at the start of the capture, with the repack fix, instead of failing at export. The new test failed before the change.
  • Validation:
    • oxfmt is clean; typecheck shows only the API-contract dependency missing locally.
    • vitest over the desktop folders, the build MCP tests and the build-computer provider tests: 1,081 passed, 22 skipped.
    • The branch merges cleanly into main.

Maintainer follow-up

Each solve now owns a private local sandbox root. After a helper stops, REA retries teardown for sandbox folders still present and retains failed cleanup for retry. The 0.9.1464 provider can remove a folder after an unverified teardown: with a Docker command that refuses removal, its provider termination returned successfully and remove_local_work_dir removed the folder in a controlled local probe. That was the verification gap on 0.9.1464; it is superseded by the private-root workflow and verified teardown in pinned 0.9.1485. Native macOS host-process verification remains limited. The export targets 0.9.1464.

Current validation: five Python cleanup regressions; the solve helper, root ownership and setup/doctor source tests; the measurement and multi-task MCP boundaries; check:fast, format:check, docs:check and package inclusion passed. With the real 0.9.1464 provider, a Linux container using Docker Desktop removed an owned Docker container and a marked Linux process, and a macOS run removed the container while retaining its folder because host-process cleanup could not be verified. This was not a full run_bundle SIGKILL or paid model solve.

Integrated the owning selector-format and setup process-lifetime fixes together with the corrected seed/export parents. JSON records retain JSON semantics through matching and task evaluation; setup retains cancellation and cleanup ownership through finalization. Multi-task and pass-rate workflows are validated with controlled source/MCP fixtures. Task hashing follows the shared runtime-resource limit and reports capacity distinctly. Staged results use the shared writer and retain retry ownership; affordable solves are not stopped by floating-point rounding. Known returned sandbox identities retain targeted cleanup ownership, while killed helpers without an identity report unknown cleanup. The former upstream identity blocker is addressed for the pinned local provider; see the current cleanup verification below. Additional checks passed: npm run docs:check.

At head abdb2d6e4b, the affected tests below completed with 163 passed (163); npm run check:fast, npm run format:check and git diff --check passed. This includes the updated parent stack and main through 25d3c7237f. Checks use source/domain and controlled boundary fixtures; no real desktop provider qualification was performed. Current-head CI and review remain required before auto-merge.

node node_modules/vitest/vitest.mjs run src/application/desktop/DesktopBuildSolves.test.ts src/application/desktop/DesktopSeveralTasks.test.ts src/domain/desktop/taskDifficulty.test.ts tests/boundary/mcp/buildMeasureMcp.test.ts tests/boundary/mcp/buildSeveralTasksMcp.test.ts src/domain/desktop/appTaskChecks.test.ts src/domain/desktop/viewPointer.test.ts src/application/desktop/DesktopTaskExport.test.ts src/application/desktop/DesktopTaskDraftNotes.test.ts src/domain/desktop/stateRoles.test.ts

Upstream sandbox-identity blocker: scaleapi/agentenv-framework#546 . The issue documents source inspection of published 0.9.1431 and 0.9.1464; it does not claim a real Docker leak reproduction. The original local cleanup gap is addressed by upstream #555/#561 and pinned 0.9.1485; see the current verification below. Auto-merge has not been enabled.

agentenv 0.9.1485 cleanup (a48513b..e586f18)

  • Cleanup through agentenv's own listing: solve_cleanup.py terminates each sandbox agentenv lists under the solve's private root with local_sandbox_ids() (feat(local): a supervisor can terminate the sandboxes of a run it killed scaleapi/agentenv-framework#555), rather than parsing folder names. The Docker and host-process checks stay. A test of a sandbox id the old pattern rejected failed first.
  • Verified once the root is empty (98b0a25b2): the solve's sandboxes are all local under that root. With fix(local): retain sandbox folders after incomplete teardown scaleapi/agentenv-framework#561 in 0.9.1485, agentenv keeps a sandbox's folder until its teardown is confirmed. So when the helper ends without a report and REA's cleanup empties the root, cleanup is verified: a cancelled solve ends as a cancellation, not as cleanup_incomplete.
  • A solve stopped before it reports (e586f183f): a helper ended by a signal, whether killed from outside or stopped at its deadline, now counts as a failed solve with unknown cost, which the measure records and stops on. Before, the measure refused with invalid_input and advice to fix the model endpoint. A helper that exits on its own without a report still ends the measure as a solve that could not run.
  • Pin: desktop bundles, setup's --install-agentenv and the doctor target 0.9.1485, since 0.9.1484 failed to publish. a48513bac adds solve_cleanup.py to the shipped-helpers test.
  • On an arm64 Ubuntu 24.04 host with 0.9.1485:
    • A Gnote bundle held all four witness stages.

    • In each case below every container, network and the private root were gone within 5 s, and no measure reported cleanup_incomplete:

      • the solve helper's process group SIGKILLed during the deploy probe;
      • the same, mid-solve;
      • an MCP cancel mid-solve;
      • REA's own cancel mid-solve;
      • a cancel followed by a SIGKILL 0.3 s later.

      Both cancels returned cancelled with cleanup complete.

    • The SIGKILL runs returned the refusal that e586f183f replaces. That commit is covered by unit tests, not by the VM.

    • An earlier run on 0.9.1482 showed a KeePassXC bundle exported on 0.9.1431 still deploys.

Merge with main (996ea01)

Main now holds the sibling features this branch was built beside: downloads (#9215), readers and the screen judge (#9217), and secrets (#9225). Every feature from both sides stays, and the desktop pin stays at 0.9.1485.

  • Readers run at each task's own end and near miss.
  • A screen judge compares the seed with the task's own end, and checks file changes from the previous task's end, so a later task isn't refused for an earlier one's change. New tests cover both.
  • Withheld-file checks apply to every task. The export, capture and summary paths take several task files and withheld secrets together, and the build tools register measure_build_task beside record_build_consent.
  • Main's tests that exported one task now pass tasks: [{ task }].
  • Fixes main lost, kept here: these commits were listed in fix(desktop): find ids in archive entries and draft nothing for a state directory's lock #9210's and feat(desktop): install an app from an AppImage or an upstream tarball, pinned by its SHA-256 #9215's squash messages but dropped when main was merged into those branches:
    • verify registered build host paths;
    • match placed notes by whole text;
    • AppImage and tarball docs;
    • a few tests.
  • The inherited main failures are fixed by merging 8dfc47b107: the duplicate ADB schema declaration, desktop-secret composite keys, and unused code reported by Knip.

CI repairs

Merged main's 8dfc47b107 fixes, then addressed failures exposed by the full and canceled test-shard logs:

  • Replace the obsolete whole-verifier digest with the existing executable file-check test, including a computer without command-execution capability. Repair two witness-guide links.
  • Preserve selected alias target kinds through export, create empty known-directory targets, retain unknown/file targets, and remove the duplicate Python alias helper.
  • Check Windows cleanup's rejected write and absent output rather than an obsolete error string.
  • Preserve the caller's exact HAR path in environment-build Evidence; its bundle now matches the three-command workflow even with /./ in the selection.
  • Classify oversized Ghidra endpoint records consistently on Windows and POSIX, retaining cleanup ownership.
  • Align model extent-rejection, IDA discovery-coverage, and CLI source-closure assertions with their current contracts.

Executed locally with Node 24.22.1 and npm 11.16.0: check:fast, check:ci:fast, docs:check, docs:build, and test:python passed. Focused validation passed: 154 model acceptance/boundary tests, 79 reproduced boundary cases, 193 related lifecycle/provider tests, 103 desktop Python/install boundary tests, 90 export service/domain tests, 25 reader/equivalence tests, and 20 several-task/measurement/verifier tests. The settings-alias and model-acceptance failures were reproduced locally before repair.

Earlier GitHub runs passed the full source suite, static checks, build, documentation, Python producers, and native Windows conformance. Desktop/IDA simulations do not establish real-provider behavior. The full CI run for 57b3a3a050 passed. A new run is required after the current main merge.

Author readiness

Human owner: @earakely-scale

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-07T01:51:13.218600Z
We couldn’t complete this review. Comment @codex review to try again.
6946454 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@greptile-apps

greptile-apps Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5 Tier: plus

[Medium impact] The PR extends REA's desktop-build workflow from capture through paid task measurement.

Not ready to merge: sandbox cleanup and the stale export assertions remain unresolved, and the new recovery issues must also be fixed.

Findings

  1. P1 Stuck computers stay running ▶
  2. P1 Gzip tests expect removed field ▶
  3. P2 Failed closes lose cleanup ▶
  4. P2 Cancellation hides completed writes ▶
Fix with agent prompt
### Issue 1
src/desktop/AgentEnvSolveHelper.ts:192-201
When a solve remains stuck after `STOP_MS`, `supervisor.stop()` kills its process group, but that does not remove its Docker computer. This path retains only failed process cleanup. It never keeps the sandbox identity or an owner that can remove the computer, so `closeAll()` cannot retry its teardown after the helper is killed.

Share the sandbox cleanup ownership used by `ComputerHelper.release()`, including fallback removal and retry. `AGENTS.md` requires retaining ownership when cleanup fails.

### Issue 2
src/application/desktop/DesktopGzipDocument.test.ts:301-303
This new assertion expects `normalized_result.task`, but `exportDesktopApplication()` returns only `tasks`. The successful gzip archive case in `DesktopArchiveTask.test.ts` has the same stale expectation. Both assertions fail when the exports succeed, breaking the desktop test suites.

Update both to expect `tasks: [{ failing_on_seed: ... }]`.

### Issue 3
src/application/desktop/DesktopBuildSolves.ts:349-359
If closing a task, README or difficulty file fails, `readHostFile` throws `ArtifactFileCleanupFailure`, which is not an `AnalysisError`. This catch wraps it without copying its cleanup facts or keeping its retry lease in `resources`. The caller sees an ordinary measurement error, and closing REA cannot retry the handle.

Retain the lease and bind it to the returned error, including during cancellation. `AGENTS.md` requires keeping ownership when cleanup fails; this requirement must be satisfied before merging.

### Issue 4
src/application/Setup.ts:537-538
Cancellation during `installSkill` discards its returned result before setup reports it. A failure with rollback details or recovery backup paths becomes a bare cancellation error. `configureDetectedClients` similarly loses its recorded client results when cancellation arrives after a write.

Keep completed actions and their failure details on the cancellation error before unwinding. `AGENTS.md` requires preserving local evidence, partial facts and meaningful failure reasons; this requirement must be satisfied before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds several tasks per desktop capture and budget-aware measurements of each task’s pass rate.

  • One desktop capture can produce several tasks with separate end checks.
  • Operators can measure a task's pass rate within a dollar budget.
  • Task checks can describe more saved changes and wait for app files to settle.
  • REA setup and diagnostics now cover the desktop build tools.

Previous-thread context:

  • earakely-scale initially deferred root-array suggestions to the stack; morluto later reported and supplied the fix.
  • earakely-scale disputed the JSON selector finding as inherited from feat(desktop): name a record by a nested field or a child's attribute #9185. Greptile withdrew it; the current stack includes the format-aware fix.
  • morluto says cleanup without an early sandbox identity remains a merge blocker, not an accepted risk.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Capture one app] --> B[Seed and task end checkpoints]
  B --> C[Check each task at seed and its end]
  C --> D[Check optional near miss]
  D --> E[Export tasks and verifiers]
  E --> F[Witness each task on a fresh computer]
  F --> G[Measure fresh solves]
  G --> H{Band decided or budget stop?}
  H -->|No| G
  H -->|Yes| I[difficulty.json and README]
Loading

Reviews (11) · Last reviewed commit: "Merge remote-tracking branch 'live-5600/..." · Reviewed by Greptile

Comment thread package.json Resolved
Comment thread src/desktop/AgentEnvSolveHelper.ts Resolved
Comment thread src/desktop/AgentEnvSolveHelper.ts Resolved
Comment thread src/domain/desktop/taskDifficulty.ts Resolved
Comment thread src/domain/desktop/appTaskDraftJson.ts Resolved
Comment thread src/desktop/AgentEnvSolveHelper.ts Resolved
morluto and others added 9 commits November 8, 2026 08:52
…ain) into morluto#9216

The stack's seed check views and root-array counts cover this branch's own
versions, and its split controller argument retires the 96 KiB task refusal;
the several-tasks test now expects the stack's export schema probe first.
Several tasks pass --task-end to the build computer's export, so a runtime
packed before it now fails at the start of the capture with the repack fix,
as a runtime without synthetic seeds already did, rather than at export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xfmt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The records describe grew past the 150-line lint limit with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/domain/desktop/viewPointer.ts Resolved
Comment thread src/desktop/DesktopBuildSetupHost.ts Resolved
morluto and others added 7 commits November 8, 2026 16:16
…ound =

The desktop entry parser accepts `Exec = notes %F` and an indented
`TryExec =notes`, so the install script now matches those keys the same way
instead of leaving the original program in place. Adds the feature's tasks.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A solve that returned an outcome dropped the runner's error, so a run that
failed to deploy, solve or verify was retried with no reason recorded. The
record, the measure result and difficulty.json now carry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dy, and keep a failed action's client setting

- The Hopper install gets setup's signal: Linux already honoured it, and the
  macOS installer now stops its downloads and stops before mounting or copying.
- runSetup checks the signal after the final doctor and close, so a
  cancellation there raises instead of returning ready.
- A failed pack or agentenv install no longer writes its archive or runner to
  client configs, so the setting the client already had stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leanup

A cancellation during the final doctor or host close carried only the applied
actions, dropping each client's result and backup path and the doctor report.
It now carries the whole result as setup_result, as doctor's cancellation
carries its report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morluto and others added 3 commits November 9, 2026 09:14
# Conflicts:
#	.agents/skills/reverse-engineer-anything/references/desktop-build.md
#	docs/desktop-application-capture.md
#	docs/desktop-application-export.md
#	docs/desktop-application-state.md
#	docs/desktop-application-tasks.md
#	docs/desktop-build-computer.md
#	docs/desktop-build-witness.md
#	docs/specs/desktop-app-environments/build-witness/spec.md
#	docs/specs/desktop-app-environments/build-witness/verification.md
#	src/application/desktop/DesktopApplicationCapture.ts
#	src/application/desktop/DesktopApplicationExport.test.ts
#	src/application/desktop/DesktopApplicationExport.ts
#	src/application/desktop/DesktopArchiveTask.test.ts
#	src/application/desktop/DesktopBuildBundle.ts
#	src/application/desktop/DesktopBuildCapture.ts
#	src/application/desktop/DesktopBuildComputers.ts
#	src/application/desktop/DesktopBuildExport.ts
#	src/application/desktop/DesktopBuildOutputRead.test.ts
#	src/application/desktop/DesktopBuildSummary.ts
#	src/application/desktop/DesktopBuildWitness.test.ts
#	src/application/desktop/DesktopExportFiles.ts
#	src/application/desktop/DesktopExportLaunch.ts
#	src/application/desktop/DesktopGzipDocument.test.ts
#	src/application/desktop/DesktopTaskDraft.ts
#	src/application/desktop/DesktopTaskDraftNotes.test.ts
#	src/application/desktop/DesktopTaskExport.test.ts
#	src/application/desktop/DesktopTaskExport.ts
#	src/composition/capabilityServices.ts
#	src/contracts/desktopBuildToolContracts.ts
#	src/desktop/DesktopApplicationSession.test.ts
#	src/domain/desktop/appExport.test.ts
#	src/domain/desktop/appExport.ts
#	src/domain/desktop/appExportPython.ts
#	src/domain/desktop/appExportUnowned.test.ts
#	src/domain/desktop/appExportUnowned.ts
#	src/domain/desktop/appTaskChecks.fixture.ts
#	src/domain/desktop/appTaskChecks.ts
#	src/domain/desktop/appTaskDraft.fixture.ts
#	src/domain/desktop/appTaskDraft.ts
#	src/domain/desktop/appTaskDraftAdded.test.ts
#	src/domain/desktop/appTaskDraftAdded.ts
#	src/domain/desktop/appTaskDraftFileNotes.test.ts
#	src/domain/desktop/appTaskDraftFileNotes.ts
#	src/domain/desktop/appTaskDraftGuards.test.ts
#	src/domain/desktop/appTaskDraftGuards.ts
#	src/domain/desktop/appTaskDraftLineGuards.ts
#	src/domain/desktop/appTaskDraftNamed.ts
#	src/domain/desktop/appTaskDraftNoise.test.ts
#	src/domain/desktop/appTaskDraftNoise.ts
#	src/domain/desktop/appTaskDraftNotes.ts
#	src/domain/desktop/appTaskDraftPositional.test.ts
#	src/domain/desktop/appTaskDraftScalars.ts
#	src/domain/desktop/appTaskDraftText.ts
#	src/domain/desktop/appTaskDraftTextAdded.test.ts
#	src/domain/desktop/appTaskDraftValues.test.ts
#	src/domain/desktop/appTaskDraftValues.ts
#	src/domain/desktop/appTaskPython.ts
#	src/domain/desktop/attributeRank.ts
#	src/domain/desktop/desktopCapture.test.ts
#	src/domain/desktop/desktopCapture.ts
#	src/domain/desktop/positionalKeys.ts
#	src/domain/desktop/runtimeFiles.ts
#	src/domain/desktop/seedDocument.test.ts
#	src/domain/desktop/seedScrub.ts
#	src/domain/desktop/selectorCandidates.ts
#	src/domain/desktop/stateChanges.ts
#	src/domain/desktop/stateFormats.ts
#	src/domain/desktop/stateRoles.test.ts
#	src/domain/desktop/stateRoles.ts
#	src/domain/desktop/viewPointer.test.ts
#	src/domain/desktop/viewPointer.ts
#	src/server/registerDesktopBuildTools.ts
#	tests/boundary/mcp/buildCaptureDraftMcp.test.ts
#	tests/boundary/mcp/buildCaptureMcp.test.ts
#	tests/boundary/mcp/buildCaptureRetryMcp.test.ts
#	tests/boundary/mcp/buildCaptureStatePathsMcp.test.ts
#	tests/boundary/mcp/buildComputerMcp.test.ts
#	tests/boundary/providers/desktopExportPython.test.ts
#	tests/fixtures/buildCaptureFakes.ts
#	tests/fixtures/buildComputerMcp.ts
#	tests/fixtures/fakeBuildComputerRea.ts
Desktop bundles, setup's --install-agentenv and the doctor's exact-version
check now target 0.9.1464, which keeps an exported app's data/get call,
holds the computer fixture above boot dialogs, leaves an unresolved setting's
build unverified rather than failed, and mounts lock packages instead of
copying them into the image. The web environment lane stays on 0.9.1431.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 16 commits November 9, 2026 09:24
…tenv helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0.9.1482

agentenv-framework 0.9.1482 lists the local sandboxes under a root
(local_sandbox_ids) and rebuilds one killed while it wrote its port map for
teardown. The solve cleanup helper now terminates each sandbox agentenv
lists, keeping its Docker and host-process checks, and reports anything else
left under the root. Desktop bundles, setup's --install-agentenv and the
doctor's version check target 0.9.1482.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s empty

A solve's sandboxes are all local under its private root, and agentenv now
keeps a sandbox's folder until its teardown is confirmed. So when the helper
ends without a report and REA's cleanup empties that root, the cleanup is
verified: a cancelled solve ends as a cancellation and a solve past its
deadline as a failed run, rather than as cleanup the measure cannot vouch for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.9.1485 keeps a local sandbox's folder until its teardown is confirmed, which
the solve cleanup's verified result relies on. (0.9.1484 failed to publish.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as a refused request

A helper killed from outside or stopped at its deadline left no report, so the
measure refused with invalid input and told the caller to fix its model
endpoint. A helper a signal ended now gives a failed solve of unknown cost,
which the measure records and stops on; one that exits on its own without a
report still ends the measure as a solve that could not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main now holds the downloads, readers and screen judge, and secrets features
beside which this branch was built. Every feature from both sides stays, and
the desktop pin stays at 0.9.1485:
- Several tasks: readers run at each task's own end and near miss; a screen
  judge compares the seed with the task's own end and checks file changes
  from the previous task's end; withheld-file checks apply to every task.
- The export, capture and summary paths take both several task files and
  withheld secrets; the build tools register measure_build_task beside
  record_build_consent.
- Main's tests that exported one task now pass tasks: [{ task }].
@morluto
morluto enabled auto-merge (squash) November 10, 2026 09:15
@morluto
morluto disabled auto-merge November 10, 2026 10:11
# Conflicts:
#	scripts/verify-windows-native.mjs
#	src/ghidra/GhidraTransport.ts
#	src/windows/WindowsPrivateRuntime.test.ts
#	tests/boundary/providers/desktopTaskPython.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Retain sandbox cleanup ownership after a forced desktop solve stop

3 participants