Local workflow implementation evidence

Objective: implement and verify roadmap M1-M5 and local portions of M7. Remote VM implementation (M6 and remote adapter expansion in M7) is deferred by the user. All other requirements remain in scope. Do not treat a mock harness demo as proof of a real agent/VM workflow.

Acceptance ledger

M1 (audited in docs/milestone-audit.md): versioned configuration, generic DAG, identities, transition predicates, transactional store, idempotent commands/events, Bubble Tea shell, harness compatibility spike.
M2 (macOS arm64 Lima support matrix; audited in docs/milestone-audit.md): real local VM provider, dedicated Docker daemon per executing revision, guest-only source/data, repository SHA/submodule/LFS pins, Compose, recovery receipts, two-run isolation demonstration.
M3 (audited in docs/milestone-audit.md): versioned invocation plugin protocol/locks, transitive resolution, prepare/probe/execute/renew/cleanup, harness/Git/browser/dataset adapters, engine-enforced Plan readiness, drift recovery, secret redaction.
M4: real worker/supervisor lifecycle, continuations and recovery, configurable scheduling, artifact/checkpoint verification, commit-bound QA/approval, idempotent PR broker, live TUI views, detach/restart and real feature demonstration.
M5: checkpoint/data restoration, rewind/invalidation/reuse, draining/fenced superseded workers, isolated overlapping revisions and capacity queue, revisioned plugin edits, full TUI review/actions, repeated-rewind and two-client checks.
M7 local: parallel ready nodes, budgets/priorities/joins/retry, separate writer worktrees/merge ownership, isolated mutable fixtures/child runtimes, multiple workers, provider/plugin conformance, second harness, MCP/API operations.
Delivery: documentation/examples describe actual supported behavior; build, vet, race tests, integration/e2e evidence; requirement-by-requirement completion audit.

Current status (latest integration pass)

The daemon constructs engine.Engine with the production localexec.Backend. A real two-repository feature has completed all six stages with QA and approved GitHub output; a separate real rewind fixture has verified overlapping revision VMs, historical drain, coordinator restart, and selected Code source/Compose/data restoration through QA. Invocation plugins have executable readiness, terminal-process recovery and explicit resource repair. Plan now retains structured discovered capabilities as admission obligations. Local MCP commands share daemon replay/version checks and support scoped/read-only inspection. Nested source/LFS restoration, explicit source joins and two-child-VM isolation now have real guest evidence. The parallel backend is enabled with independently reconciled child ownership and readiness. This remains an incomplete workflow release: the complete parallel-agent fan-out/join demonstration, source publication/data replay edge cases, live steering/preview, conformance and a second harness remain unfinished. Non-Codex harnesses are still explicitly rejected by the current adapter.

See the final section for current acceptance evidence and remaining integration work. Earlier sections are historical records and may describe gaps that have since been closed.

Current work

Initial authoritative inspection: local Compose CLI and named environment metadata exist; no workflow code. Docker/OrbStack, Codex 0.154.0, Claude 2.1.268, Go 1.25 and Homebrew are available. Lima is absent. Planning documentation is an existing uncommitted user-authorized change and must be preserved.

Implementation is in progress. No milestone has been marked delivered.

Implemented and verified in the first implementation pass

internal/workflow: strict v2 parsing and feature template overrides; v1 conversion; stable DAG validation; run/revision/attempt/checkpoint model; mandatory built-in readiness; expiry/runtime/config invalidation; output, supervisor, QA, approval and PR transition rules; exact QA-to-final commit matching; draining rewind; retry/cancel history.
internal/runstore: SQLite 1-schema store (modernc v1.58.0), transactional snapshot/event/receipt commits, optimistic versions, operation replay/conflict rejection, content-addressed artifacts with checksum verification and file/directory sync.
internal/daemon: private Unix socket, single-owner OS lock, versioned HTTP API, resumable events, separate client; user actions cannot submit agent checkpoints.
internal/tui: Bubble Tea v2.0.9 dashboard shell and API actions, stage/run navigation, readiness, conversation, checkpoint/test/service/graph views, resize, terminal-output sanitization, detach.
cmd/envctl/workflow.go: auto-started coordinator, ui, daemon serve/status, run create/validate/list/show/readiness/checkpoints/message/rewind/cancel/approve/priority/events. Created runs remain queued: the scheduler/guest runner is not integrated yet.
internal/runtime: Lima provider and validated pinned Ubuntu 26.04 guest configuration, lifecycle/exec, dedicated daemon inspection, no mounts/agent forwarding/automatic port exposure. Provisioning and isolation still require the real-VM acceptance test below.
internal/repository: independent source clones and SHA pins, preparation receipts/retries, source archive, submodule/LFS preparation paths. Tests cover branch movement, independence, multiple repos, retry and interrupted preparation. More submodule/LFS edge-case tests remain.
internal/plugin: descriptor/source locks, scoped capability resolution, cycle/conflict checks, credential references, JSON executable protocol, redacted evidence. Tests run a real local fixture executable for protocol behavior; guest plugin installation and built-in adapters remain to implement.

Evidence: go test ./... and go vet ./... passed. Race tests passed for workflow, store, daemon and TUI; expanded race checks including plugin/repository are being run. Built CLI smoke passed for idempotent creation, persistence, stale-client rejection, event/readiness inspection and cancel. Real PTY smoke rendered Bubble Tea, detached with q, and verified the coordinator still served requests. Smoke data is in /tmp/envctl-smoke-85cjrdwb; smoke daemons were stopped.

Real VM evidence (completed)

Initial session 71601 failed at its 20-minute context deadline: the cold 898 MB image download used approximately 19 minutes. Cleanup removed the test VM. The verified image is cached; subsequent runs used it.
Two-VM isolation passed, session 53195, ENVCTL_VM_TEST=1 go test ./internal/runtime -run '^TestLocalVMIsolation$' -v -count=1 -timeout=65m, 515.8 seconds. Guests envctl-test-a-20260911062447 and envctl-test-b-20260911062447 had Docker daemon IDs b9de28bc-5a68-4125-b2e2-17afc06c0403 and 003e42b3-934d-4c84-8546-3515988f13da. Docker 29.1.3, Compose 2.40.3+ds1-0ubuntu1. A volume in A was absent from B; host mount/SSH socket checks passed; stop/start retained A's daemon identity. Both VMs were removed by test cleanup.
Guest process reconnect passed, session 21738, ENVCTL_VM_TEST=1 go test ./internal/guestjob -run '^TestGuestJobReconnect$' -v -count=1 -timeout=35m, 189.9 seconds. envctl-jobtest-20260911063617 ran a real systemd-owned process after the submitting SSH command exited. A new client submitted the identical ID and observed exactly one execution. Output redacted a synthetic secret. The result receipt survived a VM stop/start. Cleanup removed the VM.
Lima 2.2.0 is installed via Homebrew. Guest support exercised here is macOS arm64/VZ with pinned Ubuntu 26.04. Linux/QEMU and amd64 are not yet exercised.
Those guest tests used the earlier forwarding template. Logs exposed an unexpected UDP mDNS forwarding rule because Lima defaults each rule to TCP. The source now explicitly uses proto: any; the rendered template validates with the installed Lima CLI and a regression check enforces both protocols. The next real guest demo should verify the corrected forwarding behavior.

Second implementation pass

internal/guestjob: embedded standard-library Python runner installed inside a guest; root-private request/result journal; dedicated envctl-agent execution identity; fixed systemd units; request digest conflict detection; atomic intent/results; at-most-one live execution lock; timeout/process-group cleanup; reconnect/cancel; bounded output and cursor reads; streaming secret redaction including split pipe reads. Interrupted work is reported, not silently replayed. Python protocol tests execute actual subprocesses and cover lost acknowledgements, request conflicts, interruption, UTF-8 cursors, redaction, timeout, and repeat execution. Real guest evidence is above.
internal/engine: asynchronous per-revision reconciliation, VM reservation/capacity queue, immutable source-pin checks, executable readiness, durable job reconciliation, retry/backoff, output/evidence verification, human gate enforcement, publication serialization, historical draining checkpoints, and visible attempt-budget exhaustion. Tests use a fixture backend and real SQLite/artifact storage; these are coordinator acceptance tests, not proof of real agent execution.
Engine tests cover a six-stage workflow with final approval, Plan retaining output while readiness remains unresolved, lost dispatch acknowledgement across coordinator recreation, corrupted evidence recovery, rewind during provisioning, draining at a human gate without publication, failed draining worker retrying only its original stage, explicit budget exhaustion, and parallel branches joining exact checkpoint IDs while one branch retries.
internal/runstore: per-run publication guard serializes external effects with rewind/cancel while avoiding a SQLite transaction during network calls. External broker reconciliation remains mandatory for crashes after remote success. Tests show rewind waits for an in-flight publication and old revisions cannot subsequently publish.
internal/daemon: execution-coordinator lifecycle hook runs only under the daemon ownership lock; coordinator failure stops the API and propagates its error. The CLI does not supply the production backend yet, so newly created runs still remain queued.
internal/workflow: preparation-aware rewind/cancel, frozen drain-node scope, historical-only checkpoints, delayed retries, source pins, recovery evidence, and preparation/readiness timestamps. Draining failures retain their stage instead of silently superseding it. Completed revisions become superseded on rewind so their VM can be released before consuming another capacity slot.
internal/runtime: lifecycle/exec ownership checks require a valid coordinator reservation; forwarding disables TCP and UDP explicitly.
internal/tui: recovery reason/evidence/retry time is visible alongside stage/readiness state.

Latest checks: full go test ./..., go vet ./..., CLI build, and race tests for engine, daemon, store, TUI, workflow, guestjob, and runtime passed. Earlier plugin/repository race checks also passed. No live VM test remains at this point.

Remaining integration / known limitations

All original M1-M5 and local M7 requirements remain in scope. None of the milestone checkboxes above is complete yet.

Implement the production execution backend and connect the engine to the CLI-created daemon. Scheduler and guest job journal now exist and are tested independently; the CLI still does not dispatch work.
Finish VM ownership/atomic provisioning receipts, toolchain replay pins, source transfer, Compose rendering/health/endpoints, and broader real stack/repository isolation checks. Two-VM daemon/volume/restart evidence now passes. Guest firewall is defense in depth; do not overclaim adversarial isolation from guest-side firewall rules alone.
Install plugins in guests; add built-in harness, Git/PR, browser, database snapshot/restore and emulator adapters; invocation plugin CLI/TUI mutation and complete Plan readiness/recovery integration.
Implement real Codex and Claude adapter execution/continuation (both host CLIs were inspected; no real model call has run yet). Keep credentials scoped and out of logs. Worker and supervisor execute in the isolated runtime, not the host repo.
Broker publication outside worker VM, verify authoritative artifacts/check results, approval and idempotent external operations. Execute a real feature acceptance demo; fixture results alone are insufficient.
Complete checkpoint data/source restoration, overlaps/queues/repeated rewind/plugin revisions, multi-client review, rich diff/evidence views, and active agent event streaming.
Implement parallel nodes, separate writable worktrees, joins/merge verification, isolated mutable test fixtures/child runtimes, priorities/budgets/recovery, conformance and MCP.
Add coverage for output-schema validation, plugin schemas/capability subsets, submodule/LFS edge cases, and provision/restore crash points. Reconcile docs as behavior becomes usable, then run the full completion audit.

Dependency retrieval: default proxy.golang.org timed out and goproxy.io had a checksum-proxy error. GOPROXY=https://goproxy.cn,https://goproxy.io,direct succeeded with checksum verification enabled. No global Go proxy configuration was changed.

Next harness integration inputs

Local help was rechecked for Codex 0.154.0 and Claude Code 2.1.268. Codex supports exec --json --output-schema --output-last-message --ignore-user-config and explicit exec resume <session-id> -; Claude supports print/stream-json, JSON schema, explicit session IDs/resume, setting-source restriction and strict MCP configuration. Existing Codex ChatGPT authentication is present at the standard auth-file location; no OpenAI or Anthropic API key is currently in this process environment. Credential values were not printed or copied by these checks. No real model call has run yet.

Official Codex release metadata: https://github.com/openai/codex/releases/tag/rust-v0.154.0

Linux arm64 archive: https://github.com/openai/codex/releases/download/rust-v0.154.0/codex-aarch64-unknown-linux-musl.tar.gz, SHA256 583b48df32804213bdcd338c2e5adb06b34340821fa757a726cc0a524fa33c27, 91,768,360 bytes.
Linux amd64 archive: https://github.com/openai/codex/releases/download/rust-v0.154.0/codex-x86_64-unknown-linux-musl.tar.gz, SHA256 d7e18b2597ae8f242f5f31ee9e90deef48dbc9edd634d9868fb6435d08c07f02, 98,981,886 bytes.

Digests came from the release-assets API, not from an unverified downloaded binary. Installation should preserve these pins and use the guest-owned harness home. Claude structured-output documentation was opened at https://code.claude.com/docs/en/agent-sdk/structured-outputs; inspect its exact response contract before implementing the second adapter.

Third implementation pass / live acceptance to resume

Added internal/agent: checksum-pinned Codex 0.154.0 installation in the VM, drift check for the installed binary, scoped credential references and guest role homes, structured job requests, explicit session-ID resume, bounded JSON result reading, and worker/supervisor output contracts. Workers cannot author authoritative commits/checks/review/approval fields. Contract, credential-reference, session, and scope tests pass. Claude remains to implement.
A real Codex job ran in envctl-agenttest-20260911065614 and produced a structured response and session identity. The independent unittest discover command failed with exit 5, before supervisor/resume verification. This is not a passing feature demo. That VM was cleaned up. The fixture now explicitly requires root-level test_slug.py, captures check diagnostics, and sends failed-check evidence back through explicit session continuation, with a bounded correction allowance.
An earlier real agent test VM booted according to Lima but lacked Docker. Lima.Ensure now independently checks bootstrap completion/Docker, repairs an incomplete bootstrap, and records actionable diagnostics. A fault-injection test proves failed bootstrap retry reuses the same VM. Apt uses IPv4 and retries; Lima startup allows 30 minutes for cold package retrieval.
Added internal/repository/guest.go: digest-verified source archive import, revision-scoped source directories, separate assignment worktrees/branches with preparation receipts, repeat assignment preserving live edits, actual commit capture, read-only source guards, Git bundle export/restore. Path-boundary tests pass. A real two-repository worktree/bundle fixture has been added to the agent acceptance test; its real-guest result is still pending.
Rewind now carries original repository pins into the new revision unless the repository URL/ref or explicit base SHA changes. Regression tests pass.
Extracted compose.Transform to reuse existing label/container-name/port/environment transformations on projects normalized inside the guest, without reading host source paths. Legacy rendering tests and a guest-path preservation test pass. Exposed dockerx.ParsePS for guest-collected status output. Production guest Compose orchestration remains to implement.
Lima's exact forwarding matcher requires all of guestIP: 0.0.0.0, guestIPMustBeZero: false, proto: any, and ignore: true to cover wildcard and loopback TCP/UDP listeners. Earlier templates still forwarded a subset of listeners. The current template has all four fields and validates. Source evidence: https://github.com/lima-vm/lima/blob/v2.2.0/pkg/portfwd/forward.go and https://github.com/lima-vm/lima/blob/v2.2.0/pkg/hostagent/port.go. A real positive-in-guest/negative-on-host TCP/UDP fixture was added after the current acceptance binary started, so it must run on the next test invocation before claiming the forwarding fix verified.

Latest real acceptance result

Passed: exec session 64030, TestRealCodexWorkerSupervisorAndResume, 99.48 seconds on the already-provisioned VM. The full invoked command was:

ENVCTL_AGENT_TEST=1 ENVCTL_AGENT_STATE_DIR=/tmp/envctl-agent-acceptance ENVCTL_AGENT_VM=envctl-agent-acceptance ENVCTL_AGENT_KEEP_VM=1 go test ./internal/agent -run '^TestRealCodexWorkerSupervisorAndResume$' -v -count=1 -timeout=40m

This run proved:

Positive guest-local TCP/UDP listeners on both wildcard and loopback addresses stayed inaccessible from host loopback, after allowing time for Lima's asynchronous forwarding detection.
Two repositories imported at exact pins; separate writer worktrees did not overwrite siblings; repeated assignment preserved live edits; read-only stages rejected edits; captured commits exported as Git bundles and restored exact contents into a different revision source/worktree; host source files remained unchanged.
A real Codex worker implemented the slug fixture and its tests. Independent unittest discovery and fixed assertions passed. A separate real supervisor inspected and accepted it. Explicit continuation of worker session 01a08f59-7073-7a83-b959-867ca896ce48 completed successfully.

Failures found and fixed during this acceptance:

Session 65790 failed on host-created AppleDouble files (._pack-*.idx) inside the Git archive. repository.Archive now uses Go's tar writer, preserving repository files/modes while excluding OS xattrs/owner identity and normalizing timestamps. Portable/deterministic archive tests and the real bundle fixture pass.
Session 33216 passed network/repository checks but failed real tool execution because the separately-distributed codex-code-mode-host was missing. The worker correctly returned accepted=false. This was a harness installation error, not a test-layout error. The installer now installs and checksums the main binary and its required companion, and the subsequent real feature/supervisor/resume test passed.

Companion release pins from the official 0.154.0 release-assets API:

codex-code-mode-host-aarch64-unknown-linux-musl.tar.gz: SHA256 20aefa302c2022b496e32911bf954a5f76c7fd749c6bdb9fbd711e32b66dcbfa.
codex-code-mode-host-x86_64-unknown-linux-musl.tar.gz: SHA256 a68df7cca23c6da7cde175677df7de61c73a234add1333a1254b86d641af01f7.

No test session is live now. The private acceptance VM envctl-agent-acceptance is deliberately retained for the next harness/backend integration checks, with provider ownership state under /tmp/envctl-agent-acceptance. Reuse only for bounded fixtures; product workflow runs must continue to receive their own revision VMs. Destroy this owned test VM before final delivery. Do not print guest request or authentication files; they contain scoped credentials. The latest installer source also sets both installed binaries to root ownership (upstream tar archives retained UID 1001).

Current unit/race/vet checks pass for changed packages; the legacy Compose suite also passes after extracting the guest-safe transformation. The acceptance above proves the components, not a six-stage production workflow. CLI-created runs still queue because the real execution backend is not connected. No whole milestone is claimed complete without its full exit audit.

Primary next work: implement and connect the production backend (source/Compose/plugin preparation, real worker/supervisor/check dispatch and reconciliation, durable source/data checkpoints, PR broker), then complete invocation plugins, TUI review/streaming, parallel merge/data ownership, second harness, conformance and MCP. Also correct engine event scoping: store events currently default to the run's current revision even when the engine mutates a draining older revision; archived events need their actual target revision.

Fourth implementation pass: production local execution connection

Implemented:

internal/gueststack: guest-only Compose normalization; pure canonical-model decoding through loader.Transform; reuse of existing rendering transformations with guest-loopback port mode; immutable root-owned configuration/receipt; actual up/health/status/down; preserved volumes on ordinary stop/down; no host Compose paths or Docker context. Raw interpolated configuration/diagnostics are not returned as errors or artifacts.
internal/localexec: production backend for engine preparation, readiness, dispatch/reconciliation, cancellation, release, and an external capability/publication interface. The CLI-created daemon now attaches this backend. Real stage jobs use Codex worker/supervisor identities, explicit persisted sessions, separate repository worktrees, independent configured checks, source capture and bundles, supervisor review bound to the exact result, and coordinator-owned attempt phase/log/cursor receipts. Failed verified attempts can resume their explicit worker session from captured source. Publication is deliberately unsatisfied until a real broker is configured; no fake PR capability is admitted.
Runtime service/Compose-digest reporting into persisted workflow state. Known missing executable QA checks now appear as workflow.checks readiness work before Plan can advance.
workflow.Result.Sources: repository-keyed Git bundle references, covered by work/review digests and checked for repository coverage. Engine acceptance verifies retained bundle bytes, not only document artifacts. Restore loads bundles into guest object storage without overwriting another assignment's files.
Scoped store mutations correctly attribute draining-revision events; scoped operation replay cannot be reused for another revision. Existing unscoped command receipt hashing remains unchanged.
Harness/check processes set PYTHONDONTWRITEBYTECODE=1, preventing Python inspection from creating unintended untracked source files during read-only stages. Credential redaction covers structured result artifacts, and partial-dispatch cancellation skips absent or terminal jobs.

Verified:

Real guest Compose lifecycle PASS, session 15224, TestRealGuestComposeLifecycle, 27.84s, using the explicitly selected retained acceptance VM. Proved literal $literal survives interpolation, active health/guest HTTP endpoint, loopback-only published binding, immutable receipt replay after source corruption, volume preservation on restart, and detection of stopped services. Its stack and volume were removed by cleanup.
Real production-backend Task/Plan and restart PASS, session 81633, TestRealLocalPlanningAndCoordinatorRecovery, 86.77s, resumed from a real interrupted execution. Authoritative store: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-local-exec-run_6c734f19434761e5fd2d17f0. Real worker and separate supervisor produced an accepted Task checkpoint and a reviewed Plan result. The engine held Plan because publication lacked a prepared binding; no Design/Code/QA/change worker was dispatched. Store/backend/engine recreation retained the guest job and explicit session. Fixture stack cleanup succeeded; the shared acceptance VM remains retained for bounded tests.
Earlier integration sessions 59872 and 16919 were intentionally stopped after concrete defects were identified, with their scoped agent jobs cancelled and stacks removed. Defects: Python bytecode dirtied read-only sources; supervisor context was recomputed instead of using the worker's frozen assignment; source bundles were invalid as ordinary dotted artifact names. Session 74019 terminated on a nil-map panic after JSON omitted empty Sources; load now reconstructs the map, regression-tested, and session 81633 resumed its already-completed worker rather than rerunning it.
Full Go tests, vet, build and whitespace checks passed; race checks passed for localexec, gueststack, engine, runstore, workflow and agent. Targeted tests cover receipt reconstruction/input rebinding, conflicting join refusal, credential artifact redaction, partial-dispatch cancel, exact source-bundle review binding, missing-bundle rejection, and historical event scope.
Built-CLI smoke created a run through the Unix API and observed production backend dispatch into an explicit unsupported-provider recovery item. Evidence: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-cli-exec-2ix_r01j. Smoke coordinator was stopped. This proves the CLI/daemon wiring, not full feature completion.

No integration process remains live after session 81633. The retained VM is still envctl-agent-acceptance, ownership state /tmp/envctl-agent-acceptance; destroy it before final delivery. Guest requests and role authentication files contain scoped credentials and must never be printed.

Next required work (full original scope remains active):

Implement the real PR/Git publication broker and non-publishing destination readiness, preserving exact commits, approval, idempotency, and rewind fencing; run a full six-stage real feature demo. Task/Plan acceptance above is intentionally not a substitute for it.
Finish guest plugin installation/lifecycle, frozen invocation locks, dynamic CLI/TUI attachment, executable requirement inventory and plan amendments, credential renewal, browser and database/emulator adapters.
Exercise the backend's Design/Code/QA pipeline with real checks, failed-test correction, checkpoint restore across revision VMs, and crash reconciliation. Add broader submodule/LFS handling and exact toolchain/image replay evidence.
Propagate guest service endpoint environment to workers/checks and implement explicit local preview forwarding. Current guest-loopback services are private and service health is reported; a host preview URL is not yet implemented.
Complete live steering/event streaming, rich diffs/history/comparison/approval UX, consistent dataset restore, rewind/drain/source reuse validation, overlap/capacity/repeated-rewind and two-client demos.
Implement local M7 parallel writer/merge/fixture/child-runtime ownership, global capacity/budgets/priorities, another harness, conformance and MCP. The current production backend refuses parallel assignments rather than sharing mutable stack state.
Audit every milestone exit requirement before marking any checkbox or the goal complete. No whole milestone has been marked complete.

Fifth implementation pass: real publication, invocation plugins, and interrupted source

Implemented:

internal/publication: coordinator-side GitHub broker using host-owned Git object stores and gh authentication. Plan checks repository access, source/base identity, PR lookup and authenticated receive-pack dry run without creating a branch. Destination/base SHA locks and per-repository publication intents bind the output to the reviewed result. Publication requires exact QA/approval, creates revision-specific branches only when absent, reconciles lost PR-create acknowledgements, rejects moved/colliding branches and base drift, and creates verified draft PRs. Unchanged read-only dependencies remain in the approved commit set without requiring artificial PRs. Current PR bodies include the complete commit set and retained QA evidence digests; this body enrichment was added after the real feature binary started, so its extra body fields were not part of that live demonstration.
workflow.PublicationTarget and revision output-branch naming. TUI Changes includes per-repository source/checkpoint commits, output branches, destination and PR URLs. The production backend now uses the real broker. Local sources need an explicit publication target; a direct GitHub branch source can resolve its target automatically.
Pinned guest Docker Buildx package 0.30.1-0ubuntu1 preparation for Compose builds, with a guest tool receipt. This is currently tied to the supported Ubuntu guest; it does not establish portable toolchain replay across providers/images.
plugin.Freeze: bounded, digest-verified package retention; transitive capability resolution; frozen selected capabilities/prerequisites and credential references. JSON-schema configuration validation normalizes YAML/native integer inputs. Packages have a 32 MiB/4096-file limit and reject links/non-regular files.
CLI run create --plugin-file, run plugin add --file, and run plugin remove --id; Bubble Tea p/P and Readiness bindings. API attachment freezes a package, creates a new revision and reopens Plan, preserving active old bindings. Identical bindings do not create extra revisions. Command receipt replay precedes rereading mutable/deleted package sources; API tests cover two invocations, stale/live revision preservation, removal, and lost responses.
Real guest plugin prepare/probe/execute/renew/cleanup protocol. Root-owned package installation validates source bytes, modes, and platform; mutable plugin state is separate. Systemd job identities and coordinator operation receipts preserve in-flight work across transport loss. Structured stdout and stderr remain separate; credentials enter stdin and are sanitized before evidence serialization. Prerequisites and credentials gate dispatch, successful preparation is reused, expiring preparation invokes supported renewal, and cleanup runs in reverse dependency order.
Production readiness includes frozen plugin capabilities and descriptor prerequisites. Built-in verification/publication bindings cannot be replaced by arbitrary plugin claims. Plugin credential values are also included in the worker/check artifact redactor. Terminal executable/protocol errors still require explicit recovery/revision work; generalized plugin process retry and external-effect compensation are not yet implemented.
Early-worker recovery captures allowed tracked/untracked edits into retained Git bundles even when the worker exits before producing structured output. A separate recovery record preserves these commits without creating a proposal or checkpoint. New attempts restore the retained work into fresh worktrees, record their failed-attempt lineage, and resume a known session when one exists. Read-only inputs are restored from accepted pins; source-export transport failures keep the old worktree pending recovery.

Verified:

Real six-stage, two-repository feature PASS, session 84097, TestRealTwoRepositoryFeatureThroughPR, 623.01s. Command: ENVCTL_FEATURE_TEST=1 ENVCTL_AGENT_STATE_DIR=/tmp/envctl-agent-acceptance ENVCTL_AGENT_VM=envctl-agent-acceptance ENVCTL_GITHUB_REPOSITORY=sam-bretz/envctl go test ./internal/localexec -run '^TestRealTwoRepositoryFeatureThroughPR$' -v -count=1 -timeout=27m. Retained store: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-feature-acceptance_ac7579e41fbf610c27571ff8. Real Codex workers/supervisors, independent API and HTTP-client unittest checks, six accepted checkpoints, complete two-repository source bundles, and coordinator reconstruction during Code passed. The explicit acceptance-test actor exercised approval; this is fixture approval, not a claim of human review. Verified PRs were https://github.com/sam-bretz/envctl/pull/1 and /pull/2. Cleanup closed both fixture PRs, deleted their owned output/base branches, removed the stack/volumes, and wrote acceptance-cleaned.json. PR bases were isolated fixture branches, not main.
The first full-feature setup failed configuration validation before a workflow existed because checks referenced repositories absent from the initial template. The fixture now validates before external setup. Its two created base branches were checked against their exact fixture SHAs and removed; the failed setup store has acceptance-setup-cleaned.json.
Real non-publishing GitHub readiness PASS, session 11715, 4.03s. Verified no branch appeared after repository/source/base/PR and authenticated Git push dry-run checks. Publication unit tests use real source/bare Git repositories and a fake hosting API to prove lost-create-response reconciliation, one PR POST, moved-branch refusal, exact approval/QA, base drift, and unchanged dependency handling. A non-publishing probe cannot guarantee future network availability or every hosting-policy outcome.
Real guest Compose lifecycle PASS after Buildx preparation, session 42860, 24.97s. Scoped stack cleanup succeeded. The package was installed and its actual version checked on the retained test guest.
Real guest plugin lifecycle/reconnect PASS, session 51171, TestRealGuestPluginLifecycleAndReconnect, 5.18s. Proved missing pre-prepare readiness, prepare/probe/execute/renew/cleanup, caller disconnect after the job was actually running, one execution on reconnect, escaped-secret redaction, no host source writes, and tampered guest-package rejection. Fixture package/state was removed.
Real production plugin readiness component PASS, session 65432, TestRealInvocationPluginReadiness, 2.77s. Missing dependency/credential prevented operation dispatch; adding the credential allowed real preparation/probing; backend reconstruction reused preparation; an expired lease invoked real renewal; cleanup replay passed. Built-in prerequisite observations were fixture inputs to this component test, so it is not yet the complete M3 agent/attachment demonstration. Fixture package/state was removed.
Real early worker subprocess exit recovery PASS, session 80504, TestRealEarlyWorkerExitPreservesUncommittedSource, 0.90s. Injected exit 7 before structured output; recovered tracked and untracked files after deleting the old worktree and recreating the backend; prepared a new attempt from retained source; original host files remained unchanged. No model session was fabricated and no checkpoint was accepted. This is a guest process/source fault-injection test; the separate earlier real Codex session-resume test remains the model continuation evidence. Fixture source/receipts were removed.

Final checks: full go test ./..., go vet ./..., CLI build, plugin command help smoke, and git diff --check passed (session 79418). Race tests passed for plugin, localexec, publication, engine, runstore, daemon, TUI and workflow (session 57658); daemon/localexec races and targeted checks were repeated after the final attachment-idempotency and feature-test audit edits (session 87422). No whole milestone is marked delivered.

Next required work:

Add real browser/test, database snapshot/restore and emulator adapters and connect their operation/data contracts to checkpoint capture/restore. Run the complete missing-QA-capability → invocation attachment → real Plan completion demonstration.
Finish dynamic requirement inventory/plan amendments, active message delivery, terminal plugin recovery, credential renewal edge cases, and source/toolchain/image/submodule/LFS replay coverage.
Complete M5 against the real backend: active worker drain, distinct revision VMs, source/data restore, repeated rewind and capacity queue, plugin edits, two-client review, diffs and checkpoint comparison.
Finish local M7: separate parallel writers, verified merge ownership, isolated mutable tests/child runtimes, limits/priorities/no-progress strategies, second harness, provider/plugin conformance and MCP/API exposure.
The real feature fixture now persists coordinator-restart evidence and requires Code to be running at the observed restart. A later resume cannot satisfy that assertion merely by finding a completed Code attempt. This test audit improvement was compiled after the passing live run and will be exercised on the next complete feature fixture; the existing run's active restart was verified from its live observations.
Full milestone exit audit and delivery documentation. Remote provider implementation remains excluded; local requirements above remain in scope.

The owned acceptance VM envctl-agent-acceptance remains retained under /tmp/envctl-agent-acceptance for subsequent bounded fixtures. No integration process remains live after the tests above. Never print its authentication/job-request files, and destroy this VM before final delivery. Real workflow defaults still allocate dedicated revision VMs; shared use of this explicitly selected fixture VM is only a test optimization.

Sixth implementation pass: PostgreSQL checkpoint data

Previous turn classification: progress. The fifth pass changed executable state and obtained real publication/plugin/source-recovery evidence. No blocker is being claimed; the full M1-M5/local M7 scope remains active.

Implemented:

workflow.DataConfig, PostgreSQL dataset definitions, seed repository/file references, application SQL verification, and an explicit writer-quiescence list. Reject reserved/connection-string database targets, seed path escapes, duplicate/aliased datasets, unknown seed repositories, and multi-dataset configurations without a writer procedure. Optional empty configuration is omitted from JSON using Go's omitzero, preserving existing invocation/attempt digests.
internal/checkpoint.Postgres: real container-local seed, probe, custom-format dump, and database reconstruction/restore. Inputs and outputs are bounded and checksummed; dump format, exact pg_dump version, configuration identity, and retained verification artifacts bind restoration. Commands resolve one container from the selected guest Compose project, use its local PostgreSQL socket, and run as durable guest jobs. SQL seeds are limited to 1 MiB by the backend; dumps are limited to 64 MiB. No host database or Docker context is used.
The backend obtains SQL seed bytes with git show <pinned-SHA>:<seed-file>, retains them, seeds the guest database, and exercises a real dump restoration during Plan. Subsequent readiness probes perform application verification without resetting active work. Initial dataset restoration can inherit the selected checkpoint's retained dump.
Stage startup restores direct predecessor dataset inputs. Conflicting dataset joins require an explicit merge/fixture policy. Stage completion captures data after checks and before independent supervisor review. Dataset manifests and artifacts participate in the work digest and engine artifact verification. Approved change retains the immutable QA dataset and cannot substitute different data even if repository commits match.
Durable writer-quiescence ownership prevents readiness or retry dispatch from auto-starting writers during an unfinished data operation. Restart waits for actual service health; completion syncs marker removal. The baseline completion/marker boundary is recoverable. The provider remains serial until M7 ownership is implemented.
CLI run rewind <run> --to plan --config <complete-v2-file> creates an invocation configuration revision, allowing Plan requirements to be amended without editing project defaults. Existing API revision/version checks and package freezing apply.

Real evidence:

PostgreSQL checkpoint/restore/isolation PASS, session 73149, TestRealPostgresCheckpointRestoreAndIsolation, 8.83s. Two distinct Compose PostgreSQL services and volumes inside the explicitly selected acceptance VM. Proved SQL seed, versioned retained dump, restoration into the other stack, independent source data, rejection of failed application verification, failed restore after target rebuild, reconstruction from retained bytes after adapter recreation, removal of extra objects absent from the snapshot, and replay of the same seed operation/dump. Both stacks and volumes were removed. This proves data isolation within the guest, not the complete two-revision VM rewind demonstration.
The pinned test image was pulled from Docker Hub as postgres:17.6-alpine, resolving to postgres@sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94; the fixture uses the digest. Actual dump tool version was checked as PostgreSQL 17.6.
Initial backend dataset test session 32530 failed after successful seeding/restoration because the installed Compose start command does not implement --wait. Actual guest help confirmed its supported flags. Writer restart now uses plain start plus bounded service-state/health polling.
Production dataset readiness/stage integration PASS, session 78537, TestRealDatasetReadinessStageRestoreAndQuiescence, 92.84s. Proved pinned seed content despite a changed checkout file, real Plan seed/restore verification, durable quiescence across backend recreation, no readiness auto-restart of stopped writers, stage capture, and QA startup restoring the exact Code dataset while removing later stray objects. Fixture stacks, volumes, source and dataset directories were cleaned. This component test uses real PostgreSQL and guest operations but does not dispatch model jobs; the earlier six-stage model demo did not yet include datasets.
Unit tests cover manifest completeness, artifact sizes/media, configuration rebinding, changed-data review invalidation, exact QA-to-approved dataset identity, retained dump/evidence verification, unsafe configuration and old receipt serialization compatibility.

Final verification: full Go tests, vet, build and whitespace checks passed (session 96257). Race tests passed for checkpoint, localexec, gueststack, workflow, engine, daemon and plugin (session 89721). The PostgreSQL integration passed again after tightening retained-evidence/media validation (session 84722, 8.83s), with cleanup complete. The final CLI amendment flag was built and checked through command help (session 88374). No live integration process remains.

Current remaining data work (not a completed M3/M5 claim): cluster-wide role/ownership/ACL restoration; the HTTP/emulator fixture and browser adapter; generic plugin-backed data adapter operations; a real multi-dataset consistency demonstration; automatic recovery generations for terminal data jobs; dataset diff/review/export UX; and real cross-VM rewind/capacity/drain acceptance. The v1 PostgreSQL dump intentionally omits original ownership and grants and therefore does not prove arbitrary-cluster replay. Temporary implementation limits remain visible rather than being treated as fulfilled requirements.

The acceptance VM is still retained and owned by /tmp/envctl-agent-acceptance. Destroy it before final delivery. No milestone checkbox is complete. Full scope, including remaining M7 local work, is unchanged.

Seventh implementation pass: browser checks, emulator data, Plan drain, and review

Previous goal turn classification: progress in the planning documents; implementation was still incomplete. This pass resumes the full M1-M5/local M7 objective. No milestone checkbox is complete and no external blocker is being claimed.

Implemented this pass:

Generalized the durable checkpoint transport to checkpoint.Client (retaining Postgres as an alias). PostgreSQL and http-fixture have distinct validated formats/media types, application verification, seed media, and commit/review-bound dataset manifests. Old PostgreSQL request/config identities remain unchanged when the new optional verify_key field is absent.
Added the embedded HTTP JSON record emulator, pinned Python image, atomic seed scaffold (envctl run fixture init), lock-consistent capture/restore, application verification before replacement, removal of absent records, and recovery over corrupt target state. The service is non-root and restore is container-local rather than an HTTP management endpoint.
The bundled builtin:playwright package prepares pinned Chromium/Node tooling, probes the VM-local application, runs interactions/assertions, and retains screenshots. Workflow checks support exactly one command or invocation-plugin executor; missing plugin bindings leave Plan incomplete. The browser implementation begun before the intervening planning turn is now included in the integrated acceptance path.
Fixed superseded Plan drainage: reviewed outputs can be archived historically even if the old binding remains unresolved. Ordinary Plan admission still requires executable readiness. Historical checkpoints cannot supply runnable DAG dependencies, and normal Accept on draining work routes through historical archival.
Readiness source selection now uses the selected restoration checkpoint's commit set, so a revised Compose definition/Dockerfile is not replaced with the original repository baseline. Missing/incomplete/historical selected inputs fail explicitly. Unit coverage passes; complete separate-VM restored-stack acceptance remains outstanding.
Bubble Tea now supports revision browsing ([/]), a History panel, artifact selection (,/.), readable checkpoint/reviewer/approval/dataset summaries, and compact layouts. Historical browsing cannot mutate current execution. Late artifact responses are scoped to their view; steering entered for a replaced revision is rejected. Source diffs and complete comparison remain unfinished.
Added envctl run artifact <digest> --output <new-file>. The client checks download hashes and rejects transfer truncation/oversize; exports are atomic/private and refuse existing files or symlinks.
Added per-attempt agent-owned scratch directories and corrected read-only-stage instructions to forbid new untracked report directories in source. The live feature test exposed that prompt/guard mismatch; its first QA attempt was correctly rejected for .qa-artifacts/, and the existing retry completed QA successfully. Scratch setup has an explicit assertion in the production dataset integration test; final regression evidence is recorded below.

Completed acceptance evidence:

HTTP emulator component PASS, session 50768: 35.75s plus scaffold test. Real HTTP mutation, retained seed/capture, separate Compose volumes, exact restore, failed application verification, and replay after adapter recreation.
HTTP emulator recovery and PostgreSQL regression PASS, session 58261: HTTP 36.17s, PostgreSQL 8.80s. Added corrupt-target recovery; both real adapters cleaned their fixture stacks and volumes. This is separate-stack isolation inside the explicitly reused acceptance VM, not evidence of separate-VM isolation.
CLI scaffold PASS: built binary created a nested fixture, docker compose config --quiet validated its repository-root build context, and repeated initialization refused overwrite. The temporary fixture directory was removed.
Full Go suite, vet and build passed (session 16377); workflow/engine/localexec/TUI/checkpoint race tests passed (65416). Later artifact-client/export race tests passed (90195), and targeted tests passed after scratch-directory changes. Run final checks after any remaining edits.
Astro documentation build passed (31987). The docs workspace had no npm lockfile: npm ci was unavailable, so dependencies were installed without creating a lockfile and the site built successfully. Fixtures guide and agent guide document implemented commands and remaining limitations.

Completed integrated acceptance:

PASS, process session 81856, TestRealFeatureWithBrowserDatasetsAndPlanAttachment, 1254.34s. Six real stages, three revision histories, two repositories, browser QA, both dataset manifests, coordinator restart, QA failure/retry, fixture approval, PR heads and cleanup were verified.
State: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-feature-acceptance_71670c412724c803f9693956.
Run: run_0e112bf9063ef487b4be9476; final current revision: rev_0ebd93eade3aee586a29a247.
Original rev_7cd596772ae1a1c78aac8fa0: real worker/supervisor Plan output held with no browser binding. Its reviewed Plan is now historical.
Intermediate rev_3e9a2ef4278196f830b4d966: API attachment of the pinned Playwright package with deliberately unavailable port 18090 produced a real Chromium probe failed response. Plan remained unaccepted, and its reviewed output is now historical.
Current revision: API replacement pointing at port 18088 passed the Chromium probe; Plan, Design, Code and QA are accepted. The coordinator was recreated while real Code was running, with durable coordinator-restart.json evidence. QA has three passing checks and two datasets. Approved-change completed with the exact QA commit set and dataset digest. PRs #3 and #4 were verified and closed; fixture branches and stack/plugin resources were removed.
Fixture-only branches/PRs target sam-bretz/envctl. Successful completion closes PRs, deletes fixture base/output branches, removes plugin resources, and tears down the current stack. Failed/timeout execution retains its store and guest journals for same-run resumption via ENVCTL_FEATURE_RESUME.
This bounded test reuses the selected VM only after each superseded Plan drains and its old stack is removed. It does not substitute for the separate-VM overlap/drain/capacity demonstration required by M5.

Remaining full scope includes generic plugin terminal-failure recovery and Plan requirement discovery/amendment; complete PostgreSQL role/ownership/ACL and broader adapter replay; multi-service application consistency; actual cross-VM restored-stack and rewind/capacity demos; source diffs/full TUI comparison and live steering/progress; and all remaining local M7 parallel writer/merge/fixture/child-runtime/budget/second-harness/conformance/MCP work. The acceptance VM envctl-agent-acceptance remains owned by /tmp/envctl-agent-acceptance for ongoing work and must be destroyed before final delivery. Never print its authentication or guest request files.

Post-run audit: acceptance-audit.json in the retained feature state verifies 27 distinct artifact checksums/sizes, all three QA checks, both datasets, exact approved/QA commits and dataset identity, closed PR states, and deletion of all four fixture branches. API SHA aca01a1ad42a42b675c72af372a4797adf54b496; client SHA 0afa6442b5ffb51445bd52a3da461919afe07763. The decoded review/browser-qa.png was inspected: the real page shows inputs 2 and 3 and result 5. Guest docker ps was empty after full-feature cleanup, before the final isolated regression began.

Final scratch/data regression initially failed (session 3297, 35.06s) because the reusable test helper used constant attempt IDs. The guest correctly rejected the old attempt_one_before receipt from a different revision. Test fixtures now use globally unique attempt IDs, matching production identities. Replacement session 25591 PASS, 93.06s: pinned seeding, Plan restore readiness, durable writer quiescence across backend recreation, private agent-owned scratch directory, Code data capture, and exact QA predecessor restoration all passed. No product guard was relaxed.

Final verification: latest full Go suite, vet and CLI build passed (session 86321); artifact download/export and TUI race coverage passed (90195), with earlier workflow/engine/localexec/checkpoint race coverage (65416). Latest documentation build passed (1546). Embedded Python syntax and CLI scaffold/Compose validation passed. Completed acceptance jobs are terminal; no live feature continuation is pending. Full milestone acceptance, including cross-VM rewind and production parallel execution, remains incomplete.

Eighth implementation pass: plugin recovery and retained checkpoint comparison

Previous goal turn classification: progress in the authoritative planning documents. This implementation pass retains the full M1-M5/local M7 objective. No milestone is marked complete and no external blocker is claimed.

Implemented and verified:

Plugin guest execution distinguishes confirmed terminal process/protocol failures from transport uncertainty. Terminal failures produce sanitized retained evidence, increment a durable process generation, and retry with exponential backoff under the invocation's attempt limit. The external plugin operation ID remains stable so its implementation can reconcile effects. Generation zero preserves existing job identities; transport failure reconnects to the existing process. Fresh lifecycle operations archive their preceding completed receipts. Exhaustion remains unresolved planning/recovery work, never passing readiness.
Real terminal recovery PASS, session 35600, TestRealPluginTerminalRecoveryPreservesOperationIdentity, 7.00s. The fixture wrote one durable effect, exited 7 with a synthetic credential on stderr, returned invalid JSON on the next process, then succeeded. Recreating the backend between failures retained identity, sanitized evidence and backoff; three processes performed one effect, and success replay launched no new process. The descriptor originally lacked its mandatory probe operation; it was corrected before the successful guest run. Fixture package/state cleanup completed.
Real lifecycle/reconnect regression PASS, session 27300, TestRealGuestPluginLifecycleAndReconnect, 5.17s. Guest prepare/probe/execute/renew/cleanup, observed in-flight disconnect, single execution on reconnect, escaped-secret redaction and package tamper refusal still pass. Fixture resources were cleaned.
Coordinator fault-injection tests verify transport loss after successful submission does not increment generation or launch a second process; terminal retry exhaustion prevents further dispatch after backend recreation; failure artifacts remain available; fresh probes archive previous evidence.
New internal/review reconstructs source comparisons from checksum-verified retained Git bundles in temporary bare repositories. It uses no VM, source checkout or remote access, disables inherited Git configuration/external diff drivers, validates exact commits, bounds patches to 1 MiB per repository, and preserves metadata when source is unavailable. Comparisons include both checkpoint records, objectives/configuration identities, checks, supervisor review, approvals, source commits, and artifact/dataset identities.
Read-only daemon GET /v1/runs/{id}/diff and CLI envctl run diff <run> --node <stage> [--revision <id>] [--from <checkpoint-id>] [--json]. An omitted base uses the target revision's source pins; explicit checkpoint bases must belong to the same run. This API does not mutate workflow state.
Bubble Tea d opens comparison, b selects a recorded checkpoint as the base, B resets it, and Escape closes loaded review. Bases can span historical revisions and remain client-local. Late diff responses cannot replace a selected artifact, appear in another panel, or reopen a closed comparison. Existing artifact and steering scope checks remain intact.
Retained real feature comparison PASS, TestRetainedFeatureCheckpointComparison, 0.22s. Store /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-feature-acceptance_71670c412724c803f9693956, run run_0e112bf9063ef487b4be9476. Both repository diffs are available and nonempty from Plan to Code. QA matches Code source and exposes three checks and two datasets. Historical/current Plan configuration identities differ as expected for plugin amendment. The complete workflow digest was unchanged. Results are retained in acceptance-review.json; this test never connects to the VM, source repositories, harness or publication API.
Real Git unit fixtures remove their original checkout before review, compare across revisions, inject inherited external-diff/Git-dir settings, and verify no external command ran and no workflow mutation occurred. Additional coverage verifies corrupt source reporting, output limits, terminal escape filtering, API scope and historical TUI navigation.

Final checks: full Go suite, vet, CLI build, command help and diff whitespace PASS (session 90528); race coverage PASS for plugin, localexec, review, daemon, TUI and CLI (25309); Astro documentation build PASS (50843). Latest binary: /tmp/envctl-review-cli. All processes started by this pass are terminal. README, agent guide, CLI reference and runtime design describe the new behavior and limitations.

Still required for the full objective: structured Plan requirement discovery/amendment; renewal and repair of prepared plugin resources that later disappear; complete PostgreSQL role/ownership/ACL and generic dataset replay; application-consistent multi-service snapshots and terminal data-operation recovery; separate-VM active rewind/drain/restore/capacity/repeated-rewind demonstrations; live steering and detailed progress/local preview access; remaining review usability and comparisons beyond artifact/dataset identities; and local M7 concurrent writers, verified joins, isolated mutable fixtures/child runtimes, assignment budgets/priorities, another harness, conformance and MCP. No narrow component test above establishes those requirements.

The owned acceptance VM envctl-agent-acceptance remains retained under /tmp/envctl-agent-acceptance for further local acceptance work. Its credentials and guest request files must not be printed. Destroy it before final goal delivery; no new VM, branch or PR was allocated this pass.

Ninth implementation pass: dedicated-VM rewind acceptance and offline baselines

Previous goal turn classification: verified implementation progress on plugin recovery and checkpoint comparison. Full M1-M5/local M7 scope remains active.

Implemented repository.Cache and connected it to local runtime preparation. It retains complete baseline archives outside revision VMs, indexed by source URL and exact commit. Archive and index publication are synced and atomic; each reuse validates size and checksum. Output branch/repository metadata come from the new invocation. Unpinned refs still resolve against their source, while pinned restores can proceed after deleting both the original repository and the earlier preparation checkout. A missing/corrupt recorded archive is surfaced rather than silently replaced.

Verification so far:

Real Git retention tests prove offline recovery, current-ref resolution for unpinned invocations, and corruption/missing-archive refusal. Repository/localexec race tests passed (62917).
Real retained baseline guest import PASS, session 85685, TestRealRetainedBaselineGuestImport, 0.84s. Original and host preparation directories were deleted before two fresh guest imports. The first guest worktree was changed and committed; the second still imported the original pin and file contents. Fixture directories/receipts were cleaned. This component test reused the separate idle envctl-agent-acceptance VM; it is not the dedicated-VM isolation demonstration.
Full Go suite, vet and CLI build passed (53426). /tmp/envctl-rewind-cli contains the current retention implementation.

Dedicated-VM integration is running, not yet accepted. Process session 84063, TestRealActiveRewindAcrossDedicatedVMs, command ENVCTL_REWIND_TEST=1 go test ./internal/localexec -run '^TestRealActiveRewindAcrossDedicatedVMs$' -v -count=1 -timeout=45m. Retained state: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-rewind-acceptance_e182e44cfed44c37179a7d6a. The test supports same-run continuation through ENVCTL_REWIND_RESUME=<state-directory>; never start another fixture because an observation call times out.

Observed: original revision rev_296f225f6c27d0cf24bdc907 completed real Task, Plan and Design checkpoints and entered Code. API rewind to Plan created rev_8cb8d0226b869ef574668122; original Code is draining and the replacement VM is preparing with only Task reused. The intended remaining assertions are simultaneous distinct VMs/daemons/data, independent client review, coordinator recreation, historical old Code completion, a subsequent QA rewind from accepted Code, and source/Compose/dataset restoration in another VM. The local five-stage fixture intentionally has no publication destination; earlier M4 evidence covers six-stage PR output. This test binary was built before baseline-cache wiring, so its rewind results must not be described as coverage of the new cache path.

On success the fixture destroys its owned VMs and writes acceptance-cleaned.json. On failure it retains jobs/state for diagnosis and continuation. Do not print any guest authentication or job request files. No remote VM/provider or PR is created by this fixture. Update this section with terminal evidence before claiming the integrated assertions passed.

Ninth-pass continuation: session 84063 terminated after 993.14s because its assertion sampled replacement dataset contents while Plan readiness was still pending. The assertion now waits for successful current readiness before inspecting baseline data. This does not relax the engine's Plan checkpoint gate. Session 12794 resumes the same retained run using ENVCTL_REWIND_RESUME; it does not replace live guest jobs. It has verified simultaneous independent revision VMs/Docker daemon identities/private data, independent Bubble Tea client selections, coordinator recreation, and original Code completion as historical work. Replacement Code completed and a second API rewind has created rev_9952f25fee5d16d669509c76 to restore Code into a third VM. Final restore and completion assertions remain pending. Unlike the first binary, the resumed binary includes baseline retention.

Additional baseline coverage preserves the exact legacy import tar during migration, even if harmless Git metadata subsequently changes. Repository/engine race tests passed (38503). TestCapacityWaitAndRepeatedRewindSurviveSchedulerRecreation passed with race detection (87702): a one-VM capacity queues replacement revisions, repeated rewind supersedes the earlier queued revision without allocating it, and the old Code result is retained historically before capacity is released to the newest revision.

Tenth implementation pass: explicit plugin resource repair

Resource disappearance is now a distinct protocol-level recovery path. Only failed probes may request a declared prepare or renew operation. The coordinator durably records the failed probe ID and preceding lifecycle operation before dispatch; lost acknowledgements reconnect to the replacement operation and fresh verification probe. Repeated unsuccessful repair/verification cycles obey backoff and the invocation attempt limit. A passing fresh probe resets the incident budget. Invalid recovery requests are terminal protocol errors, and unspecified connection failures do not trigger resource mutation.

Real guest resource repair PASS, session 54895, TestRealPluginRepairsDisappearedResource, 4.13s. After successful preparation, the fixture's resource was deleted. Its real probe failed and requested preparation; backend recreation preserved the intent; exactly one replacement effect plus a fresh probe restored readiness. Original evidence remained retained; cleanup completed.
Localexec/plugin race tests PASS, session 85716. Fault injection covers lost repair and verification acknowledgements, separate prepare/renew recovery, fresh verification, unchanged external-operation identity on reconnect, bounded repeated failures, and failed probes without repair requests.
The built-in Playwright probe now requests preparation for a missing tools volume or pinned image. Its expanded real Chromium acceptance test is running in the dedicated browser acceptance process (terminal result pending); it removes only its invocation-owned resources, then verifies automatic repair and a fresh browser check. No portability or complete milestone claim follows from these component tests.

Dedicated-VM rewind integration PASS, session 12794, 634.16s for the resumed process. The original fixture (same retained run) completed terminal QA in the restored revision and verified historical old Code, no old QA, separate daemon/source/data ownership, two independent review clients, coordinator restart, and exact selected Code source/Compose/dataset restoration. The fixture destroyed all its owned revision VMs and wrote acceptance-cleaned.json. This is the M5 integrated rewind evidence; other M5 requirements and local M7 still remain.

Expanded real Chromium acceptance PASS, session 92180, TestRealBrowserCheckPlugin, 35.98s. Missing binding, real preparation/probe, input interactions, failed assertion, screenshot dimensions, and restart replay passed. The test removed its owned browser containers/tools volume, observed failed readiness and a durable prepare request, recreated the backend, repaired the resources and passed a fresh browser interaction check. Final plugin/stack/directory cleanup completed. Only the separate component acceptance VM remains retained for further work.

Eleventh implementation pass: structured Plan requirement discovery

New Plan harness contracts require a structured requirement array naming capabilities, affected DAG nodes and reasons. Supervisor review binds that inventory to the work digest. The workflow validates names, scope and bounded inventories; discovery adds admission obligations without weakening configured requirements. The engine persists reviewed discoveries before its readiness wait, immediately schedules a readiness refresh, and retains the Plan result instead of repeating the worker. Reconnect uses each already submitted invocation's frozen schema. Same-objective/same-DAG revisions retain discoveries, including plugin attachment; an objective or DAG amendment reopens scope discovery.

CLI readiness and Bubble Tea Readiness show the inventory; checkpoint summaries and comparisons retain reviewed requirements. Scheduler tests prove a connection absent from the configured graph holds Plan across coordinator recreation, cannot dispatch downstream, and advances only after its probe passes. State/contract tests cover invalid inventories, non-Plan declarations, changed supervisor digests, retry retention and amended objectives. Full Go suite passed (92000); vet passed. Further race/docs checks and the real agent acceptance result remain pending.

Real Plan discovery acceptance running, session 97031, TestRealPlanDiscoversAdditionalReadinessRequirement, in the separate component VM. Retained state /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-local-exec-run_47c3c8a2bfa0a0e8b3a2be86; resume via ENVCTL_PLAN_DISCOVERY_RESUME with the same state/VM, never replace a live job after an observation timeout. The fixture asks for arithmetic.fixture for QA without declaring it in the configuration. It must appear in the real reviewed Plan and executable admission inventory, with no downstream dispatch. Coordinator recreation during the real worker has already been observed; final assertion remains pending.

Real Plan discovery PASS, session 97031, 146.38s. The real worker/supervisor pair returned arithmetic.fixture for QA with a reason. The coordinator retained it in the inventory and unresolved readiness; Task was checkpointed, Plan retained its result, and no downstream attempt was created. Coordinator recreation during live work passed. The fixture's Compose stack was stopped; no model job remains active. Stored source/artifacts remain available for later audit.

Eleventh-pass checks: full Go suite/vet passed (92000 and direct vet); affected-package race checks passed (69068); Astro build passed (63675). These results precede the MCP addition below.

Twelfth implementation pass: local agent MCP interface

Added internal/mcpserver and envctl mcp serve over the existing daemon client. The server has no scheduler or store authority of its own. Tools expose run inspection, event cursors, retained diffs, scoped artifact previews, creation and a restricted set of existing mutation commands. Stable caller operation IDs and exact version/revision fields preserve daemon replay/conflict semantics. Optional fixed-run/read-only modes restrict the registered tool surface; artifacts must be referenced by the selected run. Approval/publication are excluded from agent tools.

The official MCP Go SDK is pinned at v1.7.0. Protocol tests use real MCP sessions against the HTTP daemon and actual SQLite store; they pass for two-client replay, stale version/revision rejection, rewind history, event cursors, read-only registration, cross-run access refusal, artifact references/truncation, unsupported actions and malformed inputs. No test coordinator provisions VMs or starts agents. Compiled CLI stdio acceptance and race/full checks are running; record terminal results before claiming completion.

Twelfth-pass terminal evidence:

Compiled CLI stdio PASS, session 95932, TestRealCLIStdioReconnectLeavesCoordinatorRunning, 0.27s. Real process handshake, scoped mutation, MCP disconnect, read-only reconnect, and independent coordinator lifetime passed. The temporary Unix-socket API/store were shut down and removed; no VM or agent was created.
MCP/daemon/CLI race coverage PASS, session 43911; expanded MCP plugin-attachment/config-amendment replay race test PASS (40594). Config replay reconstructs paths from the originally requested revision, preserving operation identity even after another client changes the current directory/configuration.
Full Go suite PASS, session 86875; vet and diff whitespace PASS; Astro build PASS, session 32234. CLI /tmp/envctl-mcp-cli builds and exposes the new command help. The source test added after the full suite passed its own race run; no product code changed after that suite.

All acceptance processes started in this turn are terminal. The dedicated rewind VMs have been removed; the separate envctl-agent-acceptance VM remains owned by /tmp/envctl-agent-acceptance for subsequent local acceptance work and must be destroyed before final delivery. No remote runtime, branch or PR was created this turn.

Remaining requirement audit: M1-M5/local M7 are still the complete active objective. The integrated M4 and M5 paths now have real evidence, but this does not finish every milestone work package. Still required are complete source/submodule/LFS replay; credential rotation during pending operations and further renewal cases; full database role/ownership/ACL and generic dataset restoration, cross-service consistency and terminal data-operation recovery; remaining restore-crash cases; live steering, detailed progress and local previews; production parallel writers with verified merges, isolated mutable branch fixtures/child runtimes, per-assignment budgets/priorities and partial-branch recovery; an additional harness and conformance coverage; and final release documentation/acceptance audit. Local MCP command exposure is implemented and tested; remote adapters remain the only excluded delivery track.

Thirteenth implementation pass: portable submodule and LFS checkpoints

Previous turn classification: progress, with verified dedicated-VM rewind, resource repair, structured Plan inventory and local MCP commands. The full M1-M5/local M7 goal remains active.

Added a versioned source companion artifact alongside each main Git bundle. It retains nested submodule bundles and LFS content by checksum, binds their manifest to the root SHA, and validates every declared parent gitlink and LFS pointer during offline hydration. The companion is included in the work digest, retained-artifact verification, failed-worker recovery, checkpoint restoration, review metadata and scoped MCP artifact references. New assignment submodules have independent Git repositories and files; main worktrees continue to share only their content-addressed Git/LFS stores inside their revision VM. Existing assignment receipts preserve live edits on replay.

Recursive capture commits module work before parent gitlinks and rejects read-only nested changes. Resolver preparation now retains complete module paths (including spaces) and fetches/hydrates LFS in nested repositories as well as the root. Bundle restoration explicitly disables recursive submodule fetching. Companion hydration stages clones outside the working tree, installs them atomically, replaces LFS files atomically, and verifies refreshed indexes against the exact commit. A damaged guest companion cache can be rebuilt from verified source or restored from retained bytes. The format has explicit 64 MiB / 256-repository limits.

Evidence:

Nested offline source replay PASS, session 53882, 7.61s: two levels of submodules and LFS after deleting original repositories, read-only rejection, new nested LFS commits, and another complete offline restoration. An early test exposed LFS index stat data after materialization; the implementation now refreshes only verified LFS paths and checks the staged tree remains exactly HEAD.
Resolver nested module/LFS PASS, session 47756, 4.03s. Exact recursive paths and hydrated data survived preparation. The test explicitly permits only its own local file-transport fixtures; production code does not enable file transport for arbitrary module URLs.
New module and root LFS addition PASS, session 78906, 2.57s. A new module and LFS file added after the initial root commit were captured, then restored after deleting both original repositories.
Process-death hydration recovery PASS, session 54793, 6.95s. Separate subprocesses terminate immediately after LFS replacement or a staged module checkout; retry restores clean exact content without remote origins.
Initial real guest run 31411 failed because Git bundle fetch implicitly attempted recursive module fetches. The explicit no-recursion fix passed in 57792 (4.74s). Expanded real guest acceptance PASS, session 31101, 5.08s, adds corrupted-cache repair. Two independent assignments retained separate module/LFS content; replay preserved uncommitted writer edits; read-only capture rejected them; recursive commits and both artifacts restored correctly after deleting the first revision and all original host repositories. Fixture directories and receipts were cleaned. This component test uses the separate owned acceptance VM and does not independently establish cross-VM scheduling.
Full Go suite and affected-package race checks passed (32591, 58176); vet passed. Those results precede the final LFS configuration refinements and added-new-module test. Final checks remain to be recorded below.

Remaining source work includes publication of new LFS objects/submodule commits, richer nested-source review, and wider provider/toolchain portability evidence. Local source restoration is now implemented with the tested cases above; that must not be described as completion of remote Git publication semantics or the full milestone audit. Production parallel branch stack ownership/verified joins, a second harness, remaining data/credential recovery and live interaction work remain in scope.

Final thirteenth-pass checks: full Go suite PASS (36167), vet and CLI build PASS, Astro docs build PASS (17240). The final real guest regression PASS (1968, 5.34s) includes all latest source companion validation and LFS configuration changes. Repository tests also verify new module/LFS additions and source preparation with nested module paths. Final affected-package race tests PASS (19231), covering repository, local execution, workflow, engine, MCP, review and TUI. Latest binary: /tmp/envctl-source-cli.

Source companion restore is now wired through production local execution and failed-worker recovery, with mandatory retention checks. The next large delivery item remains local M7 production parallel execution: independent stack/data ownership, verified source joins and assignment scheduling/budgets. Publication of LFS/submodule objects must also be completed so the new local source support can participate safely in end-to-end PR output. The active objective remains all M1-M5/local M7; no milestone or goal is marked complete by this pass.

All test/build processes in this pass are terminal, and the guest source fixtures were cleaned. The separate owned component VM remains available for subsequent milestone acceptance; no VM, PR or branch was allocated by this pass.

Fourteenth implementation pass: explicit verified joins

M7 starting evidence reviewed this pass: the generic engine already schedules independent ready nodes and preserves sibling attempts during retry, but the production adapter rejects concurrent execution. Source input joins reject unequal SHAs; data joins reject unequal snapshot manifests. MCP is implemented. Remaining M7 packages include child VM/stack/data ownership and capacity, multiple workers, second harness and conformance. This pass's bounded demonstration is real divergent Git histories, a conflict, retained input restoration, restart/replay preserving resolution, and coordinator verification of both exact ancestors. It does not prove parallel VM execution.

Added join.repositories (repository -> starting predecessor) and join.datasets (dataset -> selected predecessor). Configuration follows possible writer identities through the DAG and rejects an ambiguous source join before launch. Repository merge nodes require writable ownership, code/custom kind and executable checks. All dataset joins require explicit choices. Incoming sources hydrate into independent offline copies. The worker performs conflict resolution; merge_parents binds every incoming SHA into the work digest and supervisor assessment. Both local execution and the engine verify ancestry from the retained bundle in an isolated host Git object store. Replacement refs, ambient Git configuration, missing prerequisites and dropped input history cannot authorize a checkpoint. Review/TUI views display source merge parents.

Initial package run 38306 found the older scheduler fixture used an implicit writable join; it was updated to declare a merge stage before QA. A compile typo in the new checkpoint assertion was also corrected. Targeted workflow/localexec/engine/TUI/review checks then passed (13560).

Real guest acceptance first failed (75390) because this guest's Git LFS returns files: null for repositories with no LFS files. The companion reader now accepts that empty-inventory representation while rejecting non-array values. The corrected real guest join acceptance passed, session 38972, TestRealGuestVerifiedJoinAndReplay, 3.40s. It removes original source and branch worktrees, restores both retained inputs, rejects unmerged output, creates and resolves an actual conflict, recreates the backend and replays preparation without overwriting the resolution, verifies merged ancestry, and checks both input copies remain unchanged. All owned guest fixture paths/receipts were cleaned. No agent or VM was allocated by this component test.

Final full/race/vet/build and documentation checks are pending below. The goal remains all M1-M5/local M7; no milestone is marked complete. Concurrent production dispatch is deliberately still rejected until independent child runtime/stack/data ownership is ready. Source publication, credential/data recovery, live interaction and the remaining release audit also remain in scope.

Final fourteenth-pass evidence: full Go suite PASS (48576), affected-package race tests PASS (12759: workflow, engine, localexec, repository, review and TUI), vet and CLI build PASS, Astro build PASS (45699), and whitespace checks pass. Expanded repository merge tests pass (91496), covering dropped branches, real conflict resolution, hostile ambient Git settings, forged replacement refs, incomplete bundles, cancellation, and SHA-256 repositories. The latest CLI binary is /tmp/envctl-join-cli. All tests/builds from this pass are terminal. The real guest fixture cleaned its own paths; the previously owned component VM remains for subsequent acceptance and must be destroyed at final delivery.

Next M7 ownership work: model and reserve child runtimes durably, count them against per-run VM limits across draining revisions, isolate per-child Compose/data/plugin receipts, revalidate child capabilities before dispatch, retain failed-branch recovery without affecting siblings, and release children only after their work/evidence is retained. The current per-revision reconciliation lock and recovery state also need review so a slow branch preparation or reconnect cannot indefinitely starve another branch. Do not remove the production parallel-execution guard until these behaviors have integration evidence.

Fifteenth implementation pass: child runtime ownership and scheduling

Previous turn classification: verified progress on explicit source/data joins. M7 starting evidence is recorded in the fourteenth pass. The remaining complete objective is unchanged: all M1-M5/local M7, with remote runtime expansion excluded.

Added per-node child runtime records, per-child readiness/recovery and independently locked reconciliation. Parallel assignments reserve VM capacity in the same transaction as attempt creation. Reservations include parent and child VMs across draining revisions; failed release retains capacity. A branch retry reuses its child VM and session namespace. Worker state remains preparing until that child's actual capabilities have passing current evidence. Parent probes do not authorize a child. The engine rejects an aliased Docker daemon or changed source pins. Child results must retain all repository bundles before checkpoint acceptance can release their VM.

Intermediate child runtimes release capacity after checkpoint acceptance. Remaining application runtimes on a completed current run stay available for inspection, matching serial execution; rewind or explicit cancellation releases them. Cancellation now also tears down a completed run's retained runtime without deleting checkpoint history. Rewind creates fresh child ownership and drains historical work in the old children. The TUI Services/Readiness panels and CLI expose child state; scoped MCP artifact reads include child readiness/recovery evidence.

The backend places stack, dataset, attempt and plugin receipts beneath each child owner. Child dataset preparation/restoration uses its exact predecessor snapshots, and plugin requests carry an optional child runtime_id namespace. Parent and child publication probes share a serialized per-revision broker object store; unrelated revisions remain independent. The root parallel-execution guard is still present while real VM isolation acceptance runs.

Evidence so far: child capacity/priority/retry/release, readiness, transport recovery and independently blocked provisioning tests passed under race detection (68559); rewind/cancellation extensions passed (25581). Full Go suite passed (14169) before the final admission/retention refinements. Affected-package race tests passed (28873); engine/workflow/daemon race tests passed after completed-runtime preservation (46051). Expanded publication concurrency and Docker-alias rejection tests passed (84041). These are coordinator/protocol fixtures and do not establish model reasoning or actual VM isolation.

Live real child VM acceptance: process 7331, TestRealChildVMSourceAndDatasetIsolation. State: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-child-acceptance_3d9847bc0cd9ec8584206d96. Owned guests: envctl-rev-6db740b94988e9c87dfd0e3b-left and envctl-rev-6db740b94988e9c87dfd0e3b-right. Both are confirmed Running; left's database is healthy, while right is finishing harness/bootstrap dependencies. Re-poll this exact process; no restart is justified by an observation timeout. The test will verify independent source/data/quiescence/replay and release both VMs. It freezes worker requests but does not launch model jobs. Final terminal evidence and the remaining end-to-end parallel-agent demonstration must be recorded before broader claims.

Terminal real child VM acceptance PASS, process 7331, 677.72s. Both child VMs restored the same retained baseline after deletion of the original source. Left's database mutation and source edit remained private; right retained its seed and source. Left's durable quiescence marker did not prevent right's dataset capture. Backend recreation preserved both namespaces and live data; releasing left did not affect right. Both VMs were then released, including idempotent cleanup replay. Retained acceptance-proof.json records distinct Docker IDs 50e45554-9b61-4724-98df-9a0cc75bccb5 and 761be6d6-f7ce-4afe-9531-535245867ee2. A subsequent Lima inventory confirms both are absent. The separate older envctl-agent-acceptance VM remains available and must be destroyed before final goal delivery.

After this evidence, the production parallel guard was removed. The adapter instead requires every parallel downstream worker to have its own prepared child matching the node identity; it cannot fall back to the parent or borrow a sibling. Engine admission additionally checks child image and repository pins against the revision. The real component test above precedes these final admission-only refinements; targeted race tests verify them. Full Go suite passed (54596), affected-package races passed (68066), and vet passed before the final backend guard replacement/image-pin refinement. Final check results follow below. No full milestone or goal completion is claimed.

Final fifteenth-pass checks: full Go suite PASS (87019), child engine race tests PASS (98065, including source/image drift), localexec/engine races PASS (88876 and 56878), vet and CLI build PASS, Astro documentation build PASS (11087), and whitespace checks pass. Latest binary: /tmp/envctl-parallel-cli. All processes started in this pass, including real VM acceptance 7331, are terminal. Both newly allocated child VMs are removed. No remote runtime, PR or external Git branch was created.

Next acceptance should exercise the actual production engine and real worker/supervisor pairs through parallel branches, one independent retry, source merge, QA and coordinator restart, with VM/source/data identities retained. The foundation now has scheduler fault tests plus separate real source-join and child-VM isolation evidence; those must not be substituted for the complete combined agent demonstration. A second harness/conformance, source publication for new LFS/submodule objects, remaining credential/data recovery, live interaction/preview and the final M1-M5/local-M7 requirement audit remain in scope.

Sixteenth implementation pass: combined parallel-agent acceptance

Previous turn classification: progress, with child runtime scheduling and real two-VM source/data isolation evidence. The complete M1-M5/local M7 goal remains active. This pass adds a resumable production-engine acceptance test, TestRealParallelAgentsRetryMergeAndCoordinatorRestart, selected by ENVCTL_PARALLEL_TEST=1 and resumable with ENVCTL_PARALLEL_RESUME pointing at the retained state directory.

The custom DAG is Task -> Plan -> left/right Code branches -> explicit source/data join -> QA. Both real workers wait on coordinator-owned barriers so the test can observe simultaneous live jobs in different VMs. It restarts the engine/backend, releases left, records a deliberately failed fixed check, allows a real retry while the same right job stays live, and then releases right. Both branches change a conflicting source line and independently write the same dataset key. The merge must retain both input SHAs and the explicitly selected left dataset; fixed checks and separate supervisors review each accepted stage. The original host fixture is deleted after Plan, requiring later VMs to use retained pins. Successful completion verifies all checkpoint/source/data evidence, then cancels through the API and confirms all fixture VMs have been removed. Failures retain the same run and jobs for inspection/resume. No PR or remote runtime is part of this local M7 demonstration; M4 PR evidence is tracked separately above.

The fixture definition/validation test passes. Real execution is starting; record its process handle, state directory and terminal outcome below. An observation timeout is not permission to launch another fixture or worker.

Observation after context recovery: the same acceptance process is alive as host Go PID 51462, test PID 51469. Retained state is /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-parallel-acceptance_a5c171a6916655e737742b1b, run run_aa6109f986162dc09c012e7d, root guest envctl-rev-a863883712c7ed0194d20f15. The original tool session handle was not available after context recovery; process inspection and the durable proof file identified this existing launch. No replacement test or worker was launched. At this observation the proof has not yet recorded parallel overlap or verification. Preserve the process and state; no terminal outcome is claimed.

Seventeenth implementation pass: credential rotation during pending plugin work

Previous goal turn classification: planning documentation progress and a verified wait on live test PID 51469. The full M1-M5/local M7 objective remains active. This pass continues observing that same parallel acceptance process and implements the outstanding M3 pending-operation credential case.

Plugin reconnect now validates the frozen guest execution identity without exporting its original credentials. The existing root-private request is submitted unchanged to reconcile any durable start intent, then its output is observed. Only credential values/redaction lists are excluded from the reconnect comparison; command, directory, environment, timeout and every other plugin payload field must match. Original receipt digest validation remains mandatory. Missing operations may resolve current credentials and submit; transport uncertainty cannot authorize a new submission. Interrupted jobs remain interrupted. New operations still check credential availability before creating their host intent; pending operations can reconnect even after the credential file is removed. Host receipts retain no credential values.

Real guest lifecycle/rotation PASS, session 30719, TestRealGuestPluginLifecycleAndReconnect, 5.79s, in the separate owned component VM. The test observed the actual execute job running, disconnected its caller, rotated then removed the available credential, rejected changed operation input, and recovered the original result with one execution and the original credential generation. New renewal failed while credentials were unavailable and succeeded with the rotated credential when restored. Original secret output remained redacted. Package integrity and lifecycle cleanup passed; no model job or new VM was allocated by this test.

Actual Python journal/reconnect tests pass (64910) for completed, pending, interrupted and missing operations, unchanged frozen requests, one start after pending intent, changed execution fields, JSON boolean/integer configuration differences, and corrupt receipts. The initial local fixture failed because macOS resolves /var through /private/var; normalizing the fixture's work root corrected the test. Local backend lifecycle tests pass (99758) across prepare/probe/execute/renew/cleanup, lost observation, lost reconnect, backend recreation, credential-file removal, new-operation refusal, rotated credentials and secret-free archived receipts. Affected package tests pass; full/race/vet/build results remain pending below.

Parallel acceptance observation: Task and Plan are checkpointed, the original host source was removed as designed, and left/right child VMs are running with their assignments still preparing. Same test PID 51469 remains live. Parallel overlap/retry/merge/QA and final cleanup are not yet established.

Final seventeenth-pass checks: full Go suite, vet and CLI build PASS (5336); plugin/localexec/guestjob/engine race checks PASS (49960); Astro build PASS (81143); whitespace checks pass. Latest CLI binary: /tmp/envctl-plugin-reconnect-cli. Real local-backend plugin regressions PASS (40023, 15.69s total): terminal process/protocol recovery preserves one external effect across three generations, disappeared-resource repair survives backend recreation with a fresh probe, and dependency/credential admission plus lease renewal and cleanup work in the guest. Component plugin fixture cleanup completed. These checks follow the final product-code change.

All tests/builds launched in the seventeenth pass are terminal. The earlier parallel acceptance remains live as test PID 51469 (14m31s at latest authoritative process inspection); it has not been restarted or declared successful. Its parent and both preparing child VMs remain owned by its retained state directory above. The separate component VM is still retained for subsequent local acceptance and must be destroyed before final delivery.

Next work remains the combined parallel-agent terminal outcome, second harness and adapter conformance, new LFS/submodule publication, remaining database/restore cases, live steering/progress/local preview, and full M1-M5/local-M7 release audit. A read-only local check found Claude installed and authenticated through claude.ai; no credentials were exported and no Claude process was dispatched. This is discovery for the additional-harness work, not adapter conformance evidence. The complete active objective is unchanged.

Eighteenth pass: second harness conformance and stalled parallel acceptance

Handover from the Codex session to Claude Code. Full go test ./..., go vet ./... and build passed on the inherited tree before any change.

Codex quota exhausted. Both parallel-acceptance workers (attempt_3f42467785728e555c8a4b51, attempt_9910809f69e403268b00a43e) failed at 13:11 UTC with the ChatGPT usage-limit error (reset stated as 2026-09-17 10:59 local). The coordinator process (PID 51469) had also exited, before the restart assertion. The retained run cannot satisfy its own "left still live at barrier release" assertion, so it is not resumable; a fresh run is required. Its parent and two child VMs (envctl-rev-a863883712c7ed0194d20f15, -6157fe4ca13f, -187e87df03ab) remain running and owned by /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-parallel-acceptance_a5c171a6916655e737742b1b. Every Codex-harness acceptance is blocked until the quota resets.
Claude conformance, first failure was the test. TestRealHarnessConformance (retained state /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-harness-conformance-268907026) read agent-owned 0600 files as the login user. The files held the exact marker; the check now reads them as root. On resume, cross-directory explicit session continuation, the independent supervisor and cancellation passed.
Claude conformance, second failure was a real credential defect. Recovery failed with OAuth session expired and could not be refreshed. The adapter copied the host claude.ai login (keychain access and refresh token) into each guest role home once (writePrivate keep mode). The host's refresh at about 07:36 rotated the refresh token and revoked the worker home's 06:12 copy. The race also runs the other way: a guest refresh can revoke the host login. ClaudeCredential now accepts only non-rotating credentials: ANTHROPIC_API_KEY, or a claude setup-token CLAUDE_CODE_OAUTH_TOKEN, via env:/oauth-env:/file:. It rejects keychain and claudeAiOauth session files with actionable errors. Unit tests cover these cases; agent/localexec race tests pass. The conformance run must be repeated with a long-lived token before the second harness counts as verified.
The copied host Claude login files in envctl-agent-acceptance (/work/envctl/harness-homes/claude-*/.credentials.json) were deleted by the user; none remain in any harness home.
Not yet investigated: whether the Codex adapter's once-per-role-home auth.json copy has the same rotation hazard for ChatGPT refresh tokens.
Real Claude harness conformance PASS, fresh run, TestRealHarnessConformance with ENVCTL_HARNESS_KIND=claude and a claude setup-token credential file reference, 38.94s. State: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-harness-conformance-1480410392. Verified real tool use, structured results, live adapter recreation against a running guest job, cross-directory explicit session resume, an independent supervisor session, coordinator cancellation and recovery in the same session. Afterwards no .credentials.json existed in any guest harness home, and no job output or result contained the token. This is harness conformance, not a Claude-worker workflow run through the engine.
Setup documentation: the Install page now documents the claude setup-token process and credential references; the agent guide tells agents never to request or print the token.

Nineteenth pass: parallel acceptance, live interaction, publication objects, recovery

Work was merged into workflow-runtime from isolated worktree branches. Each merge was followed by full go test ./..., go vet ./... and affected-package race tests, all passing.

Real parallel-agent acceptance PASS with Claude workers and supervisors: TestRealParallelAgentsRetryMergeAndCoordinatorRestart, 928.90s, ENVCTL_PARALLEL_HARNESS=claude with a claude setup-token file credential. State: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-parallel-acceptance_0fdf6610a8722988c5acf6b7, run run_524daf7dd276a35c074f6828, with acceptance-cleaned.json.
- Live overlap in distinct child VMs.
- Coordinator/backend recreation while both real workers ran.
- The left branch failed its fixed check, then retried and checkpointed while the original right attempt (attempt_ee4438e9b3db97751f7c80a5) stayed live and isolated.
- A verified merge retained both input SHAs, with the explicitly selected left dataset.
- QA passed after the original host source had been deleted.
- Five runtimes had distinct Docker daemons (parent, left, right, merge, QA). All were removed by the fixture.
Test-harness deadlock found and fixed. Earlier Codex (PID 51469) and Claude runs both stalled right after overlap. The restart helper's goroutine sent to the shared done variable, which stopEngine sets to nil before waiting. A goroutine dump showed the send blocked on a nil channel; the engine itself had stopped promptly. The helper now sends on its own channel (1b58d19). Those two runs' VMs (envctl-rev-a863883712c7ed0194d20f15*, envctl-rev-d1d5df4924c05c0fdb05b4a1*) remain for manual deletion.
Live steering and progress.
- Running attempts expose bounded, redacted progress parsed from Codex and Claude event streams.
- Messages reach a running agent by durable interrupt-and-resume of the same harness session, limited to 8 resumes per role per attempt.
- Supervisors see all stage messages. The TUI and run show show delivery status.
- Real Claude steering PASS, TestRealLiveSteeringResumesRunningClaudeWorker, 12.7s, state /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-steering-acceptance-1541908574.
- Follow-up 9f48608: progress is stored outside the versioned run document (a progress table overlaid on reads, stripped on writes). Writing it every few seconds had bumped the run version and would make version-fenced TUI/CLI commands conflict.
Per-node budgets. limits: {max_attempts, attempt_seconds} on a node overrides the revision-wide values; digests are unchanged when unset. This also fixed an engine bug: an exhausted node moved the revision to needs-attention even while a sibling with remaining budget was waiting out its retry backoff, stranding the sibling. Fan-out rewind tests confirm branch-local rewind reuses the unaffected sibling checkpoint, invalidates the join and QA, and drains in-flight work historically.
Submodule and Git LFS publication. The broker validates the retained companion, verifies committed gitlinks and LFS pointers, and publishes changed submodules deepest first: git lfs push, a create-only branch and a reconciled draft PR in the submodule repository. It uploads the root's LFS objects before the root branch. Verified with local bare repositories and git-lfs 3.7.1 file:// remotes only. Real GitHub LFS/submodule publication has not been exercised.
VM restart and restore-crash recovery.
- Readiness restarts a stopped owned VM under its reservation and refuses a changed Docker identity.
- Terminal dataset jobs get durable recovery generations with evidence, backoff and budget.
- Guest source preparation discards receipt-less partial worktrees and serializes with orphaned scripts using flock.
- Real PASS, TestRealGuestRestartAndRestoreCrashRecovery, 146.28s, dedicated VM destroyed. State: /var/folders/bd/mtrzdck92l10k6xpmqxtq2w00000gn/T/envctl-restart-acceptance_83003d8328f449f8df20fd5f. It covered VM stop/start with the same daemon and healthy stack with intact volume data, coordinator death during a live PostgreSQL restore, and a SIGKILLed restore replaced by exactly one generation-1 job.
Codex credentials. OpenAI's CI/CD guidance forbids sharing one auth.json across machines or concurrent jobs (refresh-token rotation), which the Codex adapter currently does per guest role home. The Install page now directs Codex users to API keys. Whether to enforce API-key-only Codex credentials awaits the user's decision.
Regression tests. Non-terminal and --json invocations never launch the TUI or touch coordinator state. The TUI plugin attach/remove keys submit revision- and version-bound actions.
Long-lived credentials only (user decision). Codex now matches Claude. It accepts a Codex access token (CODEX_ACCESS_TOKEN, access-env: or a file: JSON) or an OpenAI API key, and refuses ChatGPT login files with rotating tokens. Credentials travel only in the job environment, and job start removes any login file previously copied into a guest role home (auth.json for Codex, .credentials.json for Claude). Pinned Codex 0.154.0 supports CODEX_ACCESS_TOKEN. Codex access tokens require a ChatGPT Business or Enterprise workspace; the user's current login is a personal plan. The Install page documents both token flows. The six leftover envctl-rev-* VMs and the six merged agent worktrees were removed.

FilesExpand file tree

implementation-progress.md

Latest commit

History

implementation-progress.md

File metadata and controls

Local workflow implementation evidence

Acceptance ledger

Current status (latest integration pass)

Current work

Implemented and verified in the first implementation pass

Real VM evidence (completed)

Second implementation pass

Remaining integration / known limitations

Next harness integration inputs

Third implementation pass / live acceptance to resume

Latest real acceptance result

Fourth implementation pass: production local execution connection

Fifth implementation pass: real publication, invocation plugins, and interrupted source

Sixth implementation pass: PostgreSQL checkpoint data

Seventh implementation pass: browser checks, emulator data, Plan drain, and review

Eighth implementation pass: plugin recovery and retained checkpoint comparison

Ninth implementation pass: dedicated-VM rewind acceptance and offline baselines

Tenth implementation pass: explicit plugin resource repair

Eleventh implementation pass: structured Plan requirement discovery

Twelfth implementation pass: local agent MCP interface

Thirteenth implementation pass: portable submodule and LFS checkpoints

Fourteenth implementation pass: explicit verified joins

Fifteenth implementation pass: child runtime ownership and scheduling

Sixteenth implementation pass: combined parallel-agent acceptance

Seventeenth implementation pass: credential rotation during pending plugin work

Eighteenth pass: second harness conformance and stalled parallel acceptance

Nineteenth pass: parallel acceptance, live interaction, publication objects, recovery