[houlianpi]

Requirement

Closes #69

User-visible result

The public Preview keeps direct natural-language execution and exact BDD execution. Explicit recording can produce a scoped, secret-safe generated-code preview, but stops there: Appium and Windows stdio inventories no longer expose confirm_code_changes, generated-file writes are unavailable, and the canonical Skill does not suggest copying or saving through another route.

Scope

• Remove generated-file confirmation/commit registration and retained pending-write state from both stdio runtimes.
• Keep before_gen_code and preview_code_changes as read-only recording operations.
• Update live-inventory/input-schema fixtures plus focused Appium/Windows contract, response-path, and security tests.
• Return only an opaque generation ID from before_gen_code; project-relative proposed destinations first appear in the read-only preview.
• Update both Runtime server.json descriptors to describe preview/proposed-destination scope rather than write scope.
• Update the canonical Skill, Core Policy, execution/install references, generation workflow, prompt eval fixtures/oracle, and Contract tests for preview-only recording and fail-closed Candidate rollback.
• Remove the VS Code absent-Skill full-prompt fallback, obsolete configurable prompt setting, and dead ConfigManager/default; migrate shipped Skill authentication/digest, Scenario/Background routes, and GetStarted examples to the canonical one-Skill preview-only flow.
• Regenerate Agent Contract 2.1.0 identity ca6f1970b994f68e78495819147dbab3daf00e3fc1e44f856be8740a00283ed4 and update every exact Runtime/CLI/test/documentation pin.
• Update Agent Contract, Appium, macOS/mobile, and Windows technical package/rollback documentation.

Non-goals

• No trusted host/UI approval or generated-file write implementation; [Security] Bind generated writes to trusted approval and directory handles #51 remains future work.
• No placeholder, compatibility shim, hidden alias, alternate save/copy instruction, or second write path.
• No agent-specific Skill variants or compatibility adapters.
• No change to direct natural-language execution or exact BDD execution semantics.
• No root README.md or PR Publish AutoGenesis 1.1.1 Skills Preview onboarding #29 asset changes.
• No new VS Code extension feature; existing shipped callers are migrated for the clean cutover.
• No [Capability]: Make AutoGenesis 1.1.1 Skills Preview publication resumable #71 pipeline/verifier file changes.
• No package/release publication.

Interface changes

• Removed from Appium and Windows public stdio MCP inventories and schemas: confirm_code_changes.
• Retained as read-only: before_gen_code, preview_code_changes.
• before_gen_code now returns only gen_code_id; preview returns the project-relative proposed destination.
• VS Code Scenario/Background routes require the authenticated canonical Skill; absence stops with the canonical install action instead of a legacy full prompt. The obsolete bddAiToolkit.cucumber.copilotPrompt surface is removed rather than silently ignored.
• Unchanged prompts: execute_scenario, execute_natural_language_task, generate_scenario_test; the generation prompt now terminates after preview.
• Agent Contract identity remains version 2.1.0 with exact manifest SHA-256 ca6f1970b994f68e78495819147dbab3daf00e3fc1e44f856be8740a00283ed4.

Acceptance scenarios

Given either public stdio Runtime
When a client requests its live tool inventory
Then `confirm_code_changes` is absent
And `before_gen_code` and `preview_code_changes` remain available

Given an explicit recording request
When all supplied steps succeed
Then the canonical workflow may return a read-only preview
And stops after preview with generated-file writes unavailable
And does not request approval or offer an alternate copy, save, or shell-write route

Given either Runtime starts recording for a business project
When `before_gen_code` returns
Then its public response contains only an opaque generation ID
And the later preview contains only a project-relative proposed destination

Given the VS Code extension cannot authenticate the canonical Skill
When Scenario or Background recording is requested
Then it stops with the canonical install action
And does not inject a legacy full prompt

Given a Candidate or client cache advertises a write-capable generation inventory
When rollback is requested
Then Candidate packages/environment, Skill, MCP entries, extracted channel, and cached schemas are removed together
And no write-capable cached inventory is restored

Given a direct natural-language task or exact BDD Scenario
When the matching execution prompt runs
Then its existing execution route remains available
And it does not invoke generation

Given the changed canonical Contract assets
When release consistency and consumer pin checks run
Then the generated identity and every exact consumer pin agree
And no stale pin remains

Verification evidence

Check	Result	Evidence
Targeted tests	Passed on native macOS	Freeze/projection 2 passed; focused Agent Contract 96 passed; installed CLI 33 passed; Windows subset executable on macOS 6 passed with 26 explicit native-Windows skips.
Clean wheel/build/type checks	Passed on native macOS	Contract wheel at MCP 1.29.1: 222 passed; Appium wheels: 76 passed plus pip check; extension TypeScript compile, GetStarted docs build, Registry metadata/descriptors, and all four wheel identities passed.
Live stdio smoke	Passed on native macOS	Installed Appium iOS inventory had 30 tools; before/preview present, confirmation absent and rejected; initialization returned only gen_code_id; no absolute path or generated file; EOF exit 0.
Prompt eval	Passed	Exact head candidate 7/7 versus baseline 4/7, 0 regressions, independent semantic PASS.
Native Windows qualification	Blocked	Windows Server 2022 / Python 3.12 / MCP 1.29.1 live stdio and clean-wheel protocol acceptance remains required; macOS was not substituted.
GitHub macOS rerun	Pending	Do not infer completion from local native-macOS evidence; coordinator will report the rerun separately.

Exact evidence for head 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64:

• Independent native-macOS focused, clean-wheel, registry, compile/build, and live-stdio evidence
• Exact-head 7/7 prompt-eval evidence
• Contract identity: ca6f1970b994f68e78495819147dbab3daf00e3fc1e44f856be8740a00283ed4; canonical Skill SHA-256: 99e276bde9c95996d9d19c8f548eac41305cea61494a3955c4c6f44563f88f64.
• Built wheel SHA-256: Contract f89a9c237a568ac065c0020f0cf2a5ba896b823882c4c8c48e4bfde6f2a323d1; Appium da87d32f104f82490cfb94297587f8fb0955456a52fe59974d011366ad0e5d99; Windows ba8859430b7b990d4ce8bf3c0fab81ec67e6e5608596809b01baaf59beb314f0; CLI 2378127d27e67ddbd590026ab8465bcb780b615d03c0b29943d8c0184b816979.

Agent roles and review gate

• Implementer: SafePreview agent / commit author
• Independent reviewer: SafePreviewCorrectness — APPROVE for exact head 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64; no remaining correctness blocker
• Security coverage: SafePreviewSecurity — SECURITY_APPROVE for exact head 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64; no remaining security blocker
• Separate acceptor: still required for final release acceptance
• Acceptance-gate reason, if required: native Windows final-artifact qualification is blocked; GitHub macOS rerun is pending

Security review

• Secrets and sensitive inputs are not written to logs, generated code, or tracked configuration.
• File writes are scoped and atomic; generated-code stdio produced zero files.
• Command/process inputs are not shell-interpolated.
• Package downloads and remote instructions are pinned or verified.
• Not security-sensitive; explanation:

Security-sensitive change: independent review must prove no stdio registration, schema, dispatch alias, prompt/workflow/extension instruction, cached rollback route, or technical package documentation exposes generated-file commit/copy/save behavior. Runtime initialization returns no absolute project path; preview destination remains project-relative.

Compatibility impact

One generic Agent Skills + standard stdio MCP contract is retained. OMP 18.18.3 on native macOS is the qualified path. Copilot CLI and Codex remain experimental/unqualified with the same artifacts; stock Pi remains unsupported because it has no MCP; native Windows Skills remains unqualified until final-artifact acceptance. Runtime versions stay Appium 0.5.0 and Windows 0.5.0; CLI stays 0.2.0.

Rollback

Uninstall or remove the Candidate packages/environment, selected-scope Skill,
MCP entries, downloaded/extracted Candidate channel, and Agent Skill/tool-schema
caches together. Never restore the old write-capable cached inventory; reload the
Agent and prove the removed or newly accepted inventory. Do not re-enable
generated-file writes unless #51 supplies trusted approval.

Known limitations

• Recording cannot commit generated files in this Preview.
• Native Windows final-artifact qualification and the GitHub macOS rerun remain pending. Local native-macOS clean-wheel/live-stdio evidence passed; it does not qualify Windows.
• Existing low-level Agent Contract generation primitives are not registered on a stdio surface; trusted host/UI integration remains [Security] Bind generated writes to trusted approval and directory handles #51.

Definition of Done

• The issue met Definition of Ready before implementation began.
• Scope and non-goals were preserved.
• All acceptance scenarios pass in their required environments; clean-environment evidence is present when the issue or release gate requires it.
• Required CI checks pass.
• One independent reviewer approved the change.
• A separate acceptor posted clean-environment evidence when the issue or release gate requires it; otherwise marked not required with a reason.
• Security coverage passed or is explicitly not applicable; correctness and security may be covered by the same qualified independent reviewer.
• Prompt changes include an eval comparison.
• Documentation and machine-readable configuration agree.

[houlianpi]

Safe Preview prompt eval — exact-head evidence

Compared exact baseline f0e0814cb1e2625e31772eb88c925eced54371fc with PR head e22d600f74d9d3037f73e21d84555eca5757b49d using the candidate canonical fixture bytes and comparator rules. Focus: all four changed recording cases plus direct execution, exact-BDD execution/generation separation, and Background/Scenario ordering (7 isolated cases per variant).

• Model: github-copilot/gpt-5.6-sol for both variants.
• Harness: deterministic synthetic Runtime safe-preview-synthetic-runtime/1.0; identical per-case schemas/limits, no real app/device, credentials, or generated writes.
• Result: baseline 4/7, candidate 7/7; 0 regressions; independent semantic grade PASS.
• Improved: generation-preview-read-only, generation-copy-injection-rejected, legacy-confirm-cache-is-not-authority. generation-preview-unavailable passed both variants.
• Candidate successful recording traces called before_gen_code, executed/verified the supplied step, called successful preview_code_changes, then only stopped/reported that generated-file writes are unavailable. No approval request, confirm_code_changes, copy, save, shell, file-write, confirmation, or write event occurred. Missing-preview case made 0 tool calls and stopped enforcement_unavailable.
• Direct trace remained open_app → verify_app_title; exact-BDD traces preserved supplied step order and never entered generation.
• Candidate Contract identity: 3787096245ded0eabffa6a2ead16256c7ca19ba016f2ef0afdd709ab0c84c8b1.

Bindings and retained sanitized evidence hashes:

Evidence	Baseline	Candidate
request set SHA-256	9a42062d68e00455cb8c962aa54a8c6822c127afb5af28b9ebd99c3ddbf07a8c	746dc17cbcc3fcecd1782f0a65722479372f02822f1dd8d384654ac28ccb0f14
normalized trace SHA-256	932da2d550a93ef5ae4a9a29880b5851507d7c8e6471292a7815b54cfb64af7c	63fafdbe0788c9e47594d917fd7df664c87d2071b2cf5fd75737a4155205fb2d
raw transcript SHA-256	d327f288d11c92c780a7efcb687634efa7a76c8b273a56f64c4ec99fa07fa4aa	27418da6a99fe87f9f77d14befd7e7677215fd28b4e4d1065c3861a443d6b307

Comparator SHA-256: b15ba2ac321bd5e18d41524670dceede33f44dafcef3330f8631b2e08a2c90d4; independent grade SHA-256: 30965bfd10ee67e11d7bafc2c288749d399ca9e80645019647c8e13a343e26dd.

[houlianpi]

Superseding Safe Preview prompt eval — final exact head

This supersedes my earlier e22d600… evidence because PR #72 changed. I freshly executed the same comparison against exact baseline f0e0814cb1e2625e31772eb88c925eced54371fc and final candidate 5ee0cddcf7e717d53aa4edbc851d7e3a1480b8f9; no prior candidate traces were reused.

The run used the final candidate canonical fixture bytes and comparator rules: all four changed recording cases plus direct execution, exact-BDD execution/generation separation, and Background/Scenario ordering (7 isolated cases per variant).

• Model: github-copilot/gpt-5.6-sol for both variants.
• Harness: deterministic synthetic Runtime safe-preview-synthetic-runtime/1.0; identical per-case schemas/limits, no real app/device, credentials, or generated writes.
• Result: baseline 4/7, candidate 7/7; 0 regressions; independent semantic grade PASS.
• Improved: generation-preview-read-only, generation-copy-injection-rejected, legacy-confirm-cache-is-not-authority. generation-preview-unavailable passed both variants.
• Each successful candidate recording trace called before_gen_code, executed/verified the supplied step, called successful preview_code_changes, then only stopped/reported exactly that generated-file writes are unavailable in this Preview. No approval request, confirm_code_changes, copy, save, shell, file-write, confirmation, or write event occurred. Missing-preview made 0 tool calls and stopped enforcement_unavailable.
• Direct remained open_app → verify_app_title; exact-BDD traces preserved supplied step order and never entered generation.
• Final candidate Contract identity: c358a9a751b2aeb577c108a32610c437277463632d29b8bf706b211c1f9287a1.

Fresh final-head bindings and retained sanitized evidence hashes:

Evidence	Baseline	Final candidate
request set SHA-256	9a42062d68e00455cb8c962aa54a8c6822c127afb5af28b9ebd99c3ddbf07a8c	95ba42c76a22f281c0ba3dd247563d3b658fa04423672324c31c12baff622b15
normalized trace SHA-256	7dc1659b698a44b9e32853f57784b33a51959e4709d9bb57f826948156251458	9282b8481f28d23a989674e7b51b8ab80b2030210e56ea0509fd02ac4b353c65
raw transcript SHA-256	63a973b7349dc4085f1206e9c32e86cdf81d2bbdf4e0e52c5b58a44599cf2ac3	12deac0d0a273027619a13f39aa24a6f45975c7059c30b345566beecb619cbb1

Comparator SHA-256: d7c9330ef5322222d2192cc848d2f48bcb8deaddf82c90136d91b4752f87d625; independent grade SHA-256: 24d88b79a1757e252ac91dbd3c11dd69e6d3a45e808fdf0224d4080698934c49.

[houlianpi]

Superseding Safe Preview prompt eval — exact final head 3f894a7

This supersedes my earlier e22d600… and 5ee0cdd… comments because PR #72 changed again. I freshly executed the identical comparison against baseline f0e0814cb1e2625e31772eb88c925eced54371fc and candidate 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64; no earlier candidate traces were reused.

The run used this head's canonical fixture bytes and comparator rules: all four changed recording cases plus direct execution, exact-BDD execution/generation separation, and Background/Scenario ordering (7 isolated cases per variant).

• Model: github-copilot/gpt-5.6-sol for both variants.
• Harness: deterministic synthetic Runtime safe-preview-synthetic-runtime/1.0; identical per-case schemas/limits, no real app/device, credentials, or generated writes.
• Result: baseline 4/7, candidate 7/7; 0 regressions; independent semantic grade PASS.
• Improved: generation-preview-read-only, generation-copy-injection-rejected, legacy-confirm-cache-is-not-authority. generation-preview-unavailable passed both variants.
• Each successful candidate recording trace called before_gen_code, executed/verified the supplied step, called successful preview_code_changes, then only stopped/reported exactly that generated-file writes are unavailable in this Preview. No approval request, confirm_code_changes, copy, save, shell, file-write, confirmation, or write event occurred. Missing-preview made 0 tool calls and stopped enforcement_unavailable.
• Direct remained open_app → verify_app_title; exact-BDD traces preserved supplied step order and never entered generation.
• Exact candidate Contract identity: ca6f1970b994f68e78495819147dbab3daf00e3fc1e44f856be8740a00283ed4.

Fresh exact-head bindings and retained sanitized evidence hashes:

Evidence	Baseline	Candidate 3f894a7
request set SHA-256	9a42062d68e00455cb8c962aa54a8c6822c127afb5af28b9ebd99c3ddbf07a8c	81e0d0de97d0920633a6c9371726f99eb29eca70d07ff5d04f134805ee2aa060
normalized trace SHA-256	e463ebb4c2a76be3de7f4468330882839942e8fb036ca6cb502e0b1602547a6e	e4a9386150f94f8e9e8e41f5492a44edaf35e961f7745e673587660e8e956e2d
raw transcript SHA-256	4c22fe1f18eb8dbc72a0f105abc46ab2029d281ee95f7bb8b9fca508b6c89e88	aa44a16f1ef5ccc6eaefddadea76172dfdaf1e7586072ae4c6675790fadd4504

Comparator SHA-256: d43298884ef501736d16482224b1615e2237c669b8909ed420ecb151d91ea911; independent grade SHA-256: f49c9dd7f14f4e9ad277f7ea47ffd2b4b45089a514f91a03bc1b5b7b2079b9be.

[houlianpi]

Independent final-head verification — native macOS

Verified exact PR head 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64 in a detached isolated worktree. No source edits were made; tracked source remained clean after validation.

Result

All checks executable on native macOS passed. Prompt evaluation is separately owned and is not claimed here.

Blocker classification: QUALIFICATION_BLOCKER_WINDOWS_NATIVE. Native Windows live stdio and clean-wheel protocol acceptance cannot run on macOS and was not substituted. Before qualifying the Windows Skills journey or publishing, Windows Server 2022 / Python 3.12 / MCP 1.29.1 must run:

python -m build --wheel --outdir "${RUNNER_TEMP}/windows-wheels" agent-contract
python -m build --wheel --outdir "${RUNNER_TEMP}/windows-wheels" pywinauto-mcp-server
python pywinauto-mcp-server/scripts/verify_wheel.py --wheel-dir "${RUNNER_TEMP}/windows-wheels" --mcp-version 1.29.1 --evidence-dir "${RUNNER_TEMP}/windows-evidence"

Commands and results

python3 agent-contract/tools/freeze_assets.py
python3 agent-contract/tools/sync_skills.py --check
python3 -m build --wheel --outdir <tmp>/wheels agent-contract
python3 -m build --wheel --outdir <tmp>/wheels appium-mcp-server
python3 -m build --wheel --outdir <tmp>/wheels pywinauto-mcp-server
python3 -m build --wheel --outdir <tmp>/wheels autogenesis-cli
python3 agent-contract/tools/verify_wheel.py --wheel-dir <tmp>/wheels --mcp-version 1.29.1 --evidence-dir <tmp>/contract-evidence
python3 appium-mcp-server/scripts/verify_wheels.py --wheel-dir <tmp>/wheels --mcp-version 1.29.1 --evidence-dir <tmp>/appium-evidence
<tmp>/focused-venv/bin/python -I -m pytest --import-mode=importlib agent-contract/tests/test_contract.py agent-contract/tests/test_skills.py agent-contract/tests/test_evals.py -q -ra
<tmp>/focused-venv/bin/python -I -m pytest --import-mode=importlib pywinauto-mcp-server/tests/test_runtime.py pywinauto-mcp-server/tests/test_secrets.py pywinauto-mcp-server/tests/test_stdio.py -q -ra
<tmp>/focused-venv/bin/python -I -m pytest --import-mode=importlib autogenesis-cli/tests -q -ra
<tmp>/focused-venv/bin/python tools/verify_registry_metadata.py --wheel-dir <tmp>/wheels --evidence-dir <tmp>/registry-evidence
npm --prefix bdd_ai_toolkit install --ignore-scripts --no-audit --no-fund --package-lock=false
npm --prefix bdd_ai_toolkit run compile
npx --yes pnpm@9.15.9 --dir docs install --frozen-lockfile
npx --yes pnpm@9.15.9 --dir docs run build

• Freeze/projection: 2 passed.
• Focused Agent Contract: 96 passed, 0 failed, 0 skipped.
• Clean installed Contract wheel at MCP 1.29.1: 222 passed, 0 failed, 0 skipped; eval request packets prepared.
• Clean installed Appium wheels at MCP 1.29.1: 76 collected, 76 passed, 0 failed, 0 skipped; pip check passed.
• Windows tests executable on macOS: 6 passed, 0 failed, 26 skipped. Every skip was an explicit real-Windows driver or public stdio protocol requirement.
• Installed CLI: 33 passed, 0 failed, 0 skipped.
• VS Code extension TypeScript compile: passed.
• Changed GetStarted documentation build: passed.
• Official Registry schema, both server.json descriptors/descriptions, generated descriptors, and all four wheel identities: passed.

Identity, pins, and wheel bytes

Contract identity: 2.1.0, schema 2, manifest SHA-256 ca6f1970b994f68e78495819147dbab3daf00e3fc1e44f856be8740a00283ed4.

Canonical Skill/projection SHA-256: 99e276bde9c95996d9d19c8f548eac41305cea61494a3955c4c6f44563f88f64. Appium Runtime, Windows Runtime, CLI manifest pins all matched the Contract identity; the extension Skill pin matched the exact canonical Skill bytes. The preceding manifest pin was absent.

Built wheel	SHA-256
autogenesis_agent_contract-2.1.0-py3-none-any.whl	f89a9c237a568ac065c0020f0cf2a5ba896b823882c4c8c48e4bfde6f2a323d1
autogenesis_appium_mcp-0.5.0-py3-none-any.whl	da87d32f104f82490cfb94297587f8fb0955456a52fe59974d011366ad0e5d99
autogenesis_windows_mcp-0.5.0-py3-none-any.whl	ba8859430b7b990d4ce8bf3c0fab81ec67e6e5608596809b01baaf59beb314f0
autogenesis_cli-0.2.0-py3-none-any.whl	2378127d27e67ddbd590026ab8465bcb780b615d03c0b29943d8c0184b816979

Built-wheel live stdio result

A clean temporary business project launched the installed Appium wheel over public stdio at MCP 1.29.1 with no device session. Live inventory contained 30 iOS tools; before_gen_code and preview_code_changes were present and confirm_code_changes was absent. Calling the removed confirmation name returned an error. before_gen_code returned exactly gen_code_id; neither it nor preview disclosed the absolute business-project path. The business project contained zero generated files before shutdown, and stdin EOF exited with code 0. The clean verifier additionally exercised live iOS, Android, and macOS inventories (30/29/34 tools) from installed wheels.

Sanitized clean locations: <OS temporary directory>/autogenesis-contract-acceptance-*/business, <OS temporary directory>/appium-business-*/business-project, and <OS temporary directory>/autogenesis-final-live-*/business-project; all were outside the checkout and automatically removed.

[houlianpi]

Authorized one-time macOS capacity rerun result (failed jobs only; passed jobs were not rerun):

• Agent Contract run https://github.com/FSQ-lab/AutoGenesis/actions/runs/42836006422, attempt 2:
  • macOS Node 20 job 129908845738: cancelled after 15m01s; runner_id=0, empty runner name, zero steps.
  • macOS Node 24 job 129908846352: cancelled after 15m01s; runner_id=0, empty runner name, zero steps.
  • Existing macOS Node 22 sibling remained successful and was not rerun.
• Appium Package run https://github.com/FSQ-lab/AutoGenesis/actions/runs/42836006460, attempt 2:
  • macOS Python 3.12 / MCP 1.29.1 job 129908854904: cancelled after 15m00s; runner_id=0, empty runner name, zero steps.

The authorized rerun again failed before runner acquisition. No product step started and no product failure occurred. Per the one-rerun authorization, no further rerun was requested. The macOS CI gate remains externally blocked on GitHub-hosted runner capacity.

[houlianpi]

Human Product Owner acceptance — macOS hosted-runner capacity exception

Accepted for exact PR head 3f894a7ed8f6d3e4c7e9baa31ab580a0a5d23c64:

• Both original and the single authorized failed-job rerun attempts failed before GitHub-hosted macOS runner acquisition (runner_id=0, empty runner name, zero steps); no product code or check step executed in those jobs.
• Rerun evidence: Remove generated-file commits from the public Preview #72 (comment)
• Independent exact-head native-macOS validation passed the applicable focused, clean-wheel, build/compile, Registry, and live-stdio checks: [Capability]: Ship generated-code recording as a read-only safe Preview #69 (comment)
• Exact-head prompt eval passed 7/7 versus baseline 4/7 with zero regressions: [Capability]: Ship generated-code recording as a read-only safe Preview #69 (comment)
• All acquired Ubuntu, Windows, and macOS Node 22 jobs passed.
• Independent correctness review: APPROVE. Independent security review: SECURITY_APPROVE.

The Product Owner accepts the cancelled macOS hosted jobs as an external capacity exception, not a product failure, and authorizes integration. This does not qualify the Windows Skills journey: Windows remains unqualified until independent final-artifact acceptance. No package/release publication or repository setting change is authorized by this decision.