[melmennaoui]

Summary

Updates the experimental docker-sandboxes-env skill (0.1.0 → 0.2.0) by adapting sbxenv from docker/sandbox-kit-spec at 75b2696a. docker-sandboxes-env already owns sbxenv.yaml, so this extends that skill instead of adding a second one for the same file.

• Create/update workflow: workspace signals → kit search → composition → kit and environment args → credentials and bindings → pin → checklist. The summary is in SKILL.md; the full procedure is in references/authoring-workflow.md.
• scripts/kits.py (copied unchanged; SHA-256 1ca05d5f…): a live Docker Hub helper for searching kits, summarizing descriptors, upgrading by version order and pinning digests.
• Refreshed for sbx v0.51.0 and the published environment files docs:
  • -f/--force replaces -y/--auto-approve; it approves one run without recording consent.
  • env exec offers to create a missing sandbox interactively, and fails non-interactively.
  • Documents ~/.sbxenv.yaml user defaults, MCP command approval, which changes need a recreate, recovery after lost environment state, and per-command flags.
  • Removes the v2-era "kit spec.yaml" wording.
• references/env-schema-fields.md now covers every top-level field.
• Safety guidance kept: the env-file rename gap, never force-approving untrusted content, no literal secrets, the rememberHostCommands caution, and the destroy-plan recheck after preRemove.
• Evals: Prompt 7 now uses --force. New prompts cover scaffolding (8), upgrading kits (9) and non-interactive exec (10). Prompt 9 has static checks against the helper.
• Live scenario skills/docker-sandboxes-env/evals/scaffold-pin-plan.yaml (credential-free): shell agent plus the Playwright kit from the live catalog, pinned to a digest, previewed with sbx env plan. Create, run and --force are forbidden.
• Plan caveat: on sbx v0.51.0, sbx env plan doesn't resolve kit references; an invented kit or a missing tag plans cleanly. The skill now says to verify references with the helper first.
• Offline tests: scripts/test_kits_helper.py covers release ordering, the prerelease gate, update and pin behavior, local-source handling and failure exits. Breaking the version ordering or forcing exit 0 makes the tests fail.

Why a minor bump: owns and use_when gain kit selection and pinning inside sbxenv.yaml. That adds scope; existing routing doesn't change, and the sibling skills already send sbxenv.yaml work here. Maintainers may prefer a major bump.

Verification

• VERSION_CHECK_BASE_SHA=56001ac5… task: 250 unit tests OK; 270 eval checks pass, 0 fail, 23 skipped; version policy and DCO OK; links and verification scripts OK.

• Helper run against live Docker Hub on 2026-11-10 with Python 3.9: search, describe, update, pin, plus missing-repository and connection-failure exits.

• sbx env plan on the asset with sbx v0.51.0 and an isolated app name: the plan renders with the documented <agent>-<workspace-basename> name. Plan-only probes showed that an unknown field fails, while a nonexistent tag and an invented kit both plan with exit 0.

• CI live scenarios: scaffold-pin-plan passes 1/1 on both Claude (sonnet) and Codex, alongside the existing lifecycle scenario. Grading checks that the skill was invoked, that a plan succeeded, that nothing was created or force-approved, and that the answer names a sbx/playwright:…@sha256: reference. It doesn't check which tool resolved the digest.

• Runtime runbook (checks/verification.md, steps 0–8) on sbx v0.51.0, macOS arm64, with an isolated app and interactive approvals: all steps pass. Results are in checks/runtime-results.md:

  • initialize ran twice and postCreate once, and the unchanged hook file was asked about again;
  • the file without hooks re-applied silently;
  • a failing preRemove only warned and removal went ahead;
  • non-interactive exec on a missing sandbox exited 125 with "not found for this environment";
  • writing the env file at the mount root gave Read-only file system.

Questions for Sandboxes reviewers

• Rename gap: with the env file in a subdirectory of a read/write mount, sbx env plan warns that "the sandbox can rename" its directory. On v0.51.0 / macOS, though, mv of that directory failed with Read-only file system, because the file is its own read-only bind mount. Is the warning still accurate (other platforms or another attack path), or is the docs/plan text stale? The skill keeps the documented guidance until you confirm.
• Sign-in scope: sign-in is per --app-name. A plain sbx login didn't sign in the isolated app. The sibling skills' runbooks say the opposite and will get a follow-up fix.

Not verified / limitations

• The runtime runbook was run once, on sbx v0.51.0 / macOS arm64 only (see below). Not covered: cloud, MCP, secrets/bindings provisioning, --skip-host-commands, rememberHostCommands, adoption recovery, Linux/Windows.
• No live model evals.
• Known limits of the helper, inherited from the source and documented in references/sources.md:
  • it doesn't check the scheme of URLs the server returns;
  • it treats any reference ending in .zip as local;
  • date-style tags would outrank dotted releases.
• Some kit-side rules (one workload per composition, create-phase-only args on published kits) come from the source skill and helper, not the environment-file docs.
• Some cross-checks used docker/sandboxes 7d55286d, which is internal; public readers can only verify claims against the docs and the help output.

Follow-ups

• docker-sandboxes-lifecycle and docker-sandboxes-network-credentials still mention "kit spec.yaml" (v2 wording). They are not changed here.

[docker-gordon]

🔴 Critical issues

Adds a live kit selection and pinning workflow for sandbox environments. The new workflow and eval cleanup have failure-path risks.

3 findings

[medium] Describe the release selected for the environment

Location: skills/docker-sandboxes-env/references/authoring-workflow.md:62

The workflow describes the untagged candidate, which resolves to latest, before selecting newestRelease. If those tags point to different versions, the subsequent steps use the wrong version's arguments, credentials, and host grants. The final checklist permits pin instead of describe, so it does not correct this. Describe the selected release before configuring it.

Confidence: strong (100/100)

[medium] Do not report a successful pin without a digest

Location: skills/docker-sandboxes-env/scripts/kits.py:573

If a registry returns HTTP 200 without a Docker-Content-Digest header, manifest returns None. Both pin_one and pinned update_one then format a reference without a digest and report success. This can silently remove an existing pin or claim to pin an unpinned reference. Reject a missing digest when a pinned result is requested.

Confidence: strong (100/100)

[high] Limit cleanup to sandboxes owned by this scenario

Location: skills/docker-sandboxes-env/evals/scaffold-pin-plan.yaml:59

This cleanup applies to every sandbox that appears during the scenario, not just one created by the agent. The harness takes a global sbx ls snapshot and force-removes the difference. A sandbox another user or process creates while the eval runs can therefore be deleted. Also, if the initial listing fails, the harness treats the snapshot as empty and can remove existing sandboxes when the later listing succeeds. Isolate the scenario's resources or restrict cleanup to a verified scenario-owned identity.

Confidence: strong (100/100)

1 recommended validation

Check a helper-pinned kit during sandbox creation

What to check:

Reuse a run for this commit that creates a disposable environment with a helper-pinned public kit. If none exists, create one with the reviewed reference and check that the kit resolves successfully, then remove it.

Why:

The new live scenario runs only sbx env plan and forbids creation. SKILL.md states that plan does not resolve kit references. I can't check whether a creation test passed for this commit.

8 files omitted

8 out of scope

• "CHANGELOG.md"
• "README.md"
• "evals/docker-sandboxes-env.md"
• "skills/docker-sandboxes-env/SKILL.md"
• "skills/docker-sandboxes-env/checks/verification.md"
• "skills/docker-sandboxes-env/references/authoring-workflow.md"
• "skills/docker-sandboxes-env/references/env-schema-fields.md"
• "skills/docker-sandboxes-env/references/sources.md"

_{Finding IDs: f-9e505441b319a69a3187037d (describe-selected-release), f-10eb84f96525f5410f1217a1 (missing-digest-removes-pin), f-61c2ff38621662eb0003bb9d (cleanup-removes-unrelated-sandboxes).}

[docker-gordon]

[high] Limit cleanup to sandboxes owned by this scenario

This cleanup applies to every sandbox that appears during the scenario, not just one created by the agent. The harness takes a global sbx ls snapshot and force-removes the difference. A sandbox another user or process creates while the eval runs can therefore be deleted. Also, if the initial listing fails, the harness treats the snapshot as empty and can remove existing sandboxes when the later listing succeeds. Isolate the scenario's resources or restrict cleanup to a verified scenario-owned identity.

_{Confidence: 🟢 strong · 100/100.}

_{Finding f-61c2ff38621662eb0003bb9d}

[docker-gordon]

[medium] Describe the release selected for the environment

The workflow describes the untagged candidate, which resolves to latest, before selecting newestRelease. If those tags point to different versions, the subsequent steps use the wrong version's arguments, credentials, and host grants. The final checklist permits pin instead of describe, so it does not correct this. Describe the selected release before configuring it.

_{Confidence: 🟢 strong · 100/100.}

_{Finding f-9e505441b319a69a3187037d}

[docker-gordon]

[medium] Do not report a successful pin without a digest

If a registry returns HTTP 200 without a Docker-Content-Digest header, manifest returns None. Both pin_one and pinned update_one then format a reference without a digest and report success. This can silently remove an existing pin or claim to pin an unpinned reference. Reject a missing digest when a pinned result is requested.

_{Confidence: 🟢 strong · 100/100.}

_{Finding f-10eb84f96525f5410f1217a1}