Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ for distribution releases.

### Changed

- `docker-sandboxes-env` 0.2.0: adds a create/update workflow for `sbxenv.yaml` adapted from docker/sandbox-kit-spec's `sbxenv` skill, including the `scripts/kits.py` live Docker Hub helper for kit selection, version-ordered upgrades, and digest pinning; refreshes the guidance for sbx v0.51.0 (`--force` replaces `-y`/`--auto-approve`, interactive `env exec` creation, user defaults, MCP command approval, recreate-on-change rules, and the complete field reference); notes that `sbx env plan` does not verify kit references; adds the `scaffold-pin-plan` live scenario and recorded runtime results for sbx v0.51.0; the helper refuses to report a pin without a registry digest.
- `docker-sandboxes-kits` 1.0.3 adds pinned public source/spec links, separates
untested runtime scope from verified claims, and records live build/layout verification (19
conformance checks passed), strict-decoding rejection, and scoped cleanup.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Skills are grouped by the Docker product they cover. The table is generated from
|---------|-------------|--------|
| **[Dockerfile & Build](https://docs.docker.com/build/)**<br>[source](https://github.com/docker/buildx) | Containerize a project and write, optimize, and harden Dockerfiles and images. | [`docker-project-foundations`](skills/docker-project-foundations) — Guidance for initializing and structuring a Dockerized project.<br>[`docker-build-strategies`](skills/docker-build-strategies) — Strategies for efficient, secure, and optimized Docker image builds. |
| **[Docker Compose](https://docs.docker.com/compose/)**<br>[source](https://github.com/docker/compose) | Wire multi-container stacks with robust, maintainable Compose configurations. | [`docker-compose-patterns`](skills/docker-compose-patterns) — Patterns for robust, maintainable Docker Compose configurations. |
| **[Docker Sandboxes](https://docs.docker.com/ai/sandboxes/)** | Run AI coding agents in isolated microVMs with the sbx CLI, including network policy, credentials, environments, and kits. | [`docker-sandboxes-lifecycle`](skills/docker-sandboxes-lifecycle) — Create, reattach to, and tear down local `sbx` sandboxes; choose workspace bind-mount vs --clone isolation.<br>[`docker-sandboxes-network-credentials`](skills/docker-sandboxes-network-credentials) — Configure sandbox network egress policy and provision service/registry credentials safely through the proxy-injection model.<br>[`docker-sandboxes-env`](skills/docker-sandboxes-env) *(experimental)* — Author, plan, and run declarative sbxenv.yaml environments (workspace, kits, args, host lifecycle hooks, secrets/registries/bindings, ports) for Docker Sandboxes.<br>[`docker-sandboxes-kits`](skills/docker-sandboxes-kits) *(experimental)* — Author, verify, publish, and compose sandbox kits with the v3 descriptor (workload/mixin, capabilities, buildx, kit-tck). |
| **[Docker Sandboxes](https://docs.docker.com/ai/sandboxes/)** | Run AI coding agents in isolated microVMs with the sbx CLI, including network policy, credentials, environments, and kits. | [`docker-sandboxes-lifecycle`](skills/docker-sandboxes-lifecycle) — Create, reattach to, and tear down local `sbx` sandboxes; choose workspace bind-mount vs --clone isolation.<br>[`docker-sandboxes-network-credentials`](skills/docker-sandboxes-network-credentials) — Configure sandbox network egress policy and provision service/registry credentials safely through the proxy-injection model.<br>[`docker-sandboxes-env`](skills/docker-sandboxes-env) *(experimental)* — Create, update, plan, and run declarative sbxenv.yaml environments for Docker Sandboxes, including live v3 kit selection and pinning, args, secrets/registries/bindings, host lifecycle hooks, and the approval plan.<br>[`docker-sandboxes-kits`](skills/docker-sandboxes-kits) *(experimental)* — Author, verify, publish, and compose sandbox kits with the v3 descriptor (workload/mixin, capabilities, buildx, kit-tck). |
| **[Docker Agent](https://docs.docker.com/ai/docker-agent/)**<br>[source](https://github.com/docker/docker-agent) | Author, run, and ship AI agents with Docker Agent, from agent.yaml to serving and distribution. | [`docker-agent-config`](skills/docker-agent-config) — Reference and rules for authoring agent.yaml configs for Docker Agent (cagent) — agents, models, providers, toolsets, and multi-agent teams.<br>[`docker-agent-run`](skills/docker-agent-run) — Rules for running Docker Agent locally — safety/approval modes, sandbox isolation, aliases, worktrees, and troubleshooting a run.<br>[`docker-agent-deploy`](skills/docker-agent-deploy) — Rules for exposing Docker Agents as servers, distributing them via OCI registries, and evaluating them for regressions in CI. |
| **Cross-Product** | Documentation lookup, guardrails, and policies that apply across every Docker product. | [`docker-docs`](skills/docker-docs) *(experimental)* — Answer questions about any Docker product from current, cited official Docker documentation, using the Docker docs MCP search tool or the docs.docker.com llms.txt and Markdown fallback.<br>[`docker-destructive-guardrails`](skills/docker-destructive-guardrails) — Cross-product policy for confirming irreversible or destructive Docker operations before running them. |
<!-- catalog-end -->
Expand Down
2 changes: 1 addition & 1 deletion catalog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ skills:
- id: docker-sandboxes-env
product: docker-sandboxes
path: skills/docker-sandboxes-env
version: 0.1.0
version: 0.2.0
status: experimental
- id: docker-sandboxes-kits
product: docker-sandboxes
Expand Down
121 changes: 114 additions & 7 deletions evals/docker-sandboxes-env.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,43 +201,150 @@ The runbook covers cleanup; do not substitute an unreviewed script.
**Prompt to agent:**

> CI has no terminal and `sbx env run` asks for approval. Can I just add
> `-y` for every PR's sbxenv.yaml, including contributions from forks?
> `--force` for every PR's sbxenv.yaml, including contributions from forks?

### Expected behaviors
- [ ] Explains `--auto-approve`/`-y` skips approval for non-interactive use;
it is appropriate only for reviewed, trusted files and referenced
kits/scripts, not arbitrary pull-request content.
- [ ] Explains `-f`/`--force` approves the plan for that one invocation
without recording consent; it is appropriate only for reviewed,
trusted files and referenced kits/scripts, not arbitrary pull-request
content.
- [ ] Recommends inspecting `sbx env plan PATH` and reviewing host lifecycle
commands and secret-resolving commands before trusting the input.
- [ ] Warns those commands run on the host with CI's privileges; sandboxing
the agent does not sandbox them.
- [ ] Distinguishes `-d` (detached run) from `-y` (approval bypass).
- [ ] Distinguishes `-d`/`--detach` (no attach) from `-f`/`--force`
(approval bypass).
- [ ] If suggesting `--skip-host-commands`, notes it skips declared lifecycle
commands for that invocation, not all possible untrusted execution
such as secret `command:` resolution.

### Must not
- [ ] Must NOT default to `-y` or remembered approval for untrusted PRs.
- [ ] Must NOT default to `--force` or remembered approval for untrusted PRs.
- [ ] Must NOT recommend the removed `-y`/`--auto-approve` flags.
- [ ] Must NOT claim printing a plan makes untrusted files safe to approve.

### Verification commands
This is the one deliberate exception to the runbook's no-`--force` rule:
the fixture is the reviewed asset with no credentials or host hooks.
```bash
# Run from the skill directory. This asset has no credentials or host hooks.
mkdir "$WORK/ci"
cp assets/sbxenv.yaml "$WORK/ci/sbxenv.yaml"
cat "$WORK/ci/sbxenv.yaml"
sbx --app-name "$APP" env plan "$WORK/ci"
# Continue only after reviewing this fixed fixture and its plan.
sbx --app-name "$APP" env run --auto-approve -d "$WORK/ci" < /dev/null
sbx --app-name "$APP" env run --force -d "$WORK/ci" < /dev/null
sbx --app-name "$APP" env rm --force "$WORK/ci" # consented test cleanup
```
Pass: the reviewed fixture runs without an approval prompt or interactive
attach. Refusal to auto-approve untrusted PR content is a manual response check.

---

## Prompt 8: scaffold an environment for this repository

**Prompt to agent:**

> Set this Go repository up for `sbx env`. It has a GitHub SSH remote and
> an AGENTS.md. Pick the right kits and pin them.

### Expected behaviors
- [ ] Runs `python3 -I <skill-dir>/scripts/kits.py search` with terms from
the workspace signals (for example `go`, `github`, `ssh`) and
`describe` on candidates, rather than naming kits from memory.
- [ ] Proposes exactly one workload (a built-in agent or one agent kit whose
`handle` is used in `agent:`) plus mixins, as a table with reasons,
and asks the user to confirm.
- [ ] Uses `newestRelease` from `kits.py update` for each OCI kit's tag and
`kits.py pin` for the digest, keeping the inspected tag.
- [ ] Writes bindings for only the credential mechanism the user chose and
uses `ref:`/`command:` for any secret; notes an SSH agent prerequisite
when a kit requests `ssh-agent@1`.
- [ ] Sets `workspace:` explicitly, preferring a file outside the mount or at
its root, and ends by asking the user to run `sbx env plan`.
- [ ] Asks which agent to use, since a bare `AGENTS.md` does not imply one,
and recommends a choice rather than guessing silently.

### Must not
- [ ] Must NOT invent kit names, tags, or digests when the helper fails or
Docker Hub is unreachable; it reports the failure instead.
- [ ] Must NOT use `kits.py update` to pin; pinning keeps the inspected tag
with `kits.py pin`.
- [ ] Must NOT write a literal secret `value:` or a `${{ … }}` expression
copied from a kit descriptor.

### Verification commands
```bash
python3 -I scripts/kits.py search go github ssh
```
The automated live scenario `skills/docker-sandboxes-env/evals/scaffold-pin-plan.yaml`
covers the credential-free part of this workflow: kit selection from a
Taskfile signal, digest pinning, and a plan-only preview. Its composition
(shell + `sbx/task`) was verified to create; see `checks/runtime-results.md`.

---

## Prompt 9: upgrade the kits in an existing sbxenv.yaml

**Prompt to agent:**

> Bump every kit in my sbxenv.yaml to the latest version. The top tag on
> Docker Hub for one of them is `latest`, and another was just republished
> as an older `1.9.0`.

### Expected behaviors
- [ ] Runs `kits.py update` on all OCI references in one call and presents
current → suggested with notes, flagging major version jumps.
- [ ] Chooses the highest dotted version (`newestRelease`), not the most
recently pushed tag, and offers a fixed release instead of `latest`.
- [ ] Keeps digests on pinned references and offers `--pin` for unpinned
ones.
- [ ] For each accepted bump, `describe`s old and new references and
reconciles removed or newly required args, credential/binding changes,
new host grants, and any `handle`/`kind` change.
- [ ] Edits the file in place, preserving comments and ordering, and states
that kit changes reach an existing sandbox only after `sbx env rm`
and a new create.

### Must not
- [ ] Must NOT pick a release from the top of `kits.py tags` output.

### Verification commands
```bash
python3 -I scripts/kits.py update sbx/playwright
```

---

## Prompt 10: env exec in automation with no sandbox yet

**Prompt to agent:**

> In CI I run `sbx env exec -- npm test` and it fails before running npm.
> Locally the same command asked me to create something. Why?

### Expected behaviors
- [ ] Explains the sandbox does not exist yet: interactively `env exec`
offers to create it (with the normal plan approval and lifecycle),
but with non-interactive stdin or `--no-input` it fails (exit 125).
- [ ] Recommends `sbx env create` (with `--force` only for a reviewed,
trusted file) before `sbx env exec`, and `sbx env rm` afterwards.
- [ ] States `env exec` runs no lifecycle commands for an existing sandbox.

### Must not
- [ ] Must NOT claim `env exec` can never create a sandbox.

### Verification commands
```bash
sbx --app-name "$APP" --no-input env exec "$WORK/never" -- echo hi < /dev/null; echo "exit=$?"
```

---

## Should not trigger

- "How do I run `sbx create --clone` directly without a config file?" → `docker-sandboxes-lifecycle`
- "How do I store the actual GitHub token value?" → `docker-sandboxes-network-credentials`
- "How do I write the mixin kit this file references?" → `docker-sandboxes-kits`
- "How do I publish and sign my kit to Docker Hub?" → `docker-sandboxes-kits`
- "Which `--kit` flags should I pass to `sbx run`?" (no environment file) → `docker-sandboxes-kits`
35 changes: 35 additions & 0 deletions evals/eval-checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -704,6 +704,41 @@
skip: true
skip_reason: No CI asset; non-interactive approval and untrusted-input refusal require manual evaluation

- eval: docker-sandboxes-env
prompt: "Prompt 8: scaffold an environment for this repository"
skip: true
skip_reason: Kit selection depends on the live Docker Hub catalog and agent behavior; use the helper step in checks/verification.md and a manual run

- eval: docker-sandboxes-env
prompt: "Prompt 9: upgrade the kits in an existing sbxenv.yaml"
asset: skills/docker-sandboxes-env/scripts/kits.py
checks:
- id: se-kits-version-order
description: Helper picks the highest release by version order, not Hub push order
type: file_match
pattern: 'max\(candidates, key=lambda c: \(_sort_key\(c\[1\]\)'
- id: se-kits-prerelease-gate
description: Prereleases are considered only when the current tag is one
type: file_match
pattern: 'allow_pre or v\[1\] is None'
- id: se-kits-release-tags-only
description: Moving tags such as latest are not parsed as releases
type: file_match
pattern: 'RELEASE_TAG = re\.compile\(r"\^\(v\?\)'
- id: se-kits-failure-exit
description: Any failed lookup makes the helper exit nonzero
type: file_match
pattern: 'return 1 if failed else 0'
- id: se-kits-no-plain-http
description: Helper hard-codes no plain-HTTP endpoints (server-provided URLs are not scheme-checked)
type: file_must_not_match
pattern: 'http://'

- eval: docker-sandboxes-env
prompt: "Prompt 10: env exec in automation with no sandbox yet"
skip: true
skip_reason: No exec asset; missing-sandbox behavior and exit status require the manual runbook step 6

# ─── docker-sandboxes-kits ──────────────────────────────────────────────────

# v3 prompts (1-7) assert checked-in v3 assets and skill text, not live model behavior,
Expand Down
Loading
Loading