fix(docker): harness isolation via COPY/PRUNE + GRADE-OUTSIDE. Close the criteria/grader leak

.github/workflows/pr-checks.yml

@@ -405,6 +405,18 @@ jobs:
       - name: Build coder-eval-agent base Docker image
         run: make docker-image
 
+      # Docker isolation detectors (COPY/PRUNE + GRADE-OUTSIDE). The daemon-less
+      # set is the load-bearing CI sensor: host-unchanged proxy (no rw host-original
+      # mount) + criteria absence + baked-image scan + no-uid-drop-machinery guard.
+      - name: Run docker isolation detectors (daemon-less)
+        run: make test-docker-detectors
+
+      # Exit-criterion sensor: a real docker run must leave the host byte-for-byte
+      # AND metadata-identical. A daemon + the base image are present on this
+      # runner, so the -m live variant executes here (Linux-authoritative).
+      - name: Run docker host-unchanged live detector
+        run: .venv/bin/pytest tests/test_docker_host_unchanged.py -m live -p no:cacheprovider
+
       - name: Build BYOD template Docker image
         run: docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images
+.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images test-docker-detectors
 
 # Single source of the installed coder-eval version (used to tag the docker
 # images). Referenced lazily inside the docker recipes, so it doesn't run on
@@ -100,6 +100,10 @@ coder-eval-runtime:  ## Build the relocatable runtime kit image (COPY --from sou
 docker-images: docker-image coder-eval-runtime  ## Build BOTH base images (agent for rebase + runtime kit for inject); no creds
 	@echo "Built coder-eval-agent + coder-eval-runtime — ready for both rebase and inject tasks."
 
+test-docker-detectors:  ## Run the docker isolation detectors (host-unchanged proxy + criteria absence + baked-image scan). Daemon-less; CI-cheap.
+	uv run pytest tests/test_docker_host_unchanged.py tests/test_docker_criteria_isolation.py \
+		tests/test_docker_image_no_answer_leak.py -m "not live"
+
 docker-image-full:  ## Build with the UiPath extra (opt-in; uipath resolves from public PyPI, no credentials needed). Codex is always baked in.
 	@VERSION=$$($(VERSION_CMD)); \
 	echo "Building coder-eval-agent:$$VERSION (full: + uipath extra)"; \

docs/DOCKER_ISOLATION.md

@@ -89,8 +89,10 @@ coder-eval run task.yaml -D sandbox.docker.image=my-team/image:latest
 The default when you set nothing is `coder-eval-agent:<installed package version>`.
 
 A worked example ships in-tree: `tasks/byod_smoke_test.yaml` runs against
-`templates/byod_smoke_test/Dockerfile`, which extends the framework image and drops a marker file
-that the task's success criterion then asserts — proving the custom image was actually used. (The
+`templates/byod_smoke_test/Dockerfile`, which extends the framework image and drops a marker file at
+`/opt/byod_marker`. Because that path is outside the copied-out `/work` workspace, the task has the
+agent (which runs inside the container) surface the marker into its workspace, and the criterion then
+asserts it host-side — proving the custom image was actually used. (The
 `byod_*` names here mean "Bring Your Own **Docker**"; they are unrelated to the
 [Bring Your Own Dataset](DATASETS.md) guide, which is about fanning one task out over data rows.)
 
@@ -282,28 +284,97 @@ The host's run dir is bind-mounted **read-write** into the container at the same
 - **Do not** point `--run-dir` at a sensitive parent (e.g. `$HOME` directly, `/etc`, a repo root). Use a dedicated `runs/` subtree.
 - The default (`runs/<timestamp>/`) is safe.
 
+## Isolation model: COPY/PRUNE + GRADE-OUTSIDE
+
+Under `--driver docker` the agent container receives **only read-only or copied inputs and its own throwaway workspace** — it never gets rw access to a host original, and never read access to grading material. Grading runs **outside the agent's reach**, on the host, after the container exits. The guiding rule is: **never chmod a host bind mount; give the agent only copies or read-only inputs.**
+
+**Exit criterion:** a run leaves every host file **byte-for-byte AND metadata-identical** (contents, uid/gid, mode, mtime, symlink targets). Detector A below is the sensor for it.
+
+Three coordinated moves close the criteria/grader leak by **absence**, not by permission:
+
+1. **COPY/PRUNE the agent's inputs.** For each plugin, the host stages a *sanitized bundle copy* (`project_plugin_for_agent` — `skills`/`commands`/`agents`/`hooks`/`.claude-plugin` only, from the `PLUGIN_AGENT_ALLOWED_SUBDIRS` allowlist) and mounts that copy **read-only** at `/work/skills` (`CONTAINER_SKILL_DOCS_DIR`). The raw `$SKILLS_REPO_PATH` checkout, the reference, and the host task dir are **not mounted into the agent container at all**. The staged `task.yaml` is criteria-stripped via `agent_safe_dump()` (`success_criteria: []`, `reference: null`) and `context.json`'s `source_yaml` is nulled — so no grading material is in the agent's mount namespace.
+2. **GRADE OUTSIDE the agent's reach (host).** The container runs the **agent only**; its artifacts cross the boundary via the `/work/output` bind mount. After the container exits, the host grades the copied-out artifacts through the orchestrator's evaluate-only re-grade path (`regrade_on_host`), using the full, unstripped `TaskDefinition` it still holds — with `TASK_DIR` pointing at the **real host task dir**, so `run_command`/`file_check` graders resolve `$TASK_DIR/check_*.py` against the host grader, never agent-written content. Only runs whose final status is `SUCCESS`/`FAILURE`/`MAX_TURNS_EXHAUSTED` are re-graded (an explicit allowlist); a terminal agent-side failure (`ERROR`/`TIMEOUT`/`BUILD_FAILED`/budget) stands untouched.
+3. **`~/.uipath` copy-then-mount.** Like `~/.claude`, `~/.uipath` is forwarded as a throwaway rw **copy**, never the host original — so an agent can never overwrite the host credential.
+
+### Detector A — host-unchanged-after-run
+
+`tests/test_docker_host_unchanged.py`. Two variants: a **daemon-less proxy** (always runs in CI) that asserts no `-v` mount source is a host original mounted rw — only staging copies, `/work/input` (`:ro`), and `/work/output` — proving there is no rw host mount to mutate; and a **daemon-gated real-run** (`-m live`) that snapshots content hash + `os.lstat` metadata (mode, uid, gid, mtime) + symlink targets (including the mount root itself) of the host skills / task dir / reference before and after a real run and asserts they are **byte-for-byte AND metadata-identical**. The real-run check is **Linux-authoritative** (native overlayfs); macOS/Windows Docker Desktop's uid-remap masks host mutation.
+
+### Detector B — zero-grading-material-in-agent-mount
+
+`tests/test_docker_criteria_isolation.py`. Stages a task carrying real criteria + a plugin bundling grader/reference material, scans the **entire agent mount view** (`/work/input` + the sanitized skills copy) and asserts zero grading-material hits (criteria values, `check_*.py`, `RESOLUTION.md`, `reference_agents/`, reference values). A positive control asserts the **host** still holds the full criteria, so a vacuous "staged nothing" bug cannot pass.
+
+## Residual leaks
+
+Allowlist-by-absence has **no DAC backstop** (there is no permission barrier — the agent simply never receives the material), so the boundary correctness is load-bearing:
+
+1. **Prune-boundary miss.** A plugin that puts answers *inside* an allowed dir (e.g. `skills/answers.md`) defeats the prune — `PLUGIN_AGENT_ALLOWED_SUBDIRS` is a coder_eval-side guess about what is answer-free. Durable fix (cross-repo follow-up): push the agent-bundle boundary into the skills repo (a manifest declaring the agent-safe surface).
+2. **Reference/golden material inside the bundle.** A plugin bundling a reference solution under an allowed subtree ships to the agent. Detector B catches known sentinels, not an unknown golden file — reinforces risk 1.
+3. **Un-stripped `task.yaml` fields / author-pointed mounts.** `agent_safe_dump` strips only `success_criteria`/`reference`. A task author who hides expected values in `initial_prompt`/`system_prompt`/pre-post commands/`metadata` leaks them to the agent (semantic, not mechanically enforceable — see the `agent_safe_dump` docstring). The remaining agent-container mounts are `template_sources[]` dirs, a stray absolute `system_prompt_file` (normally inlined+nulled at load), and any `sandbox.docker.extra_mounts` entries, all mounted `:ro`. All three now go through the **grader-dir overlap guard**: a mount whose source equals, contains, or is contained by the host task dir (`rt.task_file.parent` — holds `check_*.py` / reference / unstripped criteria) is a hard error, so the task dir can no longer be re-exposed that way. The residual risk is a mount pointed at *another* answer-bearing location outside the task tree — the guard can't know about it, so keep template/system-prompt/extra-mount paths off any grading material.
+4. **Grade-outside boundary bleed.** If the host re-grade read agent-written content as if it were the reference, grading integrity would be compromised. Mitigation: the re-grade `Sandbox.task_dir` is the real host task dir (`rt.task_file.parent`), never the agent workspace (Detector-adjacent test in `tests/test_docker_regrade.py`).
+5. **Baked image content.** Mocks/tooling baked into `docker/Dockerfile` must not encode task-specific expected values — authoring invariant + the baked-image scan (`tests/test_docker_image_no_answer_leak.py`).
+6. **Env signposts.** `TASK_DIR`/`SKILLS_REPO_PATH` live on the grader (host) env only — never in the agent container's env (which has no task-dir/skills-repo mount to point at anyway).
+
+## Harness-outside: `pre_run` and `post_run` run on the host
+
+`pre_run` / `post_run` commands invoke helper scripts under the skills-repo `tests/` tree (seed generators, fixture copies, cloud teardown) — the same tree that is **never mounted into the agent container** (it carries graders + criteria). So under `--driver docker` both phases run **on the host**, where the full repo + credentials (`SKILLS_REPO_PATH` etc.) already live — the **same trust boundary as grading**. The container runs the **agent turn only**; the in-container orchestrator skips both phases.
+
+- **`pre_run` runs host-side, before the container**, with `cwd` = a per-task **staging dir** (`staging/workspace_seed/`). Every seed a `pre_run` produces (a `seed.json`, a `cp -r …_fixtures/<proj>` directory tree) lands there. The staging dir is mounted **read-only** at `/work/seed` (`CONTAINER_WORKSPACE_SEED_DIR`); the in-container orchestrator copies its contents into the agent workspace **after** template materialization and **before** the agent starts (`Sandbox.seed_from`, recursive + size-bounded). **Collision policy: a seed entry wins over a colliding template starter** — identical to the tempdir ordering, where `pre_run` runs after `_setup_template`. A required (`fail_on_error`) `pre_run` failure aborts **before** the container starts, so no LLM budget is spent on a broken environment; the failure is recorded on an `ERROR` result with `pre_run_results` populated.
+- **`post_run` runs host-side, after the container exits**, with `cwd` = the copied-out workspace (so a teardown that reads a seeded `seed.json` sees it — the seed round-trips out of the container). It is informational (non-fatal) and runs best-effort regardless of grade, so cloud resources are torn down even for `ERROR`/`TIMEOUT` runs.
+
+The helper scripts and the raw repo tree stay host-side throughout — moving the phases out of the container does **not** re-open the leak.
+
+### Interim carve-out: 6 tasks must use `--driver tempdir`
+
+A handful of `uipath-agents/coded/` tasks have a `pre_run` that builds a virtualenv (`uv sync`) bundled with live-tenant provisioning (`uip codedagent setup --force`). Those must run **inside** the container (the venv's absolute paths are non-portable off the host; provisioning needs the in-container CLIs), which host-side `pre_run` cannot yet do. Until in-container `pre_run` execution lands, a resolution-time guard hard-errors these under `--driver docker` (matching `uv sync` / `uip codedagent setup` in a `pre_run` command) with a redirect: **run them with `--driver tempdir`.** The guard reads the *resolved* driver, so a CLI `--driver docker` is honored. All other docker tasks (seed / fixture-copy `pre_run`, cloud-teardown `post_run`) run host-side unchanged.
+
 ## Boundary
 
 | Layer | Location |
 |---|---|
+| **`pre_run` (seeds the agent workspace)** | **host, before the container (staging dir → `/work/seed` :ro → copied into the workspace)** |
 | Agent process (Claude Code SDK) | inside container |
-| Sandbox + per-row criterion checking | inside container |
-| **`task.json` serialization** | **container → host bind mount** |
+| Sandbox setup + agent turn | inside container |
+| **`task.json` (agent trajectory) serialization** | **container → host bind mount** |
+| **Criterion checking / grading (GRADE-OUTSIDE)** | **host, after the container exits** |
+| **`post_run` (teardown over the copied-out workspace)** | **host, after the container exits** |
 | Per-criterion `aggregate()` (P/R/F1, suite thresholds) | host |
 | Reports, run summary, experiment rollups | host |
 
-`task.json` is the only artifact crossing the boundary. Aggregation reads it via the existing host pipeline unchanged.
+`task.json` is the only artifact crossing the boundary (agent trajectory + artifacts). The host re-grade merges the real grades onto it and re-persists it, so the on-disk record carries both the trajectory and the authoritative grade.
 
 ## Limitations
 
 - **Relative template paths**: `template_sources[].path` is resolved to a host absolute path *before* staging, so it won't exist inside the container unless you also forward the parent dir via `sandbox.docker.extra_mounts`.
 - **No container reuse across tasks**: each task = one fresh container. Adds ~1–3 s startup overhead per task; negligible vs. LLM latency.
 - **macOS Keychain auth**: not reachable from the container; set `ANTHROPIC_API_KEY` (direct) or Bedrock credentials instead.
 
+### Early stop (`stop_early`) is not supported under `--driver docker`
+
+Criterion-level early stop (a `stop_early:` block, driven by the `EarlyStopWatcher`)
+relies on **live criterion verdicts computed during the agent turn**. Under COPY/PRUNE
++ GRADE-OUTSIDE the container runs the agent with the criteria **stripped**, and grading
+happens on the host **after** the container exits — so the in-container watcher can never
+arm. A `stop_early:` block is therefore a **no-op under docker**: the run does not stop
+early. `DockerRunner` logs a loud warning when a task arms early stop under docker, so it
+is a documented, signposted limitation rather than a silent one.
+
+**Verdict is unaffected.** The host re-grade still grades the full criteria, and a run
+that completes naturally gates strict-AND — the same authoritative outcome, just without
+the early cutoff (a cost/time optimization) and its telemetry.
+
+The leak-free way to make early stop work under docker is a **host-side watcher**: the
+host already receives the container's per-tool-call event stream and already signals the
+container via the heartbeat channel, and the agent already supports cooperative stop — so
+the watcher can run on the host (where the full criteria live, never entering the
+container), compute verdicts against the real criteria, and cooperatively signal the
+container to stop. That is the intended follow-up; until then, run `stop_early` suites
+with `--driver tempdir`.
+
 ## Architecture
 
 The host's `DockerRunner` (`coder_eval/isolation/docker_runner.py`) renders the `docker run` argv, bind-mounts task inputs at `/work/input`, allocates an output dir at `/work/output`, and tails container stdout into `docker.log` in the task's run dir.
 
-Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
+Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the *criteria-stripped* staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`) to execute the **agent turn only**, and writes `task.json` to the output mount. The host then re-grades the copied-out artifacts (`regrade_on_host`) against the full criteria it holds, merges the authoritative grades onto the trajectory, and feeds the existing aggregation pipeline. The container never receives the criteria, reference, or graders (see [Isolation model](#isolation-model-copyprune--grade-outside)).
 
 A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.

src/coder_eval/cli/plan_command.py

@@ -59,6 +59,7 @@ def plan_command(
     check_api_keys()
 
     # Lazy import to avoid circular dependency at module level
+    from ..orchestration.docker_guard import DockerPreRunHostUnsafeError, validate_docker_pre_run_host_safety
     from ..orchestration.early_stop import EarlyStopConfigError, validate_early_stop
     from ..orchestration.experiment import DEFAULT_EXPERIMENT_PATH, load_experiment, resolve_task_for_variant
 
@@ -136,6 +137,8 @@ def plan_command(
                     resolved, _lineage, _ = resolve_task_for_variant(default_exp, task, exp_def, variant)
                     # Early-stop guardrails (no-op unless a criterion carries a stop_early: block).
                     validate_early_stop(resolved)
+                    # Interim docker guard: docker pre_run that must run in-container.
+                    validate_docker_pre_run_host_safety(resolved)
                     agent_type = str(resolved.agent.type) if resolved.agent else "unknown"
                     agent_model = resolved.agent.model if resolved.agent else None
                     model_str = f" ({agent_model})" if agent_model else ""
@@ -145,6 +148,11 @@ def plan_command(
                     # failures, which stay soft): flip the plan exit code.
                     console.print(f"    [red]Variant '{variant.variant_id}': early-stop config error - {e}[/red]")
                     all_valid = False
+                except DockerPreRunHostUnsafeError as e:
+                    # Interim docker guard: same hard-error treatment (flip the
+                    # exit code) — a docker pre_run that must run in-container.
+                    console.print(f"    [red]Variant '{variant.variant_id}': docker config error - {e}[/red]")
+                    all_valid = False
                 except Exception as e:
                     console.print(f"    [red]Variant '{variant.variant_id}': resolution failed - {e}[/red]")