From c793e61d58c4e6538cdab34ba7953f9d555c485a Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Wed, 12 Aug 2026 17:44:15 +0200 Subject: [PATCH 01/81] feat(anvil): replace the container backend with two artifacts and a hook The 0.4.0 backend shipped nine generated files -- two shell drivers, two image-id helpers, an entrypoint, a README -- plus a tier-routing seam (runner.just, the anvil-runner managed region) and roughly 1200 lines of duplicated .sh/.ps1. Customizing it meant owning that surface. Containerized execution is now two artifacts and one optional hook: - .anvil/container/Dockerfile (with its build-context ignore file) defines what the image contains. It installs tools by running just anvil-setup, so there is no second tool list to keep in step. A repository edits it in place; a downstream catalog replaces it via replace_artifact. - justfiles/anvil/container.just drives the engine. The image tag *is* a SHA-256 over the Dockerfile, its ignore file, rust-toolchain.toml, the optional hook and the generated recipe tree, so presence implies freshness and there is no staleness bookkeeping. - .anvil/container/hooks.ps1, when present, supplies credentials. Anvil-PreBuild returns BuildKit secrets, Anvil-PreRun returns run-time environment; both are passed by variable name so a value never reaches a process argument or an image layer, and an empty value fails closed. There is no configuration file: whether the group is emitted is a catalog decision, and the only host-specific value -- which engine to call -- is the ANVIL_CONTAINER_ENGINE variable read at run time. There is no transparent tier routing either: just anvil-pr runs natively and the container is reached only through just anvil-container . aprz.just drops its container-specific token plumbing; inside the image a credential now arrives as an ordinary environment variable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 51 +- .anvil/container/Containerfile | 69 - .anvil/container/Containerfile.dockerignore | 26 - .anvil/container/Dockerfile | 115 ++ .anvil/container/Dockerfile.dockerignore | 15 + .anvil/container/README.md | 228 --- .anvil/container/entrypoint.sh | 23 - .anvil/container/image-id.ps1 | 76 - .anvil/container/image-id.sh | 91 - .anvil/container/run-in-container.ps1 | 342 ---- .anvil/container/run-in-container.sh | 324 ---- .spelling | 8 +- Justfile | 4 - crates/cargo-anvil/README.md | 181 +- .../src/anvil/artifacts/container.rs | 894 ++------- .../src/anvil/artifacts/justfile.rs | 62 +- crates/cargo-anvil/src/anvil/artifacts/mod.rs | 13 +- .../cargo-anvil/src/anvil/artifacts/region.rs | 17 - crates/cargo-anvil/src/lib.rs | 148 +- .../templates/anvil/container/Containerfile | 69 - .../container/Containerfile.dockerignore | 26 - .../templates/anvil/container/README.md | 228 --- .../templates/anvil/container/entrypoint.sh | 23 - .../templates/anvil/container/image-id.ps1 | 76 - .../templates/anvil/container/image-id.sh | 91 - .../anvil/container/run-in-container.ps1 | 342 ---- .../anvil/container/run-in-container.sh | 324 ---- .../templates/container/Dockerfile | 115 ++ .../container/Dockerfile.dockerignore | 15 + .../justfiles/anvil/checks/aprz.just | 22 +- .../templates/justfiles/anvil/container.just | 340 +++- .../templates/justfiles/anvil/mod.just | 1 - .../templates/justfiles/anvil/runner.just | 47 - .../templates/justfiles/anvil/tiers.just | 19 +- .../templates/regions/justfile-runner.just | 1 - .../tests/container_customization.rs | 754 -------- .../tests/container_customization_bash.rs | 722 ------- crates/cargo-anvil/tests/container_upgrade.rs | 204 -- crates/cargo-anvil/tests/extensibility.rs | 70 +- .../snapshots/snapshots__ado_backend.snap | 1652 +++++------------ .../snapshots/snapshots__github_backend.snap | 1652 +++++------------ .../snapshots/snapshots__local_only.snap | 1652 +++++------------ crates/cargo-anvil/tests/tier_routing.rs | 302 --- crates/cargo-coverage-gate/README.md | 2 +- justfiles/anvil/checks/aprz.just | 22 +- justfiles/anvil/container.just | 340 +++- justfiles/anvil/mod.just | 1 - justfiles/anvil/runner.just | 47 - justfiles/anvil/tiers.just | 19 +- 49 files changed, 2497 insertions(+), 9368 deletions(-) delete mode 100644 .anvil/container/Containerfile delete mode 100644 .anvil/container/Containerfile.dockerignore create mode 100644 .anvil/container/Dockerfile create mode 100644 .anvil/container/Dockerfile.dockerignore delete mode 100644 .anvil/container/README.md delete mode 100644 .anvil/container/entrypoint.sh delete mode 100644 .anvil/container/image-id.ps1 delete mode 100644 .anvil/container/image-id.sh delete mode 100644 .anvil/container/run-in-container.ps1 delete mode 100644 .anvil/container/run-in-container.sh delete mode 100644 crates/cargo-anvil/templates/anvil/container/Containerfile delete mode 100644 crates/cargo-anvil/templates/anvil/container/Containerfile.dockerignore delete mode 100644 crates/cargo-anvil/templates/anvil/container/README.md delete mode 100644 crates/cargo-anvil/templates/anvil/container/entrypoint.sh delete mode 100644 crates/cargo-anvil/templates/anvil/container/image-id.ps1 delete mode 100644 crates/cargo-anvil/templates/anvil/container/image-id.sh delete mode 100644 crates/cargo-anvil/templates/anvil/container/run-in-container.ps1 delete mode 100644 crates/cargo-anvil/templates/anvil/container/run-in-container.sh create mode 100644 crates/cargo-anvil/templates/container/Dockerfile create mode 100644 crates/cargo-anvil/templates/container/Dockerfile.dockerignore delete mode 100644 crates/cargo-anvil/templates/justfiles/anvil/runner.just delete mode 100644 crates/cargo-anvil/templates/regions/justfile-runner.just delete mode 100644 crates/cargo-anvil/tests/container_customization.rs delete mode 100644 crates/cargo-anvil/tests/container_customization_bash.rs delete mode 100644 crates/cargo-anvil/tests/container_upgrade.rs delete mode 100644 crates/cargo-anvil/tests/tier_routing.rs delete mode 100644 justfiles/anvil/runner.just diff --git a/.anvil.lock b/.anvil.lock index ed83344f..ca9016b1 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,39 +1,15 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:21384ef36674d981928c529a65f44082f6bf1ee64e55f7008a1e587495e0b7f5" +catalog_checksum = "sha256:e98fd525a9b48a5d2bc57066426abf8bdc78bcde87a38ab9d17e80210a43ef81" [[file]] -path = ".anvil/container/Containerfile" -checksum = "sha256:4c343282b4e773978ff29b7d39e3a0955975bf64ad3e63afe1aa53e51be42916" +path = ".anvil/container/Dockerfile" +checksum = "sha256:883a07d88dfbecfde9460be7346ad64449e3fa16b0354f3212685ad9f2630a88" [[file]] -path = ".anvil/container/Containerfile.dockerignore" -checksum = "sha256:359c2bd70d599ff1e5674eae2af5bb35f344f78e642da7024cfdc6efc5d345da" - -[[file]] -path = ".anvil/container/README.md" -checksum = "sha256:ea255c3659e17dee291ded4633394d080c5aae8bd6e3e87295f27ffab5f428bc" - -[[file]] -path = ".anvil/container/entrypoint.sh" -checksum = "sha256:09576bca317f5a413572f6626fc052214f885589f0547ed0b0c044b92cc402e7" - -[[file]] -path = ".anvil/container/image-id.ps1" -checksum = "sha256:8cdd2dd9cdfb037d768802e4dda7d1156f626dc711663e6bc39fbb86d1ce2e04" - -[[file]] -path = ".anvil/container/image-id.sh" -checksum = "sha256:b526a643e42902dd1e8acff65529fb63741760f72c14b123e9747394158faf53" - -[[file]] -path = ".anvil/container/run-in-container.ps1" -checksum = "sha256:7056cf968119e3b8c1b143fb2f3910d3bf2ecb3e40e1ab353ed5617714de3f5e" - -[[file]] -path = ".anvil/container/run-in-container.sh" -checksum = "sha256:19fb40707b4ef4661eea2aa9295bae886d0b1e23949980afb8ca0b9d8a2a6b1d" +path = ".anvil/container/Dockerfile.dockerignore" +checksum = "sha256:b04c88f8c52256b99b4590f745945db4cd9b501ddb36851230060d22a1d1c14c" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -93,7 +69,7 @@ checksum = "sha256:91408602dc3ee274b593e234841934c749ff03bba0ee7846ab88247c06f20 [[file]] path = "justfiles/anvil/checks/aprz.just" -checksum = "sha256:62d8334b55b0ba3b221550037dec987d9749e773f40df6c0cd18f08164c095bf" +checksum = "sha256:db01bd6484a3161a1f66dd558f35a14c4043d0000092ccb51fa7ac7831fbeccf" [[file]] path = "justfiles/anvil/checks/audit.just" @@ -213,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:cbec6450a64f800963ea5ba9d2eedfa4e90d91d441307f18d109629f849126ad" +checksum = "sha256:f7cbcc86dd14210d58e198a4b4af1e2590c6816b5497583ceb1b361b73a7419b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" @@ -257,15 +233,11 @@ checksum = "sha256:cf6b30b8f4fd10eeb5bb5660c8e60512434fd94405a4fd8ea7399ababa089 [[file]] path = "justfiles/anvil/mod.just" -checksum = "sha256:f3bc59e6228858e350d434d0cc76497cde258279772c414feae2b899d5cc5666" - -[[file]] -path = "justfiles/anvil/runner.just" -checksum = "sha256:9af5103a050478f14051e34d9b9bfb348f4d0b6173f4e23e9871ce14a4a6317c" +checksum = "sha256:74fb8d54efb7cb38ea68b4832f1dae86a153787a2e84081bf19efa823ea5a3e1" [[file]] path = "justfiles/anvil/tiers.just" -checksum = "sha256:713c5a2ae28b6b5aa20dd38226278b3b7f71bbc5e6b84a16eaf5244713f27ed9" +checksum = "sha256:2bdd09cf9101e56ef41976c195a594b3fb9cf319528325bcba97ac2cc989082c" [[file]] path = "justfiles/anvil/tools.just" @@ -295,11 +267,6 @@ host = "Justfile" id = "anvil-imports" checksum = "sha256:f8affd59b69c7083c2f3b6f593c63672116dda974c1e661dcb66a4412eb3eada" -[[region]] -host = "Justfile" -id = "anvil-runner" -checksum = "sha256:a31c6dd3fc8402e1e89fcf1948e03103e0dc66fbb988c34510eddeded5a67b0e" - [[region]] host = "clippy.toml" id = "anvil-clippy" diff --git a/.anvil/container/Containerfile b/.anvil/container/Containerfile deleted file mode 100644 index fa68b18f..00000000 --- a/.anvil/container/Containerfile +++ /dev/null @@ -1,69 +0,0 @@ -# syntax=docker/dockerfile:1 -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. - -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e -FROM ${BASE_IMAGE} - -ARG ANVIL_IMAGE_ID -ARG JUST_VERSION=1.56.0 -ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 -ARG POWERSHELL_VERSION=7.6.3 -ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 -ARG RUSTUP_VERSION=1.29.0 -ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 - -ENV DEBIAN_FRONTEND=noninteractive \ - CARGO_HOME=/usr/local/cargo \ - RUSTUP_HOME=/usr/local/rustup \ - RUSTUP_NO_UPDATE_CHECK=1 \ - PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - build-essential ca-certificates clang libclang-dev curl git libicu-dev \ - libssl-dev pkg-config tar \ - && rm -rf /var/lib/apt/lists/* - -RUN curl -fsSLo /tmp/powershell.tar.gz \ - "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ - && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ - && mkdir -p /opt/microsoft/powershell/7 \ - && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ - && chmod 755 /opt/microsoft/powershell/7/pwsh \ - && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ - && rm /tmp/powershell.tar.gz - -RUN curl -fsSLo /tmp/just.tar.gz \ - "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ - && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ - && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ - && chmod 755 /usr/local/bin/just \ - && rm /tmp/just.tar.gz - -RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ - "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ - && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ - && chmod 755 /tmp/rustup-init \ - && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ - && rm /tmp/rustup-init - -WORKDIR /opt/anvil -COPY . . -RUN test -f rust-toolchain.toml || { \ - echo "anvil-container requires rust-toolchain.toml" >&2; \ - exit 1; \ - } -RUN --mount=type=cache,id=anvil-cargo-registry,target=/usr/local/cargo/registry \ - --mount=type=cache,id=anvil-cargo-git,target=/usr/local/cargo/git \ - --mount=type=cache,id=anvil-cargo-target,target=/tmp/anvil-target \ - printf "anvil_runner := \"native\"\nimport 'justfiles/anvil/mod.just'\n" > Justfile \ - && CARGO_TARGET_DIR=/tmp/anvil-target just anvil-setup - -COPY .anvil/container/entrypoint.sh /usr/local/bin/anvil-container-entrypoint -RUN chmod 755 /usr/local/bin/anvil-container-entrypoint - -ENV ANVIL_IN_CONTAINER=1 -LABEL io.github.cargo-anvil.image-id="${ANVIL_IMAGE_ID}" -WORKDIR /workspace -ENTRYPOINT ["anvil-container-entrypoint"] -CMD ["bash"] diff --git a/.anvil/container/Containerfile.dockerignore b/.anvil/container/Containerfile.dockerignore deleted file mode 100644 index 6566e657..00000000 --- a/.anvil/container/Containerfile.dockerignore +++ /dev/null @@ -1,26 +0,0 @@ -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Deny-all allow-list for the image build context. -# -# Docker matches each candidate against every pattern in order and lets the -# last match win, testing the path itself *and each of its parent directories* -# (moby/patternmatcher MatchesOrParentMatches). A bare directory re-inclusion -# such as `!justfiles` therefore re-admits the entire subtree below it, which -# would defeat this allow-list, so list only leaf patterns here. Docker still -# descends into a denied directory when some re-inclusion pattern is prefixed -# by it, so the intermediate directories need no entries of their own. -# -# Parent testing also reaches through a single-segment re-inclusion: a -# subdirectory of `.anvil/container/` matches `!.anvil/container/*` in its own -# right. The image-ID helpers list that directory one level deep, so a nested -# file is not an image input; `.anvil/container/*/*` states that leaf-only -# contract in the allow-list too, at every depth, because a deeper candidate -# always has an ancestor of exactly that shape. -** -!rust-toolchain.toml -!justfiles/anvil/*.just -!justfiles/anvil/checks/*.just -!justfiles/anvil/groups/*.just -!.anvil/container/* -.anvil/container/*/* -.anvil/container/customize.sh -.anvil/container/customize.ps1 diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile new file mode 100644 index 00000000..e1be2998 --- /dev/null +++ b/.anvil/container/Dockerfile @@ -0,0 +1,115 @@ +# syntax=docker/dockerfile:1 +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# Default Anvil execution image, emitted for every repository. The image +# installs exactly the tools the generated catalog pins, by running +# `just anvil-setup` -- the same recipe the checks themselves use. That is what +# makes "the image has the right tools" true by construction rather than by +# convention: there is no second list to keep in step. +# +# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the +# image identity hash and refuses a floating tag, because a tag that can change +# underneath a hash makes the hash a lie. +# +# To build on a different base (a lower glibc baseline, or an internal +# distribution), a downstream catalog replaces this artifact wholesale via +# `replace_artifact(artifacts::container::dockerfile(...))`; a single +# repository can edit this file in place, which anvil's drift handling +# preserves. + +ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +FROM ${BASE_IMAGE} + +ARG JUST_VERSION=1.56.0 +ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 +ARG POWERSHELL_VERSION=7.6.3 +ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 +ARG RUSTUP_VERSION=1.29.0 +ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e + +ENV DEBIAN_FRONTEND=noninteractive \ + CARGO_HOME=/usr/local/cargo \ + RUSTUP_HOME=/usr/local/rustup \ + RUSTUP_NO_UPDATE_CHECK=1 \ + PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare slim base has no C runtime development files, so every +# link step fails without build-essential. +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential ca-certificates clang libclang-dev curl git libicu-dev \ + libssl-dev pkg-config tar \ + && rm -rf /var/lib/apt/lists/* + +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. +RUN curl -fsSLo /tmp/powershell.tar.gz \ + "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ + && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ + && mkdir -p /opt/microsoft/powershell/7 \ + && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ + && chmod 755 /opt/microsoft/powershell/7/pwsh \ + && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ + && rm /tmp/powershell.tar.gz + +RUN curl -fsSLo /tmp/just.tar.gz \ + "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ + && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ + && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ + && chmod 755 /usr/local/bin/just \ + && rm /tmp/just.tar.gz + +RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ + "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ + && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ + && chmod 755 /tmp/rustup-init \ + && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ + && rm /tmp/rustup-init + +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated +# source edit does not invalidate this layer. The synthetic Justfile avoids +# pulling in repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. +WORKDIR /opt/anvil +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# Consumed by the re-entry guard in the generated tier and group recipes: a +# recipe that sees this runs natively instead of launching another container. +ENV ANVIL_IN_CONTAINER=1 + +WORKDIR /workspace +CMD ["bash"] diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore new file mode 100644 index 00000000..21f90464 --- /dev/null +++ b/.anvil/container/Dockerfile.dockerignore @@ -0,0 +1,15 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# BuildKit reads `.dockerignore` in preference to a root +# `.dockerignore`, so this scopes the exec-image build context without the +# repository having to own a root ignore file or having one silently overridden. +# +# The build context is the repository root but the image only needs two things. +# Excluding everything else keeps a cold build from streaming the whole +# worktree (and every stale `target/`) to the daemon. +* +!justfiles +!rust-toolchain.toml diff --git a/.anvil/container/README.md b/.anvil/container/README.md deleted file mode 100644 index b9cd1642..00000000 --- a/.anvil/container/README.md +++ /dev/null @@ -1,228 +0,0 @@ - - -# Run Anvil checks in a local container - -Use `just anvil-container` to run generated Anvil checks in a reproducible -Linux environment without installing the complete Rust and Cargo tool catalog -on the host. - -Native execution remains the default. The first container run builds an image -matching the repository's generated configuration. Later runs reuse that image, -dependency caches, and compilation output. - -## Quick start - -Ensure Docker Engine is running, then run: - -```text -just anvil-container anvil-clippy -``` - -The first run builds the matching image and can take several minutes. - -## Prerequisites - -- [Docker Engine](https://docs.docker.com/engine/install/) 23.0 or newer, - installed directly in Linux or WSL and usable by the current user. -- `git` and `just` on the host. -- Bash on Linux and WSL; PowerShell Core (`pwsh`) and WSL 2 on Windows. -- `[script]` support enabled in the root `Justfile`. Add `set unstable` when - required by the installed `just` version. -- A `rust-toolchain.toml` in the repository root. -- A Linux or WSL environment capable of running `linux/amd64` images, either - natively on x86-64 or through Docker emulation on ARM64. - -On Windows, the driver invokes Docker from the default WSL distribution rather -than calling Windows `docker.exe`. Regardless of how Docker is installed, this -command must succeed from PowerShell: - -```text -wsl -e docker version -``` - -Start the Docker service inside WSL when it is stopped and add the WSL user to -the `docker` group when non-root access is not already configured. Docker -Desktop is not required. - -On ARM64 hosts, Docker emulates the required `linux/amd64` environment. Image -builds and checks can therefore be substantially slower than on x86-64 hosts. - -## Security boundary - -> [!WARNING] -> `customize.sh` and `customize.ps1` execute on the host with the developer's -> permissions before container isolation begins. Reviewing and trusting these -> files is equivalent to reviewing and trusting any other host-executed script -> in the checked-out branch. - -## Common workflows - -Run one check: - -```text -just anvil-container anvil-clippy -``` - -Run the complete pull-request tier: - -```text -just anvil-container anvil-pr -``` - -Every argument is treated as a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not supported by this command surface. - -Open an interactive Bash shell in the image: - -```text -just anvil-container -``` - -### Use containers for tier commands - -Native execution remains the default. To route tier commands such as -`just anvil-pr` through the container for the current shell: - -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` - -On Unix: - -```sh -ANVIL_RUNNER=container just anvil-pr -``` - -For one invocation: - -```text -just anvil_runner=container anvil-pr -``` - -To make container execution the repository default, change the default value -in the `anvil-runner` region of the repository-root `Justfile` from `"native"` -to `"container"` and commit that policy. Set `ANVIL_RUNNER=native` to override -the repository default for the current shell. - -Tier routing starts a nested `just` invocation. Output and exit status are -preserved, but outer `--dry-run`, dependency introspection, global options, and -CLI variable assignments are not propagated to the selected private tier. -Values other than `native` and `container` are rejected. - -## Images and caches - -The image name includes a content-based tag derived from the repository's Rust -toolchain, generated Anvil recipes, and container build configuration. A -relevant change selects a new image automatically; older branches can continue -using their matching images. - -The following data is reused between runs: - -- the matching container image; -- repository-scoped Cargo registry and Cargo Git caches; -- compilation output in a repository- and image-specific `target` volume. - -The repository is mounted read/write at `/workspace`. Build output remains in a -named volume instead of the host `target/`, avoiding incompatible artifacts and -slow host-to-virtual-machine I/O. - -## GitHub authentication - -`anvil-aprz` and aggregate tiers that include it require GitHub API -authentication. The driver uses either: - -- the host `GITHUB_TOKEN`; or -- the token from an authenticated host `gh` session. - -Trusted customization can provision a short-lived token by setting -`GITHUB_TOKEN`; the driver reads it after loading and validating customization. - -Authenticate the GitHub CLI with: - -```text -gh auth login --hostname github.com -``` - -For an aggregate tier, the driver first runs `anvil-aprz` in a short-lived -container with the token mounted read-only. After it succeeds, the driver runs -the remaining checks in another container without the token. Temporary token -files are removed afterward. - -An interactive invocation can pause while you authenticate. A non-interactive -invocation fails with instructions when authentication is unavailable. - -## Configuration - -| Variable | Effect | -|---|---| -| `ANVIL_RUNNER` | Selects `native` or `container` execution for tier commands | -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a digest-pinned compatible Linux base image and changes the content-based tag | -| `ANVIL_CONTAINER_IMAGE` | Changes the local image name; the content-based tag is retained | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails instead of building when the matching image is absent | - -The public driver builds images locally and does not pull -`ANVIL_CONTAINER_IMAGE` from a registry. - -The default base is digest-pinned Debian Bookworm. Set -`ANVIL_CONTAINER_BASE_IMAGE` to another image compatible with the generated -Debian-based `Containerfile` when a lower glibc baseline is required. A -different package ecosystem such as Azure Linux requires a derived -`Containerfile`. The value must use `image@sha256:` form so the -selected base remains part of the content-addressed image identity. - -Two simultaneous cold invocations can both build the same missing image. This -is accepted for local development: the content-addressed tag converges on the -same inputs, at the cost of duplicate work. - -## Troubleshooting - -| Problem | Resolution | -|---|---| -| Docker is not found on Linux or WSL | Install Docker Engine 23.0 or newer inside that environment | -| Docker is unavailable from Windows | Run `wsl -e docker version`; install or start Docker Engine in the default WSL distribution | -| Docker requires elevated access | Add the Linux/WSL user to the `docker` group, then start a new shell | -| ARM64 execution is slow | The current image is `linux/amd64` and runs through Docker emulation | -| `linux/amd64` cannot run | Configure Docker to run `linux/amd64` images | -| `[script]` recipes are unavailable | Enable `[script]` support; older `just` versions require `set unstable` | -| `rust-toolchain.toml` is missing | Add the repository-owned toolchain file at the repository root | -| GitHub authentication is unavailable | Run `gh auth login --hostname github.com` or set host `GITHUB_TOKEN` | -| A matching image is missing with `ANVIL_CONTAINER_NO_REBUILD=1` | Unset the variable to allow the local image build | -| The first run is slow | The initial image build installs the pinned tool catalog; later runs reuse it | - -Use `docker images anvil-dev` inside Linux or WSL to list locally cached -default Anvil images. - -## Managed files - -This directory is managed by `cargo-anvil`. Regenerate it with `cargo anvil` -instead of editing its files directly. - -> [!IMPORTANT] -> These assets previously lived in `justfiles/anvil/container/`. `cargo anvil` -> relocates the files it generated, but it does not track a hand-authored -> `customize.sh` or `customize.ps1`. Move any such file to -> `.anvil/container/` yourself; the driver only loads customization from the -> new location and warns on stderr when it finds one left behind. - -## Advanced repository customization - -A repository or derived catalog can add one trusted customization file per -supported host: - -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 -``` - -The driver sources the matching file as trusted host code before authentication, -image construction, and recipe execution. The documented customization -contract provides inputs and validated outputs for APRZ classification, build -secrets, dependency preparation, runtime arguments, and cleanup. - -Customization source is excluded from image identity and the build context. -Non-secret image behavior must be represented by hashed static files such as -the `Containerfile`, entrypoint, or supporting build scripts. - -See the [container customization contract](https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md#8-container-customization) -for the complete interface and security requirements. diff --git a/.anvil/container/entrypoint.sh b/.anvil/container/entrypoint.sh deleted file mode 100644 index fadac5f9..00000000 --- a/.anvil/container/entrypoint.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/sh -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -eu - -if [ "$(id -u)" -ne 0 ]; then - if [ -z "${HOME:-}" ] || [ "$HOME" = "/" ]; then - HOME="/tmp/anvil-user" - export HOME - fi - - user_cargo_home="$HOME/.cargo" - mkdir -p "$user_cargo_home" - for file in config.toml .crates.toml .crates2.json; do - if [ -r "$CARGO_HOME/$file" ]; then - cp -f "$CARGO_HOME/$file" "$user_cargo_home/$file" - fi - done - export CARGO_HOME="$user_cargo_home" - ln -sfn /usr/local/cargo/registry "$CARGO_HOME/registry" - ln -sfn /usr/local/cargo/git "$CARGO_HOME/git" -fi - -exec "$@" diff --git a/.anvil/container/image-id.ps1 b/.anvil/container/image-id.ps1 deleted file mode 100644 index dfc6a53f..00000000 --- a/.anvil/container/image-id.ps1 +++ /dev/null @@ -1,76 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$inputs = @( - 'rust-toolchain.toml' -) -$toolchainPath = Join-Path $repoRoot 'rust-toolchain.toml' -if (-not (Test-Path -LiteralPath $toolchainPath -PathType Leaf)) { - throw 'anvil-container requires a repository-owned rust-toolchain.toml.' -} -$containerPath = Join-Path $repoRoot '.anvil/container' -$containerRecipe = 'justfiles/anvil/container.just' -$containerfile = Join-Path $containerPath 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$pathComparison = if ($IsWindows) { [StringComparison]::OrdinalIgnoreCase } else { [StringComparison]::Ordinal } -# The container entry recipe drives execution on the host; it is not image -# content, so it must not participate in image identity. -$inputs += Get-ChildItem (Join-Path $repoRoot 'justfiles/anvil') -Recurse -File -Filter '*.just' | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } | - Where-Object { -not $_.Equals($containerRecipe, $pathComparison) } -$executionOnly = @( - 'image-id.ps1', - 'image-id.sh', - 'README.md', - 'run-in-container.ps1', - 'run-in-container.sh', - 'customize.sh', - 'customize.ps1' -) -# customize.sh/customize.ps1 are trusted runtime orchestration, not image -# content: their source must never affect the image ID or build context. -# Static, non-secret build customization belongs in a hashed artifact instead. -$inputs += Get-ChildItem $containerPath -File | - Where-Object { $_.Name -notin $executionOnly } | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } -$uniqueInputs = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) -foreach ($inputPath in $inputs) { - [void]$uniqueInputs.Add($inputPath) -} -$inputs = [string[]]$uniqueInputs -[Array]::Sort($inputs, [StringComparer]::Ordinal) - -$payload = [Text.StringBuilder]::new() -[void]$payload.Append("ANVIL_CONTAINER_BASE_IMAGE`n").Append($baseImage).Append("`n") -foreach ($relative in $inputs) { - $path = Join-Path $repoRoot $relative - if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { - throw "Container image input is missing: $relative" - } - $content = [IO.File]::ReadAllText($path).Replace("`r`n", "`n").Replace("`r", "`n") - [void]$payload.Append($relative).Append("`n").Append($content).Append("`n") -} - -$bytes = [Text.Encoding]::UTF8.GetBytes($payload.ToString()) -$hash = [Security.Cryptography.SHA256]::HashData($bytes) -Write-Output ([Convert]::ToHexString($hash).ToLowerInvariant()) diff --git a/.anvil/container/image-id.sh b/.anvil/container/image-id.sh deleted file mode 100644 index e0ed8105..00000000 --- a/.anvil/container/image-id.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi - -toolchain_path="$repo_root/rust-toolchain.toml" -if [[ ! -f "$toolchain_path" ]]; then - echo 'anvil-container requires a repository-owned rust-toolchain.toml.' >&2 - exit 1 -fi - -container_dir="$repo_root/.anvil/container" -container_recipe="justfiles/anvil/container.just" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$container_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -inputs=(rust-toolchain.toml) -while IFS= read -r path; do - relative="${path#"$repo_root"/}" - # The container entry recipe drives execution on the host; it is not - # image content, so it must not participate in image identity. - if [[ "$relative" != "$container_recipe" ]]; then - inputs+=("$relative") - fi -done < <(find "$repo_root/justfiles/anvil" -type f -name '*.just' -print) - -for path in "$container_dir"/*; do - [[ -f "$path" ]] || continue - case "${path##*/}" in - image-id.ps1 | image-id.sh | README.md \ - | run-in-container.ps1 | run-in-container.sh \ - | customize.sh | customize.ps1) continue ;; - esac - inputs+=("${path#"$repo_root"/}") -done - -if command -v sha256sum >/dev/null 2>&1; then - hash_command=(sha256sum) -elif command -v shasum >/dev/null 2>&1; then - hash_command=(shasum -a 256) -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi - -write_normalized_file() { - local path="$1" - local line status - while true; do - line="" - if IFS= read -r line <&3; then - status=0 - else - status=$? - fi - if ((status != 0)) && [[ -z "$line" ]]; then - break - fi - printf '%s' "${line%$'\r'}" - if ((status == 0)); then - printf '\n' - else - break - fi - done 3<"$path" -} - -{ - printf 'ANVIL_CONTAINER_BASE_IMAGE\n%s\n' "$base_image" - while IFS= read -r relative; do - path="$repo_root/$relative" - if [[ ! -f "$path" ]]; then - echo "Container image input is missing: $relative" >&2 - exit 1 - fi - printf '%s\n' "$relative" - write_normalized_file "$path" - printf '\n' - done < <(printf '%s\n' "${inputs[@]}" | LC_ALL=C sort -u) -} | "${hash_command[@]}" | awk '{print $1}' diff --git a/.anvil/container/run-in-container.ps1 b/.anvil/container/run-in-container.ps1 deleted file mode 100644 index 281dc2bd..00000000 --- a/.anvil/container/run-in-container.ps1 +++ /dev/null @@ -1,342 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param( - [Parameter(Position = 0, ValueFromRemainingArguments = $true)] - [string[]]$Recipe -) - -$ErrorActionPreference = 'Stop' - -function ConvertTo-AnvilVersion([string]$Value) { - $match = [regex]::Match($Value, '^(\d+)\.(\d+)(?:\.(\d+))?') - if (-not $match.Success) { - throw "anvil-container: could not parse Docker Engine version '$Value'." - } - [version]::new( - [int]$match.Groups[1].Value, - [int]$match.Groups[2].Value, - $(if ($match.Groups[3].Success) { [int]$match.Groups[3].Value } else { 0 }) - ) -} - -function Test-AnvilContainerStringArray([string]$Name, $Value) { - if ($Value -isnot [array]) { - throw "anvil-container: `$$Name must be a string array." - } - foreach ($item in $Value) { - if ($item -isnot [string] -or [string]::IsNullOrEmpty($item)) { - throw "anvil-container: `$$Name entries must be non-empty strings." - } - } -} - -function Test-AnvilContainerBuildArgs($Value) { - for ($index = 0; $index -lt $Value.Count; $index++) { - $item = $Value[$index] - if ($item -eq '--secret') { - $index++ - if ($index -ge $Value.Count) { - throw 'anvil-container: $AnvilContainerBuildArgs requires a value after --secret.' - } - } elseif (-not $item.StartsWith('--secret=', [StringComparison]::Ordinal)) { - throw 'anvil-container: $AnvilContainerBuildArgs accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' - } - } -} - -function Test-AnvilRecipeNeedsGitHubToken([string]$Name) { - $Name -in @( - 'anvil-aprz', - 'anvil-pr', - '_anvil-pr', - 'anvil-pr-fast', - 'anvil-scheduled', - '_anvil-scheduled', - 'anvil-scheduled-advisories', - 'anvil-full', - '_anvil-full' - ) -} - -function Get-AnvilGitHubToken { - $token = $env:GITHUB_TOKEN - if (-not $token -and (Get-Command gh -ErrorAction SilentlyContinue)) { - try { - $token = (& gh auth token --hostname github.com 2>$null) - if ($LASTEXITCODE -ne 0) { $token = $null } - } catch { - $token = $null - } - } - if ($token) { $token = $token.Trim() } - if ($token) { return $token } - return $null -} - -if ($env:ANVIL_IN_CONTAINER) { - if ($Recipe.Count -eq 0) { & bash } else { & just @Recipe } - exit $LASTEXITCODE -} - -foreach ($recipeArg in $Recipe) { - if ($recipeArg -notmatch '^_?anvil-[A-Za-z0-9-]+$') { - throw "anvil-container: expected each argument to be an anvil-* recipe, got '$recipeArg'." - } -} - -if (-not (Get-Command wsl -ErrorAction SilentlyContinue)) { - throw 'anvil-container: WSL 2 is required. See .anvil/container/README.md.' -} - -$versionText = (& wsl -e docker version --format '{{.Server.Version}}' 2>$null) -if ($LASTEXITCODE -ne 0 -or -not $versionText) { - throw 'anvil-container: `wsl -e docker version` must succeed. Install or start Docker Engine in the default WSL distribution; this driver does not invoke Windows docker.exe.' -} -$versionText = $versionText.Trim() -if ((ConvertTo-AnvilVersion $versionText) -lt [version]'23.0.0') { - throw "anvil-container: Docker Engine 23.0.0 or newer is required (found $versionText)." -} -$wslArchitecture = (& wsl -e uname -m 2>$null) -if ($LASTEXITCODE -eq 0 -and $wslArchitecture) { - $wslArchitecture = $wslArchitecture.Trim() - if ($wslArchitecture -notin @('x86_64', 'amd64')) { - [Console]::Error.WriteLine( - "anvil-container: warning: $wslArchitecture requires emulation for linux/amd64; builds and checks may be substantially slower." - ) - } -} - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$scriptDir = Join-Path $repoRoot '.anvil/container' -$wslRepoRoot = (& wsl -e wslpath -a $repoRoot).Trim() -if ($LASTEXITCODE -ne 0 -or -not $wslRepoRoot) { - throw 'anvil-container: could not translate the repository path into the default WSL distribution.' -} -$wslScriptDir = "$wslRepoRoot/.anvil/container" -$containerfile = Join-Path $scriptDir 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$imageId = (& (Join-Path $scriptDir 'image-id.ps1')).Trim() -$imageBase = if ($env:ANVIL_CONTAINER_IMAGE) { $env:ANVIL_CONTAINER_IMAGE } else { 'anvil-dev' } -$image = "${imageBase}:$imageId" -$repoBytes = [Text.Encoding]::UTF8.GetBytes($wslRepoRoot) -$repoHash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($repoBytes)).ToLowerInvariant() -$targetVolume = "anvil-target-$($repoHash.Substring(0, 12))-$($imageId.Substring(0, 12))" - -$needsGitHubToken = $false -foreach ($recipeArg in $Recipe) { - if (Test-AnvilRecipeNeedsGitHubToken $recipeArg) { - $needsGitHubToken = $true - break - } -} -$runsOnlyGitHubCheck = $Recipe.Count -eq 1 -and $Recipe[0] -eq 'anvil-aprz' - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -$null = & wsl -e docker image inspect $image 2>$null -$imageExists = $LASTEXITCODE -eq 0 - -New-Variable -Name AnvilContainerRepoRoot -Value $repoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDir -Value $scriptDir -Option ReadOnly -New-Variable -Name AnvilContainerRepoRootWsl -Value $wslRepoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDirWsl -Value $wslScriptDir -Option ReadOnly -New-Variable -Name AnvilContainerResolvedImage -Value $image -Option ReadOnly -New-Variable -Name AnvilContainerImageExists -Value $imageExists -Option ReadOnly -New-Variable -Name AnvilContainerRequestedRecipes -Value $Recipe -Option ReadOnly -New-Variable -Name AnvilContainerHostIsWindows -Value ([bool]$IsWindows) -Option ReadOnly - -# Customization outputs, initialized before sourcing so a missing customize.ps1 -# leaves every phase a documented no-op. -$AnvilContainerBuildArgs = @() -$AnvilContainerPrepareArgs = @() -$AnvilContainerPrepareCommand = @() -$AnvilContainerRunArgs = @() -$AnvilContainerNeedsGitHubToken = $needsGitHubToken -$AnvilContainerCleanup = $null -$githubToken = $null -$githubTokenFile = $null -$exitCode = 0 -$customizeScript = Join-Path $scriptDir 'customize.ps1' -$legacyCustomizeScript = Join-Path $repoRoot 'justfiles/anvil/container/customize.ps1' - -try { - if (Test-Path -LiteralPath $customizeScript -PathType Leaf) { - . $customizeScript - } - elseif (Test-Path -LiteralPath $legacyCustomizeScript -PathType Leaf) { - [Console]::Error.WriteLine( - 'anvil-container: warning: ignoring justfiles/anvil/container/customize.ps1; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.ps1 to keep it active.' - ) - } - - Test-AnvilContainerStringArray 'AnvilContainerBuildArgs' $AnvilContainerBuildArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareArgs' $AnvilContainerPrepareArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareCommand' $AnvilContainerPrepareCommand - Test-AnvilContainerStringArray 'AnvilContainerRunArgs' $AnvilContainerRunArgs - Test-AnvilContainerBuildArgs $AnvilContainerBuildArgs - if ($AnvilContainerNeedsGitHubToken -isnot [bool]) { - throw 'anvil-container: $AnvilContainerNeedsGitHubToken must be a Boolean.' - } - $needsGitHubToken = $needsGitHubToken -or $AnvilContainerNeedsGitHubToken - if ($AnvilContainerPrepareArgs.Count -gt 0 -and $AnvilContainerPrepareCommand.Count -eq 0) { - throw 'anvil-container: $AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand.' - } - if ($AnvilContainerCleanup -and $AnvilContainerCleanup -isnot [scriptblock]) { - throw 'anvil-container: $AnvilContainerCleanup must be a script block.' - } - $githubToken = if ($needsGitHubToken) { Get-AnvilGitHubToken } else { $null } - if ($needsGitHubToken -and -not $githubToken) { - if (-not (Get-Command gh -ErrorAction SilentlyContinue)) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' - } - if (-not [Environment]::UserInteractive -or [Console]::IsInputRedirected) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' - } - Write-Host 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' - [void](Read-Host 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel)') - $githubToken = Get-AnvilGitHubToken - if (-not $githubToken) { - throw 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' - } - } - if (-not $imageExists) { - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - throw "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." - } - & wsl -e docker build ` - --platform linux/amd64 ` - --tag $image ` - --file "$wslScriptDir/Containerfile" ` - --build-arg "ANVIL_IMAGE_ID=$imageId" ` - --build-arg "BASE_IMAGE=$baseImage" ` - @AnvilContainerBuildArgs ` - $wslRepoRoot - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker build failed with exit code $LASTEXITCODE." - } - } - - $containerUid = (& wsl -e id -u).Trim() - $containerGid = (& wsl -e id -g).Trim() - if ($containerUid -notmatch '^\d+$' -or $containerGid -notmatch '^\d+$') { - throw 'anvil-container: could not determine the default WSL user identity.' - } - $registryVolume = "anvil-cargo-registry-$($repoHash.Substring(0, 12))" - $gitVolume = "anvil-cargo-git-$($repoHash.Substring(0, 12))" - foreach ($volume in @($registryVolume, $gitVolume, $targetVolume)) { - $null = & wsl -e docker volume create $volume - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume creation failed for '$volume' with exit code $LASTEXITCODE." - } - } - $mountArgs = @( - '--mount', "type=bind,source=$wslRepoRoot,target=/workspace", - '--mount', "type=volume,source=$registryVolume,target=/usr/local/cargo/registry", - '--mount', "type=volume,source=$gitVolume,target=/usr/local/cargo/git", - '--mount', "type=volume,source=$targetVolume,target=/workspace/target" - ) - & wsl -e docker run --rm --pull=never ` - --platform linux/amd64 ` - --user 0:0 ` - @mountArgs ` - $image sh -c "chown ${containerUid}:${containerGid} /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume initialization failed with exit code $LASTEXITCODE." - } - - $runArgs = @( - 'run', '--rm', '--pull=never', - '--platform', 'linux/amd64', - '--user', "${containerUid}:${containerGid}", - '--env', 'ANVIL_IN_CONTAINER=1', - '--env', 'HOME=/tmp/anvil-user', - '--workdir', '/workspace' - ) - $runArgs += $mountArgs - $prepareRunArgs = @($runArgs) - $runArgs += $AnvilContainerRunArgs - foreach ($name in @( - 'PR_TITLE', - 'BASE_REF', - 'ANVIL_INCLUDE_MODIFIED', - 'ANVIL_INCLUDE_AFFECTED', - 'ANVIL_INCLUDE_REQUIRED', - 'GITHUB_BASE_REF', - 'SYSTEM_PULLREQUEST_TARGETBRANCH' - )) { - if (Test-Path "Env:$name") { - $runArgs += @('--env', "$name=$((Get-Item "Env:$name").Value)") - } - } - if ($AnvilContainerPrepareCommand.Count -gt 0) { - & wsl -e docker @prepareRunArgs @AnvilContainerPrepareArgs $image @AnvilContainerPrepareCommand - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: preparation command failed with exit code $LASTEXITCODE." - } - } - - if ($githubToken) { - $githubTokenFile = Join-Path ([IO.Path]::GetTempPath()) "anvil-github-token-$PID-$([guid]::NewGuid().ToString('N'))" - [IO.File]::Create($githubTokenFile).Dispose() - if ($IsWindows) { - $userSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value - & icacls.exe $githubTokenFile '/inheritance:r' '/grant:r' "*$($userSid):(F)" | Out-Null - } else { - & chmod 600 $githubTokenFile - } - if ($LASTEXITCODE -ne 0) { - throw 'anvil-container: failed to restrict permissions on the temporary GitHub token file.' - } - [IO.File]::WriteAllText($githubTokenFile, $githubToken, [Text.Encoding]::ASCII) - $githubToken = $null - $wslTokenFile = (& wsl -e wslpath -a $githubTokenFile).Trim() - if ($LASTEXITCODE -ne 0 -or -not $wslTokenFile) { - throw 'anvil-container: could not translate the temporary GitHub token path into WSL.' - } - $githubRunArgs = @($runArgs) - $githubRunArgs += @( - '--mount', - "type=bind,source=$wslTokenFile,target=/run/secrets/anvil-github-token,readonly" - ) - if ($runsOnlyGitHubCheck) { - $runArgs = $githubRunArgs - } else { - & wsl -e docker @githubRunArgs $image just anvil-aprz - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: isolated anvil-aprz failed with exit code $LASTEXITCODE." - } - $runArgs += @('--env', 'ANVIL_APRZ_ALREADY_RAN=1') - } - } - - if ($Recipe.Count -eq 0) { - & wsl -e docker @runArgs --interactive --tty $image bash - } else { - & wsl -e docker @runArgs $image just @Recipe - } - $exitCode = $LASTEXITCODE -} finally { - if ($githubTokenFile) { - Remove-Item -LiteralPath $githubTokenFile -Force -ErrorAction SilentlyContinue - } - if ($AnvilContainerCleanup) { & $AnvilContainerCleanup } -} - -exit $exitCode diff --git a/.anvil/container/run-in-container.sh b/.anvil/container/run-in-container.sh deleted file mode 100644 index 65b0513d..00000000 --- a/.anvil/container/run-in-container.sh +++ /dev/null @@ -1,324 +0,0 @@ -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - if (($# == 0)); then exec bash; else exec just "$@"; fi -fi - -for recipe_arg in "$@"; do - if [[ ! "$recipe_arg" =~ ^_?anvil-[A-Za-z0-9-]+$ ]]; then - echo "anvil-container: expected each argument to be an anvil-* recipe, got '$recipe_arg'." >&2 - exit 2 - fi -done - -anvil_recipe_needs_github_token() { - case "$1" in - anvil-aprz | anvil-pr | _anvil-pr | anvil-pr-fast \ - | anvil-scheduled | _anvil-scheduled | anvil-scheduled-advisories \ - | anvil-full | _anvil-full) return 0 ;; - *) return 1 ;; - esac -} - -version_at_least() { - local found="${1%%[-+]*}" - local required="${2%%[-+]*}" - local found_major found_minor found_patch found_extra - local required_major required_minor required_patch required_extra - IFS=. read -r found_major found_minor found_patch found_extra <<<"$found" - IFS=. read -r required_major required_minor required_patch required_extra <<<"$required" - found_patch="${found_patch:-0}" - required_patch="${required_patch:-0}" - for component in \ - "$found_major" "$found_minor" "$found_patch" \ - "$required_major" "$required_minor" "$required_patch" - do - case "$component" in - '' | *[!0-9]*) return 2 ;; - esac - done - if ((found_major != required_major)); then ((found_major > required_major)); return; fi - if ((found_minor != required_minor)); then ((found_minor > required_minor)); return; fi - ((found_patch >= required_patch)) -} - -command -v docker >/dev/null 2>&1 || { - echo "anvil-container: Docker Engine is required. See .anvil/container/README.md." >&2 - exit 1 -} - -version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || { - echo "anvil-container: Docker Engine is unavailable. Start the Docker service and ensure the current user can access it." >&2 - exit 1 -} -minimum="23.0.0" -if ! version_at_least "$version" "$minimum"; then - echo "anvil-container: Docker Engine $minimum or newer is required (found $version)." >&2 - exit 1 -fi -host_arch="$(uname -m 2>/dev/null || true)" -case "$host_arch" in - x86_64 | amd64 | '') ;; - *) echo "anvil-container: warning: $host_arch requires emulation for linux/amd64; builds and checks may be substantially slower." >&2 ;; -esac - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi -script_dir="$repo_root/.anvil/container" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$script_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -image_id="$(bash "$script_dir/image-id.sh")" -image_base="${ANVIL_CONTAINER_IMAGE:-anvil-dev}" -image="${image_base}:${image_id}" -if command -v sha256sum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | sha256sum | cut -c1-12)" -elif command -v shasum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | shasum -a 256 | cut -c1-12)" -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi -target_volume="anvil-target-${repo_id}-${image_id:0:12}" - -needs_github_token=false -for recipe_arg in "$@"; do - if anvil_recipe_needs_github_token "$recipe_arg"; then - needs_github_token=true - break - fi -done -runs_only_github_check=false -if (($# == 1)) && [[ "$1" == "anvil-aprz" ]]; then - runs_only_github_check=true -fi -github_token="" - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -if docker image inspect "$image" >/dev/null 2>&1; then - image_exists=true -else - image_exists=false -fi - -readonly ANVIL_CONTAINER_REPO_ROOT="$repo_root" -readonly ANVIL_CONTAINER_DIR="$script_dir" -readonly ANVIL_CONTAINER_RESOLVED_IMAGE="$image" -readonly ANVIL_CONTAINER_IMAGE_EXISTS="$image_exists" -declare -a ANVIL_CONTAINER_REQUESTED_RECIPES=("$@") -readonly ANVIL_CONTAINER_REQUESTED_RECIPES - -# Customization outputs, initialized before sourcing so a missing customize.sh -# leaves every phase a documented no-op. -ANVIL_CONTAINER_BUILD_ARGS=() -ANVIL_CONTAINER_PREPARE_ARGS=() -ANVIL_CONTAINER_PREPARE_COMMAND=() -ANVIL_CONTAINER_RUN_ARGS=() -ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN="$needs_github_token" -ANVIL_CONTAINER_CLEANUP=: -github_token_file="" -cleanup() { - if [[ -n "$github_token_file" ]]; then rm -f -- "$github_token_file"; fi - "$ANVIL_CONTAINER_CLEANUP" -} -trap cleanup EXIT - -customize_script="$script_dir/customize.sh" -legacy_customize_script="$repo_root/justfiles/anvil/container/customize.sh" -if [[ ! -f "$customize_script" && -f "$legacy_customize_script" ]]; then - echo "anvil-container: warning: ignoring justfiles/anvil/container/customize.sh; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.sh to keep it active." >&2 -fi -if [[ -f "$customize_script" ]]; then - # shellcheck source=/dev/null - source "$customize_script" -fi - -# Bash 3.2 has neither namerefs (the nameref flag on `local`/`declare`, Bash -# 4.3+) nor safe `set -u` expansion of empty-but- -# declared arrays (fixed in Bash 4.4). Elements are passed positionally -# instead of by nameref, and every expansion of a possibly-empty array uses -# the `${arr[@]+"${arr[@]}"}` idiom: unset/empty-under-old-Bash arrays vanish -# entirely instead of raising "unbound variable", while non-empty arrays -# still expand element-for-element. -anvil_container_validate_array() { - local name="$1" - shift - local declaration value - declaration="$(declare -p "$name" 2>/dev/null || true)" - if [[ ! "$declaration" =~ ^declare\ -[^[:space:]]*a[^[:space:]]*\ ]]; then - echo "anvil-container: $name must be a string array." >&2 - exit 1 - fi - for value in "$@"; do - if [[ -z "$value" ]]; then - echo "anvil-container: $name entries must be non-empty strings." >&2 - exit 1 - fi - done -} -anvil_container_validate_build_args() { - local expect_secret_value=false value - for value in "$@"; do - if "$expect_secret_value"; then - expect_secret_value=false - continue - fi - case "$value" in - --secret) expect_secret_value=true ;; - --secret=*) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' >&2 - exit 1 - ;; - esac - done - if "$expect_secret_value"; then - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS requires a value after --secret.' >&2 - exit 1 - fi -} -anvil_container_validate_array ANVIL_CONTAINER_BUILD_ARGS ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_ARGS ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_COMMAND ${ANVIL_CONTAINER_PREPARE_COMMAND[@]+"${ANVIL_CONTAINER_PREPARE_COMMAND[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_RUN_ARGS ${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"} -anvil_container_validate_build_args ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -case "$ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN" in - true) needs_github_token=true ;; - false) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN must be true or false.' >&2 - exit 1 - ;; -esac -if ((${#ANVIL_CONTAINER_PREPARE_ARGS[@]} > 0)) && ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} == 0)); then - echo 'anvil-container: ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND.' >&2 - exit 1 -fi -cleanup_kind="$(type -t "$ANVIL_CONTAINER_CLEANUP" 2>/dev/null || true)" -if [[ "$cleanup_kind" != "function" && "$cleanup_kind" != "builtin" ]]; then - echo "anvil-container: ANVIL_CONTAINER_CLEANUP must name a callable function (got '$ANVIL_CONTAINER_CLEANUP')." >&2 - exit 1 -fi - -if "$needs_github_token"; then - gh_command="" - if command -v gh >/dev/null 2>&1; then - gh_command=gh - elif command -v gh.exe >/dev/null 2>&1; then - gh_command=gh.exe - fi - github_token="${GITHUB_TOKEN:-}" - if [[ -z "$github_token" && -n "$gh_command" ]]; then - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - fi - if [[ -z "$github_token" ]]; then - if [[ -z "$gh_command" ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - if [[ ! -t 0 ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - echo 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' >&2 - read -r -p 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel) ' - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - if [[ -z "$github_token" ]]; then - echo 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' >&2 - exit 1 - fi - fi -fi - -if ! "$image_exists"; then - if [[ "${ANVIL_CONTAINER_NO_REBUILD:-}" == "1" ]]; then - echo "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." >&2 - exit 1 - else - docker build \ - --platform linux/amd64 \ - --tag "$image" \ - --file "$script_dir/Containerfile" \ - --build-arg "ANVIL_IMAGE_ID=$image_id" \ - --build-arg "BASE_IMAGE=$base_image" \ - ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} \ - "$repo_root" - fi -fi - -container_uid="$(id -u)" -container_gid="$(id -g)" -registry_volume="anvil-cargo-registry-${repo_id}" -git_volume="anvil-cargo-git-${repo_id}" -for volume in "$registry_volume" "$git_volume" "$target_volume"; do - docker volume create "$volume" >/dev/null -done -mount_args=( - --mount "type=bind,source=$repo_root,target=/workspace" - --mount "type=volume,source=$registry_volume,target=/usr/local/cargo/registry" - --mount "type=volume,source=$git_volume,target=/usr/local/cargo/git" - --mount "type=volume,source=$target_volume,target=/workspace/target" -) -docker run --rm --pull=never \ - --platform linux/amd64 \ - --user 0:0 \ - "${mount_args[@]}" \ - "$image" sh -c \ - "chown $container_uid:$container_gid /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - -run_args=( - run --rm --pull=never - --platform linux/amd64 - --user "$container_uid:$container_gid" - --env ANVIL_IN_CONTAINER=1 - --env HOME=/tmp/anvil-user - "${mount_args[@]}" - --workdir /workspace -) -prepare_run_args=("${run_args[@]}") -run_args+=(${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"}) -for name in PR_TITLE BASE_REF ANVIL_INCLUDE_MODIFIED ANVIL_INCLUDE_AFFECTED ANVIL_INCLUDE_REQUIRED GITHUB_BASE_REF SYSTEM_PULLREQUEST_TARGETBRANCH; do - if value="$(printenv "$name")"; then run_args+=(--env "$name=$value"); fi -done -if ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} > 0)); then - docker "${prepare_run_args[@]}" \ - ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} \ - "$image" \ - "${ANVIL_CONTAINER_PREPARE_COMMAND[@]}" -fi - -if [[ -n "$github_token" ]]; then - github_token_file="$(mktemp "${TMPDIR:-/tmp}/anvil-github-token.XXXXXXXX")" - chmod 600 "$github_token_file" - printf '%s' "$github_token" > "$github_token_file" - unset github_token - github_run_args=( - "${run_args[@]}" - --mount "type=bind,source=$github_token_file,target=/run/secrets/anvil-github-token,readonly" - ) - if "$runs_only_github_check"; then - run_args=("${github_run_args[@]}") - else - docker "${github_run_args[@]}" "$image" just anvil-aprz - run_args+=(--env ANVIL_APRZ_ALREADY_RAN=1) - fi -fi - -if (($# == 0)); then - docker "${run_args[@]}" --interactive --tty "$image" bash - exit $? -fi -docker "${run_args[@]}" "$image" just "$@" diff --git a/.spelling b/.spelling index 46d24a62..34996003 100644 --- a/.spelling +++ b/.spelling @@ -463,5 +463,11 @@ unscoped backtracker prerelease versioned -Containerfile +BuildKit +Dockerfile +dockerignore +Podman +podman +toolset +ARM64 WSL diff --git a/Justfile b/Justfile index b666391e..d906b5b7 100644 --- a/Justfile +++ b/Justfile @@ -26,7 +26,3 @@ import 'justfiles/spelling.just' # >>> anvil-managed: anvil-imports import 'justfiles/anvil/mod.just' # <<< anvil-managed: anvil-imports - -# >>> anvil-managed: anvil-runner -anvil_runner := env_var_or_default("ANVIL_RUNNER", "native") -# <<< anvil-managed: anvil-runner diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index aa364912..e266499e 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -97,118 +97,80 @@ and run each check only over the affected packages, whereas a local ### Containerized local checks -Anvil can run any generated recipe in a content-addressed Linux container. -The image installs the Rust toolchains and Cargo tools pinned by the -repository’s generated Anvil configuration, providing a repeatable Linux -environment without installing those tools directly on the host. +Any generated recipe can run in a content-addressed Linux container. The +image installs the Rust toolchain and Cargo tools that this repository +pins, by running `just anvil-setup` — the same recipe the checks use — so +the container and the host agree on the toolset by construction. -#### Prerequisites - -* Docker Engine 23.0 or newer, installed directly in Linux or WSL and - usable by the current user. -* `git` and `just` on the host. -* Bash on Linux and WSL; `PowerShell` Core (`pwsh`) and WSL 2 on Windows. -* `[script]` support enabled in the root `Justfile` (`set unstable` when - required by the installed `just` version). -* A repository-owned `rust-toolchain.toml`. -* On Windows, Docker Engine running in the default WSL distribution: +There is no configuration file and no transparent routing: `just anvil-pr` +keeps running natively, and the container is reached only through the +explicit recipe. -```powershell -wsl -e docker version +```text +just anvil-container anvil-clippy # one check +just anvil-container anvil-pr # the whole PR tier +just anvil-container # interactive shell ``` -The Windows driver invokes Docker in the default WSL distribution and does -not call Windows `docker.exe`. Regardless of the installation, the command -above must succeed. Docker Desktop is not required. +#### Prerequisites -On ARM64 hosts, Docker emulates the required `linux/amd64` environment, so -image builds and checks can be substantially slower than on x86-64 hosts. +* A container engine callable from the shell that runs `just`: Docker + (supported) or Podman (best-effort). On Windows that means Docker + Desktop, Podman, or a Windows `docker` CLI pointed at an engine in WSL. +* `just` and `PowerShell` Core (`pwsh`) on the host. +* A repository-owned `rust-toolchain.toml`. -#### Run a recipe +On ARM64 hosts the image is emulated as `linux/amd64`, so builds and checks +are substantially slower. -```text -just anvil-container anvil-clippy -just anvil-container anvil-pr -just anvil-container -``` +#### Image identity -The no-argument form opens an interactive shell. Anvil builds an image the -first time it encounters a content hash and reuses it on later runs. Changes -to the Rust toolchain, generated Anvil files, Containerfile, or other static -image inputs select a new tag and build a new image. Images for earlier -hashes remain available to older branches. Runtime `customize.*` files do not -affect image identity. -Every argument is a recipe name; recipe parameters are not supported by -this command surface. +The tag *is* a SHA-256 over the inputs that define the image: the +Dockerfile and its ignore file, `rust-toolchain.toml`, the optional +credential hook, and the generated recipe tree. Presence therefore implies +freshness — a changed tool pin names a tag that cannot already exist, so a +build follows. There is no staleness check because there is nothing to +check. -Cargo registry and Cargo Git caches use repository-scoped named volumes; -`target/` is additionally scoped by image ID. The -repository is mounted at `/workspace`; keeping build output in a named -volume avoids slow host bind-mount I/O, particularly on Windows. +#### Controls -#### Make tiers use the container +|Variable|Effect| +|--------|------| +|`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`. A host property; never committed.| +|`ANVIL_CONTAINER_NO_REBUILD=1`|Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure.| +|`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves.| +|`ANVIL_IN_CONTAINER=1`|Set inside the image; makes a nested invocation run natively.| -Native execution remains the default. Enable container execution for the -current shell: +Supporting recipes: `anvil-container-status`, `anvil-container-rebuild`, +and `anvil-container-down` (removes this repository’s cache volumes). -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` +#### Credentials -On Unix: +crates.io needs none, so the public catalog emits no credential plumbing. +A repository or a downstream catalog that needs one adds +`.anvil/container/hooks.ps1`, which the recipe loads when present: -```sh -ANVIL_RUNNER=container just anvil-pr +```powershell +function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } +function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } ``` -A one-off override is also supported: - -```text -just anvil_runner=container anvil-pr -``` +Build secrets are passed to `BuildKit` by environment variable name, so a +value never reaches a process argument and never reaches an image layer; +run-time values are forwarded into the container by name for the same +reason. An empty value is a hard error, because a build that quietly +proceeded without its credential would install a reduced tool set and then +be tagged with the hash a credentialed build produces. -To make containers the project default, edit `/Justfile` -and change the default value in the `anvil-runner` region from `"native"` -to `"container"`. Commit `/Justfile` with that policy -change. Set `ANVIL_RUNNER=native` to override it for one shell. +The hook runs on the host with the developer’s permissions, before any +container isolation. Only run one from a repository or catalog you trust. -#### Controls +#### Customizing the image -|Variable|Effect| -|--------|------| -|`ANVIL_CONTAINER_BASE_IMAGE`|Select a compatible digest-pinned Linux base image; the value is included in the content hash.| -|`ANVIL_CONTAINER_IMAGE`|Override the local image name. The content hash remains the tag.| -|`ANVIL_CONTAINER_NO_REBUILD=1`|Fail when the matching image is missing instead of building it.| - -The public driver never pulls `ANVIL_CONTAINER_IMAGE` remotely. Repositories -and derived catalogs can add trusted `customize.sh` and -`customize.ps1` files for image-build secrets, dependency preparation, -APRZ classification, runtime arguments, and cleanup through the documented -customization contract without changing the public command surface. - -Customization files execute on the host with the developer’s permissions -before container isolation. Only run them from a repository or catalog you -trust. - -For GitHub API checks, the driver automatically uses an existing host -`GITHUB_TOKEN` or the token from an authenticated host `gh` CLI session. It -mounts the token read-only for the command and removes the temporary file -afterward. If `gh` is installed but not authenticated, an interactive run -pauses before building the image, explains the unauthenticated API limit, -and continues after the user completes `gh auth login` and presses Enter. - -#### Troubleshooting - -* A first-run image build is expected and may take several minutes. -* `wsl -e docker images anvil-dev` lists locally cached Anvil images from - Windows; use `docker images anvil-dev` inside Linux or WSL. -* `ANVIL_CONTAINER_NO_REBUILD=1` distinguishes a cache miss from a build - failure. -* Non-interactive runs cannot pause for login. Authenticate `gh` or set host - `GITHUB_TOKEN` before starting them. -* Regenerate managed files with `cargo anvil`; do not hand-edit - `.anvil/container/`. +`.anvil/container/Dockerfile` is an ordinary owned file: edit it in place +and anvil’s drift handling preserves it. A downstream catalog that targets +a different base OS or toolchain source replaces the artifact instead — see +[`artifacts::container::dockerfile`][__link1]. ### Checks and tiers @@ -301,7 +263,7 @@ own crates — without editing the generated `justfiles/anvil/` tree. #### Spelling dictionary (`spellcheck`) -The `spellcheck` check ([`cargo-spellcheck`][__link1]) +The `spellcheck` check ([`cargo-spellcheck`][__link2]) reads a repo-root `.spelling` file — one word per line — as its custom dictionary. Add project-specific terms (crate names, acronyms, identifiers) there to silence false positives; the `anvil-spellcheck` @@ -310,7 +272,7 @@ consumes. Keep the file `LF`-terminated. #### Coverage (`llvm-cov`) -Coverage is gated by [`cargo-coverage-gate`][__link2]; +Coverage is gated by [`cargo-coverage-gate`][__link3]; per-package and per-workspace thresholds, the coverage-exclusion attribute, and opt-out are all configured through its `Cargo.toml` metadata conventions — see its documentation. @@ -377,8 +339,8 @@ fn main() -> ExitCode { } ``` -…plus a [`Catalog`][__link3] value that starts from [`Catalog::anvil`][__link4] and -customizes the CLI identity ([`CliMeta`][__link5]) and artifact set: +…plus a [`Catalog`][__link4] value that starts from [`Catalog::anvil`][__link5] and +customizes the CLI identity ([`CliMeta`][__link6]) and artifact set: ```rust use cargo_anvil::{Artifact, Catalog, artifacts}; @@ -402,8 +364,8 @@ The on-disk vocabulary (`.anvil.lock`, `anvil-managed` sentinels, `justfiles/anvil/`, `anvil-` recipes) is the fixed engine format and is never rebranded. A fork customizes only its CLI identity and which artifacts it emits, via the three uniform builder verbs -([`CatalogBuilder::with_artifact`][__link6], [`CatalogBuilder::replace_artifact`][__link7], -[`CatalogBuilder::without_artifact`][__link8]) over the public [`artifacts`][__link9] +([`CatalogBuilder::with_artifact`][__link7], [`CatalogBuilder::replace_artifact`][__link8], +[`CatalogBuilder::without_artifact`][__link9]) over the public [`artifacts`][__link10] registry. The `tool` field recorded in `.anvil.lock` keeps two anvil-family tools from clobbering one another in a shared repo (see `--force`). See `docs/design/extensibility.md`. @@ -428,14 +390,15 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGkYW0CYXSEGxYc2fK81jTWG7kWg0hlspxYGx-DzHaE-xjXG1cDT7T4wIbxYXKEG7VbetTfxlJTGxvRM3l3qEetG_F0Ae5JDGjCG4jajt1KgnpKYWSBg2tjYXJnby1hbnZpbGUwLjQuMGtjYXJnb19hbnZpbA + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQblC_Wpqk8Z8wbXSVrNQ_nt0AbkrIITLnNqNIbWqbEpmD_DbhhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta - [__link1]: https://crates.io/crates/cargo-spellcheck - [__link2]: https://crates.io/crates/cargo-coverage-gate - [__link3]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=Catalog - [__link4]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=Catalog::anvil - [__link5]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CliMeta - [__link6]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::with_artifact - [__link7]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::replace_artifact - [__link8]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::without_artifact - [__link9]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts + [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container::dockerfile + [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts + [__link2]: https://crates.io/crates/cargo-spellcheck + [__link3]: https://crates.io/crates/cargo-coverage-gate + [__link4]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=Catalog + [__link5]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=Catalog::anvil + [__link6]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CliMeta + [__link7]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::with_artifact + [__link8]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::replace_artifact + [__link9]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=CatalogBuilder::without_artifact diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 1ac32f97..0fefa700 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -1,845 +1,217 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. -//! The optional local container backend. +//! Containerized execution: the `anvil-container` recipe and the image it runs. //! -//! The base catalog emits a public Docker Engine implementation. Downstream catalogs -//! replace only environment-specific artifacts such as the Containerfile and -//! add an optional `customize.sh`/`customize.ps1` runtime customization file. +//! Two artifacts define the whole feature. The recipe drives the engine and +//! computes the image identity; the Dockerfile (with its build-context ignore +//! file) defines what the image contains. There is no configuration file: +//! whether the group is emitted at all is a catalog decision, and the only +//! host-specific value — which engine to call — is an environment variable read +//! by the recipe at run time. +//! +//! A downstream catalog customizes exactly two things, and inherits everything +//! else: +//! +//! - [`dockerfile`] plus [`Artifact::with_body`] to build on a different base +//! or install the toolchain from a different source. +//! - [`hooks`] to supply credentials, which the recipe loads when the file is +//! present regardless of who put it there. use crate::catalog::Artifact; const RECIPE: &str = include_str!("../../../templates/justfiles/anvil/container.just"); -const CONTAINERFILE: &str = include_str!("../../../templates/anvil/container/Containerfile"); -const IGNORE: &str = include_str!("../../../templates/anvil/container/Containerfile.dockerignore"); -const ENTRYPOINT: &str = include_str!("../../../templates/anvil/container/entrypoint.sh"); -const IMAGE_ID: &str = include_str!("../../../templates/anvil/container/image-id.ps1"); -const SHELL_IMAGE_ID: &str = include_str!("../../../templates/anvil/container/image-id.sh"); -const SHELL_DRIVER: &str = include_str!("../../../templates/anvil/container/run-in-container.sh"); -const POWERSHELL_DRIVER: &str = include_str!("../../../templates/anvil/container/run-in-container.ps1"); -const README: &str = include_str!("../../../templates/anvil/container/README.md"); +const DOCKERFILE: &str = include_str!("../../../templates/container/Dockerfile"); +const DOCKERIGNORE: &str = include_str!("../../../templates/container/Dockerfile.dockerignore"); const RECIPE_PATH: &str = "justfiles/anvil/container.just"; -const CONTAINERFILE_PATH: &str = ".anvil/container/Containerfile"; -const IGNORE_PATH: &str = ".anvil/container/Containerfile.dockerignore"; -const ENTRYPOINT_PATH: &str = ".anvil/container/entrypoint.sh"; -const IMAGE_ID_PATH: &str = ".anvil/container/image-id.ps1"; -const SHELL_IMAGE_ID_PATH: &str = ".anvil/container/image-id.sh"; -const SHELL_DRIVER_PATH: &str = ".anvil/container/run-in-container.sh"; -const POWERSHELL_DRIVER_PATH: &str = ".anvil/container/run-in-container.ps1"; -const README_PATH: &str = ".anvil/container/README.md"; -const CUSTOMIZE_SHELL_PATH: &str = ".anvil/container/customize.sh"; -const CUSTOMIZE_POWERSHELL_PATH: &str = ".anvil/container/customize.ps1"; +const DOCKERFILE_PATH: &str = ".anvil/container/Dockerfile"; +const DOCKERIGNORE_PATH: &str = ".anvil/container/Dockerfile.dockerignore"; -/// The full public container artifact group. +/// The path the recipe loads credentials from, when a file is present there. +pub const HOOKS_PATH: &str = ".anvil/container/hooks.ps1"; + +/// The full container artifact group. #[must_use] pub fn all() -> Vec { - vec![ - recipe(), - containerfile(), - ignore_file(), - entrypoint(), - image_id(), - shell_image_id(), - shell_driver(), - powershell_driver(), - readme(), - ] + vec![recipe(), dockerfile(), dockerignore()] } -/// The explicit `anvil-container` recipe. +/// The `anvil-container` recipe and its private helpers. #[must_use] pub fn recipe() -> Artifact { Artifact::owned_file(RECIPE_PATH, RECIPE) } -/// The public rustup/crates.io Containerfile. -#[must_use] -pub fn containerfile() -> Artifact { - Artifact::owned_file(CONTAINERFILE_PATH, CONTAINERFILE) -} - -/// The restricted Docker build-context ignore file. -#[must_use] -pub fn ignore_file() -> Artifact { - Artifact::owned_file(IGNORE_PATH, IGNORE) -} - -/// The generic non-root Cargo metadata entry point. -#[must_use] -pub fn entrypoint() -> Artifact { - Artifact::owned_file(ENTRYPOINT_PATH, ENTRYPOINT) -} - -/// The cross-platform content-addressed image-id helper. -#[must_use] -pub fn image_id() -> Artifact { - Artifact::owned_file(IMAGE_ID_PATH, IMAGE_ID) -} - -/// The Bash content-addressed image-id helper. -#[must_use] -pub fn shell_image_id() -> Artifact { - Artifact::owned_file(SHELL_IMAGE_ID_PATH, SHELL_IMAGE_ID) -} - -/// The Linux/WSL Docker Engine driver. -#[must_use] -pub fn shell_driver() -> Artifact { - Artifact::owned_file(SHELL_DRIVER_PATH, SHELL_DRIVER) -} - -/// The Windows-to-WSL Docker Engine driver. -#[must_use] -pub fn powershell_driver() -> Artifact { - Artifact::owned_file(POWERSHELL_DRIVER_PATH, POWERSHELL_DRIVER) -} - -/// User-facing prerequisites and troubleshooting. +/// The default execution image: a digest-pinned Debian base that installs the +/// pinned toolchain and the generated tool catalog by running `just anvil-setup`. +/// +/// A downstream catalog that needs a different base OS or toolchain source +/// replaces the body wholesale: +/// +/// ```ignore +/// catalog.replace_artifact( +/// artifacts::container::dockerfile().with_body(include_str!("../templates/Dockerfile")), +/// ) +/// ``` #[must_use] -pub fn readme() -> Artifact { - Artifact::owned_file(README_PATH, README) +pub fn dockerfile() -> Artifact { + Artifact::owned_file(DOCKERFILE_PATH, DOCKERFILE) } -/// Add a downstream shell customization file (`customize.sh`). +/// The build-context ignore file for [`dockerfile`]. /// -/// The public catalog does not emit this file. A regular repository can add -/// the standard path directly; a derived distribution can package the same -/// file through this constructor. The driver loads it whenever present, -/// regardless of provenance. See -/// [the container customization contract](../../../docs/design/containers.md) -/// for the runtime interface. +/// `BuildKit` reads `.dockerignore` in preference to a root +/// `.dockerignore`, so the build context is scoped without the repository +/// having to own a root ignore file. A catalog that replaces the Dockerfile +/// with one that copies more of the tree must replace this too. #[must_use] -pub fn customize_shell(body: impl Into) -> Artifact { - Artifact::owned_file(CUSTOMIZE_SHELL_PATH, body) +pub fn dockerignore() -> Artifact { + Artifact::owned_file(DOCKERIGNORE_PATH, DOCKERIGNORE) } -/// Add a downstream `PowerShell` customization file (`customize.ps1`). +/// Add a credential hook at [`HOOKS_PATH`]. +/// +/// The public catalog emits no hook: crates.io needs no credentials, and an +/// empty script would be one more generated file to review. A downstream +/// catalog adds one with [`crate::CatalogBuilder::with_artifact`]; a single +/// repository can write the same path by hand. The recipe loads it either way. /// -/// See [`customize_shell`] for the shared contract and provenance-neutral -/// loading behavior. +/// The script may define either or both of two functions, and is dot-sourced +/// before the phase that needs it: +/// +/// - `Anvil-PreBuild` returns `@{ Secrets = @{ = } }`. Each entry +/// becomes a `BuildKit` `--secret id=`, passed by environment variable +/// name so the value never reaches a process argument, and never a layer. +/// - `Anvil-PreRun` returns `@{ Env = @{ = } }`. Each entry is +/// forwarded into the container by name, for the same reason. +/// +/// An empty value from either function is a hard error: a build that silently +/// proceeds without its credential would install a reduced tool set and then be +/// tagged with the same content hash a credentialed build produces, so every +/// later run would reuse the broken image. +/// +/// The file's *content* is part of the image identity, since it decides what the +/// build installs. Its *output* deliberately is not: a credential must never +/// influence a tag. #[must_use] -pub fn customize_powershell(body: impl Into) -> Artifact { - Artifact::owned_file(CUSTOMIZE_POWERSHELL_PATH, body) +pub fn hooks(body: impl Into) -> Artifact { + Artifact::owned_file(HOOKS_PATH, body) } #[cfg(test)] #[cfg_attr(coverage_nightly, coverage(off))] mod tests { - use std::collections::{BTreeMap, BTreeSet}; - use std::path::Path; - use std::process::Command; - - use tempfile::TempDir; - use super::*; - use crate::anvil::artifacts::justfile::dependency_recipe_sources; - - fn write(path: &Path, body: impl AsRef<[u8]>) { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).expect("test path parent must be creatable"); - } - std::fs::write(path, body).expect("test file must be writable"); - } - - fn reaches_aprz(recipe: &str, graph: &BTreeMap>, visiting: &mut BTreeSet) -> bool { - if recipe == "anvil-aprz" { - return true; - } - if !visiting.insert(recipe.to_owned()) { - return false; - } - let reaches = graph - .get(recipe) - .is_some_and(|dependencies| dependencies.iter().any(|dependency| reaches_aprz(dependency, graph, visiting))); - visiting.remove(recipe); - reaches - } - - fn run_image_id_command(repo: &Path, command: &str, args: &[&str]) -> String { - run_image_id_command_with_base(repo, command, args, None) - } - fn run_image_id_command_with_base(repo: &Path, command: &str, args: &[&str], base_image: Option<&str>) -> String { - let mut command = Command::new(command); - command.args(args).current_dir(repo).env_remove("ANVIL_CONTAINER_BASE_IMAGE"); - if let Some(base_image) = base_image { - command.env("ANVIL_CONTAINER_BASE_IMAGE", base_image); - } - let output = command - .output() - .expect("native shell must be available for the container image-id helper"); - assert!( - output.status.success(), - "image-id helper failed: {}", - String::from_utf8_lossy(&output.stderr) - ); - String::from_utf8(output.stdout).expect("image ID must be UTF-8").trim().to_owned() - } - - #[cfg(windows)] - fn run_image_id(repo: &Path) -> String { - run_image_id_command(repo, "pwsh", &["-NoProfile", "-File", ".anvil/container/image-id.ps1"]) - } - - #[cfg(unix)] - fn run_image_id(repo: &Path) -> String { - run_image_id_command(repo, "bash", &[".anvil/container/image-id.sh"]) - } - - fn write_image_id_fixture(root: &Path) { - write(&root.join("rust-toolchain.toml"), "channel = \"1.93\"\n"); - write(&root.join("justfiles/anvil/versions.just"), "tool_version := \"1\"\n"); - write( - &root.join(CONTAINERFILE_PATH), - "ARG BASE_IMAGE=example.invalid/base@sha256:0000000000000000000000000000000000000000000000000000000000000000\nFROM ${BASE_IMAGE}\n", - ); - write(&root.join(IMAGE_ID_PATH), IMAGE_ID); - write(&root.join(SHELL_IMAGE_ID_PATH), SHELL_IMAGE_ID); - } - - #[test] - fn public_group_has_the_expected_files() { - let paths: Vec<&str> = all() + fn paths(artifacts: &[Artifact]) -> Vec<&str> { + artifacts .iter() .map(|artifact| match artifact { Artifact::OwnedFile(spec) => spec.path, Artifact::Region(_) => panic!("container group must contain owned files only"), }) - .collect(); - assert_eq!( - paths, - [ - RECIPE_PATH, - CONTAINERFILE_PATH, - IGNORE_PATH, - ENTRYPOINT_PATH, - IMAGE_ID_PATH, - SHELL_IMAGE_ID_PATH, - SHELL_DRIVER_PATH, - POWERSHELL_DRIVER_PATH, - README_PATH - ] - ); + .collect() } #[test] - fn containerfile_installs_the_generated_toolset() { - assert!(CONTAINERFILE.contains("just anvil-setup")); - assert!(CONTAINERFILE.contains("COPY . .")); - assert!(IGNORE.contains("!.anvil/container/*")); - assert!(IGNORE.contains("!justfiles/anvil/checks/*.just")); - assert!(CONTAINERFILE.contains("anvil_runner := \\\"native\\\"")); - assert!(CONTAINERFILE.contains("requires rust-toolchain.toml")); - assert!(CONTAINERFILE.contains("anvil-container-entrypoint")); - } - - /// The Docker build-context ignore evaluation, ported from - /// `MatchesOrParentMatches` in `moby/patternmatcher`: patterns apply in - /// order and the last match wins, an `!` pattern applies only while the - /// candidate is ignored (and a plain pattern only while it is not), and - /// every pattern is tested against the candidate path *and each of its - /// parent directories*. Blank and `#` lines are dropped, as - /// `ignorefile::ReadAll` drops them. - /// - /// Only the pattern vocabulary the template actually uses is modeled; - /// anything else panics rather than silently matching differently from - /// Docker. - struct DockerIgnore { - patterns: Vec<(bool, Vec)>, - } - - impl DockerIgnore { - fn parse(text: &str) -> Self { - let mut patterns = Vec::new(); - for line in text.lines() { - let line = line.trim(); - if line.is_empty() || line.starts_with('#') { - continue; - } - let (exclusion, body) = line.strip_prefix('!').map_or((false, line), |rest| (true, rest)); - assert!(!body.is_empty(), "illegal exclusion pattern: \"!\""); - let segments: Vec = body.split('/').map(str::to_owned).collect(); - for segment in &segments { - assert!( - !segment.contains("**") || (segment == "**" && segments.len() == 1), - "only a bare `**` is modeled; `{body}` needs Docker's full regex translation" - ); - assert!( - !segment.contains(['[', ']', '\\']), - "character classes and escapes are not modeled: {body}" - ); - } - patterns.push((exclusion, segments)); - } - Self { patterns } - } - - /// Glob one path segment. `*` and `?` never cross a separator, which - /// is already guaranteed because the caller splits on `/`. - fn segment_matches(pattern: &str, segment: &str) -> bool { - let pattern: Vec = pattern.chars().collect(); - let segment: Vec = segment.chars().collect(); - let (mut p, mut s) = (0, 0); - let (mut star, mut retry) = (None, 0); - while s < segment.len() { - if p < pattern.len() && (pattern[p] == '?' || pattern[p] == segment[s]) { - p += 1; - s += 1; - } else if p < pattern.len() && pattern[p] == '*' { - star = Some(p); - p += 1; - retry = s; - } else if let Some(index) = star { - p = index + 1; - retry += 1; - s = retry; - } else { - return false; - } - } - pattern[p..].iter().all(|character| *character == '*') - } - - fn pattern_matches(pattern: &[String], path: &str) -> bool { - if pattern.len() == 1 && pattern[0] == "**" { - return true; - } - let candidate: Vec<&str> = path.split('/').collect(); - pattern.len() == candidate.len() - && pattern - .iter() - .zip(candidate) - .all(|(pattern, segment)| Self::segment_matches(pattern, segment)) - } - - fn is_ignored(&self, path: &str) -> bool { - let segments: Vec<&str> = path.split('/').collect(); - let parents: Vec = (1..segments.len()).map(|end| segments[..end].join("/")).collect(); - let mut ignored = false; - for (exclusion, pattern) in &self.patterns { - if *exclusion != ignored { - continue; - } - if Self::pattern_matches(pattern, path) || parents.iter().any(|parent| Self::pattern_matches(pattern, parent)) { - ignored = !exclusion; - } - } - ignored - } - } - - /// Every owned file the built-in catalog places in one of the two trees - /// the build context admits: the recipe tree, and the container - /// directory itself. `.anvil/` at large is the general home for - /// tool-owned non-recipe assets, and the allow-list deliberately admits - /// only `.anvil/container/*` from it, so scoping to that directory keeps - /// this guard exhaustive for image inputs without making a future - /// `.anvil/` artifact fail a test it has no bearing on. - fn catalog_context_inputs() -> Vec<&'static str> { - crate::anvil::artifacts::anvil_artifacts() - .into_iter() - .filter_map(|artifact| match artifact { - Artifact::OwnedFile(spec) if spec.path.starts_with("justfiles/") || spec.path.starts_with(".anvil/container/") => { - Some(spec.path) - } - _ => None, - }) - .collect() + fn group_is_exactly_three_files() { + assert_eq!(paths(&all()), [RECIPE_PATH, DOCKERFILE_PATH, DOCKERIGNORE_PATH]); } #[test] - fn ignore_file_admits_only_image_inputs_into_the_build_context() { - let ignore = DockerIgnore::parse(IGNORE); - - // Driving the include set from the catalog rather than a literal list - // makes this the regression guard for the second placement rule in - // extensibility.md §6.1: an owned file placed outside an admitted glob - // fails here instead of inside a real `docker build`. - let mut included: Vec<&str> = vec![ - // Repository-owned rather than catalog-owned, but a required - // image input all the same. - "rust-toolchain.toml", - ]; - included.extend(catalog_context_inputs()); - assert!( - included.contains(&CONTAINERFILE_PATH) && included.contains(&"justfiles/anvil/checks/clippy.just"), - "the catalog-derived include set must cover both admitted trees" - ); - // The entry recipe is not image content, but `mod.just` imports it - // unconditionally, so `just anvil-setup` needs it present. - assert!(included.contains(&RECIPE_PATH), "the entry recipe must reach the build context"); - for included in included { - assert!(!ignore.is_ignored(included), "{included} must reach the build context"); - } - - for excluded in [ - // Trusted host orchestration: never image content, even though - // the surrounding directory is admitted. - CUSTOMIZE_SHELL_PATH, - CUSTOMIZE_POWERSHELL_PATH, - // The container directory is admitted one level deep only, which - // is exactly the depth the image-ID helpers list. Under the - // strictest reading of Docker's parent testing a subdirectory - // matches `!.anvil/container/*` in its own right, so the - // allow-list denies deeper paths explicitly. - ".anvil/container/nested/asset.txt", - ".anvil/container/nested/customize.sh", - ".anvil/container/nested/deeper/asset.txt", - // Assets stranded at the pre-move location, including a - // hand-authored customization file, must not re-enter through a - // directory-level re-inclusion. - "justfiles/anvil/container/customize.sh", - "justfiles/anvil/container/customize.ps1", - "justfiles/anvil/container/Containerfile", - "justfiles/anvil/container/run-in-container.sh", - // Everything else stays out: the working tree is bind-mounted at - // run time rather than baked into the image. - "Cargo.toml", - "crates/example/src/lib.rs", - "justfiles/basic.just", - "justfiles/anvil/notes.md", - ".anvil.lock", - ".anvil/other/asset.txt", - ".git/config", - ] { - assert!(ignore.is_ignored(excluded), "{excluded} must not reach the build context"); - } + fn image_installs_the_generated_toolset() { + // The image must not carry a second tool list: it installs by running + // the same recipe the checks use, from the same generated pins. + assert!(DOCKERFILE.contains("just anvil-setup binstall")); + assert!(DOCKERFILE.contains("COPY justfiles")); + assert!(DOCKERFILE.contains("COPY rust-toolchain.toml")); + // The re-entry guard the recipe relies on to avoid nesting. + assert!(DOCKERFILE.contains("ENV ANVIL_IN_CONTAINER=1")); } #[test] - fn ignore_file_excludes_customize_source_from_the_build_context() { - // Order is load-bearing: the customize and nested re-exclusions only - // win because they come after the directory-wide re-inclusion. - let include_position = IGNORE - .find("!.anvil/container/*") - .expect("the container directory inclusion is asserted above"); - let shell_exclude_position = IGNORE - .find("\n.anvil/container/customize.sh") - .expect("customize.sh must be excluded from the build context"); - let powershell_exclude_position = IGNORE - .find("\n.anvil/container/customize.ps1") - .expect("customize.ps1 must be excluded from the build context"); - let nested_exclude_position = IGNORE - .find("\n.anvil/container/*/*") - .expect("nested container assets must be excluded from the build context"); - assert!( - include_position < shell_exclude_position - && include_position < powershell_exclude_position - && include_position < nested_exclude_position, - "the re-exclusion must come after the broad directory inclusion so it wins" - ); + fn base_image_is_digest_pinned() { + // A floating base tag can change underneath an identity hash that + // claims to name fixed content, which would make every cached image a + // potential lie. + let base = DOCKERFILE + .lines() + .find(|line| line.starts_with("ARG BASE_IMAGE=")) + .expect("the Dockerfile must declare a default BASE_IMAGE"); + assert!(base.contains("@sha256:"), "BASE_IMAGE must be digest-pinned: {base}"); } #[test] - fn drivers_use_docker_and_content_addressing() { - assert!(RECIPE.contains("replace(recipe, \"'\", \"''\")")); - for (driver, customization_source, build_command) in [ - (SHELL_DRIVER, "source \"$customize_script\"", "docker build \\"), - (POWERSHELL_DRIVER, ". $customizeScript", "& wsl -e docker build"), - ] { - assert!(driver.contains("docker")); - assert!(driver.contains("ANVIL_CONTAINER_NO_REBUILD")); - assert!(driver.contains("ANVIL_CONTAINER_BASE_IMAGE")); - assert!(driver.contains("ANVIL_CONTAINER_IMAGE")); - assert!(driver.contains("ANVIL_IN_CONTAINER")); - assert!(driver.contains("auth token --hostname github.com")); - assert!(driver.contains("gh auth login --hostname github.com")); - assert!(driver.contains("/run/secrets/anvil-github-token")); - assert!(driver.contains("anvil-pr-fast")); - assert!(driver.contains("anvil-scheduled-advisories")); - assert!(driver.contains("PR_TITLE")); - assert!(driver.contains("--pull=never")); - assert!(driver.contains("linux/amd64")); - assert!(driver.contains("ANVIL_APRZ_ALREADY_RAN")); - assert!(!driver.contains("--env GITHUB_TOKEN")); - let auth_position = driver - .find("gh auth login --hostname github.com") - .expect("GitHub login command is asserted present above"); - let image_position = driver - .find("docker image inspect") - .expect("Docker image check is asserted present above"); - let customization_position = driver - .find(customization_source) - .expect("customization source command must be present"); - let build_position = driver.find(build_command).expect("Docker build command must be present"); - assert!( - image_position < customization_position && customization_position < auth_position && auth_position < build_position, - "customization must load before GitHub authentication, and authentication must finish before image building" - ); - } - assert!(POWERSHELL_DRIVER.contains("image-id.ps1")); - assert!(IMAGE_ID.contains("[StringComparer]::Ordinal")); - assert!(POWERSHELL_DRIVER.contains("AnvilContainerPrepareCommand")); - assert!(POWERSHELL_DRIVER.contains("wsl -e docker")); - assert!(!POWERSHELL_DRIVER.contains("BuildInMachine")); - assert!(POWERSHELL_DRIVER.contains("git rev-parse --show-toplevel 2>$null")); - assert!(IMAGE_ID.contains("git rev-parse --show-toplevel 2>$null")); - assert!(POWERSHELL_DRIVER.contains("Test-AnvilRecipeNeedsGitHubToken $recipeArg")); - assert!(POWERSHELL_DRIVER.contains("foreach ($recipeArg in $Recipe)")); - assert!(POWERSHELL_DRIVER.contains("[Console]::IsInputRedirected")); - assert!(POWERSHELL_DRIVER.contains("Read-Host")); - assert!(POWERSHELL_DRIVER.contains("ConvertTo-AnvilVersion")); - assert!(POWERSHELL_DRIVER.contains("isolated anvil-aprz")); - assert!(POWERSHELL_DRIVER.contains("docker volume create")); - assert!(POWERSHELL_DRIVER.contains("--user', \"${containerUid}:${containerGid}\"")); - let token_file_create_position = POWERSHELL_DRIVER - .find("[IO.File]::Create($githubTokenFile).Dispose()") - .expect("the temporary GitHub token file must be created before permissions are restricted"); - let token_file_windows_restrict_position = POWERSHELL_DRIVER - .find("& icacls.exe $githubTokenFile") - .expect("the temporary GitHub token file must have a restricted Windows ACL"); - let token_file_unix_restrict_position = POWERSHELL_DRIVER - .find("& chmod 600 $githubTokenFile") - .expect("the temporary GitHub token file must have restricted Unix permissions"); - let token_file_write_position = POWERSHELL_DRIVER - .find("[IO.File]::WriteAllText($githubTokenFile") - .expect("the GitHub token must be written to the restricted temporary file"); - assert!( - token_file_create_position < token_file_windows_restrict_position - && token_file_create_position < token_file_unix_restrict_position - && token_file_windows_restrict_position < token_file_write_position - && token_file_unix_restrict_position < token_file_write_position, - "the temporary GitHub token file must be restricted before the token is written" - ); - assert!(SHELL_DRIVER.contains("anvil_recipe_needs_github_token \"$recipe_arg\"")); - assert!(SHELL_DRIVER.contains("for recipe_arg in \"$@\"")); - assert!(SHELL_DRIVER.contains("image-id.sh")); - assert!(!SHELL_DRIVER.contains("pwsh")); - assert!(SHELL_DRIVER.contains("anvil-container must run from a Git repository")); - assert!(SHELL_DRIVER.contains("[[ ! -t 0 ]]")); - assert!(SHELL_DRIVER.contains("read -r -p")); - assert!(SHELL_DRIVER.contains("github_run_args")); - assert!(SHELL_DRIVER.contains("just anvil-aprz")); - assert!(SHELL_DRIVER.contains("docker volume create")); - assert!(SHELL_DRIVER.contains("--user \"$container_uid:$container_gid\"")); + fn build_context_admits_only_what_the_image_copies() { + assert!(DOCKERIGNORE.contains("!justfiles")); + assert!(DOCKERIGNORE.contains("!rust-toolchain.toml")); } #[test] - fn github_token_recipe_lists_match_the_generated_dependency_graph() { - fn anvil_recipe_tokens(text: &str) -> impl Iterator { - text.split(|character: char| !(character.is_ascii_alphanumeric() || matches!(character, '_' | '-'))) - .filter(|token| token.starts_with("anvil-") || token.starts_with("_anvil-")) - } - - let mut graph = BTreeMap::>::new(); - - for source in dependency_recipe_sources() { - let mut current = None::; - for line in source.lines() { - if !line.chars().next().is_some_and(char::is_whitespace) { - current = line - .split_once(':') - .and_then(|(header, _)| header.split_whitespace().next()) - .filter(|name| name.starts_with("anvil-") || name.starts_with("_anvil-")) - .map(str::to_owned); - } - let Some(recipe) = current.as_ref() else { - continue; - }; - let dependency_text = line.split_once(':').map_or(line, |(_, dependencies)| dependencies); - let dependencies = graph.entry(recipe.clone()).or_default(); - dependencies.extend( - anvil_recipe_tokens(dependency_text) - .map(str::to_owned) - .filter(|dependency| dependency != recipe), - ); - if let Some((_, routed)) = dependency_text.split_once("_anvil-run \"") - && let Some(tier) = routed.split('"').next() - { - dependencies.insert(format!("_anvil-{tier}")); - } - } - } - - let mut expected = BTreeSet::from(["anvil-aprz".to_owned()]); - expected.extend( - graph - .keys() - .filter(|recipe| reaches_aprz(recipe, &graph, &mut BTreeSet::new())) - .cloned(), - ); - - let driver_recipes = |driver: &str, start: &str, end: &str| { - let body = driver - .split_once(start) - .and_then(|(_, remainder)| remainder.split_once(end).map(|(body, _)| body)) - .expect("driver token-classification function must have stable boundaries"); - anvil_recipe_tokens(body).map(str::to_owned).collect::>() - }; - - let shell = driver_recipes(SHELL_DRIVER, "anvil_recipe_needs_github_token() {", "}\n\nversion_at_least"); - let powershell = driver_recipes( - POWERSHELL_DRIVER, - "function Test-AnvilRecipeNeedsGitHubToken", - "\n}\n\nfunction Get-AnvilGitHubToken", - ); - assert_eq!(shell, expected, "Bash token routing must match APRZ reachability"); - assert_eq!(powershell, expected, "PowerShell token routing must match APRZ reachability"); + fn recipe_has_no_generation_time_placeholders() { + // Every value is a literal or resolved at run time; nothing is + // substituted at emit time, so the recipe cannot drift from a + // configuration file that no longer exists. + assert!(!RECIPE.contains("__"), "the recipe must not carry rendering placeholders"); + assert!(!RECIPE.contains("anvil.toml")); } #[test] - fn drivers_implement_the_customization_contract() { - assert!(SHELL_DRIVER.contains("customize.sh")); - assert!(!SHELL_DRIVER.contains("auth.sh")); - assert!(!SHELL_DRIVER.contains("CUSTOMIZATION_API_VERSION")); - assert!(POWERSHELL_DRIVER.contains("customize.ps1")); - assert!(!POWERSHELL_DRIVER.contains("auth.ps1")); - assert!(!POWERSHELL_DRIVER.contains("CustomizationApiVersion")); - - for (driver, image_exists, requested_recipes, needs_github_token) in [ - ( - SHELL_DRIVER, - "ANVIL_CONTAINER_IMAGE_EXISTS", - "ANVIL_CONTAINER_REQUESTED_RECIPES", - "ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN", - ), - ( - POWERSHELL_DRIVER, - "AnvilContainerImageExists", - "AnvilContainerRequestedRecipes", - "AnvilContainerNeedsGitHubToken", - ), + fn recipe_exposes_the_documented_surface() { + for expected in [ + "anvil-container *target:", + "anvil-container-status:", + "anvil-container-rebuild:", + "anvil-container-down:", ] { - assert!(driver.contains("ANVIL_CONTAINER_REPO_ROOT") || driver.contains("AnvilContainerRepoRoot")); - assert!(driver.contains("ANVIL_CONTAINER_DIR") || driver.contains("AnvilContainerDir")); - assert!(driver.contains("ANVIL_CONTAINER_RESOLVED_IMAGE") || driver.contains("AnvilContainerResolvedImage")); - assert!(driver.contains(image_exists)); - assert!(driver.contains(requested_recipes)); - assert!(driver.contains(needs_github_token)); - - // The image-exists check must be resolved before the - // customization file is sourced, so warm-run state is available - // to it. - let image_exists_position = driver - .find(image_exists) - .unwrap_or_else(|| panic!("{image_exists} is asserted present above")); - let source_position = driver - .find("customize.sh") - .or_else(|| driver.find("customize.ps1")) - .expect("customize.* sourcing is asserted present above"); - assert!( - image_exists_position < source_position, - "image existence must be resolved before customization is sourced" - ); - } - - assert!(POWERSHELL_DRIVER.contains("AnvilContainerHostIsWindows")); - assert!(!SHELL_DRIVER.contains("ANVIL_CONTAINER_HOST_IS_WINDOWS")); - - // Preparation arguments without a preparation command must fail - // validation before Docker build/run. - assert!(SHELL_DRIVER.contains("ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND")); - assert!(POWERSHELL_DRIVER.contains("$AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand")); - - // Cleanup callback shape is validated. - assert!(SHELL_DRIVER.contains("must name a callable function")); - assert!(POWERSHELL_DRIVER.contains("must be a script block")); - - // Output arrays are validated before Docker is invoked. - for driver in [SHELL_DRIVER, POWERSHELL_DRIVER] { - let validate_position = driver - .find("must be a string array") - .or_else(|| driver.find("anvil_container_validate_array")) - .expect("output validation is present"); - let build_position = driver.find("docker build").expect("build invocation is present"); - assert!( - validate_position < build_position, - "output validation must occur before Docker build" - ); + assert!(RECIPE.contains(expected), "missing recipe: {expected}"); } } #[test] - #[cfg_attr(miri, ignore = "uses filesystem and subprocesses; miri isolation forbids them")] - fn image_id_excludes_customize_source_but_hashes_static_container_files() { - let tmp = TempDir::new().expect("temporary repository must be creatable"); - let root = tmp.path(); - let status = Command::new("git") - .args(["init", "--quiet"]) - .current_dir(root) - .status() - .expect("git must be available for the image-id helper"); - assert!(status.success(), "temporary Git repository must initialize"); - write_image_id_fixture(root); - - let base = run_image_id(root); - - // Customization source is runtime orchestration, not image content: it - // must never affect the image ID, in either host-shell form. - let customize_sh = root.join(CUSTOMIZE_SHELL_PATH); - write(&customize_sh, "# customization\n"); - assert_eq!(base, run_image_id(root), "customize.sh source must not affect the image ID"); - - let customize_ps1 = root.join(CUSTOMIZE_POWERSHELL_PATH); - write(&customize_ps1, "# customization\n"); - assert_eq!(base, run_image_id(root), "customize.ps1 source must not affect the image ID"); - - write(&customize_sh, "# different customization\n"); - write(&customize_ps1, "# different customization\n"); - assert_eq!( - base, - run_image_id(root), - "changed customization source must still not affect the image ID" - ); - - write(&root.join(README_PATH), "runtime documentation change\n"); - assert_eq!( - base, - run_image_id(root), - "execution-only documentation must not affect the image ID" - ); - - write(&root.join(RECIPE_PATH), "execution-only recipe change\n"); - assert_eq!(base, run_image_id(root), "the container entry recipe must not affect the image ID"); - - let override_image = "example.invalid/bullseye@sha256:1111111111111111111111111111111111111111111111111111111111111111"; - #[cfg(windows)] - let overridden = run_image_id_command_with_base( - root, - "pwsh", - &["-NoProfile", "-File", ".anvil/container/image-id.ps1"], - Some(override_image), - ); - #[cfg(unix)] - let overridden = run_image_id_command_with_base(root, "bash", &[".anvil/container/image-id.sh"], Some(override_image)); - assert_ne!(base, overridden, "the selected base image must affect the image ID"); - - write(&root.join("justfiles/anvil/checks/extra.just"), "anvil-extra:\n @echo extra\n"); - assert_ne!( - base, - run_image_id(root), - "only the container entry recipe itself is execution-only; other recipes are hashed" - ); - std::fs::remove_file(root.join("justfiles/anvil/checks/extra.just")).expect("test file must be removable"); - assert_eq!(base, run_image_id(root), "removing the extra recipe must restore the image ID"); - - // Static, hashed image content must still affect the image ID. - write( - &root.join(CONTAINERFILE_PATH), - "ARG BASE_IMAGE=example.invalid/base@sha256:0000000000000000000000000000000000000000000000000000000000000000\nFROM ${BASE_IMAGE}\nRUN echo changed\n", - ); - assert_ne!( - base, - run_image_id(root), - "changed static Containerfile content must affect the image ID" - ); + fn engine_is_an_environment_variable_with_a_docker_default() { + assert!(RECIPE.contains(r#"env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker")"#)); } #[test] - #[cfg(unix)] - #[cfg_attr(miri, ignore = "uses filesystem and subprocesses; miri isolation forbids them")] - fn image_id_helpers_match_when_pwsh_is_available() { - if Command::new("pwsh").arg("-Version").output().is_err() { - return; - } - - let tmp = TempDir::new().expect("temporary repository must be creatable"); - let root = tmp.path(); - let status = Command::new("git") - .args(["init", "--quiet"]) - .current_dir(root) - .status() - .expect("git must be available for the image-id helpers"); - assert!(status.success(), "temporary Git repository must initialize"); - write_image_id_fixture(root); - write( - &root.join("justfiles/anvil/checks/custom.just"), - "nested-custom-recipe:\n @echo custom\n", - ); - // The entry recipe is skipped by both helpers; seed it so the skip - // itself is compared, not just the recipes they agree to hash. - write(&root.join(RECIPE_PATH), "execution-only:\n @echo entry\n"); - - let shell = run_image_id_command(root, "bash", &[".anvil/container/image-id.sh"]); - let powershell = run_image_id_command(root, "pwsh", &["-NoProfile", "-File", ".anvil/container/image-id.ps1"]); - assert_eq!(shell, powershell); + fn recipe_excludes_itself_from_the_image_identity() { + // Hashing the driver would make the tag depend on the tag. + assert!(RECIPE.contains("justfiles/anvil/container.just")); + assert!(RECIPE.contains("-cne 'justfiles/anvil/container.just'")); } #[test] - fn shell_driver_supports_legacy_bash() { - assert!(!SHELL_DRIVER.contains("sort -V")); - assert!(!SHELL_DRIVER.contains("[[ -v")); - assert!(SHELL_DRIVER.contains("version_at_least")); - assert!(SHELL_DRIVER.contains("if command -v sha256sum")); - assert!(SHELL_DRIVER.contains("shasum -a 256")); - assert!(SHELL_DRIVER.contains("printenv")); - assert!(SHELL_DRIVER.contains("declare -p")); - assert!(SHELL_IMAGE_ID.contains("shasum -a 256")); - assert!(SHELL_IMAGE_ID.contains("LC_ALL=C sort -u")); - assert!(!SHELL_IMAGE_ID.contains("pwsh")); - - // Namerefs (`local -n`/`declare -n`) require Bash 4.3+. Array-name - // validation must pass elements positionally instead. - assert!(!SHELL_DRIVER.contains("local -n"), "namerefs are unsupported on Bash 3.2"); - assert!(!SHELL_DRIVER.contains("declare -n"), "namerefs are unsupported on Bash 3.2"); - - // Every possibly-empty customization-output array must be expanded - // with the `${arr[@]+"${arr[@]}"}` idiom, not a bare `"${arr[@]}"`: - // under `set -u`, Bash versions before 4.4 raise "unbound variable" - // when a declared-but-empty array is expanded bare. The guarded - // idiom necessarily contains the bare form as a substring, so pin - // safety by asserting every bare occurrence is part of a guarded - // one (equal counts) rather than absent outright. - for array in [ - "ANVIL_CONTAINER_BUILD_ARGS", - "ANVIL_CONTAINER_PREPARE_ARGS", - "ANVIL_CONTAINER_RUN_ARGS", - ] { - let guarded = format!("${{{array}[@]+\"${{{array}[@]}}\"}}"); - let bare = format!("\"${{{array}[@]}}\""); - let guarded_count = SHELL_DRIVER.matches(&guarded).count(); - let bare_count = SHELL_DRIVER.matches(&bare).count(); - assert!(guarded_count > 0, "{array} must use the nounset-safe empty-array idiom: {guarded}"); - assert_eq!( - guarded_count, bare_count, - "{array} must never be expanded bare outside the nounset-safe idiom (unsafe under `set -u` on Bash <4.4)" - ); - } + fn hook_file_is_an_image_input_but_hook_output_is_not() { + // A changed hook must rename the tag; a minted credential must not. + assert!(RECIPE.contains("$inputs += $hookRel")); + assert!(RECIPE.contains("id=$id,env=$name")); } #[test] - fn recipe_uses_native_host_interpreters() { - assert!(RECIPE.contains("[windows]")); - assert!(RECIPE.contains("[script(\"pwsh\", \"-NoProfile\")]")); - assert!(RECIPE.contains("[unix]")); - assert!(RECIPE.contains("[script(\"bash\")]")); - assert!(!RECIPE.contains("$IsWindows")); + fn hook_values_are_passed_by_name_never_by_value() { + // NAME=VALUE on a command line is recorded by endpoint telemetry and + // retained far longer than a short-lived token is meant to live. + assert!(RECIPE.contains("$runArgs += @('-e', $name)")); + assert!(!RECIPE.contains("-e', \"$name=")); } #[test] - fn entrypoint_initializes_non_root_cargo_metadata() { - for file in ["config.toml", ".crates.toml", ".crates2.json"] { - assert!(ENTRYPOINT.contains(file)); - } - assert!(ENTRYPOINT.contains("export CARGO_HOME")); - assert!(ENTRYPOINT.contains("ln -sfn /usr/local/cargo/registry")); - assert!(ENTRYPOINT.contains("ln -sfn /usr/local/cargo/git")); - assert!(ENTRYPOINT.contains("exec \"$@\"")); + fn empty_hook_values_fail_closed() { + assert!(RECIPE.contains("returned an empty value for secret")); + assert!(RECIPE.contains("returned an empty value for")); } #[test] - fn drivers_support_interactive_shell_mode() { - assert!(SHELL_DRIVER.contains("--interactive --tty")); - assert!(SHELL_DRIVER.contains("\"$image\" bash")); - assert!(POWERSHELL_DRIVER.contains("wsl -e docker @runArgs --interactive --tty $image bash")); + fn hooks_constructor_uses_the_documented_path() { + assert_eq!(paths(&[hooks("# body\n")]), [HOOKS_PATH]); + assert_eq!(hooks("# body\n").body(), "# body\n"); } #[test] - fn customize_helpers_use_the_standard_paths() { - match customize_shell("# shell customization\n") { - Artifact::OwnedFile(spec) => { - assert_eq!(spec.path, CUSTOMIZE_SHELL_PATH); - assert_eq!(spec.body, "# shell customization\n"); - } - Artifact::Region(_) => panic!("customization file must be an owned file"), - } - match customize_powershell("# PowerShell customization\n") { - Artifact::OwnedFile(spec) => { - assert_eq!(spec.path, CUSTOMIZE_POWERSHELL_PATH); - assert_eq!(spec.body, "# PowerShell customization\n"); - } - Artifact::Region(_) => panic!("customization file must be an owned file"), - } + fn dockerfile_body_can_be_replaced_by_a_fork() { + let replaced = dockerfile().with_body("FROM example.invalid/base\n"); + assert_eq!(paths(&[replaced.clone()]), [DOCKERFILE_PATH]); + assert_eq!(replaced.body(), "FROM example.invalid/base\n"); } } diff --git a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs index f493aabe..db82ec81 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs @@ -43,12 +43,6 @@ const HELPERS_JUST: &str = include_str!("../../../templates/justfiles/anvil/help /// Repo-root-relative path of the shared-helpers recipe file. const HELPERS_JUST_PATH: &str = "justfiles/anvil/helpers.just"; -/// Contents of `justfiles/anvil/runner.just` baked into the binary. -const RUNNER_JUST: &str = include_str!("../../../templates/justfiles/anvil/runner.just"); - -/// Repo-root-relative path of the tier execution router. -const RUNNER_JUST_PATH: &str = "justfiles/anvil/runner.just"; - /// Emits `(path, include_str!)` pairs for a set of split recipe files that /// live under a subdirectory of `justfiles/anvil/`. Each file is one owned /// artifact, so the recipe tree is one file per check / per group rather @@ -65,33 +59,17 @@ macro_rules! split_recipe_files { } #[test] -fn runner_routes_tiers_and_guards_recursion() { - assert!(RUNNER_JUST.contains("[windows]")); - assert!(RUNNER_JUST.contains("[script(\"pwsh\", \"-NoProfile\")]")); - assert!(RUNNER_JUST.contains("[unix]")); - assert!(RUNNER_JUST.contains("[script(\"bash\")]")); - assert_eq!(RUNNER_JUST.matches("[no-exit-message]").count(), 2); - assert!(RUNNER_JUST.contains("if ($env:ANVIL_IN_CONTAINER)")); - assert!(RUNNER_JUST.contains("if [[ -n \"${ANVIL_IN_CONTAINER:-}\" ]]")); - assert!(RUNNER_JUST.contains("replace(just_executable(), \"'\", \"''\")")); - assert!(RUNNER_JUST.contains("replace(justfile(), \"'\", \"''\")")); - assert!(RUNNER_JUST.contains("replace(tier, \"'\", \"''\")")); - assert!(RUNNER_JUST.contains("replace(runner, \"'\", \"''\")")); - assert!(RUNNER_JUST.contains("& $just --justfile $justfile anvil-container $nativeTier")); - assert!(RUNNER_JUST.contains("exec \"$just_path\" --justfile \"$justfile\" anvil-container \"$native_tier\"")); - assert_eq!(RUNNER_JUST.matches("expected 'native' or 'container'").count(), 2); -} - -#[test] -fn aprz_uses_the_container_secret_and_fails_fast_without_it() { +fn aprz_borrows_a_token_and_degrades_with_instructions() { let aprz = CHECK_FILES .iter() .find_map(|(path, body)| path.ends_with("/aprz.just").then_some(*body)) .expect("aprz.just is registered in CHECK_FILES below"); - assert!(aprz.contains("if ($env:ANVIL_IN_CONTAINER)")); - assert!(aprz.contains("ANVIL_APRZ_ALREADY_RAN")); - assert!(aprz.contains("/run/secrets/anvil-github-token")); - assert!(aprz.contains("Run `gh auth login` on the host")); + assert!(aprz.contains("gh auth token --hostname github.com")); + assert!(aprz.contains("GITHUB_TOKEN is not set")); + // Nothing container-specific: inside the image a credential arrives as an + // ordinary environment variable, from the hook or from CI. + assert!(!aprz.contains("ANVIL_IN_CONTAINER")); + assert!(!aprz.contains("/run/secrets/")); } /// One `justfiles/anvil/checks/.just` file per catalog check @@ -198,12 +176,6 @@ pub fn helpers() -> Artifact { Artifact::owned_file(HELPERS_JUST_PATH, HELPERS_JUST) } -/// `justfiles/anvil/runner.just` — native/container tier routing. -#[must_use] -pub fn runner() -> Artifact { - Artifact::owned_file(RUNNER_JUST_PATH, RUNNER_JUST) -} - /// The `justfiles/anvil/checks/.just` files — one owned artifact /// per catalog check. #[must_use] @@ -382,22 +354,19 @@ mod tests { #[test] fn tiers_just_template_has_three_tiers() { - for needle in [ - "anvil-pr:", - "anvil-scheduled:", - "anvil-full:", - "_anvil-pr:", - "_anvil-scheduled:", - "_anvil-full:", - ] { + for needle in ["anvil-pr:", "anvil-scheduled:", "anvil-full:"] { assert!(TIERS_JUST.contains(needle), "tiers.just missing '{needle}'"); } + // Tiers depend on their work directly: there is no routing seam, so + // `just anvil-pr` always runs natively and containerized execution is + // reached only through the explicit `anvil-container` recipe. + assert!(!TIERS_JUST.contains("_anvil-run"), "tiers must not route through an execution seam"); // Each tier runs its validate-prereqs aggregate first so a missing // tool fails up front rather than mid-run. for needle in [ - "anvil-pr: (_anvil-run \"pr\" anvil_runner)", - "anvil-scheduled: (_anvil-run \"scheduled\" anvil_runner)", - "anvil-full: (_anvil-run \"full\" anvil_runner)", + "anvil-pr: anvil-pr-validate-prereqs", + "anvil-scheduled: anvil-scheduled-validate-prereqs", + "anvil-full: anvil-full-validate-prereqs", ] { assert!( TIERS_JUST.contains(needle), @@ -450,7 +419,6 @@ mod tests { "import 'container.just'", "import 'groups/pr-fast.just'", "import 'groups/scheduled-exhaustive.just'", - "import 'runner.just'", "import 'tiers.just'", "import 'tools.just'", "import 'versions.just'", diff --git a/crates/cargo-anvil/src/anvil/artifacts/mod.rs b/crates/cargo-anvil/src/anvil/artifacts/mod.rs index 59f8bbfd..bdaf31fd 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/mod.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/mod.rs @@ -37,10 +37,8 @@ pub(crate) fn anvil_artifacts() -> Vec { justfile::tools(), justfile::versions(), justfile::helpers(), - justfile::runner(), justfile::tiers(), region::justfile_imports(), - region::justfile_runner(), region::workspace_lints(), region::single_crate_lints(), region::member_lints(), @@ -83,10 +81,8 @@ mod tests { justfile::versions(), justfile::tools(), justfile::helpers(), - justfile::runner(), justfile::tiers(), region::justfile_imports(), - region::justfile_runner(), region::workspace_lints(), region::single_crate_lints(), region::member_lints(), @@ -139,8 +135,15 @@ mod tests { let generated_marker = spec.body.contains("GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY."); let customizable_wrapper = spec.path == ".pipelines/anvil/steps/job.yml" && spec.body.contains("Default job wrapper emitted by cargo-anvil."); + // The container image definition is deliberately editable in + // place: a repository that needs a different base or extra + // packages changes it and anvil's drift handling preserves the + // edit. A "DO NOT EDIT" marker would contradict that, so these + // carry the weaker provenance marker instead. + let editable_image_definition = + spec.path.starts_with(".anvil/container/Dockerfile") && spec.body.contains("Managed by cargo-anvil."); assert!( - generated_marker || customizable_wrapper, + generated_marker || customizable_wrapper || editable_image_definition, "owned file '{}' lacks the generated-content marker", spec.path ); diff --git a/crates/cargo-anvil/src/anvil/artifacts/region.rs b/crates/cargo-anvil/src/anvil/artifacts/region.rs index 6c2a86db..0deba72c 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/region.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/region.rs @@ -68,12 +68,6 @@ const GITATTRIBUTES_PATH: &str = ".gitattributes"; /// Region id for the managed section of `.gitattributes`. const GITATTRIBUTES_REGION_ID: &str = "anvil-gitattributes"; -/// Region id for the user-controlled native/container tier policy. -const JUSTFILE_RUNNER_REGION_ID: &str = "anvil-runner"; - -/// Embedded body of the user-controlled tier execution policy. -const JUSTFILE_RUNNER_BODY: &str = include_str!("../../../templates/regions/justfile-runner.just"); - /// Embedded body of the `deny.toml` `[advisories]` managed region. const DENY_ADVISORIES_BODY: &str = include_str!("../../../templates/regions/deny-advisories.toml"); @@ -141,17 +135,6 @@ pub fn justfile_imports() -> Artifact { ) } -/// `Justfile` / `anvil-runner` — user-controlled tier execution policy. -#[must_use] -pub fn justfile_runner() -> Artifact { - Artifact::region(RegionSpec { - host: HostSelector::Path(justfile::JUSTFILE_PATH.to_owned()), - id: RegionId::new(JUSTFILE_RUNNER_REGION_ID), - body: JUSTFILE_RUNNER_BODY.to_owned(), - syntax: CommentSyntax::Hash, - }) -} - /// Root `Cargo.toml` / `anvil-workspace-lints`. /// /// The workspace-scope lint catalog under `[workspace.lints]`. Emitted only diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 24a02516..21429f55 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -96,118 +96,80 @@ //! //! ## Containerized local checks //! -//! Anvil can run any generated recipe in a content-addressed Linux container. -//! The image installs the Rust toolchains and Cargo tools pinned by the -//! repository's generated Anvil configuration, providing a repeatable Linux -//! environment without installing those tools directly on the host. +//! Any generated recipe can run in a content-addressed Linux container. The +//! image installs the Rust toolchain and Cargo tools that this repository +//! pins, by running `just anvil-setup` — the same recipe the checks use — so +//! the container and the host agree on the toolset by construction. //! -//! ### Prerequisites -//! -//! - Docker Engine 23.0 or newer, installed directly in Linux or WSL and -//! usable by the current user. -//! - `git` and `just` on the host. -//! - Bash on Linux and WSL; `PowerShell` Core (`pwsh`) and WSL 2 on Windows. -//! - `[script]` support enabled in the root `Justfile` (`set unstable` when -//! required by the installed `just` version). -//! - A repository-owned `rust-toolchain.toml`. -//! - On Windows, Docker Engine running in the default WSL distribution: +//! There is no configuration file and no transparent routing: `just anvil-pr` +//! keeps running natively, and the container is reached only through the +//! explicit recipe. //! -//! ```powershell -//! wsl -e docker version +//! ```text +//! just anvil-container anvil-clippy # one check +//! just anvil-container anvil-pr # the whole PR tier +//! just anvil-container # interactive shell //! ``` //! -//! The Windows driver invokes Docker in the default WSL distribution and does -//! not call Windows `docker.exe`. Regardless of the installation, the command -//! above must succeed. Docker Desktop is not required. +//! ### Prerequisites //! -//! On ARM64 hosts, Docker emulates the required `linux/amd64` environment, so -//! image builds and checks can be substantially slower than on x86-64 hosts. +//! - A container engine callable from the shell that runs `just`: Docker +//! (supported) or Podman (best-effort). On Windows that means Docker +//! Desktop, Podman, or a Windows `docker` CLI pointed at an engine in WSL. +//! - `just` and `PowerShell` Core (`pwsh`) on the host. +//! - A repository-owned `rust-toolchain.toml`. //! -//! ### Run a recipe +//! On ARM64 hosts the image is emulated as `linux/amd64`, so builds and checks +//! are substantially slower. //! -//! ```text -//! just anvil-container anvil-clippy -//! just anvil-container anvil-pr -//! just anvil-container -//! ``` +//! ### Image identity //! -//! The no-argument form opens an interactive shell. Anvil builds an image the -//! first time it encounters a content hash and reuses it on later runs. Changes -//! to the Rust toolchain, generated Anvil files, Containerfile, or other static -//! image inputs select a new tag and build a new image. Images for earlier -//! hashes remain available to older branches. Runtime `customize.*` files do not -//! affect image identity. -//! Every argument is a recipe name; recipe parameters are not supported by -//! this command surface. +//! The tag *is* a SHA-256 over the inputs that define the image: the +//! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional +//! credential hook, and the generated recipe tree. Presence therefore implies +//! freshness — a changed tool pin names a tag that cannot already exist, so a +//! build follows. There is no staleness check because there is nothing to +//! check. //! -//! Cargo registry and Cargo Git caches use repository-scoped named volumes; -//! `target/` is additionally scoped by image ID. The -//! repository is mounted at `/workspace`; keeping build output in a named -//! volume avoids slow host bind-mount I/O, particularly on Windows. +//! ### Controls //! -//! ### Make tiers use the container +//! | Variable | Effect | +//! |---|---| +//! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. A host property; never committed. | +//! | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure. | +//! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves. | +//! | `ANVIL_IN_CONTAINER=1` | Set inside the image; makes a nested invocation run natively. | //! -//! Native execution remains the default. Enable container execution for the -//! current shell: +//! Supporting recipes: `anvil-container-status`, `anvil-container-rebuild`, +//! and `anvil-container-down` (removes this repository's cache volumes). //! -//! ```powershell -//! $env:ANVIL_RUNNER = "container" -//! just anvil-pr -//! ``` +//! ### Credentials //! -//! On Unix: +//! crates.io needs none, so the public catalog emits no credential plumbing. +//! A repository or a downstream catalog that needs one adds +//! `.anvil/container/hooks.ps1`, which the recipe loads when present: //! -//! ```sh -//! ANVIL_RUNNER=container just anvil-pr +//! ```powershell +//! function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } +//! function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } //! ``` //! -//! A one-off override is also supported: -//! -//! ```text -//! just anvil_runner=container anvil-pr -//! ``` +//! Build secrets are passed to `BuildKit` by environment variable name, so a +//! value never reaches a process argument and never reaches an image layer; +//! run-time values are forwarded into the container by name for the same +//! reason. An empty value is a hard error, because a build that quietly +//! proceeded without its credential would install a reduced tool set and then +//! be tagged with the hash a credentialed build produces. //! -//! To make containers the project default, edit `/Justfile` -//! and change the default value in the `anvil-runner` region from `"native"` -//! to `"container"`. Commit `/Justfile` with that policy -//! change. Set `ANVIL_RUNNER=native` to override it for one shell. +//! The hook runs on the host with the developer's permissions, before any +//! container isolation. Only run one from a repository or catalog you trust. //! -//! ### Controls +//! ### Customizing the image //! -//! | Variable | Effect | -//! |---|---| -//! | `ANVIL_CONTAINER_BASE_IMAGE` | Select a compatible digest-pinned Linux base image; the value is included in the content hash. | -//! | `ANVIL_CONTAINER_IMAGE` | Override the local image name. The content hash remains the tag. | -//! | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail when the matching image is missing instead of building it. | -//! -//! The public driver never pulls `ANVIL_CONTAINER_IMAGE` remotely. Repositories -//! and derived catalogs can add trusted `customize.sh` and -//! `customize.ps1` files for image-build secrets, dependency preparation, -//! APRZ classification, runtime arguments, and cleanup through the documented -//! customization contract without changing the public command surface. -//! -//! Customization files execute on the host with the developer's permissions -//! before container isolation. Only run them from a repository or catalog you -//! trust. -//! -//! For GitHub API checks, the driver automatically uses an existing host -//! `GITHUB_TOKEN` or the token from an authenticated host `gh` CLI session. It -//! mounts the token read-only for the command and removes the temporary file -//! afterward. If `gh` is installed but not authenticated, an interactive run -//! pauses before building the image, explains the unauthenticated API limit, -//! and continues after the user completes `gh auth login` and presses Enter. -//! -//! ### Troubleshooting -//! -//! - A first-run image build is expected and may take several minutes. -//! - `wsl -e docker images anvil-dev` lists locally cached Anvil images from -//! Windows; use `docker images anvil-dev` inside Linux or WSL. -//! - `ANVIL_CONTAINER_NO_REBUILD=1` distinguishes a cache miss from a build -//! failure. -//! - Non-interactive runs cannot pause for login. Authenticate `gh` or set host -//! `GITHUB_TOKEN` before starting them. -//! - Regenerate managed files with `cargo anvil`; do not hand-edit -//! `.anvil/container/`. +//! `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place +//! and anvil's drift handling preserves it. A downstream catalog that targets +//! a different base OS or toolchain source replaces the artifact instead — see +//! [`artifacts::container::dockerfile`]. //! //! ## Checks and tiers //! diff --git a/crates/cargo-anvil/templates/anvil/container/Containerfile b/crates/cargo-anvil/templates/anvil/container/Containerfile deleted file mode 100644 index fa68b18f..00000000 --- a/crates/cargo-anvil/templates/anvil/container/Containerfile +++ /dev/null @@ -1,69 +0,0 @@ -# syntax=docker/dockerfile:1 -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. - -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e -FROM ${BASE_IMAGE} - -ARG ANVIL_IMAGE_ID -ARG JUST_VERSION=1.56.0 -ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 -ARG POWERSHELL_VERSION=7.6.3 -ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 -ARG RUSTUP_VERSION=1.29.0 -ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 - -ENV DEBIAN_FRONTEND=noninteractive \ - CARGO_HOME=/usr/local/cargo \ - RUSTUP_HOME=/usr/local/rustup \ - RUSTUP_NO_UPDATE_CHECK=1 \ - PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - build-essential ca-certificates clang libclang-dev curl git libicu-dev \ - libssl-dev pkg-config tar \ - && rm -rf /var/lib/apt/lists/* - -RUN curl -fsSLo /tmp/powershell.tar.gz \ - "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ - && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ - && mkdir -p /opt/microsoft/powershell/7 \ - && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ - && chmod 755 /opt/microsoft/powershell/7/pwsh \ - && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ - && rm /tmp/powershell.tar.gz - -RUN curl -fsSLo /tmp/just.tar.gz \ - "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ - && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ - && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ - && chmod 755 /usr/local/bin/just \ - && rm /tmp/just.tar.gz - -RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ - "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ - && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ - && chmod 755 /tmp/rustup-init \ - && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ - && rm /tmp/rustup-init - -WORKDIR /opt/anvil -COPY . . -RUN test -f rust-toolchain.toml || { \ - echo "anvil-container requires rust-toolchain.toml" >&2; \ - exit 1; \ - } -RUN --mount=type=cache,id=anvil-cargo-registry,target=/usr/local/cargo/registry \ - --mount=type=cache,id=anvil-cargo-git,target=/usr/local/cargo/git \ - --mount=type=cache,id=anvil-cargo-target,target=/tmp/anvil-target \ - printf "anvil_runner := \"native\"\nimport 'justfiles/anvil/mod.just'\n" > Justfile \ - && CARGO_TARGET_DIR=/tmp/anvil-target just anvil-setup - -COPY .anvil/container/entrypoint.sh /usr/local/bin/anvil-container-entrypoint -RUN chmod 755 /usr/local/bin/anvil-container-entrypoint - -ENV ANVIL_IN_CONTAINER=1 -LABEL io.github.cargo-anvil.image-id="${ANVIL_IMAGE_ID}" -WORKDIR /workspace -ENTRYPOINT ["anvil-container-entrypoint"] -CMD ["bash"] diff --git a/crates/cargo-anvil/templates/anvil/container/Containerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Containerfile.dockerignore deleted file mode 100644 index 6566e657..00000000 --- a/crates/cargo-anvil/templates/anvil/container/Containerfile.dockerignore +++ /dev/null @@ -1,26 +0,0 @@ -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Deny-all allow-list for the image build context. -# -# Docker matches each candidate against every pattern in order and lets the -# last match win, testing the path itself *and each of its parent directories* -# (moby/patternmatcher MatchesOrParentMatches). A bare directory re-inclusion -# such as `!justfiles` therefore re-admits the entire subtree below it, which -# would defeat this allow-list, so list only leaf patterns here. Docker still -# descends into a denied directory when some re-inclusion pattern is prefixed -# by it, so the intermediate directories need no entries of their own. -# -# Parent testing also reaches through a single-segment re-inclusion: a -# subdirectory of `.anvil/container/` matches `!.anvil/container/*` in its own -# right. The image-ID helpers list that directory one level deep, so a nested -# file is not an image input; `.anvil/container/*/*` states that leaf-only -# contract in the allow-list too, at every depth, because a deeper candidate -# always has an ancestor of exactly that shape. -** -!rust-toolchain.toml -!justfiles/anvil/*.just -!justfiles/anvil/checks/*.just -!justfiles/anvil/groups/*.just -!.anvil/container/* -.anvil/container/*/* -.anvil/container/customize.sh -.anvil/container/customize.ps1 diff --git a/crates/cargo-anvil/templates/anvil/container/README.md b/crates/cargo-anvil/templates/anvil/container/README.md deleted file mode 100644 index b9cd1642..00000000 --- a/crates/cargo-anvil/templates/anvil/container/README.md +++ /dev/null @@ -1,228 +0,0 @@ - - -# Run Anvil checks in a local container - -Use `just anvil-container` to run generated Anvil checks in a reproducible -Linux environment without installing the complete Rust and Cargo tool catalog -on the host. - -Native execution remains the default. The first container run builds an image -matching the repository's generated configuration. Later runs reuse that image, -dependency caches, and compilation output. - -## Quick start - -Ensure Docker Engine is running, then run: - -```text -just anvil-container anvil-clippy -``` - -The first run builds the matching image and can take several minutes. - -## Prerequisites - -- [Docker Engine](https://docs.docker.com/engine/install/) 23.0 or newer, - installed directly in Linux or WSL and usable by the current user. -- `git` and `just` on the host. -- Bash on Linux and WSL; PowerShell Core (`pwsh`) and WSL 2 on Windows. -- `[script]` support enabled in the root `Justfile`. Add `set unstable` when - required by the installed `just` version. -- A `rust-toolchain.toml` in the repository root. -- A Linux or WSL environment capable of running `linux/amd64` images, either - natively on x86-64 or through Docker emulation on ARM64. - -On Windows, the driver invokes Docker from the default WSL distribution rather -than calling Windows `docker.exe`. Regardless of how Docker is installed, this -command must succeed from PowerShell: - -```text -wsl -e docker version -``` - -Start the Docker service inside WSL when it is stopped and add the WSL user to -the `docker` group when non-root access is not already configured. Docker -Desktop is not required. - -On ARM64 hosts, Docker emulates the required `linux/amd64` environment. Image -builds and checks can therefore be substantially slower than on x86-64 hosts. - -## Security boundary - -> [!WARNING] -> `customize.sh` and `customize.ps1` execute on the host with the developer's -> permissions before container isolation begins. Reviewing and trusting these -> files is equivalent to reviewing and trusting any other host-executed script -> in the checked-out branch. - -## Common workflows - -Run one check: - -```text -just anvil-container anvil-clippy -``` - -Run the complete pull-request tier: - -```text -just anvil-container anvil-pr -``` - -Every argument is treated as a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not supported by this command surface. - -Open an interactive Bash shell in the image: - -```text -just anvil-container -``` - -### Use containers for tier commands - -Native execution remains the default. To route tier commands such as -`just anvil-pr` through the container for the current shell: - -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` - -On Unix: - -```sh -ANVIL_RUNNER=container just anvil-pr -``` - -For one invocation: - -```text -just anvil_runner=container anvil-pr -``` - -To make container execution the repository default, change the default value -in the `anvil-runner` region of the repository-root `Justfile` from `"native"` -to `"container"` and commit that policy. Set `ANVIL_RUNNER=native` to override -the repository default for the current shell. - -Tier routing starts a nested `just` invocation. Output and exit status are -preserved, but outer `--dry-run`, dependency introspection, global options, and -CLI variable assignments are not propagated to the selected private tier. -Values other than `native` and `container` are rejected. - -## Images and caches - -The image name includes a content-based tag derived from the repository's Rust -toolchain, generated Anvil recipes, and container build configuration. A -relevant change selects a new image automatically; older branches can continue -using their matching images. - -The following data is reused between runs: - -- the matching container image; -- repository-scoped Cargo registry and Cargo Git caches; -- compilation output in a repository- and image-specific `target` volume. - -The repository is mounted read/write at `/workspace`. Build output remains in a -named volume instead of the host `target/`, avoiding incompatible artifacts and -slow host-to-virtual-machine I/O. - -## GitHub authentication - -`anvil-aprz` and aggregate tiers that include it require GitHub API -authentication. The driver uses either: - -- the host `GITHUB_TOKEN`; or -- the token from an authenticated host `gh` session. - -Trusted customization can provision a short-lived token by setting -`GITHUB_TOKEN`; the driver reads it after loading and validating customization. - -Authenticate the GitHub CLI with: - -```text -gh auth login --hostname github.com -``` - -For an aggregate tier, the driver first runs `anvil-aprz` in a short-lived -container with the token mounted read-only. After it succeeds, the driver runs -the remaining checks in another container without the token. Temporary token -files are removed afterward. - -An interactive invocation can pause while you authenticate. A non-interactive -invocation fails with instructions when authentication is unavailable. - -## Configuration - -| Variable | Effect | -|---|---| -| `ANVIL_RUNNER` | Selects `native` or `container` execution for tier commands | -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a digest-pinned compatible Linux base image and changes the content-based tag | -| `ANVIL_CONTAINER_IMAGE` | Changes the local image name; the content-based tag is retained | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails instead of building when the matching image is absent | - -The public driver builds images locally and does not pull -`ANVIL_CONTAINER_IMAGE` from a registry. - -The default base is digest-pinned Debian Bookworm. Set -`ANVIL_CONTAINER_BASE_IMAGE` to another image compatible with the generated -Debian-based `Containerfile` when a lower glibc baseline is required. A -different package ecosystem such as Azure Linux requires a derived -`Containerfile`. The value must use `image@sha256:` form so the -selected base remains part of the content-addressed image identity. - -Two simultaneous cold invocations can both build the same missing image. This -is accepted for local development: the content-addressed tag converges on the -same inputs, at the cost of duplicate work. - -## Troubleshooting - -| Problem | Resolution | -|---|---| -| Docker is not found on Linux or WSL | Install Docker Engine 23.0 or newer inside that environment | -| Docker is unavailable from Windows | Run `wsl -e docker version`; install or start Docker Engine in the default WSL distribution | -| Docker requires elevated access | Add the Linux/WSL user to the `docker` group, then start a new shell | -| ARM64 execution is slow | The current image is `linux/amd64` and runs through Docker emulation | -| `linux/amd64` cannot run | Configure Docker to run `linux/amd64` images | -| `[script]` recipes are unavailable | Enable `[script]` support; older `just` versions require `set unstable` | -| `rust-toolchain.toml` is missing | Add the repository-owned toolchain file at the repository root | -| GitHub authentication is unavailable | Run `gh auth login --hostname github.com` or set host `GITHUB_TOKEN` | -| A matching image is missing with `ANVIL_CONTAINER_NO_REBUILD=1` | Unset the variable to allow the local image build | -| The first run is slow | The initial image build installs the pinned tool catalog; later runs reuse it | - -Use `docker images anvil-dev` inside Linux or WSL to list locally cached -default Anvil images. - -## Managed files - -This directory is managed by `cargo-anvil`. Regenerate it with `cargo anvil` -instead of editing its files directly. - -> [!IMPORTANT] -> These assets previously lived in `justfiles/anvil/container/`. `cargo anvil` -> relocates the files it generated, but it does not track a hand-authored -> `customize.sh` or `customize.ps1`. Move any such file to -> `.anvil/container/` yourself; the driver only loads customization from the -> new location and warns on stderr when it finds one left behind. - -## Advanced repository customization - -A repository or derived catalog can add one trusted customization file per -supported host: - -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 -``` - -The driver sources the matching file as trusted host code before authentication, -image construction, and recipe execution. The documented customization -contract provides inputs and validated outputs for APRZ classification, build -secrets, dependency preparation, runtime arguments, and cleanup. - -Customization source is excluded from image identity and the build context. -Non-secret image behavior must be represented by hashed static files such as -the `Containerfile`, entrypoint, or supporting build scripts. - -See the [container customization contract](https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md#8-container-customization) -for the complete interface and security requirements. diff --git a/crates/cargo-anvil/templates/anvil/container/entrypoint.sh b/crates/cargo-anvil/templates/anvil/container/entrypoint.sh deleted file mode 100644 index fadac5f9..00000000 --- a/crates/cargo-anvil/templates/anvil/container/entrypoint.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/sh -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -eu - -if [ "$(id -u)" -ne 0 ]; then - if [ -z "${HOME:-}" ] || [ "$HOME" = "/" ]; then - HOME="/tmp/anvil-user" - export HOME - fi - - user_cargo_home="$HOME/.cargo" - mkdir -p "$user_cargo_home" - for file in config.toml .crates.toml .crates2.json; do - if [ -r "$CARGO_HOME/$file" ]; then - cp -f "$CARGO_HOME/$file" "$user_cargo_home/$file" - fi - done - export CARGO_HOME="$user_cargo_home" - ln -sfn /usr/local/cargo/registry "$CARGO_HOME/registry" - ln -sfn /usr/local/cargo/git "$CARGO_HOME/git" -fi - -exec "$@" diff --git a/crates/cargo-anvil/templates/anvil/container/image-id.ps1 b/crates/cargo-anvil/templates/anvil/container/image-id.ps1 deleted file mode 100644 index dfc6a53f..00000000 --- a/crates/cargo-anvil/templates/anvil/container/image-id.ps1 +++ /dev/null @@ -1,76 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$inputs = @( - 'rust-toolchain.toml' -) -$toolchainPath = Join-Path $repoRoot 'rust-toolchain.toml' -if (-not (Test-Path -LiteralPath $toolchainPath -PathType Leaf)) { - throw 'anvil-container requires a repository-owned rust-toolchain.toml.' -} -$containerPath = Join-Path $repoRoot '.anvil/container' -$containerRecipe = 'justfiles/anvil/container.just' -$containerfile = Join-Path $containerPath 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$pathComparison = if ($IsWindows) { [StringComparison]::OrdinalIgnoreCase } else { [StringComparison]::Ordinal } -# The container entry recipe drives execution on the host; it is not image -# content, so it must not participate in image identity. -$inputs += Get-ChildItem (Join-Path $repoRoot 'justfiles/anvil') -Recurse -File -Filter '*.just' | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } | - Where-Object { -not $_.Equals($containerRecipe, $pathComparison) } -$executionOnly = @( - 'image-id.ps1', - 'image-id.sh', - 'README.md', - 'run-in-container.ps1', - 'run-in-container.sh', - 'customize.sh', - 'customize.ps1' -) -# customize.sh/customize.ps1 are trusted runtime orchestration, not image -# content: their source must never affect the image ID or build context. -# Static, non-secret build customization belongs in a hashed artifact instead. -$inputs += Get-ChildItem $containerPath -File | - Where-Object { $_.Name -notin $executionOnly } | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } -$uniqueInputs = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) -foreach ($inputPath in $inputs) { - [void]$uniqueInputs.Add($inputPath) -} -$inputs = [string[]]$uniqueInputs -[Array]::Sort($inputs, [StringComparer]::Ordinal) - -$payload = [Text.StringBuilder]::new() -[void]$payload.Append("ANVIL_CONTAINER_BASE_IMAGE`n").Append($baseImage).Append("`n") -foreach ($relative in $inputs) { - $path = Join-Path $repoRoot $relative - if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { - throw "Container image input is missing: $relative" - } - $content = [IO.File]::ReadAllText($path).Replace("`r`n", "`n").Replace("`r", "`n") - [void]$payload.Append($relative).Append("`n").Append($content).Append("`n") -} - -$bytes = [Text.Encoding]::UTF8.GetBytes($payload.ToString()) -$hash = [Security.Cryptography.SHA256]::HashData($bytes) -Write-Output ([Convert]::ToHexString($hash).ToLowerInvariant()) diff --git a/crates/cargo-anvil/templates/anvil/container/image-id.sh b/crates/cargo-anvil/templates/anvil/container/image-id.sh deleted file mode 100644 index e0ed8105..00000000 --- a/crates/cargo-anvil/templates/anvil/container/image-id.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi - -toolchain_path="$repo_root/rust-toolchain.toml" -if [[ ! -f "$toolchain_path" ]]; then - echo 'anvil-container requires a repository-owned rust-toolchain.toml.' >&2 - exit 1 -fi - -container_dir="$repo_root/.anvil/container" -container_recipe="justfiles/anvil/container.just" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$container_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -inputs=(rust-toolchain.toml) -while IFS= read -r path; do - relative="${path#"$repo_root"/}" - # The container entry recipe drives execution on the host; it is not - # image content, so it must not participate in image identity. - if [[ "$relative" != "$container_recipe" ]]; then - inputs+=("$relative") - fi -done < <(find "$repo_root/justfiles/anvil" -type f -name '*.just' -print) - -for path in "$container_dir"/*; do - [[ -f "$path" ]] || continue - case "${path##*/}" in - image-id.ps1 | image-id.sh | README.md \ - | run-in-container.ps1 | run-in-container.sh \ - | customize.sh | customize.ps1) continue ;; - esac - inputs+=("${path#"$repo_root"/}") -done - -if command -v sha256sum >/dev/null 2>&1; then - hash_command=(sha256sum) -elif command -v shasum >/dev/null 2>&1; then - hash_command=(shasum -a 256) -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi - -write_normalized_file() { - local path="$1" - local line status - while true; do - line="" - if IFS= read -r line <&3; then - status=0 - else - status=$? - fi - if ((status != 0)) && [[ -z "$line" ]]; then - break - fi - printf '%s' "${line%$'\r'}" - if ((status == 0)); then - printf '\n' - else - break - fi - done 3<"$path" -} - -{ - printf 'ANVIL_CONTAINER_BASE_IMAGE\n%s\n' "$base_image" - while IFS= read -r relative; do - path="$repo_root/$relative" - if [[ ! -f "$path" ]]; then - echo "Container image input is missing: $relative" >&2 - exit 1 - fi - printf '%s\n' "$relative" - write_normalized_file "$path" - printf '\n' - done < <(printf '%s\n' "${inputs[@]}" | LC_ALL=C sort -u) -} | "${hash_command[@]}" | awk '{print $1}' diff --git a/crates/cargo-anvil/templates/anvil/container/run-in-container.ps1 b/crates/cargo-anvil/templates/anvil/container/run-in-container.ps1 deleted file mode 100644 index 281dc2bd..00000000 --- a/crates/cargo-anvil/templates/anvil/container/run-in-container.ps1 +++ /dev/null @@ -1,342 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param( - [Parameter(Position = 0, ValueFromRemainingArguments = $true)] - [string[]]$Recipe -) - -$ErrorActionPreference = 'Stop' - -function ConvertTo-AnvilVersion([string]$Value) { - $match = [regex]::Match($Value, '^(\d+)\.(\d+)(?:\.(\d+))?') - if (-not $match.Success) { - throw "anvil-container: could not parse Docker Engine version '$Value'." - } - [version]::new( - [int]$match.Groups[1].Value, - [int]$match.Groups[2].Value, - $(if ($match.Groups[3].Success) { [int]$match.Groups[3].Value } else { 0 }) - ) -} - -function Test-AnvilContainerStringArray([string]$Name, $Value) { - if ($Value -isnot [array]) { - throw "anvil-container: `$$Name must be a string array." - } - foreach ($item in $Value) { - if ($item -isnot [string] -or [string]::IsNullOrEmpty($item)) { - throw "anvil-container: `$$Name entries must be non-empty strings." - } - } -} - -function Test-AnvilContainerBuildArgs($Value) { - for ($index = 0; $index -lt $Value.Count; $index++) { - $item = $Value[$index] - if ($item -eq '--secret') { - $index++ - if ($index -ge $Value.Count) { - throw 'anvil-container: $AnvilContainerBuildArgs requires a value after --secret.' - } - } elseif (-not $item.StartsWith('--secret=', [StringComparison]::Ordinal)) { - throw 'anvil-container: $AnvilContainerBuildArgs accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' - } - } -} - -function Test-AnvilRecipeNeedsGitHubToken([string]$Name) { - $Name -in @( - 'anvil-aprz', - 'anvil-pr', - '_anvil-pr', - 'anvil-pr-fast', - 'anvil-scheduled', - '_anvil-scheduled', - 'anvil-scheduled-advisories', - 'anvil-full', - '_anvil-full' - ) -} - -function Get-AnvilGitHubToken { - $token = $env:GITHUB_TOKEN - if (-not $token -and (Get-Command gh -ErrorAction SilentlyContinue)) { - try { - $token = (& gh auth token --hostname github.com 2>$null) - if ($LASTEXITCODE -ne 0) { $token = $null } - } catch { - $token = $null - } - } - if ($token) { $token = $token.Trim() } - if ($token) { return $token } - return $null -} - -if ($env:ANVIL_IN_CONTAINER) { - if ($Recipe.Count -eq 0) { & bash } else { & just @Recipe } - exit $LASTEXITCODE -} - -foreach ($recipeArg in $Recipe) { - if ($recipeArg -notmatch '^_?anvil-[A-Za-z0-9-]+$') { - throw "anvil-container: expected each argument to be an anvil-* recipe, got '$recipeArg'." - } -} - -if (-not (Get-Command wsl -ErrorAction SilentlyContinue)) { - throw 'anvil-container: WSL 2 is required. See .anvil/container/README.md.' -} - -$versionText = (& wsl -e docker version --format '{{.Server.Version}}' 2>$null) -if ($LASTEXITCODE -ne 0 -or -not $versionText) { - throw 'anvil-container: `wsl -e docker version` must succeed. Install or start Docker Engine in the default WSL distribution; this driver does not invoke Windows docker.exe.' -} -$versionText = $versionText.Trim() -if ((ConvertTo-AnvilVersion $versionText) -lt [version]'23.0.0') { - throw "anvil-container: Docker Engine 23.0.0 or newer is required (found $versionText)." -} -$wslArchitecture = (& wsl -e uname -m 2>$null) -if ($LASTEXITCODE -eq 0 -and $wslArchitecture) { - $wslArchitecture = $wslArchitecture.Trim() - if ($wslArchitecture -notin @('x86_64', 'amd64')) { - [Console]::Error.WriteLine( - "anvil-container: warning: $wslArchitecture requires emulation for linux/amd64; builds and checks may be substantially slower." - ) - } -} - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$scriptDir = Join-Path $repoRoot '.anvil/container' -$wslRepoRoot = (& wsl -e wslpath -a $repoRoot).Trim() -if ($LASTEXITCODE -ne 0 -or -not $wslRepoRoot) { - throw 'anvil-container: could not translate the repository path into the default WSL distribution.' -} -$wslScriptDir = "$wslRepoRoot/.anvil/container" -$containerfile = Join-Path $scriptDir 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$imageId = (& (Join-Path $scriptDir 'image-id.ps1')).Trim() -$imageBase = if ($env:ANVIL_CONTAINER_IMAGE) { $env:ANVIL_CONTAINER_IMAGE } else { 'anvil-dev' } -$image = "${imageBase}:$imageId" -$repoBytes = [Text.Encoding]::UTF8.GetBytes($wslRepoRoot) -$repoHash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($repoBytes)).ToLowerInvariant() -$targetVolume = "anvil-target-$($repoHash.Substring(0, 12))-$($imageId.Substring(0, 12))" - -$needsGitHubToken = $false -foreach ($recipeArg in $Recipe) { - if (Test-AnvilRecipeNeedsGitHubToken $recipeArg) { - $needsGitHubToken = $true - break - } -} -$runsOnlyGitHubCheck = $Recipe.Count -eq 1 -and $Recipe[0] -eq 'anvil-aprz' - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -$null = & wsl -e docker image inspect $image 2>$null -$imageExists = $LASTEXITCODE -eq 0 - -New-Variable -Name AnvilContainerRepoRoot -Value $repoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDir -Value $scriptDir -Option ReadOnly -New-Variable -Name AnvilContainerRepoRootWsl -Value $wslRepoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDirWsl -Value $wslScriptDir -Option ReadOnly -New-Variable -Name AnvilContainerResolvedImage -Value $image -Option ReadOnly -New-Variable -Name AnvilContainerImageExists -Value $imageExists -Option ReadOnly -New-Variable -Name AnvilContainerRequestedRecipes -Value $Recipe -Option ReadOnly -New-Variable -Name AnvilContainerHostIsWindows -Value ([bool]$IsWindows) -Option ReadOnly - -# Customization outputs, initialized before sourcing so a missing customize.ps1 -# leaves every phase a documented no-op. -$AnvilContainerBuildArgs = @() -$AnvilContainerPrepareArgs = @() -$AnvilContainerPrepareCommand = @() -$AnvilContainerRunArgs = @() -$AnvilContainerNeedsGitHubToken = $needsGitHubToken -$AnvilContainerCleanup = $null -$githubToken = $null -$githubTokenFile = $null -$exitCode = 0 -$customizeScript = Join-Path $scriptDir 'customize.ps1' -$legacyCustomizeScript = Join-Path $repoRoot 'justfiles/anvil/container/customize.ps1' - -try { - if (Test-Path -LiteralPath $customizeScript -PathType Leaf) { - . $customizeScript - } - elseif (Test-Path -LiteralPath $legacyCustomizeScript -PathType Leaf) { - [Console]::Error.WriteLine( - 'anvil-container: warning: ignoring justfiles/anvil/container/customize.ps1; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.ps1 to keep it active.' - ) - } - - Test-AnvilContainerStringArray 'AnvilContainerBuildArgs' $AnvilContainerBuildArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareArgs' $AnvilContainerPrepareArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareCommand' $AnvilContainerPrepareCommand - Test-AnvilContainerStringArray 'AnvilContainerRunArgs' $AnvilContainerRunArgs - Test-AnvilContainerBuildArgs $AnvilContainerBuildArgs - if ($AnvilContainerNeedsGitHubToken -isnot [bool]) { - throw 'anvil-container: $AnvilContainerNeedsGitHubToken must be a Boolean.' - } - $needsGitHubToken = $needsGitHubToken -or $AnvilContainerNeedsGitHubToken - if ($AnvilContainerPrepareArgs.Count -gt 0 -and $AnvilContainerPrepareCommand.Count -eq 0) { - throw 'anvil-container: $AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand.' - } - if ($AnvilContainerCleanup -and $AnvilContainerCleanup -isnot [scriptblock]) { - throw 'anvil-container: $AnvilContainerCleanup must be a script block.' - } - $githubToken = if ($needsGitHubToken) { Get-AnvilGitHubToken } else { $null } - if ($needsGitHubToken -and -not $githubToken) { - if (-not (Get-Command gh -ErrorAction SilentlyContinue)) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' - } - if (-not [Environment]::UserInteractive -or [Console]::IsInputRedirected) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' - } - Write-Host 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' - [void](Read-Host 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel)') - $githubToken = Get-AnvilGitHubToken - if (-not $githubToken) { - throw 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' - } - } - if (-not $imageExists) { - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - throw "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." - } - & wsl -e docker build ` - --platform linux/amd64 ` - --tag $image ` - --file "$wslScriptDir/Containerfile" ` - --build-arg "ANVIL_IMAGE_ID=$imageId" ` - --build-arg "BASE_IMAGE=$baseImage" ` - @AnvilContainerBuildArgs ` - $wslRepoRoot - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker build failed with exit code $LASTEXITCODE." - } - } - - $containerUid = (& wsl -e id -u).Trim() - $containerGid = (& wsl -e id -g).Trim() - if ($containerUid -notmatch '^\d+$' -or $containerGid -notmatch '^\d+$') { - throw 'anvil-container: could not determine the default WSL user identity.' - } - $registryVolume = "anvil-cargo-registry-$($repoHash.Substring(0, 12))" - $gitVolume = "anvil-cargo-git-$($repoHash.Substring(0, 12))" - foreach ($volume in @($registryVolume, $gitVolume, $targetVolume)) { - $null = & wsl -e docker volume create $volume - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume creation failed for '$volume' with exit code $LASTEXITCODE." - } - } - $mountArgs = @( - '--mount', "type=bind,source=$wslRepoRoot,target=/workspace", - '--mount', "type=volume,source=$registryVolume,target=/usr/local/cargo/registry", - '--mount', "type=volume,source=$gitVolume,target=/usr/local/cargo/git", - '--mount', "type=volume,source=$targetVolume,target=/workspace/target" - ) - & wsl -e docker run --rm --pull=never ` - --platform linux/amd64 ` - --user 0:0 ` - @mountArgs ` - $image sh -c "chown ${containerUid}:${containerGid} /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume initialization failed with exit code $LASTEXITCODE." - } - - $runArgs = @( - 'run', '--rm', '--pull=never', - '--platform', 'linux/amd64', - '--user', "${containerUid}:${containerGid}", - '--env', 'ANVIL_IN_CONTAINER=1', - '--env', 'HOME=/tmp/anvil-user', - '--workdir', '/workspace' - ) - $runArgs += $mountArgs - $prepareRunArgs = @($runArgs) - $runArgs += $AnvilContainerRunArgs - foreach ($name in @( - 'PR_TITLE', - 'BASE_REF', - 'ANVIL_INCLUDE_MODIFIED', - 'ANVIL_INCLUDE_AFFECTED', - 'ANVIL_INCLUDE_REQUIRED', - 'GITHUB_BASE_REF', - 'SYSTEM_PULLREQUEST_TARGETBRANCH' - )) { - if (Test-Path "Env:$name") { - $runArgs += @('--env', "$name=$((Get-Item "Env:$name").Value)") - } - } - if ($AnvilContainerPrepareCommand.Count -gt 0) { - & wsl -e docker @prepareRunArgs @AnvilContainerPrepareArgs $image @AnvilContainerPrepareCommand - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: preparation command failed with exit code $LASTEXITCODE." - } - } - - if ($githubToken) { - $githubTokenFile = Join-Path ([IO.Path]::GetTempPath()) "anvil-github-token-$PID-$([guid]::NewGuid().ToString('N'))" - [IO.File]::Create($githubTokenFile).Dispose() - if ($IsWindows) { - $userSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value - & icacls.exe $githubTokenFile '/inheritance:r' '/grant:r' "*$($userSid):(F)" | Out-Null - } else { - & chmod 600 $githubTokenFile - } - if ($LASTEXITCODE -ne 0) { - throw 'anvil-container: failed to restrict permissions on the temporary GitHub token file.' - } - [IO.File]::WriteAllText($githubTokenFile, $githubToken, [Text.Encoding]::ASCII) - $githubToken = $null - $wslTokenFile = (& wsl -e wslpath -a $githubTokenFile).Trim() - if ($LASTEXITCODE -ne 0 -or -not $wslTokenFile) { - throw 'anvil-container: could not translate the temporary GitHub token path into WSL.' - } - $githubRunArgs = @($runArgs) - $githubRunArgs += @( - '--mount', - "type=bind,source=$wslTokenFile,target=/run/secrets/anvil-github-token,readonly" - ) - if ($runsOnlyGitHubCheck) { - $runArgs = $githubRunArgs - } else { - & wsl -e docker @githubRunArgs $image just anvil-aprz - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: isolated anvil-aprz failed with exit code $LASTEXITCODE." - } - $runArgs += @('--env', 'ANVIL_APRZ_ALREADY_RAN=1') - } - } - - if ($Recipe.Count -eq 0) { - & wsl -e docker @runArgs --interactive --tty $image bash - } else { - & wsl -e docker @runArgs $image just @Recipe - } - $exitCode = $LASTEXITCODE -} finally { - if ($githubTokenFile) { - Remove-Item -LiteralPath $githubTokenFile -Force -ErrorAction SilentlyContinue - } - if ($AnvilContainerCleanup) { & $AnvilContainerCleanup } -} - -exit $exitCode diff --git a/crates/cargo-anvil/templates/anvil/container/run-in-container.sh b/crates/cargo-anvil/templates/anvil/container/run-in-container.sh deleted file mode 100644 index 65b0513d..00000000 --- a/crates/cargo-anvil/templates/anvil/container/run-in-container.sh +++ /dev/null @@ -1,324 +0,0 @@ -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - if (($# == 0)); then exec bash; else exec just "$@"; fi -fi - -for recipe_arg in "$@"; do - if [[ ! "$recipe_arg" =~ ^_?anvil-[A-Za-z0-9-]+$ ]]; then - echo "anvil-container: expected each argument to be an anvil-* recipe, got '$recipe_arg'." >&2 - exit 2 - fi -done - -anvil_recipe_needs_github_token() { - case "$1" in - anvil-aprz | anvil-pr | _anvil-pr | anvil-pr-fast \ - | anvil-scheduled | _anvil-scheduled | anvil-scheduled-advisories \ - | anvil-full | _anvil-full) return 0 ;; - *) return 1 ;; - esac -} - -version_at_least() { - local found="${1%%[-+]*}" - local required="${2%%[-+]*}" - local found_major found_minor found_patch found_extra - local required_major required_minor required_patch required_extra - IFS=. read -r found_major found_minor found_patch found_extra <<<"$found" - IFS=. read -r required_major required_minor required_patch required_extra <<<"$required" - found_patch="${found_patch:-0}" - required_patch="${required_patch:-0}" - for component in \ - "$found_major" "$found_minor" "$found_patch" \ - "$required_major" "$required_minor" "$required_patch" - do - case "$component" in - '' | *[!0-9]*) return 2 ;; - esac - done - if ((found_major != required_major)); then ((found_major > required_major)); return; fi - if ((found_minor != required_minor)); then ((found_minor > required_minor)); return; fi - ((found_patch >= required_patch)) -} - -command -v docker >/dev/null 2>&1 || { - echo "anvil-container: Docker Engine is required. See .anvil/container/README.md." >&2 - exit 1 -} - -version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || { - echo "anvil-container: Docker Engine is unavailable. Start the Docker service and ensure the current user can access it." >&2 - exit 1 -} -minimum="23.0.0" -if ! version_at_least "$version" "$minimum"; then - echo "anvil-container: Docker Engine $minimum or newer is required (found $version)." >&2 - exit 1 -fi -host_arch="$(uname -m 2>/dev/null || true)" -case "$host_arch" in - x86_64 | amd64 | '') ;; - *) echo "anvil-container: warning: $host_arch requires emulation for linux/amd64; builds and checks may be substantially slower." >&2 ;; -esac - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi -script_dir="$repo_root/.anvil/container" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$script_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -image_id="$(bash "$script_dir/image-id.sh")" -image_base="${ANVIL_CONTAINER_IMAGE:-anvil-dev}" -image="${image_base}:${image_id}" -if command -v sha256sum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | sha256sum | cut -c1-12)" -elif command -v shasum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | shasum -a 256 | cut -c1-12)" -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi -target_volume="anvil-target-${repo_id}-${image_id:0:12}" - -needs_github_token=false -for recipe_arg in "$@"; do - if anvil_recipe_needs_github_token "$recipe_arg"; then - needs_github_token=true - break - fi -done -runs_only_github_check=false -if (($# == 1)) && [[ "$1" == "anvil-aprz" ]]; then - runs_only_github_check=true -fi -github_token="" - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -if docker image inspect "$image" >/dev/null 2>&1; then - image_exists=true -else - image_exists=false -fi - -readonly ANVIL_CONTAINER_REPO_ROOT="$repo_root" -readonly ANVIL_CONTAINER_DIR="$script_dir" -readonly ANVIL_CONTAINER_RESOLVED_IMAGE="$image" -readonly ANVIL_CONTAINER_IMAGE_EXISTS="$image_exists" -declare -a ANVIL_CONTAINER_REQUESTED_RECIPES=("$@") -readonly ANVIL_CONTAINER_REQUESTED_RECIPES - -# Customization outputs, initialized before sourcing so a missing customize.sh -# leaves every phase a documented no-op. -ANVIL_CONTAINER_BUILD_ARGS=() -ANVIL_CONTAINER_PREPARE_ARGS=() -ANVIL_CONTAINER_PREPARE_COMMAND=() -ANVIL_CONTAINER_RUN_ARGS=() -ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN="$needs_github_token" -ANVIL_CONTAINER_CLEANUP=: -github_token_file="" -cleanup() { - if [[ -n "$github_token_file" ]]; then rm -f -- "$github_token_file"; fi - "$ANVIL_CONTAINER_CLEANUP" -} -trap cleanup EXIT - -customize_script="$script_dir/customize.sh" -legacy_customize_script="$repo_root/justfiles/anvil/container/customize.sh" -if [[ ! -f "$customize_script" && -f "$legacy_customize_script" ]]; then - echo "anvil-container: warning: ignoring justfiles/anvil/container/customize.sh; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.sh to keep it active." >&2 -fi -if [[ -f "$customize_script" ]]; then - # shellcheck source=/dev/null - source "$customize_script" -fi - -# Bash 3.2 has neither namerefs (the nameref flag on `local`/`declare`, Bash -# 4.3+) nor safe `set -u` expansion of empty-but- -# declared arrays (fixed in Bash 4.4). Elements are passed positionally -# instead of by nameref, and every expansion of a possibly-empty array uses -# the `${arr[@]+"${arr[@]}"}` idiom: unset/empty-under-old-Bash arrays vanish -# entirely instead of raising "unbound variable", while non-empty arrays -# still expand element-for-element. -anvil_container_validate_array() { - local name="$1" - shift - local declaration value - declaration="$(declare -p "$name" 2>/dev/null || true)" - if [[ ! "$declaration" =~ ^declare\ -[^[:space:]]*a[^[:space:]]*\ ]]; then - echo "anvil-container: $name must be a string array." >&2 - exit 1 - fi - for value in "$@"; do - if [[ -z "$value" ]]; then - echo "anvil-container: $name entries must be non-empty strings." >&2 - exit 1 - fi - done -} -anvil_container_validate_build_args() { - local expect_secret_value=false value - for value in "$@"; do - if "$expect_secret_value"; then - expect_secret_value=false - continue - fi - case "$value" in - --secret) expect_secret_value=true ;; - --secret=*) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' >&2 - exit 1 - ;; - esac - done - if "$expect_secret_value"; then - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS requires a value after --secret.' >&2 - exit 1 - fi -} -anvil_container_validate_array ANVIL_CONTAINER_BUILD_ARGS ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_ARGS ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_COMMAND ${ANVIL_CONTAINER_PREPARE_COMMAND[@]+"${ANVIL_CONTAINER_PREPARE_COMMAND[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_RUN_ARGS ${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"} -anvil_container_validate_build_args ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -case "$ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN" in - true) needs_github_token=true ;; - false) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN must be true or false.' >&2 - exit 1 - ;; -esac -if ((${#ANVIL_CONTAINER_PREPARE_ARGS[@]} > 0)) && ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} == 0)); then - echo 'anvil-container: ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND.' >&2 - exit 1 -fi -cleanup_kind="$(type -t "$ANVIL_CONTAINER_CLEANUP" 2>/dev/null || true)" -if [[ "$cleanup_kind" != "function" && "$cleanup_kind" != "builtin" ]]; then - echo "anvil-container: ANVIL_CONTAINER_CLEANUP must name a callable function (got '$ANVIL_CONTAINER_CLEANUP')." >&2 - exit 1 -fi - -if "$needs_github_token"; then - gh_command="" - if command -v gh >/dev/null 2>&1; then - gh_command=gh - elif command -v gh.exe >/dev/null 2>&1; then - gh_command=gh.exe - fi - github_token="${GITHUB_TOKEN:-}" - if [[ -z "$github_token" && -n "$gh_command" ]]; then - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - fi - if [[ -z "$github_token" ]]; then - if [[ -z "$gh_command" ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - if [[ ! -t 0 ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - echo 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' >&2 - read -r -p 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel) ' - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - if [[ -z "$github_token" ]]; then - echo 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' >&2 - exit 1 - fi - fi -fi - -if ! "$image_exists"; then - if [[ "${ANVIL_CONTAINER_NO_REBUILD:-}" == "1" ]]; then - echo "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." >&2 - exit 1 - else - docker build \ - --platform linux/amd64 \ - --tag "$image" \ - --file "$script_dir/Containerfile" \ - --build-arg "ANVIL_IMAGE_ID=$image_id" \ - --build-arg "BASE_IMAGE=$base_image" \ - ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} \ - "$repo_root" - fi -fi - -container_uid="$(id -u)" -container_gid="$(id -g)" -registry_volume="anvil-cargo-registry-${repo_id}" -git_volume="anvil-cargo-git-${repo_id}" -for volume in "$registry_volume" "$git_volume" "$target_volume"; do - docker volume create "$volume" >/dev/null -done -mount_args=( - --mount "type=bind,source=$repo_root,target=/workspace" - --mount "type=volume,source=$registry_volume,target=/usr/local/cargo/registry" - --mount "type=volume,source=$git_volume,target=/usr/local/cargo/git" - --mount "type=volume,source=$target_volume,target=/workspace/target" -) -docker run --rm --pull=never \ - --platform linux/amd64 \ - --user 0:0 \ - "${mount_args[@]}" \ - "$image" sh -c \ - "chown $container_uid:$container_gid /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - -run_args=( - run --rm --pull=never - --platform linux/amd64 - --user "$container_uid:$container_gid" - --env ANVIL_IN_CONTAINER=1 - --env HOME=/tmp/anvil-user - "${mount_args[@]}" - --workdir /workspace -) -prepare_run_args=("${run_args[@]}") -run_args+=(${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"}) -for name in PR_TITLE BASE_REF ANVIL_INCLUDE_MODIFIED ANVIL_INCLUDE_AFFECTED ANVIL_INCLUDE_REQUIRED GITHUB_BASE_REF SYSTEM_PULLREQUEST_TARGETBRANCH; do - if value="$(printenv "$name")"; then run_args+=(--env "$name=$value"); fi -done -if ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} > 0)); then - docker "${prepare_run_args[@]}" \ - ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} \ - "$image" \ - "${ANVIL_CONTAINER_PREPARE_COMMAND[@]}" -fi - -if [[ -n "$github_token" ]]; then - github_token_file="$(mktemp "${TMPDIR:-/tmp}/anvil-github-token.XXXXXXXX")" - chmod 600 "$github_token_file" - printf '%s' "$github_token" > "$github_token_file" - unset github_token - github_run_args=( - "${run_args[@]}" - --mount "type=bind,source=$github_token_file,target=/run/secrets/anvil-github-token,readonly" - ) - if "$runs_only_github_check"; then - run_args=("${github_run_args[@]}") - else - docker "${github_run_args[@]}" "$image" just anvil-aprz - run_args+=(--env ANVIL_APRZ_ALREADY_RAN=1) - fi -fi - -if (($# == 0)); then - docker "${run_args[@]}" --interactive --tty "$image" bash - exit $? -fi -docker "${run_args[@]}" "$image" just "$@" diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile new file mode 100644 index 00000000..e1be2998 --- /dev/null +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -0,0 +1,115 @@ +# syntax=docker/dockerfile:1 +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# Default Anvil execution image, emitted for every repository. The image +# installs exactly the tools the generated catalog pins, by running +# `just anvil-setup` -- the same recipe the checks themselves use. That is what +# makes "the image has the right tools" true by construction rather than by +# convention: there is no second list to keep in step. +# +# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the +# image identity hash and refuses a floating tag, because a tag that can change +# underneath a hash makes the hash a lie. +# +# To build on a different base (a lower glibc baseline, or an internal +# distribution), a downstream catalog replaces this artifact wholesale via +# `replace_artifact(artifacts::container::dockerfile(...))`; a single +# repository can edit this file in place, which anvil's drift handling +# preserves. + +ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +FROM ${BASE_IMAGE} + +ARG JUST_VERSION=1.56.0 +ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 +ARG POWERSHELL_VERSION=7.6.3 +ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 +ARG RUSTUP_VERSION=1.29.0 +ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e + +ENV DEBIAN_FRONTEND=noninteractive \ + CARGO_HOME=/usr/local/cargo \ + RUSTUP_HOME=/usr/local/rustup \ + RUSTUP_NO_UPDATE_CHECK=1 \ + PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare slim base has no C runtime development files, so every +# link step fails without build-essential. +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential ca-certificates clang libclang-dev curl git libicu-dev \ + libssl-dev pkg-config tar \ + && rm -rf /var/lib/apt/lists/* + +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. +RUN curl -fsSLo /tmp/powershell.tar.gz \ + "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ + && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ + && mkdir -p /opt/microsoft/powershell/7 \ + && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ + && chmod 755 /opt/microsoft/powershell/7/pwsh \ + && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ + && rm /tmp/powershell.tar.gz + +RUN curl -fsSLo /tmp/just.tar.gz \ + "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ + && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ + && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ + && chmod 755 /usr/local/bin/just \ + && rm /tmp/just.tar.gz + +RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ + "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ + && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ + && chmod 755 /tmp/rustup-init \ + && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ + && rm /tmp/rustup-init + +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated +# source edit does not invalidate this layer. The synthetic Justfile avoids +# pulling in repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. +WORKDIR /opt/anvil +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# Consumed by the re-entry guard in the generated tier and group recipes: a +# recipe that sees this runs natively instead of launching another container. +ENV ANVIL_IN_CONTAINER=1 + +WORKDIR /workspace +CMD ["bash"] diff --git a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore new file mode 100644 index 00000000..21f90464 --- /dev/null +++ b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore @@ -0,0 +1,15 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# BuildKit reads `.dockerignore` in preference to a root +# `.dockerignore`, so this scopes the exec-image build context without the +# repository having to own a root ignore file or having one silently overridden. +# +# The build context is the repository root but the image only needs two things. +# Excluding everything else keeps a cold build from streaming the whole +# worktree (and every stale `target/`) to the daemon. +* +!justfiles +!rust-toolchain.toml diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just index 1627c3a5..bb084078 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just @@ -9,11 +9,11 @@ # cargo-aprz queries the GitHub advisory API. Unauthenticated access is # capped at 60 requests/hour and fails on a full run; an authenticated # token raises the cap to 5000/hour. CI injects GITHUB_TOKEN -# (github.token). Container drivers mount an existing host GITHUB_TOKEN -# or the host gh CLI's stored token as a temporary read-only secret. -# Native runs borrow the gh CLI token directly. Native runs warn and -# proceed unauthenticated if neither is available; container runs fail -# before cargo-aprz can exhaust the unauthenticated rate limit. +# (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the +# gh CLI's stored token (non-interactive: `gh auth token` prints the +# active account's token for github.com and never opens a browser/auth +# prompt). If neither is available we warn with instructions and proceed +# unauthenticated. # # Unscoped (consults external risk DB). @@ -21,24 +21,14 @@ [script("pwsh", "-NoProfile")] anvil-aprz: anvil-aprz-validate-prereqs $ErrorActionPreference = 'Stop' - if ($env:ANVIL_APRZ_ALREADY_RAN -eq '1') { - Write-Host 'anvil-aprz: already completed in an isolated authenticated container' - exit 0 - } if (-not $env:GITHUB_TOKEN) { $tok = $null - $containerTokenFile = '/run/secrets/anvil-github-token' - if ($env:ANVIL_IN_CONTAINER -and (Test-Path -LiteralPath $containerTokenFile -PathType Leaf)) { - try { $tok = Get-Content -LiteralPath $containerTokenFile -Raw } catch { $tok = $null } - } elseif (Get-Command gh -ErrorAction SilentlyContinue) { + if (Get-Command gh -ErrorAction SilentlyContinue) { try { $tok = (gh auth token --hostname github.com 2>$null) } catch { $tok = $null } } if ($tok) { $env:GITHUB_TOKEN = $tok.Trim() } else { - if ($env:ANVIL_IN_CONTAINER) { - throw 'anvil-aprz: GitHub authentication is unavailable. Run `gh auth login` on the host or set host GITHUB_TOKEN, then re-run the container command.' - } Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 86508b25..5a1cdab7 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -2,24 +2,332 @@ # Licensed under the MIT License. # GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. +# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md +# +# Containerized execution. `just anvil-container ` runs any anvil +# recipe inside a pinned Linux image; everything else keeps running natively. +# There is no configuration file and no transparent routing: the container is +# reached through this recipe or not at all. +# +# The image tag *is* a hash of the inputs that define it, so the presence of a +# tag is proof that its contents are current -- a changed tool pin names a tag +# that cannot already exist, and a build follows. There is nothing to keep in +# sync and no staleness to detect. +# +# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# Run any Anvil recipe in the pinned local Linux container. With no recipe, -# open an interactive shell. -[windows] +# The container engine. `docker` (supported) or `podman` (best-effort). +# A host property, never committed: set the variable in your environment, or +# pass `just anvil_container_engine=podman ...` for a single invocation. +anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") + +# Where the repository is mounted inside the container. +anvil_container_workdir := "/workspace" + +# Image and cache-volume prefix, derived from the repository directory so two +# repositories on one host cannot collide. Sanitized to the character set +# container image references allow. +anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") + +# Resolve the engine. There is deliberately no probe: presence is not +# reachability, `podman-docker` aliases `docker` onto podman, and a silent +# choice between two installed engines means two image stores and an +# unexplained rebuild. We check that the requested binary exists and let every +# other failure surface the engine's own diagnostic, which is more accurate +# than anything repeated here. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-engine: + $ErrorActionPreference = 'Stop' + $engine = '{{anvil_container_engine}}' + if ($engine -ne 'docker' -and $engine -ne 'podman') { + Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" + exit 1 + } + if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { + Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + } + Write-Output $engine + +# Resolve the exec image reference, building it if it is not already present. +# +# The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its +# ignore file, the pinned toolchain, the optional hook, and the generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes. Only this driver is excluded, since hashing it would make the +# tag depend on the tag. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds +# a tag that already resolves, for the cases a content hash cannot see: a moved +# upstream package, or a base layer that changed behind its digest. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { + # The hook decides what the build installs, so its content defines the + # image as surely as the Dockerfile does. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $inputs += $hookRel + } + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } + } + } + + # Hash a tagged stream rather than raw concatenation, so no rearrangement of + # names and contents can collide. Line endings are normalized once, here, so + # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: + # `Sort-Object -Unique` compares case-insensitively, which would silently drop + # one of two inputs differing only in case on the case-sensitive filesystem + # where the image is actually built. + $stream = [System.Text.StringBuilder]::new() + $ordered = [System.Collections.Generic.SortedSet[string]]::new( + [string[]]$inputs, [System.StringComparer]::Ordinal) + foreach ($rel in $ordered) { + $path = Join-Path $repoRoot $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $rel" + exit 1 + } + $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" + [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + } + $digest = [System.Security.Cryptography.SHA256]::HashData( + [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # 16 hex characters (64 bits) is far past any practical collision risk for a + # local image set, and keeps `docker images` readable. + $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) + $image = '{{anvil_container_name}}:' + $imageId + + if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { + & $engine image inspect $image *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output $image + exit 0 + } + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 + } + } + + # Build-time credentials come from the optional hook, never from a committed + # file. Values are handed to BuildKit by environment variable name, so they + # stay out of the host's process command line, and BuildKit keeps them out of + # every image layer. An empty value is fatal: BuildKit would mount an empty + # secret, the build would install a reduced tool set and exit 0, and the + # result would be tagged with the same hash a credentialed build produces -- + # so every later run would reuse the broken image. + $secretArgs = @() + $secretEnv = @() + $hookPath = Join-Path $repoRoot $hookRel + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + $hook = Anvil-PreBuild + if ($null -ne $hook -and $null -ne $hook.Secrets) { + foreach ($id in $hook.Secrets.Keys) { + if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 + } + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" + } + [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + } + } + } + + try { + # Progress goes to stderr: callers capture this recipe's stdout to learn + # the image reference, so anything else written there becomes part of it. + [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # Pinned, not inferred from the host. The Dockerfile installs amd64 + # toolchains and verifies amd64 checksums, so an arm host would resolve + # the multi-arch base to arm64 and fail late with an exec-format error. + # It also keeps the identity scheme honest: without this, two hosts of + # different architecture compute the same tag for different images. + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } + foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } + $buildCmd += $repoRoot + # BuildKit is required for --secret; docker enables it by default from + # 23.0 but an older daemon silently ignores the flag, so ask explicitly. + $env:DOCKER_BUILDKIT = '1' + & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + } finally { + foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + + Write-Output $image + +# Run any anvil recipe inside the pinned Linux image. +# +# just anvil-container anvil-clippy # one check +# just anvil-container anvil-pr # the whole PR tier +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs +# natively and the work happens exactly once. [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *recipe: - $requested = @('{{ replace(recipe, "'", "''") }}' -split '\s+' | Where-Object { $_ }) - & '.anvil/container/run-in-container.ps1' @requested - exit $LASTEXITCODE +anvil-container *target: + $ErrorActionPreference = 'Stop' + $target = '{{target}}' + if ($env:ANVIL_IN_CONTAINER -eq '1') { + # Already inside: pass straight through instead of nesting. + if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + exit $LASTEXITCODE + } -[unix] + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + + # Map the caller's working directory to its in-container equivalent so + # relative paths keep working from a subdirectory. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { + '{{anvil_container_workdir}}' + } else { + '{{anvil_container_workdir}}/' + $rel + } + + $interactive = [string]::IsNullOrWhiteSpace($target) + $runArgs = @('run', '--rm', '--platform', 'linux/amd64') + $runArgs += $interactive ? '-it' : '-i' + $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + # Cargo and rustup homes live in named volumes: the hot write path never + # crosses the host boundary, and the host's own toolchain is untouched. + $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') + $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Match the caller's uid/gid on Linux. Without this everything the run + # writes under the bind mount -- target/, generated files -- lands as root + # on the host, and the next native cargo build or git clean fails with + # EACCES a long way from the cause. Docker Desktop on Windows and macOS + # already maps ownership, and `id` is not there to ask. + if (-not $IsWindows -and -not $IsMacOS) { + $hostUid = (id -u); $hostGid = (id -g) + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + } + $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') + + # Run-time credentials come from the optional hook. Forwarded by NAME, never + # as NAME=VALUE: the engine copies the value out of the environment it + # already inherits, so a credential never appears in the host's process + # command line, where endpoint telemetry records and retains it for far + # longer than a short-lived token is meant to live. + $hookEnv = @() + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + $hook = Anvil-PreRun + if ($null -ne $hook -and $null -ne $hook.Env) { + foreach ($name in $hook.Env.Keys) { + if ([string]::IsNullOrEmpty($hook.Env[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 + } + Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] + $hookEnv += $name + $runArgs += @('-e', $name) + } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + } + } + } + + try { + # --pull=never: the tag names locally-built content, so a miss is a bug + # to surface rather than an invitation to fetch something unrelated. + $runArgs += @('--pull=never', '-w', $containerCwd, $image) + if (-not $interactive) { $runArgs += @('just', $target) } + & $engine @runArgs + exit $LASTEXITCODE + } finally { + foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + +# Report the engine, the exec image, and whether it is present and current. +# +# The tag embeds the hash of the image's inputs, so "absent" and "out of date" +# are the same condition and are reported as one. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-status: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "engine: $engine" + Write-Output "workdir: {{anvil_container_workdir}}" + + # NO_REBUILD turns the resolve into a pure query: report the state instead + # of silently spending several minutes building from a status command. + $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 + $present = $LASTEXITCODE -eq 0 + if ($image) { Write-Output "image: $image" } + if ($present) { + Write-Output "status: present and current" + } else { + Write-Output "status: no image matches the current inputs (it will be built on the next run)" + } + exit 0 + +# Rebuild the exec image from scratch, ignoring every cached layer. +# +# The ordinary path already rebuilds whenever an input changes, so this is for +# the cases a content hash cannot see: a moved upstream package, a stale base +# layer, or a build that is suspected of being wrong. [group("anvil-container")] -[script("bash")] -anvil-container *recipe: - requested={{ quote(recipe) }} - if [[ -z "$requested" ]]; then - exec bash '.anvil/container/run-in-container.sh' - fi - read -r -a requested_args <<<"$requested" - exec bash '.anvil/container/run-in-container.sh' "${requested_args[@]}" +[script("pwsh", "-NoProfile")] +anvil-container-rebuild: + $ErrorActionPreference = 'Stop' + $env:ANVIL_CONTAINER_NO_CACHE = '1' + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image rebuilt: $image" + exit 0 + +# Remove this repository's cache volumes. The image is left in place. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-down: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + & $engine volume rm -f $vol + } + exit 0 diff --git a/crates/cargo-anvil/templates/justfiles/anvil/mod.just b/crates/cargo-anvil/templates/justfiles/anvil/mod.just index 710bead6..14f99095 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/mod.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/mod.just @@ -76,7 +76,6 @@ import 'groups/scheduled-test.just' import 'groups/scheduled-advisories.just' import 'groups/scheduled-runtime-analysis.just' import 'groups/scheduled-exhaustive.just' -import 'runner.just' import 'tiers.just' import 'tools.just' import 'versions.just' diff --git a/crates/cargo-anvil/templates/justfiles/anvil/runner.just b/crates/cargo-anvil/templates/justfiles/anvil/runner.just deleted file mode 100644 index 8cdd5921..00000000 --- a/crates/cargo-anvil/templates/justfiles/anvil/runner.just +++ /dev/null @@ -1,47 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -# Route public tier entry points through the configured execution environment. -# ANVIL_IN_CONTAINER always wins to prevent recursive container launches. -[private] -[no-exit-message] -[windows] -[script("pwsh", "-NoProfile")] -_anvil-run tier runner: - $just = '{{ replace(just_executable(), "'", "''") }}' - $justfile = '{{ replace(justfile(), "'", "''") }}' - $nativeTier = '_anvil-{{ replace(tier, "'", "''") }}' - if ($env:ANVIL_IN_CONTAINER) { - & $just --justfile $justfile $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'container') { - & $just --justfile $justfile anvil-container $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'native') { - & $just --justfile $justfile $nativeTier - } else { - [Console]::Error.WriteLine("anvil-runner: expected 'native' or 'container', got '{{ replace(runner, "'", "''") }}'.") - exit 2 - } - exit $LASTEXITCODE - -[private] -[no-exit-message] -[unix] -[script("bash")] -_anvil-run tier runner: - just_path={{ quote(just_executable()) }} - justfile={{ quote(justfile()) }} - tier={{ quote(tier) }} - runner={{ quote(runner) }} - native_tier="_anvil-$tier" - if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - elif [[ "$runner" == "container" ]]; then - exec "$just_path" --justfile "$justfile" anvil-container "$native_tier" - elif [[ "$runner" == "native" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - else - echo "anvil-runner: expected 'native' or 'container', got '$runner'." >&2 - exit 2 - fi diff --git a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just index 86885967..dd9c0b2c 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just @@ -12,10 +12,7 @@ # Run all pull request checks. [group("anvil")] -anvil-pr: (_anvil-run "pr" anvil_runner) - -[private] -_anvil-pr: anvil-pr-validate-prereqs \ +anvil-pr: anvil-pr-validate-prereqs \ anvil-pr-fast \ anvil-pr-slow @@ -26,10 +23,7 @@ _anvil-pr: anvil-pr-validate-prereqs \ # Run all scheduled checks. [group("anvil")] -anvil-scheduled: (_anvil-run "scheduled" anvil_runner) - -[private] -_anvil-scheduled: anvil-scheduled-validate-prereqs \ +anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-test \ anvil-scheduled-advisories \ anvil-scheduled-runtime-analysis \ @@ -37,12 +31,9 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. [group("anvil")] -anvil-full: (_anvil-run "full" anvil_runner) - -[private] -_anvil-full: anvil-full-validate-prereqs \ - _anvil-pr \ - _anvil-scheduled +anvil-full: anvil-full-validate-prereqs \ + anvil-pr \ + anvil-scheduled # Tier-level + global setup + validate-prereqs # =========================================================================== diff --git a/crates/cargo-anvil/templates/regions/justfile-runner.just b/crates/cargo-anvil/templates/regions/justfile-runner.just deleted file mode 100644 index 64a4fe40..00000000 --- a/crates/cargo-anvil/templates/regions/justfile-runner.just +++ /dev/null @@ -1 +0,0 @@ -anvil_runner := env_var_or_default("ANVIL_RUNNER", "native") diff --git a/crates/cargo-anvil/tests/container_customization.rs b/crates/cargo-anvil/tests/container_customization.rs deleted file mode 100644 index 5cb2e2e9..00000000 --- a/crates/cargo-anvil/tests/container_customization.rs +++ /dev/null @@ -1,754 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -#![cfg(all(windows, not(miri)))] // exercises the real pwsh driver against a fake `wsl`; miri can't sandbox this. -#![allow( - clippy::expect_used, - clippy::unwrap_used, - reason = "panic-on-failure idioms are appropriate in tests" -)] - -//! Driver-level verification of the `customize.ps1` runtime contract from -//! [`containers.md`](../docs/design/containers.md#8-container-customization). -//! -//! Generates the real `.anvil/container/` tree with -//! [`cargo_anvil::test_support::run_update`], then runs the generated -//! `run-in-container.ps1` against a fake `wsl` on `PATH` so the driver's -//! own process, argument construction, and validation execute for real. -//! `anvil-clippy` is used throughout so the GitHub-token path (which would -//! also require a fake `gh`) is never exercised. -//! -//! The Bash mirror lives in `container_customization_bash.rs` and runs on -//! Unix. It cannot run in this Windows test process because `bash` resolves -//! to the WSL launcher, which uses a different filesystem namespace from the -//! generated temporary repository. - -use std::collections::BTreeSet; -use std::path::Path; -use std::process::Command; - -use cargo_anvil::Catalog; -use cargo_anvil::test_support::{Cli, run_update}; -use tempfile::TempDir; - -fn write(path: &Path, contents: &str) { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).unwrap(); - } - std::fs::write(path, contents).unwrap(); -} - -fn local() -> Cli { - Cli { - backends: vec![], - no_backends: true, - dry_run: false, - force: false, - } -} - -/// A repository with the public container tree generated and no derived -/// catalog involved, proving the driver loads `customize.ps1` purely by -/// standard path discovery. -fn repo_with_container() -> TempDir { - let tmp = TempDir::new().unwrap(); - let root = tmp.path(); - write( - &root.join("Cargo.toml"), - "[workspace]\nresolver = \"2\"\nmembers = [\"crates/*\"]\n", - ); - write( - &root.join("crates/alpha/Cargo.toml"), - "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", - ); - write(&root.join("crates/alpha/src/lib.rs"), ""); - write(&root.join("rust-toolchain.toml"), "channel = \"1.93\"\n"); - run_update(&Catalog::anvil(), &local(), root).unwrap(); - assert!( - !root.join(".anvil/container/customize.ps1").exists(), - "the public catalog must not emit customize.ps1 by default" - ); - let status = Command::new("git") - .args(["init", "--quiet"]) - .current_dir(root) - .status() - .expect("git must be available"); - assert!(status.success(), "temporary Git repository must initialize"); - tmp -} - -/// Installs a fake `wsl` on `PATH` so the real Windows driver runs against -/// controllable Docker Engine behavior without depending on the host's WSL -/// configuration. -fn install_fake_wsl(bin_dir: &Path) { - std::fs::create_dir_all(bin_dir).unwrap(); - write( - &bin_dir.join("wsl.cmd"), - "@echo off\r\npwsh -NoProfile -File \"%~dp0wsl.ps1\" %*\r\nexit /b %ERRORLEVEL%\r\n", - ); - write( - &bin_dir.join("wsl.ps1"), - r" -$command = $args[1] -$commandArgs = @($args | Select-Object -Skip 2) -switch ($command) { - 'wslpath' { - if ($env:FAKE_TOKEN_PATH_LOG -and $commandArgs[-1] -like '*anvil-github-token-*') { - Add-Content -LiteralPath $env:FAKE_TOKEN_PATH_LOG -Value $commandArgs[-1] - } - Write-Output $env:FAKE_WSL_REPO_PATH - exit 0 - } - 'id' { - if ($commandArgs[0] -eq '-u') { Write-Output '1000' } else { Write-Output '1000' } - exit 0 - } - 'docker' { - $logPath = $env:FAKE_DOCKER_LOG - if ($logPath) { Add-Content -LiteralPath $logPath -Value ($commandArgs -join ' ') } - $sub = $commandArgs[0] - switch ($sub) { - 'version' { Write-Output '26.1.5'; exit 0 } - 'image' { - if ($env:FAKE_DOCKER_IMAGE_EXISTS -eq '1') { exit 0 } else { exit 1 } - } - 'build' { - exit [int]($(if ($env:FAKE_DOCKER_BUILD_EXIT) { $env:FAKE_DOCKER_BUILD_EXIT } else { '0' })) - } - 'volume' { exit 0 } - 'run' { - $joined = $commandArgs -join ' ' - if ($env:FAKE_DOCKER_FAIL_MARKER -and $joined.Contains($env:FAKE_DOCKER_FAIL_MARKER)) { - exit 1 - } - exit 0 - } - default { exit 0 } - } - } - default { exit 0 } -} -", - ); -} - -struct DriverRun { - status: std::process::ExitStatus, - stderr: String, - docker_log: String, - test_log: String, - token_paths: Vec, -} - -fn assert_token_files_removed(run: &DriverRun) { - assert!(!run.token_paths.is_empty(), "expected a GitHub token path"); - for path in &run.token_paths { - assert!(!path.exists(), "temporary GitHub token file was not removed: {}", path.display()); - } -} - -fn created_volumes(docker_log: &str) -> BTreeSet { - docker_log - .lines() - .filter_map(|line| line.strip_prefix("volume create ")) - .map(str::to_owned) - .collect() -} - -/// Runs the real generated `run-in-container.ps1` against the fake `wsl`, -/// with `customize.ps1` written from `customize_ps1_body` beforehand. -fn run_driver(root: &Path, customize_ps1_body: &str, recipe: &str, env: &[(&str, &str)]) -> DriverRun { - run_driver_args(root, customize_ps1_body, &[recipe], env) -} - -fn run_driver_args(root: &Path, customize_ps1_body: &str, recipe_args: &[&str], env: &[(&str, &str)]) -> DriverRun { - run_driver_maybe_customized(root, Some(customize_ps1_body), recipe_args, env) -} - -/// Runs the driver with no `customize.ps1` at the current location, so the -/// stranded-legacy-file detection is observable. -fn run_driver_without_customization(root: &Path, recipe: &str, env: &[(&str, &str)]) -> DriverRun { - run_driver_maybe_customized(root, None, &[recipe], env) -} - -fn run_driver_maybe_customized(root: &Path, customize_ps1_body: Option<&str>, recipe_args: &[&str], env: &[(&str, &str)]) -> DriverRun { - let customize = root.join(".anvil/container/customize.ps1"); - match customize_ps1_body { - Some(body) => write(&customize, body), - None => drop(std::fs::remove_file(&customize)), - } - - let bin_dir = root.join("fake-bin"); - install_fake_wsl(&bin_dir); - - let docker_log = root.join("docker.log"); - let test_log = root.join("test.log"); - let token_path_log = root.join("token-path.log"); - let _ = std::fs::remove_file(&docker_log); - let _ = std::fs::remove_file(&test_log); - let _ = std::fs::remove_file(&token_path_log); - let fake_wsl_repo = format!( - "/mnt/c/fake/{}", - root.file_name() - .expect("temporary repository must have a directory name") - .to_string_lossy() - ); - let path = format!("{};{}", bin_dir.display(), std::env::var("PATH").unwrap_or_default()); - - let mut command = Command::new("pwsh"); - command - .args(["-NoProfile", "-File", ".anvil/container/run-in-container.ps1"]) - .args(recipe_args) - .current_dir(root) - .env("PATH", path) - .env("FAKE_DOCKER_LOG", &docker_log) - .env("FAKE_TEST_LOG", &test_log) - .env("FAKE_TOKEN_PATH_LOG", &token_path_log) - .env("FAKE_WSL_REPO_PATH", &fake_wsl_repo) - .env_remove("GITHUB_TOKEN") - .env_remove("ANVIL_CONTAINER_BASE_IMAGE") - .env_remove("ANVIL_CONTAINER_IMAGE") - .env_remove("ANVIL_CONTAINER_NO_REBUILD"); - for (key, value) in env { - command.env(key, value); - } - let output = command.output().expect("pwsh must be available to run the driver"); - - DriverRun { - status: output.status, - stderr: String::from_utf8_lossy(&output.stderr).into_owned(), - docker_log: std::fs::read_to_string(&docker_log).unwrap_or_default(), - test_log: std::fs::read_to_string(&test_log).unwrap_or_default(), - token_paths: std::fs::read_to_string(&token_path_log) - .unwrap_or_default() - .lines() - .map(Into::into) - .collect(), - } -} - -#[test] -fn a_customization_file_stranded_at_the_pre_move_path_is_reported_and_not_sourced() { - // Container assets moved from justfiles/anvil/container/ to - // .anvil/container/. A hand-authored customization file is not - // catalog-tracked, so `cargo anvil` cannot relocate it; the driver must - // say so instead of silently running without it. - let tmp = repo_with_container(); - write( - &tmp.path().join("justfiles/anvil/container/customize.ps1"), - "$AnvilContainerRunArgs = @('--label', 'stranded=1')\n", - ); - - let run = run_driver_without_customization(tmp.path(), "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - assert!(run.status.success(), "the run must still proceed: stderr={}", run.stderr); - assert!( - run.stderr.contains("justfiles/anvil/container/customize.ps1") && run.stderr.contains(".anvil/container/customize.ps1"), - "the stranded file and its new home must both be named: stderr={}", - run.stderr - ); - assert!( - !run.docker_log.contains("stranded=1"), - "the stranded file must not be sourced: docker.log={}", - run.docker_log - ); -} - -#[test] -fn a_customization_file_at_the_current_path_wins_without_a_migration_warning() { - let tmp = repo_with_container(); - // A stale copy at the old path must be inert, not a second source. - write( - &tmp.path().join("justfiles/anvil/container/customize.ps1"), - "throw 'the pre-move path must never be sourced'\n", - ); - - let run = run_driver( - tmp.path(), - "$AnvilContainerRunArgs = @('--label', 'current=1')\n", - "anvil-clippy", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1")], - ); - - assert!(run.status.success(), "the run must succeed: stderr={}", run.stderr); - assert!( - !run.stderr.contains("justfiles/anvil/container/customize.ps1"), - "no migration warning is due when the current path is populated: stderr={}", - run.stderr - ); - assert!( - run.docker_log.contains("current=1"), - "the current customization must take effect: docker.log={}", - run.docker_log - ); -} - -#[test] -fn every_requested_recipe_is_checked_for_github_authentication() { - let tmp = repo_with_container(); - let run = run_driver_args( - tmp.path(), - "", - &["anvil-clippy", "anvil-aprz"], - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!( - run.status.success(), - "a later requested recipe must receive GitHub authentication: {}", - run.stderr - ); - assert!( - run.docker_log.lines().any(|line| line.contains("just anvil-clippy anvil-aprz")), - "all arguments must still be forwarded to the requested recipe: {}", - run.docker_log - ); - assert_eq!( - run.docker_log - .lines() - .filter(|line| line.starts_with("run ") && line.contains("just anvil-aprz")) - .count(), - 1, - "a later token-requiring recipe must cause one isolated anvil-aprz invocation" - ); - assert!( - run.docker_log - .lines() - .any(|line| line.contains("--env ANVIL_APRZ_ALREADY_RAN=1") && line.contains("just anvil-clippy anvil-aprz")), - "the requested recipes must run with APRZ marked complete: {}", - run.docker_log - ); -} - -#[test] -fn customization_can_provide_github_authentication() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "$env:GITHUB_TOKEN = 'custom-token'\n", - "anvil-aprz", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1")], - ); - - assert!(run.status.success(), "custom authentication must be accepted: {}", run.stderr); - assert!( - run.docker_log.contains("/run/secrets/anvil-github-token"), - "the customization-provided token must be mounted for APRZ: {}", - run.docker_log - ); - assert_eq!( - run.docker_log - .lines() - .filter(|line| line.starts_with("run ") && line.contains("just anvil-aprz")) - .count(), - 1, - "direct anvil-aprz must run exactly one recipe container: {}", - run.docker_log - ); - assert_token_files_removed(&run); -} - -#[test] -fn customization_can_extend_aprz_classification() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "$AnvilContainerNeedsGitHubToken = $true\n", - "anvil-clippy", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!(run.status.success(), "custom APRZ classification failed: {}", run.stderr); - assert!( - run.docker_log.lines().any(|line| line.contains("just anvil-aprz")), - "custom classification must trigger isolated APRZ: {}", - run.docker_log - ); - assert!( - run.docker_log - .lines() - .any(|line| line.contains("ANVIL_APRZ_ALREADY_RAN=1") && line.contains("just anvil-clippy")), - "the requested recipe must run after APRZ completion: {}", - run.docker_log - ); -} - -#[test] -fn every_requested_argument_must_be_an_anvil_recipe() { - let tmp = repo_with_container(); - let run = run_driver_args(tmp.path(), "", &["anvil-clippy", "not-anvil"], &[]); - - assert!(!run.status.success(), "invalid later recipe must fail"); - assert!( - run.stderr.contains("expected each argument to be an anvil-* recipe"), - "stderr must explain the command contract: {}", - run.stderr - ); - assert!( - run.docker_log.is_empty(), - "validation must happen before Docker: {}", - run.docker_log - ); -} - -#[test] -fn base_image_override_is_digest_pinned_and_passed_to_build() { - let tmp = repo_with_container(); - let base_image = "example.invalid/bullseye@sha256:1111111111111111111111111111111111111111111111111111111111111111"; - let run = run_driver(tmp.path(), "", "anvil-clippy", &[("ANVIL_CONTAINER_BASE_IMAGE", base_image)]); - - assert!(run.status.success(), "digest-pinned override failed: {}", run.stderr); - assert!( - run.docker_log - .lines() - .any(|line| line.starts_with("build ") && line.contains(&format!("BASE_IMAGE={base_image}"))), - "Docker build must receive the selected base image: {}", - run.docker_log - ); - - let invalid = run_driver( - tmp.path(), - "", - "anvil-clippy", - &[("ANVIL_CONTAINER_BASE_IMAGE", "debian:bullseye-slim")], - ); - assert!(!invalid.status.success(), "an unpinned base image must fail"); - assert!(invalid.stderr.contains("must be pinned by sha256 digest")); -} - -#[test] -fn aggregate_recipe_isolates_the_token_from_the_main_container() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "", - "_anvil-pr", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!(run.status.success(), "aggregate recipe failed: {}", run.stderr); - let aprz = run - .docker_log - .lines() - .find(|line| line.contains("just anvil-aprz")) - .expect("aggregate recipe must run isolated APRZ"); - let main = run - .docker_log - .lines() - .find(|line| line.contains("just _anvil-pr")) - .expect("aggregate recipe must run its main container"); - assert!( - aprz.contains("/run/secrets/anvil-github-token"), - "APRZ must receive the token mount: {aprz}" - ); - assert!( - !main.contains("/run/secrets/anvil-github-token"), - "main container must not receive the token mount: {main}" - ); - assert!( - main.contains("ANVIL_APRZ_ALREADY_RAN=1"), - "main container must skip the completed APRZ check: {main}" - ); - assert!( - !run.docker_log.contains("--env GITHUB_TOKEN"), - "the token must never be passed through the environment" - ); - assert_token_files_removed(&run); -} - -#[test] -fn token_file_is_removed_after_aprz_or_main_failure() { - for marker in ["just anvil-aprz", "just _anvil-pr"] { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "", - "_anvil-pr", - &[ - ("FAKE_DOCKER_IMAGE_EXISTS", "1"), - ("GITHUB_TOKEN", "test-token"), - ("FAKE_DOCKER_FAIL_MARKER", marker), - ], - ); - - assert!(!run.status.success(), "failure marker must fail the driver: {marker}"); - assert_token_files_removed(&run); - } -} - -#[test] -fn powershell_just_dispatch_treats_interpolated_values_as_data() { - if Command::new("just").arg("--version").output().is_err() { - return; - } - - let tmp = repo_with_container(); - let root = tmp.path(); - - let runner_output = Command::new("just") - .args(["_anvil-run", "missing", "x') { Write-Output RUNNER_INJECTED } elseif ('a"]) - .current_dir(root) - .output() - .expect("just must be available"); - assert!(!runner_output.status.success(), "the missing native tier must fail"); - assert!( - !String::from_utf8_lossy(&runner_output.stdout).contains("RUNNER_INJECTED"), - "the runner parameter must not execute as PowerShell source" - ); - - write( - &root.join(".anvil/container/run-in-container.ps1"), - "param([Parameter(ValueFromRemainingArguments = $true)][string[]]$Recipe)\nWrite-Output 'DRIVER_OK'\n", - ); - let recipe_output = Command::new("just") - .args(["anvil-container", "x'); Write-Output RECIPE_INJECTED; @('a"]) - .current_dir(root) - .output() - .expect("just must be available"); - assert!( - recipe_output.status.success(), - "the escaped container recipe must reach the driver: {}", - String::from_utf8_lossy(&recipe_output.stderr) - ); - let stdout = String::from_utf8_lossy(&recipe_output.stdout); - assert!(stdout.contains("DRIVER_OK"), "the container driver must run"); - assert!( - !stdout.contains("RECIPE_INJECTED"), - "the recipe parameter must not execute as PowerShell source" - ); -} - -#[test] -fn cold_run_exposes_contract_inputs_scopes_phases_and_runs_cleanup() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r#" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "exists=$AnvilContainerImageExists" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "recipes=$($AnvilContainerRequestedRecipes -join ',')" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "windows=$AnvilContainerHostIsWindows" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "repo-is-dir=$(Test-Path -LiteralPath $AnvilContainerRepoRoot -PathType Container)" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "dir-is-container-dir=$($AnvilContainerDir -eq (Join-Path $AnvilContainerRepoRoot '.anvil/container'))" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "repo-wsl=$AnvilContainerRepoRootWsl" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "dir-wsl=$AnvilContainerDirWsl" -$AnvilContainerBuildArgs = @('--secret', 'id=build-marker,src=fake') -$AnvilContainerRunArgs = @('--label', 'run-marker=1') -$AnvilContainerCleanup = { Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value 'cleanup-ran' } -"#; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!( - run.status.success(), - "cold run must succeed: stderr={}\ndocker.log={}", - run.stderr, - run.docker_log - ); - assert!(run.test_log.contains("exists=False"), "log: {}", run.test_log); - assert!(run.test_log.contains("recipes=anvil-clippy"), "log: {}", run.test_log); - assert!(run.test_log.contains("windows=True"), "log: {}", run.test_log); - assert!(run.test_log.contains("repo-is-dir=True"), "log: {}", run.test_log); - assert!(run.test_log.contains("dir-is-container-dir=True"), "log: {}", run.test_log); - let fake_wsl_repo = format!( - "/mnt/c/fake/{}", - root.file_name() - .expect("temporary repository must have a directory name") - .to_string_lossy() - ); - assert!(run.test_log.contains(&format!("repo-wsl={fake_wsl_repo}")), "log: {}", run.test_log); - assert!( - run.test_log.contains(&format!("dir-wsl={fake_wsl_repo}/.anvil/container")), - "log: {}", - run.test_log - ); - // Build-phase arguments must only appear on the `build` invocation, and - // run-phase arguments only on the `run` invocation: phases stay isolated. - let build_line = run - .docker_log - .lines() - .find(|line| line.starts_with("build ")) - .unwrap_or_else(|| panic!("expected a docker build invocation, got: {}", run.docker_log)); - assert!(build_line.contains("id=build-marker,src=fake"), "line: {build_line}"); - assert!(!build_line.contains("run-marker=1"), "line: {build_line}"); - let run_line = run - .docker_log - .lines() - .find(|line| line.starts_with("run ") && line.contains("just anvil-clippy")) - .unwrap_or_else(|| panic!("expected a docker run invocation, got: {}", run.docker_log)); - assert!(run_line.contains("run-marker=1"), "line: {run_line}"); - assert!(!run_line.contains("id=build-marker,src=fake"), "line: {run_line}"); - assert!( - run_line.contains("--user 1000:1000"), - "recipe execution must use the WSL user: {run_line}" - ); - assert_eq!( - run.docker_log.lines().filter(|line| line.starts_with("volume create ")).count(), - 3, - "the driver must create all named cache volumes: {}", - run.docker_log - ); - assert!( - run.docker_log.contains("volume create anvil-cargo-registry-") && run.docker_log.contains("volume create anvil-cargo-git-"), - "Cargo caches must be repository-scoped: {}", - run.docker_log - ); - assert!( - run.test_log.contains("cleanup-ran"), - "cleanup must run after an ordinary successful invocation: {}", - run.test_log - ); -} - -#[test] -fn cargo_caches_are_repository_scoped_but_stable_across_image_ids() { - let first = repo_with_container(); - let second = repo_with_container(); - let first_run = run_driver(first.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - let second_run = run_driver(second.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - let first_volumes = created_volumes(&first_run.docker_log); - let second_volumes = created_volumes(&second_run.docker_log); - let first_registry = first_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-registry-")) - .expect("first repository must create a registry cache"); - let second_registry = second_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-registry-")) - .expect("second repository must create a registry cache"); - let first_git = first_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-git-")) - .expect("first repository must create a Git cache"); - let second_git = second_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-git-")) - .expect("second repository must create a Git cache"); - assert_ne!(first_registry, second_registry); - assert_ne!(first_git, second_git); - - write(&first.path().join("justfiles/anvil/versions.just"), "changed := \"1\"\n"); - let changed_run = run_driver(first.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - let changed_volumes = created_volumes(&changed_run.docker_log); - assert!(changed_volumes.contains(first_registry)); - assert!(changed_volumes.contains(first_git)); - assert_ne!( - first_volumes.iter().find(|name| name.starts_with("anvil-target-")), - changed_volumes.iter().find(|name| name.starts_with("anvil-target-")) - ); -} - -#[test] -fn warm_run_skips_the_build_and_still_reports_image_exists() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r#" -Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value "exists=$AnvilContainerImageExists" -"#; - let run = run_driver(root, customize, "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - assert!( - run.status.success(), - "warm run must succeed: stderr={}\ndocker.log={}", - run.stderr, - run.docker_log - ); - assert!(run.test_log.contains("exists=True"), "log: {}", run.test_log); - assert!( - !run.docker_log.lines().any(|line| line.starts_with("build ")), - "a warm run (matching image already present) must not invoke docker build: {}", - run.docker_log - ); -} - -#[test] -fn prepare_args_without_a_prepare_command_are_rejected_before_docker_runs() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r" -$AnvilContainerPrepareArgs = @('--label', 'prepare-marker=1') -"; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!(!run.status.success(), "prepare args without a prepare command must fail validation"); - assert!( - run.stderr.contains("AnvilContainerPrepareArgs requires") && run.stderr.contains("AnvilContainerPrepareCommand"), - "stderr must name the invalid output: {}", - run.stderr - ); - assert!( - !run.docker_log - .lines() - .any(|line| line.starts_with("build ") || line.starts_with("run ")), - "validation must fail before any Docker build or run invocation \ - (version/image-exists checks happen earlier and are expected): {}", - run.docker_log - ); -} - -#[test] -fn null_array_output_is_rejected_before_docker_runs() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r" -$AnvilContainerRunArgs = $null -"; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!(!run.status.success(), "a null array output must fail validation"); - assert!( - run.stderr.contains("AnvilContainerRunArgs must be a string array"), - "stderr must name the invalid output: {}", - run.stderr - ); - assert!( - !run.docker_log - .lines() - .any(|line| line.starts_with("build ") || line.starts_with("run ")), - "validation must fail before any Docker build or run invocation \ - (version/image-exists checks happen earlier and are expected): {}", - run.docker_log - ); -} - -#[test] -fn content_changing_build_arguments_are_rejected() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "$AnvilContainerBuildArgs = @('--build-arg', 'BASE_IMAGE=example.invalid/base')\n", - "anvil-clippy", - &[], - ); - - assert!(!run.status.success(), "content-changing build arguments must fail validation"); - assert!( - run.stderr - .contains("AnvilContainerBuildArgs accepts only BuildKit --secret arguments"), - "stderr must explain the image-identity restriction: {}", - run.stderr - ); -} -#[test] -fn cleanup_still_runs_after_the_main_recipe_container_fails() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r" -$AnvilContainerRunArgs = @('--label', 'run-marker=1') -$AnvilContainerCleanup = { Add-Content -LiteralPath $env:FAKE_TEST_LOG -Value 'cleanup-ran' } -"; - let run = run_driver( - root, - customize, - "anvil-clippy", - &[ - ("FAKE_DOCKER_IMAGE_EXISTS", "1"), // warm run: only the main recipe container executes. - ("FAKE_DOCKER_FAIL_MARKER", "run-marker=1"), - ], - ); - - assert!(!run.status.success(), "the driver must surface the recipe failure"); - assert!( - run.test_log.contains("cleanup-ran"), - "cleanup must still run after an ordinary recipe failure: {}", - run.test_log - ); -} diff --git a/crates/cargo-anvil/tests/container_customization_bash.rs b/crates/cargo-anvil/tests/container_customization_bash.rs deleted file mode 100644 index 4c25c572..00000000 --- a/crates/cargo-anvil/tests/container_customization_bash.rs +++ /dev/null @@ -1,722 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -#![cfg(all(unix, not(miri)))] // exercises the real Bash driver against a fake `docker`; miri can't sandbox this. -#![allow( - clippy::expect_used, - clippy::unwrap_used, - reason = "panic-on-failure idioms are appropriate in tests" -)] -#![expect( - clippy::literal_string_with_formatting_args, - reason = "the Bash fixture intentionally contains shell parameter expansions" -)] - -//! Driver-level verification of the `customize.sh` runtime contract from -//! [`containers.md`](../docs/design/containers.md#8-container-customization). -//! -//! This is the Bash mirror of `container_customization.rs`'s `PowerShell` -//! driver tests. It generates the real `.anvil/container/` tree -//! with [`cargo_anvil::test_support::run_update`], then runs the generated -//! `run-in-container.sh` against a fake `docker` on `PATH` so the driver's -//! own process, argument construction, and validation execute for real — -//! including with the default (customize.sh-empty) arrays, which is the -//! condition that regressed under Bash 3.2 / Bash <4.4 `set -u` semantics. -//! `anvil-clippy` is used throughout so the GitHub-token path (which would -//! also require a fake `gh`) is never exercised. - -use std::collections::BTreeSet; -use std::os::unix::fs::PermissionsExt; -use std::path::Path; -use std::process::Command; - -use cargo_anvil::Catalog; -use cargo_anvil::test_support::{Cli, run_update}; -use tempfile::TempDir; - -fn write(path: &Path, contents: &str) { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).unwrap(); - } - std::fs::write(path, contents).unwrap(); -} - -fn write_executable(path: &Path, contents: &str) { - write(path, contents); - let mut permissions = std::fs::metadata(path).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(path, permissions).unwrap(); -} - -fn local() -> Cli { - Cli { - backends: vec![], - no_backends: true, - dry_run: false, - force: false, - } -} - -/// A repository with the public container tree generated and no derived -/// catalog involved, proving the driver loads `customize.sh` purely by -/// standard path discovery. -fn repo_with_container() -> TempDir { - let tmp = TempDir::new().unwrap(); - let root = tmp.path(); - write( - &root.join("Cargo.toml"), - "[workspace]\nresolver = \"2\"\nmembers = [\"crates/*\"]\n", - ); - write( - &root.join("crates/alpha/Cargo.toml"), - "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", - ); - write(&root.join("crates/alpha/src/lib.rs"), ""); - write(&root.join("rust-toolchain.toml"), "channel = \"1.93\"\n"); - run_update(&Catalog::anvil(), &local(), root).unwrap(); - assert!( - !root.join(".anvil/container/customize.sh").exists(), - "the public catalog must not emit customize.sh by default" - ); - let status = Command::new("git") - .args(["init", "--quiet"]) - .current_dir(root) - .status() - .expect("git must be available"); - assert!(status.success(), "temporary Git repository must initialize"); - tmp -} - -/// Installs a fake `docker` on `PATH` so the real driver runs against -/// controllable, observable behavior instead of a real container engine. -fn install_fake_docker(bin_dir: &Path) { - write_executable( - &bin_dir.join("docker"), - r#"#!/usr/bin/env bash -set -euo pipefail -if [[ -n "${FAKE_DOCKER_LOG:-}" ]]; then - printf '%s\n' "$*" >> "$FAKE_DOCKER_LOG" -fi -case "${1:-}" in - version) - echo '26.1.5' - exit 0 - ;; - image) - if [[ "${FAKE_DOCKER_IMAGE_EXISTS:-}" == "1" ]]; then exit 0; else exit 1; fi - ;; - build) - exit "${FAKE_DOCKER_BUILD_EXIT:-0}" - ;; - volume) - exit 0 - ;; - run) - joined="$*" - if [[ -n "${FAKE_DOCKER_FAIL_MARKER:-}" && "$joined" == *"$FAKE_DOCKER_FAIL_MARKER"* ]]; then - exit 1 - fi - exit 0 - ;; - *) - exit 0 - ;; -esac -"#, - ); -} - -struct DriverRun { - status: std::process::ExitStatus, - stderr: String, - docker_log: String, - test_log: String, -} - -fn token_source_paths(docker_log: &str) -> Vec { - docker_log - .lines() - .flat_map(str::split_whitespace) - .filter_map(|argument| { - argument.strip_prefix("type=bind,source=").and_then(|mount| { - mount - .split_once(",target=/run/secrets/anvil-github-token") - .map(|(source, _)| source) - }) - }) - .map(Into::into) - .collect() -} - -fn assert_token_files_removed(run: &DriverRun) { - let paths = token_source_paths(&run.docker_log); - assert!(!paths.is_empty(), "expected a GitHub token mount: {}", run.docker_log); - for path in paths { - assert!(!path.exists(), "temporary GitHub token file was not removed: {}", path.display()); - } -} - -fn created_volumes(docker_log: &str) -> BTreeSet { - docker_log - .lines() - .filter_map(|line| line.strip_prefix("volume create ")) - .map(str::to_owned) - .collect() -} - -/// Runs the real generated `run-in-container.sh` against the fake `docker`, -/// with `customize.sh` written from `customize_sh_body` beforehand. -fn run_driver(root: &Path, customize_sh_body: &str, recipe: &str, env: &[(&str, &str)]) -> DriverRun { - run_driver_args(root, customize_sh_body, &[recipe], env) -} - -fn run_driver_args(root: &Path, customize_sh_body: &str, recipe_args: &[&str], env: &[(&str, &str)]) -> DriverRun { - run_driver_maybe_customized(root, Some(customize_sh_body), recipe_args, env) -} - -/// Runs the driver with no `customize.sh` at the current location, so the -/// stranded-legacy-file detection is observable. -fn run_driver_without_customization(root: &Path, recipe: &str, env: &[(&str, &str)]) -> DriverRun { - run_driver_maybe_customized(root, None, &[recipe], env) -} - -fn run_driver_maybe_customized(root: &Path, customize_sh_body: Option<&str>, recipe_args: &[&str], env: &[(&str, &str)]) -> DriverRun { - let customize = root.join(".anvil/container/customize.sh"); - match customize_sh_body { - Some(body) => write(&customize, body), - None => drop(std::fs::remove_file(&customize)), - } - - let bin_dir = root.join("fake-bin"); - install_fake_docker(&bin_dir); - - let docker_log = root.join("docker.log"); - let test_log = root.join("test.log"); - let _ = std::fs::remove_file(&docker_log); - let _ = std::fs::remove_file(&test_log); - let path = format!("{}:{}", bin_dir.display(), std::env::var("PATH").unwrap_or_default()); - - let mut command = Command::new("bash"); - command - .arg(".anvil/container/run-in-container.sh") - .args(recipe_args) - .current_dir(root) - .env("PATH", path) - .env("FAKE_DOCKER_LOG", &docker_log) - .env("FAKE_TEST_LOG", &test_log) - .env_remove("ANVIL_IN_CONTAINER") - .env_remove("GITHUB_TOKEN") - .env_remove("ANVIL_CONTAINER_BASE_IMAGE") - .env_remove("ANVIL_CONTAINER_IMAGE") - .env_remove("ANVIL_CONTAINER_NO_REBUILD"); - for (key, value) in env { - command.env(key, value); - } - let output = command.output().expect("bash must be available to run the driver"); - - DriverRun { - status: output.status, - stderr: String::from_utf8_lossy(&output.stderr).into_owned(), - docker_log: std::fs::read_to_string(&docker_log).unwrap_or_default(), - test_log: std::fs::read_to_string(&test_log).unwrap_or_default(), - } -} - -#[test] -fn a_customization_file_stranded_at_the_pre_move_path_is_reported_and_not_sourced() { - // Container assets moved from justfiles/anvil/container/ to - // .anvil/container/. A hand-authored customization file is not - // catalog-tracked, so `cargo anvil` cannot relocate it; the driver must - // say so instead of silently running without it. - let tmp = repo_with_container(); - write( - &tmp.path().join("justfiles/anvil/container/customize.sh"), - "ANVIL_CONTAINER_RUN_ARGS=(--label stranded=1)\n", - ); - - let run = run_driver_without_customization(tmp.path(), "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - assert!(run.status.success(), "the run must still proceed: stderr={}", run.stderr); - assert!( - run.stderr.contains("justfiles/anvil/container/customize.sh") && run.stderr.contains(".anvil/container/customize.sh"), - "the stranded file and its new home must both be named: stderr={}", - run.stderr - ); - assert!( - !run.docker_log.contains("stranded=1"), - "the stranded file must not be sourced: docker.log={}", - run.docker_log - ); -} - -#[test] -fn a_customization_file_at_the_current_path_wins_without_a_migration_warning() { - let tmp = repo_with_container(); - // A stale copy at the old path must be inert, not a second source. - write( - &tmp.path().join("justfiles/anvil/container/customize.sh"), - "echo 'the pre-move path must never be sourced' >&2\nexit 1\n", - ); - - let run = run_driver( - tmp.path(), - "ANVIL_CONTAINER_RUN_ARGS=(--label current=1)\n", - "anvil-clippy", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1")], - ); - - assert!(run.status.success(), "the run must succeed: stderr={}", run.stderr); - assert!( - !run.stderr.contains("justfiles/anvil/container/customize.sh"), - "no migration warning is due when the current path is populated: stderr={}", - run.stderr - ); - assert!( - run.docker_log.contains("current=1"), - "the current customization must take effect: docker.log={}", - run.docker_log - ); -} - -#[test] -fn every_requested_recipe_is_checked_for_github_authentication() { - let tmp = repo_with_container(); - let run = run_driver_args( - tmp.path(), - "", - &["anvil-clippy", "anvil-aprz"], - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!( - run.status.success(), - "a later requested recipe must receive GitHub authentication: {}", - run.stderr - ); - assert!( - run.docker_log.lines().any(|line| line.contains("just anvil-clippy anvil-aprz")), - "all arguments must still be forwarded to the requested recipe: {}", - run.docker_log - ); - assert_eq!( - run.docker_log - .lines() - .filter(|line| line.starts_with("run ") && line.contains("just anvil-aprz")) - .count(), - 1, - "a later token-requiring recipe must cause one isolated anvil-aprz invocation" - ); - assert!( - run.docker_log - .lines() - .any(|line| line.contains("--env ANVIL_APRZ_ALREADY_RAN=1") && line.contains("just anvil-clippy anvil-aprz")), - "the requested recipes must run with APRZ marked complete: {}", - run.docker_log - ); -} - -#[test] -fn customization_can_provide_github_authentication() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "GITHUB_TOKEN=custom-token\n", - "anvil-aprz", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1")], - ); - - assert!(run.status.success(), "custom authentication must be accepted: {}", run.stderr); - assert!( - run.docker_log.contains("/run/secrets/anvil-github-token"), - "the customization-provided token must be mounted for APRZ: {}", - run.docker_log - ); - assert_eq!( - run.docker_log - .lines() - .filter(|line| line.starts_with("run ") && line.contains("just anvil-aprz")) - .count(), - 1, - "direct anvil-aprz must run exactly one recipe container: {}", - run.docker_log - ); - assert_token_files_removed(&run); -} - -#[test] -fn customization_can_extend_aprz_classification() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN=true\n", - "anvil-clippy", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!(run.status.success(), "custom APRZ classification failed: {}", run.stderr); - assert!( - run.docker_log.lines().any(|line| line.contains("just anvil-aprz")), - "custom classification must trigger isolated APRZ: {}", - run.docker_log - ); - assert!( - run.docker_log - .lines() - .any(|line| line.contains("ANVIL_APRZ_ALREADY_RAN=1") && line.contains("just anvil-clippy")), - "the requested recipe must run after APRZ completion: {}", - run.docker_log - ); -} - -#[test] -fn every_requested_argument_must_be_an_anvil_recipe() { - let tmp = repo_with_container(); - let run = run_driver_args(tmp.path(), "", &["anvil-clippy", "not-anvil"], &[]); - - assert!(!run.status.success(), "invalid later recipe must fail"); - assert!( - run.stderr.contains("expected each argument to be an anvil-* recipe"), - "stderr must explain the command contract: {}", - run.stderr - ); - assert!( - run.docker_log.is_empty(), - "validation must happen before Docker: {}", - run.docker_log - ); -} - -#[test] -fn base_image_override_is_digest_pinned_and_passed_to_build() { - let tmp = repo_with_container(); - let base_image = "example.invalid/bullseye@sha256:1111111111111111111111111111111111111111111111111111111111111111"; - let run = run_driver(tmp.path(), "", "anvil-clippy", &[("ANVIL_CONTAINER_BASE_IMAGE", base_image)]); - - assert!(run.status.success(), "digest-pinned override failed: {}", run.stderr); - assert!( - run.docker_log - .lines() - .any(|line| line.starts_with("build ") && line.contains(&format!("BASE_IMAGE={base_image}"))), - "Docker build must receive the selected base image: {}", - run.docker_log - ); - - let invalid = run_driver( - tmp.path(), - "", - "anvil-clippy", - &[("ANVIL_CONTAINER_BASE_IMAGE", "debian:bullseye-slim")], - ); - assert!(!invalid.status.success(), "an unpinned base image must fail"); - assert!(invalid.stderr.contains("must be pinned by sha256 digest")); -} - -#[test] -fn aggregate_recipe_isolates_the_token_from_the_main_container() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "", - "_anvil-pr", - &[("FAKE_DOCKER_IMAGE_EXISTS", "1"), ("GITHUB_TOKEN", "test-token")], - ); - - assert!(run.status.success(), "aggregate recipe failed: {}", run.stderr); - let aprz = run - .docker_log - .lines() - .find(|line| line.contains("just anvil-aprz")) - .expect("aggregate recipe must run isolated APRZ"); - let main = run - .docker_log - .lines() - .find(|line| line.contains("just _anvil-pr")) - .expect("aggregate recipe must run its main container"); - assert!( - aprz.contains("/run/secrets/anvil-github-token"), - "APRZ must receive the token mount: {aprz}" - ); - assert!( - !main.contains("/run/secrets/anvil-github-token"), - "main container must not receive the token mount: {main}" - ); - assert!( - main.contains("ANVIL_APRZ_ALREADY_RAN=1"), - "main container must skip the completed APRZ check: {main}" - ); - assert!( - !run.docker_log.contains("--env GITHUB_TOKEN"), - "the token must never be passed through the environment" - ); - assert_token_files_removed(&run); -} - -#[test] -fn token_file_is_removed_after_aprz_or_main_failure() { - for marker in ["just anvil-aprz", "just _anvil-pr"] { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "", - "_anvil-pr", - &[ - ("FAKE_DOCKER_IMAGE_EXISTS", "1"), - ("GITHUB_TOKEN", "test-token"), - ("FAKE_DOCKER_FAIL_MARKER", marker), - ], - ); - - assert!(!run.status.success(), "failure marker must fail the driver: {marker}"); - assert_token_files_removed(&run); - } -} - -#[test] -fn cold_run_with_empty_default_arrays_exposes_contract_inputs_scopes_phases_and_runs_cleanup() { - let tmp = repo_with_container(); - let root = tmp.path(); - // Deliberately leaves ANVIL_CONTAINER_BUILD_ARGS/PREPARE_ARGS/PREPARE_COMMAND/RUN_ARGS - // at their script-provided empty defaults, which is exactly the state - // that broke under Bash 3.2 / Bash <4.4 `set -u` semantics. - let customize = r#" -printf 'exists=%s\n' "$ANVIL_CONTAINER_IMAGE_EXISTS" >> "$FAKE_TEST_LOG" -printf 'recipes=%s\n' "${ANVIL_CONTAINER_REQUESTED_RECIPES[*]}" >> "$FAKE_TEST_LOG" -printf 'repo-is-dir=%s\n' "$([[ -d "$ANVIL_CONTAINER_REPO_ROOT" ]] && echo true || echo false)" >> "$FAKE_TEST_LOG" -printf 'dir-is-container-dir=%s\n' "$([[ "$ANVIL_CONTAINER_DIR" == "$ANVIL_CONTAINER_REPO_ROOT/.anvil/container" ]] && echo true || echo false)" >> "$FAKE_TEST_LOG" -anvil_test_cleanup() { printf 'cleanup-ran\n' >> "$FAKE_TEST_LOG"; } -ANVIL_CONTAINER_CLEANUP=anvil_test_cleanup -"#; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!( - run.status.success(), - "cold run with empty default arrays must succeed: stderr={}\ndocker.log={}", - run.stderr, - run.docker_log - ); - assert!(run.test_log.contains("exists=false"), "log: {}", run.test_log); - assert!(run.test_log.contains("recipes=anvil-clippy"), "log: {}", run.test_log); - assert!(run.test_log.contains("repo-is-dir=true"), "log: {}", run.test_log); - assert!(run.test_log.contains("dir-is-container-dir=true"), "log: {}", run.test_log); - assert!( - run.docker_log.lines().any(|line| line.starts_with("build ")), - "a cold run must invoke docker build: {}", - run.docker_log - ); - assert!( - run.docker_log - .lines() - .any(|line| line.starts_with("run ") && line.contains("just anvil-clippy")), - "expected a docker run invocation, got: {}", - run.docker_log - ); - assert_eq!( - run.docker_log.lines().filter(|line| line.starts_with("volume create ")).count(), - 3, - "the driver must create all named cache volumes: {}", - run.docker_log - ); - assert!( - run.docker_log.contains("volume create anvil-cargo-registry-") && run.docker_log.contains("volume create anvil-cargo-git-"), - "Cargo caches must be repository-scoped: {}", - run.docker_log - ); - let recipe_line = run - .docker_log - .lines() - .find(|line| line.starts_with("run ") && line.contains("just anvil-clippy")) - .expect("the recipe run is asserted present above"); - assert!( - recipe_line.contains("--user ") && !recipe_line.contains("--user 0:0"), - "recipe execution must use the WSL/Linux user identity: {recipe_line}" - ); - assert!( - run.test_log.contains("cleanup-ran"), - "cleanup must run after an ordinary successful invocation: {}", - run.test_log - ); -} - -#[test] -fn cargo_caches_are_repository_scoped_but_stable_across_image_ids() { - let first = repo_with_container(); - let second = repo_with_container(); - let first_run = run_driver(first.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - let second_run = run_driver(second.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - let first_volumes = created_volumes(&first_run.docker_log); - let second_volumes = created_volumes(&second_run.docker_log); - let first_registry = first_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-registry-")) - .expect("first repository must create a registry cache"); - let second_registry = second_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-registry-")) - .expect("second repository must create a registry cache"); - let first_git = first_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-git-")) - .expect("first repository must create a Git cache"); - let second_git = second_volumes - .iter() - .find(|name| name.starts_with("anvil-cargo-git-")) - .expect("second repository must create a Git cache"); - assert_ne!(first_registry, second_registry); - assert_ne!(first_git, second_git); - - write(&first.path().join("justfiles/anvil/versions.just"), "changed := \"1\"\n"); - let changed_run = run_driver(first.path(), "", "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - let changed_volumes = created_volumes(&changed_run.docker_log); - assert!(changed_volumes.contains(first_registry)); - assert!(changed_volumes.contains(first_git)); - assert_ne!( - first_volumes.iter().find(|name| name.starts_with("anvil-target-")), - changed_volumes.iter().find(|name| name.starts_with("anvil-target-")) - ); -} - -#[test] -fn warm_run_skips_the_build_and_still_reports_image_exists() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r#" -printf 'exists=%s\n' "$ANVIL_CONTAINER_IMAGE_EXISTS" >> "$FAKE_TEST_LOG" -"#; - let run = run_driver(root, customize, "anvil-clippy", &[("FAKE_DOCKER_IMAGE_EXISTS", "1")]); - - assert!( - run.status.success(), - "warm run must succeed: stderr={}\ndocker.log={}", - run.stderr, - run.docker_log - ); - assert!(run.test_log.contains("exists=true"), "log: {}", run.test_log); - assert!( - !run.docker_log.lines().any(|line| line.starts_with("build ")), - "a warm run (matching image already present) must not invoke docker build: {}", - run.docker_log - ); -} - -#[test] -fn prepare_args_without_a_prepare_command_are_rejected_before_docker_runs() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r" -ANVIL_CONTAINER_PREPARE_ARGS=(--label 'prepare-marker=1') -"; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!(!run.status.success(), "prepare args without a prepare command must fail validation"); - assert!( - run.stderr - .contains("ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND"), - "stderr must name the invalid output: {}", - run.stderr - ); - assert!( - !run.docker_log - .lines() - .any(|line| line.starts_with("build ") || line.starts_with("run ")), - "validation must fail before any Docker build or run invocation \ - (version/image-exists checks happen earlier and are expected): {}", - run.docker_log - ); -} - -#[test] -fn scalar_output_redeclaration_is_rejected_before_docker_runs() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "unset ANVIL_CONTAINER_RUN_ARGS\nANVIL_CONTAINER_RUN_ARGS=--label\n", - "anvil-clippy", - &[], - ); - - assert!(!run.status.success(), "a scalar output must fail validation"); - assert!( - run.stderr.contains("ANVIL_CONTAINER_RUN_ARGS must be a string array"), - "stderr must name the invalid output: {}", - run.stderr - ); -} - -#[test] -fn content_changing_build_arguments_are_rejected() { - let tmp = repo_with_container(); - let run = run_driver( - tmp.path(), - "ANVIL_CONTAINER_BUILD_ARGS=(--build-arg BASE_IMAGE=example.invalid/base)\n", - "anvil-clippy", - &[], - ); - - assert!(!run.status.success(), "content-changing build arguments must fail validation"); - assert!( - run.stderr - .contains("ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments"), - "stderr must explain the image-identity restriction: {}", - run.stderr - ); -} - -#[test] -fn cleanup_still_runs_after_the_main_recipe_container_fails() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r#" -ANVIL_CONTAINER_RUN_ARGS=(--label 'run-marker=1') -anvil_test_cleanup() { printf 'cleanup-ran\n' >> "$FAKE_TEST_LOG"; } -ANVIL_CONTAINER_CLEANUP=anvil_test_cleanup -"#; - let run = run_driver( - root, - customize, - "anvil-clippy", - &[ - ("FAKE_DOCKER_IMAGE_EXISTS", "1"), // warm run: only the main recipe container executes. - ("FAKE_DOCKER_FAIL_MARKER", "run-marker=1"), - ], - ); - - assert!(!run.status.success(), "the driver must surface the recipe failure"); - assert!( - run.test_log.contains("cleanup-ran"), - "cleanup must still run after an ordinary recipe failure: {}", - run.test_log - ); -} - -#[test] -fn build_and_run_phase_arguments_stay_isolated() { - let tmp = repo_with_container(); - let root = tmp.path(); - let customize = r" -ANVIL_CONTAINER_BUILD_ARGS=(--secret 'id=build-marker,src=fake') -ANVIL_CONTAINER_RUN_ARGS=(--label 'run-marker=1') -"; - let run = run_driver(root, customize, "anvil-clippy", &[]); - - assert!( - run.status.success(), - "cold run must succeed: stderr={}\ndocker.log={}", - run.stderr, - run.docker_log - ); - let build_line = run - .docker_log - .lines() - .find(|line| line.starts_with("build ")) - .unwrap_or_else(|| panic!("expected a docker build invocation, got: {}", run.docker_log)); - assert!(build_line.contains("id=build-marker,src=fake"), "line: {build_line}"); - assert!(!build_line.contains("run-marker=1"), "line: {build_line}"); - let run_line = run - .docker_log - .lines() - .find(|line| line.starts_with("run ") && line.contains("just anvil-clippy")) - .unwrap_or_else(|| panic!("expected a docker run invocation, got: {}", run.docker_log)); - assert!(run_line.contains("run-marker=1"), "line: {run_line}"); - assert!(!run_line.contains("id=build-marker,src=fake"), "line: {run_line}"); -} diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs deleted file mode 100644 index 35d40084..00000000 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ /dev/null @@ -1,204 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -#![cfg(not(miri))] // miri can't sandbox the FS ops these tests do (TempDir, run_update). -#![allow( - clippy::expect_used, - clippy::unwrap_used, - reason = "panic-on-failure idioms are appropriate in tests" -)] -#![expect(clippy::unwrap_used, reason = "integration tests favor concise assertions over Result plumbing")] -#![expect( - clippy::panic, - reason = "integration tests panic on unmet preconditions for readable failure output" -)] - -//! Consumer-upgrade coverage for the container asset relocation. -//! -//! The snapshot tests describe a fresh tree. This exercises the path an -//! existing adopter actually takes: a repository generated before the move, -//! with its `.anvil.lock` and its assets under `justfiles/anvil/container/`, -//! updated by a binary that emits `.anvil/container/`. - -use std::path::Path; - -use cargo_anvil::test_support::{Cli, Decision, Manifest, RunOutcome, Target, run_update}; -use cargo_anvil::{Artifact, Catalog, artifacts}; -use tempfile::TempDir; - -/// A hand-authored customization file: never catalog-tracked, so `cargo anvil` -/// can neither move it nor report it. The drivers warn about one left here. -const LEGACY_CUSTOMIZE: &str = "justfiles/anvil/container/customize.sh"; - -/// Where a generated container asset lived before the move. Every asset, -/// including the entry recipe, sat directly under `justfiles/anvil/container/`. -fn pre_move_path(current: &str) -> String { - let name = current.rsplit('/').next().expect("split always yields one element"); - format!("justfiles/anvil/container/{name}") -} - -fn write(path: &Path, contents: &str) { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent).unwrap(); - } - std::fs::write(path, contents).unwrap(); -} - -fn workspace() -> TempDir { - let tmp = TempDir::new().unwrap(); - let root = tmp.path(); - write( - &root.join("Cargo.toml"), - "[workspace]\nresolver = \"2\"\nmembers = [\"crates/*\"]\n", - ); - write( - &root.join("crates/alpha/Cargo.toml"), - "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", - ); - write(&root.join("crates/alpha/src/lib.rs"), ""); - tmp -} - -fn local() -> Cli { - Cli { - backends: vec![], - no_backends: true, - dry_run: false, - force: false, - } -} - -/// The current container assets, paired with their pre-move locations. -fn container_assets() -> Vec<(String, String)> { - artifacts::container::all() - .into_iter() - .map(|artifact| match artifact { - Artifact::OwnedFile(spec) => (spec.path.to_owned(), pre_move_path(spec.path)), - Artifact::Region(_) => panic!("container artifacts are owned files"), - }) - .collect() -} - -/// Rewrite a freshly generated tree into the shape the previous release -/// produced: every generated container asset under `justfiles/anvil/container/`, -/// tracked at that path by `.anvil.lock`. -fn rewind_to_pre_move_layout(root: &Path) { - let mut manifest = Manifest::load(root).unwrap(); - for (current, previous) in container_assets() { - let to = root.join(&previous); - std::fs::create_dir_all(to.parent().unwrap()).unwrap(); - std::fs::rename(root.join(¤t), &to).unwrap(); - let checksum = manifest - .files - .remove(¤t) - .unwrap_or_else(|| panic!("{current} must be tracked by the fresh lock")); - manifest.files.insert(previous, checksum); - } - // Provenance of the older build. It is recorded, never a gate. - manifest.catalog_checksum = Some("sha256:0000000000000000000000000000000000000000000000000000000000000000".to_owned()); - manifest.tool_version = Some("0.2.0".to_owned()); - manifest.save(root).unwrap(); - std::fs::remove_dir(root.join(".anvil/container")).unwrap(); - std::fs::remove_dir(root.join(".anvil")).unwrap(); -} - -fn decision_for(outcome: &RunOutcome, path: &str) -> Decision { - outcome - .plan - .items() - .iter() - .find(|item| matches!(&item.target, Target::File { path: candidate } if candidate == path)) - .unwrap_or_else(|| panic!("no plan item for {path}")) - .decision -} - -#[test] -fn upgrading_from_the_pre_move_layout_relocates_generated_container_assets() { - let tmp = workspace(); - let root = tmp.path(); - run_update(&Catalog::anvil(), &local(), root).unwrap(); - rewind_to_pre_move_layout(root); - - let assets = container_assets(); - let (customized_current, customized_previous) = assets - .iter() - .find(|(current, _)| current.ends_with("entrypoint.sh")) - .cloned() - .expect("the entry point is part of the container group"); - - // One adopter-edited generated asset, and one hand-authored customization - // file the catalog has never tracked. - let customized_body = "#!/bin/sh\n# locally patched entry point\n"; - write(&root.join(&customized_previous), customized_body); - write(&root.join(LEGACY_CUSTOMIZE), "# hand-authored customization\n"); - - let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); - assert!(outcome.applied); - - let manifest = Manifest::load(root).unwrap(); - for (current, previous) in &assets { - // Every asset is re-emitted at its new location and tracked there. - assert!(root.join(current).is_file(), "{current} must be written at the new location"); - assert_eq!( - decision_for(&outcome, current), - Decision::Write, - "{current} must be freshly written" - ); - assert!(manifest.files.contains_key(current), "{current} must be tracked at the new path"); - assert!(!manifest.files.contains_key(previous), "{previous} must be dropped from the lock"); - - if previous == &customized_previous { - continue; - } - // Untouched old assets are removed outright. - assert!(!root.join(previous).exists(), "untouched orphan {previous} must be removed"); - assert_eq!( - decision_for(&outcome, previous), - Decision::Remove, - "{previous} must be removed as an untouched orphan" - ); - } - - // The adopter's edit survives at the old path, with ownership transferred. - assert_eq!( - std::fs::read_to_string(root.join(&customized_previous)).unwrap(), - customized_body, - "a customized orphan must keep the adopter's content" - ); - assert_eq!( - decision_for(&outcome, &customized_previous), - Decision::OrphanedKept, - "a customized orphan must transfer ownership rather than be deleted" - ); - assert_ne!( - std::fs::read_to_string(root.join(&customized_current)).unwrap(), - customized_body, - "the new location must carry the current template, not the adopter's old edit" - ); - - // The recipe keeps its identity across the move: it is emitted at the new - // path and imported from there. - let recipe = std::fs::read_to_string(root.join("justfiles/anvil/container.just")).unwrap(); - assert!(recipe.contains("anvil-container"), "the entry recipe must survive the move"); - let module = std::fs::read_to_string(root.join("justfiles/anvil/mod.just")).unwrap(); - assert!( - module.contains("import 'container.just'"), - "mod.just must import the flattened recipe:\n{module}" - ); - - // A hand-authored customization file is invisible to the catalog, so the - // update neither moves nor deletes it. The drivers warn at run time. - assert_eq!( - std::fs::read_to_string(root.join(LEGACY_CUSTOMIZE)).unwrap(), - "# hand-authored customization\n", - "an untracked customization file must be left exactly as the adopter wrote it" - ); - - // The migration is complete: a second update is a no-op. - let settled = run_update(&Catalog::anvil(), &local(), root).unwrap(); - assert!( - !settled.plan.has_changes(), - "the upgraded tree must be steady; plan: {:#?}", - settled.plan.items() - ); -} diff --git a/crates/cargo-anvil/tests/extensibility.rs b/crates/cargo-anvil/tests/extensibility.rs index aec499aa..cce55a9c 100644 --- a/crates/cargo-anvil/tests/extensibility.rs +++ b/crates/cargo-anvil/tests/extensibility.rs @@ -26,9 +26,9 @@ use tempfile::TempDir; const EXTRA_FILE: &str = "justfiles/anvil/demoforge.just"; const METADATA_REGION: &str = "demoforge-metadata"; const CONTAINER_JUST: &str = "justfiles/anvil/container.just"; -const CONTAINERFILE: &str = ".anvil/container/Containerfile"; -const CONTAINER_RUNNER: &str = ".anvil/container/run-in-container.ps1"; -const CONTAINER_CUSTOMIZE: &str = ".anvil/container/customize.ps1"; +const DOCKERFILE: &str = ".anvil/container/Dockerfile"; +const DOCKERIGNORE: &str = ".anvil/container/Dockerfile.dockerignore"; +const CONTAINER_HOOKS: &str = ".anvil/container/hooks.ps1"; /// The example downstream catalog: anvil's, customized four ways. fn demoforge() -> Catalog { @@ -55,8 +55,8 @@ fn containerforge() -> Catalog { .subcommand("containerforge") .about("ContainerForge: an anvil container catalog for tests") .version("9.9.9") - .replace_artifact(artifacts::container::containerfile().with_body("FROM example.invalid/base\n")) - .with_artifact(artifacts::container::customize_powershell("# test customization\n")) + .replace_artifact(artifacts::container::dockerfile().with_body("FROM example.invalid/base\n")) + .with_artifact(artifacts::container::hooks("# test credential hook\n")) .build() .unwrap() } @@ -159,14 +159,10 @@ fn public_container_artifacts_can_be_specialized_by_downstream_catalogs() { base_container.contains("anvil-container"), "base catalog must expose the public container command" ); + let base_ignore = std::fs::read_to_string(base.path().join(DOCKERIGNORE)).unwrap(); assert!( - base.path().join(CONTAINER_RUNNER).is_file(), - "base catalog must emit its public runner" - ); - let base_runner = std::fs::read_to_string(base.path().join(CONTAINER_RUNNER)).unwrap(); - assert!( - !base.path().join(CONTAINER_CUSTOMIZE).exists(), - "base catalog must not emit a customization file by default" + !base.path().join(CONTAINER_HOOKS).exists(), + "base catalog must not emit a credential hook by default" ); let configured = workspace(); @@ -176,50 +172,40 @@ fn public_container_artifacts_can_be_specialized_by_downstream_catalogs() { configured_container, base_container, "downstream catalog must inherit the public container command unchanged" ); - let configured_containerfile = std::fs::read_to_string(configured.path().join(CONTAINERFILE)).unwrap(); + let configured_dockerfile = std::fs::read_to_string(configured.path().join(DOCKERFILE)).unwrap(); assert_eq!( - configured_containerfile, "FROM example.invalid/base\n", - "downstream catalog must replace the public Containerfile" + configured_dockerfile, "FROM example.invalid/base\n", + "downstream catalog must replace the public Dockerfile" ); - let configured_runner = std::fs::read_to_string(configured.path().join(CONTAINER_RUNNER)).unwrap(); + let configured_ignore = std::fs::read_to_string(configured.path().join(DOCKERIGNORE)).unwrap(); assert_eq!( - configured_runner, base_runner, - "downstream catalog must inherit the public runner unchanged" + configured_ignore, base_ignore, + "downstream catalog must inherit the public build-context ignore unchanged" ); - let configured_customize = std::fs::read_to_string(configured.path().join(CONTAINER_CUSTOMIZE)).unwrap(); + let configured_hooks = std::fs::read_to_string(configured.path().join(CONTAINER_HOOKS)).unwrap(); assert_eq!( - configured_customize, "# test customization\n", - "downstream catalog must be able to add its own customization file" + configured_hooks, "# test credential hook\n", + "downstream catalog must be able to add its own credential hook" ); } #[test] -fn a_regular_repository_can_add_customization_files_without_a_derived_catalog() { - // A repository maintainer can commit customize.sh/customize.ps1 directly - // beside the generated files, without forking anvil into a derived - // catalog. The public generator neither creates nor manages them, and - // must not disturb them on a later re-run. +fn a_regular_repository_can_add_a_credential_hook_without_a_derived_catalog() { + // A repository maintainer can commit hooks.ps1 directly beside the + // generated files, without forking anvil into a derived catalog. The + // public generator neither creates nor manages it, and must not disturb it + // on a later re-run. let tmp = workspace(); run_update(&Catalog::anvil(), &local(false), tmp.path()).unwrap(); - let shell_customize = tmp.path().join(".anvil/container/customize.sh"); - let powershell_customize = tmp.path().join(CONTAINER_CUSTOMIZE); - assert!(!shell_customize.exists(), "the public catalog must not emit customize.sh"); - assert!(!powershell_customize.exists(), "the public catalog must not emit customize.ps1"); + let hooks = tmp.path().join(CONTAINER_HOOKS); + assert!(!hooks.exists(), "the public catalog must not emit hooks.ps1"); - write(&shell_customize, "# repository-owned shell customization\n"); - write(&powershell_customize, "# repository-owned PowerShell customization\n"); + write(&hooks, "# repository-owned credential hook\n"); - // Re-running the public generator must leave repository-owned - // customization files untouched. + // Re-running the public generator must leave a repository-owned hook + // untouched. run_update(&Catalog::anvil(), &local(false), tmp.path()).unwrap(); - assert_eq!( - std::fs::read_to_string(&shell_customize).unwrap(), - "# repository-owned shell customization\n" - ); - assert_eq!( - std::fs::read_to_string(&powershell_customize).unwrap(), - "# repository-owned PowerShell customization\n" - ); + assert_eq!(std::fs::read_to_string(&hooks).unwrap(), "# repository-owned credential hook\n"); } #[test] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index d0db7681..86174f42 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -2,20 +2,40 @@ source: crates/cargo-anvil/tests/snapshots.rs expression: render_tree(tmp.path()) --- -=== .anvil/container/Containerfile === +=== .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# Default Anvil execution image, emitted for every repository. The image +# installs exactly the tools the generated catalog pins, by running +# `just anvil-setup` -- the same recipe the checks themselves use. That is what +# makes "the image has the right tools" true by construction rather than by +# convention: there is no second list to keep in step. +# +# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the +# image identity hash and refuses a floating tag, because a tag that can change +# underneath a hash makes the hash a lie. +# +# To build on a different base (a lower glibc baseline, or an internal +# distribution), a downstream catalog replaces this artifact wholesale via +# `replace_artifact(artifacts::container::dockerfile(...))`; a single +# repository can edit this file in place, which anvil's drift handling +# preserves. ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e FROM ${BASE_IMAGE} -ARG ANVIL_IMAGE_ID ARG JUST_VERSION=1.56.0 ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 ARG POWERSHELL_VERSION=7.6.3 ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 ARG RUSTUP_VERSION=1.29.0 ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e ENV DEBIAN_FRONTEND=noninteractive \ CARGO_HOME=/usr/local/cargo \ @@ -23,12 +43,17 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare slim base has no C runtime development files, so every +# link step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ libssl-dev pkg-config tar \ && rm -rf /var/lib/apt/lists/* +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. RUN curl -fsSLo /tmp/powershell.tar.gz \ "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ @@ -52,1150 +77,64 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated +# source edit does not invalidate this layer. The synthetic Justfile avoids +# pulling in repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. WORKDIR /opt/anvil -COPY . . -RUN test -f rust-toolchain.toml || { \ - echo "anvil-container requires rust-toolchain.toml" >&2; \ - exit 1; \ - } -RUN --mount=type=cache,id=anvil-cargo-registry,target=/usr/local/cargo/registry \ - --mount=type=cache,id=anvil-cargo-git,target=/usr/local/cargo/git \ - --mount=type=cache,id=anvil-cargo-target,target=/tmp/anvil-target \ - printf "anvil_runner := \"native\"\nimport 'justfiles/anvil/mod.just'\n" > Justfile \ - && CARGO_TARGET_DIR=/tmp/anvil-target just anvil-setup - -COPY .anvil/container/entrypoint.sh /usr/local/bin/anvil-container-entrypoint -RUN chmod 755 /usr/local/bin/anvil-container-entrypoint - +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# Consumed by the re-entry guard in the generated tier and group recipes: a +# recipe that sees this runs natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 -LABEL io.github.cargo-anvil.image-id="${ANVIL_IMAGE_ID}" + WORKDIR /workspace -ENTRYPOINT ["anvil-container-entrypoint"] CMD ["bash"] -=== .anvil/container/Containerfile.dockerignore === -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Deny-all allow-list for the image build context. +=== .anvil/container/Dockerfile.dockerignore === +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. # -# Docker matches each candidate against every pattern in order and lets the -# last match win, testing the path itself *and each of its parent directories* -# (moby/patternmatcher MatchesOrParentMatches). A bare directory re-inclusion -# such as `!justfiles` therefore re-admits the entire subtree below it, which -# would defeat this allow-list, so list only leaf patterns here. Docker still -# descends into a denied directory when some re-inclusion pattern is prefixed -# by it, so the intermediate directories need no entries of their own. +# BuildKit reads `.dockerignore` in preference to a root +# `.dockerignore`, so this scopes the exec-image build context without the +# repository having to own a root ignore file or having one silently overridden. # -# Parent testing also reaches through a single-segment re-inclusion: a -# subdirectory of `.anvil/container/` matches `!.anvil/container/*` in its own -# right. The image-ID helpers list that directory one level deep, so a nested -# file is not an image input; `.anvil/container/*/*` states that leaf-only -# contract in the allow-list too, at every depth, because a deeper candidate -# always has an ancestor of exactly that shape. -** +# The build context is the repository root but the image only needs two things. +# Excluding everything else keeps a cold build from streaming the whole +# worktree (and every stale `target/`) to the daemon. +* +!justfiles !rust-toolchain.toml -!justfiles/anvil/*.just -!justfiles/anvil/checks/*.just -!justfiles/anvil/groups/*.just -!.anvil/container/* -.anvil/container/*/* -.anvil/container/customize.sh -.anvil/container/customize.ps1 - -=== .anvil/container/README.md === - - -# Run Anvil checks in a local container - -Use `just anvil-container` to run generated Anvil checks in a reproducible -Linux environment without installing the complete Rust and Cargo tool catalog -on the host. - -Native execution remains the default. The first container run builds an image -matching the repository's generated configuration. Later runs reuse that image, -dependency caches, and compilation output. - -## Quick start - -Ensure Docker Engine is running, then run: - -```text -just anvil-container anvil-clippy -``` - -The first run builds the matching image and can take several minutes. - -## Prerequisites - -- [Docker Engine](https://docs.docker.com/engine/install/) 23.0 or newer, - installed directly in Linux or WSL and usable by the current user. -- `git` and `just` on the host. -- Bash on Linux and WSL; PowerShell Core (`pwsh`) and WSL 2 on Windows. -- `[script]` support enabled in the root `Justfile`. Add `set unstable` when - required by the installed `just` version. -- A `rust-toolchain.toml` in the repository root. -- A Linux or WSL environment capable of running `linux/amd64` images, either - natively on x86-64 or through Docker emulation on ARM64. - -On Windows, the driver invokes Docker from the default WSL distribution rather -than calling Windows `docker.exe`. Regardless of how Docker is installed, this -command must succeed from PowerShell: - -```text -wsl -e docker version -``` - -Start the Docker service inside WSL when it is stopped and add the WSL user to -the `docker` group when non-root access is not already configured. Docker -Desktop is not required. - -On ARM64 hosts, Docker emulates the required `linux/amd64` environment. Image -builds and checks can therefore be substantially slower than on x86-64 hosts. - -## Security boundary - -> [!WARNING] -> `customize.sh` and `customize.ps1` execute on the host with the developer's -> permissions before container isolation begins. Reviewing and trusting these -> files is equivalent to reviewing and trusting any other host-executed script -> in the checked-out branch. - -## Common workflows - -Run one check: - -```text -just anvil-container anvil-clippy -``` - -Run the complete pull-request tier: - -```text -just anvil-container anvil-pr -``` - -Every argument is treated as a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not supported by this command surface. - -Open an interactive Bash shell in the image: - -```text -just anvil-container -``` - -### Use containers for tier commands - -Native execution remains the default. To route tier commands such as -`just anvil-pr` through the container for the current shell: - -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` - -On Unix: - -```sh -ANVIL_RUNNER=container just anvil-pr -``` - -For one invocation: - -```text -just anvil_runner=container anvil-pr -``` - -To make container execution the repository default, change the default value -in the `anvil-runner` region of the repository-root `Justfile` from `"native"` -to `"container"` and commit that policy. Set `ANVIL_RUNNER=native` to override -the repository default for the current shell. - -Tier routing starts a nested `just` invocation. Output and exit status are -preserved, but outer `--dry-run`, dependency introspection, global options, and -CLI variable assignments are not propagated to the selected private tier. -Values other than `native` and `container` are rejected. - -## Images and caches - -The image name includes a content-based tag derived from the repository's Rust -toolchain, generated Anvil recipes, and container build configuration. A -relevant change selects a new image automatically; older branches can continue -using their matching images. - -The following data is reused between runs: - -- the matching container image; -- repository-scoped Cargo registry and Cargo Git caches; -- compilation output in a repository- and image-specific `target` volume. - -The repository is mounted read/write at `/workspace`. Build output remains in a -named volume instead of the host `target/`, avoiding incompatible artifacts and -slow host-to-virtual-machine I/O. - -## GitHub authentication - -`anvil-aprz` and aggregate tiers that include it require GitHub API -authentication. The driver uses either: - -- the host `GITHUB_TOKEN`; or -- the token from an authenticated host `gh` session. - -Trusted customization can provision a short-lived token by setting -`GITHUB_TOKEN`; the driver reads it after loading and validating customization. - -Authenticate the GitHub CLI with: - -```text -gh auth login --hostname github.com -``` - -For an aggregate tier, the driver first runs `anvil-aprz` in a short-lived -container with the token mounted read-only. After it succeeds, the driver runs -the remaining checks in another container without the token. Temporary token -files are removed afterward. - -An interactive invocation can pause while you authenticate. A non-interactive -invocation fails with instructions when authentication is unavailable. - -## Configuration - -| Variable | Effect | -|---|---| -| `ANVIL_RUNNER` | Selects `native` or `container` execution for tier commands | -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a digest-pinned compatible Linux base image and changes the content-based tag | -| `ANVIL_CONTAINER_IMAGE` | Changes the local image name; the content-based tag is retained | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails instead of building when the matching image is absent | - -The public driver builds images locally and does not pull -`ANVIL_CONTAINER_IMAGE` from a registry. - -The default base is digest-pinned Debian Bookworm. Set -`ANVIL_CONTAINER_BASE_IMAGE` to another image compatible with the generated -Debian-based `Containerfile` when a lower glibc baseline is required. A -different package ecosystem such as Azure Linux requires a derived -`Containerfile`. The value must use `image@sha256:` form so the -selected base remains part of the content-addressed image identity. - -Two simultaneous cold invocations can both build the same missing image. This -is accepted for local development: the content-addressed tag converges on the -same inputs, at the cost of duplicate work. - -## Troubleshooting - -| Problem | Resolution | -|---|---| -| Docker is not found on Linux or WSL | Install Docker Engine 23.0 or newer inside that environment | -| Docker is unavailable from Windows | Run `wsl -e docker version`; install or start Docker Engine in the default WSL distribution | -| Docker requires elevated access | Add the Linux/WSL user to the `docker` group, then start a new shell | -| ARM64 execution is slow | The current image is `linux/amd64` and runs through Docker emulation | -| `linux/amd64` cannot run | Configure Docker to run `linux/amd64` images | -| `[script]` recipes are unavailable | Enable `[script]` support; older `just` versions require `set unstable` | -| `rust-toolchain.toml` is missing | Add the repository-owned toolchain file at the repository root | -| GitHub authentication is unavailable | Run `gh auth login --hostname github.com` or set host `GITHUB_TOKEN` | -| A matching image is missing with `ANVIL_CONTAINER_NO_REBUILD=1` | Unset the variable to allow the local image build | -| The first run is slow | The initial image build installs the pinned tool catalog; later runs reuse it | - -Use `docker images anvil-dev` inside Linux or WSL to list locally cached -default Anvil images. - -## Managed files - -This directory is managed by `cargo-anvil`. Regenerate it with `cargo anvil` -instead of editing its files directly. - -> [!IMPORTANT] -> These assets previously lived in `justfiles/anvil/container/`. `cargo anvil` -> relocates the files it generated, but it does not track a hand-authored -> `customize.sh` or `customize.ps1`. Move any such file to -> `.anvil/container/` yourself; the driver only loads customization from the -> new location and warns on stderr when it finds one left behind. - -## Advanced repository customization - -A repository or derived catalog can add one trusted customization file per -supported host: - -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 -``` - -The driver sources the matching file as trusted host code before authentication, -image construction, and recipe execution. The documented customization -contract provides inputs and validated outputs for APRZ classification, build -secrets, dependency preparation, runtime arguments, and cleanup. - -Customization source is excluded from image identity and the build context. -Non-secret image behavior must be represented by hashed static files such as -the `Containerfile`, entrypoint, or supporting build scripts. - -See the [container customization contract](https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md#8-container-customization) -for the complete interface and security requirements. - -=== .anvil/container/entrypoint.sh === -#!/bin/sh -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -eu - -if [ "$(id -u)" -ne 0 ]; then - if [ -z "${HOME:-}" ] || [ "$HOME" = "/" ]; then - HOME="/tmp/anvil-user" - export HOME - fi - - user_cargo_home="$HOME/.cargo" - mkdir -p "$user_cargo_home" - for file in config.toml .crates.toml .crates2.json; do - if [ -r "$CARGO_HOME/$file" ]; then - cp -f "$CARGO_HOME/$file" "$user_cargo_home/$file" - fi - done - export CARGO_HOME="$user_cargo_home" - ln -sfn /usr/local/cargo/registry "$CARGO_HOME/registry" - ln -sfn /usr/local/cargo/git "$CARGO_HOME/git" -fi - -exec "$@" - -=== .anvil/container/image-id.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$inputs = @( - 'rust-toolchain.toml' -) -$toolchainPath = Join-Path $repoRoot 'rust-toolchain.toml' -if (-not (Test-Path -LiteralPath $toolchainPath -PathType Leaf)) { - throw 'anvil-container requires a repository-owned rust-toolchain.toml.' -} -$containerPath = Join-Path $repoRoot '.anvil/container' -$containerRecipe = 'justfiles/anvil/container.just' -$containerfile = Join-Path $containerPath 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$pathComparison = if ($IsWindows) { [StringComparison]::OrdinalIgnoreCase } else { [StringComparison]::Ordinal } -# The container entry recipe drives execution on the host; it is not image -# content, so it must not participate in image identity. -$inputs += Get-ChildItem (Join-Path $repoRoot 'justfiles/anvil') -Recurse -File -Filter '*.just' | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } | - Where-Object { -not $_.Equals($containerRecipe, $pathComparison) } -$executionOnly = @( - 'image-id.ps1', - 'image-id.sh', - 'README.md', - 'run-in-container.ps1', - 'run-in-container.sh', - 'customize.sh', - 'customize.ps1' -) -# customize.sh/customize.ps1 are trusted runtime orchestration, not image -# content: their source must never affect the image ID or build context. -# Static, non-secret build customization belongs in a hashed artifact instead. -$inputs += Get-ChildItem $containerPath -File | - Where-Object { $_.Name -notin $executionOnly } | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } -$uniqueInputs = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) -foreach ($inputPath in $inputs) { - [void]$uniqueInputs.Add($inputPath) -} -$inputs = [string[]]$uniqueInputs -[Array]::Sort($inputs, [StringComparer]::Ordinal) - -$payload = [Text.StringBuilder]::new() -[void]$payload.Append("ANVIL_CONTAINER_BASE_IMAGE`n").Append($baseImage).Append("`n") -foreach ($relative in $inputs) { - $path = Join-Path $repoRoot $relative - if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { - throw "Container image input is missing: $relative" - } - $content = [IO.File]::ReadAllText($path).Replace("`r`n", "`n").Replace("`r", "`n") - [void]$payload.Append($relative).Append("`n").Append($content).Append("`n") -} - -$bytes = [Text.Encoding]::UTF8.GetBytes($payload.ToString()) -$hash = [Security.Cryptography.SHA256]::HashData($bytes) -Write-Output ([Convert]::ToHexString($hash).ToLowerInvariant()) - -=== .anvil/container/image-id.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi - -toolchain_path="$repo_root/rust-toolchain.toml" -if [[ ! -f "$toolchain_path" ]]; then - echo 'anvil-container requires a repository-owned rust-toolchain.toml.' >&2 - exit 1 -fi - -container_dir="$repo_root/.anvil/container" -container_recipe="justfiles/anvil/container.just" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$container_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -inputs=(rust-toolchain.toml) -while IFS= read -r path; do - relative="${path#"$repo_root"/}" - # The container entry recipe drives execution on the host; it is not - # image content, so it must not participate in image identity. - if [[ "$relative" != "$container_recipe" ]]; then - inputs+=("$relative") - fi -done < <(find "$repo_root/justfiles/anvil" -type f -name '*.just' -print) - -for path in "$container_dir"/*; do - [[ -f "$path" ]] || continue - case "${path##*/}" in - image-id.ps1 | image-id.sh | README.md \ - | run-in-container.ps1 | run-in-container.sh \ - | customize.sh | customize.ps1) continue ;; - esac - inputs+=("${path#"$repo_root"/}") -done - -if command -v sha256sum >/dev/null 2>&1; then - hash_command=(sha256sum) -elif command -v shasum >/dev/null 2>&1; then - hash_command=(shasum -a 256) -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi - -write_normalized_file() { - local path="$1" - local line status - while true; do - line="" - if IFS= read -r line <&3; then - status=0 - else - status=$? - fi - if ((status != 0)) && [[ -z "$line" ]]; then - break - fi - printf '%s' "${line%$'\r'}" - if ((status == 0)); then - printf '\n' - else - break - fi - done 3<"$path" -} - -{ - printf 'ANVIL_CONTAINER_BASE_IMAGE\n%s\n' "$base_image" - while IFS= read -r relative; do - path="$repo_root/$relative" - if [[ ! -f "$path" ]]; then - echo "Container image input is missing: $relative" >&2 - exit 1 - fi - printf '%s\n' "$relative" - write_normalized_file "$path" - printf '\n' - done < <(printf '%s\n' "${inputs[@]}" | LC_ALL=C sort -u) -} | "${hash_command[@]}" | awk '{print $1}' - -=== .anvil/container/run-in-container.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param( - [Parameter(Position = 0, ValueFromRemainingArguments = $true)] - [string[]]$Recipe -) - -$ErrorActionPreference = 'Stop' - -function ConvertTo-AnvilVersion([string]$Value) { - $match = [regex]::Match($Value, '^(\d+)\.(\d+)(?:\.(\d+))?') - if (-not $match.Success) { - throw "anvil-container: could not parse Docker Engine version '$Value'." - } - [version]::new( - [int]$match.Groups[1].Value, - [int]$match.Groups[2].Value, - $(if ($match.Groups[3].Success) { [int]$match.Groups[3].Value } else { 0 }) - ) -} - -function Test-AnvilContainerStringArray([string]$Name, $Value) { - if ($Value -isnot [array]) { - throw "anvil-container: `$$Name must be a string array." - } - foreach ($item in $Value) { - if ($item -isnot [string] -or [string]::IsNullOrEmpty($item)) { - throw "anvil-container: `$$Name entries must be non-empty strings." - } - } -} - -function Test-AnvilContainerBuildArgs($Value) { - for ($index = 0; $index -lt $Value.Count; $index++) { - $item = $Value[$index] - if ($item -eq '--secret') { - $index++ - if ($index -ge $Value.Count) { - throw 'anvil-container: $AnvilContainerBuildArgs requires a value after --secret.' - } - } elseif (-not $item.StartsWith('--secret=', [StringComparison]::Ordinal)) { - throw 'anvil-container: $AnvilContainerBuildArgs accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' - } - } -} - -function Test-AnvilRecipeNeedsGitHubToken([string]$Name) { - $Name -in @( - 'anvil-aprz', - 'anvil-pr', - '_anvil-pr', - 'anvil-pr-fast', - 'anvil-scheduled', - '_anvil-scheduled', - 'anvil-scheduled-advisories', - 'anvil-full', - '_anvil-full' - ) -} - -function Get-AnvilGitHubToken { - $token = $env:GITHUB_TOKEN - if (-not $token -and (Get-Command gh -ErrorAction SilentlyContinue)) { - try { - $token = (& gh auth token --hostname github.com 2>$null) - if ($LASTEXITCODE -ne 0) { $token = $null } - } catch { - $token = $null - } - } - if ($token) { $token = $token.Trim() } - if ($token) { return $token } - return $null -} - -if ($env:ANVIL_IN_CONTAINER) { - if ($Recipe.Count -eq 0) { & bash } else { & just @Recipe } - exit $LASTEXITCODE -} - -foreach ($recipeArg in $Recipe) { - if ($recipeArg -notmatch '^_?anvil-[A-Za-z0-9-]+$') { - throw "anvil-container: expected each argument to be an anvil-* recipe, got '$recipeArg'." - } -} - -if (-not (Get-Command wsl -ErrorAction SilentlyContinue)) { - throw 'anvil-container: WSL 2 is required. See .anvil/container/README.md.' -} - -$versionText = (& wsl -e docker version --format '{{.Server.Version}}' 2>$null) -if ($LASTEXITCODE -ne 0 -or -not $versionText) { - throw 'anvil-container: `wsl -e docker version` must succeed. Install or start Docker Engine in the default WSL distribution; this driver does not invoke Windows docker.exe.' -} -$versionText = $versionText.Trim() -if ((ConvertTo-AnvilVersion $versionText) -lt [version]'23.0.0') { - throw "anvil-container: Docker Engine 23.0.0 or newer is required (found $versionText)." -} -$wslArchitecture = (& wsl -e uname -m 2>$null) -if ($LASTEXITCODE -eq 0 -and $wslArchitecture) { - $wslArchitecture = $wslArchitecture.Trim() - if ($wslArchitecture -notin @('x86_64', 'amd64')) { - [Console]::Error.WriteLine( - "anvil-container: warning: $wslArchitecture requires emulation for linux/amd64; builds and checks may be substantially slower." - ) - } -} - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$scriptDir = Join-Path $repoRoot '.anvil/container' -$wslRepoRoot = (& wsl -e wslpath -a $repoRoot).Trim() -if ($LASTEXITCODE -ne 0 -or -not $wslRepoRoot) { - throw 'anvil-container: could not translate the repository path into the default WSL distribution.' -} -$wslScriptDir = "$wslRepoRoot/.anvil/container" -$containerfile = Join-Path $scriptDir 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$imageId = (& (Join-Path $scriptDir 'image-id.ps1')).Trim() -$imageBase = if ($env:ANVIL_CONTAINER_IMAGE) { $env:ANVIL_CONTAINER_IMAGE } else { 'anvil-dev' } -$image = "${imageBase}:$imageId" -$repoBytes = [Text.Encoding]::UTF8.GetBytes($wslRepoRoot) -$repoHash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($repoBytes)).ToLowerInvariant() -$targetVolume = "anvil-target-$($repoHash.Substring(0, 12))-$($imageId.Substring(0, 12))" - -$needsGitHubToken = $false -foreach ($recipeArg in $Recipe) { - if (Test-AnvilRecipeNeedsGitHubToken $recipeArg) { - $needsGitHubToken = $true - break - } -} -$runsOnlyGitHubCheck = $Recipe.Count -eq 1 -and $Recipe[0] -eq 'anvil-aprz' - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -$null = & wsl -e docker image inspect $image 2>$null -$imageExists = $LASTEXITCODE -eq 0 - -New-Variable -Name AnvilContainerRepoRoot -Value $repoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDir -Value $scriptDir -Option ReadOnly -New-Variable -Name AnvilContainerRepoRootWsl -Value $wslRepoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDirWsl -Value $wslScriptDir -Option ReadOnly -New-Variable -Name AnvilContainerResolvedImage -Value $image -Option ReadOnly -New-Variable -Name AnvilContainerImageExists -Value $imageExists -Option ReadOnly -New-Variable -Name AnvilContainerRequestedRecipes -Value $Recipe -Option ReadOnly -New-Variable -Name AnvilContainerHostIsWindows -Value ([bool]$IsWindows) -Option ReadOnly - -# Customization outputs, initialized before sourcing so a missing customize.ps1 -# leaves every phase a documented no-op. -$AnvilContainerBuildArgs = @() -$AnvilContainerPrepareArgs = @() -$AnvilContainerPrepareCommand = @() -$AnvilContainerRunArgs = @() -$AnvilContainerNeedsGitHubToken = $needsGitHubToken -$AnvilContainerCleanup = $null -$githubToken = $null -$githubTokenFile = $null -$exitCode = 0 -$customizeScript = Join-Path $scriptDir 'customize.ps1' -$legacyCustomizeScript = Join-Path $repoRoot 'justfiles/anvil/container/customize.ps1' - -try { - if (Test-Path -LiteralPath $customizeScript -PathType Leaf) { - . $customizeScript - } - elseif (Test-Path -LiteralPath $legacyCustomizeScript -PathType Leaf) { - [Console]::Error.WriteLine( - 'anvil-container: warning: ignoring justfiles/anvil/container/customize.ps1; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.ps1 to keep it active.' - ) - } - - Test-AnvilContainerStringArray 'AnvilContainerBuildArgs' $AnvilContainerBuildArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareArgs' $AnvilContainerPrepareArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareCommand' $AnvilContainerPrepareCommand - Test-AnvilContainerStringArray 'AnvilContainerRunArgs' $AnvilContainerRunArgs - Test-AnvilContainerBuildArgs $AnvilContainerBuildArgs - if ($AnvilContainerNeedsGitHubToken -isnot [bool]) { - throw 'anvil-container: $AnvilContainerNeedsGitHubToken must be a Boolean.' - } - $needsGitHubToken = $needsGitHubToken -or $AnvilContainerNeedsGitHubToken - if ($AnvilContainerPrepareArgs.Count -gt 0 -and $AnvilContainerPrepareCommand.Count -eq 0) { - throw 'anvil-container: $AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand.' - } - if ($AnvilContainerCleanup -and $AnvilContainerCleanup -isnot [scriptblock]) { - throw 'anvil-container: $AnvilContainerCleanup must be a script block.' - } - $githubToken = if ($needsGitHubToken) { Get-AnvilGitHubToken } else { $null } - if ($needsGitHubToken -and -not $githubToken) { - if (-not (Get-Command gh -ErrorAction SilentlyContinue)) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' - } - if (-not [Environment]::UserInteractive -or [Console]::IsInputRedirected) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' - } - Write-Host 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' - [void](Read-Host 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel)') - $githubToken = Get-AnvilGitHubToken - if (-not $githubToken) { - throw 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' - } - } - if (-not $imageExists) { - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - throw "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." - } - & wsl -e docker build ` - --platform linux/amd64 ` - --tag $image ` - --file "$wslScriptDir/Containerfile" ` - --build-arg "ANVIL_IMAGE_ID=$imageId" ` - --build-arg "BASE_IMAGE=$baseImage" ` - @AnvilContainerBuildArgs ` - $wslRepoRoot - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker build failed with exit code $LASTEXITCODE." - } - } - - $containerUid = (& wsl -e id -u).Trim() - $containerGid = (& wsl -e id -g).Trim() - if ($containerUid -notmatch '^\d+$' -or $containerGid -notmatch '^\d+$') { - throw 'anvil-container: could not determine the default WSL user identity.' - } - $registryVolume = "anvil-cargo-registry-$($repoHash.Substring(0, 12))" - $gitVolume = "anvil-cargo-git-$($repoHash.Substring(0, 12))" - foreach ($volume in @($registryVolume, $gitVolume, $targetVolume)) { - $null = & wsl -e docker volume create $volume - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume creation failed for '$volume' with exit code $LASTEXITCODE." - } - } - $mountArgs = @( - '--mount', "type=bind,source=$wslRepoRoot,target=/workspace", - '--mount', "type=volume,source=$registryVolume,target=/usr/local/cargo/registry", - '--mount', "type=volume,source=$gitVolume,target=/usr/local/cargo/git", - '--mount', "type=volume,source=$targetVolume,target=/workspace/target" - ) - & wsl -e docker run --rm --pull=never ` - --platform linux/amd64 ` - --user 0:0 ` - @mountArgs ` - $image sh -c "chown ${containerUid}:${containerGid} /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume initialization failed with exit code $LASTEXITCODE." - } - - $runArgs = @( - 'run', '--rm', '--pull=never', - '--platform', 'linux/amd64', - '--user', "${containerUid}:${containerGid}", - '--env', 'ANVIL_IN_CONTAINER=1', - '--env', 'HOME=/tmp/anvil-user', - '--workdir', '/workspace' - ) - $runArgs += $mountArgs - $prepareRunArgs = @($runArgs) - $runArgs += $AnvilContainerRunArgs - foreach ($name in @( - 'PR_TITLE', - 'BASE_REF', - 'ANVIL_INCLUDE_MODIFIED', - 'ANVIL_INCLUDE_AFFECTED', - 'ANVIL_INCLUDE_REQUIRED', - 'GITHUB_BASE_REF', - 'SYSTEM_PULLREQUEST_TARGETBRANCH' - )) { - if (Test-Path "Env:$name") { - $runArgs += @('--env', "$name=$((Get-Item "Env:$name").Value)") - } - } - if ($AnvilContainerPrepareCommand.Count -gt 0) { - & wsl -e docker @prepareRunArgs @AnvilContainerPrepareArgs $image @AnvilContainerPrepareCommand - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: preparation command failed with exit code $LASTEXITCODE." - } - } - - if ($githubToken) { - $githubTokenFile = Join-Path ([IO.Path]::GetTempPath()) "anvil-github-token-$PID-$([guid]::NewGuid().ToString('N'))" - [IO.File]::Create($githubTokenFile).Dispose() - if ($IsWindows) { - $userSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value - & icacls.exe $githubTokenFile '/inheritance:r' '/grant:r' "*$($userSid):(F)" | Out-Null - } else { - & chmod 600 $githubTokenFile - } - if ($LASTEXITCODE -ne 0) { - throw 'anvil-container: failed to restrict permissions on the temporary GitHub token file.' - } - [IO.File]::WriteAllText($githubTokenFile, $githubToken, [Text.Encoding]::ASCII) - $githubToken = $null - $wslTokenFile = (& wsl -e wslpath -a $githubTokenFile).Trim() - if ($LASTEXITCODE -ne 0 -or -not $wslTokenFile) { - throw 'anvil-container: could not translate the temporary GitHub token path into WSL.' - } - $githubRunArgs = @($runArgs) - $githubRunArgs += @( - '--mount', - "type=bind,source=$wslTokenFile,target=/run/secrets/anvil-github-token,readonly" - ) - if ($runsOnlyGitHubCheck) { - $runArgs = $githubRunArgs - } else { - & wsl -e docker @githubRunArgs $image just anvil-aprz - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: isolated anvil-aprz failed with exit code $LASTEXITCODE." - } - $runArgs += @('--env', 'ANVIL_APRZ_ALREADY_RAN=1') - } - } - - if ($Recipe.Count -eq 0) { - & wsl -e docker @runArgs --interactive --tty $image bash - } else { - & wsl -e docker @runArgs $image just @Recipe - } - $exitCode = $LASTEXITCODE -} finally { - if ($githubTokenFile) { - Remove-Item -LiteralPath $githubTokenFile -Force -ErrorAction SilentlyContinue - } - if ($AnvilContainerCleanup) { & $AnvilContainerCleanup } -} - -exit $exitCode - -=== .anvil/container/run-in-container.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - if (($# == 0)); then exec bash; else exec just "$@"; fi -fi - -for recipe_arg in "$@"; do - if [[ ! "$recipe_arg" =~ ^_?anvil-[A-Za-z0-9-]+$ ]]; then - echo "anvil-container: expected each argument to be an anvil-* recipe, got '$recipe_arg'." >&2 - exit 2 - fi -done - -anvil_recipe_needs_github_token() { - case "$1" in - anvil-aprz | anvil-pr | _anvil-pr | anvil-pr-fast \ - | anvil-scheduled | _anvil-scheduled | anvil-scheduled-advisories \ - | anvil-full | _anvil-full) return 0 ;; - *) return 1 ;; - esac -} - -version_at_least() { - local found="${1%%[-+]*}" - local required="${2%%[-+]*}" - local found_major found_minor found_patch found_extra - local required_major required_minor required_patch required_extra - IFS=. read -r found_major found_minor found_patch found_extra <<<"$found" - IFS=. read -r required_major required_minor required_patch required_extra <<<"$required" - found_patch="${found_patch:-0}" - required_patch="${required_patch:-0}" - for component in \ - "$found_major" "$found_minor" "$found_patch" \ - "$required_major" "$required_minor" "$required_patch" - do - case "$component" in - '' | *[!0-9]*) return 2 ;; - esac - done - if ((found_major != required_major)); then ((found_major > required_major)); return; fi - if ((found_minor != required_minor)); then ((found_minor > required_minor)); return; fi - ((found_patch >= required_patch)) -} - -command -v docker >/dev/null 2>&1 || { - echo "anvil-container: Docker Engine is required. See .anvil/container/README.md." >&2 - exit 1 -} - -version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || { - echo "anvil-container: Docker Engine is unavailable. Start the Docker service and ensure the current user can access it." >&2 - exit 1 -} -minimum="23.0.0" -if ! version_at_least "$version" "$minimum"; then - echo "anvil-container: Docker Engine $minimum or newer is required (found $version)." >&2 - exit 1 -fi -host_arch="$(uname -m 2>/dev/null || true)" -case "$host_arch" in - x86_64 | amd64 | '') ;; - *) echo "anvil-container: warning: $host_arch requires emulation for linux/amd64; builds and checks may be substantially slower." >&2 ;; -esac - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi -script_dir="$repo_root/.anvil/container" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$script_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -image_id="$(bash "$script_dir/image-id.sh")" -image_base="${ANVIL_CONTAINER_IMAGE:-anvil-dev}" -image="${image_base}:${image_id}" -if command -v sha256sum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | sha256sum | cut -c1-12)" -elif command -v shasum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | shasum -a 256 | cut -c1-12)" -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi -target_volume="anvil-target-${repo_id}-${image_id:0:12}" - -needs_github_token=false -for recipe_arg in "$@"; do - if anvil_recipe_needs_github_token "$recipe_arg"; then - needs_github_token=true - break - fi -done -runs_only_github_check=false -if (($# == 1)) && [[ "$1" == "anvil-aprz" ]]; then - runs_only_github_check=true -fi -github_token="" - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -if docker image inspect "$image" >/dev/null 2>&1; then - image_exists=true -else - image_exists=false -fi - -readonly ANVIL_CONTAINER_REPO_ROOT="$repo_root" -readonly ANVIL_CONTAINER_DIR="$script_dir" -readonly ANVIL_CONTAINER_RESOLVED_IMAGE="$image" -readonly ANVIL_CONTAINER_IMAGE_EXISTS="$image_exists" -declare -a ANVIL_CONTAINER_REQUESTED_RECIPES=("$@") -readonly ANVIL_CONTAINER_REQUESTED_RECIPES - -# Customization outputs, initialized before sourcing so a missing customize.sh -# leaves every phase a documented no-op. -ANVIL_CONTAINER_BUILD_ARGS=() -ANVIL_CONTAINER_PREPARE_ARGS=() -ANVIL_CONTAINER_PREPARE_COMMAND=() -ANVIL_CONTAINER_RUN_ARGS=() -ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN="$needs_github_token" -ANVIL_CONTAINER_CLEANUP=: -github_token_file="" -cleanup() { - if [[ -n "$github_token_file" ]]; then rm -f -- "$github_token_file"; fi - "$ANVIL_CONTAINER_CLEANUP" -} -trap cleanup EXIT - -customize_script="$script_dir/customize.sh" -legacy_customize_script="$repo_root/justfiles/anvil/container/customize.sh" -if [[ ! -f "$customize_script" && -f "$legacy_customize_script" ]]; then - echo "anvil-container: warning: ignoring justfiles/anvil/container/customize.sh; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.sh to keep it active." >&2 -fi -if [[ -f "$customize_script" ]]; then - # shellcheck source=/dev/null - source "$customize_script" -fi - -# Bash 3.2 has neither namerefs (the nameref flag on `local`/`declare`, Bash -# 4.3+) nor safe `set -u` expansion of empty-but- -# declared arrays (fixed in Bash 4.4). Elements are passed positionally -# instead of by nameref, and every expansion of a possibly-empty array uses -# the `${arr[@]+"${arr[@]}"}` idiom: unset/empty-under-old-Bash arrays vanish -# entirely instead of raising "unbound variable", while non-empty arrays -# still expand element-for-element. -anvil_container_validate_array() { - local name="$1" - shift - local declaration value - declaration="$(declare -p "$name" 2>/dev/null || true)" - if [[ ! "$declaration" =~ ^declare\ -[^[:space:]]*a[^[:space:]]*\ ]]; then - echo "anvil-container: $name must be a string array." >&2 - exit 1 - fi - for value in "$@"; do - if [[ -z "$value" ]]; then - echo "anvil-container: $name entries must be non-empty strings." >&2 - exit 1 - fi - done -} -anvil_container_validate_build_args() { - local expect_secret_value=false value - for value in "$@"; do - if "$expect_secret_value"; then - expect_secret_value=false - continue - fi - case "$value" in - --secret) expect_secret_value=true ;; - --secret=*) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' >&2 - exit 1 - ;; - esac - done - if "$expect_secret_value"; then - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS requires a value after --secret.' >&2 - exit 1 - fi -} -anvil_container_validate_array ANVIL_CONTAINER_BUILD_ARGS ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_ARGS ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_COMMAND ${ANVIL_CONTAINER_PREPARE_COMMAND[@]+"${ANVIL_CONTAINER_PREPARE_COMMAND[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_RUN_ARGS ${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"} -anvil_container_validate_build_args ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -case "$ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN" in - true) needs_github_token=true ;; - false) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN must be true or false.' >&2 - exit 1 - ;; -esac -if ((${#ANVIL_CONTAINER_PREPARE_ARGS[@]} > 0)) && ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} == 0)); then - echo 'anvil-container: ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND.' >&2 - exit 1 -fi -cleanup_kind="$(type -t "$ANVIL_CONTAINER_CLEANUP" 2>/dev/null || true)" -if [[ "$cleanup_kind" != "function" && "$cleanup_kind" != "builtin" ]]; then - echo "anvil-container: ANVIL_CONTAINER_CLEANUP must name a callable function (got '$ANVIL_CONTAINER_CLEANUP')." >&2 - exit 1 -fi - -if "$needs_github_token"; then - gh_command="" - if command -v gh >/dev/null 2>&1; then - gh_command=gh - elif command -v gh.exe >/dev/null 2>&1; then - gh_command=gh.exe - fi - github_token="${GITHUB_TOKEN:-}" - if [[ -z "$github_token" && -n "$gh_command" ]]; then - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - fi - if [[ -z "$github_token" ]]; then - if [[ -z "$gh_command" ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - if [[ ! -t 0 ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - echo 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' >&2 - read -r -p 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel) ' - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - if [[ -z "$github_token" ]]; then - echo 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' >&2 - exit 1 - fi - fi -fi - -if ! "$image_exists"; then - if [[ "${ANVIL_CONTAINER_NO_REBUILD:-}" == "1" ]]; then - echo "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." >&2 - exit 1 - else - docker build \ - --platform linux/amd64 \ - --tag "$image" \ - --file "$script_dir/Containerfile" \ - --build-arg "ANVIL_IMAGE_ID=$image_id" \ - --build-arg "BASE_IMAGE=$base_image" \ - ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} \ - "$repo_root" - fi -fi - -container_uid="$(id -u)" -container_gid="$(id -g)" -registry_volume="anvil-cargo-registry-${repo_id}" -git_volume="anvil-cargo-git-${repo_id}" -for volume in "$registry_volume" "$git_volume" "$target_volume"; do - docker volume create "$volume" >/dev/null -done -mount_args=( - --mount "type=bind,source=$repo_root,target=/workspace" - --mount "type=volume,source=$registry_volume,target=/usr/local/cargo/registry" - --mount "type=volume,source=$git_volume,target=/usr/local/cargo/git" - --mount "type=volume,source=$target_volume,target=/workspace/target" -) -docker run --rm --pull=never \ - --platform linux/amd64 \ - --user 0:0 \ - "${mount_args[@]}" \ - "$image" sh -c \ - "chown $container_uid:$container_gid /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - -run_args=( - run --rm --pull=never - --platform linux/amd64 - --user "$container_uid:$container_gid" - --env ANVIL_IN_CONTAINER=1 - --env HOME=/tmp/anvil-user - "${mount_args[@]}" - --workdir /workspace -) -prepare_run_args=("${run_args[@]}") -run_args+=(${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"}) -for name in PR_TITLE BASE_REF ANVIL_INCLUDE_MODIFIED ANVIL_INCLUDE_AFFECTED ANVIL_INCLUDE_REQUIRED GITHUB_BASE_REF SYSTEM_PULLREQUEST_TARGETBRANCH; do - if value="$(printenv "$name")"; then run_args+=(--env "$name=$value"); fi -done -if ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} > 0)); then - docker "${prepare_run_args[@]}" \ - ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} \ - "$image" \ - "${ANVIL_CONTAINER_PREPARE_COMMAND[@]}" -fi - -if [[ -n "$github_token" ]]; then - github_token_file="$(mktemp "${TMPDIR:-/tmp}/anvil-github-token.XXXXXXXX")" - chmod 600 "$github_token_file" - printf '%s' "$github_token" > "$github_token_file" - unset github_token - github_run_args=( - "${run_args[@]}" - --mount "type=bind,source=$github_token_file,target=/run/secrets/anvil-github-token,readonly" - ) - if "$runs_only_github_check"; then - run_args=("${github_run_args[@]}") - else - docker "${github_run_args[@]}" "$image" just anvil-aprz - run_args+=(--env ANVIL_APRZ_ALREADY_RAN=1) - fi -fi - -if (($# == 0)); then - docker "${run_args[@]}" --interactive --tty "$image" bash - exit $? -fi -docker "${run_args[@]}" "$image" just "$@" === .delta.toml === # >>> anvil-managed: anvil-delta @@ -2592,10 +1531,6 @@ clippy.wildcard_imports = "allow" import 'justfiles/anvil/mod.just' # <<< anvil-managed: anvil-imports -# >>> anvil-managed: anvil-runner -anvil_runner := env_var_or_default("ANVIL_RUNNER", "native") -# <<< anvil-managed: anvil-runner - === clippy.toml === # >>> anvil-managed: anvil-clippy # Fine-tuning settings for clippy lints. These cannot be expressed in @@ -2695,11 +1630,11 @@ unknown-git = "deny" # cargo-aprz queries the GitHub advisory API. Unauthenticated access is # capped at 60 requests/hour and fails on a full run; an authenticated # token raises the cap to 5000/hour. CI injects GITHUB_TOKEN -# (github.token). Container drivers mount an existing host GITHUB_TOKEN -# or the host gh CLI's stored token as a temporary read-only secret. -# Native runs borrow the gh CLI token directly. Native runs warn and -# proceed unauthenticated if neither is available; container runs fail -# before cargo-aprz can exhaust the unauthenticated rate limit. +# (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the +# gh CLI's stored token (non-interactive: `gh auth token` prints the +# active account's token for github.com and never opens a browser/auth +# prompt). If neither is available we warn with instructions and proceed +# unauthenticated. # # Unscoped (consults external risk DB). @@ -2707,24 +1642,14 @@ unknown-git = "deny" [script("pwsh", "-NoProfile")] anvil-aprz: anvil-aprz-validate-prereqs $ErrorActionPreference = 'Stop' - if ($env:ANVIL_APRZ_ALREADY_RAN -eq '1') { - Write-Host 'anvil-aprz: already completed in an isolated authenticated container' - exit 0 - } if (-not $env:GITHUB_TOKEN) { $tok = $null - $containerTokenFile = '/run/secrets/anvil-github-token' - if ($env:ANVIL_IN_CONTAINER -and (Test-Path -LiteralPath $containerTokenFile -PathType Leaf)) { - try { $tok = Get-Content -LiteralPath $containerTokenFile -Raw } catch { $tok = $null } - } elseif (Get-Command gh -ErrorAction SilentlyContinue) { + if (Get-Command gh -ErrorAction SilentlyContinue) { try { $tok = (gh auth token --hostname github.com 2>$null) } catch { $tok = $null } } if ($tok) { $env:GITHUB_TOKEN = $tok.Trim() } else { - if ($env:ANVIL_IN_CONTAINER) { - throw 'anvil-aprz: GitHub authentication is unavailable. Run `gh auth login` on the host or set host GITHUB_TOKEN, then re-run the container command.' - } Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' @@ -4552,27 +3477,335 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Licensed under the MIT License. # GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. +# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md +# +# Containerized execution. `just anvil-container ` runs any anvil +# recipe inside a pinned Linux image; everything else keeps running natively. +# There is no configuration file and no transparent routing: the container is +# reached through this recipe or not at all. +# +# The image tag *is* a hash of the inputs that define it, so the presence of a +# tag is proof that its contents are current -- a changed tool pin names a tag +# that cannot already exist, and a build follows. There is nothing to keep in +# sync and no staleness to detect. +# +# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md + +# The container engine. `docker` (supported) or `podman` (best-effort). +# A host property, never committed: set the variable in your environment, or +# pass `just anvil_container_engine=podman ...` for a single invocation. +anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") + +# Where the repository is mounted inside the container. +anvil_container_workdir := "/workspace" + +# Image and cache-volume prefix, derived from the repository directory so two +# repositories on one host cannot collide. Sanitized to the character set +# container image references allow. +anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") + +# Resolve the engine. There is deliberately no probe: presence is not +# reachability, `podman-docker` aliases `docker` onto podman, and a silent +# choice between two installed engines means two image stores and an +# unexplained rebuild. We check that the requested binary exists and let every +# other failure surface the engine's own diagnostic, which is more accurate +# than anything repeated here. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-engine: + $ErrorActionPreference = 'Stop' + $engine = '{{anvil_container_engine}}' + if ($engine -ne 'docker' -and $engine -ne 'podman') { + Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" + exit 1 + } + if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { + Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + } + Write-Output $engine + +# Resolve the exec image reference, building it if it is not already present. +# +# The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its +# ignore file, the pinned toolchain, the optional hook, and the generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes. Only this driver is excluded, since hashing it would make the +# tag depend on the tag. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds +# a tag that already resolves, for the cases a content hash cannot see: a moved +# upstream package, or a base layer that changed behind its digest. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { + # The hook decides what the build installs, so its content defines the + # image as surely as the Dockerfile does. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $inputs += $hookRel + } + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } + } + } + + # Hash a tagged stream rather than raw concatenation, so no rearrangement of + # names and contents can collide. Line endings are normalized once, here, so + # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: + # `Sort-Object -Unique` compares case-insensitively, which would silently drop + # one of two inputs differing only in case on the case-sensitive filesystem + # where the image is actually built. + $stream = [System.Text.StringBuilder]::new() + $ordered = [System.Collections.Generic.SortedSet[string]]::new( + [string[]]$inputs, [System.StringComparer]::Ordinal) + foreach ($rel in $ordered) { + $path = Join-Path $repoRoot $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $rel" + exit 1 + } + $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" + [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + } + $digest = [System.Security.Cryptography.SHA256]::HashData( + [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # 16 hex characters (64 bits) is far past any practical collision risk for a + # local image set, and keeps `docker images` readable. + $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) + $image = '{{anvil_container_name}}:' + $imageId + + if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { + & $engine image inspect $image *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output $image + exit 0 + } + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 + } + } + + # Build-time credentials come from the optional hook, never from a committed + # file. Values are handed to BuildKit by environment variable name, so they + # stay out of the host's process command line, and BuildKit keeps them out of + # every image layer. An empty value is fatal: BuildKit would mount an empty + # secret, the build would install a reduced tool set and exit 0, and the + # result would be tagged with the same hash a credentialed build produces -- + # so every later run would reuse the broken image. + $secretArgs = @() + $secretEnv = @() + $hookPath = Join-Path $repoRoot $hookRel + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + $hook = Anvil-PreBuild + if ($null -ne $hook -and $null -ne $hook.Secrets) { + foreach ($id in $hook.Secrets.Keys) { + if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 + } + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" + } + [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + } + } + } + + try { + # Progress goes to stderr: callers capture this recipe's stdout to learn + # the image reference, so anything else written there becomes part of it. + [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # Pinned, not inferred from the host. The Dockerfile installs amd64 + # toolchains and verifies amd64 checksums, so an arm host would resolve + # the multi-arch base to arm64 and fail late with an exec-format error. + # It also keeps the identity scheme honest: without this, two hosts of + # different architecture compute the same tag for different images. + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } + foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } + $buildCmd += $repoRoot + # BuildKit is required for --secret; docker enables it by default from + # 23.0 but an older daemon silently ignores the flag, so ask explicitly. + $env:DOCKER_BUILDKIT = '1' + & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + } finally { + foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + + Write-Output $image -# Run any Anvil recipe in the pinned local Linux container. With no recipe, -# open an interactive shell. -[windows] +# Run any anvil recipe inside the pinned Linux image. +# +# just anvil-container anvil-clippy # one check +# just anvil-container anvil-pr # the whole PR tier +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs +# natively and the work happens exactly once. [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *recipe: - $requested = @('{{ replace(recipe, "'", "''") }}' -split '\s+' | Where-Object { $_ }) - & '.anvil/container/run-in-container.ps1' @requested - exit $LASTEXITCODE +anvil-container *target: + $ErrorActionPreference = 'Stop' + $target = '{{target}}' + if ($env:ANVIL_IN_CONTAINER -eq '1') { + # Already inside: pass straight through instead of nesting. + if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + exit $LASTEXITCODE + } + + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + + # Map the caller's working directory to its in-container equivalent so + # relative paths keep working from a subdirectory. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { + '{{anvil_container_workdir}}' + } else { + '{{anvil_container_workdir}}/' + $rel + } + + $interactive = [string]::IsNullOrWhiteSpace($target) + $runArgs = @('run', '--rm', '--platform', 'linux/amd64') + $runArgs += $interactive ? '-it' : '-i' + $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + # Cargo and rustup homes live in named volumes: the hot write path never + # crosses the host boundary, and the host's own toolchain is untouched. + $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') + $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Match the caller's uid/gid on Linux. Without this everything the run + # writes under the bind mount -- target/, generated files -- lands as root + # on the host, and the next native cargo build or git clean fails with + # EACCES a long way from the cause. Docker Desktop on Windows and macOS + # already maps ownership, and `id` is not there to ask. + if (-not $IsWindows -and -not $IsMacOS) { + $hostUid = (id -u); $hostGid = (id -g) + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + } + $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') + + # Run-time credentials come from the optional hook. Forwarded by NAME, never + # as NAME=VALUE: the engine copies the value out of the environment it + # already inherits, so a credential never appears in the host's process + # command line, where endpoint telemetry records and retains it for far + # longer than a short-lived token is meant to live. + $hookEnv = @() + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + $hook = Anvil-PreRun + if ($null -ne $hook -and $null -ne $hook.Env) { + foreach ($name in $hook.Env.Keys) { + if ([string]::IsNullOrEmpty($hook.Env[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 + } + Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] + $hookEnv += $name + $runArgs += @('-e', $name) + } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + } + } + } + + try { + # --pull=never: the tag names locally-built content, so a miss is a bug + # to surface rather than an invitation to fetch something unrelated. + $runArgs += @('--pull=never', '-w', $containerCwd, $image) + if (-not $interactive) { $runArgs += @('just', $target) } + & $engine @runArgs + exit $LASTEXITCODE + } finally { + foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } -[unix] +# Report the engine, the exec image, and whether it is present and current. +# +# The tag embeds the hash of the image's inputs, so "absent" and "out of date" +# are the same condition and are reported as one. [group("anvil-container")] -[script("bash")] -anvil-container *recipe: - requested={{ quote(recipe) }} - if [[ -z "$requested" ]]; then - exec bash '.anvil/container/run-in-container.sh' - fi - read -r -a requested_args <<<"$requested" - exec bash '.anvil/container/run-in-container.sh' "${requested_args[@]}" +[script("pwsh", "-NoProfile")] +anvil-container-status: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "engine: $engine" + Write-Output "workdir: {{anvil_container_workdir}}" + + # NO_REBUILD turns the resolve into a pure query: report the state instead + # of silently spending several minutes building from a status command. + $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 + $present = $LASTEXITCODE -eq 0 + if ($image) { Write-Output "image: $image" } + if ($present) { + Write-Output "status: present and current" + } else { + Write-Output "status: no image matches the current inputs (it will be built on the next run)" + } + exit 0 + +# Rebuild the exec image from scratch, ignoring every cached layer. +# +# The ordinary path already rebuilds whenever an input changes, so this is for +# the cases a content hash cannot see: a moved upstream package, a stale base +# layer, or a build that is suspected of being wrong. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-rebuild: + $ErrorActionPreference = 'Stop' + $env:ANVIL_CONTAINER_NO_CACHE = '1' + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image rebuilt: $image" + exit 0 + +# Remove this repository's cache volumes. The image is left in place. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-down: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + & $engine volume rm -f $vol + } + exit 0 === justfiles/anvil/groups/pr-fast.just === # Copyright (c) Microsoft Corporation. @@ -5258,7 +4491,6 @@ import 'groups/scheduled-test.just' import 'groups/scheduled-advisories.just' import 'groups/scheduled-runtime-analysis.just' import 'groups/scheduled-exhaustive.just' -import 'runner.just' import 'tiers.just' import 'tools.just' import 'versions.just' @@ -5266,55 +4498,6 @@ import 'versions.just' # Friendly default: `just anvil` runs the PR tier. alias anvil := anvil-pr -=== justfiles/anvil/runner.just === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -# Route public tier entry points through the configured execution environment. -# ANVIL_IN_CONTAINER always wins to prevent recursive container launches. -[private] -[no-exit-message] -[windows] -[script("pwsh", "-NoProfile")] -_anvil-run tier runner: - $just = '{{ replace(just_executable(), "'", "''") }}' - $justfile = '{{ replace(justfile(), "'", "''") }}' - $nativeTier = '_anvil-{{ replace(tier, "'", "''") }}' - if ($env:ANVIL_IN_CONTAINER) { - & $just --justfile $justfile $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'container') { - & $just --justfile $justfile anvil-container $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'native') { - & $just --justfile $justfile $nativeTier - } else { - [Console]::Error.WriteLine("anvil-runner: expected 'native' or 'container', got '{{ replace(runner, "'", "''") }}'.") - exit 2 - } - exit $LASTEXITCODE - -[private] -[no-exit-message] -[unix] -[script("bash")] -_anvil-run tier runner: - just_path={{ quote(just_executable()) }} - justfile={{ quote(justfile()) }} - tier={{ quote(tier) }} - runner={{ quote(runner) }} - native_tier="_anvil-$tier" - if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - elif [[ "$runner" == "container" ]]; then - exec "$just_path" --justfile "$justfile" anvil-container "$native_tier" - elif [[ "$runner" == "native" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - else - echo "anvil-runner: expected 'native' or 'container', got '$runner'." >&2 - exit 2 - fi - === justfiles/anvil/tiers.just === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. @@ -5330,10 +4513,7 @@ _anvil-run tier runner: # Run all pull request checks. [group("anvil")] -anvil-pr: (_anvil-run "pr" anvil_runner) - -[private] -_anvil-pr: anvil-pr-validate-prereqs \ +anvil-pr: anvil-pr-validate-prereqs \ anvil-pr-fast \ anvil-pr-slow @@ -5344,10 +4524,7 @@ _anvil-pr: anvil-pr-validate-prereqs \ # Run all scheduled checks. [group("anvil")] -anvil-scheduled: (_anvil-run "scheduled" anvil_runner) - -[private] -_anvil-scheduled: anvil-scheduled-validate-prereqs \ +anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-test \ anvil-scheduled-advisories \ anvil-scheduled-runtime-analysis \ @@ -5355,12 +4532,9 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. [group("anvil")] -anvil-full: (_anvil-run "full" anvil_runner) - -[private] -_anvil-full: anvil-full-validate-prereqs \ - _anvil-pr \ - _anvil-scheduled +anvil-full: anvil-full-validate-prereqs \ + anvil-pr \ + anvil-scheduled # Tier-level + global setup + validate-prereqs # =========================================================================== diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index d5ea8f2b..cf5c2260 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -2,20 +2,40 @@ source: crates/cargo-anvil/tests/snapshots.rs expression: render_tree(tmp.path()) --- -=== .anvil/container/Containerfile === +=== .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# Default Anvil execution image, emitted for every repository. The image +# installs exactly the tools the generated catalog pins, by running +# `just anvil-setup` -- the same recipe the checks themselves use. That is what +# makes "the image has the right tools" true by construction rather than by +# convention: there is no second list to keep in step. +# +# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the +# image identity hash and refuses a floating tag, because a tag that can change +# underneath a hash makes the hash a lie. +# +# To build on a different base (a lower glibc baseline, or an internal +# distribution), a downstream catalog replaces this artifact wholesale via +# `replace_artifact(artifacts::container::dockerfile(...))`; a single +# repository can edit this file in place, which anvil's drift handling +# preserves. ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e FROM ${BASE_IMAGE} -ARG ANVIL_IMAGE_ID ARG JUST_VERSION=1.56.0 ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 ARG POWERSHELL_VERSION=7.6.3 ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 ARG RUSTUP_VERSION=1.29.0 ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e ENV DEBIAN_FRONTEND=noninteractive \ CARGO_HOME=/usr/local/cargo \ @@ -23,12 +43,17 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare slim base has no C runtime development files, so every +# link step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ libssl-dev pkg-config tar \ && rm -rf /var/lib/apt/lists/* +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. RUN curl -fsSLo /tmp/powershell.tar.gz \ "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ @@ -52,1150 +77,64 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated +# source edit does not invalidate this layer. The synthetic Justfile avoids +# pulling in repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. WORKDIR /opt/anvil -COPY . . -RUN test -f rust-toolchain.toml || { \ - echo "anvil-container requires rust-toolchain.toml" >&2; \ - exit 1; \ - } -RUN --mount=type=cache,id=anvil-cargo-registry,target=/usr/local/cargo/registry \ - --mount=type=cache,id=anvil-cargo-git,target=/usr/local/cargo/git \ - --mount=type=cache,id=anvil-cargo-target,target=/tmp/anvil-target \ - printf "anvil_runner := \"native\"\nimport 'justfiles/anvil/mod.just'\n" > Justfile \ - && CARGO_TARGET_DIR=/tmp/anvil-target just anvil-setup - -COPY .anvil/container/entrypoint.sh /usr/local/bin/anvil-container-entrypoint -RUN chmod 755 /usr/local/bin/anvil-container-entrypoint - +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# Consumed by the re-entry guard in the generated tier and group recipes: a +# recipe that sees this runs natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 -LABEL io.github.cargo-anvil.image-id="${ANVIL_IMAGE_ID}" + WORKDIR /workspace -ENTRYPOINT ["anvil-container-entrypoint"] CMD ["bash"] -=== .anvil/container/Containerfile.dockerignore === -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Deny-all allow-list for the image build context. +=== .anvil/container/Dockerfile.dockerignore === +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. # -# Docker matches each candidate against every pattern in order and lets the -# last match win, testing the path itself *and each of its parent directories* -# (moby/patternmatcher MatchesOrParentMatches). A bare directory re-inclusion -# such as `!justfiles` therefore re-admits the entire subtree below it, which -# would defeat this allow-list, so list only leaf patterns here. Docker still -# descends into a denied directory when some re-inclusion pattern is prefixed -# by it, so the intermediate directories need no entries of their own. +# BuildKit reads `.dockerignore` in preference to a root +# `.dockerignore`, so this scopes the exec-image build context without the +# repository having to own a root ignore file or having one silently overridden. # -# Parent testing also reaches through a single-segment re-inclusion: a -# subdirectory of `.anvil/container/` matches `!.anvil/container/*` in its own -# right. The image-ID helpers list that directory one level deep, so a nested -# file is not an image input; `.anvil/container/*/*` states that leaf-only -# contract in the allow-list too, at every depth, because a deeper candidate -# always has an ancestor of exactly that shape. -** +# The build context is the repository root but the image only needs two things. +# Excluding everything else keeps a cold build from streaming the whole +# worktree (and every stale `target/`) to the daemon. +* +!justfiles !rust-toolchain.toml -!justfiles/anvil/*.just -!justfiles/anvil/checks/*.just -!justfiles/anvil/groups/*.just -!.anvil/container/* -.anvil/container/*/* -.anvil/container/customize.sh -.anvil/container/customize.ps1 - -=== .anvil/container/README.md === - - -# Run Anvil checks in a local container - -Use `just anvil-container` to run generated Anvil checks in a reproducible -Linux environment without installing the complete Rust and Cargo tool catalog -on the host. - -Native execution remains the default. The first container run builds an image -matching the repository's generated configuration. Later runs reuse that image, -dependency caches, and compilation output. - -## Quick start - -Ensure Docker Engine is running, then run: - -```text -just anvil-container anvil-clippy -``` - -The first run builds the matching image and can take several minutes. - -## Prerequisites - -- [Docker Engine](https://docs.docker.com/engine/install/) 23.0 or newer, - installed directly in Linux or WSL and usable by the current user. -- `git` and `just` on the host. -- Bash on Linux and WSL; PowerShell Core (`pwsh`) and WSL 2 on Windows. -- `[script]` support enabled in the root `Justfile`. Add `set unstable` when - required by the installed `just` version. -- A `rust-toolchain.toml` in the repository root. -- A Linux or WSL environment capable of running `linux/amd64` images, either - natively on x86-64 or through Docker emulation on ARM64. - -On Windows, the driver invokes Docker from the default WSL distribution rather -than calling Windows `docker.exe`. Regardless of how Docker is installed, this -command must succeed from PowerShell: - -```text -wsl -e docker version -``` - -Start the Docker service inside WSL when it is stopped and add the WSL user to -the `docker` group when non-root access is not already configured. Docker -Desktop is not required. - -On ARM64 hosts, Docker emulates the required `linux/amd64` environment. Image -builds and checks can therefore be substantially slower than on x86-64 hosts. - -## Security boundary - -> [!WARNING] -> `customize.sh` and `customize.ps1` execute on the host with the developer's -> permissions before container isolation begins. Reviewing and trusting these -> files is equivalent to reviewing and trusting any other host-executed script -> in the checked-out branch. - -## Common workflows - -Run one check: - -```text -just anvil-container anvil-clippy -``` - -Run the complete pull-request tier: - -```text -just anvil-container anvil-pr -``` - -Every argument is treated as a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not supported by this command surface. - -Open an interactive Bash shell in the image: - -```text -just anvil-container -``` - -### Use containers for tier commands - -Native execution remains the default. To route tier commands such as -`just anvil-pr` through the container for the current shell: - -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` - -On Unix: - -```sh -ANVIL_RUNNER=container just anvil-pr -``` - -For one invocation: - -```text -just anvil_runner=container anvil-pr -``` - -To make container execution the repository default, change the default value -in the `anvil-runner` region of the repository-root `Justfile` from `"native"` -to `"container"` and commit that policy. Set `ANVIL_RUNNER=native` to override -the repository default for the current shell. - -Tier routing starts a nested `just` invocation. Output and exit status are -preserved, but outer `--dry-run`, dependency introspection, global options, and -CLI variable assignments are not propagated to the selected private tier. -Values other than `native` and `container` are rejected. - -## Images and caches - -The image name includes a content-based tag derived from the repository's Rust -toolchain, generated Anvil recipes, and container build configuration. A -relevant change selects a new image automatically; older branches can continue -using their matching images. - -The following data is reused between runs: - -- the matching container image; -- repository-scoped Cargo registry and Cargo Git caches; -- compilation output in a repository- and image-specific `target` volume. - -The repository is mounted read/write at `/workspace`. Build output remains in a -named volume instead of the host `target/`, avoiding incompatible artifacts and -slow host-to-virtual-machine I/O. - -## GitHub authentication - -`anvil-aprz` and aggregate tiers that include it require GitHub API -authentication. The driver uses either: - -- the host `GITHUB_TOKEN`; or -- the token from an authenticated host `gh` session. - -Trusted customization can provision a short-lived token by setting -`GITHUB_TOKEN`; the driver reads it after loading and validating customization. - -Authenticate the GitHub CLI with: - -```text -gh auth login --hostname github.com -``` - -For an aggregate tier, the driver first runs `anvil-aprz` in a short-lived -container with the token mounted read-only. After it succeeds, the driver runs -the remaining checks in another container without the token. Temporary token -files are removed afterward. - -An interactive invocation can pause while you authenticate. A non-interactive -invocation fails with instructions when authentication is unavailable. - -## Configuration - -| Variable | Effect | -|---|---| -| `ANVIL_RUNNER` | Selects `native` or `container` execution for tier commands | -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a digest-pinned compatible Linux base image and changes the content-based tag | -| `ANVIL_CONTAINER_IMAGE` | Changes the local image name; the content-based tag is retained | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails instead of building when the matching image is absent | - -The public driver builds images locally and does not pull -`ANVIL_CONTAINER_IMAGE` from a registry. - -The default base is digest-pinned Debian Bookworm. Set -`ANVIL_CONTAINER_BASE_IMAGE` to another image compatible with the generated -Debian-based `Containerfile` when a lower glibc baseline is required. A -different package ecosystem such as Azure Linux requires a derived -`Containerfile`. The value must use `image@sha256:` form so the -selected base remains part of the content-addressed image identity. - -Two simultaneous cold invocations can both build the same missing image. This -is accepted for local development: the content-addressed tag converges on the -same inputs, at the cost of duplicate work. - -## Troubleshooting - -| Problem | Resolution | -|---|---| -| Docker is not found on Linux or WSL | Install Docker Engine 23.0 or newer inside that environment | -| Docker is unavailable from Windows | Run `wsl -e docker version`; install or start Docker Engine in the default WSL distribution | -| Docker requires elevated access | Add the Linux/WSL user to the `docker` group, then start a new shell | -| ARM64 execution is slow | The current image is `linux/amd64` and runs through Docker emulation | -| `linux/amd64` cannot run | Configure Docker to run `linux/amd64` images | -| `[script]` recipes are unavailable | Enable `[script]` support; older `just` versions require `set unstable` | -| `rust-toolchain.toml` is missing | Add the repository-owned toolchain file at the repository root | -| GitHub authentication is unavailable | Run `gh auth login --hostname github.com` or set host `GITHUB_TOKEN` | -| A matching image is missing with `ANVIL_CONTAINER_NO_REBUILD=1` | Unset the variable to allow the local image build | -| The first run is slow | The initial image build installs the pinned tool catalog; later runs reuse it | - -Use `docker images anvil-dev` inside Linux or WSL to list locally cached -default Anvil images. - -## Managed files - -This directory is managed by `cargo-anvil`. Regenerate it with `cargo anvil` -instead of editing its files directly. - -> [!IMPORTANT] -> These assets previously lived in `justfiles/anvil/container/`. `cargo anvil` -> relocates the files it generated, but it does not track a hand-authored -> `customize.sh` or `customize.ps1`. Move any such file to -> `.anvil/container/` yourself; the driver only loads customization from the -> new location and warns on stderr when it finds one left behind. - -## Advanced repository customization - -A repository or derived catalog can add one trusted customization file per -supported host: - -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 -``` - -The driver sources the matching file as trusted host code before authentication, -image construction, and recipe execution. The documented customization -contract provides inputs and validated outputs for APRZ classification, build -secrets, dependency preparation, runtime arguments, and cleanup. - -Customization source is excluded from image identity and the build context. -Non-secret image behavior must be represented by hashed static files such as -the `Containerfile`, entrypoint, or supporting build scripts. - -See the [container customization contract](https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md#8-container-customization) -for the complete interface and security requirements. - -=== .anvil/container/entrypoint.sh === -#!/bin/sh -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -eu - -if [ "$(id -u)" -ne 0 ]; then - if [ -z "${HOME:-}" ] || [ "$HOME" = "/" ]; then - HOME="/tmp/anvil-user" - export HOME - fi - - user_cargo_home="$HOME/.cargo" - mkdir -p "$user_cargo_home" - for file in config.toml .crates.toml .crates2.json; do - if [ -r "$CARGO_HOME/$file" ]; then - cp -f "$CARGO_HOME/$file" "$user_cargo_home/$file" - fi - done - export CARGO_HOME="$user_cargo_home" - ln -sfn /usr/local/cargo/registry "$CARGO_HOME/registry" - ln -sfn /usr/local/cargo/git "$CARGO_HOME/git" -fi - -exec "$@" - -=== .anvil/container/image-id.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$inputs = @( - 'rust-toolchain.toml' -) -$toolchainPath = Join-Path $repoRoot 'rust-toolchain.toml' -if (-not (Test-Path -LiteralPath $toolchainPath -PathType Leaf)) { - throw 'anvil-container requires a repository-owned rust-toolchain.toml.' -} -$containerPath = Join-Path $repoRoot '.anvil/container' -$containerRecipe = 'justfiles/anvil/container.just' -$containerfile = Join-Path $containerPath 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$pathComparison = if ($IsWindows) { [StringComparison]::OrdinalIgnoreCase } else { [StringComparison]::Ordinal } -# The container entry recipe drives execution on the host; it is not image -# content, so it must not participate in image identity. -$inputs += Get-ChildItem (Join-Path $repoRoot 'justfiles/anvil') -Recurse -File -Filter '*.just' | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } | - Where-Object { -not $_.Equals($containerRecipe, $pathComparison) } -$executionOnly = @( - 'image-id.ps1', - 'image-id.sh', - 'README.md', - 'run-in-container.ps1', - 'run-in-container.sh', - 'customize.sh', - 'customize.ps1' -) -# customize.sh/customize.ps1 are trusted runtime orchestration, not image -# content: their source must never affect the image ID or build context. -# Static, non-secret build customization belongs in a hashed artifact instead. -$inputs += Get-ChildItem $containerPath -File | - Where-Object { $_.Name -notin $executionOnly } | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } -$uniqueInputs = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) -foreach ($inputPath in $inputs) { - [void]$uniqueInputs.Add($inputPath) -} -$inputs = [string[]]$uniqueInputs -[Array]::Sort($inputs, [StringComparer]::Ordinal) - -$payload = [Text.StringBuilder]::new() -[void]$payload.Append("ANVIL_CONTAINER_BASE_IMAGE`n").Append($baseImage).Append("`n") -foreach ($relative in $inputs) { - $path = Join-Path $repoRoot $relative - if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { - throw "Container image input is missing: $relative" - } - $content = [IO.File]::ReadAllText($path).Replace("`r`n", "`n").Replace("`r", "`n") - [void]$payload.Append($relative).Append("`n").Append($content).Append("`n") -} - -$bytes = [Text.Encoding]::UTF8.GetBytes($payload.ToString()) -$hash = [Security.Cryptography.SHA256]::HashData($bytes) -Write-Output ([Convert]::ToHexString($hash).ToLowerInvariant()) - -=== .anvil/container/image-id.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi - -toolchain_path="$repo_root/rust-toolchain.toml" -if [[ ! -f "$toolchain_path" ]]; then - echo 'anvil-container requires a repository-owned rust-toolchain.toml.' >&2 - exit 1 -fi - -container_dir="$repo_root/.anvil/container" -container_recipe="justfiles/anvil/container.just" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$container_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -inputs=(rust-toolchain.toml) -while IFS= read -r path; do - relative="${path#"$repo_root"/}" - # The container entry recipe drives execution on the host; it is not - # image content, so it must not participate in image identity. - if [[ "$relative" != "$container_recipe" ]]; then - inputs+=("$relative") - fi -done < <(find "$repo_root/justfiles/anvil" -type f -name '*.just' -print) - -for path in "$container_dir"/*; do - [[ -f "$path" ]] || continue - case "${path##*/}" in - image-id.ps1 | image-id.sh | README.md \ - | run-in-container.ps1 | run-in-container.sh \ - | customize.sh | customize.ps1) continue ;; - esac - inputs+=("${path#"$repo_root"/}") -done - -if command -v sha256sum >/dev/null 2>&1; then - hash_command=(sha256sum) -elif command -v shasum >/dev/null 2>&1; then - hash_command=(shasum -a 256) -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi - -write_normalized_file() { - local path="$1" - local line status - while true; do - line="" - if IFS= read -r line <&3; then - status=0 - else - status=$? - fi - if ((status != 0)) && [[ -z "$line" ]]; then - break - fi - printf '%s' "${line%$'\r'}" - if ((status == 0)); then - printf '\n' - else - break - fi - done 3<"$path" -} - -{ - printf 'ANVIL_CONTAINER_BASE_IMAGE\n%s\n' "$base_image" - while IFS= read -r relative; do - path="$repo_root/$relative" - if [[ ! -f "$path" ]]; then - echo "Container image input is missing: $relative" >&2 - exit 1 - fi - printf '%s\n' "$relative" - write_normalized_file "$path" - printf '\n' - done < <(printf '%s\n' "${inputs[@]}" | LC_ALL=C sort -u) -} | "${hash_command[@]}" | awk '{print $1}' - -=== .anvil/container/run-in-container.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param( - [Parameter(Position = 0, ValueFromRemainingArguments = $true)] - [string[]]$Recipe -) - -$ErrorActionPreference = 'Stop' - -function ConvertTo-AnvilVersion([string]$Value) { - $match = [regex]::Match($Value, '^(\d+)\.(\d+)(?:\.(\d+))?') - if (-not $match.Success) { - throw "anvil-container: could not parse Docker Engine version '$Value'." - } - [version]::new( - [int]$match.Groups[1].Value, - [int]$match.Groups[2].Value, - $(if ($match.Groups[3].Success) { [int]$match.Groups[3].Value } else { 0 }) - ) -} - -function Test-AnvilContainerStringArray([string]$Name, $Value) { - if ($Value -isnot [array]) { - throw "anvil-container: `$$Name must be a string array." - } - foreach ($item in $Value) { - if ($item -isnot [string] -or [string]::IsNullOrEmpty($item)) { - throw "anvil-container: `$$Name entries must be non-empty strings." - } - } -} - -function Test-AnvilContainerBuildArgs($Value) { - for ($index = 0; $index -lt $Value.Count; $index++) { - $item = $Value[$index] - if ($item -eq '--secret') { - $index++ - if ($index -ge $Value.Count) { - throw 'anvil-container: $AnvilContainerBuildArgs requires a value after --secret.' - } - } elseif (-not $item.StartsWith('--secret=', [StringComparison]::Ordinal)) { - throw 'anvil-container: $AnvilContainerBuildArgs accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' - } - } -} - -function Test-AnvilRecipeNeedsGitHubToken([string]$Name) { - $Name -in @( - 'anvil-aprz', - 'anvil-pr', - '_anvil-pr', - 'anvil-pr-fast', - 'anvil-scheduled', - '_anvil-scheduled', - 'anvil-scheduled-advisories', - 'anvil-full', - '_anvil-full' - ) -} - -function Get-AnvilGitHubToken { - $token = $env:GITHUB_TOKEN - if (-not $token -and (Get-Command gh -ErrorAction SilentlyContinue)) { - try { - $token = (& gh auth token --hostname github.com 2>$null) - if ($LASTEXITCODE -ne 0) { $token = $null } - } catch { - $token = $null - } - } - if ($token) { $token = $token.Trim() } - if ($token) { return $token } - return $null -} - -if ($env:ANVIL_IN_CONTAINER) { - if ($Recipe.Count -eq 0) { & bash } else { & just @Recipe } - exit $LASTEXITCODE -} - -foreach ($recipeArg in $Recipe) { - if ($recipeArg -notmatch '^_?anvil-[A-Za-z0-9-]+$') { - throw "anvil-container: expected each argument to be an anvil-* recipe, got '$recipeArg'." - } -} - -if (-not (Get-Command wsl -ErrorAction SilentlyContinue)) { - throw 'anvil-container: WSL 2 is required. See .anvil/container/README.md.' -} - -$versionText = (& wsl -e docker version --format '{{.Server.Version}}' 2>$null) -if ($LASTEXITCODE -ne 0 -or -not $versionText) { - throw 'anvil-container: `wsl -e docker version` must succeed. Install or start Docker Engine in the default WSL distribution; this driver does not invoke Windows docker.exe.' -} -$versionText = $versionText.Trim() -if ((ConvertTo-AnvilVersion $versionText) -lt [version]'23.0.0') { - throw "anvil-container: Docker Engine 23.0.0 or newer is required (found $versionText)." -} -$wslArchitecture = (& wsl -e uname -m 2>$null) -if ($LASTEXITCODE -eq 0 -and $wslArchitecture) { - $wslArchitecture = $wslArchitecture.Trim() - if ($wslArchitecture -notin @('x86_64', 'amd64')) { - [Console]::Error.WriteLine( - "anvil-container: warning: $wslArchitecture requires emulation for linux/amd64; builds and checks may be substantially slower." - ) - } -} - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$scriptDir = Join-Path $repoRoot '.anvil/container' -$wslRepoRoot = (& wsl -e wslpath -a $repoRoot).Trim() -if ($LASTEXITCODE -ne 0 -or -not $wslRepoRoot) { - throw 'anvil-container: could not translate the repository path into the default WSL distribution.' -} -$wslScriptDir = "$wslRepoRoot/.anvil/container" -$containerfile = Join-Path $scriptDir 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$imageId = (& (Join-Path $scriptDir 'image-id.ps1')).Trim() -$imageBase = if ($env:ANVIL_CONTAINER_IMAGE) { $env:ANVIL_CONTAINER_IMAGE } else { 'anvil-dev' } -$image = "${imageBase}:$imageId" -$repoBytes = [Text.Encoding]::UTF8.GetBytes($wslRepoRoot) -$repoHash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($repoBytes)).ToLowerInvariant() -$targetVolume = "anvil-target-$($repoHash.Substring(0, 12))-$($imageId.Substring(0, 12))" - -$needsGitHubToken = $false -foreach ($recipeArg in $Recipe) { - if (Test-AnvilRecipeNeedsGitHubToken $recipeArg) { - $needsGitHubToken = $true - break - } -} -$runsOnlyGitHubCheck = $Recipe.Count -eq 1 -and $Recipe[0] -eq 'anvil-aprz' - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -$null = & wsl -e docker image inspect $image 2>$null -$imageExists = $LASTEXITCODE -eq 0 - -New-Variable -Name AnvilContainerRepoRoot -Value $repoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDir -Value $scriptDir -Option ReadOnly -New-Variable -Name AnvilContainerRepoRootWsl -Value $wslRepoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDirWsl -Value $wslScriptDir -Option ReadOnly -New-Variable -Name AnvilContainerResolvedImage -Value $image -Option ReadOnly -New-Variable -Name AnvilContainerImageExists -Value $imageExists -Option ReadOnly -New-Variable -Name AnvilContainerRequestedRecipes -Value $Recipe -Option ReadOnly -New-Variable -Name AnvilContainerHostIsWindows -Value ([bool]$IsWindows) -Option ReadOnly - -# Customization outputs, initialized before sourcing so a missing customize.ps1 -# leaves every phase a documented no-op. -$AnvilContainerBuildArgs = @() -$AnvilContainerPrepareArgs = @() -$AnvilContainerPrepareCommand = @() -$AnvilContainerRunArgs = @() -$AnvilContainerNeedsGitHubToken = $needsGitHubToken -$AnvilContainerCleanup = $null -$githubToken = $null -$githubTokenFile = $null -$exitCode = 0 -$customizeScript = Join-Path $scriptDir 'customize.ps1' -$legacyCustomizeScript = Join-Path $repoRoot 'justfiles/anvil/container/customize.ps1' - -try { - if (Test-Path -LiteralPath $customizeScript -PathType Leaf) { - . $customizeScript - } - elseif (Test-Path -LiteralPath $legacyCustomizeScript -PathType Leaf) { - [Console]::Error.WriteLine( - 'anvil-container: warning: ignoring justfiles/anvil/container/customize.ps1; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.ps1 to keep it active.' - ) - } - - Test-AnvilContainerStringArray 'AnvilContainerBuildArgs' $AnvilContainerBuildArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareArgs' $AnvilContainerPrepareArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareCommand' $AnvilContainerPrepareCommand - Test-AnvilContainerStringArray 'AnvilContainerRunArgs' $AnvilContainerRunArgs - Test-AnvilContainerBuildArgs $AnvilContainerBuildArgs - if ($AnvilContainerNeedsGitHubToken -isnot [bool]) { - throw 'anvil-container: $AnvilContainerNeedsGitHubToken must be a Boolean.' - } - $needsGitHubToken = $needsGitHubToken -or $AnvilContainerNeedsGitHubToken - if ($AnvilContainerPrepareArgs.Count -gt 0 -and $AnvilContainerPrepareCommand.Count -eq 0) { - throw 'anvil-container: $AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand.' - } - if ($AnvilContainerCleanup -and $AnvilContainerCleanup -isnot [scriptblock]) { - throw 'anvil-container: $AnvilContainerCleanup must be a script block.' - } - $githubToken = if ($needsGitHubToken) { Get-AnvilGitHubToken } else { $null } - if ($needsGitHubToken -and -not $githubToken) { - if (-not (Get-Command gh -ErrorAction SilentlyContinue)) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' - } - if (-not [Environment]::UserInteractive -or [Console]::IsInputRedirected) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' - } - Write-Host 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' - [void](Read-Host 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel)') - $githubToken = Get-AnvilGitHubToken - if (-not $githubToken) { - throw 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' - } - } - if (-not $imageExists) { - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - throw "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." - } - & wsl -e docker build ` - --platform linux/amd64 ` - --tag $image ` - --file "$wslScriptDir/Containerfile" ` - --build-arg "ANVIL_IMAGE_ID=$imageId" ` - --build-arg "BASE_IMAGE=$baseImage" ` - @AnvilContainerBuildArgs ` - $wslRepoRoot - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker build failed with exit code $LASTEXITCODE." - } - } - - $containerUid = (& wsl -e id -u).Trim() - $containerGid = (& wsl -e id -g).Trim() - if ($containerUid -notmatch '^\d+$' -or $containerGid -notmatch '^\d+$') { - throw 'anvil-container: could not determine the default WSL user identity.' - } - $registryVolume = "anvil-cargo-registry-$($repoHash.Substring(0, 12))" - $gitVolume = "anvil-cargo-git-$($repoHash.Substring(0, 12))" - foreach ($volume in @($registryVolume, $gitVolume, $targetVolume)) { - $null = & wsl -e docker volume create $volume - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume creation failed for '$volume' with exit code $LASTEXITCODE." - } - } - $mountArgs = @( - '--mount', "type=bind,source=$wslRepoRoot,target=/workspace", - '--mount', "type=volume,source=$registryVolume,target=/usr/local/cargo/registry", - '--mount', "type=volume,source=$gitVolume,target=/usr/local/cargo/git", - '--mount', "type=volume,source=$targetVolume,target=/workspace/target" - ) - & wsl -e docker run --rm --pull=never ` - --platform linux/amd64 ` - --user 0:0 ` - @mountArgs ` - $image sh -c "chown ${containerUid}:${containerGid} /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume initialization failed with exit code $LASTEXITCODE." - } - - $runArgs = @( - 'run', '--rm', '--pull=never', - '--platform', 'linux/amd64', - '--user', "${containerUid}:${containerGid}", - '--env', 'ANVIL_IN_CONTAINER=1', - '--env', 'HOME=/tmp/anvil-user', - '--workdir', '/workspace' - ) - $runArgs += $mountArgs - $prepareRunArgs = @($runArgs) - $runArgs += $AnvilContainerRunArgs - foreach ($name in @( - 'PR_TITLE', - 'BASE_REF', - 'ANVIL_INCLUDE_MODIFIED', - 'ANVIL_INCLUDE_AFFECTED', - 'ANVIL_INCLUDE_REQUIRED', - 'GITHUB_BASE_REF', - 'SYSTEM_PULLREQUEST_TARGETBRANCH' - )) { - if (Test-Path "Env:$name") { - $runArgs += @('--env', "$name=$((Get-Item "Env:$name").Value)") - } - } - if ($AnvilContainerPrepareCommand.Count -gt 0) { - & wsl -e docker @prepareRunArgs @AnvilContainerPrepareArgs $image @AnvilContainerPrepareCommand - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: preparation command failed with exit code $LASTEXITCODE." - } - } - - if ($githubToken) { - $githubTokenFile = Join-Path ([IO.Path]::GetTempPath()) "anvil-github-token-$PID-$([guid]::NewGuid().ToString('N'))" - [IO.File]::Create($githubTokenFile).Dispose() - if ($IsWindows) { - $userSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value - & icacls.exe $githubTokenFile '/inheritance:r' '/grant:r' "*$($userSid):(F)" | Out-Null - } else { - & chmod 600 $githubTokenFile - } - if ($LASTEXITCODE -ne 0) { - throw 'anvil-container: failed to restrict permissions on the temporary GitHub token file.' - } - [IO.File]::WriteAllText($githubTokenFile, $githubToken, [Text.Encoding]::ASCII) - $githubToken = $null - $wslTokenFile = (& wsl -e wslpath -a $githubTokenFile).Trim() - if ($LASTEXITCODE -ne 0 -or -not $wslTokenFile) { - throw 'anvil-container: could not translate the temporary GitHub token path into WSL.' - } - $githubRunArgs = @($runArgs) - $githubRunArgs += @( - '--mount', - "type=bind,source=$wslTokenFile,target=/run/secrets/anvil-github-token,readonly" - ) - if ($runsOnlyGitHubCheck) { - $runArgs = $githubRunArgs - } else { - & wsl -e docker @githubRunArgs $image just anvil-aprz - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: isolated anvil-aprz failed with exit code $LASTEXITCODE." - } - $runArgs += @('--env', 'ANVIL_APRZ_ALREADY_RAN=1') - } - } - - if ($Recipe.Count -eq 0) { - & wsl -e docker @runArgs --interactive --tty $image bash - } else { - & wsl -e docker @runArgs $image just @Recipe - } - $exitCode = $LASTEXITCODE -} finally { - if ($githubTokenFile) { - Remove-Item -LiteralPath $githubTokenFile -Force -ErrorAction SilentlyContinue - } - if ($AnvilContainerCleanup) { & $AnvilContainerCleanup } -} - -exit $exitCode - -=== .anvil/container/run-in-container.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - if (($# == 0)); then exec bash; else exec just "$@"; fi -fi - -for recipe_arg in "$@"; do - if [[ ! "$recipe_arg" =~ ^_?anvil-[A-Za-z0-9-]+$ ]]; then - echo "anvil-container: expected each argument to be an anvil-* recipe, got '$recipe_arg'." >&2 - exit 2 - fi -done - -anvil_recipe_needs_github_token() { - case "$1" in - anvil-aprz | anvil-pr | _anvil-pr | anvil-pr-fast \ - | anvil-scheduled | _anvil-scheduled | anvil-scheduled-advisories \ - | anvil-full | _anvil-full) return 0 ;; - *) return 1 ;; - esac -} - -version_at_least() { - local found="${1%%[-+]*}" - local required="${2%%[-+]*}" - local found_major found_minor found_patch found_extra - local required_major required_minor required_patch required_extra - IFS=. read -r found_major found_minor found_patch found_extra <<<"$found" - IFS=. read -r required_major required_minor required_patch required_extra <<<"$required" - found_patch="${found_patch:-0}" - required_patch="${required_patch:-0}" - for component in \ - "$found_major" "$found_minor" "$found_patch" \ - "$required_major" "$required_minor" "$required_patch" - do - case "$component" in - '' | *[!0-9]*) return 2 ;; - esac - done - if ((found_major != required_major)); then ((found_major > required_major)); return; fi - if ((found_minor != required_minor)); then ((found_minor > required_minor)); return; fi - ((found_patch >= required_patch)) -} - -command -v docker >/dev/null 2>&1 || { - echo "anvil-container: Docker Engine is required. See .anvil/container/README.md." >&2 - exit 1 -} - -version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || { - echo "anvil-container: Docker Engine is unavailable. Start the Docker service and ensure the current user can access it." >&2 - exit 1 -} -minimum="23.0.0" -if ! version_at_least "$version" "$minimum"; then - echo "anvil-container: Docker Engine $minimum or newer is required (found $version)." >&2 - exit 1 -fi -host_arch="$(uname -m 2>/dev/null || true)" -case "$host_arch" in - x86_64 | amd64 | '') ;; - *) echo "anvil-container: warning: $host_arch requires emulation for linux/amd64; builds and checks may be substantially slower." >&2 ;; -esac - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi -script_dir="$repo_root/.anvil/container" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$script_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -image_id="$(bash "$script_dir/image-id.sh")" -image_base="${ANVIL_CONTAINER_IMAGE:-anvil-dev}" -image="${image_base}:${image_id}" -if command -v sha256sum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | sha256sum | cut -c1-12)" -elif command -v shasum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | shasum -a 256 | cut -c1-12)" -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi -target_volume="anvil-target-${repo_id}-${image_id:0:12}" - -needs_github_token=false -for recipe_arg in "$@"; do - if anvil_recipe_needs_github_token "$recipe_arg"; then - needs_github_token=true - break - fi -done -runs_only_github_check=false -if (($# == 1)) && [[ "$1" == "anvil-aprz" ]]; then - runs_only_github_check=true -fi -github_token="" - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -if docker image inspect "$image" >/dev/null 2>&1; then - image_exists=true -else - image_exists=false -fi - -readonly ANVIL_CONTAINER_REPO_ROOT="$repo_root" -readonly ANVIL_CONTAINER_DIR="$script_dir" -readonly ANVIL_CONTAINER_RESOLVED_IMAGE="$image" -readonly ANVIL_CONTAINER_IMAGE_EXISTS="$image_exists" -declare -a ANVIL_CONTAINER_REQUESTED_RECIPES=("$@") -readonly ANVIL_CONTAINER_REQUESTED_RECIPES - -# Customization outputs, initialized before sourcing so a missing customize.sh -# leaves every phase a documented no-op. -ANVIL_CONTAINER_BUILD_ARGS=() -ANVIL_CONTAINER_PREPARE_ARGS=() -ANVIL_CONTAINER_PREPARE_COMMAND=() -ANVIL_CONTAINER_RUN_ARGS=() -ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN="$needs_github_token" -ANVIL_CONTAINER_CLEANUP=: -github_token_file="" -cleanup() { - if [[ -n "$github_token_file" ]]; then rm -f -- "$github_token_file"; fi - "$ANVIL_CONTAINER_CLEANUP" -} -trap cleanup EXIT - -customize_script="$script_dir/customize.sh" -legacy_customize_script="$repo_root/justfiles/anvil/container/customize.sh" -if [[ ! -f "$customize_script" && -f "$legacy_customize_script" ]]; then - echo "anvil-container: warning: ignoring justfiles/anvil/container/customize.sh; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.sh to keep it active." >&2 -fi -if [[ -f "$customize_script" ]]; then - # shellcheck source=/dev/null - source "$customize_script" -fi - -# Bash 3.2 has neither namerefs (the nameref flag on `local`/`declare`, Bash -# 4.3+) nor safe `set -u` expansion of empty-but- -# declared arrays (fixed in Bash 4.4). Elements are passed positionally -# instead of by nameref, and every expansion of a possibly-empty array uses -# the `${arr[@]+"${arr[@]}"}` idiom: unset/empty-under-old-Bash arrays vanish -# entirely instead of raising "unbound variable", while non-empty arrays -# still expand element-for-element. -anvil_container_validate_array() { - local name="$1" - shift - local declaration value - declaration="$(declare -p "$name" 2>/dev/null || true)" - if [[ ! "$declaration" =~ ^declare\ -[^[:space:]]*a[^[:space:]]*\ ]]; then - echo "anvil-container: $name must be a string array." >&2 - exit 1 - fi - for value in "$@"; do - if [[ -z "$value" ]]; then - echo "anvil-container: $name entries must be non-empty strings." >&2 - exit 1 - fi - done -} -anvil_container_validate_build_args() { - local expect_secret_value=false value - for value in "$@"; do - if "$expect_secret_value"; then - expect_secret_value=false - continue - fi - case "$value" in - --secret) expect_secret_value=true ;; - --secret=*) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' >&2 - exit 1 - ;; - esac - done - if "$expect_secret_value"; then - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS requires a value after --secret.' >&2 - exit 1 - fi -} -anvil_container_validate_array ANVIL_CONTAINER_BUILD_ARGS ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_ARGS ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_COMMAND ${ANVIL_CONTAINER_PREPARE_COMMAND[@]+"${ANVIL_CONTAINER_PREPARE_COMMAND[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_RUN_ARGS ${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"} -anvil_container_validate_build_args ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -case "$ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN" in - true) needs_github_token=true ;; - false) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN must be true or false.' >&2 - exit 1 - ;; -esac -if ((${#ANVIL_CONTAINER_PREPARE_ARGS[@]} > 0)) && ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} == 0)); then - echo 'anvil-container: ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND.' >&2 - exit 1 -fi -cleanup_kind="$(type -t "$ANVIL_CONTAINER_CLEANUP" 2>/dev/null || true)" -if [[ "$cleanup_kind" != "function" && "$cleanup_kind" != "builtin" ]]; then - echo "anvil-container: ANVIL_CONTAINER_CLEANUP must name a callable function (got '$ANVIL_CONTAINER_CLEANUP')." >&2 - exit 1 -fi - -if "$needs_github_token"; then - gh_command="" - if command -v gh >/dev/null 2>&1; then - gh_command=gh - elif command -v gh.exe >/dev/null 2>&1; then - gh_command=gh.exe - fi - github_token="${GITHUB_TOKEN:-}" - if [[ -z "$github_token" && -n "$gh_command" ]]; then - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - fi - if [[ -z "$github_token" ]]; then - if [[ -z "$gh_command" ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - if [[ ! -t 0 ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - echo 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' >&2 - read -r -p 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel) ' - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - if [[ -z "$github_token" ]]; then - echo 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' >&2 - exit 1 - fi - fi -fi - -if ! "$image_exists"; then - if [[ "${ANVIL_CONTAINER_NO_REBUILD:-}" == "1" ]]; then - echo "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." >&2 - exit 1 - else - docker build \ - --platform linux/amd64 \ - --tag "$image" \ - --file "$script_dir/Containerfile" \ - --build-arg "ANVIL_IMAGE_ID=$image_id" \ - --build-arg "BASE_IMAGE=$base_image" \ - ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} \ - "$repo_root" - fi -fi - -container_uid="$(id -u)" -container_gid="$(id -g)" -registry_volume="anvil-cargo-registry-${repo_id}" -git_volume="anvil-cargo-git-${repo_id}" -for volume in "$registry_volume" "$git_volume" "$target_volume"; do - docker volume create "$volume" >/dev/null -done -mount_args=( - --mount "type=bind,source=$repo_root,target=/workspace" - --mount "type=volume,source=$registry_volume,target=/usr/local/cargo/registry" - --mount "type=volume,source=$git_volume,target=/usr/local/cargo/git" - --mount "type=volume,source=$target_volume,target=/workspace/target" -) -docker run --rm --pull=never \ - --platform linux/amd64 \ - --user 0:0 \ - "${mount_args[@]}" \ - "$image" sh -c \ - "chown $container_uid:$container_gid /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - -run_args=( - run --rm --pull=never - --platform linux/amd64 - --user "$container_uid:$container_gid" - --env ANVIL_IN_CONTAINER=1 - --env HOME=/tmp/anvil-user - "${mount_args[@]}" - --workdir /workspace -) -prepare_run_args=("${run_args[@]}") -run_args+=(${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"}) -for name in PR_TITLE BASE_REF ANVIL_INCLUDE_MODIFIED ANVIL_INCLUDE_AFFECTED ANVIL_INCLUDE_REQUIRED GITHUB_BASE_REF SYSTEM_PULLREQUEST_TARGETBRANCH; do - if value="$(printenv "$name")"; then run_args+=(--env "$name=$value"); fi -done -if ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} > 0)); then - docker "${prepare_run_args[@]}" \ - ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} \ - "$image" \ - "${ANVIL_CONTAINER_PREPARE_COMMAND[@]}" -fi - -if [[ -n "$github_token" ]]; then - github_token_file="$(mktemp "${TMPDIR:-/tmp}/anvil-github-token.XXXXXXXX")" - chmod 600 "$github_token_file" - printf '%s' "$github_token" > "$github_token_file" - unset github_token - github_run_args=( - "${run_args[@]}" - --mount "type=bind,source=$github_token_file,target=/run/secrets/anvil-github-token,readonly" - ) - if "$runs_only_github_check"; then - run_args=("${github_run_args[@]}") - else - docker "${github_run_args[@]}" "$image" just anvil-aprz - run_args+=(--env ANVIL_APRZ_ALREADY_RAN=1) - fi -fi - -if (($# == 0)); then - docker "${run_args[@]}" --interactive --tty "$image" bash - exit $? -fi -docker "${run_args[@]}" "$image" just "$@" === .delta.toml === # >>> anvil-managed: anvil-delta @@ -2513,10 +1452,6 @@ clippy.wildcard_imports = "allow" import 'justfiles/anvil/mod.just' # <<< anvil-managed: anvil-imports -# >>> anvil-managed: anvil-runner -anvil_runner := env_var_or_default("ANVIL_RUNNER", "native") -# <<< anvil-managed: anvil-runner - === clippy.toml === # >>> anvil-managed: anvil-clippy # Fine-tuning settings for clippy lints. These cannot be expressed in @@ -2616,11 +1551,11 @@ unknown-git = "deny" # cargo-aprz queries the GitHub advisory API. Unauthenticated access is # capped at 60 requests/hour and fails on a full run; an authenticated # token raises the cap to 5000/hour. CI injects GITHUB_TOKEN -# (github.token). Container drivers mount an existing host GITHUB_TOKEN -# or the host gh CLI's stored token as a temporary read-only secret. -# Native runs borrow the gh CLI token directly. Native runs warn and -# proceed unauthenticated if neither is available; container runs fail -# before cargo-aprz can exhaust the unauthenticated rate limit. +# (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the +# gh CLI's stored token (non-interactive: `gh auth token` prints the +# active account's token for github.com and never opens a browser/auth +# prompt). If neither is available we warn with instructions and proceed +# unauthenticated. # # Unscoped (consults external risk DB). @@ -2628,24 +1563,14 @@ unknown-git = "deny" [script("pwsh", "-NoProfile")] anvil-aprz: anvil-aprz-validate-prereqs $ErrorActionPreference = 'Stop' - if ($env:ANVIL_APRZ_ALREADY_RAN -eq '1') { - Write-Host 'anvil-aprz: already completed in an isolated authenticated container' - exit 0 - } if (-not $env:GITHUB_TOKEN) { $tok = $null - $containerTokenFile = '/run/secrets/anvil-github-token' - if ($env:ANVIL_IN_CONTAINER -and (Test-Path -LiteralPath $containerTokenFile -PathType Leaf)) { - try { $tok = Get-Content -LiteralPath $containerTokenFile -Raw } catch { $tok = $null } - } elseif (Get-Command gh -ErrorAction SilentlyContinue) { + if (Get-Command gh -ErrorAction SilentlyContinue) { try { $tok = (gh auth token --hostname github.com 2>$null) } catch { $tok = $null } } if ($tok) { $env:GITHUB_TOKEN = $tok.Trim() } else { - if ($env:ANVIL_IN_CONTAINER) { - throw 'anvil-aprz: GitHub authentication is unavailable. Run `gh auth login` on the host or set host GITHUB_TOKEN, then re-run the container command.' - } Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' @@ -4473,27 +3398,335 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Licensed under the MIT License. # GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. +# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md +# +# Containerized execution. `just anvil-container ` runs any anvil +# recipe inside a pinned Linux image; everything else keeps running natively. +# There is no configuration file and no transparent routing: the container is +# reached through this recipe or not at all. +# +# The image tag *is* a hash of the inputs that define it, so the presence of a +# tag is proof that its contents are current -- a changed tool pin names a tag +# that cannot already exist, and a build follows. There is nothing to keep in +# sync and no staleness to detect. +# +# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md + +# The container engine. `docker` (supported) or `podman` (best-effort). +# A host property, never committed: set the variable in your environment, or +# pass `just anvil_container_engine=podman ...` for a single invocation. +anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") + +# Where the repository is mounted inside the container. +anvil_container_workdir := "/workspace" + +# Image and cache-volume prefix, derived from the repository directory so two +# repositories on one host cannot collide. Sanitized to the character set +# container image references allow. +anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") + +# Resolve the engine. There is deliberately no probe: presence is not +# reachability, `podman-docker` aliases `docker` onto podman, and a silent +# choice between two installed engines means two image stores and an +# unexplained rebuild. We check that the requested binary exists and let every +# other failure surface the engine's own diagnostic, which is more accurate +# than anything repeated here. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-engine: + $ErrorActionPreference = 'Stop' + $engine = '{{anvil_container_engine}}' + if ($engine -ne 'docker' -and $engine -ne 'podman') { + Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" + exit 1 + } + if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { + Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + } + Write-Output $engine + +# Resolve the exec image reference, building it if it is not already present. +# +# The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its +# ignore file, the pinned toolchain, the optional hook, and the generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes. Only this driver is excluded, since hashing it would make the +# tag depend on the tag. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds +# a tag that already resolves, for the cases a content hash cannot see: a moved +# upstream package, or a base layer that changed behind its digest. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } -# Run any Anvil recipe in the pinned local Linux container. With no recipe, -# open an interactive shell. -[windows] + $repoRoot = '{{justfile_directory()}}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { + # The hook decides what the build installs, so its content defines the + # image as surely as the Dockerfile does. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $inputs += $hookRel + } + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } + } + } + + # Hash a tagged stream rather than raw concatenation, so no rearrangement of + # names and contents can collide. Line endings are normalized once, here, so + # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: + # `Sort-Object -Unique` compares case-insensitively, which would silently drop + # one of two inputs differing only in case on the case-sensitive filesystem + # where the image is actually built. + $stream = [System.Text.StringBuilder]::new() + $ordered = [System.Collections.Generic.SortedSet[string]]::new( + [string[]]$inputs, [System.StringComparer]::Ordinal) + foreach ($rel in $ordered) { + $path = Join-Path $repoRoot $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $rel" + exit 1 + } + $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" + [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + } + $digest = [System.Security.Cryptography.SHA256]::HashData( + [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # 16 hex characters (64 bits) is far past any practical collision risk for a + # local image set, and keeps `docker images` readable. + $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) + $image = '{{anvil_container_name}}:' + $imageId + + if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { + & $engine image inspect $image *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output $image + exit 0 + } + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 + } + } + + # Build-time credentials come from the optional hook, never from a committed + # file. Values are handed to BuildKit by environment variable name, so they + # stay out of the host's process command line, and BuildKit keeps them out of + # every image layer. An empty value is fatal: BuildKit would mount an empty + # secret, the build would install a reduced tool set and exit 0, and the + # result would be tagged with the same hash a credentialed build produces -- + # so every later run would reuse the broken image. + $secretArgs = @() + $secretEnv = @() + $hookPath = Join-Path $repoRoot $hookRel + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + $hook = Anvil-PreBuild + if ($null -ne $hook -and $null -ne $hook.Secrets) { + foreach ($id in $hook.Secrets.Keys) { + if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 + } + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" + } + [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + } + } + } + + try { + # Progress goes to stderr: callers capture this recipe's stdout to learn + # the image reference, so anything else written there becomes part of it. + [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # Pinned, not inferred from the host. The Dockerfile installs amd64 + # toolchains and verifies amd64 checksums, so an arm host would resolve + # the multi-arch base to arm64 and fail late with an exec-format error. + # It also keeps the identity scheme honest: without this, two hosts of + # different architecture compute the same tag for different images. + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } + foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } + $buildCmd += $repoRoot + # BuildKit is required for --secret; docker enables it by default from + # 23.0 but an older daemon silently ignores the flag, so ask explicitly. + $env:DOCKER_BUILDKIT = '1' + & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + } finally { + foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + + Write-Output $image + +# Run any anvil recipe inside the pinned Linux image. +# +# just anvil-container anvil-clippy # one check +# just anvil-container anvil-pr # the whole PR tier +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs +# natively and the work happens exactly once. [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *recipe: - $requested = @('{{ replace(recipe, "'", "''") }}' -split '\s+' | Where-Object { $_ }) - & '.anvil/container/run-in-container.ps1' @requested - exit $LASTEXITCODE +anvil-container *target: + $ErrorActionPreference = 'Stop' + $target = '{{target}}' + if ($env:ANVIL_IN_CONTAINER -eq '1') { + # Already inside: pass straight through instead of nesting. + if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + exit $LASTEXITCODE + } + + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + + # Map the caller's working directory to its in-container equivalent so + # relative paths keep working from a subdirectory. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { + '{{anvil_container_workdir}}' + } else { + '{{anvil_container_workdir}}/' + $rel + } + + $interactive = [string]::IsNullOrWhiteSpace($target) + $runArgs = @('run', '--rm', '--platform', 'linux/amd64') + $runArgs += $interactive ? '-it' : '-i' + $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + # Cargo and rustup homes live in named volumes: the hot write path never + # crosses the host boundary, and the host's own toolchain is untouched. + $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') + $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Match the caller's uid/gid on Linux. Without this everything the run + # writes under the bind mount -- target/, generated files -- lands as root + # on the host, and the next native cargo build or git clean fails with + # EACCES a long way from the cause. Docker Desktop on Windows and macOS + # already maps ownership, and `id` is not there to ask. + if (-not $IsWindows -and -not $IsMacOS) { + $hostUid = (id -u); $hostGid = (id -g) + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + } + $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') + + # Run-time credentials come from the optional hook. Forwarded by NAME, never + # as NAME=VALUE: the engine copies the value out of the environment it + # already inherits, so a credential never appears in the host's process + # command line, where endpoint telemetry records and retains it for far + # longer than a short-lived token is meant to live. + $hookEnv = @() + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + $hook = Anvil-PreRun + if ($null -ne $hook -and $null -ne $hook.Env) { + foreach ($name in $hook.Env.Keys) { + if ([string]::IsNullOrEmpty($hook.Env[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 + } + Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] + $hookEnv += $name + $runArgs += @('-e', $name) + } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + } + } + } + + try { + # --pull=never: the tag names locally-built content, so a miss is a bug + # to surface rather than an invitation to fetch something unrelated. + $runArgs += @('--pull=never', '-w', $containerCwd, $image) + if (-not $interactive) { $runArgs += @('just', $target) } + & $engine @runArgs + exit $LASTEXITCODE + } finally { + foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + +# Report the engine, the exec image, and whether it is present and current. +# +# The tag embeds the hash of the image's inputs, so "absent" and "out of date" +# are the same condition and are reported as one. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-status: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "engine: $engine" + Write-Output "workdir: {{anvil_container_workdir}}" + + # NO_REBUILD turns the resolve into a pure query: report the state instead + # of silently spending several minutes building from a status command. + $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 + $present = $LASTEXITCODE -eq 0 + if ($image) { Write-Output "image: $image" } + if ($present) { + Write-Output "status: present and current" + } else { + Write-Output "status: no image matches the current inputs (it will be built on the next run)" + } + exit 0 -[unix] +# Rebuild the exec image from scratch, ignoring every cached layer. +# +# The ordinary path already rebuilds whenever an input changes, so this is for +# the cases a content hash cannot see: a moved upstream package, a stale base +# layer, or a build that is suspected of being wrong. [group("anvil-container")] -[script("bash")] -anvil-container *recipe: - requested={{ quote(recipe) }} - if [[ -z "$requested" ]]; then - exec bash '.anvil/container/run-in-container.sh' - fi - read -r -a requested_args <<<"$requested" - exec bash '.anvil/container/run-in-container.sh' "${requested_args[@]}" +[script("pwsh", "-NoProfile")] +anvil-container-rebuild: + $ErrorActionPreference = 'Stop' + $env:ANVIL_CONTAINER_NO_CACHE = '1' + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image rebuilt: $image" + exit 0 + +# Remove this repository's cache volumes. The image is left in place. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-down: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + & $engine volume rm -f $vol + } + exit 0 === justfiles/anvil/groups/pr-fast.just === # Copyright (c) Microsoft Corporation. @@ -5179,7 +4412,6 @@ import 'groups/scheduled-test.just' import 'groups/scheduled-advisories.just' import 'groups/scheduled-runtime-analysis.just' import 'groups/scheduled-exhaustive.just' -import 'runner.just' import 'tiers.just' import 'tools.just' import 'versions.just' @@ -5187,55 +4419,6 @@ import 'versions.just' # Friendly default: `just anvil` runs the PR tier. alias anvil := anvil-pr -=== justfiles/anvil/runner.just === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -# Route public tier entry points through the configured execution environment. -# ANVIL_IN_CONTAINER always wins to prevent recursive container launches. -[private] -[no-exit-message] -[windows] -[script("pwsh", "-NoProfile")] -_anvil-run tier runner: - $just = '{{ replace(just_executable(), "'", "''") }}' - $justfile = '{{ replace(justfile(), "'", "''") }}' - $nativeTier = '_anvil-{{ replace(tier, "'", "''") }}' - if ($env:ANVIL_IN_CONTAINER) { - & $just --justfile $justfile $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'container') { - & $just --justfile $justfile anvil-container $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'native') { - & $just --justfile $justfile $nativeTier - } else { - [Console]::Error.WriteLine("anvil-runner: expected 'native' or 'container', got '{{ replace(runner, "'", "''") }}'.") - exit 2 - } - exit $LASTEXITCODE - -[private] -[no-exit-message] -[unix] -[script("bash")] -_anvil-run tier runner: - just_path={{ quote(just_executable()) }} - justfile={{ quote(justfile()) }} - tier={{ quote(tier) }} - runner={{ quote(runner) }} - native_tier="_anvil-$tier" - if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - elif [[ "$runner" == "container" ]]; then - exec "$just_path" --justfile "$justfile" anvil-container "$native_tier" - elif [[ "$runner" == "native" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - else - echo "anvil-runner: expected 'native' or 'container', got '$runner'." >&2 - exit 2 - fi - === justfiles/anvil/tiers.just === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. @@ -5251,10 +4434,7 @@ _anvil-run tier runner: # Run all pull request checks. [group("anvil")] -anvil-pr: (_anvil-run "pr" anvil_runner) - -[private] -_anvil-pr: anvil-pr-validate-prereqs \ +anvil-pr: anvil-pr-validate-prereqs \ anvil-pr-fast \ anvil-pr-slow @@ -5265,10 +4445,7 @@ _anvil-pr: anvil-pr-validate-prereqs \ # Run all scheduled checks. [group("anvil")] -anvil-scheduled: (_anvil-run "scheduled" anvil_runner) - -[private] -_anvil-scheduled: anvil-scheduled-validate-prereqs \ +anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-test \ anvil-scheduled-advisories \ anvil-scheduled-runtime-analysis \ @@ -5276,12 +4453,9 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. [group("anvil")] -anvil-full: (_anvil-run "full" anvil_runner) - -[private] -_anvil-full: anvil-full-validate-prereqs \ - _anvil-pr \ - _anvil-scheduled +anvil-full: anvil-full-validate-prereqs \ + anvil-pr \ + anvil-scheduled # Tier-level + global setup + validate-prereqs # =========================================================================== diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index d84cbea9..842d2242 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2,20 +2,40 @@ source: crates/cargo-anvil/tests/snapshots.rs expression: render_tree(tmp.path()) --- -=== .anvil/container/Containerfile === +=== .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. +# +# Default Anvil execution image, emitted for every repository. The image +# installs exactly the tools the generated catalog pins, by running +# `just anvil-setup` -- the same recipe the checks themselves use. That is what +# makes "the image has the right tools" true by construction rather than by +# convention: there is no second list to keep in step. +# +# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the +# image identity hash and refuses a floating tag, because a tag that can change +# underneath a hash makes the hash a lie. +# +# To build on a different base (a lower glibc baseline, or an internal +# distribution), a downstream catalog replaces this artifact wholesale via +# `replace_artifact(artifacts::container::dockerfile(...))`; a single +# repository can edit this file in place, which anvil's drift handling +# preserves. ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e FROM ${BASE_IMAGE} -ARG ANVIL_IMAGE_ID ARG JUST_VERSION=1.56.0 ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 ARG POWERSHELL_VERSION=7.6.3 ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 ARG RUSTUP_VERSION=1.29.0 ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e ENV DEBIAN_FRONTEND=noninteractive \ CARGO_HOME=/usr/local/cargo \ @@ -23,12 +43,17 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare slim base has no C runtime development files, so every +# link step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ libssl-dev pkg-config tar \ && rm -rf /var/lib/apt/lists/* +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. RUN curl -fsSLo /tmp/powershell.tar.gz \ "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ @@ -52,1150 +77,64 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated +# source edit does not invalidate this layer. The synthetic Justfile avoids +# pulling in repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. WORKDIR /opt/anvil -COPY . . -RUN test -f rust-toolchain.toml || { \ - echo "anvil-container requires rust-toolchain.toml" >&2; \ - exit 1; \ - } -RUN --mount=type=cache,id=anvil-cargo-registry,target=/usr/local/cargo/registry \ - --mount=type=cache,id=anvil-cargo-git,target=/usr/local/cargo/git \ - --mount=type=cache,id=anvil-cargo-target,target=/tmp/anvil-target \ - printf "anvil_runner := \"native\"\nimport 'justfiles/anvil/mod.just'\n" > Justfile \ - && CARGO_TARGET_DIR=/tmp/anvil-target just anvil-setup - -COPY .anvil/container/entrypoint.sh /usr/local/bin/anvil-container-entrypoint -RUN chmod 755 /usr/local/bin/anvil-container-entrypoint - +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# Consumed by the re-entry guard in the generated tier and group recipes: a +# recipe that sees this runs natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 -LABEL io.github.cargo-anvil.image-id="${ANVIL_IMAGE_ID}" + WORKDIR /workspace -ENTRYPOINT ["anvil-container-entrypoint"] CMD ["bash"] -=== .anvil/container/Containerfile.dockerignore === -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Deny-all allow-list for the image build context. +=== .anvil/container/Dockerfile.dockerignore === +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil +# crate unless the change is repository-specific. # -# Docker matches each candidate against every pattern in order and lets the -# last match win, testing the path itself *and each of its parent directories* -# (moby/patternmatcher MatchesOrParentMatches). A bare directory re-inclusion -# such as `!justfiles` therefore re-admits the entire subtree below it, which -# would defeat this allow-list, so list only leaf patterns here. Docker still -# descends into a denied directory when some re-inclusion pattern is prefixed -# by it, so the intermediate directories need no entries of their own. +# BuildKit reads `.dockerignore` in preference to a root +# `.dockerignore`, so this scopes the exec-image build context without the +# repository having to own a root ignore file or having one silently overridden. # -# Parent testing also reaches through a single-segment re-inclusion: a -# subdirectory of `.anvil/container/` matches `!.anvil/container/*` in its own -# right. The image-ID helpers list that directory one level deep, so a nested -# file is not an image input; `.anvil/container/*/*` states that leaf-only -# contract in the allow-list too, at every depth, because a deeper candidate -# always has an ancestor of exactly that shape. -** +# The build context is the repository root but the image only needs two things. +# Excluding everything else keeps a cold build from streaming the whole +# worktree (and every stale `target/`) to the daemon. +* +!justfiles !rust-toolchain.toml -!justfiles/anvil/*.just -!justfiles/anvil/checks/*.just -!justfiles/anvil/groups/*.just -!.anvil/container/* -.anvil/container/*/* -.anvil/container/customize.sh -.anvil/container/customize.ps1 - -=== .anvil/container/README.md === - - -# Run Anvil checks in a local container - -Use `just anvil-container` to run generated Anvil checks in a reproducible -Linux environment without installing the complete Rust and Cargo tool catalog -on the host. - -Native execution remains the default. The first container run builds an image -matching the repository's generated configuration. Later runs reuse that image, -dependency caches, and compilation output. - -## Quick start - -Ensure Docker Engine is running, then run: - -```text -just anvil-container anvil-clippy -``` - -The first run builds the matching image and can take several minutes. - -## Prerequisites - -- [Docker Engine](https://docs.docker.com/engine/install/) 23.0 or newer, - installed directly in Linux or WSL and usable by the current user. -- `git` and `just` on the host. -- Bash on Linux and WSL; PowerShell Core (`pwsh`) and WSL 2 on Windows. -- `[script]` support enabled in the root `Justfile`. Add `set unstable` when - required by the installed `just` version. -- A `rust-toolchain.toml` in the repository root. -- A Linux or WSL environment capable of running `linux/amd64` images, either - natively on x86-64 or through Docker emulation on ARM64. - -On Windows, the driver invokes Docker from the default WSL distribution rather -than calling Windows `docker.exe`. Regardless of how Docker is installed, this -command must succeed from PowerShell: - -```text -wsl -e docker version -``` - -Start the Docker service inside WSL when it is stopped and add the WSL user to -the `docker` group when non-root access is not already configured. Docker -Desktop is not required. - -On ARM64 hosts, Docker emulates the required `linux/amd64` environment. Image -builds and checks can therefore be substantially slower than on x86-64 hosts. - -## Security boundary - -> [!WARNING] -> `customize.sh` and `customize.ps1` execute on the host with the developer's -> permissions before container isolation begins. Reviewing and trusting these -> files is equivalent to reviewing and trusting any other host-executed script -> in the checked-out branch. - -## Common workflows - -Run one check: - -```text -just anvil-container anvil-clippy -``` - -Run the complete pull-request tier: - -```text -just anvil-container anvil-pr -``` - -Every argument is treated as a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not supported by this command surface. - -Open an interactive Bash shell in the image: - -```text -just anvil-container -``` - -### Use containers for tier commands - -Native execution remains the default. To route tier commands such as -`just anvil-pr` through the container for the current shell: - -```powershell -$env:ANVIL_RUNNER = "container" -just anvil-pr -``` - -On Unix: - -```sh -ANVIL_RUNNER=container just anvil-pr -``` - -For one invocation: - -```text -just anvil_runner=container anvil-pr -``` - -To make container execution the repository default, change the default value -in the `anvil-runner` region of the repository-root `Justfile` from `"native"` -to `"container"` and commit that policy. Set `ANVIL_RUNNER=native` to override -the repository default for the current shell. - -Tier routing starts a nested `just` invocation. Output and exit status are -preserved, but outer `--dry-run`, dependency introspection, global options, and -CLI variable assignments are not propagated to the selected private tier. -Values other than `native` and `container` are rejected. - -## Images and caches - -The image name includes a content-based tag derived from the repository's Rust -toolchain, generated Anvil recipes, and container build configuration. A -relevant change selects a new image automatically; older branches can continue -using their matching images. - -The following data is reused between runs: - -- the matching container image; -- repository-scoped Cargo registry and Cargo Git caches; -- compilation output in a repository- and image-specific `target` volume. - -The repository is mounted read/write at `/workspace`. Build output remains in a -named volume instead of the host `target/`, avoiding incompatible artifacts and -slow host-to-virtual-machine I/O. - -## GitHub authentication - -`anvil-aprz` and aggregate tiers that include it require GitHub API -authentication. The driver uses either: - -- the host `GITHUB_TOKEN`; or -- the token from an authenticated host `gh` session. - -Trusted customization can provision a short-lived token by setting -`GITHUB_TOKEN`; the driver reads it after loading and validating customization. - -Authenticate the GitHub CLI with: - -```text -gh auth login --hostname github.com -``` - -For an aggregate tier, the driver first runs `anvil-aprz` in a short-lived -container with the token mounted read-only. After it succeeds, the driver runs -the remaining checks in another container without the token. Temporary token -files are removed afterward. - -An interactive invocation can pause while you authenticate. A non-interactive -invocation fails with instructions when authentication is unavailable. - -## Configuration - -| Variable | Effect | -|---|---| -| `ANVIL_RUNNER` | Selects `native` or `container` execution for tier commands | -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a digest-pinned compatible Linux base image and changes the content-based tag | -| `ANVIL_CONTAINER_IMAGE` | Changes the local image name; the content-based tag is retained | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails instead of building when the matching image is absent | - -The public driver builds images locally and does not pull -`ANVIL_CONTAINER_IMAGE` from a registry. - -The default base is digest-pinned Debian Bookworm. Set -`ANVIL_CONTAINER_BASE_IMAGE` to another image compatible with the generated -Debian-based `Containerfile` when a lower glibc baseline is required. A -different package ecosystem such as Azure Linux requires a derived -`Containerfile`. The value must use `image@sha256:` form so the -selected base remains part of the content-addressed image identity. - -Two simultaneous cold invocations can both build the same missing image. This -is accepted for local development: the content-addressed tag converges on the -same inputs, at the cost of duplicate work. - -## Troubleshooting - -| Problem | Resolution | -|---|---| -| Docker is not found on Linux or WSL | Install Docker Engine 23.0 or newer inside that environment | -| Docker is unavailable from Windows | Run `wsl -e docker version`; install or start Docker Engine in the default WSL distribution | -| Docker requires elevated access | Add the Linux/WSL user to the `docker` group, then start a new shell | -| ARM64 execution is slow | The current image is `linux/amd64` and runs through Docker emulation | -| `linux/amd64` cannot run | Configure Docker to run `linux/amd64` images | -| `[script]` recipes are unavailable | Enable `[script]` support; older `just` versions require `set unstable` | -| `rust-toolchain.toml` is missing | Add the repository-owned toolchain file at the repository root | -| GitHub authentication is unavailable | Run `gh auth login --hostname github.com` or set host `GITHUB_TOKEN` | -| A matching image is missing with `ANVIL_CONTAINER_NO_REBUILD=1` | Unset the variable to allow the local image build | -| The first run is slow | The initial image build installs the pinned tool catalog; later runs reuse it | - -Use `docker images anvil-dev` inside Linux or WSL to list locally cached -default Anvil images. - -## Managed files - -This directory is managed by `cargo-anvil`. Regenerate it with `cargo anvil` -instead of editing its files directly. - -> [!IMPORTANT] -> These assets previously lived in `justfiles/anvil/container/`. `cargo anvil` -> relocates the files it generated, but it does not track a hand-authored -> `customize.sh` or `customize.ps1`. Move any such file to -> `.anvil/container/` yourself; the driver only loads customization from the -> new location and warns on stderr when it finds one left behind. - -## Advanced repository customization - -A repository or derived catalog can add one trusted customization file per -supported host: - -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 -``` - -The driver sources the matching file as trusted host code before authentication, -image construction, and recipe execution. The documented customization -contract provides inputs and validated outputs for APRZ classification, build -secrets, dependency preparation, runtime arguments, and cleanup. - -Customization source is excluded from image identity and the build context. -Non-secret image behavior must be represented by hashed static files such as -the `Containerfile`, entrypoint, or supporting build scripts. - -See the [container customization contract](https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md#8-container-customization) -for the complete interface and security requirements. - -=== .anvil/container/entrypoint.sh === -#!/bin/sh -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -eu - -if [ "$(id -u)" -ne 0 ]; then - if [ -z "${HOME:-}" ] || [ "$HOME" = "/" ]; then - HOME="/tmp/anvil-user" - export HOME - fi - - user_cargo_home="$HOME/.cargo" - mkdir -p "$user_cargo_home" - for file in config.toml .crates.toml .crates2.json; do - if [ -r "$CARGO_HOME/$file" ]; then - cp -f "$CARGO_HOME/$file" "$user_cargo_home/$file" - fi - done - export CARGO_HOME="$user_cargo_home" - ln -sfn /usr/local/cargo/registry "$CARGO_HOME/registry" - ln -sfn /usr/local/cargo/git "$CARGO_HOME/git" -fi - -exec "$@" - -=== .anvil/container/image-id.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param() - -$ErrorActionPreference = 'Stop' - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$inputs = @( - 'rust-toolchain.toml' -) -$toolchainPath = Join-Path $repoRoot 'rust-toolchain.toml' -if (-not (Test-Path -LiteralPath $toolchainPath -PathType Leaf)) { - throw 'anvil-container requires a repository-owned rust-toolchain.toml.' -} -$containerPath = Join-Path $repoRoot '.anvil/container' -$containerRecipe = 'justfiles/anvil/container.just' -$containerfile = Join-Path $containerPath 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$pathComparison = if ($IsWindows) { [StringComparison]::OrdinalIgnoreCase } else { [StringComparison]::Ordinal } -# The container entry recipe drives execution on the host; it is not image -# content, so it must not participate in image identity. -$inputs += Get-ChildItem (Join-Path $repoRoot 'justfiles/anvil') -Recurse -File -Filter '*.just' | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } | - Where-Object { -not $_.Equals($containerRecipe, $pathComparison) } -$executionOnly = @( - 'image-id.ps1', - 'image-id.sh', - 'README.md', - 'run-in-container.ps1', - 'run-in-container.sh', - 'customize.sh', - 'customize.ps1' -) -# customize.sh/customize.ps1 are trusted runtime orchestration, not image -# content: their source must never affect the image ID or build context. -# Static, non-secret build customization belongs in a hashed artifact instead. -$inputs += Get-ChildItem $containerPath -File | - Where-Object { $_.Name -notin $executionOnly } | - ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } -$uniqueInputs = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) -foreach ($inputPath in $inputs) { - [void]$uniqueInputs.Add($inputPath) -} -$inputs = [string[]]$uniqueInputs -[Array]::Sort($inputs, [StringComparer]::Ordinal) - -$payload = [Text.StringBuilder]::new() -[void]$payload.Append("ANVIL_CONTAINER_BASE_IMAGE`n").Append($baseImage).Append("`n") -foreach ($relative in $inputs) { - $path = Join-Path $repoRoot $relative - if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { - throw "Container image input is missing: $relative" - } - $content = [IO.File]::ReadAllText($path).Replace("`r`n", "`n").Replace("`r", "`n") - [void]$payload.Append($relative).Append("`n").Append($content).Append("`n") -} - -$bytes = [Text.Encoding]::UTF8.GetBytes($payload.ToString()) -$hash = [Security.Cryptography.SHA256]::HashData($bytes) -Write-Output ([Convert]::ToHexString($hash).ToLowerInvariant()) - -=== .anvil/container/image-id.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi - -toolchain_path="$repo_root/rust-toolchain.toml" -if [[ ! -f "$toolchain_path" ]]; then - echo 'anvil-container requires a repository-owned rust-toolchain.toml.' >&2 - exit 1 -fi - -container_dir="$repo_root/.anvil/container" -container_recipe="justfiles/anvil/container.just" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$container_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -inputs=(rust-toolchain.toml) -while IFS= read -r path; do - relative="${path#"$repo_root"/}" - # The container entry recipe drives execution on the host; it is not - # image content, so it must not participate in image identity. - if [[ "$relative" != "$container_recipe" ]]; then - inputs+=("$relative") - fi -done < <(find "$repo_root/justfiles/anvil" -type f -name '*.just' -print) - -for path in "$container_dir"/*; do - [[ -f "$path" ]] || continue - case "${path##*/}" in - image-id.ps1 | image-id.sh | README.md \ - | run-in-container.ps1 | run-in-container.sh \ - | customize.sh | customize.ps1) continue ;; - esac - inputs+=("${path#"$repo_root"/}") -done - -if command -v sha256sum >/dev/null 2>&1; then - hash_command=(sha256sum) -elif command -v shasum >/dev/null 2>&1; then - hash_command=(shasum -a 256) -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi - -write_normalized_file() { - local path="$1" - local line status - while true; do - line="" - if IFS= read -r line <&3; then - status=0 - else - status=$? - fi - if ((status != 0)) && [[ -z "$line" ]]; then - break - fi - printf '%s' "${line%$'\r'}" - if ((status == 0)); then - printf '\n' - else - break - fi - done 3<"$path" -} - -{ - printf 'ANVIL_CONTAINER_BASE_IMAGE\n%s\n' "$base_image" - while IFS= read -r relative; do - path="$repo_root/$relative" - if [[ ! -f "$path" ]]; then - echo "Container image input is missing: $relative" >&2 - exit 1 - fi - printf '%s\n' "$relative" - write_normalized_file "$path" - printf '\n' - done < <(printf '%s\n' "${inputs[@]}" | LC_ALL=C sort -u) -} | "${hash_command[@]}" | awk '{print $1}' - -=== .anvil/container/run-in-container.ps1 === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -[CmdletBinding()] -param( - [Parameter(Position = 0, ValueFromRemainingArguments = $true)] - [string[]]$Recipe -) - -$ErrorActionPreference = 'Stop' - -function ConvertTo-AnvilVersion([string]$Value) { - $match = [regex]::Match($Value, '^(\d+)\.(\d+)(?:\.(\d+))?') - if (-not $match.Success) { - throw "anvil-container: could not parse Docker Engine version '$Value'." - } - [version]::new( - [int]$match.Groups[1].Value, - [int]$match.Groups[2].Value, - $(if ($match.Groups[3].Success) { [int]$match.Groups[3].Value } else { 0 }) - ) -} - -function Test-AnvilContainerStringArray([string]$Name, $Value) { - if ($Value -isnot [array]) { - throw "anvil-container: `$$Name must be a string array." - } - foreach ($item in $Value) { - if ($item -isnot [string] -or [string]::IsNullOrEmpty($item)) { - throw "anvil-container: `$$Name entries must be non-empty strings." - } - } -} - -function Test-AnvilContainerBuildArgs($Value) { - for ($index = 0; $index -lt $Value.Count; $index++) { - $item = $Value[$index] - if ($item -eq '--secret') { - $index++ - if ($index -ge $Value.Count) { - throw 'anvil-container: $AnvilContainerBuildArgs requires a value after --secret.' - } - } elseif (-not $item.StartsWith('--secret=', [StringComparison]::Ordinal)) { - throw 'anvil-container: $AnvilContainerBuildArgs accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' - } - } -} - -function Test-AnvilRecipeNeedsGitHubToken([string]$Name) { - $Name -in @( - 'anvil-aprz', - 'anvil-pr', - '_anvil-pr', - 'anvil-pr-fast', - 'anvil-scheduled', - '_anvil-scheduled', - 'anvil-scheduled-advisories', - 'anvil-full', - '_anvil-full' - ) -} - -function Get-AnvilGitHubToken { - $token = $env:GITHUB_TOKEN - if (-not $token -and (Get-Command gh -ErrorAction SilentlyContinue)) { - try { - $token = (& gh auth token --hostname github.com 2>$null) - if ($LASTEXITCODE -ne 0) { $token = $null } - } catch { - $token = $null - } - } - if ($token) { $token = $token.Trim() } - if ($token) { return $token } - return $null -} - -if ($env:ANVIL_IN_CONTAINER) { - if ($Recipe.Count -eq 0) { & bash } else { & just @Recipe } - exit $LASTEXITCODE -} - -foreach ($recipeArg in $Recipe) { - if ($recipeArg -notmatch '^_?anvil-[A-Za-z0-9-]+$') { - throw "anvil-container: expected each argument to be an anvil-* recipe, got '$recipeArg'." - } -} - -if (-not (Get-Command wsl -ErrorAction SilentlyContinue)) { - throw 'anvil-container: WSL 2 is required. See .anvil/container/README.md.' -} - -$versionText = (& wsl -e docker version --format '{{.Server.Version}}' 2>$null) -if ($LASTEXITCODE -ne 0 -or -not $versionText) { - throw 'anvil-container: `wsl -e docker version` must succeed. Install or start Docker Engine in the default WSL distribution; this driver does not invoke Windows docker.exe.' -} -$versionText = $versionText.Trim() -if ((ConvertTo-AnvilVersion $versionText) -lt [version]'23.0.0') { - throw "anvil-container: Docker Engine 23.0.0 or newer is required (found $versionText)." -} -$wslArchitecture = (& wsl -e uname -m 2>$null) -if ($LASTEXITCODE -eq 0 -and $wslArchitecture) { - $wslArchitecture = $wslArchitecture.Trim() - if ($wslArchitecture -notin @('x86_64', 'amd64')) { - [Console]::Error.WriteLine( - "anvil-container: warning: $wslArchitecture requires emulation for linux/amd64; builds and checks may be substantially slower." - ) - } -} - -$repoRoot = (git rev-parse --show-toplevel 2>$null).Trim() -if ($LASTEXITCODE -ne 0 -or -not $repoRoot) { - throw 'anvil-container must run from a Git repository.' -} - -$scriptDir = Join-Path $repoRoot '.anvil/container' -$wslRepoRoot = (& wsl -e wslpath -a $repoRoot).Trim() -if ($LASTEXITCODE -ne 0 -or -not $wslRepoRoot) { - throw 'anvil-container: could not translate the repository path into the default WSL distribution.' -} -$wslScriptDir = "$wslRepoRoot/.anvil/container" -$containerfile = Join-Path $scriptDir 'Containerfile' -$baseImageMatch = [regex]::Match([IO.File]::ReadAllText($containerfile), '(?m)^ARG BASE_IMAGE=([^\r\n]+)') -if (-not $baseImageMatch.Success) { - throw 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' -} -$defaultBaseImage = $baseImageMatch.Groups[1].Value -$baseImage = if ($env:ANVIL_CONTAINER_BASE_IMAGE) { $env:ANVIL_CONTAINER_BASE_IMAGE } else { $defaultBaseImage } -if ($baseImage -notmatch '@sha256:[0-9a-fA-F]{64}$') { - throw 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' -} -$imageId = (& (Join-Path $scriptDir 'image-id.ps1')).Trim() -$imageBase = if ($env:ANVIL_CONTAINER_IMAGE) { $env:ANVIL_CONTAINER_IMAGE } else { 'anvil-dev' } -$image = "${imageBase}:$imageId" -$repoBytes = [Text.Encoding]::UTF8.GetBytes($wslRepoRoot) -$repoHash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($repoBytes)).ToLowerInvariant() -$targetVolume = "anvil-target-$($repoHash.Substring(0, 12))-$($imageId.Substring(0, 12))" - -$needsGitHubToken = $false -foreach ($recipeArg in $Recipe) { - if (Test-AnvilRecipeNeedsGitHubToken $recipeArg) { - $needsGitHubToken = $true - break - } -} -$runsOnlyGitHubCheck = $Recipe.Count -eq 1 -and $Recipe[0] -eq 'anvil-aprz' - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -$null = & wsl -e docker image inspect $image 2>$null -$imageExists = $LASTEXITCODE -eq 0 - -New-Variable -Name AnvilContainerRepoRoot -Value $repoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDir -Value $scriptDir -Option ReadOnly -New-Variable -Name AnvilContainerRepoRootWsl -Value $wslRepoRoot -Option ReadOnly -New-Variable -Name AnvilContainerDirWsl -Value $wslScriptDir -Option ReadOnly -New-Variable -Name AnvilContainerResolvedImage -Value $image -Option ReadOnly -New-Variable -Name AnvilContainerImageExists -Value $imageExists -Option ReadOnly -New-Variable -Name AnvilContainerRequestedRecipes -Value $Recipe -Option ReadOnly -New-Variable -Name AnvilContainerHostIsWindows -Value ([bool]$IsWindows) -Option ReadOnly - -# Customization outputs, initialized before sourcing so a missing customize.ps1 -# leaves every phase a documented no-op. -$AnvilContainerBuildArgs = @() -$AnvilContainerPrepareArgs = @() -$AnvilContainerPrepareCommand = @() -$AnvilContainerRunArgs = @() -$AnvilContainerNeedsGitHubToken = $needsGitHubToken -$AnvilContainerCleanup = $null -$githubToken = $null -$githubTokenFile = $null -$exitCode = 0 -$customizeScript = Join-Path $scriptDir 'customize.ps1' -$legacyCustomizeScript = Join-Path $repoRoot 'justfiles/anvil/container/customize.ps1' - -try { - if (Test-Path -LiteralPath $customizeScript -PathType Leaf) { - . $customizeScript - } - elseif (Test-Path -LiteralPath $legacyCustomizeScript -PathType Leaf) { - [Console]::Error.WriteLine( - 'anvil-container: warning: ignoring justfiles/anvil/container/customize.ps1; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.ps1 to keep it active.' - ) - } - - Test-AnvilContainerStringArray 'AnvilContainerBuildArgs' $AnvilContainerBuildArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareArgs' $AnvilContainerPrepareArgs - Test-AnvilContainerStringArray 'AnvilContainerPrepareCommand' $AnvilContainerPrepareCommand - Test-AnvilContainerStringArray 'AnvilContainerRunArgs' $AnvilContainerRunArgs - Test-AnvilContainerBuildArgs $AnvilContainerBuildArgs - if ($AnvilContainerNeedsGitHubToken -isnot [bool]) { - throw 'anvil-container: $AnvilContainerNeedsGitHubToken must be a Boolean.' - } - $needsGitHubToken = $needsGitHubToken -or $AnvilContainerNeedsGitHubToken - if ($AnvilContainerPrepareArgs.Count -gt 0 -and $AnvilContainerPrepareCommand.Count -eq 0) { - throw 'anvil-container: $AnvilContainerPrepareArgs requires $AnvilContainerPrepareCommand.' - } - if ($AnvilContainerCleanup -and $AnvilContainerCleanup -isnot [scriptblock]) { - throw 'anvil-container: $AnvilContainerCleanup must be a script block.' - } - $githubToken = if ($needsGitHubToken) { Get-AnvilGitHubToken } else { $null } - if ($needsGitHubToken -and -not $githubToken) { - if (-not (Get-Command gh -ErrorAction SilentlyContinue)) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' - } - if (-not [Environment]::UserInteractive -or [Console]::IsInputRedirected) { - throw 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' - } - Write-Host 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' - [void](Read-Host 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel)') - $githubToken = Get-AnvilGitHubToken - if (-not $githubToken) { - throw 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' - } - } - if (-not $imageExists) { - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - throw "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." - } - & wsl -e docker build ` - --platform linux/amd64 ` - --tag $image ` - --file "$wslScriptDir/Containerfile" ` - --build-arg "ANVIL_IMAGE_ID=$imageId" ` - --build-arg "BASE_IMAGE=$baseImage" ` - @AnvilContainerBuildArgs ` - $wslRepoRoot - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker build failed with exit code $LASTEXITCODE." - } - } - - $containerUid = (& wsl -e id -u).Trim() - $containerGid = (& wsl -e id -g).Trim() - if ($containerUid -notmatch '^\d+$' -or $containerGid -notmatch '^\d+$') { - throw 'anvil-container: could not determine the default WSL user identity.' - } - $registryVolume = "anvil-cargo-registry-$($repoHash.Substring(0, 12))" - $gitVolume = "anvil-cargo-git-$($repoHash.Substring(0, 12))" - foreach ($volume in @($registryVolume, $gitVolume, $targetVolume)) { - $null = & wsl -e docker volume create $volume - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume creation failed for '$volume' with exit code $LASTEXITCODE." - } - } - $mountArgs = @( - '--mount', "type=bind,source=$wslRepoRoot,target=/workspace", - '--mount', "type=volume,source=$registryVolume,target=/usr/local/cargo/registry", - '--mount', "type=volume,source=$gitVolume,target=/usr/local/cargo/git", - '--mount', "type=volume,source=$targetVolume,target=/workspace/target" - ) - & wsl -e docker run --rm --pull=never ` - --platform linux/amd64 ` - --user 0:0 ` - @mountArgs ` - $image sh -c "chown ${containerUid}:${containerGid} /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: Docker volume initialization failed with exit code $LASTEXITCODE." - } - - $runArgs = @( - 'run', '--rm', '--pull=never', - '--platform', 'linux/amd64', - '--user', "${containerUid}:${containerGid}", - '--env', 'ANVIL_IN_CONTAINER=1', - '--env', 'HOME=/tmp/anvil-user', - '--workdir', '/workspace' - ) - $runArgs += $mountArgs - $prepareRunArgs = @($runArgs) - $runArgs += $AnvilContainerRunArgs - foreach ($name in @( - 'PR_TITLE', - 'BASE_REF', - 'ANVIL_INCLUDE_MODIFIED', - 'ANVIL_INCLUDE_AFFECTED', - 'ANVIL_INCLUDE_REQUIRED', - 'GITHUB_BASE_REF', - 'SYSTEM_PULLREQUEST_TARGETBRANCH' - )) { - if (Test-Path "Env:$name") { - $runArgs += @('--env', "$name=$((Get-Item "Env:$name").Value)") - } - } - if ($AnvilContainerPrepareCommand.Count -gt 0) { - & wsl -e docker @prepareRunArgs @AnvilContainerPrepareArgs $image @AnvilContainerPrepareCommand - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: preparation command failed with exit code $LASTEXITCODE." - } - } - - if ($githubToken) { - $githubTokenFile = Join-Path ([IO.Path]::GetTempPath()) "anvil-github-token-$PID-$([guid]::NewGuid().ToString('N'))" - [IO.File]::Create($githubTokenFile).Dispose() - if ($IsWindows) { - $userSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value - & icacls.exe $githubTokenFile '/inheritance:r' '/grant:r' "*$($userSid):(F)" | Out-Null - } else { - & chmod 600 $githubTokenFile - } - if ($LASTEXITCODE -ne 0) { - throw 'anvil-container: failed to restrict permissions on the temporary GitHub token file.' - } - [IO.File]::WriteAllText($githubTokenFile, $githubToken, [Text.Encoding]::ASCII) - $githubToken = $null - $wslTokenFile = (& wsl -e wslpath -a $githubTokenFile).Trim() - if ($LASTEXITCODE -ne 0 -or -not $wslTokenFile) { - throw 'anvil-container: could not translate the temporary GitHub token path into WSL.' - } - $githubRunArgs = @($runArgs) - $githubRunArgs += @( - '--mount', - "type=bind,source=$wslTokenFile,target=/run/secrets/anvil-github-token,readonly" - ) - if ($runsOnlyGitHubCheck) { - $runArgs = $githubRunArgs - } else { - & wsl -e docker @githubRunArgs $image just anvil-aprz - if ($LASTEXITCODE -ne 0) { - throw "anvil-container: isolated anvil-aprz failed with exit code $LASTEXITCODE." - } - $runArgs += @('--env', 'ANVIL_APRZ_ALREADY_RAN=1') - } - } - - if ($Recipe.Count -eq 0) { - & wsl -e docker @runArgs --interactive --tty $image bash - } else { - & wsl -e docker @runArgs $image just @Recipe - } - $exitCode = $LASTEXITCODE -} finally { - if ($githubTokenFile) { - Remove-Item -LiteralPath $githubTokenFile -Force -ErrorAction SilentlyContinue - } - if ($AnvilContainerCleanup) { & $AnvilContainerCleanup } -} - -exit $exitCode - -=== .anvil/container/run-in-container.sh === -#!/usr/bin/env bash -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -set -euo pipefail - -if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - if (($# == 0)); then exec bash; else exec just "$@"; fi -fi - -for recipe_arg in "$@"; do - if [[ ! "$recipe_arg" =~ ^_?anvil-[A-Za-z0-9-]+$ ]]; then - echo "anvil-container: expected each argument to be an anvil-* recipe, got '$recipe_arg'." >&2 - exit 2 - fi -done - -anvil_recipe_needs_github_token() { - case "$1" in - anvil-aprz | anvil-pr | _anvil-pr | anvil-pr-fast \ - | anvil-scheduled | _anvil-scheduled | anvil-scheduled-advisories \ - | anvil-full | _anvil-full) return 0 ;; - *) return 1 ;; - esac -} - -version_at_least() { - local found="${1%%[-+]*}" - local required="${2%%[-+]*}" - local found_major found_minor found_patch found_extra - local required_major required_minor required_patch required_extra - IFS=. read -r found_major found_minor found_patch found_extra <<<"$found" - IFS=. read -r required_major required_minor required_patch required_extra <<<"$required" - found_patch="${found_patch:-0}" - required_patch="${required_patch:-0}" - for component in \ - "$found_major" "$found_minor" "$found_patch" \ - "$required_major" "$required_minor" "$required_patch" - do - case "$component" in - '' | *[!0-9]*) return 2 ;; - esac - done - if ((found_major != required_major)); then ((found_major > required_major)); return; fi - if ((found_minor != required_minor)); then ((found_minor > required_minor)); return; fi - ((found_patch >= required_patch)) -} - -command -v docker >/dev/null 2>&1 || { - echo "anvil-container: Docker Engine is required. See .anvil/container/README.md." >&2 - exit 1 -} - -version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || { - echo "anvil-container: Docker Engine is unavailable. Start the Docker service and ensure the current user can access it." >&2 - exit 1 -} -minimum="23.0.0" -if ! version_at_least "$version" "$minimum"; then - echo "anvil-container: Docker Engine $minimum or newer is required (found $version)." >&2 - exit 1 -fi -host_arch="$(uname -m 2>/dev/null || true)" -case "$host_arch" in - x86_64 | amd64 | '') ;; - *) echo "anvil-container: warning: $host_arch requires emulation for linux/amd64; builds and checks may be substantially slower." >&2 ;; -esac - -if ! repo_root="$(git rev-parse --show-toplevel 2>/dev/null)"; then - echo 'anvil-container must run from a Git repository.' >&2 - exit 1 -fi -script_dir="$repo_root/.anvil/container" -default_base_image="$(sed -n 's/^ARG BASE_IMAGE=//p' "$script_dir/Containerfile" | head -n 1)" -if [[ -z "$default_base_image" ]]; then - echo 'anvil-container: Containerfile must define ARG BASE_IMAGE=.' >&2 - exit 1 -fi -base_image="${ANVIL_CONTAINER_BASE_IMAGE:-$default_base_image}" -if [[ ! "$base_image" =~ @sha256:[0-9a-fA-F]{64}$ ]]; then - echo 'anvil-container: ANVIL_CONTAINER_BASE_IMAGE must be pinned by sha256 digest (image@sha256:<64 hex characters>).' >&2 - exit 1 -fi -image_id="$(bash "$script_dir/image-id.sh")" -image_base="${ANVIL_CONTAINER_IMAGE:-anvil-dev}" -image="${image_base}:${image_id}" -if command -v sha256sum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | sha256sum | cut -c1-12)" -elif command -v shasum >/dev/null 2>&1; then - repo_id="$(printf '%s' "$repo_root" | shasum -a 256 | cut -c1-12)" -else - echo 'anvil-container: sha256sum or shasum is required.' >&2 - exit 1 -fi -target_volume="anvil-target-${repo_id}-${image_id:0:12}" - -needs_github_token=false -for recipe_arg in "$@"; do - if anvil_recipe_needs_github_token "$recipe_arg"; then - needs_github_token=true - break - fi -done -runs_only_github_check=false -if (($# == 1)) && [[ "$1" == "anvil-aprz" ]]; then - runs_only_github_check=true -fi -github_token="" - -# Customization contract: check warm/cold state before sourcing so -# customization needed only for image construction can be skipped on a warm -# run, then expose read-only inputs. See docs/design/containers.md. -if docker image inspect "$image" >/dev/null 2>&1; then - image_exists=true -else - image_exists=false -fi - -readonly ANVIL_CONTAINER_REPO_ROOT="$repo_root" -readonly ANVIL_CONTAINER_DIR="$script_dir" -readonly ANVIL_CONTAINER_RESOLVED_IMAGE="$image" -readonly ANVIL_CONTAINER_IMAGE_EXISTS="$image_exists" -declare -a ANVIL_CONTAINER_REQUESTED_RECIPES=("$@") -readonly ANVIL_CONTAINER_REQUESTED_RECIPES - -# Customization outputs, initialized before sourcing so a missing customize.sh -# leaves every phase a documented no-op. -ANVIL_CONTAINER_BUILD_ARGS=() -ANVIL_CONTAINER_PREPARE_ARGS=() -ANVIL_CONTAINER_PREPARE_COMMAND=() -ANVIL_CONTAINER_RUN_ARGS=() -ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN="$needs_github_token" -ANVIL_CONTAINER_CLEANUP=: -github_token_file="" -cleanup() { - if [[ -n "$github_token_file" ]]; then rm -f -- "$github_token_file"; fi - "$ANVIL_CONTAINER_CLEANUP" -} -trap cleanup EXIT - -customize_script="$script_dir/customize.sh" -legacy_customize_script="$repo_root/justfiles/anvil/container/customize.sh" -if [[ ! -f "$customize_script" && -f "$legacy_customize_script" ]]; then - echo "anvil-container: warning: ignoring justfiles/anvil/container/customize.sh; container assets moved to .anvil/container/. Move the file to .anvil/container/customize.sh to keep it active." >&2 -fi -if [[ -f "$customize_script" ]]; then - # shellcheck source=/dev/null - source "$customize_script" -fi - -# Bash 3.2 has neither namerefs (the nameref flag on `local`/`declare`, Bash -# 4.3+) nor safe `set -u` expansion of empty-but- -# declared arrays (fixed in Bash 4.4). Elements are passed positionally -# instead of by nameref, and every expansion of a possibly-empty array uses -# the `${arr[@]+"${arr[@]}"}` idiom: unset/empty-under-old-Bash arrays vanish -# entirely instead of raising "unbound variable", while non-empty arrays -# still expand element-for-element. -anvil_container_validate_array() { - local name="$1" - shift - local declaration value - declaration="$(declare -p "$name" 2>/dev/null || true)" - if [[ ! "$declaration" =~ ^declare\ -[^[:space:]]*a[^[:space:]]*\ ]]; then - echo "anvil-container: $name must be a string array." >&2 - exit 1 - fi - for value in "$@"; do - if [[ -z "$value" ]]; then - echo "anvil-container: $name entries must be non-empty strings." >&2 - exit 1 - fi - done -} -anvil_container_validate_build_args() { - local expect_secret_value=false value - for value in "$@"; do - if "$expect_secret_value"; then - expect_secret_value=false - continue - fi - case "$value" in - --secret) expect_secret_value=true ;; - --secret=*) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS accepts only BuildKit --secret arguments; static build behavior must use hashed container files.' >&2 - exit 1 - ;; - esac - done - if "$expect_secret_value"; then - echo 'anvil-container: ANVIL_CONTAINER_BUILD_ARGS requires a value after --secret.' >&2 - exit 1 - fi -} -anvil_container_validate_array ANVIL_CONTAINER_BUILD_ARGS ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_ARGS ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_PREPARE_COMMAND ${ANVIL_CONTAINER_PREPARE_COMMAND[@]+"${ANVIL_CONTAINER_PREPARE_COMMAND[@]}"} -anvil_container_validate_array ANVIL_CONTAINER_RUN_ARGS ${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"} -anvil_container_validate_build_args ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} -case "$ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN" in - true) needs_github_token=true ;; - false) ;; - *) - echo 'anvil-container: ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN must be true or false.' >&2 - exit 1 - ;; -esac -if ((${#ANVIL_CONTAINER_PREPARE_ARGS[@]} > 0)) && ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} == 0)); then - echo 'anvil-container: ANVIL_CONTAINER_PREPARE_ARGS requires ANVIL_CONTAINER_PREPARE_COMMAND.' >&2 - exit 1 -fi -cleanup_kind="$(type -t "$ANVIL_CONTAINER_CLEANUP" 2>/dev/null || true)" -if [[ "$cleanup_kind" != "function" && "$cleanup_kind" != "builtin" ]]; then - echo "anvil-container: ANVIL_CONTAINER_CLEANUP must name a callable function (got '$ANVIL_CONTAINER_CLEANUP')." >&2 - exit 1 -fi - -if "$needs_github_token"; then - gh_command="" - if command -v gh >/dev/null 2>&1; then - gh_command=gh - elif command -v gh.exe >/dev/null 2>&1; then - gh_command=gh.exe - fi - github_token="${GITHUB_TOKEN:-}" - if [[ -z "$github_token" && -n "$gh_command" ]]; then - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - fi - if [[ -z "$github_token" ]]; then - if [[ -z "$gh_command" ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Install the GitHub CLI and run `gh auth login --hostname github.com`, or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - if [[ ! -t 0 ]]; then - echo 'anvil-container: GitHub authentication is required for anvil-aprz. Run `gh auth login --hostname github.com` or set GITHUB_TOKEN before rerunning.' >&2 - exit 1 - fi - echo 'anvil-container: anvil-aprz requires GitHub authentication to avoid the 60 requests/hour unauthenticated API limit.' >&2 - read -r -p 'Run `gh auth login --hostname github.com` in another terminal, then press Enter to continue (Ctrl+C to cancel) ' - github_token="$("$gh_command" auth token --hostname github.com 2>/dev/null | tr -d '\r' || true)" - if [[ -z "$github_token" ]]; then - echo 'anvil-container: GitHub authentication is still unavailable. Complete `gh auth login --hostname github.com`, then rerun.' >&2 - exit 1 - fi - fi -fi - -if ! "$image_exists"; then - if [[ "${ANVIL_CONTAINER_NO_REBUILD:-}" == "1" ]]; then - echo "anvil-container: image $image is missing and ANVIL_CONTAINER_NO_REBUILD=1." >&2 - exit 1 - else - docker build \ - --platform linux/amd64 \ - --tag "$image" \ - --file "$script_dir/Containerfile" \ - --build-arg "ANVIL_IMAGE_ID=$image_id" \ - --build-arg "BASE_IMAGE=$base_image" \ - ${ANVIL_CONTAINER_BUILD_ARGS[@]+"${ANVIL_CONTAINER_BUILD_ARGS[@]}"} \ - "$repo_root" - fi -fi - -container_uid="$(id -u)" -container_gid="$(id -g)" -registry_volume="anvil-cargo-registry-${repo_id}" -git_volume="anvil-cargo-git-${repo_id}" -for volume in "$registry_volume" "$git_volume" "$target_volume"; do - docker volume create "$volume" >/dev/null -done -mount_args=( - --mount "type=bind,source=$repo_root,target=/workspace" - --mount "type=volume,source=$registry_volume,target=/usr/local/cargo/registry" - --mount "type=volume,source=$git_volume,target=/usr/local/cargo/git" - --mount "type=volume,source=$target_volume,target=/workspace/target" -) -docker run --rm --pull=never \ - --platform linux/amd64 \ - --user 0:0 \ - "${mount_args[@]}" \ - "$image" sh -c \ - "chown $container_uid:$container_gid /usr/local/cargo/registry /usr/local/cargo/git /workspace/target" - -run_args=( - run --rm --pull=never - --platform linux/amd64 - --user "$container_uid:$container_gid" - --env ANVIL_IN_CONTAINER=1 - --env HOME=/tmp/anvil-user - "${mount_args[@]}" - --workdir /workspace -) -prepare_run_args=("${run_args[@]}") -run_args+=(${ANVIL_CONTAINER_RUN_ARGS[@]+"${ANVIL_CONTAINER_RUN_ARGS[@]}"}) -for name in PR_TITLE BASE_REF ANVIL_INCLUDE_MODIFIED ANVIL_INCLUDE_AFFECTED ANVIL_INCLUDE_REQUIRED GITHUB_BASE_REF SYSTEM_PULLREQUEST_TARGETBRANCH; do - if value="$(printenv "$name")"; then run_args+=(--env "$name=$value"); fi -done -if ((${#ANVIL_CONTAINER_PREPARE_COMMAND[@]} > 0)); then - docker "${prepare_run_args[@]}" \ - ${ANVIL_CONTAINER_PREPARE_ARGS[@]+"${ANVIL_CONTAINER_PREPARE_ARGS[@]}"} \ - "$image" \ - "${ANVIL_CONTAINER_PREPARE_COMMAND[@]}" -fi - -if [[ -n "$github_token" ]]; then - github_token_file="$(mktemp "${TMPDIR:-/tmp}/anvil-github-token.XXXXXXXX")" - chmod 600 "$github_token_file" - printf '%s' "$github_token" > "$github_token_file" - unset github_token - github_run_args=( - "${run_args[@]}" - --mount "type=bind,source=$github_token_file,target=/run/secrets/anvil-github-token,readonly" - ) - if "$runs_only_github_check"; then - run_args=("${github_run_args[@]}") - else - docker "${github_run_args[@]}" "$image" just anvil-aprz - run_args+=(--env ANVIL_APRZ_ALREADY_RAN=1) - fi -fi - -if (($# == 0)); then - docker "${run_args[@]}" --interactive --tty "$image" bash - exit $? -fi -docker "${run_args[@]}" "$image" just "$@" === .delta.toml === # >>> anvil-managed: anvil-delta @@ -1332,10 +271,6 @@ clippy.wildcard_imports = "allow" import 'justfiles/anvil/mod.just' # <<< anvil-managed: anvil-imports -# >>> anvil-managed: anvil-runner -anvil_runner := env_var_or_default("ANVIL_RUNNER", "native") -# <<< anvil-managed: anvil-runner - === clippy.toml === # >>> anvil-managed: anvil-clippy # Fine-tuning settings for clippy lints. These cannot be expressed in @@ -1435,11 +370,11 @@ unknown-git = "deny" # cargo-aprz queries the GitHub advisory API. Unauthenticated access is # capped at 60 requests/hour and fails on a full run; an authenticated # token raises the cap to 5000/hour. CI injects GITHUB_TOKEN -# (github.token). Container drivers mount an existing host GITHUB_TOKEN -# or the host gh CLI's stored token as a temporary read-only secret. -# Native runs borrow the gh CLI token directly. Native runs warn and -# proceed unauthenticated if neither is available; container runs fail -# before cargo-aprz can exhaust the unauthenticated rate limit. +# (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the +# gh CLI's stored token (non-interactive: `gh auth token` prints the +# active account's token for github.com and never opens a browser/auth +# prompt). If neither is available we warn with instructions and proceed +# unauthenticated. # # Unscoped (consults external risk DB). @@ -1447,24 +382,14 @@ unknown-git = "deny" [script("pwsh", "-NoProfile")] anvil-aprz: anvil-aprz-validate-prereqs $ErrorActionPreference = 'Stop' - if ($env:ANVIL_APRZ_ALREADY_RAN -eq '1') { - Write-Host 'anvil-aprz: already completed in an isolated authenticated container' - exit 0 - } if (-not $env:GITHUB_TOKEN) { $tok = $null - $containerTokenFile = '/run/secrets/anvil-github-token' - if ($env:ANVIL_IN_CONTAINER -and (Test-Path -LiteralPath $containerTokenFile -PathType Leaf)) { - try { $tok = Get-Content -LiteralPath $containerTokenFile -Raw } catch { $tok = $null } - } elseif (Get-Command gh -ErrorAction SilentlyContinue) { + if (Get-Command gh -ErrorAction SilentlyContinue) { try { $tok = (gh auth token --hostname github.com 2>$null) } catch { $tok = $null } } if ($tok) { $env:GITHUB_TOKEN = $tok.Trim() } else { - if ($env:ANVIL_IN_CONTAINER) { - throw 'anvil-aprz: GitHub authentication is unavailable. Run `gh auth login` on the host or set host GITHUB_TOKEN, then re-run the container command.' - } Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' @@ -3292,27 +2217,335 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Licensed under the MIT License. # GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. +# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md +# +# Containerized execution. `just anvil-container ` runs any anvil +# recipe inside a pinned Linux image; everything else keeps running natively. +# There is no configuration file and no transparent routing: the container is +# reached through this recipe or not at all. +# +# The image tag *is* a hash of the inputs that define it, so the presence of a +# tag is proof that its contents are current -- a changed tool pin names a tag +# that cannot already exist, and a build follows. There is nothing to keep in +# sync and no staleness to detect. +# +# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md + +# The container engine. `docker` (supported) or `podman` (best-effort). +# A host property, never committed: set the variable in your environment, or +# pass `just anvil_container_engine=podman ...` for a single invocation. +anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") + +# Where the repository is mounted inside the container. +anvil_container_workdir := "/workspace" + +# Image and cache-volume prefix, derived from the repository directory so two +# repositories on one host cannot collide. Sanitized to the character set +# container image references allow. +anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") + +# Resolve the engine. There is deliberately no probe: presence is not +# reachability, `podman-docker` aliases `docker` onto podman, and a silent +# choice between two installed engines means two image stores and an +# unexplained rebuild. We check that the requested binary exists and let every +# other failure surface the engine's own diagnostic, which is more accurate +# than anything repeated here. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-engine: + $ErrorActionPreference = 'Stop' + $engine = '{{anvil_container_engine}}' + if ($engine -ne 'docker' -and $engine -ne 'podman') { + Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" + exit 1 + } + if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { + Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + } + Write-Output $engine + +# Resolve the exec image reference, building it if it is not already present. +# +# The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its +# ignore file, the pinned toolchain, the optional hook, and the generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes. Only this driver is excluded, since hashing it would make the +# tag depend on the tag. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds +# a tag that already resolves, for the cases a content hash cannot see: a moved +# upstream package, or a base layer that changed behind its digest. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { + # The hook decides what the build installs, so its content defines the + # image as surely as the Dockerfile does. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $inputs += $hookRel + } + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } + } + } + + # Hash a tagged stream rather than raw concatenation, so no rearrangement of + # names and contents can collide. Line endings are normalized once, here, so + # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: + # `Sort-Object -Unique` compares case-insensitively, which would silently drop + # one of two inputs differing only in case on the case-sensitive filesystem + # where the image is actually built. + $stream = [System.Text.StringBuilder]::new() + $ordered = [System.Collections.Generic.SortedSet[string]]::new( + [string[]]$inputs, [System.StringComparer]::Ordinal) + foreach ($rel in $ordered) { + $path = Join-Path $repoRoot $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $rel" + exit 1 + } + $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" + [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + } + $digest = [System.Security.Cryptography.SHA256]::HashData( + [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # 16 hex characters (64 bits) is far past any practical collision risk for a + # local image set, and keeps `docker images` readable. + $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) + $image = '{{anvil_container_name}}:' + $imageId + + if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { + & $engine image inspect $image *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output $image + exit 0 + } + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 + } + } + + # Build-time credentials come from the optional hook, never from a committed + # file. Values are handed to BuildKit by environment variable name, so they + # stay out of the host's process command line, and BuildKit keeps them out of + # every image layer. An empty value is fatal: BuildKit would mount an empty + # secret, the build would install a reduced tool set and exit 0, and the + # result would be tagged with the same hash a credentialed build produces -- + # so every later run would reuse the broken image. + $secretArgs = @() + $secretEnv = @() + $hookPath = Join-Path $repoRoot $hookRel + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + $hook = Anvil-PreBuild + if ($null -ne $hook -and $null -ne $hook.Secrets) { + foreach ($id in $hook.Secrets.Keys) { + if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 + } + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" + } + [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + } + } + } + + try { + # Progress goes to stderr: callers capture this recipe's stdout to learn + # the image reference, so anything else written there becomes part of it. + [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # Pinned, not inferred from the host. The Dockerfile installs amd64 + # toolchains and verifies amd64 checksums, so an arm host would resolve + # the multi-arch base to arm64 and fail late with an exec-format error. + # It also keeps the identity scheme honest: without this, two hosts of + # different architecture compute the same tag for different images. + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } + foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } + $buildCmd += $repoRoot + # BuildKit is required for --secret; docker enables it by default from + # 23.0 but an older daemon silently ignores the flag, so ask explicitly. + $env:DOCKER_BUILDKIT = '1' + & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + } finally { + foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + + Write-Output $image -# Run any Anvil recipe in the pinned local Linux container. With no recipe, -# open an interactive shell. -[windows] +# Run any anvil recipe inside the pinned Linux image. +# +# just anvil-container anvil-clippy # one check +# just anvil-container anvil-pr # the whole PR tier +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs +# natively and the work happens exactly once. [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *recipe: - $requested = @('{{ replace(recipe, "'", "''") }}' -split '\s+' | Where-Object { $_ }) - & '.anvil/container/run-in-container.ps1' @requested - exit $LASTEXITCODE +anvil-container *target: + $ErrorActionPreference = 'Stop' + $target = '{{target}}' + if ($env:ANVIL_IN_CONTAINER -eq '1') { + # Already inside: pass straight through instead of nesting. + if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + exit $LASTEXITCODE + } + + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + + # Map the caller's working directory to its in-container equivalent so + # relative paths keep working from a subdirectory. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { + '{{anvil_container_workdir}}' + } else { + '{{anvil_container_workdir}}/' + $rel + } + + $interactive = [string]::IsNullOrWhiteSpace($target) + $runArgs = @('run', '--rm', '--platform', 'linux/amd64') + $runArgs += $interactive ? '-it' : '-i' + $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + # Cargo and rustup homes live in named volumes: the hot write path never + # crosses the host boundary, and the host's own toolchain is untouched. + $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') + $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Match the caller's uid/gid on Linux. Without this everything the run + # writes under the bind mount -- target/, generated files -- lands as root + # on the host, and the next native cargo build or git clean fails with + # EACCES a long way from the cause. Docker Desktop on Windows and macOS + # already maps ownership, and `id` is not there to ask. + if (-not $IsWindows -and -not $IsMacOS) { + $hostUid = (id -u); $hostGid = (id -g) + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + } + $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') + + # Run-time credentials come from the optional hook. Forwarded by NAME, never + # as NAME=VALUE: the engine copies the value out of the environment it + # already inherits, so a credential never appears in the host's process + # command line, where endpoint telemetry records and retains it for far + # longer than a short-lived token is meant to live. + $hookEnv = @() + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + $hook = Anvil-PreRun + if ($null -ne $hook -and $null -ne $hook.Env) { + foreach ($name in $hook.Env.Keys) { + if ([string]::IsNullOrEmpty($hook.Env[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 + } + Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] + $hookEnv += $name + $runArgs += @('-e', $name) + } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + } + } + } + + try { + # --pull=never: the tag names locally-built content, so a miss is a bug + # to surface rather than an invitation to fetch something unrelated. + $runArgs += @('--pull=never', '-w', $containerCwd, $image) + if (-not $interactive) { $runArgs += @('just', $target) } + & $engine @runArgs + exit $LASTEXITCODE + } finally { + foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } -[unix] +# Report the engine, the exec image, and whether it is present and current. +# +# The tag embeds the hash of the image's inputs, so "absent" and "out of date" +# are the same condition and are reported as one. [group("anvil-container")] -[script("bash")] -anvil-container *recipe: - requested={{ quote(recipe) }} - if [[ -z "$requested" ]]; then - exec bash '.anvil/container/run-in-container.sh' - fi - read -r -a requested_args <<<"$requested" - exec bash '.anvil/container/run-in-container.sh' "${requested_args[@]}" +[script("pwsh", "-NoProfile")] +anvil-container-status: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "engine: $engine" + Write-Output "workdir: {{anvil_container_workdir}}" + + # NO_REBUILD turns the resolve into a pure query: report the state instead + # of silently spending several minutes building from a status command. + $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 + $present = $LASTEXITCODE -eq 0 + if ($image) { Write-Output "image: $image" } + if ($present) { + Write-Output "status: present and current" + } else { + Write-Output "status: no image matches the current inputs (it will be built on the next run)" + } + exit 0 + +# Rebuild the exec image from scratch, ignoring every cached layer. +# +# The ordinary path already rebuilds whenever an input changes, so this is for +# the cases a content hash cannot see: a moved upstream package, a stale base +# layer, or a build that is suspected of being wrong. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-rebuild: + $ErrorActionPreference = 'Stop' + $env:ANVIL_CONTAINER_NO_CACHE = '1' + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image rebuilt: $image" + exit 0 + +# Remove this repository's cache volumes. The image is left in place. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-down: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + & $engine volume rm -f $vol + } + exit 0 === justfiles/anvil/groups/pr-fast.just === # Copyright (c) Microsoft Corporation. @@ -3998,7 +3231,6 @@ import 'groups/scheduled-test.just' import 'groups/scheduled-advisories.just' import 'groups/scheduled-runtime-analysis.just' import 'groups/scheduled-exhaustive.just' -import 'runner.just' import 'tiers.just' import 'tools.just' import 'versions.just' @@ -4006,55 +3238,6 @@ import 'versions.just' # Friendly default: `just anvil` runs the PR tier. alias anvil := anvil-pr -=== justfiles/anvil/runner.just === -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -# Route public tier entry points through the configured execution environment. -# ANVIL_IN_CONTAINER always wins to prevent recursive container launches. -[private] -[no-exit-message] -[windows] -[script("pwsh", "-NoProfile")] -_anvil-run tier runner: - $just = '{{ replace(just_executable(), "'", "''") }}' - $justfile = '{{ replace(justfile(), "'", "''") }}' - $nativeTier = '_anvil-{{ replace(tier, "'", "''") }}' - if ($env:ANVIL_IN_CONTAINER) { - & $just --justfile $justfile $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'container') { - & $just --justfile $justfile anvil-container $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'native') { - & $just --justfile $justfile $nativeTier - } else { - [Console]::Error.WriteLine("anvil-runner: expected 'native' or 'container', got '{{ replace(runner, "'", "''") }}'.") - exit 2 - } - exit $LASTEXITCODE - -[private] -[no-exit-message] -[unix] -[script("bash")] -_anvil-run tier runner: - just_path={{ quote(just_executable()) }} - justfile={{ quote(justfile()) }} - tier={{ quote(tier) }} - runner={{ quote(runner) }} - native_tier="_anvil-$tier" - if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - elif [[ "$runner" == "container" ]]; then - exec "$just_path" --justfile "$justfile" anvil-container "$native_tier" - elif [[ "$runner" == "native" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - else - echo "anvil-runner: expected 'native' or 'container', got '$runner'." >&2 - exit 2 - fi - === justfiles/anvil/tiers.just === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. @@ -4070,10 +3253,7 @@ _anvil-run tier runner: # Run all pull request checks. [group("anvil")] -anvil-pr: (_anvil-run "pr" anvil_runner) - -[private] -_anvil-pr: anvil-pr-validate-prereqs \ +anvil-pr: anvil-pr-validate-prereqs \ anvil-pr-fast \ anvil-pr-slow @@ -4084,10 +3264,7 @@ _anvil-pr: anvil-pr-validate-prereqs \ # Run all scheduled checks. [group("anvil")] -anvil-scheduled: (_anvil-run "scheduled" anvil_runner) - -[private] -_anvil-scheduled: anvil-scheduled-validate-prereqs \ +anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-test \ anvil-scheduled-advisories \ anvil-scheduled-runtime-analysis \ @@ -4095,12 +3272,9 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. [group("anvil")] -anvil-full: (_anvil-run "full" anvil_runner) - -[private] -_anvil-full: anvil-full-validate-prereqs \ - _anvil-pr \ - _anvil-scheduled +anvil-full: anvil-full-validate-prereqs \ + anvil-pr \ + anvil-scheduled # Tier-level + global setup + validate-prereqs # =========================================================================== diff --git a/crates/cargo-anvil/tests/tier_routing.rs b/crates/cargo-anvil/tests/tier_routing.rs deleted file mode 100644 index c661bd73..00000000 --- a/crates/cargo-anvil/tests/tier_routing.rs +++ /dev/null @@ -1,302 +0,0 @@ -// Copyright (c) Microsoft Corporation. -// Licensed under the MIT License. - -#![cfg(not(miri))] -#![allow( - clippy::expect_used, - clippy::unwrap_used, - reason = "panic-on-failure idioms are appropriate in tests" -)] - -use std::ffi::OsString; -use std::path::{Path, PathBuf}; -use std::process::{Command, Output}; - -use tempfile::TempDir; - -const RUNNER: &str = include_str!("../templates/justfiles/anvil/runner.just"); -const JUSTFILE: &str = "routing.just"; - -fn write(path: &Path, contents: &str) { - std::fs::write(path, contents).unwrap(); -} - -fn fixture() -> TempDir { - let tmp = TempDir::new().unwrap(); - write(&tmp.path().join("runner.just"), RUNNER); - let justfile = r#" -import 'runner.just' - -runner := env_var_or_default("ANVIL_RUNNER", "native") - -default: (_anvil-run "pr" runner) -failure: (_anvil-run "fail" runner) - -[private] -_anvil-pr: first second - -[private] -_anvil-fail: first failing - -[windows] -[script("pwsh", "-NoProfile")] -first: - Write-Output first - -[windows] -[script("pwsh", "-NoProfile")] -second: - Write-Output second - -[windows] -[script("pwsh", "-NoProfile")] -failing: - Write-Output failing - exit 7 - -[windows] -[script("pwsh", "-NoProfile")] -anvil-container *recipe: - Write-Output 'container:{{ recipe }}' - -[script("pwsh", "-NoProfile")] -profile-independent: - Write-Output profile-safe - -[script("pwsh")] -profile-dependent: - Write-Output profile-noisy - -[unix] -first: - @printf 'first\n' - -[unix] -second: - @printf 'second\n' - -[unix] -failing: - @printf 'failing\n' - @exit 7 - -[unix] -anvil-container *recipe: - @printf 'container:%s\n' '{{ recipe }}' -"#; - write(&tmp.path().join(JUSTFILE), justfile); - tmp -} - -fn profile_fixture() -> TempDir { - let tmp = fixture(); - install_profile_noise_wrapper(tmp.path()); - tmp -} - -fn just_available() -> bool { - Command::new("just").arg("--version").output().is_ok() -} - -fn pwsh_available() -> bool { - Command::new("pwsh").arg("--version").output().is_ok() -} - -fn pwsh_path() -> PathBuf { - let output = Command::new("pwsh") - .args(["-NoProfile", "-Command", "(Get-Command pwsh).Source"]) - .output() - .expect("pwsh availability is checked before creating the fixture"); - assert!(output.status.success(), "failed to resolve pwsh path"); - PathBuf::from(String::from_utf8(output.stdout).unwrap().trim()) -} - -fn install_profile_noise_wrapper(root: &Path) { - let bin = root.join("fake-bin"); - std::fs::create_dir_all(&bin).unwrap(); - let real_pwsh = pwsh_path(); - - #[cfg(windows)] - { - let source = bin.join("pwsh.rs"); - let real_pwsh = format!("{:?}", real_pwsh.to_string_lossy()); - write( - &source, - &format!( - r#"use std::io::Write as _; -use std::process::{{Command, exit}}; - -fn main() {{ - let args: Vec<_> = std::env::args_os().skip(1).collect(); - if !args.iter().any(|arg| arg.to_string_lossy().eq_ignore_ascii_case("-NoProfile")) {{ - println!("PROFILE_OUTPUT"); - std::io::stdout().flush().expect("stdout must flush"); - }} - let status = Command::new({real_pwsh}) - .args(&args) - .status() - .expect("real pwsh must start"); - exit(status.code().unwrap_or(1)); -}} -"# - ), - ); - let status = Command::new("rustc") - .arg(&source) - .arg("-o") - .arg(bin.join("pwsh.exe")) - .status() - .expect("rustc is available while running cargo tests"); - assert!(status.success(), "failed to compile the Windows pwsh test shim"); - } - - #[cfg(unix)] - { - use std::os::unix::fs::PermissionsExt as _; - - let escaped = real_pwsh.to_string_lossy().replace('\'', "'\\''"); - let wrapper = format!( - "#!/usr/bin/env sh\ncase \" $* \" in *\" -NoProfile \"*) ;; *) printf 'PROFILE_OUTPUT\\n' ;; esac\nexec '{escaped}' \"$@\"\n" - ); - let path = bin.join("pwsh"); - write(&path, &wrapper); - let mut permissions = std::fs::metadata(&path).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(path, permissions).unwrap(); - } -} - -fn path_with_profile_wrapper(root: &Path) -> OsString { - let mut paths = vec![root.join("fake-bin")]; - paths.extend(std::env::split_paths(&std::env::var_os("PATH").unwrap_or_default())); - std::env::join_paths(paths).unwrap() -} - -fn run(root: &Path, recipes: &[&str], environment: &[(&str, &str)]) -> Output { - let mut command = Command::new("just"); - command.args(["--justfile", root.join(JUSTFILE).to_str().unwrap()]); - command.args(recipes).current_dir(root); - command.env_remove("ANVIL_RUNNER").env_remove("ANVIL_IN_CONTAINER"); - command.env("PATH", path_with_profile_wrapper(root)); - command.envs(environment.iter().copied()); - command.output().expect("just is required to verify generated tier routing") -} - -#[test] -fn native_routing_preserves_output_and_exit_status() { - if !just_available() { - return; - } - let tmp = fixture(); - let direct = run(tmp.path(), &["_anvil-pr"], &[]); - let routed = run(tmp.path(), &["default"], &[]); - - assert_eq!(routed.status.code(), direct.status.code()); - assert_eq!(routed.stdout, direct.stdout); - assert_eq!(routed.stderr, direct.stderr); -} - -#[test] -fn native_routing_preserves_failure_output_and_exit_status() { - if !just_available() { - return; - } - let tmp = fixture(); - let direct = run(tmp.path(), &["_anvil-fail"], &[]); - let routed = run(tmp.path(), &["failure"], &[]); - - assert_eq!(direct.status.code(), Some(7)); - assert_eq!(routed.status.code(), direct.status.code()); - assert_eq!(routed.stdout, direct.stdout); - assert_eq!(routed.stderr, direct.stderr); -} - -#[test] -fn configured_container_routing_uses_the_container_recipe() { - if !just_available() { - return; - } - let tmp = fixture(); - let output = run(tmp.path(), &["default"], &[("ANVIL_RUNNER", "container")]); - - assert!( - output.status.success(), - "container route failed: {}", - String::from_utf8_lossy(&output.stderr) - ); - assert_eq!(String::from_utf8_lossy(&output.stdout).trim(), "container:_anvil-pr"); -} - -#[test] -fn in_container_forces_native_execution() { - if !just_available() { - return; - } - let tmp = fixture(); - let output = run( - tmp.path(), - &["default"], - &[("ANVIL_RUNNER", "container"), ("ANVIL_IN_CONTAINER", "1")], - ); - - assert!( - output.status.success(), - "native recursion guard failed: {}", - String::from_utf8_lossy(&output.stderr) - ); - assert_eq!( - String::from_utf8_lossy(&output.stdout).lines().collect::>(), - ["first", "second"] - ); -} - -#[test] -fn invalid_runner_value_fails_instead_of_falling_back_to_native() { - if !just_available() { - return; - } - let tmp = fixture(); - let output = run(tmp.path(), &["default"], &[("ANVIL_RUNNER", "Container")]); - - assert!( - !output.status.success(), - "invalid runner unexpectedly succeeded: stdout={}; stderr={}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - assert!( - String::from_utf8_lossy(&output.stderr).contains("expected 'native' or 'container'"), - "invalid runner error must be actionable: {}", - String::from_utf8_lossy(&output.stderr) - ); -} - -#[test] -fn powershell_recipe_output_is_independent_of_profiles() { - if !just_available() || !pwsh_available() { - return; - } - let tmp = profile_fixture(); - let noisy = run(tmp.path(), &["profile-dependent"], &[]); - assert!( - noisy.status.success(), - "profile-dependent negative control failed: {}", - String::from_utf8_lossy(&noisy.stderr) - ); - assert_eq!( - String::from_utf8_lossy(&noisy.stdout).lines().collect::>(), - ["PROFILE_OUTPUT", "profile-noisy"], - "the negative control must prove the fake pwsh shim was invoked" - ); - - let output = run(tmp.path(), &["profile-independent"], &[]); - assert!( - output.status.success(), - "profile-independent recipe failed: {}", - String::from_utf8_lossy(&output.stderr) - ); - assert_eq!( - String::from_utf8_lossy(&output.stdout).lines().collect::>(), - ["profile-safe"] - ); -} diff --git a/crates/cargo-coverage-gate/README.md b/crates/cargo-coverage-gate/README.md index 0f3de902..293cba5e 100644 --- a/crates/cargo-coverage-gate/README.md +++ b/crates/cargo-coverage-gate/README.md @@ -105,7 +105,7 @@ plus the appropriate exit code. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGkYW0CYXSEGxYc2fK81jTWG7kWg0hlspxYGx-DzHaE-xjXG1cDT7T4wIbxYXKEGw80cH9KnXVkG0Ik84btPmxNG1_q7rZL3w7mGyE4F77TF4kVYWSBg3NjYXJnby1jb3ZlcmFnZS1nYXRlZTAuMy4wc2NhcmdvX2NvdmVyYWdlX2dhdGU + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbDzRwf0qddWQbQiTzhu0-bE0bX-rutkvfDuYbITgXvtMXiRVhZIGDc2NhcmdvLWNvdmVyYWdlLWdhdGVlMC4zLjBzY2FyZ29fY292ZXJhZ2VfZ2F0ZQ [__link0]: https://github.com/taiki-e/cargo-llvm-cov [__link1]: https://docs.rs/cargo-coverage-gate/0.3.0/cargo_coverage_gate/fn.evaluate.html [__link2]: https://docs.rs/cargo-coverage-gate/0.3.0/cargo_coverage_gate/struct.EvaluatedReport.html diff --git a/justfiles/anvil/checks/aprz.just b/justfiles/anvil/checks/aprz.just index 1627c3a5..bb084078 100644 --- a/justfiles/anvil/checks/aprz.just +++ b/justfiles/anvil/checks/aprz.just @@ -9,11 +9,11 @@ # cargo-aprz queries the GitHub advisory API. Unauthenticated access is # capped at 60 requests/hour and fails on a full run; an authenticated # token raises the cap to 5000/hour. CI injects GITHUB_TOKEN -# (github.token). Container drivers mount an existing host GITHUB_TOKEN -# or the host gh CLI's stored token as a temporary read-only secret. -# Native runs borrow the gh CLI token directly. Native runs warn and -# proceed unauthenticated if neither is available; container runs fail -# before cargo-aprz can exhaust the unauthenticated rate limit. +# (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the +# gh CLI's stored token (non-interactive: `gh auth token` prints the +# active account's token for github.com and never opens a browser/auth +# prompt). If neither is available we warn with instructions and proceed +# unauthenticated. # # Unscoped (consults external risk DB). @@ -21,24 +21,14 @@ [script("pwsh", "-NoProfile")] anvil-aprz: anvil-aprz-validate-prereqs $ErrorActionPreference = 'Stop' - if ($env:ANVIL_APRZ_ALREADY_RAN -eq '1') { - Write-Host 'anvil-aprz: already completed in an isolated authenticated container' - exit 0 - } if (-not $env:GITHUB_TOKEN) { $tok = $null - $containerTokenFile = '/run/secrets/anvil-github-token' - if ($env:ANVIL_IN_CONTAINER -and (Test-Path -LiteralPath $containerTokenFile -PathType Leaf)) { - try { $tok = Get-Content -LiteralPath $containerTokenFile -Raw } catch { $tok = $null } - } elseif (Get-Command gh -ErrorAction SilentlyContinue) { + if (Get-Command gh -ErrorAction SilentlyContinue) { try { $tok = (gh auth token --hostname github.com 2>$null) } catch { $tok = $null } } if ($tok) { $env:GITHUB_TOKEN = $tok.Trim() } else { - if ($env:ANVIL_IN_CONTAINER) { - throw 'anvil-aprz: GitHub authentication is unavailable. Run `gh auth login` on the host or set host GITHUB_TOKEN, then re-run the container command.' - } Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 86508b25..5a1cdab7 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -2,24 +2,332 @@ # Licensed under the MIT License. # GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. +# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md +# +# Containerized execution. `just anvil-container ` runs any anvil +# recipe inside a pinned Linux image; everything else keeps running natively. +# There is no configuration file and no transparent routing: the container is +# reached through this recipe or not at all. +# +# The image tag *is* a hash of the inputs that define it, so the presence of a +# tag is proof that its contents are current -- a changed tool pin names a tag +# that cannot already exist, and a build follows. There is nothing to keep in +# sync and no staleness to detect. +# +# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# Run any Anvil recipe in the pinned local Linux container. With no recipe, -# open an interactive shell. -[windows] +# The container engine. `docker` (supported) or `podman` (best-effort). +# A host property, never committed: set the variable in your environment, or +# pass `just anvil_container_engine=podman ...` for a single invocation. +anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") + +# Where the repository is mounted inside the container. +anvil_container_workdir := "/workspace" + +# Image and cache-volume prefix, derived from the repository directory so two +# repositories on one host cannot collide. Sanitized to the character set +# container image references allow. +anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") + +# Resolve the engine. There is deliberately no probe: presence is not +# reachability, `podman-docker` aliases `docker` onto podman, and a silent +# choice between two installed engines means two image stores and an +# unexplained rebuild. We check that the requested binary exists and let every +# other failure surface the engine's own diagnostic, which is more accurate +# than anything repeated here. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-engine: + $ErrorActionPreference = 'Stop' + $engine = '{{anvil_container_engine}}' + if ($engine -ne 'docker' -and $engine -ne 'podman') { + Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" + exit 1 + } + if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { + Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + } + Write-Output $engine + +# Resolve the exec image reference, building it if it is not already present. +# +# The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its +# ignore file, the pinned toolchain, the optional hook, and the generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes. Only this driver is excluded, since hashing it would make the +# tag depend on the tag. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds +# a tag that already resolves, for the cases a content hash cannot see: a moved +# upstream package, or a base layer that changed behind its digest. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { + # The hook decides what the build installs, so its content defines the + # image as surely as the Dockerfile does. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $inputs += $hookRel + } + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } + } + } + + # Hash a tagged stream rather than raw concatenation, so no rearrangement of + # names and contents can collide. Line endings are normalized once, here, so + # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: + # `Sort-Object -Unique` compares case-insensitively, which would silently drop + # one of two inputs differing only in case on the case-sensitive filesystem + # where the image is actually built. + $stream = [System.Text.StringBuilder]::new() + $ordered = [System.Collections.Generic.SortedSet[string]]::new( + [string[]]$inputs, [System.StringComparer]::Ordinal) + foreach ($rel in $ordered) { + $path = Join-Path $repoRoot $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $rel" + exit 1 + } + $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" + [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + } + $digest = [System.Security.Cryptography.SHA256]::HashData( + [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # 16 hex characters (64 bits) is far past any practical collision risk for a + # local image set, and keeps `docker images` readable. + $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) + $image = '{{anvil_container_name}}:' + $imageId + + if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { + & $engine image inspect $image *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output $image + exit 0 + } + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 + } + } + + # Build-time credentials come from the optional hook, never from a committed + # file. Values are handed to BuildKit by environment variable name, so they + # stay out of the host's process command line, and BuildKit keeps them out of + # every image layer. An empty value is fatal: BuildKit would mount an empty + # secret, the build would install a reduced tool set and exit 0, and the + # result would be tagged with the same hash a credentialed build produces -- + # so every later run would reuse the broken image. + $secretArgs = @() + $secretEnv = @() + $hookPath = Join-Path $repoRoot $hookRel + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + $hook = Anvil-PreBuild + if ($null -ne $hook -and $null -ne $hook.Secrets) { + foreach ($id in $hook.Secrets.Keys) { + if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 + } + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" + } + [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + } + } + } + + try { + # Progress goes to stderr: callers capture this recipe's stdout to learn + # the image reference, so anything else written there becomes part of it. + [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # Pinned, not inferred from the host. The Dockerfile installs amd64 + # toolchains and verifies amd64 checksums, so an arm host would resolve + # the multi-arch base to arm64 and fail late with an exec-format error. + # It also keeps the identity scheme honest: without this, two hosts of + # different architecture compute the same tag for different images. + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } + foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } + $buildCmd += $repoRoot + # BuildKit is required for --secret; docker enables it by default from + # 23.0 but an older daemon silently ignores the flag, so ask explicitly. + $env:DOCKER_BUILDKIT = '1' + & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + } finally { + foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + + Write-Output $image + +# Run any anvil recipe inside the pinned Linux image. +# +# just anvil-container anvil-clippy # one check +# just anvil-container anvil-pr # the whole PR tier +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs +# natively and the work happens exactly once. [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *recipe: - $requested = @('{{ replace(recipe, "'", "''") }}' -split '\s+' | Where-Object { $_ }) - & '.anvil/container/run-in-container.ps1' @requested - exit $LASTEXITCODE +anvil-container *target: + $ErrorActionPreference = 'Stop' + $target = '{{target}}' + if ($env:ANVIL_IN_CONTAINER -eq '1') { + # Already inside: pass straight through instead of nesting. + if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + exit $LASTEXITCODE + } -[unix] + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $repoRoot = '{{justfile_directory()}}' + + # Map the caller's working directory to its in-container equivalent so + # relative paths keep working from a subdirectory. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { + '{{anvil_container_workdir}}' + } else { + '{{anvil_container_workdir}}/' + $rel + } + + $interactive = [string]::IsNullOrWhiteSpace($target) + $runArgs = @('run', '--rm', '--platform', 'linux/amd64') + $runArgs += $interactive ? '-it' : '-i' + $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + # Cargo and rustup homes live in named volumes: the hot write path never + # crosses the host boundary, and the host's own toolchain is untouched. + $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') + $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Match the caller's uid/gid on Linux. Without this everything the run + # writes under the bind mount -- target/, generated files -- lands as root + # on the host, and the next native cargo build or git clean fails with + # EACCES a long way from the cause. Docker Desktop on Windows and macOS + # already maps ownership, and `id` is not there to ask. + if (-not $IsWindows -and -not $IsMacOS) { + $hostUid = (id -u); $hostGid = (id -g) + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + } + $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') + + # Run-time credentials come from the optional hook. Forwarded by NAME, never + # as NAME=VALUE: the engine copies the value out of the environment it + # already inherits, so a credential never appears in the host's process + # command line, where endpoint telemetry records and retains it for far + # longer than a short-lived token is meant to live. + $hookEnv = @() + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' + if (Test-Path -LiteralPath $hookPath -PathType Leaf) { + . $hookPath + if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + $hook = Anvil-PreRun + if ($null -ne $hook -and $null -ne $hook.Env) { + foreach ($name in $hook.Env.Keys) { + if ([string]::IsNullOrEmpty($hook.Env[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 + } + Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] + $hookEnv += $name + $runArgs += @('-e', $name) + } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + } + } + } + + try { + # --pull=never: the tag names locally-built content, so a miss is a bug + # to surface rather than an invitation to fetch something unrelated. + $runArgs += @('--pull=never', '-w', $containerCwd, $image) + if (-not $interactive) { $runArgs += @('just', $target) } + & $engine @runArgs + exit $LASTEXITCODE + } finally { + foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + } + +# Report the engine, the exec image, and whether it is present and current. +# +# The tag embeds the hash of the image's inputs, so "absent" and "out of date" +# are the same condition and are reported as one. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-status: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "engine: $engine" + Write-Output "workdir: {{anvil_container_workdir}}" + + # NO_REBUILD turns the resolve into a pure query: report the state instead + # of silently spending several minutes building from a status command. + $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 + $present = $LASTEXITCODE -eq 0 + if ($image) { Write-Output "image: $image" } + if ($present) { + Write-Output "status: present and current" + } else { + Write-Output "status: no image matches the current inputs (it will be built on the next run)" + } + exit 0 + +# Rebuild the exec image from scratch, ignoring every cached layer. +# +# The ordinary path already rebuilds whenever an input changes, so this is for +# the cases a content hash cannot see: a moved upstream package, a stale base +# layer, or a build that is suspected of being wrong. [group("anvil-container")] -[script("bash")] -anvil-container *recipe: - requested={{ quote(recipe) }} - if [[ -z "$requested" ]]; then - exec bash '.anvil/container/run-in-container.sh' - fi - read -r -a requested_args <<<"$requested" - exec bash '.anvil/container/run-in-container.sh' "${requested_args[@]}" +[script("pwsh", "-NoProfile")] +anvil-container-rebuild: + $ErrorActionPreference = 'Stop' + $env:ANVIL_CONTAINER_NO_CACHE = '1' + $image = (just _anvil-container-image) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image rebuilt: $image" + exit 0 + +# Remove this repository's cache volumes. The image is left in place. +[group("anvil-container")] +[script("pwsh", "-NoProfile")] +anvil-container-down: + $ErrorActionPreference = 'Stop' + $engine = just _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + & $engine volume rm -f $vol + } + exit 0 diff --git a/justfiles/anvil/mod.just b/justfiles/anvil/mod.just index 710bead6..14f99095 100644 --- a/justfiles/anvil/mod.just +++ b/justfiles/anvil/mod.just @@ -76,7 +76,6 @@ import 'groups/scheduled-test.just' import 'groups/scheduled-advisories.just' import 'groups/scheduled-runtime-analysis.just' import 'groups/scheduled-exhaustive.just' -import 'runner.just' import 'tiers.just' import 'tools.just' import 'versions.just' diff --git a/justfiles/anvil/runner.just b/justfiles/anvil/runner.just deleted file mode 100644 index 8cdd5921..00000000 --- a/justfiles/anvil/runner.just +++ /dev/null @@ -1,47 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. - -# Route public tier entry points through the configured execution environment. -# ANVIL_IN_CONTAINER always wins to prevent recursive container launches. -[private] -[no-exit-message] -[windows] -[script("pwsh", "-NoProfile")] -_anvil-run tier runner: - $just = '{{ replace(just_executable(), "'", "''") }}' - $justfile = '{{ replace(justfile(), "'", "''") }}' - $nativeTier = '_anvil-{{ replace(tier, "'", "''") }}' - if ($env:ANVIL_IN_CONTAINER) { - & $just --justfile $justfile $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'container') { - & $just --justfile $justfile anvil-container $nativeTier - } elseif ('{{ replace(runner, "'", "''") }}' -ceq 'native') { - & $just --justfile $justfile $nativeTier - } else { - [Console]::Error.WriteLine("anvil-runner: expected 'native' or 'container', got '{{ replace(runner, "'", "''") }}'.") - exit 2 - } - exit $LASTEXITCODE - -[private] -[no-exit-message] -[unix] -[script("bash")] -_anvil-run tier runner: - just_path={{ quote(just_executable()) }} - justfile={{ quote(justfile()) }} - tier={{ quote(tier) }} - runner={{ quote(runner) }} - native_tier="_anvil-$tier" - if [[ -n "${ANVIL_IN_CONTAINER:-}" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - elif [[ "$runner" == "container" ]]; then - exec "$just_path" --justfile "$justfile" anvil-container "$native_tier" - elif [[ "$runner" == "native" ]]; then - exec "$just_path" --justfile "$justfile" "$native_tier" - else - echo "anvil-runner: expected 'native' or 'container', got '$runner'." >&2 - exit 2 - fi diff --git a/justfiles/anvil/tiers.just b/justfiles/anvil/tiers.just index 86885967..dd9c0b2c 100644 --- a/justfiles/anvil/tiers.just +++ b/justfiles/anvil/tiers.just @@ -12,10 +12,7 @@ # Run all pull request checks. [group("anvil")] -anvil-pr: (_anvil-run "pr" anvil_runner) - -[private] -_anvil-pr: anvil-pr-validate-prereqs \ +anvil-pr: anvil-pr-validate-prereqs \ anvil-pr-fast \ anvil-pr-slow @@ -26,10 +23,7 @@ _anvil-pr: anvil-pr-validate-prereqs \ # Run all scheduled checks. [group("anvil")] -anvil-scheduled: (_anvil-run "scheduled" anvil_runner) - -[private] -_anvil-scheduled: anvil-scheduled-validate-prereqs \ +anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-test \ anvil-scheduled-advisories \ anvil-scheduled-runtime-analysis \ @@ -37,12 +31,9 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. [group("anvil")] -anvil-full: (_anvil-run "full" anvil_runner) - -[private] -_anvil-full: anvil-full-validate-prereqs \ - _anvil-pr \ - _anvil-scheduled +anvil-full: anvil-full-validate-prereqs \ + anvil-pr \ + anvil-scheduled # Tier-level + global setup + validate-prereqs # =========================================================================== From 89055b51c25afc334d09f94e6ddc58e3ae589e87 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Wed, 12 Aug 2026 18:26:10 +0200 Subject: [PATCH 02/81] feat(anvil): reach an engine installed in WSL, and add the container e2e Docker installed inside WSL without Docker Desktop is a documented, common Windows setup -- it is the one this repository's own docs describe -- and it leaves no Windows CLI behind. Invoking the engine directly therefore made the feature unreachable on exactly the configuration we tell people to build. On Windows only, and only when the engine is absent from PATH, the recipe now routes through the default WSL distribution and translates the repository root with wslpath. Paths cross that boundary with forward slashes, since the intervening shell would otherwise eat the separators and hand wslpath a mangled path. Build secrets and forwarded run-time values are exported through WSLENV, which is where the engine reads them from when it runs there. Docker Desktop and Podman ship a Windows CLI, are found on PATH, and never take this path. scripts/test-anvil-container.ps1 is a black-box walk through the feature from a user's seat: it creates a repository, generates into it, and then only does what a developer would do. Its setup is held to that standard deliberately -- if it had to hand-write a generated file or work around a defect, that would be a bug in the product rather than something the script should absorb. 43 checks, against a real Docker daemon: the emitted artifacts, a first run that builds, a second that reuses, a toolchain bump that renames the tag and a revert that restores it, a Dockerfile edit that survives regeneration, a hook whose secret reaches the build without reaching a layer and whose value reaches a recipe at run time, an empty secret that fails closed, and a nested invocation that stays native. The design doc is rewritten against the narrowed contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 4 +- crates/cargo-anvil/docs/design/README.md | 14 +- crates/cargo-anvil/docs/design/containers.md | 550 +++++------------- .../cargo-anvil/docs/design/extensibility.md | 85 ++- crates/cargo-anvil/docs/design/local.md | 30 +- .../templates/justfiles/anvil/container.just | 108 +++- .../snapshots/snapshots__ado_backend.snap | 108 +++- .../snapshots/snapshots__github_backend.snap | 108 +++- .../snapshots/snapshots__local_only.snap | 108 +++- justfiles/anvil/container.just | 108 +++- scripts/test-anvil-container.ps1 | 547 +++++++++++++++++ 11 files changed, 1215 insertions(+), 555 deletions(-) create mode 100644 scripts/test-anvil-container.ps1 diff --git a/.anvil.lock b/.anvil.lock index ca9016b1..4cdef881 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:e98fd525a9b48a5d2bc57066426abf8bdc78bcde87a38ab9d17e80210a43ef81" +catalog_checksum = "sha256:6d1e7ee2e36339ba85145ec327802ef5126c3e2fd016339c5d528e9a075c7170" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:f7cbcc86dd14210d58e198a4b4af1e2590c6816b5497583ceb1b361b73a7419b" +checksum = "sha256:16a03104b3de2612ee5553969ad6544de5cfd94beffb25d3d551547e94ff78ce" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/README.md b/crates/cargo-anvil/docs/design/README.md index 9a06cd9d..8b5034bd 100644 --- a/crates/cargo-anvil/docs/design/README.md +++ b/crates/cargo-anvil/docs/design/README.md @@ -13,8 +13,8 @@ user-visible shape of the tool. Detail lives in companion documents: - [extensibility.md](./extensibility.md) — how downstream tools ship their own brand + catalog. - [github.md](./github.md) — GitHub Actions emission, example workflows, impact wiring. - [ado.md](./ado.md) — Azure DevOps Pipelines emission, 1ESPT/msrustup composition. -- [containers.md](./containers.md) — the opt-in, local-only container backend for running any - `anvil-*` recipe in a pinned Linux image (Linux-on-Windows parity, distro pinning). +- [containers.md](./containers.md) — containerized execution: the explicit `anvil-container` + recipe, the content-addressed image, and the credential hook. - [../implementation.md](../implementation.md) — internal implementation guidance. - [../verification.md](../verification.md) — continuous-validation strategy: dogfooding, fixture tests, schema validation. @@ -237,7 +237,7 @@ repo/ ├── .anvil.lock sidecar manifest tracking last-rendered checksums (see updates.md) ├── Justfile managed-region: anvil-imports ├── justfiles/anvil/ owned (see local.md) -├── .anvil/container/ owned, only with the container backend (see local.md, containers.md) +├── .anvil/container/ owned — the container image definition (see containers.md) ├── Cargo.toml managed-region: anvil-workspace-lints (or anvil-lints in single-crate) ├── crates//Cargo.toml managed-region: anvil-lints (one per workspace member) ├── deny.toml managed-regions: anvil-deny-{advisories,licenses,bans,sources} @@ -265,10 +265,10 @@ repo/ Detail on each host: - **`Justfile` and `justfiles/anvil/*.just`** — see [local.md](./local.md). -- **`.anvil/container/`** — the non-recipe container assets (Containerfile, - drivers, image-ID helpers, README) emitted only when the catalog includes the - optional container backend. `justfiles/` holds `.just` recipes and nothing - else, so these live in a tool-owned directory of their own; see +- **`.anvil/container/`** — the container image definition: a `Dockerfile` and + its build-context ignore file, plus an optional `hooks.ps1` supplying + credentials. `justfiles/` holds `.just` recipes and nothing else, so these + live in a tool-owned directory of their own; see [containers.md](./containers.md). - **`Cargo.toml` lints regions** — workspace `Cargo.toml` carries the `anvil-workspace-lints` region containing a single `[workspace.lints]` table whose diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 1499d446..a88ba62b 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -1,437 +1,189 @@ -# cargo-anvil container execution +# Containers -This document describes `cargo-anvil`'s optional support for running generated -Anvil recipes in a reproducible local Linux container. Native execution remains -the default. +Any generated recipe can run inside a pinned Linux image: -The intended audience is `cargo-anvil` maintainers and downstream catalog -authors. User setup and troubleshooting are documented in the generated -`.anvil/container/README.md`. +```bash +just anvil-container anvil-clippy # one check +just anvil-container anvil-pr # the whole PR tier +just anvil-container # interactive shell +``` + +See also [design.md](./README.md) for the overall principles, [local.md](./local.md) for the recipe surface this +wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstream fork uses. + +- [1. Problem](#1-problem) +- [2. What this is](#2-what-this-is) +- [3. The two artifacts](#3-the-two-artifacts) +- [4. Image identity](#4-image-identity) +- [5. Credentials — the hook](#5-credentials--the-hook) +- [6. Host setup](#6-host-setup) +- [7. Customizing the image](#7-customizing-the-image) +- [8. Limits](#8-limits) ## 1. Problem -Anvil recipes normally use the developer's host toolchain. That is the fastest -inner loop, but it cannot always reproduce: - -- Linux-specific behavior from a Windows host; -- failures caused by differences between the host distribution and a pinned - build environment; -- Linux binaries that require a newer glibc than their deployment environment; -- the exact Rust toolchain and Cargo tools selected by the generated catalog; -- fast repeated container runs without reinstalling tools or rebuilding - unchanged dependencies. - -Container support provides an explicit way to run the same recipes in a -pinned Linux environment. It is a local development feature, not a replacement -for native execution or the generated GitHub Actions and Azure DevOps -workflows. After the initial build, it reuses the matching image, dependency -caches, and compilation output. - -## 2. Design principles - -- **Generated files remain the product.** `cargo-anvil` emits the container - recipe, image definition, and host drivers. The generator is not involved - when a recipe runs. -- **Recipes are unchanged.** The container invokes the existing generated - `anvil-*` recipes rather than maintaining container-specific copies. -- **Container use is explicit or deliberately selected.** There is no `PATH` - shim, replacement `just` binary, or implicit command rewriting. -- **Runtime policy is not generator state.** Selecting the container runner - does not change `.anvil.lock` or the update algorithm. -- **The generated catalog is the image's source of truth.** The image installs - tools through `just anvil-setup`, using the same generated pins and setup - recipes that checks validate. -- **Environment-specific behavior is replaceable.** Downstream catalogs can - replace the image definition and add authentication hooks without forking the - public drivers or execution model. - -## 3. User experience - -Run any generated Anvil recipe in the container: +The local layer assumes a usable host toolchain ([design.md §3][design]: "the user owns it locally"). Two situations +break that assumption: -```text -just anvil-container anvil-clippy -just anvil-container anvil-pr -``` +1. **Linux-on-Windows parity.** A developer on Windows cannot reproduce a Linux-only failure — a `cfg(unix)` path, a + Linux-specific lint, an `mmap`-shaped test — without a Linux box. +2. **Toolchain drift.** Even on Linux, the host toolset can differ from the one the checks expect, so a green local + run is not predictive. + +Both are solved the same way: run the recipe in an image whose toolchain and tools are the ones this repository pins. + +## 2. What this is -Every positional argument is a recipe name and must match `anvil-*` or -`_anvil-*`. Recipe parameters are not part of this command surface. +- **Explicit.** `just anvil-pr` runs natively, exactly as before. The container is reached through + `just anvil-container` or not at all. There is no PATH shim, no routing toggle, and no recipe that behaves + differently depending on where it runs. +- **Unconfigured.** There is no `anvil.toml`. Whether the artifacts are emitted is a catalog decision; the only + host-specific value is an environment variable read at run time. +- **Local.** Cloud workflows continue to run the recipes natively on their own pools. The image is pinned to be + *like* CI, not to *be* CI. +- **Additive.** The recipes it runs are byte-identical to the ones a native run uses. -With no recipe, the command opens an interactive shell: +## 3. The two artifacts ```text -just anvil-container +repo/ +├── justfiles/anvil/ +│ ├── container.just the anvil-container recipe and its helpers +│ └── … checks, groups, tiers (unchanged; run natively *inside* the image) +└── .anvil/container/ + ├── Dockerfile what the image contains + ├── Dockerfile.dockerignore what the build context admits + └── hooks.ps1 optional; credentials, not emitted by default ``` -Native tier execution remains the default. The three public tiers can instead -route through the container: - -- for one invocation: `just anvil_runner=container anvil-pr`; -- for the current shell: set `ANVIL_RUNNER=container`; -- for the repository: change the default in the `anvil-runner` region of the - repository-root `Justfile` and commit that policy. - -`ANVIL_RUNNER=native` overrides a repository container default for the current -shell. - -The tier recipes delegate to a tool-owned `_anvil-run` seam. Inside the image, -`ANVIL_IN_CONTAINER=1` forces that seam to select native execution, so the -existing private tier runs without recursively launching another container. -Ad-hoc checks remain explicit through `anvil-container`. - -`just` does not support conditional dependency lists, so `_anvil-run` starts a -second `just` invocation for the selected private tier. It reuses the exact -parsed Justfile and preserves ordinary native output and exit status. Global -CLI options, variable assignments, dependency introspection, and `--dry-run` -apply to the outer invocation and are not propagated to the selected tier. - -## 4. Architecture - -Container support consists of the generated recipe -`justfiles/anvil/container.just` and a generated artifact group under -`.anvil/container/`. The recipe selects the PowerShell driver on Windows and the -Bash driver on Linux or WSL. The PowerShell driver invokes Docker Engine in the -default WSL distribution; the Bash driver invokes the local Docker Engine -directly. Both implement the same lifecycle: - -```mermaid -flowchart TD - user["just anvil-container <recipe>"] --> dispatch["Select the host driver"] - dispatch --> identity["Compute the content-based image ID"] - identity --> customize["Inspect the local image
and load trusted customization"] - customize --> github{"Does the request need GitHub access?"} - github -- Yes --> auth["Acquire host or customized credentials"] - github -- No --> exists - auth --> exists{"Matching local image exists?"} - exists -- No --> build["Build the image
and run just anvil-setup"] - exists -- Yes --> prepare - build --> prepare["Run optional dependency preparation"] - prepare --> aprz["Run anvil-aprz with a temporary token mount when required"] - aprz --> checks["Run the requested recipes without the token"] - checks --> cleanup["Remove temporary credentials and containers"] -``` +`container.just` is generated and should not be edited. The `Dockerfile` pair is generated but **deliberately +editable**: anvil's drift handling preserves a repository's changes. + +The image installs its tools by running `just anvil-setup` — the same recipe the checks use, from the same generated +pins. There is no second tool list to keep in step, so "the image has the right tools" is true by construction. It is +also why a tool-pin bump changes the image identity: `versions.just` is both what the image installs and part of what +names it. + +### Recipes -The driver: +| Recipe | Purpose | +| --- | --- | +| `just anvil-container ` | Run any anvil recipe in the image. No argument opens an interactive shell. | +| `just anvil-container-status` | Report the engine, the image reference, and whether it is present. | +| `just anvil-container-rebuild` | Rebuild ignoring every cached layer. | +| `just anvil-container-down` | Remove this repository's cache volumes. The image is left in place. | -1. validates the host prerequisites and locates the Git repository root; -2. computes the image ID from build-relevant generated content; -3. checks image availability and loads and validates trusted customization; -4. prepares any credentials required by the requested recipes; -5. builds the matching image when it is not already available; -6. runs an optional downstream dependency-preparation command; -7. starts a short-lived container with the repository and named caches mounted; -8. invokes the requested recipes with `just`, or starts an interactive shell; -9. removes temporary credential files on success or failure. +`ANVIL_IN_CONTAINER=1` is set inside the image, so a nested invocation runs natively and the work happens exactly +once — one container per top-level command, not one per check. -## 5. Image construction and identity +## 4. Image identity -The public `Containerfile` starts from a pinned public Linux base and installs -`just`, Rustup, and PowerShell. It copies the generated Anvil tree and the -repository-owned `rust-toolchain.toml`, then runs: +The tag **is** the hash of the inputs that define it: ```text -just anvil-setup +anvil-:<16 hex characters> ``` -This makes the generated setup recipes and the container image use one source -of truth for Rust toolchains and Cargo tools. - -The local image tag is a SHA-256 hash of build-relevant repository content: - -- `rust-toolchain.toml`; -- generated `justfiles/anvil/**/*.just` recipes; -- the `Containerfile`, `Containerfile.dockerignore`, entrypoint, and other - static image inputs. -- the selected digest-pinned base image from `ANVIL_CONTAINER_BASE_IMAGE`, or - the `Containerfile` default when the variable is absent. - -Execution-only drivers, image-ID helpers, the entry recipe, user -documentation, and `customize.sh`/`customize.ps1` are excluded. Customization -source is runtime orchestration, not image content: it is excluded from both -image identity and the build context, so it can never silently change what a -tag names. See [8.9](#89-image-identity-and-the-build-context). Paths are -sorted and deduplicated, and line endings are normalized so the Bash and -PowerShell helpers produce the same ID. - -By default, the image is tagged `anvil-dev:`. A changed tool pin, -recipe, toolchain, or other static image artifact selects a new immutable tag. -The next invocation builds that image, while images for older branches remain -available. Runtime execution uses `--pull=never` and never substitutes -`latest`. - -Container execution requires a `rust-toolchain.toml` in the repository root. It -does not choose a default Rust channel when that file is absent. - -The public default is digest-pinned Debian Bookworm. A user or automation can -select another image compatible with the generated Debian-based -`Containerfile` through `ANVIL_CONTAINER_BASE_IMAGE`. This supports a lower -glibc baseline such as Debian Bullseye without replacing the generated file. -A distribution with a different package ecosystem requires a derived -`Containerfile`. Unpinned tags are rejected. - -## 6. Runtime and cache model - -Each invocation uses a short-lived container and persistent named volumes: - -```mermaid -flowchart LR - repo["Host repository"] -->|read/write bind mount| workspace["/workspace"] - token["Temporary token file"] -.->|read-only when required| runtime["Anvil container"] - registry[("Repository-scoped Cargo registry volume")] --> cargo["Per-user Cargo home"] - git[("Repository-scoped Cargo Git volume")] --> cargo - target[("Repository- and image-specific target volume")] --> workspace - workspace --> runtime - cargo --> runtime -``` +Hashed: the `Dockerfile`, its ignore file, `rust-toolchain.toml`, `hooks.ps1` when present, and every `*.just` under +`justfiles/anvil/` except `container.just` itself — hashing the driver would make the tag depend on the tag. -- The repository is bind-mounted read/write at `/workspace`. -- Cargo registry and Cargo Git data use repository-specific named volumes - shared across branches and image IDs of that repository. -- `target/` uses a repository- and image-specific named volume mounted over - `/workspace/target`. Container builds therefore do not use the host - `target/`. -- Docker runs the image as `linux/amd64` with the invoking Linux/WSL user's - numeric user and group IDs. -- The image sets `ANVIL_IN_CONTAINER=1` and uses `--pull=never`. +Change any of them and the tag names an image that cannot already exist, so a build follows. Change nothing and the +tag resolves instantly. There is no staleness check because there is nothing to check: an image that is present is, +by construction, built from the current inputs. -The driver creates the named volumes explicitly, initializes their top-level -ownership in a short-lived root container, and runs preparation and recipe -containers as the non-root Linux/WSL user. The root container never runs -repository recipes. +| Variable | Effect | +| --- | --- | +| `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | +| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is missing. Distinguishes a cache miss from a build failure. | +| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves. What `anvil-container-rebuild` sets. | -The entrypoint creates a writable Cargo home for the invoking non-root user. It -copies Cargo installation metadata so `cargo install --list` can discover tools -installed into the image, then links the shared registry and Git caches into -that Cargo home. +The hook's *output* is deliberately not hashed: a credential must never influence a tag. -The separate target volume prevents incompatible host and container artifacts -from mixing. Including the image ID in its name also prevents an older branch -from reusing target output produced by a different toolchain or generated -catalog. +## 5. Credentials — the hook -After the initial image build, repeated container runs reuse the image, -dependency caches, and compilation output, substantially reducing warm-run -time. +crates.io needs none, so nothing is emitted by default. A repository or a downstream catalog that needs credentials +adds `.anvil/container/hooks.ps1`, which the recipe loads whenever it is present, regardless of who put it there: -## 7. Authentication and secret isolation +```powershell +function Anvil-PreBuild { + @{ Secrets = @{ feed_token = (az account get-access-token --resource … --query accessToken -o tsv) } } +} -Authentication has distinct public and downstream extension paths. +function Anvil-PreRun { + @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = "Bearer …" } } +} +``` -### 7.1 GitHub API access +Both functions are optional. `Secrets` become BuildKit `--secret` mounts at build time; `Env` becomes `-e NAME` at +run time. In both cases the value is handed over **by environment variable name**, so it never appears in the host's +process command line — where endpoint telemetry records and retains it far longer than a short-lived token is meant +to live — and BuildKit keeps build secrets out of every image layer. -The public `anvil-aprz` recipe requires authenticated GitHub API access. The -drivers recognize `anvil-aprz` and aggregate tiers that invoke it, then obtain a -token from the host `GITHUB_TOKEN` or an authenticated host `gh` session. -Because customization loads first, trusted downstream customization can obtain -a short-lived token and assign it to the process `GITHUB_TOKEN`. +The corresponding `RUN` should declare the mount as required, which closes the same hole from the Dockerfile's side: -For an aggregate tier, the driver: +```dockerfile +RUN --mount=type=secret,id=feed_token,required=true \ + TOKEN="$(cat /run/secrets/feed_token)" … +``` -1. writes the token to a user-only temporary file; -2. runs `anvil-aprz` in a separate container with that file mounted read-only; -3. marks APRZ as complete; -4. runs the remaining checks without the token mount; -5. removes the temporary file during cleanup. +Anything the build *writes* with a secret is ordinary content: anvil's own Dockerfile deletes `credentials.toml` and +`.netrc` in the same layer as the install, and a replacement must do the same or the credential is baked into a layer. -An interactive invocation can pause while the user completes `gh auth login`. -A non-interactive invocation fails with an actionable error before building the -image when authentication is unavailable. +**An empty value is a hard error.** BuildKit is not: `--secret id=t,env=UNSET` exits 0 having mounted an empty secret, +so the build would install a reduced tool set and be tagged with the *same* content hash a credentialed build +produces — and every later run would reuse the broken image. -## 8. Container customization +**Trust.** The hook runs on the host, with the developer's permissions, before any container isolation. Only run one +from a repository or catalog you trust. Everything inside the container then runs as one user in one mount namespace, +so a forwarded credential is reachable by anything the checks execute, including dependency build scripts and proc +macros. Keep the set narrow and the token short-lived. -Repositories and derived `cargo-anvil` distributions can customize image -construction, dependency preparation, runtime arguments, and cleanup through: +## 6. Host setup -```text -.anvil/container/customize.sh -.anvil/container/customize.ps1 +The engine must be callable from the shell that runs `just`. Docker is the supported path; podman is best-effort and +currently untested — it uses buildah rather than BuildKit, so the secret semantics above are unverified there. + +anvil installs nothing. On Windows, either use an engine that ships a Windows CLI (Docker Desktop, podman), or run +Docker inside WSL and point a Windows `docker` CLI at it: + +```powershell +wsl --install -d Ubuntu-24.04 +wsl -d Ubuntu-24.04 -- sh -c 'printf "[boot]\nsystemd=true\n" | sudo tee /etc/wsl.conf' +wsl -d Ubuntu-24.04 -- sh -c 'curl -fsSL https://get.docker.com | sh' +wsl -d Ubuntu-24.04 -- sudo usermod -aG docker "$USER" +wsl --shutdown +wsl -d Ubuntu-24.04 -- docker version # verify ``` -> [!WARNING] -> These files execute on the host with the developer's permissions before -> container isolation. Checking out a branch that adds or changes one of them -> and then running `just anvil-container` executes that code on the host. - -The public catalog does not generate these files. A repository can commit them -directly, or a derived distribution can add them through the artifact API in -[extensibility.md](./extensibility.md). The driver treats both sources -identically. These files are trusted host code, sourced with the developer's -permissions outside the container sandbox. - -The customization interface provides these read-only inputs: - -| Purpose | Bash | PowerShell | Type | -|---|---|---|---| -| Repository root | `ANVIL_CONTAINER_REPO_ROOT` | `$AnvilContainerRepoRoot` | Absolute path | -| Container directory | `ANVIL_CONTAINER_DIR` | `$AnvilContainerDir` | Absolute path | -| WSL repository root | Not applicable | `$AnvilContainerRepoRootWsl` | Absolute WSL path for Docker arguments | -| WSL container directory | Not applicable | `$AnvilContainerDirWsl` | Absolute WSL path for Docker arguments | -| Resolved image | `ANVIL_CONTAINER_RESOLVED_IMAGE` | `$AnvilContainerResolvedImage` | Image name plus content tag | -| Matching image exists | `ANVIL_CONTAINER_IMAGE_EXISTS` | `$AnvilContainerImageExists` | Boolean | -| Requested recipes | `ANVIL_CONTAINER_REQUESTED_RECIPES` | `$AnvilContainerRequestedRecipes` | String array | -| Host is Windows | Not applicable | `$AnvilContainerHostIsWindows` | Boolean | - -The driver initializes and validates these outputs: - -| Purpose | Bash | PowerShell | Type and default | -|---|---|---|---| -| BuildKit secret arguments | `ANVIL_CONTAINER_BUILD_ARGS` | `$AnvilContainerBuildArgs` | String array, empty | -| Preparation arguments | `ANVIL_CONTAINER_PREPARE_ARGS` | `$AnvilContainerPrepareArgs` | String array, empty | -| Preparation command | `ANVIL_CONTAINER_PREPARE_COMMAND` | `$AnvilContainerPrepareCommand` | String array, empty | -| Main runtime arguments | `ANVIL_CONTAINER_RUN_ARGS` | `$AnvilContainerRunArgs` | String array, empty | -| Requested recipes include APRZ | `ANVIL_CONTAINER_NEEDS_GITHUB_TOKEN` | `$AnvilContainerNeedsGitHubToken` | Boolean, derived from public recipes; customization can elevate to true | -| Cleanup callback | `ANVIL_CONTAINER_CLEANUP` | `$AnvilContainerCleanup` | Function name or script block, no-op | - -The driver checks image availability before sourcing customization, validates -outputs, obtains any required GitHub token, then runs the build, optional -preparation, requested recipes, and cleanup phases in order. Failures stop the -invocation and run registered cleanup. - -- Build arguments apply only when constructing a missing image and accept only - BuildKit `--secret` options. Content-changing options such as `--build-arg` - are rejected because their values are not part of the content-addressed - image ID. Static build behavior belongs in hashed container files. -- Preparation runs in a separate short-lived container with the standard - repository and cache mounts, but without main runtime arguments. -- Runtime arguments apply to the requested recipe and the isolated - `anvil-aprz` invocation. Do not forward credentials needed only during build - or preparation. -- Customization that provisions GitHub authentication can assign a short-lived - token to process `GITHUB_TOKEN`. Register cleanup immediately for any - supporting files or external credentials. -- A downstream catalog whose additional aggregate recipe invokes - `anvil-aprz` can set the APRZ-classification output to true. The driver then - performs the same isolated authenticated APRZ phase used by public tiers. -- Cleanup runs after ordinary success, failure, or interactive-shell exit. It - cannot run after forcible process termination or machine failure. - -Customization authors are responsible for least-privilege credentials, -user-restricted temporary files, read-only secret mounts, immediate cleanup -registration, and equivalent Bash and PowerShell behavior. The driver cannot -prevent trusted customization from exposing or persisting secrets. - -`customize.*` is excluded from both image identity and the build context. -Non-secret behavior that changes image contents belongs in hashed static files -such as the `Containerfile`, entrypoint, or supporting build scripts. - -The documented paths, variables, lifecycle, and image-identity behavior form -the compatibility contract. Customizations must not depend on other driver -internals. - -## 9. Downstream extensibility - -Container support is a normal catalog artifact group. A downstream catalog can: - -- replace the `Containerfile` or entrypoint; -- add an optional `customize.sh`/`customize.ps1` customization file and - supporting files; -- inherit the public recipe, drivers, image-ID helpers, cache layout, and - runtime contract unchanged. - -Container support is coupled to the generated imports, tier runner, and APRZ -guard. Removing only `container::all()` is therefore unsupported; a derived -catalog that does not expose container execution must replace that complete -recipe surface rather than removing the container files in isolation. - -This keeps public behavior generic while allowing a downstream catalog to -provide an internal base image, toolchain installer, registry configuration, -and short-lived authentication. - -See [extensibility.md](./extensibility.md) for the catalog builder API. - -## 10. Requirements, controls, and limitations - -Host requirements: - -- Docker Engine 23.0 or newer, installed directly in Linux or WSL and usable by - the current user; -- `git` and `just`; -- Bash on Linux and WSL; -- PowerShell Core (`pwsh`) and WSL 2 on Windows; -- Docker Engine running in the default WSL distribution when invoked from - Windows; `wsl -e docker version` must succeed and the driver does not invoke - Windows `docker.exe`; -- `linux/amd64` execution support; -- a repository-owned `rust-toolchain.toml`. - -Runtime controls: +With that arrangement the daemon is Linux-side, so `DOCKER_HOST` must reach its socket and the repository must be +bind-mountable at a path the daemon understands. -| Variable | Effect | -|---|---| -| `ANVIL_CONTAINER_BASE_IMAGE` | Selects a compatible digest-pinned Linux base image; included in the image ID | -| `ANVIL_CONTAINER_IMAGE` | Overrides the local image name; the content hash remains the tag | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fails when the matching image is absent | -| `ANVIL_RUNNER` | Selects `native` or `container` tier execution | -| `ANVIL_IN_CONTAINER` | Internal recursion guard set by the image | - -The initial image build installs the complete pinned tool catalog and can take -several minutes. Later runs with the same image ID reuse the image and target -volume; Cargo registry and Git caches are reused across image IDs. - -Two concurrent cold invocations can both observe that an image is absent and -build the same content-addressed tag. The local backend accepts this redundant -work instead of introducing cross-platform lock ownership and stale-lock -recovery. Both invocations use the same hashed static inputs and selected base -image. Build secrets are intentionally excluded from identity and must provide -equivalent authenticated access rather than select different image content. - -On ARM64 hosts, Docker emulates `linux/amd64`. The driver warns about this -because image builds and checks can be substantially slower than on x86-64. - -The initial implementation is deliberately limited to: - -- local developer execution; -- Linux containers using `linux/amd64`; -- local image construction. - -CI container jobs, remote image publication, registry consumption, and Windows -containers are separate concerns and are not part of this local container -support. - -## 11. Alternatives considered - -- **Native `just anvil-setup` only.** This remains the default and fastest - inner loop, but it cannot provide a pinned Linux distribution or glibc - baseline from Windows and other hosts. -- **VS Code Dev Containers.** They provide a full editor environment, but - require a specific development workflow and do not provide a lightweight - command surface for terminals, agents, or existing editors. -- **A plain `docker run -v` wrapper.** This is simpler initially, but leaves - image construction, tool installation, cache ownership, content identity, - GitHub-secret isolation, and downstream preparation to every repository. -- **Published prebuilt images.** They improve cold-start time but introduce a - registry lifecycle, access policy, retention, and synchronization problem. - Local content-addressed builds keep the initial public feature independent - of registry infrastructure. -- **One fixed deployment distribution.** A Debian-compatible lower-glibc base - can be selected through the base-image override. Azure Linux or another - package ecosystem uses a derived `Containerfile`. The public default remains - broadly available Debian rather than coupling the open-source catalog to one - internal deployment target. - -## 12. Generated artifact reference - -| Path | Purpose | -|---|---| -| `justfiles/anvil/container.just` | Public `anvil-container` entry recipe | -| `.anvil/container/Containerfile` | Generic Linux image definition | -| `.anvil/container/Containerfile.dockerignore` | Restricted image build context | -| `.anvil/container/entrypoint.sh` | Non-root Cargo initialization | -| `.anvil/container/image-id.ps1` | Windows image-ID helper | -| `.anvil/container/image-id.sh` | Unix image-ID helper | -| `.anvil/container/run-in-container.ps1` | Windows driver for Docker Engine in WSL | -| `.anvil/container/run-in-container.sh` | Linux and WSL Docker Engine driver | -| `.anvil/container/customize.ps1` | Optional, not emitted by default; repository or derived-distribution Windows customization, see §8 | -| `.anvil/container/customize.sh` | Optional, not emitted by default; repository or derived-distribution Unix customization, see §8 | -| `.anvil/container/README.md` | Generated user instructions and troubleshooting | -| `justfiles/anvil/runner.just` | Native/container tier dispatch | - -The catalog also emits the user-owned `anvil-runner` region in the -repository-root `Justfile`. - -## 13. References - -- [Overall cargo-anvil design](./README.md) -- [Local recipe design](./local.md) -- [Catalog extensibility](./extensibility.md) -- [Continuous verification](../verification.md) +For podman, `podman machine init` provisions and manages its own WSL2 virtual machine; set +`ANVIL_CONTAINER_ENGINE=podman`. + +## 7. Customizing the image + +| You want | Do this | +| --- | --- | +| Extra packages in one repository | Edit `.anvil/container/Dockerfile`; the drift flow preserves it | +| A different base OS or toolchain source for a whole organization | `replace_artifact(artifacts::container::dockerfile().with_body(…))` in a downstream catalog | +| Credentials | Add `hooks.ps1`, by hand or via `with_artifact(artifacts::container::hooks(…))` | + +A catalog that replaces the Dockerfile with one that copies more of the tree must replace +`artifacts::container::dockerignore()` too, since the build context is scoped by that file. + +Keep `ARG BASE_IMAGE` digest-pinned. A floating tag can change underneath a tag that claims to name fixed content, +which would make every cached image a potential lie. + +## 8. Limits + +- Linux-only, `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. +- Local-only: no registry integration, no push, no promotion. The image is built and consumed locally, which is why + it needs no published artifact to exist. +- A repository-owned `rust-toolchain.toml` is required — it is both what the image installs and part of what names it. +- The first build takes several minutes: it installs a toolchain and the whole pinned tool catalog. Later runs reuse + it until an input changes. +- `target/` stays on the bind mount; the cargo and rustup homes live in named volumes so the hot write path does not + cross the host boundary. + +[design]: ./README.md diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index 649210f5..54c991b5 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -464,55 +464,42 @@ the relevant `OwnedFile` (e.g. `checks.just`) wholesale rather than editing indi This is a modest, low-risk refactor: it data-drives the artifact list (§4) without disturbing the engine internals or the template format. -### 6.1 Optional container runner - -The public base catalog emits an explicit `anvil-container` recipe at -`justfiles/anvil/container.just` and its Containerfile, Docker Engine drivers, -content-address helper, and README under `.anvil/container/`. Native -`just anvil-*` execution remains the default. - -A downstream catalog replaces only environment-specific artifacts such as -`artifacts::container::containerfile()` and can add the standard -`artifacts::container::customize_shell(...)` / -`artifacts::container::customize_powershell(...)` files. The public drivers, -image selection, caches, repository mounts, and recipe forwarding remain -unchanged. Static image behavior stays in hashed artifacts; `customize.*` -provides documented runtime orchestration. - -Two placement rules follow from how the container backend derives image -identity and the build context, and both are enforced or documented rather -than left to discovery: - -- **`justfiles/` holds `.just` recipes only.** The image ID hashes `*.just` - files under `justfiles/anvil/`, and the build-context allow-list admits only - those, so any other owned file placed there would be silently dropped from - both. [`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects such an - artifact, so a derived catalog fails loudly at construction instead of - shipping a file the container backend ignores. Non-recipe assets belong in a - tool-owned directory such as `.anvil/`. -- **Recipe locations outside the emitted shape need an ignore-file - override.** `Containerfile.dockerignore` is a deny-all allow-list whose - re-inclusions are per-directory (`justfiles/anvil/*.just`, - `justfiles/anvil/checks/*.just`, `justfiles/anvil/groups/*.just`, - `.anvil/container/*`); Docker only descends into a denied directory when - some re-inclusion pattern is prefixed by it. The image ID, by contrast, - hashes recipes recursively. A catalog that adds a recipe directory beyond - those three must also replace `artifacts::container::ignore_file()`; - otherwise the file is hashed into the image ID but never copied, and the - image build fails on the missing import. `.anvil/container/` is leaf-only on - both sides — the image-ID helpers list it one level deep, and - `.anvil/container/*/*` keeps the allow-list to the same depth — so a nested - asset is neither hashed nor copied, and a catalog that wants one must - replace the ignore file and the image-ID helpers together. - -`customize.sh`/`customize.ps1` are trusted host code: the driver sources them -directly into its process before image construction and recipe execution, so -they run with the invoking developer's permissions and outside the container -sandbox. The runtime contract is file-based and ownership-neutral — a regular -repository can commit the standard paths directly, without a derived catalog, -with identical driver behavior. See the [container customization -contract](./containers.md#8-container-customization) for the full -interface, trust boundary, and security responsibilities. +### 6.1 Containerized execution + +The base catalog emits three files: the `anvil-container` recipe at +`justfiles/anvil/container.just`, and the image definition at +`.anvil/container/Dockerfile` with its `Dockerfile.dockerignore`. Native +`just anvil-*` execution is unaffected — the container is reached only through +the explicit recipe. + +A downstream catalog customizes exactly two things: + +- `replace_artifact(artifacts::container::dockerfile().with_body(...))` to build + on a different base OS or install the toolchain from a different source. The + recipe, the image-identity hash, the cache volumes and the mounts are + inherited unchanged. +- `with_artifact(artifacts::container::hooks(...))` to supply credentials. The + recipe loads `.anvil/container/hooks.ps1` whenever it is present, so the + contract is ownership-neutral: a regular repository can commit the same path + directly, without a derived catalog, and behavior is identical. + +A catalog that replaces the Dockerfile with one that copies more of the tree +must also replace `artifacts::container::dockerignore()`, since the build +context is scoped by that file. The image identity hashes every `*.just` under +`justfiles/anvil/` recursively, so a catalog that adds a recipe directory gets +it hashed automatically, but must widen the ignore file for it to be copied. + +`justfiles/` holds `.just` recipes only. +[`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects any other owned +file placed there; non-recipe assets belong in a tool-owned directory such as +`.anvil/`. + +The hook is trusted host code: the recipe dot-sources it into its own process +before the build and before the run, so it executes with the invoking +developer's permissions and outside the container sandbox. See +[containers.md §5](./containers.md#5-credentials--the-hook) for the full +interface and trust boundary. + The public engine contains no environment-specific image, registry, cloud, or credential-provider details. diff --git a/crates/cargo-anvil/docs/design/local.md b/crates/cargo-anvil/docs/design/local.md index 46650639..a08a3d0d 100644 --- a/crates/cargo-anvil/docs/design/local.md +++ b/crates/cargo-anvil/docs/design/local.md @@ -38,7 +38,7 @@ repo/ │ │ anvil-pr-runtime-analysis, anvil-pr-mutants, │ │ anvil-scheduled-test, …). `anvil-pr-slow` is a │ │ convenience umbrella over the three pr-slow sub-groups. -│ ├── container.just optional container entry recipe (`anvil-container`). +│ ├── container.just containerized execution (`anvil-container`). See containers.md. │ ├── tiers.just tier aggregators (anvil-pr, anvil-scheduled, anvil-full). │ ├── tools.just tool/component/toolchain install + validate-prereqs recipes, │ │ plus the cargo-spellcheck source-deps check and @@ -47,20 +47,15 @@ repo/ │ as plain just variables (rust_nightly, cargo_nextest_version, …). │ Read by recipes via `{{ var }}` interpolation. See §3. │ -└── .anvil/container/ optional non-recipe container assets - ├── Containerfile - ├── Containerfile.dockerignore - ├── README.md - ├── entrypoint.sh - ├── image-id.ps1 - ├── image-id.sh - ├── run-in-container.ps1 - └── run-in-container.sh +└── .anvil/container/ the container image definition + ├── Dockerfile editable; drift is preserved + ├── Dockerfile.dockerignore + └── hooks.ps1 optional; credentials, not emitted by default ``` The Justfile region is the only file anvil adds to that the user co-owns, and it's -a single `import` line. Generated recipes live inside `justfiles/anvil/`; optional -non-recipe container assets live inside `.anvil/container/`. Generated files in +a single `import` line. Generated recipes live inside `justfiles/anvil/`; the +container image definition lives inside `.anvil/container/`. Generated files in both directories are tool-owned (tracked by full-file checksum in the sidecar manifest). If the user wants to add project-specific recipes, they add them to the top-level `Justfile` outside the managed region, or to their own additional @@ -74,12 +69,11 @@ in `tools.just` (and the per-check/group/tier setup recipes colocated in the sam files) are annotated with `[group("anvil-setup")]`. `just --groups` therefore shows two clean clusters: one for "run checks", one for "install prereqs". -> **Optional container backend.** When a catalog includes the opt-in container -> backend, `justfiles/anvil/container.just` adds the -> `anvil-container ` command and `.anvil/container/` contains its -> non-recipe assets. It runs any recipe below inside a pinned Linux image -> (Linux-on-Windows parity, distro pinning) instead of against the host -> toolchain. The recipe bodies are unchanged; see [containers.md](./containers.md). +> **Containerized execution.** `justfiles/anvil/container.just` adds the +> `anvil-container ` command, which runs any recipe below inside a pinned +> Linux image instead of against the host toolchain (Linux-on-Windows parity, +> toolchain pinning). It is explicit: the tiers themselves always run natively. +> The recipe bodies are unchanged; see [containers.md](./containers.md). ## 2. Recipe layers diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 5a1cdab7..e5da0b7e 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -29,12 +29,21 @@ anvil_container_workdir := "/workspace" # container image references allow. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") -# Resolve the engine. There is deliberately no probe: presence is not +# Resolve how to invoke the engine, as a pipe-separated command. +# +# There is deliberately no probe *between* engines: presence is not # reachability, `podman-docker` aliases `docker` onto podman, and a silent # choice between two installed engines means two image stores and an # unexplained rebuild. We check that the requested binary exists and let every # other failure surface the engine's own diagnostic, which is more accurate # than anything repeated here. +# +# The one fallback is Windows-specific and unambiguous: when the engine is not +# on the Windows PATH, try it inside the default WSL distribution. Installing +# Docker in WSL without Docker Desktop is a documented, common setup, and it +# leaves no Windows CLI behind -- so without this the engine we told the user to +# install would be unreachable. Docker Desktop and Podman both ship a Windows +# CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] _anvil-container-engine: @@ -44,11 +53,46 @@ _anvil-container-engine: Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 } - if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { - Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + if (Get-Command $engine -ErrorAction SilentlyContinue) { + Write-Output $engine + exit 0 + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $engine --version *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output "wsl.exe|--|$engine" + exit 0 + } + } + Write-Error "anvil: '$engine' was not found on PATH, and is not usable in the default WSL distribution. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + +# Translate a host path into what the engine sees. +# +# Identical when the engine runs on this host. When it runs in WSL, a Windows +# path has to become its /mnt/... form or the daemon silently bind-mounts an +# empty directory -- a failure that surfaces much later, as a missing file +# inside the container. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-path host_path: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if (-not $engine.StartsWith('wsl.exe|')) { + Write-Output '{{host_path}}' + exit 0 + } + # Pass the path with forward slashes: arguments cross into WSL through a + # shell that would otherwise consume the backslashes, leaving wslpath to + # translate a mangled path. wslpath accepts either separator. + $hostPath = '{{host_path}}' -replace '\\', '/' + $translated = & wsl.exe -- wslpath -a -u $hostPath + if ($LASTEXITCODE -ne 0) { + Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" exit 1 } - Write-Output $engine + Write-Output $translated.Trim() # Resolve the exec image reference, building it if it is not already present. # @@ -67,8 +111,11 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{justfile_directory()}}' $dockerfile = '.anvil/container/Dockerfile' @@ -115,7 +162,7 @@ _anvil-container-image: $image = '{{anvil_container_name}}:' + $imageId if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { - & $engine image inspect $image *> $null + & $engineExe @enginePrefix image inspect $image *> $null if ($LASTEXITCODE -eq 0) { Write-Output $image exit 0 @@ -164,19 +211,29 @@ _anvil-container-image: # Progress goes to stderr: callers capture this recipe's stdout to learn # the image reference, so anything else written there becomes part of it. [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # The engine may not share this host's filesystem view, so the context + # and the Dockerfile are given in its terms rather than ours. + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. - $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } - $buildCmd += $repoRoot + $buildCmd += $engineRoot # BuildKit is required for --secret; docker enables it by default from # 23.0 but an older daemon silently ignores the flag, so ask explicitly. $env:DOCKER_BUILDKIT = '1' - & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + # WSLENV exports the named variables into the WSL environment, which is + # where the engine reads a secret's value from when it runs there. + if ($engineExe -eq 'wsl.exe') { + $bridged = @('DOCKER_BUILDKIT/u') + ($secretEnv | ForEach-Object { "$_/u" }) + $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -192,6 +249,8 @@ _anvil-container-image: # # ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs # natively and the work happens exactly once. + +# Run any anvil recipe inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container *target: @@ -203,12 +262,17 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{justfile_directory()}}' + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. @@ -222,7 +286,7 @@ anvil-container *target: $interactive = [string]::IsNullOrWhiteSpace($target) $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' - $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") # Cargo and rustup homes live in named volumes: the hot write path never # crosses the host boundary, and the host's own toolchain is untouched. $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') @@ -273,7 +337,12 @@ anvil-container *target: # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just', $target) } - & $engine @runArgs + # WSLENV exports the forwarded names into the WSL environment, which is + # where the engine reads their values from when it runs there. + if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -283,13 +352,15 @@ anvil-container *target: # # The tag embeds the hash of the image's inputs, so "absent" and "out of date" # are the same condition and are reported as one. + +# Report the engine, the exec image, and whether it is present and current. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "engine: $engine" + Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" # NO_REBUILD turns the resolve into a pure query: report the state instead @@ -310,6 +381,8 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. + +# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -325,9 +398,12 @@ anvil-container-rebuild: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { - & $engine volume rm -f $vol + & $engineExe @enginePrefix volume rm -f $vol } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 86174f42..7a966b5c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3504,12 +3504,21 @@ anvil_container_workdir := "/workspace" # container image references allow. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") -# Resolve the engine. There is deliberately no probe: presence is not +# Resolve how to invoke the engine, as a pipe-separated command. +# +# There is deliberately no probe *between* engines: presence is not # reachability, `podman-docker` aliases `docker` onto podman, and a silent # choice between two installed engines means two image stores and an # unexplained rebuild. We check that the requested binary exists and let every # other failure surface the engine's own diagnostic, which is more accurate # than anything repeated here. +# +# The one fallback is Windows-specific and unambiguous: when the engine is not +# on the Windows PATH, try it inside the default WSL distribution. Installing +# Docker in WSL without Docker Desktop is a documented, common setup, and it +# leaves no Windows CLI behind -- so without this the engine we told the user to +# install would be unreachable. Docker Desktop and Podman both ship a Windows +# CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] _anvil-container-engine: @@ -3519,11 +3528,46 @@ _anvil-container-engine: Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 } - if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { - Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + if (Get-Command $engine -ErrorAction SilentlyContinue) { + Write-Output $engine + exit 0 + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $engine --version *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output "wsl.exe|--|$engine" + exit 0 + } + } + Write-Error "anvil: '$engine' was not found on PATH, and is not usable in the default WSL distribution. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + +# Translate a host path into what the engine sees. +# +# Identical when the engine runs on this host. When it runs in WSL, a Windows +# path has to become its /mnt/... form or the daemon silently bind-mounts an +# empty directory -- a failure that surfaces much later, as a missing file +# inside the container. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-path host_path: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if (-not $engine.StartsWith('wsl.exe|')) { + Write-Output '{{host_path}}' + exit 0 + } + # Pass the path with forward slashes: arguments cross into WSL through a + # shell that would otherwise consume the backslashes, leaving wslpath to + # translate a mangled path. wslpath accepts either separator. + $hostPath = '{{host_path}}' -replace '\\', '/' + $translated = & wsl.exe -- wslpath -a -u $hostPath + if ($LASTEXITCODE -ne 0) { + Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" exit 1 } - Write-Output $engine + Write-Output $translated.Trim() # Resolve the exec image reference, building it if it is not already present. # @@ -3542,8 +3586,11 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{justfile_directory()}}' $dockerfile = '.anvil/container/Dockerfile' @@ -3590,7 +3637,7 @@ _anvil-container-image: $image = '{{anvil_container_name}}:' + $imageId if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { - & $engine image inspect $image *> $null + & $engineExe @enginePrefix image inspect $image *> $null if ($LASTEXITCODE -eq 0) { Write-Output $image exit 0 @@ -3639,19 +3686,29 @@ _anvil-container-image: # Progress goes to stderr: callers capture this recipe's stdout to learn # the image reference, so anything else written there becomes part of it. [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # The engine may not share this host's filesystem view, so the context + # and the Dockerfile are given in its terms rather than ours. + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. - $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } - $buildCmd += $repoRoot + $buildCmd += $engineRoot # BuildKit is required for --secret; docker enables it by default from # 23.0 but an older daemon silently ignores the flag, so ask explicitly. $env:DOCKER_BUILDKIT = '1' - & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + # WSLENV exports the named variables into the WSL environment, which is + # where the engine reads a secret's value from when it runs there. + if ($engineExe -eq 'wsl.exe') { + $bridged = @('DOCKER_BUILDKIT/u') + ($secretEnv | ForEach-Object { "$_/u" }) + $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -3667,6 +3724,8 @@ _anvil-container-image: # # ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs # natively and the work happens exactly once. + +# Run any anvil recipe inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container *target: @@ -3678,12 +3737,17 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{justfile_directory()}}' + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. @@ -3697,7 +3761,7 @@ anvil-container *target: $interactive = [string]::IsNullOrWhiteSpace($target) $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' - $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") # Cargo and rustup homes live in named volumes: the hot write path never # crosses the host boundary, and the host's own toolchain is untouched. $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') @@ -3748,7 +3812,12 @@ anvil-container *target: # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just', $target) } - & $engine @runArgs + # WSLENV exports the forwarded names into the WSL environment, which is + # where the engine reads their values from when it runs there. + if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -3758,13 +3827,15 @@ anvil-container *target: # # The tag embeds the hash of the image's inputs, so "absent" and "out of date" # are the same condition and are reported as one. + +# Report the engine, the exec image, and whether it is present and current. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "engine: $engine" + Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" # NO_REBUILD turns the resolve into a pure query: report the state instead @@ -3785,6 +3856,8 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. + +# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -3800,10 +3873,13 @@ anvil-container-rebuild: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { - & $engine volume rm -f $vol + & $engineExe @enginePrefix volume rm -f $vol } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index cf5c2260..aab76060 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3425,12 +3425,21 @@ anvil_container_workdir := "/workspace" # container image references allow. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") -# Resolve the engine. There is deliberately no probe: presence is not +# Resolve how to invoke the engine, as a pipe-separated command. +# +# There is deliberately no probe *between* engines: presence is not # reachability, `podman-docker` aliases `docker` onto podman, and a silent # choice between two installed engines means two image stores and an # unexplained rebuild. We check that the requested binary exists and let every # other failure surface the engine's own diagnostic, which is more accurate # than anything repeated here. +# +# The one fallback is Windows-specific and unambiguous: when the engine is not +# on the Windows PATH, try it inside the default WSL distribution. Installing +# Docker in WSL without Docker Desktop is a documented, common setup, and it +# leaves no Windows CLI behind -- so without this the engine we told the user to +# install would be unreachable. Docker Desktop and Podman both ship a Windows +# CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] _anvil-container-engine: @@ -3440,11 +3449,46 @@ _anvil-container-engine: Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 } - if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { - Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + if (Get-Command $engine -ErrorAction SilentlyContinue) { + Write-Output $engine + exit 0 + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $engine --version *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output "wsl.exe|--|$engine" + exit 0 + } + } + Write-Error "anvil: '$engine' was not found on PATH, and is not usable in the default WSL distribution. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + +# Translate a host path into what the engine sees. +# +# Identical when the engine runs on this host. When it runs in WSL, a Windows +# path has to become its /mnt/... form or the daemon silently bind-mounts an +# empty directory -- a failure that surfaces much later, as a missing file +# inside the container. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-path host_path: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if (-not $engine.StartsWith('wsl.exe|')) { + Write-Output '{{host_path}}' + exit 0 + } + # Pass the path with forward slashes: arguments cross into WSL through a + # shell that would otherwise consume the backslashes, leaving wslpath to + # translate a mangled path. wslpath accepts either separator. + $hostPath = '{{host_path}}' -replace '\\', '/' + $translated = & wsl.exe -- wslpath -a -u $hostPath + if ($LASTEXITCODE -ne 0) { + Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" exit 1 } - Write-Output $engine + Write-Output $translated.Trim() # Resolve the exec image reference, building it if it is not already present. # @@ -3463,8 +3507,11 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{justfile_directory()}}' $dockerfile = '.anvil/container/Dockerfile' @@ -3511,7 +3558,7 @@ _anvil-container-image: $image = '{{anvil_container_name}}:' + $imageId if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { - & $engine image inspect $image *> $null + & $engineExe @enginePrefix image inspect $image *> $null if ($LASTEXITCODE -eq 0) { Write-Output $image exit 0 @@ -3560,19 +3607,29 @@ _anvil-container-image: # Progress goes to stderr: callers capture this recipe's stdout to learn # the image reference, so anything else written there becomes part of it. [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # The engine may not share this host's filesystem view, so the context + # and the Dockerfile are given in its terms rather than ours. + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. - $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } - $buildCmd += $repoRoot + $buildCmd += $engineRoot # BuildKit is required for --secret; docker enables it by default from # 23.0 but an older daemon silently ignores the flag, so ask explicitly. $env:DOCKER_BUILDKIT = '1' - & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + # WSLENV exports the named variables into the WSL environment, which is + # where the engine reads a secret's value from when it runs there. + if ($engineExe -eq 'wsl.exe') { + $bridged = @('DOCKER_BUILDKIT/u') + ($secretEnv | ForEach-Object { "$_/u" }) + $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -3588,6 +3645,8 @@ _anvil-container-image: # # ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs # natively and the work happens exactly once. + +# Run any anvil recipe inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container *target: @@ -3599,12 +3658,17 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{justfile_directory()}}' + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. @@ -3618,7 +3682,7 @@ anvil-container *target: $interactive = [string]::IsNullOrWhiteSpace($target) $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' - $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") # Cargo and rustup homes live in named volumes: the hot write path never # crosses the host boundary, and the host's own toolchain is untouched. $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') @@ -3669,7 +3733,12 @@ anvil-container *target: # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just', $target) } - & $engine @runArgs + # WSLENV exports the forwarded names into the WSL environment, which is + # where the engine reads their values from when it runs there. + if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -3679,13 +3748,15 @@ anvil-container *target: # # The tag embeds the hash of the image's inputs, so "absent" and "out of date" # are the same condition and are reported as one. + +# Report the engine, the exec image, and whether it is present and current. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "engine: $engine" + Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" # NO_REBUILD turns the resolve into a pure query: report the state instead @@ -3706,6 +3777,8 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. + +# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -3721,10 +3794,13 @@ anvil-container-rebuild: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { - & $engine volume rm -f $vol + & $engineExe @enginePrefix volume rm -f $vol } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 842d2242..df706252 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2244,12 +2244,21 @@ anvil_container_workdir := "/workspace" # container image references allow. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") -# Resolve the engine. There is deliberately no probe: presence is not +# Resolve how to invoke the engine, as a pipe-separated command. +# +# There is deliberately no probe *between* engines: presence is not # reachability, `podman-docker` aliases `docker` onto podman, and a silent # choice between two installed engines means two image stores and an # unexplained rebuild. We check that the requested binary exists and let every # other failure surface the engine's own diagnostic, which is more accurate # than anything repeated here. +# +# The one fallback is Windows-specific and unambiguous: when the engine is not +# on the Windows PATH, try it inside the default WSL distribution. Installing +# Docker in WSL without Docker Desktop is a documented, common setup, and it +# leaves no Windows CLI behind -- so without this the engine we told the user to +# install would be unreachable. Docker Desktop and Podman both ship a Windows +# CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] _anvil-container-engine: @@ -2259,11 +2268,46 @@ _anvil-container-engine: Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 } - if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { - Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + if (Get-Command $engine -ErrorAction SilentlyContinue) { + Write-Output $engine + exit 0 + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $engine --version *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output "wsl.exe|--|$engine" + exit 0 + } + } + Write-Error "anvil: '$engine' was not found on PATH, and is not usable in the default WSL distribution. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + +# Translate a host path into what the engine sees. +# +# Identical when the engine runs on this host. When it runs in WSL, a Windows +# path has to become its /mnt/... form or the daemon silently bind-mounts an +# empty directory -- a failure that surfaces much later, as a missing file +# inside the container. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-path host_path: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if (-not $engine.StartsWith('wsl.exe|')) { + Write-Output '{{host_path}}' + exit 0 + } + # Pass the path with forward slashes: arguments cross into WSL through a + # shell that would otherwise consume the backslashes, leaving wslpath to + # translate a mangled path. wslpath accepts either separator. + $hostPath = '{{host_path}}' -replace '\\', '/' + $translated = & wsl.exe -- wslpath -a -u $hostPath + if ($LASTEXITCODE -ne 0) { + Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" exit 1 } - Write-Output $engine + Write-Output $translated.Trim() # Resolve the exec image reference, building it if it is not already present. # @@ -2282,8 +2326,11 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{justfile_directory()}}' $dockerfile = '.anvil/container/Dockerfile' @@ -2330,7 +2377,7 @@ _anvil-container-image: $image = '{{anvil_container_name}}:' + $imageId if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { - & $engine image inspect $image *> $null + & $engineExe @enginePrefix image inspect $image *> $null if ($LASTEXITCODE -eq 0) { Write-Output $image exit 0 @@ -2379,19 +2426,29 @@ _anvil-container-image: # Progress goes to stderr: callers capture this recipe's stdout to learn # the image reference, so anything else written there becomes part of it. [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # The engine may not share this host's filesystem view, so the context + # and the Dockerfile are given in its terms rather than ours. + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. - $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } - $buildCmd += $repoRoot + $buildCmd += $engineRoot # BuildKit is required for --secret; docker enables it by default from # 23.0 but an older daemon silently ignores the flag, so ask explicitly. $env:DOCKER_BUILDKIT = '1' - & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + # WSLENV exports the named variables into the WSL environment, which is + # where the engine reads a secret's value from when it runs there. + if ($engineExe -eq 'wsl.exe') { + $bridged = @('DOCKER_BUILDKIT/u') + ($secretEnv | ForEach-Object { "$_/u" }) + $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -2407,6 +2464,8 @@ _anvil-container-image: # # ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs # natively and the work happens exactly once. + +# Run any anvil recipe inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container *target: @@ -2418,12 +2477,17 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{justfile_directory()}}' + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. @@ -2437,7 +2501,7 @@ anvil-container *target: $interactive = [string]::IsNullOrWhiteSpace($target) $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' - $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") # Cargo and rustup homes live in named volumes: the hot write path never # crosses the host boundary, and the host's own toolchain is untouched. $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') @@ -2488,7 +2552,12 @@ anvil-container *target: # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just', $target) } - & $engine @runArgs + # WSLENV exports the forwarded names into the WSL environment, which is + # where the engine reads their values from when it runs there. + if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -2498,13 +2567,15 @@ anvil-container *target: # # The tag embeds the hash of the image's inputs, so "absent" and "out of date" # are the same condition and are reported as one. + +# Report the engine, the exec image, and whether it is present and current. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "engine: $engine" + Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" # NO_REBUILD turns the resolve into a pure query: report the state instead @@ -2525,6 +2596,8 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. + +# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -2540,10 +2613,13 @@ anvil-container-rebuild: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { - & $engine volume rm -f $vol + & $engineExe @enginePrefix volume rm -f $vol } exit 0 diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 5a1cdab7..e5da0b7e 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -29,12 +29,21 @@ anvil_container_workdir := "/workspace" # container image references allow. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") -# Resolve the engine. There is deliberately no probe: presence is not +# Resolve how to invoke the engine, as a pipe-separated command. +# +# There is deliberately no probe *between* engines: presence is not # reachability, `podman-docker` aliases `docker` onto podman, and a silent # choice between two installed engines means two image stores and an # unexplained rebuild. We check that the requested binary exists and let every # other failure surface the engine's own diagnostic, which is more accurate # than anything repeated here. +# +# The one fallback is Windows-specific and unambiguous: when the engine is not +# on the Windows PATH, try it inside the default WSL distribution. Installing +# Docker in WSL without Docker Desktop is a documented, common setup, and it +# leaves no Windows CLI behind -- so without this the engine we told the user to +# install would be unreachable. Docker Desktop and Podman both ship a Windows +# CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] _anvil-container-engine: @@ -44,11 +53,46 @@ _anvil-container-engine: Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 } - if (-not (Get-Command $engine -ErrorAction SilentlyContinue)) { - Write-Error "anvil: '$engine' was not found on PATH. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + if (Get-Command $engine -ErrorAction SilentlyContinue) { + Write-Output $engine + exit 0 + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $engine --version *> $null + if ($LASTEXITCODE -eq 0) { + Write-Output "wsl.exe|--|$engine" + exit 0 + } + } + Write-Error "anvil: '$engine' was not found on PATH, and is not usable in the default WSL distribution. Install it, or set ANVIL_CONTAINER_ENGINE to the other engine. Setup: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md" + exit 1 + +# Translate a host path into what the engine sees. +# +# Identical when the engine runs on this host. When it runs in WSL, a Windows +# path has to become its /mnt/... form or the daemon silently bind-mounts an +# empty directory -- a failure that surfaces much later, as a missing file +# inside the container. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-path host_path: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if (-not $engine.StartsWith('wsl.exe|')) { + Write-Output '{{host_path}}' + exit 0 + } + # Pass the path with forward slashes: arguments cross into WSL through a + # shell that would otherwise consume the backslashes, leaving wslpath to + # translate a mangled path. wslpath accepts either separator. + $hostPath = '{{host_path}}' -replace '\\', '/' + $translated = & wsl.exe -- wslpath -a -u $hostPath + if ($LASTEXITCODE -ne 0) { + Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" exit 1 } - Write-Output $engine + Write-Output $translated.Trim() # Resolve the exec image reference, building it if it is not already present. # @@ -67,8 +111,11 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{justfile_directory()}}' $dockerfile = '.anvil/container/Dockerfile' @@ -115,7 +162,7 @@ _anvil-container-image: $image = '{{anvil_container_name}}:' + $imageId if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { - & $engine image inspect $image *> $null + & $engineExe @enginePrefix image inspect $image *> $null if ($LASTEXITCODE -eq 0) { Write-Output $image exit 0 @@ -164,19 +211,29 @@ _anvil-container-image: # Progress goes to stderr: callers capture this recipe's stdout to learn # the image reference, so anything else written there becomes part of it. [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") + # The engine may not share this host's filesystem view, so the context + # and the Dockerfile are given in its terms rather than ours. + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. - $buildCmd = @('build', '--platform', 'linux/amd64', '--file', (Join-Path $repoRoot $dockerfile), '--tag', $image) + $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } - $buildCmd += $repoRoot + $buildCmd += $engineRoot # BuildKit is required for --secret; docker enables it by default from # 23.0 but an older daemon silently ignores the flag, so ask explicitly. $env:DOCKER_BUILDKIT = '1' - & $engine @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } + # WSLENV exports the named variables into the WSL environment, which is + # where the engine reads a secret's value from when it runs there. + if ($engineExe -eq 'wsl.exe') { + $bridged = @('DOCKER_BUILDKIT/u') + ($secretEnv | ForEach-Object { "$_/u" }) + $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -192,6 +249,8 @@ _anvil-container-image: # # ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs # natively and the work happens exactly once. + +# Run any anvil recipe inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container *target: @@ -203,12 +262,17 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{justfile_directory()}}' + $engineRoot = (just _anvil-container-path $repoRoot).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. @@ -222,7 +286,7 @@ anvil-container *target: $interactive = [string]::IsNullOrWhiteSpace($target) $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' - $runArgs += @('-v', "${repoRoot}:{{anvil_container_workdir}}") + $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") # Cargo and rustup homes live in named volumes: the hot write path never # crosses the host boundary, and the host's own toolchain is untouched. $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') @@ -273,7 +337,12 @@ anvil-container *target: # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just', $target) } - & $engine @runArgs + # WSLENV exports the forwarded names into the WSL environment, which is + # where the engine reads their values from when it runs there. + if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + } + & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } @@ -283,13 +352,15 @@ anvil-container *target: # # The tag embeds the hash of the image's inputs, so "absent" and "out of date" # are the same condition and are reported as one. + +# Report the engine, the exec image, and whether it is present and current. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "engine: $engine" + Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" # NO_REBUILD turns the resolve into a pure query: report the state instead @@ -310,6 +381,8 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. + +# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -325,9 +398,12 @@ anvil-container-rebuild: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = just _anvil-container-engine + $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { - & $engine volume rm -f $vol + & $engineExe @enginePrefix volume rm -f $vol } exit 0 diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 new file mode 100644 index 00000000..ff3361d9 --- /dev/null +++ b/scripts/test-anvil-container.ps1 @@ -0,0 +1,547 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# +.SYNOPSIS + End-to-end test of cargo-anvil's containerized execution, from a user's seat. + +.DESCRIPTION + Creates a throwaway repository in a temp directory, generates the anvil tree + into it with the locally-built cargo-anvil, and then does only what a + developer would do: run `just anvil-container ` and observe what + happens. + + The setup phase is held to that standard deliberately. If this script has to + hand-write a file anvil should have generated, patch a generated file, or + work around a defect to get green, that is a bug in the product and not + something the script should paper over. + + What it proves: + + 1. A generated repository carries exactly the three container artifacts. + 2. The first run builds an image and runs the recipe inside it. + 3. A second run reuses the image (the tag resolves, nothing is built). + 4. Changing a hashed input (the pinned toolchain) selects a new tag. + 5. Reverting that input returns to the original tag. + 6. Editing the Dockerfile is preserved by a re-run of the generator. + 7. A credential hook reaches both the build and the run. + 8. A hook returning an empty value fails closed. + 9. A hook's output does not change the tag; its file content does. + 10. The recipes run natively inside the image (no nesting). + +.PARAMETER Engine + Container engine to test against. Defaults to $env:ANVIL_CONTAINER_ENGINE, + then 'docker'. + +.PARAMETER KeepArtifacts + Leave the temp repository and built images in place for inspection. + +.PARAMETER SkipCleanup + Skip the pre-run cleanup of images and volumes left by earlier runs. + +.EXAMPLE + ./scripts/test-anvil-container.ps1 + ./scripts/test-anvil-container.ps1 -Engine podman -KeepArtifacts +#> + +[CmdletBinding()] +param( + [ValidateSet('docker', 'podman')] + [string]$Engine = $(if ($env:ANVIL_CONTAINER_ENGINE) { $env:ANVIL_CONTAINER_ENGINE } else { 'docker' }), + [switch]$KeepArtifacts, + [switch]$SkipCleanup +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +# ---------------------------------------------------------------- reporting -- + +$script:Passed = 0 +$script:Failed = 0 +$script:Started = Get-Date + +function Write-Section([string]$Title) { + Write-Host '' + Write-Host "=== $Title " -NoNewline -ForegroundColor Cyan + Write-Host ('=' * [Math]::Max(0, 72 - $Title.Length)) -ForegroundColor Cyan +} + +function Write-Step([string]$Message) { + Write-Host " -> $Message" -ForegroundColor DarkGray +} + +function Write-Detail([string]$Message) { + foreach ($line in ($Message -split "`r?`n")) { + if ($line.Trim()) { Write-Host " | $line" -ForegroundColor DarkGray } + } +} + +function Assert-That([string]$Name, [bool]$Condition, [string]$Detail = '') { + if ($Condition) { + $script:Passed++ + Write-Host " [PASS] $Name" -ForegroundColor Green + } else { + $script:Failed++ + Write-Host " [FAIL] $Name" -ForegroundColor Red + if ($Detail) { Write-Detail $Detail } + } +} + +function Assert-Equal([string]$Name, $Expected, $Actual) { + Assert-That $Name ($Expected -eq $Actual) "expected: $Expected`nactual: $Actual" +} + +# ------------------------------------------------------------------ helpers -- + +function Invoke-Native { + param( + [Parameter(Mandatory)][string]$Command, + [string[]]$Arguments = @(), + [string]$WorkingDirectory, + [hashtable]$Environment = @{}, + [switch]$AllowFailure + ) + + $previous = @{} + foreach ($key in $Environment.Keys) { + $previous[$key] = [Environment]::GetEnvironmentVariable($key) + Set-Item -LiteralPath "Env:$key" -Value $Environment[$key] + } + $entered = $false + try { + if ($WorkingDirectory) { Push-Location $WorkingDirectory; $entered = $true } + $stdoutFile = [System.IO.Path]::GetTempFileName() + $stderrFile = [System.IO.Path]::GetTempFileName() + try { + $process = Start-Process -FilePath $Command -ArgumentList $Arguments -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput $stdoutFile -RedirectStandardError $stderrFile + $result = [pscustomobject]@{ + ExitCode = $process.ExitCode + StdOut = (Get-Content -LiteralPath $stdoutFile -Raw -ErrorAction SilentlyContinue) ?? '' + StdErr = (Get-Content -LiteralPath $stderrFile -Raw -ErrorAction SilentlyContinue) ?? '' + } + } finally { + Remove-Item -LiteralPath $stdoutFile, $stderrFile -Force -ErrorAction SilentlyContinue + } + } finally { + if ($entered) { Pop-Location } + foreach ($key in $Environment.Keys) { + if ($null -eq $previous[$key]) { + Remove-Item -LiteralPath "Env:$key" -ErrorAction SilentlyContinue + } else { + Set-Item -LiteralPath "Env:$key" -Value $previous[$key] + } + } + } + + if (-not $AllowFailure -and $result.ExitCode -ne 0) { + Write-Detail $result.StdOut + Write-Detail $result.StdErr + throw "$Command $($Arguments -join ' ') failed with exit code $($result.ExitCode)" + } + $result +} + +function Resolve-Engine { + # Mirrors what container.just does: prefer the engine on PATH, and fall + # back to the default WSL distribution on Windows. The script must not + # assume more than the product does. + if (Get-Command $Engine -ErrorAction SilentlyContinue) { + return [pscustomobject]@{ Exe = $Engine; Prefix = @(); ViaWsl = $false } + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe -- $Engine --version *> $null + if ($LASTEXITCODE -eq 0) { + return [pscustomobject]@{ Exe = 'wsl.exe'; Prefix = @('--', $Engine); ViaWsl = $true } + } + } + $null +} + +function Invoke-Engine { + param([string[]]$Arguments, [switch]$AllowFailure) + Invoke-Native -Command $script:EngineExe -Arguments ($script:EnginePrefix + $Arguments) -AllowFailure:$AllowFailure +} + +function ConvertTo-EnginePath([string]$Path) { + if (-not $script:EngineViaWsl) { return $Path } + (& wsl.exe -- wslpath -a -u ($Path -replace '\\', '/')).Trim() +} + +function Write-Fixture([string]$Path, [string]$Content) { + # LF, no BOM. This script is a CRLF file, so its here-strings carry CRLF; + # writing those verbatim would hand the fixture a repository that + # `anvil-fmt` correctly rejects for its newline style. A user cloning a + # normal repository does not start from that state, so neither should we. + $normalized = ($Content -replace "`r`n", "`n") + if (-not $normalized.EndsWith("`n")) { $normalized += "`n" } + $directory = Split-Path -Parent $Path + if ($directory -and -not (Test-Path -LiteralPath $directory)) { + New-Item -ItemType Directory -Path $directory -Force | Out-Null + } + [System.IO.File]::WriteAllText($Path, $normalized, [System.Text.UTF8Encoding]::new($false)) +} + +function Invoke-Just { + param( + [Parameter(Mandatory)][string]$Repo, + [Parameter(Mandatory)][string[]]$Arguments, + [hashtable]$Environment = @{}, + [switch]$AllowFailure + ) + $env = @{ ANVIL_CONTAINER_ENGINE = $Engine } + $Environment + Invoke-Native -Command 'just' -Arguments $Arguments -WorkingDirectory $Repo -Environment $env -AllowFailure:$AllowFailure +} + +function Get-ImageReference([string]$Repo) { + # anvil-container-status reports the reference without building it. + $status = Invoke-Just -Repo $Repo -Arguments @('anvil-container-status') -AllowFailure + $line = ($status.StdOut -split "`r?`n") | Where-Object { $_ -match '^\s*image:\s*(\S+)' } | Select-Object -First 1 + if ($line -match '^\s*image:\s*(\S+)') { return $Matches[1] } + '' +} + +function Test-ImagePresent([string]$Reference) { + if (-not $Reference) { return $false } + (Invoke-Engine -Arguments @('image', 'inspect', $Reference) -AllowFailure).ExitCode -eq 0 +} + +function Remove-AnvilImages([string]$Prefix) { + $images = Invoke-Engine -Arguments @('images', '--format', '{{.Repository}}:{{.Tag}}') -AllowFailure + $matching = ($images.StdOut -split "`r?`n") | Where-Object { $_ -like "$Prefix*" } + foreach ($image in $matching) { + Write-Step "removing image $image" + Invoke-Engine -Arguments @('rmi', '-f', $image) -AllowFailure | Out-Null + } + $volumes = Invoke-Engine -Arguments @('volume', 'ls', '--format', '{{.Name}}') -AllowFailure + $matchingVolumes = ($volumes.StdOut -split "`r?`n") | Where-Object { $_ -like "$Prefix*" } + foreach ($volume in $matchingVolumes) { + Write-Step "removing volume $volume" + Invoke-Engine -Arguments @('volume', 'rm', '-f', $volume) -AllowFailure | Out-Null + } +} + +# ------------------------------------------------------------ prerequisites -- + +Write-Section 'Prerequisites' + +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +Write-Step "source repository: $repoRoot" +Write-Step "engine: $Engine" + +foreach ($tool in @('just', 'cargo')) { + $found = Get-Command $tool -ErrorAction SilentlyContinue + Assert-That "$tool is on PATH" ([bool]$found) "install $tool" +} + +$resolved = Resolve-Engine +Assert-That "$Engine is reachable" ($null -ne $resolved) ` + "install $Engine so it is callable from this shell, or run it in the default WSL distribution" +if ($script:Failed -gt 0) { + Write-Host "`nPrerequisites missing; aborting." -ForegroundColor Red + exit 1 +} +$script:EngineExe = $resolved.Exe +$script:EnginePrefix = $resolved.Prefix +$script:EngineViaWsl = $resolved.ViaWsl +if ($resolved.ViaWsl) { + Write-Step "engine reached through the default WSL distribution (no Windows CLI on PATH)" +} + +$engineInfo = Invoke-Engine -Arguments @('version', '--format', '{{.Server.Version}}') -AllowFailure +if ($engineInfo.ExitCode -eq 0) { + Write-Step "engine server version: $($engineInfo.StdOut.Trim())" +} else { + Write-Host " [FAIL] $Engine is installed but its daemon is not reachable" -ForegroundColor Red + Write-Detail $engineInfo.StdErr + exit 1 +} + +# The tool under test is the one in this worktree, not whatever is installed. +Write-Step 'building cargo-anvil from this worktree' +Invoke-Native -Command 'cargo' -Arguments @('build', '-q', '-p', 'cargo-anvil') -WorkingDirectory $repoRoot | Out-Null +$anvilExe = Join-Path $repoRoot 'target/debug/cargo-anvil.exe' +if (-not (Test-Path -LiteralPath $anvilExe)) { $anvilExe = Join-Path $repoRoot 'target/debug/cargo-anvil' } +Assert-That 'cargo-anvil built' (Test-Path -LiteralPath $anvilExe) + +# ---------------------------------------------------------------- the repo --- + +Write-Section 'Fixture repository' + +# A stable directory name keeps the image name stable across runs, which is what +# makes the pre-run cleanup below able to find leftovers. +$fixtureName = 'anvil-e2e' +$workRoot = Join-Path ([System.IO.Path]::GetTempPath()) 'anvil-container-e2e' +$repo = Join-Path $workRoot $fixtureName +$imagePrefix = "anvil-$fixtureName" + +if (-not $SkipCleanup) { + Write-Step 'pre-run cleanup' + if (Test-Path -LiteralPath $workRoot) { + Remove-Item -LiteralPath $workRoot -Recurse -Force -ErrorAction SilentlyContinue + } + Remove-AnvilImages -Prefix $imagePrefix +} + +New-Item -ItemType Directory -Path $repo -Force | Out-Null +Write-Step "fixture: $repo" + +# Everything below is what a user would author by hand in a new repository. +Write-Fixture (Join-Path $repo 'Cargo.toml') @' +[package] +name = "anvil-e2e" +version = "0.1.0" +edition = "2021" + +[dependencies] +'@ +New-Item -ItemType Directory -Path (Join-Path $repo 'src') -Force | Out-Null +Write-Fixture (Join-Path $repo 'src/lib.rs') @' +//! A fixture crate for the container end-to-end test. + +/// Adds two numbers. +#[must_use] +pub const fn add(left: u64, right: u64) -> u64 { + left + right +} +'@ +Write-Fixture (Join-Path $repo 'rust-toolchain.toml') @' +[toolchain] +channel = "1.95" +'@ +Write-Fixture (Join-Path $repo 'Justfile') @' +set unstable + +# A repository-owned recipe, to prove that forwarded values arrive. +e2e-show-env: + @echo "E2E:$ANVIL_E2E_RUNTIME" +'@ + +Invoke-Native -Command 'git' -Arguments @('init', '-q') -WorkingDirectory $repo | Out-Null + +Write-Step 'generating the anvil tree (cargo anvil --no-backends)' +$generate = Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo +Write-Detail (($generate.StdOut -split "`r?`n" | Select-Object -Last 3) -join "`n") + +# ------------------------------------------------------- 1. what was emitted -- + +Write-Section '1. Generated artifacts' + +$dockerfile = Join-Path $repo '.anvil/container/Dockerfile' +$dockerignore = Join-Path $repo '.anvil/container/Dockerfile.dockerignore' +$containerJust = Join-Path $repo 'justfiles/anvil/container.just' +$hooks = Join-Path $repo '.anvil/container/hooks.ps1' + +Assert-That 'Dockerfile emitted' (Test-Path -LiteralPath $dockerfile) +Assert-That 'Dockerfile.dockerignore emitted' (Test-Path -LiteralPath $dockerignore) +Assert-That 'container.just emitted' (Test-Path -LiteralPath $containerJust) +Assert-That 'no hook emitted by default' (-not (Test-Path -LiteralPath $hooks)) +Assert-That 'no config file emitted' (-not (Test-Path -LiteralPath (Join-Path $repo 'anvil.toml'))) +Assert-That 'no runner seam emitted' (-not (Test-Path -LiteralPath (Join-Path $repo 'justfiles/anvil/runner.just'))) + +$containerDir = Get-ChildItem -LiteralPath (Join-Path $repo '.anvil/container') -File +Assert-Equal 'container directory holds exactly two files' 2 $containerDir.Count + +$justList = Invoke-Just -Repo $repo -Arguments @('--list') +Assert-That 'anvil-container is discoverable in just --list' ($justList.StdOut -match 'anvil-container') + +# ------------------------------------------------------------- 2. first run -- + +Write-Section '2. First run builds the image' + +$reference = Get-ImageReference -Repo $repo +Assert-That 'status reports an image reference' ($reference -like "$imagePrefix*") "got: '$reference'" +Assert-That 'image is absent before the first run' (-not (Test-ImagePresent $reference)) + +Write-Step "building and running (this takes several minutes on a cold cache)" +$firstRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') +Write-Detail (($firstRun.StdErr -split "`r?`n" | Select-Object -Last 4) -join "`n") + +Assert-Equal 'anvil-fmt succeeds inside the container' 0 $firstRun.ExitCode +Assert-That 'the run reported building the image' ($firstRun.StdErr -match 'building .*(inputs changed|first run)') +Assert-That 'image is present afterwards' (Test-ImagePresent $reference) + +# ------------------------------------------------------------ 3. second run -- + +Write-Section '3. Second run reuses the image' + +$secondRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') +Assert-Equal 'anvil-fmt succeeds again' 0 $secondRun.ExitCode +Assert-That 'nothing was rebuilt' (-not ($secondRun.StdErr -match 'building ')) $secondRun.StdErr +Assert-Equal 'the reference is unchanged' $reference (Get-ImageReference -Repo $repo) + +$status = Invoke-Just -Repo $repo -Arguments @('anvil-container-status') +Assert-That 'status reports present and current' ($status.StdOut -match 'present and current') $status.StdOut +Assert-That 'status reports the selected engine' ($status.StdOut -match "engine:\s+.*$Engine") $status.StdOut + +# ------------------------------------------------------ 4/5. hashed inputs --- + +Write-Section '4. A changed input selects a new tag' + +$toolchainPath = Join-Path $repo 'rust-toolchain.toml' +$originalToolchain = Get-Content -LiteralPath $toolchainPath -Raw +Write-Fixture $toolchainPath @' +[toolchain] +channel = "1.94" +'@ + +$bumped = Get-ImageReference -Repo $repo +Assert-That 'the reference changed with the toolchain' ($bumped -ne $reference) "before: $reference`nafter: $bumped" +Assert-That 'the new tag is not already present' (-not (Test-ImagePresent $bumped)) + +Write-Section '5. Reverting the input returns to the original tag' + +Write-Fixture $toolchainPath $originalToolchain +$reverted = Get-ImageReference -Repo $repo +Assert-Equal 'the original reference is restored' $reference $reverted +Assert-That 'the original image is still present' (Test-ImagePresent $reverted) + +# ------------------------------------------------- 6. editing the Dockerfile -- + +Write-Section '6. A repository can edit the Dockerfile' + +$dockerfileBody = Get-Content -LiteralPath $dockerfile -Raw +Write-Fixture $dockerfile ($dockerfileBody + "`n# a repository-owned edit`n") +$editedReference = Get-ImageReference -Repo $repo +Assert-That 'editing the Dockerfile selects a new tag' ($editedReference -ne $reference) + +Write-Step 're-running the generator over the edited file' +Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo -AllowFailure | Out-Null +$afterRegen = Get-Content -LiteralPath $dockerfile -Raw +Assert-That 'the edit survives regeneration' ($afterRegen -match 'a repository-owned edit') ` + 'anvil must preserve a user-modified owned file' + +Write-Fixture $dockerfile $dockerfileBody +Assert-Equal 'restoring the Dockerfile restores the tag' $reference (Get-ImageReference -Repo $repo) + +# ----------------------------------------------------------------- 7. hook --- + +Write-Section '7. The credential hook reaches build and run' + +# A user writes this file by hand; the public catalog does not emit one. +Write-Fixture $hooks @' +function Anvil-PreBuild { + @{ Secrets = @{ e2e_token = 'build-secret-value' } } +} + +function Anvil-PreRun { + @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } +} +'@ + +$hookReference = Get-ImageReference -Repo $repo +Assert-That 'adding a hook selects a new tag' ($hookReference -ne $reference) ` + "the hook file's content must be part of the image identity" + +# The default Dockerfile does not consume the secret, so prove the wiring by +# having the image read it. This is a fixture-side Dockerfile edit, which is a +# supported user action (proved in section 6). +$secretStanza = @' + +# --- e2e: prove the build secret arrives and never lands in a layer --- +RUN --mount=type=secret,id=e2e_token,required=true \ + test -s /run/secrets/e2e_token \ + && echo "e2e: secret length $(wc -c < /run/secrets/e2e_token)" +'@ +Write-Fixture $dockerfile ($dockerfileBody + $secretStanza) + +Write-Step 'rebuilding with the hook active' +$hookRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +Assert-Equal 'the run with a hook succeeds' 0 $hookRun.ExitCode +Assert-That 'the hook announced itself at build time' ($hookRun.StdErr -match 'Anvil-PreBuild') +Assert-That 'the build secret was declared' ($hookRun.StdErr -match 'build secrets: e2e_token') +Assert-That 'the hook announced itself at run time' ($hookRun.StdErr -match 'Anvil-PreRun') +Assert-That 'forwarded names are reported' ($hookRun.StdErr -match 'forwarding env: ANVIL_E2E_RUNTIME') + +$secretReference = Get-ImageReference -Repo $repo +$layers = Invoke-Engine -Arguments @('history', '--no-trunc', $secretReference) -AllowFailure +Assert-That 'the secret value is absent from every image layer' ` + (-not ($layers.StdOut -match 'build-secret-value')) 'a build secret must never reach a layer' + +Write-Step 'checking that the forwarded value arrives inside the container' +$showEnv = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-env') -AllowFailure +Assert-That 'the run-time value reaches a recipe in the container' ` + ($showEnv.StdOut -match 'E2E:run-value') "stdout: $($showEnv.StdOut)`nstderr: $($showEnv.StdErr)" + +# ------------------------------------------------------ 8. hook fails closed -- + +Write-Section '8. An empty hook value fails closed' + +Write-Fixture $hooks @' +function Anvil-PreBuild { + @{ Secrets = @{ e2e_token = '' } } +} +'@ + +$emptyHook = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +Assert-That 'an empty secret aborts the run' ($emptyHook.ExitCode -ne 0) ` + 'BuildKit would mount an empty secret and exit 0, tagging a degraded image with a valid hash' +Assert-That 'the failure names the offending secret' ($emptyHook.StdErr -match "empty value for secret 'e2e_token'") ` + $emptyHook.StdErr + +# ------------------------------------------- 9. hook output is not the tag --- + +Write-Section '9. Hook output does not change the tag' + +Write-Fixture $hooks @' +function Anvil-PreBuild { + @{ Secrets = @{ e2e_token = 'build-secret-value' } } +} + +function Anvil-PreRun { + @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } +} +'@ +Assert-Equal 'restoring the hook restores the tag' $secretReference (Get-ImageReference -Repo $repo) + +# Same file length, different minted value: the tag must not move. +Write-Fixture $hooks @' +function Anvil-PreBuild { + @{ Secrets = @{ e2e_token = 'BUILD-SECRET-VALUE' } } +} + +function Anvil-PreRun { + @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } +} +'@ +$mutatedValue = Get-ImageReference -Repo $repo +Assert-That 'a changed hook body still changes the tag' ($mutatedValue -ne $secretReference) ` + 'the hook file is a hashed input' + +# ------------------------------------------------------ 10. no nested runs --- + +Write-Section '10. Recipes run natively inside the image' + +$nested = Invoke-Engine -Arguments @( + 'run', '--rm', '-e', 'ANVIL_IN_CONTAINER=1', '-v', "$(ConvertTo-EnginePath $repo):/workspace", '-w', '/workspace', + $secretReference, 'just', 'anvil-container', 'anvil-fmt' +) -AllowFailure +Assert-Equal 'anvil-container passes through inside the image' 0 $nested.ExitCode +Assert-That 'no engine was invoked from inside the container' ` + (-not ($nested.StdErr -match 'building |Cannot connect to the Docker daemon')) $nested.StdErr + +# --------------------------------------------------------------- teardown ---- + +Write-Section 'Teardown' + +if ($KeepArtifacts) { + Write-Step "keeping $repo and images matching $imagePrefix*" +} else { + Write-Step 'removing cache volumes via anvil-container-down' + Invoke-Just -Repo $repo -Arguments @('anvil-container-down') -AllowFailure | Out-Null + Remove-AnvilImages -Prefix $imagePrefix + Remove-Item -LiteralPath $workRoot -Recurse -Force -ErrorAction SilentlyContinue + Write-Step 'removed the fixture repository' +} + +$elapsed = (Get-Date) - $script:Started +Write-Host '' +Write-Host ('-' * 78) +$summary = "{0}/{1} checks passed in {2:mm\:ss}" -f $script:Passed, ($script:Passed + $script:Failed), $elapsed +if ($script:Failed -eq 0) { + Write-Host "PASS $summary" -ForegroundColor Green + exit 0 +} +Write-Host "FAIL $summary ($($script:Failed) failed)" -ForegroundColor Red +exit 1 From 2ad32cd39ad5a6c0efbd8d45cbabf5761afd4bbc Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 09:55:26 +0200 Subject: [PATCH 03/81] docs(anvil): give containerization one owner, and split host setup per engine The customization story was stated three times -- in containers.md, in extensibility.md 6.1, and in the crate docs that generate the README. That happened because main already carried a container subsection in extensibility.md for the old backend, and it was rewritten in place rather than reconsidered. Three copies of the same contract drift, and the one that drifts is whichever the reader happens to find. containers.md now owns it: the levers, who each one is for, and the coupling between the Dockerfile and its ignore file, including what a fork inherits rather than replaces. extensibility.md 6.1 keeps only what is genuinely a rule of the extensibility system -- justfiles/ holds .just recipes and nothing else, enforced by CatalogBuilder::build -- and explains that containerized execution is why. The customization detail is a pointer. The crate docs keep a sentence, since a README reader needs "you can change the image", not the fork API. Host setup gains a subsection per engine. Docker and podman need different things on Windows, and the old prose buried that: Docker Desktop and podman both ship a Windows CLI, while Docker-in-WSL does not and is reached through the WSL fallback. Each path is now written out, and podman's status is stated plainly -- wired up, expected to work, verified by nothing, with the BuildKit assumptions in the credential path called out as the specific risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/cargo-anvil/README.md | 11 ++- crates/cargo-anvil/docs/design/containers.md | 99 ++++++++++++++++--- .../cargo-anvil/docs/design/extensibility.md | 54 ++++------ crates/cargo-anvil/src/lib.rs | 7 +- 4 files changed, 111 insertions(+), 60 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index e266499e..1f2a1dee 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -168,9 +168,10 @@ container isolation. Only run one from a repository or catalog you trust. #### Customizing the image `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place -and anvil’s drift handling preserves it. A downstream catalog that targets -a different base OS or toolchain source replaces the artifact instead — see -[`artifacts::container::dockerfile`][__link1]. +for extra packages, and anvil’s drift handling preserves the change. A +downstream catalog that needs a different base OS or toolchain source for +every repository it manages replaces the artifact instead — see +[`artifacts::container`][__link1] and the design doc. ### Checks and tiers @@ -390,9 +391,9 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQblC_Wpqk8Z8wbXSVrNQ_nt0AbkrIITLnNqNIbWqbEpmD_DbhhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbIHyE2bP5dGcbfv5a5XBW8mUbpBv2VBT_2G0bCUddUAFHBklhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta - [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container::dockerfile + [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts [__link2]: https://crates.io/crates/cargo-spellcheck [__link3]: https://crates.io/crates/cargo-coverage-gate diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index a88ba62b..4a1d5e39 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -17,6 +17,8 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [4. Image identity](#4-image-identity) - [5. Credentials — the hook](#5-credentials--the-hook) - [6. Host setup](#6-host-setup) + - [6.1 Docker](#61-docker) + - [6.2 Podman](#62-podman) - [7. Customizing the image](#7-customizing-the-image) - [8. Limits](#8-limits) @@ -140,11 +142,37 @@ macros. Keep the set narrow and the token short-lived. ## 6. Host setup -The engine must be callable from the shell that runs `just`. Docker is the supported path; podman is best-effort and -currently untested — it uses buildah rather than BuildKit, so the secret semantics above are unverified there. +anvil installs nothing. It calls the engine you selected and lets that engine's own diagnostics +surface when something is wrong — the one exception is a missing binary, which is reported with +the variable to set and a pointer here. -anvil installs nothing. On Windows, either use an engine that ships a Windows CLI (Docker Desktop, podman), or run -Docker inside WSL and point a Windows `docker` CLI at it: +The engine must be **callable from the shell that runs `just`**. On Windows there is one +exception, and it is automatic: if the engine is not on `PATH`, anvil retries it inside the +default WSL distribution and translates the repository path with `wslpath`. That exists because +Docker installed in WSL leaves no Windows CLI behind, which would otherwise make the setup this +page recommends unusable. + +| | Docker | Podman | +| --- | --- | --- | +| Status | **supported** — what the e2e validates and what CI uses | best-effort, **untested** | +| Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | +| Builder | BuildKit | buildah | + +Podman is wired up and expected to work, but nothing has verified it. The credential path in +§5 is the specific risk: buildah is not BuildKit, so `# syntax=docker/dockerfile:1`, +`--mount=type=secret,required=true`, and the fail-closed-on-empty-secret behaviour are all +unconfirmed there. Rootless uid mapping differs too (`--userns keep-id` rather than +`--user $(id -u)`). Treat a podman failure as "not yet supported", not as a regression. + +### 6.1 Docker + +**Linux.** Install Docker Engine from your distribution or `get.docker.com`, add yourself to the +`docker` group, and you are done. + +**Windows — Docker Desktop.** Nothing to configure. `docker` is on `PATH`, so anvil calls it +directly. + +**Windows — Docker Engine in WSL** (no Docker Desktop, no licence question): ```powershell wsl --install -d Ubuntu-24.04 @@ -155,22 +183,61 @@ wsl --shutdown wsl -d Ubuntu-24.04 -- docker version # verify ``` -With that arrangement the daemon is Linux-side, so `DOCKER_HOST` must reach its socket and the repository must be -bind-mountable at a path the daemon understands. +That is the whole setup: no Windows `docker` CLI is needed, because anvil reaches the engine +through `wsl.exe` when it finds none on `PATH`. `just` and `pwsh` stay on Windows — the +distribution needs only Docker. -For podman, `podman machine init` provisions and manages its own WSL2 virtual machine; set -`ANVIL_CONTAINER_ENGINE=podman`. +If you *do* install a Windows `docker` CLI and point `DOCKER_HOST` at the WSL socket, anvil uses +it directly and the WSL fallback never engages. In that arrangement the daemon is Linux-side, so +the repository must be bind-mountable at a path that daemon understands. -## 7. Customizing the image +### 6.2 Podman -| You want | Do this | -| --- | --- | -| Extra packages in one repository | Edit `.anvil/container/Dockerfile`; the drift flow preserves it | -| A different base OS or toolchain source for a whole organization | `replace_artifact(artifacts::container::dockerfile().with_body(…))` in a downstream catalog | -| Credentials | Add `hooks.ps1`, by hand or via `with_artifact(artifacts::container::hooks(…))` | +**Linux.** Install podman and set `ANVIL_CONTAINER_ENGINE=podman`. + +**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and +`podman.exe` is on `PATH`, so anvil calls it directly and the WSL fallback never engages. + +```powershell +winget install RedHat.Podman-Desktop # or the podman CLI alone +podman machine init +podman machine start +$env:ANVIL_CONTAINER_ENGINE = 'podman' +``` + +## 7. Customizing the image -A catalog that replaces the Dockerfile with one that copies more of the tree must replace -`artifacts::container::dockerignore()` too, since the build context is scoped by that file. +Two audiences, three levers. A **repository** owns its own copy of the emitted files; a +**downstream catalog** (an anvil fork — see [extensibility.md](./extensibility.md)) changes what +every repository it manages receives. + +| You want | Do this | Who | +| --- | --- | --- | +| Extra packages, one repository | Edit `.anvil/container/Dockerfile` in place; the drift flow preserves it | repository | +| A different base OS or toolchain source, everywhere | `replace_artifact(artifacts::container::dockerfile().with_body(…))` | catalog | +| Credentials | Write `.anvil/container/hooks.ps1`, or ship one with `with_artifact(artifacts::container::hooks(…))` | either | +| No container support at all | `without_artifact` each of the three artifacts | catalog | + +Editing the Dockerfile in a single repository is supported but noisy: anvil keeps proposing its +own version against a file it can see has diverged. A fork that wants the change everywhere +should replace the artifact instead. + +The hook is loaded by path, not by provenance: `container.just` sources +`.anvil/container/hooks.ps1` whenever it exists, so a hand-written file and one shipped by a +catalog behave identically. That is deliberate — it lets a repository try a credential flow +before anyone commits to forking the catalog for it. + +**Coupled artifacts.** The Dockerfile and its ignore file move together. A replacement that +`COPY`s more of the tree must also replace `artifacts::container::dockerignore()`, or the extra +files are excluded from the build context and the build fails on a missing path. The recipe +tree needs no such care: the image identity hashes every `*.just` under `justfiles/anvil/` +recursively, so a catalog that adds a recipe directory gets it hashed automatically — but the +same widening rule applies before it can be copied. + +**What a fork does *not* touch.** The recipe, the identity hash, the cache volumes, the mounts +and the uid mapping are inherited unchanged. Substrate is the worked example: a different base +OS and a different toolchain source, expressed as one Dockerfile replacement plus one hook, and +nothing else. Keep `ARG BASE_IMAGE` digest-pinned. A floating tag can change underneath a tag that claims to name fixed content, which would make every cached image a potential lie. diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index 54c991b5..d7ebd4a9 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -464,42 +464,24 @@ the relevant `OwnedFile` (e.g. `checks.just`) wholesale rather than editing indi This is a modest, low-risk refactor: it data-drives the artifact list (§4) without disturbing the engine internals or the template format. -### 6.1 Containerized execution - -The base catalog emits three files: the `anvil-container` recipe at -`justfiles/anvil/container.just`, and the image definition at -`.anvil/container/Dockerfile` with its `Dockerfile.dockerignore`. Native -`just anvil-*` execution is unaffected — the container is reached only through -the explicit recipe. - -A downstream catalog customizes exactly two things: - -- `replace_artifact(artifacts::container::dockerfile().with_body(...))` to build - on a different base OS or install the toolchain from a different source. The - recipe, the image-identity hash, the cache volumes and the mounts are - inherited unchanged. -- `with_artifact(artifacts::container::hooks(...))` to supply credentials. The - recipe loads `.anvil/container/hooks.ps1` whenever it is present, so the - contract is ownership-neutral: a regular repository can commit the same path - directly, without a derived catalog, and behavior is identical. - -A catalog that replaces the Dockerfile with one that copies more of the tree -must also replace `artifacts::container::dockerignore()`, since the build -context is scoped by that file. The image identity hashes every `*.just` under -`justfiles/anvil/` recursively, so a catalog that adds a recipe directory gets -it hashed automatically, but must widen the ignore file for it to be copied. - -`justfiles/` holds `.just` recipes only. -[`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects any other owned -file placed there; non-recipe assets belong in a tool-owned directory such as -`.anvil/`. - -The hook is trusted host code: the recipe dot-sources it into its own process -before the build and before the run, so it executes with the invoking -developer's permissions and outside the container sandbox. See -[containers.md §5](./containers.md#5-credentials--the-hook) for the full -interface and trust boundary. - +### 6.1 Placement: `justfiles/` holds recipes only + +One placement rule is enforced rather than left to discovery, because getting it +wrong fails in a confusing place. `justfiles/anvil/` may contain `.just` recipes +and nothing else: [`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects +any other owned file under that prefix, so a derived catalog fails loudly at +construction instead of shipping a file whose absence is only noticed later. + +The reason is containerized execution. The image identity hashes every `*.just` +under `justfiles/anvil/` recursively, and the build context admits that tree, +so a non-recipe file placed there would be silently dropped from the image while +still appearing to be managed. Non-recipe assets belong in a tool-owned +directory of their own, such as `.anvil/`. + +Containerized execution is itself an ordinary artifact group, customized with +the same `replace_artifact` / `with_artifact` / `without_artifact` levers as +anything else; the artifacts it exposes and the contract each one carries are +specified in [containers.md](./containers.md#7-customizing-the-image). The public engine contains no environment-specific image, registry, cloud, or credential-provider details. diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 21429f55..826e46a1 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -167,9 +167,10 @@ //! ### Customizing the image //! //! `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place -//! and anvil's drift handling preserves it. A downstream catalog that targets -//! a different base OS or toolchain source replaces the artifact instead — see -//! [`artifacts::container::dockerfile`]. +//! for extra packages, and anvil's drift handling preserves the change. A +//! downstream catalog that needs a different base OS or toolchain source for +//! every repository it manages replaces the artifact instead — see +//! [`artifacts::container`] and the design doc. //! //! ## Checks and tiers //! From d349dd47856b7f2e5799db6f17a86cc66f005577 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 11:30:48 +0200 Subject: [PATCH 04/81] fix(anvil): unbreak CI, and report podman's build-secret limitation honestly Three fixes found by CI and by running the e2e against podman. dependency_recipe_sources became dead when the old container tests were removed. CI builds with -D warnings, so dead_code is a hard error there and a silent warning locally; the function had no remaining caller and is deleted. "natively" was missing from the dictionary, which is the whole of the spell-check failure. cargo-spellcheck cannot run on this machine (it needs libclang), so the word was found by CI rather than locally. Running the e2e with -Engine podman surfaced a genuine engine defect: podman 6.0.2 on Windows cannot mount a build secret at all. It composes its own temp path from the build context after translating it into its machine's view, then joins it with a Windows separator, and fails before the build starts. A four-line Dockerfile reproduces it with no anvil involved, and `src=` fails identically to `env=`, so there is nothing to work around on our side. The secret plumbing therefore stays as it was -- by environment variable name, so the value never touches disk. The build failure path gains one hint, conditioned on "secrets were passed and the build failed" rather than on the engine, because that error names neither the secret nor the engine. The e2e now skips the three hook sections on podman-for-Windows with the reason stated, rather than reporting a known engine limitation as failure every run, and its image cleanup matches anywhere in a reference: podman reports images fully qualified (localhost/anvil-...) where docker does not, so the old prefix match left images behind and the next run saw a warm cache it expected to be cold. Validation: docker 43/43, podman 31/31 (3 hook sections skipped), cargo test -p cargo-anvil --all-features under -D warnings, --dry-run clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 4 +- .spelling | 1 + crates/cargo-anvil/docs/design/containers.md | 40 ++++++++++++++----- .../src/anvil/artifacts/justfile.rs | 5 --- .../templates/justfiles/anvil/container.just | 14 ++++++- .../snapshots/snapshots__ado_backend.snap | 14 ++++++- .../snapshots/snapshots__github_backend.snap | 14 ++++++- .../snapshots/snapshots__local_only.snap | 14 ++++++- justfiles/anvil/container.just | 14 ++++++- scripts/test-anvil-container.ps1 | 23 +++++++++-- 10 files changed, 118 insertions(+), 25 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 4cdef881..29c42557 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:6d1e7ee2e36339ba85145ec327802ef5126c3e2fd016339c5d528e9a075c7170" +catalog_checksum = "sha256:c75aaeeb73b8f323152ff9505e8106e07ce75ba34a14ed72d368eb0d3dce3aa5" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:16a03104b3de2612ee5553969ad6544de5cfd94beffb25d3d551547e94ff78ce" +checksum = "sha256:d9ca3794252781912c18f9d2bd0159c578c348b0acbab24012869e329963ee73" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.spelling b/.spelling index 34996003..4b1db43b 100644 --- a/.spelling +++ b/.spelling @@ -469,5 +469,6 @@ dockerignore Podman podman toolset +natively ARM64 WSL diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 4a1d5e39..d82e8144 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -116,10 +116,16 @@ function Anvil-PreRun { } ``` -Both functions are optional. `Secrets` become BuildKit `--secret` mounts at build time; `Env` becomes `-e NAME` at -run time. In both cases the value is handed over **by environment variable name**, so it never appears in the host's -process command line — where endpoint telemetry records and retains it far longer than a short-lived token is meant -to live — and BuildKit keeps build secrets out of every image layer. +Both functions are optional. `Secrets` become `--secret id=…,src=…` mounts at build time; `Env` becomes +`-e NAME` at run time. Neither value ever appears as a command-line argument: a build secret is written to a +private temp file that is removed as soon as the build ends, and a run-time value is forwarded **by name**, so the +engine copies it from the environment it already inherits. Command lines are recorded by endpoint telemetry and +retained far longer than a short-lived token is meant to live. The engine also keeps build secrets out of every +image layer. + +`src=` is used rather than `env=` because it is the form both engines implement. Podman for Windows translates the +build context into its machine's view and then composes the `env=` temp path with a Windows separator, producing a +path it cannot open; a file anvil creates itself has no such step. The corresponding `RUN` should declare the mount as required, which closes the same hole from the Dockerfile's side: @@ -154,15 +160,29 @@ page recommends unusable. | | Docker | Podman | | --- | --- | --- | -| Status | **supported** — what the e2e validates and what CI uses | best-effort, **untested** | +| Status | **supported** — what the e2e validates and what CI uses | works, with one exception below | | Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | | Builder | BuildKit | buildah | -Podman is wired up and expected to work, but nothing has verified it. The credential path in -§5 is the specific risk: buildah is not BuildKit, so `# syntax=docker/dockerfile:1`, -`--mount=type=secret,required=true`, and the fail-closed-on-empty-secret behaviour are all -unconfirmed there. Rootless uid mapping differs too (`--userns keep-id` rather than -`--user $(id -u)`). Treat a podman failure as "not yet supported", not as a regression. +Podman has been run through the same end-to-end test as docker: it builds the image, computes and +reuses the content-addressed tag, and runs recipes. Rootless uid mapping differs (`--userns +keep-id` rather than `--user $(id -u)`), which anvil does not currently set for podman. + +**Build secrets do not work on podman for Windows.** Podman composes its own temp path from the +build context after translating it into its machine's view, and joins it with a Windows +separator, so any `--secret` fails before the build starts: + +```text +Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 +``` + +This is not something anvil can work around — `src=` and `env=` fail identically, and a +four-line Dockerfile reproduces it with no anvil involved. It affects only a repository that +supplies `Anvil-PreBuild` (§5); the public catalog ships no hook, so ordinary use is unaffected. +Use docker if you need build-time credentials on Windows. + +Docker also carries more mileage: it is what CI uses and what the e2e runs by default. Prefer it +if you have no reason not to. ### 6.1 Docker diff --git a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs index db82ec81..5a2e7a74 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs @@ -133,11 +133,6 @@ const TIERS_JUST: &str = include_str!("../../../templates/justfiles/anvil/tiers. /// Repo-root-relative path of the tier aggregator file. const TIERS_JUST_PATH: &str = "justfiles/anvil/tiers.just"; -#[cfg(test)] -pub(crate) fn dependency_recipe_sources() -> impl Iterator { - std::iter::once(TIERS_JUST).chain(GROUP_FILES.iter().map(|(_, body)| *body)) -} - /// Embedded body of the `anvil-imports` region in the user's Justfile. pub(crate) const JUSTFILE_IMPORTS_BODY: &str = include_str!("../../../templates/regions/justfile-imports.just"); diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index e5da0b7e..6b72f747 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -234,7 +234,19 @@ _anvil-container-image: $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if ($LASTEXITCODE -ne 0) { + # Build secrets are the part of this path engines implement least + # consistently -- podman on Windows cannot mount one at all, and + # fails with a path error that names neither the secret nor the + # engine. Say so once, rather than leaving that to be rediscovered. + if ($secretArgs.Count -gt 0) { + [Console]::Error.WriteLine( + "anvil: the build passed $($secretArgs.Count) secret(s) from $hookRel. " + + "If the failure above is about a temp file or a path, the engine may not support " + + "build secrets on this host; see docs/design/containers.md.") + } + exit $LASTEXITCODE + } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 7a966b5c..4396e1c8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3709,7 +3709,19 @@ _anvil-container-image: $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if ($LASTEXITCODE -ne 0) { + # Build secrets are the part of this path engines implement least + # consistently -- podman on Windows cannot mount one at all, and + # fails with a path error that names neither the secret nor the + # engine. Say so once, rather than leaving that to be rediscovered. + if ($secretArgs.Count -gt 0) { + [Console]::Error.WriteLine( + "anvil: the build passed $($secretArgs.Count) secret(s) from $hookRel. " + + "If the failure above is about a temp file or a path, the engine may not support " + + "build secrets on this host; see docs/design/containers.md.") + } + exit $LASTEXITCODE + } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index aab76060..4b539ccd 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3630,7 +3630,19 @@ _anvil-container-image: $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if ($LASTEXITCODE -ne 0) { + # Build secrets are the part of this path engines implement least + # consistently -- podman on Windows cannot mount one at all, and + # fails with a path error that names neither the secret nor the + # engine. Say so once, rather than leaving that to be rediscovered. + if ($secretArgs.Count -gt 0) { + [Console]::Error.WriteLine( + "anvil: the build passed $($secretArgs.Count) secret(s) from $hookRel. " + + "If the failure above is about a temp file or a path, the engine may not support " + + "build secrets on this host; see docs/design/containers.md.") + } + exit $LASTEXITCODE + } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index df706252..be1e8923 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2449,7 +2449,19 @@ _anvil-container-image: $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if ($LASTEXITCODE -ne 0) { + # Build secrets are the part of this path engines implement least + # consistently -- podman on Windows cannot mount one at all, and + # fails with a path error that names neither the secret nor the + # engine. Say so once, rather than leaving that to be rediscovered. + if ($secretArgs.Count -gt 0) { + [Console]::Error.WriteLine( + "anvil: the build passed $($secretArgs.Count) secret(s) from $hookRel. " + + "If the failure above is about a temp file or a path, the engine may not support " + + "build secrets on this host; see docs/design/containers.md.") + } + exit $LASTEXITCODE + } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index e5da0b7e..6b72f747 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -234,7 +234,19 @@ _anvil-container-image: $env:WSLENV = (@($env:WSLENV) + $bridged | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @buildCmd | ForEach-Object { [Console]::Error.WriteLine($_) } - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + if ($LASTEXITCODE -ne 0) { + # Build secrets are the part of this path engines implement least + # consistently -- podman on Windows cannot mount one at all, and + # fails with a path error that names neither the secret nor the + # engine. Say so once, rather than leaving that to be rediscovered. + if ($secretArgs.Count -gt 0) { + [Console]::Error.WriteLine( + "anvil: the build passed $($secretArgs.Count) secret(s) from $hookRel. " + + "If the failure above is about a temp file or a path, the engine may not support " + + "build secrets on this host; see docs/design/containers.md.") + } + exit $LASTEXITCODE + } } finally { foreach ($name in $secretEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } } diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index ff3361d9..e022a9d3 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -209,13 +209,15 @@ function Test-ImagePresent([string]$Reference) { function Remove-AnvilImages([string]$Prefix) { $images = Invoke-Engine -Arguments @('images', '--format', '{{.Repository}}:{{.Tag}}') -AllowFailure - $matching = ($images.StdOut -split "`r?`n") | Where-Object { $_ -like "$Prefix*" } + # Podman reports images fully qualified (`localhost/anvil-…`), docker does + # not, so match anywhere in the reference rather than at the start. + $matching = ($images.StdOut -split "`r?`n") | Where-Object { $_ -like "*$Prefix*" } foreach ($image in $matching) { Write-Step "removing image $image" Invoke-Engine -Arguments @('rmi', '-f', $image) -AllowFailure | Out-Null } $volumes = Invoke-Engine -Arguments @('volume', 'ls', '--format', '{{.Name}}') -AllowFailure - $matchingVolumes = ($volumes.StdOut -split "`r?`n") | Where-Object { $_ -like "$Prefix*" } + $matchingVolumes = ($volumes.StdOut -split "`r?`n") | Where-Object { $_ -like "*$Prefix*" } foreach ($volume in $matchingVolumes) { Write-Step "removing volume $volume" Invoke-Engine -Arguments @('volume', 'rm', '-f', $volume) -AllowFailure | Out-Null @@ -419,6 +421,16 @@ Assert-Equal 'restoring the Dockerfile restores the tag' $reference (Get-ImageRe Write-Section '7. The credential hook reaches build and run' +# podman on Windows cannot mount a build secret at all: it composes its own temp +# path from the already-translated build context and joins it with a Windows +# separator. That is an engine defect with no client-side workaround, documented +# in docs/design/containers.md. Reporting it as a failure every run would train +# the reader to ignore red, so it is called out and skipped. +$buildSecretsSupported = -not ($Engine -eq 'podman' -and $IsWindows) +if (-not $buildSecretsSupported) { + Write-Step 'skipping the hook sections: podman on Windows cannot mount build secrets' + Write-Step 'everything above is engine-agnostic and has already run' +} else { # A user writes this file by hand; the public catalog does not emit one. Write-Fixture $hooks @' function Anvil-PreBuild { @@ -509,13 +521,18 @@ $mutatedValue = Get-ImageReference -Repo $repo Assert-That 'a changed hook body still changes the tag' ($mutatedValue -ne $secretReference) ` 'the hook file is a hashed input' +} # end of the build-secret sections (7-9) + # ------------------------------------------------------ 10. no nested runs --- Write-Section '10. Recipes run natively inside the image' +# Sections 7-9 build the image that carries the secret stanza; without them the +# current reference is the plain one. +$nestedReference = if ($buildSecretsSupported) { $secretReference } else { Get-ImageReference -Repo $repo } $nested = Invoke-Engine -Arguments @( 'run', '--rm', '-e', 'ANVIL_IN_CONTAINER=1', '-v', "$(ConvertTo-EnginePath $repo):/workspace", '-w', '/workspace', - $secretReference, 'just', 'anvil-container', 'anvil-fmt' + $nestedReference, 'just', 'anvil-container', 'anvil-fmt' ) -AllowFailure Assert-Equal 'anvil-container passes through inside the image' 0 $nested.ExitCode Assert-That 'no engine was invoked from inside the container' ` From cb72c66aa23a9d66ee618d8cf5296a07b9e9f66e Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 11:34:55 +0200 Subject: [PATCH 05/81] test(anvil): actually prove that a minted credential cannot change the tag Section 9 of the e2e was headed "hook output does not change the tag" and did not test it. One assertion rewrote the hook to byte-identical contents, which only shows the hash is deterministic; the other changed the hook body and asserted the tag *did* move, directly contradicting the comment above it. Both were really re-testing that file content is hashed, which section 7 already covers, so the invariant the design leans on -- a credential must never influence a tag -- was unverified by a test that claimed to cover it. Proving it needs the hook file to be byte-identical while what it returns differs, so the fixture hook now reads its value from the environment and the reference is resolved twice with two different values. That matters beyond tidiness: if a minted value reached the hash, two developers holding different tokens would compute different images from identical inputs, and every token rotation would force a rebuild. The file-content half is kept as its own assertion, since both halves are load-bearing and they pull in opposite directions. Found by an independent review pass over the PR. docker 44/44. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/test-anvil-container.ps1 | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index e022a9d3..d076bf57 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -26,7 +26,7 @@ 6. Editing the Dockerfile is preserved by a re-run of the generator. 7. A credential hook reaches both the build and the run. 8. A hook returning an empty value fails closed. - 9. A hook's output does not change the tag; its file content does. + 9. A hook's returned value does not change the tag; its file content does. 10. The recipes run natively inside the image (no nesting). .PARAMETER Engine @@ -194,9 +194,10 @@ function Invoke-Just { Invoke-Native -Command 'just' -Arguments $Arguments -WorkingDirectory $Repo -Environment $env -AllowFailure:$AllowFailure } -function Get-ImageReference([string]$Repo) { +function Get-ImageReference { + param([Parameter(Mandatory)][string]$Repo, [hashtable]$Environment = @{}) # anvil-container-status reports the reference without building it. - $status = Invoke-Just -Repo $Repo -Arguments @('anvil-container-status') -AllowFailure + $status = Invoke-Just -Repo $Repo -Arguments @('anvil-container-status') -Environment $Environment -AllowFailure $line = ($status.StdOut -split "`r?`n") | Where-Object { $_ -match '^\s*image:\s*(\S+)' } | Select-Object -First 1 if ($line -match '^\s*image:\s*(\S+)') { return $Matches[1] } '' @@ -507,19 +508,32 @@ function Anvil-PreRun { '@ Assert-Equal 'restoring the hook restores the tag' $secretReference (Get-ImageReference -Repo $repo) -# Same file length, different minted value: the tag must not move. +# The invariant that matters: a *minted* credential must never influence the +# tag, or two developers holding different tokens would compute different +# images from identical inputs -- and a rotated token would force a rebuild. +# Proving it needs the hook file to be byte-identical while what it returns +# differs, so the value is read from the environment rather than written into +# the file. Changing the file instead would only re-prove that file content is +# hashed, which section 7 already covers. Write-Fixture $hooks @' function Anvil-PreBuild { - @{ Secrets = @{ e2e_token = 'BUILD-SECRET-VALUE' } } + @{ Secrets = @{ e2e_token = $env:ANVIL_E2E_MINT } } } function Anvil-PreRun { @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } } '@ -$mutatedValue = Get-ImageReference -Repo $repo -Assert-That 'a changed hook body still changes the tag' ($mutatedValue -ne $secretReference) ` - 'the hook file is a hashed input' +$mintedA = Get-ImageReference -Repo $repo -Environment @{ ANVIL_E2E_MINT = 'first-minted-value' } +$mintedB = Get-ImageReference -Repo $repo -Environment @{ ANVIL_E2E_MINT = 'a-completely-different-second-value' } +Assert-That 'the tag is stable across two different minted values' ` + ($mintedA -and $mintedA -eq $mintedB) "first: $mintedA`nsecond: $mintedB" + +# ...while the file that produces those values is itself hashed, so a changed +# hook still renames the image. +$hookBodyChanged = $mintedA -ne $secretReference +Assert-That 'a changed hook body still changes the tag' $hookBodyChanged ` + "the hook file is a hashed input; before: $secretReference, after: $mintedA" } # end of the build-secret sections (7-9) From db364dc49977061fefe5e980930ee27edccb3d97 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 11:38:36 +0200 Subject: [PATCH 06/81] fix(anvil): satisfy cloned-ref-to-slice-refs in the container tests clippy::cloned_ref_to_slice_refs fires on \paths(&[replaced.clone()])\, and the workspace denies warnings. The clone was pointless anyway -- the slice only needs to borrow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/cargo-anvil/src/anvil/artifacts/container.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 0fefa700..7492b7b0 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -211,7 +211,7 @@ mod tests { #[test] fn dockerfile_body_can_be_replaced_by_a_fork() { let replaced = dockerfile().with_body("FROM example.invalid/base\n"); - assert_eq!(paths(&[replaced.clone()]), [DOCKERFILE_PATH]); + assert_eq!(paths(std::slice::from_ref(&replaced)), [DOCKERFILE_PATH]); assert_eq!(replaced.body(), "FROM example.invalid/base\n"); } } From baaf7eb2511262ab535a321d40c96d1a85581ead Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 12:36:03 +0200 Subject: [PATCH 07/81] fix(anvil): escape interpolations, preserve recipe arguments, and close review gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent review pass over the PR found two functional regressions against behaviour the deleted runner.just explicitly had, plus a set of comments and assertions that claimed more than the code delivered. Every `just` value pasted into a PowerShell literal is escaped again. The removed runner.just wrapped each interpolation in replace(…, "'", "''") and the new driver had none, so a repository path containing an apostrophe broke every anvil-container recipe, and `target` let the remainder run as host PowerShell -- outside any container, and before the engine check. The image name and workdir stay unescaped because the first is regex-sanitized at definition and the second is a literal; a unit test now asserts nothing else slips through, since this convention has been lost once already. `*target` is split back into argv. Joining its parts into one argument made `anvil-container anvil-setup binstall` look for a recipe named "anvil-setup binstall", and around fifty generated recipes take a parameter -- including the one the image's own Dockerfile runs. ANVIL_CONTAINER_NO_REBUILD is checked outside the NO_CACHE guard. Nested inside it, a developer with NO_CACHE exported got a from-scratch build out of `anvil-container-status`, which is precisely what NO_REBUILD exists to prevent. The e2e's "no secret in any image layer" assertion could not fail: `history` reports the command that created each layer, not its contents, and the value is never a build argument. It now greps the image filesystem, where a written secret would actually land. Documentation corrected where it outran the code: containers.md §5 still described the src= transport that was reverted; the Dockerfile claimed the recipe refuses a floating base image, which it never checks; the COPY comment overstated what is copied; the cache-volume comment claimed collision-freedom the basename cannot provide; extensibility.md kept a conclusion whose mechanism had changed underneath it; and the crate docs omitted the WSL-only engine path this feature exists to support. Podman's second limitation -- it reads only a context-root ignore file, so the whole worktree is streamed -- is documented beside the build-secret one. docker 46/46; cargo test and clippy --all-targets --all-features clean under -D warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 6 +- .anvil/container/Dockerfile | 15 ++-- crates/cargo-anvil/README.md | 7 +- crates/cargo-anvil/docs/design/containers.md | 25 ++++--- .../cargo-anvil/docs/design/extensibility.md | 9 +-- .../src/anvil/artifacts/container.rs | 58 ++++++++++++++++ crates/cargo-anvil/src/lib.rs | 5 +- .../templates/container/Dockerfile | 15 ++-- .../templates/justfiles/anvil/container.just | 53 +++++++++------ .../snapshots/snapshots__ado_backend.snap | 68 +++++++++++-------- .../snapshots/snapshots__github_backend.snap | 68 +++++++++++-------- .../snapshots/snapshots__local_only.snap | 68 +++++++++++-------- justfiles/anvil/container.just | 53 +++++++++------ scripts/test-anvil-container.ps1 | 20 +++++- 14 files changed, 312 insertions(+), 158 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 29c42557..817ab32a 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:c75aaeeb73b8f323152ff9505e8106e07ce75ba34a14ed72d368eb0d3dce3aa5" +catalog_checksum = "sha256:970f6a55ed74190d1ef848fc016c529027fd016037e36f09a6027c30d2f70c05" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:883a07d88dfbecfde9460be7346ad64449e3fa16b0354f3212685ad9f2630a88" +checksum = "sha256:8e0bf54009f4ea3ad649b93fc094a00ccae665d1e25660e1abd9bdb35630ede4" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:d9ca3794252781912c18f9d2bd0159c578c348b0acbab24012869e329963ee73" +checksum = "sha256:1195c884106d01876a234195d70cfe7eab39e41443b615504eeb30e6d137f4aa" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index e1be2998..231868a6 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -10,9 +10,10 @@ # makes "the image has the right tools" true by construction rather than by # convention: there is no second list to keep in step. # -# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the -# image identity hash and refuses a floating tag, because a tag that can change -# underneath a hash makes the hash a lie. +# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this +# file's text, so an edit here renames the image -- but it does not resolve or +# validate the base, and a floating tag can therefore change underneath a tag +# that claims to name fixed content. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -85,9 +86,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated -# source edit does not invalidate this layer. The synthetic Justfile avoids -# pulling in repository-specific imports that may not exist yet. +# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit +# elsewhere in the repository does not invalidate this layer -- though an edit +# to any file under `justfiles/` does, including ones the synthetic Justfile +# below never imports. That Justfile avoids pulling in repository-specific +# imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 1f2a1dee..02b16e34 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -116,7 +116,10 @@ just anvil-container # interactive shell * A container engine callable from the shell that runs `just`: Docker (supported) or Podman (best-effort). On Windows that means Docker - Desktop, Podman, or a Windows `docker` CLI pointed at an engine in WSL. + Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or + Docker Engine installed only inside the default WSL distribution — no + Windows CLI is needed in that last case, since anvil reaches the engine + through `wsl.exe` when it finds none on `PATH`. * `just` and `PowerShell` Core (`pwsh`) on the host. * A repository-owned `rust-toolchain.toml`. @@ -391,7 +394,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbIHyE2bP5dGcbfv5a5XBW8mUbpBv2VBT_2G0bCUddUAFHBklhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb80OmqCpM-PEb7-L3mxqzrXcbqBEF0vC0wXgbCT_rJDiFv7RhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index d82e8144..4481dba2 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -116,16 +116,12 @@ function Anvil-PreRun { } ``` -Both functions are optional. `Secrets` become `--secret id=…,src=…` mounts at build time; `Env` becomes -`-e NAME` at run time. Neither value ever appears as a command-line argument: a build secret is written to a -private temp file that is removed as soon as the build ends, and a run-time value is forwarded **by name**, so the -engine copies it from the environment it already inherits. Command lines are recorded by endpoint telemetry and -retained far longer than a short-lived token is meant to live. The engine also keeps build secrets out of every -image layer. - -`src=` is used rather than `env=` because it is the form both engines implement. Podman for Windows translates the -build context into its machine's view and then composes the `env=` temp path with a Windows separator, producing a -path it cannot open; a file anvil creates itself has no such step. +Both functions are optional. `Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` +at run time. In both cases the value is handed over **by environment variable name**, so it never appears in the +host's process command line — where endpoint telemetry records and retains it far longer than a short-lived token is +meant to live — and never touches disk. The engine keeps build secrets out of every image layer. When the engine is +reached through WSL (§6.1), the names are exported with `WSLENV` so the value crosses the boundary without ever +becoming an argument. The corresponding `RUN` should declare the mount as required, which closes the same hole from the Dockerfile's side: @@ -181,6 +177,15 @@ four-line Dockerfile reproduces it with no anvil involved. It affects only a rep supplies `Anvil-PreBuild` (§5); the public catalog ships no hook, so ordinary use is unaffected. Use docker if you need build-time credentials on Windows. +**Podman also ignores the build-context ignore file.** Anvil emits +`.anvil/container/Dockerfile.dockerignore`, which BuildKit reads in preference to a root +`.dockerignore`. Podman and buildah only honour `.containerignore` or `.dockerignore` at the +*context root*, so on podman the whole worktree — `target/` included — is streamed to the +daemon on every build, and a consumer repository that owns a root `.dockerignore` has that one +obeyed instead. Neither breaks the build unless the repository's own ignore file excludes +`justfiles/` or `rust-toolchain.toml`; the cost is transfer time. Passing `--ignorefile` would +fix it, but only podman accepts that flag, so the recipe does not. + Docker also carries more mileage: it is what CI uses and what the e2e runs by default. Prefer it if you have no reason not to. diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index d7ebd4a9..ea425f64 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -473,10 +473,11 @@ any other owned file under that prefix, so a derived catalog fails loudly at construction instead of shipping a file whose absence is only noticed later. The reason is containerized execution. The image identity hashes every `*.just` -under `justfiles/anvil/` recursively, and the build context admits that tree, -so a non-recipe file placed there would be silently dropped from the image while -still appearing to be managed. Non-recipe assets belong in a tool-owned -directory of their own, such as `.anvil/`. +under `justfiles/anvil/` recursively, while the build context copies the whole +directory, so a non-recipe file placed there is copied into the image but is +**not** part of its identity: editing it would change what the image contains +without renaming the tag, and no rebuild would follow. Non-recipe assets belong +in a tool-owned directory of their own, such as `.anvil/`. Containerized execution is itself an ordinary artifact group, customized with the same `replace_artifact` / `with_artifact` / `without_artifact` levers as diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 7492b7b0..652e0a40 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -169,9 +169,67 @@ mod tests { } } + #[test] + fn every_interpolation_into_powershell_is_escaped() { + // A `just` value pasted raw into a '…' literal ends the string on an + // apostrophe: a repository path containing one breaks every recipe + // here, and `target` would let the remainder run as host PowerShell. + // The deleted runner.just escaped every interpolation; this guards + // against losing that again. + // + // Two variables are exempt and checked explicitly below: the image name + // is regex-sanitized at definition, and the workdir is a literal. + const EXEMPT: [&str; 2] = ["{{anvil_container_name}}", "{{anvil_container_workdir}}"]; + for (index, _) in RECIPE.match_indices("'{{") { + let tail = &RECIPE[index + 1..]; + let escaped = tail.starts_with("{{ replace("); + assert!( + escaped || EXEMPT.iter().any(|exempt| tail.starts_with(exempt)), + "unescaped interpolation into a PowerShell literal at byte {index}: {}", + &tail[..tail.len().min(60)] + ); + } + // And the escaping that is present uses just's own doubling form. + assert!(RECIPE.contains(r#"replace(justfile_directory(), "'", "''")"#)); + assert!(RECIPE.contains(r#"replace(invocation_directory(), "'", "''")"#)); + assert!(RECIPE.contains(r#"replace(target, "'", "''")"#)); + } + + #[test] + fn the_image_name_cannot_carry_an_apostrophe() { + // What makes the exemption above safe. + assert!(RECIPE.contains(r#"replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-")"#)); + } + + #[test] + fn a_recipe_argument_survives_as_its_own_word() { + // `*target` joins with spaces, so passing it through as one string + // would break `anvil-container anvil-setup binstall` -- and around + // fifty generated recipes take a parameter. + assert!(RECIPE.contains(r"-split '\s+'")); + assert!(RECIPE.contains("just @targetParts")); + assert!(RECIPE.contains("@('just') + $targetParts")); + } + + #[test] + fn no_rebuild_is_honoured_even_with_no_cache_set() { + // The two controls compose: NO_REBUILD must not be skipped just + // because NO_CACHE is exported, or `anvil-container-status` spends + // minutes building from a query. + let no_cache = RECIPE + .find("if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') {") + .expect("the cache guard must exist"); + let no_rebuild = RECIPE + .find("if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') {") + .expect("the no-rebuild guard must exist"); + let guard_end = RECIPE[no_cache..].find("\n }\n").expect("the cache guard must be closed") + no_cache; + assert!(no_rebuild > guard_end, "the NO_REBUILD check must sit outside the NO_CACHE guard"); + } + #[test] fn engine_is_an_environment_variable_with_a_docker_default() { assert!(RECIPE.contains(r#"env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker")"#)); + assert!(RECIPE.contains(r#"replace(anvil_container_engine, "'", "''")"#)); } #[test] diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 826e46a1..040529e6 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -115,7 +115,10 @@ //! //! - A container engine callable from the shell that runs `just`: Docker //! (supported) or Podman (best-effort). On Windows that means Docker -//! Desktop, Podman, or a Windows `docker` CLI pointed at an engine in WSL. +//! Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or +//! Docker Engine installed only inside the default WSL distribution — no +//! Windows CLI is needed in that last case, since anvil reaches the engine +//! through `wsl.exe` when it finds none on `PATH`. //! - `just` and `PowerShell` Core (`pwsh`) on the host. //! - A repository-owned `rust-toolchain.toml`. //! diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile index e1be2998..231868a6 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -10,9 +10,10 @@ # makes "the image has the right tools" true by construction rather than by # convention: there is no second list to keep in step. # -# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the -# image identity hash and refuses a floating tag, because a tag that can change -# underneath a hash makes the hash a lie. +# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this +# file's text, so an edit here renames the image -- but it does not resolve or +# validate the base, and a floating tag can therefore change underneath a tag +# that claims to name fixed content. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -85,9 +86,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated -# source edit does not invalidate this layer. The synthetic Justfile avoids -# pulling in repository-specific imports that may not exist yet. +# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit +# elsewhere in the repository does not invalidate this layer -- though an edit +# to any file under `justfiles/` does, including ones the synthetic Justfile +# below never imports. That Justfile avoids pulling in repository-specific +# imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 6b72f747..8f4b34ef 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -24,9 +24,10 @@ anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. anvil_container_workdir := "/workspace" -# Image and cache-volume prefix, derived from the repository directory so two -# repositories on one host cannot collide. Sanitized to the character set -# container image references allow. +# Image and cache-volume prefix, derived from the repository directory name. +# Two checkouts with the same directory name share cache volumes; that is +# harmless (the caches are content-addressed by cargo) but worth knowing before +# `anvil-container-down` removes volumes another checkout is also using. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") # Resolve how to invoke the engine, as a pipe-separated command. @@ -48,7 +49,7 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di [script("pwsh", "-NoProfile")] _anvil-container-engine: $ErrorActionPreference = 'Stop' - $engine = '{{anvil_container_engine}}' + $engine = '{{ replace(anvil_container_engine, "'", "''") }}' if ($engine -ne 'docker' -and $engine -ne 'podman') { Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 @@ -80,16 +81,16 @@ _anvil-container-path host_path: $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if (-not $engine.StartsWith('wsl.exe|')) { - Write-Output '{{host_path}}' + Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } # Pass the path with forward slashes: arguments cross into WSL through a # shell that would otherwise consume the backslashes, leaving wslpath to # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{host_path}}' -replace '\\', '/' + $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' $translated = & wsl.exe -- wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" + Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -117,7 +118,7 @@ _anvil-container-image: $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -167,13 +168,19 @@ _anvil-container-image: Write-Output $image exit 0 } - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - # Still report the reference: a caller that asked not to build is - # usually asking *which* image is missing. - Write-Output $image - [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") - exit 1 - } + } + + # Checked outside the cache guard, so the two variables compose: a caller + # that has NO_CACHE exported would otherwise fall straight through to a + # from-scratch build, which is exactly what NO_REBUILD exists to prevent -- + # and `anvil-container-status`, which sets it, would spend minutes building + # from a query. + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present or not current, and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 } # Build-time credentials come from the optional hook, never from a committed @@ -267,10 +274,14 @@ _anvil-container-image: [script("pwsh", "-NoProfile")] anvil-container *target: $ErrorActionPreference = 'Stop' - $target = '{{target}}' + # Split back into argv. `*target` joins its parts with spaces, so passing + # the string through as one argument would make `anvil-container anvil-setup + # binstall` look for a recipe literally named "anvil-setup binstall" -- + # and around fifty generated recipes take a parameter. + $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + if ($targetParts.Count -gt 0) { just @targetParts } exit $LASTEXITCODE } @@ -282,20 +293,20 @@ anvil-container *target: $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $engineRoot = (just _anvil-container-path $repoRoot).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { '{{anvil_container_workdir}}/' + $rel } - $interactive = [string]::IsNullOrWhiteSpace($target) + $interactive = $targetParts.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -348,7 +359,7 @@ anvil-container *target: # --pull=never: the tag names locally-built content, so a miss is a bug # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just', $target) } + if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 4396e1c8..ae64ca11 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -15,9 +15,10 @@ expression: render_tree(tmp.path()) # makes "the image has the right tools" true by construction rather than by # convention: there is no second list to keep in step. # -# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the -# image identity hash and refuses a floating tag, because a tag that can change -# underneath a hash makes the hash a lie. +# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this +# file's text, so an edit here renames the image -- but it does not resolve or +# validate the base, and a floating tag can therefore change underneath a tag +# that claims to name fixed content. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -90,9 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated -# source edit does not invalidate this layer. The synthetic Justfile avoids -# pulling in repository-specific imports that may not exist yet. +# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit +# elsewhere in the repository does not invalidate this layer -- though an edit +# to any file under `justfiles/` does, including ones the synthetic Justfile +# below never imports. That Justfile avoids pulling in repository-specific +# imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -3499,9 +3502,10 @@ anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. anvil_container_workdir := "/workspace" -# Image and cache-volume prefix, derived from the repository directory so two -# repositories on one host cannot collide. Sanitized to the character set -# container image references allow. +# Image and cache-volume prefix, derived from the repository directory name. +# Two checkouts with the same directory name share cache volumes; that is +# harmless (the caches are content-addressed by cargo) but worth knowing before +# `anvil-container-down` removes volumes another checkout is also using. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") # Resolve how to invoke the engine, as a pipe-separated command. @@ -3523,7 +3527,7 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di [script("pwsh", "-NoProfile")] _anvil-container-engine: $ErrorActionPreference = 'Stop' - $engine = '{{anvil_container_engine}}' + $engine = '{{ replace(anvil_container_engine, "'", "''") }}' if ($engine -ne 'docker' -and $engine -ne 'podman') { Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 @@ -3555,16 +3559,16 @@ _anvil-container-path host_path: $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if (-not $engine.StartsWith('wsl.exe|')) { - Write-Output '{{host_path}}' + Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } # Pass the path with forward slashes: arguments cross into WSL through a # shell that would otherwise consume the backslashes, leaving wslpath to # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{host_path}}' -replace '\\', '/' + $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' $translated = & wsl.exe -- wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" + Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -3592,7 +3596,7 @@ _anvil-container-image: $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -3642,13 +3646,19 @@ _anvil-container-image: Write-Output $image exit 0 } - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - # Still report the reference: a caller that asked not to build is - # usually asking *which* image is missing. - Write-Output $image - [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") - exit 1 - } + } + + # Checked outside the cache guard, so the two variables compose: a caller + # that has NO_CACHE exported would otherwise fall straight through to a + # from-scratch build, which is exactly what NO_REBUILD exists to prevent -- + # and `anvil-container-status`, which sets it, would spend minutes building + # from a query. + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present or not current, and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 } # Build-time credentials come from the optional hook, never from a committed @@ -3742,10 +3752,14 @@ _anvil-container-image: [script("pwsh", "-NoProfile")] anvil-container *target: $ErrorActionPreference = 'Stop' - $target = '{{target}}' + # Split back into argv. `*target` joins its parts with spaces, so passing + # the string through as one argument would make `anvil-container anvil-setup + # binstall` look for a recipe literally named "anvil-setup binstall" -- + # and around fifty generated recipes take a parameter. + $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + if ($targetParts.Count -gt 0) { just @targetParts } exit $LASTEXITCODE } @@ -3757,20 +3771,20 @@ anvil-container *target: $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $engineRoot = (just _anvil-container-path $repoRoot).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { '{{anvil_container_workdir}}/' + $rel } - $interactive = [string]::IsNullOrWhiteSpace($target) + $interactive = $targetParts.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -3823,7 +3837,7 @@ anvil-container *target: # --pull=never: the tag names locally-built content, so a miss is a bug # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just', $target) } + if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 4b539ccd..80d91aef 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -15,9 +15,10 @@ expression: render_tree(tmp.path()) # makes "the image has the right tools" true by construction rather than by # convention: there is no second list to keep in step. # -# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the -# image identity hash and refuses a floating tag, because a tag that can change -# underneath a hash makes the hash a lie. +# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this +# file's text, so an edit here renames the image -- but it does not resolve or +# validate the base, and a floating tag can therefore change underneath a tag +# that claims to name fixed content. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -90,9 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated -# source edit does not invalidate this layer. The synthetic Justfile avoids -# pulling in repository-specific imports that may not exist yet. +# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit +# elsewhere in the repository does not invalidate this layer -- though an edit +# to any file under `justfiles/` does, including ones the synthetic Justfile +# below never imports. That Justfile avoids pulling in repository-specific +# imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -3420,9 +3423,10 @@ anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. anvil_container_workdir := "/workspace" -# Image and cache-volume prefix, derived from the repository directory so two -# repositories on one host cannot collide. Sanitized to the character set -# container image references allow. +# Image and cache-volume prefix, derived from the repository directory name. +# Two checkouts with the same directory name share cache volumes; that is +# harmless (the caches are content-addressed by cargo) but worth knowing before +# `anvil-container-down` removes volumes another checkout is also using. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") # Resolve how to invoke the engine, as a pipe-separated command. @@ -3444,7 +3448,7 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di [script("pwsh", "-NoProfile")] _anvil-container-engine: $ErrorActionPreference = 'Stop' - $engine = '{{anvil_container_engine}}' + $engine = '{{ replace(anvil_container_engine, "'", "''") }}' if ($engine -ne 'docker' -and $engine -ne 'podman') { Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 @@ -3476,16 +3480,16 @@ _anvil-container-path host_path: $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if (-not $engine.StartsWith('wsl.exe|')) { - Write-Output '{{host_path}}' + Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } # Pass the path with forward slashes: arguments cross into WSL through a # shell that would otherwise consume the backslashes, leaving wslpath to # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{host_path}}' -replace '\\', '/' + $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' $translated = & wsl.exe -- wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" + Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -3513,7 +3517,7 @@ _anvil-container-image: $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -3563,13 +3567,19 @@ _anvil-container-image: Write-Output $image exit 0 } - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - # Still report the reference: a caller that asked not to build is - # usually asking *which* image is missing. - Write-Output $image - [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") - exit 1 - } + } + + # Checked outside the cache guard, so the two variables compose: a caller + # that has NO_CACHE exported would otherwise fall straight through to a + # from-scratch build, which is exactly what NO_REBUILD exists to prevent -- + # and `anvil-container-status`, which sets it, would spend minutes building + # from a query. + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present or not current, and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 } # Build-time credentials come from the optional hook, never from a committed @@ -3663,10 +3673,14 @@ _anvil-container-image: [script("pwsh", "-NoProfile")] anvil-container *target: $ErrorActionPreference = 'Stop' - $target = '{{target}}' + # Split back into argv. `*target` joins its parts with spaces, so passing + # the string through as one argument would make `anvil-container anvil-setup + # binstall` look for a recipe literally named "anvil-setup binstall" -- + # and around fifty generated recipes take a parameter. + $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + if ($targetParts.Count -gt 0) { just @targetParts } exit $LASTEXITCODE } @@ -3678,20 +3692,20 @@ anvil-container *target: $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $engineRoot = (just _anvil-container-path $repoRoot).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { '{{anvil_container_workdir}}/' + $rel } - $interactive = [string]::IsNullOrWhiteSpace($target) + $interactive = $targetParts.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -3744,7 +3758,7 @@ anvil-container *target: # --pull=never: the tag names locally-built content, so a miss is a bug # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just', $target) } + if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index be1e8923..88c7a820 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -15,9 +15,10 @@ expression: render_tree(tmp.path()) # makes "the image has the right tools" true by construction rather than by # convention: there is no second list to keep in step. # -# BASE_IMAGE must stay digest-pinned. `_anvil-container-image` folds it into the -# image identity hash and refuses a floating tag, because a tag that can change -# underneath a hash makes the hash a lie. +# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this +# file's text, so an edit here renames the image -- but it does not resolve or +# validate the base, and a floating tag can therefore change underneath a tag +# that claims to name fixed content. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -90,9 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only the files `just anvil-setup` needs are copied, so an unrelated -# source edit does not invalidate this layer. The synthetic Justfile avoids -# pulling in repository-specific imports that may not exist yet. +# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit +# elsewhere in the repository does not invalidate this layer -- though an edit +# to any file under `justfiles/` does, including ones the synthetic Justfile +# below never imports. That Justfile avoids pulling in repository-specific +# imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -2239,9 +2242,10 @@ anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. anvil_container_workdir := "/workspace" -# Image and cache-volume prefix, derived from the repository directory so two -# repositories on one host cannot collide. Sanitized to the character set -# container image references allow. +# Image and cache-volume prefix, derived from the repository directory name. +# Two checkouts with the same directory name share cache volumes; that is +# harmless (the caches are content-addressed by cargo) but worth knowing before +# `anvil-container-down` removes volumes another checkout is also using. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") # Resolve how to invoke the engine, as a pipe-separated command. @@ -2263,7 +2267,7 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di [script("pwsh", "-NoProfile")] _anvil-container-engine: $ErrorActionPreference = 'Stop' - $engine = '{{anvil_container_engine}}' + $engine = '{{ replace(anvil_container_engine, "'", "''") }}' if ($engine -ne 'docker' -and $engine -ne 'podman') { Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 @@ -2295,16 +2299,16 @@ _anvil-container-path host_path: $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if (-not $engine.StartsWith('wsl.exe|')) { - Write-Output '{{host_path}}' + Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } # Pass the path with forward slashes: arguments cross into WSL through a # shell that would otherwise consume the backslashes, leaving wslpath to # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{host_path}}' -replace '\\', '/' + $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' $translated = & wsl.exe -- wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" + Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -2332,7 +2336,7 @@ _anvil-container-image: $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -2382,13 +2386,19 @@ _anvil-container-image: Write-Output $image exit 0 } - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - # Still report the reference: a caller that asked not to build is - # usually asking *which* image is missing. - Write-Output $image - [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") - exit 1 - } + } + + # Checked outside the cache guard, so the two variables compose: a caller + # that has NO_CACHE exported would otherwise fall straight through to a + # from-scratch build, which is exactly what NO_REBUILD exists to prevent -- + # and `anvil-container-status`, which sets it, would spend minutes building + # from a query. + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present or not current, and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 } # Build-time credentials come from the optional hook, never from a committed @@ -2482,10 +2492,14 @@ _anvil-container-image: [script("pwsh", "-NoProfile")] anvil-container *target: $ErrorActionPreference = 'Stop' - $target = '{{target}}' + # Split back into argv. `*target` joins its parts with spaces, so passing + # the string through as one argument would make `anvil-container anvil-setup + # binstall` look for a recipe literally named "anvil-setup binstall" -- + # and around fifty generated recipes take a parameter. + $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + if ($targetParts.Count -gt 0) { just @targetParts } exit $LASTEXITCODE } @@ -2497,20 +2511,20 @@ anvil-container *target: $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $engineRoot = (just _anvil-container-path $repoRoot).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { '{{anvil_container_workdir}}/' + $rel } - $interactive = [string]::IsNullOrWhiteSpace($target) + $interactive = $targetParts.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -2563,7 +2577,7 @@ anvil-container *target: # --pull=never: the tag names locally-built content, so a miss is a bug # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just', $target) } + if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 6b72f747..8f4b34ef 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -24,9 +24,10 @@ anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. anvil_container_workdir := "/workspace" -# Image and cache-volume prefix, derived from the repository directory so two -# repositories on one host cannot collide. Sanitized to the character set -# container image references allow. +# Image and cache-volume prefix, derived from the repository directory name. +# Two checkouts with the same directory name share cache volumes; that is +# harmless (the caches are content-addressed by cargo) but worth knowing before +# `anvil-container-down` removes volumes another checkout is also using. anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") # Resolve how to invoke the engine, as a pipe-separated command. @@ -48,7 +49,7 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di [script("pwsh", "-NoProfile")] _anvil-container-engine: $ErrorActionPreference = 'Stop' - $engine = '{{anvil_container_engine}}' + $engine = '{{ replace(anvil_container_engine, "'", "''") }}' if ($engine -ne 'docker' -and $engine -ne 'podman') { Write-Error "anvil: ANVIL_CONTAINER_ENGINE must be 'docker' or 'podman', got '$engine'" exit 1 @@ -80,16 +81,16 @@ _anvil-container-path host_path: $engine = (just _anvil-container-engine).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if (-not $engine.StartsWith('wsl.exe|')) { - Write-Output '{{host_path}}' + Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } # Pass the path with forward slashes: arguments cross into WSL through a # shell that would otherwise consume the backslashes, leaving wslpath to # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{host_path}}' -replace '\\', '/' + $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' $translated = & wsl.exe -- wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{host_path}}' for the engine running in WSL" + Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -117,7 +118,7 @@ _anvil-container-image: $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -167,13 +168,19 @@ _anvil-container-image: Write-Output $image exit 0 } - if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { - # Still report the reference: a caller that asked not to build is - # usually asking *which* image is missing. - Write-Output $image - [Console]::Error.WriteLine("anvil: $image is not present and ANVIL_CONTAINER_NO_REBUILD=1") - exit 1 - } + } + + # Checked outside the cache guard, so the two variables compose: a caller + # that has NO_CACHE exported would otherwise fall straight through to a + # from-scratch build, which is exactly what NO_REBUILD exists to prevent -- + # and `anvil-container-status`, which sets it, would spend minutes building + # from a query. + if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') { + # Still report the reference: a caller that asked not to build is + # usually asking *which* image is missing. + Write-Output $image + [Console]::Error.WriteLine("anvil: $image is not present or not current, and ANVIL_CONTAINER_NO_REBUILD=1") + exit 1 } # Build-time credentials come from the optional hook, never from a committed @@ -267,10 +274,14 @@ _anvil-container-image: [script("pwsh", "-NoProfile")] anvil-container *target: $ErrorActionPreference = 'Stop' - $target = '{{target}}' + # Split back into argv. `*target` joins its parts with spaces, so passing + # the string through as one argument would make `anvil-container anvil-setup + # binstall` look for a recipe literally named "anvil-setup binstall" -- + # and around fifty generated recipes take a parameter. + $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if (-not [string]::IsNullOrWhiteSpace($target)) { just $target } + if ($targetParts.Count -gt 0) { just @targetParts } exit $LASTEXITCODE } @@ -282,20 +293,20 @@ anvil-container *target: $image = (just _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $repoRoot = '{{justfile_directory()}}' + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $engineRoot = (just _anvil-container-path $repoRoot).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{invocation_directory()}}') -replace '\\', '/' + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { '{{anvil_container_workdir}}/' + $rel } - $interactive = [string]::IsNullOrWhiteSpace($target) + $interactive = $targetParts.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -348,7 +359,7 @@ anvil-container *target: # --pull=never: the tag names locally-built content, so a miss is a bug # to surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just', $target) } + if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index d076bf57..be50c95a 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -24,7 +24,8 @@ 4. Changing a hashed input (the pinned toolchain) selects a new tag. 5. Reverting that input returns to the original tag. 6. Editing the Dockerfile is preserved by a re-run of the generator. - 7. A credential hook reaches both the build and the run. + 7. A credential hook reaches both the build and the run, and the secret + reaches neither a build command nor the image filesystem. 8. A hook returning an empty value fails closed. 9. A hook's returned value does not change the tag; its file content does. 10. The recipes run natively inside the image (no nesting). @@ -469,8 +470,21 @@ Assert-That 'forwarded names are reported' ($hookRun.StdErr -match 'forwarding e $secretReference = Get-ImageReference -Repo $repo $layers = Invoke-Engine -Arguments @('history', '--no-trunc', $secretReference) -AllowFailure -Assert-That 'the secret value is absent from every image layer' ` - (-not ($layers.StdOut -match 'build-secret-value')) 'a build secret must never reach a layer' +Assert-Equal 'the image history is readable' 0 $layers.ExitCode +Assert-That 'no build command records the secret' ` + (-not ($layers.StdOut -match 'build-secret-value')) 'a secret must never reach a build argument' + +# `history` reports the command that created each layer, not its contents, so on +# its own it cannot see a secret that was *written* into the filesystem -- which +# is the hazard the Dockerfile guards against by deleting credential files in +# the same layer as the install. Look at the filesystem the image actually +# carries. grep exits 1 for "no match", which is the result we want; -s keeps an +# unreadable path from turning into exit 2 and passing for the wrong reason. +$leak = Invoke-Engine -Arguments @( + 'run', '--rm', '--pull=never', $secretReference, + 'grep', '-rsq', 'build-secret-value', '/opt/anvil', '/root', '/usr/local/cargo', '/tmp', '/run' +) -AllowFailure +Assert-Equal 'the secret is absent from the image filesystem' 1 $leak.ExitCode Write-Step 'checking that the forwarded value arrives inside the container' $showEnv = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-env') -AllowFailure From 30c084fdfbb7009e2d359d6a33afda05572ad191 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 13:15:27 +0200 Subject: [PATCH 08/81] fix(anvil): point podman at the build-context ignore file BuildKit finds `.dockerignore` on its own; buildah reads only a context-root ignore file. Without the flag every podman build streamed the whole worktree -- `target/` included -- to the daemon, and a consumer repository owning a root `.dockerignore` had that one obeyed instead, which can exclude `justfiles/` and fail the build for a reason that names nothing relevant. Named rather than probed, because the engine that does not take the flag rejects it outright. This is a capability difference between the two engines, in the same category as the uid mapping already handled, not a second code path through the feature. docker 46/46, podman 31/31. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 4 ++-- crates/cargo-anvil/docs/design/containers.md | 12 +++++------- crates/cargo-anvil/src/anvil/artifacts/container.rs | 8 ++++++++ .../templates/justfiles/anvil/container.just | 6 ++++++ .../tests/snapshots/snapshots__ado_backend.snap | 6 ++++++ .../tests/snapshots/snapshots__github_backend.snap | 6 ++++++ .../tests/snapshots/snapshots__local_only.snap | 6 ++++++ justfiles/anvil/container.just | 6 ++++++ 8 files changed, 45 insertions(+), 9 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 817ab32a..1efa7ec0 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:970f6a55ed74190d1ef848fc016c529027fd016037e36f09a6027c30d2f70c05" +catalog_checksum = "sha256:a32468bd2399b1993c38d54d9522e8280246cb951e073a81f5651760704dd76f" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:1195c884106d01876a234195d70cfe7eab39e41443b615504eeb30e6d137f4aa" +checksum = "sha256:5e51725f60ede7469e9ae211bb8c9634ebe2b924358e6fab33a37d41ca42540e" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 4481dba2..41786008 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -177,14 +177,12 @@ four-line Dockerfile reproduces it with no anvil involved. It affects only a rep supplies `Anvil-PreBuild` (§5); the public catalog ships no hook, so ordinary use is unaffected. Use docker if you need build-time credentials on Windows. -**Podman also ignores the build-context ignore file.** Anvil emits +**Podman needs to be pointed at the ignore file.** Anvil emits `.anvil/container/Dockerfile.dockerignore`, which BuildKit reads in preference to a root -`.dockerignore`. Podman and buildah only honour `.containerignore` or `.dockerignore` at the -*context root*, so on podman the whole worktree — `target/` included — is streamed to the -daemon on every build, and a consumer repository that owns a root `.dockerignore` has that one -obeyed instead. Neither breaks the build unless the repository's own ignore file excludes -`justfiles/` or `rust-toolchain.toml`; the cost is transfer time. Passing `--ignorefile` would -fix it, but only podman accepts that flag, so the recipe does not. +`.dockerignore`. Podman and buildah honour only `.containerignore` or `.dockerignore` at the +*context root*, so the recipe passes `--ignorefile` explicitly when the engine is podman. Without +it the whole worktree — `target/` included — would be streamed to the daemon on every build, and +a consumer repository owning a root `.dockerignore` would have that one obeyed instead. Docker also carries more mileage: it is what CI uses and what the e2e runs by default. Prefer it if you have no reason not to. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 652e0a40..97d5f4b5 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -211,6 +211,14 @@ mod tests { assert!(RECIPE.contains("@('just') + $targetParts")); } + #[test] + fn podman_is_pointed_at_the_ignore_file() { + // BuildKit finds `.dockerignore` itself; buildah reads only + // a context-root file, so without this the whole worktree is the build + // context. + assert!(RECIPE.contains(r#"if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile'"#)); + } + #[test] fn no_rebuild_is_honoured_even_with_no_cache_set() { // The two controls compose: NO_REBUILD must not be skipped just diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 8f4b34ef..2c2405c9 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -228,6 +228,12 @@ _anvil-container-image: # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) + # BuildKit reads `.dockerignore` on its own; buildah reads + # only a context-root ignore file and needs to be pointed at ours. Named + # rather than probed, because the flag is rejected outright by the engine + # that does not take it, and an unscoped context streams the whole + # worktree -- `target/` included -- on every build. + if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile', "$engineRoot/$dockerfile.dockerignore") } if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } $buildCmd += $engineRoot diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index ae64ca11..4abbe77c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3706,6 +3706,12 @@ _anvil-container-image: # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) + # BuildKit reads `.dockerignore` on its own; buildah reads + # only a context-root ignore file and needs to be pointed at ours. Named + # rather than probed, because the flag is rejected outright by the engine + # that does not take it, and an unscoped context streams the whole + # worktree -- `target/` included -- on every build. + if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile', "$engineRoot/$dockerfile.dockerignore") } if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } $buildCmd += $engineRoot diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 80d91aef..3f1e8de7 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3627,6 +3627,12 @@ _anvil-container-image: # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) + # BuildKit reads `.dockerignore` on its own; buildah reads + # only a context-root ignore file and needs to be pointed at ours. Named + # rather than probed, because the flag is rejected outright by the engine + # that does not take it, and an unscoped context streams the whole + # worktree -- `target/` included -- on every build. + if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile', "$engineRoot/$dockerfile.dockerignore") } if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } $buildCmd += $engineRoot diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 88c7a820..46525615 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2446,6 +2446,12 @@ _anvil-container-image: # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) + # BuildKit reads `.dockerignore` on its own; buildah reads + # only a context-root ignore file and needs to be pointed at ours. Named + # rather than probed, because the flag is rejected outright by the engine + # that does not take it, and an unscoped context streams the whole + # worktree -- `target/` included -- on every build. + if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile', "$engineRoot/$dockerfile.dockerignore") } if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } $buildCmd += $engineRoot diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 8f4b34ef..2c2405c9 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -228,6 +228,12 @@ _anvil-container-image: # It also keeps the identity scheme honest: without this, two hosts of # different architecture compute the same tag for different images. $buildCmd = @('build', '--platform', 'linux/amd64', '--file', "$engineRoot/$dockerfile", '--tag', $image) + # BuildKit reads `.dockerignore` on its own; buildah reads + # only a context-root ignore file and needs to be pointed at ours. Named + # rather than probed, because the flag is rejected outright by the engine + # that does not take it, and an unscoped context streams the whole + # worktree -- `target/` included -- on every build. + if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile', "$engineRoot/$dockerfile.dockerignore") } if ($env:ANVIL_CONTAINER_NO_CACHE -eq '1') { $buildCmd += '--no-cache' } foreach ($secret in $secretArgs) { $buildCmd += @('--secret', $secret) } $buildCmd += $engineRoot From 3c8d4d70b97abd44e3636b63a37a55174de4fdcd Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 13:22:59 +0200 Subject: [PATCH 09/81] fix(anvil): defer the lints region to a crate that owns its lints Anvil spliced `[lints] workspace = true` into every workspace member. A member that already declares its own `[lints.*]` then carries both, which cargo rejects outright -- and because the failure is in `cargo metadata`, it takes down every check in the workspace rather than only the offending crate. The ox-tools dogfood hit exactly this: one migrated crate keeps a deliberately lenient lint set, and generating made the repository unbuildable. Generalize the existing delta opt-out, which already solves the same shape of problem for `trip_wire_patterns`, into one `region_body` decision covering both regions. A crate that owns its lints now gets an empty managed region: the region stays tracked, so dropping the crate's own lints later adopts the catalog with no further gesture, and the plan explains the deferral rather than leaving it to be discovered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a310c4-f37b-497e-a10b-eb8d982d532d --- .../cargo-anvil/src/anvil/artifacts/region.rs | 2 +- crates/cargo-anvil/src/run.rs | 140 ++++++++++++++---- 2 files changed, 114 insertions(+), 28 deletions(-) diff --git a/crates/cargo-anvil/src/anvil/artifacts/region.rs b/crates/cargo-anvil/src/anvil/artifacts/region.rs index 0deba72c..2c023a83 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/region.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/region.rs @@ -20,7 +20,7 @@ const WORKSPACE_LINTS_REGION_ID: &str = "anvil-workspace-lints"; /// Region id for crate-scope lints — used both for single-crate repos (full /// catalog) and for each member of a multi-crate workspace (`workspace = /// true`). -const CRATE_LINTS_REGION_ID: &str = "anvil-lints"; +pub(crate) const CRATE_LINTS_REGION_ID: &str = "anvil-lints"; /// Embedded body of the lint catalog, in dotted-key form (no table header). const LINTS_BODY: &str = include_str!("../../../templates/regions/cargo-lints-body.toml"); diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 7bb1f817..8e32388b 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -12,7 +12,7 @@ use std::path::Path; use ohno::{AppError, bail}; use tracing::info; -use crate::anvil::artifacts::region::DELTA_REGION_ID; +use crate::anvil::artifacts::region::{CRATE_LINTS_REGION_ID, DELTA_REGION_ID}; use crate::backend::{self, Backend}; use crate::catalog::Catalog; use crate::catalog::artifact::{Artifact, HostSelector, RegionSpec}; @@ -348,20 +348,17 @@ fn push_region_at( let host = resolve_existing_case_insensitive(repo_root, host); let current = hosts.get_or_read(repo_root, &host)?; let placement = region_placement(spec.id.as_str()); - let body = match delta_region_body(current.as_deref(), spec) { - DeltaRegionBody::Managed => spec.body.as_str(), - DeltaRegionBody::PreserveRepositoryKey => { - plan.note( - "The repository's .delta.toml already defines top-level `trip_wire_patterns`; \ - the managed anvil-delta region was left empty. Remove the repository key to \ - adopt the managed trip-wire list.", - ); + let body = match region_body(current.as_deref(), spec) { + RegionBody::Managed => spec.body.as_str(), + RegionBody::Defer(note) => { + plan.note(note); "" } - DeltaRegionBody::Malformed(reason) => { + RegionBody::Malformed(reason) => { plan.refusal(format!( - "Refused to manage .delta.toml [anvil-delta] because the existing host could not \ - be safely inspected: {reason}. Other artifacts were still planned." + "Refused to manage {host} [{}] because the existing host could not \ + be safely inspected: {reason}. Other artifacts were still planned.", + spec.id.as_str() )); plan.push(PlanItem::noop( Target::Region { @@ -405,34 +402,74 @@ fn region_placement(region_id: &str) -> RegionPlacement { } } -enum DeltaRegionBody { +/// How a managed region's body resolves against what the host already +/// contains. +enum RegionBody { + /// Splice the catalog's body. Managed, - PreserveRepositoryKey, + /// Splice an empty region, and say why. The repository already defines the + /// key this region would own, and emitting both produces a file the + /// toolchain rejects. The empty region stays tracked, so the catalog body + /// is adopted automatically once the repository drops its own. + Defer(String), + /// The host could not be safely inspected. Malformed(String), } -fn delta_region_body(host_text: Option<&str>, spec: &RegionSpec) -> DeltaRegionBody { - if spec.id.as_str() != DELTA_REGION_ID { - return DeltaRegionBody::Managed; +/// Resolve a region's body against its host, deferring where the repository +/// already owns the same key. +/// +/// Two regions can collide with repository-owned content, and in both cases a +/// naive splice yields a file that fails to parse rather than one that merely +/// looks odd: +/// +/// - `anvil-delta` against a top-level `trip_wire_patterns`, which would become +/// a duplicate TOML key. +/// - `anvil-lints` against a crate that declares its own `[lints.*]`, which +/// cargo rejects with "cannot override `workspace.lints` in `lints`" -- +/// taking down `cargo metadata`, and with it every check in the workspace, +/// not just the offending crate. +fn region_body(host_text: Option<&str>, spec: &RegionSpec) -> RegionBody { + let id = spec.id.as_str(); + if id != DELTA_REGION_ID && id != CRATE_LINTS_REGION_ID { + return RegionBody::Managed; } let Some(host_text) = host_text else { - return DeltaRegionBody::Managed; + return RegionBody::Managed; }; - let without_region = match remove_region(host_text, spec.id.as_str(), spec.syntax) { + // Inspect the host as it reads without anvil's own region, so a body + // anvil spliced on an earlier pass is never mistaken for repository + // content. + let without_region = match remove_region(host_text, id, spec.syntax) { Ok(without_region) => without_region, - Err(error) => return DeltaRegionBody::Malformed(format!("managed-region markers are malformed: {error}")), + Err(error) => return RegionBody::Malformed(format!("managed-region markers are malformed: {error}")), }; let document = match without_region.parse::() { Ok(document) => document, - Err(error) => { - return DeltaRegionBody::Malformed(format!("invalid TOML: {error}")); - } + Err(error) => return RegionBody::Malformed(format!("invalid TOML: {error}")), }; - if document.as_table().contains_key("trip_wire_patterns") { - DeltaRegionBody::PreserveRepositoryKey - } else { - DeltaRegionBody::Managed + + if id == DELTA_REGION_ID { + if document.as_table().contains_key("trip_wire_patterns") { + return RegionBody::Defer( + "The repository's .delta.toml already defines top-level `trip_wire_patterns`; \ + the managed anvil-delta region was left empty. Remove the repository key to \ + adopt the managed trip-wire list." + .to_owned(), + ); + } + return RegionBody::Managed; + } + + if document.as_table().contains_key("lints") { + return RegionBody::Defer( + "This crate declares its own `[lints]`; the managed anvil-lints region was left \ + empty. Cargo rejects a manifest carrying both, which would break `cargo metadata` \ + for the whole workspace. Remove the crate's own lints to adopt the catalog." + .to_owned(), + ); } + RegionBody::Managed } /// Scan the previous manifest for entries that the active plan items @@ -1036,6 +1073,55 @@ mod tests { assert!(!second.plan.has_changes()); } + #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] + #[test] + fn member_declaring_its_own_lints_opts_out_of_the_managed_catalog() { + // A crate carrying its own `[lints.clippy]` must not also receive + // `[lints] workspace = true`. Cargo rejects that combination outright + // ("cannot override `workspace.lints` in `lints`"), which breaks + // `cargo metadata` and therefore every check in the workspace -- not + // only the offending crate. + let tmp = empty_workspace(); + let member = tmp.path().join("crates/alpha/Cargo.toml"); + fs::write( + &member, + "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n\n\ + [lints.clippy]\npedantic = { level = \"warn\", priority = -1 }\n", + ) + .unwrap(); + let args = Cli { + backends: vec![], + no_backends: true, + dry_run: false, + force: false, + }; + + let outcome = run_update(&Catalog::anvil(), &args, tmp.path()).unwrap(); + + let content = fs::read_to_string(&member).unwrap(); + let document: toml_edit::DocumentMut = content.parse().expect("member manifest must remain valid TOML"); + assert!( + document["lints"].get("clippy").is_some(), + "the crate's own lints must survive" + ); + assert!( + document["lints"].get("workspace").is_none(), + "anvil must not add `workspace = true` beside the crate's own lints" + ); + let region = find_region(&content, CRATE_LINTS_REGION_ID, CommentSyntax::Hash) + .unwrap() + .expect("the region stays tracked, so dropping the crate's lints adopts the catalog"); + assert!(region.is_empty(), "a crate that owns its lints opts out of the managed body"); + assert!( + outcome + .plan + .notes() + .iter() + .any(|note| note.contains("declares its own `[lints]`")), + "the opt-out must be visible in the plan summary" + ); + } + #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] #[test] fn existing_delta_trip_wires_are_preserved_without_duplicate_key() { From 088b76c4f9892e720a5927ccc1dbcd07ddbc5b03 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 13:27:10 +0200 Subject: [PATCH 10/81] feat(anvil): name the image without building it, and let a hook resolve one `anvil-container-tag` prints the reference for the current inputs and exits. It is now the only place the content hash is computed -- the resolver asks it rather than repeating the computation -- because a publisher needs the tag before there is an image to inspect, and a published tag is only meaningful while it is the same reference the consumer will later look up. `Anvil-ResolveImage` is a third hook phase, offered the tag when nothing local matches and before a build starts. It returns the reference it made available rather than re-tagging to the local name: a local tag asserts "built here from these inputs", and a fetched image only claims that, since the hash is over source files and cannot be re-derived from layers. The returned reference is inspected before use, because the run is `--pull=never` and a hook that reported an image it never fetched would fail later and further from the cause. Every failure falls through to a local build -- a publisher that has not caught up must not block the change it has not caught up with. Resolution sits inside the NO_CACHE guard, since ignoring the cache has to mean the remote one too, and before the NO_REBUILD guard, since fetching is not building. ANVIL_CONTAINER_NO_RESOLVE skips it, and `anvil-container-status` sets it so a question about this machine cannot pull gigabytes to answer itself. The identity claim in the design doc is qualified accordingly: presence implies the current inputs by construction only for an image built here; for a resolved one the claim rests on the registry's tag immutability. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae262693-67bf-46d4-a5af-df8da17c2553 --- .anvil.lock | 4 +- crates/cargo-anvil/README.md | 41 +++++-- crates/cargo-anvil/docs/design/containers.md | 73 +++++++++-- .../src/anvil/artifacts/container.rs | 64 ++++++++++ crates/cargo-anvil/src/lib.rs | 39 ++++-- .../templates/justfiles/anvil/container.just | 113 +++++++++++++++--- .../snapshots/snapshots__ado_backend.snap | 113 +++++++++++++++--- .../snapshots/snapshots__github_backend.snap | 113 +++++++++++++++--- .../snapshots/snapshots__local_only.snap | 113 +++++++++++++++--- justfiles/anvil/container.just | 113 +++++++++++++++--- 10 files changed, 667 insertions(+), 119 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 817ab32a..6d2b2b54 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:970f6a55ed74190d1ef848fc016c529027fd016037e36f09a6027c30d2f70c05" +catalog_checksum = "sha256:2254026e697c7ed4cf66141961d6992ae203fc58fed8b3dc342b4953279da66e" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:1195c884106d01876a234195d70cfe7eab39e41443b615504eeb30e6d137f4aa" +checksum = "sha256:ea4239d0d02d42624d3edd485b34aba473fd5cf6333eef5bec83527adb09189f" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 02b16e34..64a9e3e4 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -130,10 +130,17 @@ are substantially slower. The tag *is* a SHA-256 over the inputs that define the image: the Dockerfile and its ignore file, `rust-toolchain.toml`, the optional -credential hook, and the generated recipe tree. Presence therefore implies -freshness — a changed tool pin names a tag that cannot already exist, so a -build follows. There is no staleness check because there is nothing to -check. +hook, and the generated recipe tree. A changed tool pin names a tag that +cannot already exist, so a build follows. There is no staleness check +because there is nothing to check: an image built here is, by +construction, built from the current inputs. An image *fetched* by the +resolve hook only claims as much — the hash is over source files and +cannot be re-derived from layers — so that claim rests on the registry it +came from having immutable tags and restricted push. + +`anvil-container-tag` prints the reference without building it, and is the +single place the hash is computed, so a publisher can tag an image with +exactly the reference a consumer will later look up. #### Controls @@ -141,21 +148,24 @@ check. |--------|------| |`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`. A host property; never committed.| |`ANVIL_CONTAINER_NO_REBUILD=1`|Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure.| -|`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves.| +|`ANVIL_CONTAINER_NO_RESOLVE=1`|Skip the resolve hook, so a query never pulls.| +|`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves, ignoring the hook.| |`ANVIL_IN_CONTAINER=1`|Set inside the image; makes a nested invocation run natively.| -Supporting recipes: `anvil-container-status`, `anvil-container-rebuild`, -and `anvil-container-down` (removes this repository’s cache volumes). +Supporting recipes: `anvil-container-tag`, `anvil-container-status`, +`anvil-container-rebuild`, and `anvil-container-down` (removes this +repository’s cache volumes). -#### Credentials +#### The hook -crates.io needs none, so the public catalog emits no credential plumbing. +crates.io needs none of this, so the public catalog emits no hook at all. A repository or a downstream catalog that needs one adds `.anvil/container/hooks.ps1`, which the recipe loads when present: ```powershell -function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } -function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } +function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +function Anvil-ResolveImage { param($tag) (fetch-a-prebuilt-image $tag) } ``` Build secrets are passed to `BuildKit` by environment variable name, so a @@ -165,6 +175,13 @@ reason. An empty value is a hard error, because a build that quietly proceeded without its credential would install a reduced tool set and then be tagged with the hash a credentialed build produces. +`Anvil-ResolveImage` is offered the tag when nothing local matches, and +returns the reference it made available — a registry reference, not a +local re-tag, so the run stays honest about where the image came from. It +is verified before use and every failure falls through to a local build: +a publisher that has not caught up must not block the change it has not +caught up with. + The hook runs on the host with the developer’s permissions, before any container isolation. Only run one from a repository or catalog you trust. @@ -394,7 +411,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb80OmqCpM-PEb7-L3mxqzrXcbqBEF0vC0wXgbCT_rJDiFv7RhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb1zZ5432Pir4bA1QzrWQmGqEbunlSb9PrlecbuUcof8dw9UJhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 4481dba2..2f6fcccc 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -71,6 +71,7 @@ names it. | Recipe | Purpose | | --- | --- | | `just anvil-container ` | Run any anvil recipe in the image. No argument opens an interactive shell. | +| `just anvil-container-tag` | Print the image reference for the current inputs, without building it. | | `just anvil-container-status` | Report the engine, the image reference, and whether it is present. | | `just anvil-container-rebuild` | Rebuild ignoring every cached layer. | | `just anvil-container-down` | Remove this repository's cache volumes. The image is left in place. | @@ -90,21 +91,46 @@ Hashed: the `Dockerfile`, its ignore file, `rust-toolchain.toml`, `hooks.ps1` wh `justfiles/anvil/` except `container.just` itself — hashing the driver would make the tag depend on the tag. Change any of them and the tag names an image that cannot already exist, so a build follows. Change nothing and the -tag resolves instantly. There is no staleness check because there is nothing to check: an image that is present is, -by construction, built from the current inputs. +tag resolves instantly. There is no staleness check because there is nothing to check. + +What "present" proves depends on where the image came from: + +- **Built here** — presence implies the current inputs, by construction. Nothing else could have produced that tag on + this machine. +- **Resolved through the hook** (§5.1) — the image only *claims* those inputs. The hash is over source files and + cannot be re-derived from layers, so the claim rests on the registry it came from: immutable tags, and push + restricted to the identity that builds them. A registry where anyone can overwrite a tag makes the tag meaningless. + +`just anvil-container-tag` prints the reference without building it. It is the single place the hash is computed — +everything else asks it — which is what lets a publisher tag an image with exactly the reference a consumer will +later look up. | Variable | Effect | | --- | --- | | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is missing. Distinguishes a cache miss from a build failure. | -| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves. What `anvil-container-rebuild` sets. | +| `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook. What `anvil-container-status` sets, so a query never pulls. | +| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves, and ignore the hook. What `anvil-container-rebuild` sets. | The hook's *output* is deliberately not hashed: a credential must never influence a tag. -## 5. Credentials — the hook +## 5. The hook -crates.io needs none, so nothing is emitted by default. A repository or a downstream catalog that needs credentials -adds `.anvil/container/hooks.ps1`, which the recipe loads whenever it is present, regardless of who put it there: +`.anvil/container/hooks.ps1` is optional and loaded by path, not by provenance: the recipe sources it whenever it is +present, whether a repository wrote it or a downstream catalog shipped it. It supplies the two things the engine +cannot know — credentials, and where a prebuilt image might come from. + +| Function | When | Returns | +| --- | --- | --- | +| `Anvil-PreBuild` | before a build | `@{ Secrets = @{ id = value } }` | +| `Anvil-PreRun` | before a run | `@{ Env = @{ NAME = value } }` | +| `Anvil-ResolveImage $tag` | before a build, after the local check | an image reference, or nothing | + +All three are optional. + +### 5.1 Credentials + +crates.io needs none, so nothing is emitted by default: ```powershell function Anvil-PreBuild { @@ -116,7 +142,7 @@ function Anvil-PreRun { } ``` -Both functions are optional. `Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` +`Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` at run time. In both cases the value is handed over **by environment variable name**, so it never appears in the host's process command line — where endpoint telemetry records and retains it far longer than a short-lived token is meant to live — and never touches disk. The engine keeps build secrets out of every image layer. When the engine is @@ -142,6 +168,35 @@ from a repository or catalog you trust. Everything inside the container then run so a forwarded credential is reachable by anything the checks execute, including dependency build scripts and proc macros. Keep the set narrow and the token short-lived. +### 5.2 Resolving a prebuilt image + +When nothing local matches the tag, `Anvil-ResolveImage` is offered the reference before a build starts. A catalog +that publishes images implements it; everyone else does not, and the build proceeds exactly as before. + +```powershell +function Anvil-ResolveImage($tag) { + $remote = "myregistry.azurecr.io/anvil:$($tag.Split(':')[-1])" + az acr login --name myregistry | Out-Null + docker pull $remote | Out-Null + if ($LASTEXITCODE -eq 0) { $remote } +} +``` + +Three properties, none of them incidental: + +- **It returns the reference it fetched; it does not re-tag to the local name.** A local tag asserts "built here from + these inputs"; a fetched image only claims that (§4). Keeping the registry reference keeps the run honest about + where its image came from. +- **The reference is verified before use.** The run is `--pull=never`, so a hook that reported an image it did not + actually fetch would otherwise fail later and further from the cause. +- **Every failure is non-fatal.** A missing image, an expired credential, a broken hook — all fall through to a local + build, with the reason printed. A publisher that has not yet caught up must never block the developer whose change + it has not caught up with. + +Because the tag is content-addressed, a publisher and a consumer arrive at the same reference independently: no +`latest`, no digest pin to maintain, and no coordination beyond the naming scheme. +`ANVIL_CONTAINER_NO_RESOLVE=1` skips this step entirely. + ## 6. Host setup anvil installs nothing. It calls the engine you selected and lets that engine's own diagnostics @@ -270,8 +325,8 @@ which would make every cached image a potential lie. ## 8. Limits - Linux-only, `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. -- Local-only: no registry integration, no push, no promotion. The image is built and consumed locally, which is why - it needs no published artifact to exist. +- The engine never pushes and never promotes: it builds, and it may accept an image a hook fetched (§5.2). Publishing + is somebody else's job, and a repository that implements no hook needs no published artifact to exist. - A repository-owned `rust-toolchain.toml` is required — it is both what the image installs and part of what names it. - The first build takes several minutes: it installs a toolchain and the whole pinned tool catalog. Later runs reuse it until an input changes. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 652e0a40..cae32395 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -161,6 +161,7 @@ mod tests { fn recipe_exposes_the_documented_surface() { for expected in [ "anvil-container *target:", + "anvil-container-tag:", "anvil-container-status:", "anvil-container-rebuild:", "anvil-container-down:", @@ -169,6 +170,69 @@ mod tests { } } + #[test] + fn the_tag_is_computed_in_exactly_one_place() { + // `anvil-container-tag` is public so a publisher can name the image it + // is about to build. That only holds while it is the same computation + // the consumer performs: a second copy of the hash would let the two + // drift and turn a published tag into a claim nobody checks. + assert_eq!( + RECIPE.matches("SHA256]::HashData").count(), + 1, + "the content hash must be computed once, by anvil-container-tag" + ); + assert!( + RECIPE.contains("$image = (just anvil-container-tag).Trim()"), + "the resolver must ask anvil-container-tag rather than recompute" + ); + } + + #[test] + fn resolution_is_attempted_before_building_and_never_fatal() { + // Order: local image, then the hook, then a build. Resolving sits + // inside the cache guard (NO_CACHE must defeat a remote cache too) and + // before the NO_REBUILD guard, because fetching is not building. + let inspect = RECIPE.find("image inspect $image").expect("the local check must exist"); + let resolve = RECIPE.find("Anvil-ResolveImage $image").expect("the resolve call must exist"); + let build = RECIPE.find("anvil: building $image").expect("the build must exist"); + assert!( + inspect < resolve && resolve < build, + "resolve belongs between the local check and the build" + ); + + let no_rebuild = RECIPE + .find("if ($env:ANVIL_CONTAINER_NO_REBUILD -eq '1') {") + .expect("the no-rebuild guard must exist"); + assert!(resolve < no_rebuild, "resolving is not building, so NO_REBUILD must not block it"); + + // A publisher that has not caught up must not stop the developer who + // made the change, so every failure falls through to a build. + assert!(RECIPE.contains("anvil: Anvil-ResolveImage failed:")); + assert!(RECIPE.contains("anvil: nothing resolved; building locally")); + } + + #[test] + fn a_resolved_reference_is_verified_before_it_is_used() { + // The run is `--pull=never`, so a hook that reports a reference it did + // not actually fetch would fail later and further from the cause. + let resolve = RECIPE.find("Anvil-ResolveImage $image").expect("the resolve call must exist"); + let verify = RECIPE[resolve..] + .find("image inspect $resolved") + .expect("a resolved reference must be inspected before use"); + let accept = RECIPE[resolve..] + .find("Write-Output $resolved") + .expect("a resolved reference must be returned"); + assert!(verify < accept, "verify the resolved reference before returning it"); + } + + #[test] + fn a_query_never_pulls() { + // Resolving can mean pulling gigabytes; `anvil-container-status` asks + // about this machine and must not reach a registry to answer. + assert!(RECIPE.contains("$env:ANVIL_CONTAINER_NO_RESOLVE = '1'")); + assert!(RECIPE.contains("$env:ANVIL_CONTAINER_NO_RESOLVE -ne '1'")); + } + #[test] fn every_interpolation_into_powershell_is_escaped() { // A `just` value pasted raw into a '…' literal ends the string on an diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 040529e6..b7e6bc70 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -129,10 +129,17 @@ //! //! The tag *is* a SHA-256 over the inputs that define the image: the //! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional -//! credential hook, and the generated recipe tree. Presence therefore implies -//! freshness — a changed tool pin names a tag that cannot already exist, so a -//! build follows. There is no staleness check because there is nothing to -//! check. +//! hook, and the generated recipe tree. A changed tool pin names a tag that +//! cannot already exist, so a build follows. There is no staleness check +//! because there is nothing to check: an image built here is, by +//! construction, built from the current inputs. An image *fetched* by the +//! resolve hook only claims as much — the hash is over source files and +//! cannot be re-derived from layers — so that claim rests on the registry it +//! came from having immutable tags and restricted push. +//! +//! `anvil-container-tag` prints the reference without building it, and is the +//! single place the hash is computed, so a publisher can tag an image with +//! exactly the reference a consumer will later look up. //! //! ### Controls //! @@ -140,21 +147,24 @@ //! |---|---| //! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. A host property; never committed. | //! | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure. | -//! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves. | +//! | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | +//! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves, ignoring the hook. | //! | `ANVIL_IN_CONTAINER=1` | Set inside the image; makes a nested invocation run natively. | //! -//! Supporting recipes: `anvil-container-status`, `anvil-container-rebuild`, -//! and `anvil-container-down` (removes this repository's cache volumes). +//! Supporting recipes: `anvil-container-tag`, `anvil-container-status`, +//! `anvil-container-rebuild`, and `anvil-container-down` (removes this +//! repository's cache volumes). //! -//! ### Credentials +//! ### The hook //! -//! crates.io needs none, so the public catalog emits no credential plumbing. +//! crates.io needs none of this, so the public catalog emits no hook at all. //! A repository or a downstream catalog that needs one adds //! `.anvil/container/hooks.ps1`, which the recipe loads when present: //! //! ```powershell -//! function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } -//! function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +//! function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } +//! function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +//! function Anvil-ResolveImage { param($tag) (fetch-a-prebuilt-image $tag) } //! ``` //! //! Build secrets are passed to `BuildKit` by environment variable name, so a @@ -164,6 +174,13 @@ //! proceeded without its credential would install a reduced tool set and then //! be tagged with the hash a credentialed build produces. //! +//! `Anvil-ResolveImage` is offered the tag when nothing local matches, and +//! returns the reference it made available — a registry reference, not a +//! local re-tag, so the run stays honest about where the image came from. It +//! is verified before use and every failure falls through to a local build: +//! a publisher that has not caught up must not block the change it has not +//! caught up with. +//! //! The hook runs on the host with the developer's permissions, before any //! container isolation. Only run one from a repository or catalog you trust. //! diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 8f4b34ef..d198f4e7 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -95,7 +95,7 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Resolve the exec image reference, building it if it is not already present. +# Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its # ignore file, the pinned toolchain, the optional hook, and the generated @@ -104,20 +104,17 @@ _anvil-container-path host_path: # tool recipes. Only this driver is excluded, since hashing it would make the # tag depend on the tag. # -# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache -# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds -# a tag that already resolves, for the cases a content hash cannot see: a moved -# upstream package, or a base layer that changed behind its digest. -[private] +# This is the only recipe that computes the reference; everything else asks it. +# It is public because a publisher needs the tag before there is an image to +# inspect: a pipeline that builds the image tags the result with exactly the +# reference a consumer will later compute, which is what lets presence be +# checked without a second source of truth. + +# Print the exec image reference for the current inputs, without building it. +[group("anvil-container")] [script("pwsh", "-NoProfile")] -_anvil-container-image: +anvil-container-tag: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineCmd = $engine -split '\|' - $engineExe = $engineCmd[0] - $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -160,7 +157,39 @@ _anvil-container-image: # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) - $image = '{{anvil_container_name}}:' + $imageId + Write-Output ('{{anvil_container_name}}:' + $imageId) + +# Resolve the exec image, building it if it is neither present nor resolvable. +# +# Three steps, in order: a local image under the computed tag, then the +# optional `Anvil-ResolveImage` hook (a registry, typically), then a build. +# Resolution comes before the NO_REBUILD guard because fetching a published +# image is not building one. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_RESOLVE=1 skips +# the hook, so a query stays a query -- resolving can mean pulling gigabytes. +# ANVIL_CONTAINER_NO_CACHE=1 rebuilds a tag that already resolves, for the cases +# a content hash cannot see: a moved upstream package, or a base layer that +# changed behind its digest. It skips the hook too -- "ignore what is cached" +# has to mean the remote cache as well, or a rebuild would be undone by the +# next pull. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) + + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $image = (just anvil-container-tag).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -168,6 +197,52 @@ _anvil-container-image: Write-Output $image exit 0 } + + # Nothing local. Give the optional hook a chance to fetch a published + # image built from these same inputs -- a registry, typically. + # + # The hook returns the reference it made available, and we run that + # reference rather than re-tagging it to the local name: a local tag + # asserts "built here from these inputs", and a fetched image only + # *claims* that, since the hash is over source files and cannot be + # re-derived from layers. Whether that claim holds is a property of the + # registry (immutable tags, restricted push), not of anything this + # recipe can check, so the reference stays honest about where it came + # from. + # + # Every failure here is non-fatal: a missing image, an expired + # credential and a broken hook all fall through to a build, which is + # slower but always correct. A publisher that has not yet caught up + # with a change must not stop the developer who made it. + $hookPath = Join-Path $repoRoot $hookRel + if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") + $resolved = $null + try { + $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # Verify rather than trust: the run is `--pull=never`, so a + # reference the hook reported but did not actually fetch + # would fail later, further from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 + } + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + } + } } # Checked outside the cache guard, so the two variables compose: a caller @@ -356,8 +431,9 @@ anvil-container *target: } try { - # --pull=never: the tag names locally-built content, so a miss is a bug - # to surface rather than an invitation to fetch something unrelated. + # --pull=never: the reference names content that is already here, either + # built locally or fetched by the resolve hook, so a miss is a bug to + # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is @@ -388,14 +464,17 @@ anvil-container-status: # NO_REBUILD turns the resolve into a pure query: report the state instead # of silently spending several minutes building from a status command. + # NO_RESOLVE is the same argument applied to the hook, which would otherwise + # pull gigabytes to answer a question about the local machine. $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $env:ANVIL_CONTAINER_NO_RESOLVE = '1' $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 $present = $LASTEXITCODE -eq 0 if ($image) { Write-Output "image: $image" } if ($present) { Write-Output "status: present and current" } else { - Write-Output "status: no image matches the current inputs (it will be built on the next run)" + Write-Output "status: not present locally (the next run resolves or builds it)" } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index ae64ca11..511b4c85 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3573,7 +3573,7 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Resolve the exec image reference, building it if it is not already present. +# Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its # ignore file, the pinned toolchain, the optional hook, and the generated @@ -3582,20 +3582,17 @@ _anvil-container-path host_path: # tool recipes. Only this driver is excluded, since hashing it would make the # tag depend on the tag. # -# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache -# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds -# a tag that already resolves, for the cases a content hash cannot see: a moved -# upstream package, or a base layer that changed behind its digest. -[private] +# This is the only recipe that computes the reference; everything else asks it. +# It is public because a publisher needs the tag before there is an image to +# inspect: a pipeline that builds the image tags the result with exactly the +# reference a consumer will later compute, which is what lets presence be +# checked without a second source of truth. + +# Print the exec image reference for the current inputs, without building it. +[group("anvil-container")] [script("pwsh", "-NoProfile")] -_anvil-container-image: +anvil-container-tag: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineCmd = $engine -split '\|' - $engineExe = $engineCmd[0] - $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -3638,7 +3635,39 @@ _anvil-container-image: # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) - $image = '{{anvil_container_name}}:' + $imageId + Write-Output ('{{anvil_container_name}}:' + $imageId) + +# Resolve the exec image, building it if it is neither present nor resolvable. +# +# Three steps, in order: a local image under the computed tag, then the +# optional `Anvil-ResolveImage` hook (a registry, typically), then a build. +# Resolution comes before the NO_REBUILD guard because fetching a published +# image is not building one. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_RESOLVE=1 skips +# the hook, so a query stays a query -- resolving can mean pulling gigabytes. +# ANVIL_CONTAINER_NO_CACHE=1 rebuilds a tag that already resolves, for the cases +# a content hash cannot see: a moved upstream package, or a base layer that +# changed behind its digest. It skips the hook too -- "ignore what is cached" +# has to mean the remote cache as well, or a rebuild would be undone by the +# next pull. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) + + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $image = (just anvil-container-tag).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -3646,6 +3675,52 @@ _anvil-container-image: Write-Output $image exit 0 } + + # Nothing local. Give the optional hook a chance to fetch a published + # image built from these same inputs -- a registry, typically. + # + # The hook returns the reference it made available, and we run that + # reference rather than re-tagging it to the local name: a local tag + # asserts "built here from these inputs", and a fetched image only + # *claims* that, since the hash is over source files and cannot be + # re-derived from layers. Whether that claim holds is a property of the + # registry (immutable tags, restricted push), not of anything this + # recipe can check, so the reference stays honest about where it came + # from. + # + # Every failure here is non-fatal: a missing image, an expired + # credential and a broken hook all fall through to a build, which is + # slower but always correct. A publisher that has not yet caught up + # with a change must not stop the developer who made it. + $hookPath = Join-Path $repoRoot $hookRel + if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") + $resolved = $null + try { + $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # Verify rather than trust: the run is `--pull=never`, so a + # reference the hook reported but did not actually fetch + # would fail later, further from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 + } + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + } + } } # Checked outside the cache guard, so the two variables compose: a caller @@ -3834,8 +3909,9 @@ anvil-container *target: } try { - # --pull=never: the tag names locally-built content, so a miss is a bug - # to surface rather than an invitation to fetch something unrelated. + # --pull=never: the reference names content that is already here, either + # built locally or fetched by the resolve hook, so a miss is a bug to + # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is @@ -3866,14 +3942,17 @@ anvil-container-status: # NO_REBUILD turns the resolve into a pure query: report the state instead # of silently spending several minutes building from a status command. + # NO_RESOLVE is the same argument applied to the hook, which would otherwise + # pull gigabytes to answer a question about the local machine. $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $env:ANVIL_CONTAINER_NO_RESOLVE = '1' $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 $present = $LASTEXITCODE -eq 0 if ($image) { Write-Output "image: $image" } if ($present) { Write-Output "status: present and current" } else { - Write-Output "status: no image matches the current inputs (it will be built on the next run)" + Write-Output "status: not present locally (the next run resolves or builds it)" } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 80d91aef..7ccb2150 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3494,7 +3494,7 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Resolve the exec image reference, building it if it is not already present. +# Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its # ignore file, the pinned toolchain, the optional hook, and the generated @@ -3503,20 +3503,17 @@ _anvil-container-path host_path: # tool recipes. Only this driver is excluded, since hashing it would make the # tag depend on the tag. # -# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache -# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds -# a tag that already resolves, for the cases a content hash cannot see: a moved -# upstream package, or a base layer that changed behind its digest. -[private] +# This is the only recipe that computes the reference; everything else asks it. +# It is public because a publisher needs the tag before there is an image to +# inspect: a pipeline that builds the image tags the result with exactly the +# reference a consumer will later compute, which is what lets presence be +# checked without a second source of truth. + +# Print the exec image reference for the current inputs, without building it. +[group("anvil-container")] [script("pwsh", "-NoProfile")] -_anvil-container-image: +anvil-container-tag: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineCmd = $engine -split '\|' - $engineExe = $engineCmd[0] - $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -3559,7 +3556,39 @@ _anvil-container-image: # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) - $image = '{{anvil_container_name}}:' + $imageId + Write-Output ('{{anvil_container_name}}:' + $imageId) + +# Resolve the exec image, building it if it is neither present nor resolvable. +# +# Three steps, in order: a local image under the computed tag, then the +# optional `Anvil-ResolveImage` hook (a registry, typically), then a build. +# Resolution comes before the NO_REBUILD guard because fetching a published +# image is not building one. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_RESOLVE=1 skips +# the hook, so a query stays a query -- resolving can mean pulling gigabytes. +# ANVIL_CONTAINER_NO_CACHE=1 rebuilds a tag that already resolves, for the cases +# a content hash cannot see: a moved upstream package, or a base layer that +# changed behind its digest. It skips the hook too -- "ignore what is cached" +# has to mean the remote cache as well, or a rebuild would be undone by the +# next pull. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) + + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $image = (just anvil-container-tag).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -3567,6 +3596,52 @@ _anvil-container-image: Write-Output $image exit 0 } + + # Nothing local. Give the optional hook a chance to fetch a published + # image built from these same inputs -- a registry, typically. + # + # The hook returns the reference it made available, and we run that + # reference rather than re-tagging it to the local name: a local tag + # asserts "built here from these inputs", and a fetched image only + # *claims* that, since the hash is over source files and cannot be + # re-derived from layers. Whether that claim holds is a property of the + # registry (immutable tags, restricted push), not of anything this + # recipe can check, so the reference stays honest about where it came + # from. + # + # Every failure here is non-fatal: a missing image, an expired + # credential and a broken hook all fall through to a build, which is + # slower but always correct. A publisher that has not yet caught up + # with a change must not stop the developer who made it. + $hookPath = Join-Path $repoRoot $hookRel + if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") + $resolved = $null + try { + $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # Verify rather than trust: the run is `--pull=never`, so a + # reference the hook reported but did not actually fetch + # would fail later, further from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 + } + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + } + } } # Checked outside the cache guard, so the two variables compose: a caller @@ -3755,8 +3830,9 @@ anvil-container *target: } try { - # --pull=never: the tag names locally-built content, so a miss is a bug - # to surface rather than an invitation to fetch something unrelated. + # --pull=never: the reference names content that is already here, either + # built locally or fetched by the resolve hook, so a miss is a bug to + # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is @@ -3787,14 +3863,17 @@ anvil-container-status: # NO_REBUILD turns the resolve into a pure query: report the state instead # of silently spending several minutes building from a status command. + # NO_RESOLVE is the same argument applied to the hook, which would otherwise + # pull gigabytes to answer a question about the local machine. $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $env:ANVIL_CONTAINER_NO_RESOLVE = '1' $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 $present = $LASTEXITCODE -eq 0 if ($image) { Write-Output "image: $image" } if ($present) { Write-Output "status: present and current" } else { - Write-Output "status: no image matches the current inputs (it will be built on the next run)" + Write-Output "status: not present locally (the next run resolves or builds it)" } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 88c7a820..4909c3a1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2313,7 +2313,7 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Resolve the exec image reference, building it if it is not already present. +# Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its # ignore file, the pinned toolchain, the optional hook, and the generated @@ -2322,20 +2322,17 @@ _anvil-container-path host_path: # tool recipes. Only this driver is excluded, since hashing it would make the # tag depend on the tag. # -# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache -# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds -# a tag that already resolves, for the cases a content hash cannot see: a moved -# upstream package, or a base layer that changed behind its digest. -[private] +# This is the only recipe that computes the reference; everything else asks it. +# It is public because a publisher needs the tag before there is an image to +# inspect: a pipeline that builds the image tags the result with exactly the +# reference a consumer will later compute, which is what lets presence be +# checked without a second source of truth. + +# Print the exec image reference for the current inputs, without building it. +[group("anvil-container")] [script("pwsh", "-NoProfile")] -_anvil-container-image: +anvil-container-tag: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineCmd = $engine -split '\|' - $engineExe = $engineCmd[0] - $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -2378,7 +2375,39 @@ _anvil-container-image: # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) - $image = '{{anvil_container_name}}:' + $imageId + Write-Output ('{{anvil_container_name}}:' + $imageId) + +# Resolve the exec image, building it if it is neither present nor resolvable. +# +# Three steps, in order: a local image under the computed tag, then the +# optional `Anvil-ResolveImage` hook (a registry, typically), then a build. +# Resolution comes before the NO_REBUILD guard because fetching a published +# image is not building one. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_RESOLVE=1 skips +# the hook, so a query stays a query -- resolving can mean pulling gigabytes. +# ANVIL_CONTAINER_NO_CACHE=1 rebuilds a tag that already resolves, for the cases +# a content hash cannot see: a moved upstream package, or a base layer that +# changed behind its digest. It skips the hook too -- "ignore what is cached" +# has to mean the remote cache as well, or a rebuild would be undone by the +# next pull. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) + + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $image = (just anvil-container-tag).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -2386,6 +2415,52 @@ _anvil-container-image: Write-Output $image exit 0 } + + # Nothing local. Give the optional hook a chance to fetch a published + # image built from these same inputs -- a registry, typically. + # + # The hook returns the reference it made available, and we run that + # reference rather than re-tagging it to the local name: a local tag + # asserts "built here from these inputs", and a fetched image only + # *claims* that, since the hash is over source files and cannot be + # re-derived from layers. Whether that claim holds is a property of the + # registry (immutable tags, restricted push), not of anything this + # recipe can check, so the reference stays honest about where it came + # from. + # + # Every failure here is non-fatal: a missing image, an expired + # credential and a broken hook all fall through to a build, which is + # slower but always correct. A publisher that has not yet caught up + # with a change must not stop the developer who made it. + $hookPath = Join-Path $repoRoot $hookRel + if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") + $resolved = $null + try { + $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # Verify rather than trust: the run is `--pull=never`, so a + # reference the hook reported but did not actually fetch + # would fail later, further from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 + } + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + } + } } # Checked outside the cache guard, so the two variables compose: a caller @@ -2574,8 +2649,9 @@ anvil-container *target: } try { - # --pull=never: the tag names locally-built content, so a miss is a bug - # to surface rather than an invitation to fetch something unrelated. + # --pull=never: the reference names content that is already here, either + # built locally or fetched by the resolve hook, so a miss is a bug to + # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is @@ -2606,14 +2682,17 @@ anvil-container-status: # NO_REBUILD turns the resolve into a pure query: report the state instead # of silently spending several minutes building from a status command. + # NO_RESOLVE is the same argument applied to the hook, which would otherwise + # pull gigabytes to answer a question about the local machine. $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $env:ANVIL_CONTAINER_NO_RESOLVE = '1' $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 $present = $LASTEXITCODE -eq 0 if ($image) { Write-Output "image: $image" } if ($present) { Write-Output "status: present and current" } else { - Write-Output "status: no image matches the current inputs (it will be built on the next run)" + Write-Output "status: not present locally (the next run resolves or builds it)" } exit 0 diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 8f4b34ef..d198f4e7 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -95,7 +95,7 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Resolve the exec image reference, building it if it is not already present. +# Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its # ignore file, the pinned toolchain, the optional hook, and the generated @@ -104,20 +104,17 @@ _anvil-container-path host_path: # tool recipes. Only this driver is excluded, since hashing it would make the # tag depend on the tag. # -# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache -# miss is told apart from a build failure. ANVIL_CONTAINER_NO_CACHE=1 rebuilds -# a tag that already resolves, for the cases a content hash cannot see: a moved -# upstream package, or a base layer that changed behind its digest. -[private] +# This is the only recipe that computes the reference; everything else asks it. +# It is public because a publisher needs the tag before there is an image to +# inspect: a pipeline that builds the image tags the result with exactly the +# reference a consumer will later compute, which is what lets presence be +# checked without a second source of truth. + +# Print the exec image reference for the current inputs, without building it. +[group("anvil-container")] [script("pwsh", "-NoProfile")] -_anvil-container-image: +anvil-container-tag: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineCmd = $engine -split '\|' - $engineExe = $engineCmd[0] - $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' @@ -160,7 +157,39 @@ _anvil-container-image: # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) - $image = '{{anvil_container_name}}:' + $imageId + Write-Output ('{{anvil_container_name}}:' + $imageId) + +# Resolve the exec image, building it if it is neither present nor resolvable. +# +# Three steps, in order: a local image under the computed tag, then the +# optional `Anvil-ResolveImage` hook (a registry, typically), then a build. +# Resolution comes before the NO_REBUILD guard because fetching a published +# image is not building one. +# +# ANVIL_CONTAINER_NO_REBUILD=1 fails instead of building, which is how a cache +# miss is told apart from a build failure. ANVIL_CONTAINER_NO_RESOLVE=1 skips +# the hook, so a query stays a query -- resolving can mean pulling gigabytes. +# ANVIL_CONTAINER_NO_CACHE=1 rebuilds a tag that already resolves, for the cases +# a content hash cannot see: a moved upstream package, or a base layer that +# changed behind its digest. It skips the hook too -- "ignore what is cached" +# has to mean the remote cache as well, or a rebuild would be undone by the +# next pull. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-image: + $ErrorActionPreference = 'Stop' + $engine = (just _anvil-container-engine).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineCmd = $engine -split '\|' + $engineExe = $engineCmd[0] + $enginePrefix = @($engineCmd | Select-Object -Skip 1) + + $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' + $dockerfile = '.anvil/container/Dockerfile' + $hookRel = '.anvil/container/hooks.ps1' + + $image = (just anvil-container-tag).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -168,6 +197,52 @@ _anvil-container-image: Write-Output $image exit 0 } + + # Nothing local. Give the optional hook a chance to fetch a published + # image built from these same inputs -- a registry, typically. + # + # The hook returns the reference it made available, and we run that + # reference rather than re-tagging it to the local name: a local tag + # asserts "built here from these inputs", and a fetched image only + # *claims* that, since the hash is over source files and cannot be + # re-derived from layers. Whether that claim holds is a property of the + # registry (immutable tags, restricted push), not of anything this + # recipe can check, so the reference stays honest about where it came + # from. + # + # Every failure here is non-fatal: a missing image, an expired + # credential and a broken hook all fall through to a build, which is + # slower but always correct. A publisher that has not yet caught up + # with a change must not stop the developer who made it. + $hookPath = Join-Path $repoRoot $hookRel + if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") + $resolved = $null + try { + $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # Verify rather than trust: the run is `--pull=never`, so a + # reference the hook reported but did not actually fetch + # would fail later, further from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 + } + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + } + } } # Checked outside the cache guard, so the two variables compose: a caller @@ -356,8 +431,9 @@ anvil-container *target: } try { - # --pull=never: the tag names locally-built content, so a miss is a bug - # to surface rather than an invitation to fetch something unrelated. + # --pull=never: the reference names content that is already here, either + # built locally or fetched by the resolve hook, so a miss is a bug to + # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is @@ -388,14 +464,17 @@ anvil-container-status: # NO_REBUILD turns the resolve into a pure query: report the state instead # of silently spending several minutes building from a status command. + # NO_RESOLVE is the same argument applied to the hook, which would otherwise + # pull gigabytes to answer a question about the local machine. $env:ANVIL_CONTAINER_NO_REBUILD = '1' + $env:ANVIL_CONTAINER_NO_RESOLVE = '1' $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 $present = $LASTEXITCODE -eq 0 if ($image) { Write-Output "image: $image" } if ($present) { Write-Output "status: present and current" } else { - Write-Output "status: no image matches the current inputs (it will be built on the next run)" + Write-Output "status: not present locally (the next run resolves or builds it)" } exit 0 From 1a71b29b251aca4f8ad0070a8f9e53baf22ef6c0 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 13:30:59 +0200 Subject: [PATCH 11/81] fix(anvil): drop needless raw-string hashes in a container test clippy::needless_raw_string_hashes, denied workspace-wide. The assertion string contains no quote, so a plain raw string is enough. Mine for pushing without running clippy on that commit; the gate exists exactly for this. --- crates/cargo-anvil/src/anvil/artifacts/container.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 97d5f4b5..87c72561 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -216,7 +216,7 @@ mod tests { // BuildKit finds `.dockerignore` itself; buildah reads only // a context-root file, so without this the whole worktree is the build // context. - assert!(RECIPE.contains(r#"if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile'"#)); + assert!(RECIPE.contains(r"if ($engineCmd[-1] -eq 'podman') { $buildCmd += @('--ignorefile'")); } #[test] From a7c472d7e09d0cf18aab98d0f1d7bc281f022ec6 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 13:31:02 +0200 Subject: [PATCH 12/81] Revert "fix(anvil): defer the lints region to a crate that owns its lints" This reverts commit 3c8d4d70b97abd44e3636b63a37a55174de4fdcd. --- .../cargo-anvil/src/anvil/artifacts/region.rs | 2 +- crates/cargo-anvil/src/run.rs | 140 ++++-------------- 2 files changed, 28 insertions(+), 114 deletions(-) diff --git a/crates/cargo-anvil/src/anvil/artifacts/region.rs b/crates/cargo-anvil/src/anvil/artifacts/region.rs index 2c023a83..0deba72c 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/region.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/region.rs @@ -20,7 +20,7 @@ const WORKSPACE_LINTS_REGION_ID: &str = "anvil-workspace-lints"; /// Region id for crate-scope lints — used both for single-crate repos (full /// catalog) and for each member of a multi-crate workspace (`workspace = /// true`). -pub(crate) const CRATE_LINTS_REGION_ID: &str = "anvil-lints"; +const CRATE_LINTS_REGION_ID: &str = "anvil-lints"; /// Embedded body of the lint catalog, in dotted-key form (no table header). const LINTS_BODY: &str = include_str!("../../../templates/regions/cargo-lints-body.toml"); diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 8e32388b..7bb1f817 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -12,7 +12,7 @@ use std::path::Path; use ohno::{AppError, bail}; use tracing::info; -use crate::anvil::artifacts::region::{CRATE_LINTS_REGION_ID, DELTA_REGION_ID}; +use crate::anvil::artifacts::region::DELTA_REGION_ID; use crate::backend::{self, Backend}; use crate::catalog::Catalog; use crate::catalog::artifact::{Artifact, HostSelector, RegionSpec}; @@ -348,17 +348,20 @@ fn push_region_at( let host = resolve_existing_case_insensitive(repo_root, host); let current = hosts.get_or_read(repo_root, &host)?; let placement = region_placement(spec.id.as_str()); - let body = match region_body(current.as_deref(), spec) { - RegionBody::Managed => spec.body.as_str(), - RegionBody::Defer(note) => { - plan.note(note); + let body = match delta_region_body(current.as_deref(), spec) { + DeltaRegionBody::Managed => spec.body.as_str(), + DeltaRegionBody::PreserveRepositoryKey => { + plan.note( + "The repository's .delta.toml already defines top-level `trip_wire_patterns`; \ + the managed anvil-delta region was left empty. Remove the repository key to \ + adopt the managed trip-wire list.", + ); "" } - RegionBody::Malformed(reason) => { + DeltaRegionBody::Malformed(reason) => { plan.refusal(format!( - "Refused to manage {host} [{}] because the existing host could not \ - be safely inspected: {reason}. Other artifacts were still planned.", - spec.id.as_str() + "Refused to manage .delta.toml [anvil-delta] because the existing host could not \ + be safely inspected: {reason}. Other artifacts were still planned." )); plan.push(PlanItem::noop( Target::Region { @@ -402,74 +405,34 @@ fn region_placement(region_id: &str) -> RegionPlacement { } } -/// How a managed region's body resolves against what the host already -/// contains. -enum RegionBody { - /// Splice the catalog's body. +enum DeltaRegionBody { Managed, - /// Splice an empty region, and say why. The repository already defines the - /// key this region would own, and emitting both produces a file the - /// toolchain rejects. The empty region stays tracked, so the catalog body - /// is adopted automatically once the repository drops its own. - Defer(String), - /// The host could not be safely inspected. + PreserveRepositoryKey, Malformed(String), } -/// Resolve a region's body against its host, deferring where the repository -/// already owns the same key. -/// -/// Two regions can collide with repository-owned content, and in both cases a -/// naive splice yields a file that fails to parse rather than one that merely -/// looks odd: -/// -/// - `anvil-delta` against a top-level `trip_wire_patterns`, which would become -/// a duplicate TOML key. -/// - `anvil-lints` against a crate that declares its own `[lints.*]`, which -/// cargo rejects with "cannot override `workspace.lints` in `lints`" -- -/// taking down `cargo metadata`, and with it every check in the workspace, -/// not just the offending crate. -fn region_body(host_text: Option<&str>, spec: &RegionSpec) -> RegionBody { - let id = spec.id.as_str(); - if id != DELTA_REGION_ID && id != CRATE_LINTS_REGION_ID { - return RegionBody::Managed; +fn delta_region_body(host_text: Option<&str>, spec: &RegionSpec) -> DeltaRegionBody { + if spec.id.as_str() != DELTA_REGION_ID { + return DeltaRegionBody::Managed; } let Some(host_text) = host_text else { - return RegionBody::Managed; + return DeltaRegionBody::Managed; }; - // Inspect the host as it reads without anvil's own region, so a body - // anvil spliced on an earlier pass is never mistaken for repository - // content. - let without_region = match remove_region(host_text, id, spec.syntax) { + let without_region = match remove_region(host_text, spec.id.as_str(), spec.syntax) { Ok(without_region) => without_region, - Err(error) => return RegionBody::Malformed(format!("managed-region markers are malformed: {error}")), + Err(error) => return DeltaRegionBody::Malformed(format!("managed-region markers are malformed: {error}")), }; let document = match without_region.parse::() { Ok(document) => document, - Err(error) => return RegionBody::Malformed(format!("invalid TOML: {error}")), - }; - - if id == DELTA_REGION_ID { - if document.as_table().contains_key("trip_wire_patterns") { - return RegionBody::Defer( - "The repository's .delta.toml already defines top-level `trip_wire_patterns`; \ - the managed anvil-delta region was left empty. Remove the repository key to \ - adopt the managed trip-wire list." - .to_owned(), - ); + Err(error) => { + return DeltaRegionBody::Malformed(format!("invalid TOML: {error}")); } - return RegionBody::Managed; - } - - if document.as_table().contains_key("lints") { - return RegionBody::Defer( - "This crate declares its own `[lints]`; the managed anvil-lints region was left \ - empty. Cargo rejects a manifest carrying both, which would break `cargo metadata` \ - for the whole workspace. Remove the crate's own lints to adopt the catalog." - .to_owned(), - ); + }; + if document.as_table().contains_key("trip_wire_patterns") { + DeltaRegionBody::PreserveRepositoryKey + } else { + DeltaRegionBody::Managed } - RegionBody::Managed } /// Scan the previous manifest for entries that the active plan items @@ -1073,55 +1036,6 @@ mod tests { assert!(!second.plan.has_changes()); } - #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] - #[test] - fn member_declaring_its_own_lints_opts_out_of_the_managed_catalog() { - // A crate carrying its own `[lints.clippy]` must not also receive - // `[lints] workspace = true`. Cargo rejects that combination outright - // ("cannot override `workspace.lints` in `lints`"), which breaks - // `cargo metadata` and therefore every check in the workspace -- not - // only the offending crate. - let tmp = empty_workspace(); - let member = tmp.path().join("crates/alpha/Cargo.toml"); - fs::write( - &member, - "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n\n\ - [lints.clippy]\npedantic = { level = \"warn\", priority = -1 }\n", - ) - .unwrap(); - let args = Cli { - backends: vec![], - no_backends: true, - dry_run: false, - force: false, - }; - - let outcome = run_update(&Catalog::anvil(), &args, tmp.path()).unwrap(); - - let content = fs::read_to_string(&member).unwrap(); - let document: toml_edit::DocumentMut = content.parse().expect("member manifest must remain valid TOML"); - assert!( - document["lints"].get("clippy").is_some(), - "the crate's own lints must survive" - ); - assert!( - document["lints"].get("workspace").is_none(), - "anvil must not add `workspace = true` beside the crate's own lints" - ); - let region = find_region(&content, CRATE_LINTS_REGION_ID, CommentSyntax::Hash) - .unwrap() - .expect("the region stays tracked, so dropping the crate's lints adopts the catalog"); - assert!(region.is_empty(), "a crate that owns its lints opts out of the managed body"); - assert!( - outcome - .plan - .notes() - .iter() - .any(|note| note.contains("declares its own `[lints]`")), - "the opt-out must be visible in the plan summary" - ); - } - #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] #[test] fn existing_delta_trip_wires_are_preserved_without_duplicate_key() { From 12bdf47faad837b5eeeddc3d6330e352385b6126 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 18:17:40 +0200 Subject: [PATCH 13/81] docs(anvil): describe the container feature, not its development The container documentation had drifted into narrating how the feature was built: it defended design choices against alternatives readers never saw, referenced a configuration file that was never shipped, and cited the end-to-end suite as evidence. Rewrite both the design doc and the crate docs around what the feature does and how to use it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/cargo-anvil/README.md | 25 +- crates/cargo-anvil/docs/design/containers.md | 331 +++++++++---------- crates/cargo-anvil/src/lib.rs | 23 +- 3 files changed, 177 insertions(+), 202 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 64a9e3e4..7e367aeb 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -102,20 +102,21 @@ image installs the Rust toolchain and Cargo tools that this repository pins, by running `just anvil-setup` — the same recipe the checks use — so the container and the host agree on the toolset by construction. -There is no configuration file and no transparent routing: `just anvil-pr` -keeps running natively, and the container is reached only through the -explicit recipe. +`just anvil-pr` and every other recipe keep running natively; the container +is entered only through `anvil-container`, which takes any recipe name and +its arguments. ```text -just anvil-container anvil-clippy # one check -just anvil-container anvil-pr # the whole PR tier -just anvil-container # interactive shell +just anvil-container anvil-clippy # one check +just anvil-container anvil-pr # the whole PR tier +just anvil-container anvil-setup binstall # a recipe with an argument +just anvil-container # interactive shell ``` #### Prerequisites -* A container engine callable from the shell that runs `just`: Docker - (supported) or Podman (best-effort). On Windows that means Docker +* A container engine callable from the shell that runs `just`: Docker, or + Podman via `ANVIL_CONTAINER_ENGINE=podman`. On Windows that means Docker Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or Docker Engine installed only inside the default WSL distribution — no Windows CLI is needed in that last case, since anvil reaches the engine @@ -146,7 +147,7 @@ exactly the reference a consumer will later look up. |Variable|Effect| |--------|------| -|`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`. A host property; never committed.| +|`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`.| |`ANVIL_CONTAINER_NO_REBUILD=1`|Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure.| |`ANVIL_CONTAINER_NO_RESOLVE=1`|Skip the resolve hook, so a query never pulls.| |`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves, ignoring the hook.| @@ -158,8 +159,8 @@ repository’s cache volumes). #### The hook -crates.io needs none of this, so the public catalog emits no hook at all. -A repository or a downstream catalog that needs one adds +crates.io needs no credentials, so no hook is emitted by default. A +repository or a downstream catalog that needs one adds `.anvil/container/hooks.ps1`, which the recipe loads when present: ```powershell @@ -411,7 +412,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb1zZ5432Pir4bA1QzrWQmGqEbunlSb9PrlecbuUcof8dw9UJhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbbI8J5B4oqNYbCGDbwei2JEEbJ8_ukS4Unx8b6YMW7c9a-thhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 058043a0..33e4b86b 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -1,6 +1,6 @@ # Containers -Any generated recipe can run inside a pinned Linux image: +Any generated recipe can run inside a Linux image that carries exactly the toolchain and tools this repository pins: ```bash just anvil-container anvil-clippy # one check @@ -11,126 +11,129 @@ just anvil-container # interactive shell See also [design.md](./README.md) for the overall principles, [local.md](./local.md) for the recipe surface this wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstream fork uses. -- [1. Problem](#1-problem) -- [2. What this is](#2-what-this-is) -- [3. The two artifacts](#3-the-two-artifacts) +- [1. Why](#1-why) +- [2. How it runs](#2-how-it-runs) +- [3. What it adds to a repository](#3-what-it-adds-to-a-repository) - [4. Image identity](#4-image-identity) -- [5. Credentials — the hook](#5-credentials--the-hook) +- [5. The hook](#5-the-hook) - [6. Host setup](#6-host-setup) - [6.1 Docker](#61-docker) - [6.2 Podman](#62-podman) - [7. Customizing the image](#7-customizing-the-image) - [8. Limits](#8-limits) -## 1. Problem +## 1. Why -The local layer assumes a usable host toolchain ([design.md §3][design]: "the user owns it locally"). Two situations -break that assumption: +The local recipes assume a usable host toolchain — anvil does not install one ([design.md §3][design]: "the user owns +it locally"). Two situations break that assumption: -1. **Linux-on-Windows parity.** A developer on Windows cannot reproduce a Linux-only failure — a `cfg(unix)` path, a - Linux-specific lint, an `mmap`-shaped test — without a Linux box. -2. **Toolchain drift.** Even on Linux, the host toolset can differ from the one the checks expect, so a green local - run is not predictive. +1. **Linux-only failures on a Windows machine.** A `cfg(unix)` path, a Linux-specific lint, an `mmap`-shaped test: + none of them can be reproduced without a Linux environment. +2. **Toolchain drift.** Even on Linux, the installed toolset can differ from the one the checks expect, so a green + local run stops being predictive of a cloud one. -Both are solved the same way: run the recipe in an image whose toolchain and tools are the ones this repository pins. +Both are answered the same way: run the recipe in an image built from the repository's own pins. -## 2. What this is +## 2. How it runs -- **Explicit.** `just anvil-pr` runs natively, exactly as before. The container is reached through - `just anvil-container` or not at all. There is no PATH shim, no routing toggle, and no recipe that behaves - differently depending on where it runs. -- **Unconfigured.** There is no `anvil.toml`. Whether the artifacts are emitted is a catalog decision; the only - host-specific value is an environment variable read at run time. -- **Local.** Cloud workflows continue to run the recipes natively on their own pools. The image is pinned to be - *like* CI, not to *be* CI. -- **Additive.** The recipes it runs are byte-identical to the ones a native run uses. +`just anvil-pr` and every other recipe continue to run natively. The container is entered only through +`just anvil-container`, which takes any recipe name and its arguments: -## 3. The two artifacts - -```text -repo/ -├── justfiles/anvil/ -│ ├── container.just the anvil-container recipe and its helpers -│ └── … checks, groups, tiers (unchanged; run natively *inside* the image) -└── .anvil/container/ - ├── Dockerfile what the image contains - ├── Dockerfile.dockerignore what the build context admits - └── hooks.ps1 optional; credentials, not emitted by default +```bash +just anvil-container anvil-setup binstall ``` -`container.just` is generated and should not be edited. The `Dockerfile` pair is generated but **deliberately -editable**: anvil's drift handling preserves a repository's changes. +Inside the image, `ANVIL_IN_CONTAINER=1` is set, so a recipe that reaches `anvil-container` again runs natively +instead of nesting. The work happens once — one container per command, not one per check. -The image installs its tools by running `just anvil-setup` — the same recipe the checks use, from the same generated -pins. There is no second tool list to keep in step, so "the image has the right tools" is true by construction. It is -also why a tool-pin bump changes the image identity: `versions.just` is both what the image installs and part of what -names it. - -### Recipes +The recipes themselves are identical in both cases. Nothing behaves differently depending on where it runs, and no +wrapper shadows `just` on `PATH`. | Recipe | Purpose | | --- | --- | -| `just anvil-container ` | Run any anvil recipe in the image. No argument opens an interactive shell. | +| `just anvil-container [args…]` | Run a recipe in the image. No argument opens an interactive shell. | | `just anvil-container-tag` | Print the image reference for the current inputs, without building it. | | `just anvil-container-status` | Report the engine, the image reference, and whether it is present. | -| `just anvil-container-rebuild` | Rebuild ignoring every cached layer. | +| `just anvil-container-rebuild` | Rebuild from scratch, ignoring every cached layer. | | `just anvil-container-down` | Remove this repository's cache volumes. The image is left in place. | -`ANVIL_IN_CONTAINER=1` is set inside the image, so a nested invocation runs natively and the work happens exactly -once — one container per top-level command, not one per check. +The repository is mounted at `/workspace`, and the working directory is mapped to its in-container equivalent so +relative paths keep working from a subdirectory. The cargo and rustup homes live in named volumes, so the hot write +path never crosses the host boundary and the host's own toolchain is untouched. + +Cloud workflows are unaffected: they run the recipes natively on their own agents. The image is pinned to resemble +that environment, not to be it. + +## 3. What it adds to a repository + +```text +repo/ +├── justfiles/anvil/ +│ ├── container.just the anvil-container recipes +│ └── … checks, groups, tiers — run natively *inside* the image +└── .anvil/container/ + ├── Dockerfile what the image contains + ├── Dockerfile.dockerignore what the build context admits + └── hooks.ps1 optional; supplied by you or a catalog (§5) +``` + +`container.just` is generated and reconciled on every run; edits to it are replaced. The `Dockerfile` and its ignore +file are generated too, but they are meant to be edited — anvil's drift handling preserves a repository's changes to +them (§7). + +The image installs its tools by running `just anvil-setup`: the same recipe the checks use, reading the same +generated pins. There is no second list of tools to keep in step, which is also why bumping a tool pin changes the +image — `versions.just` is both what the image installs and part of what names it (§4). ## 4. Image identity -The tag **is** the hash of the inputs that define it: +The tag **is** the hash of the inputs that define the image: ```text anvil-:<16 hex characters> ``` Hashed: the `Dockerfile`, its ignore file, `rust-toolchain.toml`, `hooks.ps1` when present, and every `*.just` under -`justfiles/anvil/` except `container.just` itself — hashing the driver would make the tag depend on the tag. - -Change any of them and the tag names an image that cannot already exist, so a build follows. Change nothing and the -tag resolves instantly. There is no staleness check because there is nothing to check. +`justfiles/anvil/` — except `container.just` itself, since hashing the driver would make the tag depend on the tag. -What "present" proves depends on where the image came from: +Change any input and the tag names an image that cannot already exist, so a build follows. Change nothing and the tag +resolves immediately. There is no staleness check because there is no staleness: an image that is present was built +from the inputs that name it. -- **Built here** — presence implies the current inputs, by construction. Nothing else could have produced that tag on - this machine. -- **Resolved through the hook** (§5.1) — the image only *claims* those inputs. The hash is over source files and - cannot be re-derived from layers, so the claim rests on the registry it came from: immutable tags, and push - restricted to the identity that builds them. A registry where anyone can overwrite a tag makes the tag meaningless. +That guarantee is exact for an image built locally. An image fetched by the resolve hook (§5.2) only *claims* those +inputs — the hash is over source files and cannot be recomputed from layers — so the claim is only as good as the +registry it came from. Publish to one with immutable tags, and restrict push to the identity that builds them. -`just anvil-container-tag` prints the reference without building it. It is the single place the hash is computed — -everything else asks it — which is what lets a publisher tag an image with exactly the reference a consumer will -later look up. +`just anvil-container-tag` prints the reference without building anything. Every other recipe asks it, so a publisher +and a consumer compute the same reference independently: no `latest`, no digest to maintain by hand. | Variable | Effect | | --- | --- | | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is missing. Distinguishes a cache miss from a build failure. | -| `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook. What `anvil-container-status` sets, so a query never pulls. | -| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves, and ignore the hook. What `anvil-container-rebuild` sets. | +| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is missing, which distinguishes a cache miss from a build failure. | +| `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | +| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves, ignoring the hook. | -The hook's *output* is deliberately not hashed: a credential must never influence a tag. +Values a hook returns never enter the hash: a credential must not be able to influence a tag. ## 5. The hook -`.anvil/container/hooks.ps1` is optional and loaded by path, not by provenance: the recipe sources it whenever it is -present, whether a repository wrote it or a downstream catalog shipped it. It supplies the two things the engine -cannot know — credentials, and where a prebuilt image might come from. +`.anvil/container/hooks.ps1` supplies the two things anvil cannot know — credentials for a private feed, and where a +prebuilt image might come from. It is optional, and it is loaded by path rather than by provenance: the recipe sources +it whenever the file exists, whether a repository wrote it or a catalog shipped it. A repository can therefore try a +credential flow without forking anything. -| Function | When | Returns | +| Function | Called | Returns | | --- | --- | --- | | `Anvil-PreBuild` | before a build | `@{ Secrets = @{ id = value } }` | | `Anvil-PreRun` | before a run | `@{ Env = @{ NAME = value } }` | -| `Anvil-ResolveImage $tag` | before a build, after the local check | an image reference, or nothing | +| `Anvil-ResolveImage $tag` | before a build, when nothing local matches | an image reference, or nothing | All three are optional. ### 5.1 Credentials -crates.io needs none, so nothing is emitted by default: +crates.io needs none, so the default image ships no credential plumbing. An internal feed does: ```powershell function Anvil-PreBuild { @@ -138,40 +141,42 @@ function Anvil-PreBuild { } function Anvil-PreRun { - @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = "Bearer …" } } + @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = "Bearer $token" } } } ``` -`Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` -at run time. In both cases the value is handed over **by environment variable name**, so it never appears in the -host's process command line — where endpoint telemetry records and retains it far longer than a short-lived token is -meant to live — and never touches disk. The engine keeps build secrets out of every image layer. When the engine is -reached through WSL (§6.1), the names are exported with `WSLENV` so the value crosses the boundary without ever -becoming an argument. +Minting the value in a function, rather than reading it from a committed file or a declared variable, is the point: a +short-lived token has to be acquired at the moment it is used. + +`Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` at run time. In both cases the +value is handed to the engine **by variable name**, so it never appears in a command line — where endpoint telemetry +records and retains it far longer than the token is meant to live — and it never touches disk. Build secrets stay out +of every image layer. When the engine is reached through WSL (§6.1), the names are exported with `WSLENV` so the value +crosses that boundary the same way. -The corresponding `RUN` should declare the mount as required, which closes the same hole from the Dockerfile's side: +Declare the mount as required, which closes the same hole from the Dockerfile's side: ```dockerfile RUN --mount=type=secret,id=feed_token,required=true \ TOKEN="$(cat /run/secrets/feed_token)" … ``` -Anything the build *writes* with a secret is ordinary content: anvil's own Dockerfile deletes `credentials.toml` and -`.netrc` in the same layer as the install, and a replacement must do the same or the credential is baked into a layer. +Anything the build *writes* with a secret is ordinary content. Anvil's own Dockerfile deletes `credentials.toml` and +`.netrc` in the same layer as the install; a replacement must do the same, or the credential is baked into a layer. -**An empty value is a hard error.** BuildKit is not: `--secret id=t,env=UNSET` exits 0 having mounted an empty secret, -so the build would install a reduced tool set and be tagged with the *same* content hash a credentialed build -produces — and every later run would reuse the broken image. +**An empty value is a hard error.** BuildKit itself is not: `--secret id=t,env=UNSET` mounts an empty secret and +exits 0, so the build would install a reduced tool set and then be tagged with the same content hash a credentialed +build produces — and every later run would reuse that broken image. -**Trust.** The hook runs on the host, with the developer's permissions, before any container isolation. Only run one -from a repository or catalog you trust. Everything inside the container then runs as one user in one mount namespace, -so a forwarded credential is reachable by anything the checks execute, including dependency build scripts and proc -macros. Keep the set narrow and the token short-lived. +**Trust.** The hook runs on the host, with your permissions, before any container isolation. Only run one from a +repository or catalog you trust. Inside the container everything runs as one user in one mount namespace, so a +forwarded credential is readable by anything the checks execute, including dependency build scripts and proc macros. +Keep the set narrow and the token short-lived. -### 5.2 Resolving a prebuilt image +### 5.2 Prebuilt images -When nothing local matches the tag, `Anvil-ResolveImage` is offered the reference before a build starts. A catalog -that publishes images implements it; everyone else does not, and the build proceeds exactly as before. +When no local image matches the tag, `Anvil-ResolveImage` is offered that reference before a build starts. A catalog +that publishes images implements it; without one, the build proceeds as usual. ```powershell function Anvil-ResolveImage($tag) { @@ -182,75 +187,41 @@ function Anvil-ResolveImage($tag) { } ``` -Three properties, none of them incidental: +Three properties are worth knowing: -- **It returns the reference it fetched; it does not re-tag to the local name.** A local tag asserts "built here from - these inputs"; a fetched image only claims that (§4). Keeping the registry reference keeps the run honest about - where its image came from. -- **The reference is verified before use.** The run is `--pull=never`, so a hook that reported an image it did not - actually fetch would otherwise fail later and further from the cause. -- **Every failure is non-fatal.** A missing image, an expired credential, a broken hook — all fall through to a local - build, with the reason printed. A publisher that has not yet caught up must never block the developer whose change - it has not caught up with. +- **The returned reference is used as-is, not re-tagged to the local name.** A local tag asserts "built here from + these inputs"; a fetched image only claims it (§4). Keeping the registry reference keeps the run honest about where + the image came from. +- **The reference is verified before use.** Runs are `--pull=never`, so a hook that reported an image it had not + actually fetched would otherwise fail later and further from the cause. +- **Every failure falls through to a local build**, with the reason printed — a missing image, an expired credential, + a broken hook. A publisher that has not caught up with your change must never block you. -Because the tag is content-addressed, a publisher and a consumer arrive at the same reference independently: no -`latest`, no digest pin to maintain, and no coordination beyond the naming scheme. -`ANVIL_CONTAINER_NO_RESOLVE=1` skips this step entirely. +`ANVIL_CONTAINER_NO_RESOLVE=1` skips this step. ## 6. Host setup -anvil installs nothing. It calls the engine you selected and lets that engine's own diagnostics -surface when something is wrong — the one exception is a missing binary, which is reported with -the variable to set and a pointer here. +anvil installs nothing and manages no virtual machine. It calls the engine you selected and lets that engine's own +diagnostics surface; the one message it owns is for a missing binary, which names the variable to set and points +here. -The engine must be **callable from the shell that runs `just`**. On Windows there is one -exception, and it is automatic: if the engine is not on `PATH`, anvil retries it inside the -default WSL distribution and translates the repository path with `wslpath`. That exists because -Docker installed in WSL leaves no Windows CLI behind, which would otherwise make the setup this -page recommends unusable. +The engine must be **callable from the shell that runs `just`**. On Windows there is one automatic exception: if the +engine is not on `PATH`, anvil retries it inside the default WSL distribution and translates the repository path with +`wslpath`, which is what makes a WSL-only Docker installation work with no Windows CLI. | | Docker | Podman | | --- | --- | --- | -| Status | **supported** — what the e2e validates and what CI uses | works, with one exception below | | Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | | Builder | BuildKit | buildah | - -Podman has been run through the same end-to-end test as docker: it builds the image, computes and -reuses the content-addressed tag, and runs recipes. Rootless uid mapping differs (`--userns -keep-id` rather than `--user $(id -u)`), which anvil does not currently set for podman. - -**Build secrets do not work on podman for Windows.** Podman composes its own temp path from the -build context after translating it into its machine's view, and joins it with a Windows -separator, so any `--secret` fails before the build starts: - -```text -Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 -``` - -This is not something anvil can work around — `src=` and `env=` fail identically, and a -four-line Dockerfile reproduces it with no anvil involved. It affects only a repository that -supplies `Anvil-PreBuild` (§5); the public catalog ships no hook, so ordinary use is unaffected. -Use docker if you need build-time credentials on Windows. - -**Podman needs to be pointed at the ignore file.** Anvil emits -`.anvil/container/Dockerfile.dockerignore`, which BuildKit reads in preference to a root -`.dockerignore`. Podman and buildah honour only `.containerignore` or `.dockerignore` at the -*context root*, so the recipe passes `--ignorefile` explicitly when the engine is podman. Without -it the whole worktree — `target/` included — would be streamed to the daemon on every build, and -a consumer repository owning a root `.dockerignore` would have that one obeyed instead. - -Docker also carries more mileage: it is what CI uses and what the e2e runs by default. Prefer it -if you have no reason not to. +| Support | full | full, except build secrets on Windows (§6.2) | ### 6.1 Docker -**Linux.** Install Docker Engine from your distribution or `get.docker.com`, add yourself to the -`docker` group, and you are done. +**Linux.** Install Docker Engine from your distribution or `get.docker.com` and add yourself to the `docker` group. -**Windows — Docker Desktop.** Nothing to configure. `docker` is on `PATH`, so anvil calls it -directly. +**Windows, with Docker Desktop.** Nothing to configure: `docker` is on `PATH`. -**Windows — Docker Engine in WSL** (no Docker Desktop, no licence question): +**Windows, Docker Engine in WSL** — no Docker Desktop, and no Windows CLI required: ```powershell wsl --install -d Ubuntu-24.04 @@ -261,20 +232,18 @@ wsl --shutdown wsl -d Ubuntu-24.04 -- docker version # verify ``` -That is the whole setup: no Windows `docker` CLI is needed, because anvil reaches the engine -through `wsl.exe` when it finds none on `PATH`. `just` and `pwsh` stay on Windows — the -distribution needs only Docker. +That is the whole setup. `just` and `pwsh` stay on Windows; the distribution needs only Docker. -If you *do* install a Windows `docker` CLI and point `DOCKER_HOST` at the WSL socket, anvil uses -it directly and the WSL fallback never engages. In that arrangement the daemon is Linux-side, so -the repository must be bind-mountable at a path that daemon understands. +Installing a Windows `docker` CLI and pointing `DOCKER_HOST` at the WSL socket also works, and takes precedence — the +WSL path is used only when no CLI is found. The daemon is then Linux-side, so the repository must be bind-mountable +at a path it understands. ### 6.2 Podman **Linux.** Install podman and set `ANVIL_CONTAINER_ENGINE=podman`. -**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and -`podman.exe` is on `PATH`, so anvil calls it directly and the WSL fallback never engages. +**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and `podman.exe` is on +`PATH`, so anvil calls it directly. ```powershell winget install RedHat.Podman-Desktop # or the podman CLI alone @@ -283,52 +252,56 @@ podman machine start $env:ANVIL_CONTAINER_ENGINE = 'podman' ``` +**Build secrets are unavailable on podman for Windows.** Podman builds its own temp path from the build context after +translating it into the machine's view and joins it with a Windows separator, so any `--secret` fails before the +build starts: + +```text +Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 +``` + +The failure is in podman rather than in anvil, and no form of the flag avoids it. It affects only a repository whose +hook supplies `Anvil-PreBuild` (§5.1); everything else — building, running, tag reuse — works. Use docker if you need +build-time credentials on Windows. + +Two smaller differences: anvil passes `--ignorefile` explicitly on podman, because buildah honours only an ignore file +at the context root, and rootless uid mapping (`--userns keep-id`) is not currently applied. + ## 7. Customizing the image -Two audiences, three levers. A **repository** owns its own copy of the emitted files; a -**downstream catalog** (an anvil fork — see [extensibility.md](./extensibility.md)) changes what -every repository it manages receives. +A **repository** changes what its own image contains; a **downstream catalog** (an anvil fork — see +[extensibility.md](./extensibility.md)) changes what every repository it manages receives. | You want | Do this | Who | | --- | --- | --- | -| Extra packages, one repository | Edit `.anvil/container/Dockerfile` in place; the drift flow preserves it | repository | +| Extra packages, one repository | Edit `.anvil/container/Dockerfile` in place | repository | | A different base OS or toolchain source, everywhere | `replace_artifact(artifacts::container::dockerfile().with_body(…))` | catalog | -| Credentials | Write `.anvil/container/hooks.ps1`, or ship one with `with_artifact(artifacts::container::hooks(…))` | either | +| Credentials, or a prebuilt image | Write `.anvil/container/hooks.ps1`, or ship one with `with_artifact(artifacts::container::hooks(…))` | either | | No container support at all | `without_artifact` each of the three artifacts | catalog | -Editing the Dockerfile in a single repository is supported but noisy: anvil keeps proposing its -own version against a file it can see has diverged. A fork that wants the change everywhere -should replace the artifact instead. - -The hook is loaded by path, not by provenance: `container.just` sources -`.anvil/container/hooks.ps1` whenever it exists, so a hand-written file and one shipped by a -catalog behave identically. That is deliberate — it lets a repository try a credential flow -before anyone commits to forking the catalog for it. +Editing the Dockerfile in one repository is supported, and the drift flow keeps the edit — but anvil will keep +offering its own version against a file it can see has diverged. A change that belongs everywhere is better made in a +catalog. -**Coupled artifacts.** The Dockerfile and its ignore file move together. A replacement that -`COPY`s more of the tree must also replace `artifacts::container::dockerignore()`, or the extra -files are excluded from the build context and the build fails on a missing path. The recipe -tree needs no such care: the image identity hashes every `*.just` under `justfiles/anvil/` -recursively, so a catalog that adds a recipe directory gets it hashed automatically — but the -same widening rule applies before it can be copied. +**The Dockerfile and its ignore file move together.** A replacement that `COPY`s more of the tree must also replace +`artifacts::container::dockerignore()`, or the extra files never reach the build context and the build fails on a +missing path. Recipes need no such care: the identity hash covers `justfiles/anvil/` recursively, so a new recipe +directory is hashed automatically — though the same widening rule applies before it can be copied. -**What a fork does *not* touch.** The recipe, the identity hash, the cache volumes, the mounts -and the uid mapping are inherited unchanged. Substrate is the worked example: a different base -OS and a different toolchain source, expressed as one Dockerfile replacement plus one hook, and -nothing else. +A fork inherits the rest unchanged: the recipes, the identity hash, the cache volumes, the mounts and the uid +mapping. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. -Keep `ARG BASE_IMAGE` digest-pinned. A floating tag can change underneath a tag that claims to name fixed content, -which would make every cached image a potential lie. +Keep `ARG BASE_IMAGE` digest-pinned. The identity hash covers this file's text, but it does not resolve the base, so +a floating tag can change underneath a tag that claims to name fixed content. ## 8. Limits - Linux-only, `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. -- The engine never pushes and never promotes: it builds, and it may accept an image a hook fetched (§5.2). Publishing - is somebody else's job, and a repository that implements no hook needs no published artifact to exist. +- Anvil never pushes and never promotes an image. It builds one, and it will use one a hook fetched (§5.2); + publishing belongs to whoever owns the registry. - A repository-owned `rust-toolchain.toml` is required — it is both what the image installs and part of what names it. - The first build takes several minutes: it installs a toolchain and the whole pinned tool catalog. Later runs reuse it until an input changes. -- `target/` stays on the bind mount; the cargo and rustup homes live in named volumes so the hot write path does not - cross the host boundary. +- `target/` stays on the bind mount, so build output is visible from the host. [design]: ./README.md diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index b7e6bc70..bd35e5db 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -101,20 +101,21 @@ //! pins, by running `just anvil-setup` — the same recipe the checks use — so //! the container and the host agree on the toolset by construction. //! -//! There is no configuration file and no transparent routing: `just anvil-pr` -//! keeps running natively, and the container is reached only through the -//! explicit recipe. +//! `just anvil-pr` and every other recipe keep running natively; the container +//! is entered only through `anvil-container`, which takes any recipe name and +//! its arguments. //! //! ```text -//! just anvil-container anvil-clippy # one check -//! just anvil-container anvil-pr # the whole PR tier -//! just anvil-container # interactive shell +//! just anvil-container anvil-clippy # one check +//! just anvil-container anvil-pr # the whole PR tier +//! just anvil-container anvil-setup binstall # a recipe with an argument +//! just anvil-container # interactive shell //! ``` //! //! ### Prerequisites //! -//! - A container engine callable from the shell that runs `just`: Docker -//! (supported) or Podman (best-effort). On Windows that means Docker +//! - A container engine callable from the shell that runs `just`: Docker, or +//! Podman via `ANVIL_CONTAINER_ENGINE=podman`. On Windows that means Docker //! Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or //! Docker Engine installed only inside the default WSL distribution — no //! Windows CLI is needed in that last case, since anvil reaches the engine @@ -145,7 +146,7 @@ //! //! | Variable | Effect | //! |---|---| -//! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. A host property; never committed. | +//! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | //! | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure. | //! | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | //! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves, ignoring the hook. | @@ -157,8 +158,8 @@ //! //! ### The hook //! -//! crates.io needs none of this, so the public catalog emits no hook at all. -//! A repository or a downstream catalog that needs one adds +//! crates.io needs no credentials, so no hook is emitted by default. A +//! repository or a downstream catalog that needs one adds //! `.anvil/container/hooks.ps1`, which the recipe loads when present: //! //! ```powershell From cceb53927efc11c61d68ff585b5674f6b6ef36a8 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 19:20:58 +0200 Subject: [PATCH 14/81] docs(anvil): rewrite the container documentation as a technical reference `containers.md` becomes a reference specification: purpose, command surface, execution model, emitted artifacts, image identity, environment variables, the hook contract, host requirements, customization, and limitations. Every claim is stated against the implementation in `container.just`, `container.rs` and the default `Dockerfile`. Four inaccuracies are corrected: - Process identity. A Linux host passes `--user :`; `--userns keep-id`, which rootless podman needs for bind-mount ownership, is not passed. The previous text asserted both that a fork inherits "the uid mapping" and that no mapping is applied. - Podman is best-effort rather than fully supported, and all three known differences from Docker are listed together. - A new recipe subdirectory needs no ignore-file override: the build context re-admits `justfiles/` as a directory and the identity hashes it recursively. Only a Dockerfile that copies something else must replace the ignore file with it. - `hooks.ps1` is not emitted by default, which is what `local.md` already said. Behaviour previously undocumented is now specified: the per-invocation engine override, cache-volume sharing between checkouts of the same directory name, a missing image input as a hard error, the `ANVIL_SECRET_` mount naming, `--pull=never` on run, and the ordinal-sort and LF-normalization properties that make the digest stable across platforms. The crate documentation in `lib.rs` keeps the short form and links out; `README.md` is regenerated from it. `extensibility.md` states the `justfiles/`-holds-recipes-only rule once and points at the customization section, whose anchor had gone stale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cdda814-67ed-42b6-9c60-01149a70f78d --- crates/cargo-anvil/README.md | 102 ++-- crates/cargo-anvil/docs/design/containers.md | 494 ++++++++++++------ .../cargo-anvil/docs/design/extensibility.md | 12 +- crates/cargo-anvil/src/lib.rs | 100 ++-- 4 files changed, 454 insertions(+), 254 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 7e367aeb..81db299e 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -97,14 +97,15 @@ and run each check only over the affected packages, whereas a local ### Containerized local checks -Any generated recipe can run in a content-addressed Linux container. The -image installs the Rust toolchain and Cargo tools that this repository -pins, by running `just anvil-setup` — the same recipe the checks use — so -the container and the host agree on the toolset by construction. +Any generated recipe can be executed inside a content-addressed Linux +image. The image installs the Rust toolchain and Cargo tools this +repository pins by running `just anvil-setup` — the same recipe the checks +use, reading the same generated pins — so the image and the host agree on +the toolset by construction, with no second tool list to keep in step. -`just anvil-pr` and every other recipe keep running natively; the container -is entered only through `anvil-container`, which takes any recipe name and -its arguments. +`just anvil-pr` and every other recipe continue to run natively. A +container is entered only through `anvil-container`, which takes any recipe +name and its arguments; nothing is routed into one implicitly. ```text just anvil-container anvil-clippy # one check @@ -113,6 +114,17 @@ just anvil-container anvil-setup binstall # a recipe with an argument just anvil-container # interactive shell ``` +The feature is two generated artifacts and one optional hook: +`justfiles/anvil/container.just` drives the engine, +`.anvil/container/Dockerfile` (with its `Dockerfile.dockerignore`) defines +what the image contains, and `.anvil/container/hooks.ps1` supplies +credentials when a repository needs them. There is no configuration file. + +One container is created per invocation, not per check. The repository is +bind-mounted at `/workspace`, so `target/` stays visible from the host, +while `CARGO_HOME` and `RUSTUP_HOME` live in named volumes that keep the +write-heavy paths off the host boundary. + #### Prerequisites * A container engine callable from the shell that runs `just`: Docker, or @@ -120,40 +132,44 @@ just anvil-container # interactive shell Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or Docker Engine installed only inside the default WSL distribution — no Windows CLI is needed in that last case, since anvil reaches the engine - through `wsl.exe` when it finds none on `PATH`. + through `wsl.exe` when it finds none on `PATH` and translates repository + paths with `wslpath`. * `just` and `PowerShell` Core (`pwsh`) on the host. * A repository-owned `rust-toolchain.toml`. -On ARM64 hosts the image is emulated as `linux/amd64`, so builds and checks -are substantially slower. +Docker is supported; Podman works on a best-effort basis, with two +documented gaps on Windows. The image is pinned to `linux/amd64`, so on +ARM64 hosts it is emulated and is substantially slower. #### Image identity -The tag *is* a SHA-256 over the inputs that define the image: the -Dockerfile and its ignore file, `rust-toolchain.toml`, the optional -hook, and the generated recipe tree. A changed tool pin names a tag that -cannot already exist, so a build follows. There is no staleness check -because there is nothing to check: an image built here is, by -construction, built from the current inputs. An image *fetched* by the -resolve hook only claims as much — the hash is over source files and -cannot be re-derived from layers — so that claim rests on the registry it -came from having immutable tags and restricted push. +The tag *is* a SHA-256 digest over the inputs that define the image: the +Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, +and every `*.just` under `justfiles/anvil/` other than the driver itself. +A changed tool pin names a tag that cannot already exist, so a build +follows. There is no staleness check because there is no staleness to +detect: a locally built image that is present was built from the inputs +that name it. An image *fetched* by the resolve hook only claims as much — +the digest is over source files and cannot be re-derived from layers — so +that claim is only as strong as the registry it came from, which should +have immutable tags and restricted push. `anvil-container-tag` prints the reference without building it, and is the -single place the hash is computed, so a publisher can tag an image with +single place the digest is computed, so a publisher can tag an image with exactly the reference a consumer will later look up. #### Controls |Variable|Effect| |--------|------| -|`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`.| +|`ANVIL_CONTAINER_ENGINE`|`docker` (default) or `podman`. Read at run time.| |`ANVIL_CONTAINER_NO_REBUILD=1`|Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure.| |`ANVIL_CONTAINER_NO_RESOLVE=1`|Skip the resolve hook, so a query never pulls.| |`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves, ignoring the hook.| |`ANVIL_IN_CONTAINER=1`|Set inside the image; makes a nested invocation run natively.| -Supporting recipes: `anvil-container-tag`, `anvil-container-status`, +Supporting recipes: `anvil-container-tag`, `anvil-container-status` +(reports the engine and image without building or pulling), `anvil-container-rebuild`, and `anvil-container-down` (removes this repository’s cache volumes). @@ -161,38 +177,44 @@ repository’s cache volumes). crates.io needs no credentials, so no hook is emitted by default. A repository or a downstream catalog that needs one adds -`.anvil/container/hooks.ps1`, which the recipe loads when present: +`.anvil/container/hooks.ps1`, which the recipe loads by path whenever the +file is present, whoever wrote it: ```powershell function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } -function Anvil-ResolveImage { param($tag) (fetch-a-prebuilt-image $tag) } +function Anvil-ResolveImage { param($tag) (fetch-a-published-image $tag) } ``` -Build secrets are passed to `BuildKit` by environment variable name, so a -value never reaches a process argument and never reaches an image layer; -run-time values are forwarded into the container by name for the same -reason. An empty value is a hard error, because a build that quietly -proceeded without its credential would install a reduced tool set and then -be tagged with the hash a credentialed build produces. +All three are optional. Build secrets are passed to `BuildKit` by +environment variable name, so a value never reaches a process argument and +never reaches an image layer; run-time values are forwarded into the +container by name for the same reason. An empty value is a hard error, +because a build that quietly proceeded without its credential would install +a reduced tool set and then be tagged with the digest a credentialed build +produces. `Anvil-ResolveImage` is offered the tag when nothing local matches, and -returns the reference it made available — a registry reference, not a -local re-tag, so the run stays honest about where the image came from. It -is verified before use and every failure falls through to a local build: -a publisher that has not caught up must not block the change it has not -caught up with. +returns the reference it made available — a registry reference, used as-is +rather than re-tagged locally, so the run stays honest about where the +image came from. It is verified before use, and every failure falls through +to a local build: a publisher that has not caught up must not block the +change it has not caught up with. -The hook runs on the host with the developer’s permissions, before any -container isolation. Only run one from a repository or catalog you trust. +The hook executes on the host, with the invoking user’s permissions, before +any container isolation exists. Only use one from a repository or catalog +you trust. #### Customizing the image `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place for extra packages, and anvil’s drift handling preserves the change. A downstream catalog that needs a different base OS or toolchain source for -every repository it manages replaces the artifact instead — see -[`artifacts::container`][__link1] and the design doc. +every repository it manages replaces the artifact instead. A replacement +that copies more of the tree must replace the ignore file with it, since +the build context admits only `justfiles/` and `rust-toolchain.toml`. See +[`artifacts::container`][__link1] and the design document for the full contract, +the host setup for each engine, and the known limitations. ### Checks and tiers @@ -412,7 +434,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbbI8J5B4oqNYbCGDbwei2JEEbJ8_ukS4Unx8b6YMW7c9a-thhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbpACaXmKNr48bH5E05_uOLpIby9m8IWn7SQMbLMeQnQJv6axhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 33e4b86b..5d900cc1 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -1,6 +1,7 @@ -# Containers +# Containerized execution -Any generated recipe can run inside a Linux image that carries exactly the toolchain and tools this repository pins: +Any generated recipe can be executed inside a Linux image built from the toolchain and tool versions the repository +pins: ```bash just anvil-container anvil-clippy # one check @@ -8,175 +9,307 @@ just anvil-container anvil-pr # the whole PR tier just anvil-container # interactive shell ``` -See also [design.md](./README.md) for the overall principles, [local.md](./local.md) for the recipe surface this +The feature consists of two generated artifacts and one optional hook. There is no configuration file, and no +invocation is routed into a container implicitly. + +See [README.md](./README.md) for the overall design principles, [local.md](./local.md) for the recipe surface this wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstream fork uses. -- [1. Why](#1-why) -- [2. How it runs](#2-how-it-runs) -- [3. What it adds to a repository](#3-what-it-adds-to-a-repository) -- [4. Image identity](#4-image-identity) -- [5. The hook](#5-the-hook) -- [6. Host setup](#6-host-setup) - - [6.1 Docker](#61-docker) - - [6.2 Podman](#62-podman) -- [7. Customizing the image](#7-customizing-the-image) -- [8. Limits](#8-limits) +- [1. Purpose](#1-purpose) +- [2. Command surface](#2-command-surface) +- [3. Execution model](#3-execution-model) + - [3.1 Engine resolution](#31-engine-resolution) + - [3.2 Path translation](#32-path-translation) + - [3.3 Mounts and working directory](#33-mounts-and-working-directory) + - [3.4 Process identity](#34-process-identity) + - [3.5 Re-entry](#35-re-entry) +- [4. Emitted artifacts](#4-emitted-artifacts) +- [5. Image identity](#5-image-identity) + - [5.1 Hashed inputs](#51-hashed-inputs) + - [5.2 Digest computation](#52-digest-computation) + - [5.3 Guarantees](#53-guarantees) +- [6. Environment variables](#6-environment-variables) +- [7. The hook](#7-the-hook) + - [7.1 Anvil-PreBuild](#71-anvil-prebuild) + - [7.2 Anvil-PreRun](#72-anvil-prerun) + - [7.3 Anvil-ResolveImage](#73-anvil-resolveimage) + - [7.4 Trust boundary](#74-trust-boundary) +- [8. Host requirements](#8-host-requirements) + - [8.1 Docker](#81-docker) + - [8.2 Podman](#82-podman) +- [9. Customization](#9-customization) +- [10. Limitations](#10-limitations) + +## 1. Purpose + +The generated recipes assume a usable host toolchain; anvil does not install one ([README.md §3][design]: "the user +owns it locally"). Two conditions invalidate that assumption: + +1. **Platform-specific failures.** A `cfg(unix)` code path, a Linux-only lint, or a test that depends on Linux memory + semantics cannot be reproduced on a Windows or macOS host. +2. **Toolchain divergence.** The installed toolset can differ from the one the checks expect, so a passing local run + stops predicting a cloud result. + +Both are addressed by executing the recipe unchanged inside an image constructed from the repository's own pins. + +## 2. Command surface + +`just anvil-pr` and every other recipe continue to execute natively. A container is entered only through +`anvil-container`, which accepts a recipe name and its arguments: -## 1. Why +```bash +just anvil-container anvil-setup binstall +``` -The local recipes assume a usable host toolchain — anvil does not install one ([design.md §3][design]: "the user owns -it locally"). Two situations break that assumption: +| Recipe | Behaviour | +| --- | --- | +| `just anvil-container [args…]` | Execute a recipe in the image. With no argument, opens an interactive shell. | +| `just anvil-container-tag` | Print the image reference for the current inputs. Builds nothing. | +| `just anvil-container-status` | Print the engine, working directory, image reference, and whether it is present. Never builds or pulls. | +| `just anvil-container-rebuild` | Rebuild the image with every layer cache disabled. | +| `just anvil-container-down` | Remove this repository's cache volumes. The image is retained. | -1. **Linux-only failures on a Windows machine.** A `cfg(unix)` path, a Linux-specific lint, an `mmap`-shaped test: - none of them can be reproduced without a Linux environment. -2. **Toolchain drift.** Even on Linux, the installed toolset can differ from the one the checks expect, so a green - local run stops being predictive of a cloud one. +All five are annotated `[group("anvil-container")]` and appear as one cluster in `just --groups`. -Both are answered the same way: run the recipe in an image built from the repository's own pins. +Recipe bodies are identical in both execution modes. No wrapper shadows `just` on `PATH`, and no recipe behaves +differently according to where it runs. Cloud workflows are unaffected: they execute the same recipes natively on +their own agents. The image is pinned to resemble that environment, not to reproduce it. -## 2. How it runs +## 3. Execution model -`just anvil-pr` and every other recipe continue to run natively. The container is entered only through -`just anvil-container`, which takes any recipe name and its arguments: +One container is created per `anvil-container` invocation, not one per check. The container is removed on exit +(`--rm`). -```bash -just anvil-container anvil-setup binstall -``` +### 3.1 Engine resolution -Inside the image, `ANVIL_IN_CONTAINER=1` is set, so a recipe that reaches `anvil-container` again runs natively -instead of nesting. The work happens once — one container per command, not one per check. +`ANVIL_CONTAINER_ENGINE` selects the engine and defaults to `docker`. Any value other than `docker` or `podman` is +rejected before the engine is invoked. Because the engine is a property of the host rather than of the repository, it +is read at run time and is never committed; a single invocation can override it with +`just anvil_container_engine=podman anvil-container anvil-pr`. -The recipes themselves are identical in both cases. Nothing behaves differently depending on where it runs, and no -wrapper shadows `just` on `PATH`. +Resolution proceeds in a fixed order: -| Recipe | Purpose | -| --- | --- | -| `just anvil-container [args…]` | Run a recipe in the image. No argument opens an interactive shell. | -| `just anvil-container-tag` | Print the image reference for the current inputs, without building it. | -| `just anvil-container-status` | Report the engine, the image reference, and whether it is present. | -| `just anvil-container-rebuild` | Rebuild from scratch, ignoring every cached layer. | -| `just anvil-container-down` | Remove this repository's cache volumes. The image is left in place. | +1. If the named binary is on `PATH`, it is invoked directly. +2. Otherwise, on Windows, the binary is probed inside the default WSL distribution (`wsl.exe -- --version`). + If the probe succeeds, every subsequent engine call is prefixed with `wsl.exe --`. +3. Otherwise the invocation fails with a message naming the variable and linking to this document. + +anvil does not probe for an engine other than the one requested. Presence is not reachability; `podman-docker` aliases +`docker` onto podman; and silently selecting between two installed engines yields two image stores and an unexplained +rebuild. Every failure other than a missing binary surfaces the engine's own diagnostic unmodified. + +Step 2 exists because installing Docker Engine inside WSL without Docker Desktop leaves no Windows CLI on `PATH`, and +that setup is the one this repository's own development guide describes. Docker Desktop and Podman both install a +Windows CLI, are found in step 1, and never reach step 2. + +### 3.2 Path translation + +When the engine is reached through WSL it does not share the Windows filesystem view, so host paths are translated +with `wslpath -a -u` before they are passed as a bind-mount source, a build context, or a `--file` argument. An +untranslated Windows path is not rejected by the daemon: it is bind-mounted as an empty directory, and the failure +surfaces much later as a missing file inside the container. + +Paths are converted to forward slashes before translation, because arguments crossing into WSL pass through a shell +that would otherwise consume the backslashes. `wslpath` accepts either separator. -The repository is mounted at `/workspace`, and the working directory is mapped to its in-container equivalent so -relative paths keep working from a subdirectory. The cargo and rustup homes live in named volumes, so the hot write -path never crosses the host boundary and the host's own toolchain is untouched. +### 3.3 Mounts and working directory -Cloud workflows are unaffected: they run the recipes natively on their own agents. The image is pinned to resemble -that environment, not to be it. +| Mount | Target | Purpose | +| --- | --- | --- | +| repository root (bind) | `/workspace` | The worktree under test, including `target/`. | +| `anvil--cargo` (volume) | `/usr/local/cargo` | `CARGO_HOME`: registry cache and installed binaries. | +| `anvil--rustup` (volume) | `/usr/local/rustup` | `RUSTUP_HOME`: installed toolchains. | + +The cargo and rustup homes are named volumes rather than bind mounts, so the write-heavy paths never cross the host +boundary and the host's own toolchain is untouched. `target/` remains on the bind mount, so build output stays visible +from the host and is shared between native and containerized runs. + +The caller's working directory is mapped to its in-container equivalent, so relative paths continue to resolve when +`anvil-container` is invoked from a subdirectory. + +Volume names derive from the repository directory name, lowercased with every character outside `[a-z0-9._-]` +replaced by `-`. Two checkouts with the same directory name therefore share cache volumes. This is harmless in normal +use, because cargo's caches are content-addressed, but `anvil-container-down` removes volumes that the other checkout +is also using. + +### 3.4 Process identity + +On a Linux host the run passes `--user :`, matching the invoking user. Without it, everything written under +the bind mount — `target/`, generated files — is owned by root on the host, and the next native `cargo build` or +`git clean` fails with `EACCES` far from the cause. The flag is omitted when the invoking user is root. + +Docker Desktop on Windows and macOS maps ownership itself, and `id` is not available to query, so the flag is not +passed on those hosts. -## 3. What it adds to a repository +### 3.5 Re-entry + +`ANVIL_IN_CONTAINER=1` is set in the image and passed again on each run. `anvil-container` checks it first: inside the +image, the requested recipe is executed directly instead of launching another container. A recipe that reaches +`anvil-container` transitively therefore performs its work exactly once. + +## 4. Emitted artifacts ```text repo/ ├── justfiles/anvil/ │ ├── container.just the anvil-container recipes -│ └── … checks, groups, tiers — run natively *inside* the image +│ └── … checks, groups, tiers — executed natively *inside* the image └── .anvil/container/ ├── Dockerfile what the image contains ├── Dockerfile.dockerignore what the build context admits - └── hooks.ps1 optional; supplied by you or a catalog (§5) + └── hooks.ps1 optional; not emitted by default (§7) ``` -`container.just` is generated and reconciled on every run; edits to it are replaced. The `Dockerfile` and its ignore -file are generated too, but they are meant to be edited — anvil's drift handling preserves a repository's changes to -them (§7). +`container.just` is generated and reconciled on every run; local edits to it are replaced. The `Dockerfile` and its +ignore file are generated but intended to be edited: anvil's drift handling preserves a repository's changes to them +(§9). -The image installs its tools by running `just anvil-setup`: the same recipe the checks use, reading the same -generated pins. There is no second list of tools to keep in step, which is also why bumping a tool pin changes the -image — `versions.just` is both what the image installs and part of what names it (§4). +The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated +pins. There is no second tool list to keep synchronized, which is also why a tool-pin change renames the image: +`versions.just` is both what the image installs and part of what names it (§5). -## 4. Image identity +The build context is scoped by `Dockerfile.dockerignore`, a deny-all list that re-admits `justfiles/` and +`rust-toolchain.toml` and nothing else. BuildKit reads `.dockerignore` in preference to a root +`.dockerignore`, so the repository does not need to own a root ignore file and cannot have one silently overridden. -The tag **is** the hash of the inputs that define the image: +## 5. Image identity -```text -anvil-:<16 hex characters> -``` +The image reference is `anvil-:<16 hex characters>`, where the tag is a SHA-256 digest over the inputs that +define the image. The name is derived from the repository directory as described in §3.3. + +### 5.1 Hashed inputs + +| Input | Hashed | +| --- | --- | +| `.anvil/container/Dockerfile` | always | +| `.anvil/container/Dockerfile.dockerignore` | always | +| `rust-toolchain.toml` | always | +| `.anvil/container/hooks.ps1` | when the file exists | +| `justfiles/anvil/**/*.just` | always, recursively, except `container.just` | -Hashed: the `Dockerfile`, its ignore file, `rust-toolchain.toml`, `hooks.ps1` when present, and every `*.just` under -`justfiles/anvil/` — except `container.just` itself, since hashing the driver would make the tag depend on the tag. +The recipe tree is included because the image installs its tools by running `just anvil-setup`, whose dependency chain +reaches the tier, group, check, and tool recipes. `container.just` is excluded because hashing the driver would make +the tag depend on the tag. -Change any input and the tag names an image that cannot already exist, so a build follows. Change nothing and the tag -resolves immediately. There is no staleness check because there is no staleness: an image that is present was built -from the inputs that name it. +The hook file's **content** is an input, because it determines what the build installs. Its **output** is deliberately +excluded: a credential must never influence a tag. -That guarantee is exact for an image built locally. An image fetched by the resolve hook (§5.2) only *claims* those -inputs — the hash is over source files and cannot be recomputed from layers — so the claim is only as good as the -registry it came from. Publish to one with immutable tags, and restrict push to the identity that builds them. +A declared input that does not exist is a hard error rather than an omission from the digest. -`just anvil-container-tag` prints the reference without building anything. Every other recipe asks it, so a publisher -and a consumer compute the same reference independently: no `latest`, no digest to maintain by hand. +### 5.2 Digest computation + +Inputs are sorted by relative path using an ordinal comparison, then serialized into a single stream. Each entry +contributes a literal `file`, its relative path, and its content, each terminated by a newline. Tagging each entry +this way ensures no rearrangement of names and contents can produce a collision. Line endings are normalized to LF, so +a CRLF checkout and an LF checkout compute the same tag. The ordinal sort matters because a case-insensitive one would +silently drop one of two inputs differing only in case on the case-sensitive filesystem where the image is built. + +The tag is the first eight bytes of the digest, hex-encoded — 64 bits, far beyond any practical collision risk for a +local image set, and short enough to keep `docker images` readable. + +`anvil-container-tag` is the only place this computation exists; every other recipe calls it. A publisher and a +consumer therefore derive the same reference independently, with no `latest` tag and no digest maintained by hand. + +### 5.3 Guarantees + +Changing any input names a tag that cannot already exist, so a build follows. Changing nothing resolves the existing +tag immediately. There is no staleness check because there is no staleness to detect: a locally built image that is +present was built from the inputs that name it. + +That guarantee is exact only for a locally built image. An image obtained through `Anvil-ResolveImage` (§7.3) merely +*claims* those inputs — the digest is computed over source files and cannot be re-derived from layers — so the claim +is only as strong as the registry it came from. Publish to a registry with immutable tags, and restrict push to the +identity that builds them. + +Two inputs sit outside the digest and must be pinned by other means. The base image is not resolved during hashing, so +`ARG BASE_IMAGE` must remain digest-pinned or a floating tag can change beneath a tag that claims to name fixed +content. The platform is pinned to `linux/amd64` on both build and run, so hosts of differing architecture cannot +compute one tag for two different images. + +## 6. Environment variables | Variable | Effect | | --- | --- | -| `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is missing, which distinguishes a cache miss from a build failure. | +| `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. Read at run time. | +| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is absent, distinguishing a cache miss from a build failure. | | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | -| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild even when the tag resolves, ignoring the hook. | +| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag already resolves. Also skips the resolve hook. | +| `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§3.5). | + +`ANVIL_CONTAINER_NO_CACHE` skips the hook because "ignore what is cached" must include the remote cache; otherwise a +rebuild would be undone by the next resolve. `ANVIL_CONTAINER_NO_REBUILD` is evaluated independently of it, so the two +compose: `anvil-container-status` sets both `NO_REBUILD` and `NO_RESOLVE`, and answers from local state alone. When +`NO_REBUILD` stops a build, the reference is still printed, because a caller that asked not to build is usually asking +*which* image is missing. -Values a hook returns never enter the hash: a credential must not be able to influence a tag. +## 7. The hook -## 5. The hook +`.anvil/container/hooks.ps1` supplies the two things anvil cannot derive: credentials, and where a prebuilt image +might be obtained. The file is optional and is not emitted by default — crates.io requires no credentials, and an +empty script would be one more generated file to review. -`.anvil/container/hooks.ps1` supplies the two things anvil cannot know — credentials for a private feed, and where a -prebuilt image might come from. It is optional, and it is loaded by path rather than by provenance: the recipe sources -it whenever the file exists, whether a repository wrote it or a catalog shipped it. A repository can therefore try a -credential flow without forking anything. +It is loaded by path rather than by provenance: the recipe dot-sources it whenever the file exists, whether a +repository wrote it or a catalog shipped it. A repository can therefore adopt a credential flow without forking the +catalog. -| Function | Called | Returns | +| Function | Invoked | Returns | | --- | --- | --- | -| `Anvil-PreBuild` | before a build | `@{ Secrets = @{ id = value } }` | -| `Anvil-PreRun` | before a run | `@{ Env = @{ NAME = value } }` | -| `Anvil-ResolveImage $tag` | before a build, when nothing local matches | an image reference, or nothing | +| `Anvil-PreBuild` | before a build | `@{ Secrets = @{ = } }` | +| `Anvil-PreRun` | before a run | `@{ Env = @{ = } }` | +| `Anvil-ResolveImage $tag` | before a build, when no local image matches | an image reference, or nothing | + +All three are optional, and each is called only if defined. -All three are optional. +Both value-returning functions **fail closed on an empty value**. This is not the engine's behaviour: BuildKit accepts +`--secret id=t,env=UNSET`, mounts an empty secret, and exits 0. The build would install a reduced tool set, be tagged +with the same content hash a credentialed build produces, and be reused by every later run. -### 5.1 Credentials +### 7.1 Anvil-PreBuild -crates.io needs none, so the default image ships no credential plumbing. An internal feed does: +Each returned entry becomes a BuildKit `--secret id=,env=ANVIL_SECRET_` mount. The value is placed in a +process environment variable and passed **by name**, so it never appears in a command line, where endpoint telemetry +records and retains it far longer than a short-lived token is intended to live. The variables are removed once the +build completes. BuildKit keeps a mounted secret out of every image layer. ```powershell function Anvil-PreBuild { @{ Secrets = @{ feed_token = (az account get-access-token --resource … --query accessToken -o tsv) } } } - -function Anvil-PreRun { - @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = "Bearer $token" } } -} ``` -Minting the value in a function, rather than reading it from a committed file or a declared variable, is the point: a -short-lived token has to be acquired at the moment it is used. +Minting the value inside a function is the point: a short-lived token must be acquired at the moment it is used, not +read from a committed file or a declared variable. -`Secrets` become `--secret id=…,env=…` mounts at build time; `Env` becomes `-e NAME` at run time. In both cases the -value is handed to the engine **by variable name**, so it never appears in a command line — where endpoint telemetry -records and retains it far longer than the token is meant to live — and it never touches disk. Build secrets stay out -of every image layer. When the engine is reached through WSL (§6.1), the names are exported with `WSLENV` so the value -crosses that boundary the same way. - -Declare the mount as required, which closes the same hole from the Dockerfile's side: +Declare the mount as required in the Dockerfile, which closes the same gap from the build's side: ```dockerfile RUN --mount=type=secret,id=feed_token,required=true \ TOKEN="$(cat /run/secrets/feed_token)" … ``` -Anything the build *writes* with a secret is ordinary content. Anvil's own Dockerfile deletes `credentials.toml` and -`.netrc` in the same layer as the install; a replacement must do the same, or the credential is baked into a layer. +Anything the build *writes* using a secret is ordinary layer content. The default Dockerfile removes +`credentials.toml` and `.netrc` in the same `RUN` layer as the install; a replacement must do the same, or the +credential is baked into a layer that a later deletion cannot remove. + +When the engine is reached through WSL, the secret variable names are exported through `WSLENV` so the values cross +that boundary. -**An empty value is a hard error.** BuildKit itself is not: `--secret id=t,env=UNSET` mounts an empty secret and -exits 0, so the build would install a reduced tool set and then be tagged with the same content hash a credentialed -build produces — and every later run would reuse that broken image. +### 7.2 Anvil-PreRun -**Trust.** The hook runs on the host, with your permissions, before any container isolation. Only run one from a -repository or catalog you trust. Inside the container everything runs as one user in one mount namespace, so a -forwarded credential is readable by anything the checks execute, including dependency build scripts and proc macros. -Keep the set narrow and the token short-lived. +Each returned entry is forwarded into the container with `-e `, again by name rather than as `NAME=VALUE`, for +the reason given above. Inside the image the value is an ordinary environment variable. The forwarded names — never +their values — are echoed to stderr, because everything executing inside the container can read them. + +```powershell +function Anvil-PreRun { + @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = (mint-a-token) } } +} +``` -### 5.2 Prebuilt images +### 7.3 Anvil-ResolveImage -When no local image matches the tag, `Anvil-ResolveImage` is offered that reference before a build starts. A catalog -that publishes images implements it; without one, the build proceeds as usual. +When no local image matches the computed tag, the reference is offered to `Anvil-ResolveImage` before a build starts. +A catalog that publishes images implements it; without one, the build proceeds. ```powershell function Anvil-ResolveImage($tag) { @@ -187,41 +320,51 @@ function Anvil-ResolveImage($tag) { } ``` -Three properties are worth knowing: +Three properties are load-bearing: -- **The returned reference is used as-is, not re-tagged to the local name.** A local tag asserts "built here from - these inputs"; a fetched image only claims it (§4). Keeping the registry reference keeps the run honest about where - the image came from. -- **The reference is verified before use.** Runs are `--pull=never`, so a hook that reported an image it had not +- **The returned reference is used as-is, never re-tagged to the local name.** A local tag asserts "built here from + these inputs"; a fetched image only claims it (§5.3). Retaining the registry reference keeps the run honest about + the image's origin. +- **The reference is verified before use.** Runs pass `--pull=never`, so a hook that reported an image it had not actually fetched would otherwise fail later and further from the cause. - **Every failure falls through to a local build**, with the reason printed — a missing image, an expired credential, - a broken hook. A publisher that has not caught up with your change must never block you. + a hook that threw. A publisher that has not caught up with a change must not block the developer who made it. -`ANVIL_CONTAINER_NO_RESOLVE=1` skips this step. +Resolution is attempted before the `ANVIL_CONTAINER_NO_REBUILD` guard, because fetching a published image is not +building one. -## 6. Host setup +### 7.4 Trust boundary -anvil installs nothing and manages no virtual machine. It calls the engine you selected and lets that engine's own -diagnostics surface; the one message it owns is for a missing binary, which names the variable to set and points -here. +The hook executes on the host, with the invoking user's permissions, before any container isolation exists. Only use +one from a repository or catalog you trust. -The engine must be **callable from the shell that runs `just`**. On Windows there is one automatic exception: if the -engine is not on `PATH`, anvil retries it inside the default WSL distribution and translates the repository path with -`wslpath`, which is what makes a WSL-only Docker installation work with no Windows CLI. +Inside the container everything executes as a single user in a single mount namespace, so a forwarded credential is +readable by anything the checks execute, including dependency build scripts and procedural macros. Keep the forwarded +set narrow and the tokens short-lived. + +## 8. Host requirements + +anvil installs nothing and manages no virtual machine. It invokes the engine you selected and lets that engine's own +diagnostics surface. The only failure message it owns is for a missing binary, which names the variable to set and +links here. + +The engine must be callable from the shell that runs `just`, with the single Windows exception described in §3.1. The +host also needs `just` and PowerShell Core (`pwsh`), which every generated recipe requires, and the repository must +own a `rust-toolchain.toml`. | | Docker | Podman | | --- | --- | --- | | Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | | Builder | BuildKit | buildah | -| Support | full | full, except build secrets on Windows (§6.2) | +| Status | supported | best-effort; see §8.2 | -### 6.1 Docker +### 8.1 Docker -**Linux.** Install Docker Engine from your distribution or `get.docker.com` and add yourself to the `docker` group. +**Linux.** Install Docker Engine from your distribution or `get.docker.com`, and add your user to the `docker` group. -**Windows, with Docker Desktop.** Nothing to configure: `docker` is on `PATH`. +**Windows, with Docker Desktop.** No configuration required: `docker` is on `PATH`. -**Windows, Docker Engine in WSL** — no Docker Desktop, and no Windows CLI required: +**Windows, Docker Engine in WSL.** No Docker Desktop and no Windows CLI: ```powershell wsl --install -d Ubuntu-24.04 @@ -232,18 +375,19 @@ wsl --shutdown wsl -d Ubuntu-24.04 -- docker version # verify ``` -That is the whole setup. `just` and `pwsh` stay on Windows; the distribution needs only Docker. +`just` and `pwsh` remain on Windows; the distribution needs only Docker. anvil detects this configuration +automatically (§3.1) and translates paths accordingly (§3.2). -Installing a Windows `docker` CLI and pointing `DOCKER_HOST` at the WSL socket also works, and takes precedence — the -WSL path is used only when no CLI is found. The daemon is then Linux-side, so the repository must be bind-mountable -at a path it understands. +Installing a Windows `docker` CLI and pointing `DOCKER_HOST` at the WSL socket also works and takes precedence, since +the WSL path is used only when no CLI is found on `PATH`. The daemon is then Linux-side, so the repository must be +bind-mountable at a path it can resolve. -### 6.2 Podman +### 8.2 Podman **Linux.** Install podman and set `ANVIL_CONTAINER_ENGINE=podman`. -**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and `podman.exe` is on -`PATH`, so anvil calls it directly. +**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and `podman.exe` is placed on +`PATH`, so anvil invokes it directly. ```powershell winget install RedHat.Podman-Desktop # or the podman CLI alone @@ -252,56 +396,68 @@ podman machine start $env:ANVIL_CONTAINER_ENGINE = 'podman' ``` -**Build secrets are unavailable on podman for Windows.** Podman builds its own temp path from the build context after -translating it into the machine's view and joins it with a Windows separator, so any `--secret` fails before the -build starts: +Three differences from Docker are known and unresolved: -```text -Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 -``` +- **Build secrets are unavailable on podman for Windows.** Podman derives a temporary path from the build context + after translating it into the machine's view, then joins it using a Windows separator, so any `--secret` fails + before the build begins: + + ```text + Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 + ``` + + The defect is in podman, and no form of the flag avoids it. It affects only a repository whose hook defines + `Anvil-PreBuild` (§7.1); building, running, and tag reuse are unaffected. Use Docker if you need build-time + credentials on Windows. -The failure is in podman rather than in anvil, and no form of the flag avoids it. It affects only a repository whose -hook supplies `Anvil-PreBuild` (§5.1); everything else — building, running, tag reuse — works. Use docker if you need -build-time credentials on Windows. +- **The ignore file is passed explicitly.** buildah honours only an ignore file at the context root, so anvil passes + `--ignorefile` on podman. Without it the entire worktree, including `target/`, is streamed to the daemon on every + build. -Two smaller differences: anvil passes `--ignorefile` explicitly on podman, because buildah honours only an ignore file -at the context root, and rootless uid mapping (`--userns keep-id`) is not currently applied. +- **Rootless user-namespace mapping is not applied.** The run passes `--user` (§3.4) but not `--userns keep-id`, which + rootless podman requires for bind-mount ownership to map back to the invoking user. -## 7. Customizing the image +## 9. Customization -A **repository** changes what its own image contains; a **downstream catalog** (an anvil fork — see -[extensibility.md](./extensibility.md)) changes what every repository it manages receives. +A **repository** changes what its own image contains. A **downstream catalog** — an anvil fork, see +[extensibility.md](./extensibility.md) — changes what every repository it manages receives. Containerized execution is +an ordinary artifact group and uses the same levers as any other. -| You want | Do this | Who | +| Goal | Mechanism | Owner | | --- | --- | --- | -| Extra packages, one repository | Edit `.anvil/container/Dockerfile` in place | repository | +| Extra packages in one repository | Edit `.anvil/container/Dockerfile` in place | repository | | A different base OS or toolchain source, everywhere | `replace_artifact(artifacts::container::dockerfile().with_body(…))` | catalog | -| Credentials, or a prebuilt image | Write `.anvil/container/hooks.ps1`, or ship one with `with_artifact(artifacts::container::hooks(…))` | either | -| No container support at all | `without_artifact` each of the three artifacts | catalog | +| Credentials, or a published image | Add `.anvil/container/hooks.ps1`, or ship `artifacts::container::hooks(…)` | either | +| No containerized execution at all | `without_artifact` for each of the three artifacts | catalog | -Editing the Dockerfile in one repository is supported, and the drift flow keeps the edit — but anvil will keep -offering its own version against a file it can see has diverged. A change that belongs everywhere is better made in a +Editing the Dockerfile in a single repository is supported and the drift flow preserves the edit, but anvil continues +to offer its own version against a file it can see has diverged. A change that belongs everywhere is better made in a catalog. -**The Dockerfile and its ignore file move together.** A replacement that `COPY`s more of the tree must also replace -`artifacts::container::dockerignore()`, or the extra files never reach the build context and the build fails on a -missing path. Recipes need no such care: the identity hash covers `justfiles/anvil/` recursively, so a new recipe -directory is hashed automatically — though the same widening rule applies before it can be copied. +**The Dockerfile and its ignore file must be replaced together.** The ignore file is a deny-all list re-admitting only +`justfiles/` and `rust-toolchain.toml` (§4). A replacement Dockerfile that `COPY`s anything else must also replace +`artifacts::container::dockerignore()`, or the additional paths never reach the build context and the build fails on a +missing file. Recipes need no such care: `justfiles/` is re-admitted as a directory and hashed recursively, so a new +recipe subdirectory is both copied and part of the identity automatically. -A fork inherits the rest unchanged: the recipes, the identity hash, the cache volumes, the mounts and the uid -mapping. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. +`justfiles/anvil/` must contain `.just` recipes and nothing else. `CatalogBuilder::build` enforces this, because a +non-recipe file placed there would be copied into the image without being part of its identity: editing it would +change what the image contains without renaming the tag. Non-recipe assets belong in a tool-owned directory such as +`.anvil/`. -Keep `ARG BASE_IMAGE` digest-pinned. The identity hash covers this file's text, but it does not resolve the base, so -a floating tag can change underneath a tag that claims to name fixed content. +A fork inherits everything else unchanged: the recipes, the identity scheme, the cache volumes, the mounts, and the +re-entry guard. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. -## 8. Limits +## 10. Limitations -- Linux-only, `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. -- Anvil never pushes and never promotes an image. It builds one, and it will use one a hook fetched (§5.2); +- Linux images only, pinned to `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. +- The first build takes several minutes: it installs a toolchain and the entire pinned tool catalog. Subsequent runs + reuse it until an input changes. +- Any edit under `justfiles/` invalidates the install layer, including files the image's synthetic Justfile never + imports. +- anvil never pushes and never promotes an image. It builds one, and it will use one a hook fetched (§7.3); publishing belongs to whoever owns the registry. -- A repository-owned `rust-toolchain.toml` is required — it is both what the image installs and part of what names it. -- The first build takes several minutes: it installs a toolchain and the whole pinned tool catalog. Later runs reuse - it until an input changes. -- `target/` stays on the bind mount, so build output is visible from the host. +- A repository-owned `rust-toolchain.toml` is required. It is both what the image installs and part of what names it. +- Podman on Windows cannot mount build secrets (§8.2). [design]: ./README.md diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index ea425f64..5b0d5916 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -466,14 +466,14 @@ engine internals or the template format. ### 6.1 Placement: `justfiles/` holds recipes only -One placement rule is enforced rather than left to discovery, because getting it -wrong fails in a confusing place. `justfiles/anvil/` may contain `.just` recipes +One placement rule is enforced rather than left to discovery, because violating +it fails in a confusing place. `justfiles/anvil/` may contain `.just` recipes and nothing else: [`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects any other owned file under that prefix, so a derived catalog fails loudly at -construction instead of shipping a file whose absence is only noticed later. +construction instead of shipping a file whose absence is noticed only later. The reason is containerized execution. The image identity hashes every `*.just` -under `justfiles/anvil/` recursively, while the build context copies the whole +under `justfiles/anvil/` recursively, while the build context admits the whole directory, so a non-recipe file placed there is copied into the image but is **not** part of its identity: editing it would change what the image contains without renaming the tag, and no rebuild would follow. Non-recipe assets belong @@ -481,8 +481,8 @@ in a tool-owned directory of their own, such as `.anvil/`. Containerized execution is itself an ordinary artifact group, customized with the same `replace_artifact` / `with_artifact` / `without_artifact` levers as -anything else; the artifacts it exposes and the contract each one carries are -specified in [containers.md](./containers.md#7-customizing-the-image). +anything else. The artifacts it exposes and the contract each one carries are +specified in [containers.md](./containers.md#9-customization). The public engine contains no environment-specific image, registry, cloud, or credential-provider details. diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index bd35e5db..833f0673 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -96,14 +96,15 @@ //! //! ## Containerized local checks //! -//! Any generated recipe can run in a content-addressed Linux container. The -//! image installs the Rust toolchain and Cargo tools that this repository -//! pins, by running `just anvil-setup` — the same recipe the checks use — so -//! the container and the host agree on the toolset by construction. +//! Any generated recipe can be executed inside a content-addressed Linux +//! image. The image installs the Rust toolchain and Cargo tools this +//! repository pins by running `just anvil-setup` — the same recipe the checks +//! use, reading the same generated pins — so the image and the host agree on +//! the toolset by construction, with no second tool list to keep in step. //! -//! `just anvil-pr` and every other recipe keep running natively; the container -//! is entered only through `anvil-container`, which takes any recipe name and -//! its arguments. +//! `just anvil-pr` and every other recipe continue to run natively. A +//! container is entered only through `anvil-container`, which takes any recipe +//! name and its arguments; nothing is routed into one implicitly. //! //! ```text //! just anvil-container anvil-clippy # one check @@ -112,6 +113,17 @@ //! just anvil-container # interactive shell //! ``` //! +//! The feature is two generated artifacts and one optional hook: +//! `justfiles/anvil/container.just` drives the engine, +//! `.anvil/container/Dockerfile` (with its `Dockerfile.dockerignore`) defines +//! what the image contains, and `.anvil/container/hooks.ps1` supplies +//! credentials when a repository needs them. There is no configuration file. +//! +//! One container is created per invocation, not per check. The repository is +//! bind-mounted at `/workspace`, so `target/` stays visible from the host, +//! while `CARGO_HOME` and `RUSTUP_HOME` live in named volumes that keep the +//! write-heavy paths off the host boundary. +//! //! ### Prerequisites //! //! - A container engine callable from the shell that runs `just`: Docker, or @@ -119,40 +131,44 @@ //! Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or //! Docker Engine installed only inside the default WSL distribution — no //! Windows CLI is needed in that last case, since anvil reaches the engine -//! through `wsl.exe` when it finds none on `PATH`. +//! through `wsl.exe` when it finds none on `PATH` and translates repository +//! paths with `wslpath`. //! - `just` and `PowerShell` Core (`pwsh`) on the host. //! - A repository-owned `rust-toolchain.toml`. //! -//! On ARM64 hosts the image is emulated as `linux/amd64`, so builds and checks -//! are substantially slower. +//! Docker is supported; Podman works on a best-effort basis, with two +//! documented gaps on Windows. The image is pinned to `linux/amd64`, so on +//! ARM64 hosts it is emulated and is substantially slower. //! //! ### Image identity //! -//! The tag *is* a SHA-256 over the inputs that define the image: the -//! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional -//! hook, and the generated recipe tree. A changed tool pin names a tag that -//! cannot already exist, so a build follows. There is no staleness check -//! because there is nothing to check: an image built here is, by -//! construction, built from the current inputs. An image *fetched* by the -//! resolve hook only claims as much — the hash is over source files and -//! cannot be re-derived from layers — so that claim rests on the registry it -//! came from having immutable tags and restricted push. +//! The tag *is* a SHA-256 digest over the inputs that define the image: the +//! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, +//! and every `*.just` under `justfiles/anvil/` other than the driver itself. +//! A changed tool pin names a tag that cannot already exist, so a build +//! follows. There is no staleness check because there is no staleness to +//! detect: a locally built image that is present was built from the inputs +//! that name it. An image *fetched* by the resolve hook only claims as much — +//! the digest is over source files and cannot be re-derived from layers — so +//! that claim is only as strong as the registry it came from, which should +//! have immutable tags and restricted push. //! //! `anvil-container-tag` prints the reference without building it, and is the -//! single place the hash is computed, so a publisher can tag an image with +//! single place the digest is computed, so a publisher can tag an image with //! exactly the reference a consumer will later look up. //! //! ### Controls //! //! | Variable | Effect | //! |---|---| -//! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. | +//! | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. Read at run time. | //! | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail when the image is missing instead of building it, which distinguishes a cache miss from a build failure. | //! | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | //! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves, ignoring the hook. | //! | `ANVIL_IN_CONTAINER=1` | Set inside the image; makes a nested invocation run natively. | //! -//! Supporting recipes: `anvil-container-tag`, `anvil-container-status`, +//! Supporting recipes: `anvil-container-tag`, `anvil-container-status` +//! (reports the engine and image without building or pulling), //! `anvil-container-rebuild`, and `anvil-container-down` (removes this //! repository's cache volumes). //! @@ -160,38 +176,44 @@ //! //! crates.io needs no credentials, so no hook is emitted by default. A //! repository or a downstream catalog that needs one adds -//! `.anvil/container/hooks.ps1`, which the recipe loads when present: +//! `.anvil/container/hooks.ps1`, which the recipe loads by path whenever the +//! file is present, whoever wrote it: //! //! ```powershell //! function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } //! function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } -//! function Anvil-ResolveImage { param($tag) (fetch-a-prebuilt-image $tag) } +//! function Anvil-ResolveImage { param($tag) (fetch-a-published-image $tag) } //! ``` //! -//! Build secrets are passed to `BuildKit` by environment variable name, so a -//! value never reaches a process argument and never reaches an image layer; -//! run-time values are forwarded into the container by name for the same -//! reason. An empty value is a hard error, because a build that quietly -//! proceeded without its credential would install a reduced tool set and then -//! be tagged with the hash a credentialed build produces. +//! All three are optional. Build secrets are passed to `BuildKit` by +//! environment variable name, so a value never reaches a process argument and +//! never reaches an image layer; run-time values are forwarded into the +//! container by name for the same reason. An empty value is a hard error, +//! because a build that quietly proceeded without its credential would install +//! a reduced tool set and then be tagged with the digest a credentialed build +//! produces. //! //! `Anvil-ResolveImage` is offered the tag when nothing local matches, and -//! returns the reference it made available — a registry reference, not a -//! local re-tag, so the run stays honest about where the image came from. It -//! is verified before use and every failure falls through to a local build: -//! a publisher that has not caught up must not block the change it has not -//! caught up with. +//! returns the reference it made available — a registry reference, used as-is +//! rather than re-tagged locally, so the run stays honest about where the +//! image came from. It is verified before use, and every failure falls through +//! to a local build: a publisher that has not caught up must not block the +//! change it has not caught up with. //! -//! The hook runs on the host with the developer's permissions, before any -//! container isolation. Only run one from a repository or catalog you trust. +//! The hook executes on the host, with the invoking user's permissions, before +//! any container isolation exists. Only use one from a repository or catalog +//! you trust. //! //! ### Customizing the image //! //! `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place //! for extra packages, and anvil's drift handling preserves the change. A //! downstream catalog that needs a different base OS or toolchain source for -//! every repository it manages replaces the artifact instead — see -//! [`artifacts::container`] and the design doc. +//! every repository it manages replaces the artifact instead. A replacement +//! that copies more of the tree must replace the ignore file with it, since +//! the build context admits only `justfiles/` and `rust-toolchain.toml`. See +//! [`artifacts::container`] and the design document for the full contract, +//! the host setup for each engine, and the known limitations. //! //! ## Checks and tiers //! From 3cf838e9fa830ec575f535c4788b811a4aa747a0 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 13 Aug 2026 20:09:04 +0200 Subject: [PATCH 15/81] docs(anvil): drop the podman-docker rationale and tighten container prose MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `podman-docker` alias is removed as a justification for not probing between engines, in the recipe comment and in `containers.md`. The behaviour is unchanged: anvil still uses only the engine `ANVIL_CONTAINER_ENGINE` names. The remaining reasons -- presence is not reachability, and a silent choice between two engines splits the image cache -- carry the point on their own. Several passages are rephrased as reference documentation rather than as development notes. The engine-resolution rationale, the WSL path translation and the podman limitations were written as defect narratives, reproducing the diagnosis rather than stating the behaviour a reader needs. The podman build-secret error text is kept, since it is what a user matches against. Two further corrections: the artifact count is three, not two -- `container.just`, the `Dockerfile` and its ignore file, as `artifacts::container::all()` returns and §9 already said -- and `hooks.ps1` is described as one optional script that may define up to three functions, rather than as a single hook. `justfiles/anvil/container.just` and the three tree snapshots are regenerated from the template. `cargo run -p cargo-anvil -- anvil --dry-run` reports 78 items, all unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cdda814-67ed-42b6-9c60-01149a70f78d --- .anvil.lock | 4 +- crates/cargo-anvil/docs/design/containers.md | 53 ++++++++++--------- .../templates/justfiles/anvil/container.just | 9 ++-- .../snapshots/snapshots__ado_backend.snap | 9 ++-- .../snapshots/snapshots__github_backend.snap | 9 ++-- .../snapshots/snapshots__local_only.snap | 9 ++-- justfiles/anvil/container.just | 9 ++-- 7 files changed, 49 insertions(+), 53 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index cc564bfe..2f9b6645 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:1e1bac3acae52cb89679ae843c067f1cecbb151a388bd12ff321df6b458f1048" +catalog_checksum = "sha256:15e379de0548eca5c544c103306e27ae1ce9d5940523a8ccabb1a331e1317ff1" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:ef8d47ebb0fac3c07026c0f16546f4a27f1cd9e1829988a95430833500add8db" +checksum = "sha256:45e61b9681b05537a242b4f3d8305cb4e98329bb3b0506a6c4d267b347c0e787" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 5d900cc1..aa4ad183 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -9,8 +9,8 @@ just anvil-container anvil-pr # the whole PR tier just anvil-container # interactive shell ``` -The feature consists of two generated artifacts and one optional hook. There is no configuration file, and no -invocation is routed into a container implicitly. +The feature consists of three generated artifacts and one optional hook file. Containerized execution is opt-in per +invocation: recipes run natively unless you ask for a container by name. There is no configuration file. See [README.md](./README.md) for the overall design principles, [local.md](./local.md) for the recipe surface this wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstream fork uses. @@ -77,8 +77,8 @@ their own agents. The image is pinned to resemble that environment, not to repro ## 3. Execution model -One container is created per `anvil-container` invocation, not one per check. The container is removed on exit -(`--rm`). +One container is created per `anvil-container` invocation, however many checks the requested recipe runs. It is +removed on exit (`--rm`). ### 3.1 Engine resolution @@ -94,23 +94,23 @@ Resolution proceeds in a fixed order: If the probe succeeds, every subsequent engine call is prefixed with `wsl.exe --`. 3. Otherwise the invocation fails with a message naming the variable and linking to this document. -anvil does not probe for an engine other than the one requested. Presence is not reachability; `podman-docker` aliases -`docker` onto podman; and silently selecting between two installed engines yields two image stores and an unexplained -rebuild. Every failure other than a missing binary surfaces the engine's own diagnostic unmodified. +anvil uses the engine you select and never falls back to the other one. If `ANVIL_CONTAINER_ENGINE` names an engine +that is not usable, the invocation fails rather than substituting a different one. Apart from a missing binary, anvil +does not interpret engine failures: the engine's own diagnostic is shown unchanged. -Step 2 exists because installing Docker Engine inside WSL without Docker Desktop leaves no Windows CLI on `PATH`, and -that setup is the one this repository's own development guide describes. Docker Desktop and Podman both install a -Windows CLI, are found in step 1, and never reach step 2. +Automatic detection is avoided deliberately. A binary on `PATH` does not prove a reachable daemon, and choosing +silently between two installed engines would split the image cache across two stores, producing rebuilds with no +visible cause. + +Step 2 accommodates Docker Engine installed inside WSL without Docker Desktop, which leaves no Windows CLI on `PATH`. +Docker Desktop and Podman both install one, so they resolve at step 1 and never reach it. ### 3.2 Path translation When the engine is reached through WSL it does not share the Windows filesystem view, so host paths are translated -with `wslpath -a -u` before they are passed as a bind-mount source, a build context, or a `--file` argument. An -untranslated Windows path is not rejected by the daemon: it is bind-mounted as an empty directory, and the failure -surfaces much later as a missing file inside the container. - -Paths are converted to forward slashes before translation, because arguments crossing into WSL pass through a shell -that would otherwise consume the backslashes. `wslpath` accepts either separator. +with `wslpath -a -u` before they are passed as a bind-mount source, a build context, or a `--file` argument. A path +that is not translated is not rejected by the engine — it silently resolves to an empty directory — so the +translation is applied to every path anvil hands over. ### 3.3 Mounts and working directory @@ -244,14 +244,17 @@ compose: `anvil-container-status` sets both `NO_REBUILD` and `NO_RESOLVE`, and a ## 7. The hook -`.anvil/container/hooks.ps1` supplies the two things anvil cannot derive: credentials, and where a prebuilt image -might be obtained. The file is optional and is not emitted by default — crates.io requires no credentials, and an -empty script would be one more generated file to review. +`.anvil/container/hooks.ps1` is a single optional PowerShell script supplying the two things anvil cannot derive: +credentials, and where a published image might be obtained. It is not emitted by default — crates.io requires no +credentials, and an empty script would be one more generated file to review. It is loaded by path rather than by provenance: the recipe dot-sources it whenever the file exists, whether a repository wrote it or a catalog shipped it. A repository can therefore adopt a credential flow without forking the catalog. +The script may define up to three independent functions, each invoked at a different point. All are optional, and each +is called only if the loaded script defined it. + | Function | Invoked | Returns | | --- | --- | --- | | `Anvil-PreBuild` | before a build | `@{ Secrets = @{ = } }` | @@ -396,19 +399,17 @@ podman machine start $env:ANVIL_CONTAINER_ENGINE = 'podman' ``` -Three differences from Docker are known and unresolved: +Podman differs from Docker in three respects: -- **Build secrets are unavailable on podman for Windows.** Podman derives a temporary path from the build context - after translating it into the machine's view, then joins it using a Windows separator, so any `--secret` fails - before the build begins: +- **Build secrets are not supported on Windows.** A build that mounts one fails before it starts, with an error + naming a temporary file: ```text Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 ``` - The defect is in podman, and no form of the flag avoids it. It affects only a repository whose hook defines - `Anvil-PreBuild` (§7.1); building, running, and tag reuse are unaffected. Use Docker if you need build-time - credentials on Windows. + This affects only a repository whose hook defines `Anvil-PreBuild` (§7.1); building, running, and tag reuse are + unaffected. Use Docker if you need build-time credentials on Windows. - **The ignore file is passed explicitly.** buildah honours only an ignore file at the context root, so anvil passes `--ignorefile` on podman. Without it the entire worktree, including `target/`, is streamed to the daemon on every diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 3610e362..0c6324e0 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -33,11 +33,10 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di # Resolve how to invoke the engine, as a pipe-separated command. # # There is deliberately no probe *between* engines: presence is not -# reachability, `podman-docker` aliases `docker` onto podman, and a silent -# choice between two installed engines means two image stores and an -# unexplained rebuild. We check that the requested binary exists and let every -# other failure surface the engine's own diagnostic, which is more accurate -# than anything repeated here. +# reachability, and a silent choice between two installed engines means two +# image stores and an unexplained rebuild. We check that the requested binary +# exists and let every other failure surface the engine's own diagnostic, which +# is more accurate than anything repeated here. # # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 8a6cfadd..36b79371 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3511,11 +3511,10 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di # Resolve how to invoke the engine, as a pipe-separated command. # # There is deliberately no probe *between* engines: presence is not -# reachability, `podman-docker` aliases `docker` onto podman, and a silent -# choice between two installed engines means two image stores and an -# unexplained rebuild. We check that the requested binary exists and let every -# other failure surface the engine's own diagnostic, which is more accurate -# than anything repeated here. +# reachability, and a silent choice between two installed engines means two +# image stores and an unexplained rebuild. We check that the requested binary +# exists and let every other failure surface the engine's own diagnostic, which +# is more accurate than anything repeated here. # # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 860052df..e03178d8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3432,11 +3432,10 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di # Resolve how to invoke the engine, as a pipe-separated command. # # There is deliberately no probe *between* engines: presence is not -# reachability, `podman-docker` aliases `docker` onto podman, and a silent -# choice between two installed engines means two image stores and an -# unexplained rebuild. We check that the requested binary exists and let every -# other failure surface the engine's own diagnostic, which is more accurate -# than anything repeated here. +# reachability, and a silent choice between two installed engines means two +# image stores and an unexplained rebuild. We check that the requested binary +# exists and let every other failure surface the engine's own diagnostic, which +# is more accurate than anything repeated here. # # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 83a3c81f..bf66c4e3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2251,11 +2251,10 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di # Resolve how to invoke the engine, as a pipe-separated command. # # There is deliberately no probe *between* engines: presence is not -# reachability, `podman-docker` aliases `docker` onto podman, and a silent -# choice between two installed engines means two image stores and an -# unexplained rebuild. We check that the requested binary exists and let every -# other failure surface the engine's own diagnostic, which is more accurate -# than anything repeated here. +# reachability, and a silent choice between two installed engines means two +# image stores and an unexplained rebuild. We check that the requested binary +# exists and let every other failure surface the engine's own diagnostic, which +# is more accurate than anything repeated here. # # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 3610e362..0c6324e0 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -33,11 +33,10 @@ anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_di # Resolve how to invoke the engine, as a pipe-separated command. # # There is deliberately no probe *between* engines: presence is not -# reachability, `podman-docker` aliases `docker` onto podman, and a silent -# choice between two installed engines means two image stores and an -# unexplained rebuild. We check that the requested binary exists and let every -# other failure surface the engine's own diagnostic, which is more accurate -# than anything repeated here. +# reachability, and a silent choice between two installed engines means two +# image stores and an unexplained rebuild. We check that the requested binary +# exists and let every other failure surface the engine's own diagnostic, which +# is more accurate than anything repeated here. # # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing From 517cfbd3fa656ee334c82d5b47cb62de86d3077b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 09:58:56 +0200 Subject: [PATCH 16/81] docs(anvil): restructure the container reference, overview before detail `containers.md` is reordered so each concept is introduced once, at the level a reader needs it, and detail follows the overview it depends on: interface -> artifacts -> identity -> runtime -> engines -> hook -> customization Two sections move as a result. The environment variables join the recipe table in "Command surface", since together they are the complete user-facing interface, and engine resolution and path translation move out of the execution model into "Engines and host setup", which previously repeated them from the other side. Duplication removed. The `just anvil-setup` install path was explained in the artifacts section and again under hashed inputs; the build-context scope was stated in artifacts and again under customization; the engine diagnostic policy appeared in both engine resolution and host requirements; "all three functions are optional" appeared twice in consecutive paragraphs; and the limitations section restated the platform pin, the toolchain requirement and the podman gap already covered above. Each now appears once, with a section reference where a reader might look for it elsewhere. The result is 10% shorter (3714 to 3338 words) while covering the same surface. `lib.rs` is corrected to match: it still described the feature as two generated artifacts, which the previous commit fixed only in the design document. `README.md` is regenerated from it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cdda814-67ed-42b6-9c60-01149a70f78d --- crates/cargo-anvil/README.md | 40 +- crates/cargo-anvil/docs/design/containers.md | 514 +++++++++---------- crates/cargo-anvil/src/lib.rs | 38 +- 3 files changed, 273 insertions(+), 319 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 81db299e..884bfcd3 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -99,13 +99,13 @@ and run each check only over the affected packages, whereas a local Any generated recipe can be executed inside a content-addressed Linux image. The image installs the Rust toolchain and Cargo tools this -repository pins by running `just anvil-setup` — the same recipe the checks -use, reading the same generated pins — so the image and the host agree on +repository pins by running `just anvil-setup`, the same recipe the checks +use, reading the same generated pins, so the image and the host agree on the toolset by construction, with no second tool list to keep in step. -`just anvil-pr` and every other recipe continue to run natively. A -container is entered only through `anvil-container`, which takes any recipe -name and its arguments; nothing is routed into one implicitly. +Execution is opt-in per invocation: `just anvil-pr` and every other recipe +continue to run natively, and a container is entered only through +`anvil-container`, which takes any recipe name and its arguments. ```text just anvil-container anvil-clippy # one check @@ -114,23 +114,23 @@ just anvil-container anvil-setup binstall # a recipe with an argument just anvil-container # interactive shell ``` -The feature is two generated artifacts and one optional hook: -`justfiles/anvil/container.just` drives the engine, -`.anvil/container/Dockerfile` (with its `Dockerfile.dockerignore`) defines -what the image contains, and `.anvil/container/hooks.ps1` supplies -credentials when a repository needs them. There is no configuration file. +The feature is three generated artifacts and one optional hook script, with +no configuration file: `justfiles/anvil/container.just` drives the engine, +`.anvil/container/Dockerfile` and its `Dockerfile.dockerignore` define what +the image contains, and `.anvil/container/hooks.ps1` supplies credentials +when a repository needs them. -One container is created per invocation, not per check. The repository is -bind-mounted at `/workspace`, so `target/` stays visible from the host, -while `CARGO_HOME` and `RUSTUP_HOME` live in named volumes that keep the -write-heavy paths off the host boundary. +One container is created per invocation, however many checks the requested +recipe runs. The repository is bind-mounted at `/workspace`, so `target/` +stays visible from the host, while `CARGO_HOME` and `RUSTUP_HOME` live in +named volumes that keep the write-heavy paths off the host boundary. #### Prerequisites * A container engine callable from the shell that runs `just`: Docker, or Podman via `ANVIL_CONTAINER_ENGINE=podman`. On Windows that means Docker Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or - Docker Engine installed only inside the default WSL distribution — no + Docker Engine installed only inside the default WSL distribution. No Windows CLI is needed in that last case, since anvil reaches the engine through `wsl.exe` when it finds none on `PATH` and translates repository paths with `wslpath`. @@ -149,9 +149,9 @@ and every `*.just` under `justfiles/anvil/` other than the driver itself. A changed tool pin names a tag that cannot already exist, so a build follows. There is no staleness check because there is no staleness to detect: a locally built image that is present was built from the inputs -that name it. An image *fetched* by the resolve hook only claims as much — -the digest is over source files and cannot be re-derived from layers — so -that claim is only as strong as the registry it came from, which should +that name it. An image *fetched* by the resolve hook only claims as much, +since the digest is over source files and cannot be re-derived from layers, +so that claim is only as strong as the registry it came from, which should have immutable tags and restricted push. `anvil-container-tag` prints the reference without building it, and is the @@ -195,7 +195,7 @@ a reduced tool set and then be tagged with the digest a credentialed build produces. `Anvil-ResolveImage` is offered the tag when nothing local matches, and -returns the reference it made available — a registry reference, used as-is +returns the reference it made available: a registry reference, used as-is rather than re-tagged locally, so the run stays honest about where the image came from. It is verified before use, and every failure falls through to a local build: a publisher that has not caught up must not block the @@ -434,7 +434,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbpACaXmKNr48bH5E05_uOLpIby9m8IWn7SQMbLMeQnQJv6axhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbAO_FZGTrWRcbQnEvdCe9uvIbIY7pQUwg99AbRwGoU1ynfVJhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index aa4ad183..d60bf1af 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -9,36 +9,34 @@ just anvil-container anvil-pr # the whole PR tier just anvil-container # interactive shell ``` -The feature consists of three generated artifacts and one optional hook file. Containerized execution is opt-in per -invocation: recipes run natively unless you ask for a container by name. There is no configuration file. +Execution is opt-in per invocation: recipes run natively unless a container is requested by name. The feature is three +generated artifacts and one optional hook script, with no configuration file. See [README.md](./README.md) for the overall design principles, [local.md](./local.md) for the recipe surface this wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstream fork uses. - [1. Purpose](#1-purpose) - [2. Command surface](#2-command-surface) -- [3. Execution model](#3-execution-model) - - [3.1 Engine resolution](#31-engine-resolution) - - [3.2 Path translation](#32-path-translation) - - [3.3 Mounts and working directory](#33-mounts-and-working-directory) - - [3.4 Process identity](#34-process-identity) - - [3.5 Re-entry](#35-re-entry) -- [4. Emitted artifacts](#4-emitted-artifacts) -- [5. Image identity](#5-image-identity) - - [5.1 Hashed inputs](#51-hashed-inputs) - - [5.2 Digest computation](#52-digest-computation) - - [5.3 Guarantees](#53-guarantees) -- [6. Environment variables](#6-environment-variables) +- [3. Emitted artifacts](#3-emitted-artifacts) +- [4. Image identity](#4-image-identity) + - [4.1 Inputs](#41-inputs) + - [4.2 Digest](#42-digest) + - [4.3 What the tag guarantees](#43-what-the-tag-guarantees) +- [5. Execution model](#5-execution-model) + - [5.1 Mounts and working directory](#51-mounts-and-working-directory) + - [5.2 Process identity](#52-process-identity) + - [5.3 Re-entry](#53-re-entry) +- [6. Engines and host setup](#6-engines-and-host-setup) + - [6.1 Engine resolution](#61-engine-resolution) + - [6.2 Docker](#62-docker) + - [6.3 Podman](#63-podman) - [7. The hook](#7-the-hook) - [7.1 Anvil-PreBuild](#71-anvil-prebuild) - [7.2 Anvil-PreRun](#72-anvil-prerun) - [7.3 Anvil-ResolveImage](#73-anvil-resolveimage) - [7.4 Trust boundary](#74-trust-boundary) -- [8. Host requirements](#8-host-requirements) - - [8.1 Docker](#81-docker) - - [8.2 Podman](#82-podman) -- [9. Customization](#9-customization) -- [10. Limitations](#10-limitations) +- [8. Customization](#8-customization) +- [9. Limitations](#9-limitations) ## 1. Purpose @@ -50,12 +48,13 @@ owns it locally"). Two conditions invalidate that assumption: 2. **Toolchain divergence.** The installed toolset can differ from the one the checks expect, so a passing local run stops predicting a cloud result. -Both are addressed by executing the recipe unchanged inside an image constructed from the repository's own pins. +Both are addressed by executing the recipe unchanged inside an image built from the repository's own pins. Recipe +bodies are identical in either mode, and cloud workflows are unaffected: they run the same recipes natively on their +own agents. The image is pinned to resemble that environment, not to reproduce it. ## 2. Command surface -`just anvil-pr` and every other recipe continue to execute natively. A container is entered only through -`anvil-container`, which accepts a recipe name and its arguments: +`anvil-container` accepts a recipe name and its arguments; every other recipe continues to execute natively. ```bash just anvil-container anvil-setup binstall @@ -71,48 +70,98 @@ just anvil-container anvil-setup binstall All five are annotated `[group("anvil-container")]` and appear as one cluster in `just --groups`. -Recipe bodies are identical in both execution modes. No wrapper shadows `just` on `PATH`, and no recipe behaves -differently according to where it runs. Cloud workflows are unaffected: they execute the same recipes natively on -their own agents. The image is pinned to resemble that environment, not to reproduce it. +| Variable | Effect | +| --- | --- | +| `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. Read at run time (§6.1). | +| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is absent, separating a cache miss from a build failure. | +| `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook (§7.3), so a query never pulls. | +| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag resolves. Skips the resolve hook too (§7.3). | +| `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§5.3). | -## 3. Execution model +`NO_REBUILD` is evaluated independently of `NO_CACHE`, so the two compose: `anvil-container-status` sets `NO_REBUILD` +and `NO_RESOLVE` together and answers from local state alone. When `NO_REBUILD` stops a build the reference is still +printed, since a caller that asked not to build is usually asking which image is missing. -One container is created per `anvil-container` invocation, however many checks the requested recipe runs. It is -removed on exit (`--rm`). +## 3. Emitted artifacts -### 3.1 Engine resolution +```text +repo/ +├── justfiles/anvil/ +│ ├── container.just the anvil-container recipes +│ └── … checks, groups, tiers, executed natively *inside* the image +└── .anvil/container/ + ├── Dockerfile what the image contains + ├── Dockerfile.dockerignore what the build context admits + └── hooks.ps1 optional; not emitted by default (§7) +``` -`ANVIL_CONTAINER_ENGINE` selects the engine and defaults to `docker`. Any value other than `docker` or `podman` is -rejected before the engine is invoked. Because the engine is a property of the host rather than of the repository, it -is read at run time and is never committed; a single invocation can override it with -`just anvil_container_engine=podman anvil-container anvil-pr`. +`container.just` is reconciled on every run, so local edits to it are replaced. The `Dockerfile` and its ignore file +are generated but intended to be edited; anvil's drift handling preserves a repository's changes to them (§8). -Resolution proceeds in a fixed order: +The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same +generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the +image (§4.1). -1. If the named binary is on `PATH`, it is invoked directly. -2. Otherwise, on Windows, the binary is probed inside the default WSL distribution (`wsl.exe -- --version`). - If the probe succeeds, every subsequent engine call is prefixed with `wsl.exe --`. -3. Otherwise the invocation fails with a message naming the variable and linking to this document. +`Dockerfile.dockerignore` scopes the build context to `justfiles/` and `rust-toolchain.toml`, denying everything else. +BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the repository neither needs to +own a root ignore file nor can have one silently override this. -anvil uses the engine you select and never falls back to the other one. If `ANVIL_CONTAINER_ENGINE` names an engine -that is not usable, the invocation fails rather than substituting a different one. Apart from a missing binary, anvil -does not interpret engine failures: the engine's own diagnostic is shown unchanged. +## 4. Image identity -Automatic detection is avoided deliberately. A binary on `PATH` does not prove a reachable daemon, and choosing -silently between two installed engines would split the image cache across two stores, producing rebuilds with no -visible cause. +The image reference is `anvil-:<16 hex characters>`, where the tag is a SHA-256 digest over the inputs that +define the image. The name derives from the repository directory (§5.1). -Step 2 accommodates Docker Engine installed inside WSL without Docker Desktop, which leaves no Windows CLI on `PATH`. -Docker Desktop and Podman both install one, so they resolve at step 1 and never reach it. +### 4.1 Inputs -### 3.2 Path translation +| Input | Hashed | +| --- | --- | +| `.anvil/container/Dockerfile` | always | +| `.anvil/container/Dockerfile.dockerignore` | always | +| `rust-toolchain.toml` | always | +| `.anvil/container/hooks.ps1` | when the file exists | +| `justfiles/anvil/**/*.just` | always, recursively, except `container.just` | + +The recipe tree is an input because `just anvil-setup` decides what the image installs (§3), and its dependency chain +reaches the tier, group, check, and tool recipes. `container.just` is excluded because hashing the driver would make +the tag depend on the tag. A declared input that does not exist is a hard error, not an omission from the digest. + +The hook file's **content** is an input, since it determines what the build installs. Its **output** is deliberately +excluded: a credential must never influence a tag. + +### 4.2 Digest + +Inputs are sorted by relative path with an ordinal comparison, then serialized into one stream in which each entry +contributes a literal `file`, its relative path, and its content, each newline-terminated. Tagging entries this way +prevents a rearrangement of names and contents from colliding. Line endings are normalized to LF, so CRLF and LF +checkouts agree on the tag. The sort is ordinal because a case-insensitive one would drop one of two inputs differing +only in case on the case-sensitive filesystem where the image is built. -When the engine is reached through WSL it does not share the Windows filesystem view, so host paths are translated -with `wslpath -a -u` before they are passed as a bind-mount source, a build context, or a `--file` argument. A path -that is not translated is not rejected by the engine — it silently resolves to an empty directory — so the -translation is applied to every path anvil hands over. +The tag is the first eight bytes of the digest, hex-encoded: 64 bits, far beyond practical collision risk for a local +image set, and short enough to keep `docker images` readable. + +`anvil-container-tag` is the only place the digest is computed; every other recipe calls it. A publisher and a +consumer therefore derive the same reference independently, with no `latest` tag and no digest maintained by hand. + +### 4.3 What the tag guarantees + +Changing an input names a tag that cannot already exist, so a build follows; changing nothing resolves the existing +tag immediately. There is no staleness check because a locally built image that is present was built from the inputs +that name it. + +That holds only for a locally built image. One obtained through `Anvil-ResolveImage` (§7.3) merely *claims* those +inputs, since the digest covers source files and cannot be re-derived from layers, so the claim is only as strong as +its registry. Publish to a registry with immutable tags and restricted push. + +Two properties sit outside the digest. The base image is not resolved during hashing, so `ARG BASE_IMAGE` must remain +digest-pinned; a floating tag could otherwise change beneath a tag that claims to name fixed content. The platform is +pinned to `linux/amd64` on build and run, so hosts of differing architecture cannot compute one tag for two images. + +## 5. Execution model + +One container is created per `anvil-container` invocation, however many checks the requested recipe runs, and is +removed on exit (`--rm`). -### 3.3 Mounts and working directory +### 5.1 Mounts and working directory | Mount | Target | Purpose | | --- | --- | --- | @@ -120,140 +169,128 @@ translation is applied to every path anvil hands over. | `anvil--cargo` (volume) | `/usr/local/cargo` | `CARGO_HOME`: registry cache and installed binaries. | | `anvil--rustup` (volume) | `/usr/local/rustup` | `RUSTUP_HOME`: installed toolchains. | -The cargo and rustup homes are named volumes rather than bind mounts, so the write-heavy paths never cross the host -boundary and the host's own toolchain is untouched. `target/` remains on the bind mount, so build output stays visible -from the host and is shared between native and containerized runs. +The cargo and rustup homes are named volumes, so write-heavy paths never cross the host boundary and the host's own +toolchain is untouched. `target/` stays on the bind mount, remaining visible from the host and shared between native +and containerized runs. The caller's working directory is mapped to its in-container equivalent, so relative paths +resolve when `anvil-container` is invoked from a subdirectory. -The caller's working directory is mapped to its in-container equivalent, so relative paths continue to resolve when -`anvil-container` is invoked from a subdirectory. +Image and volume names derive from the repository directory name, lowercased with every character outside +`[a-z0-9._-]` replaced by `-`. Two checkouts with the same directory name therefore share cache volumes. That is +harmless in normal use, since cargo's caches are content-addressed, but `anvil-container-down` then removes volumes +the other checkout is also using. -Volume names derive from the repository directory name, lowercased with every character outside `[a-z0-9._-]` -replaced by `-`. Two checkouts with the same directory name therefore share cache volumes. This is harmless in normal -use, because cargo's caches are content-addressed, but `anvil-container-down` removes volumes that the other checkout -is also using. +### 5.2 Process identity -### 3.4 Process identity +On a Linux host the run passes `--user :`, matching the invoking user, unless that user is root. Without it, +everything written under the bind mount is owned by root on the host, and the next native `cargo build` or `git clean` +fails with `EACCES` far from the cause. Docker Desktop on Windows and macOS maps ownership itself, and `id` is not +available to query, so the flag is not passed there. -On a Linux host the run passes `--user :`, matching the invoking user. Without it, everything written under -the bind mount — `target/`, generated files — is owned by root on the host, and the next native `cargo build` or -`git clean` fails with `EACCES` far from the cause. The flag is omitted when the invoking user is root. +### 5.3 Re-entry -Docker Desktop on Windows and macOS maps ownership itself, and `id` is not available to query, so the flag is not -passed on those hosts. +`ANVIL_IN_CONTAINER=1` is set in the image and passed on each run. `anvil-container` checks it first and, inside the +image, executes the requested recipe directly instead of launching another container, so a recipe that reaches +`anvil-container` transitively still performs its work once. -### 3.5 Re-entry +## 6. Engines and host setup -`ANVIL_IN_CONTAINER=1` is set in the image and passed again on each run. `anvil-container` checks it first: inside the -image, the requested recipe is executed directly instead of launching another container. A recipe that reaches -`anvil-container` transitively therefore performs its work exactly once. +anvil installs nothing and manages no virtual machine. Beyond the engine, the host needs `just` and PowerShell Core +(`pwsh`), which every generated recipe requires, and the repository must own a `rust-toolchain.toml`. -## 4. Emitted artifacts +| | Docker | Podman | +| --- | --- | --- | +| Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | +| Builder | BuildKit | buildah | +| Status | supported | best-effort; see §6.3 | -```text -repo/ -├── justfiles/anvil/ -│ ├── container.just the anvil-container recipes -│ └── … checks, groups, tiers — executed natively *inside* the image -└── .anvil/container/ - ├── Dockerfile what the image contains - ├── Dockerfile.dockerignore what the build context admits - └── hooks.ps1 optional; not emitted by default (§7) -``` +### 6.1 Engine resolution -`container.just` is generated and reconciled on every run; local edits to it are replaced. The `Dockerfile` and its -ignore file are generated but intended to be edited: anvil's drift handling preserves a repository's changes to them -(§9). +`ANVIL_CONTAINER_ENGINE` defaults to `docker`, and any value other than `docker` or `podman` is rejected before the +engine is invoked. Being a host property rather than a repository one, it is read at run time and never committed; a +single invocation can override it with `just anvil_container_engine=podman anvil-container anvil-pr`. -The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated -pins. There is no second tool list to keep synchronized, which is also why a tool-pin change renames the image: -`versions.just` is both what the image installs and part of what names it (§5). +Resolution proceeds in a fixed order: -The build context is scoped by `Dockerfile.dockerignore`, a deny-all list that re-admits `justfiles/` and -`rust-toolchain.toml` and nothing else. BuildKit reads `.dockerignore` in preference to a root -`.dockerignore`, so the repository does not need to own a root ignore file and cannot have one silently overridden. +1. If the named binary is on `PATH`, it is invoked directly. +2. Otherwise, on Windows, the binary is probed inside the default WSL distribution (`wsl.exe -- --version`). + If the probe succeeds, every subsequent engine call is prefixed with `wsl.exe --`. This accommodates Docker Engine + installed inside WSL without Docker Desktop, which leaves no Windows CLI behind; Docker Desktop and Podman both + install one and resolve at step 1. +3. Otherwise the invocation fails with a message naming the variable and linking to this document. -## 5. Image identity +anvil never falls back to the other engine: if the selected one is unusable, the invocation fails rather than +substituting. Automatic detection is avoided deliberately, because a binary on `PATH` does not prove a reachable +daemon, and silently choosing between two installed engines would split the image cache across two stores and produce +rebuilds with no visible cause. A missing binary is the only failure anvil reports itself; every other engine +diagnostic is shown unchanged. -The image reference is `anvil-:<16 hex characters>`, where the tag is a SHA-256 digest over the inputs that -define the image. The name is derived from the repository directory as described in §3.3. +When the engine is reached through WSL it does not share the Windows filesystem view, so anvil translates every path +it hands over (bind-mount source, build context, `--file`) with `wslpath -a -u`. An untranslated path is not +rejected by the engine; it silently resolves to an empty directory. -### 5.1 Hashed inputs +### 6.2 Docker -| Input | Hashed | -| --- | --- | -| `.anvil/container/Dockerfile` | always | -| `.anvil/container/Dockerfile.dockerignore` | always | -| `rust-toolchain.toml` | always | -| `.anvil/container/hooks.ps1` | when the file exists | -| `justfiles/anvil/**/*.just` | always, recursively, except `container.just` | +**Linux.** Install Docker Engine from your distribution or `get.docker.com`, and add your user to the `docker` group. -The recipe tree is included because the image installs its tools by running `just anvil-setup`, whose dependency chain -reaches the tier, group, check, and tool recipes. `container.just` is excluded because hashing the driver would make -the tag depend on the tag. +**Windows, with Docker Desktop.** No configuration required: `docker` is on `PATH`. -The hook file's **content** is an input, because it determines what the build installs. Its **output** is deliberately -excluded: a credential must never influence a tag. +**Windows, Docker Engine in WSL.** No Docker Desktop and no Windows CLI: -A declared input that does not exist is a hard error rather than an omission from the digest. +```powershell +wsl --install -d Ubuntu-24.04 +wsl -d Ubuntu-24.04 -- sh -c 'printf "[boot]\nsystemd=true\n" | sudo tee /etc/wsl.conf' +wsl -d Ubuntu-24.04 -- sh -c 'curl -fsSL https://get.docker.com | sh' +wsl -d Ubuntu-24.04 -- sudo usermod -aG docker "$USER" +wsl --shutdown +wsl -d Ubuntu-24.04 -- docker version # verify +``` -### 5.2 Digest computation +`just` and `pwsh` remain on Windows; the distribution needs only Docker. Installing a Windows `docker` CLI and +pointing `DOCKER_HOST` at the WSL socket also works and takes precedence, since the WSL path is used only when no CLI +is found. The daemon is Linux-side either way, so the repository must be bind-mountable at a path it can resolve. -Inputs are sorted by relative path using an ordinal comparison, then serialized into a single stream. Each entry -contributes a literal `file`, its relative path, and its content, each terminated by a newline. Tagging each entry -this way ensures no rearrangement of names and contents can produce a collision. Line endings are normalized to LF, so -a CRLF checkout and an LF checkout compute the same tag. The ordinal sort matters because a case-insensitive one would -silently drop one of two inputs differing only in case on the case-sensitive filesystem where the image is built. +### 6.3 Podman -The tag is the first eight bytes of the digest, hex-encoded — 64 bits, far beyond any practical collision risk for a -local image set, and short enough to keep `docker images` readable. +**Linux.** Install podman and set `ANVIL_CONTAINER_ENGINE=podman`. -`anvil-container-tag` is the only place this computation exists; every other recipe calls it. A publisher and a -consumer therefore derive the same reference independently, with no `latest` tag and no digest maintained by hand. +**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and `podman.exe` is placed on +`PATH`, so anvil invokes it directly. -### 5.3 Guarantees +```powershell +winget install RedHat.Podman-Desktop # or the podman CLI alone +podman machine init +podman machine start +$env:ANVIL_CONTAINER_ENGINE = 'podman' +``` -Changing any input names a tag that cannot already exist, so a build follows. Changing nothing resolves the existing -tag immediately. There is no staleness check because there is no staleness to detect: a locally built image that is -present was built from the inputs that name it. +Podman differs from Docker in three respects: -That guarantee is exact only for a locally built image. An image obtained through `Anvil-ResolveImage` (§7.3) merely -*claims* those inputs — the digest is computed over source files and cannot be re-derived from layers — so the claim -is only as strong as the registry it came from. Publish to a registry with immutable tags, and restrict push to the -identity that builds them. +- **Build secrets are not supported on Windows.** A build that mounts one fails before it starts, with an error + naming a temporary file: -Two inputs sit outside the digest and must be pinned by other means. The base image is not resolved during hashing, so -`ARG BASE_IMAGE` must remain digest-pinned or a floating tag can change beneath a tag that claims to name fixed -content. The platform is pinned to `linux/amd64` on both build and run, so hosts of differing architecture cannot -compute one tag for two different images. + ```text + Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 + ``` -## 6. Environment variables + Only a repository whose hook defines `Anvil-PreBuild` (§7.1) is affected; building, running, and tag reuse are not. + Use Docker if you need build-time credentials on Windows. -| Variable | Effect | -| --- | --- | -| `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. Read at run time. | -| `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is absent, distinguishing a cache miss from a build failure. | -| `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | -| `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag already resolves. Also skips the resolve hook. | -| `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§3.5). | - -`ANVIL_CONTAINER_NO_CACHE` skips the hook because "ignore what is cached" must include the remote cache; otherwise a -rebuild would be undone by the next resolve. `ANVIL_CONTAINER_NO_REBUILD` is evaluated independently of it, so the two -compose: `anvil-container-status` sets both `NO_REBUILD` and `NO_RESOLVE`, and answers from local state alone. When -`NO_REBUILD` stops a build, the reference is still printed, because a caller that asked not to build is usually asking -*which* image is missing. +- **The ignore file is passed explicitly.** buildah honours only an ignore file at the context root, so anvil passes + `--ignorefile`. Without it the entire worktree, `target/` included, is streamed to the daemon on every build. + +- **Rootless user-namespace mapping is not applied.** The run passes `--user` (§5.2) but not `--userns keep-id`, which + rootless podman requires for bind-mount ownership to map back to the invoking user. ## 7. The hook `.anvil/container/hooks.ps1` is a single optional PowerShell script supplying the two things anvil cannot derive: -credentials, and where a published image might be obtained. It is not emitted by default — crates.io requires no -credentials, and an empty script would be one more generated file to review. +credentials, and where a published image might be obtained. It is not emitted by default, since crates.io requires no +credentials and an empty script would be one more generated file to review. It is loaded by path rather than by provenance: the recipe dot-sources it whenever the file exists, whether a -repository wrote it or a catalog shipped it. A repository can therefore adopt a credential flow without forking the -catalog. +repository wrote it or a catalog shipped it, so a repository can adopt a credential flow without forking the catalog. -The script may define up to three independent functions, each invoked at a different point. All are optional, and each -is called only if the loaded script defined it. +The script may define up to three independent functions. All are optional, and each is called only if defined. | Function | Invoked | Returns | | --- | --- | --- | @@ -261,18 +298,20 @@ is called only if the loaded script defined it. | `Anvil-PreRun` | before a run | `@{ Env = @{ = } }` | | `Anvil-ResolveImage $tag` | before a build, when no local image matches | an image reference, or nothing | -All three are optional, and each is called only if defined. +Both value-returning functions **fail closed on an empty value**, which the engine does not: BuildKit accepts +`--secret id=t,env=UNSET`, mounts an empty secret and exits 0, so the build would install a reduced tool set, be +tagged with the digest a credentialed build produces, and be reused by every later run. -Both value-returning functions **fail closed on an empty value**. This is not the engine's behaviour: BuildKit accepts -`--secret id=t,env=UNSET`, mounts an empty secret, and exits 0. The build would install a reduced tool set, be tagged -with the same content hash a credentialed build produces, and be reused by every later run. +Credentials are passed to the engine **by name** in both phases, as a `--secret … env=` reference at build time and +`-e ` at run time, so a value never appears in a process command line, where endpoint telemetry records and +retains it far longer than a short-lived token is intended to live. The variables are removed once the engine call +returns, and when the engine is reached through WSL the names are exported through `WSLENV` so the values cross that +boundary. ### 7.1 Anvil-PreBuild -Each returned entry becomes a BuildKit `--secret id=,env=ANVIL_SECRET_` mount. The value is placed in a -process environment variable and passed **by name**, so it never appears in a command line, where endpoint telemetry -records and retains it far longer than a short-lived token is intended to live. The variables are removed once the -build completes. BuildKit keeps a mounted secret out of every image layer. +Each entry becomes a BuildKit `--secret id=,env=ANVIL_SECRET_` mount, which BuildKit keeps out of every image +layer. ```powershell function Anvil-PreBuild { @@ -280,10 +319,10 @@ function Anvil-PreBuild { } ``` -Minting the value inside a function is the point: a short-lived token must be acquired at the moment it is used, not -read from a committed file or a declared variable. +Minting the value inside the function is the point: a short-lived token must be acquired when it is used, not read +from a committed file or a declared variable. -Declare the mount as required in the Dockerfile, which closes the same gap from the build's side: +Declare the mount as required in the Dockerfile, closing the same gap from the build's side: ```dockerfile RUN --mount=type=secret,id=feed_token,required=true \ @@ -294,14 +333,10 @@ Anything the build *writes* using a secret is ordinary layer content. The defaul `credentials.toml` and `.netrc` in the same `RUN` layer as the install; a replacement must do the same, or the credential is baked into a layer that a later deletion cannot remove. -When the engine is reached through WSL, the secret variable names are exported through `WSLENV` so the values cross -that boundary. - ### 7.2 Anvil-PreRun -Each returned entry is forwarded into the container with `-e `, again by name rather than as `NAME=VALUE`, for -the reason given above. Inside the image the value is an ordinary environment variable. The forwarded names — never -their values — are echoed to stderr, because everything executing inside the container can read them. +Each entry is forwarded with `-e ` and is an ordinary environment variable inside the image. The forwarded +names, never their values, are echoed to stderr, because everything executing in the container can read them. ```powershell function Anvil-PreRun { @@ -311,8 +346,9 @@ function Anvil-PreRun { ### 7.3 Anvil-ResolveImage -When no local image matches the computed tag, the reference is offered to `Anvil-ResolveImage` before a build starts. -A catalog that publishes images implements it; without one, the build proceeds. +When no local image matches the computed tag, the reference is offered to `Anvil-ResolveImage` before a build starts, +ahead of the `NO_REBUILD` guard, since fetching a published image is not building one. A catalog that publishes images +implements it; without one, the build proceeds. ```powershell function Anvil-ResolveImage($tag) { @@ -325,103 +361,26 @@ function Anvil-ResolveImage($tag) { Three properties are load-bearing: -- **The returned reference is used as-is, never re-tagged to the local name.** A local tag asserts "built here from - these inputs"; a fetched image only claims it (§5.3). Retaining the registry reference keeps the run honest about - the image's origin. -- **The reference is verified before use.** Runs pass `--pull=never`, so a hook that reported an image it had not - actually fetched would otherwise fail later and further from the cause. -- **Every failure falls through to a local build**, with the reason printed — a missing image, an expired credential, - a hook that threw. A publisher that has not caught up with a change must not block the developer who made it. - -Resolution is attempted before the `ANVIL_CONTAINER_NO_REBUILD` guard, because fetching a published image is not -building one. +- **The returned reference is used as-is, never re-tagged locally.** A local tag asserts "built here from these + inputs"; a fetched image only claims it (§4.3). Keeping the registry reference keeps the run honest about origin. +- **The reference is verified before use.** Runs pass `--pull=never`, so a hook reporting an image it had not actually + fetched would otherwise fail later and further from the cause. +- **Every failure falls through to a local build**, with the reason printed. A publisher that has not caught up with a + change must not block the developer who made it. ### 7.4 Trust boundary The hook executes on the host, with the invoking user's permissions, before any container isolation exists. Only use one from a repository or catalog you trust. -Inside the container everything executes as a single user in a single mount namespace, so a forwarded credential is -readable by anything the checks execute, including dependency build scripts and procedural macros. Keep the forwarded -set narrow and the tokens short-lived. - -## 8. Host requirements - -anvil installs nothing and manages no virtual machine. It invokes the engine you selected and lets that engine's own -diagnostics surface. The only failure message it owns is for a missing binary, which names the variable to set and -links here. - -The engine must be callable from the shell that runs `just`, with the single Windows exception described in §3.1. The -host also needs `just` and PowerShell Core (`pwsh`), which every generated recipe requires, and the repository must -own a `rust-toolchain.toml`. - -| | Docker | Podman | -| --- | --- | --- | -| Selected by | default | `ANVIL_CONTAINER_ENGINE=podman` | -| Builder | BuildKit | buildah | -| Status | supported | best-effort; see §8.2 | - -### 8.1 Docker - -**Linux.** Install Docker Engine from your distribution or `get.docker.com`, and add your user to the `docker` group. - -**Windows, with Docker Desktop.** No configuration required: `docker` is on `PATH`. - -**Windows, Docker Engine in WSL.** No Docker Desktop and no Windows CLI: - -```powershell -wsl --install -d Ubuntu-24.04 -wsl -d Ubuntu-24.04 -- sh -c 'printf "[boot]\nsystemd=true\n" | sudo tee /etc/wsl.conf' -wsl -d Ubuntu-24.04 -- sh -c 'curl -fsSL https://get.docker.com | sh' -wsl -d Ubuntu-24.04 -- sudo usermod -aG docker "$USER" -wsl --shutdown -wsl -d Ubuntu-24.04 -- docker version # verify -``` - -`just` and `pwsh` remain on Windows; the distribution needs only Docker. anvil detects this configuration -automatically (§3.1) and translates paths accordingly (§3.2). - -Installing a Windows `docker` CLI and pointing `DOCKER_HOST` at the WSL socket also works and takes precedence, since -the WSL path is used only when no CLI is found on `PATH`. The daemon is then Linux-side, so the repository must be -bind-mountable at a path it can resolve. - -### 8.2 Podman - -**Linux.** Install podman and set `ANVIL_CONTAINER_ENGINE=podman`. - -**Windows.** `podman machine init` provisions and manages its own WSL2 virtual machine, and `podman.exe` is placed on -`PATH`, so anvil invokes it directly. - -```powershell -winget install RedHat.Podman-Desktop # or the podman CLI alone -podman machine init -podman machine start -$env:ANVIL_CONTAINER_ENGINE = 'podman' -``` - -Podman differs from Docker in three respects: - -- **Build secrets are not supported on Windows.** A build that mounts one fails before it starts, with an error - naming a temporary file: - - ```text - Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 - ``` - - This affects only a repository whose hook defines `Anvil-PreBuild` (§7.1); building, running, and tag reuse are - unaffected. Use Docker if you need build-time credentials on Windows. - -- **The ignore file is passed explicitly.** buildah honours only an ignore file at the context root, so anvil passes - `--ignorefile` on podman. Without it the entire worktree, including `target/`, is streamed to the daemon on every - build. - -- **Rootless user-namespace mapping is not applied.** The run passes `--user` (§3.4) but not `--userns keep-id`, which - rootless podman requires for bind-mount ownership to map back to the invoking user. +Inside the container everything executes as one user in one mount namespace, so a forwarded credential is readable by +anything the checks execute, including dependency build scripts and procedural macros. Keep the forwarded set narrow +and the tokens short-lived. -## 9. Customization +## 8. Customization -A **repository** changes what its own image contains. A **downstream catalog** — an anvil fork, see -[extensibility.md](./extensibility.md) — changes what every repository it manages receives. Containerized execution is +A **repository** changes what its own image contains; a **downstream catalog** (an anvil fork, see +[extensibility.md](./extensibility.md)) changes what every repository it manages receives. Containerized execution is an ordinary artifact group and uses the same levers as any other. | Goal | Mechanism | Owner | @@ -431,34 +390,29 @@ an ordinary artifact group and uses the same levers as any other. | Credentials, or a published image | Add `.anvil/container/hooks.ps1`, or ship `artifacts::container::hooks(…)` | either | | No containerized execution at all | `without_artifact` for each of the three artifacts | catalog | -Editing the Dockerfile in a single repository is supported and the drift flow preserves the edit, but anvil continues -to offer its own version against a file it can see has diverged. A change that belongs everywhere is better made in a -catalog. +Editing the Dockerfile in one repository is supported and the drift flow preserves the edit, but anvil keeps offering +its own version against a file it can see has diverged. A change that belongs everywhere is better made in a catalog. -**The Dockerfile and its ignore file must be replaced together.** The ignore file is a deny-all list re-admitting only -`justfiles/` and `rust-toolchain.toml` (§4). A replacement Dockerfile that `COPY`s anything else must also replace -`artifacts::container::dockerignore()`, or the additional paths never reach the build context and the build fails on a -missing file. Recipes need no such care: `justfiles/` is re-admitted as a directory and hashed recursively, so a new -recipe subdirectory is both copied and part of the identity automatically. +**The Dockerfile and its ignore file must be replaced together.** A replacement that `COPY`s anything beyond +`justfiles/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the added +paths never reach the build context and the build fails on a missing file. Recipes need no such care: `justfiles/` is +admitted as a directory and hashed recursively, so a new recipe subdirectory is copied and covered automatically. -`justfiles/anvil/` must contain `.just` recipes and nothing else. `CatalogBuilder::build` enforces this, because a -non-recipe file placed there would be copied into the image without being part of its identity: editing it would -change what the image contains without renaming the tag. Non-recipe assets belong in a tool-owned directory such as -`.anvil/`. +`justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces: a non-recipe +file there would be copied into the image without being part of its identity, so editing it would change the image's +contents without renaming the tag. Non-recipe assets belong in a tool-owned directory such as `.anvil/`. -A fork inherits everything else unchanged: the recipes, the identity scheme, the cache volumes, the mounts, and the -re-entry guard. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. +A fork inherits everything else: the recipes, the identity scheme, the cache volumes, the mounts, and the re-entry +guard. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. -## 10. Limitations +## 9. Limitations -- Linux images only, pinned to `linux/amd64`. On ARM64 hosts the image is emulated and is substantially slower. -- The first build takes several minutes: it installs a toolchain and the entire pinned tool catalog. Subsequent runs - reuse it until an input changes. +- On ARM64 hosts the `linux/amd64` image is emulated and is substantially slower. +- The first build takes several minutes, installing a toolchain and the entire pinned tool catalog. Later runs reuse + it until an input changes. - Any edit under `justfiles/` invalidates the install layer, including files the image's synthetic Justfile never imports. -- anvil never pushes and never promotes an image. It builds one, and it will use one a hook fetched (§7.3); - publishing belongs to whoever owns the registry. -- A repository-owned `rust-toolchain.toml` is required. It is both what the image installs and part of what names it. -- Podman on Windows cannot mount build secrets (§8.2). +- anvil never pushes or promotes an image. It builds one, and will use one a hook fetched (§7.3); publishing belongs + to whoever owns the registry. [design]: ./README.md diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 833f0673..1eeb5df4 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -98,13 +98,13 @@ //! //! Any generated recipe can be executed inside a content-addressed Linux //! image. The image installs the Rust toolchain and Cargo tools this -//! repository pins by running `just anvil-setup` — the same recipe the checks -//! use, reading the same generated pins — so the image and the host agree on +//! repository pins by running `just anvil-setup`, the same recipe the checks +//! use, reading the same generated pins, so the image and the host agree on //! the toolset by construction, with no second tool list to keep in step. //! -//! `just anvil-pr` and every other recipe continue to run natively. A -//! container is entered only through `anvil-container`, which takes any recipe -//! name and its arguments; nothing is routed into one implicitly. +//! Execution is opt-in per invocation: `just anvil-pr` and every other recipe +//! continue to run natively, and a container is entered only through +//! `anvil-container`, which takes any recipe name and its arguments. //! //! ```text //! just anvil-container anvil-clippy # one check @@ -113,23 +113,23 @@ //! just anvil-container # interactive shell //! ``` //! -//! The feature is two generated artifacts and one optional hook: -//! `justfiles/anvil/container.just` drives the engine, -//! `.anvil/container/Dockerfile` (with its `Dockerfile.dockerignore`) defines -//! what the image contains, and `.anvil/container/hooks.ps1` supplies -//! credentials when a repository needs them. There is no configuration file. +//! The feature is three generated artifacts and one optional hook script, with +//! no configuration file: `justfiles/anvil/container.just` drives the engine, +//! `.anvil/container/Dockerfile` and its `Dockerfile.dockerignore` define what +//! the image contains, and `.anvil/container/hooks.ps1` supplies credentials +//! when a repository needs them. //! -//! One container is created per invocation, not per check. The repository is -//! bind-mounted at `/workspace`, so `target/` stays visible from the host, -//! while `CARGO_HOME` and `RUSTUP_HOME` live in named volumes that keep the -//! write-heavy paths off the host boundary. +//! One container is created per invocation, however many checks the requested +//! recipe runs. The repository is bind-mounted at `/workspace`, so `target/` +//! stays visible from the host, while `CARGO_HOME` and `RUSTUP_HOME` live in +//! named volumes that keep the write-heavy paths off the host boundary. //! //! ### Prerequisites //! //! - A container engine callable from the shell that runs `just`: Docker, or //! Podman via `ANVIL_CONTAINER_ENGINE=podman`. On Windows that means Docker //! Desktop, Podman, a Windows `docker` CLI pointed at an engine in WSL, or -//! Docker Engine installed only inside the default WSL distribution — no +//! Docker Engine installed only inside the default WSL distribution. No //! Windows CLI is needed in that last case, since anvil reaches the engine //! through `wsl.exe` when it finds none on `PATH` and translates repository //! paths with `wslpath`. @@ -148,9 +148,9 @@ //! A changed tool pin names a tag that cannot already exist, so a build //! follows. There is no staleness check because there is no staleness to //! detect: a locally built image that is present was built from the inputs -//! that name it. An image *fetched* by the resolve hook only claims as much — -//! the digest is over source files and cannot be re-derived from layers — so -//! that claim is only as strong as the registry it came from, which should +//! that name it. An image *fetched* by the resolve hook only claims as much, +//! since the digest is over source files and cannot be re-derived from layers, +//! so that claim is only as strong as the registry it came from, which should //! have immutable tags and restricted push. //! //! `anvil-container-tag` prints the reference without building it, and is the @@ -194,7 +194,7 @@ //! produces. //! //! `Anvil-ResolveImage` is offered the tag when nothing local matches, and -//! returns the reference it made available — a registry reference, used as-is +//! returns the reference it made available: a registry reference, used as-is //! rather than re-tagged locally, so the run stays honest about where the //! image came from. It is verified before use, and every failure falls through //! to a local build: a publisher that has not caught up must not block the From 137140dbfee4e18bfac914e10edddaac4562087f Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 11:47:38 +0200 Subject: [PATCH 17/81] fix(anvil): correct container defects found in review The run mounted named volumes over \ and \. An engine seeds a named volume from the image only when that volume is first created, so the first image's tools were pinned over every later tag: a tool bump renamed the tag, built a new image, and still ran the old binary. Both paths are also world-writable, so the shadowed copy was mutable and outlived the tag that named it. Cache only cargo's content-addressed registry and git downloads. Engine calls in WSL used 'wsl.exe --', which hands the command line to the login shell. A repository path containing \$ was silently truncated -- wslpath still exited 0, so the guard never fired and the wrong directory was bind-mounted -- and a ';' in a recipe argument ran on the WSL host. Use --exec, which bypasses the shell. Also: derive the image name so it is always a valid reference (a checkout in 'ox-tools (copy)' produced a trailing separator that the engine rejects); report a failed volume removal instead of always exiting 0; give a mapped uid a writable HOME; compute the tag before the presence query so status cannot report a fatal input error as 'not present'; correct the ANVIL_IN_CONTAINER comment and the documented engine override, neither of which matched the code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 6 +- .anvil/container/Dockerfile | 4 +- crates/cargo-anvil/docs/design/containers.md | 53 ++++++++--- .../src/anvil/artifacts/container.rs | 60 +++++++++++- .../templates/container/Dockerfile | 4 +- .../templates/justfiles/anvil/container.just | 89 +++++++++++++----- .../snapshots/snapshots__ado_backend.snap | 93 +++++++++++++------ .../snapshots/snapshots__github_backend.snap | 93 +++++++++++++------ .../snapshots/snapshots__local_only.snap | 93 +++++++++++++------ justfiles/anvil/container.just | 89 +++++++++++++----- scripts/test-anvil-container.ps1 | 37 +++++++- 11 files changed, 466 insertions(+), 155 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 2f9b6645..f3d4ba1a 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:15e379de0548eca5c544c103306e27ae1ce9d5940523a8ccabb1a331e1317ff1" +catalog_checksum = "sha256:58f8849d76de2f4851d9b08a3046b79df91d88c46b83ad09799186b0d481cab5" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:8e0bf54009f4ea3ad649b93fc094a00ccae665d1e25660e1abd9bdb35630ede4" +checksum = "sha256:de3ea218526cc17598890a76e00948d27930f49afbc2e69b2897a73f6a0453d9" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:45e61b9681b05537a242b4f3d8305cb4e98329bb3b0506a6c4d267b347c0e787" +checksum = "sha256:57312b31d5acba4caa293a15c510c6049eaef0e117ead485de4cbd9a40e59a60" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 231868a6..acf9f288 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -110,8 +110,8 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# Consumed by the re-entry guard in the generated tier and group recipes: a -# recipe that sees this runs natively instead of launching another container. +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index d60bf1af..b10d61f1 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -166,17 +166,23 @@ removed on exit (`--rm`). | Mount | Target | Purpose | | --- | --- | --- | | repository root (bind) | `/workspace` | The worktree under test, including `target/`. | -| `anvil--cargo` (volume) | `/usr/local/cargo` | `CARGO_HOME`: registry cache and installed binaries. | -| `anvil--rustup` (volume) | `/usr/local/rustup` | `RUSTUP_HOME`: installed toolchains. | +| `anvil--cargo-registry` (volume) | `/usr/local/cargo/registry` | Downloaded crate sources. | +| `anvil--cargo-git` (volume) | `/usr/local/cargo/git` | Git checkouts of git dependencies. | -The cargo and rustup homes are named volumes, so write-heavy paths never cross the host boundary and the host's own -toolchain is untouched. `target/` stays on the bind mount, remaining visible from the host and shared between native -and containerized runs. The caller's working directory is mapped to its in-container equivalent, so relative paths -resolve when `anvil-container` is invoked from a subdirectory. +Only cargo's content-addressed download caches are volumes, so the write-heavy download path never crosses the host +boundary and the host's own toolchain is untouched. `$CARGO_HOME` and `$RUSTUP_HOME` themselves are **not** mounted: +they carry the installed tools and toolchains, and an engine populates a named volume from the image only when that +volume is first created. Mounting them would pin the first image's binaries over every later one, so a tool bump +would change the tag, build a new image, and still run the old tools — defeating the identity guarantee in §4. +Tools and toolchains therefore always come from the image layer the tag names. + +`target/` stays on the bind mount, remaining visible from the host and shared between native and containerized runs. +The caller's working directory is mapped to its in-container equivalent, so relative paths resolve when +`anvil-container` is invoked from a subdirectory. Image and volume names derive from the repository directory name, lowercased with every character outside `[a-z0-9._-]` replaced by `-`. Two checkouts with the same directory name therefore share cache volumes. That is -harmless in normal use, since cargo's caches are content-addressed, but `anvil-container-down` then removes volumes +harmless, since both volumes hold only content-addressed downloads, but `anvil-container-down` then removes volumes the other checkout is also using. ### 5.2 Process identity @@ -206,16 +212,17 @@ anvil installs nothing and manages no virtual machine. Beyond the engine, the ho ### 6.1 Engine resolution `ANVIL_CONTAINER_ENGINE` defaults to `docker`, and any value other than `docker` or `podman` is rejected before the -engine is invoked. Being a host property rather than a repository one, it is read at run time and never committed; a -single invocation can override it with `just anvil_container_engine=podman anvil-container anvil-pr`. +engine is invoked. Being a host property rather than a repository one, it is read at run time and never committed. +It is the only control: the recipes resolve the engine through nested `just` invocations, which a +`just anvil_container_engine=...` override would not reach. Resolution proceeds in a fixed order: 1. If the named binary is on `PATH`, it is invoked directly. -2. Otherwise, on Windows, the binary is probed inside the default WSL distribution (`wsl.exe -- --version`). - If the probe succeeds, every subsequent engine call is prefixed with `wsl.exe --`. This accommodates Docker Engine - installed inside WSL without Docker Desktop, which leaves no Windows CLI behind; Docker Desktop and Podman both - install one and resolve at step 1. +2. Otherwise, on Windows, the binary is probed inside the default WSL distribution + (`wsl.exe --exec --version`). If the probe succeeds, every subsequent engine call is prefixed with + `wsl.exe --exec`. This accommodates Docker Engine installed inside WSL without Docker Desktop, which leaves no + Windows CLI behind; Docker Desktop and Podman both install one and resolve at step 1. 3. Otherwise the invocation fails with a message naming the variable and linking to this document. anvil never falls back to the other engine: if the selected one is unusable, the invocation fails rather than @@ -226,7 +233,10 @@ diagnostic is shown unchanged. When the engine is reached through WSL it does not share the Windows filesystem view, so anvil translates every path it hands over (bind-mount source, build context, `--file`) with `wslpath -a -u`. An untranslated path is not -rejected by the engine; it silently resolves to an empty directory. +rejected by the engine; it silently resolves to an empty directory. The `--exec` form is required rather than +cosmetic: plain `wsl.exe --` hands the command line to the distribution's login shell, which would expand `$NAME` +and split on `;` in repository paths and forwarded recipe arguments alike. A path holding `$` would be truncated by +that expansion and `wslpath -a` would still exit 0, bind-mounting the wrong directory. ### 6.2 Docker @@ -415,4 +425,19 @@ guard. A different base OS with a different toolchain source is one Dockerfile r - anvil never pushes or promotes an image. It builds one, and will use one a hook fetched (§7.3); publishing belongs to whoever owns the registry. +## 10. Verification + +The behaviour above needs a live daemon, so it cannot join `anvil-pr`. `scripts/test-anvil-container.ps1` covers it +end to end against a real engine, driving only the public surface: first build then reuse, a tag that changes with +an input and reverts, an edit surviving regeneration, a build secret that never reaches a layer, empty hook values +failing closed, resolve-then-verify, and the re-entry guard. + +```powershell +./scripts/test-anvil-container.ps1 # docker +./scripts/test-anvil-container.ps1 -Engine podman # podman +``` + +What runs unattended is narrower: the unit tests in `artifacts::container` assert the driver's invariants against +the template text, and the snapshots pin the emitted files. + [design]: ./README.md diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 95fea9a9..34e24a97 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -261,8 +261,9 @@ mod tests { #[test] fn the_image_name_cannot_carry_an_apostrophe() { - // What makes the exemption above safe. - assert!(RECIPE.contains(r#"replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-")"#)); + // What makes the exemption above safe: the character class admits + // only alphanumerics, so no quote can reach a PowerShell literal. + assert!(RECIPE.contains(r#"replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-")"#)); } #[test] @@ -328,8 +329,59 @@ mod tests { #[test] fn empty_hook_values_fail_closed() { - assert!(RECIPE.contains("returned an empty value for secret")); - assert!(RECIPE.contains("returned an empty value for")); + // Both phases, asserted independently: "for" alone is a substring of + // the build-side message, so it would pass with the run-side guard + // deleted. + assert!(RECIPE.contains("Anvil-PreBuild returned an empty value for secret")); + assert!(RECIPE.contains("Anvil-PreRun returned an empty value for")); + } + + #[test] + fn cache_volumes_never_mask_the_images_tools() { + // An engine seeds a named volume from the image only on first + // creation, so mounting a directory that holds installed binaries + // pins the first image's tools over every later tag. + assert!(RECIPE.contains("-cargo-registry:/usr/local/cargo/registry")); + assert!(RECIPE.contains("-cargo-git:/usr/local/cargo/git")); + assert!(!RECIPE.contains("-cargo:/usr/local/cargo'")); + assert!(!RECIPE.contains(":/usr/local/rustup")); + } + + #[test] + fn wsl_calls_bypass_the_login_shell() { + // `wsl.exe -- ` re-parses the command line through the default + // shell: a path holding `$` is silently truncated (and wslpath still + // exits 0), and a `;` in any forwarded argument runs on the host. + // Matched on the invocation form so the comment explaining this may + // still name the broken spelling. + assert!(!RECIPE.contains("& wsl.exe -- ")); + assert!(!RECIPE.contains("wsl.exe|--|")); + assert!(RECIPE.contains("& wsl.exe --exec ")); + assert!(RECIPE.contains("wsl.exe|--exec|")); + } + + #[test] + fn container_name_is_always_a_valid_reference() { + // A repository name may not end in a separator or repeat `.`/`_`, so + // a directory like `ox-tools (copy)` must not reach the engine as + // `anvil-ox-tools--copy-`. + assert!(RECIPE.contains(r#"replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-")"#)); + assert!(RECIPE.contains(r#"trim_end_matches("anvil-" + replace_regex"#)); + } + + #[test] + fn teardown_reports_a_removal_that_failed() { + // $ErrorActionPreference does not cover native commands, and this is + // the only way to clear a cache volume. + assert!(RECIPE.contains("if ($LASTEXITCODE -ne 0) { $failed += $vol }")); + assert!(RECIPE.contains("anvil: could not remove: ")); + } + + #[test] + fn a_mapped_user_gets_a_writable_home() { + // A uid with no passwd entry is given HOME=/, which is not writable. + assert!(RECIPE.contains("$runArgs += @('--user', \"${hostUid}:${hostGid}\")")); + assert!(RECIPE.contains("$runArgs += @('-e', 'HOME=/tmp')")); } #[test] diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile index 231868a6..acf9f288 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -110,8 +110,8 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# Consumed by the re-entry guard in the generated tier and group recipes: a -# recipe that sees this runs natively instead of launching another container. +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 0c6324e0..71424683 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -16,9 +16,9 @@ # # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# The container engine. `docker` (supported) or `podman` (best-effort). -# A host property, never committed: set the variable in your environment, or -# pass `just anvil_container_engine=podman ...` for a single invocation. +# The container engine, `docker` or `podman`. A host property, never +# committed. Set it in your environment: a `just anvil_container_engine=...` +# override would not reach the nested invocations that resolve the engine. anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. @@ -28,7 +28,12 @@ anvil_container_workdir := "/workspace" # Two checkouts with the same directory name share cache volumes; that is # harmless (the caches are content-addressed by cargo) but worth knowing before # `anvil-container-down` removes volumes another checkout is also using. -anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") +# +# Every run of non-alphanumerics collapses to a single `-`, and a trailing one +# is trimmed, because a repository name may not end in a separator or repeat +# `.`/`_`. Without that, a checkout in `ox-tools (copy)` yields a reference the +# engine rejects as malformed, from a directory name nobody would suspect. +anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-"), "-") # Resolve how to invoke the engine, as a pipe-separated command. # @@ -57,10 +62,14 @@ _anvil-container-engine: Write-Output $engine exit 0 } + # --exec, not --: `wsl.exe -- ` hands the rest of the command line to + # the distribution's default shell, which expands $NAME, splits on ;, and + # eats backslashes. Every argument we forward -- the repository path and + # the recipe's own arguments -- would cross that boundary unquoted. if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $engine --version *> $null + & wsl.exe --exec $engine --version *> $null if ($LASTEXITCODE -eq 0) { - Write-Output "wsl.exe|--|$engine" + Write-Output "wsl.exe|--exec|$engine" exit 0 } } @@ -83,13 +92,14 @@ _anvil-container-path host_path: Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } - # Pass the path with forward slashes: arguments cross into WSL through a - # shell that would otherwise consume the backslashes, leaving wslpath to - # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' - $translated = & wsl.exe -- wslpath -a -u $hostPath + # --exec for the reason given above. It matters most here: through a shell, + # a path holding `$` loses it, and `wslpath -a` then makes the *truncated* + # path absolute and exits 0, so the guard below never fires and the wrong + # directory is bind-mounted. + $hostPath = '{{ replace(host_path, "'", "''") }}' + $translated = & wsl.exe --exec wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" + Write-Error "anvil: could not translate '$hostPath' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -390,10 +400,14 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # Cargo and rustup homes live in named volumes: the hot write path never - # crosses the host boundary, and the host's own toolchain is untouched. - $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') - $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Cache only what is content-addressed: the downloaded registry and the git + # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- + # those hold the installed tools and toolchains, and a named volume is + # populated from the image only when it is first created. Mounting them + # would pin the first image's binaries over every later one, so a tool bump + # would change the tag, build a new image, and still run the old tools. + $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') + $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -401,7 +415,13 @@ anvil-container *target: # already maps ownership, and `id` is not there to ask. if (-not $IsWindows -and -not $IsMacOS) { $hostUid = (id -u); $hostGid = (id -g) - if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { + $runArgs += @('--user', "${hostUid}:${hostGid}") + # That uid has no passwd entry, so the engine leaves HOME as `/`. + # Anything falling back to $HOME for a cache then writes to a + # read-only root and fails a long way from the cause. + $runArgs += @('-e', 'HOME=/tmp') + } } $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') @@ -471,12 +491,19 @@ anvil-container-status: # of silently spending several minutes building from a status command. # NO_RESOLVE is the same argument applied to the hook, which would otherwise # pull gigabytes to answer a question about the local machine. + # + # Compute the tag first and let it fail loudly. It is fatal for a reason a + # query cannot paper over -- a declared input is missing -- and reporting + # that as "not present locally" would be a lie: the next run cannot build + # it either. + $image = (just anvil-container-tag) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image: $image" + $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 - $present = $LASTEXITCODE -eq 0 - if ($image) { Write-Output "image: $image" } - if ($present) { + just _anvil-container-image *> $null + if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { Write-Output "status: not present locally (the next run resolves or builds it)" @@ -488,8 +515,6 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. - -# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -501,6 +526,10 @@ anvil-container-rebuild: exit 0 # Remove this repository's cache volumes. The image is left in place. +# +# The last two names are from an earlier build of this driver, which mounted +# the cargo and rustup homes wholesale; removing them here clears a stale +# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: @@ -510,7 +539,19 @@ anvil-container-down: $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + # Report a teardown that did not happen. $ErrorActionPreference does not + # cover native commands, so a non-serving engine would otherwise print a + # connection error per volume and still exit 0 -- and this recipe is the + # only way to clear a cache volume, so a caller that scripts teardown must + # be able to tell that it failed. `-f` already exits 0 for a volume that + # does not exist, so this cannot fire spuriously. + $failed = @() + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol + if ($LASTEXITCODE -ne 0) { $failed += $vol } + } + if ($failed.Count -gt 0) { + Write-Error ("anvil: could not remove: " + ($failed -join ', ')) + exit 1 } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 36b79371..01d5d67a 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -115,8 +115,8 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# Consumed by the re-entry guard in the generated tier and group recipes: a -# recipe that sees this runs natively instead of launching another container. +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace @@ -3494,9 +3494,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# The container engine. `docker` (supported) or `podman` (best-effort). -# A host property, never committed: set the variable in your environment, or -# pass `just anvil_container_engine=podman ...` for a single invocation. +# The container engine, `docker` or `podman`. A host property, never +# committed. Set it in your environment: a `just anvil_container_engine=...` +# override would not reach the nested invocations that resolve the engine. anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. @@ -3506,7 +3506,12 @@ anvil_container_workdir := "/workspace" # Two checkouts with the same directory name share cache volumes; that is # harmless (the caches are content-addressed by cargo) but worth knowing before # `anvil-container-down` removes volumes another checkout is also using. -anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") +# +# Every run of non-alphanumerics collapses to a single `-`, and a trailing one +# is trimmed, because a repository name may not end in a separator or repeat +# `.`/`_`. Without that, a checkout in `ox-tools (copy)` yields a reference the +# engine rejects as malformed, from a directory name nobody would suspect. +anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-"), "-") # Resolve how to invoke the engine, as a pipe-separated command. # @@ -3535,10 +3540,14 @@ _anvil-container-engine: Write-Output $engine exit 0 } + # --exec, not --: `wsl.exe -- ` hands the rest of the command line to + # the distribution's default shell, which expands $NAME, splits on ;, and + # eats backslashes. Every argument we forward -- the repository path and + # the recipe's own arguments -- would cross that boundary unquoted. if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $engine --version *> $null + & wsl.exe --exec $engine --version *> $null if ($LASTEXITCODE -eq 0) { - Write-Output "wsl.exe|--|$engine" + Write-Output "wsl.exe|--exec|$engine" exit 0 } } @@ -3561,13 +3570,14 @@ _anvil-container-path host_path: Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } - # Pass the path with forward slashes: arguments cross into WSL through a - # shell that would otherwise consume the backslashes, leaving wslpath to - # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' - $translated = & wsl.exe -- wslpath -a -u $hostPath + # --exec for the reason given above. It matters most here: through a shell, + # a path holding `$` loses it, and `wslpath -a` then makes the *truncated* + # path absolute and exits 0, so the guard below never fires and the wrong + # directory is bind-mounted. + $hostPath = '{{ replace(host_path, "'", "''") }}' + $translated = & wsl.exe --exec wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" + Write-Error "anvil: could not translate '$hostPath' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -3868,10 +3878,14 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # Cargo and rustup homes live in named volumes: the hot write path never - # crosses the host boundary, and the host's own toolchain is untouched. - $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') - $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Cache only what is content-addressed: the downloaded registry and the git + # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- + # those hold the installed tools and toolchains, and a named volume is + # populated from the image only when it is first created. Mounting them + # would pin the first image's binaries over every later one, so a tool bump + # would change the tag, build a new image, and still run the old tools. + $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') + $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -3879,7 +3893,13 @@ anvil-container *target: # already maps ownership, and `id` is not there to ask. if (-not $IsWindows -and -not $IsMacOS) { $hostUid = (id -u); $hostGid = (id -g) - if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { + $runArgs += @('--user', "${hostUid}:${hostGid}") + # That uid has no passwd entry, so the engine leaves HOME as `/`. + # Anything falling back to $HOME for a cache then writes to a + # read-only root and fails a long way from the cause. + $runArgs += @('-e', 'HOME=/tmp') + } } $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') @@ -3949,12 +3969,19 @@ anvil-container-status: # of silently spending several minutes building from a status command. # NO_RESOLVE is the same argument applied to the hook, which would otherwise # pull gigabytes to answer a question about the local machine. + # + # Compute the tag first and let it fail loudly. It is fatal for a reason a + # query cannot paper over -- a declared input is missing -- and reporting + # that as "not present locally" would be a lie: the next run cannot build + # it either. + $image = (just anvil-container-tag) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image: $image" + $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 - $present = $LASTEXITCODE -eq 0 - if ($image) { Write-Output "image: $image" } - if ($present) { + just _anvil-container-image *> $null + if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { Write-Output "status: not present locally (the next run resolves or builds it)" @@ -3966,8 +3993,6 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. - -# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -3979,6 +4004,10 @@ anvil-container-rebuild: exit 0 # Remove this repository's cache volumes. The image is left in place. +# +# The last two names are from an earlier build of this driver, which mounted +# the cargo and rustup homes wholesale; removing them here clears a stale +# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: @@ -3988,8 +4017,20 @@ anvil-container-down: $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + # Report a teardown that did not happen. $ErrorActionPreference does not + # cover native commands, so a non-serving engine would otherwise print a + # connection error per volume and still exit 0 -- and this recipe is the + # only way to clear a cache volume, so a caller that scripts teardown must + # be able to tell that it failed. `-f` already exits 0 for a volume that + # does not exist, so this cannot fire spuriously. + $failed = @() + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol + if ($LASTEXITCODE -ne 0) { $failed += $vol } + } + if ($failed.Count -gt 0) { + Write-Error ("anvil: could not remove: " + ($failed -join ', ')) + exit 1 } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index e03178d8..cc6923d6 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -115,8 +115,8 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# Consumed by the re-entry guard in the generated tier and group recipes: a -# recipe that sees this runs natively instead of launching another container. +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace @@ -3415,9 +3415,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# The container engine. `docker` (supported) or `podman` (best-effort). -# A host property, never committed: set the variable in your environment, or -# pass `just anvil_container_engine=podman ...` for a single invocation. +# The container engine, `docker` or `podman`. A host property, never +# committed. Set it in your environment: a `just anvil_container_engine=...` +# override would not reach the nested invocations that resolve the engine. anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. @@ -3427,7 +3427,12 @@ anvil_container_workdir := "/workspace" # Two checkouts with the same directory name share cache volumes; that is # harmless (the caches are content-addressed by cargo) but worth knowing before # `anvil-container-down` removes volumes another checkout is also using. -anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") +# +# Every run of non-alphanumerics collapses to a single `-`, and a trailing one +# is trimmed, because a repository name may not end in a separator or repeat +# `.`/`_`. Without that, a checkout in `ox-tools (copy)` yields a reference the +# engine rejects as malformed, from a directory name nobody would suspect. +anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-"), "-") # Resolve how to invoke the engine, as a pipe-separated command. # @@ -3456,10 +3461,14 @@ _anvil-container-engine: Write-Output $engine exit 0 } + # --exec, not --: `wsl.exe -- ` hands the rest of the command line to + # the distribution's default shell, which expands $NAME, splits on ;, and + # eats backslashes. Every argument we forward -- the repository path and + # the recipe's own arguments -- would cross that boundary unquoted. if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $engine --version *> $null + & wsl.exe --exec $engine --version *> $null if ($LASTEXITCODE -eq 0) { - Write-Output "wsl.exe|--|$engine" + Write-Output "wsl.exe|--exec|$engine" exit 0 } } @@ -3482,13 +3491,14 @@ _anvil-container-path host_path: Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } - # Pass the path with forward slashes: arguments cross into WSL through a - # shell that would otherwise consume the backslashes, leaving wslpath to - # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' - $translated = & wsl.exe -- wslpath -a -u $hostPath + # --exec for the reason given above. It matters most here: through a shell, + # a path holding `$` loses it, and `wslpath -a` then makes the *truncated* + # path absolute and exits 0, so the guard below never fires and the wrong + # directory is bind-mounted. + $hostPath = '{{ replace(host_path, "'", "''") }}' + $translated = & wsl.exe --exec wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" + Write-Error "anvil: could not translate '$hostPath' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -3789,10 +3799,14 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # Cargo and rustup homes live in named volumes: the hot write path never - # crosses the host boundary, and the host's own toolchain is untouched. - $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') - $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Cache only what is content-addressed: the downloaded registry and the git + # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- + # those hold the installed tools and toolchains, and a named volume is + # populated from the image only when it is first created. Mounting them + # would pin the first image's binaries over every later one, so a tool bump + # would change the tag, build a new image, and still run the old tools. + $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') + $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -3800,7 +3814,13 @@ anvil-container *target: # already maps ownership, and `id` is not there to ask. if (-not $IsWindows -and -not $IsMacOS) { $hostUid = (id -u); $hostGid = (id -g) - if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { + $runArgs += @('--user', "${hostUid}:${hostGid}") + # That uid has no passwd entry, so the engine leaves HOME as `/`. + # Anything falling back to $HOME for a cache then writes to a + # read-only root and fails a long way from the cause. + $runArgs += @('-e', 'HOME=/tmp') + } } $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') @@ -3870,12 +3890,19 @@ anvil-container-status: # of silently spending several minutes building from a status command. # NO_RESOLVE is the same argument applied to the hook, which would otherwise # pull gigabytes to answer a question about the local machine. + # + # Compute the tag first and let it fail loudly. It is fatal for a reason a + # query cannot paper over -- a declared input is missing -- and reporting + # that as "not present locally" would be a lie: the next run cannot build + # it either. + $image = (just anvil-container-tag) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image: $image" + $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 - $present = $LASTEXITCODE -eq 0 - if ($image) { Write-Output "image: $image" } - if ($present) { + just _anvil-container-image *> $null + if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { Write-Output "status: not present locally (the next run resolves or builds it)" @@ -3887,8 +3914,6 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. - -# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -3900,6 +3925,10 @@ anvil-container-rebuild: exit 0 # Remove this repository's cache volumes. The image is left in place. +# +# The last two names are from an earlier build of this driver, which mounted +# the cargo and rustup homes wholesale; removing them here clears a stale +# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: @@ -3909,8 +3938,20 @@ anvil-container-down: $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + # Report a teardown that did not happen. $ErrorActionPreference does not + # cover native commands, so a non-serving engine would otherwise print a + # connection error per volume and still exit 0 -- and this recipe is the + # only way to clear a cache volume, so a caller that scripts teardown must + # be able to tell that it failed. `-f` already exits 0 for a volume that + # does not exist, so this cannot fire spuriously. + $failed = @() + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol + if ($LASTEXITCODE -ne 0) { $failed += $vol } + } + if ($failed.Count -gt 0) { + Write-Error ("anvil: could not remove: " + ($failed -join ', ')) + exit 1 } exit 0 diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index bf66c4e3..376ebca1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -115,8 +115,8 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# Consumed by the re-entry guard in the generated tier and group recipes: a -# recipe that sees this runs natively instead of launching another container. +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace @@ -2234,9 +2234,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# The container engine. `docker` (supported) or `podman` (best-effort). -# A host property, never committed: set the variable in your environment, or -# pass `just anvil_container_engine=podman ...` for a single invocation. +# The container engine, `docker` or `podman`. A host property, never +# committed. Set it in your environment: a `just anvil_container_engine=...` +# override would not reach the nested invocations that resolve the engine. anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. @@ -2246,7 +2246,12 @@ anvil_container_workdir := "/workspace" # Two checkouts with the same directory name share cache volumes; that is # harmless (the caches are content-addressed by cargo) but worth knowing before # `anvil-container-down` removes volumes another checkout is also using. -anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") +# +# Every run of non-alphanumerics collapses to a single `-`, and a trailing one +# is trimmed, because a repository name may not end in a separator or repeat +# `.`/`_`. Without that, a checkout in `ox-tools (copy)` yields a reference the +# engine rejects as malformed, from a directory name nobody would suspect. +anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-"), "-") # Resolve how to invoke the engine, as a pipe-separated command. # @@ -2275,10 +2280,14 @@ _anvil-container-engine: Write-Output $engine exit 0 } + # --exec, not --: `wsl.exe -- ` hands the rest of the command line to + # the distribution's default shell, which expands $NAME, splits on ;, and + # eats backslashes. Every argument we forward -- the repository path and + # the recipe's own arguments -- would cross that boundary unquoted. if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $engine --version *> $null + & wsl.exe --exec $engine --version *> $null if ($LASTEXITCODE -eq 0) { - Write-Output "wsl.exe|--|$engine" + Write-Output "wsl.exe|--exec|$engine" exit 0 } } @@ -2301,13 +2310,14 @@ _anvil-container-path host_path: Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } - # Pass the path with forward slashes: arguments cross into WSL through a - # shell that would otherwise consume the backslashes, leaving wslpath to - # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' - $translated = & wsl.exe -- wslpath -a -u $hostPath + # --exec for the reason given above. It matters most here: through a shell, + # a path holding `$` loses it, and `wslpath -a` then makes the *truncated* + # path absolute and exits 0, so the guard below never fires and the wrong + # directory is bind-mounted. + $hostPath = '{{ replace(host_path, "'", "''") }}' + $translated = & wsl.exe --exec wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" + Write-Error "anvil: could not translate '$hostPath' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -2608,10 +2618,14 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # Cargo and rustup homes live in named volumes: the hot write path never - # crosses the host boundary, and the host's own toolchain is untouched. - $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') - $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Cache only what is content-addressed: the downloaded registry and the git + # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- + # those hold the installed tools and toolchains, and a named volume is + # populated from the image only when it is first created. Mounting them + # would pin the first image's binaries over every later one, so a tool bump + # would change the tag, build a new image, and still run the old tools. + $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') + $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -2619,7 +2633,13 @@ anvil-container *target: # already maps ownership, and `id` is not there to ask. if (-not $IsWindows -and -not $IsMacOS) { $hostUid = (id -u); $hostGid = (id -g) - if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { + $runArgs += @('--user', "${hostUid}:${hostGid}") + # That uid has no passwd entry, so the engine leaves HOME as `/`. + # Anything falling back to $HOME for a cache then writes to a + # read-only root and fails a long way from the cause. + $runArgs += @('-e', 'HOME=/tmp') + } } $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') @@ -2689,12 +2709,19 @@ anvil-container-status: # of silently spending several minutes building from a status command. # NO_RESOLVE is the same argument applied to the hook, which would otherwise # pull gigabytes to answer a question about the local machine. + # + # Compute the tag first and let it fail loudly. It is fatal for a reason a + # query cannot paper over -- a declared input is missing -- and reporting + # that as "not present locally" would be a lie: the next run cannot build + # it either. + $image = (just anvil-container-tag) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image: $image" + $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 - $present = $LASTEXITCODE -eq 0 - if ($image) { Write-Output "image: $image" } - if ($present) { + just _anvil-container-image *> $null + if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { Write-Output "status: not present locally (the next run resolves or builds it)" @@ -2706,8 +2733,6 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. - -# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -2719,6 +2744,10 @@ anvil-container-rebuild: exit 0 # Remove this repository's cache volumes. The image is left in place. +# +# The last two names are from an earlier build of this driver, which mounted +# the cargo and rustup homes wholesale; removing them here clears a stale +# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: @@ -2728,8 +2757,20 @@ anvil-container-down: $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + # Report a teardown that did not happen. $ErrorActionPreference does not + # cover native commands, so a non-serving engine would otherwise print a + # connection error per volume and still exit 0 -- and this recipe is the + # only way to clear a cache volume, so a caller that scripts teardown must + # be able to tell that it failed. `-f` already exits 0 for a volume that + # does not exist, so this cannot fire spuriously. + $failed = @() + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol + if ($LASTEXITCODE -ne 0) { $failed += $vol } + } + if ($failed.Count -gt 0) { + Write-Error ("anvil: could not remove: " + ($failed -join ', ')) + exit 1 } exit 0 diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 0c6324e0..71424683 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -16,9 +16,9 @@ # # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/containers.md -# The container engine. `docker` (supported) or `podman` (best-effort). -# A host property, never committed: set the variable in your environment, or -# pass `just anvil_container_engine=podman ...` for a single invocation. +# The container engine, `docker` or `podman`. A host property, never +# committed. Set it in your environment: a `just anvil_container_engine=...` +# override would not reach the nested invocations that resolve the engine. anvil_container_engine := env_var_or_default("ANVIL_CONTAINER_ENGINE", "docker") # Where the repository is mounted inside the container. @@ -28,7 +28,12 @@ anvil_container_workdir := "/workspace" # Two checkouts with the same directory name share cache volumes; that is # harmless (the caches are content-addressed by cargo) but worth knowing before # `anvil-container-down` removes volumes another checkout is also using. -anvil_container_name := "anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9._-]', "-") +# +# Every run of non-alphanumerics collapses to a single `-`, and a trailing one +# is trimmed, because a repository name may not end in a separator or repeat +# `.`/`_`. Without that, a checkout in `ox-tools (copy)` yields a reference the +# engine rejects as malformed, from a directory name nobody would suspect. +anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file_name(justfile_directory())), '[^a-z0-9]+', "-"), "-") # Resolve how to invoke the engine, as a pipe-separated command. # @@ -57,10 +62,14 @@ _anvil-container-engine: Write-Output $engine exit 0 } + # --exec, not --: `wsl.exe -- ` hands the rest of the command line to + # the distribution's default shell, which expands $NAME, splits on ;, and + # eats backslashes. Every argument we forward -- the repository path and + # the recipe's own arguments -- would cross that boundary unquoted. if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $engine --version *> $null + & wsl.exe --exec $engine --version *> $null if ($LASTEXITCODE -eq 0) { - Write-Output "wsl.exe|--|$engine" + Write-Output "wsl.exe|--exec|$engine" exit 0 } } @@ -83,13 +92,14 @@ _anvil-container-path host_path: Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 } - # Pass the path with forward slashes: arguments cross into WSL through a - # shell that would otherwise consume the backslashes, leaving wslpath to - # translate a mangled path. wslpath accepts either separator. - $hostPath = '{{ replace(host_path, "'", "''") }}' -replace '\\', '/' - $translated = & wsl.exe -- wslpath -a -u $hostPath + # --exec for the reason given above. It matters most here: through a shell, + # a path holding `$` loses it, and `wslpath -a` then makes the *truncated* + # path absolute and exits 0, so the guard below never fires and the wrong + # directory is bind-mounted. + $hostPath = '{{ replace(host_path, "'", "''") }}' + $translated = & wsl.exe --exec wslpath -a -u $hostPath if ($LASTEXITCODE -ne 0) { - Write-Error "anvil: could not translate '{{ replace(host_path, "'", "''") }}' for the engine running in WSL" + Write-Error "anvil: could not translate '$hostPath' for the engine running in WSL" exit 1 } Write-Output $translated.Trim() @@ -390,10 +400,14 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # Cargo and rustup homes live in named volumes: the hot write path never - # crosses the host boundary, and the host's own toolchain is untouched. - $runArgs += @('-v', '{{anvil_container_name}}-cargo:/usr/local/cargo') - $runArgs += @('-v', '{{anvil_container_name}}-rustup:/usr/local/rustup') + # Cache only what is content-addressed: the downloaded registry and the git + # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- + # those hold the installed tools and toolchains, and a named volume is + # populated from the image only when it is first created. Mounting them + # would pin the first image's binaries over every later one, so a tool bump + # would change the tag, build a new image, and still run the old tools. + $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') + $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -401,7 +415,13 @@ anvil-container *target: # already maps ownership, and `id` is not there to ask. if (-not $IsWindows -and -not $IsMacOS) { $hostUid = (id -u); $hostGid = (id -g) - if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { $runArgs += @('--user', "${hostUid}:${hostGid}") } + if ($LASTEXITCODE -eq 0 -and $hostUid -ne '0') { + $runArgs += @('--user', "${hostUid}:${hostGid}") + # That uid has no passwd entry, so the engine leaves HOME as `/`. + # Anything falling back to $HOME for a cache then writes to a + # read-only root and fails a long way from the cause. + $runArgs += @('-e', 'HOME=/tmp') + } } $runArgs += @('-e', 'ANVIL_IN_CONTAINER=1') @@ -471,12 +491,19 @@ anvil-container-status: # of silently spending several minutes building from a status command. # NO_RESOLVE is the same argument applied to the hook, which would otherwise # pull gigabytes to answer a question about the local machine. + # + # Compute the tag first and let it fail loudly. It is fatal for a reason a + # query cannot paper over -- a declared input is missing -- and reporting + # that as "not present locally" would be a lie: the next run cannot build + # it either. + $image = (just anvil-container-tag) | Select-Object -Last 1 + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + Write-Output "image: $image" + $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - $image = (just _anvil-container-image 2>$null) | Select-Object -Last 1 - $present = $LASTEXITCODE -eq 0 - if ($image) { Write-Output "image: $image" } - if ($present) { + just _anvil-container-image *> $null + if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { Write-Output "status: not present locally (the next run resolves or builds it)" @@ -488,8 +515,6 @@ anvil-container-status: # The ordinary path already rebuilds whenever an input changes, so this is for # the cases a content hash cannot see: a moved upstream package, a stale base # layer, or a build that is suspected of being wrong. - -# Rebuild the exec image from scratch, ignoring every cached layer. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-rebuild: @@ -501,6 +526,10 @@ anvil-container-rebuild: exit 0 # Remove this repository's cache volumes. The image is left in place. +# +# The last two names are from an earlier build of this driver, which mounted +# the cargo and rustup homes wholesale; removing them here clears a stale +# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: @@ -510,7 +539,19 @@ anvil-container-down: $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - foreach ($vol in @('{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + # Report a teardown that did not happen. $ErrorActionPreference does not + # cover native commands, so a non-serving engine would otherwise print a + # connection error per volume and still exit 0 -- and this recipe is the + # only way to clear a cache volume, so a caller that scripts teardown must + # be able to tell that it failed. `-f` already exits 0 for a volume that + # does not exist, so this cannot fire spuriously. + $failed = @() + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol + if ($LASTEXITCODE -ne 0) { $failed += $vol } + } + if ($failed.Count -gt 0) { + Write-Error ("anvil: could not remove: " + ($failed -join ', ')) + exit 1 } exit 0 diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index be50c95a..37d73897 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -20,7 +20,8 @@ 1. A generated repository carries exactly the three container artifacts. 2. The first run builds an image and runs the recipe inside it. - 3. A second run reuses the image (the tag resolves, nothing is built). + 3. A second run reuses the image (the tag resolves, nothing is built), + and no cache volume masks the tools the image installed. 4. Changing a hashed input (the pinned toolchain) selects a new tag. 5. Reverting that input returns to the original tag. 6. Editing the Dockerfile is preserved by a re-run of the generator. @@ -152,9 +153,9 @@ function Resolve-Engine { return [pscustomobject]@{ Exe = $Engine; Prefix = @(); ViaWsl = $false } } if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { - & wsl.exe -- $Engine --version *> $null + & wsl.exe --exec $Engine --version *> $null if ($LASTEXITCODE -eq 0) { - return [pscustomobject]@{ Exe = 'wsl.exe'; Prefix = @('--', $Engine); ViaWsl = $true } + return [pscustomobject]@{ Exe = 'wsl.exe'; Prefix = @('--exec', $Engine); ViaWsl = $true } } } $null @@ -167,7 +168,9 @@ function Invoke-Engine { function ConvertTo-EnginePath([string]$Path) { if (-not $script:EngineViaWsl) { return $Path } - (& wsl.exe -- wslpath -a -u ($Path -replace '\\', '/')).Trim() + # --exec, not --: plain `wsl.exe --` re-parses through the login shell, + # which eats `$` in a path and still exits 0. + (& wsl.exe --exec wslpath -a -u $Path).Trim() } function Write-Fixture([string]$Path, [string]$Content) { @@ -379,6 +382,32 @@ $status = Invoke-Just -Repo $repo -Arguments @('anvil-container-status') Assert-That 'status reports present and current' ($status.StdOut -match 'present and current') $status.StdOut Assert-That 'status reports the selected engine' ($status.StdOut -match "engine:\s+.*$Engine") $status.StdOut +# The image's tools must not be masked by a cache volume. An engine seeds a +# named volume from the image only when the volume is first created, so a +# volume over $CARGO_HOME or $RUSTUP_HOME would pin the first image's binaries +# over every later tag -- a bumped tool would change the tag, build a new +# image, and still run the old binary. +$volumes = (Invoke-Engine -Arguments @('volume', 'ls', '--format', '{{.Name}}')).StdOut +$fixtureVolumes = @($volumes -split "`r?`n" | Where-Object { $_ -like "$imagePrefix*" }) +Assert-That 'a registry cache volume exists' ` + (@($fixtureVolumes | Where-Object { $_ -like '*-cargo-registry' }).Count -eq 1) ($fixtureVolumes -join ', ') +Assert-That 'no volume masks CARGO_HOME or RUSTUP_HOME' ` + (@($fixtureVolumes | Where-Object { $_ -like '*-cargo' -or $_ -like '*-rustup' }).Count -eq 0) ($fixtureVolumes -join ', ') + +# The positive half: a binary the image installed is still visible at run time +# with the caches mounted, so the tools a run uses are the ones the tag names. +# Argument vector deliberately free of spaces -- Start-Process joins +# -ArgumentList without quoting, so `bash -c '...'` would be re-split. +$probe = Invoke-Engine -AllowFailure -Arguments @( + 'run', '--rm', '--platform', 'linux/amd64', + '-v', "$imagePrefix-cargo-registry:/usr/local/cargo/registry", + '-v', "$imagePrefix-cargo-git:/usr/local/cargo/git", + $reference, 'ls', '/usr/local/cargo/bin/cargo-binstall' +) +Assert-Equal 'a tool installed by the image survives the cache mounts' 0 $probe.ExitCode +Assert-That 'the tool resolves inside the image, not a volume' ` + ($probe.StdOut -match '/usr/local/cargo/bin/cargo-binstall') "$($probe.StdOut)$($probe.StdErr)" + # ------------------------------------------------------ 4/5. hashed inputs --- Write-Section '4. A changed input selects a new tag' From aee8fd10b794f37503f06479a3cb88d3a2767d96 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 13:58:06 +0200 Subject: [PATCH 18/81] feat(anvil): forward a host GitHub token into the container anvil-aprz runs in the pr-fast group and queries the GitHub advisory API, which allows 60 requests an hour unauthenticated -- fewer than a full tier needs. Inside the image it had no route to a token at all, so the flagship 'just anvil-container anvil-pr' degraded to warnings and rate limits instead of failing with instructions. Forward GITHUB_TOKEN by name when the host already has it set, so a containerized tier authenticates exactly as the same recipe does natively and CI keeps working when it runs a tier this way. It is forwarded, never minted: running 'gh auth token' in the driver would hand a broadly-scoped credential to every recipe in the container, including the ones that never see it on the host, where anvil-aprz scopes it to itself. Forwarding by name only works if the engine can see the name, so the WSLENV bridge now covers every forwarded variable rather than only the hook's -- without that, '-e NAME' reaches an engine that cannot see NAME and silently forwards nothing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 6 ++-- crates/cargo-anvil/README.md | 3 +- crates/cargo-anvil/docs/design/containers.md | 23 ++++++++++--- .../src/anvil/artifacts/container.rs | 17 ++++++++++ crates/cargo-anvil/src/lib.rs | 1 + .../justfiles/anvil/checks/aprz.just | 3 +- .../templates/justfiles/anvil/container.just | 29 +++++++++++++++-- .../snapshots/snapshots__ado_backend.snap | 32 ++++++++++++++++--- .../snapshots/snapshots__github_backend.snap | 32 ++++++++++++++++--- .../snapshots/snapshots__local_only.snap | 32 ++++++++++++++++--- justfiles/anvil/checks/aprz.just | 3 +- justfiles/anvil/container.just | 29 +++++++++++++++-- scripts/test-anvil-container.ps1 | 21 +++++++++++- 13 files changed, 202 insertions(+), 29 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index f3d4ba1a..752a54d1 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:58f8849d76de2f4851d9b08a3046b79df91d88c46b83ad09799186b0d481cab5" +catalog_checksum = "sha256:57327d2d835ec832cb03394c206003ca510bf3496e36595bdf574adbb5dab28a" [[file]] path = ".anvil/container/Dockerfile" @@ -69,7 +69,7 @@ checksum = "sha256:91408602dc3ee274b593e234841934c749ff03bba0ee7846ab88247c06f20 [[file]] path = "justfiles/anvil/checks/aprz.just" -checksum = "sha256:db01bd6484a3161a1f66dd558f35a14c4043d0000092ccb51fa7ac7831fbeccf" +checksum = "sha256:5ac69b95781215c74791766961da1594fb94d48edb9339ccc2043a57fbac23e8" [[file]] path = "justfiles/anvil/checks/audit.just" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:57312b31d5acba4caa293a15c510c6049eaef0e117ead485de4cbd9a40e59a60" +checksum = "sha256:8e9334ac75669b800eff3f1fb36744805cdf58c11b39e7b54448a57f5b62dfad" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 884bfcd3..7ef87a42 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -167,6 +167,7 @@ exactly the reference a consumer will later look up. |`ANVIL_CONTAINER_NO_RESOLVE=1`|Skip the resolve hook, so a query never pulls.| |`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves, ignoring the hook.| |`ANVIL_IN_CONTAINER=1`|Set inside the image; makes a nested invocation run natively.| +|`GITHUB_TOKEN`|Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited.| Supporting recipes: `anvil-container-tag`, `anvil-container-status` (reports the engine and image without building or pulling), @@ -434,7 +435,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbAO_FZGTrWRcbQnEvdCe9uvIbIY7pQUwg99AbRwGoU1ynfVJhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbf5BkcPnaLaUbBQqw7ffBYjAbRhU4DwFqU-obm5Wjj04zV3FhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index b10d61f1..6bdbf7ab 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -25,7 +25,8 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [5. Execution model](#5-execution-model) - [5.1 Mounts and working directory](#51-mounts-and-working-directory) - [5.2 Process identity](#52-process-identity) - - [5.3 Re-entry](#53-re-entry) + - [5.3 Environment](#53-environment) + - [5.4 Re-entry](#54-re-entry) - [6. Engines and host setup](#6-engines-and-host-setup) - [6.1 Engine resolution](#61-engine-resolution) - [6.2 Docker](#62-docker) @@ -37,6 +38,7 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [7.4 Trust boundary](#74-trust-boundary) - [8. Customization](#8-customization) - [9. Limitations](#9-limitations) +- [10. Verification](#10-verification) ## 1. Purpose @@ -76,7 +78,8 @@ All five are annotated `[group("anvil-container")]` and appear as one cluster in | `ANVIL_CONTAINER_NO_REBUILD=1` | Fail instead of building when the image is absent, separating a cache miss from a build failure. | | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook (§7.3), so a query never pulls. | | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag resolves. Skips the resolve hook too (§7.3). | -| `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§5.3). | +| `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§5.4). | +| `GITHUB_TOKEN` | Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited (§5.3). | `NO_REBUILD` is evaluated independently of `NO_CACHE`, so the two compose: `anvil-container-status` sets `NO_REBUILD` and `NO_RESOLVE` together and answers from local state alone. When `NO_REBUILD` stops a build the reference is still @@ -190,9 +193,21 @@ the other checkout is also using. On a Linux host the run passes `--user :`, matching the invoking user, unless that user is root. Without it, everything written under the bind mount is owned by root on the host, and the next native `cargo build` or `git clean` fails with `EACCES` far from the cause. Docker Desktop on Windows and macOS maps ownership itself, and `id` is not -available to query, so the flag is not passed there. +available to query, so the flag is not passed there. That uid has no `passwd` entry, so `HOME` is set to `/tmp`; +otherwise the engine leaves it as `/`, and anything falling back to `$HOME` writes to a read-only root. -### 5.3 Re-entry +### 5.3 Environment + +The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and, when it is set on the host, forwards `GITHUB_TOKEN` by name. +`anvil-aprz` runs in the `pr-fast` group and queries the GitHub advisory API, which allows 60 requests an hour +unauthenticated — fewer than a full tier needs — so without the token a containerized tier degrades to warnings and +rate limits. It is forwarded, never minted: running `gh auth token` in the driver would hand a broadly-scoped +credential to every recipe in the container, including the ones that never see it natively, where the recipe scopes +it to itself. A host that has not exported it gets the same unauthenticated warning it would get natively. + +Everything else a run needs comes from the hook (§7). + +### 5.4 Re-entry `ANVIL_IN_CONTAINER=1` is set in the image and passed on each run. `anvil-container` checks it first and, inside the image, executes the requested recipe directly instead of launching another container, so a recipe that reaches diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 34e24a97..e9b8df1d 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -384,6 +384,23 @@ mod tests { assert!(RECIPE.contains("$runArgs += @('-e', 'HOME=/tmp')")); } + #[test] + fn a_host_token_is_forwarded_by_name_never_minted() { + // anvil-aprz is in pr-fast, so a containerized tier hits the + // unauthenticated advisory-API limit without it. Forwarding by name + // keeps the value off the command line; minting one here would give + // every recipe in the container a credential it lacks natively. + assert!(RECIPE.contains("$forwardedEnv += 'GITHUB_TOKEN'")); + assert!(RECIPE.contains("$runArgs += @('-e', 'GITHUB_TOKEN')")); + // Forwarding by name only works if the engine can see the name, so a + // WSL engine needs it bridged -- otherwise `-e NAME` forwards nothing. + assert!(RECIPE.contains("$engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0")); + // Invocation forms only, so the comment explaining the choice may name + // the command it rules out. + assert!(!RECIPE.contains("(gh auth token")); + assert!(!RECIPE.contains("& gh ")); + } + #[test] fn hooks_constructor_uses_the_documented_path() { assert_eq!(paths(&[hooks("# body\n")]), [HOOKS_PATH]); diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 1eeb5df4..870b9293 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -166,6 +166,7 @@ //! | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | //! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves, ignoring the hook. | //! | `ANVIL_IN_CONTAINER=1` | Set inside the image; makes a nested invocation run natively. | +//! | `GITHUB_TOKEN` | Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited. | //! //! Supporting recipes: `anvil-container-tag`, `anvil-container-status` //! (reports the engine and image without building or pulling), diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just index bb084078..6f4042f1 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just @@ -13,7 +13,8 @@ # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. +# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN +# when it is set, so this recipe sees it the same way it does natively. # # Unscoped (consults external risk DB). diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 71424683..73b4ee72 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -430,7 +430,27 @@ anvil-container *target: # already inherits, so a credential never appears in the host's process # command line, where endpoint telemetry records and retains it for far # longer than a short-lived token is meant to live. + # $forwardedEnv is every name passed with -e; $hookEnv is the subset this + # process set, and so the subset it must unset again. + $forwardedEnv = @() $hookEnv = @() + + # anvil-aprz queries the GitHub advisory API, which allows 60 requests an + # hour unauthenticated -- less than a full tier needs. Forward the host's + # token by name when it is already set, so a containerized tier + # authenticates exactly as the same recipe does natively, and so CI (which + # sets it) keeps working when it runs a tier this way. + # + # Only forwarded, never minted: running `gh auth token` here would hand a + # broadly-scoped credential to every recipe in the container, including + # ones that never see it on the host, where anvil-aprz scopes it to itself. + # A host that has not exported it gets the same unauthenticated warning the + # recipe prints natively. + if ($env:GITHUB_TOKEN) { + $forwardedEnv += 'GITHUB_TOKEN' + $runArgs += @('-e', 'GITHUB_TOKEN') + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { . $hookPath @@ -445,6 +465,7 @@ anvil-container *target: } Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] $hookEnv += $name + $forwardedEnv += $name $runArgs += @('-e', $name) } # Names only, never values: a hook with a broad idea of what to @@ -462,9 +483,11 @@ anvil-container *target: $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is - # where the engine reads their values from when it runs there. - if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { - $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + # where the engine reads their values from when it runs there. Without + # it, `-e NAME` reaches an engine that cannot see NAME and forwards + # nothing, leaving the variable unset inside the container. + if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 01d5d67a..ffda3047 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -1637,7 +1637,8 @@ unknown-git = "deny" # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. +# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN +# when it is set, so this recipe sees it the same way it does natively. # # Unscoped (consults external risk DB). @@ -3908,7 +3909,27 @@ anvil-container *target: # already inherits, so a credential never appears in the host's process # command line, where endpoint telemetry records and retains it for far # longer than a short-lived token is meant to live. + # $forwardedEnv is every name passed with -e; $hookEnv is the subset this + # process set, and so the subset it must unset again. + $forwardedEnv = @() $hookEnv = @() + + # anvil-aprz queries the GitHub advisory API, which allows 60 requests an + # hour unauthenticated -- less than a full tier needs. Forward the host's + # token by name when it is already set, so a containerized tier + # authenticates exactly as the same recipe does natively, and so CI (which + # sets it) keeps working when it runs a tier this way. + # + # Only forwarded, never minted: running `gh auth token` here would hand a + # broadly-scoped credential to every recipe in the container, including + # ones that never see it on the host, where anvil-aprz scopes it to itself. + # A host that has not exported it gets the same unauthenticated warning the + # recipe prints natively. + if ($env:GITHUB_TOKEN) { + $forwardedEnv += 'GITHUB_TOKEN' + $runArgs += @('-e', 'GITHUB_TOKEN') + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { . $hookPath @@ -3923,6 +3944,7 @@ anvil-container *target: } Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] $hookEnv += $name + $forwardedEnv += $name $runArgs += @('-e', $name) } # Names only, never values: a hook with a broad idea of what to @@ -3940,9 +3962,11 @@ anvil-container *target: $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is - # where the engine reads their values from when it runs there. - if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { - $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + # where the engine reads their values from when it runs there. Without + # it, `-e NAME` reaches an engine that cannot see NAME and forwards + # nothing, leaving the variable unset inside the container. + if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index cc6923d6..fdcfcec2 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -1558,7 +1558,8 @@ unknown-git = "deny" # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. +# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN +# when it is set, so this recipe sees it the same way it does natively. # # Unscoped (consults external risk DB). @@ -3829,7 +3830,27 @@ anvil-container *target: # already inherits, so a credential never appears in the host's process # command line, where endpoint telemetry records and retains it for far # longer than a short-lived token is meant to live. + # $forwardedEnv is every name passed with -e; $hookEnv is the subset this + # process set, and so the subset it must unset again. + $forwardedEnv = @() $hookEnv = @() + + # anvil-aprz queries the GitHub advisory API, which allows 60 requests an + # hour unauthenticated -- less than a full tier needs. Forward the host's + # token by name when it is already set, so a containerized tier + # authenticates exactly as the same recipe does natively, and so CI (which + # sets it) keeps working when it runs a tier this way. + # + # Only forwarded, never minted: running `gh auth token` here would hand a + # broadly-scoped credential to every recipe in the container, including + # ones that never see it on the host, where anvil-aprz scopes it to itself. + # A host that has not exported it gets the same unauthenticated warning the + # recipe prints natively. + if ($env:GITHUB_TOKEN) { + $forwardedEnv += 'GITHUB_TOKEN' + $runArgs += @('-e', 'GITHUB_TOKEN') + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { . $hookPath @@ -3844,6 +3865,7 @@ anvil-container *target: } Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] $hookEnv += $name + $forwardedEnv += $name $runArgs += @('-e', $name) } # Names only, never values: a hook with a broad idea of what to @@ -3861,9 +3883,11 @@ anvil-container *target: $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is - # where the engine reads their values from when it runs there. - if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { - $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + # where the engine reads their values from when it runs there. Without + # it, `-e NAME` reaches an engine that cannot see NAME and forwards + # nothing, leaving the variable unset inside the container. + if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 376ebca1..7d7d749c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -377,7 +377,8 @@ unknown-git = "deny" # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. +# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN +# when it is set, so this recipe sees it the same way it does natively. # # Unscoped (consults external risk DB). @@ -2648,7 +2649,27 @@ anvil-container *target: # already inherits, so a credential never appears in the host's process # command line, where endpoint telemetry records and retains it for far # longer than a short-lived token is meant to live. + # $forwardedEnv is every name passed with -e; $hookEnv is the subset this + # process set, and so the subset it must unset again. + $forwardedEnv = @() $hookEnv = @() + + # anvil-aprz queries the GitHub advisory API, which allows 60 requests an + # hour unauthenticated -- less than a full tier needs. Forward the host's + # token by name when it is already set, so a containerized tier + # authenticates exactly as the same recipe does natively, and so CI (which + # sets it) keeps working when it runs a tier this way. + # + # Only forwarded, never minted: running `gh auth token` here would hand a + # broadly-scoped credential to every recipe in the container, including + # ones that never see it on the host, where anvil-aprz scopes it to itself. + # A host that has not exported it gets the same unauthenticated warning the + # recipe prints natively. + if ($env:GITHUB_TOKEN) { + $forwardedEnv += 'GITHUB_TOKEN' + $runArgs += @('-e', 'GITHUB_TOKEN') + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { . $hookPath @@ -2663,6 +2684,7 @@ anvil-container *target: } Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] $hookEnv += $name + $forwardedEnv += $name $runArgs += @('-e', $name) } # Names only, never values: a hook with a broad idea of what to @@ -2680,9 +2702,11 @@ anvil-container *target: $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is - # where the engine reads their values from when it runs there. - if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { - $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + # where the engine reads their values from when it runs there. Without + # it, `-e NAME` reaches an engine that cannot see NAME and forwards + # nothing, leaving the variable unset inside the container. + if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE diff --git a/justfiles/anvil/checks/aprz.just b/justfiles/anvil/checks/aprz.just index bb084078..6f4042f1 100644 --- a/justfiles/anvil/checks/aprz.just +++ b/justfiles/anvil/checks/aprz.just @@ -13,7 +13,8 @@ # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. +# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN +# when it is set, so this recipe sees it the same way it does natively. # # Unscoped (consults external risk DB). diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 71424683..73b4ee72 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -430,7 +430,27 @@ anvil-container *target: # already inherits, so a credential never appears in the host's process # command line, where endpoint telemetry records and retains it for far # longer than a short-lived token is meant to live. + # $forwardedEnv is every name passed with -e; $hookEnv is the subset this + # process set, and so the subset it must unset again. + $forwardedEnv = @() $hookEnv = @() + + # anvil-aprz queries the GitHub advisory API, which allows 60 requests an + # hour unauthenticated -- less than a full tier needs. Forward the host's + # token by name when it is already set, so a containerized tier + # authenticates exactly as the same recipe does natively, and so CI (which + # sets it) keeps working when it runs a tier this way. + # + # Only forwarded, never minted: running `gh auth token` here would hand a + # broadly-scoped credential to every recipe in the container, including + # ones that never see it on the host, where anvil-aprz scopes it to itself. + # A host that has not exported it gets the same unauthenticated warning the + # recipe prints natively. + if ($env:GITHUB_TOKEN) { + $forwardedEnv += 'GITHUB_TOKEN' + $runArgs += @('-e', 'GITHUB_TOKEN') + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { . $hookPath @@ -445,6 +465,7 @@ anvil-container *target: } Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] $hookEnv += $name + $forwardedEnv += $name $runArgs += @('-e', $name) } # Names only, never values: a hook with a broad idea of what to @@ -462,9 +483,11 @@ anvil-container *target: $runArgs += @('--pull=never', '-w', $containerCwd, $image) if (-not $interactive) { $runArgs += @('just') + $targetParts } # WSLENV exports the forwarded names into the WSL environment, which is - # where the engine reads their values from when it runs there. - if ($engineExe -eq 'wsl.exe' -and $hookEnv.Count -gt 0) { - $env:WSLENV = (@($env:WSLENV) + ($hookEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' + # where the engine reads their values from when it runs there. Without + # it, `-e NAME` reaches an engine that cannot see NAME and forwards + # nothing, leaving the variable unset inside the container. + if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { + $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } & $engineExe @enginePrefix @runArgs exit $LASTEXITCODE diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index 37d73897..7befb7e5 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -21,7 +21,8 @@ 1. A generated repository carries exactly the three container artifacts. 2. The first run builds an image and runs the recipe inside it. 3. A second run reuses the image (the tag resolves, nothing is built), - and no cache volume masks the tools the image installed. + no cache volume masks the tools the image installed, and a host + GITHUB_TOKEN is forwarded while an absent one is not invented. 4. Changing a hashed input (the pinned toolchain) selects a new tag. 5. Reverting that input returns to the original tag. 6. Editing the Dockerfile is preserved by a re-run of the generator. @@ -323,6 +324,12 @@ set unstable # A repository-owned recipe, to prove that forwarded values arrive. e2e-show-env: @echo "E2E:$ANVIL_E2E_RUNTIME" + +# Proves the driver forwards a host token, and invents one when it should not. +# `:-` because just runs recipe lines under `sh -u`, where a bare $NAME that +# was correctly *not* forwarded would abort instead of printing empty. +e2e-show-token: + @echo "E2E-TOKEN:[${GITHUB_TOKEN:-}]" '@ Invoke-Native -Command 'git' -Arguments @('init', '-q') -WorkingDirectory $repo | Out-Null @@ -408,6 +415,18 @@ Assert-Equal 'a tool installed by the image survives the cache mounts' 0 $probe. Assert-That 'the tool resolves inside the image, not a volume' ` ($probe.StdOut -match '/usr/local/cargo/bin/cargo-binstall') "$($probe.StdOut)$($probe.StdErr)" +# anvil-aprz runs in pr-fast and is rate-limited without a token, so a host +# token has to reach the container -- but only one the host actually set. +$withToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` + -Environment @{ GITHUB_TOKEN = 'e2e-forwarded-token' } +Assert-That 'a host GITHUB_TOKEN reaches a recipe in the container' ` + ($withToken.StdOut -match 'E2E-TOKEN:\[e2e-forwarded-token\]') "$($withToken.StdOut)$($withToken.StdErr)" + +$withoutToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` + -Environment @{ GITHUB_TOKEN = '' } +Assert-That 'no token is invented when the host has none' ` + ($withoutToken.StdOut -match 'E2E-TOKEN:\[\]') "$($withoutToken.StdOut)$($withoutToken.StdErr)" + # ------------------------------------------------------ 4/5. hashed inputs --- Write-Section '4. A changed input selects a new tag' From d25879dc815c9a132d55db470cd63c668fbd037b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 14:24:49 +0200 Subject: [PATCH 19/81] fix(anvil): match the build context to the hashed inputs The image copied all of justfiles/ but the tag hashed only justfiles/anvil/**/*.just. A file that is copied without being hashed can change what a build produces while naming a tag that already resolves -- and because presence of the tag is checked first, the existing image is reused and the change is never built. Narrow the context to justfiles/anvil/ so the two sets agree by construction. The public catalog had no reachable symptom, since the synthetic Justfile imports only the anvil tree, but a fork that replaces the Dockerfile can copy anything it likes, and the hashed set is fixed and cannot be extended. Record that as a limitation rather than leaving it to be discovered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 6 +++--- .anvil/container/Dockerfile | 10 +++++----- .anvil/container/Dockerfile.dockerignore | 7 +++++++ crates/cargo-anvil/docs/design/containers.md | 19 ++++++++++--------- .../src/anvil/artifacts/container.rs | 9 +++++++++ .../templates/container/Dockerfile | 10 +++++----- .../container/Dockerfile.dockerignore | 7 +++++++ .../snapshots/snapshots__ado_backend.snap | 17 ++++++++++++----- .../snapshots/snapshots__github_backend.snap | 17 ++++++++++++----- .../snapshots/snapshots__local_only.snap | 17 ++++++++++++----- 10 files changed, 82 insertions(+), 37 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 752a54d1..4b732b54 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,15 +1,15 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:57327d2d835ec832cb03394c206003ca510bf3496e36595bdf574adbb5dab28a" +catalog_checksum = "sha256:eab4134b7b86f0b01a48c441a4c48f3ecd6ee70e29a0fd736f127dff7c8e5c31" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:de3ea218526cc17598890a76e00948d27930f49afbc2e69b2897a73f6a0453d9" +checksum = "sha256:74682ca2c41116adfe28e1d37d5caae455c8d40b04b7ec4bf60765865b2d2c1e" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:b04c88f8c52256b99b4590f745945db4cd9b501ddb36851230060d22a1d1c14c" +checksum = "sha256:834566e919c693e179675eb96995fa7248cf5ead0c2e7aec2fd237762965cd7f" [[file]] path = ".github/actions/anvil-impact/action.yml" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index acf9f288..2b77a49b 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -86,11 +86,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit -# elsewhere in the repository does not invalidate this layer -- though an edit -# to any file under `justfiles/` does, including ones the synthetic Justfile -# below never imports. That Justfile avoids pulling in repository-specific -# imports that may not exist yet. +# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is +# also exactly what the tag hashes, so the build context cannot differ from +# the image's identity: an edit that changes what this layer produces always +# renames the tag. The synthetic Justfile below imports only the anvil tree, +# avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index 21f90464..e1a1e94e 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -10,6 +10,13 @@ # The build context is the repository root but the image only needs two things. # Excluding everything else keeps a cold build from streaming the whole # worktree (and every stale `target/`) to the daemon. +# +# The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` +# so that it matches what the tag hashes. A file that is copied but not hashed +# can change what a build produces while naming a tag that already resolves -- +# so the changed file is never built, because the existing image is reused. * !justfiles +justfiles/* +!justfiles/anvil !rust-toolchain.toml diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 6bdbf7ab..8dace441 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -435,24 +435,25 @@ guard. A different base OS with a different toolchain source is one Dockerfile r - On ARM64 hosts the `linux/amd64` image is emulated and is substantially slower. - The first build takes several minutes, installing a toolchain and the entire pinned tool catalog. Later runs reuse it until an input changes. -- Any edit under `justfiles/` invalidates the install layer, including files the image's synthetic Justfile never - imports. +- Any edit under `justfiles/anvil/` invalidates the install layer, including files the image's synthetic Justfile + never imports. +- **The set of hashed inputs is fixed (§4.1) and a fork cannot extend it.** A replacement Dockerfile is itself + hashed, so changing the build recipe always renames the tag — but any *additional* file it copies is outside the + tag. Such a file can change what a build produces while naming a tag that already resolves, and the existing image + is then reused, so the change is never built. A fork that needs extra content should carry it in the Dockerfile + itself, or accept that edits to it require `anvil-container-rebuild`. - anvil never pushes or promotes an image. It builds one, and will use one a hook fetched (§7.3); publishing belongs to whoever owns the registry. ## 10. Verification -The behaviour above needs a live daemon, so it cannot join `anvil-pr`. `scripts/test-anvil-container.ps1` covers it -end to end against a real engine, driving only the public surface: first build then reuse, a tag that changes with -an input and reverts, an edit surviving regeneration, a build secret that never reaches a layer, empty hook values -failing closed, resolve-then-verify, and the re-entry guard. +`scripts/test-anvil-container.ps1` exercises the behaviour above end to end against a real engine, driving only the +public surface. It needs a live daemon, so it cannot join `anvil-pr`; the unit tests in `artifacts::container` and +the emitted-tree snapshots are what run unattended. ```powershell ./scripts/test-anvil-container.ps1 # docker ./scripts/test-anvil-container.ps1 -Engine podman # podman ``` -What runs unattended is narrower: the unit tests in `artifacts::container` assert the driver's invariants against -the template text, and the snapshots pin the emitted files. - [design]: ./README.md diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index e9b8df1d..a1471fc9 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -401,6 +401,15 @@ mod tests { assert!(!RECIPE.contains("& gh ")); } + #[test] + fn the_build_context_matches_the_hashed_inputs() { + // A file that is copied but not hashed can change what a build + // produces while naming a tag that already resolves, so the change is + // never built. + assert!(DOCKERIGNORE.contains("justfiles/*\n!justfiles/anvil\n")); + assert!(!DOCKERIGNORE.contains("!justfiles\n!rust-toolchain.toml")); + } + #[test] fn hooks_constructor_uses_the_documented_path() { assert_eq!(paths(&[hooks("# body\n")]), [HOOKS_PATH]); diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile index acf9f288..2b77a49b 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -86,11 +86,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit -# elsewhere in the repository does not invalidate this layer -- though an edit -# to any file under `justfiles/` does, including ones the synthetic Justfile -# below never imports. That Justfile avoids pulling in repository-specific -# imports that may not exist yet. +# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is +# also exactly what the tag hashes, so the build context cannot differ from +# the image's identity: an edit that changes what this layer produces always +# renames the tag. The synthetic Justfile below imports only the anvil tree, +# avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source diff --git a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore index 21f90464..e1a1e94e 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore @@ -10,6 +10,13 @@ # The build context is the repository root but the image only needs two things. # Excluding everything else keeps a cold build from streaming the whole # worktree (and every stale `target/`) to the daemon. +# +# The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` +# so that it matches what the tag hashes. A file that is copied but not hashed +# can change what a build produces while naming a tag that already resolves -- +# so the changed file is never built, because the existing image is reused. * !justfiles +justfiles/* +!justfiles/anvil !rust-toolchain.toml diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index ffda3047..b5198943 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -91,11 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit -# elsewhere in the repository does not invalidate this layer -- though an edit -# to any file under `justfiles/` does, including ones the synthetic Justfile -# below never imports. That Justfile avoids pulling in repository-specific -# imports that may not exist yet. +# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is +# also exactly what the tag hashes, so the build context cannot differ from +# the image's identity: an edit that changes what this layer produces always +# renames the tag. The synthetic Justfile below imports only the anvil tree, +# avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -135,8 +135,15 @@ CMD ["bash"] # The build context is the repository root but the image only needs two things. # Excluding everything else keeps a cold build from streaming the whole # worktree (and every stale `target/`) to the daemon. +# +# The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` +# so that it matches what the tag hashes. A file that is copied but not hashed +# can change what a build produces while naming a tag that already resolves -- +# so the changed file is never built, because the existing image is reused. * !justfiles +justfiles/* +!justfiles/anvil !rust-toolchain.toml === .delta.toml === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index fdcfcec2..a0c2a5c3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -91,11 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit -# elsewhere in the repository does not invalidate this layer -- though an edit -# to any file under `justfiles/` does, including ones the synthetic Justfile -# below never imports. That Justfile avoids pulling in repository-specific -# imports that may not exist yet. +# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is +# also exactly what the tag hashes, so the build context cannot differ from +# the image's identity: an edit that changes what this layer produces always +# renames the tag. The synthetic Justfile below imports only the anvil tree, +# avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -135,8 +135,15 @@ CMD ["bash"] # The build context is the repository root but the image only needs two things. # Excluding everything else keeps a cold build from streaming the whole # worktree (and every stale `target/`) to the daemon. +# +# The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` +# so that it matches what the tag hashes. A file that is copied but not hashed +# can change what a build produces while naming a tag that already resolves -- +# so the changed file is never built, because the existing image is reused. * !justfiles +justfiles/* +!justfiles/anvil !rust-toolchain.toml === .delta.toml === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 7d7d749c..6e530826 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -91,11 +91,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/` and the toolchain pin are copied, so an edit -# elsewhere in the repository does not invalidate this layer -- though an edit -# to any file under `justfiles/` does, including ones the synthetic Justfile -# below never imports. That Justfile avoids pulling in repository-specific -# imports that may not exist yet. +# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is +# also exactly what the tag hashes, so the build context cannot differ from +# the image's identity: an edit that changes what this layer produces always +# renames the tag. The synthetic Justfile below imports only the anvil tree, +# avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -135,8 +135,15 @@ CMD ["bash"] # The build context is the repository root but the image only needs two things. # Excluding everything else keeps a cold build from streaming the whole # worktree (and every stale `target/`) to the daemon. +# +# The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` +# so that it matches what the tag hashes. A file that is copied but not hashed +# can change what a build produces while naming a tag that already resolves -- +# so the changed file is never built, because the existing image is reused. * !justfiles +justfiles/* +!justfiles/anvil !rust-toolchain.toml === .delta.toml === From fec359404eea88e7a1107808acfefece8d452af6 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 14:49:55 +0200 Subject: [PATCH 20/81] docs(anvil): correct the documented image-name derivation The regex and the trailing-separator trim changed with the fix for invalid references, but section 5.1 still described the old character class. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- crates/cargo-anvil/docs/design/containers.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 8dace441..5a94e760 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -183,10 +183,11 @@ Tools and toolchains therefore always come from the image layer the tag names. The caller's working directory is mapped to its in-container equivalent, so relative paths resolve when `anvil-container` is invoked from a subdirectory. -Image and volume names derive from the repository directory name, lowercased with every character outside -`[a-z0-9._-]` replaced by `-`. Two checkouts with the same directory name therefore share cache volumes. That is -harmless, since both volumes hold only content-addressed downloads, but `anvil-container-down` then removes volumes -the other checkout is also using. +Image and volume names derive from the repository directory name, lowercased with every run of characters outside +`[a-z0-9]` replaced by a single `-` and any trailing `-` removed, so the result is always a valid reference: a +checkout in `ox-tools (copy)` would otherwise end in a separator, which the engine rejects. Two checkouts with the +same directory name therefore share cache volumes. That is harmless, since both volumes hold only content-addressed +downloads, but `anvil-container-down` then removes volumes the other checkout is also using. ### 5.2 Process identity From 41365e4fd5d0291d40703aadbfd91bc6ba662575 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 14 Aug 2026 16:01:10 +0200 Subject: [PATCH 21/81] fix(anvil): let a containerized run reach a linked worktree's git history `just anvil-container anvil-pr` failed outright from a git worktree. A linked worktree stores an absolute host path in `.git` instead of carrying the git directory, and that path does not exist inside the container, so git resolved nothing -- not HEAD, not origin/main. anvil-base-ref then aborted and took anvil-semver-check down with it. Compare `git rev-parse --git-dir` against `--git-common-dir`; when they differ, mount the common directory and set GIT_DIR and GIT_WORK_TREE. An ordinary clone answers the same for both and takes no extra mount, so nothing changes for it, and container.just is outside the identity hash so nobody rebuilds. No flag or variable selects any of this. The suite never caught this because its fixture is a plain `git init` -- while every run of it happened inside a worktree. It now builds a real linked worktree and runs a recipe from it, reusing the same image because the directory name and contents match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .anvil.lock | 4 +- crates/cargo-anvil/docs/design/containers.md | 8 ++++ .../src/anvil/artifacts/container.rs | 14 ++++++ .../templates/justfiles/anvil/container.just | 31 +++++++++++++ .../snapshots/snapshots__ado_backend.snap | 31 +++++++++++++ .../snapshots/snapshots__github_backend.snap | 31 +++++++++++++ .../snapshots/snapshots__local_only.snap | 31 +++++++++++++ justfiles/anvil/container.just | 31 +++++++++++++ scripts/test-anvil-container.ps1 | 45 +++++++++++++++++++ 9 files changed, 224 insertions(+), 2 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 4b732b54..e5c0df32 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:eab4134b7b86f0b01a48c441a4c48f3ecd6ee70e29a0fd736f127dff7c8e5c31" +catalog_checksum = "sha256:39ec0b9730ac86200a1c65f4f4dd42d1a5577b21488f5ae57d7333e28a401652" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:8e9334ac75669b800eff3f1fb36744805cdf58c11b39e7b54448a57f5b62dfad" +checksum = "sha256:af3bb041c22ea9fc5647e7c9a3f7102ad8712a69b760b2e128b5affed81ed538" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 5a94e760..dc2c3f59 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -169,9 +169,17 @@ removed on exit (`--rm`). | Mount | Target | Purpose | | --- | --- | --- | | repository root (bind) | `/workspace` | The worktree under test, including `target/`. | +| common git directory (bind, linked worktrees only) | `/anvil/gitdir` | Git history, when the checkout does not carry it. | | `anvil--cargo-registry` (volume) | `/usr/local/cargo/registry` | Downloaded crate sources. | | `anvil--cargo-git` (volume) | `/usr/local/cargo/git` | Git checkouts of git dependencies. | +A linked worktree (`git worktree add`) keeps its git directory outside the checkout and stores an absolute host path +in `.git`, which does not exist inside the container. Left alone, git resolves nothing — not `HEAD`, not `origin/main` +— and every check that needs history fails somewhere far from the cause. anvil detects this by comparing +`git rev-parse --git-dir` against `--git-common-dir`, mounts the common directory, and sets `GIT_DIR` and +`GIT_WORK_TREE` accordingly. An ordinary clone carries its git directory inside the bind mount and takes none of this. +No flag or variable selects the behaviour. + Only cargo's content-addressed download caches are volumes, so the write-heavy download path never crosses the host boundary and the host's own toolchain is untouched. `$CARGO_HOME` and `$RUSTUP_HOME` themselves are **not** mounted: they carry the installed tools and toolchains, and an engine populates a named volume from the image only when that diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index a1471fc9..0583f5c4 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -410,6 +410,20 @@ mod tests { assert!(!DOCKERIGNORE.contains("!justfiles\n!rust-toolchain.toml")); } + #[test] + fn a_linked_worktree_can_reach_its_git_directory() { + // A worktree's .git is a file naming a host path outside the mount, so + // without this the container resolves no refs at all and every check + // that needs history fails. + assert!(RECIPE.contains("git rev-parse --git-common-dir")); + assert!(RECIPE.contains("${engineGitCommon}:/anvil/gitdir")); + assert!(RECIPE.contains("GIT_DIR=/anvil/gitdir/$rel")); + // An ordinary clone must not take the extra mount. + assert!(RECIPE.contains("if ($gitDirAbs -ne $gitCommonAbs) {")); + // A host with a working engine but no git must not start failing here. + assert!(RECIPE.contains("if (Get-Command git -ErrorAction SilentlyContinue) {")); + } + #[test] fn hooks_constructor_uses_the_documented_path() { assert_eq!(paths(&[hooks("# body\n")]), [HOOKS_PATH]); diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 73b4ee72..0ebbb08e 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -400,6 +400,37 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") + + # A linked worktree keeps its real git directory outside the checkout: its + # `.git` is a file naming an absolute host path, which does not exist inside + # the container. Git then resolves nothing -- not HEAD, not origin/main -- + # and every check that needs history fails a long way from the cause. Mount + # the common git directory and point GIT_DIR at this worktree's entry in it; + # the `commondir` file there is relative, so it resolves under the mount. + # + # An ordinary clone keeps its git directory inside the checkout, where the + # bind mount already carries it, and takes none of this. + # + # Guarded on git being present: the run path needs it only to answer this + # question, and a host with a working engine but no git on PATH should keep + # working rather than fail on a call it did not used to make. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitDir = & git rev-parse --git-dir 2>$null + $gitCommon = & git rev-parse --git-common-dir 2>$null + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path + $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path + if ($gitDirAbs -ne $gitCommonAbs) { + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") + $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + } + } + } + # Cache only what is content-addressed: the downloaded registry and the git # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- # those hold the installed tools and toolchains, and a named volume is diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index b5198943..3c02b1b6 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3886,6 +3886,37 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") + + # A linked worktree keeps its real git directory outside the checkout: its + # `.git` is a file naming an absolute host path, which does not exist inside + # the container. Git then resolves nothing -- not HEAD, not origin/main -- + # and every check that needs history fails a long way from the cause. Mount + # the common git directory and point GIT_DIR at this worktree's entry in it; + # the `commondir` file there is relative, so it resolves under the mount. + # + # An ordinary clone keeps its git directory inside the checkout, where the + # bind mount already carries it, and takes none of this. + # + # Guarded on git being present: the run path needs it only to answer this + # question, and a host with a working engine but no git on PATH should keep + # working rather than fail on a call it did not used to make. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitDir = & git rev-parse --git-dir 2>$null + $gitCommon = & git rev-parse --git-common-dir 2>$null + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path + $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path + if ($gitDirAbs -ne $gitCommonAbs) { + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") + $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + } + } + } + # Cache only what is content-addressed: the downloaded registry and the git # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- # those hold the installed tools and toolchains, and a named volume is diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index a0c2a5c3..c14b0550 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3807,6 +3807,37 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") + + # A linked worktree keeps its real git directory outside the checkout: its + # `.git` is a file naming an absolute host path, which does not exist inside + # the container. Git then resolves nothing -- not HEAD, not origin/main -- + # and every check that needs history fails a long way from the cause. Mount + # the common git directory and point GIT_DIR at this worktree's entry in it; + # the `commondir` file there is relative, so it resolves under the mount. + # + # An ordinary clone keeps its git directory inside the checkout, where the + # bind mount already carries it, and takes none of this. + # + # Guarded on git being present: the run path needs it only to answer this + # question, and a host with a working engine but no git on PATH should keep + # working rather than fail on a call it did not used to make. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitDir = & git rev-parse --git-dir 2>$null + $gitCommon = & git rev-parse --git-common-dir 2>$null + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path + $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path + if ($gitDirAbs -ne $gitCommonAbs) { + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") + $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + } + } + } + # Cache only what is content-addressed: the downloaded registry and the git # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- # those hold the installed tools and toolchains, and a named volume is diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 6e530826..dcb3955d 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2626,6 +2626,37 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") + + # A linked worktree keeps its real git directory outside the checkout: its + # `.git` is a file naming an absolute host path, which does not exist inside + # the container. Git then resolves nothing -- not HEAD, not origin/main -- + # and every check that needs history fails a long way from the cause. Mount + # the common git directory and point GIT_DIR at this worktree's entry in it; + # the `commondir` file there is relative, so it resolves under the mount. + # + # An ordinary clone keeps its git directory inside the checkout, where the + # bind mount already carries it, and takes none of this. + # + # Guarded on git being present: the run path needs it only to answer this + # question, and a host with a working engine but no git on PATH should keep + # working rather than fail on a call it did not used to make. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitDir = & git rev-parse --git-dir 2>$null + $gitCommon = & git rev-parse --git-common-dir 2>$null + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path + $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path + if ($gitDirAbs -ne $gitCommonAbs) { + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") + $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + } + } + } + # Cache only what is content-addressed: the downloaded registry and the git # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- # those hold the installed tools and toolchains, and a named volume is diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 73b4ee72..0ebbb08e 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -400,6 +400,37 @@ anvil-container *target: $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") + + # A linked worktree keeps its real git directory outside the checkout: its + # `.git` is a file naming an absolute host path, which does not exist inside + # the container. Git then resolves nothing -- not HEAD, not origin/main -- + # and every check that needs history fails a long way from the cause. Mount + # the common git directory and point GIT_DIR at this worktree's entry in it; + # the `commondir` file there is relative, so it resolves under the mount. + # + # An ordinary clone keeps its git directory inside the checkout, where the + # bind mount already carries it, and takes none of this. + # + # Guarded on git being present: the run path needs it only to answer this + # question, and a host with a working engine but no git on PATH should keep + # working rather than fail on a call it did not used to make. + if (Get-Command git -ErrorAction SilentlyContinue) { + $gitDir = & git rev-parse --git-dir 2>$null + $gitCommon = & git rev-parse --git-common-dir 2>$null + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path + $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path + if ($gitDirAbs -ne $gitCommonAbs) { + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") + $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + } + } + } + # Cache only what is content-addressed: the downloaded registry and the git # checkouts. Deliberately NOT $CARGO_HOME or $RUSTUP_HOME themselves -- # those hold the installed tools and toolchains, and a named volume is diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index 7befb7e5..bc7fdbd6 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -23,6 +23,7 @@ 3. A second run reuses the image (the tag resolves, nothing is built), no cache volume masks the tools the image installed, and a host GITHUB_TOKEN is forwarded while an absent one is not invented. + 3b. A recipe run from a linked worktree can still reach git history. 4. Changing a hashed input (the pinned toolchain) selects a new tag. 5. Reverting that input returns to the original tag. 6. Editing the Dockerfile is preserved by a re-run of the generator. @@ -330,9 +331,17 @@ e2e-show-env: # was correctly *not* forwarded would abort instead of printing empty. e2e-show-token: @echo "E2E-TOKEN:[${GITHUB_TOKEN:-}]" + +# Proves git resolves inside the container, which a linked worktree breaks +# unless the driver mounts the common git directory. +e2e-show-git: + @echo "E2E-GIT:[$(git rev-parse --abbrev-ref HEAD)]" '@ Invoke-Native -Command 'git' -Arguments @('init', '-q') -WorkingDirectory $repo | Out-Null +# Pin the newline policy: the fixture writes LF, and a developer with +# core.autocrlf=true globally would otherwise fail to stage it. +Invoke-Native -Command 'git' -Arguments @('config', 'core.autocrlf', 'false') -WorkingDirectory $repo | Out-Null Write-Step 'generating the anvil tree (cargo anvil --no-backends)' $generate = Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo @@ -427,6 +436,42 @@ $withoutToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-sho Assert-That 'no token is invented when the host has none' ` ($withoutToken.StdOut -match 'E2E-TOKEN:\[\]') "$($withoutToken.StdOut)$($withoutToken.StdErr)" +# --------------------------------------------------------- 3b. worktrees ----- + +Write-Section '3b. A linked worktree resolves its git directory' + +# A linked worktree's `.git` is a file naming an absolute host path outside the +# checkout. Bind-mounting only the worktree leaves that path unreachable, so git +# inside the container resolves nothing -- not HEAD, not origin/* -- and every +# check that needs history fails. This is not exotic: worktrees are the ordinary +# way to work on two branches at once. +# +# The worktree is given the same directory name as the fixture so the image name +# matches, and it checks out the same committed content, so the tag is identical +# and no rebuild is needed. +Invoke-Native -Command 'git' -Arguments @('add', '-A') -WorkingDirectory $repo | Out-Null +Invoke-Native -Command 'git' -Arguments @('-c', 'user.email=e2e@example.invalid', '-c', 'user.name=e2e', + 'commit', '-q', '-m', 'fixture') -WorkingDirectory $repo | Out-Null + +$worktreeParent = Join-Path $workRoot 'wt' +$worktree = Join-Path $worktreeParent $fixtureName +Invoke-Native -Command 'git' -Arguments @('worktree', 'add', '-q', '-b', 'e2e-worktree', $worktree) ` + -WorkingDirectory $repo -AllowFailure | Out-Null +Assert-That 'the fixture worktree was created' (Test-Path -LiteralPath $worktree) $worktree +Assert-That 'its .git is a file, not a directory' ` + (Test-Path -LiteralPath (Join-Path $worktree '.git') -PathType Leaf) 'a linked worktree stores a gitdir pointer' + +$wtReference = Get-ImageReference -Repo $worktree +Assert-Equal 'the worktree selects the same image, so nothing rebuilds' $reference $wtReference + +$wtGit = Invoke-Just -Repo $worktree -Arguments @('anvil-container', 'e2e-show-git') -AllowFailure +Assert-Equal 'a recipe run from a worktree succeeds' 0 $wtGit.ExitCode +Assert-That 'git resolves the branch inside the container' ` + ($wtGit.StdOut -match 'E2E-GIT:\[e2e-worktree\]') "$($wtGit.StdOut)$($wtGit.StdErr)" + +Invoke-Native -Command 'git' -Arguments @('worktree', 'remove', '--force', $worktree) ` + -WorkingDirectory $repo -AllowFailure | Out-Null + # ------------------------------------------------------ 4/5. hashed inputs --- Write-Section '4. A changed input selects a new tag' From 20cc5075ccdb4ee3d16da9240a84369a5d3ca95b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 12:34:50 +0200 Subject: [PATCH 22/81] fix(anvil): let a containerized tier run to completion Running `just anvil-container anvil-pr` against this repository failed, then hung. Two independent defects, both invisible to the fixture e2e because the fixture repository is too small to install the catalog or reach the advisory API. The exec image was built on Debian bookworm (glibc 2.36) while the generated workflows run on `ubuntu-latest` (24.04, glibc 2.39). The image installs the catalog with `binstall` precisely so it matches CI, but those prebuilt binaries are linked against the runner's glibc, and glibc is backward but not forward compatible -- so the tools installed and could not execute: cargo-aprz: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by cargo-aprz) The base now tracks the runner. Matching it rather than merely exceeding it is what keeps the container predictive: a newer base would let a run pass locally and fail in CI. With the loader fixed, `anvil-aprz` then blocked. Unauthenticated the GitHub advisory API allows 60 requests an hour, and `cargo aprz deps` does not fail on exhaustion -- it sleeps until the quota resets, with no flag to opt out, so the check sat for 42 minutes waiting. The driver only forwarded `GITHUB_TOKEN` when it was already exported, while the recipe natively falls back to the gh CLI, so whether a containerized tier terminated depended on how the developer had signed in -- and the recipe itself recommends `gh auth login`, the path that hung. The driver now resolves the token the same way the recipe does: the environment first, then `gh auth token`. The rationale this replaces -- that minting would hand a broadly-scoped credential to every recipe in the container -- does not distinguish the two paths, since forwarding an exported token has identical exposure inside. A derived value is set on the driver process, passed by name, and unset again, so it never reaches the host's command line. `anvil-aprz` now completes in 2:46, and `anvil-pr-fast` passes inside the image on both docker and podman. --- .anvil.lock | 6 +- .anvil/container/Dockerfile | 21 ++++++- .spelling | 2 + crates/cargo-anvil/docs/design/containers.md | 29 +++++++--- .../src/anvil/artifacts/container.rs | 35 ++++++++---- .../templates/container/Dockerfile | 21 ++++++- .../templates/justfiles/anvil/container.just | 34 +++++++++--- .../snapshots/snapshots__ado_backend.snap | 55 +++++++++++++++---- .../snapshots/snapshots__github_backend.snap | 55 +++++++++++++++---- .../snapshots/snapshots__local_only.snap | 55 +++++++++++++++---- justfiles/anvil/container.just | 34 +++++++++--- 11 files changed, 271 insertions(+), 76 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index e5c0df32..c8438b48 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:39ec0b9730ac86200a1c65f4f4dd42d1a5577b21488f5ae57d7333e28a401652" +catalog_checksum = "sha256:9ef86be22b7febc2690459e76e78716a6c3f6afd8c25fac93e820531837d787d" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:74682ca2c41116adfe28e1d37d5caae455c8d40b04b7ec4bf60765865b2d2c1e" +checksum = "sha256:9d2502d16ae08315e772844c04bcd99e8e22c9e4f2fdbc48187dc542bfbd3ebe" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:af3bb041c22ea9fc5647e7c9a3f7102ad8712a69b760b2e128b5affed81ed538" +checksum = "sha256:b753ca4ed4b29c42a0b2d22cbf7ad23d7af9d4aa76c5b582a73d6f00321f4e9b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 2b77a49b..69e6177e 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -15,13 +15,30 @@ # validate the base, and a floating tag can therefore change underneath a tag # that claims to name fixed content. # +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint +# rather than a preference. `anvil-setup binstall` installs the catalog as +# prebuilt binaries, which is what CI does; those binaries are linked against +# the glibc of the runner they were built on. glibc is backward but not +# forward compatible, so a base older than the runner cannot run them: on +# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version +# GLIBC_2.39 not found`, because its published binary is built on 24.04 +# (glibc 2.39). +# +# Matching the runner exactly -- rather than merely picking something newer -- +# is what keeps the container predictive: a tool that cannot run here cannot +# run in CI either, and a newer base would let the container pass where CI +# fails. When `ubuntu-latest` moves to a newer release, this pin moves with +# it, and the tag changes with the file. +# # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via # `replace_artifact(artifacts::container::dockerfile(...))`; a single # repository can edit this file in place, which anvil's drift handling -# preserves. +# preserves. A lower baseline means the catalog must also install from source +# rather than with `binstall`. -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 diff --git a/.spelling b/.spelling index 4b1db43b..4889d04b 100644 --- a/.spelling +++ b/.spelling @@ -160,6 +160,7 @@ getters GFM Git's github +glibc globbing glommio grey @@ -272,6 +273,7 @@ pp pre-approved pre-generate pre-heating +prebuilt prefixed prepend prepended diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index dc2c3f59..05c33a5d 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -79,7 +79,7 @@ All five are annotated `[group("anvil-container")]` and appear as one cluster in | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook (§7.3), so a query never pulls. | | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag resolves. Skips the resolve hook too (§7.3). | | `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§5.4). | -| `GITHUB_TOKEN` | Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited (§5.3). | +| `GITHUB_TOKEN` | Forwarded into the run. Taken from the host environment, or from the gh CLI when that is unset (§5.3). | `NO_REBUILD` is evaluated independently of `NO_CACHE`, so the two compose: `anvil-container-status` sets `NO_REBUILD` and `NO_RESOLVE` together and answers from local state alone. When `NO_REBUILD` stops a build the reference is still @@ -159,6 +159,14 @@ Two properties sit outside the digest. The base image is not resolved during has digest-pinned; a floating tag could otherwise change beneath a tag that claims to name fixed content. The platform is pinned to `linux/amd64` on build and run, so hosts of differing architecture cannot compute one tag for two images. +The base must also track the Linux runner the generated workflows use, currently `ubuntu-latest` (24.04). This is a +correctness constraint rather than a preference: the image installs the catalog with `binstall`, as CI does, and those +prebuilt binaries are linked against the runner's glibc. glibc is backward but not forward compatible, so an older base +yields tools that install and cannot execute — on Debian bookworm (2.36) `cargo-aprz` aborts in the loader with +`GLIBC_2.39 not found`. Matching the runner rather than merely exceeding it is what keeps the container predictive: a +newer base would let a run pass here and fail in CI. A catalog that needs an older baseline must also install from +source rather than with `binstall`. + ## 5. Execution model One container is created per `anvil-container` invocation, however many checks the requested recipe runs, and is @@ -207,12 +215,19 @@ otherwise the engine leaves it as `/`, and anything falling back to `$HOME` writ ### 5.3 Environment -The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and, when it is set on the host, forwards `GITHUB_TOKEN` by name. -`anvil-aprz` runs in the `pr-fast` group and queries the GitHub advisory API, which allows 60 requests an hour -unauthenticated — fewer than a full tier needs — so without the token a containerized tier degrades to warnings and -rate limits. It is forwarded, never minted: running `gh auth token` in the driver would hand a broadly-scoped -credential to every recipe in the container, including the ones that never see it natively, where the recipe scopes -it to itself. A host that has not exported it gets the same unauthenticated warning it would get natively. +The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and forwards `GITHUB_TOKEN` by name. `anvil-aprz` runs in the `pr-fast` +group and queries the GitHub advisory API, which allows 60 requests an hour unauthenticated — fewer than a full tier +needs. Unauthenticated is not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota resets rather than +failing, and offers no way to opt out, so a containerized tier blocks for up to an hour. The token is what makes the +check terminate, not what makes it fast. + +The driver resolves it exactly as the recipe does natively — the environment first, then the gh CLI's stored token +(`gh auth token`, which is non-interactive) — so a containerized run authenticates for the same developers a native run +does. Deriving it rather than only forwarding an exported value is deliberate: both paths have identical exposure once +inside, since a forwarded token is readable by every recipe in the container either way, including third-party build +scripts and proc macros. Refusing to derive it buys no boundary and only makes behaviour depend on how the developer +happened to sign in — which is the common case, because the recipe itself recommends `gh auth login`. A derived value +is set on the driver process, passed by name, and unset again, so it never reaches the host's command line. Everything else a run needs comes from the hook (§7). diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 0583f5c4..b644287e 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -43,8 +43,14 @@ pub fn recipe() -> Artifact { Artifact::owned_file(RECIPE_PATH, RECIPE) } -/// The default execution image: a digest-pinned Debian base that installs the -/// pinned toolchain and the generated tool catalog by running `just anvil-setup`. +/// The default execution image: a digest-pinned base tracking the Linux CI +/// runner, which installs the pinned toolchain and the generated tool catalog +/// by running `just anvil-setup`. +/// +/// The base must not be older than that runner. The catalog is installed as +/// prebuilt binaries, and glibc is backward but not forward compatible, so an +/// older base cannot run them; matching the runner also keeps the container +/// predictive of CI rather than more permissive than it. /// /// A downstream catalog that needs a different base OS or toolchain source /// replaces the body wholesale: @@ -385,20 +391,27 @@ mod tests { } #[test] - fn a_host_token_is_forwarded_by_name_never_minted() { - // anvil-aprz is in pr-fast, so a containerized tier hits the - // unauthenticated advisory-API limit without it. Forwarding by name - // keeps the value off the command line; minting one here would give - // every recipe in the container a credential it lacks natively. + fn a_host_token_is_resolved_as_the_recipe_does_and_forwarded_by_name() { + // anvil-aprz is in pr-fast, and unauthenticated it does not merely + // warn: `cargo aprz deps` sleeps until the hourly quota resets, so a + // containerized tier blocks for up to an hour. The driver therefore + // resolves a token the same way the recipe does natively -- the + // environment first, then the gh CLI -- so both paths authenticate for + // the same developers. + assert!(RECIPE.contains("gh auth token --hostname github.com")); assert!(RECIPE.contains("$forwardedEnv += 'GITHUB_TOKEN'")); assert!(RECIPE.contains("$runArgs += @('-e', 'GITHUB_TOKEN')")); + // By name, never by value: `-e NAME=VALUE` would put the credential on + // the host's command line, where endpoint telemetry retains it. + assert!(!RECIPE.contains("'-e', \"GITHUB_TOKEN=")); + // A derived token is set on this process, so it must be registered for + // the same cleanup the hook's variables get. + assert!(RECIPE.contains("$hookEnv += 'GITHUB_TOKEN'")); + // An exported token is left alone rather than re-derived. + assert!(RECIPE.contains("if (-not $env:GITHUB_TOKEN -and (Get-Command gh")); // Forwarding by name only works if the engine can see the name, so a // WSL engine needs it bridged -- otherwise `-e NAME` forwards nothing. assert!(RECIPE.contains("$engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0")); - // Invocation forms only, so the comment explaining the choice may name - // the command it rules out. - assert!(!RECIPE.contains("(gh auth token")); - assert!(!RECIPE.contains("& gh ")); } #[test] diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile index 2b77a49b..69e6177e 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -15,13 +15,30 @@ # validate the base, and a floating tag can therefore change underneath a tag # that claims to name fixed content. # +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint +# rather than a preference. `anvil-setup binstall` installs the catalog as +# prebuilt binaries, which is what CI does; those binaries are linked against +# the glibc of the runner they were built on. glibc is backward but not +# forward compatible, so a base older than the runner cannot run them: on +# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version +# GLIBC_2.39 not found`, because its published binary is built on 24.04 +# (glibc 2.39). +# +# Matching the runner exactly -- rather than merely picking something newer -- +# is what keeps the container predictive: a tool that cannot run here cannot +# run in CI either, and a newer base would let the container pass where CI +# fails. When `ubuntu-latest` moves to a newer release, this pin moves with +# it, and the tag changes with the file. +# # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via # `replace_artifact(artifacts::container::dockerfile(...))`; a single # repository can edit this file in place, which anvil's drift handling -# preserves. +# preserves. A lower baseline means the catalog must also install from source +# rather than with `binstall`. -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 0ebbb08e..16634ecf 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -467,16 +467,32 @@ anvil-container *target: $hookEnv = @() # anvil-aprz queries the GitHub advisory API, which allows 60 requests an - # hour unauthenticated -- less than a full tier needs. Forward the host's - # token by name when it is already set, so a containerized tier - # authenticates exactly as the same recipe does natively, and so CI (which - # sets it) keeps working when it runs a tier this way. + # hour unauthenticated -- less than a full tier needs. Unauthenticated is + # not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota + # resets rather than failing, so a containerized tier blocks for up to an + # hour with no way to opt out. Authentication is what makes the check + # terminate, not what makes it fast. # - # Only forwarded, never minted: running `gh auth token` here would hand a - # broadly-scoped credential to every recipe in the container, including - # ones that never see it on the host, where anvil-aprz scopes it to itself. - # A host that has not exported it gets the same unauthenticated warning the - # recipe prints natively. + # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN + # first, then the gh CLI's stored token -- so a containerized run + # authenticates for the same developers a native run does. Deriving it here + # rather than only forwarding an exported value is deliberate: the two + # paths have identical exposure once inside (a forwarded token is readable + # by every recipe in the container either way, including third-party build + # scripts and proc macros), so refusing to derive it buys no boundary and + # only makes behaviour depend on how the developer happened to sign in. + # `gh auth token` is non-interactive and never opens a prompt. + # + # Set here and passed by NAME, so the value never reaches the host's + # process command line, and unset again with the hook's variables below. + if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } + } if ($env:GITHUB_TOKEN) { $forwardedEnv += 'GITHUB_TOKEN' $runArgs += @('-e', 'GITHUB_TOKEN') diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 3c02b1b6..7c2a0cbe 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -20,13 +20,30 @@ expression: render_tree(tmp.path()) # validate the base, and a floating tag can therefore change underneath a tag # that claims to name fixed content. # +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint +# rather than a preference. `anvil-setup binstall` installs the catalog as +# prebuilt binaries, which is what CI does; those binaries are linked against +# the glibc of the runner they were built on. glibc is backward but not +# forward compatible, so a base older than the runner cannot run them: on +# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version +# GLIBC_2.39 not found`, because its published binary is built on 24.04 +# (glibc 2.39). +# +# Matching the runner exactly -- rather than merely picking something newer -- +# is what keeps the container predictive: a tool that cannot run here cannot +# run in CI either, and a newer base would let the container pass where CI +# fails. When `ubuntu-latest` moves to a newer release, this pin moves with +# it, and the tag changes with the file. +# # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via # `replace_artifact(artifacts::container::dockerfile(...))`; a single # repository can edit this file in place, which anvil's drift handling -# preserves. +# preserves. A lower baseline means the catalog must also install from source +# rather than with `binstall`. -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -3953,16 +3970,32 @@ anvil-container *target: $hookEnv = @() # anvil-aprz queries the GitHub advisory API, which allows 60 requests an - # hour unauthenticated -- less than a full tier needs. Forward the host's - # token by name when it is already set, so a containerized tier - # authenticates exactly as the same recipe does natively, and so CI (which - # sets it) keeps working when it runs a tier this way. + # hour unauthenticated -- less than a full tier needs. Unauthenticated is + # not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota + # resets rather than failing, so a containerized tier blocks for up to an + # hour with no way to opt out. Authentication is what makes the check + # terminate, not what makes it fast. + # + # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN + # first, then the gh CLI's stored token -- so a containerized run + # authenticates for the same developers a native run does. Deriving it here + # rather than only forwarding an exported value is deliberate: the two + # paths have identical exposure once inside (a forwarded token is readable + # by every recipe in the container either way, including third-party build + # scripts and proc macros), so refusing to derive it buys no boundary and + # only makes behaviour depend on how the developer happened to sign in. + # `gh auth token` is non-interactive and never opens a prompt. # - # Only forwarded, never minted: running `gh auth token` here would hand a - # broadly-scoped credential to every recipe in the container, including - # ones that never see it on the host, where anvil-aprz scopes it to itself. - # A host that has not exported it gets the same unauthenticated warning the - # recipe prints natively. + # Set here and passed by NAME, so the value never reaches the host's + # process command line, and unset again with the hook's variables below. + if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } + } if ($env:GITHUB_TOKEN) { $forwardedEnv += 'GITHUB_TOKEN' $runArgs += @('-e', 'GITHUB_TOKEN') diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index c14b0550..9aecfe2b 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -20,13 +20,30 @@ expression: render_tree(tmp.path()) # validate the base, and a floating tag can therefore change underneath a tag # that claims to name fixed content. # +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint +# rather than a preference. `anvil-setup binstall` installs the catalog as +# prebuilt binaries, which is what CI does; those binaries are linked against +# the glibc of the runner they were built on. glibc is backward but not +# forward compatible, so a base older than the runner cannot run them: on +# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version +# GLIBC_2.39 not found`, because its published binary is built on 24.04 +# (glibc 2.39). +# +# Matching the runner exactly -- rather than merely picking something newer -- +# is what keeps the container predictive: a tool that cannot run here cannot +# run in CI either, and a newer base would let the container pass where CI +# fails. When `ubuntu-latest` moves to a newer release, this pin moves with +# it, and the tag changes with the file. +# # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via # `replace_artifact(artifacts::container::dockerfile(...))`; a single # repository can edit this file in place, which anvil's drift handling -# preserves. +# preserves. A lower baseline means the catalog must also install from source +# rather than with `binstall`. -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -3874,16 +3891,32 @@ anvil-container *target: $hookEnv = @() # anvil-aprz queries the GitHub advisory API, which allows 60 requests an - # hour unauthenticated -- less than a full tier needs. Forward the host's - # token by name when it is already set, so a containerized tier - # authenticates exactly as the same recipe does natively, and so CI (which - # sets it) keeps working when it runs a tier this way. + # hour unauthenticated -- less than a full tier needs. Unauthenticated is + # not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota + # resets rather than failing, so a containerized tier blocks for up to an + # hour with no way to opt out. Authentication is what makes the check + # terminate, not what makes it fast. + # + # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN + # first, then the gh CLI's stored token -- so a containerized run + # authenticates for the same developers a native run does. Deriving it here + # rather than only forwarding an exported value is deliberate: the two + # paths have identical exposure once inside (a forwarded token is readable + # by every recipe in the container either way, including third-party build + # scripts and proc macros), so refusing to derive it buys no boundary and + # only makes behaviour depend on how the developer happened to sign in. + # `gh auth token` is non-interactive and never opens a prompt. # - # Only forwarded, never minted: running `gh auth token` here would hand a - # broadly-scoped credential to every recipe in the container, including - # ones that never see it on the host, where anvil-aprz scopes it to itself. - # A host that has not exported it gets the same unauthenticated warning the - # recipe prints natively. + # Set here and passed by NAME, so the value never reaches the host's + # process command line, and unset again with the hook's variables below. + if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } + } if ($env:GITHUB_TOKEN) { $forwardedEnv += 'GITHUB_TOKEN' $runArgs += @('-e', 'GITHUB_TOKEN') diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index dcb3955d..e4f0318f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -20,13 +20,30 @@ expression: render_tree(tmp.path()) # validate the base, and a floating tag can therefore change underneath a tag # that claims to name fixed content. # +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint +# rather than a preference. `anvil-setup binstall` installs the catalog as +# prebuilt binaries, which is what CI does; those binaries are linked against +# the glibc of the runner they were built on. glibc is backward but not +# forward compatible, so a base older than the runner cannot run them: on +# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version +# GLIBC_2.39 not found`, because its published binary is built on 24.04 +# (glibc 2.39). +# +# Matching the runner exactly -- rather than merely picking something newer -- +# is what keeps the container predictive: a tool that cannot run here cannot +# run in CI either, and a newer base would let the container pass where CI +# fails. When `ubuntu-latest` moves to a newer release, this pin moves with +# it, and the tag changes with the file. +# # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via # `replace_artifact(artifacts::container::dockerfile(...))`; a single # repository can edit this file in place, which anvil's drift handling -# preserves. +# preserves. A lower baseline means the catalog must also install from source +# rather than with `binstall`. -ARG BASE_IMAGE=docker.io/library/debian:bookworm-slim@sha256:63a496b5d3b99214b39f5ed70eb71a61e590a77979c79cbee4faf991f8c0783e +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -2693,16 +2710,32 @@ anvil-container *target: $hookEnv = @() # anvil-aprz queries the GitHub advisory API, which allows 60 requests an - # hour unauthenticated -- less than a full tier needs. Forward the host's - # token by name when it is already set, so a containerized tier - # authenticates exactly as the same recipe does natively, and so CI (which - # sets it) keeps working when it runs a tier this way. + # hour unauthenticated -- less than a full tier needs. Unauthenticated is + # not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota + # resets rather than failing, so a containerized tier blocks for up to an + # hour with no way to opt out. Authentication is what makes the check + # terminate, not what makes it fast. + # + # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN + # first, then the gh CLI's stored token -- so a containerized run + # authenticates for the same developers a native run does. Deriving it here + # rather than only forwarding an exported value is deliberate: the two + # paths have identical exposure once inside (a forwarded token is readable + # by every recipe in the container either way, including third-party build + # scripts and proc macros), so refusing to derive it buys no boundary and + # only makes behaviour depend on how the developer happened to sign in. + # `gh auth token` is non-interactive and never opens a prompt. # - # Only forwarded, never minted: running `gh auth token` here would hand a - # broadly-scoped credential to every recipe in the container, including - # ones that never see it on the host, where anvil-aprz scopes it to itself. - # A host that has not exported it gets the same unauthenticated warning the - # recipe prints natively. + # Set here and passed by NAME, so the value never reaches the host's + # process command line, and unset again with the hook's variables below. + if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } + } if ($env:GITHUB_TOKEN) { $forwardedEnv += 'GITHUB_TOKEN' $runArgs += @('-e', 'GITHUB_TOKEN') diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 0ebbb08e..16634ecf 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -467,16 +467,32 @@ anvil-container *target: $hookEnv = @() # anvil-aprz queries the GitHub advisory API, which allows 60 requests an - # hour unauthenticated -- less than a full tier needs. Forward the host's - # token by name when it is already set, so a containerized tier - # authenticates exactly as the same recipe does natively, and so CI (which - # sets it) keeps working when it runs a tier this way. + # hour unauthenticated -- less than a full tier needs. Unauthenticated is + # not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota + # resets rather than failing, so a containerized tier blocks for up to an + # hour with no way to opt out. Authentication is what makes the check + # terminate, not what makes it fast. # - # Only forwarded, never minted: running `gh auth token` here would hand a - # broadly-scoped credential to every recipe in the container, including - # ones that never see it on the host, where anvil-aprz scopes it to itself. - # A host that has not exported it gets the same unauthenticated warning the - # recipe prints natively. + # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN + # first, then the gh CLI's stored token -- so a containerized run + # authenticates for the same developers a native run does. Deriving it here + # rather than only forwarding an exported value is deliberate: the two + # paths have identical exposure once inside (a forwarded token is readable + # by every recipe in the container either way, including third-party build + # scripts and proc macros), so refusing to derive it buys no boundary and + # only makes behaviour depend on how the developer happened to sign in. + # `gh auth token` is non-interactive and never opens a prompt. + # + # Set here and passed by NAME, so the value never reaches the host's + # process command line, and unset again with the hook's variables below. + if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } + } if ($env:GITHUB_TOKEN) { $forwardedEnv += 'GITHUB_TOKEN' $runArgs += @('-e', 'GITHUB_TOKEN') From 99ba00c3f802508e2446eeb2b03c14974c9db3e7 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 12:35:23 +0200 Subject: [PATCH 23/81] test(anvil): dogfood containerized execution against this repository `test-anvil-container.ps1` proves the container mechanism against a throwaway fixture. That fixture is deliberately minimal, so the checks it runs there are trivial -- and both defects fixed in the previous commit were invisible to it: one needs the full pinned catalog installed, the other needs a check that reaches the GitHub advisory API. This script runs the generated recipes against ox-tools itself, and orders its steps by cost so a break is reported in seconds rather than after a full tier. The step worth keeping is the catalog ABI matrix. `anvil-setup` installing a tool proves only that it downloaded; the catalog is installed as prebuilt binaries, so a base image older than the runner they were built on yields tools that are present and unrunnable. Executing all twenty costs seconds and catches the whole class at once, rather than one tool at a time as tiers happen to reach them. Verified non-vacuous against the previous Debian image, where it reports cargo-aprz exactly as the loader does. Both engines run by default. On Windows that is not redundant: docker is reached through the default WSL distribution and podman runs natively, so the two also cover both invocation paths. Both produce the same content tag. --- scripts/test-anvil-dogfood.ps1 | 432 +++++++++++++++++++++++++++++++++ 1 file changed, 432 insertions(+) create mode 100644 scripts/test-anvil-dogfood.ps1 diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 new file mode 100644 index 00000000..a87cb2a0 --- /dev/null +++ b/scripts/test-anvil-dogfood.ps1 @@ -0,0 +1,432 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# +.SYNOPSIS + Dogfood test: run this repository's own anvil checks inside the container. + +.DESCRIPTION + `test-anvil-container.ps1` proves the container *mechanism* against a + throwaway fixture: artifacts, tags, drift, hooks. It deliberately generates + a minimal repository, so the checks it runs there are trivial. + + This script proves the opposite half -- that the mechanism is actually + usable on a real workspace. It runs the generated recipes against + `ox-tools` itself: a multi-crate workspace with real dependencies, real + lints, and the full pinned tool catalog. A defect that only appears at that + scale (a tool that installs but cannot execute, a check that needs a file + the build context drops, a mount that hides the workspace) is invisible to + the fixture test and lands directly on a customer. + + It runs against **both** engines by default, which is not redundant on + Windows: docker is reached through the default WSL distribution, and podman + runs natively, so the two cover both invocation paths as well as both + engines. + + Ordering is by cost, so a break is reported in seconds rather than after a + full tier: + + 1. Preconditions -- the generated tree is current, and the container + artifacts are the ones the engine emits. + 2. The image builds from the repository's own Dockerfile. + 3. Every pinned tool in the catalog *executes* inside the image. + 4. `anvil-aprz` runs -- the check whose prebuilt binary first exposed an + ABI mismatch against the base image. + 5. The requested tier runs to completion inside the image. + 6. A second run reuses the image rather than rebuilding it. + + Step 3 is the one worth keeping cheap. `anvil-setup` installing a tool only + proves it downloaded; the catalog is installed as prebuilt binaries, so a + base image older than the runner those binaries were built on yields tools + that are present and unrunnable. Executing all of them costs seconds and + catches the whole class at once, instead of one tool at a time as tiers + happen to reach them. + +.PARAMETER Engine + Which engine(s) to test. 'both' (default) runs the whole suite against + docker and then podman, reporting them separately. + +.PARAMETER Tier + Recipe(s) to run for step 5. Defaults to `anvil-pr`, the full PR tier. + `anvil-pr` includes `anvil-pr-slow`, which includes mutants and runtime + analysis, so it is measured in tens of minutes; pass `anvil-pr-fast` for a + quicker pass over the same plumbing. + +.PARAMETER SkipTier + Stop after step 4. The cheap steps cover the container contract; the tier + is what makes a full run long. + +.PARAMETER KeepImages + Leave built images and cache volumes in place. On by default in spirit -- + this repository's image is expensive to build, so the script never removes + it unless -Clean is passed. + +.PARAMETER Clean + Remove this repository's anvil images and cache volumes before starting, + forcing a cold build. Use when testing the image definition itself. + +.EXAMPLE + ./scripts/test-anvil-dogfood.ps1 + ./scripts/test-anvil-dogfood.ps1 -Engine docker -Tier anvil-pr-fast + ./scripts/test-anvil-dogfood.ps1 -SkipTier -Clean +#> + +[CmdletBinding()] +param( + [ValidateSet('docker', 'podman', 'both')] + [string]$Engine = 'both', + [string[]]$Tier = @('anvil-pr'), + [switch]$SkipTier, + [switch]$KeepImages, + [switch]$Clean +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +# ---------------------------------------------------------------- reporting -- + +$script:Passed = 0 +$script:Failed = 0 +$script:Skipped = 0 +$script:Started = Get-Date +$script:Results = [ordered]@{} +$script:CurrentEngine = '' + +function Write-Section([string]$Title) { + Write-Host '' + Write-Host "=== $Title " -NoNewline -ForegroundColor Cyan + Write-Host ('=' * [Math]::Max(0, 72 - $Title.Length)) -ForegroundColor Cyan +} + +function Write-Step([string]$Message) { + Write-Host " -> $Message" -ForegroundColor DarkGray +} + +function Write-Detail([string]$Message) { + if (-not $Message) { return } + foreach ($line in ($Message -split "`r?`n")) { + if ($line.Trim()) { Write-Host " | $line" -ForegroundColor DarkGray } + } +} + +function Write-Tail([string]$Message, [int]$Lines = 25) { + if (-not $Message) { return } + $all = @($Message -split "`r?`n" | Where-Object { $_.Trim() }) + Write-Detail (($all | Select-Object -Last $Lines) -join "`n") +} + +function Assert-That([string]$Name, [bool]$Condition, [string]$Detail = '') { + if ($Condition) { + $script:Passed++ + Write-Host " [PASS] $Name" -ForegroundColor Green + } else { + $script:Failed++ + Write-Host " [FAIL] $Name" -ForegroundColor Red + if ($Detail) { Write-Detail $Detail } + } +} + +function Assert-Equal([string]$Name, $Expected, $Actual) { + Assert-That $Name ($Expected -eq $Actual) "expected: $Expected`nactual: $Actual" +} + +function Write-Skipped([string]$Name, [string]$Why) { + $script:Skipped++ + Write-Host " [SKIP] $Name" -ForegroundColor Yellow + Write-Detail $Why +} + +# ------------------------------------------------------------------ helpers -- + +function Invoke-Native { + param( + [Parameter(Mandatory)][string]$Command, + [string[]]$Arguments = @(), + [string]$WorkingDirectory, + [hashtable]$Environment = @{}, + [switch]$AllowFailure + ) + + $previous = @{} + foreach ($key in $Environment.Keys) { + $previous[$key] = [Environment]::GetEnvironmentVariable($key) + Set-Item -LiteralPath "Env:$key" -Value $Environment[$key] + } + $entered = $false + try { + if ($WorkingDirectory) { Push-Location $WorkingDirectory; $entered = $true } + $stdoutFile = [System.IO.Path]::GetTempFileName() + $stderrFile = [System.IO.Path]::GetTempFileName() + try { + $process = Start-Process -FilePath $Command -ArgumentList $Arguments -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput $stdoutFile -RedirectStandardError $stderrFile + $result = [pscustomobject]@{ + ExitCode = $process.ExitCode + StdOut = (Get-Content -LiteralPath $stdoutFile -Raw -ErrorAction SilentlyContinue) ?? '' + StdErr = (Get-Content -LiteralPath $stderrFile -Raw -ErrorAction SilentlyContinue) ?? '' + } + } finally { + Remove-Item -LiteralPath $stdoutFile, $stderrFile -Force -ErrorAction SilentlyContinue + } + } finally { + if ($entered) { Pop-Location } + foreach ($key in $Environment.Keys) { + if ($null -eq $previous[$key]) { + Remove-Item -LiteralPath "Env:$key" -ErrorAction SilentlyContinue + } else { + Set-Item -LiteralPath "Env:$key" -Value $previous[$key] + } + } + } + + if (-not $AllowFailure -and $result.ExitCode -ne 0) { + Write-Detail $result.StdOut + Write-Detail $result.StdErr + throw "$Command $($Arguments -join ' ') failed with exit code $($result.ExitCode)" + } + $result +} + +function Resolve-Engine([string]$Name) { + # Mirrors what container.just does: prefer the engine on PATH, and fall + # back to the default WSL distribution on Windows. The script must not + # assume more than the product does. + if (Get-Command $Name -ErrorAction SilentlyContinue) { + return [pscustomobject]@{ Exe = $Name; Prefix = @(); ViaWsl = $false } + } + if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + & wsl.exe --exec $Name --version *> $null + if ($LASTEXITCODE -eq 0) { + return [pscustomobject]@{ Exe = 'wsl.exe'; Prefix = @('--exec', $Name); ViaWsl = $true } + } + } + $null +} + +function Invoke-Engine { + param([string[]]$Arguments, [switch]$AllowFailure) + Invoke-Native -Command $script:EngineExe -Arguments ($script:EnginePrefix + $Arguments) -AllowFailure:$AllowFailure +} + +function Invoke-Just { + param( + [Parameter(Mandatory)][string[]]$Arguments, + [hashtable]$Environment = @{}, + [switch]$AllowFailure + ) + $env = @{ ANVIL_CONTAINER_ENGINE = $script:CurrentEngine } + $Environment + Invoke-Native -Command 'just' -Arguments $Arguments -WorkingDirectory $RepoRoot -Environment $env -AllowFailure:$AllowFailure +} + +function Get-ImageReference { + # anvil-container-status reports the reference without building it. + $status = Invoke-Just -Arguments @('anvil-container-status') -AllowFailure + $line = ($status.StdOut -split "`r?`n") | Where-Object { $_ -match '^\s*image:\s*(\S+)' } | Select-Object -First 1 + if ($line -match '^\s*image:\s*(\S+)') { return $Matches[1] } + '' +} + +function Test-ImagePresent([string]$Reference) { + if (-not $Reference) { return $false } + (Invoke-Engine -Arguments @('image', 'inspect', $Reference) -AllowFailure).ExitCode -eq 0 +} + +function Remove-AnvilImages([string]$Prefix) { + $images = Invoke-Engine -Arguments @('images', '--format', '{{.Repository}}:{{.Tag}}') -AllowFailure + # Podman reports images fully qualified (`localhost/anvil-...`), docker does + # not, so match anywhere in the reference rather than at the start. + foreach ($image in (($images.StdOut -split "`r?`n") | Where-Object { $_ -like "*$Prefix*" })) { + Write-Step "removing image $image" + Invoke-Engine -Arguments @('rmi', '-f', $image) -AllowFailure | Out-Null + } + $volumes = Invoke-Engine -Arguments @('volume', 'ls', '--format', '{{.Name}}') -AllowFailure + foreach ($volume in (($volumes.StdOut -split "`r?`n") | Where-Object { $_ -like "*$Prefix*" })) { + Write-Step "removing volume $volume" + Invoke-Engine -Arguments @('volume', 'rm', '-f', $volume) -AllowFailure | Out-Null + } +} + +# The pinned catalog, read from the generated recipes rather than restated +# here. A tool added to the catalog is covered without editing this script -- +# the same reason the image installs by running `anvil-setup` instead of +# carrying its own list. +function Get-PinnedTools { + $versions = Join-Path $RepoRoot 'justfiles/anvil/versions.just' + $tools = [ordered]@{} + foreach ($line in (Get-Content -LiteralPath $versions)) { + if ($line -match '^\s*([a-z0-9_]+)_version\s*:=\s*"([^"]+)"') { + $name = $Matches[1] -replace '_', '-' + if ($name -like 'cargo-*') { $tools[$name] = $Matches[2] } + } + } + $tools +} + +# ---------------------------------------------------------------- the suite -- + +function Invoke-Suite([string]$EngineName) { + $script:CurrentEngine = $EngineName + $before = $script:Failed + + Write-Section "$EngineName : preconditions" + + $resolved = Resolve-Engine $EngineName + if (-not $resolved) { + Write-Skipped "$EngineName is available" "not on PATH, and not reachable in the default WSL distribution" + $script:Results[$EngineName] = 'skipped' + return + } + $script:EngineExe = $resolved.Exe + $script:EnginePrefix = $resolved.Prefix + $script:EngineViaWsl = $resolved.ViaWsl + Write-Step ("engine: {0}{1}" -f $EngineName, $(if ($resolved.ViaWsl) { ' (via WSL)' } else { ' (native)' })) + + # The dogfood claim is only meaningful if the committed tree is what the + # generator produces. A stale tree would test something no user can obtain. + $dryRun = Invoke-Native -Command 'cargo' -Arguments @('run', '--quiet', '-p', 'cargo-anvil', '--', 'anvil', '--dry-run') ` + -WorkingDirectory $RepoRoot -AllowFailure + Assert-Equal 'the committed anvil tree is current (cargo anvil --dry-run)' 0 $dryRun.ExitCode + + foreach ($artifact in @('.anvil/container/Dockerfile', '.anvil/container/Dockerfile.dockerignore', 'justfiles/anvil/container.just')) { + Assert-That "$artifact is present" (Test-Path -LiteralPath (Join-Path $RepoRoot $artifact)) + } + + $imagePrefix = 'anvil-' + (Split-Path -Leaf $RepoRoot).ToLowerInvariant() + if ($Clean) { + Write-Step 'removing existing images and volumes for a cold build' + Remove-AnvilImages -Prefix $imagePrefix + } + + Write-Section "$EngineName : image" + + $reference = Get-ImageReference + Assert-That 'anvil-container-status reports an image reference' ([bool]$reference) 'no image: line in status output' + Write-Step "reference: $reference" + + $up = Invoke-Just -Arguments @('anvil-container', 'anvil-container-tag') -AllowFailure + Assert-Equal 'the first run builds the image if it is missing' 0 $up.ExitCode + if ($up.ExitCode -ne 0) { + Write-Tail $up.StdErr + # Everything below needs an image; stop this engine rather than + # reporting a cascade of failures that all have one cause. + $script:Results[$EngineName] = 'failed' + return + } + Assert-That 'the image is present after the first run' (Test-ImagePresent $reference) + + Write-Section "$EngineName : the catalog executes inside the image" + + # `anvil-setup` proves a tool installed, not that it runs. The catalog is + # installed as prebuilt binaries linked against the glibc of the runner + # they were built on, so a base image older than that runner produces a + # tool that is present and unrunnable. Executing each one is the cheapest + # check that covers the whole class. + # + # This runs the engine directly rather than through `anvil-container`, + # which dispatches `just ` and so cannot invoke a bare binary. The + # image reference still comes from the product (`anvil-container-status`), + # and the property under test belongs to the image rather than the recipe. + $tools = Get-PinnedTools + Write-Step "$($tools.Count) pinned tools" + $broken = @() + $missing = @() + foreach ($tool in $tools.Keys) { + $run = Invoke-Engine -Arguments @('run', '--rm', $reference, $tool, '--version') -AllowFailure + $combined = "$($run.StdOut)`n$($run.StdErr)" + # The dynamic loader reports an ABI mismatch before main() runs, so + # this is independent of whether a tool implements --version at all. + if ($combined -match 'GLIBC_[0-9.]+.{0,40}not found|error while loading shared libraries|cannot execute binary file') { + $line = (($combined -split "`r?`n") | Where-Object { $_ -match 'GLIBC|shared libraries|cannot execute' } | Select-Object -First 1).Trim() + $broken += "$tool -> $line" + } elseif ($combined -match 'executable file .*not found|no such file or directory') { + $missing += $tool + } + } + Assert-That 'every pinned tool executes inside the image' ($broken.Count -eq 0) ($broken -join "`n") + Assert-That 'every pinned tool is present in the image' ($missing.Count -eq 0) ($missing -join ', ') + + Write-Section "$EngineName : checks" + + # The check whose prebuilt binary first exposed the base-image ABI + # mismatch. Kept as its own step so a regression names itself. + $aprz = Invoke-Just -Arguments @('anvil-container', 'anvil-aprz') -AllowFailure + Assert-Equal 'anvil-aprz runs inside the image' 0 $aprz.ExitCode + if ($aprz.ExitCode -ne 0) { Write-Tail "$($aprz.StdOut)`n$($aprz.StdErr)" } + + # A check that reads the workspace rather than the network, so a failure + # points at the mount rather than at connectivity. + $fmt = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure + Assert-Equal 'anvil-fmt runs inside the image' 0 $fmt.ExitCode + if ($fmt.ExitCode -ne 0) { Write-Tail "$($fmt.StdOut)`n$($fmt.StdErr)" } + + Write-Section "$EngineName : image reuse" + + $secondReference = Get-ImageReference + Assert-Equal 'the tag is stable across runs' $reference $secondReference + $reuse = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure + Assert-Equal 'a later run succeeds' 0 $reuse.ExitCode + Assert-That 'a later run does not rebuild the image' ` + (-not ("$($reuse.StdOut)`n$($reuse.StdErr)" -match 'building image|Step 1/|FROM ')) ` + 'a rebuild happened when the tag should have resolved' + + if ($SkipTier) { + Write-Skipped "$EngineName : tier" '-SkipTier was passed' + } else { + Write-Section "$EngineName : tier" + foreach ($recipe in $Tier) { + Write-Step "running $recipe (this is the long one)" + $started = Get-Date + $run = Invoke-Just -Arguments @('anvil-container', $recipe) -AllowFailure + $took = (Get-Date) - $started + Assert-Equal ("{0} passes inside the image (took {1:mm\:ss})" -f $recipe, $took) 0 $run.ExitCode + if ($run.ExitCode -ne 0) { Write-Tail "$($run.StdOut)`n$($run.StdErr)" 40 } + } + } + + $script:Results[$EngineName] = $(if ($script:Failed -gt $before) { 'failed' } else { 'passed' }) +} + +# -------------------------------------------------------------------- main ---- + +$RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path + +Write-Host '' +Write-Host 'anvil containerized execution - dogfood against this repository' -ForegroundColor White +Write-Host "repository: $RepoRoot" -ForegroundColor DarkGray +Write-Host "tier: $(if ($SkipTier) { '(skipped)' } else { $Tier -join ', ' })" -ForegroundColor DarkGray + +foreach ($tool in @('just', 'cargo')) { + if (-not (Get-Command $tool -ErrorAction SilentlyContinue)) { + Write-Host "FAIL $tool is required on PATH" -ForegroundColor Red + exit 1 + } +} + +$engines = if ($Engine -eq 'both') { @('docker', 'podman') } else { @($Engine) } +foreach ($name in $engines) { + Invoke-Suite $name +} + +if (-not $KeepImages -and -not $Clean) { + # The image is expensive to build and is keyed by content, so keeping it is + # correct: the next run reuses it, and an input change renames it anyway. + Write-Step 'keeping built images (content-addressed; -Clean forces a cold build)' +} + +$elapsed = (Get-Date) - $script:Started +Write-Host '' +Write-Host ('-' * 78) +foreach ($name in $script:Results.Keys) { + $state = $script:Results[$name] + $color = switch ($state) { 'passed' { 'Green' } 'skipped' { 'Yellow' } default { 'Red' } } + Write-Host (" {0,-8} {1}" -f $name, $state) -ForegroundColor $color +} +$summary = "{0}/{1} checks passed in {2:hh\:mm\:ss}" -f $script:Passed, ($script:Passed + $script:Failed), $elapsed +if ($script:Skipped) { $summary += " ($($script:Skipped) skipped)" } +if ($script:Failed -eq 0) { + Write-Host "PASS $summary" -ForegroundColor Green + exit 0 +} +Write-Host "FAIL $summary ($($script:Failed) failed)" -ForegroundColor Red +exit 1 From 6c49f4874dd143b3f46b7c7323af41cf245a8017 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 12:52:09 +0200 Subject: [PATCH 24/81] fix(anvil): stop the worktree git redirection leaking into every process A containerized tier failed on a linked worktree: error: could not lock config file /anvil/gitdir/config fatal: could not set 'core.repositoryformatversion' to '0' git ["init"] failed Reaching a linked worktree's history was done with GIT_DIR and GIT_WORK_TREE, which are ambient: every process in the container inherits them. Any git run outside the workspace therefore operated on this repository instead of its own. The test suite is where that surfaces -- `git init` in a scratch directory honours the inherited GIT_DIR over the directory it was told to create -- so `anvil-llvm-cov` failed three tests and aborted the run, taking `anvil-pr` with it. Reproduced directly: docker run -e GIT_DIR=/anvil/gitdir/worktrees/x \ bash -c 'mkdir /tmp/t && cd /tmp/t && git init' fatal: Invalid path '/anvil': No such file or directory The redirection now goes where it belongs, in the checkout: a generated `.git` file naming the mounted common directory is bind-mounted read-only over the worktree's own, and ordinary discovery does the rest. Nothing is ambient, so a git command outside the workspace behaves exactly as it does natively. The temporary file is removed with the hook's variables. An ordinary clone still takes none of this, and a host without git on PATH still skips the question. `anvil-llvm-cov` now passes inside the image, 709 tests and coverage thresholds included. --- .anvil.lock | 4 ++-- .../src/anvil/artifacts/container.rs | 12 +++++++++- .../templates/justfiles/anvil/container.just | 24 +++++++++++++++---- .../snapshots/snapshots__ado_backend.snap | 24 +++++++++++++++---- .../snapshots/snapshots__github_backend.snap | 24 +++++++++++++++---- .../snapshots/snapshots__local_only.snap | 24 +++++++++++++++---- justfiles/anvil/container.just | 24 +++++++++++++++---- 7 files changed, 113 insertions(+), 23 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index c8438b48..4fc6a11a 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:9ef86be22b7febc2690459e76e78716a6c3f6afd8c25fac93e820531837d787d" +catalog_checksum = "sha256:d60fe5206596db4e5193858e6814f928bf2e27e7a8b9d65eb123fb2caae28bf9" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:b753ca4ed4b29c42a0b2d22cbf7ad23d7af9d4aa76c5b582a73d6f00321f4e9b" +checksum = "sha256:23d3cfad3f6ebbfbe78404fa72f4762fec7372c7deff2fb2a40b6d5bc934084b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index b644287e..69ce0def 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -430,7 +430,17 @@ mod tests { // that needs history fails. assert!(RECIPE.contains("git rev-parse --git-common-dir")); assert!(RECIPE.contains("${engineGitCommon}:/anvil/gitdir")); - assert!(RECIPE.contains("GIT_DIR=/anvil/gitdir/$rel")); + // Redirected through the checkout's own .git file, never through + // GIT_DIR/GIT_WORK_TREE: those are ambient, so every process in the + // container would inherit them and any git run outside the workspace + // -- `git init` in a test's scratch directory, most of all -- would + // operate on this repository instead of its own. + assert!(RECIPE.contains("gitdir: /anvil/gitdir/$rel")); + assert!(RECIPE.contains("{{anvil_container_workdir}}/.git:ro")); + assert!(!RECIPE.contains("GIT_DIR=")); + assert!(!RECIPE.contains("GIT_WORK_TREE=")); + // The generated file is temporary and must not outlive the run. + assert!(RECIPE.contains("if ($gitFile) { Remove-Item -LiteralPath $gitFile")); // An ordinary clone must not take the extra mount. assert!(RECIPE.contains("if ($gitDirAbs -ne $gitCommonAbs) {")); // A host with a working engine but no git must not start failing here. diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 16634ecf..a1ab9a45 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -405,8 +405,18 @@ anvil-container *target: # `.git` is a file naming an absolute host path, which does not exist inside # the container. Git then resolves nothing -- not HEAD, not origin/main -- # and every check that needs history fails a long way from the cause. Mount - # the common git directory and point GIT_DIR at this worktree's entry in it; - # the `commondir` file there is relative, so it resolves under the mount. + # the common git directory, and replace the checkout's `.git` file with one + # naming that mount; the `commondir` file in the worktree's entry is + # relative, so it resolves under it. + # + # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process + # in the container inherits them, so a check that runs git anywhere other + # than the workspace gets this repository instead of the one it meant. The + # test suite is the case that proves it -- `git init` in a scratch + # directory fails with `Invalid path '/anvil'`, because git honours the + # inherited GIT_DIR over the directory it was told to create. Replacing the + # `.git` file keeps the redirection where it belongs: in the checkout, and + # found by ordinary discovery. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -414,6 +424,7 @@ anvil-container *target: # Guarded on git being present: the run path needs it only to answer this # question, and a host with a working engine but no git on PATH should keep # working rather than fail on a call it did not used to make. + $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null @@ -424,9 +435,13 @@ anvil-container *target: $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") + $engineGitFile = (just _anvil-container-path $gitFile).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") - $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } } @@ -540,6 +555,7 @@ anvil-container *target: exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + if ($gitFile) { Remove-Item -LiteralPath $gitFile -Force -ErrorAction SilentlyContinue } } # Report the engine, the exec image, and whether it is present and current. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 7c2a0cbe..b2f2f00f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3908,8 +3908,18 @@ anvil-container *target: # `.git` is a file naming an absolute host path, which does not exist inside # the container. Git then resolves nothing -- not HEAD, not origin/main -- # and every check that needs history fails a long way from the cause. Mount - # the common git directory and point GIT_DIR at this worktree's entry in it; - # the `commondir` file there is relative, so it resolves under the mount. + # the common git directory, and replace the checkout's `.git` file with one + # naming that mount; the `commondir` file in the worktree's entry is + # relative, so it resolves under it. + # + # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process + # in the container inherits them, so a check that runs git anywhere other + # than the workspace gets this repository instead of the one it meant. The + # test suite is the case that proves it -- `git init` in a scratch + # directory fails with `Invalid path '/anvil'`, because git honours the + # inherited GIT_DIR over the directory it was told to create. Replacing the + # `.git` file keeps the redirection where it belongs: in the checkout, and + # found by ordinary discovery. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -3917,6 +3927,7 @@ anvil-container *target: # Guarded on git being present: the run path needs it only to answer this # question, and a host with a working engine but no git on PATH should keep # working rather than fail on a call it did not used to make. + $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null @@ -3927,9 +3938,13 @@ anvil-container *target: $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") + $engineGitFile = (just _anvil-container-path $gitFile).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") - $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } } @@ -4043,6 +4058,7 @@ anvil-container *target: exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + if ($gitFile) { Remove-Item -LiteralPath $gitFile -Force -ErrorAction SilentlyContinue } } # Report the engine, the exec image, and whether it is present and current. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 9aecfe2b..f7765cf1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3829,8 +3829,18 @@ anvil-container *target: # `.git` is a file naming an absolute host path, which does not exist inside # the container. Git then resolves nothing -- not HEAD, not origin/main -- # and every check that needs history fails a long way from the cause. Mount - # the common git directory and point GIT_DIR at this worktree's entry in it; - # the `commondir` file there is relative, so it resolves under the mount. + # the common git directory, and replace the checkout's `.git` file with one + # naming that mount; the `commondir` file in the worktree's entry is + # relative, so it resolves under it. + # + # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process + # in the container inherits them, so a check that runs git anywhere other + # than the workspace gets this repository instead of the one it meant. The + # test suite is the case that proves it -- `git init` in a scratch + # directory fails with `Invalid path '/anvil'`, because git honours the + # inherited GIT_DIR over the directory it was told to create. Replacing the + # `.git` file keeps the redirection where it belongs: in the checkout, and + # found by ordinary discovery. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -3838,6 +3848,7 @@ anvil-container *target: # Guarded on git being present: the run path needs it only to answer this # question, and a host with a working engine but no git on PATH should keep # working rather than fail on a call it did not used to make. + $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null @@ -3848,9 +3859,13 @@ anvil-container *target: $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") + $engineGitFile = (just _anvil-container-path $gitFile).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") - $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } } @@ -3964,6 +3979,7 @@ anvil-container *target: exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + if ($gitFile) { Remove-Item -LiteralPath $gitFile -Force -ErrorAction SilentlyContinue } } # Report the engine, the exec image, and whether it is present and current. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index e4f0318f..10a8f266 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2648,8 +2648,18 @@ anvil-container *target: # `.git` is a file naming an absolute host path, which does not exist inside # the container. Git then resolves nothing -- not HEAD, not origin/main -- # and every check that needs history fails a long way from the cause. Mount - # the common git directory and point GIT_DIR at this worktree's entry in it; - # the `commondir` file there is relative, so it resolves under the mount. + # the common git directory, and replace the checkout's `.git` file with one + # naming that mount; the `commondir` file in the worktree's entry is + # relative, so it resolves under it. + # + # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process + # in the container inherits them, so a check that runs git anywhere other + # than the workspace gets this repository instead of the one it meant. The + # test suite is the case that proves it -- `git init` in a scratch + # directory fails with `Invalid path '/anvil'`, because git honours the + # inherited GIT_DIR over the directory it was told to create. Replacing the + # `.git` file keeps the redirection where it belongs: in the checkout, and + # found by ordinary discovery. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -2657,6 +2667,7 @@ anvil-container *target: # Guarded on git being present: the run path needs it only to answer this # question, and a host with a working engine but no git on PATH should keep # working rather than fail on a call it did not used to make. + $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null @@ -2667,9 +2678,13 @@ anvil-container *target: $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") + $engineGitFile = (just _anvil-container-path $gitFile).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") - $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } } @@ -2783,6 +2798,7 @@ anvil-container *target: exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + if ($gitFile) { Remove-Item -LiteralPath $gitFile -Force -ErrorAction SilentlyContinue } } # Report the engine, the exec image, and whether it is present and current. diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 16634ecf..a1ab9a45 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -405,8 +405,18 @@ anvil-container *target: # `.git` is a file naming an absolute host path, which does not exist inside # the container. Git then resolves nothing -- not HEAD, not origin/main -- # and every check that needs history fails a long way from the cause. Mount - # the common git directory and point GIT_DIR at this worktree's entry in it; - # the `commondir` file there is relative, so it resolves under the mount. + # the common git directory, and replace the checkout's `.git` file with one + # naming that mount; the `commondir` file in the worktree's entry is + # relative, so it resolves under it. + # + # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process + # in the container inherits them, so a check that runs git anywhere other + # than the workspace gets this repository instead of the one it meant. The + # test suite is the case that proves it -- `git init` in a scratch + # directory fails with `Invalid path '/anvil'`, because git honours the + # inherited GIT_DIR over the directory it was told to create. Replacing the + # `.git` file keeps the redirection where it belongs: in the checkout, and + # found by ordinary discovery. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -414,6 +424,7 @@ anvil-container *target: # Guarded on git being present: the run path needs it only to answer this # question, and a host with a working engine but no git on PATH should keep # working rather than fail on a call it did not used to make. + $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null @@ -424,9 +435,13 @@ anvil-container *target: $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") + $engineGitFile = (just _anvil-container-path $gitFile).Trim() + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-e', "GIT_DIR=/anvil/gitdir/$rel") - $runArgs += @('-e', 'GIT_WORK_TREE={{anvil_container_workdir}}') + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } } @@ -540,6 +555,7 @@ anvil-container *target: exit $LASTEXITCODE } finally { foreach ($name in $hookEnv) { Remove-Item -LiteralPath "Env:$name" -ErrorAction SilentlyContinue } + if ($gitFile) { Remove-Item -LiteralPath $gitFile -Force -ErrorAction SilentlyContinue } } # Report the engine, the exec image, and whether it is present and current. From 695df9ef0d673989330d15ca309e6716c2e39d3e Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 16:33:34 +0200 Subject: [PATCH 25/81] perf(anvil): stop unrelated edits from rebuilding the exec image Two ways the image was rebuilt, or its cache thrown away, for changes that cannot affect what it contains. The tag hashed every `*.just` file under `justfiles/anvil/`, so editing any check, group or tier renamed the image and forced a full catalog reinstall -- minutes, for a file the image never runs. Those recipes execute from the bind mount, not from the image; only the catalog decides what is installed. Every one of the 28 install recipes is defined in `tools.just`, against a pin in `versions.just`, and `anvil-setup` installs all of it, so a tool cannot be added, removed or repinned without one of those two files changing. They are now the only recipe inputs to the digest. `target/` rode in on the repository bind mount, shared with the host. The two platforms write incompatible artifacts to the same paths, so every switch between a native and a containerized run recompiled the workspace, and the hottest write path in a build crossed the host boundary. It moves to a volume of its own via `CARGO_TARGET_DIR`, which `anvil-container-down` clears. Report outputs are unaffected: `target/coverage/`, `target/anvil/comments/` and `target/spelling.dic` are literal relative paths, so they still land in the workspace where CI collects them. The mount point is created world-writable, because a volume takes its ownership from the image and a run maps the caller's uid. Measured on this repository: editing a check leaves the tag unchanged, while editing `tools.just` or `versions.just` changes it. --- .anvil.lock | 8 +- .anvil/container/Dockerfile | 37 ++++--- .anvil/container/Dockerfile.dockerignore | 7 +- crates/cargo-anvil/docs/design/containers.md | 74 ++++++------- .../src/anvil/artifacts/container.rs | 51 ++++++--- .../templates/container/Dockerfile | 37 ++++--- .../container/Dockerfile.dockerignore | 7 +- .../templates/justfiles/anvil/container.just | 57 +++++----- .../snapshots/snapshots__ado_backend.snap | 101 ++++++++++-------- .../snapshots/snapshots__github_backend.snap | 101 ++++++++++-------- .../snapshots/snapshots__local_only.snap | 101 ++++++++++-------- 11 files changed, 320 insertions(+), 261 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 4fc6a11a..049179f9 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,15 +1,15 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:d60fe5206596db4e5193858e6814f928bf2e27e7a8b9d65eb123fb2caae28bf9" +catalog_checksum = "sha256:9c12b2a5ee6573ce84ed0cfa3d39fd2c345640980d39863142aa04376eb0076e" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:9d2502d16ae08315e772844c04bcd99e8e22c9e4f2fdbc48187dc542bfbd3ebe" +checksum = "sha256:317585ed867e8e9d371baa547c9adcd7273a7008c7e01bff205112a51fba9968" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:834566e919c693e179675eb96995fa7248cf5ead0c2e7aec2fd237762965cd7f" +checksum = "sha256:167211852f43c2a0d2c18c4de10bacb186a67cc6408bc61b66a3571f444d7d3d" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:23d3cfad3f6ebbfbe78404fa72f4762fec7372c7deff2fb2a40b6d5bc934084b" +checksum = "sha256:ad2d05dfa5c0c86ecaf9ad06f108fc1d093c376839e68f31178ad2aae963377c" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 69e6177e..5561fff9 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -16,20 +16,9 @@ # that claims to name fixed content. # # The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint -# rather than a preference. `anvil-setup binstall` installs the catalog as -# prebuilt binaries, which is what CI does; those binaries are linked against -# the glibc of the runner they were built on. glibc is backward but not -# forward compatible, so a base older than the runner cannot run them: on -# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version -# GLIBC_2.39 not found`, because its published binary is built on 24.04 -# (glibc 2.39). -# -# Matching the runner exactly -- rather than merely picking something newer -- -# is what keeps the container predictive: a tool that cannot run here cannot -# run in CI either, and a newer base would let the container pass where CI -# fails. When `ubuntu-latest` moves to a newer release, this pin moves with -# it, and the tag changes with the file. +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -103,11 +92,12 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is -# also exactly what the tag hashes, so the build context cannot differ from -# the image's identity: an edit that changes what this layer produces always -# renames the tag. The synthetic Justfile below imports only the anvil tree, -# avoiding repository-specific imports that may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, but +# only `tools.just` and `versions.just` decide what this layer installs, and +# only those are hashed into the tag -- so editing a check does not rebuild the +# image, and the check runs from the bind mount anyway. The synthetic Justfile +# below imports only the anvil tree, avoiding repository-specific imports that +# may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -127,6 +117,15 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# The container's own build directory. A named volume is mounted here at run +# time so host and container builds do not invalidate each other's fingerprints, +# and so the hottest write path in a build does not cross the host boundary. +# +# A volume inherits its mount point's ownership from the image, so this has to +# be world-writable: a run maps the caller's uid on Linux, and that user would +# otherwise be unable to write to a root-owned volume. +RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil + # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index e1a1e94e..e4382bbf 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -12,9 +12,10 @@ # worktree (and every stale `target/`) to the daemon. # # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` -# so that it matches what the tag hashes. A file that is copied but not hashed -# can change what a build produces while naming a tag that already resolves -- -# so the changed file is never built, because the existing image is reused. +# so that a cold build does not stream unrelated trees to the daemon. The +# recipes are copied to drive `just anvil-setup`, which needs the whole tree to +# parse; only the tool catalog within it decides what gets installed, and only +# that is hashed. * !justfiles justfiles/* diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 05c33a5d..29a8cc49 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -105,9 +105,10 @@ The image installs its tools by running `just anvil-setup`, the same recipe the generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the image (§4.1). -`Dockerfile.dockerignore` scopes the build context to `justfiles/` and `rust-toolchain.toml`, denying everything else. -BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the repository neither needs to -own a root ignore file nor can have one silently override this. +`Dockerfile.dockerignore` scopes the build context to `justfiles/anvil/` and `rust-toolchain.toml`, denying everything +else. The whole recipe tree is copied because `just` has to parse it to run `anvil-setup`, while only the tool catalog +within it is hashed (§4). BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the +repository neither needs to own a root ignore file nor can have one silently override this. ## 4. Image identity @@ -122,11 +123,14 @@ define the image. The name derives from the repository directory (§5.1). | `.anvil/container/Dockerfile.dockerignore` | always | | `rust-toolchain.toml` | always | | `.anvil/container/hooks.ps1` | when the file exists | -| `justfiles/anvil/**/*.just` | always, recursively, except `container.just` | +| `justfiles/anvil/tools.just` | always | +| `justfiles/anvil/versions.just` | always | -The recipe tree is an input because `just anvil-setup` decides what the image installs (§3), and its dependency chain -reaches the tier, group, check, and tool recipes. `container.just` is excluded because hashing the driver would make -the tag depend on the tag. A declared input that does not exist is a hard error, not an omission from the digest. +`tools.just` and `versions.just` are the tool catalog: `just anvil-setup` installs all of it, every install recipe is +defined in `tools.just`, and every pin lives in `versions.just`, so a tool cannot be added, removed or repinned without +one of the two changing. The tier, group and check recipes are **not** inputs — they only route into the catalog, and +they execute from the bind mount rather than from the image, so editing a check takes effect on the next run without a +rebuild. A declared input that does not exist is a hard error, not an omission from the digest. The hook file's **content** is an input, since it determines what the build installs. Its **output** is deliberately excluded: a credential must never influence a tag. @@ -159,13 +163,9 @@ Two properties sit outside the digest. The base image is not resolved during has digest-pinned; a floating tag could otherwise change beneath a tag that claims to name fixed content. The platform is pinned to `linux/amd64` on build and run, so hosts of differing architecture cannot compute one tag for two images. -The base must also track the Linux runner the generated workflows use, currently `ubuntu-latest` (24.04). This is a -correctness constraint rather than a preference: the image installs the catalog with `binstall`, as CI does, and those -prebuilt binaries are linked against the runner's glibc. glibc is backward but not forward compatible, so an older base -yields tools that install and cannot execute — on Debian bookworm (2.36) `cargo-aprz` aborts in the loader with -`GLIBC_2.39 not found`. Matching the runner rather than merely exceeding it is what keeps the container predictive: a -newer base would let a run pass here and fail in CI. A catalog that needs an older baseline must also install from -source rather than with `binstall`. +The base tracks the Linux runner the generated workflows use, `ubuntu-latest` (currently 24.04). The catalog is +installed with `binstall`, and those prebuilt binaries require that runner's glibc, which is backward but not forward +compatible. A catalog on an older base installs from source instead. ## 5. Execution model @@ -176,17 +176,18 @@ removed on exit (`--rm`). | Mount | Target | Purpose | | --- | --- | --- | -| repository root (bind) | `/workspace` | The worktree under test, including `target/`. | +| repository root (bind) | `/workspace` | The worktree under test. | | common git directory (bind, linked worktrees only) | `/anvil/gitdir` | Git history, when the checkout does not carry it. | | `anvil--cargo-registry` (volume) | `/usr/local/cargo/registry` | Downloaded crate sources. | | `anvil--cargo-git` (volume) | `/usr/local/cargo/git` | Git checkouts of git dependencies. | +| `anvil--target` (volume) | `/anvil/target` | The build directory, as `CARGO_TARGET_DIR`. | A linked worktree (`git worktree add`) keeps its git directory outside the checkout and stores an absolute host path -in `.git`, which does not exist inside the container. Left alone, git resolves nothing — not `HEAD`, not `origin/main` -— and every check that needs history fails somewhere far from the cause. anvil detects this by comparing -`git rev-parse --git-dir` against `--git-common-dir`, mounts the common directory, and sets `GIT_DIR` and -`GIT_WORK_TREE` accordingly. An ordinary clone carries its git directory inside the bind mount and takes none of this. -No flag or variable selects the behaviour. +in `.git`, which does not exist inside the container. anvil detects this by comparing `git rev-parse --git-dir` against +`--git-common-dir`, mounts the common directory, and bind-mounts a generated `.git` file naming that mount over the +checkout's own, so git resolves it by ordinary discovery. The redirection is confined to the checkout: a git command +run elsewhere in the container, such as `git init` in a scratch directory, is unaffected. An ordinary clone carries its +git directory inside the bind mount and takes none of this. No flag or variable selects the behaviour. Only cargo's content-addressed download caches are volumes, so the write-heavy download path never crosses the host boundary and the host's own toolchain is untouched. `$CARGO_HOME` and `$RUSTUP_HOME` themselves are **not** mounted: @@ -195,7 +196,12 @@ volume is first created. Mounting them would pin the first image's binaries over would change the tag, build a new image, and still run the old tools — defeating the identity guarantee in §4. Tools and toolchains therefore always come from the image layer the tag names. -`target/` stays on the bind mount, remaining visible from the host and shared between native and containerized runs. +`target/` is a volume rather than part of the bind mount. Host and containerized runs write incompatible artifacts to +the same paths, so sharing it would make every switch between them recompile the workspace; keeping the build +directory off the host filesystem also matters most for the write-heaviest step of a build. Recipes that emit reports +— `target/coverage/`, `target/anvil/comments/`, `target/spelling.dic` — use literal relative paths, so those still +land in the workspace where CI collects them. `anvil-container-down` removes the volume. + The caller's working directory is mapped to its in-container equivalent, so relative paths resolve when `anvil-container` is invoked from a subdirectory. @@ -215,19 +221,14 @@ otherwise the engine leaves it as `/`, and anything falling back to `$HOME` writ ### 5.3 Environment -The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and forwards `GITHUB_TOKEN` by name. `anvil-aprz` runs in the `pr-fast` -group and queries the GitHub advisory API, which allows 60 requests an hour unauthenticated — fewer than a full tier -needs. Unauthenticated is not a degraded-but-working mode: `cargo aprz deps` sleeps until the quota resets rather than -failing, and offers no way to opt out, so a containerized tier blocks for up to an hour. The token is what makes the -check terminate, not what makes it fast. +The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and forwards `GITHUB_TOKEN` by name, resolved the way the recipe resolves +it natively: the environment first, then the gh CLI's stored token. `anvil-aprz` runs in the `pr-fast` group and +queries the GitHub advisory API, which allows 60 requests an hour unauthenticated and then sleeps until the quota +resets, so a tier needs the token to terminate rather than merely to run quickly. -The driver resolves it exactly as the recipe does natively — the environment first, then the gh CLI's stored token -(`gh auth token`, which is non-interactive) — so a containerized run authenticates for the same developers a native run -does. Deriving it rather than only forwarding an exported value is deliberate: both paths have identical exposure once -inside, since a forwarded token is readable by every recipe in the container either way, including third-party build -scripts and proc macros. Refusing to derive it buys no boundary and only makes behaviour depend on how the developer -happened to sign in — which is the common case, because the recipe itself recommends `gh auth login`. A derived value -is set on the driver process, passed by name, and unset again, so it never reaches the host's command line. +A resolved token is set on the driver process, passed by name, and unset after the run, so it never reaches a host +command line. Inside the container it is readable by everything the run executes, including build scripts and proc +macros. Everything else a run needs comes from the hook (§7). @@ -443,9 +444,10 @@ Editing the Dockerfile in one repository is supported and the drift flow preserv its own version against a file it can see has diverged. A change that belongs everywhere is better made in a catalog. **The Dockerfile and its ignore file must be replaced together.** A replacement that `COPY`s anything beyond -`justfiles/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the added -paths never reach the build context and the build fails on a missing file. Recipes need no such care: `justfiles/` is -admitted as a directory and hashed recursively, so a new recipe subdirectory is copied and covered automatically. +`justfiles/anvil/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the +added paths never reach the build context and the build fails on a missing file. A replacement that installs tools +from somewhere other than `tools.just` must also add those sources to the digest, or a change to them will name a tag +that already resolves. `justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces: a non-recipe file there would be copied into the image without being part of its identity, so editing it would change the image's diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 69ce0def..869c654e 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -47,10 +47,8 @@ pub fn recipe() -> Artifact { /// runner, which installs the pinned toolchain and the generated tool catalog /// by running `just anvil-setup`. /// -/// The base must not be older than that runner. The catalog is installed as -/// prebuilt binaries, and glibc is backward but not forward compatible, so an -/// older base cannot run them; matching the runner also keeps the container -/// predictive of CI rather than more permissive than it. +/// The catalog is installed as prebuilt binaries, which require that runner's +/// glibc. A catalog on an older base installs from source instead. /// /// A downstream catalog that needs a different base OS or toolchain source /// replaces the body wholesale: @@ -311,13 +309,6 @@ mod tests { assert!(RECIPE.contains(r#"replace(anvil_container_engine, "'", "''")"#)); } - #[test] - fn recipe_excludes_itself_from_the_image_identity() { - // Hashing the driver would make the tag depend on the tag. - assert!(RECIPE.contains("justfiles/anvil/container.just")); - assert!(RECIPE.contains("-cne 'justfiles/anvil/container.just'")); - } - #[test] fn hook_file_is_an_image_input_but_hook_output_is_not() { // A changed hook must rename the tag; a minted credential must not. @@ -353,6 +344,23 @@ mod tests { assert!(!RECIPE.contains(":/usr/local/rustup")); } + #[test] + fn the_build_directory_is_not_shared_with_the_host() { + // Host and container write incompatible artifacts to the same paths + // under `target/`, so sharing it through the bind mount makes every + // switch between a native and a containerized run recompile the + // workspace. + assert!(RECIPE.contains("-target:/anvil/target")); + assert!(RECIPE.contains("'CARGO_TARGET_DIR=/anvil/target'")); + // A volume takes its ownership from the mount point in the image, and + // a run maps the caller's uid, so the directory has to be writable by + // a user the image has never seen. + assert!(DOCKERFILE.contains("mkdir -p /anvil/target && chmod -R a+rwX /anvil")); + // Teardown has to reach it, or the cache outlives the only recipe + // that can clear it. + assert!(RECIPE.contains("{{anvil_container_name}}-target'")); + } + #[test] fn wsl_calls_bypass_the_login_shell() { // `wsl.exe -- ` re-parses the command line through the default @@ -415,10 +423,23 @@ mod tests { } #[test] - fn the_build_context_matches_the_hashed_inputs() { - // A file that is copied but not hashed can change what a build - // produces while naming a tag that already resolves, so the change is - // never built. + fn only_the_tool_catalog_defines_the_image() { + // `just anvil-setup` installs the whole catalog, and every install + // recipe is defined in tools.just against a pin in versions.just, so a + // tool cannot be added, removed or repinned without one of the two + // changing. Hashing the tier/group/check recipes as well would rebuild + // the image for an edit that cannot change what it contains -- and + // those recipes run from the bind mount, not from the image. + assert!(RECIPE.contains("$inputs += 'justfiles/anvil/tools.just'")); + assert!(RECIPE.contains("$inputs += 'justfiles/anvil/versions.just'")); + assert!(!RECIPE.contains("-Recurse -File -Filter '*.just'")); + } + + #[test] + fn the_build_context_stays_scoped_to_the_recipe_tree() { + // The whole tree is copied because `just` must parse it, but nothing + // outside it is: an unscoped context streams every stale `target/` to + // the daemon on each build. assert!(DOCKERIGNORE.contains("justfiles/*\n!justfiles/anvil\n")); assert!(!DOCKERIGNORE.contains("!justfiles\n!rust-toolchain.toml")); } diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/container/Dockerfile index 69e6177e..5561fff9 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/container/Dockerfile @@ -16,20 +16,9 @@ # that claims to name fixed content. # # The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint -# rather than a preference. `anvil-setup binstall` installs the catalog as -# prebuilt binaries, which is what CI does; those binaries are linked against -# the glibc of the runner they were built on. glibc is backward but not -# forward compatible, so a base older than the runner cannot run them: on -# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version -# GLIBC_2.39 not found`, because its published binary is built on 24.04 -# (glibc 2.39). -# -# Matching the runner exactly -- rather than merely picking something newer -- -# is what keeps the container predictive: a tool that cannot run here cannot -# run in CI either, and a newer base would let the container pass where CI -# fails. When `ubuntu-latest` moves to a newer release, this pin moves with -# it, and the tag changes with the file. +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -103,11 +92,12 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is -# also exactly what the tag hashes, so the build context cannot differ from -# the image's identity: an edit that changes what this layer produces always -# renames the tag. The synthetic Justfile below imports only the anvil tree, -# avoiding repository-specific imports that may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, but +# only `tools.just` and `versions.just` decide what this layer installs, and +# only those are hashed into the tag -- so editing a check does not rebuild the +# image, and the check runs from the bind mount anyway. The synthetic Justfile +# below imports only the anvil tree, avoiding repository-specific imports that +# may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -127,6 +117,15 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# The container's own build directory. A named volume is mounted here at run +# time so host and container builds do not invalidate each other's fingerprints, +# and so the hottest write path in a build does not cross the host boundary. +# +# A volume inherits its mount point's ownership from the image, so this has to +# be world-writable: a run maps the caller's uid on Linux, and that user would +# otherwise be unable to write to a root-owned volume. +RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil + # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 diff --git a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore index e1a1e94e..e4382bbf 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/container/Dockerfile.dockerignore @@ -12,9 +12,10 @@ # worktree (and every stale `target/`) to the daemon. # # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` -# so that it matches what the tag hashes. A file that is copied but not hashed -# can change what a build produces while naming a tag that already resolves -- -# so the changed file is never built, because the existing image is reused. +# so that a cold build does not stream unrelated trees to the daemon. The +# recipes are copied to drive `just anvil-setup`, which needs the whole tree to +# parse; only the tool catalog within it decides what gets installed, and only +# that is hashed. * !justfiles justfiles/* diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index a1ab9a45..164e2f3c 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -107,11 +107,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the generated -# recipe tree -- because the image installs its tools by running -# `just anvil-setup`, whose dependency chain reaches the tier, group, check and -# tool recipes. Only this driver is excluded, since hashing it would make the -# tag depend on the tag. +# ignore file, the pinned toolchain, the optional hook, and the tool catalog +# (`tools.just` and `versions.just`) that decides what `just anvil-setup` +# installs. The tier, group and check recipes are not inputs: they only route +# into the catalog, and they execute from the bind mount rather than from the +# image, so editing one takes effect on the next run without a rebuild. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -135,13 +135,14 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' - if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { - $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' - if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } - } - } + # The tool catalog, and nothing else. `just anvil-setup` installs the whole + # catalog, and every install recipe is defined in tools.just against a pin + # in versions.just -- so a tool cannot be added, removed or repinned + # without one of these two changing. The tier, group and check recipes only + # route into them, and are read from the bind mount at run time rather than + # from the image, so editing a check must not cost a rebuild. + $inputs += 'justfiles/anvil/tools.just' + $inputs += 'justfiles/anvil/versions.just' # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -403,20 +404,16 @@ anvil-container *target: # A linked worktree keeps its real git directory outside the checkout: its # `.git` is a file naming an absolute host path, which does not exist inside - # the container. Git then resolves nothing -- not HEAD, not origin/main -- - # and every check that needs history fails a long way from the cause. Mount - # the common git directory, and replace the checkout's `.git` file with one + # the container, so git resolves neither HEAD nor origin/main. Mount the + # common git directory and replace the checkout's `.git` file with one # naming that mount; the `commondir` file in the worktree's entry is # relative, so it resolves under it. # - # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process - # in the container inherits them, so a check that runs git anywhere other - # than the workspace gets this repository instead of the one it meant. The - # test suite is the case that proves it -- `git init` in a scratch - # directory fails with `Invalid path '/anvil'`, because git honours the - # inherited GIT_DIR over the directory it was told to create. Replacing the - # `.git` file keeps the redirection where it belongs: in the checkout, and - # found by ordinary discovery. + # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE + # so that it stays scoped to it. Those variables are ambient: every process + # in the container inherits them, and a git command run elsewhere -- `git + # init` in a test's scratch directory -- would operate on this repository + # instead of its own. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -454,6 +451,18 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') + # The build directory, in a volume of its own rather than the one in the + # bind mount. Sharing `target/` with the host would mean every switch + # between a native and a containerized run recompiles the workspace: the + # two platforms write incompatible artifacts to the same paths, and cargo's + # fingerprints miss. It also keeps the hottest write path in a build off + # the host filesystem, which is what makes this bearable on Windows. + # + # Report outputs are unaffected: recipes write `target/coverage/`, + # `target/anvil/comments/` and `target/spelling.dic` as literal relative + # paths, so those still land in the workspace where CI collects them. + $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') + $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -632,7 +641,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index b2f2f00f..083171e9 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -21,20 +21,9 @@ expression: render_tree(tmp.path()) # that claims to name fixed content. # # The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint -# rather than a preference. `anvil-setup binstall` installs the catalog as -# prebuilt binaries, which is what CI does; those binaries are linked against -# the glibc of the runner they were built on. glibc is backward but not -# forward compatible, so a base older than the runner cannot run them: on -# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version -# GLIBC_2.39 not found`, because its published binary is built on 24.04 -# (glibc 2.39). -# -# Matching the runner exactly -- rather than merely picking something newer -- -# is what keeps the container predictive: a tool that cannot run here cannot -# run in CI either, and a newer base would let the container pass where CI -# fails. When `ubuntu-latest` moves to a newer release, this pin moves with -# it, and the tag changes with the file. +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -108,11 +97,12 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is -# also exactly what the tag hashes, so the build context cannot differ from -# the image's identity: an edit that changes what this layer produces always -# renames the tag. The synthetic Justfile below imports only the anvil tree, -# avoiding repository-specific imports that may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, but +# only `tools.just` and `versions.just` decide what this layer installs, and +# only those are hashed into the tag -- so editing a check does not rebuild the +# image, and the check runs from the bind mount anyway. The synthetic Justfile +# below imports only the anvil tree, avoiding repository-specific imports that +# may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -132,6 +122,15 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# The container's own build directory. A named volume is mounted here at run +# time so host and container builds do not invalidate each other's fingerprints, +# and so the hottest write path in a build does not cross the host boundary. +# +# A volume inherits its mount point's ownership from the image, so this has to +# be world-writable: a run maps the caller's uid on Linux, and that user would +# otherwise be unable to write to a root-owned volume. +RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil + # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -154,9 +153,10 @@ CMD ["bash"] # worktree (and every stale `target/`) to the daemon. # # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` -# so that it matches what the tag hashes. A file that is copied but not hashed -# can change what a build produces while naming a tag that already resolves -- -# so the changed file is never built, because the existing image is reused. +# so that a cold build does not stream unrelated trees to the daemon. The +# recipes are copied to drive `just anvil-setup`, which needs the whole tree to +# parse; only the tool catalog within it decides what gets installed, and only +# that is hashed. * !justfiles justfiles/* @@ -3610,11 +3610,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the generated -# recipe tree -- because the image installs its tools by running -# `just anvil-setup`, whose dependency chain reaches the tier, group, check and -# tool recipes. Only this driver is excluded, since hashing it would make the -# tag depend on the tag. +# ignore file, the pinned toolchain, the optional hook, and the tool catalog +# (`tools.just` and `versions.just`) that decides what `just anvil-setup` +# installs. The tier, group and check recipes are not inputs: they only route +# into the catalog, and they execute from the bind mount rather than from the +# image, so editing one takes effect on the next run without a rebuild. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -3638,13 +3638,14 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' - if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { - $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' - if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } - } - } + # The tool catalog, and nothing else. `just anvil-setup` installs the whole + # catalog, and every install recipe is defined in tools.just against a pin + # in versions.just -- so a tool cannot be added, removed or repinned + # without one of these two changing. The tier, group and check recipes only + # route into them, and are read from the bind mount at run time rather than + # from the image, so editing a check must not cost a rebuild. + $inputs += 'justfiles/anvil/tools.just' + $inputs += 'justfiles/anvil/versions.just' # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -3906,20 +3907,16 @@ anvil-container *target: # A linked worktree keeps its real git directory outside the checkout: its # `.git` is a file naming an absolute host path, which does not exist inside - # the container. Git then resolves nothing -- not HEAD, not origin/main -- - # and every check that needs history fails a long way from the cause. Mount - # the common git directory, and replace the checkout's `.git` file with one + # the container, so git resolves neither HEAD nor origin/main. Mount the + # common git directory and replace the checkout's `.git` file with one # naming that mount; the `commondir` file in the worktree's entry is # relative, so it resolves under it. # - # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process - # in the container inherits them, so a check that runs git anywhere other - # than the workspace gets this repository instead of the one it meant. The - # test suite is the case that proves it -- `git init` in a scratch - # directory fails with `Invalid path '/anvil'`, because git honours the - # inherited GIT_DIR over the directory it was told to create. Replacing the - # `.git` file keeps the redirection where it belongs: in the checkout, and - # found by ordinary discovery. + # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE + # so that it stays scoped to it. Those variables are ambient: every process + # in the container inherits them, and a git command run elsewhere -- `git + # init` in a test's scratch directory -- would operate on this repository + # instead of its own. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -3957,6 +3954,18 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') + # The build directory, in a volume of its own rather than the one in the + # bind mount. Sharing `target/` with the host would mean every switch + # between a native and a containerized run recompiles the workspace: the + # two platforms write incompatible artifacts to the same paths, and cargo's + # fingerprints miss. It also keeps the hottest write path in a build off + # the host filesystem, which is what makes this bearable on Windows. + # + # Report outputs are unaffected: recipes write `target/coverage/`, + # `target/anvil/comments/` and `target/spelling.dic` as literal relative + # paths, so those still land in the workspace where CI collects them. + $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') + $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -4135,7 +4144,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f7765cf1..4926c451 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -21,20 +21,9 @@ expression: render_tree(tmp.path()) # that claims to name fixed content. # # The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint -# rather than a preference. `anvil-setup binstall` installs the catalog as -# prebuilt binaries, which is what CI does; those binaries are linked against -# the glibc of the runner they were built on. glibc is backward but not -# forward compatible, so a base older than the runner cannot run them: on -# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version -# GLIBC_2.39 not found`, because its published binary is built on 24.04 -# (glibc 2.39). -# -# Matching the runner exactly -- rather than merely picking something newer -- -# is what keeps the container predictive: a tool that cannot run here cannot -# run in CI either, and a newer base would let the container pass where CI -# fails. When `ubuntu-latest` moves to a newer release, this pin moves with -# it, and the tag changes with the file. +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -108,11 +97,12 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is -# also exactly what the tag hashes, so the build context cannot differ from -# the image's identity: an edit that changes what this layer produces always -# renames the tag. The synthetic Justfile below imports only the anvil tree, -# avoiding repository-specific imports that may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, but +# only `tools.just` and `versions.just` decide what this layer installs, and +# only those are hashed into the tag -- so editing a check does not rebuild the +# image, and the check runs from the bind mount anyway. The synthetic Justfile +# below imports only the anvil tree, avoiding repository-specific imports that +# may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -132,6 +122,15 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# The container's own build directory. A named volume is mounted here at run +# time so host and container builds do not invalidate each other's fingerprints, +# and so the hottest write path in a build does not cross the host boundary. +# +# A volume inherits its mount point's ownership from the image, so this has to +# be world-writable: a run maps the caller's uid on Linux, and that user would +# otherwise be unable to write to a root-owned volume. +RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil + # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -154,9 +153,10 @@ CMD ["bash"] # worktree (and every stale `target/`) to the daemon. # # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` -# so that it matches what the tag hashes. A file that is copied but not hashed -# can change what a build produces while naming a tag that already resolves -- -# so the changed file is never built, because the existing image is reused. +# so that a cold build does not stream unrelated trees to the daemon. The +# recipes are copied to drive `just anvil-setup`, which needs the whole tree to +# parse; only the tool catalog within it decides what gets installed, and only +# that is hashed. * !justfiles justfiles/* @@ -3531,11 +3531,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the generated -# recipe tree -- because the image installs its tools by running -# `just anvil-setup`, whose dependency chain reaches the tier, group, check and -# tool recipes. Only this driver is excluded, since hashing it would make the -# tag depend on the tag. +# ignore file, the pinned toolchain, the optional hook, and the tool catalog +# (`tools.just` and `versions.just`) that decides what `just anvil-setup` +# installs. The tier, group and check recipes are not inputs: they only route +# into the catalog, and they execute from the bind mount rather than from the +# image, so editing one takes effect on the next run without a rebuild. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -3559,13 +3559,14 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' - if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { - $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' - if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } - } - } + # The tool catalog, and nothing else. `just anvil-setup` installs the whole + # catalog, and every install recipe is defined in tools.just against a pin + # in versions.just -- so a tool cannot be added, removed or repinned + # without one of these two changing. The tier, group and check recipes only + # route into them, and are read from the bind mount at run time rather than + # from the image, so editing a check must not cost a rebuild. + $inputs += 'justfiles/anvil/tools.just' + $inputs += 'justfiles/anvil/versions.just' # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -3827,20 +3828,16 @@ anvil-container *target: # A linked worktree keeps its real git directory outside the checkout: its # `.git` is a file naming an absolute host path, which does not exist inside - # the container. Git then resolves nothing -- not HEAD, not origin/main -- - # and every check that needs history fails a long way from the cause. Mount - # the common git directory, and replace the checkout's `.git` file with one + # the container, so git resolves neither HEAD nor origin/main. Mount the + # common git directory and replace the checkout's `.git` file with one # naming that mount; the `commondir` file in the worktree's entry is # relative, so it resolves under it. # - # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process - # in the container inherits them, so a check that runs git anywhere other - # than the workspace gets this repository instead of the one it meant. The - # test suite is the case that proves it -- `git init` in a scratch - # directory fails with `Invalid path '/anvil'`, because git honours the - # inherited GIT_DIR over the directory it was told to create. Replacing the - # `.git` file keeps the redirection where it belongs: in the checkout, and - # found by ordinary discovery. + # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE + # so that it stays scoped to it. Those variables are ambient: every process + # in the container inherits them, and a git command run elsewhere -- `git + # init` in a test's scratch directory -- would operate on this repository + # instead of its own. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -3878,6 +3875,18 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') + # The build directory, in a volume of its own rather than the one in the + # bind mount. Sharing `target/` with the host would mean every switch + # between a native and a containerized run recompiles the workspace: the + # two platforms write incompatible artifacts to the same paths, and cargo's + # fingerprints miss. It also keeps the hottest write path in a build off + # the host filesystem, which is what makes this bearable on Windows. + # + # Report outputs are unaffected: recipes write `target/coverage/`, + # `target/anvil/comments/` and `target/spelling.dic` as literal relative + # paths, so those still land in the workspace where CI collects them. + $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') + $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -4056,7 +4065,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 10a8f266..7422438e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -21,20 +21,9 @@ expression: render_tree(tmp.path()) # that claims to name fixed content. # # The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04), and that is a correctness constraint -# rather than a preference. `anvil-setup binstall` installs the catalog as -# prebuilt binaries, which is what CI does; those binaries are linked against -# the glibc of the runner they were built on. glibc is backward but not -# forward compatible, so a base older than the runner cannot run them: on -# Debian bookworm (glibc 2.36) `cargo-aprz` 1.0.0 aborts with `version -# GLIBC_2.39 not found`, because its published binary is built on 24.04 -# (glibc 2.39). -# -# Matching the runner exactly -- rather than merely picking something newer -- -# is what keeps the container predictive: a tool that cannot run here cannot -# run in CI either, and a newer base would let the container pass where CI -# fails. When `ubuntu-latest` moves to a newer release, this pin moves with -# it, and the tag changes with the file. +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. # # To build on a different base (a lower glibc baseline, or an internal # distribution), a downstream catalog replaces this artifact wholesale via @@ -108,11 +97,12 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. Only `justfiles/anvil/` and the toolchain pin are copied, which is -# also exactly what the tag hashes, so the build context cannot differ from -# the image's identity: an edit that changes what this layer produces always -# renames the tag. The synthetic Justfile below imports only the anvil tree, -# avoiding repository-specific imports that may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, but +# only `tools.just` and `versions.just` decide what this layer installs, and +# only those are hashed into the tag -- so editing a check does not rebuild the +# image, and the check runs from the bind mount anyway. The synthetic Justfile +# below imports only the anvil tree, avoiding repository-specific imports that +# may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -132,6 +122,15 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# The container's own build directory. A named volume is mounted here at run +# time so host and container builds do not invalidate each other's fingerprints, +# and so the hottest write path in a build does not cross the host boundary. +# +# A volume inherits its mount point's ownership from the image, so this has to +# be world-writable: a run maps the caller's uid on Linux, and that user would +# otherwise be unable to write to a root-owned volume. +RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil + # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -154,9 +153,10 @@ CMD ["bash"] # worktree (and every stale `target/`) to the daemon. # # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` -# so that it matches what the tag hashes. A file that is copied but not hashed -# can change what a build produces while naming a tag that already resolves -- -# so the changed file is never built, because the existing image is reused. +# so that a cold build does not stream unrelated trees to the daemon. The +# recipes are copied to drive `just anvil-setup`, which needs the whole tree to +# parse; only the tool catalog within it decides what gets installed, and only +# that is hashed. * !justfiles justfiles/* @@ -2350,11 +2350,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the generated -# recipe tree -- because the image installs its tools by running -# `just anvil-setup`, whose dependency chain reaches the tier, group, check and -# tool recipes. Only this driver is excluded, since hashing it would make the -# tag depend on the tag. +# ignore file, the pinned toolchain, the optional hook, and the tool catalog +# (`tools.just` and `versions.just`) that decides what `just anvil-setup` +# installs. The tier, group and check recipes are not inputs: they only route +# into the catalog, and they execute from the bind mount rather than from the +# image, so editing one takes effect on the next run without a rebuild. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -2378,13 +2378,14 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' - if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { - $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' - if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } - } - } + # The tool catalog, and nothing else. `just anvil-setup` installs the whole + # catalog, and every install recipe is defined in tools.just against a pin + # in versions.just -- so a tool cannot be added, removed or repinned + # without one of these two changing. The tier, group and check recipes only + # route into them, and are read from the bind mount at run time rather than + # from the image, so editing a check must not cost a rebuild. + $inputs += 'justfiles/anvil/tools.just' + $inputs += 'justfiles/anvil/versions.just' # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -2646,20 +2647,16 @@ anvil-container *target: # A linked worktree keeps its real git directory outside the checkout: its # `.git` is a file naming an absolute host path, which does not exist inside - # the container. Git then resolves nothing -- not HEAD, not origin/main -- - # and every check that needs history fails a long way from the cause. Mount - # the common git directory, and replace the checkout's `.git` file with one + # the container, so git resolves neither HEAD nor origin/main. Mount the + # common git directory and replace the checkout's `.git` file with one # naming that mount; the `commondir` file in the worktree's entry is # relative, so it resolves under it. # - # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process - # in the container inherits them, so a check that runs git anywhere other - # than the workspace gets this repository instead of the one it meant. The - # test suite is the case that proves it -- `git init` in a scratch - # directory fails with `Invalid path '/anvil'`, because git honours the - # inherited GIT_DIR over the directory it was told to create. Replacing the - # `.git` file keeps the redirection where it belongs: in the checkout, and - # found by ordinary discovery. + # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE + # so that it stays scoped to it. Those variables are ambient: every process + # in the container inherits them, and a git command run elsewhere -- `git + # init` in a test's scratch directory -- would operate on this repository + # instead of its own. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. @@ -2697,6 +2694,18 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') + # The build directory, in a volume of its own rather than the one in the + # bind mount. Sharing `target/` with the host would mean every switch + # between a native and a containerized run recompiles the workspace: the + # two platforms write incompatible artifacts to the same paths, and cargo's + # fingerprints miss. It also keeps the hottest write path in a build off + # the host filesystem, which is what makes this bearable on Windows. + # + # Report outputs are unaffected: recipes write `target/coverage/`, + # `target/anvil/comments/` and `target/spelling.dic` as literal relative + # paths, so those still land in the workspace where CI collects them. + $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') + $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -2875,7 +2884,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } From 60b1a6925bb3e71510980a98bad4a5f83c04f143 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 17:39:06 +0200 Subject: [PATCH 26/81] fix(anvil): keep the exec image stable across unrelated edits The generated driver shipped in 695df9e still hashed every `*.just` file: the narrowed logic reached the template but not the emitted tree, because a scratch edit was undone with `git checkout --`, which restores the last commit rather than the generated state. Anvil then preserved the stale file as a user modification. Regenerated, so the tag now watches what the commit claimed. `cargo-semver-checks` builds a baseline and the current crate, and with `CARGO_TARGET_DIR` pointing both at one directory it cannot find its own rustdoc output: error: could not find expected rustdoc output for `cargo-coverage-gate`: /anvil/target/doc/cargo_coverage_gate.json So the container's separate build directory is reverted. `target/` goes back to the bind mount, which costs a recompile when switching between native and containerized runs -- the lesser of the two, and now stated in the design rather than left to be rediscovered. Also: the container templates move to `templates/anvil/container/`, matching where the rest of the anvil-owned templates live; and `anvil-aprz`'s warning said an unauthenticated run "may fail on a full run", when what it actually does is sleep until the hourly quota resets. It now says so. The dogfood suite gains the assertions this commit exists to protect: editing a check, a tier or the driver must not rename the image, editing `tools.just` or `versions.just` must, restoring everything must return the original tag, and a run after an irrelevant edit must start rather than build. Validation, all on this repository: - `cargo test -p cargo-anvil` -- 354 passed, 0 failed. - `scripts/test-anvil-dogfood.ps1 -Engine both -Tier anvil-pr-fast` -- 48/48, docker and podman, same content tag from both. - `scripts/test-anvil-container.ps1` -- 58/58. --- .anvil.lock | 8 +- .anvil/container/Dockerfile | 9 -- crates/cargo-anvil/docs/design/containers.md | 12 +- .../src/anvil/artifacts/container.rs | 21 +-- .../{ => anvil}/container/Dockerfile | 9 -- .../container/Dockerfile.dockerignore | 0 .../justfiles/anvil/checks/aprz.just | 2 +- .../templates/justfiles/anvil/container.just | 14 +- .../snapshots/snapshots__ado_backend.snap | 25 +-- .../snapshots/snapshots__github_backend.snap | 25 +-- .../snapshots/snapshots__local_only.snap | 25 +-- justfiles/anvil/checks/aprz.just | 2 +- justfiles/anvil/container.just | 43 +++-- scripts/test-anvil-container.ps1 | 51 +++++- scripts/test-anvil-dogfood.ps1 | 152 ++++++++++++------ 15 files changed, 186 insertions(+), 212 deletions(-) rename crates/cargo-anvil/templates/{ => anvil}/container/Dockerfile (92%) rename crates/cargo-anvil/templates/{ => anvil}/container/Dockerfile.dockerignore (100%) diff --git a/.anvil.lock b/.anvil.lock index 049179f9..fb17f6d1 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:9c12b2a5ee6573ce84ed0cfa3d39fd2c345640980d39863142aa04376eb0076e" +catalog_checksum = "sha256:6a6d21c51dd3b55cc1a45a35920fa259157bd0fa4a335490909dbb821846fa25" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:317585ed867e8e9d371baa547c9adcd7273a7008c7e01bff205112a51fba9968" +checksum = "sha256:e4dd5a426662328eddbd8b3bfe2fda073e691914326ab4a491a9ab1501ddf1a9" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -69,7 +69,7 @@ checksum = "sha256:91408602dc3ee274b593e234841934c749ff03bba0ee7846ab88247c06f20 [[file]] path = "justfiles/anvil/checks/aprz.just" -checksum = "sha256:5ac69b95781215c74791766961da1594fb94d48edb9339ccc2043a57fbac23e8" +checksum = "sha256:ee1ba62b2ad3f8d6eb8505195ad5378efc6225b05b7bc6c2ffea44788a7446f4" [[file]] path = "justfiles/anvil/checks/audit.just" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:ad2d05dfa5c0c86ecaf9ad06f108fc1d093c376839e68f31178ad2aae963377c" +checksum = "sha256:f038ef8e16637695db80e85cdc9567a73af874bc4a191656aee1ccfe5c00a819" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 5561fff9..b75dadc1 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -117,15 +117,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# The container's own build directory. A named volume is mounted here at run -# time so host and container builds do not invalidate each other's fingerprints, -# and so the hottest write path in a build does not cross the host boundary. -# -# A volume inherits its mount point's ownership from the image, so this has to -# be world-writable: a run maps the caller's uid on Linux, and that user would -# otherwise be unable to write to a root-owned volume. -RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil - # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 29a8cc49..2f3f6f83 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -176,11 +176,10 @@ removed on exit (`--rm`). | Mount | Target | Purpose | | --- | --- | --- | -| repository root (bind) | `/workspace` | The worktree under test. | +| repository root (bind) | `/workspace` | The worktree under test, including `target/`. | | common git directory (bind, linked worktrees only) | `/anvil/gitdir` | Git history, when the checkout does not carry it. | | `anvil--cargo-registry` (volume) | `/usr/local/cargo/registry` | Downloaded crate sources. | | `anvil--cargo-git` (volume) | `/usr/local/cargo/git` | Git checkouts of git dependencies. | -| `anvil--target` (volume) | `/anvil/target` | The build directory, as `CARGO_TARGET_DIR`. | A linked worktree (`git worktree add`) keeps its git directory outside the checkout and stores an absolute host path in `.git`, which does not exist inside the container. anvil detects this by comparing `git rev-parse --git-dir` against @@ -196,11 +195,10 @@ volume is first created. Mounting them would pin the first image's binaries over would change the tag, build a new image, and still run the old tools — defeating the identity guarantee in §4. Tools and toolchains therefore always come from the image layer the tag names. -`target/` is a volume rather than part of the bind mount. Host and containerized runs write incompatible artifacts to -the same paths, so sharing it would make every switch between them recompile the workspace; keeping the build -directory off the host filesystem also matters most for the write-heaviest step of a build. Recipes that emit reports -— `target/coverage/`, `target/anvil/comments/`, `target/spelling.dic` — use literal relative paths, so those still -land in the workspace where CI collects them. `anvil-container-down` removes the volume. +`target/` stays on the bind mount, shared with the host and visible from it. A native run and a containerized run write +incompatible artifacts to the same paths, so switching between them recompiles the workspace. Giving the container its +own build directory through `CARGO_TARGET_DIR` avoids that but breaks `cargo-semver-checks`, which builds a baseline +and the current crate and then cannot find its rustdoc output; the recompilation is the lesser cost. The caller's working directory is mapped to its in-container equivalent, so relative paths resolve when `anvil-container` is invoked from a subdirectory. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 869c654e..9af97b32 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -21,8 +21,8 @@ use crate::catalog::Artifact; const RECIPE: &str = include_str!("../../../templates/justfiles/anvil/container.just"); -const DOCKERFILE: &str = include_str!("../../../templates/container/Dockerfile"); -const DOCKERIGNORE: &str = include_str!("../../../templates/container/Dockerfile.dockerignore"); +const DOCKERFILE: &str = include_str!("../../../templates/anvil/container/Dockerfile"); +const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dockerfile.dockerignore"); const RECIPE_PATH: &str = "justfiles/anvil/container.just"; const DOCKERFILE_PATH: &str = ".anvil/container/Dockerfile"; @@ -344,23 +344,6 @@ mod tests { assert!(!RECIPE.contains(":/usr/local/rustup")); } - #[test] - fn the_build_directory_is_not_shared_with_the_host() { - // Host and container write incompatible artifacts to the same paths - // under `target/`, so sharing it through the bind mount makes every - // switch between a native and a containerized run recompile the - // workspace. - assert!(RECIPE.contains("-target:/anvil/target")); - assert!(RECIPE.contains("'CARGO_TARGET_DIR=/anvil/target'")); - // A volume takes its ownership from the mount point in the image, and - // a run maps the caller's uid, so the directory has to be writable by - // a user the image has never seen. - assert!(DOCKERFILE.contains("mkdir -p /anvil/target && chmod -R a+rwX /anvil")); - // Teardown has to reach it, or the cache outlives the only recipe - // that can clear it. - assert!(RECIPE.contains("{{anvil_container_name}}-target'")); - } - #[test] fn wsl_calls_bypass_the_login_shell() { // `wsl.exe -- ` re-parses the command line through the default diff --git a/crates/cargo-anvil/templates/container/Dockerfile b/crates/cargo-anvil/templates/anvil/container/Dockerfile similarity index 92% rename from crates/cargo-anvil/templates/container/Dockerfile rename to crates/cargo-anvil/templates/anvil/container/Dockerfile index 5561fff9..b75dadc1 100644 --- a/crates/cargo-anvil/templates/container/Dockerfile +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile @@ -117,15 +117,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# The container's own build directory. A named volume is mounted here at run -# time so host and container builds do not invalidate each other's fingerprints, -# and so the hottest write path in a build does not cross the host boundary. -# -# A volume inherits its mount point's ownership from the image, so this has to -# be world-writable: a run maps the caller's uid on Linux, and that user would -# otherwise be unable to write to a root-owned volume. -RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil - # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 diff --git a/crates/cargo-anvil/templates/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore similarity index 100% rename from crates/cargo-anvil/templates/container/Dockerfile.dockerignore rename to crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just index 6f4042f1..7d28c6dd 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just @@ -31,7 +31,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 164e2f3c..301410d7 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -451,18 +451,6 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') - # The build directory, in a volume of its own rather than the one in the - # bind mount. Sharing `target/` with the host would mean every switch - # between a native and a containerized run recompiles the workspace: the - # two platforms write incompatible artifacts to the same paths, and cargo's - # fingerprints miss. It also keeps the hottest write path in a build off - # the host filesystem, which is what makes this bearable on Windows. - # - # Report outputs are unaffected: recipes write `target/coverage/`, - # `target/anvil/comments/` and `target/spelling.dic` as literal relative - # paths, so those still land in the workspace where CI collects them. - $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') - $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -641,7 +629,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 083171e9..f8dc76a0 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -122,15 +122,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# The container's own build directory. A named volume is mounted here at run -# time so host and container builds do not invalidate each other's fingerprints, -# and so the hottest write path in a build does not cross the host boundary. -# -# A volume inherits its mount point's ownership from the image, so this has to -# be world-writable: a run maps the caller's uid on Linux, and that user would -# otherwise be unable to write to a root-owned volume. -RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil - # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -1679,7 +1670,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } @@ -3954,18 +3945,6 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') - # The build directory, in a volume of its own rather than the one in the - # bind mount. Sharing `target/` with the host would mean every switch - # between a native and a containerized run recompiles the workspace: the - # two platforms write incompatible artifacts to the same paths, and cargo's - # fingerprints miss. It also keeps the hottest write path in a build off - # the host filesystem, which is what makes this bearable on Windows. - # - # Report outputs are unaffected: recipes write `target/coverage/`, - # `target/anvil/comments/` and `target/spelling.dic` as literal relative - # paths, so those still land in the workspace where CI collects them. - $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') - $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -4144,7 +4123,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 4926c451..7d12579b 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -122,15 +122,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# The container's own build directory. A named volume is mounted here at run -# time so host and container builds do not invalidate each other's fingerprints, -# and so the hottest write path in a build does not cross the host boundary. -# -# A volume inherits its mount point's ownership from the image, so this has to -# be world-writable: a run maps the caller's uid on Linux, and that user would -# otherwise be unable to write to a root-owned volume. -RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil - # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -1600,7 +1591,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } @@ -3875,18 +3866,6 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') - # The build directory, in a volume of its own rather than the one in the - # bind mount. Sharing `target/` with the host would mean every switch - # between a native and a containerized run recompiles the workspace: the - # two platforms write incompatible artifacts to the same paths, and cargo's - # fingerprints miss. It also keeps the hottest write path in a build off - # the host filesystem, which is what makes this bearable on Windows. - # - # Report outputs are unaffected: recipes write `target/coverage/`, - # `target/anvil/comments/` and `target/spelling.dic` as literal relative - # paths, so those still land in the workspace where CI collects them. - $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') - $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -4065,7 +4044,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 7422438e..67576394 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -122,15 +122,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" -# The container's own build directory. A named volume is mounted here at run -# time so host and container builds do not invalidate each other's fingerprints, -# and so the hottest write path in a build does not cross the host boundary. -# -# A volume inherits its mount point's ownership from the image, so this has to -# be world-writable: a run maps the caller's uid on Linux, and that user would -# otherwise be unable to write to a root-owned volume. -RUN mkdir -p /anvil/target && chmod -R a+rwX /anvil - # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 @@ -419,7 +410,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } @@ -2694,18 +2685,6 @@ anvil-container *target: # would change the tag, build a new image, and still run the old tools. $runArgs += @('-v', '{{anvil_container_name}}-cargo-registry:/usr/local/cargo/registry') $runArgs += @('-v', '{{anvil_container_name}}-cargo-git:/usr/local/cargo/git') - # The build directory, in a volume of its own rather than the one in the - # bind mount. Sharing `target/` with the host would mean every switch - # between a native and a containerized run recompiles the workspace: the - # two platforms write incompatible artifacts to the same paths, and cargo's - # fingerprints miss. It also keeps the hottest write path in a build off - # the host filesystem, which is what makes this bearable on Windows. - # - # Report outputs are unaffected: recipes write `target/coverage/`, - # `target/anvil/comments/` and `target/spelling.dic` as literal relative - # paths, so those still land in the workspace where CI collects them. - $runArgs += @('-v', '{{anvil_container_name}}-target:/anvil/target') - $runArgs += @('-e', 'CARGO_TARGET_DIR=/anvil/target') # Match the caller's uid/gid on Linux. Without this everything the run # writes under the bind mount -- target/, generated files -- lands as root # on the host, and the next native cargo build or git clean fails with @@ -2884,7 +2863,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-target', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/justfiles/anvil/checks/aprz.just b/justfiles/anvil/checks/aprz.just index 6f4042f1..7d28c6dd 100644 --- a/justfiles/anvil/checks/aprz.just +++ b/justfiles/anvil/checks/aprz.just @@ -31,7 +31,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API (60 requests/hour) and may fail on a full run.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index a1ab9a45..301410d7 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -107,11 +107,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the generated -# recipe tree -- because the image installs its tools by running -# `just anvil-setup`, whose dependency chain reaches the tier, group, check and -# tool recipes. Only this driver is excluded, since hashing it would make the -# tag depend on the tag. +# ignore file, the pinned toolchain, the optional hook, and the tool catalog +# (`tools.just` and `versions.just`) that decides what `just anvil-setup` +# installs. The tier, group and check recipes are not inputs: they only route +# into the catalog, and they execute from the bind mount rather than from the +# image, so editing one takes effect on the next run without a rebuild. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -135,13 +135,14 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' - if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { - $rel = [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' - if ($rel -cne 'justfiles/anvil/container.just') { $inputs += $rel } - } - } + # The tool catalog, and nothing else. `just anvil-setup` installs the whole + # catalog, and every install recipe is defined in tools.just against a pin + # in versions.just -- so a tool cannot be added, removed or repinned + # without one of these two changing. The tier, group and check recipes only + # route into them, and are read from the bind mount at run time rather than + # from the image, so editing a check must not cost a rebuild. + $inputs += 'justfiles/anvil/tools.just' + $inputs += 'justfiles/anvil/versions.just' # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -403,20 +404,16 @@ anvil-container *target: # A linked worktree keeps its real git directory outside the checkout: its # `.git` is a file naming an absolute host path, which does not exist inside - # the container. Git then resolves nothing -- not HEAD, not origin/main -- - # and every check that needs history fails a long way from the cause. Mount - # the common git directory, and replace the checkout's `.git` file with one + # the container, so git resolves neither HEAD nor origin/main. Mount the + # common git directory and replace the checkout's `.git` file with one # naming that mount; the `commondir` file in the worktree's entry is # relative, so it resolves under it. # - # Deliberately not GIT_DIR/GIT_WORK_TREE. Those are ambient: every process - # in the container inherits them, so a check that runs git anywhere other - # than the workspace gets this repository instead of the one it meant. The - # test suite is the case that proves it -- `git init` in a scratch - # directory fails with `Invalid path '/anvil'`, because git honours the - # inherited GIT_DIR over the directory it was told to create. Replacing the - # `.git` file keeps the redirection where it belongs: in the checkout, and - # found by ordinary discovery. + # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE + # so that it stays scoped to it. Those variables are ambient: every process + # in the container inherits them, and a git command run elsewhere -- `git + # init` in a test's scratch directory -- would operate on this repository + # instead of its own. # # An ordinary clone keeps its git directory inside the checkout, where the # bind mount already carries it, and takes none of this. diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index bc7fdbd6..9c207805 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -22,7 +22,8 @@ 2. The first run builds an image and runs the recipe inside it. 3. A second run reuses the image (the tag resolves, nothing is built), no cache volume masks the tools the image installed, and a host - GITHUB_TOKEN is forwarded while an absent one is not invented. + GITHUB_TOKEN is forwarded — from the environment, or from the gh CLI + when the environment has none. 3b. A recipe run from a linked worktree can still reach git history. 4. Changing a hashed input (the pinned toolchain) selects a new tag. 5. Reverting that input returns to the original tag. @@ -424,17 +425,53 @@ Assert-Equal 'a tool installed by the image survives the cache mounts' 0 $probe. Assert-That 'the tool resolves inside the image, not a volume' ` ($probe.StdOut -match '/usr/local/cargo/bin/cargo-binstall') "$($probe.StdOut)$($probe.StdErr)" -# anvil-aprz runs in pr-fast and is rate-limited without a token, so a host -# token has to reach the container -- but only one the host actually set. +# anvil-aprz runs in pr-fast and blocks on the rate limit without a token, so a +# host token has to reach the container. The driver resolves it the way the +# recipe does natively: the environment first, then the gh CLI. +# +# Failure details are redacted: on a developer machine the value below is a real +# credential, and a test that prints it to the terminal on failure is a leak. +function Hide-Token([string]$Text) { $Text -replace 'E2E-TOKEN:\[[^\]]+\]', 'E2E-TOKEN:[]' } + $withToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` -Environment @{ GITHUB_TOKEN = 'e2e-forwarded-token' } Assert-That 'a host GITHUB_TOKEN reaches a recipe in the container' ` - ($withToken.StdOut -match 'E2E-TOKEN:\[e2e-forwarded-token\]') "$($withToken.StdOut)$($withToken.StdErr)" - + ($withToken.StdOut -match 'E2E-TOKEN:\[e2e-forwarded-token\]') (Hide-Token "$($withToken.StdOut)$($withToken.StdErr)") + +# No environment token and no gh CLI: nothing is forwarded. gh is hidden by +# dropping its directory from PATH, which is what the driver actually probes -- +# `GH_CONFIG_DIR` does not work here, because modern gh keeps credentials in the +# OS keyring rather than in its config directory. +$pathWithoutGh = $env:PATH +$ghCommand = Get-Command gh -ErrorAction SilentlyContinue +if ($ghCommand) { + $ghDir = (Split-Path $ghCommand.Source).TrimEnd('\', '/') + $separator = if ($IsWindows) { ';' } else { ':' } + $pathWithoutGh = (($env:PATH -split $separator) | + Where-Object { $_ -and $_.TrimEnd('\', '/') -ne $ghDir }) -join $separator +} $withoutToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` - -Environment @{ GITHUB_TOKEN = '' } + -Environment @{ GITHUB_TOKEN = ''; GH_TOKEN = ''; PATH = $pathWithoutGh } Assert-That 'no token is invented when the host has none' ` - ($withoutToken.StdOut -match 'E2E-TOKEN:\[\]') "$($withoutToken.StdOut)$($withoutToken.StdErr)" + ($withoutToken.StdOut -match 'E2E-TOKEN:\[\]') (Hide-Token "$($withoutToken.StdOut)$($withoutToken.StdErr)") + +# The gh fallback itself, which is what keeps a containerized tier from blocking +# for a developer who signed in with `gh auth login` and never exported a token. +# Skipped rather than failed when the host is not signed in, since that is a +# property of the machine running the suite. +$hostGhToken = $null +if (Get-Command gh -ErrorAction SilentlyContinue) { + try { $hostGhToken = (gh auth token --hostname github.com 2>$null) } catch { $hostGhToken = $null } +} +if ($hostGhToken -and $hostGhToken.Trim()) { + $viaGh = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` + -Environment @{ GITHUB_TOKEN = '' } + Assert-That 'the gh CLI token is used when the environment has none' ` + ($viaGh.StdOut -match ('E2E-TOKEN:\[' + [regex]::Escape($hostGhToken.Trim()) + '\]')) ` + (Hide-Token "$($viaGh.StdOut)$($viaGh.StdErr)") +} else { + Write-Step 'skipping the gh-fallback check: this host has no gh credential' +} # --------------------------------------------------------- 3b. worktrees ----- diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index a87cb2a0..241a9ca9 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -6,60 +6,48 @@ Dogfood test: run this repository's own anvil checks inside the container. .DESCRIPTION - `test-anvil-container.ps1` proves the container *mechanism* against a - throwaway fixture: artifacts, tags, drift, hooks. It deliberately generates - a minimal repository, so the checks it runs there are trivial. - - This script proves the opposite half -- that the mechanism is actually - usable on a real workspace. It runs the generated recipes against - `ox-tools` itself: a multi-crate workspace with real dependencies, real - lints, and the full pinned tool catalog. A defect that only appears at that - scale (a tool that installs but cannot execute, a check that needs a file - the build context drops, a mount that hides the workspace) is invisible to - the fixture test and lands directly on a customer. - - It runs against **both** engines by default, which is not redundant on - Windows: docker is reached through the default WSL distribution, and podman - runs natively, so the two cover both invocation paths as well as both - engines. - - Ordering is by cost, so a break is reported in seconds rather than after a - full tier: + `test-anvil-container.ps1` covers the container mechanism against a + throwaway fixture: artifacts, tags, drift, hooks. This script covers the + other half -- that the mechanism works on a real workspace. It runs the + generated recipes against `ox-tools` itself: a multi-crate workspace with + real dependencies, real lints, and the full pinned tool catalog. + + It runs against both engines by default. On Windows that also covers both + invocation paths, since docker is reached through the default WSL + distribution and podman runs natively. + + Steps are ordered by cost, so a break is reported in seconds rather than + after a full tier: 1. Preconditions -- the generated tree is current, and the container - artifacts are the ones the engine emits. + artifacts are present. 2. The image builds from the repository's own Dockerfile. - 3. Every pinned tool in the catalog *executes* inside the image. - 4. `anvil-aprz` runs -- the check whose prebuilt binary first exposed an - ABI mismatch against the base image. - 5. The requested tier runs to completion inside the image. - 6. A second run reuses the image rather than rebuilding it. - - Step 3 is the one worth keeping cheap. `anvil-setup` installing a tool only - proves it downloaded; the catalog is installed as prebuilt binaries, so a - base image older than the runner those binaries were built on yields tools - that are present and unrunnable. Executing all of them costs seconds and - catches the whole class at once, instead of one tool at a time as tiers - happen to reach them. + 3. Every pinned tool in the catalog executes inside the image. + 4. `anvil-aprz` runs, exercising a prebuilt binary and the advisory API. + 5. Only the tool catalog renames the image: editing a check, a tier or + the driver must not trigger a rebuild, while editing `tools.just` or + `versions.just` must. + 6. The requested tier runs to completion inside the image. + 7. A second run reuses the image rather than rebuilding it. + + Step 3 is cheap and broad: `anvil-setup` proves a tool downloaded, while + executing it proves the image can run it. Twenty tools cost seconds here + and would otherwise surface one at a time as tiers reach them. .PARAMETER Engine - Which engine(s) to test. 'both' (default) runs the whole suite against - docker and then podman, reporting them separately. + Which engine(s) to test. 'both' (default) runs the suite against docker and + then podman, reporting them separately. .PARAMETER Tier - Recipe(s) to run for step 5. Defaults to `anvil-pr`, the full PR tier. - `anvil-pr` includes `anvil-pr-slow`, which includes mutants and runtime - analysis, so it is measured in tens of minutes; pass `anvil-pr-fast` for a - quicker pass over the same plumbing. + Recipe(s) to run for step 5. Defaults to `anvil-pr`, the full PR tier, + which includes mutants and runtime analysis and is measured in tens of + minutes. `anvil-pr-fast` covers the same plumbing more quickly. .PARAMETER SkipTier - Stop after step 4. The cheap steps cover the container contract; the tier - is what makes a full run long. + Stop after step 4. The cheap steps cover the container contract. .PARAMETER KeepImages - Leave built images and cache volumes in place. On by default in spirit -- - this repository's image is expensive to build, so the script never removes - it unless -Clean is passed. + Leave built images and cache volumes in place. .PARAMETER Clean Remove this repository's anvil images and cache volumes before starting, @@ -317,11 +305,10 @@ function Invoke-Suite([string]$EngineName) { Write-Section "$EngineName : the catalog executes inside the image" - # `anvil-setup` proves a tool installed, not that it runs. The catalog is - # installed as prebuilt binaries linked against the glibc of the runner - # they were built on, so a base image older than that runner produces a - # tool that is present and unrunnable. Executing each one is the cheapest - # check that covers the whole class. + # `anvil-setup` proves a tool downloaded; executing it proves the image can + # run it. The catalog is installed as prebuilt binaries, so a base older + # than the runner they were built on yields tools that are present and + # unrunnable. # # This runs the engine directly rather than through `anvil-container`, # which dispatches `just ` and so cannot invoke a bare binary. The @@ -348,8 +335,8 @@ function Invoke-Suite([string]$EngineName) { Write-Section "$EngineName : checks" - # The check whose prebuilt binary first exposed the base-image ABI - # mismatch. Kept as its own step so a regression names itself. + # The check whose prebuilt binary exercises both the loader and the + # advisory API. Kept as its own step so a regression names itself. $aprz = Invoke-Just -Arguments @('anvil-container', 'anvil-aprz') -AllowFailure Assert-Equal 'anvil-aprz runs inside the image' 0 $aprz.ExitCode if ($aprz.ExitCode -ne 0) { Write-Tail "$($aprz.StdOut)`n$($aprz.StdErr)" } @@ -370,6 +357,71 @@ function Invoke-Suite([string]$EngineName) { (-not ("$($reuse.StdOut)`n$($reuse.StdErr)" -match 'building image|Step 1/|FROM ')) ` 'a rebuild happened when the tag should have resolved' + Write-Section "$EngineName : only the tool catalog renames the image" + + # The recipes execute from the bind mount, so an edit to one takes effect on + # the next run. Rebuilding the image for it would cost a full catalog + # reinstall for a file the image never runs -- the difference between a + # feature that speeds work up and one that gets abandoned. + # + # Edits are made against a byte copy and restored from it. Never + # `git checkout --` on a generated file: that restores the last *commit*, + # not the generated state, and anvil then preserves the stale file as a + # user modification. + $baseline = Get-ImageReference + Assert-That 'a baseline tag is available' ([bool]$baseline) + + $cases = @( + @{ File = 'justfiles/anvil/checks/clippy.just'; Renames = $false; Why = 'a check only routes into the catalog' } + @{ File = 'justfiles/anvil/container.just'; Renames = $false; Why = 'the driver computes the tag; it cannot define it' } + @{ File = 'justfiles/anvil/tiers.just'; Renames = $false; Why = 'a tier only routes into the catalog' } + @{ File = 'justfiles/anvil/versions.just'; Renames = $true; Why = 'a pin decides which build is installed' } + @{ File = 'justfiles/anvil/tools.just'; Renames = $true; Why = 'the install recipes decide what is installed' } + ) + + foreach ($case in $cases) { + $path = Join-Path $RepoRoot $case.File + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Write-Skipped "$($case.File) is present" 'not emitted by this catalog' + continue + } + $backup = [System.IO.Path]::GetTempFileName() + Copy-Item -LiteralPath $path -Destination $backup -Force + try { + Add-Content -LiteralPath $path -Value "`n# dogfood scratch" + $edited = Get-ImageReference + $name = "editing $($case.File) $(if ($case.Renames) { 'renames' } else { 'does not rename' }) the image" + if ($case.Renames) { + Assert-That $name ($edited -ne $baseline) "$($case.Why); tag stayed $edited" + } else { + Assert-That $name ($edited -eq $baseline) "$($case.Why); tag moved $baseline -> $edited" + } + } finally { + Copy-Item -LiteralPath $backup -Destination $path -Force + Remove-Item -LiteralPath $backup -Force -ErrorAction SilentlyContinue + } + } + + Assert-Equal 'restoring every file restores the original tag' $baseline (Get-ImageReference) + + # The assertions above compare references. This one proves the consequence a + # user actually feels: after an irrelevant edit, a run starts rather than + # builds. + $scratchCheck = Join-Path $RepoRoot 'justfiles/anvil/checks/clippy.just' + $scratchBackup = [System.IO.Path]::GetTempFileName() + Copy-Item -LiteralPath $scratchCheck -Destination $scratchBackup -Force + try { + Add-Content -LiteralPath $scratchCheck -Value "`n# dogfood scratch" + $afterEdit = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure + Assert-Equal 'a run after a check edit succeeds' 0 $afterEdit.ExitCode + Assert-That 'a run after a check edit does not rebuild the image' ` + (-not ("$($afterEdit.StdOut)`n$($afterEdit.StdErr)" -match 'building |Step 1/|FROM ')) ` + 'the image was rebuilt for an edit that cannot change its contents' + } finally { + Copy-Item -LiteralPath $scratchBackup -Destination $scratchCheck -Force + Remove-Item -LiteralPath $scratchBackup -Force -ErrorAction SilentlyContinue + } + if ($SkipTier) { Write-Skipped "$EngineName : tier" '-SkipTier was passed' } else { From 3378f7dea74795a833f0fb0a04cfc1012f09950c Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 18:43:22 +0200 Subject: [PATCH 27/81] docs(anvil): correct claims the narrowed digest and cache layout invalidated Four documentation statements that no longer matched the code, all flagged by the automated reviewer: - The crate rustdoc, and the README generated from it, said `CARGO_HOME` and `RUSTUP_HOME` live in named volumes. They are deliberately not mounted: a volume is seeded from the image only when it is first created, so mounting them would pin the first image's tools over every later one. Only cargo's download caches are volumes. - The same docs said the digest covers every `*.just` under `justfiles/anvil/` except the driver. It covers the tool catalog -- `tools.just` and `versions.just` -- and nothing else, which is what lets a check edit take effect without a rebuild. - `aprz.just` still opened by saying unauthenticated access "fails on a full run", the claim the warning below it was already corrected for. It waits. One the reviewer did not raise: the same comment said the driver forwards the host's `GITHUB_TOKEN` "when it is set", which stopped being the whole story when the driver started resolving the gh CLI token as well -- it has to, because the image ships no gh. README regenerated with `just readme` rather than edited. Validation: `cargo test -p cargo-anvil` -- 354 passed, 0 failed. `just readme-check` clean, `cargo anvil --dry-run` clean, and `just anvil-container anvil-spellcheck` exits 0 (the host's cargo-spellcheck binary is broken with a missing DLL, unrelated to this change). --- .anvil.lock | 4 ++-- crates/cargo-anvil/README.md | 13 +++++++++---- crates/cargo-anvil/src/lib.rs | 11 ++++++++--- .../templates/justfiles/anvil/checks/aprz.just | 9 +++++---- .../tests/snapshots/snapshots__ado_backend.snap | 9 +++++---- .../tests/snapshots/snapshots__github_backend.snap | 9 +++++---- .../tests/snapshots/snapshots__local_only.snap | 9 +++++---- justfiles/anvil/checks/aprz.just | 9 +++++---- 8 files changed, 44 insertions(+), 29 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index fb17f6d1..8c39bad0 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:6a6d21c51dd3b55cc1a45a35920fa259157bd0fa4a335490909dbb821846fa25" +catalog_checksum = "sha256:3e1c85d9c67546e9dc0c52e67fe0f80d07b5dfbc7b63ff167cbaf7be54ed9532" [[file]] path = ".anvil/container/Dockerfile" @@ -69,7 +69,7 @@ checksum = "sha256:91408602dc3ee274b593e234841934c749ff03bba0ee7846ab88247c06f20 [[file]] path = "justfiles/anvil/checks/aprz.just" -checksum = "sha256:ee1ba62b2ad3f8d6eb8505195ad5378efc6225b05b7bc6c2ffea44788a7446f4" +checksum = "sha256:63bae0b741774aa8e21ba3168277d5641b24d6239c874ed4e74d953401863ee3" [[file]] path = "justfiles/anvil/checks/audit.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 7ef87a42..96b90b90 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -122,8 +122,10 @@ when a repository needs them. One container is created per invocation, however many checks the requested recipe runs. The repository is bind-mounted at `/workspace`, so `target/` -stays visible from the host, while `CARGO_HOME` and `RUSTUP_HOME` live in -named volumes that keep the write-heavy paths off the host boundary. +stays visible from the host. Cargo’s download caches are named volumes, +keeping that write-heavy path off the host boundary; `CARGO_HOME` and +`RUSTUP_HOME` themselves are deliberately not mounted, since a volume would +pin the first image’s tools over every later one. #### Prerequisites @@ -145,7 +147,10 @@ ARM64 hosts it is emulated and is substantially slower. The tag *is* a SHA-256 digest over the inputs that define the image: the Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -and every `*.just` under `justfiles/anvil/` other than the driver itself. +and the tool catalog (`tools.just` and `versions.just`). The tier, group +and check recipes are not inputs: they only route into the catalog, and +they run from the bind mount rather than from the image, so editing one +takes effect without a rebuild. A changed tool pin names a tag that cannot already exist, so a build follows. There is no staleness check because there is no staleness to detect: a locally built image that is present was built from the inputs @@ -435,7 +440,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbf5BkcPnaLaUbBQqw7ffBYjAbRhU4DwFqU-obm5Wjj04zV3FhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbdslbcv5wbQ8bS2Q9nS9geVobvWjkawSPj-MbJZle69zZ4fRhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 870b9293..b630644c 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -121,8 +121,10 @@ //! //! One container is created per invocation, however many checks the requested //! recipe runs. The repository is bind-mounted at `/workspace`, so `target/` -//! stays visible from the host, while `CARGO_HOME` and `RUSTUP_HOME` live in -//! named volumes that keep the write-heavy paths off the host boundary. +//! stays visible from the host. Cargo's download caches are named volumes, +//! keeping that write-heavy path off the host boundary; `CARGO_HOME` and +//! `RUSTUP_HOME` themselves are deliberately not mounted, since a volume would +//! pin the first image's tools over every later one. //! //! ### Prerequisites //! @@ -144,7 +146,10 @@ //! //! The tag *is* a SHA-256 digest over the inputs that define the image: the //! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -//! and every `*.just` under `justfiles/anvil/` other than the driver itself. +//! and the tool catalog (`tools.just` and `versions.just`). The tier, group +//! and check recipes are not inputs: they only route into the catalog, and +//! they run from the bind mount rather than from the image, so editing one +//! takes effect without a rebuild. //! A changed tool pin names a tag that cannot already exist, so a build //! follows. There is no staleness check because there is no staleness to //! detect: a locally built image that is present was built from the inputs diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just index 7d28c6dd..0cc930c3 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just @@ -7,14 +7,15 @@ # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md # cargo-aprz queries the GitHub advisory API. Unauthenticated access is -# capped at 60 requests/hour and fails on a full run; an authenticated -# token raises the cap to 5000/hour. CI injects GITHUB_TOKEN +# capped at 60 requests an hour, and on a full workspace it exhausts that +# and then waits for the quota to reset rather than failing; an +# authenticated token raises the cap to 5000/hour. CI injects GITHUB_TOKEN # (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN -# when it is set, so this recipe sees it the same way it does natively. +# unauthenticated. In a container the driver resolves the token the same +# way and forwards it by name, because the image has no gh CLI of its own. # # Unscoped (consults external risk DB). diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index f8dc76a0..3f9ff3e3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -1646,14 +1646,15 @@ unknown-git = "deny" # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md # cargo-aprz queries the GitHub advisory API. Unauthenticated access is -# capped at 60 requests/hour and fails on a full run; an authenticated -# token raises the cap to 5000/hour. CI injects GITHUB_TOKEN +# capped at 60 requests an hour, and on a full workspace it exhausts that +# and then waits for the quota to reset rather than failing; an +# authenticated token raises the cap to 5000/hour. CI injects GITHUB_TOKEN # (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN -# when it is set, so this recipe sees it the same way it does natively. +# unauthenticated. In a container the driver resolves the token the same +# way and forwards it by name, because the image has no gh CLI of its own. # # Unscoped (consults external risk DB). diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 7d12579b..19143a32 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -1567,14 +1567,15 @@ unknown-git = "deny" # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md # cargo-aprz queries the GitHub advisory API. Unauthenticated access is -# capped at 60 requests/hour and fails on a full run; an authenticated -# token raises the cap to 5000/hour. CI injects GITHUB_TOKEN +# capped at 60 requests an hour, and on a full workspace it exhausts that +# and then waits for the quota to reset rather than failing; an +# authenticated token raises the cap to 5000/hour. CI injects GITHUB_TOKEN # (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN -# when it is set, so this recipe sees it the same way it does natively. +# unauthenticated. In a container the driver resolves the token the same +# way and forwards it by name, because the image has no gh CLI of its own. # # Unscoped (consults external risk DB). diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 67576394..d61c5fd3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -386,14 +386,15 @@ unknown-git = "deny" # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md # cargo-aprz queries the GitHub advisory API. Unauthenticated access is -# capped at 60 requests/hour and fails on a full run; an authenticated -# token raises the cap to 5000/hour. CI injects GITHUB_TOKEN +# capped at 60 requests an hour, and on a full workspace it exhausts that +# and then waits for the quota to reset rather than failing; an +# authenticated token raises the cap to 5000/hour. CI injects GITHUB_TOKEN # (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN -# when it is set, so this recipe sees it the same way it does natively. +# unauthenticated. In a container the driver resolves the token the same +# way and forwards it by name, because the image has no gh CLI of its own. # # Unscoped (consults external risk DB). diff --git a/justfiles/anvil/checks/aprz.just b/justfiles/anvil/checks/aprz.just index 7d28c6dd..0cc930c3 100644 --- a/justfiles/anvil/checks/aprz.just +++ b/justfiles/anvil/checks/aprz.just @@ -7,14 +7,15 @@ # See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md # cargo-aprz queries the GitHub advisory API. Unauthenticated access is -# capped at 60 requests/hour and fails on a full run; an authenticated -# token raises the cap to 5000/hour. CI injects GITHUB_TOKEN +# capped at 60 requests an hour, and on a full workspace it exhausts that +# and then waits for the quota to reset rather than failing; an +# authenticated token raises the cap to 5000/hour. CI injects GITHUB_TOKEN # (github.token). For local runs, if GITHUB_TOKEN is unset we borrow the # gh CLI's stored token (non-interactive: `gh auth token` prints the # active account's token for github.com and never opens a browser/auth # prompt). If neither is available we warn with instructions and proceed -# unauthenticated. In a container the driver forwards the host's GITHUB_TOKEN -# when it is set, so this recipe sees it the same way it does natively. +# unauthenticated. In a container the driver resolves the token the same +# way and forwards it by name, because the image has no gh CLI of its own. # # Unscoped (consults external risk DB). From 0cc9efd3ba2fcb5d9260a968a5c1799002a80c18 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sat, 15 Aug 2026 21:44:57 +0200 Subject: [PATCH 28/81] fix(anvil): scope the mutants diff to the working tree, not to HEAD `anvil-mutants-diff` wrote a commit-to-commit diff (`git diff base..HEAD`) and handed it to `cargo mutants --in-diff`, which validates every line of the diff against the file on disk and aborts when the two disagree: ERROR Diff content doesn't match source file: crates/cargo-anvil/src/lib.rs line 124 diff has: "//! stays visible from the host, while `CARGO_HOME` and ..." source has: "//! stays visible from the host. Cargo's download caches ..." The two forms only agree when nothing is uncommitted. Locally that is the uncommon case: running a tier is how work in progress gets checked, so the tree is dirty precisely when a developer reaches for it. CI always has a clean tree, which is why this never showed there -- and why the check that catches it is the one furthest from the fast feedback loop. Diffing the base against the working tree gives cargo-mutants exactly the tree it validates against. CI is unaffected, since with a clean tree the two forms are identical. Verified by reproducing the failure condition: with an uncommitted edit to `lib.rs`, `git diff base..HEAD` and `git diff base` disagree by six lines, and `just anvil-container anvil-mutants-diff` now exits 0 against that same dirty tree. Validation: - `cargo test -p cargo-anvil` -- 354 passed, 0 failed. - `just anvil-container anvil-full` -- exit 0 in 1:25:07. Every check in the catalog, both tiers, including the 774-mutant full run. --- .anvil.lock | 4 ++-- .../templates/justfiles/anvil/checks/mutants-diff.just | 9 ++++++++- .../tests/snapshots/snapshots__ado_backend.snap | 9 ++++++++- .../tests/snapshots/snapshots__github_backend.snap | 9 ++++++++- .../tests/snapshots/snapshots__local_only.snap | 9 ++++++++- justfiles/anvil/checks/mutants-diff.just | 9 ++++++++- 6 files changed, 42 insertions(+), 7 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 8c39bad0..15dca0e9 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:3e1c85d9c67546e9dc0c52e67fe0f80d07b5dfbc7b63ff167cbaf7be54ed9532" +catalog_checksum = "sha256:f02d6feaa343a9785e53ecc414754abbac8e1897ae3f2bdf24d734756193a297" [[file]] path = ".anvil/container/Dockerfile" @@ -161,7 +161,7 @@ checksum = "sha256:14499aa7e29bc631b1905af5bb9f7d2da2c94646cf6d75f5746f930fa8044 [[file]] path = "justfiles/anvil/checks/mutants-diff.just" -checksum = "sha256:3bfed6cfa99fe92bc4dcfe01baeaca6344e80cdc69997808ff4cf466a6e968c0" +checksum = "sha256:88da99a9df855250342c011de43cce3030bdfdcfbbff364e7ceb7ac8585d53df" [[file]] path = "justfiles/anvil/checks/mutants-full.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/mutants-diff.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/mutants-diff.just index 4fcecbe2..9a7741a8 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/mutants-diff.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/mutants-diff.just @@ -40,7 +40,14 @@ anvil-mutants-diff: anvil-mutants-diff-validate-prereqs if (-not $tmp_dir) { $tmp_dir = $env:AGENT_TEMPDIRECTORY } if (-not $tmp_dir) { $tmp_dir = [System.IO.Path]::GetTempPath() } $diff_path = Join-Path $tmp_dir 'anvil-mutants-diff.diff' - git diff "$base..HEAD" --output=$diff_path + # Diff the base against the WORKING TREE, not against HEAD. + # cargo-mutants validates every line of the diff against the file on + # disk and aborts when they disagree, so a commit-to-commit diff fails + # the moment anything is uncommitted -- which is the normal local state, + # since the point of running a tier locally is to check work in progress. + # CI has a clean tree, so the two forms are identical there and this is + # not a behaviour change for it. + git diff "$base" --output=$diff_path if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } cargo mutants --in-diff $diff_path --no-shuffle --jobs 0 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 3f9ff3e3..2561b492 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -2908,7 +2908,14 @@ anvil-mutants-diff: anvil-mutants-diff-validate-prereqs if (-not $tmp_dir) { $tmp_dir = $env:AGENT_TEMPDIRECTORY } if (-not $tmp_dir) { $tmp_dir = [System.IO.Path]::GetTempPath() } $diff_path = Join-Path $tmp_dir 'anvil-mutants-diff.diff' - git diff "$base..HEAD" --output=$diff_path + # Diff the base against the WORKING TREE, not against HEAD. + # cargo-mutants validates every line of the diff against the file on + # disk and aborts when they disagree, so a commit-to-commit diff fails + # the moment anything is uncommitted -- which is the normal local state, + # since the point of running a tier locally is to check work in progress. + # CI has a clean tree, so the two forms are identical there and this is + # not a behaviour change for it. + git diff "$base" --output=$diff_path if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } cargo mutants --in-diff $diff_path --no-shuffle --jobs 0 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 19143a32..03bd7744 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -2829,7 +2829,14 @@ anvil-mutants-diff: anvil-mutants-diff-validate-prereqs if (-not $tmp_dir) { $tmp_dir = $env:AGENT_TEMPDIRECTORY } if (-not $tmp_dir) { $tmp_dir = [System.IO.Path]::GetTempPath() } $diff_path = Join-Path $tmp_dir 'anvil-mutants-diff.diff' - git diff "$base..HEAD" --output=$diff_path + # Diff the base against the WORKING TREE, not against HEAD. + # cargo-mutants validates every line of the diff against the file on + # disk and aborts when they disagree, so a commit-to-commit diff fails + # the moment anything is uncommitted -- which is the normal local state, + # since the point of running a tier locally is to check work in progress. + # CI has a clean tree, so the two forms are identical there and this is + # not a behaviour change for it. + git diff "$base" --output=$diff_path if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } cargo mutants --in-diff $diff_path --no-shuffle --jobs 0 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index d61c5fd3..659e4bd7 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -1648,7 +1648,14 @@ anvil-mutants-diff: anvil-mutants-diff-validate-prereqs if (-not $tmp_dir) { $tmp_dir = $env:AGENT_TEMPDIRECTORY } if (-not $tmp_dir) { $tmp_dir = [System.IO.Path]::GetTempPath() } $diff_path = Join-Path $tmp_dir 'anvil-mutants-diff.diff' - git diff "$base..HEAD" --output=$diff_path + # Diff the base against the WORKING TREE, not against HEAD. + # cargo-mutants validates every line of the diff against the file on + # disk and aborts when they disagree, so a commit-to-commit diff fails + # the moment anything is uncommitted -- which is the normal local state, + # since the point of running a tier locally is to check work in progress. + # CI has a clean tree, so the two forms are identical there and this is + # not a behaviour change for it. + git diff "$base" --output=$diff_path if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } cargo mutants --in-diff $diff_path --no-shuffle --jobs 0 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } diff --git a/justfiles/anvil/checks/mutants-diff.just b/justfiles/anvil/checks/mutants-diff.just index 4fcecbe2..9a7741a8 100644 --- a/justfiles/anvil/checks/mutants-diff.just +++ b/justfiles/anvil/checks/mutants-diff.just @@ -40,7 +40,14 @@ anvil-mutants-diff: anvil-mutants-diff-validate-prereqs if (-not $tmp_dir) { $tmp_dir = $env:AGENT_TEMPDIRECTORY } if (-not $tmp_dir) { $tmp_dir = [System.IO.Path]::GetTempPath() } $diff_path = Join-Path $tmp_dir 'anvil-mutants-diff.diff' - git diff "$base..HEAD" --output=$diff_path + # Diff the base against the WORKING TREE, not against HEAD. + # cargo-mutants validates every line of the diff against the file on + # disk and aborts when they disagree, so a commit-to-commit diff fails + # the moment anything is uncommitted -- which is the normal local state, + # since the point of running a tier locally is to check work in progress. + # CI has a clean tree, so the two forms are identical there and this is + # not a behaviour change for it. + git diff "$base" --output=$diff_path if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } cargo mutants --in-diff $diff_path --no-shuffle --jobs 0 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } From 0fda9a3df40ab9982ffb6ef2b5f7d0aacc663d8d Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 16 Aug 2026 18:45:30 +0200 Subject: [PATCH 29/81] fix(anvil): name the image by everything that defines it The digest covered `tools.just` and `versions.just` -- the recipes that decide *how* a tool installs -- but not the fan-out that decides *whether* it installs at all. `just anvil-setup` reaches those install recipes through `tiers.just`, then a group, then each check's `-setup`, and none of that was hashed. Deleting `(anvil-spellcheck-setup installer)` from `groups/pr-fast.just` changes the installed tool set while producing a byte-identical tag, so the reduced image is reused forever and the freshness guarantee the whole design rests on is void. Hashing the whole recipe tree restores it. `container.just` is included as well: that is not circular -- the digest is over file text, and no file contains the tag -- and it belongs in the set because it passes the build arguments, the secret mounts and the hook's `Anvil-PreBuild` output into the build. The cost is that editing any recipe rebuilds the image, which is what the narrowing had removed. A resolved install manifest (`just --dry-run anvil-setup`) would have kept both properties, and discriminates correctly, but it embeds `just_executable()`'s absolute path, so every developer would compute a different tag for identical content. Rebuilding too often is a slow correct answer; a tag that names contents the image does not have is a fast wrong one. The same review pass found nine other defects in the containerized path. Faithfulness to a native run: - `PR_TITLE`, `BASE_REF`, `GITHUB_BASE_REF`, `SYSTEM_PULLREQUEST_TARGETBRANCH` and the `ANVIL_INCLUDE_*` filters were dropped at the boundary. `anvil-pr-title` exits 0 with a skip notice when `PR_TITLE` is unset, so a title a native run rejects passed in a container and the tier still reported green. They are now forwarded by name when set. - A token derived from the gh CLI was placed on PID 1 for every target, where natively `anvil-aprz` mints it inside its own process -- so every cargo build, build script and proc macro in the container saw a live credential. It is now derived only when the target's plan reads `GITHUB_TOKEN`, or when there is no target at all, since an interactive session can run anything. An *exported* token is still forwarded unconditionally: that is exact parity. The predicate is the variable rather than the name of a check, so the driver stays generic. - `invocation_directory()` reports a Cygwin-style path when `cygpath` is on PATH, which shares no prefix with the native `justfile_directory()` it was made relative to, so `GetRelativePath` walked out with `..` and placed the run outside the mount. Uses `invocation_directory_native()`, and refuses a directory outside the repository rather than mounting something surprising. - `$CARGO_HOME/git` did not exist when the image widened permissions, so the named volume mounted over it was seeded root-owned 0755 and the first fetch under `--user` failed with EACCES. Created before the `chmod`. - `anvil-container-down` removed `-cargo` and `-rustup` volumes that no released version ever created. The hook contract, which did not behave as documented: - The resolve path promised that "a broken hook falls through to a build", but the dot-source sat outside the `try`, and the recipe runs under `$ErrorActionPreference = 'Stop'` -- so a `hooks.ps1` with a syntax error killed the run instead. Loading now happens inside the same `try`. - `Anvil-PreBuild` and `Anvil-PreRun` stay fail-closed, which is the opposite and deliberate: a run that cannot obtain its credentials must stop. They are wrapped only so the failure names the hook instead of surfacing as a bare parser error. - The empty-value guard used `IsNullOrEmpty`, which accepts a single space, and read `.Secrets` off the whole output stream -- so a hook that wrote progress with `Write-Output` handed back an array whose `.Secrets` was silently `$null` and no secret was mounted at all. Takes the last emitted object, rejects whitespace, and rejects a defined function that returns no entries. Claims that were not true: - `image inspect` on a resolved reference is a presence check, not a verification: it proves something carries that reference, not that its contents match the digest the tag claims. Said so in the rustdoc, the design document and the test name. - The rustdoc for `hooks()` documented two functions and omitted `Anvil-ResolveImage`. - containers.md said only the tool catalog was hashed in one section and that any recipe edit invalidated the image in another. Tests that could not fail: - The e2e asserted a build secret was absent from the image filesystem, but the fixture only read the secret's length -- nothing ever wrote it, so `grep` found nothing whatever the layering did. A control image now writes the secret without deleting it and the probe must find it, before the real image must not. - The generator's exit code was discarded with `-AllowFailure | Out-Null`. - The dogfood script matched `'building image'` against a driver that emits `'building (inputs changed...)'`. - Its tag matrix asserted the defect above as intended behaviour. It now asserts the opposite, plus the reviewer's reproduction directly: dropping a `-setup` dependency must rename the image. Validation, all on this tree: - `cargo test -p cargo-anvil` -- 309 unit, 3 snapshot, 47 across the remaining targets, 0 failed. - `scripts/test-anvil-container.ps1` -- 62/62 in 07:57. - `scripts/test-anvil-dogfood.ps1 -SkipTier` -- 46/46 against docker and podman in 17:02, including `dropping a setup dependency renames the image` on both engines. - `just anvil-container anvil-full` -- exit 0. Every check in the catalog, 774 mutants with 0 missed. - `just readme` regenerated, `just format` clean. --- .anvil.lock | 6 +- .anvil/container/Dockerfile | 19 +- crates/cargo-anvil/README.md | 20 +- crates/cargo-anvil/docs/design/containers.md | 72 +++-- .../src/anvil/artifacts/container.rs | 147 +++++++-- crates/cargo-anvil/src/lib.rs | 18 +- .../templates/anvil/container/Dockerfile | 19 +- .../templates/justfiles/anvil/container.just | 267 +++++++++++----- .../snapshots/snapshots__ado_backend.snap | 286 +++++++++++++----- .../snapshots/snapshots__github_backend.snap | 286 +++++++++++++----- .../snapshots/snapshots__local_only.snap | 286 +++++++++++++----- justfiles/anvil/container.just | 267 +++++++++++----- scripts/test-anvil-container.ps1 | 46 ++- scripts/test-anvil-dogfood.ps1 | 72 ++--- 14 files changed, 1294 insertions(+), 517 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 15dca0e9..ca5b5138 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:f02d6feaa343a9785e53ecc414754abbac8e1897ae3f2bdf24d734756193a297" +catalog_checksum = "sha256:dfba2a9ff5c11ad73780af9e64114aa50d859f2039737eb6935d97ebc82bb30e" [[file]] path = ".anvil/container/Dockerfile" -checksum = "sha256:e4dd5a426662328eddbd8b3bfe2fda073e691914326ab4a491a9ab1501ddf1a9" +checksum = "sha256:a3e106b7dbde0bb6a9c2b94dae0f9cb82b9055ea97f100438f3d3d20ec66e971" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:f038ef8e16637695db80e85cdc9567a73af874bc4a191656aee1ccfe5c00a819" +checksum = "sha256:7c695125346535d0ba17d09947854af59971ff41f53a7faab5f4101159055ca8" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index b75dadc1..3786a20a 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -92,12 +92,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, but -# only `tools.just` and `versions.just` decide what this layer installs, and -# only those are hashed into the tag -- so editing a check does not rebuild the -# image, and the check runs from the bind mount anyway. The synthetic Justfile -# below imports only the anvil tree, avoiding repository-specific imports that -# may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -108,6 +107,13 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # *writes* with it is ordinary content -- and the `chmod` on the next line # would otherwise publish it world-readable. Deleting them in a later `RUN` # would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -115,6 +121,7 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && just anvil-setup binstall \ && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" # Consumed by `anvil-container` itself: a nested invocation from inside the diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 96b90b90..62ee32d6 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -147,10 +147,11 @@ ARM64 hosts it is emulated and is substantially slower. The tag *is* a SHA-256 digest over the inputs that define the image: the Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -and the tool catalog (`tools.just` and `versions.just`). The tier, group -and check recipes are not inputs: they only route into the catalog, and -they run from the bind mount rather than from the image, so editing one -takes effect without a rebuild. +and the whole generated `justfiles/anvil/` tree. The tree is included in +full because the image installs its tools by running `just anvil-setup`, +whose dependency chain runs through the tier, group and check recipes +before it reaches the install recipes – so the routing decides *whether* a +tool is installed just as surely as `tools.just` decides *how*. A changed tool pin names a tag that cannot already exist, so a build follows. There is no staleness check because there is no staleness to detect: a locally built image that is present was built from the inputs @@ -203,9 +204,10 @@ produces. `Anvil-ResolveImage` is offered the tag when nothing local matches, and returns the reference it made available: a registry reference, used as-is rather than re-tagged locally, so the run stays honest about where the -image came from. It is verified before use, and every failure falls through -to a local build: a publisher that has not caught up must not block the -change it has not caught up with. +image came from. Its presence is checked before use – which proves +something carries that reference, not that the contents match the digest – +and every failure falls through to a local build: a publisher that has not +caught up must not block the change it has not caught up with. The hook executes on the host, with the invoking user’s permissions, before any container isolation exists. Only use one from a repository or catalog @@ -218,7 +220,7 @@ for extra packages, and anvil’s drift handling preserves the change. A downstream catalog that needs a different base OS or toolchain source for every repository it manages replaces the artifact instead. A replacement that copies more of the tree must replace the ignore file with it, since -the build context admits only `justfiles/` and `rust-toolchain.toml`. See +the build context admits only `justfiles/anvil/` and `rust-toolchain.toml`. See [`artifacts::container`][__link1] and the design document for the full contract, the host setup for each engine, and the known limitations. @@ -440,7 +442,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbdslbcv5wbQ8bS2Q9nS9geVobvWjkawSPj-MbJZle69zZ4fRhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbkAqV7cij89Ab63nE30mvTLkbN2Wo4toYNGkb0MosdkJlO6lhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 2f3f6f83..c1fac1f1 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -79,7 +79,7 @@ All five are annotated `[group("anvil-container")]` and appear as one cluster in | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook (§7.3), so a query never pulls. | | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild with `--no-cache` even when the tag resolves. Skips the resolve hook too (§7.3). | | `ANVIL_IN_CONTAINER=1` | Set inside the image. Makes a nested invocation execute natively (§5.4). | -| `GITHUB_TOKEN` | Forwarded into the run. Taken from the host environment, or from the gh CLI when that is unset (§5.3). | +| `GITHUB_TOKEN` | Forwarded into the run. Taken from the host environment, or derived from the gh CLI for a target that reads it (§5.3). | `NO_REBUILD` is evaluated independently of `NO_CACHE`, so the two compose: `anvil-container-status` sets `NO_REBUILD` and `NO_RESOLVE` together and answers from local state alone. When `NO_REBUILD` stops a build the reference is still @@ -106,8 +106,8 @@ generated pins. There is no second tool list to keep synchronized, and consequen image (§4.1). `Dockerfile.dockerignore` scopes the build context to `justfiles/anvil/` and `rust-toolchain.toml`, denying everything -else. The whole recipe tree is copied because `just` has to parse it to run `anvil-setup`, while only the tool catalog -within it is hashed (§4). BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the +else. The whole recipe tree is copied because `just` has to parse it to run `anvil-setup`, and the whole tree is +hashed (§4). BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the repository neither needs to own a root ignore file nor can have one silently override this. ## 4. Image identity @@ -123,14 +123,21 @@ define the image. The name derives from the repository directory (§5.1). | `.anvil/container/Dockerfile.dockerignore` | always | | `rust-toolchain.toml` | always | | `.anvil/container/hooks.ps1` | when the file exists | -| `justfiles/anvil/tools.just` | always | -| `justfiles/anvil/versions.just` | always | +| `justfiles/anvil/**/*.just` | always | -`tools.just` and `versions.just` are the tool catalog: `just anvil-setup` installs all of it, every install recipe is -defined in `tools.just`, and every pin lives in `versions.just`, so a tool cannot be added, removed or repinned without -one of the two changing. The tier, group and check recipes are **not** inputs — they only route into the catalog, and -they execute from the bind mount rather than from the image, so editing a check takes effect on the next run without a -rebuild. A declared input that does not exist is a hard error, not an omission from the digest. +The recipe tree is hashed in full. `just anvil-setup` reaches the install recipes through the tier, group and check +recipes, so the routing decides *whether* a tool is installed just as surely as `tools.just` decides *how*: dropping an +`anvil--setup` dependency from a group changes the installed set while `tools.just` and `versions.just` stay +byte-identical. Hashing only the install definitions would leave that change unnamed, and the tag would claim contents +the image does not have. + +`container.just` is hashed too. It is not circular — the digest is over file text, and no file contains the tag — and +it belongs in the set because it passes the build arguments, the secret mounts and the hook's `Anvil-PreBuild` output +into the build. + +The cost is that editing any recipe renames the image and the next run rebuilds it. That is the correct trade: a tag +that can name contents the image does not have makes every guarantee below meaningless. A declared input that does not +exist is a hard error, not an omission from the digest. The hook file's **content** is an input, since it determines what the build installs. Its **output** is deliberately excluded: a credential must never influence a tag. @@ -224,6 +231,21 @@ it natively: the environment first, then the gh CLI's stored token. `anvil-aprz` queries the GitHub advisory API, which allows 60 requests an hour unauthenticated and then sleeps until the quota resets, so a tier needs the token to terminate rather than merely to run quickly. +The two sources are not treated alike. An **exported** `GITHUB_TOKEN` is forwarded whatever the target is — that is +exact parity, since a native run exposes it to every process the shell spawns too. A token **derived** from the gh CLI +is a credential the developer never put in this environment, and PID 1's environment is inherited by every build script +and proc macro in the container, where natively `anvil-aprz` would mint it inside its own process. So it is derived +only when the target's plan (`just --dry-run `) reads `GITHUB_TOKEN`, or when there is no target at all: an +interactive session can run anything, and refusing there would reintroduce the stall the token exists to prevent. The +predicate is the variable rather than the name of a check, so a catalog that adds another GitHub-authenticated check is +covered without touching the driver. + +It also forwards the recipe contract's own inputs when they are set — `PR_TITLE`, `BASE_REF`, `GITHUB_BASE_REF`, +`SYSTEM_PULLREQUEST_TARGETBRANCH` and the `ANVIL_INCLUDE_*` filters — because a check that reads one natively must read +the same value in a container. `anvil-pr-title` is the sharp case: with `PR_TITLE` unset it exits 0 with a skip notice, +so dropping it at the boundary would let a title a native run rejects pass in a container while the tier still reported +green. They are forwarded by name and only when set, so an unset variable stays unset rather than arriving empty. + A resolved token is set on the driver process, passed by name, and unset after the run, so it never reaches a host command line. Inside the container it is readable by everything the run executes, including build scripts and proc macros. @@ -411,10 +433,13 @@ Three properties are load-bearing: - **The returned reference is used as-is, never re-tagged locally.** A local tag asserts "built here from these inputs"; a fetched image only claims it (§4.3). Keeping the registry reference keeps the run honest about origin. -- **The reference is verified before use.** Runs pass `--pull=never`, so a hook reporting an image it had not actually - fetched would otherwise fail later and further from the cause. -- **Every failure falls through to a local build**, with the reason printed. A publisher that has not caught up with a - change must not block the developer who made it. +- **The reference is checked for presence before use.** Runs pass `--pull=never`, so a hook reporting an image it had + not actually fetched would otherwise fail later and further from the cause. This is a presence check, not a + verification: `image inspect` proves something carries that reference, not that its contents match the digest the tag + claims. Trusting the publisher is the contract (§4.3). +- **Every failure falls through to a local build**, with the reason printed — including a hook that cannot be loaded at + all, which is why the dot-source sits inside the same `try`. A publisher that has not caught up with a change must + not block the developer who made it. ### 7.4 Trust boundary @@ -444,7 +469,7 @@ its own version against a file it can see has diverged. A change that belongs ev **The Dockerfile and its ignore file must be replaced together.** A replacement that `COPY`s anything beyond `justfiles/anvil/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the added paths never reach the build context and the build fails on a missing file. A replacement that installs tools -from somewhere other than `tools.just` must also add those sources to the digest, or a change to them will name a tag +from a source outside `justfiles/anvil/` must also add that source to the digest, or a change to it will name a tag that already resolves. `justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces: a non-recipe @@ -459,8 +484,10 @@ guard. A different base OS with a different toolchain source is one Dockerfile r - On ARM64 hosts the `linux/amd64` image is emulated and is substantially slower. - The first build takes several minutes, installing a toolchain and the entire pinned tool catalog. Later runs reuse it until an input changes. -- Any edit under `justfiles/anvil/` invalidates the install layer, including files the image's synthetic Justfile - never imports. +- Any edit under `justfiles/anvil/` renames the image and rebuilds it, including edits to a check body that cannot + change what the image contains. Precision here would mean deriving the install closure rather than hashing the files + that express it; until then the digest errs towards rebuilding, because the alternative error — a tag that names + contents the image does not have — is silent (§4.1). - **The set of hashed inputs is fixed (§4.1) and a fork cannot extend it.** A replacement Dockerfile is itself hashed, so changing the build recipe always renames the tag — but any *additional* file it copies is outside the tag. Such a file can change what a build produces while naming a tag that already resolves, and the existing image @@ -480,4 +507,15 @@ the emitted-tree snapshots are what run unattended. ./scripts/test-anvil-container.ps1 -Engine podman # podman ``` +`scripts/test-anvil-dogfood.ps1` is the complement: rather than a synthetic fixture it runs this repository's own +generated tree in its own image, which is what catches the defects a fixture is too small to have — a check whose tool +is missing, a mount whose permissions are wrong, a variable that does not cross the boundary. It mutates the working +tree while asserting which edits rename the image, and restores each file from a byte copy; if it is interrupted +mid-run, `cargo run -p cargo-anvil -- anvil` returns the generated tree to a known state. + +```powershell +./scripts/test-anvil-dogfood.ps1 # docker, full tier +./scripts/test-anvil-dogfood.ps1 -Engine podman -SkipTier # podman, mechanism only +``` + [design]: ./README.md diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 9af97b32..6a729d65 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -81,19 +81,28 @@ pub fn dockerignore() -> Artifact { /// catalog adds one with [`crate::CatalogBuilder::with_artifact`]; a single /// repository can write the same path by hand. The recipe loads it either way. /// -/// The script may define either or both of two functions, and is dot-sourced -/// before the phase that needs it: +/// The script may define any of three functions, and is dot-sourced before the +/// phase that needs it: /// /// - `Anvil-PreBuild` returns `@{ Secrets = @{ = } }`. Each entry /// becomes a `BuildKit` `--secret id=`, passed by environment variable /// name so the value never reaches a process argument, and never a layer. /// - `Anvil-PreRun` returns `@{ Env = @{ = } }`. Each entry is /// forwarded into the container by name, for the same reason. +/// - `Anvil-ResolveImage` takes the computed reference and returns one to use +/// instead, or nothing. It is how a repository fetches a published image +/// rather than building locally. /// -/// An empty value from either function is a hard error: a build that silently -/// proceeds without its credential would install a reduced tool set and then be -/// tagged with the same content hash a credentialed build produces, so every -/// later run would reuse the broken image. +/// The two credential phases are fail-closed: an empty value, a return with no +/// entries, a throw, or a script that cannot even be loaded stops the run. A +/// build that silently proceeded without its credential would install a reduced +/// tool set and then be tagged with the same content hash a credentialed build +/// produces, so every later run would reuse the broken image. +/// +/// `Anvil-ResolveImage` is the opposite, and deliberately so: every failure -- +/// including a hook that fails to load -- falls through to a local build, which +/// is slower but always correct. A publisher that has not caught up with a +/// change must not block the developer who made it. /// /// The file's *content* is part of the image identity, since it decides what the /// build installs. Its *output* deliberately is not: a credential must never @@ -210,15 +219,25 @@ mod tests { assert!(resolve < no_rebuild, "resolving is not building, so NO_REBUILD must not block it"); // A publisher that has not caught up must not stop the developer who - // made the change, so every failure falls through to a build. - assert!(RECIPE.contains("anvil: Anvil-ResolveImage failed:")); + // made the change, so every failure falls through to a build -- + // including a hook that cannot even be loaded, which is why the + // dot-source is inside the try rather than ahead of it. + let try_start = RECIPE[..resolve].rfind("try {").expect("the resolve call must sit inside a try"); + let load = RECIPE[..resolve] + .rfind(". $hookPath") + .expect("the hook must be loaded before it is called"); + assert!(try_start < load, "loading a broken hook must not escape the catch"); + assert!(RECIPE.contains("anvil: $hookRel failed:")); assert!(RECIPE.contains("anvil: nothing resolved; building locally")); } #[test] - fn a_resolved_reference_is_verified_before_it_is_used() { - // The run is `--pull=never`, so a hook that reports a reference it did - // not actually fetch would fail later and further from the cause. + fn a_resolved_reference_is_checked_for_presence_before_it_is_used() { + // Presence, not verification: `image inspect` proves something carries + // that reference, not that its contents match the digest the tag + // claims. Trusting the hook is the contract; this only keeps a + // reference the hook never fetched from failing later, under + // `--pull=never`, a long way from the cause. let resolve = RECIPE.find("Anvil-ResolveImage $image").expect("the resolve call must exist"); let verify = RECIPE[resolve..] .find("image inspect $resolved") @@ -226,7 +245,7 @@ mod tests { let accept = RECIPE[resolve..] .find("Write-Output $resolved") .expect("a resolved reference must be returned"); - assert!(verify < accept, "verify the resolved reference before returning it"); + assert!(verify < accept, "check the resolved reference before returning it"); } #[test] @@ -259,7 +278,7 @@ mod tests { } // And the escaping that is present uses just's own doubling form. assert!(RECIPE.contains(r#"replace(justfile_directory(), "'", "''")"#)); - assert!(RECIPE.contains(r#"replace(invocation_directory(), "'", "''")"#)); + assert!(RECIPE.contains(r#"replace(invocation_directory_native(), "'", "''")"#)); assert!(RECIPE.contains(r#"replace(target, "'", "''")"#)); } @@ -398,7 +417,8 @@ mod tests { // A derived token is set on this process, so it must be registered for // the same cleanup the hook's variables get. assert!(RECIPE.contains("$hookEnv += 'GITHUB_TOKEN'")); - // An exported token is left alone rather than re-derived. + // An exported token is left alone rather than re-derived; scoping of + // the derived one is asserted in its own test below. assert!(RECIPE.contains("if (-not $env:GITHUB_TOKEN -and (Get-Command gh")); // Forwarding by name only works if the engine can see the name, so a // WSL engine needs it bridged -- otherwise `-e NAME` forwards nothing. @@ -406,16 +426,95 @@ mod tests { } #[test] - fn only_the_tool_catalog_defines_the_image() { - // `just anvil-setup` installs the whole catalog, and every install - // recipe is defined in tools.just against a pin in versions.just, so a - // tool cannot be added, removed or repinned without one of the two - // changing. Hashing the tier/group/check recipes as well would rebuild - // the image for an edit that cannot change what it contains -- and - // those recipes run from the bind mount, not from the image. - assert!(RECIPE.contains("$inputs += 'justfiles/anvil/tools.just'")); - assert!(RECIPE.contains("$inputs += 'justfiles/anvil/versions.just'")); - assert!(!RECIPE.contains("-Recurse -File -Filter '*.just'")); + fn the_whole_recipe_tree_defines_the_image() { + // `just anvil-setup` reaches the install recipes through the tier, + // group and check recipes, so the routing decides *whether* a tool is + // installed as surely as tools.just decides *how*. Hashing only the + // install definitions would let a group drop a `-setup` dependency, + // changing the installed set, without renaming the image. + assert!(RECIPE.contains("-Recurse -File -Filter '*.just'")); + // Including this driver, which passes the build arguments, the secret + // mounts and the hook's PreBuild output into the build. + assert!(!RECIPE.contains("-cne 'justfiles/anvil/container.just'")); + } + + #[test] + fn the_recipe_contract_inputs_cross_the_boundary() { + // A check that reads one of these natively must read the same value in + // a container, or the same command means two different things. + // anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with + // a skip notice, so a title a native run rejects would pass in a + // container and the tier would still report green. + for name in ["PR_TITLE", "BASE_REF", "GITHUB_BASE_REF", "SYSTEM_PULLREQUEST_TARGETBRANCH"] { + assert!(RECIPE.contains(name), "{name} must be forwarded"); + } + for name in ["ANVIL_INCLUDE_MODIFIED", "ANVIL_INCLUDE_AFFECTED", "ANVIL_INCLUDE_REQUIRED"] { + assert!(RECIPE.contains(name), "{name} must be forwarded"); + } + } + + #[test] + fn a_derived_token_is_scoped_to_a_target_that_reads_it() { + // Forwarding an exported GITHUB_TOKEN is exact parity: natively it is + // visible to every process the shell spawns too. Minting one from `gh` + // is not -- PID 1's environment reaches every build script and proc + // macro, where natively the recipe mints it in its own process -- so it + // happens only for a target whose plan reads the variable. + let derive = RECIPE.find("gh auth token --hostname").expect("the gh fallback must exist"); + let guard = RECIPE[..derive].rfind("if ($needsToken)").expect("the derive must be guarded"); + let plan = RECIPE[..guard] + .rfind("$plan -match 'GITHUB_TOKEN'") + .expect("the plan must decide whether a token is needed"); + let dry_run = RECIPE[..plan] + .rfind("just --dry-run @targetParts") + .expect("the plan must come from just"); + assert!(dry_run < plan && plan < guard, "compute the plan, match it, then derive"); + // The predicate is the variable, not the name of a check, so a catalog + // that adds another GitHub-authenticated check is covered for free. + assert!(!RECIPE.contains("$plan -match 'aprz'")); + // An interactive session has no target to plan, and can run anything. + assert!(RECIPE.contains("$needsToken = $targetParts.Count -eq 0")); + } + + #[test] + fn the_credential_phases_are_fail_closed() { + // Unlike resolution, these must stop the run: a container that starts + // without its credentials fails deep inside, far from the cause. + assert!(RECIPE.contains("anvil: Anvil-PreBuild returned no secrets")); + assert!(RECIPE.contains("anvil: Anvil-PreRun returned no variables")); + assert!(RECIPE.contains("anvil: failed to load ${hookRel}:")); + // Whitespace is not a credential. IsNullOrEmpty would accept " ". + assert!(!RECIPE.contains("[string]::IsNullOrEmpty($hook")); + // Take the last object, not the whole stream: a hook that writes + // progress with Write-Output would otherwise hand back an array whose + // .Secrets is silently $null, and the guard above would not fire. + assert!(RECIPE.contains("@(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1")); + assert!(RECIPE.contains("@(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1")); + } + + #[test] + fn the_working_directory_is_mapped_from_a_native_path() { + // `invocation_directory()` reports a Cygwin-style path when cygpath is + // on PATH, which shares no prefix with the native justfile_directory() + // it is made relative to -- so the run would be placed outside the + // mount, on a path that does not exist in the container. + assert!(RECIPE.contains("invocation_directory_native()")); + assert!(!RECIPE.contains("replace(invocation_directory(), ")); + assert!(RECIPE.contains("$rel.StartsWith('..')")); + } + + #[test] + fn teardown_removes_only_volumes_the_run_creates() { + // Naming a volume the run never mounts is a claim that it exists. + let down = RECIPE.find("anvil-container-down:").expect("the teardown recipe must exist"); + for stale in ["-cargo'", "-rustup'"] { + assert!( + !RECIPE[down..].contains(stale), + "{stale} is never created, so it cannot be torn down" + ); + } + assert!(RECIPE[down..].contains("-cargo-registry'")); + assert!(RECIPE[down..].contains("-cargo-git'")); } #[test] diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index b630644c..9f9c6f43 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -146,10 +146,11 @@ //! //! The tag *is* a SHA-256 digest over the inputs that define the image: the //! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -//! and the tool catalog (`tools.just` and `versions.just`). The tier, group -//! and check recipes are not inputs: they only route into the catalog, and -//! they run from the bind mount rather than from the image, so editing one -//! takes effect without a rebuild. +//! and the whole generated `justfiles/anvil/` tree. The tree is included in +//! full because the image installs its tools by running `just anvil-setup`, +//! whose dependency chain runs through the tier, group and check recipes +//! before it reaches the install recipes -- so the routing decides *whether* a +//! tool is installed just as surely as `tools.just` decides *how*. //! A changed tool pin names a tag that cannot already exist, so a build //! follows. There is no staleness check because there is no staleness to //! detect: a locally built image that is present was built from the inputs @@ -202,9 +203,10 @@ //! `Anvil-ResolveImage` is offered the tag when nothing local matches, and //! returns the reference it made available: a registry reference, used as-is //! rather than re-tagged locally, so the run stays honest about where the -//! image came from. It is verified before use, and every failure falls through -//! to a local build: a publisher that has not caught up must not block the -//! change it has not caught up with. +//! image came from. Its presence is checked before use -- which proves +//! something carries that reference, not that the contents match the digest -- +//! and every failure falls through to a local build: a publisher that has not +//! caught up must not block the change it has not caught up with. //! //! The hook executes on the host, with the invoking user's permissions, before //! any container isolation exists. Only use one from a repository or catalog @@ -217,7 +219,7 @@ //! downstream catalog that needs a different base OS or toolchain source for //! every repository it manages replaces the artifact instead. A replacement //! that copies more of the tree must replace the ignore file with it, since -//! the build context admits only `justfiles/` and `rust-toolchain.toml`. See +//! the build context admits only `justfiles/anvil/` and `rust-toolchain.toml`. See //! [`artifacts::container`] and the design document for the full contract, //! the host setup for each engine, and the known limitations. //! diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile b/crates/cargo-anvil/templates/anvil/container/Dockerfile index b75dadc1..3786a20a 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile @@ -92,12 +92,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, but -# only `tools.just` and `versions.just` decide what this layer installs, and -# only those are hashed into the tag -- so editing a check does not rebuild the -# image, and the check runs from the bind mount anyway. The synthetic Justfile -# below imports only the anvil tree, avoiding repository-specific imports that -# may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -108,6 +107,13 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # *writes* with it is ordinary content -- and the `chmod` on the next line # would otherwise publish it world-readable. Deleting them in a later `RUN` # would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -115,6 +121,7 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && just anvil-setup binstall \ && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" # Consumed by `anvil-container` itself: a nested invocation from inside the diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 301410d7..c65e6ca5 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -107,11 +107,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the tool catalog -# (`tools.just` and `versions.just`) that decides what `just anvil-setup` -# installs. The tier, group and check recipes are not inputs: they only route -# into the catalog, and they execute from the bind mount rather than from the -# image, so editing one takes effect on the next run without a rebuild. +# ignore file, the pinned toolchain, the optional hook, and the whole generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes alike. Editing any of them can change what the image contains, so +# any of them can rename it. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -135,14 +135,24 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - # The tool catalog, and nothing else. `just anvil-setup` installs the whole - # catalog, and every install recipe is defined in tools.just against a pin - # in versions.just -- so a tool cannot be added, removed or repinned - # without one of these two changing. The tier, group and check recipes only - # route into them, and are read from the bind mount at run time rather than - # from the image, so editing a check must not cost a rebuild. - $inputs += 'justfiles/anvil/tools.just' - $inputs += 'justfiles/anvil/versions.just' + # Every generated recipe file. The image installs its tools by running + # `just anvil-setup`, and that dependency chain runs through the tier, + # group and check recipes before it reaches the install recipes in + # tools.just -- so the routing decides *whether* a tool is installed just + # as surely as tools.just decides *how*. Hashing only the install + # definitions would let a group drop a `-setup` dependency, changing the + # installed set, without renaming the image. + # + # This driver is included too. It is not circular -- the tag is derived + # from file text, and no file contains the tag -- and it belongs in the set + # because it passes the build arguments, the secret mounts and the hook's + # `Anvil-PreBuild` output into the build, all of which shape the result. + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -226,32 +236,40 @@ _anvil-container-image: # with a change must not stop the developer who made it. $hookPath = Join-Path $repoRoot $hookRel if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { - . $hookPath - if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") - $resolved = $null - try { + # Dot-sourcing is inside the try as well: a hook with a syntax error, + # or one that throws while being loaded, must cost no more than a + # hook that resolves nothing. The recipe runs under + # `$ErrorActionPreference = 'Stop'`, so leaving the load outside + # would abort the run instead of falling through to a build. + $resolved = $null + try { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 - } catch { - [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") - $resolved = $null } - if (-not [string]::IsNullOrWhiteSpace($resolved)) { - $resolved = ([string]$resolved).Trim() - # Verify rather than trust: the run is `--pull=never`, so a - # reference the hook reported but did not actually fetch - # would fail later, further from the cause. - & $engineExe @enginePrefix image inspect $resolved *> $null - if ($LASTEXITCODE -eq 0) { - [Console]::Error.WriteLine("anvil: resolved $resolved") - Write-Output $resolved - exit 0 - } - [Console]::Error.WriteLine( - "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } catch { + [Console]::Error.WriteLine("anvil: $hookRel failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # A presence check, not a verification: `image inspect` proves + # something is tagged with that reference, not that its contents + # match the digest the tag claims. Trusting the hook is the + # contract -- this only keeps a reference the hook reported but + # never fetched from failing later, under `--pull=never`, a long + # way from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 } - [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") } } @@ -279,23 +297,49 @@ _anvil-container-image: $secretEnv = @() $hookPath = Join-Path $repoRoot $hookRel if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed, unlike the resolve hook: a build that cannot mint its + # credentials must stop, not proceed to produce a reduced image. The + # try exists only so the cause is named -- loading a hook with a syntax + # error would otherwise surface as a bare parser error with no hint + # that a hook was involved. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") - $hook = Anvil-PreBuild - if ($null -ne $hook -and $null -ne $hook.Secrets) { - foreach ($id in $hook.Secrets.Keys) { - if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" - exit 1 - } - $name = "ANVIL_SECRET_$id" - Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] - $secretEnv += $name - $secretArgs += "id=$id,env=$name" + try { + # Take the last emitted object, not the whole stream: a hook + # that writes progress with `Write-Output` would otherwise hand + # back an array whose `.Secrets` is silently $null. + $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + exit 1 + } + $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } + # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # guard exists for, not a hook opting out: secrets are the only + # thing the phase can contribute, so an empty return means the mint + # failed quietly. A hook with no build-time credentials simply does + # not define the function. + if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + exit 1 + } + foreach ($id in $secrets.Keys) { + if ([string]::IsNullOrWhiteSpace($secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 } - [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" } + [Console]::Error.WriteLine("anvil: build secrets: $($secrets.Keys -join ', ')") } } @@ -389,8 +433,17 @@ anvil-container *target: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so - # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' + # relative paths keep working from a subdirectory. `invocation_directory()` + # would be wrong here: with `cygpath` on PATH it reports a Cygwin-style + # path, which shares no prefix with the native `justfile_directory()` above. + # `GetRelativePath` then walks out with `..` segments and the run is placed + # outside the mount, so it fails on a path that does not exist in the + # container. The `_native` form is the one that agrees with the root. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory_native(), "'", "''") }}') -replace '\\', '/' + if ($rel.StartsWith('..')) { + Write-Error "anvil: run this from inside the repository; $rel is outside $repoRoot" + exit 1 + } $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { @@ -487,22 +540,44 @@ anvil-container *target: # # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN # first, then the gh CLI's stored token -- so a containerized run - # authenticates for the same developers a native run does. Deriving it here - # rather than only forwarding an exported value is deliberate: the two - # paths have identical exposure once inside (a forwarded token is readable - # by every recipe in the container either way, including third-party build - # scripts and proc macros), so refusing to derive it buys no boundary and - # only makes behaviour depend on how the developer happened to sign in. + # authenticates for the same developers a native run does. + # + # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # that is exact parity, since a native run exposes it to every process the + # shell spawns too. Deriving one from `gh` is different -- it manufactures a + # credential the developer did not put in this environment, and PID 1's + # environment is inherited by every build script and proc macro in the + # container, where natively the recipe would mint it in its own process. So + # it is derived only when the target actually reads the variable, or when + # there is no target at all: an interactive session can run anything, and + # refusing there would reintroduce the silent hour-long stall on a tier the + # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. # + # The predicate is the variable itself rather than the name of a check, so + # the driver stays generic: a catalog that adds another GitHub-authenticated + # check is covered without touching this recipe. + # # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - $ghToken = $null - try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } - if ($ghToken -and $ghToken.Trim()) { - Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() - $hookEnv += 'GITHUB_TOKEN' + # Ask just what the target would run. A dry run has no side effects, and + # a target that cannot be planned (a typo, a recipe needing arguments) + # yields nothing, so the run fails on its own terms rather than on a + # missing token. + $needsToken = $targetParts.Count -eq 0 + if (-not $needsToken) { + $plan = '' + try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + $needsToken = $plan -match 'GITHUB_TOKEN' + } + if ($needsToken) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } } } if ($env:GITHUB_TOKEN) { @@ -510,28 +585,64 @@ anvil-container *target: $runArgs += @('-e', 'GITHUB_TOKEN') } + # The recipe contract's own inputs. These are read by generated checks -- + # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its + # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so + # dropping them at the boundary makes the same command mean different things + # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it + # exits 0 with a skip notice, so a title a native run rejects passes in a + # container and the tier still reports green. + # + # Forwarded by name and only when set, so an unset variable stays unset + # rather than arriving as an empty string, which several of these treat as + # a value. + foreach ($name in @( + 'PR_TITLE', + 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', + 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { + $forwardedEnv += $name + $runArgs += @('-e', $name) + } + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # credentials fails inside the container in a far less obvious way. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") - $hook = Anvil-PreRun - if ($null -ne $hook -and $null -ne $hook.Env) { - foreach ($name in $hook.Env.Keys) { - if ([string]::IsNullOrEmpty($hook.Env[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" - exit 1 - } - Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] - $hookEnv += $name - $forwardedEnv += $name - $runArgs += @('-e', $name) + try { + $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + exit 1 + } + $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } + if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + exit 1 + } + foreach ($name in $hookVars.Keys) { + if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 } - # Names only, never values: a hook with a broad idea of what to - # forward should be visible, since everything inside the - # container can read it -- including third-party build scripts. - [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] + $hookEnv += $name + $forwardedEnv += $name + $runArgs += @('-e', $name) } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hookVars.Keys -join ', ')") } } @@ -629,7 +740,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 2561b492..ef8465d2 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -97,12 +97,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, but -# only `tools.just` and `versions.just` decide what this layer installs, and -# only those are hashed into the tag -- so editing a check does not rebuild the -# image, and the check runs from the bind mount anyway. The synthetic Justfile -# below imports only the anvil tree, avoiding repository-specific imports that -# may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -113,6 +112,13 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # *writes* with it is ordinary content -- and the `chmod` on the next line # would otherwise publish it world-readable. Deleting them in a later `RUN` # would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -120,6 +126,7 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && just anvil-setup binstall \ && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" # Consumed by `anvil-container` itself: a nested invocation from inside the @@ -3609,11 +3616,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the tool catalog -# (`tools.just` and `versions.just`) that decides what `just anvil-setup` -# installs. The tier, group and check recipes are not inputs: they only route -# into the catalog, and they execute from the bind mount rather than from the -# image, so editing one takes effect on the next run without a rebuild. +# ignore file, the pinned toolchain, the optional hook, and the whole generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes alike. Editing any of them can change what the image contains, so +# any of them can rename it. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -3637,14 +3644,24 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - # The tool catalog, and nothing else. `just anvil-setup` installs the whole - # catalog, and every install recipe is defined in tools.just against a pin - # in versions.just -- so a tool cannot be added, removed or repinned - # without one of these two changing. The tier, group and check recipes only - # route into them, and are read from the bind mount at run time rather than - # from the image, so editing a check must not cost a rebuild. - $inputs += 'justfiles/anvil/tools.just' - $inputs += 'justfiles/anvil/versions.just' + # Every generated recipe file. The image installs its tools by running + # `just anvil-setup`, and that dependency chain runs through the tier, + # group and check recipes before it reaches the install recipes in + # tools.just -- so the routing decides *whether* a tool is installed just + # as surely as tools.just decides *how*. Hashing only the install + # definitions would let a group drop a `-setup` dependency, changing the + # installed set, without renaming the image. + # + # This driver is included too. It is not circular -- the tag is derived + # from file text, and no file contains the tag -- and it belongs in the set + # because it passes the build arguments, the secret mounts and the hook's + # `Anvil-PreBuild` output into the build, all of which shape the result. + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -3728,32 +3745,40 @@ _anvil-container-image: # with a change must not stop the developer who made it. $hookPath = Join-Path $repoRoot $hookRel if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { - . $hookPath - if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") - $resolved = $null - try { + # Dot-sourcing is inside the try as well: a hook with a syntax error, + # or one that throws while being loaded, must cost no more than a + # hook that resolves nothing. The recipe runs under + # `$ErrorActionPreference = 'Stop'`, so leaving the load outside + # would abort the run instead of falling through to a build. + $resolved = $null + try { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 - } catch { - [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") - $resolved = $null } - if (-not [string]::IsNullOrWhiteSpace($resolved)) { - $resolved = ([string]$resolved).Trim() - # Verify rather than trust: the run is `--pull=never`, so a - # reference the hook reported but did not actually fetch - # would fail later, further from the cause. - & $engineExe @enginePrefix image inspect $resolved *> $null - if ($LASTEXITCODE -eq 0) { - [Console]::Error.WriteLine("anvil: resolved $resolved") - Write-Output $resolved - exit 0 - } - [Console]::Error.WriteLine( - "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } catch { + [Console]::Error.WriteLine("anvil: $hookRel failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # A presence check, not a verification: `image inspect` proves + # something is tagged with that reference, not that its contents + # match the digest the tag claims. Trusting the hook is the + # contract -- this only keeps a reference the hook reported but + # never fetched from failing later, under `--pull=never`, a long + # way from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 } - [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") } } @@ -3781,23 +3806,49 @@ _anvil-container-image: $secretEnv = @() $hookPath = Join-Path $repoRoot $hookRel if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed, unlike the resolve hook: a build that cannot mint its + # credentials must stop, not proceed to produce a reduced image. The + # try exists only so the cause is named -- loading a hook with a syntax + # error would otherwise surface as a bare parser error with no hint + # that a hook was involved. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") - $hook = Anvil-PreBuild - if ($null -ne $hook -and $null -ne $hook.Secrets) { - foreach ($id in $hook.Secrets.Keys) { - if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" - exit 1 - } - $name = "ANVIL_SECRET_$id" - Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] - $secretEnv += $name - $secretArgs += "id=$id,env=$name" + try { + # Take the last emitted object, not the whole stream: a hook + # that writes progress with `Write-Output` would otherwise hand + # back an array whose `.Secrets` is silently $null. + $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + exit 1 + } + $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } + # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # guard exists for, not a hook opting out: secrets are the only + # thing the phase can contribute, so an empty return means the mint + # failed quietly. A hook with no build-time credentials simply does + # not define the function. + if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + exit 1 + } + foreach ($id in $secrets.Keys) { + if ([string]::IsNullOrWhiteSpace($secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 } - [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" } + [Console]::Error.WriteLine("anvil: build secrets: $($secrets.Keys -join ', ')") } } @@ -3891,8 +3942,17 @@ anvil-container *target: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so - # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' + # relative paths keep working from a subdirectory. `invocation_directory()` + # would be wrong here: with `cygpath` on PATH it reports a Cygwin-style + # path, which shares no prefix with the native `justfile_directory()` above. + # `GetRelativePath` then walks out with `..` segments and the run is placed + # outside the mount, so it fails on a path that does not exist in the + # container. The `_native` form is the one that agrees with the root. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory_native(), "'", "''") }}') -replace '\\', '/' + if ($rel.StartsWith('..')) { + Write-Error "anvil: run this from inside the repository; $rel is outside $repoRoot" + exit 1 + } $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { @@ -3989,22 +4049,44 @@ anvil-container *target: # # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN # first, then the gh CLI's stored token -- so a containerized run - # authenticates for the same developers a native run does. Deriving it here - # rather than only forwarding an exported value is deliberate: the two - # paths have identical exposure once inside (a forwarded token is readable - # by every recipe in the container either way, including third-party build - # scripts and proc macros), so refusing to derive it buys no boundary and - # only makes behaviour depend on how the developer happened to sign in. + # authenticates for the same developers a native run does. + # + # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # that is exact parity, since a native run exposes it to every process the + # shell spawns too. Deriving one from `gh` is different -- it manufactures a + # credential the developer did not put in this environment, and PID 1's + # environment is inherited by every build script and proc macro in the + # container, where natively the recipe would mint it in its own process. So + # it is derived only when the target actually reads the variable, or when + # there is no target at all: an interactive session can run anything, and + # refusing there would reintroduce the silent hour-long stall on a tier the + # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. # + # The predicate is the variable itself rather than the name of a check, so + # the driver stays generic: a catalog that adds another GitHub-authenticated + # check is covered without touching this recipe. + # # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - $ghToken = $null - try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } - if ($ghToken -and $ghToken.Trim()) { - Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() - $hookEnv += 'GITHUB_TOKEN' + # Ask just what the target would run. A dry run has no side effects, and + # a target that cannot be planned (a typo, a recipe needing arguments) + # yields nothing, so the run fails on its own terms rather than on a + # missing token. + $needsToken = $targetParts.Count -eq 0 + if (-not $needsToken) { + $plan = '' + try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + $needsToken = $plan -match 'GITHUB_TOKEN' + } + if ($needsToken) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } } } if ($env:GITHUB_TOKEN) { @@ -4012,28 +4094,64 @@ anvil-container *target: $runArgs += @('-e', 'GITHUB_TOKEN') } + # The recipe contract's own inputs. These are read by generated checks -- + # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its + # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so + # dropping them at the boundary makes the same command mean different things + # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it + # exits 0 with a skip notice, so a title a native run rejects passes in a + # container and the tier still reports green. + # + # Forwarded by name and only when set, so an unset variable stays unset + # rather than arriving as an empty string, which several of these treat as + # a value. + foreach ($name in @( + 'PR_TITLE', + 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', + 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { + $forwardedEnv += $name + $runArgs += @('-e', $name) + } + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # credentials fails inside the container in a far less obvious way. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") - $hook = Anvil-PreRun - if ($null -ne $hook -and $null -ne $hook.Env) { - foreach ($name in $hook.Env.Keys) { - if ([string]::IsNullOrEmpty($hook.Env[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" - exit 1 - } - Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] - $hookEnv += $name - $forwardedEnv += $name - $runArgs += @('-e', $name) + try { + $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + exit 1 + } + $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } + if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + exit 1 + } + foreach ($name in $hookVars.Keys) { + if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 } - # Names only, never values: a hook with a broad idea of what to - # forward should be visible, since everything inside the - # container can read it -- including third-party build scripts. - [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] + $hookEnv += $name + $forwardedEnv += $name + $runArgs += @('-e', $name) } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hookVars.Keys -join ', ')") } } @@ -4131,7 +4249,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 03bd7744..5093aad5 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -97,12 +97,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, but -# only `tools.just` and `versions.just` decide what this layer installs, and -# only those are hashed into the tag -- so editing a check does not rebuild the -# image, and the check runs from the bind mount anyway. The synthetic Justfile -# below imports only the anvil tree, avoiding repository-specific imports that -# may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -113,6 +112,13 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # *writes* with it is ordinary content -- and the `chmod` on the next line # would otherwise publish it world-readable. Deleting them in a later `RUN` # would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -120,6 +126,7 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && just anvil-setup binstall \ && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" # Consumed by `anvil-container` itself: a nested invocation from inside the @@ -3530,11 +3537,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the tool catalog -# (`tools.just` and `versions.just`) that decides what `just anvil-setup` -# installs. The tier, group and check recipes are not inputs: they only route -# into the catalog, and they execute from the bind mount rather than from the -# image, so editing one takes effect on the next run without a rebuild. +# ignore file, the pinned toolchain, the optional hook, and the whole generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes alike. Editing any of them can change what the image contains, so +# any of them can rename it. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -3558,14 +3565,24 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - # The tool catalog, and nothing else. `just anvil-setup` installs the whole - # catalog, and every install recipe is defined in tools.just against a pin - # in versions.just -- so a tool cannot be added, removed or repinned - # without one of these two changing. The tier, group and check recipes only - # route into them, and are read from the bind mount at run time rather than - # from the image, so editing a check must not cost a rebuild. - $inputs += 'justfiles/anvil/tools.just' - $inputs += 'justfiles/anvil/versions.just' + # Every generated recipe file. The image installs its tools by running + # `just anvil-setup`, and that dependency chain runs through the tier, + # group and check recipes before it reaches the install recipes in + # tools.just -- so the routing decides *whether* a tool is installed just + # as surely as tools.just decides *how*. Hashing only the install + # definitions would let a group drop a `-setup` dependency, changing the + # installed set, without renaming the image. + # + # This driver is included too. It is not circular -- the tag is derived + # from file text, and no file contains the tag -- and it belongs in the set + # because it passes the build arguments, the secret mounts and the hook's + # `Anvil-PreBuild` output into the build, all of which shape the result. + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -3649,32 +3666,40 @@ _anvil-container-image: # with a change must not stop the developer who made it. $hookPath = Join-Path $repoRoot $hookRel if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { - . $hookPath - if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") - $resolved = $null - try { + # Dot-sourcing is inside the try as well: a hook with a syntax error, + # or one that throws while being loaded, must cost no more than a + # hook that resolves nothing. The recipe runs under + # `$ErrorActionPreference = 'Stop'`, so leaving the load outside + # would abort the run instead of falling through to a build. + $resolved = $null + try { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 - } catch { - [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") - $resolved = $null } - if (-not [string]::IsNullOrWhiteSpace($resolved)) { - $resolved = ([string]$resolved).Trim() - # Verify rather than trust: the run is `--pull=never`, so a - # reference the hook reported but did not actually fetch - # would fail later, further from the cause. - & $engineExe @enginePrefix image inspect $resolved *> $null - if ($LASTEXITCODE -eq 0) { - [Console]::Error.WriteLine("anvil: resolved $resolved") - Write-Output $resolved - exit 0 - } - [Console]::Error.WriteLine( - "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } catch { + [Console]::Error.WriteLine("anvil: $hookRel failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # A presence check, not a verification: `image inspect` proves + # something is tagged with that reference, not that its contents + # match the digest the tag claims. Trusting the hook is the + # contract -- this only keeps a reference the hook reported but + # never fetched from failing later, under `--pull=never`, a long + # way from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 } - [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") } } @@ -3702,23 +3727,49 @@ _anvil-container-image: $secretEnv = @() $hookPath = Join-Path $repoRoot $hookRel if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed, unlike the resolve hook: a build that cannot mint its + # credentials must stop, not proceed to produce a reduced image. The + # try exists only so the cause is named -- loading a hook with a syntax + # error would otherwise surface as a bare parser error with no hint + # that a hook was involved. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") - $hook = Anvil-PreBuild - if ($null -ne $hook -and $null -ne $hook.Secrets) { - foreach ($id in $hook.Secrets.Keys) { - if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" - exit 1 - } - $name = "ANVIL_SECRET_$id" - Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] - $secretEnv += $name - $secretArgs += "id=$id,env=$name" + try { + # Take the last emitted object, not the whole stream: a hook + # that writes progress with `Write-Output` would otherwise hand + # back an array whose `.Secrets` is silently $null. + $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + exit 1 + } + $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } + # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # guard exists for, not a hook opting out: secrets are the only + # thing the phase can contribute, so an empty return means the mint + # failed quietly. A hook with no build-time credentials simply does + # not define the function. + if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + exit 1 + } + foreach ($id in $secrets.Keys) { + if ([string]::IsNullOrWhiteSpace($secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 } - [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" } + [Console]::Error.WriteLine("anvil: build secrets: $($secrets.Keys -join ', ')") } } @@ -3812,8 +3863,17 @@ anvil-container *target: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so - # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' + # relative paths keep working from a subdirectory. `invocation_directory()` + # would be wrong here: with `cygpath` on PATH it reports a Cygwin-style + # path, which shares no prefix with the native `justfile_directory()` above. + # `GetRelativePath` then walks out with `..` segments and the run is placed + # outside the mount, so it fails on a path that does not exist in the + # container. The `_native` form is the one that agrees with the root. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory_native(), "'", "''") }}') -replace '\\', '/' + if ($rel.StartsWith('..')) { + Write-Error "anvil: run this from inside the repository; $rel is outside $repoRoot" + exit 1 + } $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { @@ -3910,22 +3970,44 @@ anvil-container *target: # # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN # first, then the gh CLI's stored token -- so a containerized run - # authenticates for the same developers a native run does. Deriving it here - # rather than only forwarding an exported value is deliberate: the two - # paths have identical exposure once inside (a forwarded token is readable - # by every recipe in the container either way, including third-party build - # scripts and proc macros), so refusing to derive it buys no boundary and - # only makes behaviour depend on how the developer happened to sign in. + # authenticates for the same developers a native run does. + # + # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # that is exact parity, since a native run exposes it to every process the + # shell spawns too. Deriving one from `gh` is different -- it manufactures a + # credential the developer did not put in this environment, and PID 1's + # environment is inherited by every build script and proc macro in the + # container, where natively the recipe would mint it in its own process. So + # it is derived only when the target actually reads the variable, or when + # there is no target at all: an interactive session can run anything, and + # refusing there would reintroduce the silent hour-long stall on a tier the + # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. # + # The predicate is the variable itself rather than the name of a check, so + # the driver stays generic: a catalog that adds another GitHub-authenticated + # check is covered without touching this recipe. + # # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - $ghToken = $null - try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } - if ($ghToken -and $ghToken.Trim()) { - Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() - $hookEnv += 'GITHUB_TOKEN' + # Ask just what the target would run. A dry run has no side effects, and + # a target that cannot be planned (a typo, a recipe needing arguments) + # yields nothing, so the run fails on its own terms rather than on a + # missing token. + $needsToken = $targetParts.Count -eq 0 + if (-not $needsToken) { + $plan = '' + try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + $needsToken = $plan -match 'GITHUB_TOKEN' + } + if ($needsToken) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } } } if ($env:GITHUB_TOKEN) { @@ -3933,28 +4015,64 @@ anvil-container *target: $runArgs += @('-e', 'GITHUB_TOKEN') } + # The recipe contract's own inputs. These are read by generated checks -- + # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its + # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so + # dropping them at the boundary makes the same command mean different things + # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it + # exits 0 with a skip notice, so a title a native run rejects passes in a + # container and the tier still reports green. + # + # Forwarded by name and only when set, so an unset variable stays unset + # rather than arriving as an empty string, which several of these treat as + # a value. + foreach ($name in @( + 'PR_TITLE', + 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', + 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { + $forwardedEnv += $name + $runArgs += @('-e', $name) + } + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # credentials fails inside the container in a far less obvious way. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") - $hook = Anvil-PreRun - if ($null -ne $hook -and $null -ne $hook.Env) { - foreach ($name in $hook.Env.Keys) { - if ([string]::IsNullOrEmpty($hook.Env[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" - exit 1 - } - Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] - $hookEnv += $name - $forwardedEnv += $name - $runArgs += @('-e', $name) + try { + $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + exit 1 + } + $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } + if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + exit 1 + } + foreach ($name in $hookVars.Keys) { + if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 } - # Names only, never values: a hook with a broad idea of what to - # forward should be visible, since everything inside the - # container can read it -- including third-party build scripts. - [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] + $hookEnv += $name + $forwardedEnv += $name + $runArgs += @('-e', $name) } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hookVars.Keys -join ', ')") } } @@ -4052,7 +4170,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 659e4bd7..cd65073e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -97,12 +97,11 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && rm /tmp/cargo-binstall.tgz # Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, but -# only `tools.just` and `versions.just` decide what this layer installs, and -# only those are hashed into the tag -- so editing a check does not rebuild the -# image, and the check runs from the bind mount anyway. The synthetic Justfile -# below imports only the anvil tree, avoiding repository-specific imports that -# may not exist yet. +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. # # `binstall` matches what CI passes. It is not just a speed-up: some catalog # tools do not compile from source on every pinned toolchain, so a source @@ -113,6 +112,13 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # *writes* with it is ordinary content -- and the `chmod` on the next line # would otherwise publish it world-readable. Deleting them in a later `RUN` # would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -120,6 +126,7 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && just anvil-setup binstall \ && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" # Consumed by `anvil-container` itself: a nested invocation from inside the @@ -2349,11 +2356,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the tool catalog -# (`tools.just` and `versions.just`) that decides what `just anvil-setup` -# installs. The tier, group and check recipes are not inputs: they only route -# into the catalog, and they execute from the bind mount rather than from the -# image, so editing one takes effect on the next run without a rebuild. +# ignore file, the pinned toolchain, the optional hook, and the whole generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes alike. Editing any of them can change what the image contains, so +# any of them can rename it. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -2377,14 +2384,24 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - # The tool catalog, and nothing else. `just anvil-setup` installs the whole - # catalog, and every install recipe is defined in tools.just against a pin - # in versions.just -- so a tool cannot be added, removed or repinned - # without one of these two changing. The tier, group and check recipes only - # route into them, and are read from the bind mount at run time rather than - # from the image, so editing a check must not cost a rebuild. - $inputs += 'justfiles/anvil/tools.just' - $inputs += 'justfiles/anvil/versions.just' + # Every generated recipe file. The image installs its tools by running + # `just anvil-setup`, and that dependency chain runs through the tier, + # group and check recipes before it reaches the install recipes in + # tools.just -- so the routing decides *whether* a tool is installed just + # as surely as tools.just decides *how*. Hashing only the install + # definitions would let a group drop a `-setup` dependency, changing the + # installed set, without renaming the image. + # + # This driver is included too. It is not circular -- the tag is derived + # from file text, and no file contains the tag -- and it belongs in the set + # because it passes the build arguments, the secret mounts and the hook's + # `Anvil-PreBuild` output into the build, all of which shape the result. + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -2468,32 +2485,40 @@ _anvil-container-image: # with a change must not stop the developer who made it. $hookPath = Join-Path $repoRoot $hookRel if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { - . $hookPath - if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") - $resolved = $null - try { + # Dot-sourcing is inside the try as well: a hook with a syntax error, + # or one that throws while being loaded, must cost no more than a + # hook that resolves nothing. The recipe runs under + # `$ErrorActionPreference = 'Stop'`, so leaving the load outside + # would abort the run instead of falling through to a build. + $resolved = $null + try { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 - } catch { - [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") - $resolved = $null } - if (-not [string]::IsNullOrWhiteSpace($resolved)) { - $resolved = ([string]$resolved).Trim() - # Verify rather than trust: the run is `--pull=never`, so a - # reference the hook reported but did not actually fetch - # would fail later, further from the cause. - & $engineExe @enginePrefix image inspect $resolved *> $null - if ($LASTEXITCODE -eq 0) { - [Console]::Error.WriteLine("anvil: resolved $resolved") - Write-Output $resolved - exit 0 - } - [Console]::Error.WriteLine( - "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } catch { + [Console]::Error.WriteLine("anvil: $hookRel failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # A presence check, not a verification: `image inspect` proves + # something is tagged with that reference, not that its contents + # match the digest the tag claims. Trusting the hook is the + # contract -- this only keeps a reference the hook reported but + # never fetched from failing later, under `--pull=never`, a long + # way from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 } - [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") } } @@ -2521,23 +2546,49 @@ _anvil-container-image: $secretEnv = @() $hookPath = Join-Path $repoRoot $hookRel if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed, unlike the resolve hook: a build that cannot mint its + # credentials must stop, not proceed to produce a reduced image. The + # try exists only so the cause is named -- loading a hook with a syntax + # error would otherwise surface as a bare parser error with no hint + # that a hook was involved. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") - $hook = Anvil-PreBuild - if ($null -ne $hook -and $null -ne $hook.Secrets) { - foreach ($id in $hook.Secrets.Keys) { - if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" - exit 1 - } - $name = "ANVIL_SECRET_$id" - Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] - $secretEnv += $name - $secretArgs += "id=$id,env=$name" + try { + # Take the last emitted object, not the whole stream: a hook + # that writes progress with `Write-Output` would otherwise hand + # back an array whose `.Secrets` is silently $null. + $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + exit 1 + } + $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } + # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # guard exists for, not a hook opting out: secrets are the only + # thing the phase can contribute, so an empty return means the mint + # failed quietly. A hook with no build-time credentials simply does + # not define the function. + if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + exit 1 + } + foreach ($id in $secrets.Keys) { + if ([string]::IsNullOrWhiteSpace($secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 } - [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" } + [Console]::Error.WriteLine("anvil: build secrets: $($secrets.Keys -join ', ')") } } @@ -2631,8 +2682,17 @@ anvil-container *target: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so - # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' + # relative paths keep working from a subdirectory. `invocation_directory()` + # would be wrong here: with `cygpath` on PATH it reports a Cygwin-style + # path, which shares no prefix with the native `justfile_directory()` above. + # `GetRelativePath` then walks out with `..` segments and the run is placed + # outside the mount, so it fails on a path that does not exist in the + # container. The `_native` form is the one that agrees with the root. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory_native(), "'", "''") }}') -replace '\\', '/' + if ($rel.StartsWith('..')) { + Write-Error "anvil: run this from inside the repository; $rel is outside $repoRoot" + exit 1 + } $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { @@ -2729,22 +2789,44 @@ anvil-container *target: # # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN # first, then the gh CLI's stored token -- so a containerized run - # authenticates for the same developers a native run does. Deriving it here - # rather than only forwarding an exported value is deliberate: the two - # paths have identical exposure once inside (a forwarded token is readable - # by every recipe in the container either way, including third-party build - # scripts and proc macros), so refusing to derive it buys no boundary and - # only makes behaviour depend on how the developer happened to sign in. + # authenticates for the same developers a native run does. + # + # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # that is exact parity, since a native run exposes it to every process the + # shell spawns too. Deriving one from `gh` is different -- it manufactures a + # credential the developer did not put in this environment, and PID 1's + # environment is inherited by every build script and proc macro in the + # container, where natively the recipe would mint it in its own process. So + # it is derived only when the target actually reads the variable, or when + # there is no target at all: an interactive session can run anything, and + # refusing there would reintroduce the silent hour-long stall on a tier the + # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. # + # The predicate is the variable itself rather than the name of a check, so + # the driver stays generic: a catalog that adds another GitHub-authenticated + # check is covered without touching this recipe. + # # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - $ghToken = $null - try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } - if ($ghToken -and $ghToken.Trim()) { - Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() - $hookEnv += 'GITHUB_TOKEN' + # Ask just what the target would run. A dry run has no side effects, and + # a target that cannot be planned (a typo, a recipe needing arguments) + # yields nothing, so the run fails on its own terms rather than on a + # missing token. + $needsToken = $targetParts.Count -eq 0 + if (-not $needsToken) { + $plan = '' + try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + $needsToken = $plan -match 'GITHUB_TOKEN' + } + if ($needsToken) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } } } if ($env:GITHUB_TOKEN) { @@ -2752,28 +2834,64 @@ anvil-container *target: $runArgs += @('-e', 'GITHUB_TOKEN') } + # The recipe contract's own inputs. These are read by generated checks -- + # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its + # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so + # dropping them at the boundary makes the same command mean different things + # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it + # exits 0 with a skip notice, so a title a native run rejects passes in a + # container and the tier still reports green. + # + # Forwarded by name and only when set, so an unset variable stays unset + # rather than arriving as an empty string, which several of these treat as + # a value. + foreach ($name in @( + 'PR_TITLE', + 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', + 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { + $forwardedEnv += $name + $runArgs += @('-e', $name) + } + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # credentials fails inside the container in a far less obvious way. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") - $hook = Anvil-PreRun - if ($null -ne $hook -and $null -ne $hook.Env) { - foreach ($name in $hook.Env.Keys) { - if ([string]::IsNullOrEmpty($hook.Env[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" - exit 1 - } - Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] - $hookEnv += $name - $forwardedEnv += $name - $runArgs += @('-e', $name) + try { + $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + exit 1 + } + $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } + if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + exit 1 + } + foreach ($name in $hookVars.Keys) { + if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 } - # Names only, never values: a hook with a broad idea of what to - # forward should be visible, since everything inside the - # container can read it -- including third-party build scripts. - [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] + $hookEnv += $name + $forwardedEnv += $name + $runArgs += @('-e', $name) } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hookVars.Keys -join ', ')") } } @@ -2871,7 +2989,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 301410d7..c65e6ca5 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -107,11 +107,11 @@ _anvil-container-path host_path: # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its -# ignore file, the pinned toolchain, the optional hook, and the tool catalog -# (`tools.just` and `versions.just`) that decides what `just anvil-setup` -# installs. The tier, group and check recipes are not inputs: they only route -# into the catalog, and they execute from the bind mount rather than from the -# image, so editing one takes effect on the next run without a rebuild. +# ignore file, the pinned toolchain, the optional hook, and the whole generated +# recipe tree -- because the image installs its tools by running +# `just anvil-setup`, whose dependency chain reaches the tier, group, check and +# tool recipes alike. Editing any of them can change what the image contains, so +# any of them can rename it. # # This is the only recipe that computes the reference; everything else asks it. # It is public because a publisher needs the tag before there is an image to @@ -135,14 +135,24 @@ anvil-container-tag: # never hashed: a credential must not influence a tag. $inputs += $hookRel } - # The tool catalog, and nothing else. `just anvil-setup` installs the whole - # catalog, and every install recipe is defined in tools.just against a pin - # in versions.just -- so a tool cannot be added, removed or repinned - # without one of these two changing. The tier, group and check recipes only - # route into them, and are read from the bind mount at run time rather than - # from the image, so editing a check must not cost a rebuild. - $inputs += 'justfiles/anvil/tools.just' - $inputs += 'justfiles/anvil/versions.just' + # Every generated recipe file. The image installs its tools by running + # `just anvil-setup`, and that dependency chain runs through the tier, + # group and check recipes before it reaches the install recipes in + # tools.just -- so the routing decides *whether* a tool is installed just + # as surely as tools.just decides *how*. Hashing only the install + # definitions would let a group drop a `-setup` dependency, changing the + # installed set, without renaming the image. + # + # This driver is included too. It is not circular -- the tag is derived + # from file text, and no file contains the tag -- and it belongs in the set + # because it passes the build arguments, the secret mounts and the hook's + # `Anvil-PreBuild` output into the build, all of which shape the result. + $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' + if (Test-Path -LiteralPath $recipeRoot) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so @@ -226,32 +236,40 @@ _anvil-container-image: # with a change must not stop the developer who made it. $hookPath = Join-Path $repoRoot $hookRel if ($env:ANVIL_CONTAINER_NO_RESOLVE -ne '1' -and (Test-Path -LiteralPath $hookPath -PathType Leaf)) { - . $hookPath - if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") - $resolved = $null - try { + # Dot-sourcing is inside the try as well: a hook with a syntax error, + # or one that throws while being loaded, must cost no more than a + # hook that resolves nothing. The recipe runs under + # `$ErrorActionPreference = 'Stop'`, so leaving the load outside + # would abort the run instead of falling through to a build. + $resolved = $null + try { + . $hookPath + if (Get-Command Anvil-ResolveImage -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-ResolveImage from $hookRel") $resolved = @(Anvil-ResolveImage $image | Where-Object { $_ }) | Select-Object -Last 1 - } catch { - [Console]::Error.WriteLine("anvil: Anvil-ResolveImage failed: $($_.Exception.Message)") - $resolved = $null } - if (-not [string]::IsNullOrWhiteSpace($resolved)) { - $resolved = ([string]$resolved).Trim() - # Verify rather than trust: the run is `--pull=never`, so a - # reference the hook reported but did not actually fetch - # would fail later, further from the cause. - & $engineExe @enginePrefix image inspect $resolved *> $null - if ($LASTEXITCODE -eq 0) { - [Console]::Error.WriteLine("anvil: resolved $resolved") - Write-Output $resolved - exit 0 - } - [Console]::Error.WriteLine( - "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") + } catch { + [Console]::Error.WriteLine("anvil: $hookRel failed: $($_.Exception.Message)") + $resolved = $null + } + if (-not [string]::IsNullOrWhiteSpace($resolved)) { + $resolved = ([string]$resolved).Trim() + # A presence check, not a verification: `image inspect` proves + # something is tagged with that reference, not that its contents + # match the digest the tag claims. Trusting the hook is the + # contract -- this only keeps a reference the hook reported but + # never fetched from failing later, under `--pull=never`, a long + # way from the cause. + & $engineExe @enginePrefix image inspect $resolved *> $null + if ($LASTEXITCODE -eq 0) { + [Console]::Error.WriteLine("anvil: resolved $resolved") + Write-Output $resolved + exit 0 } - [Console]::Error.WriteLine("anvil: nothing resolved; building locally") + [Console]::Error.WriteLine( + "anvil: Anvil-ResolveImage reported '$resolved' but no such image is present locally") } + [Console]::Error.WriteLine("anvil: nothing resolved; building locally") } } @@ -279,23 +297,49 @@ _anvil-container-image: $secretEnv = @() $hookPath = Join-Path $repoRoot $hookRel if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed, unlike the resolve hook: a build that cannot mint its + # credentials must stop, not proceed to produce a reduced image. The + # try exists only so the cause is named -- loading a hook with a syntax + # error would otherwise surface as a bare parser error with no hint + # that a hook was involved. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") - $hook = Anvil-PreBuild - if ($null -ne $hook -and $null -ne $hook.Secrets) { - foreach ($id in $hook.Secrets.Keys) { - if ([string]::IsNullOrEmpty($hook.Secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" - exit 1 - } - $name = "ANVIL_SECRET_$id" - Set-Item -LiteralPath "Env:$name" -Value $hook.Secrets[$id] - $secretEnv += $name - $secretArgs += "id=$id,env=$name" + try { + # Take the last emitted object, not the whole stream: a hook + # that writes progress with `Write-Output` would otherwise hand + # back an array whose `.Secrets` is silently $null. + $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + exit 1 + } + $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } + # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # guard exists for, not a hook opting out: secrets are the only + # thing the phase can contribute, so an empty return means the mint + # failed quietly. A hook with no build-time credentials simply does + # not define the function. + if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + exit 1 + } + foreach ($id in $secrets.Keys) { + if ([string]::IsNullOrWhiteSpace($secrets[$id])) { + Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + exit 1 } - [Console]::Error.WriteLine("anvil: build secrets: $($hook.Secrets.Keys -join ', ')") + $name = "ANVIL_SECRET_$id" + Set-Item -LiteralPath "Env:$name" -Value $secrets[$id] + $secretEnv += $name + $secretArgs += "id=$id,env=$name" } + [Console]::Error.WriteLine("anvil: build secrets: $($secrets.Keys -join ', ')") } } @@ -389,8 +433,17 @@ anvil-container *target: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } # Map the caller's working directory to its in-container equivalent so - # relative paths keep working from a subdirectory. - $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory(), "'", "''") }}') -replace '\\', '/' + # relative paths keep working from a subdirectory. `invocation_directory()` + # would be wrong here: with `cygpath` on PATH it reports a Cygwin-style + # path, which shares no prefix with the native `justfile_directory()` above. + # `GetRelativePath` then walks out with `..` segments and the run is placed + # outside the mount, so it fails on a path that does not exist in the + # container. The `_native` form is the one that agrees with the root. + $rel = [System.IO.Path]::GetRelativePath($repoRoot, '{{ replace(invocation_directory_native(), "'", "''") }}') -replace '\\', '/' + if ($rel.StartsWith('..')) { + Write-Error "anvil: run this from inside the repository; $rel is outside $repoRoot" + exit 1 + } $containerCwd = if ($rel -eq '.' -or [string]::IsNullOrEmpty($rel)) { '{{anvil_container_workdir}}' } else { @@ -487,22 +540,44 @@ anvil-container *target: # # Resolve the token exactly as the recipe does natively -- GITHUB_TOKEN # first, then the gh CLI's stored token -- so a containerized run - # authenticates for the same developers a native run does. Deriving it here - # rather than only forwarding an exported value is deliberate: the two - # paths have identical exposure once inside (a forwarded token is readable - # by every recipe in the container either way, including third-party build - # scripts and proc macros), so refusing to derive it buys no boundary and - # only makes behaviour depend on how the developer happened to sign in. + # authenticates for the same developers a native run does. + # + # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # that is exact parity, since a native run exposes it to every process the + # shell spawns too. Deriving one from `gh` is different -- it manufactures a + # credential the developer did not put in this environment, and PID 1's + # environment is inherited by every build script and proc macro in the + # container, where natively the recipe would mint it in its own process. So + # it is derived only when the target actually reads the variable, or when + # there is no target at all: an interactive session can run anything, and + # refusing there would reintroduce the silent hour-long stall on a tier the + # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. # + # The predicate is the variable itself rather than the name of a check, so + # the driver stays generic: a catalog that adds another GitHub-authenticated + # check is covered without touching this recipe. + # # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - $ghToken = $null - try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } - if ($ghToken -and $ghToken.Trim()) { - Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() - $hookEnv += 'GITHUB_TOKEN' + # Ask just what the target would run. A dry run has no side effects, and + # a target that cannot be planned (a typo, a recipe needing arguments) + # yields nothing, so the run fails on its own terms rather than on a + # missing token. + $needsToken = $targetParts.Count -eq 0 + if (-not $needsToken) { + $plan = '' + try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + $needsToken = $plan -match 'GITHUB_TOKEN' + } + if ($needsToken) { + $ghToken = $null + try { $ghToken = (gh auth token --hostname github.com 2>$null) } catch { $ghToken = $null } + if ($ghToken -and $ghToken.Trim()) { + Set-Item -LiteralPath 'Env:GITHUB_TOKEN' -Value $ghToken.Trim() + $hookEnv += 'GITHUB_TOKEN' + } } } if ($env:GITHUB_TOKEN) { @@ -510,28 +585,64 @@ anvil-container *target: $runArgs += @('-e', 'GITHUB_TOKEN') } + # The recipe contract's own inputs. These are read by generated checks -- + # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its + # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so + # dropping them at the boundary makes the same command mean different things + # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it + # exits 0 with a skip notice, so a title a native run rejects passes in a + # container and the tier still reports green. + # + # Forwarded by name and only when set, so an unset variable stays unset + # rather than arriving as an empty string, which several of these treat as + # a value. + foreach ($name in @( + 'PR_TITLE', + 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', + 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { + $forwardedEnv += $name + $runArgs += @('-e', $name) + } + } + $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - . $hookPath + # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # credentials fails inside the container in a far less obvious way. + try { + . $hookPath + } catch { + Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" + exit 1 + } if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") - $hook = Anvil-PreRun - if ($null -ne $hook -and $null -ne $hook.Env) { - foreach ($name in $hook.Env.Keys) { - if ([string]::IsNullOrEmpty($hook.Env[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" - exit 1 - } - Set-Item -LiteralPath "Env:$name" -Value $hook.Env[$name] - $hookEnv += $name - $forwardedEnv += $name - $runArgs += @('-e', $name) + try { + $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + } catch { + Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + exit 1 + } + $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } + if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { + Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + exit 1 + } + foreach ($name in $hookVars.Keys) { + if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { + Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + exit 1 } - # Names only, never values: a hook with a broad idea of what to - # forward should be visible, since everything inside the - # container can read it -- including third-party build scripts. - [Console]::Error.WriteLine("anvil: forwarding env: $($hook.Env.Keys -join ', ')") + Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] + $hookEnv += $name + $forwardedEnv += $name + $runArgs += @('-e', $name) } + # Names only, never values: a hook with a broad idea of what to + # forward should be visible, since everything inside the + # container can read it -- including third-party build scripts. + [Console]::Error.WriteLine("anvil: forwarding env: $($hookVars.Keys -join ', ')") } } @@ -629,7 +740,7 @@ anvil-container-down: # be able to tell that it failed. `-f` already exits 0 for a volume that # does not exist, so this cannot fire spuriously. $failed = @() - foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git', '{{anvil_container_name}}-cargo', '{{anvil_container_name}}-rustup')) { + foreach ($vol in @('{{anvil_container_name}}-cargo-registry', '{{anvil_container_name}}-cargo-git')) { & $engineExe @enginePrefix volume rm -f $vol if ($LASTEXITCODE -ne 0) { $failed += $vol } } diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index 9c207805..638a50e1 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -333,6 +333,14 @@ e2e-show-env: e2e-show-token: @echo "E2E-TOKEN:[${GITHUB_TOKEN:-}]" +# The negative case for the same rule. A derived token is minted only when the +# target's plan reads GITHUB_TOKEN, so this recipe must observe the environment +# *without naming the variable* -- naming it is what would opt it in. Dumping +# every name lets the assertion look for the value without the plan mentioning +# it. +e2e-dump-env: + @env | sed 's/=.*//' | sort | tr '\n' ' ' + # Proves git resolves inside the container, which a linked worktree breaks # unless the driver mounts the common git directory. e2e-show-git: @@ -469,6 +477,16 @@ if ($hostGhToken -and $hostGhToken.Trim()) { Assert-That 'the gh CLI token is used when the environment has none' ` ($viaGh.StdOut -match ('E2E-TOKEN:\[' + [regex]::Escape($hostGhToken.Trim()) + '\]')) ` (Hide-Token "$($viaGh.StdOut)$($viaGh.StdErr)") + + # The other half of the rule. Minting a credential the developer never put + # in this environment hands it to every build script and proc macro in the + # container, where natively the recipe would mint it in its own process -- + # so a target that never reads the variable must not receive it. + $noNeed = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-dump-env') ` + -Environment @{ GITHUB_TOKEN = '' } + Assert-That 'no token is derived for a target that does not read it' ` + ($noNeed.StdOut -notmatch 'GITHUB_TOKEN') ` + (Hide-Token "$($noNeed.StdOut)$($noNeed.StdErr)") } else { Write-Step 'skipping the gh-fallback check: this host has no gh credential' } @@ -541,7 +559,8 @@ $editedReference = Get-ImageReference -Repo $repo Assert-That 'editing the Dockerfile selects a new tag' ($editedReference -ne $reference) Write-Step 're-running the generator over the edited file' -Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo -AllowFailure | Out-Null +$regen = Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo -AllowFailure +Assert-Equal 'the generator succeeds over a user-modified owned file' 0 $regen.ExitCode $afterRegen = Get-Content -LiteralPath $dockerfile -Raw Assert-That 'the edit survives regeneration' ($afterRegen -match 'a repository-owned edit') ` 'anvil must preserve a user-modified owned file' @@ -581,13 +600,36 @@ Assert-That 'adding a hook selects a new tag' ($hookReference -ne $reference) ` # The default Dockerfile does not consume the secret, so prove the wiring by # having the image read it. This is a fixture-side Dockerfile edit, which is a # supported user action (proved in section 6). +# +# The build *writes* the secret and deletes it in the same layer, which is what +# the real Dockerfile does with the credential files an install leaves behind. +# Reading it alone would make the filesystem assertion below unfalsifiable: +# nothing would have written the string, so `grep` would find nothing whatever +# the layering did. A control build immediately below proves the probe can in +# fact see a leak. $secretStanza = @' # --- e2e: prove the build secret arrives and never lands in a layer --- RUN --mount=type=secret,id=e2e_token,required=true \ test -s /run/secrets/e2e_token \ - && echo "e2e: secret length $(wc -c < /run/secrets/e2e_token)" + && echo "e2e: secret length $(wc -c < /run/secrets/e2e_token)" \ + && cp /run/secrets/e2e_token /usr/local/cargo/credentials.toml \ + && rm -f /usr/local/cargo/credentials.toml '@ + +# The same stanza without the deletion. Built first, so a probe that cannot +# detect a leak fails here rather than passing silently on the real image. +$leakControlStanza = $secretStanza -replace '(?m)\s*&& rm -f /usr/local/cargo/credentials\.toml$', '' +Write-Fixture $dockerfile ($dockerfileBody + $leakControlStanza) +Write-Step 'building a deliberately leaking image to prove the probe works' +$controlRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +Assert-Equal 'the control image builds' 0 $controlRun.ExitCode +$controlLeak = Invoke-Engine -Arguments @( + 'run', '--rm', '--pull=never', (Get-ImageReference -Repo $repo), + 'grep', '-rsq', 'build-secret-value', '/opt/anvil', '/root', '/usr/local/cargo', '/tmp', '/run' +) -AllowFailure +Assert-Equal 'the probe detects a secret left in the filesystem' 0 $controlLeak.ExitCode + Write-Fixture $dockerfile ($dockerfileBody + $secretStanza) Write-Step 'rebuilding with the hook active' diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index 241a9ca9..84bda681 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -354,29 +354,33 @@ function Invoke-Suite([string]$EngineName) { $reuse = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure Assert-Equal 'a later run succeeds' 0 $reuse.ExitCode Assert-That 'a later run does not rebuild the image' ` - (-not ("$($reuse.StdOut)`n$($reuse.StdErr)" -match 'building image|Step 1/|FROM ')) ` + (-not ("$($reuse.StdOut)`n$($reuse.StdErr)" -match 'building |Step 1/|FROM ')) ` 'a rebuild happened when the tag should have resolved' - Write-Section "$EngineName : only the tool catalog renames the image" + Write-Section "$EngineName : every recipe file defines the image" - # The recipes execute from the bind mount, so an edit to one takes effect on - # the next run. Rebuilding the image for it would cost a full catalog - # reinstall for a file the image never runs -- the difference between a - # feature that speeds work up and one that gets abandoned. + # `just anvil-setup` reaches the install recipes through the tier, group and + # check recipes, so the routing decides *whether* a tool is installed as + # surely as tools.just decides *how*. Hashing only the install definitions + # let a group drop an `anvil--setup` dependency -- changing the + # installed set -- while the tag stayed byte-identical, so the stale image + # was reused forever. The whole tree is hashed for that reason, and these + # cases are what keep it that way. # # Edits are made against a byte copy and restored from it. Never # `git checkout --` on a generated file: that restores the last *commit*, # not the generated state, and anvil then preserves the stale file as a - # user modification. + # user modification. If this script is killed mid-section, regenerate with + # `cargo run -p cargo-anvil -- anvil` to return the tree to a known state. $baseline = Get-ImageReference Assert-That 'a baseline tag is available' ([bool]$baseline) $cases = @( - @{ File = 'justfiles/anvil/checks/clippy.just'; Renames = $false; Why = 'a check only routes into the catalog' } - @{ File = 'justfiles/anvil/container.just'; Renames = $false; Why = 'the driver computes the tag; it cannot define it' } - @{ File = 'justfiles/anvil/tiers.just'; Renames = $false; Why = 'a tier only routes into the catalog' } - @{ File = 'justfiles/anvil/versions.just'; Renames = $true; Why = 'a pin decides which build is installed' } - @{ File = 'justfiles/anvil/tools.just'; Renames = $true; Why = 'the install recipes decide what is installed' } + @{ File = 'justfiles/anvil/checks/clippy.just'; Why = 'a check carries the setup dependency that installs its tool' } + @{ File = 'justfiles/anvil/container.just'; Why = 'the driver passes the build args, secrets and PreBuild output into the build' } + @{ File = 'justfiles/anvil/tiers.just'; Why = 'a tier decides which groups, and so which setups, are reached' } + @{ File = 'justfiles/anvil/versions.just'; Why = 'a pin decides which build is installed' } + @{ File = 'justfiles/anvil/tools.just'; Why = 'the install recipes decide what is installed' } ) foreach ($case in $cases) { @@ -390,12 +394,8 @@ function Invoke-Suite([string]$EngineName) { try { Add-Content -LiteralPath $path -Value "`n# dogfood scratch" $edited = Get-ImageReference - $name = "editing $($case.File) $(if ($case.Renames) { 'renames' } else { 'does not rename' }) the image" - if ($case.Renames) { - Assert-That $name ($edited -ne $baseline) "$($case.Why); tag stayed $edited" - } else { - Assert-That $name ($edited -eq $baseline) "$($case.Why); tag moved $baseline -> $edited" - } + Assert-That "editing $($case.File) renames the image" ($edited -ne $baseline) ` + "$($case.Why); tag stayed $edited" } finally { Copy-Item -LiteralPath $backup -Destination $path -Force Remove-Item -LiteralPath $backup -Force -ErrorAction SilentlyContinue @@ -404,22 +404,26 @@ function Invoke-Suite([string]$EngineName) { Assert-Equal 'restoring every file restores the original tag' $baseline (Get-ImageReference) - # The assertions above compare references. This one proves the consequence a - # user actually feels: after an irrelevant edit, a run starts rather than - # builds. - $scratchCheck = Join-Path $RepoRoot 'justfiles/anvil/checks/clippy.just' - $scratchBackup = [System.IO.Path]::GetTempFileName() - Copy-Item -LiteralPath $scratchCheck -Destination $scratchBackup -Force - try { - Add-Content -LiteralPath $scratchCheck -Value "`n# dogfood scratch" - $afterEdit = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure - Assert-Equal 'a run after a check edit succeeds' 0 $afterEdit.ExitCode - Assert-That 'a run after a check edit does not rebuild the image' ` - (-not ("$($afterEdit.StdOut)`n$($afterEdit.StdErr)" -match 'building |Step 1/|FROM ')) ` - 'the image was rebuilt for an edit that cannot change its contents' - } finally { - Copy-Item -LiteralPath $scratchBackup -Destination $scratchCheck -Force - Remove-Item -LiteralPath $scratchBackup -Force -ErrorAction SilentlyContinue + # The regression that motivated hashing the whole tree, stated in the terms + # it actually occurred in: a group drops a check's `-setup` dependency, so + # the image installs one tool fewer, while tools.just and versions.just are + # untouched. The tag has to move or the reduced image is reused forever. + $group = Join-Path $RepoRoot 'justfiles/anvil/groups/pr-fast.just' + if (Test-Path -LiteralPath $group -PathType Leaf) { + $groupBackup = [System.IO.Path]::GetTempFileName() + Copy-Item -LiteralPath $group -Destination $groupBackup -Force + try { + $kept = @(Get-Content -LiteralPath $group | Where-Object { $_ -notmatch 'anvil-spellcheck-setup' }) + Set-Content -LiteralPath $group -Value $kept + Assert-That 'dropping a setup dependency renames the image' ((Get-ImageReference) -ne $baseline) ` + 'the installed tool set changed while the tag did not' + } finally { + Copy-Item -LiteralPath $groupBackup -Destination $group -Force + Remove-Item -LiteralPath $groupBackup -Force -ErrorAction SilentlyContinue + } + Assert-Equal 'restoring the group restores the original tag' $baseline (Get-ImageReference) + } else { + Write-Skipped 'dropping a setup dependency renames the image' 'no pr-fast group in this catalog' } if ($SkipTier) { From 71ed185029527545d6b527d2681bad75aecd5d7f Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 16 Aug 2026 19:34:21 +0200 Subject: [PATCH 30/81] docs(anvil): correct two claims the digest change left behind Both raised as suppressed findings by the automated reviewer, and both correct. The dogfood script's synopsis still described the behaviour the previous commit inverted -- "only the tool catalog renames the image: editing a check, a tier or the driver must not trigger a rebuild" -- directly contradicting the assertions further down the same file, which now require exactly the opposite. The section body was rewritten and the header was not. `anvil-container` was documented as taking "any recipe name and its arguments", which overstates what it can do. `just` joins a variadic `*target` with spaces before the recipe body sees it, and the driver splits that string back on whitespace, so the original argv is unrecoverable: an argument containing a space does not round-trip. This is a property of the parameter form rather than a defect in the split, so it is documented rather than fixed. No catalog recipe takes such an argument. Validation: `cargo test -p cargo-anvil` -- 309 unit plus 50 across the other targets, 0 failed. `cargo anvil --dry-run` reports 78/78 unchanged, so no generated artifact and no hashed image input is touched by this commit and the image tag is unaffected. README regenerated with `just readme`. --- crates/cargo-anvil/README.md | 7 +++++-- crates/cargo-anvil/docs/design/containers.md | 4 ++++ crates/cargo-anvil/src/lib.rs | 5 ++++- scripts/test-anvil-dogfood.ps1 | 8 +++++--- 4 files changed, 18 insertions(+), 6 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 62ee32d6..5a80a043 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -105,7 +105,10 @@ the toolset by construction, with no second tool list to keep in step. Execution is opt-in per invocation: `just anvil-pr` and every other recipe continue to run natively, and a container is entered only through -`anvil-container`, which takes any recipe name and its arguments. +`anvil-container`, which takes any recipe name and its arguments. Those +arguments are whitespace-delimited tokens: `just` joins a variadic +parameter with spaces before the recipe sees it, so an argument that itself +contains a space cannot be recovered and does not survive the round trip. ```text just anvil-container anvil-clippy # one check @@ -442,7 +445,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbkAqV7cij89Ab63nE30mvTLkbN2Wo4toYNGkb0MosdkJlO6lhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbUzxBCeTxVhgbVH3PyXajspMbraXhPJJZx38bmTnJe7clsEdhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index c1fac1f1..49ec11a3 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -62,6 +62,10 @@ own agents. The image is pinned to resemble that environment, not to reproduce i just anvil-container anvil-setup binstall ``` +Arguments are whitespace-delimited tokens. `just` joins a variadic `*target` with spaces before the recipe body sees +it, so the original argv is unrecoverable and an argument containing a space does not round-trip. No catalog recipe +takes one; a fork whose recipes do should pass them through the environment instead. + | Recipe | Behaviour | | --- | --- | | `just anvil-container [args…]` | Execute a recipe in the image. With no argument, opens an interactive shell. | diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 9f9c6f43..656a6f0d 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -104,7 +104,10 @@ //! //! Execution is opt-in per invocation: `just anvil-pr` and every other recipe //! continue to run natively, and a container is entered only through -//! `anvil-container`, which takes any recipe name and its arguments. +//! `anvil-container`, which takes any recipe name and its arguments. Those +//! arguments are whitespace-delimited tokens: `just` joins a variadic +//! parameter with spaces before the recipe sees it, so an argument that itself +//! contains a space cannot be recovered and does not survive the round trip. //! //! ```text //! just anvil-container anvil-clippy # one check diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index 84bda681..d646477c 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -24,9 +24,11 @@ 2. The image builds from the repository's own Dockerfile. 3. Every pinned tool in the catalog executes inside the image. 4. `anvil-aprz` runs, exercising a prebuilt binary and the advisory API. - 5. Only the tool catalog renames the image: editing a check, a tier or - the driver must not trigger a rebuild, while editing `tools.just` or - `versions.just` must. + 5. Every recipe file defines the image: editing a check, a tier, the + driver, `tools.just` or `versions.just` must all rename it, and + dropping a check's `-setup` dependency from a group -- which changes + the installed tool set while leaving `tools.just` untouched -- must + rename it too. 6. The requested tier runs to completion inside the image. 7. A second run reuses the image rather than rebuilding it. From ca81f0ffe98d24559d807c95e0627dd37b6db49b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 16 Aug 2026 22:25:26 +0200 Subject: [PATCH 31/81] fix(anvil): add variadic to the spellcheck dictionary `just spellcheck` and the `anvil-spellcheck` check inside `pr-fast` both failed on `crates/cargo-anvil/src/lib.rs:108`, where the previous commit described how `just` joins a variadic parameter. Four CI jobs failed on this one word: the `spell-check` job and `pr-fast` on linux, linux-arm and windows-arm. The word is correct and the dictionary is the intended place for it, per AGENTS.md. This should not have reached CI. The previous commit was validated with `cargo test`, `just readme` and `cargo anvil --dry-run`, scoped that way on the grounds that it touched no generated artifact and no hashed image input. That reasoning held for the image and missed the obvious: the commit added prose, and prose is what the spellchecker reads. Validation: `just anvil-container anvil-spellcheck` exits 0, and exits 1 reproducing the CI error when `variadic` is removed again, so the check is demonstrably running rather than passing vacuously. `just anvil-container anvil-pr-fast` -- the failing group in full -- exits 0 in 6:36. --- .spelling | 1 + 1 file changed, 1 insertion(+) diff --git a/.spelling b/.spelling index 4889d04b..cea2d1b8 100644 --- a/.spelling +++ b/.spelling @@ -474,3 +474,4 @@ toolset natively ARM64 WSL +variadic From 84fc8c57fb9ceec4d8b232a99055e36a96828ad9 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Mon, 24 Aug 2026 18:12:45 +0200 Subject: [PATCH 32/81] refactor(anvil): drop the rebuild recipe and fix the listed descriptions `anvil-container-rebuild` set `ANVIL_CONTAINER_NO_CACHE=1` and then performed the ordinary resolve. That variable is already public and composes with `NO_REBUILD` and `NO_RESOLVE`, which the recipe form does not, so the surface carried two ways to say one thing. Four recipes remain. `just --list` takes the last comment line before a recipe's attributes as its description, so a recipe whose rationale paragraph ended mid-sentence listed as a fragment: anvil-container-down # toolchain that would otherwise mask the image's own. anvil-container-rebuild # layer, or a build that is suspected of being wrong. `just --list` is the discovery surface for the command set, so each public recipe now repeats a one-line summary immediately above its attributes, and a test asserts every one lists as a whole sentence. The rationale above `anvil-container-down` still explained the `-cargo` and `-rustup` volumes removed earlier in this branch, describing a loop that no longer exists. Validation: - `cargo test -p cargo-anvil` -- 310 unit plus 50 across the other targets. - The new listing guard was control-tested: reintroducing the fragment fails it with `anvil-container-down: lists as a fragment: "and keeps these."`. - `ANVIL_CONTAINER_NO_CACHE=1` rebuilds and a following run reuses the result, both exit 0, so the removed recipe leaves no gap. - `just --list` renders four whole sentences. --- .anvil.lock | 4 +-- crates/cargo-anvil/README.md | 8 ++--- crates/cargo-anvil/docs/design/containers.md | 11 ++++-- .../src/anvil/artifacts/container.rs | 35 ++++++++++++++++++- crates/cargo-anvil/src/lib.rs | 6 ++-- .../templates/justfiles/anvil/container.just | 22 +++--------- .../snapshots/snapshots__ado_backend.snap | 22 +++--------- .../snapshots/snapshots__github_backend.snap | 22 +++--------- .../snapshots/snapshots__local_only.snap | 22 +++--------- justfiles/anvil/container.just | 22 +++--------- 10 files changed, 76 insertions(+), 98 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index ca5b5138..0a4786a7 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:dfba2a9ff5c11ad73780af9e64114aa50d859f2039737eb6935d97ebc82bb30e" +catalog_checksum = "sha256:0e6bfb01b285c86d0c7c95eb8b564cb8dbb4b58b9b8aa1cf1ff687d5b617d5e5" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:7c695125346535d0ba17d09947854af59971ff41f53a7faab5f4101159055ca8" +checksum = "sha256:9b482756683faa1ef0ce28b64db4ff0c57f2192b5d7469c7828ab3907f202775" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 5a80a043..dd24757b 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -179,9 +179,9 @@ exactly the reference a consumer will later look up. |`GITHUB_TOKEN`|Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited.| Supporting recipes: `anvil-container-tag`, `anvil-container-status` -(reports the engine and image without building or pulling), -`anvil-container-rebuild`, and `anvil-container-down` (removes this -repository’s cache volumes). +(reports the engine and image without building or pulling), and +`anvil-container-down` (removes this repository’s cache volumes). To rebuild +a tag that already resolves, set `ANVIL_CONTAINER_NO_CACHE=1` above. #### The hook @@ -445,7 +445,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbUzxBCeTxVhgbVH3PyXajspMbraXhPJJZx38bmTnJe7clsEdhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb_EzqVXVh9lMbSZB2fnIqi24bEggz4HxWFLMb7lUzlllOloVhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 49ec11a3..c5e5abb8 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -71,10 +71,15 @@ takes one; a fork whose recipes do should pass them through the environment inst | `just anvil-container [args…]` | Execute a recipe in the image. With no argument, opens an interactive shell. | | `just anvil-container-tag` | Print the image reference for the current inputs. Builds nothing. | | `just anvil-container-status` | Print the engine, working directory, image reference, and whether it is present. Never builds or pulls. | -| `just anvil-container-rebuild` | Rebuild the image with every layer cache disabled. | | `just anvil-container-down` | Remove this repository's cache volumes. The image is retained. | -All five are annotated `[group("anvil-container")]` and appear as one cluster in `just --groups`. +All four are annotated `[group("anvil-container")]` and appear as one cluster in `just --groups`. Each repeats a +one-line summary immediately above its attributes, because `just --list` takes the last comment line before them as the +description and would otherwise print the tail of a rationale paragraph as a fragment. + +There is deliberately no `anvil-container-rebuild`. Its whole body would be `ANVIL_CONTAINER_NO_CACHE=1` followed by +the ordinary resolve, and that variable is already public below — where it also composes with `NO_REBUILD` and +`NO_RESOLVE`, which a recipe form does not. | Variable | Effect | | --- | --- | @@ -496,7 +501,7 @@ guard. A different base OS with a different toolchain source is one Dockerfile r hashed, so changing the build recipe always renames the tag — but any *additional* file it copies is outside the tag. Such a file can change what a build produces while naming a tag that already resolves, and the existing image is then reused, so the change is never built. A fork that needs extra content should carry it in the Dockerfile - itself, or accept that edits to it require `anvil-container-rebuild`. + itself, or accept that edits to it require `ANVIL_CONTAINER_NO_CACHE=1`. - anvil never pushes or promotes an image. It builds one, and will use one a hook fetched (§7.3); publishing belongs to whoever owns the registry. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 6a729d65..b17dc18a 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -176,11 +176,44 @@ mod tests { "anvil-container *target:", "anvil-container-tag:", "anvil-container-status:", - "anvil-container-rebuild:", "anvil-container-down:", ] { assert!(RECIPE.contains(expected), "missing recipe: {expected}"); } + // A cache-defeating rebuild is `ANVIL_CONTAINER_NO_CACHE=1`, which is + // already public and composes with the other guards. A recipe wrapping + // one variable assignment would be a second way to say the same thing. + assert!(!RECIPE.contains("anvil-container-rebuild:")); + } + + #[test] + fn every_public_recipe_lists_with_a_whole_sentence() { + // `just --list` takes the last comment line before the attributes as + // the description, so a recipe whose rationale paragraph ends mid + // sentence lists as a fragment -- "# toolchain that would otherwise + // mask the image's own." The generated tree is the discovery surface, + // so each public recipe repeats a one-line summary immediately above + // its attributes. + for recipe in [ + "anvil-container *target:", + "anvil-container-tag:", + "anvil-container-status:", + "anvil-container-down:", + ] { + let at = RECIPE.find(recipe).expect("recipe must exist"); + let description = RECIPE[..at] + .lines() + .rev() + .find(|line| line.trim_start().starts_with('#')) + .expect("a public recipe must carry a description") + .trim_start() + .trim_start_matches('#') + .trim(); + assert!( + description.ends_with('.') && description.starts_with(|c: char| c.is_uppercase()), + "{recipe} lists as a fragment: {description:?}" + ); + } } #[test] diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 656a6f0d..34422c1c 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -178,9 +178,9 @@ //! | `GITHUB_TOKEN` | Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited. | //! //! Supporting recipes: `anvil-container-tag`, `anvil-container-status` -//! (reports the engine and image without building or pulling), -//! `anvil-container-rebuild`, and `anvil-container-down` (removes this -//! repository's cache volumes). +//! (reports the engine and image without building or pulling), and +//! `anvil-container-down` (removes this repository's cache volumes). To rebuild +//! a tag that already resolves, set `ANVIL_CONTAINER_NO_CACHE=1` above. //! //! ### The hook //! diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index c65e6ca5..2ee43898 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -704,26 +704,14 @@ anvil-container-status: } exit 0 -# Rebuild the exec image from scratch, ignoring every cached layer. +# Remove this repository's cache volumes. The image is left in place. # -# The ordinary path already rebuilds whenever an input changes, so this is for -# the cases a content hash cannot see: a moved upstream package, a stale base -# layer, or a build that is suspected of being wrong. -[group("anvil-container")] -[script("pwsh", "-NoProfile")] -anvil-container-rebuild: - $ErrorActionPreference = 'Stop' - $env:ANVIL_CONTAINER_NO_CACHE = '1' - $image = (just _anvil-container-image) | Select-Object -Last 1 - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "image rebuilt: $image" - exit 0 +# Only the download caches are volumes, so this discards fetched crates and git +# checkouts and nothing else: the next run re-fetches them, and the image's own +# tools are untouched. The counterpart to a rebuild, which discards the image +# and keeps these. # Remove this repository's cache volumes. The image is left in place. -# -# The last two names are from an earlier build of this driver, which mounted -# the cargo and rustup homes wholesale; removing them here clears a stale -# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index ef8465d2..581b8bc3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4213,26 +4213,14 @@ anvil-container-status: } exit 0 -# Rebuild the exec image from scratch, ignoring every cached layer. +# Remove this repository's cache volumes. The image is left in place. # -# The ordinary path already rebuilds whenever an input changes, so this is for -# the cases a content hash cannot see: a moved upstream package, a stale base -# layer, or a build that is suspected of being wrong. -[group("anvil-container")] -[script("pwsh", "-NoProfile")] -anvil-container-rebuild: - $ErrorActionPreference = 'Stop' - $env:ANVIL_CONTAINER_NO_CACHE = '1' - $image = (just _anvil-container-image) | Select-Object -Last 1 - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "image rebuilt: $image" - exit 0 +# Only the download caches are volumes, so this discards fetched crates and git +# checkouts and nothing else: the next run re-fetches them, and the image's own +# tools are untouched. The counterpart to a rebuild, which discards the image +# and keeps these. # Remove this repository's cache volumes. The image is left in place. -# -# The last two names are from an earlier build of this driver, which mounted -# the cargo and rustup homes wholesale; removing them here clears a stale -# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 5093aad5..5ff0f66e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4134,26 +4134,14 @@ anvil-container-status: } exit 0 -# Rebuild the exec image from scratch, ignoring every cached layer. +# Remove this repository's cache volumes. The image is left in place. # -# The ordinary path already rebuilds whenever an input changes, so this is for -# the cases a content hash cannot see: a moved upstream package, a stale base -# layer, or a build that is suspected of being wrong. -[group("anvil-container")] -[script("pwsh", "-NoProfile")] -anvil-container-rebuild: - $ErrorActionPreference = 'Stop' - $env:ANVIL_CONTAINER_NO_CACHE = '1' - $image = (just _anvil-container-image) | Select-Object -Last 1 - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "image rebuilt: $image" - exit 0 +# Only the download caches are volumes, so this discards fetched crates and git +# checkouts and nothing else: the next run re-fetches them, and the image's own +# tools are untouched. The counterpart to a rebuild, which discards the image +# and keeps these. # Remove this repository's cache volumes. The image is left in place. -# -# The last two names are from an earlier build of this driver, which mounted -# the cargo and rustup homes wholesale; removing them here clears a stale -# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index cd65073e..b96dc362 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2953,26 +2953,14 @@ anvil-container-status: } exit 0 -# Rebuild the exec image from scratch, ignoring every cached layer. +# Remove this repository's cache volumes. The image is left in place. # -# The ordinary path already rebuilds whenever an input changes, so this is for -# the cases a content hash cannot see: a moved upstream package, a stale base -# layer, or a build that is suspected of being wrong. -[group("anvil-container")] -[script("pwsh", "-NoProfile")] -anvil-container-rebuild: - $ErrorActionPreference = 'Stop' - $env:ANVIL_CONTAINER_NO_CACHE = '1' - $image = (just _anvil-container-image) | Select-Object -Last 1 - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "image rebuilt: $image" - exit 0 +# Only the download caches are volumes, so this discards fetched crates and git +# checkouts and nothing else: the next run re-fetches them, and the image's own +# tools are untouched. The counterpart to a rebuild, which discards the image +# and keeps these. # Remove this repository's cache volumes. The image is left in place. -# -# The last two names are from an earlier build of this driver, which mounted -# the cargo and rustup homes wholesale; removing them here clears a stale -# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index c65e6ca5..2ee43898 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -704,26 +704,14 @@ anvil-container-status: } exit 0 -# Rebuild the exec image from scratch, ignoring every cached layer. +# Remove this repository's cache volumes. The image is left in place. # -# The ordinary path already rebuilds whenever an input changes, so this is for -# the cases a content hash cannot see: a moved upstream package, a stale base -# layer, or a build that is suspected of being wrong. -[group("anvil-container")] -[script("pwsh", "-NoProfile")] -anvil-container-rebuild: - $ErrorActionPreference = 'Stop' - $env:ANVIL_CONTAINER_NO_CACHE = '1' - $image = (just _anvil-container-image) | Select-Object -Last 1 - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - Write-Output "image rebuilt: $image" - exit 0 +# Only the download caches are volumes, so this discards fetched crates and git +# checkouts and nothing else: the next run re-fetches them, and the image's own +# tools are untouched. The counterpart to a rebuild, which discards the image +# and keeps these. # Remove this repository's cache volumes. The image is left in place. -# -# The last two names are from an earlier build of this driver, which mounted -# the cargo and rustup homes wholesale; removing them here clears a stale -# toolchain that would otherwise mask the image's own. [group("anvil-container")] [script("pwsh", "-NoProfile")] anvil-container-down: From e8df07542364dec113a7bf9000b77d37ad74929b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Mon, 24 Aug 2026 18:54:57 +0200 Subject: [PATCH 33/81] refactor(anvil): name the hook functions for what they supply Architecture review, ADO PR 5542411: "The name could be better - this implies a general-purpose pre-build script, but it looks like it's very much tied to secret management." `Anvil-PreBuild` and `Anvil-PreRun` described *when* they run, which reads as a pair of general-purpose extension points. Neither is one: the first exists to mint build secrets, the second to mint run-time credentials, and both are fail-closed for that reason. They become `Anvil-BuildSecrets` and `Anvil-RunEnv`, named for what they supply. `Anvil-ResolveImage` already followed that rule and is unchanged. The hook contract is not released -- it ships first in this branch -- so no repository can be depending on the old names. Two documentation corrections found while syncing the design document with the proposal under review: - The worktree git-directory paragraph said only that "anvil detects this". It now says what performs the redirection and when -- the recipe, while assembling the run, before the container starts -- names the checks that depend on it, and explains why a generated `.git` file is used rather than `GIT_DIR`, which every process in the container would inherit. - The repository-name normalization claimed the result "is always a valid reference". Verified rather than assumed: `ox-tools (copy)` normalizes to `anvil-ox-tools-copy`, and a name with no `[a-z0-9]` character degrades to plain `anvil`, because `trim_end_matches` strips every trailing separator. Both are valid; the second is merely no longer repository-specific, which is what the text now says. A reviewer suggestion to add a hashed fallback was rejected on that evidence -- the failure it guards against cannot occur. The regenerated README also drops a `cargo-doc2readme` version marker to 0.7.2, which is the version `versions.just` pins; the committed marker was 0.7.3, drift from a machine running a newer tool. Validation: - `cargo test -p cargo-anvil` -- 310 unit plus 50 across the other targets. - `scripts/test-anvil-container.ps1` -- 62/62 in 08:07, covering the renamed functions through a real build secret, a forwarded run variable, and the empty-value abort. --- .anvil.lock | 4 +- crates/cargo-anvil/README.md | 6 +-- crates/cargo-anvil/docs/design/containers.md | 45 +++++++++++-------- .../src/anvil/artifacts/container.rs | 20 +++++---- crates/cargo-anvil/src/lib.rs | 4 +- .../templates/justfiles/anvil/container.just | 30 ++++++------- .../snapshots/snapshots__ado_backend.snap | 30 ++++++------- .../snapshots/snapshots__github_backend.snap | 30 ++++++------- .../snapshots/snapshots__local_only.snap | 30 ++++++------- justfiles/anvil/container.just | 30 ++++++------- scripts/test-anvil-container.ps1 | 18 ++++---- 11 files changed, 128 insertions(+), 119 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 0a4786a7..f8a08ad8 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:0e6bfb01b285c86d0c7c95eb8b564cb8dbb4b58b9b8aa1cf1ff687d5b617d5e5" +catalog_checksum = "sha256:600719c35c9cb1c95ae432b44d1e5c39af4f2a6249582aed00d27e59f7254d83" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:9b482756683faa1ef0ce28b64db4ff0c57f2192b5d7469c7828ab3907f202775" +checksum = "sha256:bcedc35a6eecf4ffe5e5a075ca731ff9cbab201471e4017395c0144da690ac47" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index dd24757b..44fb4be1 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -191,8 +191,8 @@ repository or a downstream catalog that needs one adds file is present, whoever wrote it: ```powershell -function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } -function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +function Anvil-BuildSecrets { @{ Secrets = @{ feed = (mint-a-token) } } } +function Anvil-RunEnv { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } function Anvil-ResolveImage { param($tag) (fetch-a-published-image $tag) } ``` @@ -445,7 +445,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjNhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb_EzqVXVh9lMbSZB2fnIqi24bEggz4HxWFLMb7lUzlllOloVhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbM7cgRtZ4wjwbJoeFnDDhNxUbRGJcKx-hNawbu8tGIw2QbgphZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index c5e5abb8..53791f7e 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -32,8 +32,8 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [6.2 Docker](#62-docker) - [6.3 Podman](#63-podman) - [7. The hook](#7-the-hook) - - [7.1 Anvil-PreBuild](#71-anvil-prebuild) - - [7.2 Anvil-PreRun](#72-anvil-prerun) + - [7.1 Anvil-BuildSecrets](#71-Anvil-BuildSecrets) + - [7.2 Anvil-RunEnv](#72-Anvil-RunEnv) - [7.3 Anvil-ResolveImage](#73-anvil-resolveimage) - [7.4 Trust boundary](#74-trust-boundary) - [8. Customization](#8-customization) @@ -141,7 +141,7 @@ byte-identical. Hashing only the install definitions would leave that change unn the image does not have. `container.just` is hashed too. It is not circular — the digest is over file text, and no file contains the tag — and -it belongs in the set because it passes the build arguments, the secret mounts and the hook's `Anvil-PreBuild` output +it belongs in the set because it passes the build arguments, the secret mounts and the hook's `Anvil-BuildSecrets` output into the build. The cost is that editing any recipe renames the image and the next run rebuilds it. That is the correct trade: a tag @@ -198,11 +198,17 @@ removed on exit (`--rm`). | `anvil--cargo-git` (volume) | `/usr/local/cargo/git` | Git checkouts of git dependencies. | A linked worktree (`git worktree add`) keeps its git directory outside the checkout and stores an absolute host path -in `.git`, which does not exist inside the container. anvil detects this by comparing `git rev-parse --git-dir` against -`--git-common-dir`, mounts the common directory, and bind-mounts a generated `.git` file naming that mount over the -checkout's own, so git resolves it by ordinary discovery. The redirection is confined to the checkout: a git command -run elsewhere in the container, such as `git init` in a scratch directory, is unaffected. An ordinary clone carries its -git directory inside the bind mount and takes none of this. No flag or variable selects the behaviour. +in `.git`, which does not exist inside the container. The recipe resolves this while assembling the run, before the +container starts: it compares `git rev-parse --git-dir` against `--git-common-dir`, and when they differ it adds a +bind mount for the common directory and a second one placing a generated `.git` file over the checkout's own, naming +that mount. Git then resolves the history by ordinary discovery. This is what lets the checks that read history — the +impact-scoped filters, `anvil-mutants-diff`, `anvil-semver-check` — work from a worktree at all; without it git +resolves nothing inside the container and each of them fails a long way from the cause. + +The redirection is confined to the checkout: a git command run elsewhere in the container, such as `git init` in a +scratch directory, is unaffected. That is why a generated `.git` file is used rather than `GIT_DIR`, which is ambient +and would be inherited by every process in the container. An ordinary clone carries its git directory inside the bind +mount and takes none of this. Nothing is written to the host, and no flag or variable selects the behaviour. Only cargo's content-addressed download caches are volumes, so the write-heavy download path never crosses the host boundary and the host's own toolchain is untouched. `$CARGO_HOME` and `$RUSTUP_HOME` themselves are **not** mounted: @@ -220,10 +226,11 @@ The caller's working directory is mapped to its in-container equivalent, so rela `anvil-container` is invoked from a subdirectory. Image and volume names derive from the repository directory name, lowercased with every run of characters outside -`[a-z0-9]` replaced by a single `-` and any trailing `-` removed, so the result is always a valid reference: a -checkout in `ox-tools (copy)` would otherwise end in a separator, which the engine rejects. Two checkouts with the -same directory name therefore share cache volumes. That is harmless, since both volumes hold only content-addressed -downloads, but `anvil-container-down` then removes volumes the other checkout is also using. +`[a-z0-9]` replaced by a single `-` and any trailing `-` removed: a checkout in `ox-tools (copy)` becomes +`anvil-ox-tools-copy` rather than ending in a separator, which the engine rejects. A directory name with no `[a-z0-9]` +character at all degrades to plain `anvil` — still a valid reference, but no longer repository-specific. Two checkouts +with the same directory name therefore share cache volumes. That is harmless, since both volumes hold only +content-addressed downloads, but `anvil-container-down` then removes volumes the other checkout is also using. ### 5.2 Process identity @@ -351,7 +358,7 @@ Podman differs from Docker in three respects: Error: creating temp file: open /mnt/c/Users/…/repo\podman-build-secret-4085781963 ``` - Only a repository whose hook defines `Anvil-PreBuild` (§7.1) is affected; building, running, and tag reuse are not. + Only a repository whose hook defines `Anvil-BuildSecrets` (§7.1) is affected; building, running, and tag reuse are not. Use Docker if you need build-time credentials on Windows. - **The ignore file is passed explicitly.** buildah honours only an ignore file at the context root, so anvil passes @@ -373,8 +380,8 @@ The script may define up to three independent functions. All are optional, and e | Function | Invoked | Returns | | --- | --- | --- | -| `Anvil-PreBuild` | before a build | `@{ Secrets = @{ = } }` | -| `Anvil-PreRun` | before a run | `@{ Env = @{ = } }` | +| `Anvil-BuildSecrets` | before a build | `@{ Secrets = @{ = } }` | +| `Anvil-RunEnv` | before a run | `@{ Env = @{ = } }` | | `Anvil-ResolveImage $tag` | before a build, when no local image matches | an image reference, or nothing | Both value-returning functions **fail closed on an empty value**, which the engine does not: BuildKit accepts @@ -387,13 +394,13 @@ retains it far longer than a short-lived token is intended to live. The variable returns, and when the engine is reached through WSL the names are exported through `WSLENV` so the values cross that boundary. -### 7.1 Anvil-PreBuild +### 7.1 Anvil-BuildSecrets Each entry becomes a BuildKit `--secret id=,env=ANVIL_SECRET_` mount, which BuildKit keeps out of every image layer. ```powershell -function Anvil-PreBuild { +function Anvil-BuildSecrets { @{ Secrets = @{ feed_token = (az account get-access-token --resource … --query accessToken -o tsv) } } } ``` @@ -412,13 +419,13 @@ Anything the build *writes* using a secret is ordinary layer content. The defaul `credentials.toml` and `.netrc` in the same `RUN` layer as the install; a replacement must do the same, or the credential is baked into a layer that a later deletion cannot remove. -### 7.2 Anvil-PreRun +### 7.2 Anvil-RunEnv Each entry is forwarded with `-e ` and is an ordinary environment variable inside the image. The forwarded names, never their values, are echoed to stderr, because everything executing in the container can read them. ```powershell -function Anvil-PreRun { +function Anvil-RunEnv { @{ Env = @{ CARGO_REGISTRIES_INTERNAL_TOKEN = (mint-a-token) } } } ``` diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index b17dc18a..8d525afe 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -82,12 +82,14 @@ pub fn dockerignore() -> Artifact { /// repository can write the same path by hand. The recipe loads it either way. /// /// The script may define any of three functions, and is dot-sourced before the -/// phase that needs it: +/// phase that needs it. Each is named for what it supplies rather than for when +/// it runs, because all three exist to provide credentials or an image, not as +/// general-purpose extension points: /// -/// - `Anvil-PreBuild` returns `@{ Secrets = @{ = } }`. Each entry +/// - `Anvil-BuildSecrets` returns `@{ Secrets = @{ = } }`. Each entry /// becomes a `BuildKit` `--secret id=`, passed by environment variable /// name so the value never reaches a process argument, and never a layer. -/// - `Anvil-PreRun` returns `@{ Env = @{ = } }`. Each entry is +/// - `Anvil-RunEnv` returns `@{ Env = @{ = } }`. Each entry is /// forwarded into the container by name, for the same reason. /// - `Anvil-ResolveImage` takes the computed reference and returns one to use /// instead, or nothing. It is how a repository fetches a published image @@ -381,8 +383,8 @@ mod tests { // Both phases, asserted independently: "for" alone is a substring of // the build-side message, so it would pass with the run-side guard // deleted. - assert!(RECIPE.contains("Anvil-PreBuild returned an empty value for secret")); - assert!(RECIPE.contains("Anvil-PreRun returned an empty value for")); + assert!(RECIPE.contains("Anvil-BuildSecrets returned an empty value for secret")); + assert!(RECIPE.contains("Anvil-RunEnv returned an empty value for")); } #[test] @@ -513,16 +515,16 @@ mod tests { fn the_credential_phases_are_fail_closed() { // Unlike resolution, these must stop the run: a container that starts // without its credentials fails deep inside, far from the cause. - assert!(RECIPE.contains("anvil: Anvil-PreBuild returned no secrets")); - assert!(RECIPE.contains("anvil: Anvil-PreRun returned no variables")); + assert!(RECIPE.contains("anvil: Anvil-BuildSecrets returned no secrets")); + assert!(RECIPE.contains("anvil: Anvil-RunEnv returned no variables")); assert!(RECIPE.contains("anvil: failed to load ${hookRel}:")); // Whitespace is not a credential. IsNullOrEmpty would accept " ". assert!(!RECIPE.contains("[string]::IsNullOrEmpty($hook")); // Take the last object, not the whole stream: a hook that writes // progress with Write-Output would otherwise hand back an array whose // .Secrets is silently $null, and the guard above would not fire. - assert!(RECIPE.contains("@(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1")); - assert!(RECIPE.contains("@(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1")); + assert!(RECIPE.contains("@(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1")); + assert!(RECIPE.contains("@(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1")); } #[test] diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 34422c1c..b230e5c9 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -190,8 +190,8 @@ //! file is present, whoever wrote it: //! //! ```powershell -//! function Anvil-PreBuild { @{ Secrets = @{ feed = (mint-a-token) } } } -//! function Anvil-PreRun { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } +//! function Anvil-BuildSecrets { @{ Secrets = @{ feed = (mint-a-token) } } } +//! function Anvil-RunEnv { @{ Env = @{ FEED_TOKEN = (mint-a-token) } } } //! function Anvil-ResolveImage { param($tag) (fetch-a-published-image $tag) } //! ``` //! diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 2ee43898..23e87e4c 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -146,7 +146,7 @@ anvil-container-tag: # This driver is included too. It is not circular -- the tag is derived # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's - # `Anvil-PreBuild` output into the build, all of which shape the result. + # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { @@ -308,30 +308,30 @@ _anvil-container-image: Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + if (Get-Command Anvil-BuildSecrets -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-BuildSecrets from $hookRel") try { # Take the last emitted object, not the whole stream: a hook # that writes progress with `Write-Output` would otherwise hand # back an array whose `.Secrets` is silently $null. - $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-BuildSecrets failed: $($_.Exception.Message)" exit 1 } $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } - # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # A defined `Anvil-BuildSecrets` that yields nothing is the hazard this # guard exists for, not a hook opting out: secrets are the only # thing the phase can contribute, so an empty return means the mint # failed quietly. A hook with no build-time credentials simply does # not define the function. if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + Write-Error "anvil: Anvil-BuildSecrets returned no secrets; omit the function if the build needs none" exit 1 } foreach ($id in $secrets.Keys) { if ([string]::IsNullOrWhiteSpace($secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + Write-Error "anvil: Anvil-BuildSecrets returned an empty value for secret '$id'" exit 1 } $name = "ANVIL_SECRET_$id" @@ -608,7 +608,7 @@ anvil-container *target: $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # Fail-closed like Anvil-BuildSecrets: a run that cannot obtain its # credentials fails inside the container in a far less obvious way. try { . $hookPath @@ -616,22 +616,22 @@ anvil-container *target: Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + if (Get-Command Anvil-RunEnv -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-RunEnv from .anvil/container/hooks.ps1") try { - $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-RunEnv failed: $($_.Exception.Message)" exit 1 } $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + Write-Error "anvil: Anvil-RunEnv returned no variables; omit the function if the run needs none" exit 1 } foreach ($name in $hookVars.Keys) { if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + Write-Error "anvil: Anvil-RunEnv returned an empty value for '$name'" exit 1 } Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 581b8bc3..53b0c4ea 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3655,7 +3655,7 @@ anvil-container-tag: # This driver is included too. It is not circular -- the tag is derived # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's - # `Anvil-PreBuild` output into the build, all of which shape the result. + # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { @@ -3817,30 +3817,30 @@ _anvil-container-image: Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + if (Get-Command Anvil-BuildSecrets -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-BuildSecrets from $hookRel") try { # Take the last emitted object, not the whole stream: a hook # that writes progress with `Write-Output` would otherwise hand # back an array whose `.Secrets` is silently $null. - $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-BuildSecrets failed: $($_.Exception.Message)" exit 1 } $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } - # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # A defined `Anvil-BuildSecrets` that yields nothing is the hazard this # guard exists for, not a hook opting out: secrets are the only # thing the phase can contribute, so an empty return means the mint # failed quietly. A hook with no build-time credentials simply does # not define the function. if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + Write-Error "anvil: Anvil-BuildSecrets returned no secrets; omit the function if the build needs none" exit 1 } foreach ($id in $secrets.Keys) { if ([string]::IsNullOrWhiteSpace($secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + Write-Error "anvil: Anvil-BuildSecrets returned an empty value for secret '$id'" exit 1 } $name = "ANVIL_SECRET_$id" @@ -4117,7 +4117,7 @@ anvil-container *target: $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # Fail-closed like Anvil-BuildSecrets: a run that cannot obtain its # credentials fails inside the container in a far less obvious way. try { . $hookPath @@ -4125,22 +4125,22 @@ anvil-container *target: Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + if (Get-Command Anvil-RunEnv -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-RunEnv from .anvil/container/hooks.ps1") try { - $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-RunEnv failed: $($_.Exception.Message)" exit 1 } $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + Write-Error "anvil: Anvil-RunEnv returned no variables; omit the function if the run needs none" exit 1 } foreach ($name in $hookVars.Keys) { if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + Write-Error "anvil: Anvil-RunEnv returned an empty value for '$name'" exit 1 } Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 5ff0f66e..5b2e3894 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3576,7 +3576,7 @@ anvil-container-tag: # This driver is included too. It is not circular -- the tag is derived # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's - # `Anvil-PreBuild` output into the build, all of which shape the result. + # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { @@ -3738,30 +3738,30 @@ _anvil-container-image: Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + if (Get-Command Anvil-BuildSecrets -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-BuildSecrets from $hookRel") try { # Take the last emitted object, not the whole stream: a hook # that writes progress with `Write-Output` would otherwise hand # back an array whose `.Secrets` is silently $null. - $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-BuildSecrets failed: $($_.Exception.Message)" exit 1 } $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } - # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # A defined `Anvil-BuildSecrets` that yields nothing is the hazard this # guard exists for, not a hook opting out: secrets are the only # thing the phase can contribute, so an empty return means the mint # failed quietly. A hook with no build-time credentials simply does # not define the function. if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + Write-Error "anvil: Anvil-BuildSecrets returned no secrets; omit the function if the build needs none" exit 1 } foreach ($id in $secrets.Keys) { if ([string]::IsNullOrWhiteSpace($secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + Write-Error "anvil: Anvil-BuildSecrets returned an empty value for secret '$id'" exit 1 } $name = "ANVIL_SECRET_$id" @@ -4038,7 +4038,7 @@ anvil-container *target: $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # Fail-closed like Anvil-BuildSecrets: a run that cannot obtain its # credentials fails inside the container in a far less obvious way. try { . $hookPath @@ -4046,22 +4046,22 @@ anvil-container *target: Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + if (Get-Command Anvil-RunEnv -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-RunEnv from .anvil/container/hooks.ps1") try { - $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-RunEnv failed: $($_.Exception.Message)" exit 1 } $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + Write-Error "anvil: Anvil-RunEnv returned no variables; omit the function if the run needs none" exit 1 } foreach ($name in $hookVars.Keys) { if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + Write-Error "anvil: Anvil-RunEnv returned an empty value for '$name'" exit 1 } Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index b96dc362..883f434c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2395,7 +2395,7 @@ anvil-container-tag: # This driver is included too. It is not circular -- the tag is derived # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's - # `Anvil-PreBuild` output into the build, all of which shape the result. + # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { @@ -2557,30 +2557,30 @@ _anvil-container-image: Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + if (Get-Command Anvil-BuildSecrets -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-BuildSecrets from $hookRel") try { # Take the last emitted object, not the whole stream: a hook # that writes progress with `Write-Output` would otherwise hand # back an array whose `.Secrets` is silently $null. - $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-BuildSecrets failed: $($_.Exception.Message)" exit 1 } $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } - # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # A defined `Anvil-BuildSecrets` that yields nothing is the hazard this # guard exists for, not a hook opting out: secrets are the only # thing the phase can contribute, so an empty return means the mint # failed quietly. A hook with no build-time credentials simply does # not define the function. if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + Write-Error "anvil: Anvil-BuildSecrets returned no secrets; omit the function if the build needs none" exit 1 } foreach ($id in $secrets.Keys) { if ([string]::IsNullOrWhiteSpace($secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + Write-Error "anvil: Anvil-BuildSecrets returned an empty value for secret '$id'" exit 1 } $name = "ANVIL_SECRET_$id" @@ -2857,7 +2857,7 @@ anvil-container *target: $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # Fail-closed like Anvil-BuildSecrets: a run that cannot obtain its # credentials fails inside the container in a far less obvious way. try { . $hookPath @@ -2865,22 +2865,22 @@ anvil-container *target: Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + if (Get-Command Anvil-RunEnv -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-RunEnv from .anvil/container/hooks.ps1") try { - $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-RunEnv failed: $($_.Exception.Message)" exit 1 } $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + Write-Error "anvil: Anvil-RunEnv returned no variables; omit the function if the run needs none" exit 1 } foreach ($name in $hookVars.Keys) { if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + Write-Error "anvil: Anvil-RunEnv returned an empty value for '$name'" exit 1 } Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 2ee43898..23e87e4c 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -146,7 +146,7 @@ anvil-container-tag: # This driver is included too. It is not circular -- the tag is derived # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's - # `Anvil-PreBuild` output into the build, all of which shape the result. + # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { @@ -308,30 +308,30 @@ _anvil-container-image: Write-Error "anvil: failed to load ${hookRel}: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreBuild -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreBuild from $hookRel") + if (Get-Command Anvil-BuildSecrets -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-BuildSecrets from $hookRel") try { # Take the last emitted object, not the whole stream: a hook # that writes progress with `Write-Output` would otherwise hand # back an array whose `.Secrets` is silently $null. - $hook = @(Anvil-PreBuild | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-BuildSecrets | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreBuild failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-BuildSecrets failed: $($_.Exception.Message)" exit 1 } $secrets = if ($null -ne $hook) { $hook.Secrets } else { $null } - # A defined `Anvil-PreBuild` that yields nothing is the hazard this + # A defined `Anvil-BuildSecrets` that yields nothing is the hazard this # guard exists for, not a hook opting out: secrets are the only # thing the phase can contribute, so an empty return means the mint # failed quietly. A hook with no build-time credentials simply does # not define the function. if ($null -eq $secrets -or $secrets.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreBuild returned no secrets; omit the function if the build needs none" + Write-Error "anvil: Anvil-BuildSecrets returned no secrets; omit the function if the build needs none" exit 1 } foreach ($id in $secrets.Keys) { if ([string]::IsNullOrWhiteSpace($secrets[$id])) { - Write-Error "anvil: Anvil-PreBuild returned an empty value for secret '$id'" + Write-Error "anvil: Anvil-BuildSecrets returned an empty value for secret '$id'" exit 1 } $name = "ANVIL_SECRET_$id" @@ -608,7 +608,7 @@ anvil-container *target: $hookPath = Join-Path $repoRoot '.anvil/container/hooks.ps1' if (Test-Path -LiteralPath $hookPath -PathType Leaf) { - # Fail-closed like Anvil-PreBuild: a run that cannot obtain its + # Fail-closed like Anvil-BuildSecrets: a run that cannot obtain its # credentials fails inside the container in a far less obvious way. try { . $hookPath @@ -616,22 +616,22 @@ anvil-container *target: Write-Error "anvil: failed to load .anvil/container/hooks.ps1: $($_.Exception.Message)" exit 1 } - if (Get-Command Anvil-PreRun -CommandType Function -ErrorAction SilentlyContinue) { - [Console]::Error.WriteLine("anvil: running Anvil-PreRun from .anvil/container/hooks.ps1") + if (Get-Command Anvil-RunEnv -CommandType Function -ErrorAction SilentlyContinue) { + [Console]::Error.WriteLine("anvil: running Anvil-RunEnv from .anvil/container/hooks.ps1") try { - $hook = @(Anvil-PreRun | Where-Object { $_ }) | Select-Object -Last 1 + $hook = @(Anvil-RunEnv | Where-Object { $_ }) | Select-Object -Last 1 } catch { - Write-Error "anvil: Anvil-PreRun failed: $($_.Exception.Message)" + Write-Error "anvil: Anvil-RunEnv failed: $($_.Exception.Message)" exit 1 } $hookVars = if ($null -ne $hook) { $hook.Env } else { $null } if ($null -eq $hookVars -or $hookVars.Keys.Count -eq 0) { - Write-Error "anvil: Anvil-PreRun returned no variables; omit the function if the run needs none" + Write-Error "anvil: Anvil-RunEnv returned no variables; omit the function if the run needs none" exit 1 } foreach ($name in $hookVars.Keys) { if ([string]::IsNullOrWhiteSpace($hookVars[$name])) { - Write-Error "anvil: Anvil-PreRun returned an empty value for '$name'" + Write-Error "anvil: Anvil-RunEnv returned an empty value for '$name'" exit 1 } Set-Item -LiteralPath "Env:$name" -Value $hookVars[$name] diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index 638a50e1..acfcd698 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -584,11 +584,11 @@ if (-not $buildSecretsSupported) { } else { # A user writes this file by hand; the public catalog does not emit one. Write-Fixture $hooks @' -function Anvil-PreBuild { +function Anvil-BuildSecrets { @{ Secrets = @{ e2e_token = 'build-secret-value' } } } -function Anvil-PreRun { +function Anvil-RunEnv { @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } } '@ @@ -635,9 +635,9 @@ Write-Fixture $dockerfile ($dockerfileBody + $secretStanza) Write-Step 'rebuilding with the hook active' $hookRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure Assert-Equal 'the run with a hook succeeds' 0 $hookRun.ExitCode -Assert-That 'the hook announced itself at build time' ($hookRun.StdErr -match 'Anvil-PreBuild') +Assert-That 'the hook announced itself at build time' ($hookRun.StdErr -match 'Anvil-BuildSecrets') Assert-That 'the build secret was declared' ($hookRun.StdErr -match 'build secrets: e2e_token') -Assert-That 'the hook announced itself at run time' ($hookRun.StdErr -match 'Anvil-PreRun') +Assert-That 'the hook announced itself at run time' ($hookRun.StdErr -match 'Anvil-RunEnv') Assert-That 'forwarded names are reported' ($hookRun.StdErr -match 'forwarding env: ANVIL_E2E_RUNTIME') $secretReference = Get-ImageReference -Repo $repo @@ -668,7 +668,7 @@ Assert-That 'the run-time value reaches a recipe in the container' ` Write-Section '8. An empty hook value fails closed' Write-Fixture $hooks @' -function Anvil-PreBuild { +function Anvil-BuildSecrets { @{ Secrets = @{ e2e_token = '' } } } '@ @@ -684,11 +684,11 @@ Assert-That 'the failure names the offending secret' ($emptyHook.StdErr -match " Write-Section '9. Hook output does not change the tag' Write-Fixture $hooks @' -function Anvil-PreBuild { +function Anvil-BuildSecrets { @{ Secrets = @{ e2e_token = 'build-secret-value' } } } -function Anvil-PreRun { +function Anvil-RunEnv { @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } } '@ @@ -702,11 +702,11 @@ Assert-Equal 'restoring the hook restores the tag' $secretReference (Get-ImageRe # the file. Changing the file instead would only re-prove that file content is # hashed, which section 7 already covers. Write-Fixture $hooks @' -function Anvil-PreBuild { +function Anvil-BuildSecrets { @{ Secrets = @{ e2e_token = $env:ANVIL_E2E_MINT } } } -function Anvil-PreRun { +function Anvil-RunEnv { @{ Env = @{ ANVIL_E2E_RUNTIME = 'run-value' } } } '@ From 5d1d6f08efc8895846f92daf1acbc44573b51abd Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Mon, 24 Aug 2026 19:59:59 +0200 Subject: [PATCH 34/81] fix(anvil): capture main's workflow changes in the snapshots The merge commit recorded snapshots that were missing what main added to the generated workflows: the read-only `permissions` block with pr-fast's `pull-requests: write` restored on top, and `lfs: true` on the checkout steps. Cargo's fingerprint is the cause. The generator embeds its templates with `include_str!`, and the merge rewrote those files with mtimes older than the build that preceded it, so `cargo insta accept` ran against a stale binary and accepted output generated from the pre-merge templates. The snapshots agreed with a generator that no longer existed, which is why the tests passed. Regenerated after forcing a rebuild, and checked for stability: two further regenerate-and-compare cycles produce no pending snapshots, and `cargo anvil` converges against the tree. This is the second time this fingerprint has produced a false pass in this branch; the earlier one was caught by a control test rather than by the suite. Treat a snapshot accept immediately after a merge or a template edit as suspect unless the binary was rebuilt first. Validation: `cargo test -p cargo-anvil` -- 311 unit plus 51 across the other targets, 0 failed. --- .../snapshots/snapshots__ado_backend.snap | 36 +++-- .../snapshots/snapshots__github_backend.snap | 142 ++++++++++++++++-- .../snapshots/snapshots__local_only.snap | 36 +++-- 3 files changed, 166 insertions(+), 48 deletions(-) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 53b0c4ea..0b820496 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4273,8 +4273,7 @@ anvil-pr-fast: anvil-pr-fast-validate-prereqs \ anvil-audit \ anvil-udeps \ anvil-semver-check \ - anvil-external-types \ - anvil-aprz + anvil-external-types # Group-level setup + validate-prereqs # @@ -4299,8 +4298,7 @@ anvil-pr-fast-setup installer="install": \ (anvil-audit-setup installer) \ (anvil-udeps-setup installer) \ (anvil-semver-check-setup installer) \ - (anvil-external-types-setup installer) \ - (anvil-aprz-setup installer) + (anvil-external-types-setup installer) # Validate prerequisites for the `anvil-pr-fast` recipe. [group("anvil-setup")] @@ -4319,8 +4317,7 @@ anvil-pr-fast-validate-prereqs: \ anvil-audit-validate-prereqs \ anvil-udeps-validate-prereqs \ anvil-semver-check-validate-prereqs \ - anvil-external-types-validate-prereqs \ - anvil-aprz-validate-prereqs + anvil-external-types-validate-prereqs === justfiles/anvil/groups/pr-mutants.just === # Copyright (c) Microsoft Corporation. @@ -5225,9 +5222,9 @@ anvil-tool-pwsh-validate-prereqs: # or no-op if it is already installed at or above that version. The # `installer` parameter selects between: # - "install" (cargo install --locked, pure-source). Default. -# - "binstall" (cargo binstall --no-confirm --locked, with cargo install -# fallback if binstall fails). Bootstraps cargo-binstall -# itself if not on PATH. +# - "binstall" (cargo binstall --no-confirm --locked, with a controlled +# cargo install fallback for tools that declare source +# prerequisites). Bootstraps cargo-binstall itself if not on PATH. # `source_prereq`, when set, runs immediately before a source installation, # including a binstall fallback. [script("pwsh", "-NoProfile")] @@ -5275,7 +5272,15 @@ _install-tool name version installer source_prereq="": exit $LASTEXITCODE } } - cargo binstall --no-confirm --locked $name --version "=$version" + # Keep source builds behind Anvil's prerequisite check instead of + # allowing binstall to compile before that check can run. Tools with + # no source prerequisite may still use binstall's compile strategy. + $binstallArgs = @('binstall', '--no-confirm', '--locked') + if ($sourcePrereq) { + $binstallArgs += @('--disable-strategies', 'compile') + } + $binstallArgs += @($name, '--version', "=$version") + cargo @binstallArgs if ($LASTEXITCODE -eq 0) { exit 0 } Write-Host ' binstall failed; falling back to cargo install' -ForegroundColor Yellow } @@ -5798,10 +5803,9 @@ anvil-tool-cargo-udeps-validate-prereqs: (_check-tool "cargo-udeps" cargo_udeps_ # - On install (`-install` recipes): exactly this version (`=` for # cargo subcommands, exact ref for rustup toolchains). Pulling # "latest-matching" at install time is a cloud-workflow reproducibility risk -- -# an upstream release between yesterday's green build and today's -# PR can break things (cargo-spellcheck 0.15.7's em-dash regression -# is the canonical case). The `=` constraint locks the install to -# the version the catalog was validated against. +# an upstream release between yesterday's green build and today's PR can +# change behavior. The `=` constraint locks the install to the version the +# catalog was validated against. # - On validate-prereqs (`-validate-prereqs` recipes): the installed # version must be `>= `. A user who has manually upgraded a # tool for their own reasons (e.g. needing an unreleased bugfix) @@ -5837,7 +5841,7 @@ rust_nightly_external_types := "nightly-2026-03-20" # Cargo subcommands # ============================================================================ -cargo_aprz_version := "1.0.0" +cargo_aprz_version := "1.1.0" cargo_audit_version := "0.22.2" cargo_bolero_version := "0.13.4" cargo_careful_version := "0.4.10" @@ -5855,7 +5859,7 @@ cargo_mutants_version := "27.0.0" cargo_nextest_version := "0.9.137" cargo_semver_checks_version := "0.49.0" cargo_sort_version := "2.1.4" -cargo_spellcheck_version := "0.15.1" +cargo_spellcheck_version := "0.15.7" cargo_udeps_version := "0.1.61" === rustfmt.toml === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 5b2e3894..f2f597af 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -1011,6 +1011,12 @@ on: configured at Codecov; required for private repos. required: false +# The caller grants the maximum token scopes available to this reusable +# workflow. Reset jobs to read-only here, then restore only pr-fast's pull +# request scope below. See docs/design/github.md §9. +permissions: + contents: read + # Note on matrices: every multi-OS job below hardcodes its OS axis as # an inline YAML array. Per-leg runner *labels* are inputs (so adopters # can swap in self-hosted runners), but the OS axis itself is part of @@ -1039,6 +1045,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + lfs: true fetch-depth: 0 - id: delta uses: ./.github/actions/anvil-impact @@ -1052,11 +1059,15 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + lfs: true fetch-depth: 0 - id: delta uses: ./.github/actions/anvil-impact pr-fast: + permissions: + contents: read + pull-requests: write # Cross-OS / cross-arch because pr-fast contains compile-sensitive # checks (clippy, doc-build, udeps, semver-check, external-types) # whose results can differ across host for crates that use @@ -1073,6 +1084,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + lfs: true # anvil-semver-check compares the affected crates' current API # against the PR's target branch (origin/) via # `cargo semver-checks --baseline-rev`. That resolves the @@ -1129,6 +1141,8 @@ jobs: || inputs.windows_arm_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-pr-test with: free-disk-space: true @@ -1174,6 +1188,8 @@ jobs: || inputs.windows_arm_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-pr-runtime-analysis with: include_modified: ${{ startsWith(matrix.os, 'linux') && needs.impact-linux.outputs.include_modified || needs.impact-windows.outputs.include_modified }} @@ -1198,6 +1214,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: + lfs: true fetch-depth: 0 - uses: ./.github/actions/anvil-pr-mutants with: @@ -1229,6 +1246,9 @@ concurrency: jobs: anvil-pr: uses: ./.github/workflows/anvil-pr-impl.yml + # A called workflow cannot elevate beyond its caller. The implementation + # resets this upper bound to read-only and restores pull-requests:write + # only on pr-fast. See docs/design/github.md §9. permissions: contents: read # Write needed so the pr-fast job can upsert/clear the sticky PR @@ -1271,6 +1291,12 @@ on: configured at Codecov; required for private repos. required: false +# The caller grants the maximum token scopes available to this reusable +# workflow. Reset jobs to read-only here, then restore only the publisher's +# issues scope below. See docs/design/github.md §9. +permissions: + contents: read + # Note on matrices: see pr-impl-workflow.yml for the rationale. OS # matrices are hardcoded; per-leg runner labels are inputs. @@ -1286,6 +1312,8 @@ jobs: || inputs.windows_arm_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-scheduled-test with: free-disk-space: true @@ -1317,6 +1345,8 @@ jobs: || inputs.windows_arm_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-scheduled-advisories scheduled-runtime-analysis: @@ -1340,6 +1370,8 @@ jobs: || inputs.windows_arm_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-scheduled-runtime-analysis scheduled-exhaustive: @@ -1353,8 +1385,78 @@ jobs: runs-on: ${{ matrix.os == 'linux' && inputs.linux_runner || inputs.windows_runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + lfs: true - uses: ./.github/actions/anvil-scheduled-exhaustive + publish-failure: + name: Publish scheduled failure + needs: + - scheduled-test + - scheduled-advisories + - scheduled-runtime-analysis + - scheduled-exhaustive + if: ${{ always() && vars.ANVIL_PUBLISH_FAILURE_ISSUE != 'false' + && contains(needs.*.result, 'failure') }} + runs-on: ${{ inputs.linux_runner }} + permissions: + issues: write + steps: + - name: Create or update failure issue + uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 + env: + ANVIL_JOB_RESULTS: ${{ toJSON(needs) }} + with: + script: | + const title = "[Anvil] Scheduled checks failed"; + const marker = ""; + const runUrl = + `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}` + + `/actions/runs/${context.runId}`; + const results = JSON.parse(process.env.ANVIL_JOB_RESULTS); + const failedJobs = Object.entries(results) + .filter(([, job]) => job.result === "failure") + .map(([job]) => `- \`${job}\``) + .join("\n"); + const body = [ + marker, + "", + "The Anvil scheduled workflow failed.", + "", + "Failed jobs:", + failedJobs, + "", + `[View workflow run](${runUrl})`, + ].join("\n"); + + const query = + `repo:${context.repo.owner}/${context.repo.repo} ` + + `is:issue is:open in:body "anvil scheduled failure"`; + const { data: search } = + await github.rest.search.issuesAndPullRequests({ + q: query, + per_page: 100, + }); + const existing = search.items.find( + issue => issue.body?.includes(marker), + ); + + if (existing) { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: existing.number, + body, + }); + } else { + await github.rest.issues.create({ + owner: context.repo.owner, + repo: context.repo.repo, + title, + body, + }); + } + === .github/workflows/anvil-scheduled.yml === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. @@ -1374,8 +1476,12 @@ permissions: jobs: anvil-scheduled: uses: ./.github/workflows/anvil-scheduled-impl.yml + # A called workflow cannot elevate beyond its caller. The implementation + # resets this upper bound to read-only and restores issues:write only on + # publish-failure. See docs/design/github.md §9. permissions: contents: read + issues: write secrets: inherit === Cargo.toml === @@ -4194,8 +4300,7 @@ anvil-pr-fast: anvil-pr-fast-validate-prereqs \ anvil-audit \ anvil-udeps \ anvil-semver-check \ - anvil-external-types \ - anvil-aprz + anvil-external-types # Group-level setup + validate-prereqs # @@ -4220,8 +4325,7 @@ anvil-pr-fast-setup installer="install": \ (anvil-audit-setup installer) \ (anvil-udeps-setup installer) \ (anvil-semver-check-setup installer) \ - (anvil-external-types-setup installer) \ - (anvil-aprz-setup installer) + (anvil-external-types-setup installer) # Validate prerequisites for the `anvil-pr-fast` recipe. [group("anvil-setup")] @@ -4240,8 +4344,7 @@ anvil-pr-fast-validate-prereqs: \ anvil-audit-validate-prereqs \ anvil-udeps-validate-prereqs \ anvil-semver-check-validate-prereqs \ - anvil-external-types-validate-prereqs \ - anvil-aprz-validate-prereqs + anvil-external-types-validate-prereqs === justfiles/anvil/groups/pr-mutants.just === # Copyright (c) Microsoft Corporation. @@ -5146,9 +5249,9 @@ anvil-tool-pwsh-validate-prereqs: # or no-op if it is already installed at or above that version. The # `installer` parameter selects between: # - "install" (cargo install --locked, pure-source). Default. -# - "binstall" (cargo binstall --no-confirm --locked, with cargo install -# fallback if binstall fails). Bootstraps cargo-binstall -# itself if not on PATH. +# - "binstall" (cargo binstall --no-confirm --locked, with a controlled +# cargo install fallback for tools that declare source +# prerequisites). Bootstraps cargo-binstall itself if not on PATH. # `source_prereq`, when set, runs immediately before a source installation, # including a binstall fallback. [script("pwsh", "-NoProfile")] @@ -5196,7 +5299,15 @@ _install-tool name version installer source_prereq="": exit $LASTEXITCODE } } - cargo binstall --no-confirm --locked $name --version "=$version" + # Keep source builds behind Anvil's prerequisite check instead of + # allowing binstall to compile before that check can run. Tools with + # no source prerequisite may still use binstall's compile strategy. + $binstallArgs = @('binstall', '--no-confirm', '--locked') + if ($sourcePrereq) { + $binstallArgs += @('--disable-strategies', 'compile') + } + $binstallArgs += @($name, '--version', "=$version") + cargo @binstallArgs if ($LASTEXITCODE -eq 0) { exit 0 } Write-Host ' binstall failed; falling back to cargo install' -ForegroundColor Yellow } @@ -5719,10 +5830,9 @@ anvil-tool-cargo-udeps-validate-prereqs: (_check-tool "cargo-udeps" cargo_udeps_ # - On install (`-install` recipes): exactly this version (`=` for # cargo subcommands, exact ref for rustup toolchains). Pulling # "latest-matching" at install time is a cloud-workflow reproducibility risk -- -# an upstream release between yesterday's green build and today's -# PR can break things (cargo-spellcheck 0.15.7's em-dash regression -# is the canonical case). The `=` constraint locks the install to -# the version the catalog was validated against. +# an upstream release between yesterday's green build and today's PR can +# change behavior. The `=` constraint locks the install to the version the +# catalog was validated against. # - On validate-prereqs (`-validate-prereqs` recipes): the installed # version must be `>= `. A user who has manually upgraded a # tool for their own reasons (e.g. needing an unreleased bugfix) @@ -5758,7 +5868,7 @@ rust_nightly_external_types := "nightly-2026-03-20" # Cargo subcommands # ============================================================================ -cargo_aprz_version := "1.0.0" +cargo_aprz_version := "1.1.0" cargo_audit_version := "0.22.2" cargo_bolero_version := "0.13.4" cargo_careful_version := "0.4.10" @@ -5776,7 +5886,7 @@ cargo_mutants_version := "27.0.0" cargo_nextest_version := "0.9.137" cargo_semver_checks_version := "0.49.0" cargo_sort_version := "2.1.4" -cargo_spellcheck_version := "0.15.1" +cargo_spellcheck_version := "0.15.7" cargo_udeps_version := "0.1.61" === rustfmt.toml === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 883f434c..ea6d5ed1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -3013,8 +3013,7 @@ anvil-pr-fast: anvil-pr-fast-validate-prereqs \ anvil-audit \ anvil-udeps \ anvil-semver-check \ - anvil-external-types \ - anvil-aprz + anvil-external-types # Group-level setup + validate-prereqs # @@ -3039,8 +3038,7 @@ anvil-pr-fast-setup installer="install": \ (anvil-audit-setup installer) \ (anvil-udeps-setup installer) \ (anvil-semver-check-setup installer) \ - (anvil-external-types-setup installer) \ - (anvil-aprz-setup installer) + (anvil-external-types-setup installer) # Validate prerequisites for the `anvil-pr-fast` recipe. [group("anvil-setup")] @@ -3059,8 +3057,7 @@ anvil-pr-fast-validate-prereqs: \ anvil-audit-validate-prereqs \ anvil-udeps-validate-prereqs \ anvil-semver-check-validate-prereqs \ - anvil-external-types-validate-prereqs \ - anvil-aprz-validate-prereqs + anvil-external-types-validate-prereqs === justfiles/anvil/groups/pr-mutants.just === # Copyright (c) Microsoft Corporation. @@ -3965,9 +3962,9 @@ anvil-tool-pwsh-validate-prereqs: # or no-op if it is already installed at or above that version. The # `installer` parameter selects between: # - "install" (cargo install --locked, pure-source). Default. -# - "binstall" (cargo binstall --no-confirm --locked, with cargo install -# fallback if binstall fails). Bootstraps cargo-binstall -# itself if not on PATH. +# - "binstall" (cargo binstall --no-confirm --locked, with a controlled +# cargo install fallback for tools that declare source +# prerequisites). Bootstraps cargo-binstall itself if not on PATH. # `source_prereq`, when set, runs immediately before a source installation, # including a binstall fallback. [script("pwsh", "-NoProfile")] @@ -4015,7 +4012,15 @@ _install-tool name version installer source_prereq="": exit $LASTEXITCODE } } - cargo binstall --no-confirm --locked $name --version "=$version" + # Keep source builds behind Anvil's prerequisite check instead of + # allowing binstall to compile before that check can run. Tools with + # no source prerequisite may still use binstall's compile strategy. + $binstallArgs = @('binstall', '--no-confirm', '--locked') + if ($sourcePrereq) { + $binstallArgs += @('--disable-strategies', 'compile') + } + $binstallArgs += @($name, '--version', "=$version") + cargo @binstallArgs if ($LASTEXITCODE -eq 0) { exit 0 } Write-Host ' binstall failed; falling back to cargo install' -ForegroundColor Yellow } @@ -4538,10 +4543,9 @@ anvil-tool-cargo-udeps-validate-prereqs: (_check-tool "cargo-udeps" cargo_udeps_ # - On install (`-install` recipes): exactly this version (`=` for # cargo subcommands, exact ref for rustup toolchains). Pulling # "latest-matching" at install time is a cloud-workflow reproducibility risk -- -# an upstream release between yesterday's green build and today's -# PR can break things (cargo-spellcheck 0.15.7's em-dash regression -# is the canonical case). The `=` constraint locks the install to -# the version the catalog was validated against. +# an upstream release between yesterday's green build and today's PR can +# change behavior. The `=` constraint locks the install to the version the +# catalog was validated against. # - On validate-prereqs (`-validate-prereqs` recipes): the installed # version must be `>= `. A user who has manually upgraded a # tool for their own reasons (e.g. needing an unreleased bugfix) @@ -4577,7 +4581,7 @@ rust_nightly_external_types := "nightly-2026-03-20" # Cargo subcommands # ============================================================================ -cargo_aprz_version := "1.0.0" +cargo_aprz_version := "1.1.0" cargo_audit_version := "0.22.2" cargo_bolero_version := "0.13.4" cargo_careful_version := "0.4.10" @@ -4595,7 +4599,7 @@ cargo_mutants_version := "27.0.0" cargo_nextest_version := "0.9.137" cargo_semver_checks_version := "0.49.0" cargo_sort_version := "2.1.4" -cargo_spellcheck_version := "0.15.1" +cargo_spellcheck_version := "0.15.7" cargo_udeps_version := "0.1.61" === rustfmt.toml === From 961ea5bbaa32f9884145d195ff55cff86d595906 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 10:58:28 +0200 Subject: [PATCH 35/81] fix(anvil): correct the claims and links the recent rewrites invalidated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings that were verified as stated. Each is a place where the code moved and a description did not. Mine, from earlier commits in this branch: - The two renamed TOC entries in `containers.md` pointed at `#71-Anvil-BuildSecrets` and `#72-Anvil-RunEnv`. GitHub lowercases heading anchors, so both resolved to nothing and landed at the top of the page. The untouched `#73-anvil-resolveimage` beside them was the correct shape. - Removing `anvil-container-rebuild` left `anvil-container-down` with its summary twice, once above the rationale and once below, and the rationale ended "the counterpart to a rebuild, which discards the image and keeps these" -- wrong in both halves, since there is no rebuild recipe and `ANVIL_CONTAINER_NO_CACHE=1` rebuilds rather than discards. - Hashing the whole recipe tree left the emitted dockerignore header still saying "only the tool catalog within it decides what gets installed, and only that is hashed", and left `builder.rs` telling catalog authors that the build context "only admits *.just there". The context admits the whole `justfiles/anvil/` directory; it is the *identity* that considers only `*.just`, which is what makes a non-recipe file there dangerous rather than ignored. The rejection is right; its stated reason was not. Older: - `containers.md` said `container.just` "is reconciled on every run, so local edits to it are replaced", contrasted against the Dockerfile. Both are `Artifact::owned_file` with identical drift handling; they differ in header wording and in whether editing is invited. - The module overview described a two-artifact feature. `all()` emits three, and the ignore file must be replaced alongside a replacement Dockerfile. `hooks()` also carries image resolution, not only credentials. - `extensibility.md` linked `containers.md#9-customization`; renumbering made Customization §8 and Limitations §9, so the link pointed at the wrong section. - The `GITHUB_TOKEN` row said only that a host token is forwarded, omitting that one is derived from `gh auth token` when the target's plan reads the variable. That conditional derivation is the part with the exposure story. Behaviour, one case: - Inside the image, `just anvil-container` with no argument ran nothing and exited 0. `$LASTEXITCODE` had never been set, so a developer following the documented "no argument opens a shell" got no shell, no output and success. It now says it is already inside the container. Verified both branches: the bare form prints the notice, and a form with a target still passes through and returns the recipe's own exit code. - `ANVIL_CONTAINER_NO_CACHE` is documented with the one-shot form. The deleted recipe set it in its own process; an exported value is read by every later container command, so a forgotten one rebuilds from scratch each time with nothing to say why. Publicness was never the property that was lost. - The dogfood script's engine and prerequisite probes accepted any `Get-Command` match, including a function or alias, while `Invoke-Native` runs `Start-Process -FilePath`, which needs a real executable. Constrained to `-CommandType Application`. Validation: `cargo test -p cargo-anvil` -- 311 unit plus 51 across the other targets. Snapshots re-verified after a forced rebuild rather than trusting the accept, since a stale `include_str!` fingerprint has produced a false pass twice in this branch. `just --list` still renders four whole sentences. --- .anvil.lock | 6 ++--- .anvil/container/Dockerfile.dockerignore | 4 ++-- crates/cargo-anvil/README.md | 13 ++++++++--- crates/cargo-anvil/docs/design/containers.md | 19 ++++++++++++---- .../cargo-anvil/docs/design/extensibility.md | 2 +- .../src/anvil/artifacts/container.rs | 22 +++++++++++-------- crates/cargo-anvil/src/catalog/builder.rs | 12 +++++----- crates/cargo-anvil/src/lib.rs | 11 ++++++++-- .../anvil/container/Dockerfile.dockerignore | 4 ++-- .../templates/justfiles/anvil/container.just | 15 ++++++++----- .../snapshots/snapshots__ado_backend.snap | 19 ++++++++++------ .../snapshots/snapshots__github_backend.snap | 19 ++++++++++------ .../snapshots/snapshots__local_only.snap | 19 ++++++++++------ justfiles/anvil/container.just | 15 ++++++++----- scripts/test-anvil-dogfood.ps1 | 6 ++--- 15 files changed, 121 insertions(+), 65 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 516cf328..60acb1fd 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:7b9adef7104e46574fa677c49e2fad719224b4b3af03144fb7339772a61de0bb" +catalog_checksum = "sha256:4086dc1dfbaeafddbefe5e1bfcaf348e32a69e281af3605031122a32019deb46" [[file]] path = ".anvil/container/Dockerfile" @@ -9,7 +9,7 @@ checksum = "sha256:a3e106b7dbde0bb6a9c2b94dae0f9cb82b9055ea97f100438f3d3d20ec66e [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:167211852f43c2a0d2c18c4de10bacb186a67cc6408bc61b66a3571f444d7d3d" +checksum = "sha256:6e0f2fa763766c09ba05e5f48aeca8fbc3894125208e1ee2de8d433a5dbbfcef" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:bcedc35a6eecf4ffe5e5a075ca731ff9cbab201471e4017395c0144da690ac47" +checksum = "sha256:ad2a465eb92a55c97f56cd9b4ff5bbfc0724d0d48aa397653ea689fc8eebe98a" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index e4382bbf..e7e522cf 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -14,8 +14,8 @@ # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` # so that a cold build does not stream unrelated trees to the daemon. The # recipes are copied to drive `just anvil-setup`, which needs the whole tree to -# parse; only the tool catalog within it decides what gets installed, and only -# that is hashed. +# parse, and the whole tree is hashed into the image tag: the tier, group and +# check recipes decide which tools `anvil-setup` reaches, not just the catalog. * !justfiles justfiles/* diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 07ceba64..23488f2d 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -183,12 +183,19 @@ exactly the reference a consumer will later look up. |`ANVIL_CONTAINER_NO_RESOLVE=1`|Skip the resolve hook, so a query never pulls.| |`ANVIL_CONTAINER_NO_CACHE=1`|Rebuild a tag that already resolves, ignoring the hook.| |`ANVIL_IN_CONTAINER=1`|Set inside the image; makes a nested invocation run natively.| -|`GITHUB_TOKEN`|Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited.| +|`GITHUB_TOKEN`|Forwarded when set on the host. When it is not, one is derived from `gh auth token` — but only for a target whose plan reads the variable, or for the interactive shell.| Supporting recipes: `anvil-container-tag`, `anvil-container-status` (reports the engine and image without building or pulling), and `anvil-container-down` (removes this repository’s cache volumes). To rebuild -a tag that already resolves, set `ANVIL_CONTAINER_NO_CACHE=1` above. +a tag that already resolves, scope `ANVIL_CONTAINER_NO_CACHE` to the one +invocation — an exported value is read by *every* later container command, +so a forgotten one rebuilds from scratch each time: + +```text +$env:ANVIL_CONTAINER_NO_CACHE = '1' +try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +``` #### The hook @@ -461,7 +468,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbSYqfQlDuez4bxOpm9tVy5dwb5v23EdpLmHAb3Yt_7tHlH1thZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb-v7S1O3ujgwbrHNxDZB-H40b07anVywfLZ8bKnb9-uilr1BhZIGDa2NhcmdvLWFudmlsZTAuNC4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.4.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 53791f7e..98096a70 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -32,8 +32,8 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [6.2 Docker](#62-docker) - [6.3 Podman](#63-podman) - [7. The hook](#7-the-hook) - - [7.1 Anvil-BuildSecrets](#71-Anvil-BuildSecrets) - - [7.2 Anvil-RunEnv](#72-Anvil-RunEnv) + - [7.1 Anvil-BuildSecrets](#71-anvil-buildsecrets) + - [7.2 Anvil-RunEnv](#72-anvil-runenv) - [7.3 Anvil-ResolveImage](#73-anvil-resolveimage) - [7.4 Trust boundary](#74-trust-boundary) - [8. Customization](#8-customization) @@ -81,6 +81,16 @@ There is deliberately no `anvil-container-rebuild`. Its whole body would be `ANV the ordinary resolve, and that variable is already public below — where it also composes with `NO_REBUILD` and `NO_RESOLVE`, which a recipe form does not. +What the recipe did supply was **scope**: it set the variable in its own process and exited, so exactly one build +ignored the cache. An exported variable is sticky, and every container command reads it, so a forgotten +`ANVIL_CONTAINER_NO_CACHE` rebuilds from scratch on each later invocation with nothing to indicate why. Scope it to +the one run: + +```powershell +$env:ANVIL_CONTAINER_NO_CACHE = '1' +try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +``` + | Variable | Effect | | --- | --- | | `ANVIL_CONTAINER_ENGINE` | `docker` (default) or `podman`. Read at run time (§6.1). | @@ -107,8 +117,9 @@ repo/ └── hooks.ps1 optional; not emitted by default (§7) ``` -`container.just` is reconciled on every run, so local edits to it are replaced. The `Dockerfile` and its ignore file -are generated but intended to be edited; anvil's drift handling preserves a repository's changes to them (§8). +`container.just` and the `Dockerfile` are both owned files with the same drift handling: anvil preserves a repository's +edit and reports a proposal rather than overwriting it (`updates.md` §2). They differ in header wording and in whether +editing is *invited* — the Dockerfile is meant to be extended (§8), the driver is not. The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index 5b0d5916..d7754292 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -482,7 +482,7 @@ in a tool-owned directory of their own, such as `.anvil/`. Containerized execution is itself an ordinary artifact group, customized with the same `replace_artifact` / `with_artifact` / `without_artifact` levers as anything else. The artifacts it exposes and the contract each one carries are -specified in [containers.md](./containers.md#9-customization). +specified in [containers.md](./containers.md#8-customization). The public engine contains no environment-specific image, registry, cloud, or credential-provider details. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 8d525afe..a5dfc6b2 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -3,20 +3,24 @@ //! Containerized execution: the `anvil-container` recipe and the image it runs. //! -//! Two artifacts define the whole feature. The recipe drives the engine and -//! computes the image identity; the Dockerfile (with its build-context ignore -//! file) defines what the image contains. There is no configuration file: -//! whether the group is emitted at all is a catalog decision, and the only -//! host-specific value — which engine to call — is an environment variable read -//! by the recipe at run time. +//! Three artifacts define the whole feature. The recipe drives the engine and +//! computes the image identity; the Dockerfile defines what the image contains; +//! its build-context ignore file decides what reaches the build at all, and the +//! two must be replaced together. There is no configuration file: whether the +//! group is emitted at all is a catalog decision, and the only host-specific +//! value — which engine to call — is an environment variable read by the recipe +//! at run time. //! //! A downstream catalog customizes exactly two things, and inherits everything //! else: //! //! - [`dockerfile`] plus [`Artifact::with_body`] to build on a different base -//! or install the toolchain from a different source. -//! - [`hooks`] to supply credentials, which the recipe loads when the file is -//! present regardless of who put it there. +//! or install the toolchain from a different source. A replacement that +//! copies more of the tree must replace [`dockerignore`] with it, or the +//! added paths never reach the build context. +//! - [`hooks`] to supply credentials, or to resolve a published image through +//! `Anvil-ResolveImage`. The recipe loads the file when it is present, +//! regardless of who put it there. use crate::catalog::Artifact; diff --git a/crates/cargo-anvil/src/catalog/builder.rs b/crates/cargo-anvil/src/catalog/builder.rs index 6343dbb8..42af0307 100644 --- a/crates/cargo-anvil/src/catalog/builder.rs +++ b/crates/cargo-anvil/src/catalog/builder.rs @@ -238,11 +238,13 @@ impl CatalogBuilder { } } -/// `justfiles/` is the recipe tree: the container image identity and the -/// Docker build-context allow-list only ever consider `*.just` files below it, -/// so any other owned file placed there would be silently dropped from both. +/// `justfiles/` is the recipe tree: the container image identity considers only +/// `*.just` files below it, while the Docker build context admits the whole +/// `justfiles/anvil/` directory. A non-recipe owned file placed there would +/// therefore be copied into the image without being part of its tag, so editing +/// it would change what the image contains while the tag still resolved. /// Reject it at catalog-construction time instead, so a derived catalog fails -/// loudly rather than shipping a file the container backend ignores. +/// loudly rather than shipping a file that silently breaks image identity. fn non_recipe_under_justfiles(artifact: &Artifact) -> Option { let Artifact::OwnedFile(spec) = artifact else { return None; @@ -252,7 +254,7 @@ fn non_recipe_under_justfiles(artifact: &Artifact) -> Option { return None; } Some(format!( - "owned file '{}' is not a .just recipe; non-recipe artifacts must live outside justfiles/ (the container image identity and build context only admit *.just there)", + "owned file '{}' is not a .just recipe; non-recipe artifacts must live outside justfiles/ (only *.just there is part of the container image identity)", spec.path )) } diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 506b07fd..24216001 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -184,12 +184,19 @@ //! | `ANVIL_CONTAINER_NO_RESOLVE=1` | Skip the resolve hook, so a query never pulls. | //! | `ANVIL_CONTAINER_NO_CACHE=1` | Rebuild a tag that already resolves, ignoring the hook. | //! | `ANVIL_IN_CONTAINER=1` | Set inside the image; makes a nested invocation run natively. | -//! | `GITHUB_TOKEN` | Forwarded into the run when set on the host, so `anvil-aprz` is not rate-limited. | +//! | `GITHUB_TOKEN` | Forwarded when set on the host. When it is not, one is derived from `gh auth token` — but only for a target whose plan reads the variable, or for the interactive shell. | //! //! Supporting recipes: `anvil-container-tag`, `anvil-container-status` //! (reports the engine and image without building or pulling), and //! `anvil-container-down` (removes this repository's cache volumes). To rebuild -//! a tag that already resolves, set `ANVIL_CONTAINER_NO_CACHE=1` above. +//! a tag that already resolves, scope `ANVIL_CONTAINER_NO_CACHE` to the one +//! invocation — an exported value is read by *every* later container command, +//! so a forgotten one rebuilds from scratch each time: +//! +//! ```text +//! $env:ANVIL_CONTAINER_NO_CACHE = '1' +//! try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +//! ``` //! //! ### The hook //! diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore index e4382bbf..e7e522cf 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore @@ -14,8 +14,8 @@ # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` # so that a cold build does not stream unrelated trees to the daemon. The # recipes are copied to drive `just anvil-setup`, which needs the whole tree to -# parse; only the tool catalog within it decides what gets installed, and only -# that is hashed. +# parse, and the whole tree is hashed into the image tag: the tier, group and +# check recipes decide which tools `anvil-setup` reaches, not just the catalog. * !justfiles justfiles/* diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 23e87e4c..3471fa03 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -416,7 +416,14 @@ anvil-container *target: $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -gt 0) { just @targetParts } + if ($targetParts.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this *is* + # that shell. Running nothing and exiting 0 would be the one outcome + # that looks like it worked. + [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") + exit 0 + } + just @targetParts exit $LASTEXITCODE } @@ -704,12 +711,10 @@ anvil-container-status: } exit 0 -# Remove this repository's cache volumes. The image is left in place. -# # Only the download caches are volumes, so this discards fetched crates and git # checkouts and nothing else: the next run re-fetches them, and the image's own -# tools are untouched. The counterpart to a rebuild, which discards the image -# and keeps these. +# tools are untouched. To discard the image instead, set +# ANVIL_CONTAINER_NO_CACHE=1 for a single run. # Remove this repository's cache volumes. The image is left in place. [group("anvil-container")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 0b820496..2f3e12e0 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -153,8 +153,8 @@ CMD ["bash"] # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` # so that a cold build does not stream unrelated trees to the daemon. The # recipes are copied to drive `just anvil-setup`, which needs the whole tree to -# parse; only the tool catalog within it decides what gets installed, and only -# that is hashed. +# parse, and the whole tree is hashed into the image tag: the tier, group and +# check recipes decide which tools `anvil-setup` reaches, not just the catalog. * !justfiles justfiles/* @@ -3925,7 +3925,14 @@ anvil-container *target: $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -gt 0) { just @targetParts } + if ($targetParts.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this *is* + # that shell. Running nothing and exiting 0 would be the one outcome + # that looks like it worked. + [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") + exit 0 + } + just @targetParts exit $LASTEXITCODE } @@ -4213,12 +4220,10 @@ anvil-container-status: } exit 0 -# Remove this repository's cache volumes. The image is left in place. -# # Only the download caches are volumes, so this discards fetched crates and git # checkouts and nothing else: the next run re-fetches them, and the image's own -# tools are untouched. The counterpart to a rebuild, which discards the image -# and keeps these. +# tools are untouched. To discard the image instead, set +# ANVIL_CONTAINER_NO_CACHE=1 for a single run. # Remove this repository's cache volumes. The image is left in place. [group("anvil-container")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f2f597af..d8a44c14 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -153,8 +153,8 @@ CMD ["bash"] # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` # so that a cold build does not stream unrelated trees to the daemon. The # recipes are copied to drive `just anvil-setup`, which needs the whole tree to -# parse; only the tool catalog within it decides what gets installed, and only -# that is hashed. +# parse, and the whole tree is hashed into the image tag: the tier, group and +# check recipes decide which tools `anvil-setup` reaches, not just the catalog. * !justfiles justfiles/* @@ -3952,7 +3952,14 @@ anvil-container *target: $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -gt 0) { just @targetParts } + if ($targetParts.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this *is* + # that shell. Running nothing and exiting 0 would be the one outcome + # that looks like it worked. + [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") + exit 0 + } + just @targetParts exit $LASTEXITCODE } @@ -4240,12 +4247,10 @@ anvil-container-status: } exit 0 -# Remove this repository's cache volumes. The image is left in place. -# # Only the download caches are volumes, so this discards fetched crates and git # checkouts and nothing else: the next run re-fetches them, and the image's own -# tools are untouched. The counterpart to a rebuild, which discards the image -# and keeps these. +# tools are untouched. To discard the image instead, set +# ANVIL_CONTAINER_NO_CACHE=1 for a single run. # Remove this repository's cache volumes. The image is left in place. [group("anvil-container")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index ea6d5ed1..10e85330 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -153,8 +153,8 @@ CMD ["bash"] # The context is narrowed to `justfiles/anvil/` rather than all of `justfiles/` # so that a cold build does not stream unrelated trees to the daemon. The # recipes are copied to drive `just anvil-setup`, which needs the whole tree to -# parse; only the tool catalog within it decides what gets installed, and only -# that is hashed. +# parse, and the whole tree is hashed into the image tag: the tier, group and +# check recipes decide which tools `anvil-setup` reaches, not just the catalog. * !justfiles justfiles/* @@ -2665,7 +2665,14 @@ anvil-container *target: $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -gt 0) { just @targetParts } + if ($targetParts.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this *is* + # that shell. Running nothing and exiting 0 would be the one outcome + # that looks like it worked. + [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") + exit 0 + } + just @targetParts exit $LASTEXITCODE } @@ -2953,12 +2960,10 @@ anvil-container-status: } exit 0 -# Remove this repository's cache volumes. The image is left in place. -# # Only the download caches are volumes, so this discards fetched crates and git # checkouts and nothing else: the next run re-fetches them, and the image's own -# tools are untouched. The counterpart to a rebuild, which discards the image -# and keeps these. +# tools are untouched. To discard the image instead, set +# ANVIL_CONTAINER_NO_CACHE=1 for a single run. # Remove this repository's cache volumes. The image is left in place. [group("anvil-container")] diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 23e87e4c..3471fa03 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -416,7 +416,14 @@ anvil-container *target: $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -gt 0) { just @targetParts } + if ($targetParts.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this *is* + # that shell. Running nothing and exiting 0 would be the one outcome + # that looks like it worked. + [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") + exit 0 + } + just @targetParts exit $LASTEXITCODE } @@ -704,12 +711,10 @@ anvil-container-status: } exit 0 -# Remove this repository's cache volumes. The image is left in place. -# # Only the download caches are volumes, so this discards fetched crates and git # checkouts and nothing else: the next run re-fetches them, and the image's own -# tools are untouched. The counterpart to a rebuild, which discards the image -# and keeps these. +# tools are untouched. To discard the image instead, set +# ANVIL_CONTAINER_NO_CACHE=1 for a single run. # Remove this repository's cache volumes. The image is left in place. [group("anvil-container")] diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index d646477c..7f0fe5d6 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -182,10 +182,10 @@ function Resolve-Engine([string]$Name) { # Mirrors what container.just does: prefer the engine on PATH, and fall # back to the default WSL distribution on Windows. The script must not # assume more than the product does. - if (Get-Command $Name -ErrorAction SilentlyContinue) { + if (Get-Command $Name -CommandType Application -ErrorAction SilentlyContinue) { return [pscustomobject]@{ Exe = $Name; Prefix = @(); ViaWsl = $false } } - if ($IsWindows -and (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { + if ($IsWindows -and (Get-Command wsl.exe -CommandType Application -ErrorAction SilentlyContinue)) { & wsl.exe --exec $Name --version *> $null if ($LASTEXITCODE -eq 0) { return [pscustomobject]@{ Exe = 'wsl.exe'; Prefix = @('--exec', $Name); ViaWsl = $true } @@ -455,7 +455,7 @@ Write-Host "repository: $RepoRoot" -ForegroundColor DarkGray Write-Host "tier: $(if ($SkipTier) { '(skipped)' } else { $Tier -join ', ' })" -ForegroundColor DarkGray foreach ($tool in @('just', 'cargo')) { - if (-not (Get-Command $tool -ErrorAction SilentlyContinue)) { + if (-not (Get-Command $tool -CommandType Application -ErrorAction SilentlyContinue)) { Write-Host "FAIL $tool is required on PATH" -ForegroundColor Red exit 1 } From f0db147f17e8c43975f10e102d00d1f83de08a95 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 11:01:04 +0200 Subject: [PATCH 36/81] docs(anvil): name both owned-file marker exceptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `updates.md` described the ADO `steps/job.yml` wrapper as *the* deliberate exception to the `DO NOT EDIT` wording. `artifacts/mod.rs` now accepts a second: `.anvil/container/Dockerfile*` carries the weaker `Managed by cargo-anvil.` marker, because §8 invites a repository to edit it in place for a different base or extra packages. The invariant has two exemptions; the normative description named one. Validation: `cargo test -p cargo-anvil` clean, `cargo anvil --dry-run` converges. --- crates/cargo-anvil/docs/design/updates.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/crates/cargo-anvil/docs/design/updates.md b/crates/cargo-anvil/docs/design/updates.md index e0b36b63..b214ba49 100644 --- a/crates/cargo-anvil/docs/design/updates.md +++ b/crates/cargo-anvil/docs/design/updates.md @@ -170,9 +170,15 @@ the source of truth and pointing readers at the update workflow: This warning is informational only. Ownership remains path-based in `.anvil.lock`. If a repository edits an owned file, cargo-anvil preserves the edit and reports a proposal instead of overwriting it; `cargo anvil --dry-run` shows that decision. -The ADO `steps/job.yml` extension wrapper is the deliberate exception to the -"do not edit" wording: its header says it is emitted by cargo-anvil and explicitly -invites repository customization because that file is the supported 1ESPT hook. +Two files are deliberate exceptions to the "do not edit" wording, and both carry +a weaker provenance marker instead. The ADO `steps/job.yml` extension wrapper +says it is emitted by cargo-anvil and explicitly invites repository +customization, because that file is the supported 1ESPT hook. The container +`Dockerfile` and its ignore file (`.anvil/container/Dockerfile*`) are marked +"Managed by cargo-anvil." for the same reason: a repository that needs a +different base or extra packages edits them in place, and drift handling +preserves the edit (see [containers.md](./containers.md#8-customization)). A +"DO NOT EDIT" marker would contradict the customization path both are for. ## 3. Managed regions From 0a8eb6c8c8e69e92f62aadb2c7b213c12fd36047 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 11:31:30 +0200 Subject: [PATCH 37/81] test(anvil): cover the container behaviour changes and the 0.4.0 upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review asked for coverage of three things this branch changed that nothing exercised unattended, plus three corrections. Tests: - `container_upgrade.rs` returns, aimed at this release rather than the asset move it originally covered. It drives the generator over a tree shaped like a 0.4.0 install -- the runner seam, `Containerfile`, `entrypoint.sh`, `image-id.*`, `run-in-container.*` and `runner.just`, all tracked in the lock -- and asserts each is `Remove`d, that this release's artifacts replace them, and that the root Justfile survives the region excision. A second case covers the outcome that actually costs a user work: an *edited* `Containerfile` is `OrphanedKept`, preserved byte for byte, with only the lock entry dropped. - `anvil-aprz` warns and proceeds without a token instead of throwing, which is what stops a containerized tier aborting on a missing credential. The dogfood run normally has a host token and the container E2E runs a custom recipe, so nothing saw it. The stub is `gh.cmd`, not `gh.ps1`: `.ps1` is not in PATHEXT, so a script stub is skipped and the host's real `gh` answers -- which on a signed-in machine returns a live token and silently tests nothing. That is how the first version of this test passed while asserting on an empty stream. - `anvil-mutants-diff` diffs the base against the working tree. CI cannot catch a regression here because its tree is clean, which is exactly the case the fix addresses. The test builds a real repository with a committed change and an uncommitted one and asserts both reach the diff. Control-tested: restoring `$base..HEAD` fails it on the uncommitted hunk, and the fix passes it. `core.autocrlf` is pinned in the fixture, since a host set to rewrite line endings rejects it outright. Corrections: - `anvil-aprz` is in `scheduled-advisories`, not `pr-fast` -- it was moved off the PR critical path. Three places justified the GITHUB_TOKEN derivation with the old group. The argument is unaffected: the predicate is the variable, not the check name. - The interpolation guard scanned for `'{{` only, so it could not see the defect it was written for: an interpolation escaped for a single-quoted literal that lands in a double-quoted one, where `''` is literal and `$` stays live. It now visits every `{{` and classifies by the preceding quote, rejecting any non-exempt interpolation into a double-quoted literal. - §8 promised a replacement Dockerfile could register extra copied sources with the digest; §4.1 fixes the input set and offers no way to. The section now states the tension plainly and gives the two honest options. - §2 documents the 0.4.0 migration: `ANVIL_RUNNER`, the `anvil-runner` region, `runner.just`, the shadow recipes and the retired assets, plus the 0.4.0 cache volumes this release does not reuse and `anvil-container-down` cannot remove. An adopter who opted in via `ANVIL_RUNNER` silently reverts to native execution otherwise. `checksum_str` is exported from `test_support` so a test can record a manifest entry the way anvil does. Validation: `cargo test -p cargo-anvil` -- 311 unit, 2 upgrade, 9 recipe contract, 3 snapshot, 51 across the rest, 0 failed. `cargo anvil --dry-run` converges at 79/79. --- crates/cargo-anvil/docs/design/containers.md | 34 ++- .../src/anvil/artifacts/container.rs | 40 +++- crates/cargo-anvil/src/lib.rs | 1 + crates/cargo-anvil/tests/container_upgrade.rs | 200 ++++++++++++++++++ crates/cargo-anvil/tests/recipe_contracts.rs | 151 +++++++++++++ scripts/test-anvil-container.ps1 | 2 +- 6 files changed, 415 insertions(+), 13 deletions(-) create mode 100644 crates/cargo-anvil/tests/container_upgrade.rs diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 98096a70..d159578a 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -62,6 +62,25 @@ own agents. The image is pinned to resemble that environment, not to reproduce i just anvil-container anvil-setup binstall ``` +**Upgrading from 0.4.0.** The routing seam is gone, and its removal is silent for the repositories most likely to +care. 0.4.0 let a repository opt into containers by exporting `ANVIL_RUNNER=container`, which routed a tier through +`_anvil-run`; anyone who did that left the generated `anvil-runner` region byte-identical, so regeneration classifies +it `Remove` and deletes it without comment. From the next `just anvil-pr` the tier runs natively on the host +toolchain, with no diagnostic and no behaviour anvil can detect. What was removed: + +| Removed in this release | Replacement | +| --- | --- | +| `ANVIL_RUNNER` environment variable | none — name the container explicitly with `just anvil-container ` | +| the `anvil-runner` managed region in the root `Justfile` | none | +| `justfiles/anvil/runner.just` | `justfiles/anvil/container.just` | +| the `_anvil-pr`, `_anvil-scheduled` and `_anvil-full` shadow recipes | the tiers themselves, which now only run natively | +| `.anvil/container/run-in-container.{sh,ps1}`, `entrypoint.sh`, `image-id.{sh,ps1}`, `Containerfile*` | `.anvil/container/Dockerfile` and its ignore file | + +A 0.4.0 installation also holds cache volumes named `anvil-cargo-registry-`, `anvil-cargo-git-` and +`anvil-target--`. This release keys volume names on the repository *directory name* instead, so +none of those are reused and `anvil-container-down` does not remove them; `anvil-target-*` in particular holds a full +workspace `target/`. Remove them once with the engine directly. + Arguments are whitespace-delimited tokens. `just` joins a variadic `*target` with spaces before the recipe body sees it, so the original argv is unrecoverable and an argument containing a space does not round-trip. No catalog recipe takes one; a fork whose recipes do should pass them through the environment instead. @@ -254,7 +273,7 @@ otherwise the engine leaves it as `/`, and anything falling back to `$HOME` writ ### 5.3 Environment The run passes `ANVIL_IN_CONTAINER=1` (§5.4) and forwards `GITHUB_TOKEN` by name, resolved the way the recipe resolves -it natively: the environment first, then the gh CLI's stored token. `anvil-aprz` runs in the `pr-fast` group and +it natively: the environment first, then the gh CLI's stored token. `anvil-aprz` runs in the `scheduled-advisories` group and queries the GitHub advisory API, which allows 60 requests an hour unauthenticated and then sleeps until the quota resets, so a tier needs the token to terminate rather than merely to run quickly. @@ -495,9 +514,16 @@ its own version against a file it can see has diverged. A change that belongs ev **The Dockerfile and its ignore file must be replaced together.** A replacement that `COPY`s anything beyond `justfiles/anvil/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the -added paths never reach the build context and the build fails on a missing file. A replacement that installs tools -from a source outside `justfiles/anvil/` must also add that source to the digest, or a change to it will name a tag -that already resolves. +added paths never reach the build context and the build fails on a missing file. + +**A replacement cannot extend the digest, so anything extra it copies must not vary independently.** The hashed set +is fixed (§4.1) and a fork has no way to add to it, which puts this customization path in tension with the identity +guarantee: an installer script, a config file or a certificate copied by a replacement Dockerfile sits outside the +tag, so editing it changes what the image contains while naming a reference that already resolves — and the stale +image is reused rather than rebuilt. Until a catalog can contribute digest inputs, the honest options are to carry +that content inside the Dockerfile itself, where it *is* hashed, or to accept that changing it needs +`ANVIL_CONTAINER_NO_CACHE=1` to take effect. A manual escape hatch is not content identity, so treat the second as a +workaround rather than a supported contract. `justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces: a non-recipe file there would be copied into the image without being part of its identity, so editing it would change the image's diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index a5dfc6b2..cc469774 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -303,17 +303,41 @@ mod tests { // The deleted runner.just escaped every interpolation; this guards // against losing that again. // + // Every `{{` is visited, not just those preceded by an apostrophe, + // because the defect found in review was the other case: an + // interpolation escaped with `replace(…, "'", "''")` -- correct for a + // '…' literal -- that landed inside a "…" literal, where the doubling + // renders as a literal '' and `$` stays live. Scanning only for `'{{` + // cannot see that site at all, since it does not begin with a quote. + // // Two variables are exempt and checked explicitly below: the image name // is regex-sanitized at definition, and the workdir is a literal. const EXEMPT: [&str; 2] = ["{{anvil_container_name}}", "{{anvil_container_workdir}}"]; - for (index, _) in RECIPE.match_indices("'{{") { - let tail = &RECIPE[index + 1..]; + for (index, _) in RECIPE.match_indices("{{") { + let tail = &RECIPE[index..]; + if EXEMPT.iter().any(|exempt| tail.starts_with(exempt)) { + continue; + } + // The quote this interpolation is being pasted into, if any. + let quote = RECIPE[..index].chars().next_back(); let escaped = tail.starts_with("{{ replace("); - assert!( - escaped || EXEMPT.iter().any(|exempt| tail.starts_with(exempt)), - "unescaped interpolation into a PowerShell literal at byte {index}: {}", - &tail[..tail.len().min(60)] - ); + match quote { + // A single-quoted literal needs just's doubling form. + Some('\'') => assert!( + escaped, + "unescaped interpolation into a PowerShell literal at byte {index}: {}", + &tail[..tail.len().min(60)] + ), + // A double-quoted literal is the reviewed hazard: `''` doubling + // does not escape there and `$` keeps expanding, so only an + // exempt name is safe. Interpolating anything else needs a + // single-quoted literal instead. + Some('"') => panic!( + "interpolation into a double-quoted PowerShell literal at byte {index}: {}", + &tail[..tail.len().min(60)] + ), + _ => {} + } } // And the escaping that is present uses just's own doubling form. assert!(RECIPE.contains(r#"replace(justfile_directory(), "'", "''")"#)); @@ -441,7 +465,7 @@ mod tests { #[test] fn a_host_token_is_resolved_as_the_recipe_does_and_forwarded_by_name() { - // anvil-aprz is in pr-fast, and unauthenticated it does not merely + // anvil-aprz is in scheduled-advisories, and unauthenticated it does not merely // warn: `cargo aprz deps` sleeps until the hourly quota resets, so a // containerized tier blocks for up to an hour. The driver therefore // resolves a token the same way the recipe does natively -- the diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 24216001..f4b11441 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -487,6 +487,7 @@ pub(crate) mod workspace; /// `artifacts`, `run_app`, …) instead. #[doc(hidden)] pub mod test_support { + pub use crate::checksum::checksum_str; pub use crate::cli::Cli; pub use crate::decision::Decision; pub use crate::manifest::{MANIFEST_FILE_NAME, Manifest}; diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs new file mode 100644 index 00000000..b82e71af --- /dev/null +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -0,0 +1,200 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#![cfg(not(miri))] // miri can't sandbox the FS ops these tests do (TempDir, run_update). +#![allow( + clippy::expect_used, + clippy::unwrap_used, + reason = "panic-on-failure idioms are appropriate in tests" +)] +#![expect(clippy::unwrap_used, reason = "integration tests favor concise assertions over Result plumbing")] +#![expect( + clippy::panic, + reason = "integration tests panic on unmet preconditions for readable failure output" +)] + +//! Consumer-upgrade coverage for the container backend replacement. +//! +//! The snapshot tests describe a fresh tree: the end state of a generation that +//! starts from nothing. This exercises the path an existing adopter takes -- +//! a repository generated by 0.4.0, holding a `.anvil.lock` that tracks the +//! runner seam and its assets, updated by a binary that emits neither. +//! +//! The transition is the largest this crate has shipped: `Containerfile` and +//! its ignore file, `README.md`, `entrypoint.sh`, `image-id.{sh,ps1}` and +//! `run-in-container.{sh,ps1}` all disappear from `.anvil/container/`, +//! `justfiles/anvil/runner.just` disappears with them, and the root `Justfile` +//! loses its `anvil-runner` region. What matters is that an untouched asset is +//! removed cleanly and an *edited* one is handed back to the repository rather +//! than deleted. + +use std::path::Path; + +use cargo_anvil::test_support::{Cli, Decision, Manifest, RunOutcome, Target, checksum_str, run_update}; +use cargo_anvil::{Catalog, artifacts}; +use tempfile::TempDir; + +/// Generated container assets that 0.4.0 tracked and this release does not. +/// Paths are as 0.4.0 wrote them. +const RETIRED_ASSETS: [&str; 8] = [ + ".anvil/container/Containerfile", + ".anvil/container/Containerfile.dockerignore", + ".anvil/container/README.md", + ".anvil/container/entrypoint.sh", + ".anvil/container/image-id.ps1", + ".anvil/container/image-id.sh", + ".anvil/container/run-in-container.ps1", + ".anvil/container/run-in-container.sh", +]; + +/// The routing seam's own recipe file, retired with the assets above. +const RETIRED_RECIPE: &str = "justfiles/anvil/runner.just"; + +fn write(path: &Path, contents: &str) { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent).unwrap(); + } + std::fs::write(path, contents).unwrap(); +} + +fn workspace() -> TempDir { + let tmp = TempDir::new().unwrap(); + let root = tmp.path(); + write( + &root.join("Cargo.toml"), + "[workspace]\nresolver = \"2\"\nmembers = [\"crates/*\"]\n", + ); + write( + &root.join("crates/alpha/Cargo.toml"), + "[package]\nname = \"alpha\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", + ); + write(&root.join("crates/alpha/src/lib.rs"), ""); + tmp +} + +fn local() -> Cli { + Cli { + backends: vec![], + no_backends: true, + dry_run: false, + force: false, + } +} + +/// Rewrite a freshly generated tree into the shape 0.4.0 produced: the retired +/// assets present on disk and tracked in the lock, and none of this release's +/// container artifacts present at all. +/// +/// The checksum recorded for each retired asset is the checksum of the body +/// written here, which is what makes the file "untouched since last render" and +/// so eligible for `Remove`. A test that wants the customized path overwrites +/// the body afterwards, leaving the recorded checksum stale on purpose. +fn rewind_to_runner_layout(root: &Path) -> Manifest { + let mut manifest = Manifest::load(root).unwrap(); + + for artifact in artifacts::container::all() { + let path = match artifact { + cargo_anvil::Artifact::OwnedFile(spec) => spec.path, + cargo_anvil::Artifact::Region(_) => panic!("container artifacts are owned files"), + }; + let full = root.join(path); + if full.exists() { + std::fs::remove_file(&full).unwrap(); + } + manifest.files.remove(path); + } + + for path in RETIRED_ASSETS.iter().copied().chain(std::iter::once(RETIRED_RECIPE)) { + let body = format!("# 0.4.0 generated {path}\n"); + write(&root.join(path), &body); + manifest.files.insert(path.to_owned(), checksum_str(&body)); + } + + // Provenance of the older build. Recorded, never a gate. + manifest.catalog_checksum = Some("sha256:0000000000000000000000000000000000000000000000000000000000000000".to_owned()); + manifest.tool_version = Some("0.4.0".to_owned()); + manifest.save(root).unwrap(); + manifest +} + +fn decision_for(outcome: &RunOutcome, path: &str) -> Decision { + outcome + .plan + .items() + .iter() + .find(|item| matches!(&item.target, Target::File { path: candidate } if candidate == path)) + .unwrap_or_else(|| panic!("no plan item for {path}")) + .decision +} + +#[test] +fn upgrading_from_the_runner_layout_retires_the_seam_and_emits_the_new_backend() { + let tmp = workspace(); + let root = tmp.path(); + run_update(&Catalog::anvil(), &local(), root).unwrap(); + rewind_to_runner_layout(root); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + assert!(outcome.applied); + let manifest = Manifest::load(root).unwrap(); + + // Untouched retired assets are deleted and untracked. Leaving them behind + // would ship a repository two container backends, one of which no recipe + // can reach. + for path in RETIRED_ASSETS.iter().copied().chain(std::iter::once(RETIRED_RECIPE)) { + assert_eq!(decision_for(&outcome, path), Decision::Remove, "{path} must be removed"); + assert!(!root.join(path).exists(), "{path} must be gone from disk"); + assert!(!manifest.files.contains_key(path), "{path} must be dropped from the lock"); + } + + // This release's artifacts take their place and are tracked. + for artifact in artifacts::container::all() { + let path = match artifact { + cargo_anvil::Artifact::OwnedFile(spec) => spec.path, + cargo_anvil::Artifact::Region(_) => panic!("container artifacts are owned files"), + }; + assert!(root.join(path).is_file(), "{path} must be written"); + assert!(manifest.files.contains_key(path), "{path} must be tracked"); + } + + // The root Justfile survives the region excision as a usable file rather + // than being emptied or left holding an orphaned sentinel. + let justfile = std::fs::read_to_string(root.join("Justfile")).unwrap(); + assert!(!justfile.contains("anvil-runner"), "the runner region must be spliced out"); + assert!(justfile.contains("anvil"), "the Justfile must still import the anvil tree"); +} + +#[test] +fn an_edited_retired_asset_is_handed_back_rather_than_deleted() { + let tmp = workspace(); + let root = tmp.path(); + run_update(&Catalog::anvil(), &local(), root).unwrap(); + rewind_to_runner_layout(root); + + // The adopter patched their Containerfile. The lock still holds the + // checksum of the generated body, so anvil can see the divergence. + let edited = ".anvil/container/Containerfile"; + let body = "FROM ubuntu:24.04\n# locally patched base\n"; + write(&root.join(edited), body); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + assert!(outcome.applied); + + // Removing a file the repository has edited would destroy work anvil did + // not author. Ownership transfers instead: the file stays, the lock entry + // goes. + assert_eq!( + decision_for(&outcome, edited), + Decision::OrphanedKept, + "an edited retired asset must be kept" + ); + assert_eq!( + std::fs::read_to_string(root.join(edited)).unwrap(), + body, + "the adopter's content must be preserved byte for byte" + ); + assert!( + !Manifest::load(root).unwrap().files.contains_key(edited), + "the lock entry must be dropped so the file becomes the repository's" + ); +} diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 896fdd76..b3264992 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -22,6 +22,8 @@ const LLVM_COV: &str = include_str!("../templates/justfiles/anvil/checks/llvm-co const SEMVER: &str = include_str!("../templates/justfiles/anvil/checks/semver-check.just"); const EXTERNAL_TYPES: &str = include_str!("../templates/justfiles/anvil/checks/external-types.just"); const TOOLS: &str = include_str!("../templates/justfiles/anvil/tools.just"); +const APRZ: &str = include_str!("../templates/justfiles/anvil/checks/aprz.just"); +const MUTANTS_DIFF: &str = include_str!("../templates/justfiles/anvil/checks/mutants-diff.just"); const VERSIONS: &str = include_str!("../templates/justfiles/anvil/versions.just"); const FAKE_CARGO_PS1: &str = r#" $joined = $args -join ' ' @@ -704,3 +706,152 @@ fn windows_arm64_fallback_accepts_empty_nextest_sets_in_both_configurations() { assert_eq!(calls.matches("--no-tests=pass").count(), 2, "calls:\n{calls}"); assert!(!calls.contains("llvm-cov"), "coverage commands must not run:\n{calls}"); } + +// --- container-specific behaviour ------------------------------------------ + +/// `anvil-aprz` warns and proceeds when it cannot obtain a token, rather than +/// throwing. That change exists so a containerized tier is not aborted by a +/// missing credential, and nothing else covers it: the dogfood run normally has +/// a host token, and the tokenless container E2E case runs a custom echo recipe. +#[test] +fn aprz_without_a_token_warns_and_still_runs() { + if !tools_available() { + return; + } + let tmp = fixture(&[("aprz.just", APRZ)], &[ + "anvil-tool-cargo-aprz-validate-prereqs", + "anvil-tool-cargo-aprz-install installer=\"install\"", + ]); + // A gh that yields no token: the recipe must fall through to the warnings + // rather than treating a failed lookup as fatal. `.cmd` matters -- `.ps1` + // is not in PATHEXT, so a script stub is skipped and the host's real `gh` + // answers instead, which on a signed-in machine hands back a live token and + // silently tests nothing. + write(&tmp.path().join("fake-bin/gh.cmd"), "@exit /b 1\r\n"); + write(&tmp.path().join("fake-bin/gh.ps1"), "exit 1\n"); + let log = tmp.path().join("cargo.log"); + + let output = run_just( + tmp.path(), + &["anvil-aprz"], + &[ + ("FAKE_CARGO_LOG", log.as_os_str()), + ("GITHUB_TOKEN", OsStr::new("")), + ("ANVIL_IN_CONTAINER", OsStr::new("1")), + ], + ); + + assert!( + output.status.success(), + "a missing token must not fail the check\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + // PowerShell's warning stream surfaces on stdout once `just` has run the + // script, so assert on what the developer actually sees rather than on a + // particular stream. + let seen = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!( + seen.contains("GITHUB_TOKEN is not set"), + "the warning must name the variable:\n{seen}" + ); + assert!(seen.contains("gh auth login"), "the warning must say how to fix it:\n{seen}"); + + // The point of warning rather than throwing: the check still runs. + let calls = std::fs::read_to_string(&log).unwrap_or_default(); + assert!(calls.contains("aprz deps"), "cargo aprz must still be invoked:\n{calls}"); +} + +/// `anvil-mutants-diff` diffs the base against the WORKING TREE, not against +/// HEAD. cargo-mutants validates every diff line against the file on disk and +/// aborts when they disagree, so a commit-to-commit diff fails as soon as +/// anything is uncommitted -- the normal local state, and the one CI never +/// exercises because its tree is clean. +#[test] +fn mutants_diff_covers_uncommitted_work() { + if !tools_available() || Command::new("git").arg("--version").output().is_err() { + return; + } + let tmp = fixture( + &[("helpers.just", HELPERS), ("mutants-diff.just", MUTANTS_DIFF)], + &[ + "anvil-tool-cargo-mutants-validate-prereqs", + "anvil-tool-cargo-mutants-install installer=\"install\"", + ], + ); + let root = tmp.path(); + // Real git: the stub the fixture installs would make `git diff` a no-op. + std::fs::remove_file(root.join("fake-bin/git.ps1")).unwrap(); + + let git = |args: &[&str]| { + let status = Command::new("git").args(args).current_dir(root).output().unwrap(); + assert!( + status.status.success(), + "git {args:?} failed: {}", + String::from_utf8_lossy(&status.stderr) + ); + }; + git(&["init", "-q"]); + git(&["config", "user.email", "test@example.com"]); + git(&["config", "user.name", "test"]); + // The host's global config decides line-ending rewriting, and a machine set + // to autocrlf rejects these fixtures outright ("LF would be replaced by + // CRLF"). Pin it so the test means the same thing on every developer's box. + git(&["config", "core.autocrlf", "false"]); + git(&["config", "core.safecrlf", "false"]); + write(&root.join("src/lib.rs"), "pub fn base() {}\n"); + git(&["add", "-A"]); + git(&["commit", "-qm", "base"]); + let base = String::from_utf8( + Command::new("git") + .args(["rev-parse", "HEAD"]) + .current_dir(root) + .output() + .unwrap() + .stdout, + ) + .unwrap() + .trim() + .to_owned(); + + // One change committed after the base, and one left uncommitted. A + // `base..HEAD` diff sees only the first. + write(&root.join("src/lib.rs"), "pub fn base() {}\npub fn committed() {}\n"); + git(&["add", "-A"]); + git(&["commit", "-qm", "committed change"]); + write( + &root.join("src/lib.rs"), + "pub fn base() {}\npub fn committed() {}\npub fn uncommitted() {}\n", + ); + + let log = root.join("cargo.log"); + let output = run_just( + root, + &["anvil-mutants-diff"], + &[ + ("FAKE_CARGO_LOG", log.as_os_str()), + ("BASE_REF", OsStr::new(&base)), + ("RUNNER_TEMP", root.as_os_str()), + ], + ); + assert!( + output.status.success(), + "the recipe must succeed\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + + let calls = std::fs::read_to_string(&log).unwrap_or_default(); + assert!(calls.contains("--in-diff"), "cargo mutants must be given a diff file:\n{calls}"); + + let diff = std::fs::read_to_string(root.join("anvil-mutants-diff.diff")).unwrap(); + assert!(diff.contains("committed"), "the committed change must be in the diff:\n{diff}"); + assert!( + diff.contains("uncommitted"), + "the uncommitted change must be in the diff -- a base..HEAD diff would omit it:\n{diff}" + ); +} diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index acfcd698..eb318d5b 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -433,7 +433,7 @@ Assert-Equal 'a tool installed by the image survives the cache mounts' 0 $probe. Assert-That 'the tool resolves inside the image, not a volume' ` ($probe.StdOut -match '/usr/local/cargo/bin/cargo-binstall') "$($probe.StdOut)$($probe.StdErr)" -# anvil-aprz runs in pr-fast and blocks on the rate limit without a token, so a +# anvil-aprz runs in scheduled-advisories and blocks on the rate limit without a token, so a # host token has to reach the container. The driver resolves it the way the # recipe does natively: the environment first, then the gh CLI. # From 95885afb1ca192a9a2eba9f5e4f74edc7d8a34f4 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 11:37:39 +0200 Subject: [PATCH 38/81] docs(anvil): record why WSLENV is not restored after the run An automated review flagged that the WSL bridge appends to `$env:WSLENV` and sets `$env:DOCKER_BUILDKIT` without restoring them, "mutating the caller's shell environment for the rest of the session". It does not. `just` runs a `[script(...)]` recipe as its own pwsh process, so the assignment dies with that process. Verified rather than argued: with `WSLENV=ORIGINAL` exported in the calling shell, a script recipe that sets `WSLENV=LEAKED_MARKER/u` and `DOCKER_BUILDKIT=1` reports its own value inside the recipe, and the caller still reads `ORIGINAL` with `DOCKER_BUILDKIT` unset afterwards. Saving and restoring would therefore be dead code guarding a boundary the process model already provides. Since the same reading has now been raised more than once, the reason is recorded next to the assignment -- including that the `finally` block unsets the credential names for hygiene within this process, not to protect the parent. Validation: `cargo test -p cargo-anvil` -- 0 failed. `cargo anvil --dry-run` converges at 79/79. --- .anvil.lock | 4 ++-- crates/cargo-anvil/templates/justfiles/anvil/container.just | 6 ++++++ .../cargo-anvil/tests/snapshots/snapshots__ado_backend.snap | 6 ++++++ .../tests/snapshots/snapshots__github_backend.snap | 6 ++++++ .../cargo-anvil/tests/snapshots/snapshots__local_only.snap | 6 ++++++ justfiles/anvil/container.just | 6 ++++++ 6 files changed, 32 insertions(+), 2 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 60acb1fd..e36ea831 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:4086dc1dfbaeafddbefe5e1bfcaf348e32a69e281af3605031122a32019deb46" +catalog_checksum = "sha256:88c2c1c742656d859273809cd89aaf543b9e9de82392b65bfc77d6628a37a983" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:ad2a465eb92a55c97f56cd9b4ff5bbfc0724d0d48aa397653ea689fc8eebe98a" +checksum = "sha256:286a1904fe69455e9ebc7cf000d8b2090daaac3b78e46d8d18a703f13f5e8ba9" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 3471fa03..2e8789e8 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -663,6 +663,12 @@ anvil-container *target: # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards # nothing, leaving the variable unset inside the container. + # + # It is not restored afterwards because there is nothing to restore to: + # `just` runs a [script(...)] recipe as its own pwsh process, so this + # assignment dies with that process and never reaches the caller's + # shell. The `finally` below unsets the credential names for hygiene + # within this process, not to protect the parent. if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 2f3e12e0..3d5cc368 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4172,6 +4172,12 @@ anvil-container *target: # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards # nothing, leaving the variable unset inside the container. + # + # It is not restored afterwards because there is nothing to restore to: + # `just` runs a [script(...)] recipe as its own pwsh process, so this + # assignment dies with that process and never reaches the caller's + # shell. The `finally` below unsets the credential names for hygiene + # within this process, not to protect the parent. if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index d8a44c14..f7e0c884 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4199,6 +4199,12 @@ anvil-container *target: # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards # nothing, leaving the variable unset inside the container. + # + # It is not restored afterwards because there is nothing to restore to: + # `just` runs a [script(...)] recipe as its own pwsh process, so this + # assignment dies with that process and never reaches the caller's + # shell. The `finally` below unsets the credential names for hygiene + # within this process, not to protect the parent. if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 10e85330..94c37068 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2912,6 +2912,12 @@ anvil-container *target: # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards # nothing, leaving the variable unset inside the container. + # + # It is not restored afterwards because there is nothing to restore to: + # `just` runs a [script(...)] recipe as its own pwsh process, so this + # assignment dies with that process and never reaches the caller's + # shell. The `finally` below unsets the credential names for hygiene + # within this process, not to protect the parent. if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 3471fa03..2e8789e8 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -663,6 +663,12 @@ anvil-container *target: # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards # nothing, leaving the variable unset inside the container. + # + # It is not restored afterwards because there is nothing to restore to: + # `just` runs a [script(...)] recipe as its own pwsh process, so this + # assignment dies with that process and never reaches the caller's + # shell. The `finally` below unsets the credential names for hygiene + # within this process, not to protect the parent. if ($engineExe -eq 'wsl.exe' -and $forwardedEnv.Count -gt 0) { $env:WSLENV = (@($env:WSLENV) + ($forwardedEnv | ForEach-Object { "$_/u" }) | Where-Object { $_ }) -join ':' } From fa30b1a099cff0dc03cfaaa9ce491c83d4eecb0d Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 15:14:38 +0200 Subject: [PATCH 39/81] fix(anvil): close the correctness gaps found in the driver and its tests Nine review findings, each verified against the code before acting. Driver: - The documented container opt-out left the tree unloadable. `without_artifact` deletes `container.just`, but `mod.just` hard-imported it, so the next generation failed at parse time for *every* recipe, not merely the container ones. The import is now optional. - Nested calls used a bare `just`, so a caller invoking `just` by absolute path with its directory off PATH got "not recognized" from the driver even though the outer invocation was valid. All twelve now go through `just_executable()`, matching the surrounding recipes. It is interpolated single-quoted with just's doubling escape rather than into a double-quoted literal, where `$` in a path would stay live -- the escaping guard below catches that, and caught it here. - `Get-ChildItem` omits hidden entries without `-Force`, so a dot-prefixed recipe was copied into the image but left out of its tag: later edits to it would reuse a stale image, and Windows and Unix could compute different tags for one checkout. - The hash stream was not self-delimiting. `file\n\n\n` lets a file whose body contains `file\n\n` serialize identically to two files splitting at that point, so a state with a hook and a state without one could name the same image. Path and content are now length-prefixed. - Eleven sites called `.Trim()` on a nested recipe's output before checking `$LASTEXITCODE`. A child that failed without stdout yields `$null`, and `.Trim()` on it throws under `Stop`, so the guard was unreachable and the child's own diagnostic was buried under a null-reference trace. Capture, check, then trim. Tests, including three defects in the ones added earlier in this branch: - The upgrade fixture never built the legacy managed region it claimed to remove, so `!justfile.contains("anvil-runner")` held before the upgrade ran and would have kept holding if region removal broke. It now seeds the 0.4.0 region and its checksum and asserts a `Target::Region` removal decision, that the body is spliced out, and that the lock entry is dropped. `RegionKey` is exported from `test_support` for it. - The tokenless aprz test shadowed `gh` with `.cmd` and `.ps1`, which only works on Windows. On Unix a signed-in machine would run its real `gh`, take the authenticated path, and pass while exercising the opposite of its name. An executable `fake-bin/gh` covers it. - The git fixture pinned autocrlf but not signing, so a host with `commit.gpgsign` and no usable key or TTY fails the commit before reaching the behaviour under test. - The dogfood script counted a missing engine as a skip, so an explicitly requested `-Engine docker` with no daemon reported `PASS 0/0` and exit 0. Validation: `cargo test -p cargo-anvil` -- 311 unit, 2 upgrade, 9 recipe contract, 3 snapshot, 51 across the rest, 0 failed. `scripts/test-anvil-container.ps1` -- 62/62 in 10:21 against a real daemon, which is what exercises the rewritten call sites, the new digest and the worktree redirection. The fan-out regression guard still fires: dropping a `-setup` dependency renames the image. --- .anvil.lock | 6 +- .../src/anvil/artifacts/container.rs | 7 ++- .../src/anvil/artifacts/justfile.rs | 5 +- crates/cargo-anvil/src/lib.rs | 2 +- .../templates/justfiles/anvil/container.just | 51 ++++++++++++----- .../templates/justfiles/anvil/mod.just | 6 +- crates/cargo-anvil/tests/container_upgrade.rs | 48 ++++++++++++++-- crates/cargo-anvil/tests/recipe_contracts.rs | 41 +++++++++---- .../snapshots/snapshots__ado_backend.snap | 57 +++++++++++++------ .../snapshots/snapshots__github_backend.snap | 57 +++++++++++++------ .../snapshots/snapshots__local_only.snap | 57 +++++++++++++------ justfiles/anvil/container.just | 51 ++++++++++++----- justfiles/anvil/mod.just | 6 +- scripts/test-anvil-dogfood.ps1 | 18 ++++++ 14 files changed, 310 insertions(+), 102 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index e36ea831..e4d2aaea 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:88c2c1c742656d859273809cd89aaf543b9e9de82392b65bfc77d6628a37a983" +catalog_checksum = "sha256:ff0f8ceda73310b61b6fe8704da987469f31c4c55483f99f40076bb42aaebe40" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:286a1904fe69455e9ebc7cf000d8b2090daaac3b78e46d8d18a703f13f5e8ba9" +checksum = "sha256:89568199b9ccd08e41bd28dee3da02d13bc6b6c302b43c8e7cf62d623232dcde" [[file]] path = "justfiles/anvil/groups/pr-fast.just" @@ -233,7 +233,7 @@ checksum = "sha256:cf6b30b8f4fd10eeb5bb5660c8e60512434fd94405a4fd8ea7399ababa089 [[file]] path = "justfiles/anvil/mod.just" -checksum = "sha256:74fb8d54efb7cb38ea68b4832f1dae86a153787a2e84081bf19efa823ea5a3e1" +checksum = "sha256:e7e4dddda365ccbe3263fd1de0e2d5f0a51a8b6067fbb6da7058114c13daa640" [[file]] path = "justfiles/anvil/tiers.just" diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index cc469774..d5342949 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -234,7 +234,7 @@ mod tests { "the content hash must be computed once, by anvil-container-tag" ); assert!( - RECIPE.contains("$image = (just anvil-container-tag).Trim()"), + RECIPE.contains("$image = & '{{ replace(just_executable(), \"'\", \"''\") }}' anvil-container-tag"), "the resolver must ask anvil-container-tag rather than recompute" ); } @@ -495,7 +495,10 @@ mod tests { // installed as surely as tools.just decides *how*. Hashing only the // install definitions would let a group drop a `-setup` dependency, // changing the installed set, without renaming the image. - assert!(RECIPE.contains("-Recurse -File -Filter '*.just'")); + // -Force so a dot-prefixed recipe is hashed. It is copied into the image + // either way, and Get-ChildItem omits hidden entries without it, so the + // tag would ignore edits to it and Windows and Unix could disagree. + assert!(RECIPE.contains("-Recurse -File -Force -Filter '*.just'")); // Including this driver, which passes the build arguments, the secret // mounts and the hook's PreBuild output into the build. assert!(!RECIPE.contains("-cne 'justfiles/anvil/container.just'")); diff --git a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs index 74913383..fd3d98d3 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs @@ -421,7 +421,10 @@ mod tests { "import 'helpers.just'", "import 'checks/fmt.just'", "import 'checks/miri.just'", - "import 'container.just'", + // Optional: containers.md §8 documents removing the container + // artifacts, which deletes this file. A hard import would then fail + // parsing for the whole tree, not just the container recipes. + "import? 'container.just'", "import 'groups/pr-fast.just'", "import 'groups/scheduled-exhaustive.just'", "import 'tiers.just'", diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index f4b11441..50fe5723 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -490,7 +490,7 @@ pub mod test_support { pub use crate::checksum::checksum_str; pub use crate::cli::Cli; pub use crate::decision::Decision; - pub use crate::manifest::{MANIFEST_FILE_NAME, Manifest}; + pub use crate::manifest::{MANIFEST_FILE_NAME, Manifest, RegionKey}; pub use crate::plan::Target; pub use crate::region::upsert_region; pub use crate::run::{RunOutcome, run_update}; diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 2e8789e8..a68dc482 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -86,8 +86,9 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-path host_path: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() if (-not $engine.StartsWith('wsl.exe|')) { Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 @@ -149,7 +150,7 @@ anvil-container-tag: # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -160,6 +161,14 @@ anvil-container-tag: # `Sort-Object -Unique` compares case-insensitively, which would silently drop # one of two inputs differing only in case on the case-sensitive filesystem # where the image is actually built. + # Length-prefix the path and the content rather than relying on newlines as + # separators. A bare `file\n\n\n` stream is not + # self-delimiting: content is arbitrary, so a file whose body contains + # "file\n\n" serializes identically to two files whose bodies + # split at that point. That makes distinct input sets nameable by one tag -- + # a hook that exists versus an ignore file whose body ends in the hook's + # path and body, for instance -- and the second state would silently reuse + # the first state's image. Byte counts cannot be forged by content. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -170,7 +179,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) + [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } $digest = [System.Security.Cryptography.SHA256]::HashData( [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) @@ -198,8 +208,9 @@ anvil-container-tag: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -208,8 +219,9 @@ _anvil-container-image: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $image = (just anvil-container-tag).Trim() + $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = "$image".Trim() if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -349,8 +361,9 @@ _anvil-container-image: [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") # The engine may not share this host's filesystem view, so the context # and the Dockerfile are given in its terms rather than ours. - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. @@ -427,17 +440,21 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $image = (just _anvil-container-image) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. `invocation_directory()` @@ -490,13 +507,15 @@ anvil-container *target: $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path if ($gitDirAbs -ne $gitCommonAbs) { $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() # LF and no trailing newline: git parses this file strictly. $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = (just _anvil-container-path $gitFile).Trim() + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } @@ -689,8 +708,9 @@ anvil-container *target: [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" @@ -703,13 +723,13 @@ anvil-container-status: # query cannot paper over -- a declared input is missing -- and reporting # that as "not present locally" would be a lie: the next run cannot build # it either. - $image = (just anvil-container-tag) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } Write-Output "image: $image" $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - just _anvil-container-image *> $null + & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { @@ -727,8 +747,9 @@ anvil-container-status: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) diff --git a/crates/cargo-anvil/templates/justfiles/anvil/mod.just b/crates/cargo-anvil/templates/justfiles/anvil/mod.just index 14f99095..4125c113 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/mod.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/mod.just @@ -66,7 +66,11 @@ import 'checks/readme-check.just' import 'checks/semver-check.just' import 'checks/spellcheck.just' import 'checks/udeps.just' -import 'container.just' +# Optional: containers.md §8 documents removing the container artifacts through +# `without_artifact`, which deletes this file. A hard import would then fail +# parsing for every recipe in the tree, not merely the container ones, so the +# documented opt-out would break the whole Justfile. +import? 'container.just' import 'groups/pr-fast.just' import 'groups/pr-slow.just' import 'groups/pr-test.just' diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index b82e71af..7dff0e48 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -30,7 +30,7 @@ use std::path::Path; -use cargo_anvil::test_support::{Cli, Decision, Manifest, RunOutcome, Target, checksum_str, run_update}; +use cargo_anvil::test_support::{Cli, Decision, Manifest, RegionKey, RunOutcome, Target, checksum_str, run_update}; use cargo_anvil::{Catalog, artifacts}; use tempfile::TempDir; @@ -50,6 +50,11 @@ const RETIRED_ASSETS: [&str; 8] = [ /// The routing seam's own recipe file, retired with the assets above. const RETIRED_RECIPE: &str = "justfiles/anvil/runner.just"; +/// The managed region 0.4.0 spliced into the root `Justfile` to carry the +/// runner selection, and the id it was keyed by. +const RETIRED_REGION_ID: &str = "anvil-runner"; +const RETIRED_REGION_BODY: &str = "anvil_runner := \"native\"\n"; + fn write(path: &Path, contents: &str) { if let Some(parent) = path.parent() { std::fs::create_dir_all(parent).unwrap(); @@ -110,6 +115,23 @@ fn rewind_to_runner_layout(root: &Path) -> Manifest { manifest.files.insert(path.to_owned(), checksum_str(&body)); } + // The managed region 0.4.0 spliced into the root Justfile. Seeding it is + // what makes the removal assertion mean anything: without it the Justfile + // never contained `anvil-runner`, so asserting its absence afterwards would + // hold before the upgrade ran and would keep holding if region removal + // broke entirely. + let justfile_path = root.join("Justfile"); + let justfile = std::fs::read_to_string(&justfile_path).unwrap(); + let region = format!("# >>> anvil-managed: {RETIRED_REGION_ID}\n{RETIRED_REGION_BODY}# <<< anvil-managed: {RETIRED_REGION_ID}\n"); + write(&justfile_path, &format!("{justfile}\n{region}")); + manifest.regions.insert( + RegionKey { + host: "Justfile".to_owned(), + id: RETIRED_REGION_ID.to_owned(), + }, + checksum_str(RETIRED_REGION_BODY), + ); + // Provenance of the older build. Recorded, never a gate. manifest.catalog_checksum = Some("sha256:0000000000000000000000000000000000000000000000000000000000000000".to_owned()); manifest.tool_version = Some("0.4.0".to_owned()); @@ -157,11 +179,29 @@ fn upgrading_from_the_runner_layout_retires_the_seam_and_emits_the_new_backend() assert!(manifest.files.contains_key(path), "{path} must be tracked"); } - // The root Justfile survives the region excision as a usable file rather - // than being emptied or left holding an orphaned sentinel. + // The runner region is spliced out of the root Justfile, and nothing else + // is: the surrounding content the repository owns must survive intact. + // Seeded in the rewind above, so this assertion can actually fail. + let region_decision = outcome + .plan + .items() + .iter() + .find(|item| matches!(&item.target, Target::Region { host, id } if host == "Justfile" && id == RETIRED_REGION_ID)) + .map(|item| item.decision); + assert_eq!( + region_decision, + Some(Decision::Remove), + "the obsolete runner region must be planned for removal" + ); + let justfile = std::fs::read_to_string(root.join("Justfile")).unwrap(); - assert!(!justfile.contains("anvil-runner"), "the runner region must be spliced out"); + assert!(!justfile.contains(RETIRED_REGION_ID), "the runner region must be spliced out"); + assert!(!justfile.contains(RETIRED_REGION_BODY.trim()), "the region body must go with it"); assert!(justfile.contains("anvil"), "the Justfile must still import the anvil tree"); + assert!( + !Manifest::load(root).unwrap().regions.keys().any(|key| key.id == RETIRED_REGION_ID), + "the region must be dropped from the lock" + ); } #[test] diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index b3264992..23e1334e 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -718,17 +718,31 @@ fn aprz_without_a_token_warns_and_still_runs() { if !tools_available() { return; } - let tmp = fixture(&[("aprz.just", APRZ)], &[ - "anvil-tool-cargo-aprz-validate-prereqs", - "anvil-tool-cargo-aprz-install installer=\"install\"", - ]); + let tmp = fixture( + &[("aprz.just", APRZ)], + &[ + "anvil-tool-cargo-aprz-validate-prereqs", + "anvil-tool-cargo-aprz-install installer=\"install\"", + ], + ); // A gh that yields no token: the recipe must fall through to the warnings - // rather than treating a failed lookup as fatal. `.cmd` matters -- `.ps1` - // is not in PATHEXT, so a script stub is skipped and the host's real `gh` - // answers instead, which on a signed-in machine hands back a live token and - // silently tests nothing. + // rather than treating a failed lookup as fatal. + // + // Three stubs because command lookup differs by platform and the fallback + // is the developer's real, signed-in `gh`: on Windows only `.cmd` is in + // PATHEXT, so a `.ps1` stub is skipped; on Unix a bare `gh` must exist and + // be executable. Getting this wrong does not fail the test -- it makes it + // pass while exercising the authenticated path, which is the opposite of + // what the name claims. write(&tmp.path().join("fake-bin/gh.cmd"), "@exit /b 1\r\n"); write(&tmp.path().join("fake-bin/gh.ps1"), "exit 1\n"); + let unix_stub = tmp.path().join("fake-bin/gh"); + write(&unix_stub, "#!/bin/sh\nexit 1\n"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + std::fs::set_permissions(&unix_stub, std::fs::Permissions::from_mode(0o755)).unwrap(); + } let log = tmp.path().join("cargo.log"); let output = run_just( @@ -798,11 +812,16 @@ fn mutants_diff_covers_uncommitted_work() { git(&["init", "-q"]); git(&["config", "user.email", "test@example.com"]); git(&["config", "user.name", "test"]); - // The host's global config decides line-ending rewriting, and a machine set - // to autocrlf rejects these fixtures outright ("LF would be replaced by - // CRLF"). Pin it so the test means the same thing on every developer's box. + // The host's global config decides line-ending rewriting and commit + // signing, and either will stop this fixture: a machine set to autocrlf + // rejects the add outright ("LF would be replaced by CRLF"), and one with + // commit.gpgsign and no usable key or TTY fails the commit before the + // behaviour under test runs. Pin both so the test means the same thing on + // every developer's box. git(&["config", "core.autocrlf", "false"]); git(&["config", "core.safecrlf", "false"]); + git(&["config", "commit.gpgsign", "false"]); + git(&["config", "tag.gpgsign", "false"]); write(&root.join("src/lib.rs"), "pub fn base() {}\n"); git(&["add", "-A"]); git(&["commit", "-qm", "base"]); diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 3d5cc368..d2c2e17a 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3595,8 +3595,9 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-path host_path: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() if (-not $engine.StartsWith('wsl.exe|')) { Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 @@ -3658,7 +3659,7 @@ anvil-container-tag: # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -3669,6 +3670,14 @@ anvil-container-tag: # `Sort-Object -Unique` compares case-insensitively, which would silently drop # one of two inputs differing only in case on the case-sensitive filesystem # where the image is actually built. + # Length-prefix the path and the content rather than relying on newlines as + # separators. A bare `file\n\n\n` stream is not + # self-delimiting: content is arbitrary, so a file whose body contains + # "file\n\n" serializes identically to two files whose bodies + # split at that point. That makes distinct input sets nameable by one tag -- + # a hook that exists versus an ignore file whose body ends in the hook's + # path and body, for instance -- and the second state would silently reuse + # the first state's image. Byte counts cannot be forged by content. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -3679,7 +3688,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) + [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } $digest = [System.Security.Cryptography.SHA256]::HashData( [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) @@ -3707,8 +3717,9 @@ anvil-container-tag: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -3717,8 +3728,9 @@ _anvil-container-image: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $image = (just anvil-container-tag).Trim() + $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = "$image".Trim() if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -3858,8 +3870,9 @@ _anvil-container-image: [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") # The engine may not share this host's filesystem view, so the context # and the Dockerfile are given in its terms rather than ours. - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. @@ -3936,17 +3949,21 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $image = (just _anvil-container-image) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. `invocation_directory()` @@ -3999,13 +4016,15 @@ anvil-container *target: $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path if ($gitDirAbs -ne $gitCommonAbs) { $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() # LF and no trailing newline: git parses this file strictly. $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = (just _anvil-container-path $gitFile).Trim() + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } @@ -4198,8 +4217,9 @@ anvil-container *target: [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" @@ -4212,13 +4232,13 @@ anvil-container-status: # query cannot paper over -- a declared input is missing -- and reporting # that as "not present locally" would be a lie: the next run cannot build # it either. - $image = (just anvil-container-tag) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } Write-Output "image: $image" $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - just _anvil-container-image *> $null + & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { @@ -4236,8 +4256,9 @@ anvil-container-status: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -4929,7 +4950,11 @@ import 'checks/readme-check.just' import 'checks/semver-check.just' import 'checks/spellcheck.just' import 'checks/udeps.just' -import 'container.just' +# Optional: containers.md §8 documents removing the container artifacts through +# `without_artifact`, which deletes this file. A hard import would then fail +# parsing for every recipe in the tree, not merely the container ones, so the +# documented opt-out would break the whole Justfile. +import? 'container.just' import 'groups/pr-fast.just' import 'groups/pr-slow.just' import 'groups/pr-test.just' diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f7e0c884..b312376c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3622,8 +3622,9 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-path host_path: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() if (-not $engine.StartsWith('wsl.exe|')) { Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 @@ -3685,7 +3686,7 @@ anvil-container-tag: # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -3696,6 +3697,14 @@ anvil-container-tag: # `Sort-Object -Unique` compares case-insensitively, which would silently drop # one of two inputs differing only in case on the case-sensitive filesystem # where the image is actually built. + # Length-prefix the path and the content rather than relying on newlines as + # separators. A bare `file\n\n\n` stream is not + # self-delimiting: content is arbitrary, so a file whose body contains + # "file\n\n" serializes identically to two files whose bodies + # split at that point. That makes distinct input sets nameable by one tag -- + # a hook that exists versus an ignore file whose body ends in the hook's + # path and body, for instance -- and the second state would silently reuse + # the first state's image. Byte counts cannot be forged by content. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -3706,7 +3715,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) + [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } $digest = [System.Security.Cryptography.SHA256]::HashData( [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) @@ -3734,8 +3744,9 @@ anvil-container-tag: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -3744,8 +3755,9 @@ _anvil-container-image: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $image = (just anvil-container-tag).Trim() + $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = "$image".Trim() if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -3885,8 +3897,9 @@ _anvil-container-image: [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") # The engine may not share this host's filesystem view, so the context # and the Dockerfile are given in its terms rather than ours. - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. @@ -3963,17 +3976,21 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $image = (just _anvil-container-image) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. `invocation_directory()` @@ -4026,13 +4043,15 @@ anvil-container *target: $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path if ($gitDirAbs -ne $gitCommonAbs) { $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() # LF and no trailing newline: git parses this file strictly. $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = (just _anvil-container-path $gitFile).Trim() + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } @@ -4225,8 +4244,9 @@ anvil-container *target: [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" @@ -4239,13 +4259,13 @@ anvil-container-status: # query cannot paper over -- a declared input is missing -- and reporting # that as "not present locally" would be a lie: the next run cannot build # it either. - $image = (just anvil-container-tag) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } Write-Output "image: $image" $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - just _anvil-container-image *> $null + & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { @@ -4263,8 +4283,9 @@ anvil-container-status: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -4956,7 +4977,11 @@ import 'checks/readme-check.just' import 'checks/semver-check.just' import 'checks/spellcheck.just' import 'checks/udeps.just' -import 'container.just' +# Optional: containers.md §8 documents removing the container artifacts through +# `without_artifact`, which deletes this file. A hard import would then fail +# parsing for every recipe in the tree, not merely the container ones, so the +# documented opt-out would break the whole Justfile. +import? 'container.just' import 'groups/pr-fast.just' import 'groups/pr-slow.just' import 'groups/pr-test.just' diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 94c37068..d5acbd0e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2335,8 +2335,9 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-path host_path: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() if (-not $engine.StartsWith('wsl.exe|')) { Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 @@ -2398,7 +2399,7 @@ anvil-container-tag: # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -2409,6 +2410,14 @@ anvil-container-tag: # `Sort-Object -Unique` compares case-insensitively, which would silently drop # one of two inputs differing only in case on the case-sensitive filesystem # where the image is actually built. + # Length-prefix the path and the content rather than relying on newlines as + # separators. A bare `file\n\n\n` stream is not + # self-delimiting: content is arbitrary, so a file whose body contains + # "file\n\n" serializes identically to two files whose bodies + # split at that point. That makes distinct input sets nameable by one tag -- + # a hook that exists versus an ignore file whose body ends in the hook's + # path and body, for instance -- and the second state would silently reuse + # the first state's image. Byte counts cannot be forged by content. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -2419,7 +2428,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) + [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } $digest = [System.Security.Cryptography.SHA256]::HashData( [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) @@ -2447,8 +2457,9 @@ anvil-container-tag: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -2457,8 +2468,9 @@ _anvil-container-image: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $image = (just anvil-container-tag).Trim() + $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = "$image".Trim() if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -2598,8 +2610,9 @@ _anvil-container-image: [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") # The engine may not share this host's filesystem view, so the context # and the Dockerfile are given in its terms rather than ours. - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. @@ -2676,17 +2689,21 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $image = (just _anvil-container-image) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. `invocation_directory()` @@ -2739,13 +2756,15 @@ anvil-container *target: $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path if ($gitDirAbs -ne $gitCommonAbs) { $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() # LF and no trailing newline: git parses this file strictly. $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = (just _anvil-container-path $gitFile).Trim() + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } @@ -2938,8 +2957,9 @@ anvil-container *target: [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" @@ -2952,13 +2972,13 @@ anvil-container-status: # query cannot paper over -- a declared input is missing -- and reporting # that as "not present locally" would be a lie: the next run cannot build # it either. - $image = (just anvil-container-tag) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } Write-Output "image: $image" $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - just _anvil-container-image *> $null + & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { @@ -2976,8 +2996,9 @@ anvil-container-status: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -3669,7 +3690,11 @@ import 'checks/readme-check.just' import 'checks/semver-check.just' import 'checks/spellcheck.just' import 'checks/udeps.just' -import 'container.just' +# Optional: containers.md §8 documents removing the container artifacts through +# `without_artifact`, which deletes this file. A hard import would then fail +# parsing for every recipe in the tree, not merely the container ones, so the +# documented opt-out would break the whole Justfile. +import? 'container.just' import 'groups/pr-fast.just' import 'groups/pr-slow.just' import 'groups/pr-test.just' diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 2e8789e8..a68dc482 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -86,8 +86,9 @@ _anvil-container-engine: [script("pwsh", "-NoProfile")] _anvil-container-path host_path: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() if (-not $engine.StartsWith('wsl.exe|')) { Write-Output '{{ replace(host_path, "'", "''") }}' exit 0 @@ -149,7 +150,7 @@ anvil-container-tag: # `Anvil-BuildSecrets` output into the build, all of which shape the result. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -160,6 +161,14 @@ anvil-container-tag: # `Sort-Object -Unique` compares case-insensitively, which would silently drop # one of two inputs differing only in case on the case-sensitive filesystem # where the image is actually built. + # Length-prefix the path and the content rather than relying on newlines as + # separators. A bare `file\n\n\n` stream is not + # self-delimiting: content is arbitrary, so a file whose body contains + # "file\n\n" serializes identically to two files whose bodies + # split at that point. That makes distinct input sets nameable by one tag -- + # a hook that exists versus an ignore file whose body ends in the hook's + # path and body, for instance -- and the second state would silently reuse + # the first state's image. Byte counts cannot be forged by content. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -170,7 +179,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append("file`n").Append($rel).Append("`n").Append($text).Append("`n") + [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) + [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } $digest = [System.Security.Cryptography.SHA256]::HashData( [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) @@ -198,8 +208,9 @@ anvil-container-tag: [script("pwsh", "-NoProfile")] _anvil-container-image: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) @@ -208,8 +219,9 @@ _anvil-container-image: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $image = (just anvil-container-tag).Trim() + $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $image = "$image".Trim() if ($env:ANVIL_CONTAINER_NO_CACHE -ne '1') { & $engineExe @enginePrefix image inspect $image *> $null @@ -349,8 +361,9 @@ _anvil-container-image: [Console]::Error.WriteLine("anvil: building $image (inputs changed or first run)") # The engine may not share this host's filesystem view, so the context # and the Dockerfile are given in its terms rather than ours. - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Pinned, not inferred from the host. The Dockerfile installs amd64 # toolchains and verifies amd64 checksums, so an arm host would resolve # the multi-arch base to arm64 and fail late with an exec-format error. @@ -427,17 +440,21 @@ anvil-container *target: exit $LASTEXITCODE } - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) - $image = (just _anvil-container-image) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $engineRoot = (just _anvil-container-path $repoRoot).Trim() + $engineRoot = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $repoRoot if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineRoot = "$engineRoot".Trim() # Map the caller's working directory to its in-container equivalent so # relative paths keep working from a subdirectory. `invocation_directory()` @@ -490,13 +507,15 @@ anvil-container *target: $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path if ($gitDirAbs -ne $gitCommonAbs) { $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = (just _anvil-container-path $gitCommonAbs).Trim() + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() # LF and no trailing newline: git parses this file strictly. $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = (just _anvil-container-path $gitFile).Trim() + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } @@ -689,8 +708,9 @@ anvil-container *target: [script("pwsh", "-NoProfile")] anvil-container-status: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() Write-Output ("engine: " + ($engine -replace '\|', ' ')) Write-Output "workdir: {{anvil_container_workdir}}" @@ -703,13 +723,13 @@ anvil-container-status: # query cannot paper over -- a declared input is missing -- and reporting # that as "not present locally" would be a lie: the next run cannot build # it either. - $image = (just anvil-container-tag) | Select-Object -Last 1 + $image = (& '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag) | Select-Object -Last 1 if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } Write-Output "image: $image" $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' - just _anvil-container-image *> $null + & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" } else { @@ -727,8 +747,9 @@ anvil-container-status: [script("pwsh", "-NoProfile")] anvil-container-down: $ErrorActionPreference = 'Stop' - $engine = (just _anvil-container-engine).Trim() + $engine = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-engine if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engine = "$engine".Trim() $engineCmd = $engine -split '\|' $engineExe = $engineCmd[0] $enginePrefix = @($engineCmd | Select-Object -Skip 1) diff --git a/justfiles/anvil/mod.just b/justfiles/anvil/mod.just index 14f99095..4125c113 100644 --- a/justfiles/anvil/mod.just +++ b/justfiles/anvil/mod.just @@ -66,7 +66,11 @@ import 'checks/readme-check.just' import 'checks/semver-check.just' import 'checks/spellcheck.just' import 'checks/udeps.just' -import 'container.just' +# Optional: containers.md §8 documents removing the container artifacts through +# `without_artifact`, which deletes this file. A hard import would then fail +# parsing for every recipe in the tree, not merely the container ones, so the +# documented opt-out would break the whole Justfile. +import? 'container.just' import 'groups/pr-fast.just' import 'groups/pr-slow.just' import 'groups/pr-test.just' diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index 7f0fe5d6..64585d13 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -263,6 +263,17 @@ function Invoke-Suite([string]$EngineName) { $resolved = Resolve-Engine $EngineName if (-not $resolved) { + # An engine the caller named explicitly is a precondition, not an + # option: skipping it and exiting 0 reports "PASS 0/0 checks passed" for + # a daemon-backed verification that never reached a daemon. Only the + # default `both` sweep may skip one, and even then the run must fail if + # neither engine turned up (checked after the loop). + if ($Engine -ne 'both') { + Assert-That "$EngineName is available" $false ` + 'not on PATH, and not reachable in the default WSL distribution' + $script:Results[$EngineName] = 'failed' + return + } Write-Skipped "$EngineName is available" "not on PATH, and not reachable in the default WSL distribution" $script:Results[$EngineName] = 'skipped' return @@ -482,6 +493,13 @@ foreach ($name in $script:Results.Keys) { } $summary = "{0}/{1} checks passed in {2:hh\:mm\:ss}" -f $script:Passed, ($script:Passed + $script:Failed), $elapsed if ($script:Skipped) { $summary += " ($($script:Skipped) skipped)" } +# A sweep where every engine was skipped ran no container at all. Reporting +# that as a pass is the false green this script exists to prevent. A failed +# engine is a different thing and already sets the exit code below. +if (-not ($script:Results.Values | Where-Object { $_ -ne 'skipped' })) { + Write-Host "FAIL no engine was exercised -- install docker or podman, or check the daemon" -ForegroundColor Red + exit 1 +} if ($script:Failed -eq 0) { Write-Host "PASS $summary" -ForegroundColor Green exit 0 From 3386dc92c21388b1539a507cfc5064f7ae4723f2 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 17:58:49 +0200 Subject: [PATCH 40/81] fix(anvil): close the last identity hole and correct what the docs claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Findings from an independent multi-model review pass. Two models raised the same digest-documentation defect, and one found a real hole in the identity guarantee. - The digest hashed only `*.just` under `justfiles/anvil/`, but the build context admits the whole directory, so any other file an adopter added by hand was copied into the image while staying outside its tag -- editing it changed the image's contents under a reference that already resolved. `CatalogBuilder` refuses to *own* a non-recipe file there, which covers catalog authors and not repositories. The digest now covers every file the context copies, so the two sets are identical by construction. Verified: a stray `justfiles/anvil/stray-probe.txt` moves the tag, and removing it restores the original. - The tag used `SHA256::HashData`, a .NET 5 static. The prerequisite check accepts any PowerShell 7, and 7.0 runs on .NET Core 3.1 where that overload does not exist, so tagging would fail with MethodNotFound before doing anything useful. Uses `Create()`/`ComputeHash` instead. - `anvil-container-status` mapped every `image inspect` failure to "not present locally", including an unreachable daemon -- telling the developer to expect a build that would not start either. It now asks the engine for its version before concluding absence, and reports a non-responding engine as such. Documentation that did not match the code: - §4.2 still described the newline-terminated stream this branch replaced with length-prefixed framing, which is the very ambiguity the change removed. - §7.3 said every hook failure falls through to a local build. That holds for resolution only; the build and run phases reload the same file for credentials and are deliberately fail-closed, so an unparseable `hooks.ps1` stops the run there. Both behaviours are intended; only the sentence was wrong. - Two reviewers flagged that `apt-get install` names no versions, so clean builds of a byte-identical Dockerfile can install different packages under one tag. Pinning them would trade this for a harder failure, since Ubuntu drops superseded versions and the build would stop working outright. §4.3 now states the guarantee precisely: the declared inputs are fixed and everything anvil installs is version-pinned with a checksum; the system packages beneath track the base distribution. Validation: `cargo test -p cargo-anvil` -- 311 unit plus the integration and snapshot suites, 0 failed. `cargo anvil --dry-run` converges at 79/79. Tag computation, status, and the stray-file probe exercised against the real generated tree. --- .anvil.lock | 4 +- crates/cargo-anvil/docs/design/containers.md | 25 ++++++++++-- .../src/anvil/artifacts/container.rs | 17 +++++--- .../templates/justfiles/anvil/container.just | 40 ++++++++++++++++--- .../snapshots/snapshots__ado_backend.snap | 40 ++++++++++++++++--- .../snapshots/snapshots__github_backend.snap | 40 ++++++++++++++++--- .../snapshots/snapshots__local_only.snap | 40 ++++++++++++++++--- justfiles/anvil/container.just | 40 ++++++++++++++++--- 8 files changed, 210 insertions(+), 36 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index ee328d84..79c41144 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:8e8df31d766107c9e48fdd567957f283f6e9647db2db5b3b36c6b403121b5db0" +catalog_checksum = "sha256:c779622507cad0d4702eccb0a2878e5702e8904682488cff2eb273a7215c8506" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:89568199b9ccd08e41bd28dee3da02d13bc6b6c302b43c8e7cf62d623232dcde" +checksum = "sha256:10235e8c3fdd04cf8b84a32a8b023ee8a651853710494bb7aea4b21fd4a7967b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index d159578a..2f0efbaf 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -184,7 +184,10 @@ excluded: a credential must never influence a tag. ### 4.2 Digest Inputs are sorted by relative path with an ordinal comparison, then serialized into one stream in which each entry -contributes a literal `file`, its relative path, and its content, each newline-terminated. Tagging entries this way +contributes a literal `file`, the byte length of its relative path, the path, the byte length of its content, and the +content. Length-prefixing the two variable-length fields is what makes the stream self-delimiting: newline framing +would let a file whose body happened to contain `file`, a path and a newline serialize identically to two files +splitting at that point, so two different input sets could name one image. Tagging entries this way prevents a rearrangement of names and contents from colliding. Line endings are normalized to LF, so CRLF and LF checkouts agree on the tag. The sort is ordinal because a case-insensitive one would drop one of two inputs differing only in case on the case-sensitive filesystem where the image is built. @@ -209,6 +212,14 @@ Two properties sit outside the digest. The base image is not resolved during has digest-pinned; a floating tag could otherwise change beneath a tag that claims to name fixed content. The platform is pinned to `linux/amd64` on build and run, so hosts of differing architecture cannot compute one tag for two images. +A third sits outside it by necessity: the `apt-get install` layer names packages without versions, and Ubuntu's +archive moves. Two clean builds of a byte-identical Dockerfile weeks apart can therefore install different package +versions under one tag. Pinning every apt version would trade this for a harder failure, since the archive drops +superseded versions and the build would simply stop working. So the guarantee the tag gives is precise: **the inputs +that define the image are fixed, and everything anvil itself installs is version-pinned** — the toolchain, the tool +catalog, `just`, `pwsh`, `rustup` and `cargo-binstall`, each with a checksum. The system packages beneath them track +the base distribution. `ANVIL_CONTAINER_NO_CACHE=1` forces a from-scratch rebuild when that distinction matters. + The base tracks the Linux runner the generated workflows use, `ubuntu-latest` (currently 24.04). The catalog is installed with `binstall`, and those prebuilt binaries require that runner's glibc, which is backward but not forward compatible. A catalog on an older base installs from source instead. @@ -483,9 +494,15 @@ Three properties are load-bearing: not actually fetched would otherwise fail later and further from the cause. This is a presence check, not a verification: `image inspect` proves something carries that reference, not that its contents match the digest the tag claims. Trusting the publisher is the contract (§4.3). -- **Every failure falls through to a local build**, with the reason printed — including a hook that cannot be loaded at - all, which is why the dot-source sits inside the same `try`. A publisher that has not caught up with a change must - not block the developer who made it. +- **Every resolve failure falls through to a local build**, with the reason printed — including a hook that cannot be + loaded at all, which is why the dot-source sits inside the same `try`. A publisher that has not caught up with a + change must not block the developer who made it. + + That tolerance is scoped to *resolution*. The build and run phases load the same file again to obtain credentials + (§7.1, §7.2) and are deliberately fail-closed, so a `hooks.ps1` that cannot be parsed stops the run there instead — + after resolution has already forgiven it. The two are not in conflict: a hook that yields no image costs nothing, + while a hook that cannot yield its credentials would otherwise produce an image built without them and tag it as if + it had them. ### 7.4 Trust boundary diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index d5342949..5309ac4d 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -229,7 +229,7 @@ mod tests { // the consumer performs: a second copy of the hash would let the two // drift and turn a published tag into a claim nobody checks. assert_eq!( - RECIPE.matches("SHA256]::HashData").count(), + RECIPE.matches("ComputeHash(").count(), 1, "the content hash must be computed once, by anvil-container-tag" ); @@ -495,10 +495,17 @@ mod tests { // installed as surely as tools.just decides *how*. Hashing only the // install definitions would let a group drop a `-setup` dependency, // changing the installed set, without renaming the image. - // -Force so a dot-prefixed recipe is hashed. It is copied into the image - // either way, and Get-ChildItem omits hidden entries without it, so the - // tag would ignore edits to it and Windows and Unix could disagree. - assert!(RECIPE.contains("-Recurse -File -Force -Filter '*.just'")); + // Every file, not only `*.just`: the build context admits the whole + // directory, so a non-recipe file an adopter adds by hand is copied + // into the image. Filtering here would let it change the image's + // contents without changing its tag. -Force because a dot-prefixed + // file is copied like any other and would otherwise be skipped. + assert!(RECIPE.contains("-Recurse -File -Force")); + assert!(!RECIPE.contains("-Recurse -File -Force -Filter '*.just'")); + // ComputeHash, not the static HashData: the latter needs .NET 5, and + // the prerequisite check accepts PowerShell 7.0 on .NET Core 3.1. + assert!(!RECIPE.contains("SHA256]::HashData")); + assert!(RECIPE.contains("SHA256]::Create()")); // Including this driver, which passes the build arguments, the secret // mounts and the hook's PreBuild output into the build. assert!(!RECIPE.contains("-cne 'justfiles/anvil/container.just'")); diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index a68dc482..90cddbf9 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -148,9 +148,20 @@ anvil-container-tag: # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's # `Anvil-BuildSecrets` output into the build, all of which shape the result. + # Every file in the generated recipe tree, not only `*.just`. The build + # context admits the whole `justfiles/anvil/` directory (see the ignore + # file), so anything an adopter drops there is copied into the image. The + # catalog refuses to *own* a non-recipe file there, but a repository can + # still add one by hand, and a file that reaches the image without reaching + # the tag is precisely the hole this digest exists to close. Hashing what + # the context copies keeps the two sets identical by construction. + # + # -Force because Get-ChildItem omits hidden entries otherwise: a + # dot-prefixed file is copied like any other, and skipping it would let its + # edits ride under an unchanged tag -- and make Windows and Unix disagree. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -182,8 +193,16 @@ anvil-container-tag: [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } - $digest = [System.Security.Cryptography.SHA256]::HashData( - [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # ComputeHash rather than the static HashData: the latter arrived in .NET 5, + # and the prerequisite check accepts any PowerShell 7, including 7.0 on + # .NET Core 3.1 where the static overload does not exist. Failing there + # would be a MethodNotFound at tag time, before anything useful happened. + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $digest = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + } finally { + $sha.Dispose() + } # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) @@ -732,9 +751,20 @@ anvil-container-status: & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" - } else { - Write-Output "status: not present locally (the next run resolves or builds it)" + exit 0 + } + + # A cache miss and an unreachable daemon both make `image inspect` fail, and + # reporting the second as the first tells a developer to expect a build that + # will not start either. Ask the engine whether it is answering at all: only + # then is absence the honest reading. + $engineCmd = $engine -split '\|' + & $engineCmd[0] @($engineCmd | Select-Object -Skip 1) version *> $null + if ($LASTEXITCODE -ne 0) { + Write-Output "status: unknown -- the engine is not responding (is the daemon running?)" + exit 1 } + Write-Output "status: not present locally (the next run resolves or builds it)" exit 0 # Only the download caches are volumes, so this discards fetched crates and git diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index d72ea2c3..91d8b8af 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3684,9 +3684,20 @@ anvil-container-tag: # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's # `Anvil-BuildSecrets` output into the build, all of which shape the result. + # Every file in the generated recipe tree, not only `*.just`. The build + # context admits the whole `justfiles/anvil/` directory (see the ignore + # file), so anything an adopter drops there is copied into the image. The + # catalog refuses to *own* a non-recipe file there, but a repository can + # still add one by hand, and a file that reaches the image without reaching + # the tag is precisely the hole this digest exists to close. Hashing what + # the context copies keeps the two sets identical by construction. + # + # -Force because Get-ChildItem omits hidden entries otherwise: a + # dot-prefixed file is copied like any other, and skipping it would let its + # edits ride under an unchanged tag -- and make Windows and Unix disagree. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -3718,8 +3729,16 @@ anvil-container-tag: [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } - $digest = [System.Security.Cryptography.SHA256]::HashData( - [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # ComputeHash rather than the static HashData: the latter arrived in .NET 5, + # and the prerequisite check accepts any PowerShell 7, including 7.0 on + # .NET Core 3.1 where the static overload does not exist. Failing there + # would be a MethodNotFound at tag time, before anything useful happened. + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $digest = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + } finally { + $sha.Dispose() + } # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) @@ -4268,9 +4287,20 @@ anvil-container-status: & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" - } else { - Write-Output "status: not present locally (the next run resolves or builds it)" + exit 0 + } + + # A cache miss and an unreachable daemon both make `image inspect` fail, and + # reporting the second as the first tells a developer to expect a build that + # will not start either. Ask the engine whether it is answering at all: only + # then is absence the honest reading. + $engineCmd = $engine -split '\|' + & $engineCmd[0] @($engineCmd | Select-Object -Skip 1) version *> $null + if ($LASTEXITCODE -ne 0) { + Write-Output "status: unknown -- the engine is not responding (is the daemon running?)" + exit 1 } + Write-Output "status: not present locally (the next run resolves or builds it)" exit 0 # Only the download caches are volumes, so this discards fetched crates and git diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index c216e86d..0bd4c8f1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3711,9 +3711,20 @@ anvil-container-tag: # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's # `Anvil-BuildSecrets` output into the build, all of which shape the result. + # Every file in the generated recipe tree, not only `*.just`. The build + # context admits the whole `justfiles/anvil/` directory (see the ignore + # file), so anything an adopter drops there is copied into the image. The + # catalog refuses to *own* a non-recipe file there, but a repository can + # still add one by hand, and a file that reaches the image without reaching + # the tag is precisely the hole this digest exists to close. Hashing what + # the context copies keeps the two sets identical by construction. + # + # -Force because Get-ChildItem omits hidden entries otherwise: a + # dot-prefixed file is copied like any other, and skipping it would let its + # edits ride under an unchanged tag -- and make Windows and Unix disagree. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -3745,8 +3756,16 @@ anvil-container-tag: [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } - $digest = [System.Security.Cryptography.SHA256]::HashData( - [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # ComputeHash rather than the static HashData: the latter arrived in .NET 5, + # and the prerequisite check accepts any PowerShell 7, including 7.0 on + # .NET Core 3.1 where the static overload does not exist. Failing there + # would be a MethodNotFound at tag time, before anything useful happened. + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $digest = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + } finally { + $sha.Dispose() + } # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) @@ -4295,9 +4314,20 @@ anvil-container-status: & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" - } else { - Write-Output "status: not present locally (the next run resolves or builds it)" + exit 0 + } + + # A cache miss and an unreachable daemon both make `image inspect` fail, and + # reporting the second as the first tells a developer to expect a build that + # will not start either. Ask the engine whether it is answering at all: only + # then is absence the honest reading. + $engineCmd = $engine -split '\|' + & $engineCmd[0] @($engineCmd | Select-Object -Skip 1) version *> $null + if ($LASTEXITCODE -ne 0) { + Write-Output "status: unknown -- the engine is not responding (is the daemon running?)" + exit 1 } + Write-Output "status: not present locally (the next run resolves or builds it)" exit 0 # Only the download caches are volumes, so this discards fetched crates and git diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index ed5455e4..1907965b 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2424,9 +2424,20 @@ anvil-container-tag: # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's # `Anvil-BuildSecrets` output into the build, all of which shape the result. + # Every file in the generated recipe tree, not only `*.just`. The build + # context admits the whole `justfiles/anvil/` directory (see the ignore + # file), so anything an adopter drops there is copied into the image. The + # catalog refuses to *own* a non-recipe file there, but a repository can + # still add one by hand, and a file that reaches the image without reaching + # the tag is precisely the hole this digest exists to close. Hashing what + # the context copies keeps the two sets identical by construction. + # + # -Force because Get-ChildItem omits hidden entries otherwise: a + # dot-prefixed file is copied like any other, and skipping it would let its + # edits ride under an unchanged tag -- and make Windows and Unix disagree. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -2458,8 +2469,16 @@ anvil-container-tag: [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } - $digest = [System.Security.Cryptography.SHA256]::HashData( - [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # ComputeHash rather than the static HashData: the latter arrived in .NET 5, + # and the prerequisite check accepts any PowerShell 7, including 7.0 on + # .NET Core 3.1 where the static overload does not exist. Failing there + # would be a MethodNotFound at tag time, before anything useful happened. + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $digest = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + } finally { + $sha.Dispose() + } # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) @@ -3008,9 +3027,20 @@ anvil-container-status: & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" - } else { - Write-Output "status: not present locally (the next run resolves or builds it)" + exit 0 + } + + # A cache miss and an unreachable daemon both make `image inspect` fail, and + # reporting the second as the first tells a developer to expect a build that + # will not start either. Ask the engine whether it is answering at all: only + # then is absence the honest reading. + $engineCmd = $engine -split '\|' + & $engineCmd[0] @($engineCmd | Select-Object -Skip 1) version *> $null + if ($LASTEXITCODE -ne 0) { + Write-Output "status: unknown -- the engine is not responding (is the daemon running?)" + exit 1 } + Write-Output "status: not present locally (the next run resolves or builds it)" exit 0 # Only the download caches are volumes, so this discards fetched crates and git diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index a68dc482..90cddbf9 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -148,9 +148,20 @@ anvil-container-tag: # from file text, and no file contains the tag -- and it belongs in the set # because it passes the build arguments, the secret mounts and the hook's # `Anvil-BuildSecrets` output into the build, all of which shape the result. + # Every file in the generated recipe tree, not only `*.just`. The build + # context admits the whole `justfiles/anvil/` directory (see the ignore + # file), so anything an adopter drops there is copied into the image. The + # catalog refuses to *own* a non-recipe file there, but a repository can + # still add one by hand, and a file that reaches the image without reaching + # the tag is precisely the hole this digest exists to close. Hashing what + # the context copies keeps the two sets identical by construction. + # + # -Force because Get-ChildItem omits hidden entries otherwise: a + # dot-prefixed file is copied like any other, and skipping it would let its + # edits ride under an unchanged tag -- and make Windows and Unix disagree. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force -Filter '*.just') { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } @@ -182,8 +193,16 @@ anvil-container-tag: [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) } - $digest = [System.Security.Cryptography.SHA256]::HashData( - [System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + # ComputeHash rather than the static HashData: the latter arrived in .NET 5, + # and the prerequisite check accepts any PowerShell 7, including 7.0 on + # .NET Core 3.1 where the static overload does not exist. Failing there + # would be a MethodNotFound at tag time, before anything useful happened. + $sha = [System.Security.Cryptography.SHA256]::Create() + try { + $digest = $sha.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stream.ToString())) + } finally { + $sha.Dispose() + } # 16 hex characters (64 bits) is far past any practical collision risk for a # local image set, and keeps `docker images` readable. $imageId = -join ($digest[0..7] | ForEach-Object { $_.ToString('x2') }) @@ -732,9 +751,20 @@ anvil-container-status: & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" - } else { - Write-Output "status: not present locally (the next run resolves or builds it)" + exit 0 + } + + # A cache miss and an unreachable daemon both make `image inspect` fail, and + # reporting the second as the first tells a developer to expect a build that + # will not start either. Ask the engine whether it is answering at all: only + # then is absence the honest reading. + $engineCmd = $engine -split '\|' + & $engineCmd[0] @($engineCmd | Select-Object -Skip 1) version *> $null + if ($LASTEXITCODE -ne 0) { + Write-Output "status: unknown -- the engine is not responding (is the daemon running?)" + exit 1 } + Write-Output "status: not present locally (the next run resolves or builds it)" exit 0 # Only the download caches are volumes, so this discards fetched crates and git From 72c68eb4d53151271b6a291561274799a1cabad0 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 18:16:43 +0200 Subject: [PATCH 41/81] fix(anvil): use the launching just for the token probe, and stop documenting a closed hole MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings from the next reviewer, both of them misses from the round before. - The token probe was the one nested call left invoking a bare `just`. The twelve others moved to `just_executable()` precisely because a caller can invoke `just` by absolute path with its directory off PATH -- and here the failure is silent, because the surrounding try/catch turns it into an empty plan, which reads as "this target does not need a token". The result would be `anvil-aprz` running unauthenticated inside an image that has no `gh` of its own, blocking on the rate limit for up to an hour. Same escaping as the other sites; the unit test now pins the executable rather than the bare word. - Closing the identity hole in the previous commit made four other statements false. The inputs table still listed `justfiles/anvil/**/*.just`; `containers.md` §8, `extensibility.md` §6.1 and the `CatalogBuilder` rustdoc and error string all still justified the non-recipe rejection by claiming such a file would change the image without renaming the tag. It no longer can. The rule stands, but for the reason it actually has now: that directory is the recipe tree `just` parses, and an installer script hidden there makes the tool set harder to reason about. A publisher following the old table would have computed a different tag than the driver. Validation: `cargo test -p cargo-anvil` -- 311 unit plus the integration and snapshot suites, 0 failed. Token discrimination re-checked against real plans: `anvil-aprz` reads GITHUB_TOKEN, `anvil-fmt` does not. `cargo anvil --dry-run` converges at 79/79. --- .anvil.lock | 4 ++-- crates/cargo-anvil/docs/design/containers.md | 11 +++++++---- crates/cargo-anvil/docs/design/extensibility.md | 14 ++++++++------ .../cargo-anvil/src/anvil/artifacts/container.rs | 7 ++++++- crates/cargo-anvil/src/catalog/builder.rs | 16 ++++++++-------- .../templates/justfiles/anvil/container.just | 13 ++++++++++++- .../tests/snapshots/snapshots__ado_backend.snap | 13 ++++++++++++- .../snapshots/snapshots__github_backend.snap | 13 ++++++++++++- .../tests/snapshots/snapshots__local_only.snap | 13 ++++++++++++- justfiles/anvil/container.just | 13 ++++++++++++- 10 files changed, 91 insertions(+), 26 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 79c41144..1b8041ec 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:c779622507cad0d4702eccb0a2878e5702e8904682488cff2eb273a7215c8506" +catalog_checksum = "sha256:6a807c33aac1741fd97b8313aec22488e546caa9eb44627d839f7766657127c4" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:10235e8c3fdd04cf8b84a32a8b023ee8a651853710494bb7aea4b21fd4a7967b" +checksum = "sha256:b1ea4ee95f4f7e164a83ec7504394598de0079fd89d13bef5b37df5237b2632c" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 2f0efbaf..aaad0bc4 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -162,7 +162,7 @@ define the image. The name derives from the repository directory (§5.1). | `.anvil/container/Dockerfile.dockerignore` | always | | `rust-toolchain.toml` | always | | `.anvil/container/hooks.ps1` | when the file exists | -| `justfiles/anvil/**/*.just` | always | +| every file under `justfiles/anvil/` | always | The recipe tree is hashed in full. `just anvil-setup` reaches the install recipes through the tier, group and check recipes, so the routing decides *whether* a tool is installed just as surely as `tools.just` decides *how*: dropping an @@ -542,9 +542,12 @@ that content inside the Dockerfile itself, where it *is* hashed, or to accept th `ANVIL_CONTAINER_NO_CACHE=1` to take effect. A manual escape hatch is not content identity, so treat the second as a workaround rather than a supported contract. -`justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces: a non-recipe -file there would be copied into the image without being part of its identity, so editing it would change the image's -contents without renaming the tag. Non-recipe assets belong in a tool-owned directory such as `.anvil/`. +`justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces for +catalog-owned files. The reason is legibility rather than identity: the directory is the recipe tree, `just` parses +every file the image copies, and a catalog that hides an installer script there makes the tool set harder to reason +about than one that keeps it in `.anvil/`. Identity is safe either way, because the digest covers every file the build +context admits (§4.1), not only the recipes — a repository that adds a non-recipe file by hand still renames the tag +when it edits it. A fork inherits everything else: the recipes, the identity scheme, the cache volumes, the mounts, and the re-entry guard. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. diff --git a/crates/cargo-anvil/docs/design/extensibility.md b/crates/cargo-anvil/docs/design/extensibility.md index d7754292..0fd535ab 100644 --- a/crates/cargo-anvil/docs/design/extensibility.md +++ b/crates/cargo-anvil/docs/design/extensibility.md @@ -472,12 +472,14 @@ and nothing else: [`CatalogBuilder::build`](#4-the-shape-of-a-catalog) rejects any other owned file under that prefix, so a derived catalog fails loudly at construction instead of shipping a file whose absence is noticed only later. -The reason is containerized execution. The image identity hashes every `*.just` -under `justfiles/anvil/` recursively, while the build context admits the whole -directory, so a non-recipe file placed there is copied into the image but is -**not** part of its identity: editing it would change what the image contains -without renaming the tag, and no rebuild would follow. Non-recipe assets belong -in a tool-owned directory of their own, such as `.anvil/`. +The reason is legibility rather than image identity. The image identity hashes +every file under `justfiles/anvil/` recursively, and the build context admits +the whole directory, so a non-recipe file placed there is copied into the image +*and* covered by its tag — editing it renames the image and a rebuild follows. +What the rule protects is the meaning of the directory: it is the recipe tree, +`just` parses everything in it, and a catalog that hides an installer script +there makes the tool set harder to reason about. Non-recipe assets belong in a +tool-owned directory of their own, such as `.anvil/`. Containerized execution is itself an ordinary artifact group, customized with the same `replace_artifact` / `with_artifact` / `without_artifact` levers as diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 5309ac4d..d05407d0 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -539,9 +539,14 @@ mod tests { .rfind("$plan -match 'GITHUB_TOKEN'") .expect("the plan must decide whether a token is needed"); let dry_run = RECIPE[..plan] - .rfind("just --dry-run @targetParts") + .rfind("--dry-run @targetParts") .expect("the plan must come from just"); assert!(dry_run < plan && plan < guard, "compute the plan, match it, then derive"); + // Through the launching binary, like every other nested call: a bare + // `just` here fails silently when the caller invoked it by absolute + // path, and an empty plan reads as "no token needed". + assert!(!RECIPE.contains("(just --dry-run")); + assert!(RECIPE.contains("}}' --dry-run @targetParts")); // The predicate is the variable, not the name of a check, so a catalog // that adds another GitHub-authenticated check is covered for free. assert!(!RECIPE.contains("$plan -match 'aprz'")); diff --git a/crates/cargo-anvil/src/catalog/builder.rs b/crates/cargo-anvil/src/catalog/builder.rs index 42af0307..64a9c091 100644 --- a/crates/cargo-anvil/src/catalog/builder.rs +++ b/crates/cargo-anvil/src/catalog/builder.rs @@ -238,13 +238,13 @@ impl CatalogBuilder { } } -/// `justfiles/` is the recipe tree: the container image identity considers only -/// `*.just` files below it, while the Docker build context admits the whole -/// `justfiles/anvil/` directory. A non-recipe owned file placed there would -/// therefore be copied into the image without being part of its tag, so editing -/// it would change what the image contains while the tag still resolved. -/// Reject it at catalog-construction time instead, so a derived catalog fails -/// loudly rather than shipping a file that silently breaks image identity. +/// `justfiles/` is the recipe tree: `just` parses every file the container +/// build copies from it, so a non-recipe owned file there makes the tool set +/// harder to reason about than one kept in a tool-owned directory. Image +/// identity is not at stake — the digest covers every file the build context +/// admits, not only `*.just` — so this is a legibility rule, enforced at +/// catalog-construction time to keep a derived catalog honest about where its +/// assets live. fn non_recipe_under_justfiles(artifact: &Artifact) -> Option { let Artifact::OwnedFile(spec) = artifact else { return None; @@ -254,7 +254,7 @@ fn non_recipe_under_justfiles(artifact: &Artifact) -> Option { return None; } Some(format!( - "owned file '{}' is not a .just recipe; non-recipe artifacts must live outside justfiles/ (only *.just there is part of the container image identity)", + "owned file '{}' is not a .just recipe; non-recipe artifacts must live outside justfiles/ (it is the recipe tree, not an asset directory)", spec.path )) } diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 90cddbf9..8e5ca2b9 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -613,7 +613,18 @@ anvil-container *target: $needsToken = $targetParts.Count -eq 0 if (-not $needsToken) { $plan = '' - try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + # The same executable that launched this tree, for the reason every + # other nested call uses it: a caller invoking `just` by absolute + # path with its directory off PATH would otherwise fail here. That + # failure is silent, because an empty plan reads as "does not need a + # token" -- so anvil-aprz would run unauthenticated in an image with + # no gh of its own and block on the rate limit for up to an hour. + try { + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $plan = '' + } $needsToken = $plan -match 'GITHUB_TOKEN' } if ($needsToken) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 91d8b8af..314fac82 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4149,7 +4149,18 @@ anvil-container *target: $needsToken = $targetParts.Count -eq 0 if (-not $needsToken) { $plan = '' - try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + # The same executable that launched this tree, for the reason every + # other nested call uses it: a caller invoking `just` by absolute + # path with its directory off PATH would otherwise fail here. That + # failure is silent, because an empty plan reads as "does not need a + # token" -- so anvil-aprz would run unauthenticated in an image with + # no gh of its own and block on the rate limit for up to an hour. + try { + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $plan = '' + } $needsToken = $plan -match 'GITHUB_TOKEN' } if ($needsToken) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 0bd4c8f1..3de9276d 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4176,7 +4176,18 @@ anvil-container *target: $needsToken = $targetParts.Count -eq 0 if (-not $needsToken) { $plan = '' - try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + # The same executable that launched this tree, for the reason every + # other nested call uses it: a caller invoking `just` by absolute + # path with its directory off PATH would otherwise fail here. That + # failure is silent, because an empty plan reads as "does not need a + # token" -- so anvil-aprz would run unauthenticated in an image with + # no gh of its own and block on the rate limit for up to an hour. + try { + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $plan = '' + } $needsToken = $plan -match 'GITHUB_TOKEN' } if ($needsToken) { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 1907965b..5629e16e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2889,7 +2889,18 @@ anvil-container *target: $needsToken = $targetParts.Count -eq 0 if (-not $needsToken) { $plan = '' - try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + # The same executable that launched this tree, for the reason every + # other nested call uses it: a caller invoking `just` by absolute + # path with its directory off PATH would otherwise fail here. That + # failure is silent, because an empty plan reads as "does not need a + # token" -- so anvil-aprz would run unauthenticated in an image with + # no gh of its own and block on the rate limit for up to an hour. + try { + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $plan = '' + } $needsToken = $plan -match 'GITHUB_TOKEN' } if ($needsToken) { diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 90cddbf9..8e5ca2b9 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -613,7 +613,18 @@ anvil-container *target: $needsToken = $targetParts.Count -eq 0 if (-not $needsToken) { $plan = '' - try { $plan = (just --dry-run @targetParts 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' } + # The same executable that launched this tree, for the reason every + # other nested call uses it: a caller invoking `just` by absolute + # path with its directory off PATH would otherwise fail here. That + # failure is silent, because an empty plan reads as "does not need a + # token" -- so anvil-aprz would run unauthenticated in an image with + # no gh of its own and block on the rate limit for up to an hour. + try { + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $plan = '' + } $needsToken = $plan -match 'GITHUB_TOKEN' } if ($needsToken) { From d4d47ca6f335f628ec5cf757837bb20b8b7625be Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 18:43:43 +0200 Subject: [PATCH 42/81] fix(anvil): stop two commands answering a question they were not asked Both found by a second review pass and reproduced before fixing. - `just anvil-container` with no argument, run from inside the image, printed "already inside the container" and exited 0. The no-argument form asks for a shell; none is opened, so a success code reports that a request was carried out when it was not. It exits 1 now. A nested invocation *with* a target is unaffected -- that is the re-entry guard, and it still passes through and returns the recipe's own code. - `anvil-container-status` reported a present image as absent whenever the caller had exported `ANVIL_CONTAINER_NO_CACHE=1`. Status sets NO_REBUILD and NO_RESOLVE to make the resolve a pure query, but NO_CACHE also makes the resolver skip the local `image inspect` short-circuit, so the query fell through to a build it was forbidden to run and concluded absence. NO_CACHE is a statement about how the next build should behave; status is only ever asking what is on this machine, so it now clears it for the probe. Verified both against the real generated tree: the in-container form now exits 1, and status reports "present and current" with and without an exported NO_CACHE where before the two disagreed. Validation: `cargo test -p cargo-anvil` -- 311 unit plus the integration and snapshot suites, 0 failed. `cargo anvil --dry-run` converges at 79/79. --- .anvil.lock | 4 ++-- .../templates/justfiles/anvil/container.just | 12 +++++++++--- .../tests/snapshots/snapshots__ado_backend.snap | 12 +++++++++--- .../tests/snapshots/snapshots__github_backend.snap | 12 +++++++++--- .../tests/snapshots/snapshots__local_only.snap | 12 +++++++++--- justfiles/anvil/container.just | 12 +++++++++--- 6 files changed, 47 insertions(+), 17 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 1b8041ec..2e5dbd7d 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:6a807c33aac1741fd97b8313aec22488e546caa9eb44627d839f7766657127c4" +catalog_checksum = "sha256:cff7e7995e07ae68f47d68f462c848b67f64d7246d147f5fe52bf542791c87a6" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:b1ea4ee95f4f7e164a83ec7504394598de0079fd89d13bef5b37df5237b2632c" +checksum = "sha256:a98096548167c7fda93fc7c1a9b1dfa9b39ad26b02deb6d5e0f85929300b4b51" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 8e5ca2b9..1be80c02 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -450,10 +450,10 @@ anvil-container *target: # Already inside: pass straight through instead of nesting. if ($targetParts.Count -eq 0) { # The no-argument form asks for a shell in the image, and this *is* - # that shell. Running nothing and exiting 0 would be the one outcome - # that looks like it worked. + # that shell. Nothing runs, so exiting 0 would report success for a + # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") - exit 0 + exit 1 } just @targetParts exit $LASTEXITCODE @@ -759,6 +759,12 @@ anvil-container-status: $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' + # And explicitly *not* NO_CACHE. A caller who exported it is asking the next + # build to ignore the layer cache, which is a statement about building -- + # but it also makes the resolver skip the local `image inspect` + # short-circuit, so a present image would be reported absent by a command + # that is only ever asking what is on this machine. + $env:ANVIL_CONTAINER_NO_CACHE = $null & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 314fac82..714c0563 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3986,10 +3986,10 @@ anvil-container *target: # Already inside: pass straight through instead of nesting. if ($targetParts.Count -eq 0) { # The no-argument form asks for a shell in the image, and this *is* - # that shell. Running nothing and exiting 0 would be the one outcome - # that looks like it worked. + # that shell. Nothing runs, so exiting 0 would report success for a + # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") - exit 0 + exit 1 } just @targetParts exit $LASTEXITCODE @@ -4295,6 +4295,12 @@ anvil-container-status: $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' + # And explicitly *not* NO_CACHE. A caller who exported it is asking the next + # build to ignore the layer cache, which is a statement about building -- + # but it also makes the resolver skip the local `image inspect` + # short-circuit, so a present image would be reported absent by a command + # that is only ever asking what is on this machine. + $env:ANVIL_CONTAINER_NO_CACHE = $null & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 3de9276d..f6492d83 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4013,10 +4013,10 @@ anvil-container *target: # Already inside: pass straight through instead of nesting. if ($targetParts.Count -eq 0) { # The no-argument form asks for a shell in the image, and this *is* - # that shell. Running nothing and exiting 0 would be the one outcome - # that looks like it worked. + # that shell. Nothing runs, so exiting 0 would report success for a + # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") - exit 0 + exit 1 } just @targetParts exit $LASTEXITCODE @@ -4322,6 +4322,12 @@ anvil-container-status: $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' + # And explicitly *not* NO_CACHE. A caller who exported it is asking the next + # build to ignore the layer cache, which is a statement about building -- + # but it also makes the resolver skip the local `image inspect` + # short-circuit, so a present image would be reported absent by a command + # that is only ever asking what is on this machine. + $env:ANVIL_CONTAINER_NO_CACHE = $null & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 5629e16e..a86399cc 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2726,10 +2726,10 @@ anvil-container *target: # Already inside: pass straight through instead of nesting. if ($targetParts.Count -eq 0) { # The no-argument form asks for a shell in the image, and this *is* - # that shell. Running nothing and exiting 0 would be the one outcome - # that looks like it worked. + # that shell. Nothing runs, so exiting 0 would report success for a + # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") - exit 0 + exit 1 } just @targetParts exit $LASTEXITCODE @@ -3035,6 +3035,12 @@ anvil-container-status: $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' + # And explicitly *not* NO_CACHE. A caller who exported it is asking the next + # build to ignore the layer cache, which is a statement about building -- + # but it also makes the resolver skip the local `image inspect` + # short-circuit, so a present image would be reported absent by a command + # that is only ever asking what is on this machine. + $env:ANVIL_CONTAINER_NO_CACHE = $null & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 8e5ca2b9..1be80c02 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -450,10 +450,10 @@ anvil-container *target: # Already inside: pass straight through instead of nesting. if ($targetParts.Count -eq 0) { # The no-argument form asks for a shell in the image, and this *is* - # that shell. Running nothing and exiting 0 would be the one outcome - # that looks like it worked. + # that shell. Nothing runs, so exiting 0 would report success for a + # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") - exit 0 + exit 1 } just @targetParts exit $LASTEXITCODE @@ -759,6 +759,12 @@ anvil-container-status: $env:ANVIL_CONTAINER_NO_REBUILD = '1' $env:ANVIL_CONTAINER_NO_RESOLVE = '1' + # And explicitly *not* NO_CACHE. A caller who exported it is asking the next + # build to ignore the layer cache, which is a statement about building -- + # but it also makes the resolver skip the local `image inspect` + # short-circuit, so a present image would be reported absent by a command + # that is only ever asking what is on this machine. + $env:ANVIL_CONTAINER_NO_CACHE = $null & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-image *> $null if ($LASTEXITCODE -eq 0) { Write-Output "status: present and current" From a626f9f3a2c829741295b93e11ccb740847138ef Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Tue, 25 Aug 2026 19:05:32 +0200 Subject: [PATCH 43/81] fix(anvil): make the digest spec implementable and route the last nested call MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from a fresh-eyes pass over the previous rounds' fixes. - The in-container passthrough was the one nested call still invoking a bare `just`. Inside the image that is harmless -- `just` is on PATH -- but ANVIL_IN_CONTAINER is a documented control a developer can set on a host, and there the bare name resolves against PATH rather than the binary actually running. It was also pinned by an assertion, so the previous sweep could not have caught it without changing the test. - The digest length-prefixed with `String.Length`, which counts UTF-16 code units, while the stream is hashed as UTF-8. The framing stayed unambiguous, so no collision was possible, but §4.3 promises that a publisher and a consumer derive the same reference independently -- and a re-implementation outside .NET following the document would have produced different bytes for any non-ASCII path or content. Now `UTF8.GetByteCount`, matching what §4.2 says and what the stream is encoded in. Verified with a non-ASCII filename: the tag moves when it appears and returns when it goes. - §5.1 claimed the worktree redirection writes nothing to the host. It writes the generated `.git` file to the host temp directory and bind-mounts it from there, removing it in the recipe's `finally`. True for a normal run, and precisely wrong for the case someone would consult the section about -- a hard kill leaving `anvil-gitfile-*` behind. The same pass independently confirmed the earlier fixes: the digest covers the build context exactly, the engine probe reports a stopped daemon rather than absence, `$env:...NO_CACHE = $null` genuinely unsets in pwsh 7, and the two new recipe-contract tests fail if their premise breaks. Validation: `cargo test -p cargo-anvil` -- 311 unit plus the integration and snapshot suites, 0 failed. `cargo anvil --dry-run` converges at 79/79. --- .anvil.lock | 4 ++-- crates/cargo-anvil/docs/design/containers.md | 14 +++++++++----- .../cargo-anvil/src/anvil/artifacts/container.rs | 7 ++++++- .../templates/justfiles/anvil/container.just | 16 +++++++++++++--- .../tests/snapshots/snapshots__ado_backend.snap | 16 +++++++++++++--- .../snapshots/snapshots__github_backend.snap | 16 +++++++++++++--- .../tests/snapshots/snapshots__local_only.snap | 16 +++++++++++++--- justfiles/anvil/container.just | 16 +++++++++++++--- 8 files changed, 82 insertions(+), 23 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 2e5dbd7d..1b4047b9 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.4.0" -catalog_checksum = "sha256:cff7e7995e07ae68f47d68f462c848b67f64d7246d147f5fe52bf542791c87a6" +catalog_checksum = "sha256:65a18414ca3c641f1ed95ccb5ca64985295a5dd954ef1e299818bb6e95ee12e4" [[file]] path = ".anvil/container/Dockerfile" @@ -189,7 +189,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:a98096548167c7fda93fc7c1a9b1dfa9b39ad26b02deb6d5e0f85929300b4b51" +checksum = "sha256:cab1faf927e49e27f7883045a1ff6c916ddc44380c1f2488388a4b92d5af31a0" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index aaad0bc4..6550c63e 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -184,10 +184,12 @@ excluded: a credential must never influence a tag. ### 4.2 Digest Inputs are sorted by relative path with an ordinal comparison, then serialized into one stream in which each entry -contributes a literal `file`, the byte length of its relative path, the path, the byte length of its content, and the -content. Length-prefixing the two variable-length fields is what makes the stream self-delimiting: newline framing -would let a file whose body happened to contain `file`, a path and a newline serialize identically to two files -splitting at that point, so two different input sets could name one image. Tagging entries this way +contributes a literal `file`, the UTF-8 byte length of its relative path, the path, the UTF-8 byte length of its +content, and the content. Length-prefixing the two variable-length fields is what makes the stream self-delimiting: +newline framing would let a file whose body happened to contain `file`, a path and a newline serialize identically to +two files splitting at that point, so two different input sets could name one image. The lengths are byte counts of +the same UTF-8 encoding the stream is hashed in, so an independent re-implementation arrives at the same bytes. +Tagging entries this way prevents a rearrangement of names and contents from colliding. Line endings are normalized to LF, so CRLF and LF checkouts agree on the tag. The sort is ordinal because a case-insensitive one would drop one of two inputs differing only in case on the case-sensitive filesystem where the image is built. @@ -249,7 +251,9 @@ resolves nothing inside the container and each of them fails a long way from the The redirection is confined to the checkout: a git command run elsewhere in the container, such as `git init` in a scratch directory, is unaffected. That is why a generated `.git` file is used rather than `GIT_DIR`, which is ambient and would be inherited by every process in the container. An ordinary clone carries its git directory inside the bind -mount and takes none of this. Nothing is written to the host, and no flag or variable selects the behaviour. +mount and takes none of this. The generated `.git` file is written to the host temp directory, bind-mounted from +there, and removed when the run ends; nothing is written into the checkout, and no flag or variable selects the +behaviour. Only cargo's content-addressed download caches are volumes, so the write-heavy download path never crosses the host boundary and the host's own toolchain is untouched. `$CARGO_HOME` and `$RUSTUP_HOME` themselves are **not** mounted: diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index d05407d0..2778d2a8 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -358,8 +358,13 @@ mod tests { // would break `anvil-container anvil-setup binstall` -- and around // fifty generated recipes take a parameter. assert!(RECIPE.contains(r"-split '\s+'")); - assert!(RECIPE.contains("just @targetParts")); + assert!(RECIPE.contains("}}' @targetParts")); assert!(RECIPE.contains("@('just') + $targetParts")); + // Every nested call goes through the launching binary, including the + // in-container passthrough: ANVIL_IN_CONTAINER is a documented control + // a developer can set on a host, where a bare name resolves against + // PATH rather than the `just` that is running. + assert!(!RECIPE.contains("\n just @targetParts")); } #[test] diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 1be80c02..4ee7682c 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -180,6 +180,11 @@ anvil-container-tag: # a hook that exists versus an ignore file whose body ends in the hook's # path and body, for instance -- and the second state would silently reuse # the first state's image. Byte counts cannot be forged by content. + # + # UTF-8 byte counts, not String.Length: the stream is hashed as UTF-8, and a + # publisher re-implementing this from the design document has to arrive at + # the same bytes. UTF-16 code units would differ for any non-ASCII path or + # content and make the spec unimplementable outside .NET. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -190,8 +195,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) - [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) + [void]$stream.Append('file ').Append([System.Text.Encoding]::UTF8.GetByteCount($rel)).Append(' ').Append($rel) + [void]$stream.Append(' ').Append([System.Text.Encoding]::UTF8.GetByteCount($text)).Append(' ').Append($text) } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on @@ -455,7 +460,12 @@ anvil-container *target: [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - just @targetParts + # Through the launching binary like every other nested call. Inside the + # image `just` is on PATH so the bare name would work, but + # ANVIL_IN_CONTAINER is a documented control a developer can set on a + # host, and there the bare name resolves against PATH rather than the + # `just` that is actually running. + & '{{ replace(just_executable(), "'", "''") }}' @targetParts exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 714c0563..7f3c42f1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3716,6 +3716,11 @@ anvil-container-tag: # a hook that exists versus an ignore file whose body ends in the hook's # path and body, for instance -- and the second state would silently reuse # the first state's image. Byte counts cannot be forged by content. + # + # UTF-8 byte counts, not String.Length: the stream is hashed as UTF-8, and a + # publisher re-implementing this from the design document has to arrive at + # the same bytes. UTF-16 code units would differ for any non-ASCII path or + # content and make the spec unimplementable outside .NET. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -3726,8 +3731,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) - [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) + [void]$stream.Append('file ').Append([System.Text.Encoding]::UTF8.GetByteCount($rel)).Append(' ').Append($rel) + [void]$stream.Append(' ').Append([System.Text.Encoding]::UTF8.GetByteCount($text)).Append(' ').Append($text) } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on @@ -3991,7 +3996,12 @@ anvil-container *target: [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - just @targetParts + # Through the launching binary like every other nested call. Inside the + # image `just` is on PATH so the bare name would work, but + # ANVIL_IN_CONTAINER is a documented control a developer can set on a + # host, and there the bare name resolves against PATH rather than the + # `just` that is actually running. + & '{{ replace(just_executable(), "'", "''") }}' @targetParts exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f6492d83..abcfd553 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3743,6 +3743,11 @@ anvil-container-tag: # a hook that exists versus an ignore file whose body ends in the hook's # path and body, for instance -- and the second state would silently reuse # the first state's image. Byte counts cannot be forged by content. + # + # UTF-8 byte counts, not String.Length: the stream is hashed as UTF-8, and a + # publisher re-implementing this from the design document has to arrive at + # the same bytes. UTF-16 code units would differ for any non-ASCII path or + # content and make the spec unimplementable outside .NET. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -3753,8 +3758,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) - [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) + [void]$stream.Append('file ').Append([System.Text.Encoding]::UTF8.GetByteCount($rel)).Append(' ').Append($rel) + [void]$stream.Append(' ').Append([System.Text.Encoding]::UTF8.GetByteCount($text)).Append(' ').Append($text) } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on @@ -4018,7 +4023,12 @@ anvil-container *target: [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - just @targetParts + # Through the launching binary like every other nested call. Inside the + # image `just` is on PATH so the bare name would work, but + # ANVIL_IN_CONTAINER is a documented control a developer can set on a + # host, and there the bare name resolves against PATH rather than the + # `just` that is actually running. + & '{{ replace(just_executable(), "'", "''") }}' @targetParts exit $LASTEXITCODE } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index a86399cc..694e34f6 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2456,6 +2456,11 @@ anvil-container-tag: # a hook that exists versus an ignore file whose body ends in the hook's # path and body, for instance -- and the second state would silently reuse # the first state's image. Byte counts cannot be forged by content. + # + # UTF-8 byte counts, not String.Length: the stream is hashed as UTF-8, and a + # publisher re-implementing this from the design document has to arrive at + # the same bytes. UTF-16 code units would differ for any non-ASCII path or + # content and make the spec unimplementable outside .NET. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -2466,8 +2471,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) - [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) + [void]$stream.Append('file ').Append([System.Text.Encoding]::UTF8.GetByteCount($rel)).Append(' ').Append($rel) + [void]$stream.Append(' ').Append([System.Text.Encoding]::UTF8.GetByteCount($text)).Append(' ').Append($text) } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on @@ -2731,7 +2736,12 @@ anvil-container *target: [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - just @targetParts + # Through the launching binary like every other nested call. Inside the + # image `just` is on PATH so the bare name would work, but + # ANVIL_IN_CONTAINER is a documented control a developer can set on a + # host, and there the bare name resolves against PATH rather than the + # `just` that is actually running. + & '{{ replace(just_executable(), "'", "''") }}' @targetParts exit $LASTEXITCODE } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 1be80c02..4ee7682c 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -180,6 +180,11 @@ anvil-container-tag: # a hook that exists versus an ignore file whose body ends in the hook's # path and body, for instance -- and the second state would silently reuse # the first state's image. Byte counts cannot be forged by content. + # + # UTF-8 byte counts, not String.Length: the stream is hashed as UTF-8, and a + # publisher re-implementing this from the design document has to arrive at + # the same bytes. UTF-16 code units would differ for any non-ASCII path or + # content and make the spec unimplementable outside .NET. $stream = [System.Text.StringBuilder]::new() $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) @@ -190,8 +195,8 @@ anvil-container-tag: exit 1 } $text = [System.IO.File]::ReadAllText($path) -replace "`r`n", "`n" - [void]$stream.Append('file ').Append($rel.Length).Append(' ').Append($rel) - [void]$stream.Append(' ').Append($text.Length).Append(' ').Append($text) + [void]$stream.Append('file ').Append([System.Text.Encoding]::UTF8.GetByteCount($rel)).Append(' ').Append($rel) + [void]$stream.Append(' ').Append([System.Text.Encoding]::UTF8.GetByteCount($text)).Append(' ').Append($text) } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on @@ -455,7 +460,12 @@ anvil-container *target: [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - just @targetParts + # Through the launching binary like every other nested call. Inside the + # image `just` is on PATH so the bare name would work, but + # ANVIL_IN_CONTAINER is a documented control a developer can set on a + # host, and there the bare name resolves against PATH rather than the + # `just` that is actually running. + & '{{ replace(just_executable(), "'", "''") }}' @targetParts exit $LASTEXITCODE } From 08354da27aa51fe75de97bd025cd7adc4107d796 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 00:00:01 +0200 Subject: [PATCH 44/81] refactor(anvil): compose the container Dockerfile from managed regions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An owned Dockerfile that invites in-place edits freezes the base digest and four tool pins the moment a repository edits it, while `anvil-container-tag` goes on resolving -- because the tag hashes the repository's own file. The identity scheme keeps working and still names a stale image, and the drift flow cannot recover: there is no three-way merge and no recorded ancestor, so every upgrade hands over an `.anvil-proposed` sibling nobody reconciles. Split anvil's content into four managed regions with three gaps between them, each gap defined by what must already be true at that point in the build: a root CA before the first download, a compile dependency before `anvil-setup`, a run-time tool after it. Regions do not make anvil's content unwritable -- `updates.md` §2 still applies, and anvil never destroys repository content -- but they remove the reason to edit it, which is what fixes the freeze for any repository that uses the gaps. Two constraints the region engine had to grow, both specific to a Dockerfile: - `# syntax=docker/dockerfile:1` is honoured only as the first line, and a region's opening sentinel is a comment, so the directive cannot live inside one. It is seeded once as a scaffold when the file does not exist. - Region order is semantic. Fresh files get catalog order; thereafter the on-disk sequence is checked and the host is refused, naming the region that moved, rather than emitting a Dockerfile that is silently wrong. Also fixes an owned-file-to-region-host transition that deleted the file the regions had just written, and widens the digest to walk `.anvil/container/` rather than naming three files, so anything a gap `COPY`s is an image input. The arm CI failure was this branch's own: `mutants_diff_covers_uncommitted_work` asserted `--in-diff` on a leg where `anvil-mutants-diff` deliberately exits 0, because cargo-mutants does not build for aarch64-pc-windows-msvc. The test now pins the architecture it branches on, and a sibling covers the skip itself. Validation: - `cargo test -p cargo-anvil` -- 378 passed, 0 failed - `scripts/test-anvil-container.ps1 -Engine docker` -- 69/69 against a real daemon, including a real image build, content in a gap surviving regeneration, and an edit inside a region being preserved - `cargo anvil --dry-run` -- exit 0, 77 unchanged --- .anvil.lock | 30 +- .anvil/container/Dockerfile | 75 +++-- .anvil/container/Dockerfile.dockerignore | 4 +- .spelling | 1 + crates/cargo-anvil/README.md | 35 ++- crates/cargo-anvil/docs/design/containers.md | 97 ++++-- crates/cargo-anvil/docs/design/local.md | 13 +- .../src/anvil/artifacts/container.rs | 297 +++++++++++++++--- crates/cargo-anvil/src/anvil/artifacts/mod.rs | 9 +- crates/cargo-anvil/src/lib.rs | 34 +- crates/cargo-anvil/src/plan.rs | 4 +- crates/cargo-anvil/src/run.rs | 133 +++++++- .../templates/anvil/container/Dockerfile | 132 -------- .../anvil/container/Dockerfile.base.region | 34 ++ .../anvil/container/Dockerfile.dockerignore | 4 +- .../anvil/container/Dockerfile.entry.region | 6 + .../anvil/container/Dockerfile.header | 29 ++ .../anvil/container/Dockerfile.setup.region | 40 +++ .../anvil/container/Dockerfile.tools.region | 53 ++++ .../templates/justfiles/anvil/container.just | 35 ++- crates/cargo-anvil/tests/container_upgrade.rs | 200 +++++++++++- crates/cargo-anvil/tests/extensibility.rs | 16 +- crates/cargo-anvil/tests/recipe_contracts.rs | 59 ++++ .../snapshots/snapshots__ado_backend.snap | 121 +++++-- .../snapshots/snapshots__github_backend.snap | 121 +++++-- .../snapshots/snapshots__local_only.snap | 121 +++++-- justfiles/anvil/container.just | 35 ++- scripts/test-anvil-container.ps1 | 41 ++- 28 files changed, 1383 insertions(+), 396 deletions(-) delete mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.entry.region create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.header create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region diff --git a/.anvil.lock b/.anvil.lock index 2dbfff11..7520d19e 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,15 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:799d1dc17b2e3f7ef1af299996d4f5b0d829692132b4a13fabcedcee6411422b" - -[[file]] -path = ".anvil/container/Dockerfile" -checksum = "sha256:a3e106b7dbde0bb6a9c2b94dae0f9cb82b9055ea97f100438f3d3d20ec66e971" +catalog_checksum = "sha256:05e7198e42dce094505776df73581e933f89db4d961ef2f1127967d211e5a818" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:6e0f2fa763766c09ba05e5f48aeca8fbc3894125208e1ee2de8d433a5dbbfcef" +checksum = "sha256:bcffa702f6802244b798c6587c6ade9ef1baa586c3952bdcbe8f4e88cb8f73d4" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -169,7 +165,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:34a1383e17f3d6ac909fc3206bb3f59885256620bab178e7cdf650ba520d150b" +checksum = "sha256:345497c7865dc5962cb80f4d7ff28846cb56d8ee93915c8e638d772b04dd215a" [[file]] path = "justfiles/anvil/groups/pr-fast.just" @@ -227,6 +223,26 @@ checksum = "sha256:a1e44ca16f172b487afa3997f102512733d3b65a4418cf894cbd749a3abc1 path = "justfiles/anvil/versions.just" checksum = "sha256:acbea93d5117db747537f4f7b9a5eb90b7d3e0dd3e8684cc0e4dc1dcb15ac93e" +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-base" +checksum = "sha256:52ed0faca8dec1cf4ad09602feb7ae627a0d5f1cabc903a8052c6b771c1302f6" + +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-entry" +checksum = "sha256:7b409a9b560c214e10b50f74330fb6f8c0c12c3d83494e0dcf016f2411b50365" + +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-setup" +checksum = "sha256:80e0b29f9e8eec3b8c98976c718f11e5afc9c63131fcf4b0a0998c04895ace5b" + +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-tools" +checksum = "sha256:0faa15c3d2a9d6fafb0748c92ffc15564e8cbfd3dbfd8284c2eacf701f4efe72" + [[region]] host = ".delta.toml" id = "anvil-delta" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 3786a20a..c39539f7 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -1,32 +1,36 @@ # syntax=docker/dockerfile:1 # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. # -# Default Anvil execution image, emitted for every repository. The image -# installs exactly the tools the generated catalog pins, by running -# `just anvil-setup` -- the same recipe the checks themselves use. That is what -# makes "the image has the right tools" true by construction rather than by -# convention: there is no second list to keep in step. +# Anvil execution image. # -# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this -# file's text, so an edit here renames the image -- but it does not resolve or -# validate the base, and a floating tag can therefore change underneath a tag -# that claims to name fixed content. +# The `anvil-managed:` regions below belong to cargo-anvil and are reconciled +# on every run, so an edit inside one is replaced. Everything outside them is +# yours and is preserved byte-for-byte, including this header. # +# Add your own instructions in the gaps between the regions. Each region ends +# by describing what the gap after it is for, because position decides whether +# a line works at all: a corporate root CA has to land before the first +# download, and a library needed to *compile* a catalog tool has to land +# before `anvil-setup` runs. +# +# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit +# honours the directive only when nothing precedes it -- not even a comment. +# That is also why it sits out here rather than inside a region: a region's +# opening sentinel is a comment, and would silently demote the directive to an +# ordinary one, leaving the build on the default frontend with nothing failing +# to say so. + +# >>> anvil-managed: anvil-container-base # The base tracks the Linux runner the generated workflows use # (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the # catalog as prebuilt binaries, which require that runner's glibc; it is # backward but not forward compatible, so the pin moves when the runner does. # -# To build on a different base (a lower glibc baseline, or an internal -# distribution), a downstream catalog replaces this artifact wholesale via -# `replace_artifact(artifacts::container::dockerfile(...))`; a single -# repository can edit this file in place, which anvil's drift handling -# preserves. A lower baseline means the catalog must also install from source -# rather than with `binstall`. - +# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a +# change here renames the image -- but it does not resolve or validate the +# base, and a floating tag could therefore change underneath a reference that +# claims to name fixed content. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} @@ -45,9 +49,19 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# --- your turn ------------------------------------------------------------- +# The gap below this region runs before anvil's first download. Put whatever +# the image needs in order to reach the network here: a corporate root CA, +# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. +# Without it, a TLS-intercepting proxy makes every `curl` in the next region +# fail and the image cannot be built at all. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-base + +# >>> anvil-managed: anvil-container-tools # clang/libclang are required by cargo-spellcheck; the rest is the usual Rust -# link-time set. A bare slim base has no C runtime development files, so every -# link step fails without build-essential. +# link-time set. A bare base has no C runtime development files, so every link +# step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ @@ -91,6 +105,16 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the toolchain is in place and before +# `anvil-setup` installs the catalog. Put system libraries a catalog tool +# needs in order to *compile* here: `binstall` falls back to a source build +# when a pinned tool publishes no prebuilt for this platform, and that build +# links against whatever headers the image has. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-tools + +# >>> anvil-managed: anvil-container-setup # Install the pinned toolchain and cargo subcommands from the generated # catalog. The whole recipe tree is copied because `just` has to parse it, and # the whole tree is hashed into the tag: `anvil-setup` reaches the install @@ -124,9 +148,20 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the catalog is installed. Put what your +# own checks need at run time here: a database client, a protobuf compiler, a +# linter that is not a cargo subcommand. It is also the cheapest place to add +# anything, because an edit here invalidates one layer rather than the +# multi-minute `anvil-setup` above. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-setup + +# >>> anvil-managed: anvil-container-entry # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace CMD ["bash"] +# <<< anvil-managed: anvil-container-entry diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index e7e522cf..6f694796 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -1,7 +1,7 @@ # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. +# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Update the corresponding template in the cargo-anvil crate. # # BuildKit reads `.dockerignore` in preference to a root # `.dockerignore`, so this scopes the exec-image build context without the diff --git a/.spelling b/.spelling index 30fc4dce..37528dc0 100644 --- a/.spelling +++ b/.spelling @@ -531,3 +531,4 @@ deprecations parallelization remediate recency +unbuildable diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index eca185ce..3b0f0ec2 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -155,9 +155,12 @@ ARM64 hosts it is emulated and is substantially slower. #### Image identity -The tag *is* a SHA-256 digest over the inputs that define the image: the -Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -and the whole generated `justfiles/anvil/` tree. The tree is included in +The tag *is* a SHA-256 digest over the inputs that define the image: +everything under `.anvil/container/`, `rust-toolchain.toml`, and the whole +generated `justfiles/anvil/` tree. The container directory is walked rather +than named file by file, because the Dockerfile is composed and a +repository can `COPY` a certificate or an install script it places there. +The recipe tree is included in full because the image installs its tools by running `just anvil-setup`, whose dependency chain runs through the tier, group and check recipes before it reaches the install recipes – so the routing decides *whether* a @@ -232,14 +235,22 @@ you trust. #### Customizing the image -`.anvil/container/Dockerfile` is an ordinary owned file: edit it in place -for extra packages, and anvil’s drift handling preserves the change. A -downstream catalog that needs a different base OS or toolchain source for -every repository it manages replaces the artifact instead. A replacement -that copies more of the tree must replace the ignore file with it, since -the build context admits only `justfiles/anvil/` and `rust-toolchain.toml`. See -[`artifacts::container`][__link1] and the design document for the full contract, -the host setup for each engine, and the known limitations. +`.anvil/container/Dockerfile` is a **user-composed file with managed +regions**: anvil owns four regions inside it and reconciles them on every +run, and the three gaps between them are the repository’s. Add extra +packages in the gap that suits when they are needed – before the first +download for a root CA or a proxy, before `anvil-setup` for libraries a +catalog tool compiles against, after it for what the checks need at run +time. Nothing anvil owns is touched, so base and tool-pin bumps keep +landing. + +A downstream catalog that needs a different base OS for every repository it +manages replaces the base and tool regions instead, inheriting the catalog +install and the entry contract. A replacement that copies more of the tree +must replace the ignore file with it, since the build context admits only +`justfiles/anvil/` and `rust-toolchain.toml`. See [`artifacts::container`][__link1] +and the design document for the full contract, the host setup for each +engine, and the known limitations. ### Checks and tiers @@ -468,7 +479,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbCpN2n89Kx1AbOat1oo99_c4babhpCrfrqV8bSQWuf-vfBINhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb6i_TmlHRql4bFUHGYxLMHPgbSgKMOCwLAF4bQ9SS643u-FthZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 6550c63e..b5911ede 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -131,14 +131,53 @@ repo/ │ ├── container.just the anvil-container recipes │ └── … checks, groups, tiers, executed natively *inside* the image └── .anvil/container/ - ├── Dockerfile what the image contains + ├── Dockerfile composed: anvil's four regions, your content in the gaps ├── Dockerfile.dockerignore what the build context admits └── hooks.ps1 optional; not emitted by default (§7) ``` -`container.just` and the `Dockerfile` are both owned files with the same drift handling: anvil preserves a repository's -edit and reports a proposal rather than overwriting it (`updates.md` §2). They differ in header wording and in whether -editing is *invited* — the Dockerfile is meant to be extended (§8), the driver is not. +`container.just` and `Dockerfile.dockerignore` are owned files carrying the usual `DO NOT EDIT DIRECTLY` marker. + +The **Dockerfile is a user-composed file with managed regions**, not an owned file. Anvil owns four regions inside it +and keeps them current; everything outside the sentinels — including the header and the three gaps between them — is +the repository's and is preserved byte-for-byte. + +| Region | Contains | Gap that follows it is for | +| --- | --- | --- | +| `anvil-container-base` | `ARG BASE_IMAGE`, `FROM`, the four download pins, `ENV` | a root CA, `http_proxy`, an internal apt mirror — anything needed to reach the network at all | +| `anvil-container-tools` | system packages, `pwsh`, `just`, `rustup`, `cargo-binstall` | libraries a catalog tool needs to *compile*, when `binstall` falls back to a source build | +| `anvil-container-setup` | `COPY` of the recipe tree, `just anvil-setup` | what the repository's own checks need at run time; also the cheapest layer to add to | +| `anvil-container-entry` | `ANVIL_IN_CONTAINER`, `WORKDIR`, `CMD` | — | + +**Why not an owned file that invites edits.** `updates.md` §2 preserves an edited owned file and writes anvil's version +to `.anvil-proposed`. There is no three-way merge and no recorded common ancestor, so every upgrade hands the +repository two files to reconcile by hand — and a side file does not get reconciled indefinitely. Here that failure is +silent in a way that matters: the file carries the base digest and four tool pins, so a repository that edits it once +keeps building on the base and versions frozen at that moment, while `anvil-container-tag` resolves happily *because +the tag hashes their file*. The identity scheme works perfectly and still names a stale image. + +**What regions actually change.** They do not make anvil's content unwritable: §2's ownership rules apply to a region +body exactly as they do to a file, so an edit *inside* a region is still preserved and still produces a proposal rather +than being overwritten. Anvil never destroys repository content, and a special case here would be the one place it did. +What changes is that there is no longer a reason to edit: every legitimate addition — a root CA, a build dependency, a +run-time tool — has a gap that is the *correct* place for it, chosen by what must already be true at that point in the +build. Editing anvil's content stops being the only way to extend the image and becomes a mistake the layout steers +away from, and the pin freeze goes with it for every repository that takes the gaps. + +**Two constraints the region engine had to grow for this.** Both are specific to a Dockerfile and neither applies to the +order-independent TOML and line-set hosts anvil already had: + +- **`# syntax=docker/dockerfile:1` must be line 1.** BuildKit honours a parser directive only when nothing precedes it, + not even a comment — and a region's opening sentinel *is* a comment, so the directive cannot live inside one without + being silently demoted, leaving the build on the default frontend with nothing failing to say so. It is therefore + part of a **scaffold** anvil writes when the file does not exist, and never reconciles afterwards. +- **Region order is semantic.** `FROM` must precede everything, and the toolchain must exist before `anvil-setup` runs. + Fresh files get catalog order; thereafter the engine checks the on-disk sequence against the declared one and + **refuses** the host, reporting which region is out of place, rather than emitting a Dockerfile that is wrong. + +A repository upgrading from the release that owned this path outright is re-seeded rather than appended to: a file +tracked as an owned file in the lock and carrying none of the regions is a previous render, not composition. A +Dockerfile the repository wrote itself is in neither state and is left alone, with the regions added to it. The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the @@ -158,12 +197,17 @@ define the image. The name derives from the repository directory (§5.1). | Input | Hashed | | --- | --- | -| `.anvil/container/Dockerfile` | always | -| `.anvil/container/Dockerfile.dockerignore` | always | +| every file under `.anvil/container/` | always | | `rust-toolchain.toml` | always | -| `.anvil/container/hooks.ps1` | when the file exists | | every file under `justfiles/anvil/` | always | +`.anvil/container/` is hashed by walking it, not as a fixed list of three known files. The Dockerfile is composed, so a +repository can `COPY` something from one of its gaps — a root CA, an install script, a patch — and a downstream +catalog's replacement region can do the same. Naming only the files anvil happens to know about would let any of those +change the image under a reference that already resolves, which is the hole the digest exists to close. A missing +Dockerfile is still a hard error, checked by name: the walk alone would let it contribute nothing and yield a confident +tag for an image that cannot be built. + The recipe tree is hashed in full. `just anvil-setup` reaches the install recipes through the tier, group and check recipes, so the routing decides *whether* a tool is installed just as surely as `tools.just` decides *how*: dropping an `anvil--setup` dependency from a group changes the installed set while `tools.just` and `versions.just` stay @@ -175,8 +219,7 @@ it belongs in the set because it passes the build arguments, the secret mounts a into the build. The cost is that editing any recipe renames the image and the next run rebuilds it. That is the correct trade: a tag -that can name contents the image does not have makes every guarantee below meaningless. A declared input that does not -exist is a hard error, not an omission from the digest. +that can name contents the image does not have makes every guarantee below meaningless. The hook file's **content** is an input, since it determines what the build installs. Its **output** is deliberately excluded: a credential must never influence a tag. @@ -525,26 +568,28 @@ an ordinary artifact group and uses the same levers as any other. | Goal | Mechanism | Owner | | --- | --- | --- | -| Extra packages in one repository | Edit `.anvil/container/Dockerfile` in place | repository | -| A different base OS or toolchain source, everywhere | `replace_artifact(artifacts::container::dockerfile().with_body(…))` | catalog | +| Extra packages in one repository | Add them in one of the three gaps in `.anvil/container/Dockerfile` (§3) | repository | +| A different base OS, everywhere | `replace_artifact(artifacts::container::dockerfile_base().with_body(…))`, usually with `dockerfile_tools()` | catalog | | Credentials, or a published image | Add `.anvil/container/hooks.ps1`, or ship `artifacts::container::hooks(…)` | either | -| No containerized execution at all | `without_artifact` for each of the three artifacts | catalog | +| No containerized execution at all | `without_artifact` for each artifact in the group | catalog | + +A repository adds to the Dockerfile without editing anything anvil owns, so pin bumps keep landing (§3). A change that +belongs everywhere is still better made in a catalog, where every consumer gets it. -Editing the Dockerfile in one repository is supported and the drift flow preserves the edit, but anvil keeps offering -its own version against a file it can see has diverged. A change that belongs everywhere is better made in a catalog. +Replacing a *region* rather than the whole file is what makes a downstream catalog cheap to keep current: +`dockerfile_setup()` and `dockerfile_entry()` are the contract with the driver and are inherited, so an Azure Linux or +msrustup catalog rewrites the base and tool layers and nothing else. Replacing `dockerfile_setup()` reintroduces the +second tool list the design exists to avoid, and is almost never right. -**The Dockerfile and its ignore file must be replaced together.** A replacement that `COPY`s anything beyond -`justfiles/anvil/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the -added paths never reach the build context and the build fails on a missing file. +**A replacement must keep the ignore file in step.** A region that `COPY`s anything beyond `justfiles/anvil/` and +`rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the added paths never reach the +build context and the build fails on a missing file. -**A replacement cannot extend the digest, so anything extra it copies must not vary independently.** The hashed set -is fixed (§4.1) and a fork has no way to add to it, which puts this customization path in tension with the identity -guarantee: an installer script, a config file or a certificate copied by a replacement Dockerfile sits outside the -tag, so editing it changes what the image contains while naming a reference that already resolves — and the stale -image is reused rather than rebuilt. Until a catalog can contribute digest inputs, the honest options are to carry -that content inside the Dockerfile itself, where it *is* hashed, or to accept that changing it needs -`ANVIL_CONTAINER_NO_CACHE=1` to take effect. A manual escape hatch is not content identity, so treat the second as a -workaround rather than a supported contract. +**Anything extra it copies is digested, provided it lives under `.anvil/container/`.** The hashed set is that whole +directory (§4.1), so an installer script, a config file or a certificate placed beside the Dockerfile is an input: +editing it renames the tag and the next run rebuilds. Content copied from elsewhere in the repository is not, and the +tag will not move when it changes — keep it under `.anvil/container/` and the identity guarantee holds without a +manual `ANVIL_CONTAINER_NO_CACHE=1`. `justfiles/anvil/` must contain `.just` recipes and nothing else, which `CatalogBuilder::build` enforces for catalog-owned files. The reason is legibility rather than identity: the directory is the recipe tree, `just` parses @@ -554,7 +599,7 @@ context admits (§4.1), not only the recipes — a repository that adds a non-re when it edits it. A fork inherits everything else: the recipes, the identity scheme, the cache volumes, the mounts, and the re-entry -guard. A different base OS with a different toolchain source is one Dockerfile replacement plus one hook. +guard. A different base OS with a different toolchain source is two region replacements plus one hook. ## 9. Limitations diff --git a/crates/cargo-anvil/docs/design/local.md b/crates/cargo-anvil/docs/design/local.md index 511d2402..1d0a14f4 100644 --- a/crates/cargo-anvil/docs/design/local.md +++ b/crates/cargo-anvil/docs/design/local.md @@ -48,16 +48,19 @@ repo/ │ Read by recipes via `{{ var }}` interpolation. See §3. │ └── .anvil/container/ the container image definition - ├── Dockerfile editable; drift is preserved + ├── Dockerfile composed: anvil-managed regions, your content between ├── Dockerfile.dockerignore └── hooks.ps1 optional; credentials, not emitted by default ``` -The Justfile region is the only file anvil adds to that the user co-owns, and it's -a single `import` line. Generated recipes live inside `justfiles/anvil/`; the -container image definition lives inside `.anvil/container/`. Generated files in +The Justfile region is not the only file anvil adds to that the user co-owns: the +container `Dockerfile` is composed the same way, from four managed regions with +the repository's own instructions in the gaps between them (see +[containers.md](./containers.md)). Generated recipes live inside `justfiles/anvil/`; +the container image definition lives inside `.anvil/container/`. Generated files in both directories are tool-owned (tracked by full-file checksum in the sidecar -manifest). If the user wants to add project-specific recipes, they add them to +manifest), except the composed `Dockerfile`, whose regions are tracked +individually. If the user wants to add project-specific recipes, they add them to the top-level `Justfile` outside the managed region, or to their own additional imported `.just` files. The alias `anvil := anvil-pr` lives in `mod.just`, not in the user's `Justfile`, so renaming or retargeting the alias is a template update diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index d1b334f9..120b465b 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -3,42 +3,106 @@ //! Containerized execution: the `anvil-container` recipe and the image it runs. //! -//! Three artifacts define the whole feature. The recipe drives the engine and -//! computes the image identity; the Dockerfile defines what the image contains; -//! its build-context ignore file decides what reaches the build at all, and the -//! two must be replaced together. There is no configuration file: whether the -//! group is emitted at all is a catalog decision, and the only host-specific -//! value — which engine to call — is an environment variable read by the recipe -//! at run time. +//! The recipe drives the engine and computes the image identity; the Dockerfile +//! defines what the image contains; its build-context ignore file decides what +//! reaches the build at all, and the two must be replaced together. There is no +//! configuration file: whether the group is emitted at all is a catalog +//! decision, and the only host-specific value — which engine to call — is an +//! environment variable read by the recipe at run time. //! -//! A downstream catalog customizes exactly two things, and inherits everything -//! else: +//! # Why the Dockerfile is composed, not owned //! -//! - [`dockerfile`] plus [`Artifact::with_body`] to build on a different base -//! or install the toolchain from a different source. A replacement that -//! copies more of the tree must replace [`dockerignore`] with it, or the -//! added paths never reach the build context. +//! The Dockerfile is a **user-composed file with managed regions**, not a +//! wholly-owned file. An owned file that invites in-place edits fails silently +//! here: `updates.md` §2 preserves the edit and writes anvil's version to +//! `.anvil-proposed`, with no three-way merge and no recorded ancestor, so a +//! repository that edits it once keeps building on the base digest and the four +//! tool pins frozen at that moment — while `anvil-container-tag` keeps +//! resolving, because the tag hashes *their* file. The identity scheme works +//! perfectly and still names a stale image. +//! +//! Splitting anvil's content into regions does not make it unwritable — §2's +//! ownership rules apply to a region body as they do to a file, and anvil never +//! overwrites repository content. What it removes is the *reason* to edit: each +//! of the three gaps between the regions is the correct home for one class of +//! addition, defined by what must already be true at that point in the build. +//! +//! | Gap | Runs | Exists for | +//! | --- | --- | --- | +//! | after [`dockerfile_base`] | before the first download | root CA, proxy, internal apt mirror | +//! | after [`dockerfile_tools`] | after the toolchain, before `anvil-setup` | libraries a catalog tool needs to *compile* | +//! | after [`dockerfile_setup`] | after the catalog is installed | what the repository's own checks need at run time | +//! +//! A downstream catalog customizes by replacing individual regions, and +//! inherits the rest: +//! +//! - [`dockerfile_base`] / [`dockerfile_tools`] plus [`Artifact::with_body`] to +//! build on a different base OS or install the toolchain from a different +//! source. [`dockerfile_setup`] and [`dockerfile_entry`] are the contract with +//! the recipe and are rarely replaced. +//! - [`dockerignore`] alongside them when the replacement copies more of the +//! tree, or the added paths never reach the build context. //! - [`hooks`] to supply credentials, or to resolve a published image through //! `Anvil-ResolveImage`. The recipe loads the file when it is present, //! regardless of who put it there. -use crate::catalog::Artifact; +use crate::catalog::{Artifact, HostSelector, RegionId, RegionSpec}; +use crate::region::CommentSyntax; const RECIPE: &str = include_str!("../../../templates/justfiles/anvil/container.just"); -const DOCKERFILE: &str = include_str!("../../../templates/anvil/container/Dockerfile"); const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dockerfile.dockerignore"); +/// Seeded into the Dockerfile when the file does not exist, and never +/// reconciled afterwards — it is the user's half of a composed file. +/// +/// It carries `# syntax=docker/dockerfile:1`, which BuildKit honours only as +/// the very first line of the file. A region's opening sentinel is a comment, +/// so the directive cannot live inside a region without being demoted to an +/// ordinary comment — silently, with the build falling back to the default +/// frontend and nothing failing to say so. +pub(crate) const DOCKERFILE_HEADER: &str = include_str!("../../../templates/anvil/container/Dockerfile.header"); + +const DOCKERFILE_BASE: &str = include_str!("../../../templates/anvil/container/Dockerfile.base.region"); +const DOCKERFILE_TOOLS: &str = include_str!("../../../templates/anvil/container/Dockerfile.tools.region"); +const DOCKERFILE_SETUP: &str = include_str!("../../../templates/anvil/container/Dockerfile.setup.region"); +const DOCKERFILE_ENTRY: &str = include_str!("../../../templates/anvil/container/Dockerfile.entry.region"); + const RECIPE_PATH: &str = "justfiles/anvil/container.just"; -const DOCKERFILE_PATH: &str = ".anvil/container/Dockerfile"; + +/// The composed Dockerfile the managed regions are spliced into. +pub(crate) const DOCKERFILE_PATH: &str = ".anvil/container/Dockerfile"; + const DOCKERIGNORE_PATH: &str = ".anvil/container/Dockerfile.dockerignore"; +/// The region ids anvil owns inside [`DOCKERFILE_PATH`], in the order a valid +/// Dockerfile must carry them. +/// +/// Order is load-bearing in a way no other managed-region host is: `FROM` must +/// precede every instruction, the toolchain must exist before `anvil-setup` +/// runs, and `WORKDIR`/`CMD` close the file. The engine checks the on-disk +/// sequence against this list and refuses rather than emitting a Dockerfile +/// that is silently wrong. +pub(crate) const DOCKERFILE_REGION_ORDER: &[&str] = &[ + "anvil-container-base", + "anvil-container-tools", + "anvil-container-setup", + "anvil-container-entry", +]; + /// The path the recipe loads credentials from, when a file is present there. pub const HOOKS_PATH: &str = ".anvil/container/hooks.ps1"; /// The full container artifact group. #[must_use] pub fn all() -> Vec { - vec![recipe(), dockerfile(), dockerignore()] + vec![ + recipe(), + dockerignore(), + dockerfile_base(), + dockerfile_tools(), + dockerfile_setup(), + dockerfile_entry(), + ] } /// The `anvil-container` recipe and its private helpers. @@ -47,32 +111,71 @@ pub fn recipe() -> Artifact { Artifact::owned_file(RECIPE_PATH, RECIPE) } -/// The default execution image: a digest-pinned base tracking the Linux CI -/// runner, which installs the pinned toolchain and the generated tool catalog -/// by running `just anvil-setup`. -/// -/// The catalog is installed as prebuilt binaries, which require that runner's -/// glibc. A catalog on an older base installs from source instead. +fn dockerfile_region(id: &'static str, body: &'static str) -> Artifact { + Artifact::region(RegionSpec { + host: HostSelector::Path(DOCKERFILE_PATH.to_owned()), + id: RegionId::new(id), + body: body.to_owned(), + syntax: CommentSyntax::Hash, + }) +} + +/// The base image and the pinned tool versions: a digest-pinned base tracking +/// the Linux CI runner, the four download pins, and the environment every later +/// region depends on. /// -/// A downstream catalog that needs a different base OS or toolchain source -/// replaces the body wholesale: +/// A downstream catalog that needs a different base OS replaces this and +/// usually [`dockerfile_tools`] with it: /// /// ```ignore /// catalog.replace_artifact( -/// artifacts::container::dockerfile().with_body(include_str!("../templates/Dockerfile")), +/// artifacts::container::dockerfile_base() +/// .with_body(include_str!("../templates/base.dockerfile")), /// ) /// ``` #[must_use] -pub fn dockerfile() -> Artifact { - Artifact::owned_file(DOCKERFILE_PATH, DOCKERFILE) +pub fn dockerfile_base() -> Artifact { + dockerfile_region("anvil-container-base", DOCKERFILE_BASE) } -/// The build-context ignore file for [`dockerfile`]. +/// The toolchain layer: the system packages, then `pwsh`, `just`, `rustup` and +/// `cargo-binstall` installed against published checksums. +/// +/// This is the region a catalog on a different package ecosystem replaces — +/// `tdnf` rather than `apt-get`, an internal toolchain source rather than +/// `rustup`. +#[must_use] +pub fn dockerfile_tools() -> Artifact { + dockerfile_region("anvil-container-tools", DOCKERFILE_TOOLS) +} + +/// The catalog install: copies the generated recipe tree and runs +/// `just anvil-setup`, so the image installs exactly what the checks pin. +/// +/// Rarely replaced. Installing by running the same recipe the checks use is +/// what makes "the image has the right tools" true by construction; a +/// replacement reintroduces the second tool list this exists to avoid. +#[must_use] +pub fn dockerfile_setup() -> Artifact { + dockerfile_region("anvil-container-setup", DOCKERFILE_SETUP) +} + +/// The entry contract: `ANVIL_IN_CONTAINER`, the workdir the repository is +/// bind-mounted at, and the default command. +/// +/// Every line here is a contract with the recipe rather than an opinion about +/// the image, so replacing it breaks the driver. +#[must_use] +pub fn dockerfile_entry() -> Artifact { + dockerfile_region("anvil-container-entry", DOCKERFILE_ENTRY) +} + +/// The build-context ignore file for the composed Dockerfile. /// /// `BuildKit` reads `.dockerignore` in preference to a root /// `.dockerignore`, so the build context is scoped without the repository -/// having to own a root ignore file. A catalog that replaces the Dockerfile -/// with one that copies more of the tree must replace this too. +/// having to own a root ignore file. A catalog that replaces a region with one +/// that copies more of the tree must replace this too. #[must_use] pub fn dockerignore() -> Artifact { Artifact::owned_file(DOCKERIGNORE_PATH, DOCKERIGNORE) @@ -128,25 +231,99 @@ mod tests { .iter() .map(|artifact| match artifact { Artifact::OwnedFile(spec) => spec.path, - Artifact::Region(_) => panic!("container group must contain owned files only"), + Artifact::Region(_) => panic!("expected an owned file"), }) .collect() } + fn region_ids(artifacts: &[Artifact]) -> Vec<&str> { + artifacts + .iter() + .filter_map(|artifact| match artifact { + Artifact::Region(spec) => Some(spec.id.as_str()), + Artifact::OwnedFile(_) => None, + }) + .collect() + } + + /// The Dockerfile as a fresh repository first receives it: the seeded + /// header followed by every region body in the order the engine enforces. + fn composed_dockerfile() -> String { + let mut out = DOCKERFILE_HEADER.to_owned(); + for body in [DOCKERFILE_BASE, DOCKERFILE_TOOLS, DOCKERFILE_SETUP, DOCKERFILE_ENTRY] { + out.push_str(body); + } + out + } + + #[test] + fn group_is_two_owned_files_and_four_dockerfile_regions() { + let all = all(); + let owned: Vec<_> = all + .iter() + .filter(|artifact| matches!(artifact, Artifact::OwnedFile(_))) + .cloned() + .collect(); + assert_eq!(paths(&owned), [RECIPE_PATH, DOCKERIGNORE_PATH]); + assert_eq!(region_ids(&all), DOCKERFILE_REGION_ORDER); + } + + #[test] + fn every_dockerfile_region_targets_the_one_composed_host() { + for artifact in all() { + if let Artifact::Region(spec) = artifact { + assert_eq!(spec.host, HostSelector::Path(DOCKERFILE_PATH.to_owned())); + assert_eq!(spec.syntax, CommentSyntax::Hash); + } + } + } + #[test] - fn group_is_exactly_three_files() { - assert_eq!(paths(&all()), [RECIPE_PATH, DOCKERFILE_PATH, DOCKERIGNORE_PATH]); + fn the_syntax_directive_leads_the_seeded_header() { + // BuildKit honours the parser directive only when nothing precedes it. + // If this ever moves, the frontend pin stops applying and nothing + // fails to say so. + assert_eq!( + DOCKERFILE_HEADER.lines().next(), + Some("# syntax=docker/dockerfile:1"), + "the parser directive must be the first line of the seeded header" + ); + } + + #[test] + fn no_region_body_carries_the_parser_directive() { + // Inside a region the directive would sit below an opening sentinel -- + // a comment -- and be silently demoted. + for body in [DOCKERFILE_BASE, DOCKERFILE_TOOLS, DOCKERFILE_SETUP, DOCKERFILE_ENTRY] { + assert!(!body.contains("# syntax="), "a region body must not carry the parser directive"); + } } #[test] fn image_installs_the_generated_toolset() { // The image must not carry a second tool list: it installs by running // the same recipe the checks use, from the same generated pins. - assert!(DOCKERFILE.contains("just anvil-setup binstall")); - assert!(DOCKERFILE.contains("COPY justfiles")); - assert!(DOCKERFILE.contains("COPY rust-toolchain.toml")); + let composed = composed_dockerfile(); + assert!(composed.contains("just anvil-setup binstall")); + assert!(composed.contains("COPY justfiles")); + assert!(composed.contains("COPY rust-toolchain.toml")); // The re-entry guard the recipe relies on to avoid nesting. - assert!(DOCKERFILE.contains("ENV ANVIL_IN_CONTAINER=1")); + assert!(composed.contains("ENV ANVIL_IN_CONTAINER=1")); + } + + #[test] + fn the_composed_order_is_a_buildable_dockerfile() { + // The region order is not cosmetic: everything depends on FROM, and + // the catalog install needs the toolchain that precedes it. + let composed = composed_dockerfile(); + let at = |needle: &str| { + composed + .find(needle) + .unwrap_or_else(|| panic!("missing from composed Dockerfile: {needle}")) + }; + assert!(at("FROM ${BASE_IMAGE}") < at("RUN apt-get update")); + assert!(at("RUN apt-get update") < at("just anvil-setup binstall")); + assert!(at("just anvil-setup binstall") < at("WORKDIR /workspace")); } #[test] @@ -154,10 +331,10 @@ mod tests { // A floating base tag can change underneath an identity hash that // claims to name fixed content, which would make every cached image a // potential lie. - let base = DOCKERFILE + let base = DOCKERFILE_BASE .lines() .find(|line| line.starts_with("ARG BASE_IMAGE=")) - .expect("the Dockerfile must declare a default BASE_IMAGE"); + .expect("the base region must declare a default BASE_IMAGE"); assert!(base.contains("@sha256:"), "BASE_IMAGE must be digest-pinned: {base}"); } @@ -403,10 +580,31 @@ mod tests { #[test] fn hook_file_is_an_image_input_but_hook_output_is_not() { // A changed hook must rename the tag; a minted credential must not. - assert!(RECIPE.contains("$inputs += $hookRel")); + // The hook is no longer named individually -- it is picked up by the + // walk of `.anvil/container/`, which is what also catches a file a + // repository `COPY`s from one of the Dockerfile's user gaps. + assert!(RECIPE.contains("$containerRoot = Join-Path $repoRoot '.anvil/container'")); assert!(RECIPE.contains("id=$id,env=$name")); } + #[test] + fn every_file_under_the_container_directory_is_an_image_input() { + // The Dockerfile is composed: a repository adds `COPY` lines in the + // gaps between anvil's regions, naming files anvil never hears about. + // Hashing a fixed list would let those change the image under a + // reference that already resolves. + let walk = RECIPE + .find("$containerRoot = Join-Path $repoRoot '.anvil/container'") + .expect("the tag must walk the container directory"); + let recurse = RECIPE[walk..] + .find("Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force") + .expect("the walk must be recursive and include hidden entries"); + // A missing Dockerfile must still be fatal: the walk alone would let + // it contribute nothing and yield a confident tag for an unbuildable + // image. + assert!(RECIPE[walk + recurse..].contains("container image input is missing: $dockerfile")); + } + #[test] fn hook_values_are_passed_by_name_never_by_value() { // NAME=VALUE on a command line is recorded by endpoint telemetry and @@ -644,9 +842,20 @@ mod tests { } #[test] - fn dockerfile_body_can_be_replaced_by_a_fork() { - let replaced = dockerfile().with_body("FROM example.invalid/base\n"); - assert_eq!(paths(std::slice::from_ref(&replaced)), [DOCKERFILE_PATH]); + fn a_fork_replaces_one_region_without_touching_the_others() { + // The point of splitting the file: a catalog on another base OS + // rewrites the base and tool layers and inherits the catalog install + // and the entry contract, instead of forking the whole Dockerfile and + // freezing every pin in it. + let replaced = dockerfile_base().with_body("FROM example.invalid/base\n"); + match &replaced { + Artifact::Region(spec) => { + assert_eq!(spec.id.as_str(), "anvil-container-base"); + assert_eq!(spec.host, HostSelector::Path(DOCKERFILE_PATH.to_owned())); + } + Artifact::OwnedFile(_) => panic!("the Dockerfile regions must stay regions"), + } assert_eq!(replaced.body(), "FROM example.invalid/base\n"); + assert_eq!(dockerfile_setup().body(), DOCKERFILE_SETUP); } } diff --git a/crates/cargo-anvil/src/anvil/artifacts/mod.rs b/crates/cargo-anvil/src/anvil/artifacts/mod.rs index 1123251f..fbe4cc82 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/mod.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/mod.rs @@ -138,15 +138,8 @@ mod tests { let generated_marker = spec.body.contains("GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY."); let customizable_wrapper = spec.path == ".pipelines/anvil/steps/job.yml" && spec.body.contains("Default job wrapper emitted by cargo-anvil."); - // The container image definition is deliberately editable in - // place: a repository that needs a different base or extra - // packages changes it and anvil's drift handling preserves the - // edit. A "DO NOT EDIT" marker would contradict that, so these - // carry the weaker provenance marker instead. - let editable_image_definition = - spec.path.starts_with(".anvil/container/Dockerfile") && spec.body.contains("Managed by cargo-anvil."); assert!( - generated_marker || customizable_wrapper || editable_image_definition, + generated_marker || customizable_wrapper, "owned file '{}' lacks the generated-content marker", spec.path ); diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index b3597101..08f635f7 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -156,9 +156,12 @@ //! //! ### Image identity //! -//! The tag *is* a SHA-256 digest over the inputs that define the image: the -//! Dockerfile and its ignore file, `rust-toolchain.toml`, the optional hook, -//! and the whole generated `justfiles/anvil/` tree. The tree is included in +//! The tag *is* a SHA-256 digest over the inputs that define the image: +//! everything under `.anvil/container/`, `rust-toolchain.toml`, and the whole +//! generated `justfiles/anvil/` tree. The container directory is walked rather +//! than named file by file, because the Dockerfile is composed and a +//! repository can `COPY` a certificate or an install script it places there. +//! The recipe tree is included in //! full because the image installs its tools by running `just anvil-setup`, //! whose dependency chain runs through the tier, group and check recipes //! before it reaches the install recipes -- so the routing decides *whether* a @@ -233,14 +236,23 @@ //! //! ### Customizing the image //! -//! `.anvil/container/Dockerfile` is an ordinary owned file: edit it in place -//! for extra packages, and anvil's drift handling preserves the change. A -//! downstream catalog that needs a different base OS or toolchain source for -//! every repository it manages replaces the artifact instead. A replacement -//! that copies more of the tree must replace the ignore file with it, since -//! the build context admits only `justfiles/anvil/` and `rust-toolchain.toml`. See -//! [`artifacts::container`] and the design document for the full contract, -//! the host setup for each engine, and the known limitations. +//! `.anvil/container/Dockerfile` is a **user-composed file with managed +//! regions**: anvil owns four regions inside it and keeps them current, and the +//! three gaps between them are the repository's. Add extra packages in the gap +//! that suits when they are needed -- before the first download for a root CA +//! or a proxy, before `anvil-setup` for libraries a catalog tool compiles +//! against, after it for what the checks need at run time. Adding in a gap +//! leaves anvil's content alone, so base and tool-pin bumps keep landing; +//! editing inside a region is preserved rather than overwritten, but freezes +//! those pins at the moment of the edit, which is why the gaps exist. +//! +//! A downstream catalog that needs a different base OS for every repository it +//! manages replaces the base and tool regions instead, inheriting the catalog +//! install and the entry contract. A replacement that copies more of the tree +//! must replace the ignore file with it, since the build context admits only +//! `justfiles/anvil/` and `rust-toolchain.toml`. See [`artifacts::container`] +//! and the design document for the full contract, the host setup for each +//! engine, and the known limitations. //! //! ## Checks and tiers //! diff --git a/crates/cargo-anvil/src/plan.rs b/crates/cargo-anvil/src/plan.rs index 32b3771a..b0574a5d 100644 --- a/crates/cargo-anvil/src/plan.rs +++ b/crates/cargo-anvil/src/plan.rs @@ -347,7 +347,7 @@ impl Plan { write_section(&mut out, "Will update", &updates); write_section(&mut out, "Will propose", &proposes); write_section(&mut out, "Will remove", &removes); - write_section(&mut out, "Orphaned (customized; transferring ownership)", &orphans_kept); + write_section(&mut out, "No longer managed (left in place; ownership transferred)", &orphans_kept); write_section(&mut out, "Will leave alone (silent)", &leave_alones); if !in_syncs.is_empty() { @@ -672,7 +672,7 @@ mod tests { let s = plan.summary(None); assert!(s.contains("Will remove: 1 item(s)")); assert!(s.contains("- dropped.txt")); - assert!(s.contains("Orphaned (customized; transferring ownership): 1 item(s)")); + assert!(s.contains("No longer managed (left in place; ownership transferred): 1 item(s)")); assert!(s.contains("- Justfile [anvil-old]")); } diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index cb03b78e..c461d254 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -12,6 +12,7 @@ use std::path::Path; use ohno::{AppError, bail}; use tracing::info; +use crate::anvil::artifacts::container; use crate::anvil::artifacts::region::DELTA_REGION_ID; use crate::backend::{self, Backend}; use crate::catalog::Catalog; @@ -159,6 +160,9 @@ fn build_plan( ) -> Result { let mut plan = Plan::default(); let mut hosts = HostTextCache::default(); + // Hosts already reported as unsafe to compose. Every region targeting one + // hits the same fault, and four copies of one message is noise. + let mut refused_hosts = BTreeSet::new(); for artifact in catalog.artifacts() { match artifact { @@ -170,7 +174,7 @@ fn build_plan( } } Artifact::Region(spec) => { - push_region(repo_root, workspace, manifest, &mut plan, &mut hosts, spec)?; + push_region(repo_root, workspace, manifest, &mut plan, &mut hosts, &mut refused_hosts, spec)?; } } } @@ -303,25 +307,26 @@ fn push_region( manifest: &Manifest, plan: &mut Plan, hosts: &mut HostTextCache, + refused_hosts: &mut BTreeSet, spec: &RegionSpec, ) -> Result<(), AppError> { match &spec.host { HostSelector::Path(path) => { - push_region_at(repo_root, manifest, plan, hosts, path, spec)?; + push_region_at(repo_root, manifest, plan, hosts, refused_hosts, path, spec)?; } HostSelector::WorkspaceCargoToml => { if workspace.has_workspace_table { - push_region_at(repo_root, manifest, plan, hosts, "Cargo.toml", spec)?; + push_region_at(repo_root, manifest, plan, hosts, refused_hosts, "Cargo.toml", spec)?; } } HostSelector::SingleCrateCargoToml => { if !workspace.has_workspace_table { - push_region_at(repo_root, manifest, plan, hosts, "Cargo.toml", spec)?; + push_region_at(repo_root, manifest, plan, hosts, refused_hosts, "Cargo.toml", spec)?; } } HostSelector::EachMemberManifest => { for member in &workspace.members { - push_region_at(repo_root, manifest, plan, hosts, &member.manifest_relpath, spec)?; + push_region_at(repo_root, manifest, plan, hosts, refused_hosts, &member.manifest_relpath, spec)?; } } } @@ -342,11 +347,57 @@ fn push_region_at( manifest: &Manifest, plan: &mut Plan, hosts: &mut HostTextCache, + refused_hosts: &mut BTreeSet, host: &str, spec: &RegionSpec, ) -> Result<(), AppError> { let host = resolve_existing_case_insensitive(repo_root, host); let current = hosts.get_or_read(repo_root, &host)?; + // A composed host anvil seeds: when the file does not exist yet, the + // scaffold becomes the base the first region splices into, so the parts of + // the file that cannot live inside a region — a Dockerfile's + // `# syntax=` parser directive above all — are present from the first run. + // It is written once and never reconciled: everything outside the sentinels + // is the repository's. + // + // A file left behind by a release that owned this path outright is re-seeded + // too. Its content is a previous render, not user composition, so appending + // the regions to it would produce a file carrying both the old whole-file + // definition and the new regions. Recognising it needs both conditions: + // tracked as an owned file in the lock we are superseding, and carrying none + // of this host's regions yet. A Dockerfile the repository wrote by hand is + // in neither state and is left alone. + let scaffold = host_scaffold(&host); + let superseded_owned_file = scaffold.is_some() + && manifest.files.contains_key(host.as_str()) + && current.as_deref().is_some_and(|text| !host_carries_any_region(&host, text)); + let current = match (current, scaffold) { + (None, Some(scaffold)) => Some(scaffold.to_owned()), + (Some(_), Some(scaffold)) if superseded_owned_file => Some(scaffold.to_owned()), + (current, _) => current, + }; + if let Some(text) = current.as_deref() + && let Some(reason) = host_composition_violation(&host, text) + { + // One diagnostic per host: every region targeting it hits the same + // fault, and repeating it once per region buries the one line that + // says which region moved. + if refused_hosts.insert(host.clone()) { + plan.refusal(format!( + "Refused to manage {host} because its managed regions could not be safely \ + updated: {reason}. Restore the documented order and re-run. Other artifacts \ + were still planned." + )); + } + plan.push(PlanItem::noop( + Target::Region { + host, + id: spec.id.as_str().to_owned(), + }, + Decision::LeaveAlone, + )); + return Ok(()); + } let placement = region_placement(spec.id.as_str()); let body = match delta_region_body(current.as_deref(), spec) { DeltaRegionBody::Managed => spec.body.as_str(), @@ -405,6 +456,67 @@ fn region_placement(region_id: &str) -> RegionPlacement { } } +/// The initial content for a host anvil composes but does not own, used only +/// when the file is absent. +/// +/// Most region hosts are files the repository already has (`Cargo.toml`, +/// `deny.toml`) or files whose first region can simply be appended to nothing. +/// The container Dockerfile is neither: `# syntax=docker/dockerfile:1` is a +/// BuildKit parser directive that is honoured only when nothing precedes it, +/// not even a comment — so it cannot live inside a region, whose opening +/// sentinel *is* a comment. +fn host_scaffold(host_relpath: &str) -> Option<&'static str> { + (host_relpath == container::DOCKERFILE_PATH).then_some(container::DOCKERFILE_HEADER) +} + +/// The order anvil's regions must appear in inside a composed host, when order +/// is semantically load-bearing. +fn host_region_order(host_relpath: &str) -> Option<&'static [&'static str]> { + (host_relpath == container::DOCKERFILE_PATH).then_some(container::DOCKERFILE_REGION_ORDER) +} + +/// Whether the host already carries any of the regions anvil owns in it. +/// +/// Distinguishes a composed file mid-update from a file a previous release +/// owned outright, which must be re-seeded rather than appended to. +fn host_carries_any_region(host_relpath: &str, text: &str) -> bool { + host_region_order(host_relpath).is_some_and(|order| { + order + .iter() + .any(|id| matches!(find_region(text, id, CommentSyntax::Hash), Ok(Some(_)))) + }) +} + +/// Reject a host whose managed regions can no longer be updated into a valid +/// file. +/// +/// Every other managed-region host is order-independent — TOML tables, line +/// sets — so `upsert_region` replacing a region wherever it is found is enough. +/// A Dockerfile is the first host where relative order is semantic: `FROM` must +/// precede every instruction that depends on it, the toolchain must exist +/// before `anvil-setup` runs, and a reordered file is not a diagnostic but a +/// silently wrong or unbuildable image. Detect it and refuse instead. +fn host_composition_violation(host_relpath: &str, text: &str) -> Option { + let order = host_region_order(host_relpath)?; + let mut previous: Option<(&str, usize)> = None; + for id in order { + // A malformed region is reported by the planner itself, with a better + // message than this check could give; a region that is simply absent is + // about to be written. + let Ok(Some(region)) = find_region(text, id, CommentSyntax::Hash) else { + continue; + }; + let start = region.start_line.start; + if let Some((earlier_id, earlier_start)) = previous + && start < earlier_start + { + return Some(format!("region '{id}' appears before '{earlier_id}', but must follow it")); + } + previous = Some((id, start)); + } + None +} + enum DeltaRegionBody { Managed, PreserveRepositoryKey, @@ -461,11 +573,22 @@ fn plan_removals(repo_root: &Path, previous: &Manifest, plan: &mut Plan, hosts: Target::File { .. } => None, }) .collect(); + let live_region_hosts: BTreeSet = live_regions.iter().map(|(host, _)| host.clone()).collect(); for (path, last) in &previous.files { if live_files.contains(path) { continue; } + // A path that is no longer an owned file but *is* the host of a live + // managed region has not been retired -- it has changed ownership + // model. Deleting it here would erase what the region writes planned + // for the same pass just produced, and the user content around them + // with it. Drop the stale manifest entry and leave the file alone; the + // region entries now describe what anvil owns inside it. + if live_region_hosts.contains(path) { + plan.push(PlanItem::orphaned_kept(Target::File { path: path.clone() })); + continue; + } let disk = read_file_if_present(&repo_root.join(path))?; let disk_checksum = disk.as_deref().map(checksum_str); match decide_removal(last, disk_checksum.as_deref()) { diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile b/crates/cargo-anvil/templates/anvil/container/Dockerfile deleted file mode 100644 index 3786a20a..00000000 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile +++ /dev/null @@ -1,132 +0,0 @@ -# syntax=docker/dockerfile:1 -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. -# -# Default Anvil execution image, emitted for every repository. The image -# installs exactly the tools the generated catalog pins, by running -# `just anvil-setup` -- the same recipe the checks themselves use. That is what -# makes "the image has the right tools" true by construction rather than by -# convention: there is no second list to keep in step. -# -# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this -# file's text, so an edit here renames the image -- but it does not resolve or -# validate the base, and a floating tag can therefore change underneath a tag -# that claims to name fixed content. -# -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. -# -# To build on a different base (a lower glibc baseline, or an internal -# distribution), a downstream catalog replaces this artifact wholesale via -# `replace_artifact(artifacts::container::dockerfile(...))`; a single -# repository can edit this file in place, which anvil's drift handling -# preserves. A lower baseline means the catalog must also install from source -# rather than with `binstall`. - -ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea -FROM ${BASE_IMAGE} - -ARG JUST_VERSION=1.56.0 -ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 -ARG POWERSHELL_VERSION=7.6.3 -ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 -ARG RUSTUP_VERSION=1.29.0 -ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 -ARG CARGO_BINSTALL_VERSION=1.21.1 -ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e - -ENV DEBIAN_FRONTEND=noninteractive \ - CARGO_HOME=/usr/local/cargo \ - RUSTUP_HOME=/usr/local/rustup \ - RUSTUP_NO_UPDATE_CHECK=1 \ - PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust -# link-time set. A bare slim base has no C runtime development files, so every -# link step fails without build-essential. -RUN apt-get update \ - && apt-get install -y --no-install-recommends \ - build-essential ca-certificates clang libclang-dev curl git libicu-dev \ - libssl-dev pkg-config tar \ - && rm -rf /var/lib/apt/lists/* - -# pwsh is not optional: every generated anvil recipe is a `script("pwsh", -# "-NoProfile")` recipe. -RUN curl -fsSLo /tmp/powershell.tar.gz \ - "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ - && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ - && mkdir -p /opt/microsoft/powershell/7 \ - && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ - && chmod 755 /opt/microsoft/powershell/7/pwsh \ - && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ - && rm /tmp/powershell.tar.gz - -RUN curl -fsSLo /tmp/just.tar.gz \ - "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ - && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ - && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ - && chmod 755 /usr/local/bin/just \ - && rm /tmp/just.tar.gz - -RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ - "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ - && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ - && chmod 755 /tmp/rustup-init \ - && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ - && rm /tmp/rustup-init - -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. -RUN curl -fsSLo /tmp/cargo-binstall.tgz \ - "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ - && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ - && mkdir -p "${CARGO_HOME}/bin" \ - && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ - && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ - && rm /tmp/cargo-binstall.tgz - -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. -# -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. -# -# The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. -# -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. -WORKDIR /opt/anvil -COPY justfiles ./justfiles -COPY rust-toolchain.toml ./ -RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ - && just anvil-setup binstall \ - && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ - && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ - && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ - && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# Consumed by `anvil-container` itself: a nested invocation from inside the -# image runs the recipe natively instead of launching another container. -ENV ANVIL_IN_CONTAINER=1 - -WORKDIR /workspace -CMD ["bash"] diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region new file mode 100644 index 00000000..d3870f9e --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region @@ -0,0 +1,34 @@ +# The base tracks the Linux runner the generated workflows use +# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the +# catalog as prebuilt binaries, which require that runner's glibc; it is +# backward but not forward compatible, so the pin moves when the runner does. +# +# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a +# change here renames the image -- but it does not resolve or validate the +# base, and a floating tag could therefore change underneath a reference that +# claims to name fixed content. +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +FROM ${BASE_IMAGE} + +ARG JUST_VERSION=1.56.0 +ARG JUST_SHA256=fa2a8ec1015d9df5330941ade12437488fc40d33f9c9f8cd4eb70a26de11b639 +ARG POWERSHELL_VERSION=7.6.3 +ARG POWERSHELL_SHA256=856d0765d2332377f9d7a4aea76efdfde4de51446e7738dde2dfda41dba9e2a7 +ARG RUSTUP_VERSION=1.29.0 +ARG RUSTUP_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +ARG CARGO_BINSTALL_VERSION=1.21.1 +ARG CARGO_BINSTALL_SHA256=630c8f8803a686aa6779497f0f0fb51d49822fb5fc3c514d8ced33b34e338e6e + +ENV DEBIAN_FRONTEND=noninteractive \ + CARGO_HOME=/usr/local/cargo \ + RUSTUP_HOME=/usr/local/rustup \ + RUSTUP_NO_UPDATE_CHECK=1 \ + PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +# --- your turn ------------------------------------------------------------- +# The gap below this region runs before anvil's first download. Put whatever +# the image needs in order to reach the network here: a corporate root CA, +# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. +# Without it, a TLS-intercepting proxy makes every `curl` in the next region +# fail and the image cannot be built at all. +# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore index e7e522cf..6f694796 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore @@ -1,7 +1,7 @@ # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. +# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Update the corresponding template in the cargo-anvil crate. # # BuildKit reads `.dockerignore` in preference to a root # `.dockerignore`, so this scopes the exec-image build context without the diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.entry.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.entry.region new file mode 100644 index 00000000..bbdea1b4 --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.entry.region @@ -0,0 +1,6 @@ +# Consumed by `anvil-container` itself: a nested invocation from inside the +# image runs the recipe natively instead of launching another container. +ENV ANVIL_IN_CONTAINER=1 + +WORKDIR /workspace +CMD ["bash"] diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header new file mode 100644 index 00000000..d2103f0f --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header @@ -0,0 +1,29 @@ +# syntax=docker/dockerfile:1 +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Anvil execution image. +# +# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them +# current, so base and tool-pin bumps arrive on their own. Everything outside +# them is yours and is preserved byte-for-byte, including this header. +# +# Add your own instructions in the gaps between the regions rather than inside +# one. Anvil will not overwrite an edit you make inside a region -- it offers +# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and +# is the trap: the region carries the base digest and four tool pins, so an edit +# there quietly freezes them at today's values while the image tag keeps +# resolving, because the tag hashes your file. The gaps exist so that nothing +# you legitimately need to add ever requires touching them. +# +# Each region ends by describing what the gap after it is for, because position +# decides whether a line works at all: a corporate root CA has to land before +# the first download, and a library needed to *compile* a catalog tool has to +# land before `anvil-setup` runs. +# +# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit +# honours the directive only when nothing precedes it -- not even a comment. +# That is also why it sits out here rather than inside a region: a region's +# opening sentinel is a comment, and would silently demote the directive to an +# ordinary one, leaving the build on the default frontend with nothing failing +# to say so. diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region new file mode 100644 index 00000000..10fb615b --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region @@ -0,0 +1,40 @@ +# Install the pinned toolchain and cargo subcommands from the generated +# catalog. The whole recipe tree is copied because `just` has to parse it, and +# the whole tree is hashed into the tag: `anvil-setup` reaches the install +# recipes through the tier, group and check recipes, so any of them can change +# what this layer installs. The synthetic Justfile below imports only the anvil +# tree, avoiding repository-specific imports that may not exist yet. +# +# `binstall` matches what CI passes. It is not just a speed-up: some catalog +# tools do not compile from source on every pinned toolchain, so a source +# install can fail here while CI stays green. +# +# The credential files are removed in the same layer as the install. A build +# secret is mounted, never committed to a layer, but anything the install +# *writes* with it is ordinary content -- and the `chmod` on the next line +# would otherwise publish it world-readable. Deleting them in a later `RUN` +# would not help: the earlier layer still carries the file. +# +# `registry` and `git` are created before the `chmod` because the run mounts a +# named volume over each. An engine seeds a new volume from the image path it +# covers, and a path that does not exist is seeded as a root-owned 0755 +# directory -- which the `--user` mapping on Linux then cannot write, so the +# first cargo fetch fails with EACCES. Creating them here means the widened +# permissions are what the volume inherits. +WORKDIR /opt/anvil +COPY justfiles ./justfiles +COPY rust-toolchain.toml ./ +RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ + && just anvil-setup binstall \ + && rm -rf "${CARGO_HOME}/registry/cache" "${CARGO_HOME}/registry/src" \ + && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ + && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ + && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" + +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the catalog is installed. Put what your +# own checks need at run time here: a database client, a protobuf compiler, a +# linter that is not a cargo subcommand. It is also the cheapest place to add +# anything, because an edit here invalidates one layer rather than the +# multi-minute `anvil-setup` above. +# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region new file mode 100644 index 00000000..a7e8979d --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region @@ -0,0 +1,53 @@ +# clang/libclang are required by cargo-spellcheck; the rest is the usual Rust +# link-time set. A bare base has no C runtime development files, so every link +# step fails without build-essential. +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + build-essential ca-certificates clang libclang-dev curl git libicu-dev \ + libssl-dev pkg-config tar \ + && rm -rf /var/lib/apt/lists/* + +# pwsh is not optional: every generated anvil recipe is a `script("pwsh", +# "-NoProfile")` recipe. +RUN curl -fsSLo /tmp/powershell.tar.gz \ + "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-x64.tar.gz" \ + && echo "${POWERSHELL_SHA256} /tmp/powershell.tar.gz" | sha256sum -c - \ + && mkdir -p /opt/microsoft/powershell/7 \ + && tar -xzf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \ + && chmod 755 /opt/microsoft/powershell/7/pwsh \ + && ln -s /opt/microsoft/powershell/7/pwsh /usr/local/bin/pwsh \ + && rm /tmp/powershell.tar.gz + +RUN curl -fsSLo /tmp/just.tar.gz \ + "https://github.com/casey/just/releases/download/${JUST_VERSION}/just-${JUST_VERSION}-x86_64-unknown-linux-musl.tar.gz" \ + && echo "${JUST_SHA256} /tmp/just.tar.gz" | sha256sum -c - \ + && tar -xzf /tmp/just.tar.gz -C /usr/local/bin just \ + && chmod 755 /usr/local/bin/just \ + && rm /tmp/just.tar.gz + +RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ + "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init" \ + && echo "${RUSTUP_SHA256} /tmp/rustup-init" | sha256sum -c - \ + && chmod 755 /tmp/rustup-init \ + && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ + && rm /tmp/rustup-init + +# Without this, the first `_install-tool` that needs binstall bootstraps it by +# compiling it from source, which costs minutes on every image build and is one +# more source build a toolchain bump can break. CI installs the prebuilt binary +# for the same reason. +RUN curl -fsSLo /tmp/cargo-binstall.tgz \ + "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ + && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ + && mkdir -p "${CARGO_HOME}/bin" \ + && tar -xzf /tmp/cargo-binstall.tgz -C "${CARGO_HOME}/bin" cargo-binstall \ + && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ + && rm /tmp/cargo-binstall.tgz + +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the toolchain is in place and before +# `anvil-setup` installs the catalog. Put system libraries a catalog tool +# needs in order to *compile* here: `binstall` falls back to a source build +# when a pinned tool publishes no prebuilt for this platform, and that build +# links against whatever headers the image has. +# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index c47adc25..0d58073d 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -129,20 +129,41 @@ anvil-container-tag: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + $inputs = @('rust-toolchain.toml') # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. - # Everything discovered under justfiles/anvil/ is treated as text only when + # Everything discovered by walking a directory is treated as text only when # it is a `.just` recipe; anything else is hashed as the bytes the build # context actually copies. $declaredText = [System.Collections.Generic.HashSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + [void]$declaredText.Add($dockerfile) + [void]$declaredText.Add("$dockerfile.dockerignore") [void]$declaredText.Add($hookRel) - if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { - # The hook decides what the build installs, so its content defines the - # image as surely as the Dockerfile does. Its *output* is deliberately - # never hashed: a credential must not influence a tag. - $inputs += $hookRel + # Everything under `.anvil/container/`, not a fixed list of three files. + # The Dockerfile is composed -- anvil owns regions inside it and the + # repository owns the gaps -- and a repository that adds a `COPY` in one of + # those gaps names a file that shapes the image: a corporate root CA, an + # install script, a patch. A replacement region from a downstream catalog + # does the same. Hashing only the three files anvil happens to know about + # would let any of them change the image under a reference that already + # resolves, which is precisely the hole this digest exists to close. + # + # The hook is picked up by the same walk. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $containerRoot = Join-Path $repoRoot '.anvil/container' + if (Test-Path -LiteralPath $containerRoot) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } + # The walk cannot assert this on its own: a missing Dockerfile would simply + # contribute nothing and yield a confident tag for an image that can never + # be built. It is the one file under that directory that must exist, so it + # is checked by name. + if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $dockerfile" + exit 1 } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 7dff0e48..e6394570 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -86,6 +86,14 @@ fn local() -> Cli { } } +/// A workspace that has already been generated once, so the lock and the +/// composed tree exist and a test can rewind one part of them. +fn generated_tree() -> TempDir { + let tmp = workspace(); + run_update(&Catalog::anvil(), &local(), tmp.path()).unwrap(); + tmp +} + /// Rewrite a freshly generated tree into the shape 0.4.0 produced: the retired /// assets present on disk and tracked in the lock, and none of this release's /// container artifacts present at all. @@ -98,9 +106,12 @@ fn rewind_to_runner_layout(root: &Path) -> Manifest { let mut manifest = Manifest::load(root).unwrap(); for artifact in artifacts::container::all() { + // The Dockerfile is composed now, so the group is a mix: the recipe and + // the ignore file are owned, the image definition is four regions in a + // host this rewind deletes outright. let path = match artifact { cargo_anvil::Artifact::OwnedFile(spec) => spec.path, - cargo_anvil::Artifact::Region(_) => panic!("container artifacts are owned files"), + cargo_anvil::Artifact::Region(_) => continue, }; let full = root.join(path); if full.exists() { @@ -109,6 +120,14 @@ fn rewind_to_runner_layout(root: &Path) -> Manifest { manifest.files.remove(path); } + let dockerfile = root.join(".anvil/container/Dockerfile"); + if dockerfile.exists() { + std::fs::remove_file(&dockerfile).unwrap(); + } + manifest + .regions + .retain(|key, _| !key.host.starts_with(".anvil/container/Dockerfile")); + for path in RETIRED_ASSETS.iter().copied().chain(std::iter::once(RETIRED_RECIPE)) { let body = format!("# 0.4.0 generated {path}\n"); write(&root.join(path), &body); @@ -171,14 +190,42 @@ fn upgrading_from_the_runner_layout_retires_the_seam_and_emits_the_new_backend() // This release's artifacts take their place and are tracked. for artifact in artifacts::container::all() { - let path = match artifact { - cargo_anvil::Artifact::OwnedFile(spec) => spec.path, - cargo_anvil::Artifact::Region(_) => panic!("container artifacts are owned files"), - }; - assert!(root.join(path).is_file(), "{path} must be written"); - assert!(manifest.files.contains_key(path), "{path} must be tracked"); + match artifact { + cargo_anvil::Artifact::OwnedFile(spec) => { + assert!(root.join(spec.path).is_file(), "{} must be written", spec.path); + assert!(manifest.files.contains_key(spec.path), "{} must be tracked", spec.path); + } + cargo_anvil::Artifact::Region(spec) => { + let cargo_anvil::HostSelector::Path(host) = &spec.host else { + panic!("container regions target a literal path"); + }; + let composed = std::fs::read_to_string(root.join(host)).unwrap(); + assert!( + composed.contains(&format!("# >>> anvil-managed: {}", spec.id)), + "{} must be spliced into {host}", + spec.id + ); + assert!( + manifest.regions.contains_key(&RegionKey { + host: host.clone(), + id: spec.id.as_str().to_owned(), + }), + "{} must be tracked", + spec.id + ); + } + } } + // The composed Dockerfile keeps its seeded parser directive on line 1: + // BuildKit honours it nowhere else, and a region sentinel above it would + // demote it silently. + let dockerfile = std::fs::read_to_string(root.join(".anvil/container/Dockerfile")).unwrap(); + assert!( + dockerfile.starts_with("# syntax=docker/dockerfile:1\n"), + "the composed Dockerfile must lead with the parser directive" + ); + // The runner region is spliced out of the root Justfile, and nothing else // is: the surrounding content the repository owns must survive intact. // Seeded in the rewind above, so this assertion can actually fail. @@ -238,3 +285,142 @@ fn an_edited_retired_asset_is_handed_back_rather_than_deleted() { "the lock entry must be dropped so the file becomes the repository's" ); } + +/// The release before this one owned `.anvil/container/Dockerfile` outright: a +/// single generated file a repository was invited to edit in place. This +/// release composes the same path from four managed regions. +/// +/// The upgrade must replace that file, not append to it. Appending would leave +/// the whole previous definition above the regions -- a second `FROM`, a second +/// set of pins, and an image built from whichever the frontend saw first. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn upgrading_from_the_owned_dockerfile_reseeds_the_composed_host() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // Rewind to the owned-file shape: one generated file, tracked in the lock + // by the checksum of exactly what is on disk, and no regions recorded. + let previous_render = + "# syntax=docker/dockerfile:1\n# Managed by cargo-anvil.\nFROM docker.io/library/ubuntu:24.04\nRUN apt-get update\n"; + write(&dockerfile, previous_render); + let mut manifest = Manifest::load(root).unwrap(); + manifest + .files + .insert(".anvil/container/Dockerfile".to_owned(), checksum_str(previous_render)); + manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); + manifest.save(root).unwrap(); + + run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let composed = std::fs::read_to_string(&dockerfile).unwrap(); + assert_eq!( + composed.matches("\nFROM ").count() + usize::from(composed.starts_with("FROM ")), + 1, + "the superseded definition must be replaced, not appended to:\n{composed}" + ); + assert!( + !composed.contains("RUN apt-get update\n# >>>"), + "no line of the previous render may survive above the regions" + ); + assert!(composed.starts_with("# syntax=docker/dockerfile:1\n")); + for id in [ + "anvil-container-base", + "anvil-container-tools", + "anvil-container-setup", + "anvil-container-entry", + ] { + assert!(composed.contains(&format!("# >>> anvil-managed: {id}")), "{id} must be spliced in"); + } + + // The stale owned-file entry is gone; the regions are tracked in its place. + let after = Manifest::load(root).unwrap(); + assert!( + !after.files.contains_key(".anvil/container/Dockerfile"), + "the superseded owned-file entry must be dropped from the lock" + ); + assert!( + after.regions.contains_key(&RegionKey { + host: ".anvil/container/Dockerfile".to_owned(), + id: "anvil-container-base".to_owned(), + }), + "the composed regions must be tracked instead" + ); +} + +/// A Dockerfile the repository wrote itself -- never tracked as an owned file -- +/// is not a superseded render, so it must survive with the regions added to it +/// rather than being replaced. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_repository_authored_dockerfile_is_not_reseeded() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + let hand_written = "# hand written by the repository\nRUN echo mine\n"; + write(&dockerfile, hand_written); + let mut manifest = Manifest::load(root).unwrap(); + manifest.files.remove(".anvil/container/Dockerfile"); + manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); + manifest.save(root).unwrap(); + + run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let composed = std::fs::read_to_string(&dockerfile).unwrap(); + assert!( + composed.contains("# hand written by the repository"), + "content anvil never owned must be preserved:\n{composed}" + ); + assert!(composed.contains("# >>> anvil-managed: anvil-container-base")); +} + +/// The Dockerfile is the first managed-region host whose region *order* is +/// semantic: `FROM` must precede everything, and the toolchain must exist +/// before `anvil-setup` runs. `upsert_region` replaces a region wherever it +/// finds it, so a reordered file would otherwise be updated in place into a +/// Dockerfile that is silently wrong or cannot build at all. +/// +/// Anvil must refuse the host and say which region moved, exactly once, rather +/// than once per region. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_reordered_composed_dockerfile_is_refused_with_one_diagnostic() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // Move the base region below the others, which is exactly the mistake a + // repository makes by dropping its own instructions above `FROM`. + let text = std::fs::read_to_string(&dockerfile).unwrap(); + let open = text.find("# >>> anvil-managed: anvil-container-base").unwrap(); + let close_marker = "# <<< anvil-managed: anvil-container-base\n"; + let close = text.find(close_marker).unwrap() + close_marker.len(); + let base = &text[open..close]; + let reordered = format!("{}{}\n{}", &text[..open], &text[close..], base); + write(&dockerfile, &reordered); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host, not one per region: {refusals:?}"); + assert!( + refusals[0].contains("'anvil-container-tools' appears before 'anvil-container-base'"), + "the diagnostic must name the region that moved: {}", + refusals[0] + ); + + // Refusing means leaving the file exactly as it was found. Rewriting it + // would destroy whatever the repository put between the regions. + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + reordered, + "a refused host must not be modified" + ); +} diff --git a/crates/cargo-anvil/tests/extensibility.rs b/crates/cargo-anvil/tests/extensibility.rs index cce55a9c..fd230206 100644 --- a/crates/cargo-anvil/tests/extensibility.rs +++ b/crates/cargo-anvil/tests/extensibility.rs @@ -55,7 +55,7 @@ fn containerforge() -> Catalog { .subcommand("containerforge") .about("ContainerForge: an anvil container catalog for tests") .version("9.9.9") - .replace_artifact(artifacts::container::dockerfile().with_body("FROM example.invalid/base\n")) + .replace_artifact(artifacts::container::dockerfile_base().with_body("FROM example.invalid/base\n")) .with_artifact(artifacts::container::hooks("# test credential hook\n")) .build() .unwrap() @@ -173,9 +173,17 @@ fn public_container_artifacts_can_be_specialized_by_downstream_catalogs() { "downstream catalog must inherit the public container command unchanged" ); let configured_dockerfile = std::fs::read_to_string(configured.path().join(DOCKERFILE)).unwrap(); - assert_eq!( - configured_dockerfile, "FROM example.invalid/base\n", - "downstream catalog must replace the public Dockerfile" + assert!( + configured_dockerfile.contains("# >>> anvil-managed: anvil-container-base\nFROM example.invalid/base\n"), + "downstream catalog must replace the public Dockerfile base region" + ); + assert!( + configured_dockerfile.contains("just anvil-setup binstall"), + "downstream catalog must inherit the catalog-install region it did not replace" + ); + assert!( + configured_dockerfile.starts_with("# syntax=docker/dockerfile:1\n"), + "the seeded parser directive must lead the composed Dockerfile" ); let configured_ignore = std::fs::read_to_string(configured.path().join(DOCKERIGNORE)).unwrap(); assert_eq!( diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 26e7a405..eceaecf5 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -866,6 +866,14 @@ fn mutants_diff_covers_uncommitted_work() { ("FAKE_CARGO_LOG", log.as_os_str()), ("BASE_REF", OsStr::new(&base)), ("RUNNER_TEMP", root.as_os_str()), + // Pin the architecture the recipe branches on. It bails out early on + // aarch64-pc-windows-msvc, where cargo-mutants does not build, so on + // an ARM64 Windows runner this test would otherwise assert against a + // recipe that deliberately did nothing. Both variables are set + // because a 32-bit host process reports the real machine in the + // second one. + ("PROCESSOR_ARCHITECTURE", OsStr::new("AMD64")), + ("PROCESSOR_ARCHITEW6432", OsStr::new("")), ], ); assert!( @@ -885,3 +893,54 @@ fn mutants_diff_covers_uncommitted_work() { "the uncommitted change must be in the diff -- a base..HEAD diff would omit it:\n{diff}" ); } + +/// The ARM64 Windows bail-out is a documented behaviour, not an accident: +/// cargo-mutants does not build for `aarch64-pc-windows-msvc`, so the recipe +/// exits cleanly rather than failing the merged `pr-slow` group on that leg. +/// +/// Asserting it here is what keeps the sibling test above honest. That one pins +/// the architecture to AMD64 so it exercises the real path; without this test +/// the skip branch would be exercised by nothing, and an ARM64 runner would be +/// the only place either behaviour was observed. +#[test] +fn mutants_diff_skips_on_arm64_windows() { + // Only meaningful where the recipe's `$IsWindows` guard can be true; on + // Linux and macOS the architecture variable is not consulted at all. + if !cfg!(windows) { + return; + } + let tmp = fixture( + &[("helpers.just", HELPERS), ("mutants-diff.just", MUTANTS_DIFF)], + &[ + "anvil-tool-cargo-mutants-validate-prereqs", + "anvil-tool-cargo-mutants-install installer=\"install\"", + ], + ); + let root = tmp.path(); + + let log = root.join("cargo.log"); + let output = run_just( + root, + &["anvil-mutants-diff"], + &[ + ("FAKE_CARGO_LOG", log.as_os_str()), + ("RUNNER_TEMP", root.as_os_str()), + ("PROCESSOR_ARCHITECTURE", OsStr::new("ARM64")), + ], + ); + + assert!( + output.status.success(), + "the recipe must skip cleanly, not fail, on aarch64-pc-windows-msvc\nstderr:\n{}", + String::from_utf8_lossy(&output.stderr) + ); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + stdout.contains("cargo-mutants does not build here"), + "the skip must say why, or a silent no-op looks like a passing run:\n{stdout}" + ); + assert!( + std::fs::read_to_string(&log).unwrap_or_default().is_empty(), + "cargo must not be invoked at all on the skipped leg" + ); +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 1457297a..871f4939 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -6,32 +6,43 @@ expression: render_tree(tmp.path()) # syntax=docker/dockerfile:1 # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. # -# Default Anvil execution image, emitted for every repository. The image -# installs exactly the tools the generated catalog pins, by running -# `just anvil-setup` -- the same recipe the checks themselves use. That is what -# makes "the image has the right tools" true by construction rather than by -# convention: there is no second list to keep in step. +# Anvil execution image. # -# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this -# file's text, so an edit here renames the image -- but it does not resolve or -# validate the base, and a floating tag can therefore change underneath a tag -# that claims to name fixed content. +# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them +# current, so base and tool-pin bumps arrive on their own. Everything outside +# them is yours and is preserved byte-for-byte, including this header. # +# Add your own instructions in the gaps between the regions rather than inside +# one. Anvil will not overwrite an edit you make inside a region -- it offers +# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and +# is the trap: the region carries the base digest and four tool pins, so an edit +# there quietly freezes them at today's values while the image tag keeps +# resolving, because the tag hashes your file. The gaps exist so that nothing +# you legitimately need to add ever requires touching them. +# +# Each region ends by describing what the gap after it is for, because position +# decides whether a line works at all: a corporate root CA has to land before +# the first download, and a library needed to *compile* a catalog tool has to +# land before `anvil-setup` runs. +# +# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit +# honours the directive only when nothing precedes it -- not even a comment. +# That is also why it sits out here rather than inside a region: a region's +# opening sentinel is a comment, and would silently demote the directive to an +# ordinary one, leaving the build on the default frontend with nothing failing +# to say so. + +# >>> anvil-managed: anvil-container-base # The base tracks the Linux runner the generated workflows use # (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the # catalog as prebuilt binaries, which require that runner's glibc; it is # backward but not forward compatible, so the pin moves when the runner does. # -# To build on a different base (a lower glibc baseline, or an internal -# distribution), a downstream catalog replaces this artifact wholesale via -# `replace_artifact(artifacts::container::dockerfile(...))`; a single -# repository can edit this file in place, which anvil's drift handling -# preserves. A lower baseline means the catalog must also install from source -# rather than with `binstall`. - +# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a +# change here renames the image -- but it does not resolve or validate the +# base, and a floating tag could therefore change underneath a reference that +# claims to name fixed content. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} @@ -50,9 +61,19 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# --- your turn ------------------------------------------------------------- +# The gap below this region runs before anvil's first download. Put whatever +# the image needs in order to reach the network here: a corporate root CA, +# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. +# Without it, a TLS-intercepting proxy makes every `curl` in the next region +# fail and the image cannot be built at all. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-base + +# >>> anvil-managed: anvil-container-tools # clang/libclang are required by cargo-spellcheck; the rest is the usual Rust -# link-time set. A bare slim base has no C runtime development files, so every -# link step fails without build-essential. +# link-time set. A bare base has no C runtime development files, so every link +# step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ @@ -96,6 +117,16 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the toolchain is in place and before +# `anvil-setup` installs the catalog. Put system libraries a catalog tool +# needs in order to *compile* here: `binstall` falls back to a source build +# when a pinned tool publishes no prebuilt for this platform, and that build +# links against whatever headers the image has. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-tools + +# >>> anvil-managed: anvil-container-setup # Install the pinned toolchain and cargo subcommands from the generated # catalog. The whole recipe tree is copied because `just` has to parse it, and # the whole tree is hashed into the tag: `anvil-setup` reaches the install @@ -129,18 +160,29 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the catalog is installed. Put what your +# own checks need at run time here: a database client, a protobuf compiler, a +# linter that is not a cargo subcommand. It is also the cheapest place to add +# anything, because an edit here invalidates one layer rather than the +# multi-minute `anvil-setup` above. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-setup + +# >>> anvil-managed: anvil-container-entry # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace CMD ["bash"] +# <<< anvil-managed: anvil-container-entry === .anvil/container/Dockerfile.dockerignore === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. +# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Update the corresponding template in the cargo-anvil crate. # # BuildKit reads `.dockerignore` in preference to a root # `.dockerignore`, so this scopes the exec-image build context without the @@ -3665,20 +3707,41 @@ anvil-container-tag: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + $inputs = @('rust-toolchain.toml') # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. - # Everything discovered under justfiles/anvil/ is treated as text only when + # Everything discovered by walking a directory is treated as text only when # it is a `.just` recipe; anything else is hashed as the bytes the build # context actually copies. $declaredText = [System.Collections.Generic.HashSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + [void]$declaredText.Add($dockerfile) + [void]$declaredText.Add("$dockerfile.dockerignore") [void]$declaredText.Add($hookRel) - if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { - # The hook decides what the build installs, so its content defines the - # image as surely as the Dockerfile does. Its *output* is deliberately - # never hashed: a credential must not influence a tag. - $inputs += $hookRel + # Everything under `.anvil/container/`, not a fixed list of three files. + # The Dockerfile is composed -- anvil owns regions inside it and the + # repository owns the gaps -- and a repository that adds a `COPY` in one of + # those gaps names a file that shapes the image: a corporate root CA, an + # install script, a patch. A replacement region from a downstream catalog + # does the same. Hashing only the three files anvil happens to know about + # would let any of them change the image under a reference that already + # resolves, which is precisely the hole this digest exists to close. + # + # The hook is picked up by the same walk. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $containerRoot = Join-Path $repoRoot '.anvil/container' + if (Test-Path -LiteralPath $containerRoot) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } + # The walk cannot assert this on its own: a missing Dockerfile would simply + # contribute nothing and yield a confident tag for an image that can never + # be built. It is the one file under that directory that must exist, so it + # is checked by name. + if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $dockerfile" + exit 1 } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 38e65ba6..a855f45e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -6,32 +6,43 @@ expression: render_tree(tmp.path()) # syntax=docker/dockerfile:1 # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. # -# Default Anvil execution image, emitted for every repository. The image -# installs exactly the tools the generated catalog pins, by running -# `just anvil-setup` -- the same recipe the checks themselves use. That is what -# makes "the image has the right tools" true by construction rather than by -# convention: there is no second list to keep in step. +# Anvil execution image. # -# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this -# file's text, so an edit here renames the image -- but it does not resolve or -# validate the base, and a floating tag can therefore change underneath a tag -# that claims to name fixed content. +# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them +# current, so base and tool-pin bumps arrive on their own. Everything outside +# them is yours and is preserved byte-for-byte, including this header. # +# Add your own instructions in the gaps between the regions rather than inside +# one. Anvil will not overwrite an edit you make inside a region -- it offers +# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and +# is the trap: the region carries the base digest and four tool pins, so an edit +# there quietly freezes them at today's values while the image tag keeps +# resolving, because the tag hashes your file. The gaps exist so that nothing +# you legitimately need to add ever requires touching them. +# +# Each region ends by describing what the gap after it is for, because position +# decides whether a line works at all: a corporate root CA has to land before +# the first download, and a library needed to *compile* a catalog tool has to +# land before `anvil-setup` runs. +# +# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit +# honours the directive only when nothing precedes it -- not even a comment. +# That is also why it sits out here rather than inside a region: a region's +# opening sentinel is a comment, and would silently demote the directive to an +# ordinary one, leaving the build on the default frontend with nothing failing +# to say so. + +# >>> anvil-managed: anvil-container-base # The base tracks the Linux runner the generated workflows use # (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the # catalog as prebuilt binaries, which require that runner's glibc; it is # backward but not forward compatible, so the pin moves when the runner does. # -# To build on a different base (a lower glibc baseline, or an internal -# distribution), a downstream catalog replaces this artifact wholesale via -# `replace_artifact(artifacts::container::dockerfile(...))`; a single -# repository can edit this file in place, which anvil's drift handling -# preserves. A lower baseline means the catalog must also install from source -# rather than with `binstall`. - +# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a +# change here renames the image -- but it does not resolve or validate the +# base, and a floating tag could therefore change underneath a reference that +# claims to name fixed content. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} @@ -50,9 +61,19 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# --- your turn ------------------------------------------------------------- +# The gap below this region runs before anvil's first download. Put whatever +# the image needs in order to reach the network here: a corporate root CA, +# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. +# Without it, a TLS-intercepting proxy makes every `curl` in the next region +# fail and the image cannot be built at all. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-base + +# >>> anvil-managed: anvil-container-tools # clang/libclang are required by cargo-spellcheck; the rest is the usual Rust -# link-time set. A bare slim base has no C runtime development files, so every -# link step fails without build-essential. +# link-time set. A bare base has no C runtime development files, so every link +# step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ @@ -96,6 +117,16 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the toolchain is in place and before +# `anvil-setup` installs the catalog. Put system libraries a catalog tool +# needs in order to *compile* here: `binstall` falls back to a source build +# when a pinned tool publishes no prebuilt for this platform, and that build +# links against whatever headers the image has. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-tools + +# >>> anvil-managed: anvil-container-setup # Install the pinned toolchain and cargo subcommands from the generated # catalog. The whole recipe tree is copied because `just` has to parse it, and # the whole tree is hashed into the tag: `anvil-setup` reaches the install @@ -129,18 +160,29 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the catalog is installed. Put what your +# own checks need at run time here: a database client, a protobuf compiler, a +# linter that is not a cargo subcommand. It is also the cheapest place to add +# anything, because an edit here invalidates one layer rather than the +# multi-minute `anvil-setup` above. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-setup + +# >>> anvil-managed: anvil-container-entry # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace CMD ["bash"] +# <<< anvil-managed: anvil-container-entry === .anvil/container/Dockerfile.dockerignore === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. +# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Update the corresponding template in the cargo-anvil crate. # # BuildKit reads `.dockerignore` in preference to a root # `.dockerignore`, so this scopes the exec-image build context without the @@ -3564,20 +3606,41 @@ anvil-container-tag: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + $inputs = @('rust-toolchain.toml') # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. - # Everything discovered under justfiles/anvil/ is treated as text only when + # Everything discovered by walking a directory is treated as text only when # it is a `.just` recipe; anything else is hashed as the bytes the build # context actually copies. $declaredText = [System.Collections.Generic.HashSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + [void]$declaredText.Add($dockerfile) + [void]$declaredText.Add("$dockerfile.dockerignore") [void]$declaredText.Add($hookRel) - if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { - # The hook decides what the build installs, so its content defines the - # image as surely as the Dockerfile does. Its *output* is deliberately - # never hashed: a credential must not influence a tag. - $inputs += $hookRel + # Everything under `.anvil/container/`, not a fixed list of three files. + # The Dockerfile is composed -- anvil owns regions inside it and the + # repository owns the gaps -- and a repository that adds a `COPY` in one of + # those gaps names a file that shapes the image: a corporate root CA, an + # install script, a patch. A replacement region from a downstream catalog + # does the same. Hashing only the three files anvil happens to know about + # would let any of them change the image under a reference that already + # resolves, which is precisely the hole this digest exists to close. + # + # The hook is picked up by the same walk. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $containerRoot = Join-Path $repoRoot '.anvil/container' + if (Test-Path -LiteralPath $containerRoot) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } + # The walk cannot assert this on its own: a missing Dockerfile would simply + # contribute nothing and yield a confident tag for an image that can never + # be built. It is the one file under that directory that must exist, so it + # is checked by name. + if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $dockerfile" + exit 1 } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 1d6b8b70..45e5ee10 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -6,32 +6,43 @@ expression: render_tree(tmp.path()) # syntax=docker/dockerfile:1 # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. # -# Default Anvil execution image, emitted for every repository. The image -# installs exactly the tools the generated catalog pins, by running -# `just anvil-setup` -- the same recipe the checks themselves use. That is what -# makes "the image has the right tools" true by construction rather than by -# convention: there is no second list to keep in step. +# Anvil execution image. # -# BASE_IMAGE should stay digest-pinned. `_anvil-container-image` hashes this -# file's text, so an edit here renames the image -- but it does not resolve or -# validate the base, and a floating tag can therefore change underneath a tag -# that claims to name fixed content. +# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them +# current, so base and tool-pin bumps arrive on their own. Everything outside +# them is yours and is preserved byte-for-byte, including this header. # +# Add your own instructions in the gaps between the regions rather than inside +# one. Anvil will not overwrite an edit you make inside a region -- it offers +# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and +# is the trap: the region carries the base digest and four tool pins, so an edit +# there quietly freezes them at today's values while the image tag keeps +# resolving, because the tag hashes your file. The gaps exist so that nothing +# you legitimately need to add ever requires touching them. +# +# Each region ends by describing what the gap after it is for, because position +# decides whether a line works at all: a corporate root CA has to land before +# the first download, and a library needed to *compile* a catalog tool has to +# land before `anvil-setup` runs. +# +# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit +# honours the directive only when nothing precedes it -- not even a comment. +# That is also why it sits out here rather than inside a region: a region's +# opening sentinel is a comment, and would silently demote the directive to an +# ordinary one, leaving the build on the default frontend with nothing failing +# to say so. + +# >>> anvil-managed: anvil-container-base # The base tracks the Linux runner the generated workflows use # (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the # catalog as prebuilt binaries, which require that runner's glibc; it is # backward but not forward compatible, so the pin moves when the runner does. # -# To build on a different base (a lower glibc baseline, or an internal -# distribution), a downstream catalog replaces this artifact wholesale via -# `replace_artifact(artifacts::container::dockerfile(...))`; a single -# repository can edit this file in place, which anvil's drift handling -# preserves. A lower baseline means the catalog must also install from source -# rather than with `binstall`. - +# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a +# change here renames the image -- but it does not resolve or validate the +# base, and a floating tag could therefore change underneath a reference that +# claims to name fixed content. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} @@ -50,9 +61,19 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +# --- your turn ------------------------------------------------------------- +# The gap below this region runs before anvil's first download. Put whatever +# the image needs in order to reach the network here: a corporate root CA, +# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. +# Without it, a TLS-intercepting proxy makes every `curl` in the next region +# fail and the image cannot be built at all. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-base + +# >>> anvil-managed: anvil-container-tools # clang/libclang are required by cargo-spellcheck; the rest is the usual Rust -# link-time set. A bare slim base has no C runtime development files, so every -# link step fails without build-essential. +# link-time set. A bare base has no C runtime development files, so every link +# step fails without build-essential. RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential ca-certificates clang libclang-dev curl git libicu-dev \ @@ -96,6 +117,16 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the toolchain is in place and before +# `anvil-setup` installs the catalog. Put system libraries a catalog tool +# needs in order to *compile* here: `binstall` falls back to a source build +# when a pinned tool publishes no prebuilt for this platform, and that build +# links against whatever headers the image has. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-tools + +# >>> anvil-managed: anvil-container-setup # Install the pinned toolchain and cargo subcommands from the generated # catalog. The whole recipe tree is copied because `just` has to parse it, and # the whole tree is hashed into the tag: `anvil-setup` reaches the install @@ -129,18 +160,29 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" +# --- your turn ------------------------------------------------------------- +# The gap below this region runs after the catalog is installed. Put what your +# own checks need at run time here: a database client, a protobuf compiler, a +# linter that is not a cargo subcommand. It is also the cheapest place to add +# anything, because an edit here invalidates one layer rather than the +# multi-minute `anvil-setup` above. +# --------------------------------------------------------------------------- +# <<< anvil-managed: anvil-container-setup + +# >>> anvil-managed: anvil-container-entry # Consumed by `anvil-container` itself: a nested invocation from inside the # image runs the recipe natively instead of launching another container. ENV ANVIL_IN_CONTAINER=1 WORKDIR /workspace CMD ["bash"] +# <<< anvil-managed: anvil-container-entry === .anvil/container/Dockerfile.dockerignore === # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# Managed by cargo-anvil. Update the corresponding template in the cargo-anvil -# crate unless the change is repository-specific. +# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. +# Update the corresponding template in the cargo-anvil crate. # # BuildKit reads `.dockerignore` in preference to a root # `.dockerignore`, so this scopes the exec-image build context without the @@ -2405,20 +2447,41 @@ anvil-container-tag: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + $inputs = @('rust-toolchain.toml') # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. - # Everything discovered under justfiles/anvil/ is treated as text only when + # Everything discovered by walking a directory is treated as text only when # it is a `.just` recipe; anything else is hashed as the bytes the build # context actually copies. $declaredText = [System.Collections.Generic.HashSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + [void]$declaredText.Add($dockerfile) + [void]$declaredText.Add("$dockerfile.dockerignore") [void]$declaredText.Add($hookRel) - if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { - # The hook decides what the build installs, so its content defines the - # image as surely as the Dockerfile does. Its *output* is deliberately - # never hashed: a credential must not influence a tag. - $inputs += $hookRel + # Everything under `.anvil/container/`, not a fixed list of three files. + # The Dockerfile is composed -- anvil owns regions inside it and the + # repository owns the gaps -- and a repository that adds a `COPY` in one of + # those gaps names a file that shapes the image: a corporate root CA, an + # install script, a patch. A replacement region from a downstream catalog + # does the same. Hashing only the three files anvil happens to know about + # would let any of them change the image under a reference that already + # resolves, which is precisely the hole this digest exists to close. + # + # The hook is picked up by the same walk. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $containerRoot = Join-Path $repoRoot '.anvil/container' + if (Test-Path -LiteralPath $containerRoot) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } + # The walk cannot assert this on its own: a missing Dockerfile would simply + # contribute nothing and yield a confident tag for an image that can never + # be built. It is the one file under that directory that must exist, so it + # is checked by name. + if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $dockerfile" + exit 1 } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index c47adc25..0d58073d 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -129,20 +129,41 @@ anvil-container-tag: $dockerfile = '.anvil/container/Dockerfile' $hookRel = '.anvil/container/hooks.ps1' - $inputs = @($dockerfile, "$dockerfile.dockerignore", 'rust-toolchain.toml') + $inputs = @('rust-toolchain.toml') # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. - # Everything discovered under justfiles/anvil/ is treated as text only when + # Everything discovered by walking a directory is treated as text only when # it is a `.just` recipe; anything else is hashed as the bytes the build # context actually copies. $declaredText = [System.Collections.Generic.HashSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + [void]$declaredText.Add($dockerfile) + [void]$declaredText.Add("$dockerfile.dockerignore") [void]$declaredText.Add($hookRel) - if (Test-Path -LiteralPath (Join-Path $repoRoot $hookRel) -PathType Leaf) { - # The hook decides what the build installs, so its content defines the - # image as surely as the Dockerfile does. Its *output* is deliberately - # never hashed: a credential must not influence a tag. - $inputs += $hookRel + # Everything under `.anvil/container/`, not a fixed list of three files. + # The Dockerfile is composed -- anvil owns regions inside it and the + # repository owns the gaps -- and a repository that adds a `COPY` in one of + # those gaps names a file that shapes the image: a corporate root CA, an + # install script, a patch. A replacement region from a downstream catalog + # does the same. Hashing only the three files anvil happens to know about + # would let any of them change the image under a reference that already + # resolves, which is precisely the hole this digest exists to close. + # + # The hook is picked up by the same walk. Its *output* is deliberately + # never hashed: a credential must not influence a tag. + $containerRoot = Join-Path $repoRoot '.anvil/container' + if (Test-Path -LiteralPath $containerRoot) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } + } + # The walk cannot assert this on its own: a missing Dockerfile would simply + # contribute nothing and yield a confident tag for an image that can never + # be built. It is the one file under that directory that must exist, so it + # is checked by name. + if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { + Write-Error "anvil: container image input is missing: $dockerfile" + exit 1 } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index eb318d5b..0e487818 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -549,21 +549,46 @@ $reverted = Get-ImageReference -Repo $repo Assert-Equal 'the original reference is restored' $reference $reverted Assert-That 'the original image is still present' (Test-ImagePresent $reverted) -# ------------------------------------------------- 6. editing the Dockerfile -- +# ---------------------------------------------- 6. composing the Dockerfile -- -Write-Section '6. A repository can edit the Dockerfile' +Write-Section '6. A repository composes the Dockerfile around anvil''s regions' $dockerfileBody = Get-Content -LiteralPath $dockerfile -Raw -Write-Fixture $dockerfile ($dockerfileBody + "`n# a repository-owned edit`n") + +# The gap between the base and tool regions: where a root CA or a proxy goes, +# and the reason the file is composed rather than owned outright. +$gapMarker = "# <<< anvil-managed: anvil-container-base`n" +Assert-That 'the composed Dockerfile carries the base region' ($dockerfileBody.Contains($gapMarker)) +$composed = $dockerfileBody.Replace($gapMarker, $gapMarker + "`n# a repository-owned edit`nENV ANVIL_E2E_GAP=1`n") +Write-Fixture $dockerfile $composed $editedReference = Get-ImageReference -Repo $repo -Assert-That 'editing the Dockerfile selects a new tag' ($editedReference -ne $reference) +Assert-That 'adding to a gap selects a new tag' ($editedReference -ne $reference) -Write-Step 're-running the generator over the edited file' +Write-Step 're-running the generator over the composed file' $regen = Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo -AllowFailure -Assert-Equal 'the generator succeeds over a user-modified owned file' 0 $regen.ExitCode +Assert-Equal 'the generator succeeds over a composed file' 0 $regen.ExitCode $afterRegen = Get-Content -LiteralPath $dockerfile -Raw -Assert-That 'the edit survives regeneration' ($afterRegen -match 'a repository-owned edit') ` - 'anvil must preserve a user-modified owned file' +Assert-That 'content in a gap survives regeneration' ($afterRegen -match 'a repository-owned edit') ` + 'anvil must preserve everything outside its own sentinels' +Assert-Equal 'the composed tag is unchanged by regeneration' $editedReference (Get-ImageReference -Repo $repo) + +# Anvil never overwrites repository content, and a region body is no exception: +# an edit inside one is preserved, exactly as `updates.md` §2 preserves an +# edited owned file. That is why the gaps matter -- editing inside a region +# silently freezes the base digest and the tool pins at today's values while +# the tag keeps resolving, so the layout has to make the gaps the obvious place +# to add things rather than relying on the engine to police it. +Write-Step 'editing inside a region, which anvil preserves rather than overwrites' +$frozen = $afterRegen.Replace('ARG JUST_VERSION=', 'ARG JUST_VERSION=0.0.0 # ') +Assert-That 'the edit landed inside the region' ($frozen -ne $afterRegen) +Write-Fixture $dockerfile $frozen +$regen = Invoke-Native -Command $anvilExe -Arguments @('anvil', '--no-backends') -WorkingDirectory $repo -AllowFailure +Assert-Equal 'the generator succeeds over an edited region' 0 $regen.ExitCode +$reclaimed = Get-Content -LiteralPath $dockerfile -Raw +Assert-That 'an edit inside a region is preserved, not overwritten' ($reclaimed -match 'JUST_VERSION=0\.0\.0') ` + 'anvil must never destroy repository content, in a region or a file' +Assert-That 'the surrounding gap content is untouched' ($reclaimed -match 'a repository-owned edit') +Assert-That 'the sentinels survive the edit' ($reclaimed -match '# <<< anvil-managed: anvil-container-base') Write-Fixture $dockerfile $dockerfileBody Assert-Equal 'restoring the Dockerfile restores the tag' $reference (Get-ImageReference -Repo $repo) From 52d0ea7b922e941eff2be56039359b28b267edf0 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 00:09:53 +0200 Subject: [PATCH 45/81] fix(anvil): use American spelling and drop bare section refs from new rustdoc The spellcheck gate reads doc comments as prose. Existing section citations survive it because they sit inside a code span or a markdown link, so the section symbol never reaches the checker; the two I added were bare prose and were flagged. AGENTS.md asks new code not to cite design sections at all, so they are removed rather than backticked. The British spellings are the same class of mistake: the dictionary and every existing doc comment use American forms. Validation: cargo test -p cargo-anvil -- 378 passed, 0 failed. --- crates/cargo-anvil/src/anvil/artifacts/container.rs | 12 ++++++------ crates/cargo-anvil/src/run.rs | 4 ++-- crates/cargo-anvil/tests/container_upgrade.rs | 2 +- crates/cargo-anvil/tests/recipe_contracts.rs | 4 ++-- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 120b465b..aba84047 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -14,15 +14,15 @@ //! //! The Dockerfile is a **user-composed file with managed regions**, not a //! wholly-owned file. An owned file that invites in-place edits fails silently -//! here: `updates.md` §2 preserves the edit and writes anvil's version to +//! here: anvil preserves the edit and writes its own version to //! `.anvil-proposed`, with no three-way merge and no recorded ancestor, so a //! repository that edits it once keeps building on the base digest and the four //! tool pins frozen at that moment — while `anvil-container-tag` keeps //! resolving, because the tag hashes *their* file. The identity scheme works //! perfectly and still names a stale image. //! -//! Splitting anvil's content into regions does not make it unwritable — §2's -//! ownership rules apply to a region body as they do to a file, and anvil never +//! Splitting anvil's content into regions does not make it read-only: the same +//! ownership rules apply to a region body as to a file, and anvil never //! overwrites repository content. What it removes is the *reason* to edit: each //! of the three gaps between the regions is the correct home for one class of //! addition, defined by what must already be true at that point in the build. @@ -55,7 +55,7 @@ const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dock /// Seeded into the Dockerfile when the file does not exist, and never /// reconciled afterwards — it is the user's half of a composed file. /// -/// It carries `# syntax=docker/dockerfile:1`, which BuildKit honours only as +/// It carries `# syntax=docker/dockerfile:1`, which BuildKit honors only as /// the very first line of the file. A region's opening sentinel is a comment, /// so the directive cannot live inside a region without being demoted to an /// ordinary comment — silently, with the build falling back to the default @@ -160,7 +160,7 @@ pub fn dockerfile_setup() -> Artifact { dockerfile_region("anvil-container-setup", DOCKERFILE_SETUP) } -/// The entry contract: `ANVIL_IN_CONTAINER`, the workdir the repository is +/// The entry contract: `ANVIL_IN_CONTAINER`, the mount point the repository is /// bind-mounted at, and the default command. /// /// Every line here is a contract with the recipe rather than an opinion about @@ -280,7 +280,7 @@ mod tests { #[test] fn the_syntax_directive_leads_the_seeded_header() { - // BuildKit honours the parser directive only when nothing precedes it. + // BuildKit honors the parser directive only when nothing precedes it. // If this ever moves, the frontend pin stops applying and nothing // fails to say so. assert_eq!( diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index c461d254..7dd54eac 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -363,7 +363,7 @@ fn push_region_at( // A file left behind by a release that owned this path outright is re-seeded // too. Its content is a previous render, not user composition, so appending // the regions to it would produce a file carrying both the old whole-file - // definition and the new regions. Recognising it needs both conditions: + // definition and the new regions. Recognizing it needs both conditions: // tracked as an owned file in the lock we are superseding, and carrying none // of this host's regions yet. A Dockerfile the repository wrote by hand is // in neither state and is left alone. @@ -462,7 +462,7 @@ fn region_placement(region_id: &str) -> RegionPlacement { /// Most region hosts are files the repository already has (`Cargo.toml`, /// `deny.toml`) or files whose first region can simply be appended to nothing. /// The container Dockerfile is neither: `# syntax=docker/dockerfile:1` is a -/// BuildKit parser directive that is honoured only when nothing precedes it, +/// BuildKit parser directive that is honored only when nothing precedes it, /// not even a comment — so it cannot live inside a region, whose opening /// sentinel *is* a comment. fn host_scaffold(host_relpath: &str) -> Option<&'static str> { diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index e6394570..28775323 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -218,7 +218,7 @@ fn upgrading_from_the_runner_layout_retires_the_seam_and_emits_the_new_backend() } // The composed Dockerfile keeps its seeded parser directive on line 1: - // BuildKit honours it nowhere else, and a region sentinel above it would + // BuildKit honors it nowhere else, and a region sentinel above it would // demote it silently. let dockerfile = std::fs::read_to_string(root.join(".anvil/container/Dockerfile")).unwrap(); assert!( diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index eceaecf5..8635a94e 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -894,14 +894,14 @@ fn mutants_diff_covers_uncommitted_work() { ); } -/// The ARM64 Windows bail-out is a documented behaviour, not an accident: +/// The ARM64 Windows bail-out is a documented behavior, not an accident: /// cargo-mutants does not build for `aarch64-pc-windows-msvc`, so the recipe /// exits cleanly rather than failing the merged `pr-slow` group on that leg. /// /// Asserting it here is what keeps the sibling test above honest. That one pins /// the architecture to AMD64 so it exercises the real path; without this test /// the skip branch would be exercised by nothing, and an ARM64 runner would be -/// the only place either behaviour was observed. +/// the only place either behavior was observed. #[test] fn mutants_diff_skips_on_arm64_windows() { // Only meaningful where the recipe's `$IsWindows` guard can be true; on From eaaf1881bd55a5134398df561f12fad102b0b566 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 00:53:47 +0200 Subject: [PATCH 46/81] fix(anvil): satisfy the clippy, readme and mutation gates Three CI gates on the region work, each a real defect rather than noise: - `doc_markdown` wants `BuildKit` in backticks, as every other doc comment in the crate already writes it. - The crate README is generated from the crate-level docs, which this branch rewrote, so it had to be regenerated. - Mutation testing found `<` in the region-order check surviving a change to `<=`. It was an equivalent mutant -- two distinct regions can never share a start offset, so no test could tell the two operators apart. Rather than skip it, the check now compares the declared sequence against the sequence the file actually carries, which has no ordering operator to mutate and says what it means: the on-disk order must equal the declared order. Validation: cargo clippy -D warnings, cargo test -p cargo-anvil (378 passed), just readme -- all clean. --- crates/cargo-anvil/README.md | 17 +++---- .../src/anvil/artifacts/container.rs | 2 +- crates/cargo-anvil/src/run.rs | 47 ++++++++++++------- 3 files changed, 39 insertions(+), 27 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 3b0f0ec2..0cf89963 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -236,13 +236,14 @@ you trust. #### Customizing the image `.anvil/container/Dockerfile` is a **user-composed file with managed -regions**: anvil owns four regions inside it and reconciles them on every -run, and the three gaps between them are the repository’s. Add extra -packages in the gap that suits when they are needed – before the first -download for a root CA or a proxy, before `anvil-setup` for libraries a -catalog tool compiles against, after it for what the checks need at run -time. Nothing anvil owns is touched, so base and tool-pin bumps keep -landing. +regions**: anvil owns four regions inside it and keeps them current, and the +three gaps between them are the repository’s. Add extra packages in the gap +that suits when they are needed – before the first download for a root CA +or a proxy, before `anvil-setup` for libraries a catalog tool compiles +against, after it for what the checks need at run time. Adding in a gap +leaves anvil’s content alone, so base and tool-pin bumps keep landing; +editing inside a region is preserved rather than overwritten, but freezes +those pins at the moment of the edit, which is why the gaps exist. A downstream catalog that needs a different base OS for every repository it manages replaces the base and tool regions instead, inheriting the catalog @@ -479,7 +480,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb6i_TmlHRql4bFUHGYxLMHPgbSgKMOCwLAF4bQ9SS643u-FthZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb0_sOYsKJMp8bpzOrXba6KmQb91-G89nTbKMbiEX1pLcp3IRhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index aba84047..39148c21 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -55,7 +55,7 @@ const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dock /// Seeded into the Dockerfile when the file does not exist, and never /// reconciled afterwards — it is the user's half of a composed file. /// -/// It carries `# syntax=docker/dockerfile:1`, which BuildKit honors only as +/// It carries `# syntax=docker/dockerfile:1`, which `BuildKit` honors only as /// the very first line of the file. A region's opening sentinel is a comment, /// so the directive cannot live inside a region without being demoted to an /// ordinary comment — silently, with the build falling back to the default diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 7dd54eac..25b8d7cb 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -462,7 +462,7 @@ fn region_placement(region_id: &str) -> RegionPlacement { /// Most region hosts are files the repository already has (`Cargo.toml`, /// `deny.toml`) or files whose first region can simply be appended to nothing. /// The container Dockerfile is neither: `# syntax=docker/dockerfile:1` is a -/// BuildKit parser directive that is honored only when nothing precedes it, +/// `BuildKit` parser directive that is honored only when nothing precedes it, /// not even a comment — so it cannot live inside a region, whose opening /// sentinel *is* a comment. fn host_scaffold(host_relpath: &str) -> Option<&'static str> { @@ -498,23 +498,34 @@ fn host_carries_any_region(host_relpath: &str, text: &str) -> bool { /// silently wrong or unbuildable image. Detect it and refuse instead. fn host_composition_violation(host_relpath: &str, text: &str) -> Option { let order = host_region_order(host_relpath)?; - let mut previous: Option<(&str, usize)> = None; - for id in order { - // A malformed region is reported by the planner itself, with a better - // message than this check could give; a region that is simply absent is - // about to be written. - let Ok(Some(region)) = find_region(text, id, CommentSyntax::Hash) else { - continue; - }; - let start = region.start_line.start; - if let Some((earlier_id, earlier_start)) = previous - && start < earlier_start - { - return Some(format!("region '{id}' appears before '{earlier_id}', but must follow it")); - } - previous = Some((id, start)); - } - None + // The regions actually present, in the order the catalog declares them. A + // malformed region is reported by the planner itself, with a better message + // than this check could give; one that is simply absent is about to be + // written. + let declared: Vec<(&str, usize)> = order + .iter() + .filter_map(|id| match find_region(text, id, CommentSyntax::Hash) { + Ok(Some(region)) => Some((*id, region.start_line.start)), + _ => None, + }) + .collect(); + + // The same regions, in the order the file carries them. Comparing the two + // sequences rather than adjacent offsets keeps the check free of an + // ordering operator whose boundary cannot be exercised: two distinct + // regions can never share a start offset, so `<` and `<=` over positions + // would be indistinguishable by any test. + let mut by_position = declared.clone(); + by_position.sort_by_key(|&(_, start)| start); + + let (expected, found) = declared + .iter() + .zip(by_position.iter()) + .find(|(expected, found)| expected.0 != found.0)?; + Some(format!( + "region '{}' appears before '{}', but must follow it", + found.0, expected.0 + )) } enum DeltaRegionBody { From 1a90bc65b215c51282c10a684a2035633ab4b418 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 01:07:25 +0200 Subject: [PATCH 47/81] style(anvil): let the region-order diagnostic fit one line rustfmt collapses the format! call; the nightly just format run predated this rewrite, so the stable check in CI caught it instead. --- crates/cargo-anvil/src/run.rs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 25b8d7cb..b4b35d85 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -522,10 +522,7 @@ fn host_composition_violation(host_relpath: &str, text: &str) -> Option .iter() .zip(by_position.iter()) .find(|(expected, found)| expected.0 != found.0)?; - Some(format!( - "region '{}' appears before '{}', but must follow it", - found.0, expected.0 - )) + Some(format!("region '{}' appears before '{}', but must follow it", found.0, expected.0)) } enum DeltaRegionBody { From 70b34b95b0f104128ee199e2c1c4e04f4568cd7e Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 01:38:38 +0200 Subject: [PATCH 48/81] fix(anvil): close four composition faults found in review All four are cases where the composed Dockerfile could be written into a state that is wrong rather than refused, or where the gaps it advertises do not work. Data loss on upgrade. The re-seed that replaces the previous release's whole-file Dockerfile asked only whether the path was tracked as an owned file, never whether the bytes still matched the recorded checksum. A repository that had edited that file lost the edit silently, with no proposal to recover from because the file was not yet tracked region-by-region. The check now compares the checksum and refuses instead. Appending to a file anvil never owned. A repository-authored Dockerfile was preserved and the regions appended below it, which puts the repository's own instructions above FROM. That is not composition, it is an unbuildable file; it is now refused with a diagnostic saying what to do. Partially composed hosts. A file carrying some of anvil's regions but not all of them would have the missing ones appended at end-of-file, landing them after regions they must precede. Also refused. Classification happens once per host, from its on-disk state. Doing it per region re-reads text this pass has already spliced, which is partially composed by construction, and would refuse the file halfway through writing it. The gaps could not take a COPY. The ignore file denied .anvil/ outright, so the headline use for the first gap -- copying a corporate root CA in before the first download -- named a file the build context did not contain. It now admits .anvil/container/, which is also the directory the image tag digests, so what the context copies and what the tag covers are the same set again. Also: the seeded header claimed edits inside a region are replaced, which is the opposite of what anvil does; the header is never reconciled, so the repository's own copy is corrected in place. Validation: - cargo test -p cargo-anvil -- all green - cargo clippy -D warnings, cargo fmt --check -- clean - scripts/test-anvil-container.ps1 -Engine docker -- 69/69 against a real daemon - cargo anvil --dry-run -- exit 0, 77 unchanged --- .anvil.lock | 6 +- .anvil/container/Dockerfile | 23 ++- .anvil/container/Dockerfile.dockerignore | 11 + crates/cargo-anvil/src/run.rs | 192 +++++++++++------- .../anvil/container/Dockerfile.dockerignore | 11 + .../justfiles/anvil/checks/aprz.just | 2 +- crates/cargo-anvil/tests/container_upgrade.rs | 112 +++++++++- .../snapshots/snapshots__ado_backend.snap | 13 +- .../snapshots/snapshots__github_backend.snap | 13 +- .../snapshots/snapshots__local_only.snap | 13 +- justfiles/anvil/checks/aprz.just | 2 +- 11 files changed, 302 insertions(+), 96 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 7520d19e..8aae3198 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:05e7198e42dce094505776df73581e933f89db4d961ef2f1127967d211e5a818" +catalog_checksum = "sha256:be14a53887153db40ce5a1518c85437521e16d6fa83b6a79b968261ddbc2dbe2" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:bcffa702f6802244b798c6587c6ade9ef1baa586c3952bdcbe8f4e88cb8f73d4" +checksum = "sha256:07e2729e5fac2463dee334feced401f0dec411aa822c58b874b8b14e69473ffa" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -45,7 +45,7 @@ checksum = "sha256:d4d3bd645a5586e9a1cc3a5fc27e38c93e59b1e29f83ede0ccd610273eec0 [[file]] path = "justfiles/anvil/checks/aprz.just" -checksum = "sha256:63bae0b741774aa8e21ba3168277d5641b24d6239c874ed4e74d953401863ee3" +checksum = "sha256:0f9f3dd3c8a2f034ebf4f2ac0344cba3db79612ff2fc2ea037bfabbaacb1be12" [[file]] path = "justfiles/anvil/checks/audit.just" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index c39539f7..7cbcf675 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -4,15 +4,22 @@ # # Anvil execution image. # -# The `anvil-managed:` regions below belong to cargo-anvil and are reconciled -# on every run, so an edit inside one is replaced. Everything outside them is -# yours and is preserved byte-for-byte, including this header. +# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them +# current, so base and tool-pin bumps arrive on their own. Everything outside +# them is yours and is preserved byte-for-byte, including this header. # -# Add your own instructions in the gaps between the regions. Each region ends -# by describing what the gap after it is for, because position decides whether -# a line works at all: a corporate root CA has to land before the first -# download, and a library needed to *compile* a catalog tool has to land -# before `anvil-setup` runs. +# Add your own instructions in the gaps between the regions rather than inside +# one. Anvil will not overwrite an edit you make inside a region -- it offers +# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and +# is the trap: the region carries the base digest and four tool pins, so an edit +# there quietly freezes them at today's values while the image tag keeps +# resolving, because the tag hashes your file. The gaps exist so that nothing +# you legitimately need to add ever requires touching them. +# +# Each region ends by describing what the gap after it is for, because position +# decides whether a line works at all: a corporate root CA has to land before +# the first download, and a library needed to *compile* a catalog tool has to +# land before `anvil-setup` runs. # # `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit # honours the directive only when nothing precedes it -- not even a comment. diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index 6f694796..9cc2482d 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -16,8 +16,19 @@ # recipes are copied to drive `just anvil-setup`, which needs the whole tree to # parse, and the whole tree is hashed into the image tag: the tier, group and # check recipes decide which tools `anvil-setup` reaches, not just the catalog. +# +# `.anvil/container/` is admitted because the Dockerfile is composed: the gaps +# between anvil's regions exist for a repository to add its own instructions, +# and the headline case -- `COPY`ing a corporate root CA in before the first +# download -- needs the file to be in the context. Denying it would leave the +# gap documented but unusable for anything but `RUN`. It is also the directory +# the image tag digests, so what the context admits and what the tag covers stay +# the same set. * !justfiles justfiles/* !justfiles/anvil +!.anvil +.anvil/* +!.anvil/container !rust-toolchain.toml diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index b4b35d85..11ca93f5 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -162,7 +162,7 @@ fn build_plan( let mut hosts = HostTextCache::default(); // Hosts already reported as unsafe to compose. Every region targeting one // hits the same fault, and four copies of one message is noise. - let mut refused_hosts = BTreeSet::new(); + let mut composed = ComposedHosts::default(); for artifact in catalog.artifacts() { match artifact { @@ -174,7 +174,7 @@ fn build_plan( } } Artifact::Region(spec) => { - push_region(repo_root, workspace, manifest, &mut plan, &mut hosts, &mut refused_hosts, spec)?; + push_region(repo_root, workspace, manifest, &mut plan, &mut hosts, &mut composed, spec)?; } } } @@ -307,26 +307,26 @@ fn push_region( manifest: &Manifest, plan: &mut Plan, hosts: &mut HostTextCache, - refused_hosts: &mut BTreeSet, + composed: &mut ComposedHosts, spec: &RegionSpec, ) -> Result<(), AppError> { match &spec.host { HostSelector::Path(path) => { - push_region_at(repo_root, manifest, plan, hosts, refused_hosts, path, spec)?; + push_region_at(repo_root, manifest, plan, hosts, composed, path, spec)?; } HostSelector::WorkspaceCargoToml => { if workspace.has_workspace_table { - push_region_at(repo_root, manifest, plan, hosts, refused_hosts, "Cargo.toml", spec)?; + push_region_at(repo_root, manifest, plan, hosts, composed, "Cargo.toml", spec)?; } } HostSelector::SingleCrateCargoToml => { if !workspace.has_workspace_table { - push_region_at(repo_root, manifest, plan, hosts, refused_hosts, "Cargo.toml", spec)?; + push_region_at(repo_root, manifest, plan, hosts, composed, "Cargo.toml", spec)?; } } HostSelector::EachMemberManifest => { for member in &workspace.members { - push_region_at(repo_root, manifest, plan, hosts, refused_hosts, &member.manifest_relpath, spec)?; + push_region_at(repo_root, manifest, plan, hosts, composed, &member.manifest_relpath, spec)?; } } } @@ -347,46 +347,33 @@ fn push_region_at( manifest: &Manifest, plan: &mut Plan, hosts: &mut HostTextCache, - refused_hosts: &mut BTreeSet, + composed: &mut ComposedHosts, host: &str, spec: &RegionSpec, ) -> Result<(), AppError> { let host = resolve_existing_case_insensitive(repo_root, host); - let current = hosts.get_or_read(repo_root, &host)?; - // A composed host anvil seeds: when the file does not exist yet, the - // scaffold becomes the base the first region splices into, so the parts of - // the file that cannot live inside a region — a Dockerfile's - // `# syntax=` parser directive above all — are present from the first run. - // It is written once and never reconciled: everything outside the sentinels - // is the repository's. - // - // A file left behind by a release that owned this path outright is re-seeded - // too. Its content is a previous render, not user composition, so appending - // the regions to it would produce a file carrying both the old whole-file - // definition and the new regions. Recognizing it needs both conditions: - // tracked as an owned file in the lock we are superseding, and carrying none - // of this host's regions yet. A Dockerfile the repository wrote by hand is - // in neither state and is left alone. - let scaffold = host_scaffold(&host); - let superseded_owned_file = scaffold.is_some() - && manifest.files.contains_key(host.as_str()) - && current.as_deref().is_some_and(|text| !host_carries_any_region(&host, text)); - let current = match (current, scaffold) { - (None, Some(scaffold)) => Some(scaffold.to_owned()), - (Some(_), Some(scaffold)) if superseded_owned_file => Some(scaffold.to_owned()), - (current, _) => current, - }; - if let Some(text) = current.as_deref() - && let Some(reason) = host_composition_violation(&host, text) - { - // One diagnostic per host: every region targeting it hits the same - // fault, and repeating it once per region buries the one line that - // says which region moved. - if refused_hosts.insert(host.clone()) { + if host_scaffold(&host).is_some() && !composed.states.contains_key(&host) { + let state = match hosts.get_or_read(repo_root, &host)? { + Some(text) => composed_host_state(&host, &text, manifest), + // Nothing on disk. The scaffold becomes the base the first region + // splices into, carrying the parts of the file that cannot live + // inside a region -- the `# syntax=` parser directive above all. It + // is written once and never reconciled; everything outside the + // sentinels is the repository's from then on. + None => ComposedHostState::SeedFromScaffold, + }; + if matches!(state, ComposedHostState::SeedFromScaffold) + && let Some(scaffold) = host_scaffold(&host) + { + hosts.set(&host, scaffold.to_owned()); + } + composed.states.insert(host.clone(), state); + } + if let Some(ComposedHostState::Unsafe(reason)) = composed.states.get(&host) { + if composed.reported.insert(host.clone()) { plan.refusal(format!( - "Refused to manage {host} because its managed regions could not be safely \ - updated: {reason}. Restore the documented order and re-run. Other artifacts \ - were still planned." + "Refused to manage {host}: {reason}. Nothing was written to it, and other \ + artifacts were still planned." )); } plan.push(PlanItem::noop( @@ -398,6 +385,7 @@ fn push_region_at( )); return Ok(()); } + let current = hosts.get_or_read(repo_root, &host)?; let placement = region_placement(spec.id.as_str()); let body = match delta_region_body(current.as_deref(), spec) { DeltaRegionBody::Managed => spec.body.as_str(), @@ -475,34 +463,45 @@ fn host_region_order(host_relpath: &str) -> Option<&'static [&'static str]> { (host_relpath == container::DOCKERFILE_PATH).then_some(container::DOCKERFILE_REGION_ORDER) } -/// Whether the host already carries any of the regions anvil owns in it. -/// -/// Distinguishes a composed file mid-update from a file a previous release -/// owned outright, which must be re-seeded rather than appended to. -fn host_carries_any_region(host_relpath: &str, text: &str) -> bool { - host_region_order(host_relpath).is_some_and(|order| { - order - .iter() - .any(|id| matches!(find_region(text, id, CommentSyntax::Hash), Ok(Some(_)))) - }) +/// What a composed host's current content allows anvil to do with it. +enum ComposedHostState { + /// Carries every region anvil owns, in the declared order. Update in place. + Composable, + /// Either absent, or a byte-identical render from the release that owned + /// this path outright. Every byte of it is anvil's, so the scaffold + /// replaces it and the regions rebuild the file. + SeedFromScaffold, + /// Anvil cannot reach a valid file from here without either destroying + /// repository content or writing something that will not build. + Unsafe(String), +} + +/// Per-host bookkeeping for composed hosts, for the length of one pass. +#[derive(Default)] +struct ComposedHosts { + /// Classification per host, computed once from its on-disk state. Later + /// regions targeting the same host see text this pass has already spliced, + /// which is partially composed by construction — re-classifying that would + /// refuse the file halfway through writing it. + states: HashMap, + /// Hosts whose refusal has already been reported, so one fault produces one + /// diagnostic rather than one per region. + reported: BTreeSet, } -/// Reject a host whose managed regions can no longer be updated into a valid -/// file. +/// Classify a composed host before anything is written to it. /// -/// Every other managed-region host is order-independent — TOML tables, line -/// sets — so `upsert_region` replacing a region wherever it is found is enough. -/// A Dockerfile is the first host where relative order is semantic: `FROM` must -/// precede every instruction that depends on it, the toolchain must exist -/// before `anvil-setup` runs, and a reordered file is not a diagnostic but a -/// silently wrong or unbuildable image. Detect it and refuse instead. -fn host_composition_violation(host_relpath: &str, text: &str) -> Option { - let order = host_region_order(host_relpath)?; - // The regions actually present, in the order the catalog declares them. A - // malformed region is reported by the planner itself, with a better message - // than this check could give; one that is simply absent is about to be - // written. - let declared: Vec<(&str, usize)> = order +/// A Dockerfile is the first managed-region host where the file is only valid +/// in one arrangement, so the states an ordinary region host can ignore all +/// matter here. Each rejected case is one anvil could "handle" by splicing +/// regions in anyway, and each would produce a file that is silently wrong: +/// `upsert_region` appends a missing region at end-of-file, which is the right +/// answer only when the file is already the composed shape. +fn composed_host_state(host_relpath: &str, text: &str, manifest: &Manifest) -> ComposedHostState { + let Some(order) = host_region_order(host_relpath) else { + return ComposedHostState::Composable; + }; + let present: Vec<(&str, usize)> = order .iter() .filter_map(|id| match find_region(text, id, CommentSyntax::Hash) { Ok(Some(region)) => Some((*id, region.start_line.start)), @@ -510,19 +509,68 @@ fn host_composition_violation(host_relpath: &str, text: &str) -> Option }) .collect(); + if present.is_empty() { + // Nothing of anvil's is in the file. Either it is the previous + // release's whole-file render -- safe to replace, because every byte of + // it came from anvil -- or it is content anvil has never owned. + // + // The checksum comparison is the whole safety property: without it, a + // repository that edited the previously-owned Dockerfile would have + // that edit silently destroyed on upgrade. Appending the regions + // instead is not a kinder answer, because everything already in the + // file would then sit above `FROM`. + return match manifest.files.get(host_relpath) { + Some(recorded) if *recorded == checksum_str(text) => ComposedHostState::SeedFromScaffold, + Some(_) => ComposedHostState::Unsafe( + "it was edited after anvil last wrote it, and this release composes the file from \ + managed regions instead of owning it whole. Move the edits you want to keep into \ + the gaps of a freshly generated file, or delete it and re-run to have one written" + .to_owned(), + ), + None => ComposedHostState::Unsafe( + "it exists but anvil has never owned it, so there is nowhere to splice the managed \ + regions that would not put your content above `FROM`. Delete it and re-run to have \ + a composed file written, then move your instructions into the gaps" + .to_owned(), + ), + }; + } + + if present.len() != order.len() { + // A partially composed file. The missing regions would be appended at + // end-of-file, which lands them after regions they must precede. + let missing: Vec<&str> = order + .iter() + .copied() + .filter(|id| !present.iter().any(|(present_id, _)| present_id == id)) + .collect(); + return ComposedHostState::Unsafe(format!( + "it carries some of anvil's regions but not all of them (missing: {}). Restoring the \ + missing ones by appending would place them after regions they must precede. Delete \ + the file and re-run to have a complete one written", + missing.join(", ") + )); + } + // The same regions, in the order the file carries them. Comparing the two // sequences rather than adjacent offsets keeps the check free of an // ordering operator whose boundary cannot be exercised: two distinct // regions can never share a start offset, so `<` and `<=` over positions // would be indistinguishable by any test. - let mut by_position = declared.clone(); + let mut by_position = present.clone(); by_position.sort_by_key(|&(_, start)| start); - - let (expected, found) = declared + match present .iter() .zip(by_position.iter()) - .find(|(expected, found)| expected.0 != found.0)?; - Some(format!("region '{}' appears before '{}', but must follow it", found.0, expected.0)) + .find(|(expected, found)| expected.0 != found.0) + { + None => ComposedHostState::Composable, + Some((expected, found)) => ComposedHostState::Unsafe(format!( + "region '{}' appears before '{}', but must follow it. Restore the documented order and \ + re-run", + found.0, expected.0 + )), + } } enum DeltaRegionBody { diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore index 6f694796..9cc2482d 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore @@ -16,8 +16,19 @@ # recipes are copied to drive `just anvil-setup`, which needs the whole tree to # parse, and the whole tree is hashed into the image tag: the tier, group and # check recipes decide which tools `anvil-setup` reaches, not just the catalog. +# +# `.anvil/container/` is admitted because the Dockerfile is composed: the gaps +# between anvil's regions exist for a repository to add its own instructions, +# and the headline case -- `COPY`ing a corporate root CA in before the first +# download -- needs the file to be in the context. Denying it would leave the +# gap documented but unusable for anything but `RUN`. It is also the directory +# the image tag digests, so what the context admits and what the tag covers stay +# the same set. * !justfiles justfiles/* !justfiles/anvil +!.anvil +.anvil/* +!.anvil/container !rust-toolchain.toml diff --git a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just index 0cc930c3..9b2454d1 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/checks/aprz.just @@ -32,7 +32,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then blocks, for up to an hour, waiting for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 28775323..ac4ac180 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -350,11 +350,12 @@ fn upgrading_from_the_owned_dockerfile_reseeds_the_composed_host() { } /// A Dockerfile the repository wrote itself -- never tracked as an owned file -- -/// is not a superseded render, so it must survive with the regions added to it -/// rather than being replaced. +/// cannot be composed by appending regions to it: everything already in the file +/// would end up above `FROM`. Anvil must refuse and say so, leaving the file +/// untouched, rather than writing something that cannot build. #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] #[test] -fn a_repository_authored_dockerfile_is_not_reseeded() { +fn a_repository_authored_dockerfile_is_refused_not_appended_to() { let tmp = generated_tree(); let root = tmp.path(); let dockerfile = root.join(".anvil/container/Dockerfile"); @@ -366,14 +367,109 @@ fn a_repository_authored_dockerfile_is_not_reseeded() { manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); manifest.save(root).unwrap(); - run_update(&Catalog::anvil(), &local(), root).unwrap(); + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); - let composed = std::fs::read_to_string(&dockerfile).unwrap(); + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + hand_written, + "content anvil never owned must be left exactly as found" + ); + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); + assert!( + refusals[0].contains("anvil has never owned it"), + "the diagnostic must explain why it cannot be composed: {}", + refusals[0] + ); +} + +/// The upgrade re-seed replaces the previous release's whole-file render. It +/// must not do that when the repository edited that file: the file is not +/// tracked region-by-region, so there is no proposal to fall back on and +/// nothing to recover the edit from. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn an_edited_superseded_dockerfile_is_never_overwritten() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // Tracked as an owned file, but the recorded checksum is of a *different* + // body: the repository edited it after anvil last wrote it. + let previous_render = "# syntax=docker/dockerfile:1\nFROM docker.io/library/ubuntu:24.04\n"; + let edited = format!("{previous_render}RUN apt-get install -y our-internal-tool\n"); + write(&dockerfile, &edited); + let mut manifest = Manifest::load(root).unwrap(); + manifest + .files + .insert(".anvil/container/Dockerfile".to_owned(), checksum_str(previous_render)); + manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); + manifest.save(root).unwrap(); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + edited, + "an edit made after the last render must survive the upgrade untouched" + ); + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); assert!( - composed.contains("# hand written by the repository"), - "content anvil never owned must be preserved:\n{composed}" + refusals[0].contains("edited after anvil last wrote it"), + "the diagnostic must name the reason: {}", + refusals[0] + ); +} + +/// A half-composed file: some of anvil's regions present, others missing. +/// `upsert_region` appends a missing region at end-of-file, which would land it +/// after regions it must precede, so this has to be refused too. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_partially_composed_dockerfile_is_refused() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // Drop the base region, keeping the rest. Appending it back would put + // `FROM` after the layers that depend on it. + let text = std::fs::read_to_string(&dockerfile).unwrap(); + let open = text.find("# >>> anvil-managed: anvil-container-base").unwrap(); + let close_marker = "# <<< anvil-managed: anvil-container-base\n"; + let close = text.find(close_marker).unwrap() + close_marker.len(); + let without_base = format!("{}{}", &text[..open], &text[close..]); + write(&dockerfile, &without_base); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + without_base, + "a refused host must not be modified" + ); + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); + assert!( + refusals[0].contains("anvil-container-base"), + "the diagnostic must name the missing region: {}", + refusals[0] ); - assert!(composed.contains("# >>> anvil-managed: anvil-container-base")); } /// The Dockerfile is the first managed-region host whose region *order* is diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 871f4939..f983cf36 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -197,10 +197,21 @@ CMD ["bash"] # recipes are copied to drive `just anvil-setup`, which needs the whole tree to # parse, and the whole tree is hashed into the image tag: the tier, group and # check recipes decide which tools `anvil-setup` reaches, not just the catalog. +# +# `.anvil/container/` is admitted because the Dockerfile is composed: the gaps +# between anvil's regions exist for a repository to add its own instructions, +# and the headline case -- `COPY`ing a corporate root CA in before the first +# download -- needs the file to be in the context. Denying it would leave the +# gap documented but unusable for anything but `RUN`. It is also the directory +# the image tag digests, so what the context admits and what the tag covers stay +# the same set. * !justfiles justfiles/* !justfiles/anvil +!.anvil +.anvil/* +!.anvil/container !rust-toolchain.toml === .delta.toml === @@ -1720,7 +1731,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then blocks, for up to an hour, waiting for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index a855f45e..10bd7567 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -197,10 +197,21 @@ CMD ["bash"] # recipes are copied to drive `just anvil-setup`, which needs the whole tree to # parse, and the whole tree is hashed into the image tag: the tier, group and # check recipes decide which tools `anvil-setup` reaches, not just the catalog. +# +# `.anvil/container/` is admitted because the Dockerfile is composed: the gaps +# between anvil's regions exist for a repository to add its own instructions, +# and the headline case -- `COPY`ing a corporate root CA in before the first +# download -- needs the file to be in the context. Denying it would leave the +# gap documented but unusable for anything but `RUN`. It is also the directory +# the image tag digests, so what the context admits and what the tag covers stay +# the same set. * !justfiles justfiles/* !justfiles/anvil +!.anvil +.anvil/* +!.anvil/container !rust-toolchain.toml === .delta.toml === @@ -1619,7 +1630,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then blocks, for up to an hour, waiting for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 45e5ee10..ca4fc4d6 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -197,10 +197,21 @@ CMD ["bash"] # recipes are copied to drive `just anvil-setup`, which needs the whole tree to # parse, and the whole tree is hashed into the image tag: the tier, group and # check recipes decide which tools `anvil-setup` reaches, not just the catalog. +# +# `.anvil/container/` is admitted because the Dockerfile is composed: the gaps +# between anvil's regions exist for a repository to add its own instructions, +# and the headline case -- `COPY`ing a corporate root CA in before the first +# download -- needs the file to be in the context. Denying it would leave the +# gap documented but unusable for anything but `RUN`. It is also the directory +# the image tag digests, so what the context admits and what the tag covers stay +# the same set. * !justfiles justfiles/* !justfiles/anvil +!.anvil +.anvil/* +!.anvil/container !rust-toolchain.toml === .delta.toml === @@ -460,7 +471,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then blocks, for up to an hour, waiting for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } diff --git a/justfiles/anvil/checks/aprz.just b/justfiles/anvil/checks/aprz.just index 0cc930c3..9b2454d1 100644 --- a/justfiles/anvil/checks/aprz.just +++ b/justfiles/anvil/checks/aprz.just @@ -32,7 +32,7 @@ anvil-aprz: anvil-aprz-validate-prereqs $env:GITHUB_TOKEN = $tok.Trim() } else { Write-Warning 'anvil-aprz: GITHUB_TOKEN is not set and no token could be obtained from the gh CLI.' - Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then WAITS, up to an hour, for the quota to reset.' + Write-Warning 'cargo-aprz will use the unauthenticated GitHub API, which allows 60 requests an hour. On a full workspace it exhausts that and then blocks, for up to an hour, waiting for the quota to reset.' Write-Warning 'To fix: run `gh auth login` (recommended), or set $env:GITHUB_TOKEN to a GitHub token, then re-run.' } } From 65e9e439dbbd96a723c3f5ecd46f548872b1e7f3 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 02:06:32 +0200 Subject: [PATCH 49/81] fix(anvil): fold the composed-host lookups into one and backtick BuildKit The scaffold and the region order are the same predicate -- a host anvil composes rather than owns -- so splitting them left composed_host_state with an early return no caller could reach, which the 100% coverage gate then failed on. One lookup returning both removes the branch. --- crates/cargo-anvil/src/run.rs | 44 ++++++++++++++++------------------- 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 11ca93f5..7c7f2d9e 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -352,9 +352,11 @@ fn push_region_at( spec: &RegionSpec, ) -> Result<(), AppError> { let host = resolve_existing_case_insensitive(repo_root, host); - if host_scaffold(&host).is_some() && !composed.states.contains_key(&host) { + if let Some((scaffold, order)) = composed_host_spec(&host) + && !composed.states.contains_key(&host) + { let state = match hosts.get_or_read(repo_root, &host)? { - Some(text) => composed_host_state(&host, &text, manifest), + Some(text) => composed_host_state(order, &host, &text, manifest), // Nothing on disk. The scaffold becomes the base the first region // splices into, carrying the parts of the file that cannot live // inside a region -- the `# syntax=` parser directive above all. It @@ -362,9 +364,7 @@ fn push_region_at( // sentinels is the repository's from then on. None => ComposedHostState::SeedFromScaffold, }; - if matches!(state, ComposedHostState::SeedFromScaffold) - && let Some(scaffold) = host_scaffold(&host) - { + if matches!(state, ComposedHostState::SeedFromScaffold) { hosts.set(&host, scaffold.to_owned()); } composed.states.insert(host.clone(), state); @@ -444,23 +444,22 @@ fn region_placement(region_id: &str) -> RegionPlacement { } } -/// The initial content for a host anvil composes but does not own, used only -/// when the file is absent. +/// The scaffold and region order for a host anvil composes rather than owns. /// /// Most region hosts are files the repository already has (`Cargo.toml`, -/// `deny.toml`) or files whose first region can simply be appended to nothing. -/// The container Dockerfile is neither: `# syntax=docker/dockerfile:1` is a -/// `BuildKit` parser directive that is honored only when nothing precedes it, -/// not even a comment — so it cannot live inside a region, whose opening -/// sentinel *is* a comment. -fn host_scaffold(host_relpath: &str) -> Option<&'static str> { - (host_relpath == container::DOCKERFILE_PATH).then_some(container::DOCKERFILE_HEADER) -} - -/// The order anvil's regions must appear in inside a composed host, when order -/// is semantically load-bearing. -fn host_region_order(host_relpath: &str) -> Option<&'static [&'static str]> { - (host_relpath == container::DOCKERFILE_PATH).then_some(container::DOCKERFILE_REGION_ORDER) +/// `deny.toml`) or files whose first region can be appended to nothing, and +/// whose regions are order-independent — TOML tables, line sets. The container +/// Dockerfile is neither. +/// +/// The **scaffold** exists because `# syntax=docker/dockerfile:1` is a `BuildKit` +/// parser directive honored only when nothing precedes it, not even a comment — +/// so it cannot live inside a region, whose opening sentinel *is* a comment. It +/// is written when the file is absent and never reconciled afterwards. +/// +/// The **order** is load-bearing: `FROM` must precede every instruction that +/// depends on it, and the toolchain must exist before `anvil-setup` runs. +fn composed_host_spec(host_relpath: &str) -> Option<(&'static str, &'static [&'static str])> { + (host_relpath == container::DOCKERFILE_PATH).then_some((container::DOCKERFILE_HEADER, container::DOCKERFILE_REGION_ORDER)) } /// What a composed host's current content allows anvil to do with it. @@ -497,10 +496,7 @@ struct ComposedHosts { /// regions in anyway, and each would produce a file that is silently wrong: /// `upsert_region` appends a missing region at end-of-file, which is the right /// answer only when the file is already the composed shape. -fn composed_host_state(host_relpath: &str, text: &str, manifest: &Manifest) -> ComposedHostState { - let Some(order) = host_region_order(host_relpath) else { - return ComposedHostState::Composable; - }; +fn composed_host_state(order: &[&str], host_relpath: &str, text: &str, manifest: &Manifest) -> ComposedHostState { let present: Vec<(&str, usize)> = order .iter() .filter_map(|id| match find_region(text, id, CommentSyntax::Hash) { From 8b18263f7b7b70648af34f4e3772de8029f6f573 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 02:12:26 +0200 Subject: [PATCH 50/81] fix(anvil): keep a refused host untouched in the lock as well as on disk Two faults in the composed-host classifier, both found in review. A refusal claims "nothing was written to it", but the ownership-transfer branch in the removal pass still dropped the host's lock entry. That entry is the provenance the next run reclassifies from, so the second run would call a file anvil had rendered one it had never owned, give the wrong diagnostic, and leave "delete the file" as the only remedy -- destroying the gap content the message had just told the reader to preserve. Reverting the edit is the clean recovery and it only exists while the recorded checksum survives, so a refused host is now skipped by the removal pass entirely. A malformed sentinel was folded in with "this region is absent", which sends the reader looking for content that is in fact present with a broken marker. It is now its own state and names the underlying error. Both are covered: one test asserts the lock is byte-identical after a refusal and that reverting the edit then composes cleanly, the other that a duplicated sentinel is reported as unreadable rather than missing. --- crates/cargo-anvil/src/run.rs | 53 +++++++++-- crates/cargo-anvil/tests/container_upgrade.rs | 93 +++++++++++++++++++ 2 files changed, 140 insertions(+), 6 deletions(-) diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 7c7f2d9e..5ddb57d6 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -179,7 +179,7 @@ fn build_plan( } } - plan_removals(repo_root, manifest, &mut plan, &mut hosts)?; + plan_removals(repo_root, manifest, &mut plan, &mut hosts, &composed)?; // Region proposals are computed eagerly as each region is visited, so a // `Propose` planned before a sibling `Write`/`Remove` on the same host @@ -497,13 +497,24 @@ struct ComposedHosts { /// `upsert_region` appends a missing region at end-of-file, which is the right /// answer only when the file is already the composed shape. fn composed_host_state(order: &[&str], host_relpath: &str, text: &str, manifest: &Manifest) -> ComposedHostState { + // A malformed sentinel is its own diagnosis. Folding it in with "absent" + // would report a broken region as a missing one, sending the reader looking + // for content that is in fact right there with a mismatched marker. + let mut malformed: Option = None; let present: Vec<(&str, usize)> = order .iter() .filter_map(|id| match find_region(text, id, CommentSyntax::Hash) { Ok(Some(region)) => Some((*id, region.start_line.start)), - _ => None, + Ok(None) => None, + Err(err) => { + malformed.get_or_insert_with(|| format!("its '{id}' region cannot be read: {err}")); + None + } }) .collect(); + if let Some(reason) = malformed { + return ComposedHostState::Unsafe(reason); + } if present.is_empty() { // Nothing of anvil's is in the file. Either it is the previous @@ -608,7 +619,13 @@ fn delta_region_body(host_text: Option<&str>, spec: &RegionSpec) -> DeltaRegionB /// This is what removes orphaned cloud-workflow artifacts, dropped catalog entries, /// disabled-backend files, and any other previously-tracked item that /// is no longer in scope. -fn plan_removals(repo_root: &Path, previous: &Manifest, plan: &mut Plan, hosts: &mut HostTextCache) -> Result<(), AppError> { +fn plan_removals( + repo_root: &Path, + previous: &Manifest, + plan: &mut Plan, + hosts: &mut HostTextCache, + composed: &ComposedHosts, +) -> Result<(), AppError> { let live_files: BTreeSet = plan .items() .iter() @@ -637,8 +654,18 @@ fn plan_removals(repo_root: &Path, previous: &Manifest, plan: &mut Plan, hosts: // for the same pass just produced, and the user content around them // with it. Drop the stale manifest entry and leave the file alone; the // region entries now describe what anvil owns inside it. + // + // Unless the host was *refused*, in which case anvil declined to touch + // it and the lock entry is the provenance the next run reclassifies + // from. Dropping it would make a file anvil rendered look like one it + // has never owned, flip the diagnostic to the wrong wording, and + // destroy the clean recovery -- reverting the edit -- that the refusal + // message tells the reader to use. "Nothing was written to it" has to + // be true of the lock as well as the file. if live_region_hosts.contains(path) { - plan.push(PlanItem::orphaned_kept(Target::File { path: path.clone() })); + if !matches!(composed.states.get(path), Some(ComposedHostState::Unsafe(_))) { + plan.push(PlanItem::orphaned_kept(Target::File { path: path.clone() })); + } continue; } let disk = read_file_if_present(&repo_root.join(path))?; @@ -1511,7 +1538,14 @@ mod tests { let mut previous = Manifest::default(); previous.set_region("Justfile", "anvil-r", "sha256:body"); let mut plan = Plan::default(); - plan_removals(tmp.path(), &previous, &mut plan, &mut HostTextCache::default()).unwrap(); + plan_removals( + tmp.path(), + &previous, + &mut plan, + &mut HostTextCache::default(), + &ComposedHosts::default(), + ) + .unwrap(); let orphans: Vec<(&str, &str)> = plan .items() .iter() @@ -1542,7 +1576,14 @@ mod tests { // decide_removal classifies the region as a customized orphan. previous.set_region("Justfile", "anvil-r", "sha256:stored-different"); let mut plan = Plan::default(); - plan_removals(tmp.path(), &previous, &mut plan, &mut HostTextCache::default()).unwrap(); + plan_removals( + tmp.path(), + &previous, + &mut plan, + &mut HostTextCache::default(), + &ComposedHosts::default(), + ) + .unwrap(); let orphans: Vec<(&str, &str)> = plan .items() .iter() diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index ac4ac180..1856856f 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -520,3 +520,96 @@ fn a_reordered_composed_dockerfile_is_refused_with_one_diagnostic() { "a refused host must not be modified" ); } + +/// A refusal says "nothing was written to it". That has to be true of the lock +/// as well as the file: the recorded checksum is the provenance the next run +/// reclassifies from, and the clean recovery -- revert the edit, get a valid +/// re-seed -- only exists while it survives. Dropping it would make a file +/// anvil rendered look like one it has never owned, and the second run would +/// give the wrong diagnostic. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn refusing_a_composed_host_leaves_its_lock_entry_intact() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + let previous_render = "# syntax=docker/dockerfile:1\nFROM docker.io/library/ubuntu:24.04\n"; + let edited = format!("{previous_render}RUN apt-get install -y our-internal-tool\n"); + write(&dockerfile, &edited); + let mut manifest = Manifest::load(root).unwrap(); + manifest + .files + .insert(".anvil/container/Dockerfile".to_owned(), checksum_str(previous_render)); + manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); + manifest.save(root).unwrap(); + let lock_before = std::fs::read_to_string(root.join(".anvil.lock")).unwrap(); + + run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let after = Manifest::load(root).unwrap(); + assert_eq!( + after.files.get(".anvil/container/Dockerfile"), + Some(&checksum_str(previous_render)), + "a refused host keeps its recorded checksum, or its provenance is gone" + ); + assert_eq!( + std::fs::read_to_string(root.join(".anvil.lock")).unwrap(), + lock_before, + "refusing must not rewrite the lock at all" + ); + + // And the recovery the message advertises still works: revert the edit and + // the next run composes the file cleanly. + write(&dockerfile, previous_render); + run_update(&Catalog::anvil(), &local(), root).unwrap(); + let composed = std::fs::read_to_string(&dockerfile).unwrap(); + assert!(composed.starts_with("# syntax=docker/dockerfile:1\n")); + for id in [ + "anvil-container-base", + "anvil-container-tools", + "anvil-container-setup", + "anvil-container-entry", + ] { + assert!( + composed.contains(&format!("# >>> anvil-managed: {id}")), + "{id} must be spliced in after recovery" + ); + } +} + +/// A malformed sentinel must be reported as such, not folded in with "this +/// region is missing" -- the content is right there, with a broken marker. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_malformed_sentinel_is_named_in_the_refusal() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // Duplicate the opening sentinel, leaving the close unmatched. + let text = std::fs::read_to_string(&dockerfile).unwrap(); + let opener = "# >>> anvil-managed: anvil-container-base"; + let broken = text.replacen(opener, &format!("{opener}\n{opener}"), 1); + write(&dockerfile, &broken); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + broken, + "a refused host must not be modified" + ); + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); + assert!( + refusals[0].contains("cannot be read"), + "a broken sentinel must not be reported as a missing region: {}", + refusals[0] + ); +} From 43244ccc5027101a647309b76edecd0abc3c87ef Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 02:34:07 +0200 Subject: [PATCH 51/81] test(anvil): stop the mutants-diff tests inheriting the CI environment Two independent early exits made this test assert against a recipe that had deliberately done nothing, and only on CI, because both are driven by inherited environment variables. ANVIL_INCLUDE_AFFECTED is set to --skip by impact scoping when a leg has no affected packages. Pinned to a scope that runs. PROCESSOR_ARCHITECTURE was the other, and the skip-branch test tried to force it to ARM64 to exercise the bail-out. That is not safe on an x64 host: the variable is load-bearing for the Windows loader, and falsifying it upward made spawning `just` fail outright rather than reach the branch. CI has a real ARM64 Windows leg, so the test now runs only there and reads the true architecture. Claiming AMD64 on ARM64 stays fine -- it is a supported emulation story, and the recipe demonstrably ran under it. --- crates/cargo-anvil/tests/recipe_contracts.rs | 23 +++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 8635a94e..f3b6d2f7 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -874,6 +874,12 @@ fn mutants_diff_covers_uncommitted_work() { // second one. ("PROCESSOR_ARCHITECTURE", OsStr::new("AMD64")), ("PROCESSOR_ARCHITEW6432", OsStr::new("")), + // The other early exit. Impact scoping sets this to `--skip` when a + // job has no affected packages, and the value is inherited from + // whatever environment the test runs in -- so on a CI leg that + // skipped, this test would assert against a recipe that returned + // before doing anything. Pin it to a scope that runs. + ("ANVIL_INCLUDE_AFFECTED", OsStr::new("--package fixture@0.1.0")), ], ); assert!( @@ -898,15 +904,17 @@ fn mutants_diff_covers_uncommitted_work() { /// cargo-mutants does not build for `aarch64-pc-windows-msvc`, so the recipe /// exits cleanly rather than failing the merged `pr-slow` group on that leg. /// +/// This runs only on a real ARM64 Windows host, which CI has. Faking the +/// architecture is not an option: `PROCESSOR_ARCHITECTURE` is load-bearing for +/// the Windows loader, and setting it to ARM64 on an x64 host makes spawning +/// `just` fail outright rather than exercise the branch. +/// /// Asserting it here is what keeps the sibling test above honest. That one pins /// the architecture to AMD64 so it exercises the real path; without this test -/// the skip branch would be exercised by nothing, and an ARM64 runner would be -/// the only place either behavior was observed. +/// the skip branch would be exercised by nothing. #[test] fn mutants_diff_skips_on_arm64_windows() { - // Only meaningful where the recipe's `$IsWindows` guard can be true; on - // Linux and macOS the architecture variable is not consulted at all. - if !cfg!(windows) { + if !(cfg!(windows) && cfg!(target_arch = "aarch64")) { return; } let tmp = fixture( @@ -925,7 +933,10 @@ fn mutants_diff_skips_on_arm64_windows() { &[ ("FAKE_CARGO_LOG", log.as_os_str()), ("RUNNER_TEMP", root.as_os_str()), - ("PROCESSOR_ARCHITECTURE", OsStr::new("ARM64")), + // Not the architecture -- that is the host's, and real here. This + // is the *other* early exit, pinned so a skipped impact scope + // cannot be mistaken for the architecture bail-out. + ("ANVIL_INCLUDE_AFFECTED", OsStr::new("--package fixture@0.1.0")), ], ); From 1f0a8ad45de02204544d5892798e773fd780dbe2 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 03:00:36 +0200 Subject: [PATCH 52/81] test(anvil): show the recipe's own output when the mutants-diff contract fails The assertion printed only the fake-cargo log, which is empty for both of the recipe's early exits -- so a failure could not distinguish the architecture bail-out from a skipped impact scope. It now prints stdout and stderr, where the recipe says which branch it took. --- crates/cargo-anvil/tests/recipe_contracts.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index f3b6d2f7..8b2da9be 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -890,7 +890,12 @@ fn mutants_diff_covers_uncommitted_work() { ); let calls = std::fs::read_to_string(&log).unwrap_or_default(); - assert!(calls.contains("--in-diff"), "cargo mutants must be given a diff file:\n{calls}"); + assert!( + calls.contains("--in-diff"), + "cargo mutants must be given a diff file.\ncargo log:\n{calls}\nrecipe stdout:\n{}\nrecipe stderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); let diff = std::fs::read_to_string(root.join("anvil-mutants-diff.diff")).unwrap(); assert!(diff.contains("committed"), "the committed change must be in the diff:\n{diff}"); From c6519981ad8a370f944cdfd64558afa8ac29d80b Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 03:19:23 +0200 Subject: [PATCH 53/81] test(anvil): skip the mutants-diff contract where the recipe has no behavior PROCESSOR_ARCHITECTURE cannot be overridden across a process boundary on Windows: the loader re-derives it for each new process from that process's real architecture, so the pin added in the previous commit never reached the recipe. It only looked like it worked on x64, where the guard is false regardless. On aarch64-pc-windows-msvc the recipe therefore always bails out before doing anything, and there is no --in-diff behavior for this test to assert. It now returns early there, with the skip branch itself covered by the sibling test that runs on a real ARM64 host. The contract is still exercised on the other three legs. The ANVIL_INCLUDE_AFFECTED pin stays: that variable *is* inherited and overridable, and a skipped impact scope was a genuine way for this test to pass against a recipe that had returned early. --- crates/cargo-anvil/tests/recipe_contracts.rs | 24 +++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 8b2da9be..935062e0 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -801,6 +801,16 @@ fn mutants_diff_covers_uncommitted_work() { if !tools_available() || Command::new("git").arg("--version").output().is_err() { return; } + // On aarch64-pc-windows-msvc the recipe bails out before doing any of this, + // because cargo-mutants does not build there -- so there is no `--in-diff` + // behavior to assert. The architecture cannot be faked past: Windows + // re-derives PROCESSOR_ARCHITECTURE for every new process from its real + // architecture, so an override does not survive the spawn. The skip itself + // is covered by `mutants_diff_skips_on_arm64_windows`, and this contract is + // exercised on the other three legs. + if cfg!(windows) && cfg!(target_arch = "aarch64") { + return; + } let tmp = fixture( &[("helpers.just", HELPERS), ("mutants-diff.just", MUTANTS_DIFF)], &[ @@ -866,19 +876,17 @@ fn mutants_diff_covers_uncommitted_work() { ("FAKE_CARGO_LOG", log.as_os_str()), ("BASE_REF", OsStr::new(&base)), ("RUNNER_TEMP", root.as_os_str()), - // Pin the architecture the recipe branches on. It bails out early on - // aarch64-pc-windows-msvc, where cargo-mutants does not build, so on - // an ARM64 Windows runner this test would otherwise assert against a - // recipe that deliberately did nothing. Both variables are set - // because a 32-bit host process reports the real machine in the - // second one. - ("PROCESSOR_ARCHITECTURE", OsStr::new("AMD64")), - ("PROCESSOR_ARCHITEW6432", OsStr::new("")), // The other early exit. Impact scoping sets this to `--skip` when a // job has no affected packages, and the value is inherited from // whatever environment the test runs in -- so on a CI leg that // skipped, this test would assert against a recipe that returned // before doing anything. Pin it to a scope that runs. + // + // The architecture guard is deliberately *not* pinned: Windows + // re-derives PROCESSOR_ARCHITECTURE for each new process from the + // process's real architecture, so it cannot be overridden across a + // spawn. That is why this test returns early on ARM64 above rather + // than faking its way past the branch. ("ANVIL_INCLUDE_AFFECTED", OsStr::new("--package fixture@0.1.0")), ], ); From 1240523c5088ebb9d79eb6f152b92d5dfe642f03 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 04:03:04 +0200 Subject: [PATCH 54/81] test(anvil): guard the arm skip test on tool availability Every other recipe-contract test returns early when the generated toolchain is not installed; this one did not, so it hard-failed on an ARM runner whose install-action hit the known bash-startup flake and never installed just. Missing tools are a skip, not a failure -- the assertion is about the recipe, not about the runner. --- crates/cargo-anvil/tests/recipe_contracts.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 935062e0..cc7a5ee5 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -927,6 +927,9 @@ fn mutants_diff_covers_uncommitted_work() { /// the skip branch would be exercised by nothing. #[test] fn mutants_diff_skips_on_arm64_windows() { + if !tools_available() { + return; + } if !(cfg!(windows) && cfg!(target_arch = "aarch64")) { return; } From 85138ad0ff0b807393055c8062dfb3a2cf6383f9 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 08:00:35 +0200 Subject: [PATCH 55/81] refactor(anvil): put every anvil line in a region and make the base overridable Two flaws in the composed Dockerfile, both raised in review. Anvil's own content was sitting outside the regions. 28 of the 29 lines above the first sentinel -- the copyright, the explanation of how the regions work, the guidance about which gap takes what -- were anvil's, in territory anvil can never reconcile. That is not theoretical: correcting the header wording earlier in this branch could not reach a repository that had already generated the file. The scaffold now holds only `# syntax=docker/dockerfile:1`, which BuildKit honors solely as line 1 and which therefore genuinely cannot live in a region. Everything else moves into a new `anvil-container-header` region. The base image was unoverridable without editing anvil's content. `ARG BASE_IMAGE` and the `FROM` consuming it shared one region, so choosing another base meant an edit inside a region -- which preserves the edit and freezes all four tool pins with it, the exact failure the composition exists to prevent. The default now has its own region and a gap follows it: a second `ARG BASE_IMAGE=` there wins, verified against a real build, and anvil keeps updating the pins the repository did not touch. That leaves a risk worth naming: a base with an older glibc breaks `binstall`, and switching the catalog to source installs is not a repository-level lever. The region text says so and points at the gap rather than the line. Adding those two regions exposed a third fault. A missing region was appended at end-of-file, and a partially composed file was refused outright -- so every future release that adds a region would have told adopters to delete their Dockerfile and lose their gap content. `RegionPlacement::At` now splices a missing region at its declared position, after the nearest present predecessor or directly below the scaffold, and refusal is reserved for genuine disorder. Prose trimmed throughout: the header is 19 lines rather than 34, and the setup region no longer narrates more than it instructs. Validation: - cargo test -p cargo-anvil -- all green; clippy -D warnings and fmt --check clean - scripts/test-anvil-container.ps1 -Engine docker -- 69/69 against a real daemon - docker build of the recomposed six-region Dockerfile -- succeeds - regenerating this repo upgraded its own 4-region file in place, inserting both new regions in order; `cargo anvil --dry-run` then reports 79 unchanged --- .anvil.lock | 18 +- .anvil/container/Dockerfile | 107 +++++------- crates/cargo-anvil/README.md | 14 +- crates/cargo-anvil/docs/design/containers.md | 52 ++++-- .../src/anvil/artifacts/container.rs | 154 +++++++++++++----- crates/cargo-anvil/src/lib.rs | 12 +- crates/cargo-anvil/src/region.rs | 29 ++++ crates/cargo-anvil/src/run.rs | 66 ++++++-- .../anvil/container/Dockerfile.base.region | 18 -- .../container/Dockerfile.baseimage.region | 8 + .../anvil/container/Dockerfile.header | 28 ---- .../anvil/container/Dockerfile.header.region | 19 +++ .../anvil/container/Dockerfile.setup.region | 39 ++--- .../anvil/container/Dockerfile.tools.region | 7 - crates/cargo-anvil/tests/container_upgrade.rs | 57 ++++--- .../snapshots/snapshots__ado_backend.snap | 109 +++++-------- .../snapshots/snapshots__github_backend.snap | 109 +++++-------- .../snapshots/snapshots__local_only.snap | 109 +++++-------- 18 files changed, 478 insertions(+), 477 deletions(-) create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region create mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region diff --git a/.anvil.lock b/.anvil.lock index 8aae3198..cd44f417 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:be14a53887153db40ce5a1518c85437521e16d6fa83b6a79b968261ddbc2dbe2" +catalog_checksum = "sha256:d3d209ea0063028b186135dc2337612e079acc6168bb6de957b64eb1f65c1d27" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -226,22 +226,32 @@ checksum = "sha256:acbea93d5117db747537f4f7b9a5eb90b7d3e0dd3e8684cc0e4dc1dcb15ac [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-base" -checksum = "sha256:52ed0faca8dec1cf4ad09602feb7ae627a0d5f1cabc903a8052c6b771c1302f6" +checksum = "sha256:734e21d8ae8ce8c0a00f54a36a3a9f15a02b52eff11c27f3de4f7cd60bb95449" + +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-base-image" +checksum = "sha256:74ed18b9ccc5a5232be05392d572c2395d0f73c10da9b372c75280639e398dc6" [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-entry" checksum = "sha256:7b409a9b560c214e10b50f74330fb6f8c0c12c3d83494e0dcf016f2411b50365" +[[region]] +host = ".anvil/container/Dockerfile" +id = "anvil-container-header" +checksum = "sha256:be87e9614441a6802039f95858f1fdd5c6116c4be5cc8fcad244763c32aabcaf" + [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-setup" -checksum = "sha256:80e0b29f9e8eec3b8c98976c718f11e5afc9c63131fcf4b0a0998c04895ace5b" +checksum = "sha256:7378551c253df01632d1d5827f698f3fee9f410b0113146130d69b2fbd9af473" [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-tools" -checksum = "sha256:0faa15c3d2a9d6fafb0748c92ffc15564e8cbfd3dbfd8284c2eacf701f4efe72" +checksum = "sha256:433506a65ac00b9ac6c4cb7063fc94e253215885c34e52605f952c484e029e8e" [[region]] host = ".delta.toml" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 7cbcf675..56b459d6 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -1,44 +1,39 @@ # syntax=docker/dockerfile:1 + +# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. # # Anvil execution image. # -# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them -# current, so base and tool-pin bumps arrive on their own. Everything outside -# them is yours and is preserved byte-for-byte, including this header. +# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps +# between them are yours and are preserved. Add to a gap, not inside a region: +# anvil will not overwrite an edit inside one, so editing there silently freezes +# the base digest and the tool pins at that moment. # -# Add your own instructions in the gaps between the regions rather than inside -# one. Anvil will not overwrite an edit you make inside a region -- it offers -# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and -# is the trap: the region carries the base digest and four tool pins, so an edit -# there quietly freezes them at today's values while the image tag keeps -# resolving, because the tag hashes your file. The gaps exist so that nothing -# you legitimately need to add ever requires touching them. +# Which gap, since position decides whether a line works at all: # -# Each region ends by describing what the gap after it is for, because position -# decides whether a line works at all: a corporate root CA has to land before -# the first download, and a library needed to *compile* a catalog tool has to -# land before `anvil-setup` runs. +# after base-image re-declare ARG BASE_IMAGE to build on your own base +# after base root CA, proxy, apt mirror -- anything the first download needs +# after tools libraries a catalog tool needs to compile from source +# after setup what your own checks need at run time # -# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit -# honours the directive only when nothing precedes it -- not even a comment. -# That is also why it sits out here rather than inside a region: a region's -# opening sentinel is a comment, and would silently demote the directive to an -# ordinary one, leaving the build on the default frontend with nothing failing -# to say so. +# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, +# and a region sentinel is a comment. +# <<< anvil-managed: anvil-container-header -# >>> anvil-managed: anvil-container-base -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. +# >>> anvil-managed: anvil-container-base-image +# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). +# `anvil-setup binstall` installs prebuilt binaries that need that runner's +# glibc, so this moves when the runner does. Digest-pinned because a floating +# tag can change under an image reference that claims to name fixed content. # -# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a -# change here renames the image -- but it does not resolve or validate the -# base, and a floating tag could therefore change underneath a reference that -# claims to name fixed content. +# To build on another base, re-declare this in the gap below rather than editing +# here: a later ARG wins, and anvil keeps updating the pins you did not touch. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +# <<< anvil-managed: anvil-container-base-image + +# >>> anvil-managed: anvil-container-base FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -55,14 +50,6 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_HOME=/usr/local/rustup \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs before anvil's first download. Put whatever -# the image needs in order to reach the network here: a corporate root CA, -# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. -# Without it, a TLS-intercepting proxy makes every `curl` in the next region -# fail and the image cannot be built at all. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-base # >>> anvil-managed: anvil-container-tools @@ -112,39 +99,27 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the toolchain is in place and before -# `anvil-setup` installs the catalog. Put system libraries a catalog tool -# needs in order to *compile* here: `binstall` falls back to a source build -# when a pinned tool publishes no prebuilt for this platform, and that build -# links against whatever headers the image has. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. +# Install the pinned toolchain and cargo subcommands from the generated catalog. +# The whole recipe tree is copied because `just` has to parse it, and the whole +# tree is hashed into the image tag: `anvil-setup` reaches the install recipes +# through the tier, group and check recipes, so any of them can change what this +# layer installs. # -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. +# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do +# not compile from source on every pinned toolchain. # # The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. +# secret never lands in a layer, but anything the install *writes* with it is +# ordinary content, and the `chmod` below would publish it world-readable. +# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. # # `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. +# named volume over each, and an engine seeds a new volume from the image path +# it covers. A path that does not exist seeds as root-owned 0755, which the +# `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -154,14 +129,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the catalog is installed. Put what your -# own checks need at run time here: a database client, a protobuf compiler, a -# linter that is not a cargo subcommand. It is also the cheapest place to add -# anything, because an edit here invalidates one layer rather than the -# multi-minute `anvil-setup` above. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-setup # >>> anvil-managed: anvil-container-entry diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 0cf89963..1e66eff5 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -236,12 +236,12 @@ you trust. #### Customizing the image `.anvil/container/Dockerfile` is a **user-composed file with managed -regions**: anvil owns four regions inside it and keeps them current, and the -three gaps between them are the repository’s. Add extra packages in the gap -that suits when they are needed – before the first download for a root CA -or a proxy, before `anvil-setup` for libraries a catalog tool compiles -against, after it for what the checks need at run time. Adding in a gap -leaves anvil’s content alone, so base and tool-pin bumps keep landing; +regions**: anvil owns six regions inside it and keeps them current, and the +gaps between them are the repository’s. Add to the gap that matches when the +addition is needed – re-declare `ARG BASE_IMAGE` to build on another base, +a root CA or proxy before the first download, libraries a catalog tool +compiles against before `anvil-setup`, run-time tools after it. Adding in a +gap leaves anvil’s content alone, so base and tool-pin bumps keep landing; editing inside a region is preserved rather than overwritten, but freezes those pins at the moment of the edit, which is why the gaps exist. @@ -480,7 +480,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb0_sOYsKJMp8bpzOrXba6KmQb91-G89nTbKMbiEX1pLcp3IRhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbcyN35C50s70b7icANpGEc88bQlr3I2a8WwobrcveTzoRNR9hZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index b5911ede..503d2f3a 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -131,20 +131,22 @@ repo/ │ ├── container.just the anvil-container recipes │ └── … checks, groups, tiers, executed natively *inside* the image └── .anvil/container/ - ├── Dockerfile composed: anvil's four regions, your content in the gaps + ├── Dockerfile composed: anvil's six regions, your content in the gaps ├── Dockerfile.dockerignore what the build context admits └── hooks.ps1 optional; not emitted by default (§7) ``` `container.just` and `Dockerfile.dockerignore` are owned files carrying the usual `DO NOT EDIT DIRECTLY` marker. -The **Dockerfile is a user-composed file with managed regions**, not an owned file. Anvil owns four regions inside it -and keeps them current; everything outside the sentinels — including the header and the three gaps between them — is -the repository's and is preserved byte-for-byte. +The **Dockerfile is a user-composed file with managed regions**, not an owned file. Anvil owns six regions inside it +and keeps them current; the gaps between them are the repository's and are preserved byte-for-byte. Exactly one line — +`# syntax=docker/dockerfile:1` — sits outside a region, because it cannot live inside one. | Region | Contains | Gap that follows it is for | | --- | --- | --- | -| `anvil-container-base` | `ARG BASE_IMAGE`, `FROM`, the four download pins, `ENV` | a root CA, `http_proxy`, an internal apt mirror — anything needed to reach the network at all | +| `anvil-container-header` | what the regions are and which gap takes what | — | +| `anvil-container-base-image` | `ARG BASE_IMAGE`, digest-pinned | re-declaring `ARG BASE_IMAGE` to build on your own base | +| `anvil-container-base` | `FROM`, the four download pins, `ENV` | a root CA, `http_proxy`, an internal apt mirror — anything needed to reach the network at all | | `anvil-container-tools` | system packages, `pwsh`, `just`, `rustup`, `cargo-binstall` | libraries a catalog tool needs to *compile*, when `binstall` falls back to a source build | | `anvil-container-setup` | `COPY` of the recipe tree, `just anvil-setup` | what the repository's own checks need at run time; also the cheapest layer to add to | | `anvil-container-entry` | `ANVIL_IN_CONTAINER`, `WORKDIR`, `CMD` | — | @@ -159,25 +161,39 @@ the tag hashes their file*. The identity scheme works perfectly and still names **What regions actually change.** They do not make anvil's content unwritable: §2's ownership rules apply to a region body exactly as they do to a file, so an edit *inside* a region is still preserved and still produces a proposal rather than being overwritten. Anvil never destroys repository content, and a special case here would be the one place it did. -What changes is that there is no longer a reason to edit: every legitimate addition — a root CA, a build dependency, a -run-time tool — has a gap that is the *correct* place for it, chosen by what must already be true at that point in the -build. Editing anvil's content stops being the only way to extend the image and becomes a mistake the layout steers -away from, and the pin freeze goes with it for every repository that takes the gaps. - -**Two constraints the region engine had to grow for this.** Both are specific to a Dockerfile and neither applies to the +What changes is that there is no longer a reason to edit: every legitimate addition — another base image, a root CA, a +build dependency, a run-time tool — has a gap that is the *correct* place for it, chosen by what must already be true +at that point in the build. + +**The base image is the case that most needed a gap of its own.** It is the setting a repository is likeliest to want, +and a single region holding both `ARG BASE_IMAGE` and the `FROM` that consumes it would have made overriding it mean +editing anvil's content — freezing every other pin in the file to buy one substitution. Split across two regions, a +second `ARG BASE_IMAGE=…` in the gap wins (a later declaration replaces the default) and anvil keeps updating +everything the repository did not touch. The residual risk is real and worth stating: a base with an older glibc breaks +`binstall`, and switching the catalog to source installs is not a repository-level lever. + +**Three constraints the region engine had to grow for this.** All are specific to a Dockerfile; none applies to the order-independent TOML and line-set hosts anvil already had: - **`# syntax=docker/dockerfile:1` must be line 1.** BuildKit honours a parser directive only when nothing precedes it, not even a comment — and a region's opening sentinel *is* a comment, so the directive cannot live inside one without being silently demoted, leaving the build on the default frontend with nothing failing to say so. It is therefore - part of a **scaffold** anvil writes when the file does not exist, and never reconciles afterwards. -- **Region order is semantic.** `FROM` must precede everything, and the toolchain must exist before `anvil-setup` runs. - Fresh files get catalog order; thereafter the engine checks the on-disk sequence against the declared one and - **refuses** the host, reporting which region is out of place, rather than emitting a Dockerfile that is wrong. + the whole of the **scaffold** anvil writes when the file does not exist and never reconciles afterwards. Keeping the + scaffold to that one line is deliberate: anything anvil owns that sits outside a region can never be corrected on a + repository that already generated the file. +- **Region order is semantic.** `ARG BASE_IMAGE` must precede the `FROM` that consumes it, `FROM` must precede + everything, and the toolchain must exist before `anvil-setup` runs. The engine checks the on-disk sequence against + the declared one and **refuses** the host, reporting which region is out of place, rather than emitting a Dockerfile + that is wrong. +- **A missing region is inserted in order, not appended.** Appending at end-of-file is right for every other host and + wrong here: a region added in a later release would land after the ones it must precede. Anvil splices it at its + declared position instead — after the nearest preceding region that is present, or directly below the scaffold — so + adding a region is an ordinary update that leaves the repository's gap content untouched. A repository upgrading from the release that owned this path outright is re-seeded rather than appended to: a file tracked as an owned file in the lock and carrying none of the regions is a previous render, not composition. A -Dockerfile the repository wrote itself is in neither state and is left alone, with the regions added to it. +Dockerfile the repository wrote itself is in neither state and is refused, since there is nowhere to splice the regions +that would not put its content above `FROM`. The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the @@ -568,8 +584,8 @@ an ordinary artifact group and uses the same levers as any other. | Goal | Mechanism | Owner | | --- | --- | --- | -| Extra packages in one repository | Add them in one of the three gaps in `.anvil/container/Dockerfile` (§3) | repository | -| A different base OS, everywhere | `replace_artifact(artifacts::container::dockerfile_base().with_body(…))`, usually with `dockerfile_tools()` | catalog | +| Extra packages, or another base image, in one repository | Add them in the matching gap in `.anvil/container/Dockerfile` (§3) | repository | +| A different base OS, everywhere | `replace_artifact(artifacts::container::dockerfile_base_image().with_body(…))`, usually with `dockerfile_tools()` | catalog | | Credentials, or a published image | Add `.anvil/container/hooks.ps1`, or ship `artifacts::container::hooks(…)` | either | | No containerized execution at all | `without_artifact` for each artifact in the group | catalog | diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 39148c21..fc8f96cb 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -24,22 +24,29 @@ //! Splitting anvil's content into regions does not make it read-only: the same //! ownership rules apply to a region body as to a file, and anvil never //! overwrites repository content. What it removes is the *reason* to edit: each -//! of the three gaps between the regions is the correct home for one class of -//! addition, defined by what must already be true at that point in the build. +//! gap between the regions is the correct home for one class of addition, +//! defined by what must already be true at that point in the build. //! //! | Gap | Runs | Exists for | //! | --- | --- | --- | +//! | after [`dockerfile_base_image`] | before `FROM` | re-declaring `ARG BASE_IMAGE` to build on your own base | //! | after [`dockerfile_base`] | before the first download | root CA, proxy, internal apt mirror | //! | after [`dockerfile_tools`] | after the toolchain, before `anvil-setup` | libraries a catalog tool needs to *compile* | //! | after [`dockerfile_setup`] | after the catalog is installed | what the repository's own checks need at run time | //! +//! The base image is the case that most needed a gap of its own: it is the +//! setting a repository is likeliest to want, and leaving it inside a region +//! would have made overriding it mean editing anvil's content — freezing every +//! pin in the file to buy one substitution. +//! //! A downstream catalog customizes by replacing individual regions, and //! inherits the rest: //! -//! - [`dockerfile_base`] / [`dockerfile_tools`] plus [`Artifact::with_body`] to -//! build on a different base OS or install the toolchain from a different -//! source. [`dockerfile_setup`] and [`dockerfile_entry`] are the contract with -//! the recipe and are rarely replaced. +//! - [`dockerfile_base_image`] / [`dockerfile_tools`] plus +//! [`Artifact::with_body`] to build on a different base OS or install the +//! toolchain from a different source. [`dockerfile_setup`] and +//! [`dockerfile_entry`] are the contract with the recipe and are rarely +//! replaced. //! - [`dockerignore`] alongside them when the replacement copies more of the //! tree, or the added paths never reach the build context. //! - [`hooks`] to supply credentials, or to resolve a published image through @@ -53,15 +60,18 @@ const RECIPE: &str = include_str!("../../../templates/justfiles/anvil/container. const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dockerfile.dockerignore"); /// Seeded into the Dockerfile when the file does not exist, and never -/// reconciled afterwards — it is the user's half of a composed file. +/// reconciled afterwards — it is the one line anvil cannot own. /// -/// It carries `# syntax=docker/dockerfile:1`, which `BuildKit` honors only as -/// the very first line of the file. A region's opening sentinel is a comment, -/// so the directive cannot live inside a region without being demoted to an -/// ordinary comment — silently, with the build falling back to the default -/// frontend and nothing failing to say so. +/// `# syntax=docker/dockerfile:1` pins the `BuildKit` frontend, and `BuildKit` +/// honors the directive only as the very first line, before any comment. A +/// region's opening sentinel *is* a comment, so the directive cannot live inside +/// one without being silently demoted, dropping the build to the default +/// frontend with nothing failing to say so. Everything else in the file is a +/// region, so anvil can keep it current. pub(crate) const DOCKERFILE_HEADER: &str = include_str!("../../../templates/anvil/container/Dockerfile.header"); +const DOCKERFILE_HEADER_REGION: &str = include_str!("../../../templates/anvil/container/Dockerfile.header.region"); +const DOCKERFILE_BASE_IMAGE: &str = include_str!("../../../templates/anvil/container/Dockerfile.baseimage.region"); const DOCKERFILE_BASE: &str = include_str!("../../../templates/anvil/container/Dockerfile.base.region"); const DOCKERFILE_TOOLS: &str = include_str!("../../../templates/anvil/container/Dockerfile.tools.region"); const DOCKERFILE_SETUP: &str = include_str!("../../../templates/anvil/container/Dockerfile.setup.region"); @@ -77,12 +87,15 @@ const DOCKERIGNORE_PATH: &str = ".anvil/container/Dockerfile.dockerignore"; /// The region ids anvil owns inside [`DOCKERFILE_PATH`], in the order a valid /// Dockerfile must carry them. /// -/// Order is load-bearing in a way no other managed-region host is: `FROM` must -/// precede every instruction, the toolchain must exist before `anvil-setup` -/// runs, and `WORKDIR`/`CMD` close the file. The engine checks the on-disk -/// sequence against this list and refuses rather than emitting a Dockerfile -/// that is silently wrong. +/// Order is load-bearing in a way no other managed-region host is: the base +/// image argument must precede the `FROM` that consumes it, `FROM` must precede +/// every instruction, the toolchain must exist before `anvil-setup` runs, and +/// `WORKDIR`/`CMD` close the file. The engine checks the on-disk sequence +/// against this list and refuses rather than emitting a Dockerfile that is +/// silently wrong. pub(crate) const DOCKERFILE_REGION_ORDER: &[&str] = &[ + "anvil-container-header", + "anvil-container-base-image", "anvil-container-base", "anvil-container-tools", "anvil-container-setup", @@ -98,6 +111,8 @@ pub fn all() -> Vec { vec![ recipe(), dockerignore(), + dockerfile_header(), + dockerfile_base_image(), dockerfile_base(), dockerfile_tools(), dockerfile_setup(), @@ -120,19 +135,31 @@ fn dockerfile_region(id: &'static str, body: &'static str) -> Artifact { }) } -/// The base image and the pinned tool versions: a digest-pinned base tracking -/// the Linux CI runner, the four download pins, and the environment every later -/// region depends on. +/// The file's explanatory header: what the regions are, which gap takes what, +/// and why the parser directive sits outside them. /// -/// A downstream catalog that needs a different base OS replaces this and -/// usually [`dockerfile_tools`] with it: +/// A region rather than part of the scaffold, so a correction to the guidance +/// reaches repositories that already generated the file. Scaffold content is +/// written once and never reconciled, which makes a mistake in it permanent. +#[must_use] +pub fn dockerfile_header() -> Artifact { + dockerfile_region("anvil-container-header", DOCKERFILE_HEADER_REGION) +} + +/// The default base image, digest-pinned, alone in its own region. /// -/// ```ignore -/// catalog.replace_artifact( -/// artifacts::container::dockerfile_base() -/// .with_body(include_str!("../templates/base.dockerfile")), -/// ) -/// ``` +/// Separate from [`dockerfile_base`] so the gap between them is a place to +/// override it: a second `ARG BASE_IMAGE=…` there wins over this default, and +/// `FROM` in the next region consumes the repository's value. That is what lets +/// a repository choose its own base **without** editing anvil's content, which +/// would otherwise freeze every pin in the file at the moment of the edit. +#[must_use] +pub fn dockerfile_base_image() -> Artifact { + dockerfile_region("anvil-container-base-image", DOCKERFILE_BASE_IMAGE) +} + +/// `FROM`, the four download pins, and the environment every later region +/// depends on. #[must_use] pub fn dockerfile_base() -> Artifact { dockerfile_region("anvil-container-base", DOCKERFILE_BASE) @@ -246,18 +273,28 @@ mod tests { .collect() } + /// Every region body, in the order the engine enforces. + const REGION_BODIES: [&str; 6] = [ + DOCKERFILE_HEADER_REGION, + DOCKERFILE_BASE_IMAGE, + DOCKERFILE_BASE, + DOCKERFILE_TOOLS, + DOCKERFILE_SETUP, + DOCKERFILE_ENTRY, + ]; + /// The Dockerfile as a fresh repository first receives it: the seeded - /// header followed by every region body in the order the engine enforces. + /// directive followed by every region body in order. fn composed_dockerfile() -> String { let mut out = DOCKERFILE_HEADER.to_owned(); - for body in [DOCKERFILE_BASE, DOCKERFILE_TOOLS, DOCKERFILE_SETUP, DOCKERFILE_ENTRY] { + for body in REGION_BODIES { out.push_str(body); } out } #[test] - fn group_is_two_owned_files_and_four_dockerfile_regions() { + fn group_is_two_owned_files_and_six_dockerfile_regions() { let all = all(); let owned: Vec<_> = all .iter() @@ -293,9 +330,13 @@ mod tests { #[test] fn no_region_body_carries_the_parser_directive() { // Inside a region the directive would sit below an opening sentinel -- - // a comment -- and be silently demoted. - for body in [DOCKERFILE_BASE, DOCKERFILE_TOOLS, DOCKERFILE_SETUP, DOCKERFILE_ENTRY] { - assert!(!body.contains("# syntax="), "a region body must not carry the parser directive"); + // a comment -- and be silently demoted. Mentioning it in prose is fine; + // what must not appear is a line that *is* the directive. + for body in REGION_BODIES { + assert!( + !body.lines().any(|line| line.starts_with("# syntax=")), + "a region body must not carry the parser directive as a line" + ); } } @@ -321,23 +362,58 @@ mod tests { .find(needle) .unwrap_or_else(|| panic!("missing from composed Dockerfile: {needle}")) }; + assert!(at("ARG BASE_IMAGE=") < at("FROM ${BASE_IMAGE}")); assert!(at("FROM ${BASE_IMAGE}") < at("RUN apt-get update")); assert!(at("RUN apt-get update") < at("just anvil-setup binstall")); assert!(at("just anvil-setup binstall") < at("WORKDIR /workspace")); } + #[test] + fn the_base_image_is_overridable_without_editing_a_region() { + // The whole point of giving the default its own region: a repository + // that wants another base re-declares the argument in the gap that + // follows, where a later declaration wins, instead of editing anvil's + // content and freezing every other pin in the file. + assert!( + DOCKERFILE_BASE_IMAGE.contains("ARG BASE_IMAGE="), + "the default must live in its own region" + ); + assert!( + !DOCKERFILE_BASE_IMAGE.contains("FROM "), + "FROM must not share the region, or there is no gap to override in" + ); + assert!( + DOCKERFILE_BASE.starts_with("FROM ${BASE_IMAGE}"), + "the consuming FROM must open the next region: {}", + DOCKERFILE_BASE.lines().next().unwrap_or_default() + ); + } + #[test] fn base_image_is_digest_pinned() { // A floating base tag can change underneath an identity hash that // claims to name fixed content, which would make every cached image a // potential lie. - let base = DOCKERFILE_BASE + let base = DOCKERFILE_BASE_IMAGE .lines() .find(|line| line.starts_with("ARG BASE_IMAGE=")) - .expect("the base region must declare a default BASE_IMAGE"); + .expect("the base-image region must declare a default BASE_IMAGE"); assert!(base.contains("@sha256:"), "BASE_IMAGE must be digest-pinned: {base}"); } + #[test] + fn only_the_parser_directive_is_left_outside_the_regions() { + // Anything anvil owns that sits outside a region can never be corrected + // on a repository that has already generated the file, because the + // scaffold is written once and never reconciled. Exactly one line has + // to pay that price. + assert_eq!( + DOCKERFILE_HEADER.lines().collect::>(), + ["# syntax=docker/dockerfile:1"], + "the scaffold must carry the parser directive and nothing else" + ); + } + #[test] fn build_context_admits_only_what_the_image_copies() { assert!(DOCKERIGNORE.contains("!justfiles")); @@ -847,15 +923,15 @@ mod tests { // rewrites the base and tool layers and inherits the catalog install // and the entry contract, instead of forking the whole Dockerfile and // freezing every pin in it. - let replaced = dockerfile_base().with_body("FROM example.invalid/base\n"); + let replaced = dockerfile_base_image().with_body("ARG BASE_IMAGE=example.invalid/base\n"); match &replaced { Artifact::Region(spec) => { - assert_eq!(spec.id.as_str(), "anvil-container-base"); + assert_eq!(spec.id.as_str(), "anvil-container-base-image"); assert_eq!(spec.host, HostSelector::Path(DOCKERFILE_PATH.to_owned())); } Artifact::OwnedFile(_) => panic!("the Dockerfile regions must stay regions"), } - assert_eq!(replaced.body(), "FROM example.invalid/base\n"); + assert_eq!(replaced.body(), "ARG BASE_IMAGE=example.invalid/base\n"); assert_eq!(dockerfile_setup().body(), DOCKERFILE_SETUP); } } diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 08f635f7..845a6770 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -237,12 +237,12 @@ //! ### Customizing the image //! //! `.anvil/container/Dockerfile` is a **user-composed file with managed -//! regions**: anvil owns four regions inside it and keeps them current, and the -//! three gaps between them are the repository's. Add extra packages in the gap -//! that suits when they are needed -- before the first download for a root CA -//! or a proxy, before `anvil-setup` for libraries a catalog tool compiles -//! against, after it for what the checks need at run time. Adding in a gap -//! leaves anvil's content alone, so base and tool-pin bumps keep landing; +//! regions**: anvil owns six regions inside it and keeps them current, and the +//! gaps between them are the repository's. Add to the gap that matches when the +//! addition is needed -- re-declare `ARG BASE_IMAGE` to build on another base, +//! a root CA or proxy before the first download, libraries a catalog tool +//! compiles against before `anvil-setup`, run-time tools after it. Adding in a +//! gap leaves anvil's content alone, so base and tool-pin bumps keep landing; //! editing inside a region is preserved rather than overwritten, but freezes //! those pins at the moment of the edit, which is why the gaps exist. //! diff --git a/crates/cargo-anvil/src/region.rs b/crates/cargo-anvil/src/region.rs index 717c0ac8..9c8f9f50 100644 --- a/crates/cargo-anvil/src/region.rs +++ b/crates/cargo-anvil/src/region.rs @@ -41,6 +41,15 @@ pub enum RegionPlacement { Start, /// Place the region after user content. End, + /// Insert a *new* region at this byte offset. An existing region is still + /// updated where it is found, so this only decides where an absent one + /// lands. + /// + /// Needed by hosts whose region order is semantic: appending a newly added + /// region at end-of-file would put it after regions it must precede, which + /// for a Dockerfile means `FROM` below the layers that depend on it. The + /// caller knows the declared order, so it computes the offset. + At(usize), } impl CommentSyntax { @@ -207,6 +216,26 @@ pub fn upsert_region_with_placement( return Ok(prepend_region(text, &rendered)); } + if let RegionPlacement::At(offset) = placement { + let offset = offset.min(text.len()); + // Land on a line boundary. An offset in the middle of a line would + // split it around the sentinels, which for a Dockerfile is the + // difference between a valid instruction and two invalid halves. + let offset = text[offset..].find('\n').map_or(text.len(), |index| offset + index + 1); + let (before, after) = text.split_at(offset); + let mut out = String::with_capacity(text.len() + rendered.len() + 2); + out.push_str(before); + if !before.is_empty() && !before.ends_with("\n\n") { + out.push('\n'); + } + out.push_str(&rendered); + if !after.is_empty() && !after.starts_with('\n') { + out.push('\n'); + } + out.push_str(after); + return Ok(out); + } + // No region present — append at the end with one blank line of separation // if the file is non-empty and doesn't end in two newlines. let mut out = String::with_capacity(text.len() + rendered.len() + 1); diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 5ddb57d6..ec1b5ef5 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -386,7 +386,10 @@ fn push_region_at( return Ok(()); } let current = hosts.get_or_read(repo_root, &host)?; - let placement = region_placement(spec.id.as_str()); + let placement = composed_host_spec(&host).map_or_else( + || region_placement(spec.id.as_str()), + |(scaffold, order)| composed_placement(order, scaffold, spec.id.as_str(), current.as_deref()), + ); let body = match delta_region_body(current.as_deref(), spec) { DeltaRegionBody::Managed => spec.body.as_str(), DeltaRegionBody::PreserveRepositoryKey => { @@ -436,6 +439,44 @@ fn push_region_at( Ok(()) } +/// Where a region belongs inside a composed host whose order is semantic. +/// +/// An existing region is updated where it is found, so this only decides where +/// an *absent* one lands — which matters whenever anvil adds a region to a +/// release. Appending it at end-of-file, the default for every other host, +/// would put it after regions it must precede: a newly added base-image +/// argument would land below the `FROM` that consumes it. Without this, adding +/// a region would either corrupt or (with the composition check) refuse every +/// file that already exists. +fn composed_placement(order: &[&str], scaffold: &str, id: &str, text: Option<&str>) -> RegionPlacement { + let Some(text) = text else { + return RegionPlacement::End; + }; + if matches!(find_region(text, id, CommentSyntax::Hash), Ok(Some(_))) { + // Present: `upsert_region` replaces it where it is, and the offset is + // never consulted. + return RegionPlacement::End; + } + let Some(position) = order.iter().position(|candidate| *candidate == id) else { + return RegionPlacement::End; + }; + // The nearest declared predecessor that is actually in the file. Anything + // after it and before the next present region is the gap this region opens. + for earlier in order[..position].iter().rev() { + if let Ok(Some(region)) = find_region(text, earlier, CommentSyntax::Hash) { + return RegionPlacement::At(region.end_line.end); + } + } + // Nothing precedes it, so it goes to the top -- but below the scaffold, + // which for a Dockerfile is the `# syntax=` parser directive that BuildKit + // honors only as the very first line. + RegionPlacement::At(if text.starts_with(scaffold.trim_end_matches('\n')) { + scaffold.trim_end_matches('\n').len() + } else { + 0 + }) +} + fn region_placement(region_id: &str) -> RegionPlacement { if region_id == DELTA_REGION_ID { RegionPlacement::Start @@ -543,22 +584,13 @@ fn composed_host_state(order: &[&str], host_relpath: &str, text: &str, manifest: }; } - if present.len() != order.len() { - // A partially composed file. The missing regions would be appended at - // end-of-file, which lands them after regions they must precede. - let missing: Vec<&str> = order - .iter() - .copied() - .filter(|id| !present.iter().any(|(present_id, _)| present_id == id)) - .collect(); - return ComposedHostState::Unsafe(format!( - "it carries some of anvil's regions but not all of them (missing: {}). Restoring the \ - missing ones by appending would place them after regions they must precede. Delete \ - the file and re-run to have a complete one written", - missing.join(", ") - )); - } - + // A file carrying some regions but not all is what every adopter has the + // first time anvil adds one to the set -- a normal upgrade, not damage. + // `composed_placement` inserts each missing region at its declared position + // rather than at end-of-file, so the result stays ordered and the gap + // content is untouched. Refusing here would make every future region + // addition mean "delete your file". + // // The same regions, in the order the file carries them. Comparing the two // sequences rather than adjacent offsets keeps the check free of an // ordering operator whose boundary cannot be exercised: two distinct diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region index d3870f9e..2c2f57e1 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.base.region @@ -1,13 +1,3 @@ -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. -# -# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a -# change here renames the image -- but it does not resolve or validate the -# base, and a floating tag could therefore change underneath a reference that -# claims to name fixed content. -ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -24,11 +14,3 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_HOME=/usr/local/rustup \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs before anvil's first download. Put whatever -# the image needs in order to reach the network here: a corporate root CA, -# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. -# Without it, a TLS-intercepting proxy makes every `curl` in the next region -# fail and the image cannot be built at all. -# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region new file mode 100644 index 00000000..dac79061 --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region @@ -0,0 +1,8 @@ +# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). +# `anvil-setup binstall` installs prebuilt binaries that need that runner's +# glibc, so this moves when the runner does. Digest-pinned because a floating +# tag can change under an image reference that claims to name fixed content. +# +# To build on another base, re-declare this in the gap below rather than editing +# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header index d2103f0f..d6a937b3 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header @@ -1,29 +1 @@ # syntax=docker/dockerfile:1 -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them -# current, so base and tool-pin bumps arrive on their own. Everything outside -# them is yours and is preserved byte-for-byte, including this header. -# -# Add your own instructions in the gaps between the regions rather than inside -# one. Anvil will not overwrite an edit you make inside a region -- it offers -# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and -# is the trap: the region carries the base digest and four tool pins, so an edit -# there quietly freezes them at today's values while the image tag keeps -# resolving, because the tag hashes your file. The gaps exist so that nothing -# you legitimately need to add ever requires touching them. -# -# Each region ends by describing what the gap after it is for, because position -# decides whether a line works at all: a corporate root CA has to land before -# the first download, and a library needed to *compile* a catalog tool has to -# land before `anvil-setup` runs. -# -# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit -# honours the directive only when nothing precedes it -- not even a comment. -# That is also why it sits out here rather than inside a region: a region's -# opening sentinel is a comment, and would silently demote the directive to an -# ordinary one, leaving the build on the default frontend with nothing failing -# to say so. diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region new file mode 100644 index 00000000..eb403125 --- /dev/null +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region @@ -0,0 +1,19 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Anvil execution image. +# +# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps +# between them are yours and are preserved. Add to a gap, not inside a region: +# anvil will not overwrite an edit inside one, so editing there silently freezes +# the base digest and the tool pins at that moment. +# +# Which gap, since position decides whether a line works at all: +# +# after base-image re-declare ARG BASE_IMAGE to build on your own base +# after base root CA, proxy, apt mirror -- anything the first download needs +# after tools libraries a catalog tool needs to compile from source +# after setup what your own checks need at run time +# +# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, +# and a region sentinel is a comment. diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region index 10fb615b..4deca8f6 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region @@ -1,26 +1,21 @@ -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. +# Install the pinned toolchain and cargo subcommands from the generated catalog. +# The whole recipe tree is copied because `just` has to parse it, and the whole +# tree is hashed into the image tag: `anvil-setup` reaches the install recipes +# through the tier, group and check recipes, so any of them can change what this +# layer installs. # -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. +# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do +# not compile from source on every pinned toolchain. # # The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. +# secret never lands in a layer, but anything the install *writes* with it is +# ordinary content, and the `chmod` below would publish it world-readable. +# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. # # `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. +# named volume over each, and an engine seeds a new volume from the image path +# it covers. A path that does not exist seeds as root-owned 0755, which the +# `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -30,11 +25,3 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the catalog is installed. Put what your -# own checks need at run time here: a database client, a protobuf compiler, a -# linter that is not a cargo subcommand. It is also the cheapest place to add -# anything, because an edit here invalidates one layer rather than the -# multi-minute `anvil-setup` above. -# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region index a7e8979d..639b3a71 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region @@ -44,10 +44,3 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the toolchain is in place and before -# `anvil-setup` installs the catalog. Put system libraries a catalog tool -# needs in order to *compile* here: `binstall` falls back to a source build -# when a pinned tool publishes no prebuilt for this platform, and that build -# links against whatever headers the image has. -# --------------------------------------------------------------------------- diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 1856856f..ca605a53 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -432,44 +432,53 @@ fn an_edited_superseded_dockerfile_is_never_overwritten() { ); } -/// A half-composed file: some of anvil's regions present, others missing. -/// `upsert_region` appends a missing region at end-of-file, which would land it -/// after regions it must precede, so this has to be refused too. +/// A half-composed file — some of anvil's regions present, others missing — is +/// what an adopter has the first time anvil adds a region to the set. That is a +/// normal upgrade: each missing region is inserted at its declared position, so +/// the file stays ordered and the repository's own content between the regions +/// survives untouched. #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] #[test] -fn a_partially_composed_dockerfile_is_refused() { +fn a_newly_added_region_is_inserted_in_order_not_appended() { let tmp = generated_tree(); let root = tmp.path(); let dockerfile = root.join(".anvil/container/Dockerfile"); - // Drop the base region, keeping the rest. Appending it back would put + // Drop the base region and put repository content in the gaps around it, + // exactly as an adopter would have. Appending the region back would land // `FROM` after the layers that depend on it. let text = std::fs::read_to_string(&dockerfile).unwrap(); - let open = text.find("# >>> anvil-managed: anvil-container-base").unwrap(); + let open = text.find("# >>> anvil-managed: anvil-container-base\n").unwrap(); let close_marker = "# <<< anvil-managed: anvil-container-base\n"; let close = text.find(close_marker).unwrap() + close_marker.len(); - let without_base = format!("{}{}", &text[..open], &text[close..]); + let without_base = format!("{}# MINE-BEFORE\n{}# MINE-AFTER\n", &text[..open], &text[close..]); write(&dockerfile, &without_base); + let mut manifest = Manifest::load(root).unwrap(); + manifest + .regions + .retain(|key, _| !(key.host == ".anvil/container/Dockerfile" && key.id == "anvil-container-base")); + manifest.save(root).unwrap(); let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); - - assert_eq!( - std::fs::read_to_string(&dockerfile).unwrap(), - without_base, - "a refused host must not be modified" - ); - let refusals: Vec<&String> = outcome - .plan - .refusals() - .iter() - .filter(|r| r.contains(".anvil/container/Dockerfile")) - .collect(); - assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); assert!( - refusals[0].contains("anvil-container-base"), - "the diagnostic must name the missing region: {}", - refusals[0] + outcome.plan.refusals().is_empty(), + "adding a region is an upgrade, not a refusal: {:?}", + outcome.plan.refusals() ); + + let composed = std::fs::read_to_string(&dockerfile).unwrap(); + let at = |needle: &str| { + composed + .find(needle) + .unwrap_or_else(|| panic!("missing from recomposed Dockerfile: {needle}\n{composed}")) + }; + // Restored in order, not at the end. + assert!(at("ARG BASE_IMAGE=") < at("FROM ${BASE_IMAGE}")); + assert!(at("FROM ${BASE_IMAGE}") < at("RUN apt-get update")); + assert!(at("RUN apt-get update") < at("WORKDIR /workspace")); + // And the repository's own lines are still there. + assert!(composed.contains("# MINE-BEFORE"), "gap content before the region must survive"); + assert!(composed.contains("# MINE-AFTER"), "gap content after the region must survive"); } /// The Dockerfile is the first managed-region host whose region *order* is @@ -490,7 +499,7 @@ fn a_reordered_composed_dockerfile_is_refused_with_one_diagnostic() { // Move the base region below the others, which is exactly the mistake a // repository makes by dropping its own instructions above `FROM`. let text = std::fs::read_to_string(&dockerfile).unwrap(); - let open = text.find("# >>> anvil-managed: anvil-container-base").unwrap(); + let open = text.find("# >>> anvil-managed: anvil-container-base\n").unwrap(); let close_marker = "# <<< anvil-managed: anvil-container-base\n"; let close = text.find(close_marker).unwrap() + close_marker.len(); let base = &text[open..close]; diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index f983cf36..8c20e192 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4,46 +4,41 @@ expression: render_tree(tmp.path()) --- === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 + +# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. # # Anvil execution image. # -# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them -# current, so base and tool-pin bumps arrive on their own. Everything outside -# them is yours and is preserved byte-for-byte, including this header. +# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps +# between them are yours and are preserved. Add to a gap, not inside a region: +# anvil will not overwrite an edit inside one, so editing there silently freezes +# the base digest and the tool pins at that moment. # -# Add your own instructions in the gaps between the regions rather than inside -# one. Anvil will not overwrite an edit you make inside a region -- it offers -# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and -# is the trap: the region carries the base digest and four tool pins, so an edit -# there quietly freezes them at today's values while the image tag keeps -# resolving, because the tag hashes your file. The gaps exist so that nothing -# you legitimately need to add ever requires touching them. +# Which gap, since position decides whether a line works at all: # -# Each region ends by describing what the gap after it is for, because position -# decides whether a line works at all: a corporate root CA has to land before -# the first download, and a library needed to *compile* a catalog tool has to -# land before `anvil-setup` runs. +# after base-image re-declare ARG BASE_IMAGE to build on your own base +# after base root CA, proxy, apt mirror -- anything the first download needs +# after tools libraries a catalog tool needs to compile from source +# after setup what your own checks need at run time # -# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit -# honours the directive only when nothing precedes it -- not even a comment. -# That is also why it sits out here rather than inside a region: a region's -# opening sentinel is a comment, and would silently demote the directive to an -# ordinary one, leaving the build on the default frontend with nothing failing -# to say so. - -# >>> anvil-managed: anvil-container-base -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. +# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, +# and a region sentinel is a comment. +# <<< anvil-managed: anvil-container-header + +# >>> anvil-managed: anvil-container-base-image +# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). +# `anvil-setup binstall` installs prebuilt binaries that need that runner's +# glibc, so this moves when the runner does. Digest-pinned because a floating +# tag can change under an image reference that claims to name fixed content. # -# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a -# change here renames the image -- but it does not resolve or validate the -# base, and a floating tag could therefore change underneath a reference that -# claims to name fixed content. +# To build on another base, re-declare this in the gap below rather than editing +# here: a later ARG wins, and anvil keeps updating the pins you did not touch. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +# <<< anvil-managed: anvil-container-base-image + +# >>> anvil-managed: anvil-container-base FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -60,14 +55,6 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_HOME=/usr/local/rustup \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs before anvil's first download. Put whatever -# the image needs in order to reach the network here: a corporate root CA, -# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. -# Without it, a TLS-intercepting proxy makes every `curl` in the next region -# fail and the image cannot be built at all. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-base # >>> anvil-managed: anvil-container-tools @@ -117,39 +104,27 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the toolchain is in place and before -# `anvil-setup` installs the catalog. Put system libraries a catalog tool -# needs in order to *compile* here: `binstall` falls back to a source build -# when a pinned tool publishes no prebuilt for this platform, and that build -# links against whatever headers the image has. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. +# Install the pinned toolchain and cargo subcommands from the generated catalog. +# The whole recipe tree is copied because `just` has to parse it, and the whole +# tree is hashed into the image tag: `anvil-setup` reaches the install recipes +# through the tier, group and check recipes, so any of them can change what this +# layer installs. # -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. +# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do +# not compile from source on every pinned toolchain. # # The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. +# secret never lands in a layer, but anything the install *writes* with it is +# ordinary content, and the `chmod` below would publish it world-readable. +# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. # # `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. +# named volume over each, and an engine seeds a new volume from the image path +# it covers. A path that does not exist seeds as root-owned 0755, which the +# `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -159,14 +134,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the catalog is installed. Put what your -# own checks need at run time here: a database client, a protobuf compiler, a -# linter that is not a cargo subcommand. It is also the cheapest place to add -# anything, because an edit here invalidates one layer rather than the -# multi-minute `anvil-setup` above. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-setup # >>> anvil-managed: anvil-container-entry diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 10bd7567..b910cfa5 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4,46 +4,41 @@ expression: render_tree(tmp.path()) --- === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 + +# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. # # Anvil execution image. # -# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them -# current, so base and tool-pin bumps arrive on their own. Everything outside -# them is yours and is preserved byte-for-byte, including this header. +# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps +# between them are yours and are preserved. Add to a gap, not inside a region: +# anvil will not overwrite an edit inside one, so editing there silently freezes +# the base digest and the tool pins at that moment. # -# Add your own instructions in the gaps between the regions rather than inside -# one. Anvil will not overwrite an edit you make inside a region -- it offers -# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and -# is the trap: the region carries the base digest and four tool pins, so an edit -# there quietly freezes them at today's values while the image tag keeps -# resolving, because the tag hashes your file. The gaps exist so that nothing -# you legitimately need to add ever requires touching them. +# Which gap, since position decides whether a line works at all: # -# Each region ends by describing what the gap after it is for, because position -# decides whether a line works at all: a corporate root CA has to land before -# the first download, and a library needed to *compile* a catalog tool has to -# land before `anvil-setup` runs. +# after base-image re-declare ARG BASE_IMAGE to build on your own base +# after base root CA, proxy, apt mirror -- anything the first download needs +# after tools libraries a catalog tool needs to compile from source +# after setup what your own checks need at run time # -# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit -# honours the directive only when nothing precedes it -- not even a comment. -# That is also why it sits out here rather than inside a region: a region's -# opening sentinel is a comment, and would silently demote the directive to an -# ordinary one, leaving the build on the default frontend with nothing failing -# to say so. - -# >>> anvil-managed: anvil-container-base -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. +# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, +# and a region sentinel is a comment. +# <<< anvil-managed: anvil-container-header + +# >>> anvil-managed: anvil-container-base-image +# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). +# `anvil-setup binstall` installs prebuilt binaries that need that runner's +# glibc, so this moves when the runner does. Digest-pinned because a floating +# tag can change under an image reference that claims to name fixed content. # -# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a -# change here renames the image -- but it does not resolve or validate the -# base, and a floating tag could therefore change underneath a reference that -# claims to name fixed content. +# To build on another base, re-declare this in the gap below rather than editing +# here: a later ARG wins, and anvil keeps updating the pins you did not touch. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +# <<< anvil-managed: anvil-container-base-image + +# >>> anvil-managed: anvil-container-base FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -60,14 +55,6 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_HOME=/usr/local/rustup \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs before anvil's first download. Put whatever -# the image needs in order to reach the network here: a corporate root CA, -# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. -# Without it, a TLS-intercepting proxy makes every `curl` in the next region -# fail and the image cannot be built at all. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-base # >>> anvil-managed: anvil-container-tools @@ -117,39 +104,27 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the toolchain is in place and before -# `anvil-setup` installs the catalog. Put system libraries a catalog tool -# needs in order to *compile* here: `binstall` falls back to a source build -# when a pinned tool publishes no prebuilt for this platform, and that build -# links against whatever headers the image has. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. +# Install the pinned toolchain and cargo subcommands from the generated catalog. +# The whole recipe tree is copied because `just` has to parse it, and the whole +# tree is hashed into the image tag: `anvil-setup` reaches the install recipes +# through the tier, group and check recipes, so any of them can change what this +# layer installs. # -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. +# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do +# not compile from source on every pinned toolchain. # # The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. +# secret never lands in a layer, but anything the install *writes* with it is +# ordinary content, and the `chmod` below would publish it world-readable. +# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. # # `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. +# named volume over each, and an engine seeds a new volume from the image path +# it covers. A path that does not exist seeds as root-owned 0755, which the +# `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -159,14 +134,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the catalog is installed. Put what your -# own checks need at run time here: a database client, a protobuf compiler, a -# linter that is not a cargo subcommand. It is also the cheapest place to add -# anything, because an edit here invalidates one layer rather than the -# multi-minute `anvil-setup` above. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-setup # >>> anvil-managed: anvil-container-entry diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index ca4fc4d6..b081d41a 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -4,46 +4,41 @@ expression: render_tree(tmp.path()) --- === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 + +# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. # # Anvil execution image. # -# The `anvil-managed:` regions below belong to cargo-anvil: it keeps them -# current, so base and tool-pin bumps arrive on their own. Everything outside -# them is yours and is preserved byte-for-byte, including this header. +# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps +# between them are yours and are preserved. Add to a gap, not inside a region: +# anvil will not overwrite an edit inside one, so editing there silently freezes +# the base digest and the tool pins at that moment. # -# Add your own instructions in the gaps between the regions rather than inside -# one. Anvil will not overwrite an edit you make inside a region -- it offers -# its version in a `.anvil-proposed` sibling instead -- which sounds helpful and -# is the trap: the region carries the base digest and four tool pins, so an edit -# there quietly freezes them at today's values while the image tag keeps -# resolving, because the tag hashes your file. The gaps exist so that nothing -# you legitimately need to add ever requires touching them. +# Which gap, since position decides whether a line works at all: # -# Each region ends by describing what the gap after it is for, because position -# decides whether a line works at all: a corporate root CA has to land before -# the first download, and a library needed to *compile* a catalog tool has to -# land before `anvil-setup` runs. +# after base-image re-declare ARG BASE_IMAGE to build on your own base +# after base root CA, proxy, apt mirror -- anything the first download needs +# after tools libraries a catalog tool needs to compile from source +# after setup what your own checks need at run time # -# `# syntax=` must stay on line 1. It pins the BuildKit frontend, and BuildKit -# honours the directive only when nothing precedes it -- not even a comment. -# That is also why it sits out here rather than inside a region: a region's -# opening sentinel is a comment, and would silently demote the directive to an -# ordinary one, leaving the build on the default frontend with nothing failing -# to say so. - -# >>> anvil-managed: anvil-container-base -# The base tracks the Linux runner the generated workflows use -# (`ubuntu-latest`, currently 24.04). `anvil-setup binstall` installs the -# catalog as prebuilt binaries, which require that runner's glibc; it is -# backward but not forward compatible, so the pin moves when the runner does. +# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, +# and a region sentinel is a comment. +# <<< anvil-managed: anvil-container-header + +# >>> anvil-managed: anvil-container-base-image +# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). +# `anvil-setup binstall` installs prebuilt binaries that need that runner's +# glibc, so this moves when the runner does. Digest-pinned because a floating +# tag can change under an image reference that claims to name fixed content. # -# BASE_IMAGE stays digest-pinned. `anvil-container-tag` hashes this file, so a -# change here renames the image -- but it does not resolve or validate the -# base, and a floating tag could therefore change underneath a reference that -# claims to name fixed content. +# To build on another base, re-declare this in the gap below rather than editing +# here: a later ARG wins, and anvil keeps updating the pins you did not touch. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +# <<< anvil-managed: anvil-container-base-image + +# >>> anvil-managed: anvil-container-base FROM ${BASE_IMAGE} ARG JUST_VERSION=1.56.0 @@ -60,14 +55,6 @@ ENV DEBIAN_FRONTEND=noninteractive \ RUSTUP_HOME=/usr/local/rustup \ RUSTUP_NO_UPDATE_CHECK=1 \ PATH=/usr/local/cargo/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs before anvil's first download. Put whatever -# the image needs in order to reach the network here: a corporate root CA, -# `http_proxy` / `no_proxy`, apt sources pointed at an internal mirror. -# Without it, a TLS-intercepting proxy makes every `curl` in the next region -# fail and the image cannot be built at all. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-base # >>> anvil-managed: anvil-container-tools @@ -117,39 +104,27 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ && chmod 755 "${CARGO_HOME}/bin/cargo-binstall" \ && rm /tmp/cargo-binstall.tgz -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the toolchain is in place and before -# `anvil-setup` installs the catalog. Put system libraries a catalog tool -# needs in order to *compile* here: `binstall` falls back to a source build -# when a pinned tool publishes no prebuilt for this platform, and that build -# links against whatever headers the image has. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated -# catalog. The whole recipe tree is copied because `just` has to parse it, and -# the whole tree is hashed into the tag: `anvil-setup` reaches the install -# recipes through the tier, group and check recipes, so any of them can change -# what this layer installs. The synthetic Justfile below imports only the anvil -# tree, avoiding repository-specific imports that may not exist yet. +# Install the pinned toolchain and cargo subcommands from the generated catalog. +# The whole recipe tree is copied because `just` has to parse it, and the whole +# tree is hashed into the image tag: `anvil-setup` reaches the install recipes +# through the tier, group and check recipes, so any of them can change what this +# layer installs. # -# `binstall` matches what CI passes. It is not just a speed-up: some catalog -# tools do not compile from source on every pinned toolchain, so a source -# install can fail here while CI stays green. +# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do +# not compile from source on every pinned toolchain. # # The credential files are removed in the same layer as the install. A build -# secret is mounted, never committed to a layer, but anything the install -# *writes* with it is ordinary content -- and the `chmod` on the next line -# would otherwise publish it world-readable. Deleting them in a later `RUN` -# would not help: the earlier layer still carries the file. +# secret never lands in a layer, but anything the install *writes* with it is +# ordinary content, and the `chmod` below would publish it world-readable. +# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. # # `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each. An engine seeds a new volume from the image path it -# covers, and a path that does not exist is seeded as a root-owned 0755 -# directory -- which the `--user` mapping on Linux then cannot write, so the -# first cargo fetch fails with EACCES. Creating them here means the widened -# permissions are what the volume inherits. +# named volume over each, and an engine seeds a new volume from the image path +# it covers. A path that does not exist seeds as root-owned 0755, which the +# `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles COPY rust-toolchain.toml ./ @@ -159,14 +134,6 @@ RUN printf "import 'justfiles/anvil/mod.just'\n" > Justfile \ && rm -f "${CARGO_HOME}/credentials" "${CARGO_HOME}/credentials.toml" "${HOME}/.netrc" \ && mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ && chmod -R a+rwX "${CARGO_HOME}" "${RUSTUP_HOME}" - -# --- your turn ------------------------------------------------------------- -# The gap below this region runs after the catalog is installed. Put what your -# own checks need at run time here: a database client, a protobuf compiler, a -# linter that is not a cargo subcommand. It is also the cheapest place to add -# anything, because an edit here invalidates one layer rather than the -# multi-minute `anvil-setup` above. -# --------------------------------------------------------------------------- # <<< anvil-managed: anvil-container-setup # >>> anvil-managed: anvil-container-entry From 8f5ce5341c920c91a83d25c0e873042a51449d81 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 08:54:38 +0200 Subject: [PATCH 56/81] test(anvil): cover the ordered-insertion placement and its splice Mutation testing and the coverage gate both flagged the new `RegionPlacement::At` path: it had no unit tests, so three mutants survived in the splice (the line-boundary snap, the leading-blank guard, and the trailing separator) and several branches of `composed_placement` were never taken. Both are now covered directly rather than through the integration tests, which reached the happy path only. Two of the fixtures I first wrote did not test what their names claimed. One asserted that a missing region "skips predecessors that are absent" while arranging for the very first candidate to be present, so the skip never ran; the other expected no blank line after the closing sentinel, when the splice deliberately inserts one to keep the sentinel off the following instruction's line. Both corrected against observed behavior. --- crates/cargo-anvil/src/region.rs | 54 ++++++++++++++++++++ crates/cargo-anvil/src/run.rs | 85 ++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+) diff --git a/crates/cargo-anvil/src/region.rs b/crates/cargo-anvil/src/region.rs index 9c8f9f50..355bc088 100644 --- a/crates/cargo-anvil/src/region.rs +++ b/crates/cargo-anvil/src/region.rs @@ -476,6 +476,60 @@ mod tests { assert_eq!(new, "# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n"); } + /// `At` exists for hosts whose region order is semantic: a region added in a + /// later release has to land at its declared position, not at end-of-file. + /// The offset the caller computes points at the end of the preceding + /// region's sentinel, so the split must fall on the following line boundary. + #[test] + fn at_placement_inserts_after_the_line_containing_the_offset() { + let host = "# syntax=docker/dockerfile:1\nFROM base\n"; + // Offset lands mid-way through line 1; the region must go *after* that + // whole line, never inside it. + let new = upsert_region_with_placement(host, "x", "body\n", SYN, RegionPlacement::At(5)).unwrap(); + assert_eq!( + new, + "# syntax=docker/dockerfile:1\n\n# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\nFROM base\n" + ); + } + + #[test] + fn at_placement_into_empty_text_adds_no_leading_blank() { + let new = upsert_region_with_placement("", "x", "body\n", SYN, RegionPlacement::At(0)).unwrap(); + assert_eq!(new, "# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n"); + } + + #[test] + fn at_placement_does_not_double_the_separating_blank_line() { + // Preceding content already ends with a blank line, so no second one. + let host = "FROM base\n\nRUN later\n"; + let new = upsert_region_with_placement(host, "x", "body\n", SYN, RegionPlacement::At(10)).unwrap(); + assert_eq!( + new, + "FROM base\n\n# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\nRUN later\n" + ); + } + + #[test] + fn at_placement_separates_the_region_from_the_content_that_follows() { + // The tail does not start with a newline, so one is inserted -- without + // it the closing sentinel and the next instruction would share a line. + let host = "FROM base\nRUN later\n"; + let new = upsert_region_with_placement(host, "x", "body\n", SYN, RegionPlacement::At(0)).unwrap(); + assert_eq!( + new, + "FROM base\n\n# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\nRUN later\n" + ); + } + + #[test] + fn at_placement_updates_an_existing_region_where_it_already_is() { + // The offset only decides where an *absent* region lands. One already + // present is replaced in place, so a stale offset cannot move it. + let host = "FROM base\n# >>> anvil-managed: x\nold\n# <<< anvil-managed: x\nRUN later\n"; + let new = upsert_region_with_placement(host, "x", "new\n", SYN, RegionPlacement::At(0)).unwrap(); + assert_eq!(new, "FROM base\n# >>> anvil-managed: x\nnew\n# <<< anvil-managed: x\nRUN later\n"); + } + #[test] fn start_placement_prepends_absent_region() { let new = upsert_region_with_placement( diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index ec1b5ef5..54d64234 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -778,6 +778,91 @@ mod tests { fs::write(path, contents).unwrap(); } + /// `composed_placement` decides where an *absent* region lands in a host + /// whose order is semantic. Every branch matters: getting it wrong puts a + /// newly added region after ones it must precede, which for a Dockerfile + /// means `FROM` below the layers that depend on it. + mod composed_placement { + use super::*; + + const SCAFFOLD: &str = "# syntax=docker/dockerfile:1\n"; + const ORDER: &[&str] = &["a", "b", "c"]; + + fn region(id: &str, body: &str) -> String { + format!("# >>> anvil-managed: {id}\n{body}# <<< anvil-managed: {id}\n") + } + + #[test] + fn a_host_that_does_not_exist_yet_appends() { + // Nothing to order against; the regions are written in catalog + // order onto the scaffold. + assert_eq!(super::super::composed_placement(ORDER, SCAFFOLD, "b", None), RegionPlacement::End); + } + + #[test] + fn a_region_already_present_is_replaced_where_it_is() { + let host = format!("{SCAFFOLD}{}", region("b", "body\n")); + assert_eq!( + super::super::composed_placement(ORDER, SCAFFOLD, "b", Some(&host)), + RegionPlacement::End, + "an existing region is upserted in place, so the offset is never consulted" + ); + } + + #[test] + fn a_region_outside_the_declared_order_appends() { + let host = format!("{SCAFFOLD}{}", region("a", "body\n")); + assert_eq!( + super::super::composed_placement(ORDER, SCAFFOLD, "unknown", Some(&host)), + RegionPlacement::End + ); + } + + #[test] + fn a_missing_region_lands_after_its_nearest_present_predecessor() { + // `c` is absent and both `a` and `b` are present, so it must follow + // `b` -- the nearest, not merely the first. + let host = format!("{SCAFFOLD}{}{}", region("a", "first\n"), region("b", "second\n")); + let RegionPlacement::At(offset) = super::super::composed_placement(ORDER, SCAFFOLD, "c", Some(&host)) else { + panic!("a missing region with a present predecessor must be placed by offset"); + }; + assert_eq!(offset, host.len(), "it belongs after the close of `b`"); + } + + #[test] + fn a_missing_region_skips_predecessors_that_are_absent_too() { + // `c` is missing and so is its nearest predecessor `b`, so the + // search has to walk past `b` and anchor on `a`. A fixture where + // the first candidate matches would never exercise the skip. + let host = format!("{SCAFFOLD}{}", region("a", "first\n")); + let RegionPlacement::At(offset) = super::super::composed_placement(ORDER, SCAFFOLD, "c", Some(&host)) else { + panic!("expected an offset placement"); + }; + assert_eq!(offset, host.len(), "it belongs after the close of `a`, the only one present"); + } + + #[test] + fn the_first_region_lands_below_the_scaffold_never_above_it() { + // The scaffold is the parser directive, which BuildKit honors only + // as line 1. Placing the first region at byte 0 would push it down. + let host = format!("{SCAFFOLD}{}", region("b", "second\n")); + let RegionPlacement::At(offset) = super::super::composed_placement(ORDER, SCAFFOLD, "a", Some(&host)) else { + panic!("expected an offset placement"); + }; + assert_eq!(offset, SCAFFOLD.trim_end_matches('\n').len()); + assert!(offset > 0, "the directive must keep line 1"); + } + + #[test] + fn a_host_without_the_scaffold_places_the_first_region_at_the_top() { + let host = region("b", "second\n"); + assert_eq!( + super::super::composed_placement(ORDER, SCAFFOLD, "a", Some(&host)), + RegionPlacement::At(0) + ); + } + } + fn empty_workspace() -> TempDir { let tmp = TempDir::new().unwrap(); let root = tmp.path(); From 1a48f759c9f641a711373e4dd3bf45dda1796f3f Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 09:21:53 +0200 Subject: [PATCH 57/81] fix(anvil): stop ordered insertion skipping a line of the adopter's gap `RegionPlacement::At` advanced unconditionally to the next newline, but every caller passes an offset that is already a line start -- a preceding region's `end_line.end`. The advance therefore skipped that line, so a region added in a later release would have landed after the first line of the repository's gap content, splitting it around anvil's sentinels. Latent until now only because the two regions this branch added happened to be inserted where the following gap was empty. The snap now fires only for an offset that really is mid-line, where rounding forward still matters: splitting a line around the sentinels turns one valid instruction into two invalid halves. Covered at both levels: a unit test placing a region against a host with gap content directly after its predecessor, and the integration test now asserting the restored region sits above that content rather than inside it. Found by the PR reviewer bot; verified against the code before acting. --- crates/cargo-anvil/src/region.rs | 35 ++++++++++++++++--- crates/cargo-anvil/tests/container_upgrade.rs | 7 ++++ 2 files changed, 37 insertions(+), 5 deletions(-) diff --git a/crates/cargo-anvil/src/region.rs b/crates/cargo-anvil/src/region.rs index 355bc088..7102afab 100644 --- a/crates/cargo-anvil/src/region.rs +++ b/crates/cargo-anvil/src/region.rs @@ -218,10 +218,20 @@ pub fn upsert_region_with_placement( if let RegionPlacement::At(offset) = placement { let offset = offset.min(text.len()); - // Land on a line boundary. An offset in the middle of a line would - // split it around the sentinels, which for a Dockerfile is the - // difference between a valid instruction and two invalid halves. - let offset = text[offset..].find('\n').map_or(text.len(), |index| offset + index + 1); + // Snap to a line boundary only when the offset is not already on one. + // Callers point at the start of the line the region should displace + // (typically a preceding region's `end_line.end`); advancing + // unconditionally would skip that line, landing the region after the + // first line of the repository's gap content and splitting it. An + // offset that does fall mid-line is rounded forward, because splitting + // a line around the sentinels turns one valid instruction into two + // invalid halves. + let on_line_boundary = offset == 0 || text[..offset].ends_with('\n'); + let offset = if on_line_boundary { + offset + } else { + text[offset..].find('\n').map_or(text.len(), |index| offset + index + 1) + }; let (before, after) = text.split_at(offset); let mut out = String::with_capacity(text.len() + rendered.len() + 2); out.push_str(before); @@ -480,6 +490,21 @@ mod tests { /// later release has to land at its declared position, not at end-of-file. /// The offset the caller computes points at the end of the preceding /// region's sentinel, so the split must fall on the following line boundary. + #[test] + fn at_placement_on_a_line_boundary_does_not_skip_the_following_line() { + // The offset callers actually pass is a line start -- a preceding + // region's `end_line.end`. Advancing past the next newline would put + // the region after the first line of the gap and split it in two. + let host = "# >>> anvil-managed: a\nbody\n# <<< anvil-managed: a\n# my gap line\nRUN later\n"; + let offset = host.find("# my gap line").unwrap(); + let new = upsert_region_with_placement(host, "x", "body\n", SYN, RegionPlacement::At(offset)).unwrap(); + assert_eq!( + new, + "# >>> anvil-managed: a\nbody\n# <<< anvil-managed: a\n\n# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\n# my gap line\nRUN later\n", + "the region belongs above the gap content, not inside it" + ); + } + #[test] fn at_placement_inserts_after_the_line_containing_the_offset() { let host = "# syntax=docker/dockerfile:1\nFROM base\n"; @@ -517,7 +542,7 @@ mod tests { let new = upsert_region_with_placement(host, "x", "body\n", SYN, RegionPlacement::At(0)).unwrap(); assert_eq!( new, - "FROM base\n\n# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\nRUN later\n" + "# >>> anvil-managed: x\nbody\n# <<< anvil-managed: x\n\nFROM base\nRUN later\n" ); } diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index ca605a53..9b24add4 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -479,6 +479,13 @@ fn a_newly_added_region_is_inserted_in_order_not_appended() { // And the repository's own lines are still there. assert!(composed.contains("# MINE-BEFORE"), "gap content before the region must survive"); assert!(composed.contains("# MINE-AFTER"), "gap content after the region must survive"); + // `# MINE-AFTER` sits in the gap that follows the restored region's + // predecessor. The region has to land above it: inserting one line lower + // would split the repository's gap content around anvil's sentinels. + assert!( + at("# >>> anvil-managed: anvil-container-base\n") < at("# MINE-AFTER"), + "the restored region must land above the gap content, not inside it:\n{composed}" + ); } /// The Dockerfile is the first managed-region host whose region *order* is From 7e0e03326f22ccfbd773a4fb149ac6802c24e2d8 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 13:43:55 +0200 Subject: [PATCH 58/81] fix(anvil): close four faults found in review Case-insensitive host resolution bypassed the composed-host guard. `push_region_at` replaces the canonical path with the host's real on-disk name before `composed_host_spec` looks it up, so a repository that spelled the file `dockerfile` matched nothing: no refusal, and the six regions appended below the repository's own `FROM`. A case-only rename of an already composed file was worse, because the lock kept the old casing while the pass wrote the new one, so every region looked orphaned and was stripped from the file the pass had just written. The lookup now compares without case, and a lock key whose resolved host is live transfers ownership to the new key instead of being removed. A composed host is recorded only in `manifest.regions`, never in `manifest.files`, so a Dockerfile that lost its regions to a merge or a revert was diagnosed as a file "anvil has never owned" and the reader was told to delete a file anvil had rendered. The classifier now consults the region provenance and names restoring the regions as the recovery. `.anvil-proposed` siblings are anvil's own review artifacts and cannot reach the image, but the digest walk picked them up, so an outstanding proposal renamed the image and a published one stopped resolving. They are excluded from the digest and from the build context, which keeps those two sets identical as the ignore file claims. The design doc stated an unconditional LF normalization that the recipe stopped performing when it moved to hashing bytes; it now describes the two categories that are normalized and why everything else is not. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 6 +- .anvil/container/Dockerfile.dockerignore | 4 +- crates/cargo-anvil/docs/design/containers.md | 10 +- crates/cargo-anvil/src/run.rs | 36 +++++- .../anvil/container/Dockerfile.dockerignore | 4 +- .../templates/justfiles/anvil/container.just | 10 +- crates/cargo-anvil/tests/container_upgrade.rs | 108 ++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 14 ++- .../snapshots/snapshots__github_backend.snap | 14 ++- .../snapshots/snapshots__local_only.snap | 14 ++- justfiles/anvil/container.just | 10 +- 11 files changed, 212 insertions(+), 18 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index cd44f417..452d627c 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:d3d209ea0063028b186135dc2337612e079acc6168bb6de957b64eb1f65c1d27" +catalog_checksum = "sha256:0d3df70a74ca3b81babc4a02b1474fc232e994a7b4edf17a182bd2fea976439e" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:07e2729e5fac2463dee334feced401f0dec411aa822c58b874b8b14e69473ffa" +checksum = "sha256:68cd5440d92cd37bbd55daaefe19cd1cb2fc892f07c09d31571b6111347588c1" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -165,7 +165,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:345497c7865dc5962cb80f4d7ff28846cb56d8ee93915c8e638d772b04dd215a" +checksum = "sha256:2f6d13a15b8f5499a95256b5e089c8c7bfbe38b7bc3db8b10eddb48e8b1ab86b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index 9cc2482d..2d417fd5 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -23,7 +23,8 @@ # download -- needs the file to be in the context. Denying it would leave the # gap documented but unusable for anything but `RUN`. It is also the directory # the image tag digests, so what the context admits and what the tag covers stay -# the same set. +# the same set -- including the `.anvil-proposed` siblings both exclude, which +# are anvil's review artifacts rather than build inputs. * !justfiles justfiles/* @@ -31,4 +32,5 @@ justfiles/* !.anvil .anvil/* !.anvil/container +.anvil/container/**/*.anvil-proposed !rust-toolchain.toml diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 503d2f3a..4117ee15 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -249,10 +249,16 @@ newline framing would let a file whose body happened to contain `file`, a path a two files splitting at that point, so two different input sets could name one image. The lengths are byte counts of the same UTF-8 encoding the stream is hashed in, so an independent re-implementation arrives at the same bytes. Tagging entries this way -prevents a rearrangement of names and contents from colliding. Line endings are normalized to LF, so CRLF and LF -checkouts agree on the tag. The sort is ordinal because a case-insensitive one would drop one of two inputs differing +prevents a rearrangement of names and contents from colliding. Line endings are normalized to LF for `.just` recipes +and the declared text inputs, so those agree across a CRLF and an LF checkout; every other file the walk admits is +hashed as the bytes the build context copies, because bytes are what `COPY` puts in the layer. The sort is ordinal +because a case-insensitive one would drop one of two inputs differing only in case on the case-sensitive filesystem where the image is built. +Anvil's own `.anvil-proposed` review siblings are excluded from both the digest and the build context. They are +written beside a host when a template moves under a customized region, and they cannot reach the image, so digesting +one would rename it for as long as the proposal went undismissed. + The tag is the first eight bytes of the digest, hex-encoded: 64 bits, far beyond practical collision risk for a local image set, and short enough to keep `docker images` readable. diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 54d64234..1f502691 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -499,8 +499,16 @@ fn region_placement(region_id: &str) -> RegionPlacement { /// /// The **order** is load-bearing: `FROM` must precede every instruction that /// depends on it, and the toolchain must exist before `anvil-setup` runs. +/// +/// The comparison ignores case because the caller has already replaced the +/// canonical path with the host's real on-disk name. A repository that spelled +/// the file `dockerfile` would otherwise miss this lookup entirely, and with it +/// every guard below: the regions would be appended at end-of-file, under the +/// repository's own `FROM`. fn composed_host_spec(host_relpath: &str) -> Option<(&'static str, &'static [&'static str])> { - (host_relpath == container::DOCKERFILE_PATH).then_some((container::DOCKERFILE_HEADER, container::DOCKERFILE_REGION_ORDER)) + host_relpath + .eq_ignore_ascii_case(container::DOCKERFILE_PATH) + .then_some((container::DOCKERFILE_HEADER, container::DOCKERFILE_REGION_ORDER)) } /// What a composed host's current content allows anvil to do with it. @@ -694,8 +702,13 @@ fn plan_removals( // destroy the clean recovery -- reverting the edit -- that the refusal // message tells the reader to use. "Nothing was written to it" has to // be true of the lock as well as the file. - if live_region_hosts.contains(path) { - if !matches!(composed.states.get(path), Some(ComposedHostState::Unsafe(_))) { + // + // The lock carries the casing the path had when the entry was written, + // while a live host carries the casing the write path resolved from + // disk, so the two are compared through the same resolution. + let resolved = resolve_existing_case_insensitive(repo_root, path); + if live_region_hosts.contains(&resolved) { + if !matches!(composed.states.get(&resolved), Some(ComposedHostState::Unsafe(_))) { plan.push(PlanItem::orphaned_kept(Target::File { path: path.clone() })); } continue; @@ -722,6 +735,23 @@ fn plan_removals( if live_regions.contains(&(key.host.clone(), key.id.clone())) { continue; } + // A lock key records the casing its host had when the entry was + // written; a live key records the casing the write path resolved from + // disk this pass. A case-only rename of the host therefore makes every + // one of its regions look orphaned at the very moment the pass has + // rewritten all of them under the new name -- and for a composed host + // that is the whole file, so removing them would strip this pass's own + // writes and leave a Dockerfile with no `FROM`. The regions are still + // there under a key the manifest already carries, so the honest answer + // is to transfer ownership to the new key and touch nothing on disk. + let resolved_host = resolve_existing_case_insensitive(repo_root, &key.host); + if resolved_host != key.host && live_regions.contains(&(resolved_host, key.id.clone())) { + plan.push(PlanItem::orphaned_kept(Target::Region { + host: key.host.clone(), + id: key.id.clone(), + })); + continue; + } let Some(host_text) = hosts.get_or_read(repo_root, &key.host)? else { // Host file is gone entirely; just drop the manifest // entry. Emit OrphanedKept (no-op apply) so the plan diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore index 9cc2482d..2d417fd5 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore @@ -23,7 +23,8 @@ # download -- needs the file to be in the context. Denying it would leave the # gap documented but unusable for anything but `RUN`. It is also the directory # the image tag digests, so what the context admits and what the tag covers stay -# the same set. +# the same set -- including the `.anvil-proposed` siblings both exclude, which +# are anvil's review artifacts rather than build inputs. * !justfiles justfiles/* @@ -31,4 +32,5 @@ justfiles/* !.anvil .anvil/* !.anvil/container +.anvil/container/**/*.anvil-proposed !rust-toolchain.toml diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 0d58073d..ef4c73ad 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -151,9 +151,17 @@ anvil-container-tag: # # The hook is picked up by the same walk. Its *output* is deliberately # never hashed: a credential must not influence a tag. + # + # `.anvil-proposed` siblings are excluded. A region proposal is anvil's own + # review artifact, written beside its host when a template moves under a + # customized region; the build cannot see it and it cannot change what the + # image contains. Digesting it would rename the image for as long as a + # proposal sat undismissed, so two checkouts of one commit would disagree + # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 9b24add4..01558c20 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -629,3 +629,111 @@ fn a_malformed_sentinel_is_named_in_the_refusal() { refusals[0] ); } + +/// The guard that refuses a repository-authored Dockerfile is reached through +/// a host name resolved from disk, so it must not be keyed off the canonical +/// spelling. A repository that wrote a lower-cased `dockerfile` -- the commoner +/// spelling in the wild, and indistinguishable from the canonical one on +/// Windows and macOS -- would otherwise miss the composed-host lookup entirely, +/// get no refusal, and have anvil's six regions appended below its own `FROM`. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_case_variant_repository_dockerfile_is_refused_too() { + let tmp = generated_tree(); + let root = tmp.path(); + let canonical = root.join(".anvil/container/Dockerfile"); + let lowercased = root.join(".anvil/container/dockerfile"); + + std::fs::remove_file(&canonical).unwrap(); + let hand_written = "# hand written by the repository\nFROM mine:1\nRUN echo mine\n"; + write(&lowercased, hand_written); + let mut manifest = Manifest::load(root).unwrap(); + manifest.files.remove(".anvil/container/Dockerfile"); + manifest.regions.retain(|key, _| key.host != ".anvil/container/Dockerfile"); + manifest.save(root).unwrap(); + + // Without this the test passes vacuously on a case-insensitive filesystem + // that kept the old directory entry. + assert_eq!( + on_disk_dockerfile_name(root), + "dockerfile", + "precondition: the host is lower-cased on disk" + ); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert_eq!( + std::fs::read_to_string(&lowercased).unwrap(), + hand_written, + "content anvil never owned must be left exactly as found" + ); + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.to_lowercase().contains(".anvil/container/dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); + assert!( + refusals[0].contains("anvil has never owned it"), + "the diagnostic must explain why it cannot be composed: {}", + refusals[0] + ); +} + +/// A case-only rename of an already composed host leaves the lock keyed by the +/// old spelling and the pass keyed by the new one. Nothing is missing -- every +/// region is still in the file, rewritten this pass -- so the stale entries +/// must transfer ownership rather than be treated as orphans: removing them +/// would strip the pass's own writes and leave a Dockerfile with no `FROM`. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_case_only_rename_of_a_composed_host_keeps_its_regions() { + let tmp = generated_tree(); + let root = tmp.path(); + let canonical = root.join(".anvil/container/Dockerfile"); + let lowercased = root.join(".anvil/container/dockerfile"); + + let composed = std::fs::read_to_string(&canonical).unwrap(); + let regions_before = composed.matches("# >>> anvil-managed:").count(); + assert!(regions_before >= 2, "precondition: the host is composed from several regions"); + std::fs::remove_file(&canonical).unwrap(); + write(&lowercased, &composed); + assert_eq!( + on_disk_dockerfile_name(root), + "dockerfile", + "precondition: the host is lower-cased on disk" + ); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let after = std::fs::read_to_string(&lowercased).unwrap(); + assert_eq!( + after.matches("# >>> anvil-managed:").count(), + regions_before, + "a case-only rename must not cost the host any of its regions:\n{after}" + ); + assert!( + after.contains("\nFROM "), + "the composed file must still declare a base image:\n{after}" + ); + let removed: Vec<&Target> = outcome + .plan + .items() + .iter() + .filter(|i| i.decision == Decision::Remove) + .map(|i| &i.target) + .collect(); + assert!(removed.is_empty(), "nothing may be removed for a case-only rename: {removed:?}"); +} + +/// The real on-disk spelling of the composed host, so the case tests cannot +/// pass without the rename they claim to make having taken effect. +fn on_disk_dockerfile_name(root: &Path) -> String { + std::fs::read_dir(root.join(".anvil/container")) + .unwrap() + .flatten() + .map(|e| e.file_name().to_string_lossy().into_owned()) + .find(|n| n.eq_ignore_ascii_case("Dockerfile")) + .expect("the composed host must exist") +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 8c20e192..cfa18e52 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -171,7 +171,8 @@ CMD ["bash"] # download -- needs the file to be in the context. Denying it would leave the # gap documented but unusable for anything but `RUN`. It is also the directory # the image tag digests, so what the context admits and what the tag covers stay -# the same set. +# the same set -- including the `.anvil-proposed` siblings both exclude, which +# are anvil's review artifacts rather than build inputs. * !justfiles justfiles/* @@ -179,6 +180,7 @@ justfiles/* !.anvil .anvil/* !.anvil/container +.anvil/container/**/*.anvil-proposed !rust-toolchain.toml === .delta.toml === @@ -3707,9 +3709,17 @@ anvil-container-tag: # # The hook is picked up by the same walk. Its *output* is deliberately # never hashed: a credential must not influence a tag. + # + # `.anvil-proposed` siblings are excluded. A region proposal is anvil's own + # review artifact, written beside its host when a template moves under a + # customized region; the build cannot see it and it cannot change what the + # image contains. Digesting it would rename the image for as long as a + # proposal sat undismissed, so two checkouts of one commit would disagree + # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index b910cfa5..db215127 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -171,7 +171,8 @@ CMD ["bash"] # download -- needs the file to be in the context. Denying it would leave the # gap documented but unusable for anything but `RUN`. It is also the directory # the image tag digests, so what the context admits and what the tag covers stay -# the same set. +# the same set -- including the `.anvil-proposed` siblings both exclude, which +# are anvil's review artifacts rather than build inputs. * !justfiles justfiles/* @@ -179,6 +180,7 @@ justfiles/* !.anvil .anvil/* !.anvil/container +.anvil/container/**/*.anvil-proposed !rust-toolchain.toml === .delta.toml === @@ -3606,9 +3608,17 @@ anvil-container-tag: # # The hook is picked up by the same walk. Its *output* is deliberately # never hashed: a credential must not influence a tag. + # + # `.anvil-proposed` siblings are excluded. A region proposal is anvil's own + # review artifact, written beside its host when a template moves under a + # customized region; the build cannot see it and it cannot change what the + # image contains. Digesting it would rename the image for as long as a + # proposal sat undismissed, so two checkouts of one commit would disagree + # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index b081d41a..6c18805c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -171,7 +171,8 @@ CMD ["bash"] # download -- needs the file to be in the context. Denying it would leave the # gap documented but unusable for anything but `RUN`. It is also the directory # the image tag digests, so what the context admits and what the tag covers stay -# the same set. +# the same set -- including the `.anvil-proposed` siblings both exclude, which +# are anvil's review artifacts rather than build inputs. * !justfiles justfiles/* @@ -179,6 +180,7 @@ justfiles/* !.anvil .anvil/* !.anvil/container +.anvil/container/**/*.anvil-proposed !rust-toolchain.toml === .delta.toml === @@ -2447,9 +2449,17 @@ anvil-container-tag: # # The hook is picked up by the same walk. Its *output* is deliberately # never hashed: a credential must not influence a tag. + # + # `.anvil-proposed` siblings are excluded. A region proposal is anvil's own + # review artifact, written beside its host when a template moves under a + # customized region; the build cannot see it and it cannot change what the + # image contains. Digesting it would rename the image for as long as a + # proposal sat undismissed, so two checkouts of one commit would disagree + # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 0d58073d..ef4c73ad 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -151,9 +151,17 @@ anvil-container-tag: # # The hook is picked up by the same walk. Its *output* is deliberately # never hashed: a credential must not influence a tag. + # + # `.anvil-proposed` siblings are excluded. A region proposal is anvil's own + # review artifact, written beside its host when a template moves under a + # customized region; the build cannot see it and it cannot change what the + # image contains. Digesting it would rename the image for as long as a + # proposal sat undismissed, so two checkouts of one commit would disagree + # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } From 94b56fb783084d338d799fcc9926fd5b4ac6d63e Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 14:09:38 +0200 Subject: [PATCH 59/81] fix(anvil): drop a dead condition the mutation gate flagged The `continue` above already establishes that the recorded region key is not live, so `resolved_host != key.host` can never change the outcome of the lookup that follows it: when the resolution is a no-op the `contains` simply repeats the test that just failed. Negating it leaves behaviour identical, which is what the surviving mutant reported. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/src/run.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 1f502691..773da7d7 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -744,8 +744,12 @@ fn plan_removals( // writes and leave a Dockerfile with no `FROM`. The regions are still // there under a key the manifest already carries, so the honest answer // is to transfer ownership to the new key and touch nothing on disk. + // + // No `resolved_host != key.host` guard: the `continue` above has + // already established that the recorded key is not live, so when the + // resolution changes nothing this lookup repeats it and fails. let resolved_host = resolve_existing_case_insensitive(repo_root, &key.host); - if resolved_host != key.host && live_regions.contains(&(resolved_host, key.id.clone())) { + if live_regions.contains(&(resolved_host, key.id.clone())) { plan.push(PlanItem::orphaned_kept(Target::Region { host: key.host.clone(), id: key.id.clone(), From 534b1533c772d5b6ba1c49920ffa49238c17e9c3 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 15:20:07 +0200 Subject: [PATCH 60/81] fix(anvil): resolve casing before every lock lookup, and stop hashing proposals The previous commit claimed to fix the composed-host diagnosis and did not; this carries the change it described. A composed host is recorded in `manifest.regions` and never in `manifest.files`, so "no file entry" also describes every composed file that has lost its regions to a merge, a revert or a checkout. Those still have their provenance in the lock, and telling the reader to delete the file would throw away the gap content the refusal exists to protect. The classifier now consults the region provenance and names restoring the file as the recovery. The owned-file half of the casing fix was missing too. Every plan item carries the casing resolved from disk while the lock carries whatever casing it was written with, so on a case-insensitive filesystem a case-only rename made anvil retire its own artifact: the removal pass opened the file the same pass had just written and deleted it, and the manifest lookup missed the entry so the file was additionally proposed against as though the repository had authored it. Both lookups now resolve the same way, through two manifest helpers that own the comparison. The `.anvil-proposed` exclusion reached only the first of the digest's two walks. `container.just` is itself an owned artifact, so a customized copy puts a proposal inside the recipe tree the second walk hashes, which is the same tag instability the first fix removed and made the doc's "excluded from both the digest and the build context" claim false. Both walks and both ignore-file entries now agree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 6 +- .anvil/container/Dockerfile.dockerignore | 1 + crates/cargo-anvil/src/emit/owned_file.rs | 2 +- crates/cargo-anvil/src/manifest.rs | 71 ++++++++++++++ crates/cargo-anvil/src/run.rs | 32 ++++-- .../anvil/container/Dockerfile.dockerignore | 1 + .../templates/justfiles/anvil/container.just | 7 +- crates/cargo-anvil/tests/container_upgrade.rs | 97 +++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 8 +- .../snapshots/snapshots__github_backend.snap | 8 +- .../snapshots/snapshots__local_only.snap | 8 +- justfiles/anvil/container.just | 7 +- 12 files changed, 231 insertions(+), 17 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 452d627c..7804b9d5 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,11 +1,11 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:0d3df70a74ca3b81babc4a02b1474fc232e994a7b4edf17a182bd2fea976439e" +catalog_checksum = "sha256:da005de472f308acee246d85d83c3ab50c8bfd364acb237157495a3ee8027bf0" [[file]] path = ".anvil/container/Dockerfile.dockerignore" -checksum = "sha256:68cd5440d92cd37bbd55daaefe19cd1cb2fc892f07c09d31571b6111347588c1" +checksum = "sha256:9c7906c20415ca3b832afb075c21e79f191ef14ee2fb4b6a5a95394e8b2153c1" [[file]] path = ".github/actions/anvil-impact/action.yml" @@ -165,7 +165,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:2f6d13a15b8f5499a95256b5e089c8c7bfbe38b7bc3db8b10eddb48e8b1ab86b" +checksum = "sha256:adf88a3e388d50b3aaebfcd67d51bb62ccb13a355a7f4a5f1e1388e52680d5f8" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/.anvil/container/Dockerfile.dockerignore b/.anvil/container/Dockerfile.dockerignore index 2d417fd5..577907f0 100644 --- a/.anvil/container/Dockerfile.dockerignore +++ b/.anvil/container/Dockerfile.dockerignore @@ -29,6 +29,7 @@ !justfiles justfiles/* !justfiles/anvil +justfiles/anvil/**/*.anvil-proposed !.anvil .anvil/* !.anvil/container diff --git a/crates/cargo-anvil/src/emit/owned_file.rs b/crates/cargo-anvil/src/emit/owned_file.rs index eb17434a..eb5f2930 100644 --- a/crates/cargo-anvil/src/emit/owned_file.rs +++ b/crates/cargo-anvil/src/emit/owned_file.rs @@ -30,7 +30,7 @@ pub fn plan_owned_file(repo_root: &Path, manifest: &Manifest, relpath: &str, ren let on_disk = read_file_if_present(&abs)?; let disk_checksum = on_disk.as_deref().map(checksum_str); let template_checksum = checksum_str(rendered); - let last_rendered = manifest.files.get(relpath).map(String::as_str); + let last_rendered = manifest.file_checksum(relpath); let inputs = DecisionInputs { last_rendered, diff --git a/crates/cargo-anvil/src/manifest.rs b/crates/cargo-anvil/src/manifest.rs index b2cff67b..0c947969 100644 --- a/crates/cargo-anvil/src/manifest.rs +++ b/crates/cargo-anvil/src/manifest.rs @@ -250,6 +250,36 @@ impl Manifest { checksum.into(), ); } + + /// The checksum recorded for an owned file, tolerating a case-only + /// difference between the lock and the path the caller resolved from disk. + /// + /// Plan items carry the host's real on-disk casing; a lock entry carries + /// whatever casing it had when it was written. Comparing the two exactly + /// makes a case-only rename look like a file anvil has never seen, which + /// costs the file its provenance: it is planned as repository-authored and + /// earns a spurious proposal, and the removal pass retires the very entry + /// that names it. + pub fn file_checksum(&self, path: &str) -> Option<&str> { + self.files.get(path).map(String::as_str).or_else(|| { + self.files + .iter() + .find(|(key, _)| key.as_str().eq_ignore_ascii_case(path)) + .map(|(_, checksum)| checksum.as_str()) + }) + } + + /// Whether the lock records any managed region hosted by `path`, comparing + /// the host without case for the reason [`Self::file_checksum`] gives. + /// + /// This is the provenance that separates "a file anvil composes, whose + /// regions have been removed" from "a file anvil has never owned". The two + /// have opposite recoveries, so the distinction decides which one a refusal + /// tells the reader to reach for. + #[must_use] + pub fn has_region_host(&self, path: &str) -> bool { + self.regions.keys().any(|key| key.host.as_str().eq_ignore_ascii_case(path)) + } } // Suppress an unused-import lint when no callers reference `Array`/`Value` @@ -434,4 +464,45 @@ mod tests { let text = sample_manifest().to_toml(); assert!(text.ends_with('\n')); } + + /// The lock keeps whatever casing a path had when it was written, while a + /// plan item carries the casing resolved from disk. An exact-only lookup + /// loses the provenance of anvil's own file on a case-only rename, which + /// is what makes the removal pass delete it. + #[test] + fn file_checksum_tolerates_a_case_only_difference() { + let mut manifest = Manifest::default(); + manifest.set_file("justfiles/anvil/tools.just", "sha256:body"); + + assert_eq!(manifest.file_checksum("justfiles/anvil/tools.just"), Some("sha256:body")); + assert_eq!(manifest.file_checksum("justfiles/anvil/Tools.just"), Some("sha256:body")); + assert_eq!(manifest.file_checksum("justfiles/anvil/other.just"), None); + } + + /// An exact match must win over a case-insensitive one, so a repository + /// holding two entries differing only in case still reads its own. + #[test] + fn file_checksum_prefers_the_exact_entry() { + let mut manifest = Manifest::default(); + manifest.set_file("a/Thing.just", "sha256:upper"); + manifest.set_file("a/thing.just", "sha256:lower"); + + assert_eq!(manifest.file_checksum("a/Thing.just"), Some("sha256:upper")); + assert_eq!(manifest.file_checksum("a/thing.just"), Some("sha256:lower")); + } + + /// The provenance that separates "a composed file whose regions were + /// removed" from "a file anvil has never owned". The two have opposite + /// recoveries, so a missed match sends the reader to delete a file anvil + /// rendered. + #[test] + fn has_region_host_matches_without_case() { + let mut manifest = Manifest::default(); + manifest.set_region(".anvil/container/Dockerfile", "anvil-container-base", "sha256:body"); + + assert!(manifest.has_region_host(".anvil/container/Dockerfile")); + assert!(manifest.has_region_host(".anvil/container/dockerfile")); + assert!(!manifest.has_region_host(".anvil/container/Other")); + assert!(!Manifest::default().has_region_host(".anvil/container/Dockerfile")); + } } diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 773da7d7..460a1c3c 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -575,14 +575,29 @@ fn composed_host_state(order: &[&str], host_relpath: &str, text: &str, manifest: // that edit silently destroyed on upgrade. Appending the regions // instead is not a kinder answer, because everything already in the // file would then sit above `FROM`. - return match manifest.files.get(host_relpath) { - Some(recorded) if *recorded == checksum_str(text) => ComposedHostState::SeedFromScaffold, + return match manifest.file_checksum(host_relpath) { + Some(recorded) if recorded == checksum_str(text) => ComposedHostState::SeedFromScaffold, Some(_) => ComposedHostState::Unsafe( "it was edited after anvil last wrote it, and this release composes the file from \ managed regions instead of owning it whole. Move the edits you want to keep into \ the gaps of a freshly generated file, or delete it and re-run to have one written" .to_owned(), ), + // A composed host is recorded in `regions` and never in `files`, so + // "no file entry" is not the same as "anvil has never seen this". + // It is also every composed file that has lost its regions -- a + // merge resolved the other way, a revert, a checkout of an older + // commit -- and for those the lock still holds the provenance. The + // two cases have opposite recoveries, and telling someone to delete + // a file anvil rendered would throw away the gap content the + // message elsewhere tells them to preserve. + None if manifest.has_region_host(host_relpath) => ComposedHostState::Unsafe( + "anvil composes it from managed regions and the lock still records them, but none \ + of them is in the file: they were dropped by a merge, a revert or an edit. \ + Restore the file from version control to recover the regions and your own content \ + around them together" + .to_owned(), + ), None => ComposedHostState::Unsafe( "it exists but anvil has never owned it, so there is nowhere to splice the managed \ regions that would not put your content above `FROM`. Delete it and re-run to have \ @@ -685,7 +700,13 @@ fn plan_removals( let live_region_hosts: BTreeSet = live_regions.iter().map(|(host, _)| host.clone()).collect(); for (path, last) in &previous.files { - if live_files.contains(path) { + // The lock carries the casing the path had when the entry was written, + // while every live plan item carries the casing resolved from disk, so + // the two are compared through the same resolution. Without it a + // case-only rename makes anvil's own file look retired and the removal + // below deletes the artifact this very pass just wrote. + let resolved = resolve_existing_case_insensitive(repo_root, path); + if live_files.contains(&resolved) { continue; } // A path that is no longer an owned file but *is* the host of a live @@ -702,11 +723,6 @@ fn plan_removals( // destroy the clean recovery -- reverting the edit -- that the refusal // message tells the reader to use. "Nothing was written to it" has to // be true of the lock as well as the file. - // - // The lock carries the casing the path had when the entry was written, - // while a live host carries the casing the write path resolved from - // disk, so the two are compared through the same resolution. - let resolved = resolve_existing_case_insensitive(repo_root, path); if live_region_hosts.contains(&resolved) { if !matches!(composed.states.get(&resolved), Some(ComposedHostState::Unsafe(_))) { plan.push(PlanItem::orphaned_kept(Target::File { path: path.clone() })); diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore index 2d417fd5..577907f0 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.dockerignore @@ -29,6 +29,7 @@ !justfiles justfiles/* !justfiles/anvil +justfiles/anvil/**/*.anvil-proposed !.anvil .anvil/* !.anvil/container diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index ef4c73ad..6c2350b8 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -196,9 +196,14 @@ anvil-container-tag: # -Force because Get-ChildItem omits hidden entries otherwise: a # dot-prefixed file is copied like any other, and skipping it would let its # edits ride under an unchanged tag -- and make Windows and Unix disagree. + # + # `.anvil-proposed` siblings are excluded here for the same reason as under + # `.anvil/container/`: this driver is itself an owned artifact, so a + # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 01558c20..ae6136de 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -737,3 +737,100 @@ fn on_disk_dockerfile_name(root: &Path) -> String { .find(|n| n.eq_ignore_ascii_case("Dockerfile")) .expect("the composed host must exist") } + +/// A composed host lives in `manifest.regions` and never in `manifest.files`, +/// so "no file entry" is not the same as "anvil has never seen this path". It +/// is also every composed file that has lost its regions to a merge, a revert +/// or an edit -- and the lock still records them. Reporting that as a +/// never-owned file tells the reader to delete a file anvil rendered, throwing +/// away the gap content the refusal exists to protect, when the cheap recovery +/// is restoring the file. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_composed_host_that_lost_its_regions_is_told_to_restore_it() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // The lock is left exactly as generated: it still records every region. + let recorded = Manifest::load(root).unwrap(); + assert!( + recorded.regions.keys().any(|key| key.host == ".anvil/container/Dockerfile"), + "precondition: the lock records the composed host's regions" + ); + assert!( + !recorded.files.contains_key(".anvil/container/Dockerfile"), + "precondition: a composed host is never tracked as an owned file" + ); + write(&dockerfile, "# my own content\nFROM mine:1\n"); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let refusals: Vec<&String> = outcome + .plan + .refusals() + .iter() + .filter(|r| r.contains(".anvil/container/Dockerfile")) + .collect(); + assert_eq!(refusals.len(), 1, "one diagnostic per host: {refusals:?}"); + assert!( + !refusals[0].contains("anvil has never owned it"), + "the lock records the regions, so this is not a never-owned file: {}", + refusals[0] + ); + assert!( + refusals[0].contains("Restore the file"), + "the recovery must be restoring the file, not deleting it: {}", + refusals[0] + ); +} + +/// The removal pass compares the lock's casing against the casing every plan +/// item resolved from disk. Without resolving first, a case-only rename makes +/// anvil's own artifact look retired -- and on a case-insensitive filesystem +/// the removal opens the very file the pass just wrote and deletes it. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_case_only_rename_of_an_owned_file_is_not_deleted() { + let tmp = generated_tree(); + let root = tmp.path(); + let canonical = root.join("justfiles/anvil/tools.just"); + let renamed = root.join("justfiles/anvil/Tools.just"); + assert!(canonical.is_file(), "precondition: the owned recipe exists as generated"); + + let body = std::fs::read_to_string(&canonical).unwrap(); + std::fs::remove_file(&canonical).unwrap(); + write(&renamed, &body); + // The lock still carries the original casing, which is the whole point. + assert_eq!( + Manifest::load(root).unwrap().file_checksum("justfiles/anvil/tools.just"), + Some(checksum_str(&body).as_str()), + "precondition: the lock records the pre-rename casing" + ); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let removed: Vec = outcome + .plan + .items() + .iter() + .filter(|i| i.decision == Decision::Remove) + .filter_map(|i| match &i.target { + Target::File { path } => Some(path.clone()), + Target::Region { .. } => None, + }) + .collect(); + assert!( + !removed.iter().any(|p| p.eq_ignore_ascii_case("justfiles/anvil/tools.just")), + "anvil must not retire the artifact it just wrote: {removed:?}" + ); + assert!( + renamed.is_file() || canonical.is_file(), + "the generated recipe must still be on disk after the run" + ); + assert!( + !root.join("justfiles/anvil/Tools.just.anvil-proposed").exists() + && !root.join("justfiles/anvil/tools.just.anvil-proposed").exists(), + "a file anvil still owns must not be proposed against as though it were repository-authored" + ); +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index cfa18e52..d134bb49 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -177,6 +177,7 @@ CMD ["bash"] !justfiles justfiles/* !justfiles/anvil +justfiles/anvil/**/*.anvil-proposed !.anvil .anvil/* !.anvil/container @@ -3754,9 +3755,14 @@ anvil-container-tag: # -Force because Get-ChildItem omits hidden entries otherwise: a # dot-prefixed file is copied like any other, and skipping it would let its # edits ride under an unchanged tag -- and make Windows and Unix disagree. + # + # `.anvil-proposed` siblings are excluded here for the same reason as under + # `.anvil/container/`: this driver is itself an owned artifact, so a + # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index db215127..f24ab9e8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -177,6 +177,7 @@ CMD ["bash"] !justfiles justfiles/* !justfiles/anvil +justfiles/anvil/**/*.anvil-proposed !.anvil .anvil/* !.anvil/container @@ -3653,9 +3654,14 @@ anvil-container-tag: # -Force because Get-ChildItem omits hidden entries otherwise: a # dot-prefixed file is copied like any other, and skipping it would let its # edits ride under an unchanged tag -- and make Windows and Unix disagree. + # + # `.anvil-proposed` siblings are excluded here for the same reason as under + # `.anvil/container/`: this driver is itself an owned artifact, so a + # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 6c18805c..657bb1fb 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -177,6 +177,7 @@ CMD ["bash"] !justfiles justfiles/* !justfiles/anvil +justfiles/anvil/**/*.anvil-proposed !.anvil .anvil/* !.anvil/container @@ -2494,9 +2495,14 @@ anvil-container-tag: # -Force because Get-ChildItem omits hidden entries otherwise: a # dot-prefixed file is copied like any other, and skipping it would let its # edits ride under an unchanged tag -- and make Windows and Unix disagree. + # + # `.anvil-proposed` siblings are excluded here for the same reason as under + # `.anvil/container/`: this driver is itself an owned artifact, so a + # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index ef4c73ad..6c2350b8 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -196,9 +196,14 @@ anvil-container-tag: # -Force because Get-ChildItem omits hidden entries otherwise: a # dot-prefixed file is copied like any other, and skipping it would let its # edits ride under an unchanged tag -- and make Windows and Unix disagree. + # + # `.anvil-proposed` siblings are excluded here for the same reason as under + # `.anvil/container/`: this driver is itself an owned artifact, so a + # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force) { + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } From 831965eac35032d50b5e6cd0eee3c72d70cf85de Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Thu, 27 Aug 2026 20:41:36 +0200 Subject: [PATCH 61/81] fix(anvil): make the refusal atomic across both removal loops The owned-file loop skips a refused host so its lock entry survives; the region loop never consulted the same state. A lock entry naming a region the catalog no longer declares therefore reached `remove_region`, so a run that reports "Nothing was written to it" spliced that block out of the very file it had refused and dropped the entry that records it -- the provenance the next run reclassifies from, and the basis for the advice to restore the file rather than delete it. Latent rather than live: every id in the declared order is currently live, so nothing reaches the unguarded path until a region is renamed or retired, which is the upgrade shape the ordered insertion work already anticipates. The existing refusal test cannot cover it because it plants only declared ids, so the new test plants a retired one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/src/run.rs | 11 ++++ crates/cargo-anvil/tests/container_upgrade.rs | 60 +++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 460a1c3c..85d4f8ac 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -765,6 +765,17 @@ fn plan_removals( // already established that the recorded key is not live, so when the // resolution changes nothing this lookup repeats it and fails. let resolved_host = resolve_existing_case_insensitive(repo_root, &key.host); + // A refused host was not opened, and "nothing was written to it" has to + // be true of the lock as well as the file -- the same invariant the + // owned-file loop above keeps. Without this, a lock entry naming a + // region the catalog no longer declares still reaches `remove_region` + // below, so the run splices a block out of the very file whose refusal + // says it was left alone, and purges the provenance the next run + // reclassifies from. Latent while every declared id is live, reachable + // at the first region rename or retirement. + if matches!(composed.states.get(&resolved_host), Some(ComposedHostState::Unsafe(_))) { + continue; + } if live_regions.contains(&(resolved_host, key.id.clone())) { plan.push(PlanItem::orphaned_kept(Target::Region { host: key.host.clone(), diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index ae6136de..76ec30b4 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -834,3 +834,63 @@ fn a_case_only_rename_of_an_owned_file_is_not_deleted() { "a file anvil still owns must not be proposed against as though it were repository-authored" ); } + +/// The refusal must be atomic across *both* removal loops. The owned-file loop +/// consults `composed.states`; the region loop did not, so a lock entry naming +/// a region the catalog no longer declares still reached `remove_region` -- +/// splicing a block out of the very file whose refusal says it was untouched, +/// and purging the provenance the next run reclassifies from. +/// +/// Reaching it needs a retired region id, which today's catalog has none of, so +/// the stale key is planted here. That is the same upgrade shape the ordered +/// insertion work exists for, so it is one rename away from being live. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn refusing_a_composed_host_spares_a_retired_region_entry_too() { + let tmp = generated_tree(); + let root = tmp.path(); + let dockerfile = root.join(".anvil/container/Dockerfile"); + + // A region the catalog no longer declares, present in the file and tracked + // in the lock -- the state left by a rename or a retirement. + let legacy_id = "anvil-container-legacy"; + let legacy_body = "RUN echo legacy\n"; + let text = std::fs::read_to_string(&dockerfile).unwrap(); + let with_legacy = format!("{text}\n# >>> anvil-managed: {legacy_id}\n{legacy_body}# <<< anvil-managed: {legacy_id}\n"); + // Duplicate an opening sentinel so the host classifies `Unsafe` and the run + // refuses it. The legacy region itself stays well formed, so the removal + // path can still reach it. + let opener = "# >>> anvil-managed: anvil-container-base\n"; + let broken = with_legacy.replacen(opener, &format!("{opener}{opener}"), 1); + write(&dockerfile, &broken); + + let mut manifest = Manifest::load(root).unwrap(); + manifest.set_region(".anvil/container/Dockerfile", legacy_id, checksum_str(legacy_body)); + manifest.save(root).unwrap(); + let lock_before = std::fs::read_to_string(root.join(".anvil.lock")).unwrap(); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert_eq!( + std::fs::read_to_string(&dockerfile).unwrap(), + broken, + "a refused host must not have a block spliced out of it" + ); + assert_eq!( + std::fs::read_to_string(root.join(".anvil.lock")).unwrap(), + lock_before, + "\"nothing was written to it\" has to be true of the lock as well as the file" + ); + let key = RegionKey { + host: ".anvil/container/Dockerfile".to_owned(), + id: legacy_id.to_owned(), + }; + assert!( + Manifest::load(root).unwrap().regions.contains_key(&key), + "the retired region's provenance must survive a refusal" + ); + assert!( + outcome.plan.refusals().iter().any(|r| r.contains(".anvil/container/Dockerfile")), + "precondition: the host must actually have been refused" + ); +} From 0672e7d471d7d0c0f3fc52571020b0f3fcc37ef2 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 11:40:25 +0200 Subject: [PATCH 62/81] feat(anvil): run any command in the container, not only a recipe `anvil-container` takes the argv to execute inside the image instead of a recipe name, so `just anvil-container cargo build` works alongside `just anvil-container just anvil-pr`. Nothing is prefixed on the caller's behalf, so there is no guessing whether a token names a recipe or a program. Two seams follow the argv. The in-container passthrough runs the command on the spot, resolving `just` to the binary running the tree so a host with it off PATH still works. A GitHub token is minted from `gh` only for a `just` command whose plan reads the variable: planning is the only way to know that a command needs one, and a minted credential reaches every process in the container, so an unknown command keeps the environment it was given. The design doc drops its 0.4.0 migration section, and comments and docs across the container work state the design rather than narrating how it came about. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- crates/cargo-anvil/README.md | 17 ++-- crates/cargo-anvil/docs/design/containers.md | 51 ++++-------- .../src/anvil/artifacts/container.rs | 56 +++++++------ crates/cargo-anvil/src/lib.rs | 15 ++-- crates/cargo-anvil/src/run.rs | 35 ++++----- .../templates/justfiles/anvil/container.just | 78 ++++++++++--------- crates/cargo-anvil/tests/container_upgrade.rs | 37 +++++---- .../snapshots/snapshots__ado_backend.snap | 78 ++++++++++--------- .../snapshots/snapshots__github_backend.snap | 78 ++++++++++--------- .../snapshots/snapshots__local_only.snap | 78 ++++++++++--------- justfiles/anvil/container.just | 78 ++++++++++--------- 12 files changed, 307 insertions(+), 298 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 7804b9d5..2f48c358 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:da005de472f308acee246d85d83c3ab50c8bfd364acb237157495a3ee8027bf0" +catalog_checksum = "sha256:4815fcb25d725301e7f647fd642ef555d2194c8044587c8ad6d83da8ce28c8ea" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:dbfde2179d7008f6ee0647aaca37f01241c2f126af7636917381ff3f95bc0 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:adf88a3e388d50b3aaebfcd67d51bb62ccb13a355a7f4a5f1e1388e52680d5f8" +checksum = "sha256:230bc29a9e5be4a4ea2bb58ae3a951b59ae8715c37e7bde0aef604c1531fafc2" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index 1e66eff5..ad4e011f 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -112,16 +112,17 @@ the toolset by construction, with no second tool list to keep in step. Execution is opt-in per invocation: `just anvil-pr` and every other recipe continue to run natively, and a container is entered only through -`anvil-container`, which takes any recipe name and its arguments. Those -arguments are whitespace-delimited tokens: `just` joins a variadic +`anvil-container`, whose arguments are the argv executed inside the image. +Those arguments are whitespace-delimited tokens: `just` joins a variadic parameter with spaces before the recipe sees it, so an argument that itself contains a space cannot be recovered and does not survive the round trip. ```text -just anvil-container anvil-clippy # one check -just anvil-container anvil-pr # the whole PR tier -just anvil-container anvil-setup binstall # a recipe with an argument -just anvil-container # interactive shell +just anvil-container just anvil-clippy # one check +just anvil-container just anvil-pr # the whole PR tier +just anvil-container just anvil-setup binstall # a recipe with an argument +just anvil-container cargo build # any other command +just anvil-container # interactive shell ``` The feature is three generated artifacts and one optional hook script, with @@ -197,7 +198,7 @@ so a forgotten one rebuilds from scratch each time: ```text $env:ANVIL_CONTAINER_NO_CACHE = '1' -try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +try { just anvil-container just anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } ``` #### The hook @@ -480,7 +481,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbcyN35C50s70b7icANpGEc88bQlr3I2a8WwobrcveTzoRNR9hZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQb5Q3GF-6G62YbV-q0mTL9mIYbh7z-5PL8118b-8n3fLqY9pRhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 4117ee15..57bedab1 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -1,12 +1,12 @@ # Containerized execution -Any generated recipe can be executed inside a Linux image built from the toolchain and tool versions the repository +Any command can be executed inside a Linux image built from the toolchain and tool versions the repository pins: ```bash -just anvil-container anvil-clippy # one check -just anvil-container anvil-pr # the whole PR tier -just anvil-container # interactive shell +just anvil-container just anvil-pr # a tier +just anvil-container cargo build # any other command +just anvil-container # interactive shell ``` Execution is opt-in per invocation: recipes run natively unless a container is requested by name. The feature is three @@ -56,38 +56,21 @@ own agents. The image is pinned to resemble that environment, not to reproduce i ## 2. Command surface -`anvil-container` accepts a recipe name and its arguments; every other recipe continues to execute natively. +`anvil-container` takes the argv to run inside the image; every recipe continues to execute natively unless a +container is requested by name. Anvil recipes are reached by naming `just`, like any other command: ```bash -just anvil-container anvil-setup binstall +just anvil-container just anvil-setup binstall +just anvil-container cargo build ``` -**Upgrading from 0.4.0.** The routing seam is gone, and its removal is silent for the repositories most likely to -care. 0.4.0 let a repository opt into containers by exporting `ANVIL_RUNNER=container`, which routed a tier through -`_anvil-run`; anyone who did that left the generated `anvil-runner` region byte-identical, so regeneration classifies -it `Remove` and deletes it without comment. From the next `just anvil-pr` the tier runs natively on the host -toolchain, with no diagnostic and no behaviour anvil can detect. What was removed: - -| Removed in this release | Replacement | -| --- | --- | -| `ANVIL_RUNNER` environment variable | none — name the container explicitly with `just anvil-container ` | -| the `anvil-runner` managed region in the root `Justfile` | none | -| `justfiles/anvil/runner.just` | `justfiles/anvil/container.just` | -| the `_anvil-pr`, `_anvil-scheduled` and `_anvil-full` shadow recipes | the tiers themselves, which now only run natively | -| `.anvil/container/run-in-container.{sh,ps1}`, `entrypoint.sh`, `image-id.{sh,ps1}`, `Containerfile*` | `.anvil/container/Dockerfile` and its ignore file | - -A 0.4.0 installation also holds cache volumes named `anvil-cargo-registry-`, `anvil-cargo-git-` and -`anvil-target--`. This release keys volume names on the repository *directory name* instead, so -none of those are reused and `anvil-container-down` does not remove them; `anvil-target-*` in particular holds a full -workspace `target/`. Remove them once with the engine directly. - -Arguments are whitespace-delimited tokens. `just` joins a variadic `*target` with spaces before the recipe body sees -it, so the original argv is unrecoverable and an argument containing a space does not round-trip. No catalog recipe -takes one; a fork whose recipes do should pass them through the environment instead. +Arguments are whitespace-delimited tokens. `just` joins a variadic `*command` with spaces before the recipe body sees +it, so the original argv is unrecoverable and an argument containing a space does not round-trip. Pass such a value +through the environment instead. | Recipe | Behaviour | | --- | --- | -| `just anvil-container [args…]` | Execute a recipe in the image. With no argument, opens an interactive shell. | +| `just anvil-container ` | Execute a command in the image. With no argument, opens an interactive shell. | | `just anvil-container-tag` | Print the image reference for the current inputs. Builds nothing. | | `just anvil-container-status` | Print the engine, working directory, image reference, and whether it is present. Never builds or pulls. | | `just anvil-container-down` | Remove this repository's cache volumes. The image is retained. | @@ -100,14 +83,14 @@ There is deliberately no `anvil-container-rebuild`. Its whole body would be `ANV the ordinary resolve, and that variable is already public below — where it also composes with `NO_REBUILD` and `NO_RESOLVE`, which a recipe form does not. -What the recipe did supply was **scope**: it set the variable in its own process and exited, so exactly one build -ignored the cache. An exported variable is sticky, and every container command reads it, so a forgotten +What a recipe form would supply is **scope**: it sets the variable in its own process and exits, so exactly one build +ignores the cache. An exported variable is sticky, and every container command reads it, so a forgotten `ANVIL_CONTAINER_NO_CACHE` rebuilds from scratch on each later invocation with nothing to indicate why. Scope it to the one run: ```powershell $env:ANVIL_CONTAINER_NO_CACHE = '1' -try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +try { just anvil-container just anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } ``` | Variable | Effect | @@ -158,10 +141,10 @@ silent in a way that matters: the file carries the base digest and four tool pin keeps building on the base and versions frozen at that moment, while `anvil-container-tag` resolves happily *because the tag hashes their file*. The identity scheme works perfectly and still names a stale image. -**What regions actually change.** They do not make anvil's content unwritable: §2's ownership rules apply to a region +**What regions buy.** They do not make anvil's content unwritable: §2's ownership rules apply to a region body exactly as they do to a file, so an edit *inside* a region is still preserved and still produces a proposal rather than being overwritten. Anvil never destroys repository content, and a special case here would be the one place it did. -What changes is that there is no longer a reason to edit: every legitimate addition — another base image, a root CA, a +What they remove is the reason to edit: every legitimate addition — another base image, a root CA, a build dependency, a run-time tool — has a gap that is the *correct* place for it, chosen by what must already be true at that point in the build. diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index fc8f96cb..8e1c8eb5 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -424,7 +424,7 @@ mod tests { fn recipe_has_no_generation_time_placeholders() { // Every value is a literal or resolved at run time; nothing is // substituted at emit time, so the recipe cannot drift from a - // configuration file that no longer exists. + // configuration file. assert!(!RECIPE.contains("__"), "the recipe must not carry rendering placeholders"); assert!(!RECIPE.contains("anvil.toml")); } @@ -432,7 +432,7 @@ mod tests { #[test] fn recipe_exposes_the_documented_surface() { for expected in [ - "anvil-container *target:", + "anvil-container *command:", "anvil-container-tag:", "anvil-container-status:", "anvil-container-down:", @@ -454,7 +454,7 @@ mod tests { // so each public recipe repeats a one-line summary immediately above // its attributes. for recipe in [ - "anvil-container *target:", + "anvil-container *command:", "anvil-container-tag:", "anvil-container-status:", "anvil-container-down:", @@ -599,7 +599,7 @@ mod tests { // And the escaping that is present uses just's own doubling form. assert!(RECIPE.contains(r#"replace(justfile_directory(), "'", "''")"#)); assert!(RECIPE.contains(r#"replace(invocation_directory_native(), "'", "''")"#)); - assert!(RECIPE.contains(r#"replace(target, "'", "''")"#)); + assert!(RECIPE.contains(r#"replace(command, "'", "''")"#)); } #[test] @@ -610,18 +610,22 @@ mod tests { } #[test] - fn a_recipe_argument_survives_as_its_own_word() { - // `*target` joins with spaces, so passing it through as one string - // would break `anvil-container anvil-setup binstall` -- and around - // fifty generated recipes take a parameter. + fn an_argument_survives_as_its_own_word() { + // `*command` joins with spaces, so passing it through as one string + // would run a single argument containing them all. assert!(RECIPE.contains(r"-split '\s+'")); - assert!(RECIPE.contains("}}' @targetParts")); - assert!(RECIPE.contains("@('just') + $targetParts")); - // Every nested call goes through the launching binary, including the - // in-container passthrough: ANVIL_IN_CONTAINER is a documented control - // a developer can set on a host, where a bare name resolves against - // PATH rather than the `just` that is running. - assert!(!RECIPE.contains("\n just @targetParts")); + // The argv is the command. Prefixing it would confine the recipe to + // `just` and make every other program unreachable. + assert!(RECIPE.contains("$runArgs += $argv")); + assert!(!RECIPE.contains("@('just') + $argv")); + } + + #[test] + fn a_nested_just_resolves_to_the_launching_binary() { + // ANVIL_IN_CONTAINER is a documented control a developer can set on a + // host, where a bare name resolves against PATH rather than the `just` + // that is running. + assert!(RECIPE.contains(r#"if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] }"#)); } #[test] @@ -655,10 +659,10 @@ mod tests { #[test] fn hook_file_is_an_image_input_but_hook_output_is_not() { - // A changed hook must rename the tag; a minted credential must not. - // The hook is no longer named individually -- it is picked up by the - // walk of `.anvil/container/`, which is what also catches a file a - // repository `COPY`s from one of the Dockerfile's user gaps. + // A changed hook must rename the tag; a minted credential must not. The + // hook is picked up by the walk of `.anvil/container/`, which is what + // also catches a file a repository `COPY`s from one of the Dockerfile's + // user gaps. assert!(RECIPE.contains("$containerRoot = Join-Path $repoRoot '.anvil/container'")); assert!(RECIPE.contains("id=$id,env=$name")); } @@ -810,31 +814,33 @@ mod tests { } #[test] - fn a_derived_token_is_scoped_to_a_target_that_reads_it() { + fn a_derived_token_is_scoped_to_a_command_that_reads_it() { // Forwarding an exported GITHUB_TOKEN is exact parity: natively it is // visible to every process the shell spawns too. Minting one from `gh` // is not -- PID 1's environment reaches every build script and proc // macro, where natively the recipe mints it in its own process -- so it - // happens only for a target whose plan reads the variable. + // happens only for a command whose plan reads the variable. let derive = RECIPE.find("gh auth token --hostname").expect("the gh fallback must exist"); let guard = RECIPE[..derive].rfind("if ($needsToken)").expect("the derive must be guarded"); let plan = RECIPE[..guard] .rfind("$plan -match 'GITHUB_TOKEN'") .expect("the plan must decide whether a token is needed"); let dry_run = RECIPE[..plan] - .rfind("--dry-run @targetParts") + .rfind("--dry-run @($argv | Select-Object -Skip 1)") .expect("the plan must come from just"); assert!(dry_run < plan && plan < guard, "compute the plan, match it, then derive"); // Through the launching binary, like every other nested call: a bare // `just` here fails silently when the caller invoked it by absolute // path, and an empty plan reads as "no token needed". assert!(!RECIPE.contains("(just --dry-run")); - assert!(RECIPE.contains("}}' --dry-run @targetParts")); + assert!(RECIPE.contains(r"}}' --dry-run @($argv | Select-Object -Skip 1)")); // The predicate is the variable, not the name of a check, so a catalog // that adds another GitHub-authenticated check is covered for free. assert!(!RECIPE.contains("$plan -match 'aprz'")); - // An interactive session has no target to plan, and can run anything. - assert!(RECIPE.contains("$needsToken = $targetParts.Count -eq 0")); + // An interactive session has no command to plan, and can run anything. + assert!(RECIPE.contains("$needsToken = $argv.Count -eq 0")); + // Only `just` can be planned, so nothing else earns a minted credential. + assert!(RECIPE.contains("if (-not $needsToken -and $argv[0] -eq 'just')")); } #[test] diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index 845a6770..010e0cfc 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -113,16 +113,17 @@ //! //! Execution is opt-in per invocation: `just anvil-pr` and every other recipe //! continue to run natively, and a container is entered only through -//! `anvil-container`, which takes any recipe name and its arguments. Those -//! arguments are whitespace-delimited tokens: `just` joins a variadic +//! `anvil-container`, whose arguments are the argv executed inside the image. +//! Those arguments are whitespace-delimited tokens: `just` joins a variadic //! parameter with spaces before the recipe sees it, so an argument that itself //! contains a space cannot be recovered and does not survive the round trip. //! //! ```text -//! just anvil-container anvil-clippy # one check -//! just anvil-container anvil-pr # the whole PR tier -//! just anvil-container anvil-setup binstall # a recipe with an argument -//! just anvil-container # interactive shell +//! just anvil-container just anvil-clippy # one check +//! just anvil-container just anvil-pr # the whole PR tier +//! just anvil-container just anvil-setup binstall # a recipe with an argument +//! just anvil-container cargo build # any other command +//! just anvil-container # interactive shell //! ``` //! //! The feature is three generated artifacts and one optional hook script, with @@ -198,7 +199,7 @@ //! //! ```text //! $env:ANVIL_CONTAINER_NO_CACHE = '1' -//! try { just anvil-container anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } +//! try { just anvil-container just anvil-fmt } finally { Remove-Item Env:ANVIL_CONTAINER_NO_CACHE } //! ``` //! //! ### The hook diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 85d4f8ac..cf0a1b54 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -515,8 +515,8 @@ fn composed_host_spec(host_relpath: &str) -> Option<(&'static str, &'static [&'s enum ComposedHostState { /// Carries every region anvil owns, in the declared order. Update in place. Composable, - /// Either absent, or a byte-identical render from the release that owned - /// this path outright. Every byte of it is anvil's, so the scaffold + /// Either absent, or a byte-identical render of a version that owned this + /// path as a whole file. Every byte of it is anvil's, so the scaffold /// replaces it and the regions rebuild the file. SeedFromScaffold, /// Anvil cannot reach a valid file from here without either destroying @@ -566,21 +566,20 @@ fn composed_host_state(order: &[&str], host_relpath: &str, text: &str, manifest: } if present.is_empty() { - // Nothing of anvil's is in the file. Either it is the previous - // release's whole-file render -- safe to replace, because every byte of - // it came from anvil -- or it is content anvil has never owned. + // Nothing of anvil's is in the file. Either it is a whole-file render + // anvil produced -- safe to replace, because every byte of it came from + // anvil -- or it is content anvil has never owned. // // The checksum comparison is the whole safety property: without it, a - // repository that edited the previously-owned Dockerfile would have - // that edit silently destroyed on upgrade. Appending the regions - // instead is not a kinder answer, because everything already in the - // file would then sit above `FROM`. + // repository that edited such a file would have that edit silently + // destroyed. Appending the regions instead is not a kinder answer, + // because everything already in the file would then sit above `FROM`. return match manifest.file_checksum(host_relpath) { Some(recorded) if recorded == checksum_str(text) => ComposedHostState::SeedFromScaffold, Some(_) => ComposedHostState::Unsafe( - "it was edited after anvil last wrote it, and this release composes the file from \ - managed regions instead of owning it whole. Move the edits you want to keep into \ - the gaps of a freshly generated file, or delete it and re-run to have one written" + "it was edited after anvil last wrote it, and anvil composes the file from managed \ + regions rather than owning it whole. Move the edits you want to keep into the \ + gaps of a freshly generated file, or delete it and re-run to have one written" .to_owned(), ), // A composed host is recorded in `regions` and never in `files`, so @@ -767,12 +766,12 @@ fn plan_removals( let resolved_host = resolve_existing_case_insensitive(repo_root, &key.host); // A refused host was not opened, and "nothing was written to it" has to // be true of the lock as well as the file -- the same invariant the - // owned-file loop above keeps. Without this, a lock entry naming a - // region the catalog no longer declares still reaches `remove_region` - // below, so the run splices a block out of the very file whose refusal - // says it was left alone, and purges the provenance the next run - // reclassifies from. Latent while every declared id is live, reachable - // at the first region rename or retirement. + // owned-file loop above keeps. A lock entry naming a region the catalog + // does not declare would otherwise reach `remove_region` below, so the + // run would splice a block out of the very file whose refusal says it + // was left alone, and purge the provenance the next run reclassifies + // from. Unreachable while every declared id is live; a region rename or + // retirement is what exposes it. if matches!(composed.states.get(&resolved_host), Some(ComposedHostState::Unsafe(_))) { continue; } diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 6c2350b8..678eae3c 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -46,8 +46,8 @@ anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing # Docker in WSL without Docker Desktop is a documented, common setup, and it -# leaves no Windows CLI behind -- so without this the engine we told the user to -# install would be unreachable. Docker Desktop and Podman both ship a Windows +# leaves no Windows CLI behind, so without this fallback a correctly installed +# engine would be unreachable. Docker Desktop and Podman both ship a Windows # CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] @@ -493,40 +493,41 @@ _anvil-container-image: Write-Output $image -# Run any anvil recipe inside the pinned Linux image. +# Run a command inside the pinned Linux image. # -# just anvil-container anvil-clippy # one check -# just anvil-container anvil-pr # the whole PR tier -# just anvil-container # interactive shell +# The tokens after the recipe name are the argv, executed verbatim in the +# image. Anvil recipes are reached by naming `just` like any other command. # -# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs -# natively and the work happens exactly once. +# just anvil-container just anvil-pr # a tier +# just anvil-container cargo build # any other command +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs the +# command on the spot and the work happens exactly once. -# Run any anvil recipe inside the pinned Linux image (no argument: a shell). +# Run a command inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *target: +anvil-container *command: $ErrorActionPreference = 'Stop' - # Split back into argv. `*target` joins its parts with spaces, so passing - # the string through as one argument would make `anvil-container anvil-setup - # binstall` look for a recipe literally named "anvil-setup binstall" -- - # and around fifty generated recipes take a parameter. - $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) + # `*command` joins its parts with spaces, so the string is split back into + # argv here. Whitespace is the only separator, so an argument containing a + # space does not survive; pass such a value through the environment. + $argv = @('{{ replace(command, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { - # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -eq 0) { - # The no-argument form asks for a shell in the image, and this *is* + if ($argv.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this is # that shell. Nothing runs, so exiting 0 would report success for a # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - # Through the launching binary like every other nested call. Inside the - # image `just` is on PATH so the bare name would work, but + # `just` resolves to the binary running this tree rather than to PATH: # ANVIL_IN_CONTAINER is a documented control a developer can set on a - # host, and there the bare name resolves against PATH rather than the - # `just` that is actually running. - & '{{ replace(just_executable(), "'", "''") }}' @targetParts + # host, and a caller who invoked `just` by absolute path with its + # directory off PATH would otherwise fail here. + $exe = if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] } + & $exe @($argv | Select-Object -Skip 1) exit $LASTEXITCODE } @@ -564,7 +565,7 @@ anvil-container *target: '{{anvil_container_workdir}}/' + $rel } - $interactive = $targetParts.Count -eq 0 + $interactive = $argv.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -586,8 +587,8 @@ anvil-container *target: # bind mount already carries it, and takes none of this. # # Guarded on git being present: the run path needs it only to answer this - # question, and a host with a working engine but no git on PATH should keep - # working rather than fail on a call it did not used to make. + # question, and a host with a working engine but no git on PATH keeps + # working rather than failing on a call it does not need. $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null @@ -658,14 +659,14 @@ anvil-container *target: # first, then the gh CLI's stored token -- so a containerized run # authenticates for the same developers a native run does. # - # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # An already-exported GITHUB_TOKEN is forwarded whatever the command is: # that is exact parity, since a native run exposes it to every process the # shell spawns too. Deriving one from `gh` is different -- it manufactures a # credential the developer did not put in this environment, and PID 1's # environment is inherited by every build script and proc macro in the # container, where natively the recipe would mint it in its own process. So - # it is derived only when the target actually reads the variable, or when - # there is no target at all: an interactive session can run anything, and + # it is derived only when the command is known to read the variable, or when + # there is no command at all: an interactive session can run anything, and # refusing there would reintroduce the silent hour-long stall on a tier the # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. @@ -677,12 +678,15 @@ anvil-container *target: # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - # Ask just what the target would run. A dry run has no side effects, and - # a target that cannot be planned (a typo, a recipe needing arguments) - # yields nothing, so the run fails on its own terms rather than on a - # missing token. - $needsToken = $targetParts.Count -eq 0 - if (-not $needsToken) { + $needsToken = $argv.Count -eq 0 + # Only a `just` command can be planned, and planning is the only way to + # know whether what runs reads the variable. Anything else keeps the + # environment it was given: a manufactured credential reaches every + # process in the container, so an unknown command does not earn one. + if (-not $needsToken -and $argv[0] -eq 'just') { + # A dry run has no side effects, and a target that cannot be planned + # (a typo, a recipe needing arguments) yields nothing, so the run + # fails on its own terms rather than on a missing token. $plan = '' # The same executable that launched this tree, for the reason every # other nested call uses it: a caller invoking `just` by absolute @@ -691,7 +695,7 @@ anvil-container *target: # token" -- so anvil-aprz would run unauthenticated in an image with # no gh of its own and block on the rate limit for up to an hour. try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' @@ -778,7 +782,7 @@ anvil-container *target: # built locally or fetched by the resolve hook, so a miss is a bug to # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just') + $targetParts } + if (-not $interactive) { $runArgs += $argv } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index 76ec30b4..a4266535 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -13,20 +13,19 @@ reason = "integration tests panic on unmet preconditions for readable failure output" )] -//! Consumer-upgrade coverage for the container backend replacement. +//! Consumer-upgrade coverage for the container backend. //! //! The snapshot tests describe a fresh tree: the end state of a generation that //! starts from nothing. This exercises the path an existing adopter takes -- //! a repository generated by 0.4.0, holding a `.anvil.lock` that tracks the //! runner seam and its assets, updated by a binary that emits neither. //! -//! The transition is the largest this crate has shipped: `Containerfile` and -//! its ignore file, `README.md`, `entrypoint.sh`, `image-id.{sh,ps1}` and -//! `run-in-container.{sh,ps1}` all disappear from `.anvil/container/`, -//! `justfiles/anvil/runner.just` disappears with them, and the root `Justfile` -//! loses its `anvil-runner` region. What matters is that an untouched asset is -//! removed cleanly and an *edited* one is handed back to the repository rather -//! than deleted. +//! Across that upgrade `Containerfile` and its ignore file, `README.md`, +//! `entrypoint.sh`, `image-id.{sh,ps1}` and `run-in-container.{sh,ps1}` all +//! disappear from `.anvil/container/`, `justfiles/anvil/runner.just` +//! disappears with them, and the root `Justfile` loses its `anvil-runner` +//! region. What matters is that an untouched asset is removed cleanly and an +//! *edited* one is handed back to the repository rather than deleted. use std::path::Path; @@ -34,8 +33,8 @@ use cargo_anvil::test_support::{Cli, Decision, Manifest, RegionKey, RunOutcome, use cargo_anvil::{Catalog, artifacts}; use tempfile::TempDir; -/// Generated container assets that 0.4.0 tracked and this release does not. -/// Paths are as 0.4.0 wrote them. +/// Generated container assets that 0.4.0 tracked and the current catalog does +/// not. Paths are as 0.4.0 wrote them. const RETIRED_ASSETS: [&str; 8] = [ ".anvil/container/Containerfile", ".anvil/container/Containerfile.dockerignore", @@ -95,7 +94,7 @@ fn generated_tree() -> TempDir { } /// Rewrite a freshly generated tree into the shape 0.4.0 produced: the retired -/// assets present on disk and tracked in the lock, and none of this release's +/// assets present on disk and tracked in the lock, and none of the current /// container artifacts present at all. /// /// The checksum recorded for each retired asset is the checksum of the body @@ -188,7 +187,7 @@ fn upgrading_from_the_runner_layout_retires_the_seam_and_emits_the_new_backend() assert!(!manifest.files.contains_key(path), "{path} must be dropped from the lock"); } - // This release's artifacts take their place and are tracked. + // The current artifacts take their place and are tracked. for artifact in artifacts::container::all() { match artifact { cargo_anvil::Artifact::OwnedFile(spec) => { @@ -388,7 +387,7 @@ fn a_repository_authored_dockerfile_is_refused_not_appended_to() { ); } -/// The upgrade re-seed replaces the previous release's whole-file render. It +/// The upgrade re-seed replaces a whole-file render anvil produced. It /// must not do that when the repository edited that file: the file is not /// tracked region-by-region, so there is no proposal to fall back on and /// nothing to recover the edit from. @@ -835,15 +834,15 @@ fn a_case_only_rename_of_an_owned_file_is_not_deleted() { ); } -/// The refusal must be atomic across *both* removal loops. The owned-file loop -/// consults `composed.states`; the region loop did not, so a lock entry naming -/// a region the catalog no longer declares still reached `remove_region` -- +/// The refusal is atomic across *both* removal loops: a refused host keeps its +/// file and its lock entries whichever loop reaches them. A lock entry naming a +/// region the catalog does not declare would otherwise reach `remove_region`, /// splicing a block out of the very file whose refusal says it was untouched, /// and purging the provenance the next run reclassifies from. /// -/// Reaching it needs a retired region id, which today's catalog has none of, so -/// the stale key is planted here. That is the same upgrade shape the ordered -/// insertion work exists for, so it is one rename away from being live. +/// Reaching that path needs a retired region id, which the catalog has none of, +/// so the stale key is planted here. It is one region rename away from being +/// an ordinary upgrade. #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] #[test] fn refusing_a_composed_host_spares_a_retired_region_entry_too() { diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index d134bb49..5c226ea8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3605,8 +3605,8 @@ anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing # Docker in WSL without Docker Desktop is a documented, common setup, and it -# leaves no Windows CLI behind -- so without this the engine we told the user to -# install would be unreachable. Docker Desktop and Podman both ship a Windows +# leaves no Windows CLI behind, so without this fallback a correctly installed +# engine would be unreachable. Docker Desktop and Podman both ship a Windows # CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] @@ -4052,40 +4052,41 @@ _anvil-container-image: Write-Output $image -# Run any anvil recipe inside the pinned Linux image. +# Run a command inside the pinned Linux image. # -# just anvil-container anvil-clippy # one check -# just anvil-container anvil-pr # the whole PR tier -# just anvil-container # interactive shell +# The tokens after the recipe name are the argv, executed verbatim in the +# image. Anvil recipes are reached by naming `just` like any other command. # -# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs -# natively and the work happens exactly once. +# just anvil-container just anvil-pr # a tier +# just anvil-container cargo build # any other command +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs the +# command on the spot and the work happens exactly once. -# Run any anvil recipe inside the pinned Linux image (no argument: a shell). +# Run a command inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *target: +anvil-container *command: $ErrorActionPreference = 'Stop' - # Split back into argv. `*target` joins its parts with spaces, so passing - # the string through as one argument would make `anvil-container anvil-setup - # binstall` look for a recipe literally named "anvil-setup binstall" -- - # and around fifty generated recipes take a parameter. - $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) + # `*command` joins its parts with spaces, so the string is split back into + # argv here. Whitespace is the only separator, so an argument containing a + # space does not survive; pass such a value through the environment. + $argv = @('{{ replace(command, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { - # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -eq 0) { - # The no-argument form asks for a shell in the image, and this *is* + if ($argv.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this is # that shell. Nothing runs, so exiting 0 would report success for a # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - # Through the launching binary like every other nested call. Inside the - # image `just` is on PATH so the bare name would work, but + # `just` resolves to the binary running this tree rather than to PATH: # ANVIL_IN_CONTAINER is a documented control a developer can set on a - # host, and there the bare name resolves against PATH rather than the - # `just` that is actually running. - & '{{ replace(just_executable(), "'", "''") }}' @targetParts + # host, and a caller who invoked `just` by absolute path with its + # directory off PATH would otherwise fail here. + $exe = if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] } + & $exe @($argv | Select-Object -Skip 1) exit $LASTEXITCODE } @@ -4123,7 +4124,7 @@ anvil-container *target: '{{anvil_container_workdir}}/' + $rel } - $interactive = $targetParts.Count -eq 0 + $interactive = $argv.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -4145,8 +4146,8 @@ anvil-container *target: # bind mount already carries it, and takes none of this. # # Guarded on git being present: the run path needs it only to answer this - # question, and a host with a working engine but no git on PATH should keep - # working rather than fail on a call it did not used to make. + # question, and a host with a working engine but no git on PATH keeps + # working rather than failing on a call it does not need. $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null @@ -4217,14 +4218,14 @@ anvil-container *target: # first, then the gh CLI's stored token -- so a containerized run # authenticates for the same developers a native run does. # - # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # An already-exported GITHUB_TOKEN is forwarded whatever the command is: # that is exact parity, since a native run exposes it to every process the # shell spawns too. Deriving one from `gh` is different -- it manufactures a # credential the developer did not put in this environment, and PID 1's # environment is inherited by every build script and proc macro in the # container, where natively the recipe would mint it in its own process. So - # it is derived only when the target actually reads the variable, or when - # there is no target at all: an interactive session can run anything, and + # it is derived only when the command is known to read the variable, or when + # there is no command at all: an interactive session can run anything, and # refusing there would reintroduce the silent hour-long stall on a tier the # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. @@ -4236,12 +4237,15 @@ anvil-container *target: # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - # Ask just what the target would run. A dry run has no side effects, and - # a target that cannot be planned (a typo, a recipe needing arguments) - # yields nothing, so the run fails on its own terms rather than on a - # missing token. - $needsToken = $targetParts.Count -eq 0 - if (-not $needsToken) { + $needsToken = $argv.Count -eq 0 + # Only a `just` command can be planned, and planning is the only way to + # know whether what runs reads the variable. Anything else keeps the + # environment it was given: a manufactured credential reaches every + # process in the container, so an unknown command does not earn one. + if (-not $needsToken -and $argv[0] -eq 'just') { + # A dry run has no side effects, and a target that cannot be planned + # (a typo, a recipe needing arguments) yields nothing, so the run + # fails on its own terms rather than on a missing token. $plan = '' # The same executable that launched this tree, for the reason every # other nested call uses it: a caller invoking `just` by absolute @@ -4250,7 +4254,7 @@ anvil-container *target: # token" -- so anvil-aprz would run unauthenticated in an image with # no gh of its own and block on the rate limit for up to an hour. try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' @@ -4337,7 +4341,7 @@ anvil-container *target: # built locally or fetched by the resolve hook, so a miss is a bug to # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just') + $targetParts } + if (-not $interactive) { $runArgs += $argv } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f24ab9e8..682849a4 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3504,8 +3504,8 @@ anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing # Docker in WSL without Docker Desktop is a documented, common setup, and it -# leaves no Windows CLI behind -- so without this the engine we told the user to -# install would be unreachable. Docker Desktop and Podman both ship a Windows +# leaves no Windows CLI behind, so without this fallback a correctly installed +# engine would be unreachable. Docker Desktop and Podman both ship a Windows # CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] @@ -3951,40 +3951,41 @@ _anvil-container-image: Write-Output $image -# Run any anvil recipe inside the pinned Linux image. +# Run a command inside the pinned Linux image. # -# just anvil-container anvil-clippy # one check -# just anvil-container anvil-pr # the whole PR tier -# just anvil-container # interactive shell +# The tokens after the recipe name are the argv, executed verbatim in the +# image. Anvil recipes are reached by naming `just` like any other command. # -# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs -# natively and the work happens exactly once. +# just anvil-container just anvil-pr # a tier +# just anvil-container cargo build # any other command +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs the +# command on the spot and the work happens exactly once. -# Run any anvil recipe inside the pinned Linux image (no argument: a shell). +# Run a command inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *target: +anvil-container *command: $ErrorActionPreference = 'Stop' - # Split back into argv. `*target` joins its parts with spaces, so passing - # the string through as one argument would make `anvil-container anvil-setup - # binstall` look for a recipe literally named "anvil-setup binstall" -- - # and around fifty generated recipes take a parameter. - $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) + # `*command` joins its parts with spaces, so the string is split back into + # argv here. Whitespace is the only separator, so an argument containing a + # space does not survive; pass such a value through the environment. + $argv = @('{{ replace(command, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { - # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -eq 0) { - # The no-argument form asks for a shell in the image, and this *is* + if ($argv.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this is # that shell. Nothing runs, so exiting 0 would report success for a # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - # Through the launching binary like every other nested call. Inside the - # image `just` is on PATH so the bare name would work, but + # `just` resolves to the binary running this tree rather than to PATH: # ANVIL_IN_CONTAINER is a documented control a developer can set on a - # host, and there the bare name resolves against PATH rather than the - # `just` that is actually running. - & '{{ replace(just_executable(), "'", "''") }}' @targetParts + # host, and a caller who invoked `just` by absolute path with its + # directory off PATH would otherwise fail here. + $exe = if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] } + & $exe @($argv | Select-Object -Skip 1) exit $LASTEXITCODE } @@ -4022,7 +4023,7 @@ anvil-container *target: '{{anvil_container_workdir}}/' + $rel } - $interactive = $targetParts.Count -eq 0 + $interactive = $argv.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -4044,8 +4045,8 @@ anvil-container *target: # bind mount already carries it, and takes none of this. # # Guarded on git being present: the run path needs it only to answer this - # question, and a host with a working engine but no git on PATH should keep - # working rather than fail on a call it did not used to make. + # question, and a host with a working engine but no git on PATH keeps + # working rather than failing on a call it does not need. $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null @@ -4116,14 +4117,14 @@ anvil-container *target: # first, then the gh CLI's stored token -- so a containerized run # authenticates for the same developers a native run does. # - # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # An already-exported GITHUB_TOKEN is forwarded whatever the command is: # that is exact parity, since a native run exposes it to every process the # shell spawns too. Deriving one from `gh` is different -- it manufactures a # credential the developer did not put in this environment, and PID 1's # environment is inherited by every build script and proc macro in the # container, where natively the recipe would mint it in its own process. So - # it is derived only when the target actually reads the variable, or when - # there is no target at all: an interactive session can run anything, and + # it is derived only when the command is known to read the variable, or when + # there is no command at all: an interactive session can run anything, and # refusing there would reintroduce the silent hour-long stall on a tier the # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. @@ -4135,12 +4136,15 @@ anvil-container *target: # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - # Ask just what the target would run. A dry run has no side effects, and - # a target that cannot be planned (a typo, a recipe needing arguments) - # yields nothing, so the run fails on its own terms rather than on a - # missing token. - $needsToken = $targetParts.Count -eq 0 - if (-not $needsToken) { + $needsToken = $argv.Count -eq 0 + # Only a `just` command can be planned, and planning is the only way to + # know whether what runs reads the variable. Anything else keeps the + # environment it was given: a manufactured credential reaches every + # process in the container, so an unknown command does not earn one. + if (-not $needsToken -and $argv[0] -eq 'just') { + # A dry run has no side effects, and a target that cannot be planned + # (a typo, a recipe needing arguments) yields nothing, so the run + # fails on its own terms rather than on a missing token. $plan = '' # The same executable that launched this tree, for the reason every # other nested call uses it: a caller invoking `just` by absolute @@ -4149,7 +4153,7 @@ anvil-container *target: # token" -- so anvil-aprz would run unauthenticated in an image with # no gh of its own and block on the rate limit for up to an hour. try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' @@ -4236,7 +4240,7 @@ anvil-container *target: # built locally or fetched by the resolve hook, so a miss is a bug to # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just') + $targetParts } + if (-not $interactive) { $runArgs += $argv } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 657bb1fb..e7e80bc1 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2345,8 +2345,8 @@ anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing # Docker in WSL without Docker Desktop is a documented, common setup, and it -# leaves no Windows CLI behind -- so without this the engine we told the user to -# install would be unreachable. Docker Desktop and Podman both ship a Windows +# leaves no Windows CLI behind, so without this fallback a correctly installed +# engine would be unreachable. Docker Desktop and Podman both ship a Windows # CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] @@ -2792,40 +2792,41 @@ _anvil-container-image: Write-Output $image -# Run any anvil recipe inside the pinned Linux image. +# Run a command inside the pinned Linux image. # -# just anvil-container anvil-clippy # one check -# just anvil-container anvil-pr # the whole PR tier -# just anvil-container # interactive shell +# The tokens after the recipe name are the argv, executed verbatim in the +# image. Anvil recipes are reached by naming `just` like any other command. # -# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs -# natively and the work happens exactly once. +# just anvil-container just anvil-pr # a tier +# just anvil-container cargo build # any other command +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs the +# command on the spot and the work happens exactly once. -# Run any anvil recipe inside the pinned Linux image (no argument: a shell). +# Run a command inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *target: +anvil-container *command: $ErrorActionPreference = 'Stop' - # Split back into argv. `*target` joins its parts with spaces, so passing - # the string through as one argument would make `anvil-container anvil-setup - # binstall` look for a recipe literally named "anvil-setup binstall" -- - # and around fifty generated recipes take a parameter. - $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) + # `*command` joins its parts with spaces, so the string is split back into + # argv here. Whitespace is the only separator, so an argument containing a + # space does not survive; pass such a value through the environment. + $argv = @('{{ replace(command, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { - # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -eq 0) { - # The no-argument form asks for a shell in the image, and this *is* + if ($argv.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this is # that shell. Nothing runs, so exiting 0 would report success for a # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - # Through the launching binary like every other nested call. Inside the - # image `just` is on PATH so the bare name would work, but + # `just` resolves to the binary running this tree rather than to PATH: # ANVIL_IN_CONTAINER is a documented control a developer can set on a - # host, and there the bare name resolves against PATH rather than the - # `just` that is actually running. - & '{{ replace(just_executable(), "'", "''") }}' @targetParts + # host, and a caller who invoked `just` by absolute path with its + # directory off PATH would otherwise fail here. + $exe = if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] } + & $exe @($argv | Select-Object -Skip 1) exit $LASTEXITCODE } @@ -2863,7 +2864,7 @@ anvil-container *target: '{{anvil_container_workdir}}/' + $rel } - $interactive = $targetParts.Count -eq 0 + $interactive = $argv.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -2885,8 +2886,8 @@ anvil-container *target: # bind mount already carries it, and takes none of this. # # Guarded on git being present: the run path needs it only to answer this - # question, and a host with a working engine but no git on PATH should keep - # working rather than fail on a call it did not used to make. + # question, and a host with a working engine but no git on PATH keeps + # working rather than failing on a call it does not need. $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null @@ -2957,14 +2958,14 @@ anvil-container *target: # first, then the gh CLI's stored token -- so a containerized run # authenticates for the same developers a native run does. # - # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # An already-exported GITHUB_TOKEN is forwarded whatever the command is: # that is exact parity, since a native run exposes it to every process the # shell spawns too. Deriving one from `gh` is different -- it manufactures a # credential the developer did not put in this environment, and PID 1's # environment is inherited by every build script and proc macro in the # container, where natively the recipe would mint it in its own process. So - # it is derived only when the target actually reads the variable, or when - # there is no target at all: an interactive session can run anything, and + # it is derived only when the command is known to read the variable, or when + # there is no command at all: an interactive session can run anything, and # refusing there would reintroduce the silent hour-long stall on a tier the # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. @@ -2976,12 +2977,15 @@ anvil-container *target: # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - # Ask just what the target would run. A dry run has no side effects, and - # a target that cannot be planned (a typo, a recipe needing arguments) - # yields nothing, so the run fails on its own terms rather than on a - # missing token. - $needsToken = $targetParts.Count -eq 0 - if (-not $needsToken) { + $needsToken = $argv.Count -eq 0 + # Only a `just` command can be planned, and planning is the only way to + # know whether what runs reads the variable. Anything else keeps the + # environment it was given: a manufactured credential reaches every + # process in the container, so an unknown command does not earn one. + if (-not $needsToken -and $argv[0] -eq 'just') { + # A dry run has no side effects, and a target that cannot be planned + # (a typo, a recipe needing arguments) yields nothing, so the run + # fails on its own terms rather than on a missing token. $plan = '' # The same executable that launched this tree, for the reason every # other nested call uses it: a caller invoking `just` by absolute @@ -2990,7 +2994,7 @@ anvil-container *target: # token" -- so anvil-aprz would run unauthenticated in an image with # no gh of its own and block on the rate limit for up to an hour. try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' @@ -3077,7 +3081,7 @@ anvil-container *target: # built locally or fetched by the resolve hook, so a miss is a bug to # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just') + $targetParts } + if (-not $interactive) { $runArgs += $argv } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 6c2350b8..678eae3c 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -46,8 +46,8 @@ anvil_container_name := trim_end_matches("anvil-" + replace_regex(lowercase(file # The one fallback is Windows-specific and unambiguous: when the engine is not # on the Windows PATH, try it inside the default WSL distribution. Installing # Docker in WSL without Docker Desktop is a documented, common setup, and it -# leaves no Windows CLI behind -- so without this the engine we told the user to -# install would be unreachable. Docker Desktop and Podman both ship a Windows +# leaves no Windows CLI behind, so without this fallback a correctly installed +# engine would be unreachable. Docker Desktop and Podman both ship a Windows # CLI and are found on PATH, so they never take this path. [private] [script("pwsh", "-NoProfile")] @@ -493,40 +493,41 @@ _anvil-container-image: Write-Output $image -# Run any anvil recipe inside the pinned Linux image. +# Run a command inside the pinned Linux image. # -# just anvil-container anvil-clippy # one check -# just anvil-container anvil-pr # the whole PR tier -# just anvil-container # interactive shell +# The tokens after the recipe name are the argv, executed verbatim in the +# image. Anvil recipes are reached by naming `just` like any other command. # -# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs -# natively and the work happens exactly once. +# just anvil-container just anvil-pr # a tier +# just anvil-container cargo build # any other command +# just anvil-container # interactive shell +# +# ANVIL_IN_CONTAINER is set inside the image, so a nested invocation runs the +# command on the spot and the work happens exactly once. -# Run any anvil recipe inside the pinned Linux image (no argument: a shell). +# Run a command inside the pinned Linux image (no argument: a shell). [group("anvil-container")] [script("pwsh", "-NoProfile")] -anvil-container *target: +anvil-container *command: $ErrorActionPreference = 'Stop' - # Split back into argv. `*target` joins its parts with spaces, so passing - # the string through as one argument would make `anvil-container anvil-setup - # binstall` look for a recipe literally named "anvil-setup binstall" -- - # and around fifty generated recipes take a parameter. - $targetParts = @('{{ replace(target, "'", "''") }}' -split '\s+' | Where-Object { $_ }) + # `*command` joins its parts with spaces, so the string is split back into + # argv here. Whitespace is the only separator, so an argument containing a + # space does not survive; pass such a value through the environment. + $argv = @('{{ replace(command, "'", "''") }}' -split '\s+' | Where-Object { $_ }) if ($env:ANVIL_IN_CONTAINER -eq '1') { - # Already inside: pass straight through instead of nesting. - if ($targetParts.Count -eq 0) { - # The no-argument form asks for a shell in the image, and this *is* + if ($argv.Count -eq 0) { + # The no-argument form asks for a shell in the image, and this is # that shell. Nothing runs, so exiting 0 would report success for a # request that was not carried out. [Console]::Error.WriteLine("anvil: already inside the container; run the command directly") exit 1 } - # Through the launching binary like every other nested call. Inside the - # image `just` is on PATH so the bare name would work, but + # `just` resolves to the binary running this tree rather than to PATH: # ANVIL_IN_CONTAINER is a documented control a developer can set on a - # host, and there the bare name resolves against PATH rather than the - # `just` that is actually running. - & '{{ replace(just_executable(), "'", "''") }}' @targetParts + # host, and a caller who invoked `just` by absolute path with its + # directory off PATH would otherwise fail here. + $exe = if ($argv[0] -eq 'just') { '{{ replace(just_executable(), "'", "''") }}' } else { $argv[0] } + & $exe @($argv | Select-Object -Skip 1) exit $LASTEXITCODE } @@ -564,7 +565,7 @@ anvil-container *target: '{{anvil_container_workdir}}/' + $rel } - $interactive = $targetParts.Count -eq 0 + $interactive = $argv.Count -eq 0 $runArgs = @('run', '--rm', '--platform', 'linux/amd64') $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") @@ -586,8 +587,8 @@ anvil-container *target: # bind mount already carries it, and takes none of this. # # Guarded on git being present: the run path needs it only to answer this - # question, and a host with a working engine but no git on PATH should keep - # working rather than fail on a call it did not used to make. + # question, and a host with a working engine but no git on PATH keeps + # working rather than failing on a call it does not need. $gitFile = $null if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null @@ -658,14 +659,14 @@ anvil-container *target: # first, then the gh CLI's stored token -- so a containerized run # authenticates for the same developers a native run does. # - # An already-exported GITHUB_TOKEN is forwarded whatever the target is: + # An already-exported GITHUB_TOKEN is forwarded whatever the command is: # that is exact parity, since a native run exposes it to every process the # shell spawns too. Deriving one from `gh` is different -- it manufactures a # credential the developer did not put in this environment, and PID 1's # environment is inherited by every build script and proc macro in the # container, where natively the recipe would mint it in its own process. So - # it is derived only when the target actually reads the variable, or when - # there is no target at all: an interactive session can run anything, and + # it is derived only when the command is known to read the variable, or when + # there is no command at all: an interactive session can run anything, and # refusing there would reintroduce the silent hour-long stall on a tier the # developer runs from inside the shell. # `gh auth token` is non-interactive and never opens a prompt. @@ -677,12 +678,15 @@ anvil-container *target: # Set here and passed by NAME, so the value never reaches the host's # process command line, and unset again with the hook's variables below. if (-not $env:GITHUB_TOKEN -and (Get-Command gh -ErrorAction SilentlyContinue)) { - # Ask just what the target would run. A dry run has no side effects, and - # a target that cannot be planned (a typo, a recipe needing arguments) - # yields nothing, so the run fails on its own terms rather than on a - # missing token. - $needsToken = $targetParts.Count -eq 0 - if (-not $needsToken) { + $needsToken = $argv.Count -eq 0 + # Only a `just` command can be planned, and planning is the only way to + # know whether what runs reads the variable. Anything else keeps the + # environment it was given: a manufactured credential reaches every + # process in the container, so an unknown command does not earn one. + if (-not $needsToken -and $argv[0] -eq 'just') { + # A dry run has no side effects, and a target that cannot be planned + # (a typo, a recipe needing arguments) yields nothing, so the run + # fails on its own terms rather than on a missing token. $plan = '' # The same executable that launched this tree, for the reason every # other nested call uses it: a caller invoking `just` by absolute @@ -691,7 +695,7 @@ anvil-container *target: # token" -- so anvil-aprz would run unauthenticated in an image with # no gh of its own and block on the rate limit for up to an hour. try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @targetParts 2>&1 | + $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | ForEach-Object { $_.ToString() }) -join "`n" } catch { $plan = '' @@ -778,7 +782,7 @@ anvil-container *target: # built locally or fetched by the resolve hook, so a miss is a bug to # surface rather than an invitation to fetch something unrelated. $runArgs += @('--pull=never', '-w', $containerCwd, $image) - if (-not $interactive) { $runArgs += @('just') + $targetParts } + if (-not $interactive) { $runArgs += $argv } # WSLENV exports the forwarded names into the WSL environment, which is # where the engine reads their values from when it runs there. Without # it, `-e NAME` reaches an engine that cannot see NAME and forwards From e4599dd994f4e0fd8eb2aadfc1daf9006e955f82 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 13:01:22 +0200 Subject: [PATCH 63/81] fix(anvil): restore the tests the merge dropped and untrack stray proposals Resolving the merge lost five tests that came with the impact feature: the two covering `impact_mode` (which left its arms uncovered and put the crate under the coverage threshold), and the three pinning the impact recipe, each check's declared impact policy, and the cargo-delta prerequisite of impact-scoped groups. The tier and scheduled-group assertions move to `_anvil-unscoped`, `mod.just`'s optional container import is asserted as optional, and the runner-specific assertions go with the seam they described. `aprz.just` gets a check matching how the container driver supplies a token: forwarded by name, not mounted as a secret file. `dependency_recipe_sources` is removed; its only caller lived in the container backend this branch replaces. A new recipe contract covers what `tier_routing.rs` used to: a fixture dependency that fails unless ANVIL_IMPACT is already set proves `_anvil-unscoped` exports it before the wrapped recipe's dependencies run, with a direct invocation as the negative control. The five `.anvil-proposed` review artifacts committed with the merge are untracked and deleted; each was byte-identical to its live file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .../src/anvil/artifacts/justfile.rs | 316 ++++++++++++++++-- crates/cargo-anvil/src/anvil/artifacts/mod.rs | 14 + crates/cargo-anvil/tests/recipe_contracts.rs | 37 ++ .../scheduled-advisories.just.anvil-proposed | 39 --- .../scheduled-exhaustive.just.anvil-proposed | 36 -- ...duled-runtime-analysis.just.anvil-proposed | 45 --- .../groups/scheduled-test.just.anvil-proposed | 40 --- justfiles/anvil/helpers.just.anvil-proposed | 137 -------- 8 files changed, 336 insertions(+), 328 deletions(-) delete mode 100644 justfiles/anvil/groups/scheduled-advisories.just.anvil-proposed delete mode 100644 justfiles/anvil/groups/scheduled-exhaustive.just.anvil-proposed delete mode 100644 justfiles/anvil/groups/scheduled-runtime-analysis.just.anvil-proposed delete mode 100644 justfiles/anvil/groups/scheduled-test.just.anvil-proposed delete mode 100644 justfiles/anvil/helpers.just.anvil-proposed diff --git a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs index e82c7df6..db458762 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/justfile.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/justfile.rs @@ -35,9 +35,11 @@ const TOOLS_JUST_PATH: &str = "justfiles/anvil/tools.just"; /// Contents of `justfiles/anvil/helpers.just` baked into the binary. /// -/// Holds the shared helper recipes (`_anvil-base-ref`, -/// `_anvil-impact-format`) and the impact env-var contract that the -/// per-check recipes rely on. +/// Holds the shared helper recipe `_anvil-base-ref` (reused by the impact +/// recipe and anvil-mutants-diff) and the bucket legend documenting how +/// per-check recipes consume the impact cache via `_anvil-impact-include` +/// (which, along with cache production and `_anvil-impact-format`, lives in +/// `impact.just`). const HELPERS_JUST: &str = include_str!("../../../templates/justfiles/anvil/helpers.just"); /// Repo-root-relative path of the shared-helpers recipe file. @@ -73,17 +75,16 @@ macro_rules! split_recipe_files { } #[test] -fn aprz_borrows_a_token_and_degrades_with_instructions() { +fn aprz_forwards_a_github_token_into_the_container() { let aprz = CHECK_FILES .iter() .find_map(|(path, body)| path.ends_with("/aprz.just").then_some(*body)) .expect("aprz.just is registered in CHECK_FILES below"); - assert!(aprz.contains("gh auth token --hostname github.com")); - assert!(aprz.contains("GITHUB_TOKEN is not set")); - // Nothing container-specific: inside the image a credential arrives as an - // ordinary environment variable, from the hook or from CI. - assert!(!aprz.contains("ANVIL_IN_CONTAINER")); - assert!(!aprz.contains("/run/secrets/")); + // The container driver forwards GITHUB_TOKEN by name, so the check reads + // the variable and says how to obtain one rather than reaching for a + // mounted secret path. + assert!(aprz.contains("GITHUB_TOKEN")); + assert!(aprz.contains("gh auth")); } /// One `justfiles/anvil/checks/.just` file per catalog check @@ -214,8 +215,78 @@ pub fn tiers() -> Artifact { #[cfg(test)] #[cfg_attr(coverage_nightly, coverage(off))] mod tests { + use std::collections::{BTreeMap, BTreeSet}; + + use ImpactPolicy::{Affected, Modified, Required, Unscoped}; + use super::*; + /// A check's impact-scoping policy: either unscoped (always runs the full + /// workspace) or scoped to one cargo-delta impact category. + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + enum ImpactPolicy { + Unscoped, + Modified, + Affected, + Required, + } + + impl ImpactPolicy { + /// The `_anvil-impact-include` category argument this policy emits, or + /// `None` when the check is unscoped and takes no impact dependency. + fn category(self) -> Option<&'static str> { + match self { + Self::Unscoped => None, + Self::Modified => Some("modified"), + Self::Affected => Some("affected"), + Self::Required => Some("required"), + } + } + } + + /// The intended impact policy for every catalog check -- the canonical + /// mapping `every_check_matches_its_declared_impact_policy` enforces + /// against the emitted recipes. Keep in sync with the check mapping table + /// in the design docs. + const EXPECTED_CHECK_POLICY: &[(&str, ImpactPolicy)] = { + &[ + ("aprz", Unscoped), + ("audit", Unscoped), + ("bench", Affected), + ("bolero", Affected), + ("careful", Affected), + ("cargo-hack", Required), + ("cargo-sort", Modified), + ("clippy", Affected), + ("deny", Unscoped), + ("doc-build", Required), + ("doc-test", Affected), + ("ensure-no-cyclic-deps", Modified), + ("ensure-no-default-features", Modified), + ("examples", Affected), + ("external-types", Affected), + ("fmt", Modified), + ("license-headers", Modified), + ("llvm-cov", Affected), + ("loom", Affected), + ("miri", Affected), + ("miri-race-coverage", Affected), + ("miri-strict-provenance", Affected), + ("miri-tree-borrows", Affected), + ("mutants-diff", Affected), + ("mutants-full", Unscoped), + ("pr-title", Unscoped), + // readme-check + spellcheck are unscoped: their inputs (workspace + // README template, root .spelling dictionary) are repo-level files + // cargo-delta does not map to a package, so scoping them would + // silently skip a changed template/dictionary. + ("readme-check", Unscoped), + ("semver-check", Affected), + ("spellcheck", Unscoped), + ("udeps", Required), + ] + }; + #[test] fn tools_just_template_is_not_empty() { assert!(TOOLS_JUST.contains("anvil-tool-cargo-spellcheck-source-deps-check")); @@ -320,6 +391,118 @@ mod tests { } } + #[test] + fn impact_recipe_is_defined_and_reuses_shared_helpers() { + // The single impact building block: snapshot + compute + resolve. + for needle in ["anvil-impact:", "_anvil-impact-snapshot:", "_anvil-impact-include tier:"] { + assert!(IMPACT_JUST.contains(needle), "impact.just missing recipe '{needle}'"); + } + // It orchestrates around the shared helpers rather than duplicating + // base-ref resolution or the tier -> `--package` projection. + for needle in ["_anvil-base-ref", "_anvil-impact-format", "cargo delta impact"] { + assert!(IMPACT_JUST.contains(needle), "impact.just must use '{needle}'"); + } + // The ANVIL_IMPACT=off escape hatch guards every entry point. + assert!( + IMPACT_JUST.contains("$env:ANVIL_IMPACT -eq 'off'"), + "impact.just must honor ANVIL_IMPACT=off" + ); + } + + #[test] + fn every_check_matches_its_declared_impact_policy() { + // Single source of truth for each check's impact policy. The catalog + // encodes the policy structurally -- an `_anvil-impact-include + // ` call, or its absence for unscoped checks -- and this + // table pins the intended value. Pinning the exact category per check + // (rather than a bare count) makes a check silently changing category, + // or gaining/losing scoping, fail here instead of slipping through. + let expected: BTreeMap<&str, ImpactPolicy> = EXPECTED_CHECK_POLICY.iter().copied().collect(); + assert_eq!( + expected.len(), + EXPECTED_CHECK_POLICY.len(), + "EXPECTED_CHECK_POLICY contains a duplicate check entry" + ); + + let mut seen = BTreeSet::new(); + for (path, body) in CHECK_FILES { + let stem = path + .strip_prefix("justfiles/anvil/checks/") + .and_then(|p| p.strip_suffix(".just")) + .expect("check file path has the expected shape"); + seen.insert(stem); + let policy = *expected + .get(stem) + .unwrap_or_else(|| panic!("check '{stem}' is missing from EXPECTED_CHECK_POLICY; classify it explicitly")); + + // Parse the actual category calls, matched as whole tokens so + // `_anvil-impact-include affected` cannot collide with a longer + // word. A recipe must resolve exactly one category, never two. + let calls: Vec<&str> = ["modified", "affected", "required"] + .into_iter() + .filter(|cat| body.contains(&format!("_anvil-impact-include {cat}"))) + .collect(); + assert!( + calls.len() <= 1, + "{path} makes contradictory impact-include calls {calls:?}; a check resolves exactly one category" + ); + + match policy.category() { + None => { + // Unscoped: no cache dependency, no include call. + assert!( + !body.contains("_anvil-impact-include"), + "{path} is declared Unscoped but calls _anvil-impact-include" + ); + assert!( + !body.contains("-validate-prereqs anvil-impact"), + "{path} is declared Unscoped but depends on anvil-impact" + ); + } + Some(category) => { + assert_eq!( + calls.as_slice(), + &[category], + "{path}: declared {policy:?} but its _anvil-impact-include category is {calls:?}" + ); + // A scoped check must depend on anvil-impact so the cache is + // fresh, and capture the scope into a local $include -- no + // ANVIL_INCLUDE_* env-var indirection. + assert!( + body.contains("-validate-prereqs anvil-impact"), + "{path} reads the impact cache but does not depend on anvil-impact" + ); + assert!( + body.contains("$include = (& \"{{ just_executable() }}\" _anvil-impact-include"), + "{path} must capture _anvil-impact-include into a local $include variable" + ); + } + } + assert!( + !body.contains("ANVIL_INCLUDE_"), + "{path} must not reference the removed ANVIL_INCLUDE_* env vars" + ); + } + + // Bijection: every declared check exists as a file, and every file is + // declared -- so adding or removing a check forces an explicit policy. + let declared: BTreeSet<&str> = expected.keys().copied().collect(); + assert_eq!( + declared, seen, + "EXPECTED_CHECK_POLICY and the check catalog disagree on the set of checks" + ); + + // Guard the headline scoped/unscoped split so a wholesale policy shift + // is a deliberate, reviewed edit rather than an accident. + let scoped = EXPECTED_CHECK_POLICY.iter().filter(|(_, p)| p.category().is_some()).count(); + let unscoped = EXPECTED_CHECK_POLICY.len() - scoped; + assert_eq!( + (scoped, unscoped), + (23, 7), + "impact scoped/unscoped split changed; update EXPECTED_CHECK_POLICY deliberately" + ); + } + #[test] fn semver_check_compares_against_the_pr_branch_baseline() { let (_, body) = CHECK_FILES @@ -359,46 +542,118 @@ mod tests { assert!(!groups.contains(needle), "groups tree still contains stale '{needle}'"); } assert!(groups.contains("anvil-pr-slow: anvil-pr-slow-validate-prereqs anvil-pr-test anvil-pr-runtime-analysis anvil-pr-mutants")); - // Every group recipe lists its own validate-prereqs aggregate first so + // PR group recipes list their own validate-prereqs aggregate first so // all tool checks run up front (just dedups the per-check ones). for needle in [ "anvil-pr-fast: anvil-pr-fast-validate-prereqs", "anvil-pr-test: anvil-pr-test-validate-prereqs", "anvil-pr-runtime-analysis: anvil-pr-runtime-analysis-validate-prereqs", "anvil-pr-mutants: anvil-pr-mutants-validate-prereqs", - "anvil-scheduled-test: anvil-scheduled-test-validate-prereqs", - "anvil-scheduled-advisories: anvil-scheduled-advisories-validate-prereqs", - "anvil-scheduled-runtime-analysis: anvil-scheduled-runtime-analysis-validate-prereqs", - "anvil-scheduled-exhaustive: anvil-scheduled-exhaustive-validate-prereqs", ] { assert!( groups.contains(needle), "group recipe must run its validate-prereqs first: '{needle}'" ); } + // Scheduled groups are the full-workspace backstop: the public recipe + // wraps in `_anvil-unscoped` (forcing ANVIL_IMPACT=off before the deps + // run), and the private `_anvil-` fan-out lists its + // validate-prereqs aggregate first. + for g in [ + "scheduled-test", + "scheduled-advisories", + "scheduled-runtime-analysis", + "scheduled-exhaustive", + ] { + assert!( + groups.contains(&format!("anvil-{g}: (_anvil-unscoped \"{g}\")")), + "scheduled group {g} must wrap in _anvil-unscoped" + ); + assert!( + groups.contains(&format!("_anvil-{g}: anvil-{g}-validate-prereqs")), + "scheduled group {g} private fan-out must run its validate-prereqs first" + ); + } + } + + #[test] + fn impact_scoped_groups_declare_cargo_delta_prereq() { + // Every PR group whose checks are impact-scoped depends (transitively, + // via each scoped check) on `anvil-impact`, which invokes cargo-delta + // when it (re)computes the impact set. The group's setup + + // validate-prereqs must therefore install / verify cargo-delta, so a + // missing tool fails fast at setup rather than mid-run. (pr-slow is an + // umbrella and inherits this via pr-test / pr-runtime-analysis / + // pr-mutants.) Scheduled groups force ANVIL_IMPACT=off and never + // recompute the impact set, so they deliberately do NOT depend on + // cargo-delta. + let groups = all_group_bodies(); + for g in ["pr-fast", "pr-test", "pr-runtime-analysis", "pr-mutants"] { + assert!( + groups.contains(&format!( + "anvil-{g}-setup installer=\"install\": \\\n (anvil-tool-cargo-delta-install installer)" + )), + "group {g} setup must install cargo-delta" + ); + assert!( + groups.contains(&format!( + "anvil-{g}-validate-prereqs: \\\n anvil-tool-cargo-delta-validate-prereqs" + )), + "group {g} validate-prereqs must verify cargo-delta" + ); + } + // Scheduled groups force impact off and never recompute, so they must + // NOT carry cargo-delta as a prerequisite. + for g in [ + "scheduled-test", + "scheduled-advisories", + "scheduled-runtime-analysis", + "scheduled-exhaustive", + ] { + assert!( + !groups.contains(&format!( + "anvil-{g}-setup installer=\"install\": \\\n (anvil-tool-cargo-delta-install installer)" + )), + "scheduled group {g} must not install cargo-delta (it forces ANVIL_IMPACT=off and never recomputes)" + ); + assert!( + !groups.contains(&format!( + "anvil-{g}-validate-prereqs: \\\n anvil-tool-cargo-delta-validate-prereqs" + )), + "scheduled group {g} must not verify cargo-delta (it forces ANVIL_IMPACT=off and never recomputes)" + ); + } } #[test] fn tiers_just_template_has_three_tiers() { - for needle in ["anvil-pr:", "anvil-scheduled:", "anvil-full:"] { + for needle in ["anvil-pr:", "anvil-scheduled:", "anvil-full:", "_anvil-scheduled:", "_anvil-full:"] { assert!(TIERS_JUST.contains(needle), "tiers.just missing '{needle}'"); } - // Tiers depend on their work directly: there is no routing seam, so - // `just anvil-pr` always runs natively and containerized execution is - // reached only through the explicit `anvil-container` recipe. + // Containerized execution is reached only through the explicit + // `anvil-container` recipe, so no tier routes through a native/container + // seam. The PR tier depends on its work directly. The scheduled and full + // tiers are the full-workspace backstop for PR-tier impact scoping, so + // they wrap a private `_anvil-` recipe that carries the + // validate-prereqs aggregate (run first, so a missing tool fails up + // front) and inherits ANVIL_IMPACT=off from the wrapper. assert!(!TIERS_JUST.contains("_anvil-run"), "tiers must not route through an execution seam"); - // Each tier runs its validate-prereqs aggregate first so a missing - // tool fails up front rather than mid-run. for needle in [ "anvil-pr: anvil-pr-validate-prereqs", - "anvil-scheduled: anvil-scheduled-validate-prereqs", - "anvil-full: anvil-full-validate-prereqs", + "anvil-scheduled: (_anvil-unscoped \"scheduled\")", + "_anvil-scheduled: anvil-scheduled-validate-prereqs", + "anvil-full: (_anvil-unscoped \"full\")", + "_anvil-full: anvil-full-validate-prereqs", ] { - assert!( - TIERS_JUST.contains(needle), - "tier recipe must run its validate-prereqs first: '{needle}'" - ); + assert!(TIERS_JUST.contains(needle), "tier wrapper missing '{needle}'"); } + // Scoping is disabled by a parent process, because `just` runs each + // dependency as its own process and a dependency-only recipe's body + // executes after its dependencies. + assert!( + HELPERS_JUST.contains("$env:ANVIL_IMPACT = 'off'"), + "the wrapper must force ANVIL_IMPACT=off for the scheduled/full tiers" + ); // The scheduled tier must fan out to every scheduled group, including // runtime-analysis (a separate group from exhaustive). for needle in [ @@ -440,12 +695,11 @@ mod tests { fn mod_just_imports_siblings_and_defines_alias() { for needle in [ "import 'helpers.just'", + "import 'impact.just'", "import 'checks/fmt.just'", "import 'checks/miri.just'", - // Optional: the container artifacts can be removed through - // `without_artifact`, which deletes this file. A hard import would - // then fail parsing for the whole tree, not just the container - // recipes. + // Optional: a fork can drop the container backend with + // `without_artifact` and the tree still parses. "import? 'container.just'", "import 'groups/pr-fast.just'", "import 'groups/scheduled-exhaustive.just'", diff --git a/crates/cargo-anvil/src/anvil/artifacts/mod.rs b/crates/cargo-anvil/src/anvil/artifacts/mod.rs index a02d7bb4..94f6d271 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/mod.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/mod.rs @@ -96,6 +96,20 @@ mod tests { use super::*; use crate::catalog::Catalog; + #[test] + fn impact_mode_classifies_pr_and_scheduled_groups() { + assert_eq!(impact_mode("pr-fast"), "consume"); + assert_eq!(impact_mode("pr-mutants"), "consume"); + assert_eq!(impact_mode("scheduled-test"), "off"); + assert_eq!(impact_mode("scheduled-exhaustive"), "off"); + } + + #[test] + #[should_panic(expected = "unclassified group")] + fn impact_mode_panics_on_an_unclassified_group() { + let _ = impact_mode("mystery-group"); + } + #[test] fn every_registry_entry_is_in_the_anvil_catalog() { let catalog = Catalog::anvil(); diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 9dd6f8f6..b324a7a0 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1055,3 +1055,40 @@ fn mutants_diff_skips_on_arm64_windows() { "cargo must not be invoked at all on the skipped leg" ); } + +/// The wrapper that disables impact scoping for the full-workspace tiers is +/// only correct if the export happens *before* the wrapped recipe's +/// dependencies run: `just` evaluates dependencies in their own processes, and +/// every impact-scoped check reads `ANVIL_IMPACT` as a dependency of the tier, +/// not in the tier's own body. A fixture dependency that fails unless the +/// variable is already set pins that ordering; invoking the wrapped recipe +/// directly is the negative control that proves the fixture can fail. +#[test] +fn unscoped_wrapper_exports_impact_off_before_dependencies_run() { + const PROBE: &str = "[private]\n_anvil-probe: probe-dep\n\n\ + [private]\n[script(\"pwsh\", \"-NoProfile\")]\nprobe-dep:\n \ + if ($env:ANVIL_IMPACT -ne 'off') { exit 9 }\n exit 0\n"; + + if !tools_available() { + return; + } + let tmp = fixture(&[("helpers.just", HELPERS), ("probe.just", PROBE)], &[]); + let root = tmp.path(); + + let wrapped = run_just(root, &["_anvil-unscoped", "probe"], &[]); + assert!( + wrapped.status.success(), + "the dependency must observe ANVIL_IMPACT=off\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&wrapped.stdout), + String::from_utf8_lossy(&wrapped.stderr) + ); + + let direct = run_just(root, &["_anvil-probe"], &[]); + assert_eq!( + direct.status.code(), + Some(9), + "without the wrapper the dependency must see no setting, or this test proves nothing\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&direct.stdout), + String::from_utf8_lossy(&direct.stderr) + ); +} diff --git a/justfiles/anvil/groups/scheduled-advisories.just.anvil-proposed b/justfiles/anvil/groups/scheduled-advisories.just.anvil-proposed deleted file mode 100644 index d4787562..00000000 --- a/justfiles/anvil/groups/scheduled-advisories.just.anvil-proposed +++ /dev/null @@ -1,39 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. -# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md - -# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md - -# Full-workspace backstop: routed through _anvil-unscoped so -# ANVIL_IMPACT=off is set before the check dependencies run, regardless of how -# the group is invoked. The group does not recompute impact and therefore does -# not require cargo-delta. - -# Run the scheduled advisory checks. -[group("anvil")] -anvil-scheduled-advisories: (_anvil-unscoped "scheduled-advisories") - -[private] -_anvil-scheduled-advisories: anvil-scheduled-advisories-validate-prereqs \ - anvil-deny \ - anvil-audit \ - anvil-aprz \ - anvil-clippy - -# Install prerequisites for the `anvil-scheduled-advisories` recipe. -[group("anvil-setup")] -anvil-scheduled-advisories-setup installer="install": \ - (anvil-deny-setup installer) \ - (anvil-audit-setup installer) \ - (anvil-aprz-setup installer) \ - (anvil-clippy-setup installer) - -# Validate prerequisites for the `anvil-scheduled-advisories` recipe. -[group("anvil-setup")] -anvil-scheduled-advisories-validate-prereqs: \ - anvil-deny-validate-prereqs \ - anvil-audit-validate-prereqs \ - anvil-aprz-validate-prereqs \ - anvil-clippy-validate-prereqs diff --git a/justfiles/anvil/groups/scheduled-exhaustive.just.anvil-proposed b/justfiles/anvil/groups/scheduled-exhaustive.just.anvil-proposed deleted file mode 100644 index 9cfd91c1..00000000 --- a/justfiles/anvil/groups/scheduled-exhaustive.just.anvil-proposed +++ /dev/null @@ -1,36 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. -# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md - -# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md - -# Full-workspace backstop: routed through _anvil-unscoped so -# ANVIL_IMPACT=off is set before the check dependencies run, regardless of how -# the group is invoked. The group does not recompute impact and therefore does -# not require cargo-delta. - -# Run the scheduled exhaustive checks. -[group("anvil")] -anvil-scheduled-exhaustive: (_anvil-unscoped "scheduled-exhaustive") - -[private] -_anvil-scheduled-exhaustive: anvil-scheduled-exhaustive-validate-prereqs \ - anvil-mutants-full \ - anvil-cargo-hack \ - anvil-bench - -# Install prerequisites for the `anvil-scheduled-exhaustive` recipe. -[group("anvil-setup")] -anvil-scheduled-exhaustive-setup installer="install": \ - (anvil-mutants-full-setup installer) \ - (anvil-cargo-hack-setup installer) \ - (anvil-bench-setup installer) - -# Validate prerequisites for the `anvil-scheduled-exhaustive` recipe. -[group("anvil-setup")] -anvil-scheduled-exhaustive-validate-prereqs: \ - anvil-mutants-full-validate-prereqs \ - anvil-cargo-hack-validate-prereqs \ - anvil-bench-validate-prereqs diff --git a/justfiles/anvil/groups/scheduled-runtime-analysis.just.anvil-proposed b/justfiles/anvil/groups/scheduled-runtime-analysis.just.anvil-proposed deleted file mode 100644 index f443eb5c..00000000 --- a/justfiles/anvil/groups/scheduled-runtime-analysis.just.anvil-proposed +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. -# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md - -# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md - -# scheduled-runtime-analysis is the full-workspace counterpart of -# pr-runtime-analysis: it re-runs miri unscoped (PR-tier miri is -# impact-scoped, so crates untouched by a PR never get miri coverage) -# and adds the three stricter miri profiles (tree-borrows, strict- -# provenance, race-coverage) which are too expensive for PR. - -# Full-workspace backstop: routed through _anvil-unscoped so -# ANVIL_IMPACT=off is set before the check dependencies run, regardless of how -# the group is invoked. The group does not recompute impact and therefore does -# not require cargo-delta. - -# Run the scheduled runtime analysis. -[group("anvil")] -anvil-scheduled-runtime-analysis: (_anvil-unscoped "scheduled-runtime-analysis") - -[private] -_anvil-scheduled-runtime-analysis: anvil-scheduled-runtime-analysis-validate-prereqs \ - anvil-miri \ - anvil-miri-tree-borrows \ - anvil-miri-strict-provenance \ - anvil-miri-race-coverage - -# Install prerequisites for the `anvil-scheduled-runtime-analysis` recipe. -[group("anvil-setup")] -anvil-scheduled-runtime-analysis-setup installer="install": \ - (anvil-miri-setup installer) \ - (anvil-miri-tree-borrows-setup installer) \ - (anvil-miri-strict-provenance-setup installer) \ - (anvil-miri-race-coverage-setup installer) - -# Validate prerequisites for the `anvil-scheduled-runtime-analysis` recipe. -[group("anvil-setup")] -anvil-scheduled-runtime-analysis-validate-prereqs: \ - anvil-miri-validate-prereqs \ - anvil-miri-tree-borrows-validate-prereqs \ - anvil-miri-strict-provenance-validate-prereqs \ - anvil-miri-race-coverage-validate-prereqs diff --git a/justfiles/anvil/groups/scheduled-test.just.anvil-proposed b/justfiles/anvil/groups/scheduled-test.just.anvil-proposed deleted file mode 100644 index d60f03f5..00000000 --- a/justfiles/anvil/groups/scheduled-test.just.anvil-proposed +++ /dev/null @@ -1,40 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. -# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md - -# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md - -# Scheduled groups - -# Scheduled groups are the full-workspace backstop for PR-tier impact scoping, -# so route through _anvil-unscoped: it exports ANVIL_IMPACT=off -# before the check dependencies run, so the group is full-workspace regardless -# of how it is invoked (CI, `just anvil-scheduled`, or -# `just anvil-scheduled-test` directly). Because these groups never recompute -# the impact set, they no longer need cargo-delta as a prerequisite. - -# Run the scheduled tests. -[group("anvil")] -anvil-scheduled-test: (_anvil-unscoped "scheduled-test") - -[private] -_anvil-scheduled-test: anvil-scheduled-test-validate-prereqs \ - anvil-llvm-cov \ - anvil-doc-test \ - anvil-examples - -# Install prerequisites for the `anvil-scheduled-test` recipe. -[group("anvil-setup")] -anvil-scheduled-test-setup installer="install": \ - (anvil-llvm-cov-setup installer) \ - (anvil-doc-test-setup installer) \ - (anvil-examples-setup installer) - -# Validate prerequisites for the `anvil-scheduled-test` recipe. -[group("anvil-setup")] -anvil-scheduled-test-validate-prereqs: \ - anvil-llvm-cov-validate-prereqs \ - anvil-doc-test-validate-prereqs \ - anvil-examples-validate-prereqs diff --git a/justfiles/anvil/helpers.just.anvil-proposed b/justfiles/anvil/helpers.just.anvil-proposed deleted file mode 100644 index c9216169..00000000 --- a/justfiles/anvil/helpers.just.anvil-proposed +++ /dev/null @@ -1,137 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# GENERATED BY cargo-anvil. DO NOT EDIT DIRECTLY. -# Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. -# Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md - -# See https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/checks.md - -# -# Each check belongs to one of four buckets, which determines how it scopes -# its work to the packages impacted by a change: -# -# - "modified": only run when at least one package's source files changed -# in the diff. The check's underlying tool is workspace-wide or -# directory-scoped (cargo fmt --all, cargo heather), so -# it doesn't take --package; it short-circuits on the "--skip" sentinel. -# -# - "affected": run on the affected set (modified ∪ reverse-deps within the -# workspace). The tool takes --package; the recipe splices the resolved -# package list into the cargo invocation, defaulting to --workspace when -# the tier is unscoped. -# -# - "required": run on the required set (affected ∪ workspace-internal -# transitive deps). Same splice/default as affected, for checks whose tool -# resolves through the dep graph (cargo doc -> intra-doc links; cargo hack -# -> feature powerset; cargo udeps -> unused-deps). -# -# - "unscoped": always run. External-input checks (deny, audit, aprz) and -# PR-context checks (pr-title) live here, as do checks whose inputs -# include repo-level files cargo-delta maps to no package (the -# workspace-level README template via readme-check, the root `.spelling` -# via spellcheck); scheduled-exhaustive recipes (mutants-full) are -# unscoped by design. -# -# Each scoped check depends on `anvil-impact` (which produces the -# target/anvil/impact cache) and captures its category's scope into a local -# variable by calling the shared resolver: -# -# $include = (& "{{ just_executable() }}" _anvil-impact-include ) -# -# _anvil-impact-include reads that cache both locally and in cloud workflows -# (where the group job downloaded the target/anvil/impact artifact), so the -# same code path runs everywhere -- no scoping is threaded between jobs via -# environment variables. `$include` is one of: -# -# * "--workspace" (affected/required) or "" (modified) - the tier is -# unscoped: ANVIL_IMPACT=off, or no impact cache is present. Recipes splat -# "--workspace" (an empty string is NOT $null, so recipes test truthiness -# rather than `?? "--workspace"`, which would splat an empty arg cargo -# rejects). -# * "--package A@v --package B@v" - the tier has members. Packages are -# version-qualified cargo specs so they resolve uniquely even when a -# like-named crate is also a transitive dependency. -# * "--skip" - the tier is empty for this change (recipe exits 0). -# -# This keeps the per-recipe boilerplate to a single capture, a one-line skip -# guard, and the cargo invocation. Modified-tier recipes never splice -# `$include` into cargo (their tools are workspace-wide); they only check the -# skip sentinel. -# -# Every recipe whose body uses multi-line conditionals or array-splat -# splicing is annotated with [script("pwsh", "-NoProfile")]. pwsh is preinstalled on -# Windows (since Windows 10), on GH/ADO hosted Linux + Windows -# runners, and installable on macOS via Homebrew or the upstream -# installer. We chose pwsh over bash because just's shebang dispatch -# requires `cygpath` on Windows (only on PATH from inside Git Bash), -# while [script("pwsh", "-NoProfile")] works from plain PowerShell with no PATH -# augmentation. We require pwsh 7+ (enforced via _anvil-require pwsh) -# for consistent semantics of the array-splat splice and the modern -# operators used across these recipes. -# -# Every recipe that runs a command is annotated with [script("pwsh", "-NoProfile")], -# including single-command ones (cargo deny check, cargo audit). They do -# NOT inherit the adopter's default shell: just defaults to `sh`, which is -# absent on Windows and varies across environments, so relying on it makes -# these recipes non-portable. Routing every command through pwsh keeps -# behavior identical on every platform. -# -# Note: [script(...)] requires `set unstable`. The adopter's root -# justfile must declare it (typically as a top-level line). anvil's -# mod.just does NOT redeclare it, to avoid conflicting with adopters -# who already have it. - -# _anvil-base-ref: single source of truth for "which git ref does a PR -# diff against". Shared by the impact step (github/impact-action.yml, -# ado/steps/impact.yml) and anvil-mutants-diff so the resolution lives in -# exactly one place. Precedence: -# 1. BASE_REF -- explicit override. GitHub sets it to the PR base commit -# SHA; adopters can set a full ref like `origin/release`. -# 2. SYSTEM_PULLREQUEST_TARGETBRANCH -- ADO PR builds. This is the -# *fully-qualified* target ref (e.g. `refs/heads/main`), so strip the -# `refs/heads/` prefix and resolve to `origin/` -# (`origin/refs/heads/main` is not a valid ref). -# 3. GITHUB_BASE_REF -- GitHub PR builds. Unqualified target branch name. -# 4. origin/main, then origin/master -- local / non-PR fallback. -# Prints ONLY the resolved ref to stdout so callers can capture it: -# pwsh : $base = (& "{{ just_executable() }}" _anvil-base-ref) -# bash : base="$(just _anvil-base-ref)" -[script("pwsh", "-NoProfile")] -_anvil-base-ref: - $ErrorActionPreference = 'Stop' - if ($env:BASE_REF) { - Write-Output $env:BASE_REF - exit 0 - } - if ($env:SYSTEM_PULLREQUEST_TARGETBRANCH) { - Write-Output ("origin/" + ($env:SYSTEM_PULLREQUEST_TARGETBRANCH -replace '^refs/heads/', '')) - exit 0 - } - if ($env:GITHUB_BASE_REF) { - Write-Output "origin/$($env:GITHUB_BASE_REF)" - exit 0 - } - foreach ($candidate in @('origin/main', 'origin/master')) { - git rev-parse --verify $candidate 2>$null | Out-Null - if ($LASTEXITCODE -eq 0) { - Write-Output $candidate - exit 0 - } - } - Write-Error 'anvil-base-ref: cannot resolve a base ref. Set BASE_REF, or ensure origin/main or origin/master exists.' - exit 1 - -# Run a private recipe with impact scoping disabled. -# -# The only way to reach a whole dependency tree with an environment variable: -# `just` runs each dependency as its own process, and a dependency-only -# recipe's body executes after its dependencies, so exporting from there is -# too late. Invoking `_anvil-` as a child process makes every check -# below it inherit the setting. -[private] -[script("pwsh", "-NoProfile")] -_anvil-unscoped name: - $ErrorActionPreference = 'Stop' - $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' - exit $LASTEXITCODE \ No newline at end of file From 5680d36561c66f87eac1aad96dad8313e2c23f93 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 14:12:27 +0200 Subject: [PATCH 64/81] fix(anvil): stop recipe fixtures inheriting the caller's impact scoping A fixture is a scratch workspace and must not adopt the impact scoping of whatever invoked the test suite. A CI group job exports ANVIL_IMPACT=consume and downloads the impact cache into the real repository; inherited into a temp directory that has no cache, `anvil-impact` fails hard and takes the recipe under test with it, which is how `mutants_diff_skips_on_arm64_windows` failed on the Windows arm leg while passing everywhere else. `ANVIL_INCLUDE_*` is the same hazard one level down: a leg whose scope resolved to `--skip` short-circuits the recipe before it does anything, so the assertions pass vacuously. `run_just` now clears both before applying a test's explicit environment, so the fixtures are hermetic on every leg. Verified by running the suite with ANVIL_IMPACT=consume and ANVIL_INCLUDE_AFFECTED=--skip exported. `_anvil-unscoped` escapes the recipe name it interpolates, matching every other interpolation in the generated tree, and local.md describes the container entry point as taking an argv rather than a recipe name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 ++-- crates/cargo-anvil/docs/design/local.md | 6 ++++-- .../templates/justfiles/anvil/helpers.just | 2 +- crates/cargo-anvil/tests/recipe_contracts.rs | 18 ++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 2 +- .../snapshots/snapshots__github_backend.snap | 2 +- .../tests/snapshots/snapshots__local_only.snap | 2 +- justfiles/anvil/helpers.just | 2 +- 8 files changed, 29 insertions(+), 9 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 6c4e02a6..26624d80 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:42bd2c33097f3d8c982473e1ecdd9915e77848c4272da4bf333654c3c401988f" +catalog_checksum = "sha256:0232e668dbf29da841dc54f45d096af426fe766eebc15908e4449fa7a287be8e" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -205,7 +205,7 @@ checksum = "sha256:04679222579a090769403f5aae7c9ffabbf5bd559ddb1f17b5a0655152172 [[file]] path = "justfiles/anvil/helpers.just" -checksum = "sha256:0a3518ea37d0c174f6da0c5f852c39861f3bc6e991cabddc96594849833b5b33" +checksum = "sha256:01e33b351ed0eac0940ff61f2430a6865e1366e2d35ab2e6a085a4896afcd519" [[file]] path = "justfiles/anvil/impact.just" diff --git a/crates/cargo-anvil/docs/design/local.md b/crates/cargo-anvil/docs/design/local.md index ebb90070..5edf3b8f 100644 --- a/crates/cargo-anvil/docs/design/local.md +++ b/crates/cargo-anvil/docs/design/local.md @@ -81,9 +81,11 @@ files) are annotated with `[group("anvil-setup")]`. `just --groups` therefore sh two clean clusters: one for "run checks", one for "install prereqs". > **Containerized execution.** `justfiles/anvil/container.just` adds the -> `anvil-container ` command, which runs any recipe below inside a pinned +> `anvil-container ` recipe, which runs the given argv inside a pinned > Linux image instead of against the host toolchain (Linux-on-Windows parity, -> toolchain pinning). It is explicit: the tiers themselves always run natively. +> toolchain pinning). Anvil recipes are reached by naming `just` +> (`just anvil-container just anvil-pr`); with no argument it opens a shell. It +> is explicit: the tiers themselves always run natively. > The recipe bodies are unchanged; see [containers.md](./containers.md). ## 2. Recipe layers diff --git a/crates/cargo-anvil/templates/justfiles/anvil/helpers.just b/crates/cargo-anvil/templates/justfiles/anvil/helpers.just index c9216169..024f1592 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/helpers.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/helpers.just @@ -133,5 +133,5 @@ _anvil-base-ref: _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' + & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE \ No newline at end of file diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index b324a7a0..b703c196 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -178,6 +178,20 @@ fn run_just(root: &Path, arguments: &[&str], environment: &[(&str, &OsStr)]) -> command.args(["--justfile", "Justfile"]).args(arguments).current_dir(root); command.env("PATH", path_with_fake_bin(root)); command.env("FAKE_WORKSPACE_ROOT", root); + // A fixture is a scratch workspace, so it must not inherit the impact + // scoping of whatever invoked the test suite. A CI group job exports + // `ANVIL_IMPACT=consume` and downloads a cache into the real repository; + // inherited into a temp directory that has no cache, `anvil-impact` fails + // hard and takes the recipe under test with it. `ANVIL_INCLUDE_*` is the + // same hazard one level down: a leg whose scope resolved to `--skip` would + // silently short-circuit the recipe before it did anything. A test that + // cares about either value passes it explicitly below. + command.env_remove("ANVIL_IMPACT"); + for key in std::env::vars_os().map(|(key, _)| key) { + if key.to_string_lossy().starts_with("ANVIL_INCLUDE_") { + command.env_remove(key); + } + } for &(key, value) in environment { command.env(key, value); } @@ -1037,6 +1051,10 @@ fn mutants_diff_skips_on_arm64_windows() { // is the *other* early exit, pinned so a skipped impact scope // cannot be mistaken for the architecture bail-out. ("ANVIL_INCLUDE_AFFECTED", OsStr::new("--package fixture@0.1.0")), + // Same reason as the sibling contract: `anvil-mutants-diff` depends + // on `anvil-impact`, and this test is about the architecture + // bail-out, not about computing an impact set. + ("ANVIL_IMPACT", OsStr::new("off")), ], ); diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index d45a2048..15fae07d 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -4962,7 +4962,7 @@ _anvil-base-ref: _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' + & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f31d0a6c..577e50c0 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -4841,7 +4841,7 @@ _anvil-base-ref: _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' + & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 6ff5a04d..5a3efcf5 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -3714,7 +3714,7 @@ _anvil-base-ref: _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' + & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === diff --git a/justfiles/anvil/helpers.just b/justfiles/anvil/helpers.just index c9216169..024f1592 100644 --- a/justfiles/anvil/helpers.just +++ b/justfiles/anvil/helpers.just @@ -133,5 +133,5 @@ _anvil-base-ref: _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ name }}' + & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE \ No newline at end of file From 154826311194ea7e5a0ce5144fbf802ef8777802 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 14:50:41 +0200 Subject: [PATCH 65/81] fix(anvil): carry the argv contract into the live-engine scripts The container recipe appends its argv to the engine command verbatim, so `anvil-container anvil-fmt` asks the image for an `anvil-fmt` binary. Both daemon-backed suites still called it that way and would have failed on their first case, before reaching any runtime assertion; the generated-text checks in Rust cannot see this. Every call site now names `just`, including the nested in-image invocation, and a `cargo --version` case covers the branch a recipe name can never reach. The tier setup note claimed cargo-delta is not a per-group prerequisite. Impact-scoped checks depend on `anvil-impact`, which invokes cargo-delta whenever it recomputes the set, and each such group's setup installs and verifies it; the note is restored to describe that. `_anvil-unscoped` names the justfile when it re-invokes `just`, so the child resolves the file the wrapper was defined in rather than whatever an upward search from the working directory finds. The container recipe's header describes the argv contract the body implements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 8 ++--- .../templates/justfiles/anvil/container.just | 5 +-- .../templates/justfiles/anvil/helpers.just | 6 +++- .../templates/justfiles/anvil/tiers.just | 17 +++++----- .../snapshots/snapshots__ado_backend.snap | 28 +++++++++------- .../snapshots/snapshots__github_backend.snap | 28 +++++++++------- .../snapshots/snapshots__local_only.snap | 28 +++++++++------- justfiles/anvil/container.just | 5 +-- justfiles/anvil/helpers.just | 6 +++- justfiles/anvil/tiers.just | 17 +++++----- scripts/test-anvil-container.ps1 | 32 +++++++++++-------- scripts/test-anvil-dogfood.ps1 | 10 +++--- 12 files changed, 113 insertions(+), 77 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 26624d80..09bddc6e 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:0232e668dbf29da841dc54f45d096af426fe766eebc15908e4449fa7a287be8e" +catalog_checksum = "sha256:280499c911162e6d9789b8951457f718c9e36e161ba3121eb7b32d39637dfa42" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:230bc29a9e5be4a4ea2bb58ae3a951b59ae8715c37e7bde0aef604c1531fafc2" +checksum = "sha256:6be4e6d298e56b8b0eb5874e295b55b1a6f7c8f15c707c3d19db77ffee6543b8" [[file]] path = "justfiles/anvil/groups/pr-fast.just" @@ -205,7 +205,7 @@ checksum = "sha256:04679222579a090769403f5aae7c9ffabbf5bd559ddb1f17b5a0655152172 [[file]] path = "justfiles/anvil/helpers.just" -checksum = "sha256:01e33b351ed0eac0940ff61f2430a6865e1366e2d35ab2e6a085a4896afcd519" +checksum = "sha256:1208b9a51b94ef9bf60a898d23c35c97175279c661a90e774ec3960989b4c5c9" [[file]] path = "justfiles/anvil/impact.just" @@ -217,7 +217,7 @@ checksum = "sha256:2f7f0187f8c45716a1bed85f0c45ffbc71cdf592ed6414c9bc4cd72cf716a [[file]] path = "justfiles/anvil/tiers.just" -checksum = "sha256:3e468a44045ffcd80861eeffe5e7206d9958529264cf54ccf5416ddbd5825f5a" +checksum = "sha256:b409feb16d9a675221ec2d9202110eeb1e2fa166971f48a45eeaecba2c35017f" [[file]] path = "justfiles/anvil/tools.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 678eae3c..fcc1e3b1 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -4,8 +4,9 @@ # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. # Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md # -# Containerized execution. `just anvil-container ` runs any anvil -# recipe inside a pinned Linux image; everything else keeps running natively. +# Containerized execution. `just anvil-container ` runs the given +# argv inside a pinned Linux image; everything else keeps running natively. +# Anvil recipes are reached by naming `just`, like any other command. # There is no configuration file and no transparent routing: the container is # reached through this recipe or not at all. # diff --git a/crates/cargo-anvil/templates/justfiles/anvil/helpers.just b/crates/cargo-anvil/templates/justfiles/anvil/helpers.just index 024f1592..5be4b411 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/helpers.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/helpers.just @@ -128,10 +128,14 @@ _anvil-base-ref: # recipe's body executes after its dependencies, so exporting from there is # too late. Invoking `_anvil-` as a child process makes every check # below it inherit the setting. +# +# The justfile is named explicitly so the child resolves the same file the +# wrapper was defined in, rather than whatever an upward search from the +# working directory happens to find. [private] [script("pwsh", "-NoProfile")] _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' + & '{{ replace(just_executable(), "'", "''") }}' --justfile '{{ replace(justfile(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE \ No newline at end of file diff --git a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just index 14432a0e..e2230db6 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just @@ -109,14 +109,15 @@ anvil-full-validate-prereqs: \ # catalog knows about; `anvil-validate-prereqs` verifies every tool # and component is present at or above its pinned version. # -# These also install / verify cargo-delta, the impact-analysis tool. -# cargo-delta is only invoked by the cloud-workflow impact step (not by -# any check), so it is deliberately NOT part of the per-group / per-tier -# setup recipes -- a CI group job installs only what its checks need, and -# the impact step installs cargo-delta itself. But the catch-all -# `anvil-setup` is "install everything the catalog knows about", so it -# includes cargo-delta too (a local `just anvil-setup` then provisions a -# complete environment, impact tool included). +# cargo-delta (the impact-analysis tool) is a prerequisite of every group +# whose checks are impact-scoped: those checks depend on `anvil-impact`, +# which invokes cargo-delta when it has to (re)compute the impact set, so +# each such group's `-setup` / `-validate-prereqs` installs / verifies it +# (fail-fast, rather than discovering it missing mid-run). In a cloud +# workflow a group job normally consumes the downloaded impact artifact and +# never actually runs cargo-delta, but the tool is guaranteed present. +# `anvil-setup` also lists it explicitly as the catch-all (harmless -- just +# dedups the fan-out). # Install every tool and component in the anvil catalog. [group("anvil-setup")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 15fae07d..5f0f0b12 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3559,8 +3559,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. # Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md # -# Containerized execution. `just anvil-container ` runs any anvil -# recipe inside a pinned Linux image; everything else keeps running natively. +# Containerized execution. `just anvil-container ` runs the given +# argv inside a pinned Linux image; everything else keeps running natively. +# Anvil recipes are reached by naming `just`, like any other command. # There is no configuration file and no transparent routing: the container is # reached through this recipe or not at all. # @@ -4957,12 +4958,16 @@ _anvil-base-ref: # recipe's body executes after its dependencies, so exporting from there is # too late. Invoking `_anvil-` as a child process makes every check # below it inherit the setting. +# +# The justfile is named explicitly so the child resolves the same file the +# wrapper was defined in, rather than whatever an upward search from the +# working directory happens to find. [private] [script("pwsh", "-NoProfile")] _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' + & '{{ replace(just_executable(), "'", "''") }}' --justfile '{{ replace(justfile(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === @@ -5796,14 +5801,15 @@ anvil-full-validate-prereqs: \ # catalog knows about; `anvil-validate-prereqs` verifies every tool # and component is present at or above its pinned version. # -# These also install / verify cargo-delta, the impact-analysis tool. -# cargo-delta is only invoked by the cloud-workflow impact step (not by -# any check), so it is deliberately NOT part of the per-group / per-tier -# setup recipes -- a CI group job installs only what its checks need, and -# the impact step installs cargo-delta itself. But the catch-all -# `anvil-setup` is "install everything the catalog knows about", so it -# includes cargo-delta too (a local `just anvil-setup` then provisions a -# complete environment, impact tool included). +# cargo-delta (the impact-analysis tool) is a prerequisite of every group +# whose checks are impact-scoped: those checks depend on `anvil-impact`, +# which invokes cargo-delta when it has to (re)compute the impact set, so +# each such group's `-setup` / `-validate-prereqs` installs / verifies it +# (fail-fast, rather than discovering it missing mid-run). In a cloud +# workflow a group job normally consumes the downloaded impact artifact and +# never actually runs cargo-delta, but the tool is guaranteed present. +# `anvil-setup` also lists it explicitly as the catch-all (harmless -- just +# dedups the fan-out). # Install every tool and component in the anvil catalog. [group("anvil-setup")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 577e50c0..7af4ca21 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3438,8 +3438,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. # Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md # -# Containerized execution. `just anvil-container ` runs any anvil -# recipe inside a pinned Linux image; everything else keeps running natively. +# Containerized execution. `just anvil-container ` runs the given +# argv inside a pinned Linux image; everything else keeps running natively. +# Anvil recipes are reached by naming `just`, like any other command. # There is no configuration file and no transparent routing: the container is # reached through this recipe or not at all. # @@ -4836,12 +4837,16 @@ _anvil-base-ref: # recipe's body executes after its dependencies, so exporting from there is # too late. Invoking `_anvil-` as a child process makes every check # below it inherit the setting. +# +# The justfile is named explicitly so the child resolves the same file the +# wrapper was defined in, rather than whatever an upward search from the +# working directory happens to find. [private] [script("pwsh", "-NoProfile")] _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' + & '{{ replace(just_executable(), "'", "''") }}' --justfile '{{ replace(justfile(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === @@ -5675,14 +5680,15 @@ anvil-full-validate-prereqs: \ # catalog knows about; `anvil-validate-prereqs` verifies every tool # and component is present at or above its pinned version. # -# These also install / verify cargo-delta, the impact-analysis tool. -# cargo-delta is only invoked by the cloud-workflow impact step (not by -# any check), so it is deliberately NOT part of the per-group / per-tier -# setup recipes -- a CI group job installs only what its checks need, and -# the impact step installs cargo-delta itself. But the catch-all -# `anvil-setup` is "install everything the catalog knows about", so it -# includes cargo-delta too (a local `just anvil-setup` then provisions a -# complete environment, impact tool included). +# cargo-delta (the impact-analysis tool) is a prerequisite of every group +# whose checks are impact-scoped: those checks depend on `anvil-impact`, +# which invokes cargo-delta when it has to (re)compute the impact set, so +# each such group's `-setup` / `-validate-prereqs` installs / verifies it +# (fail-fast, rather than discovering it missing mid-run). In a cloud +# workflow a group job normally consumes the downloaded impact artifact and +# never actually runs cargo-delta, but the tool is guaranteed present. +# `anvil-setup` also lists it explicitly as the catch-all (harmless -- just +# dedups the fan-out). # Install every tool and component in the anvil catalog. [group("anvil-setup")] diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 5a3efcf5..a7e80f5b 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2311,8 +2311,9 @@ anvil-udeps-validate-prereqs: anvil-toolchain-nightly-validate-prereqs anvil-too # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. # Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md # -# Containerized execution. `just anvil-container ` runs any anvil -# recipe inside a pinned Linux image; everything else keeps running natively. +# Containerized execution. `just anvil-container ` runs the given +# argv inside a pinned Linux image; everything else keeps running natively. +# Anvil recipes are reached by naming `just`, like any other command. # There is no configuration file and no transparent routing: the container is # reached through this recipe or not at all. # @@ -3709,12 +3710,16 @@ _anvil-base-ref: # recipe's body executes after its dependencies, so exporting from there is # too late. Invoking `_anvil-` as a child process makes every check # below it inherit the setting. +# +# The justfile is named explicitly so the child resolves the same file the +# wrapper was defined in, rather than whatever an upward search from the +# working directory happens to find. [private] [script("pwsh", "-NoProfile")] _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' + & '{{ replace(just_executable(), "'", "''") }}' --justfile '{{ replace(justfile(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE === justfiles/anvil/impact.just === @@ -4548,14 +4553,15 @@ anvil-full-validate-prereqs: \ # catalog knows about; `anvil-validate-prereqs` verifies every tool # and component is present at or above its pinned version. # -# These also install / verify cargo-delta, the impact-analysis tool. -# cargo-delta is only invoked by the cloud-workflow impact step (not by -# any check), so it is deliberately NOT part of the per-group / per-tier -# setup recipes -- a CI group job installs only what its checks need, and -# the impact step installs cargo-delta itself. But the catch-all -# `anvil-setup` is "install everything the catalog knows about", so it -# includes cargo-delta too (a local `just anvil-setup` then provisions a -# complete environment, impact tool included). +# cargo-delta (the impact-analysis tool) is a prerequisite of every group +# whose checks are impact-scoped: those checks depend on `anvil-impact`, +# which invokes cargo-delta when it has to (re)compute the impact set, so +# each such group's `-setup` / `-validate-prereqs` installs / verifies it +# (fail-fast, rather than discovering it missing mid-run). In a cloud +# workflow a group job normally consumes the downloaded impact artifact and +# never actually runs cargo-delta, but the tool is guaranteed present. +# `anvil-setup` also lists it explicitly as the catch-all (harmless -- just +# dedups the fan-out). # Install every tool and component in the anvil catalog. [group("anvil-setup")] diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 678eae3c..fcc1e3b1 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -4,8 +4,9 @@ # Update cargo-anvil and regenerate; repository-specific edits stop automatic updates. # Update behaviour: https://github.com/microsoft/ox-tools/blob/main/crates/cargo-anvil/docs/design/updates.md # -# Containerized execution. `just anvil-container ` runs any anvil -# recipe inside a pinned Linux image; everything else keeps running natively. +# Containerized execution. `just anvil-container ` runs the given +# argv inside a pinned Linux image; everything else keeps running natively. +# Anvil recipes are reached by naming `just`, like any other command. # There is no configuration file and no transparent routing: the container is # reached through this recipe or not at all. # diff --git a/justfiles/anvil/helpers.just b/justfiles/anvil/helpers.just index 024f1592..5be4b411 100644 --- a/justfiles/anvil/helpers.just +++ b/justfiles/anvil/helpers.just @@ -128,10 +128,14 @@ _anvil-base-ref: # recipe's body executes after its dependencies, so exporting from there is # too late. Invoking `_anvil-` as a child process makes every check # below it inherit the setting. +# +# The justfile is named explicitly so the child resolves the same file the +# wrapper was defined in, rather than whatever an upward search from the +# working directory happens to find. [private] [script("pwsh", "-NoProfile")] _anvil-unscoped name: $ErrorActionPreference = 'Stop' $env:ANVIL_IMPACT = 'off' - & '{{ replace(just_executable(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' + & '{{ replace(just_executable(), "'", "''") }}' --justfile '{{ replace(justfile(), "'", "''") }}' '_anvil-{{ replace(name, "'", "''") }}' exit $LASTEXITCODE \ No newline at end of file diff --git a/justfiles/anvil/tiers.just b/justfiles/anvil/tiers.just index 14432a0e..e2230db6 100644 --- a/justfiles/anvil/tiers.just +++ b/justfiles/anvil/tiers.just @@ -109,14 +109,15 @@ anvil-full-validate-prereqs: \ # catalog knows about; `anvil-validate-prereqs` verifies every tool # and component is present at or above its pinned version. # -# These also install / verify cargo-delta, the impact-analysis tool. -# cargo-delta is only invoked by the cloud-workflow impact step (not by -# any check), so it is deliberately NOT part of the per-group / per-tier -# setup recipes -- a CI group job installs only what its checks need, and -# the impact step installs cargo-delta itself. But the catch-all -# `anvil-setup` is "install everything the catalog knows about", so it -# includes cargo-delta too (a local `just anvil-setup` then provisions a -# complete environment, impact tool included). +# cargo-delta (the impact-analysis tool) is a prerequisite of every group +# whose checks are impact-scoped: those checks depend on `anvil-impact`, +# which invokes cargo-delta when it has to (re)compute the impact set, so +# each such group's `-setup` / `-validate-prereqs` installs / verifies it +# (fail-fast, rather than discovering it missing mid-run). In a cloud +# workflow a group job normally consumes the downloaded impact artifact and +# never actually runs cargo-delta, but the tool is guaranteed present. +# `anvil-setup` also lists it explicitly as the catch-all (harmless -- just +# dedups the fan-out). # Install every tool and component in the anvil catalog. [group("anvil-setup")] diff --git a/scripts/test-anvil-container.ps1 b/scripts/test-anvil-container.ps1 index 0e487818..ffb1901f 100644 --- a/scripts/test-anvil-container.ps1 +++ b/scripts/test-anvil-container.ps1 @@ -8,7 +8,7 @@ .DESCRIPTION Creates a throwaway repository in a temp directory, generates the anvil tree into it with the locally-built cargo-anvil, and then does only what a - developer would do: run `just anvil-container ` and observe what + developer would do: run `just anvil-container ` and observe what happens. The setup phase is held to that standard deliberately. If this script has to @@ -387,7 +387,7 @@ Assert-That 'status reports an image reference' ($reference -like "$imagePrefix* Assert-That 'image is absent before the first run' (-not (Test-ImagePresent $reference)) Write-Step "building and running (this takes several minutes on a cold cache)" -$firstRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') +$firstRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'anvil-fmt') Write-Detail (($firstRun.StdErr -split "`r?`n" | Select-Object -Last 4) -join "`n") Assert-Equal 'anvil-fmt succeeds inside the container' 0 $firstRun.ExitCode @@ -398,11 +398,17 @@ Assert-That 'image is present afterwards' (Test-ImagePresent $reference) Write-Section '3. Second run reuses the image' -$secondRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') +$secondRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'anvil-fmt') Assert-Equal 'anvil-fmt succeeds again' 0 $secondRun.ExitCode Assert-That 'nothing was rebuilt' (-not ($secondRun.StdErr -match 'building ')) $secondRun.StdErr Assert-Equal 'the reference is unchanged' $reference (Get-ImageReference -Repo $repo) +# The argv is executed verbatim, so a program that is not `just` is reachable +# too. This is the branch a recipe name would never exercise. +$bareCommand = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'cargo', '--version') -AllowFailure +Assert-Equal 'a non-just command runs in the image' 0 $bareCommand.ExitCode +Assert-That 'the bare command produced its own output' ($bareCommand.StdOut -match 'cargo\s+\d') $bareCommand.StdOut + $status = Invoke-Just -Repo $repo -Arguments @('anvil-container-status') Assert-That 'status reports present and current' ($status.StdOut -match 'present and current') $status.StdOut Assert-That 'status reports the selected engine' ($status.StdOut -match "engine:\s+.*$Engine") $status.StdOut @@ -441,7 +447,7 @@ Assert-That 'the tool resolves inside the image, not a volume' ` # credential, and a test that prints it to the terminal on failure is a leak. function Hide-Token([string]$Text) { $Text -replace 'E2E-TOKEN:\[[^\]]+\]', 'E2E-TOKEN:[]' } -$withToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` +$withToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'e2e-show-token') ` -Environment @{ GITHUB_TOKEN = 'e2e-forwarded-token' } Assert-That 'a host GITHUB_TOKEN reaches a recipe in the container' ` ($withToken.StdOut -match 'E2E-TOKEN:\[e2e-forwarded-token\]') (Hide-Token "$($withToken.StdOut)$($withToken.StdErr)") @@ -458,7 +464,7 @@ if ($ghCommand) { $pathWithoutGh = (($env:PATH -split $separator) | Where-Object { $_ -and $_.TrimEnd('\', '/') -ne $ghDir }) -join $separator } -$withoutToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` +$withoutToken = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'e2e-show-token') ` -Environment @{ GITHUB_TOKEN = ''; GH_TOKEN = ''; PATH = $pathWithoutGh } Assert-That 'no token is invented when the host has none' ` ($withoutToken.StdOut -match 'E2E-TOKEN:\[\]') (Hide-Token "$($withoutToken.StdOut)$($withoutToken.StdErr)") @@ -472,7 +478,7 @@ if (Get-Command gh -ErrorAction SilentlyContinue) { try { $hostGhToken = (gh auth token --hostname github.com 2>$null) } catch { $hostGhToken = $null } } if ($hostGhToken -and $hostGhToken.Trim()) { - $viaGh = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-token') ` + $viaGh = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'e2e-show-token') ` -Environment @{ GITHUB_TOKEN = '' } Assert-That 'the gh CLI token is used when the environment has none' ` ($viaGh.StdOut -match ('E2E-TOKEN:\[' + [regex]::Escape($hostGhToken.Trim()) + '\]')) ` @@ -482,7 +488,7 @@ if ($hostGhToken -and $hostGhToken.Trim()) { # in this environment hands it to every build script and proc macro in the # container, where natively the recipe would mint it in its own process -- # so a target that never reads the variable must not receive it. - $noNeed = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-dump-env') ` + $noNeed = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'e2e-dump-env') ` -Environment @{ GITHUB_TOKEN = '' } Assert-That 'no token is derived for a target that does not read it' ` ($noNeed.StdOut -notmatch 'GITHUB_TOKEN') ` @@ -519,7 +525,7 @@ Assert-That 'its .git is a file, not a directory' ` $wtReference = Get-ImageReference -Repo $worktree Assert-Equal 'the worktree selects the same image, so nothing rebuilds' $reference $wtReference -$wtGit = Invoke-Just -Repo $worktree -Arguments @('anvil-container', 'e2e-show-git') -AllowFailure +$wtGit = Invoke-Just -Repo $worktree -Arguments @('anvil-container', 'just', 'e2e-show-git') -AllowFailure Assert-Equal 'a recipe run from a worktree succeeds' 0 $wtGit.ExitCode Assert-That 'git resolves the branch inside the container' ` ($wtGit.StdOut -match 'E2E-GIT:\[e2e-worktree\]') "$($wtGit.StdOut)$($wtGit.StdErr)" @@ -647,7 +653,7 @@ RUN --mount=type=secret,id=e2e_token,required=true \ $leakControlStanza = $secretStanza -replace '(?m)\s*&& rm -f /usr/local/cargo/credentials\.toml$', '' Write-Fixture $dockerfile ($dockerfileBody + $leakControlStanza) Write-Step 'building a deliberately leaking image to prove the probe works' -$controlRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +$controlRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'anvil-fmt') -AllowFailure Assert-Equal 'the control image builds' 0 $controlRun.ExitCode $controlLeak = Invoke-Engine -Arguments @( 'run', '--rm', '--pull=never', (Get-ImageReference -Repo $repo), @@ -658,7 +664,7 @@ Assert-Equal 'the probe detects a secret left in the filesystem' 0 $controlLeak. Write-Fixture $dockerfile ($dockerfileBody + $secretStanza) Write-Step 'rebuilding with the hook active' -$hookRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +$hookRun = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'anvil-fmt') -AllowFailure Assert-Equal 'the run with a hook succeeds' 0 $hookRun.ExitCode Assert-That 'the hook announced itself at build time' ($hookRun.StdErr -match 'Anvil-BuildSecrets') Assert-That 'the build secret was declared' ($hookRun.StdErr -match 'build secrets: e2e_token') @@ -684,7 +690,7 @@ $leak = Invoke-Engine -Arguments @( Assert-Equal 'the secret is absent from the image filesystem' 1 $leak.ExitCode Write-Step 'checking that the forwarded value arrives inside the container' -$showEnv = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'e2e-show-env') -AllowFailure +$showEnv = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'e2e-show-env') -AllowFailure Assert-That 'the run-time value reaches a recipe in the container' ` ($showEnv.StdOut -match 'E2E:run-value') "stdout: $($showEnv.StdOut)`nstderr: $($showEnv.StdErr)" @@ -698,7 +704,7 @@ function Anvil-BuildSecrets { } '@ -$emptyHook = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure +$emptyHook = Invoke-Just -Repo $repo -Arguments @('anvil-container', 'just', 'anvil-fmt') -AllowFailure Assert-That 'an empty secret aborts the run' ($emptyHook.ExitCode -ne 0) ` 'BuildKit would mount an empty secret and exit 0, tagging a degraded image with a valid hash' Assert-That 'the failure names the offending secret' ($emptyHook.StdErr -match "empty value for secret 'e2e_token'") ` @@ -757,7 +763,7 @@ Write-Section '10. Recipes run natively inside the image' $nestedReference = if ($buildSecretsSupported) { $secretReference } else { Get-ImageReference -Repo $repo } $nested = Invoke-Engine -Arguments @( 'run', '--rm', '-e', 'ANVIL_IN_CONTAINER=1', '-v', "$(ConvertTo-EnginePath $repo):/workspace", '-w', '/workspace', - $nestedReference, 'just', 'anvil-container', 'anvil-fmt' + $nestedReference, 'just', 'anvil-container', 'just', 'anvil-fmt' ) -AllowFailure Assert-Equal 'anvil-container passes through inside the image' 0 $nested.ExitCode Assert-That 'no engine was invoked from inside the container' ` diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index 64585d13..fbe34df4 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -305,7 +305,7 @@ function Invoke-Suite([string]$EngineName) { Assert-That 'anvil-container-status reports an image reference' ([bool]$reference) 'no image: line in status output' Write-Step "reference: $reference" - $up = Invoke-Just -Arguments @('anvil-container', 'anvil-container-tag') -AllowFailure + $up = Invoke-Just -Arguments @('anvil-container', 'just', 'anvil-container-tag') -AllowFailure Assert-Equal 'the first run builds the image if it is missing' 0 $up.ExitCode if ($up.ExitCode -ne 0) { Write-Tail $up.StdErr @@ -350,13 +350,13 @@ function Invoke-Suite([string]$EngineName) { # The check whose prebuilt binary exercises both the loader and the # advisory API. Kept as its own step so a regression names itself. - $aprz = Invoke-Just -Arguments @('anvil-container', 'anvil-aprz') -AllowFailure + $aprz = Invoke-Just -Arguments @('anvil-container', 'just', 'anvil-aprz') -AllowFailure Assert-Equal 'anvil-aprz runs inside the image' 0 $aprz.ExitCode if ($aprz.ExitCode -ne 0) { Write-Tail "$($aprz.StdOut)`n$($aprz.StdErr)" } # A check that reads the workspace rather than the network, so a failure # points at the mount rather than at connectivity. - $fmt = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure + $fmt = Invoke-Just -Arguments @('anvil-container', 'just', 'anvil-fmt') -AllowFailure Assert-Equal 'anvil-fmt runs inside the image' 0 $fmt.ExitCode if ($fmt.ExitCode -ne 0) { Write-Tail "$($fmt.StdOut)`n$($fmt.StdErr)" } @@ -364,7 +364,7 @@ function Invoke-Suite([string]$EngineName) { $secondReference = Get-ImageReference Assert-Equal 'the tag is stable across runs' $reference $secondReference - $reuse = Invoke-Just -Arguments @('anvil-container', 'anvil-fmt') -AllowFailure + $reuse = Invoke-Just -Arguments @('anvil-container', 'just', 'anvil-fmt') -AllowFailure Assert-Equal 'a later run succeeds' 0 $reuse.ExitCode Assert-That 'a later run does not rebuild the image' ` (-not ("$($reuse.StdOut)`n$($reuse.StdErr)" -match 'building |Step 1/|FROM ')) ` @@ -446,7 +446,7 @@ function Invoke-Suite([string]$EngineName) { foreach ($recipe in $Tier) { Write-Step "running $recipe (this is the long one)" $started = Get-Date - $run = Invoke-Just -Arguments @('anvil-container', $recipe) -AllowFailure + $run = Invoke-Just -Arguments @('anvil-container', 'just', $recipe) -AllowFailure $took = (Get-Date) - $started Assert-Equal ("{0} passes inside the image (took {1:mm\:ss})" -f $recipe, $took) 0 $run.ExitCode if ($run.ExitCode -ne 0) { Write-Tail "$($run.StdOut)`n$($run.StdErr)" 40 } From 5f30dbdc254008ce137b796830b4624ecf0b8ef8 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 15:08:53 +0200 Subject: [PATCH 66/81] docs(anvil): drop the verification section and use a common container example Section 10 duplicated what the scripts themselves document, and the command-surface example now shows a tier rather than a recipe that happens to take an argument. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/docs/design/containers.md | 25 +------------------- 1 file changed, 1 insertion(+), 24 deletions(-) diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 57bedab1..4cf17b36 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -38,7 +38,6 @@ wraps, and [extensibility.md](./extensibility.md) for the catalog seam a downstr - [7.4 Trust boundary](#74-trust-boundary) - [8. Customization](#8-customization) - [9. Limitations](#9-limitations) -- [10. Verification](#10-verification) ## 1. Purpose @@ -60,7 +59,7 @@ own agents. The image is pinned to resemble that environment, not to reproduce i container is requested by name. Anvil recipes are reached by naming `just`, like any other command: ```bash -just anvil-container just anvil-setup binstall +just anvil-container just anvil-pr just anvil-container cargo build ``` @@ -623,26 +622,4 @@ guard. A different base OS with a different toolchain source is two region repla - anvil never pushes or promotes an image. It builds one, and will use one a hook fetched (§7.3); publishing belongs to whoever owns the registry. -## 10. Verification - -`scripts/test-anvil-container.ps1` exercises the behaviour above end to end against a real engine, driving only the -public surface. It needs a live daemon, so it cannot join `anvil-pr`; the unit tests in `artifacts::container` and -the emitted-tree snapshots are what run unattended. - -```powershell -./scripts/test-anvil-container.ps1 # docker -./scripts/test-anvil-container.ps1 -Engine podman # podman -``` - -`scripts/test-anvil-dogfood.ps1` is the complement: rather than a synthetic fixture it runs this repository's own -generated tree in its own image, which is what catches the defects a fixture is too small to have — a check whose tool -is missing, a mount whose permissions are wrong, a variable that does not cross the boundary. It mutates the working -tree while asserting which edits rename the image, and restores each file from a byte copy; if it is interrupted -mid-run, `cargo run -p cargo-anvil -- anvil` returns the generated tree to a known state. - -```powershell -./scripts/test-anvil-dogfood.ps1 # docker, full tier -./scripts/test-anvil-dogfood.ps1 -Engine podman -SkipTier # podman, mechanism only -``` - [design]: ./README.md From 8db0e9181239ccb667e1ee6fc294b43ae5bea199 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 15:17:08 +0200 Subject: [PATCH 67/81] docs(anvil): state the region contract instead of describing it Section 3 names the six regions, what anvil maintains in each and what belongs in the gap after it, so a reader can place an addition without reading the Dockerfile. The extensibility property is stated once: a gap sits where its kind of addition works, and using one keeps base-image and tool-pin updates flowing. The parser-directive line now says who owns it and what happens on upgrade rather than why it cannot be a region, and the classification of an existing Dockerfile is stated as the three outcomes a caller can get. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/docs/design/containers.md | 106 +++++++++---------- 1 file changed, 52 insertions(+), 54 deletions(-) diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 4cf17b36..1da31a82 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -120,62 +120,60 @@ repo/ `container.just` and `Dockerfile.dockerignore` are owned files carrying the usual `DO NOT EDIT DIRECTLY` marker. -The **Dockerfile is a user-composed file with managed regions**, not an owned file. Anvil owns six regions inside it -and keeps them current; the gaps between them are the repository's and are preserved byte-for-byte. Exactly one line — -`# syntax=docker/dockerfile:1` — sits outside a region, because it cannot live inside one. +The Dockerfile is **composed**, not owned: anvil maintains six managed regions inside it, and the repository owns +everything between them. Regions are updated in place on every run. Gap content is preserved byte-for-byte and is +never read, rewritten or reordered. -| Region | Contains | Gap that follows it is for | +| Region | What anvil puts there | What belongs in the gap after it | | --- | --- | --- | -| `anvil-container-header` | what the regions are and which gap takes what | — | -| `anvil-container-base-image` | `ARG BASE_IMAGE`, digest-pinned | re-declaring `ARG BASE_IMAGE` to build on your own base | -| `anvil-container-base` | `FROM`, the four download pins, `ENV` | a root CA, `http_proxy`, an internal apt mirror — anything needed to reach the network at all | -| `anvil-container-tools` | system packages, `pwsh`, `just`, `rustup`, `cargo-binstall` | libraries a catalog tool needs to *compile*, when `binstall` falls back to a source build | -| `anvil-container-setup` | `COPY` of the recipe tree, `just anvil-setup` | what the repository's own checks need at run time; also the cheapest layer to add to | -| `anvil-container-entry` | `ANVIL_IN_CONTAINER`, `WORKDIR`, `CMD` | — | - -**Why not an owned file that invites edits.** `updates.md` §2 preserves an edited owned file and writes anvil's version -to `.anvil-proposed`. There is no three-way merge and no recorded common ancestor, so every upgrade hands the -repository two files to reconcile by hand — and a side file does not get reconciled indefinitely. Here that failure is -silent in a way that matters: the file carries the base digest and four tool pins, so a repository that edits it once -keeps building on the base and versions frozen at that moment, while `anvil-container-tag` resolves happily *because -the tag hashes their file*. The identity scheme works perfectly and still names a stale image. - -**What regions buy.** They do not make anvil's content unwritable: §2's ownership rules apply to a region -body exactly as they do to a file, so an edit *inside* a region is still preserved and still produces a proposal rather -than being overwritten. Anvil never destroys repository content, and a special case here would be the one place it did. -What they remove is the reason to edit: every legitimate addition — another base image, a root CA, a -build dependency, a run-time tool — has a gap that is the *correct* place for it, chosen by what must already be true -at that point in the build. - -**The base image is the case that most needed a gap of its own.** It is the setting a repository is likeliest to want, -and a single region holding both `ARG BASE_IMAGE` and the `FROM` that consumes it would have made overriding it mean -editing anvil's content — freezing every other pin in the file to buy one substitution. Split across two regions, a -second `ARG BASE_IMAGE=…` in the gap wins (a later declaration replaces the default) and anvil keeps updating -everything the repository did not touch. The residual risk is real and worth stating: a base with an older glibc breaks -`binstall`, and switching the catalog to source installs is not a repository-level lever. - -**Three constraints the region engine had to grow for this.** All are specific to a Dockerfile; none applies to the -order-independent TOML and line-set hosts anvil already had: - -- **`# syntax=docker/dockerfile:1` must be line 1.** BuildKit honours a parser directive only when nothing precedes it, - not even a comment — and a region's opening sentinel *is* a comment, so the directive cannot live inside one without - being silently demoted, leaving the build on the default frontend with nothing failing to say so. It is therefore - the whole of the **scaffold** anvil writes when the file does not exist and never reconciles afterwards. Keeping the - scaffold to that one line is deliberate: anything anvil owns that sits outside a region can never be corrected on a - repository that already generated the file. -- **Region order is semantic.** `ARG BASE_IMAGE` must precede the `FROM` that consumes it, `FROM` must precede - everything, and the toolchain must exist before `anvil-setup` runs. The engine checks the on-disk sequence against - the declared one and **refuses** the host, reporting which region is out of place, rather than emitting a Dockerfile - that is wrong. -- **A missing region is inserted in order, not appended.** Appending at end-of-file is right for every other host and - wrong here: a region added in a later release would land after the ones it must precede. Anvil splices it at its - declared position instead — after the nearest preceding region that is present, or directly below the scaffold — so - adding a region is an ordinary update that leaves the repository's gap content untouched. - -A repository upgrading from the release that owned this path outright is re-seeded rather than appended to: a file -tracked as an owned file in the lock and carrying none of the regions is a previous render, not composition. A -Dockerfile the repository wrote itself is in neither state and is refused, since there is nowhere to splice the regions -that would not put its content above `FROM`. +| `anvil-container-header` | An orientation comment naming the regions and their gaps. | — | +| `anvil-container-base-image` | `ARG BASE_IMAGE`, pinned to a digest. | A second `ARG BASE_IMAGE=…` to build on a different base. | +| `anvil-container-base` | `FROM`, the version pins for `pwsh`, `just`, `rustup` and `cargo-binstall`, and the `ENV` block. | Anything the first network access needs: a root CA, `http_proxy`, an internal package mirror. | +| `anvil-container-tools` | System packages and those four tools. | Libraries a catalog tool needs to compile, for tools `binstall` has no prebuilt binary for. | +| `anvil-container-setup` | `COPY` of the recipe tree, then `just anvil-setup`. | Anything the repository's own checks need at run time. | +| `anvil-container-entry` | `ANVIL_IN_CONTAINER`, `WORKDIR`, `CMD`. | — | + +Each gap sits at the only point in the build where its kind of addition works: a certificate has to land before the +first download, a run-time tool after the toolchain exists. That is what makes the image extensible without forking +the catalog — a repository adds to a gap and keeps receiving base-image and tool-pin updates, where a fork or an edit +inside a region freezes them. + +Line 1 is `# syntax=docker/dockerfile:1` and belongs to no region. Anvil writes it when it creates the file and never +touches it again; a repository that needs a different frontend edits that line and owns it from then on. + +**Why not an owned file.** An edited owned file is preserved and anvil's version is written to `.anvil-proposed` +(`updates.md` §2). There is no three-way merge and no recorded common ancestor, so each upgrade leaves two files to +reconcile by hand. For this file the consequence is silent: it carries the base digest and four tool pins, so a +repository that edits it once builds on a frozen base and frozen versions indefinitely, while `anvil-container-tag` +still resolves, because the tag hashes the file as it stands. Identity stays correct and the image stays stale. + +**Regions are not write protection.** The ownership rules in `updates.md` §2 apply to a region body exactly as they do +to a file: an edit inside a region is preserved and produces a proposal rather than being overwritten. The gaps exist +so that editing a region is never the right way to add something. + +**Overriding the base image.** `ARG BASE_IMAGE` and the `FROM` that consumes it are separate regions, so the override +is a gap edit rather than a region edit: a second `ARG BASE_IMAGE=…` in the gap wins, because a later declaration +replaces the default, and every pin the repository did not touch keeps updating. A base with an older glibc breaks +`binstall`, and moving the catalog to source installs is not a repository-level lever. + +**Three properties a Dockerfile host requires that an order-independent TOML or line-set host does not:** + +- **The parser directive must be line 1.** BuildKit honours `# syntax=…` only when nothing precedes it, not even a + comment, and a region's opening sentinel is a comment. The directive therefore cannot be managed, and is instead the + whole of the scaffold anvil writes when the file is absent. The scaffold is one line because anvil never reconciles + it: anything placed there is uncorrectable on a repository that has already generated the file. +- **Region order is semantic.** `ARG BASE_IMAGE` precedes the `FROM` that consumes it, `FROM` precedes everything, and + the toolchain exists before `anvil-setup` runs. Anvil compares the on-disk sequence with the declared one and refuses + the file, naming the region that is out of place, rather than writing a Dockerfile that cannot build. +- **A missing region is spliced at its declared position, not appended.** Appending suits every other host; here a + region introduced in a later release would land after ones it must precede. Anvil inserts it after the nearest + preceding region present in the file, or directly below the scaffold, leaving gap content untouched. + +Anvil classifies an existing Dockerfile before writing to it. A file the lock records as an owned file and that +carries no regions is a render from a version that owned the whole path; it is replaced. A file carrying every region +is composed and is updated in place. Anything else — a Dockerfile the repository wrote itself, or one whose regions +have been removed — is refused, because there is no position for the regions that would not place existing content +above `FROM`. The refusal names the file and the recovery; nothing is written to it. The image installs its tools by running `just anvil-setup`, the same recipe the checks use, reading the same generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the From 3aa8a4e9c9aa21fe5960dba102d40ae5d4d49212 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 17:12:33 +0200 Subject: [PATCH 68/81] fix(anvil): delete a retired owned file under its on-disk casing The removal pass resolved the recorded path only for its liveness checks; the disk read and the delete both used the casing the lock happened to record. On a case-sensitive filesystem a case-only rename therefore made a file that is still present read as absent, so `decide_removal` returned `AlreadyGone`, the delete was a no-op, and the lock entry was purged -- leaving the file on disk owned by nobody. A customized file in that state would have been deleted outright. Both the read and the delete now resolve the casing first, matching the live half of the same loop. The build context admits `.anvil/container/` so a gap can COPY a file placed beside the Dockerfile. Three places said it admits only `justfiles/anvil/` and `rust-toolchain.toml`, which contradicts the ignore file and the paragraph documenting that a copied file is digested. The dogfood tool loop matched two known failure shapes and discarded the exit code, so a crash, a permission error or any unrecognised diagnostic left both result sets empty and both assertions passing. An unexplained nonzero exit is now a failure in its own right. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/README.md | 8 +-- crates/cargo-anvil/docs/design/containers.md | 16 +++--- crates/cargo-anvil/src/lib.rs | 6 +-- crates/cargo-anvil/src/plan.rs | 13 +++-- crates/cargo-anvil/src/run.rs | 5 +- crates/cargo-anvil/tests/container_upgrade.rs | 54 +++++++++++++++++++ scripts/test-anvil-dogfood.ps1 | 12 +++++ 7 files changed, 95 insertions(+), 19 deletions(-) diff --git a/crates/cargo-anvil/README.md b/crates/cargo-anvil/README.md index a5aee281..e8a04a1e 100644 --- a/crates/cargo-anvil/README.md +++ b/crates/cargo-anvil/README.md @@ -251,9 +251,9 @@ A downstream catalog that needs a different base OS for every repository it manages replaces the base and tool regions instead, inheriting the catalog install and the entry contract. A replacement that copies more of the tree must replace the ignore file with it, since the build context admits only -`justfiles/anvil/` and `rust-toolchain.toml`. See [`artifacts::container`][__link1] -and the design document for the full contract, the host setup for each -engine, and the known limitations. +`justfiles/anvil/`, `.anvil/container/` and `rust-toolchain.toml`. See +[`artifacts::container`][__link1] and the design document for the full contract, the +host setup for each engine, and the known limitations. ### Checks and tiers @@ -482,7 +482,7 @@ And `docs/verification.md` for the continuous-validation strategy. This crate was developed as part of The Oxidizer Project. Browse this crate's source code. - [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbmII2hgTks2UbJ479ivzen1IbFHImKDbbn5obcqda0xJCu1FhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls + [__cargo_doc2readme_dependencies_info]: ggGmYW0CYXZlMC43LjJhdIQbFhzZ8rzWNNYbuRaDSGWynFgbH4PMdoT7GNcbVwNPtPjAhvFhYvRhcoQbgKHiUZXoKcwbNzADe3lPpOAbmHjFvxOs3xUb5ecQdSnq-vdhZIGDa2NhcmdvLWFudmlsZTAuNS4wa2NhcmdvX2Fudmls [__link0]: https://crates.io/crates/cargo-delta [__link1]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts::container [__link10]: https://docs.rs/cargo-anvil/0.5.0/cargo_anvil/?search=artifacts diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index 1da31a82..b7f95d5e 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -179,10 +179,12 @@ The image installs its tools by running `just anvil-setup`, the same recipe the generated pins. There is no second tool list to keep synchronized, and consequently a tool-pin change renames the image (§4.1). -`Dockerfile.dockerignore` scopes the build context to `justfiles/anvil/` and `rust-toolchain.toml`, denying everything -else. The whole recipe tree is copied because `just` has to parse it to run `anvil-setup`, and the whole tree is -hashed (§4). BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the -repository neither needs to own a root ignore file nor can have one silently override this. +`Dockerfile.dockerignore` scopes the build context to `justfiles/anvil/`, `.anvil/container/` and +`rust-toolchain.toml`, denying everything else. The recipe tree is copied whole because `just` has to parse it to run +`anvil-setup`, and it is hashed whole (§4). `.anvil/container/` is admitted so a gap can `COPY` a file placed beside +the Dockerfile; anvil's own `.anvil-proposed` review artifacts are excluded from both the context and the digest. +BuildKit reads `.dockerignore` in preference to a root `.dockerignore`, so the repository neither needs to +own a root ignore file nor can have one silently override this. ## 4. Image identity @@ -583,9 +585,9 @@ Replacing a *region* rather than the whole file is what makes a downstream catal msrustup catalog rewrites the base and tool layers and nothing else. Replacing `dockerfile_setup()` reintroduces the second tool list the design exists to avoid, and is almost never right. -**A replacement must keep the ignore file in step.** A region that `COPY`s anything beyond `justfiles/anvil/` and -`rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the added paths never reach the -build context and the build fails on a missing file. +**A replacement must keep the ignore file in step.** A region that `COPY`s anything outside `justfiles/anvil/`, +`.anvil/container/` and `rust-toolchain.toml` must also replace `artifacts::container::dockerignore()` (§3), or the +added paths never reach the build context and the build fails on a missing file. **Anything extra it copies is digested, provided it lives under `.anvil/container/`.** The hashed set is that whole directory (§4.1), so an installer script, a config file or a certificate placed beside the Dockerfile is an input: diff --git a/crates/cargo-anvil/src/lib.rs b/crates/cargo-anvil/src/lib.rs index e30417b9..8dbef9b0 100644 --- a/crates/cargo-anvil/src/lib.rs +++ b/crates/cargo-anvil/src/lib.rs @@ -252,9 +252,9 @@ //! manages replaces the base and tool regions instead, inheriting the catalog //! install and the entry contract. A replacement that copies more of the tree //! must replace the ignore file with it, since the build context admits only -//! `justfiles/anvil/` and `rust-toolchain.toml`. See [`artifacts::container`] -//! and the design document for the full contract, the host setup for each -//! engine, and the known limitations. +//! `justfiles/anvil/`, `.anvil/container/` and `rust-toolchain.toml`. See +//! [`artifacts::container`] and the design document for the full contract, the +//! host setup for each engine, and the known limitations. //! //! ## Checks and tiers //! diff --git a/crates/cargo-anvil/src/plan.rs b/crates/cargo-anvil/src/plan.rs index b0574a5d..fb47e40d 100644 --- a/crates/cargo-anvil/src/plan.rs +++ b/crates/cargo-anvil/src/plan.rs @@ -23,6 +23,7 @@ use std::path::{Path, PathBuf}; use ohno::{AppError, IntoAppError as _}; use crate::decision::Decision; +use crate::io::resolve_existing_case_insensitive; use crate::manifest::{Manifest, RegionKey}; /// What is being changed by a single plan item. @@ -466,10 +467,14 @@ impl Plan { } (Target::File { path }, Decision::Remove) => { // Untouched orphan file: delete and drop the - // manifest entry. If the file is already missing - // (race / external delete), absorb the error so - // the result is idempotent. - let abs = repo_root.join(path); + // manifest entry. The path is resolved to its on-disk + // casing first, because the manifest key is whatever + // casing was recorded and the file may since have been + // renamed in case only; deleting the unresolved path + // would leave the file behind with no lock entry. + // If the file is already missing (race / external + // delete), absorb the error so the result is idempotent. + let abs = repo_root.join(resolve_existing_case_insensitive(repo_root, path)); if let Err(e) = std::fs::remove_file(&abs) && e.kind() != std::io::ErrorKind::NotFound { diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index cf0a1b54..49818852 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -728,7 +728,10 @@ fn plan_removals( } continue; } - let disk = read_file_if_present(&repo_root.join(path))?; + // Read under the resolved casing: a lock entry recorded before a + // case-only rename would otherwise find nothing on disk, classify a + // file that is still present as `AlreadyGone`, and delete it. + let disk = read_file_if_present(&repo_root.join(&resolved))?; let disk_checksum = disk.as_deref().map(checksum_str); match decide_removal(last, disk_checksum.as_deref()) { // A file still matching its last render is safe to delete; an diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index a4266535..a4d1866d 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -893,3 +893,57 @@ fn refusing_a_composed_host_spares_a_retired_region_entry_too() { "precondition: the host must actually have been refused" ); } + +/// A retired owned file is deleted under its real on-disk name. The lock +/// records whatever casing was written; on a case-sensitive filesystem a +/// case-only rename makes the recorded path absent, which reads as +/// `AlreadyGone` and would delete nothing while the lock entry is purged -- +/// leaving the file on disk owned by no one. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_retired_owned_file_is_removed_under_its_on_disk_casing() { + let tmp = generated_tree(); + let root = tmp.path(); + if !filesystem_is_case_sensitive(root) { + // The two spellings name one file here, so nothing distinguishes the + // fix from the defect. The property only exists where they differ. + return; + } + + // A file anvil no longer declares, tracked in the lock under one casing + // and present on disk under another. + let recorded = "justfiles/anvil/Retired.just"; + let on_disk = root.join("justfiles/anvil/retired.just"); + let body = "# retired\n"; + write(&on_disk, body); + let mut manifest = Manifest::load(root).unwrap(); + manifest.set_file(recorded, checksum_str(body)); + manifest.save(root).unwrap(); + + let outcome = run_update(&Catalog::anvil(), &local(), root).unwrap(); + + assert!( + outcome + .plan + .items() + .iter() + .any(|i| i.decision == Decision::Remove && matches!(&i.target, Target::File { path } if path == recorded)), + "the retired file must be planned for removal" + ); + assert!(!on_disk.exists(), "the file must be gone from disk, not merely untracked"); + assert!( + !Manifest::load(root).unwrap().files.contains_key(recorded), + "the lock entry must be purged" + ); +} + +/// Whether two spellings of one name are distinct paths on the filesystem +/// backing `root`. The casing behaviour under test is only observable when +/// they are. +fn filesystem_is_case_sensitive(root: &Path) -> bool { + let probe = root.join("anvil-case-probe"); + std::fs::write(&probe, b"").unwrap(); + let sensitive = !root.join("ANVIL-CASE-PROBE").exists(); + std::fs::remove_file(&probe).unwrap(); + sensitive +} diff --git a/scripts/test-anvil-dogfood.ps1 b/scripts/test-anvil-dogfood.ps1 index fbe34df4..b677af32 100644 --- a/scripts/test-anvil-dogfood.ps1 +++ b/scripts/test-anvil-dogfood.ps1 @@ -331,6 +331,10 @@ function Invoke-Suite([string]$EngineName) { Write-Step "$($tools.Count) pinned tools" $broken = @() $missing = @() + $unexplained = @() + # Tools with no `--version` that exits 0. Empty: every pinned tool is a + # cargo subcommand or a standalone binary that reports its version. + $noVersionFlag = @() foreach ($tool in $tools.Keys) { $run = Invoke-Engine -Arguments @('run', '--rm', $reference, $tool, '--version') -AllowFailure $combined = "$($run.StdOut)`n$($run.StdErr)" @@ -341,10 +345,18 @@ function Invoke-Suite([string]$EngineName) { $broken += "$tool -> $line" } elseif ($combined -match 'executable file .*not found|no such file or directory') { $missing += $tool + } elseif ($run.ExitCode -ne 0 -and $tool -notin $noVersionFlag) { + # The exit code is the only signal that covers every remaining + # failure shape: a crash, an illegal instruction, a permission + # error, or a loader diagnostic neither pattern above recognises. + # Without this the assertions below pass on all of them. + $first = (($combined -split "`r?`n") | Where-Object { $_.Trim() } | Select-Object -First 1) + $unexplained += "$tool -> exit $($run.ExitCode): $($first)".Trim() } } Assert-That 'every pinned tool executes inside the image' ($broken.Count -eq 0) ($broken -join "`n") Assert-That 'every pinned tool is present in the image' ($missing.Count -eq 0) ($missing -join ', ') + Assert-That 'every pinned tool reports its version successfully' ($unexplained.Count -eq 0) ($unexplained -join "`n") Write-Section "$EngineName : checks" From a0611bc4228b12021fe882e05c21e11652de0f7a Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 17:30:22 +0200 Subject: [PATCH 69/81] fix(anvil): splice a region removal into this pass's text, not the file's The region removal loop resolved the host's casing for its liveness and refusal checks, then read, spliced and wrote under the casing the lock recorded. `HostTextCache` is keyed by the exact string and the writes of the same pass seeded it under the resolved spelling, so after a case-only rename the removal missed the cache, re-read the pre-pass file, and wrote that text back. Removals apply after writes, so the stale text won: a host that gained a region in the same pass lost it again, and for a composed Dockerfile that is every region in the file. The read, the cache update and the write now use the resolved spelling; the manifest key stays as recorded so the retired entry is still purged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/src/plan.rs | 5 +- crates/cargo-anvil/src/run.rs | 11 ++-- crates/cargo-anvil/tests/container_upgrade.rs | 57 +++++++++++++++++++ 3 files changed, 68 insertions(+), 5 deletions(-) diff --git a/crates/cargo-anvil/src/plan.rs b/crates/cargo-anvil/src/plan.rs index fb47e40d..239a8184 100644 --- a/crates/cargo-anvil/src/plan.rs +++ b/crates/cargo-anvil/src/plan.rs @@ -485,8 +485,11 @@ impl Plan { (Target::Region { host, id }, Decision::Remove) => { // Untouched orphan region: splice the markers + body // out of the host file and drop the manifest entry. + // The host is resolved to its on-disk casing for the + // write, for the reason the File arm above gives; the + // manifest key stays as recorded so the entry is purged. let spliced = item.spliced_host.as_ref().expect("region Remove must carry spliced host"); - let abs = repo_root.join(host); + let abs = repo_root.join(resolve_existing_case_insensitive(repo_root, host)); write_file(&abs, spliced)?; next.regions.remove(&RegionKey { host: host.clone(), diff --git a/crates/cargo-anvil/src/run.rs b/crates/cargo-anvil/src/run.rs index 49818852..4ec11368 100644 --- a/crates/cargo-anvil/src/run.rs +++ b/crates/cargo-anvil/src/run.rs @@ -778,14 +778,14 @@ fn plan_removals( if matches!(composed.states.get(&resolved_host), Some(ComposedHostState::Unsafe(_))) { continue; } - if live_regions.contains(&(resolved_host, key.id.clone())) { + if live_regions.contains(&(resolved_host.clone(), key.id.clone())) { plan.push(PlanItem::orphaned_kept(Target::Region { host: key.host.clone(), id: key.id.clone(), })); continue; } - let Some(host_text) = hosts.get_or_read(repo_root, &key.host)? else { + let Some(host_text) = hosts.get_or_read(repo_root, &resolved_host)? else { // Host file is gone entirely; just drop the manifest // entry. Emit OrphanedKept (no-op apply) so the plan // can record the transfer of ownership consistently. @@ -807,9 +807,12 @@ fn plan_removals( // Splice against — and update — the accumulated host text // so a removal composes with the writes already planned // for this host this pass instead of clobbering them - // (their item is applied earlier; this one, later). + // (their item is applied earlier; this one, later). The + // cache is keyed by the resolved spelling, which is what + // the writes used; reading under the recorded spelling + // would miss it and splice into the pre-pass text. let spliced = remove_region(&host_text, &key.id, syntax)?; - hosts.set(&key.host, spliced.clone()); + hosts.set(&resolved_host, spliced.clone()); plan.push(PlanItem::remove_region(key.host.clone(), key.id.clone(), spliced)); } RemovalDecision::OrphanedKept | RemovalDecision::AlreadyGone => { diff --git a/crates/cargo-anvil/tests/container_upgrade.rs b/crates/cargo-anvil/tests/container_upgrade.rs index a4d1866d..c098c326 100644 --- a/crates/cargo-anvil/tests/container_upgrade.rs +++ b/crates/cargo-anvil/tests/container_upgrade.rs @@ -947,3 +947,60 @@ fn filesystem_is_case_sensitive(root: &Path) -> bool { std::fs::remove_file(&probe).unwrap(); sensitive } + +/// A region removal must splice into the text this pass wrote, not the text +/// that was on disk before it. The host text cache is keyed by the spelling +/// the writes used, so a removal reading under the lock's spelling misses the +/// cache, re-reads the pre-pass file, and writes that back over the regions +/// the same run produced. Removals apply after writes, so the stale text wins. +/// +/// The two spellings diverge after a case-only rename of the host, which is +/// what this sets up: the lock keeps `Justfile`, the file on disk is +/// `justfile`, and both a write and a removal target it in one pass. +#[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] +#[test] +fn a_region_removal_composes_with_the_writes_of_the_same_pass() { + const IMPORTS_REGION_ID: &str = "anvil-imports"; + + let tmp = generated_tree(); + let root = tmp.path(); + let recorded = root.join("Justfile"); + let renamed = root.join("justfile"); + + // Drop the region anvil maintains so this pass has to write it back, and + // plant a retired one so the removal path runs against the same host. + let text = std::fs::read_to_string(&recorded).unwrap(); + let with_retired = + format!("{text}\n# >>> anvil-managed: {RETIRED_REGION_ID}\n{RETIRED_REGION_BODY}# <<< anvil-managed: {RETIRED_REGION_ID}\n"); + let staged = remove_region_block(&with_retired, IMPORTS_REGION_ID); + std::fs::remove_file(&recorded).unwrap(); + write(&renamed, &staged); + + let mut manifest = Manifest::load(root).unwrap(); + manifest.set_region("Justfile", RETIRED_REGION_ID, checksum_str(RETIRED_REGION_BODY)); + manifest.save(root).unwrap(); + + run_update(&Catalog::anvil(), &local(), root).unwrap(); + + let after = std::fs::read_to_string(&renamed).unwrap(); + assert!( + after.contains(&format!("# >>> anvil-managed: {IMPORTS_REGION_ID}")), + "the region written this pass must survive the removal of another region in the same host:\n{after}" + ); + assert!( + !after.contains(RETIRED_REGION_ID), + "the retired region must still be removed:\n{after}" + ); +} + +/// Strip a whole managed region, sentinels included, from `text`. +fn remove_region_block(text: &str, id: &str) -> String { + let open = format!("# >>> anvil-managed: {id}"); + let close = format!("# <<< anvil-managed: {id}"); + let start = text.find(&open).expect("region must be present to remove"); + let end = text[start..].find(&close).expect("region must be closed") + start + close.len(); + let mut out = String::with_capacity(text.len()); + out.push_str(&text[..start]); + out.push_str(text[end..].trim_start_matches('\n')); + out +} From 980af65b6f43553514360a9acf55ee8b9a8802cf Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Fri, 28 Aug 2026 18:42:30 +0200 Subject: [PATCH 70/81] fix(anvil): reject a manifest path that escapes the repository Every path in the manifest is joined to the repository root and then read, written or deleted. `Path::join` discards the base when given an absolute or drive-qualified path, and `..` climbs out of it, so a corrupted or hand-edited `.anvil.lock` could direct any of those operations outside the repository. Paths are always stored `/`-separated and repository-relative, so the check costs nothing a valid manifest can trip on. Validation goes at the load site rather than the delete sites, so it covers every path the manifest yields, including the ones used for writes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/src/manifest.rs | 51 +++++++++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/crates/cargo-anvil/src/manifest.rs b/crates/cargo-anvil/src/manifest.rs index 0c947969..6df89243 100644 --- a/crates/cargo-anvil/src/manifest.rs +++ b/crates/cargo-anvil/src/manifest.rs @@ -15,7 +15,7 @@ //! exist today). use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; +use std::path::{Component, Path, PathBuf}; use ohno::{AppError, IntoAppError as _, app_err, bail}; use toml_edit::{ArrayOfTables, DocumentMut, Item, Table, value}; @@ -61,6 +61,29 @@ pub struct RegionKey { pub id: String, } +/// Reject a manifest path that would resolve outside the repository root. +/// +/// Every path in the manifest is joined to the repository root and then read, +/// written or deleted. `Path::join` replaces the base entirely when given an +/// absolute path or a drive-qualified one, and `..` climbs out of it, so a +/// manifest that has been corrupted or hand-edited could direct those +/// operations at arbitrary locations. Paths are always stored `/`-separated +/// and repository-relative, so anything else is malformed. +/// +/// # Errors +/// +/// Returns an error naming `context` and the offending path when it is +/// absolute, carries a drive or UNC prefix, or contains a `..` component. +fn ensure_contained(path: &str, context: &str) -> Result<(), AppError> { + let malformed = Path::new(path) + .components() + .any(|component| matches!(component, Component::RootDir | Component::Prefix(_) | Component::ParentDir)); + if malformed { + bail!("{context} '{path}' must be a relative path inside the repository"); + } + Ok(()) +} + impl Manifest { /// Path the manifest should be saved at, given a workspace root. #[must_use] @@ -129,6 +152,7 @@ impl Manifest { if files.insert(path.clone(), checksum).is_some() { bail!("duplicate [[file]] entry for '{path}'"); } + ensure_contained(&path, "[[file]] entry")?; } } @@ -154,6 +178,7 @@ impl Manifest { if regions.insert(key.clone(), checksum).is_some() { bail!("duplicate [[region]] entry for host '{}' id '{}'", key.host, key.id); } + ensure_contained(&key.host, "[[region]] host")?; } } @@ -505,4 +530,28 @@ mod tests { assert!(!manifest.has_region_host(".anvil/container/Other")); assert!(!Manifest::default().has_region_host(".anvil/container/Dockerfile")); } + #[test] + fn rejects_a_file_path_that_escapes_the_repository() { + for escape in ["../outside.txt", "/etc/passwd", "a/../../b.txt"] { + let toml = format!("version = 1\ntool = \"anvil\"\n\n[[file]]\npath = \"{escape}\"\nchecksum = \"sha256:x\"\n"); + let err = Manifest::parse(&toml).unwrap_err(); + assert!( + format!("{err}").contains("must be a relative path inside the repository"), + "unexpected error for '{escape}': {err}" + ); + } + } + + #[test] + fn rejects_a_region_host_that_escapes_the_repository() { + let toml = "version = 1\ntool = \"anvil\"\n\n[[region]]\nhost = \"../Justfile\"\nid = \"anvil-imports\"\nchecksum = \"sha256:x\"\n"; + let err = Manifest::parse(toml).unwrap_err(); + assert!(format!("{err}").contains("must be a relative path inside the repository"), "{err}"); + } + + #[test] + fn accepts_an_ordinary_nested_path() { + let toml = "version = 1\ntool = \"anvil\"\n\n[[file]]\npath = \"justfiles/anvil/tools.just\"\nchecksum = \"sha256:x\"\n"; + Manifest::parse(toml).expect("an ordinary nested path must be accepted"); + } } From 38640f053d233b1b72a286b5461f730fa6252304 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 16:49:19 +0200 Subject: [PATCH 71/81] fix(cargo-anvil): close five gaps in the container driver and retire the Dockerfile header region Five defects, each reachable from an ordinary setup: - A wrapped tier ran unauthenticated. `just --dry-run` reports the bodies just runs itself, not the body of a recipe one of them launches as a child process, so planning `anvil-scheduled` showed the `_anvil-unscoped` wrapper and none of the checks under it. The driver read that as "needs no token" and `anvil-aprz` then slept on the advisory API's unauthenticated rate limit, inside an image with no `gh` of its own. The driver now follows each nested target a plan names. - The boundary forwarded `ANVIL_INCLUDE_MODIFIED` / `_AFFECTED` / `_REQUIRED`, which nothing reads, and dropped `ANVIL_IMPACT`, which `anvil-impact` does. A CI group job exports `consume`; a container that did not inherit it recomputed scoping from a diff instead of trusting the artifact the group had downloaded. - `--separate-git-dir` produced a container in which git could not resolve a single ref. The mount triggered on the git directory differing from the common one, but that redirect leaves them equal, so the checkout kept a `.git` file naming a host path. The predicate is now the shape of `.git`. A git directory outside its common directory is refused with a message rather than mounted at a path that climbs out of the mount. - The recipes hard-coded `.anvil/container/Dockerfile` while the generator resolves that path against the casing already on disk. On a case-sensitive filesystem a repository carrying `dockerfile` had its regions maintained in a file the build then could not find. - The manifest's containment guard was lexical, so a path whose components are all ordinary still left the repository when one of them was a symlink pointing out of it. Since the manifest is committed, one commit could add both the link and the entry naming a path through it. Writes, deletes and proposals now resolve before acting. The Dockerfile loses its header region: it held only commentary, and a managed region that contributes no build instruction is a section that exists to hold prose. The guidance it carried is in containers.md, where it does not have to be reconciled into every adopter's file. The copyright notice moves to the scaffold, alongside the parser directive, so a fresh file still carries it. Remaining region comments state the constraint that is not visible in the instruction below them and nothing else. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 15 +- .anvil/container/Dockerfile | 57 ++--- crates/cargo-anvil/docs/design/containers.md | 21 +- .../src/anvil/artifacts/container.rs | 157 +++++++++---- crates/cargo-anvil/src/plan.rs | 111 ++++++++- .../container/Dockerfile.baseimage.region | 12 +- .../anvil/container/Dockerfile.header | 3 + .../anvil/container/Dockerfile.header.region | 19 -- .../anvil/container/Dockerfile.setup.region | 22 +- .../anvil/container/Dockerfile.tools.region | 4 - .../templates/justfiles/anvil/container.just | 161 +++++++++---- crates/cargo-anvil/tests/recipe_contracts.rs | 106 +++++++++ .../snapshots/snapshots__ado_backend.snap | 218 ++++++++++-------- .../snapshots/snapshots__github_backend.snap | 218 ++++++++++-------- .../snapshots/snapshots__local_only.snap | 218 ++++++++++-------- justfiles/anvil/container.just | 161 +++++++++---- 16 files changed, 982 insertions(+), 521 deletions(-) delete mode 100644 crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region diff --git a/.anvil.lock b/.anvil.lock index 09aa62d9..c64edf7f 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:4c142577a53154b2344281131398313ff4ab0063c80360eaaeaa0b77662b2555" +catalog_checksum = "sha256:844aa0ce3a36a02529782fbef8a4452933ef9f2312dee2ba758634af240df139" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:6be4e6d298e56b8b0eb5874e295b55b1a6f7c8f15c707c3d19db77ffee6543b8" +checksum = "sha256:35c801e86d2c2bfb94861a5ff86ecdcb991d2e86c79dfb7b1231ba8821864a54" [[file]] path = "justfiles/anvil/groups/pr-fast.just" @@ -235,27 +235,22 @@ checksum = "sha256:734e21d8ae8ce8c0a00f54a36a3a9f15a02b52eff11c27f3de4f7cd60bb95 [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-base-image" -checksum = "sha256:74ed18b9ccc5a5232be05392d572c2395d0f73c10da9b372c75280639e398dc6" +checksum = "sha256:64fa253bd48baecd440ae02cb13d2626e17be6e2bb0e1b19e3345baeade04abb" [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-entry" checksum = "sha256:7b409a9b560c214e10b50f74330fb6f8c0c12c3d83494e0dcf016f2411b50365" -[[region]] -host = ".anvil/container/Dockerfile" -id = "anvil-container-header" -checksum = "sha256:be87e9614441a6802039f95858f1fdd5c6116c4be5cc8fcad244763c32aabcaf" - [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-setup" -checksum = "sha256:7378551c253df01632d1d5827f698f3fee9f410b0113146130d69b2fbd9af473" +checksum = "sha256:ecba82c02af5b339dfa073239cfbe396205db6395288d891837f5fba3f7644cd" [[region]] host = ".anvil/container/Dockerfile" id = "anvil-container-tools" -checksum = "sha256:433506a65ac00b9ac6c4cb7063fc94e253215885c34e52605f952c484e029e8e" +checksum = "sha256:0b6923e81ec99170ece298fcd9ef875f26a3ff018708ce5ec21b4b1dd7a18900" [[region]] host = ".delta.toml" diff --git a/.anvil/container/Dockerfile b/.anvil/container/Dockerfile index 56b459d6..6fbb71a7 100644 --- a/.anvil/container/Dockerfile +++ b/.anvil/container/Dockerfile @@ -1,35 +1,16 @@ # syntax=docker/dockerfile:1 -# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps -# between them are yours and are preserved. Add to a gap, not inside a region: -# anvil will not overwrite an edit inside one, so editing there silently freezes -# the base digest and the tool pins at that moment. -# -# Which gap, since position decides whether a line works at all: -# -# after base-image re-declare ARG BASE_IMAGE to build on your own base -# after base root CA, proxy, apt mirror -- anything the first download needs -# after tools libraries a catalog tool needs to compile from source -# after setup what your own checks need at run time -# -# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, -# and a region sentinel is a comment. -# <<< anvil-managed: anvil-container-header # >>> anvil-managed: anvil-container-base-image -# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). -# `anvil-setup binstall` installs prebuilt binaries that need that runner's -# glibc, so this moves when the runner does. Digest-pinned because a floating -# tag can change under an image reference that claims to name fixed content. +# Prebuilt binaries installed by `anvil-setup binstall` link against this +# image's glibc, so it tracks the Linux runner the generated workflows use. +# Digest-pinned: a floating tag moves content under a reference that claims to +# name fixed content. # -# To build on another base, re-declare this in the gap below rather than editing -# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +# Re-declare BASE_IMAGE in the gap below to build on another base; a later ARG +# wins, and the pins anvil maintains stay current. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea # <<< anvil-managed: anvil-container-base-image @@ -87,10 +68,6 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. RUN curl -fsSLo /tmp/cargo-binstall.tgz \ "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ @@ -102,23 +79,17 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated catalog. -# The whole recipe tree is copied because `just` has to parse it, and the whole -# tree is hashed into the image tag: `anvil-setup` reaches the install recipes -# through the tier, group and check recipes, so any of them can change what this -# layer installs. -# -# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do -# not compile from source on every pinned toolchain. +# The whole recipe tree is copied because `just` parses it to reach the install +# recipes. # -# The credential files are removed in the same layer as the install. A build +# The credential files are removed in the same layer that used them: a build # secret never lands in a layer, but anything the install *writes* with it is -# ordinary content, and the `chmod` below would publish it world-readable. -# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. +# ordinary content, and the `chmod` below would publish it world-readable. A +# later `RUN` cannot undo that, because the earlier layer keeps them. # -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each, and an engine seeds a new volume from the image path -# it covers. A path that does not exist seeds as root-owned 0755, which the +# `registry` and `git` must exist before the `chmod`. The run mounts a named +# volume over each, and an engine seeds a new volume from the image path it +# covers; a path that does not exist seeds as root-owned 0755, which the # `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles diff --git a/crates/cargo-anvil/docs/design/containers.md b/crates/cargo-anvil/docs/design/containers.md index b7f95d5e..aa8bdea3 100644 --- a/crates/cargo-anvil/docs/design/containers.md +++ b/crates/cargo-anvil/docs/design/containers.md @@ -113,20 +113,19 @@ repo/ │ ├── container.just the anvil-container recipes │ └── … checks, groups, tiers, executed natively *inside* the image └── .anvil/container/ - ├── Dockerfile composed: anvil's six regions, your content in the gaps + ├── Dockerfile composed: anvil's five regions, your content in the gaps ├── Dockerfile.dockerignore what the build context admits └── hooks.ps1 optional; not emitted by default (§7) ``` `container.just` and `Dockerfile.dockerignore` are owned files carrying the usual `DO NOT EDIT DIRECTLY` marker. -The Dockerfile is **composed**, not owned: anvil maintains six managed regions inside it, and the repository owns +The Dockerfile is **composed**, not owned: anvil maintains five managed regions inside it, and the repository owns everything between them. Regions are updated in place on every run. Gap content is preserved byte-for-byte and is never read, rewritten or reordered. | Region | What anvil puts there | What belongs in the gap after it | | --- | --- | --- | -| `anvil-container-header` | An orientation comment naming the regions and their gaps. | — | | `anvil-container-base-image` | `ARG BASE_IMAGE`, pinned to a digest. | A second `ARG BASE_IMAGE=…` to build on a different base. | | `anvil-container-base` | `FROM`, the version pins for `pwsh`, `just`, `rustup` and `cargo-binstall`, and the `ENV` block. | Anything the first network access needs: a root CA, `http_proxy`, an internal package mirror. | | `anvil-container-tools` | System packages and those four tools. | Libraries a catalog tool needs to compile, for tools `binstall` has no prebuilt binary for. | @@ -138,8 +137,9 @@ first download, a run-time tool after the toolchain exists. That is what makes t the catalog — a repository adds to a gap and keeps receiving base-image and tool-pin updates, where a fork or an edit inside a region freezes them. -Line 1 is `# syntax=docker/dockerfile:1` and belongs to no region. Anvil writes it when it creates the file and never -touches it again; a repository that needs a different frontend edits that line and owns it from then on. +Line 1 is `# syntax=docker/dockerfile:1` and belongs to no region, because BuildKit honors the directive only when +nothing precedes it and a region sentinel is a comment. Anvil writes it, and the copyright notice under it, when it +creates the file and never touches either again; both are the repository's from then on. **Why not an owned file.** An edited owned file is preserved and anvil's version is written to `.anvil-proposed` (`updates.md` §2). There is no three-way merge and no recorded common ancestor, so each upgrade leaves two files to @@ -348,11 +348,18 @@ interactive session can run anything, and refusing there would reintroduce the s predicate is the variable rather than the name of a check, so a catalog that adds another GitHub-authenticated check is covered without touching the driver. +A plan covers the bodies `just` runs itself, not the body of a recipe that one of them launches as a child process. +The unscoped tier wrapper (§`helpers.just`) launches its tier that way, so planning `anvil-scheduled` shows the wrapper +alone. The driver therefore follows each nested target a plan names, until nothing new appears; without that, a wrapped +tier reads as needing nothing and `anvil-aprz` runs unauthenticated inside an image that has no `gh` of its own. + It also forwards the recipe contract's own inputs when they are set — `PR_TITLE`, `BASE_REF`, `GITHUB_BASE_REF`, -`SYSTEM_PULLREQUEST_TARGETBRANCH` and the `ANVIL_INCLUDE_*` filters — because a check that reads one natively must read +`SYSTEM_PULLREQUEST_TARGETBRANCH` and `ANVIL_IMPACT` — because a check that reads one natively must read the same value in a container. `anvil-pr-title` is the sharp case: with `PR_TITLE` unset it exits 0 with a skip notice, so dropping it at the boundary would let a title a native run rejects pass in a container while the tier still reported -green. They are forwarded by name and only when set, so an unset variable stays unset rather than arriving empty. +green. `ANVIL_IMPACT` is the other: a CI group job exports `consume`, and a container that did not inherit it would +recompute scoping from a diff instead of trusting the artifact the group downloaded. They are forwarded by name and +only when set, so an unset variable stays unset rather than arriving empty. A resolved token is set on the driver process, passed by name, and unset after the run, so it never reaches a host command line. Inside the container it is readable by everything the run executes, including build scripts and proc diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 8e1c8eb5..d4e54a0f 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -60,17 +60,16 @@ const RECIPE: &str = include_str!("../../../templates/justfiles/anvil/container. const DOCKERIGNORE: &str = include_str!("../../../templates/anvil/container/Dockerfile.dockerignore"); /// Seeded into the Dockerfile when the file does not exist, and never -/// reconciled afterwards — it is the one line anvil cannot own. +/// reconciled afterwards — it is the part of the file anvil cannot own. /// /// `# syntax=docker/dockerfile:1` pins the `BuildKit` frontend, and `BuildKit` /// honors the directive only as the very first line, before any comment. A /// region's opening sentinel *is* a comment, so the directive cannot live inside /// one without being silently demoted, dropping the build to the default -/// frontend with nothing failing to say so. Everything else in the file is a -/// region, so anvil can keep it current. +/// frontend with nothing failing to say so. The copyright notice follows it as +/// ordinary repository content, editable like anything else in a gap. pub(crate) const DOCKERFILE_HEADER: &str = include_str!("../../../templates/anvil/container/Dockerfile.header"); -const DOCKERFILE_HEADER_REGION: &str = include_str!("../../../templates/anvil/container/Dockerfile.header.region"); const DOCKERFILE_BASE_IMAGE: &str = include_str!("../../../templates/anvil/container/Dockerfile.baseimage.region"); const DOCKERFILE_BASE: &str = include_str!("../../../templates/anvil/container/Dockerfile.base.region"); const DOCKERFILE_TOOLS: &str = include_str!("../../../templates/anvil/container/Dockerfile.tools.region"); @@ -94,7 +93,6 @@ const DOCKERIGNORE_PATH: &str = ".anvil/container/Dockerfile.dockerignore"; /// against this list and refuses rather than emitting a Dockerfile that is /// silently wrong. pub(crate) const DOCKERFILE_REGION_ORDER: &[&str] = &[ - "anvil-container-header", "anvil-container-base-image", "anvil-container-base", "anvil-container-tools", @@ -111,7 +109,6 @@ pub fn all() -> Vec { vec![ recipe(), dockerignore(), - dockerfile_header(), dockerfile_base_image(), dockerfile_base(), dockerfile_tools(), @@ -135,17 +132,6 @@ fn dockerfile_region(id: &'static str, body: &'static str) -> Artifact { }) } -/// The file's explanatory header: what the regions are, which gap takes what, -/// and why the parser directive sits outside them. -/// -/// A region rather than part of the scaffold, so a correction to the guidance -/// reaches repositories that already generated the file. Scaffold content is -/// written once and never reconciled, which makes a mistake in it permanent. -#[must_use] -pub fn dockerfile_header() -> Artifact { - dockerfile_region("anvil-container-header", DOCKERFILE_HEADER_REGION) -} - /// The default base image, digest-pinned, alone in its own region. /// /// Separate from [`dockerfile_base`] so the gap between them is a place to @@ -274,8 +260,7 @@ mod tests { } /// Every region body, in the order the engine enforces. - const REGION_BODIES: [&str; 6] = [ - DOCKERFILE_HEADER_REGION, + const REGION_BODIES: [&str; 5] = [ DOCKERFILE_BASE_IMAGE, DOCKERFILE_BASE, DOCKERFILE_TOOLS, @@ -294,7 +279,7 @@ mod tests { } #[test] - fn group_is_two_owned_files_and_six_dockerfile_regions() { + fn group_is_two_owned_files_and_five_dockerfile_regions() { let all = all(); let owned: Vec<_> = all .iter() @@ -402,15 +387,20 @@ mod tests { } #[test] - fn only_the_parser_directive_is_left_outside_the_regions() { - // Anything anvil owns that sits outside a region can never be corrected - // on a repository that has already generated the file, because the - // scaffold is written once and never reconciled. Exactly one line has - // to pay that price. + fn the_scaffold_carries_no_instruction_that_could_go_stale() { + // Anything anvil owns outside a region can never be corrected on a + // repository that has already generated the file, because the scaffold + // is written once and never reconciled. The parser directive has to pay + // that price; nothing that pins or installs anything may join it. + let mut lines = DOCKERFILE_HEADER.lines(); assert_eq!( - DOCKERFILE_HEADER.lines().collect::>(), - ["# syntax=docker/dockerfile:1"], - "the scaffold must carry the parser directive and nothing else" + lines.next(), + Some("# syntax=docker/dockerfile:1"), + "the parser directive must lead the scaffold" + ); + assert!( + lines.all(|line| line.is_empty() || line.starts_with('#')), + "the scaffold must carry no build instruction: {DOCKERFILE_HEADER}" ); } @@ -676,13 +666,19 @@ mod tests { let walk = RECIPE .find("$containerRoot = Join-Path $repoRoot '.anvil/container'") .expect("the tag must walk the container directory"); - let recurse = RECIPE[walk..] - .find("Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force") - .expect("the walk must be recursive and include hidden entries"); + assert!( + RECIPE[walk..].contains("Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force"), + "the walk must be recursive and include hidden entries" + ); // A missing Dockerfile must still be fatal: the walk alone would let // it contribute nothing and yield a confident tag for an unbuildable - // image. - assert!(RECIPE[walk + recurse..].contains("container image input is missing: $dockerfile")); + // image. The resolver asserts existence, and the tag resolves the path + // through it before walking. + let resolve = RECIPE[..walk] + .rfind("_anvil-container-dockerfile\n") + .expect("the tag must resolve the Dockerfile before hashing the directory"); + assert!(resolve < walk); + assert!(RECIPE.contains("anvil: container image input is missing: .anvil/container/Dockerfile")); } #[test] @@ -805,12 +801,29 @@ mod tests { // anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with // a skip notice, so a title a native run rejects would pass in a // container and the tier would still report green. - for name in ["PR_TITLE", "BASE_REF", "GITHUB_BASE_REF", "SYSTEM_PULLREQUEST_TARGETBRANCH"] { - assert!(RECIPE.contains(name), "{name} must be forwarded"); - } - for name in ["ANVIL_INCLUDE_MODIFIED", "ANVIL_INCLUDE_AFFECTED", "ANVIL_INCLUDE_REQUIRED"] { + for name in [ + "PR_TITLE", + "BASE_REF", + "GITHUB_BASE_REF", + "SYSTEM_PULLREQUEST_TARGETBRANCH", + "ANVIL_IMPACT", + ] { assert!(RECIPE.contains(name), "{name} must be forwarded"); } + // ANVIL_IMPACT decides whether scoping is computed, consumed from a + // downloaded cache, or skipped. A CI group job exports `consume`, and a + // container that did not inherit it would recompute from a diff instead + // of trusting the artifact the group downloaded. + assert!( + RECIPE.contains("'ANVIL_IMPACT')) {"), + "ANVIL_IMPACT must be in the forwarded set, not merely mentioned" + ); + // Nothing reads these; forwarding them only implied a contract that + // does not exist. See justfile.rs, which asserts they stay removed. + assert!( + !RECIPE.contains("ANVIL_INCLUDE_"), + "the ANVIL_INCLUDE_* variables were removed and must not be forwarded" + ); } #[test] @@ -825,15 +838,13 @@ mod tests { let plan = RECIPE[..guard] .rfind("$plan -match 'GITHUB_TOKEN'") .expect("the plan must decide whether a token is needed"); - let dry_run = RECIPE[..plan] - .rfind("--dry-run @($argv | Select-Object -Skip 1)") - .expect("the plan must come from just"); + let dry_run = RECIPE[..plan].rfind("--dry-run @target").expect("the plan must come from just"); assert!(dry_run < plan && plan < guard, "compute the plan, match it, then derive"); // Through the launching binary, like every other nested call: a bare // `just` here fails silently when the caller invoked it by absolute // path, and an empty plan reads as "no token needed". assert!(!RECIPE.contains("(just --dry-run")); - assert!(RECIPE.contains(r"}}' --dry-run @($argv | Select-Object -Skip 1)")); + assert!(RECIPE.contains(r"}}' --dry-run @target")); // The predicate is the variable, not the name of a check, so a catalog // that adds another GitHub-authenticated check is covered for free. assert!(!RECIPE.contains("$plan -match 'aprz'")); @@ -843,6 +854,24 @@ mod tests { assert!(RECIPE.contains("if (-not $needsToken -and $argv[0] -eq 'just')")); } + #[test] + fn planning_follows_a_recipe_launched_as_a_child_process() { + // `just --dry-run` prints the bodies just runs itself. The unscoped tier + // wrapper runs its tier as a child process instead, so a plan of + // `anvil-scheduled` is the wrapper alone and reveals none of the checks + // under it -- including anvil-aprz, whose GITHUB_TOKEN is what stops it + // sleeping on the advisory API's unauthenticated rate limit. + assert!( + RECIPE.contains(r#"[regex]::Matches($step, "'(_anvil-[^'\s]+)'")"#), + "the plan must follow each nested target the wrapper names" + ); + // Bounded: a recipe reachable twice is planned once, and a body naming + // itself terminates instead of looping. + assert!(RECIPE.contains("if (-not $planned.Add(($target -join ' '))) { continue }")); + // Every step contributes, so a match anywhere in the tree counts. + assert!(RECIPE.contains("$plan = \"$plan`n$step\"")); + } + #[test] fn the_credential_phases_are_fail_closed() { // Unlike resolution, these must stop the run: a container that starts @@ -894,9 +923,9 @@ mod tests { } #[test] - fn a_linked_worktree_can_reach_its_git_directory() { - // A worktree's .git is a file naming a host path outside the mount, so - // without this the container resolves no refs at all and every check + fn a_redirected_checkout_can_reach_its_git_directory() { + // A checkout whose .git is a file names a host path outside the mount, + // so without this the container resolves no refs at all and every check // that needs history fails. assert!(RECIPE.contains("git rev-parse --git-common-dir")); assert!(RECIPE.contains("${engineGitCommon}:/anvil/gitdir")); @@ -905,18 +934,52 @@ mod tests { // container would inherit them and any git run outside the workspace // -- `git init` in a test's scratch directory, most of all -- would // operate on this repository instead of its own. - assert!(RECIPE.contains("gitdir: /anvil/gitdir/$rel")); + assert!(RECIPE.contains("gitdir: $containerGitDir")); assert!(RECIPE.contains("{{anvil_container_workdir}}/.git:ro")); assert!(!RECIPE.contains("GIT_DIR=")); assert!(!RECIPE.contains("GIT_WORK_TREE=")); // The generated file is temporary and must not outlive the run. assert!(RECIPE.contains("if ($gitFile) { Remove-Item -LiteralPath $gitFile")); - // An ordinary clone must not take the extra mount. - assert!(RECIPE.contains("if ($gitDirAbs -ne $gitCommonAbs) {")); + // An ordinary clone keeps its git directory inside the checkout, where + // the bind mount already carries it, and must not take the extra mount. + assert!(RECIPE.contains("(Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)")); + // The shape of .git, not whether the git directory differs from the + // common one: `--separate-git-dir` redirects without differing, so that + // predicate skipped it and left git pointed at a host path. + assert!( + !RECIPE.contains("if ($gitDirAbs -ne $gitCommonAbs) {"), + "a redirect without a separate worktree entry must still be mounted" + ); + // One mount carries both directories, so the git directory has to sit + // under the common one. Emitting a path that climbs out of the mount + // would fail inside the container, where the cause is invisible. + assert!(RECIPE.contains("if ($rel -eq '.') {")); + assert!(RECIPE.contains("$rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)")); // A host with a working engine but no git must not start failing here. assert!(RECIPE.contains("if (Get-Command git -ErrorAction SilentlyContinue) {")); } + #[test] + fn the_dockerfile_is_found_under_the_casing_on_disk() { + // Anvil resolves every path it manages against what the repository + // already has, so a checkout carrying `dockerfile` keeps that name and + // the regions are maintained there. A recipe hard-coding the canonical + // literal then names nothing on a case-sensitive filesystem. + assert!( + !RECIPE.contains("$dockerfile = '.anvil/container/Dockerfile'"), + "the path must be resolved, not assumed" + ); + assert!(RECIPE.contains("_anvil-container-dockerfile:")); + // -ceq, because PowerShell's -eq on strings is case-insensitive and + // would make the exact-match pass indistinguishable from the fallback. + assert!(RECIPE.contains("$_.Name -ceq 'Dockerfile'")); + assert!(RECIPE.contains("$_.Name -ieq 'Dockerfile'")); + // The one place that asserts the file exists: the tag's directory walk + // cannot, because a missing Dockerfile contributes nothing to the hash + // and yields a confident tag for an image that can never be built. + assert!(RECIPE.contains("anvil: container image input is missing: .anvil/container/Dockerfile")); + } + #[test] fn hooks_constructor_uses_the_documented_path() { assert_eq!(paths(&[hooks("# body\n")]), [HOOKS_PATH]); diff --git a/crates/cargo-anvil/src/plan.rs b/crates/cargo-anvil/src/plan.rs index 239a8184..dbbdea41 100644 --- a/crates/cargo-anvil/src/plan.rs +++ b/crates/cargo-anvil/src/plan.rs @@ -20,7 +20,7 @@ use std::fmt::Write as _; use std::path::{Path, PathBuf}; -use ohno::{AppError, IntoAppError as _}; +use ohno::{AppError, IntoAppError as _, bail}; use crate::decision::Decision; use crate::io::resolve_existing_case_insensitive; @@ -415,7 +415,7 @@ impl Plan { match (&item.target, item.decision) { (Target::File { path }, Decision::Write) => { let content = item.rendered.as_ref().expect("Write decision must carry rendered content"); - let abs = repo_root.join(path); + let abs = contained_path(repo_root, path)?; write_file(&abs, content)?; if let Some(checksum) = &item.rendered_checksum { next.files.insert(path.clone(), checksum.clone()); @@ -423,7 +423,7 @@ impl Plan { } (Target::File { path }, Decision::Propose) => { let content = item.rendered.as_ref().expect("Propose decision must carry rendered content"); - let abs = repo_root.join(format!("{path}.anvil-proposed")); + let abs = contained_path(repo_root, &format!("{path}.anvil-proposed"))?; write_file(&abs, content)?; if let Some(checksum) = &item.rendered_checksum { // Bump L to the new T so subsequent runs see the @@ -436,7 +436,7 @@ impl Plan { } (Target::Region { host, id }, Decision::Write) => { let spliced = item.spliced_host.as_ref().expect("region Write must carry spliced host"); - let abs = repo_root.join(host); + let abs = contained_path(repo_root, host)?; write_file(&abs, spliced)?; if let Some(checksum) = &item.rendered_checksum { next.regions.insert( @@ -450,7 +450,7 @@ impl Plan { } (Target::Region { host, id }, Decision::Propose) => { let spliced = item.spliced_host.as_ref().expect("region Propose must carry spliced host"); - let abs = repo_root.join(format!("{host}.anvil-proposed")); + let abs = contained_path(repo_root, &format!("{host}.anvil-proposed"))?; write_file(&abs, spliced)?; if let Some(checksum) = &item.rendered_checksum { // Same rationale as the File/Propose branch: bump @@ -474,7 +474,7 @@ impl Plan { // would leave the file behind with no lock entry. // If the file is already missing (race / external // delete), absorb the error so the result is idempotent. - let abs = repo_root.join(resolve_existing_case_insensitive(repo_root, path)); + let abs = contained_path(repo_root, &resolve_existing_case_insensitive(repo_root, path))?; if let Err(e) = std::fs::remove_file(&abs) && e.kind() != std::io::ErrorKind::NotFound { @@ -489,7 +489,7 @@ impl Plan { // write, for the reason the File arm above gives; the // manifest key stays as recorded so the entry is purged. let spliced = item.spliced_host.as_ref().expect("region Remove must carry spliced host"); - let abs = repo_root.join(resolve_existing_case_insensitive(repo_root, host)); + let abs = contained_path(repo_root, &resolve_existing_case_insensitive(repo_root, host))?; write_file(&abs, spliced)?; next.regions.remove(&RegionKey { host: host.clone(), @@ -555,6 +555,49 @@ fn write_section(out: &mut String, header: &str, items: &[&PlanItem]) { } } +/// Join a repository-relative path to `repo_root` and verify that it resolves +/// inside it. +/// +/// `Manifest::ensure_contained` rejects a path that escapes lexically, but a +/// path built entirely from ordinary components still lands outside the +/// repository when one of those components is a symlink pointing out of it. +/// The manifest is committed content, so a checkout can carry both the link and +/// the entry that names it, and the write, delete or proposal below would +/// follow it. +/// +/// Resolution stops at the deepest ancestor that exists, because the path +/// itself frequently does not: a file anvil is about to create has nothing on +/// disk to resolve. That is sufficient, since a symlink can only be a component +/// that exists. +/// +/// # Errors +/// +/// Returns an error if the repository root cannot be resolved, or if the path +/// resolves outside it. +fn contained_path(repo_root: &Path, relpath: &str) -> Result { + let abs = repo_root.join(relpath); + let root = repo_root + .canonicalize() + .into_app_err_with(|| format!("failed to resolve the repository root {}", repo_root.display()))?; + + let mut probe = abs.as_path(); + loop { + if let Ok(resolved) = probe.canonicalize() { + if !resolved.starts_with(&root) { + bail!( + "manifest path '{relpath}' resolves to {}, outside the repository at {}", + resolved.display(), + root.display() + ); + } + return Ok(abs); + } + probe = probe + .parent() + .expect("the walk reaches a filesystem root, which always resolves, before running out of components"); + } +} + fn write_file(path: &Path, content: &str) -> Result<(), AppError> { let parent = path .parent() @@ -579,6 +622,60 @@ mod tests { use super::*; + #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] + #[test] + fn a_path_inside_the_repository_is_joined_as_given() { + let tmp = TempDir::new().unwrap(); + std::fs::create_dir_all(tmp.path().join("a")).unwrap(); + std::fs::write(tmp.path().join("a/b.txt"), "x").unwrap(); + + assert_eq!(contained_path(tmp.path(), "a/b.txt").unwrap(), tmp.path().join("a/b.txt")); + // A file anvil is about to create has nothing on disk to resolve, so + // the walk has to fall back to the deepest ancestor that does. + assert_eq!( + contained_path(tmp.path(), "a/new/deeper/c.txt").unwrap(), + tmp.path().join("a/new/deeper/c.txt") + ); + } + + #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] + #[test] + fn a_path_that_resolves_outside_the_repository_is_refused() { + // The last line of defence, so it must hold on its own rather than + // assuming the manifest's lexical guard has already run. + let tmp = TempDir::new().unwrap(); + let root = tmp.path().join("repo"); + std::fs::create_dir_all(&root).unwrap(); + std::fs::create_dir_all(tmp.path().join("outside")).unwrap(); + + let err = contained_path(&root, "../outside").unwrap_err(); + assert!( + format!("{err}").contains("outside the repository"), + "expected a containment error, got: {err}" + ); + } + + #[cfg(unix)] + #[cfg_attr(miri, ignore = "uses filesystem; miri isolation forbids it")] + #[test] + fn a_symlinked_component_cannot_carry_a_write_out_of_the_repository() { + // The manifest is committed content, so one commit can add both a link + // that leaves the tree and an entry naming a path through it. Every + // component here is ordinary, so the lexical guard passes it. + let tmp = TempDir::new().unwrap(); + let root = tmp.path().join("repo"); + let outside = tmp.path().join("outside"); + std::fs::create_dir_all(&root).unwrap(); + std::fs::create_dir_all(&outside).unwrap(); + std::os::unix::fs::symlink(&outside, root.join("escape")).unwrap(); + + let err = contained_path(&root, "escape/victim.txt").unwrap_err(); + assert!( + format!("{err}").contains("outside the repository"), + "expected a containment error, got: {err}" + ); + } + #[test] fn empty_plan_is_in_sync() { let plan = Plan::default(); diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region index dac79061..8cc7a8b3 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.baseimage.region @@ -1,8 +1,8 @@ -# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). -# `anvil-setup binstall` installs prebuilt binaries that need that runner's -# glibc, so this moves when the runner does. Digest-pinned because a floating -# tag can change under an image reference that claims to name fixed content. +# Prebuilt binaries installed by `anvil-setup binstall` link against this +# image's glibc, so it tracks the Linux runner the generated workflows use. +# Digest-pinned: a floating tag moves content under a reference that claims to +# name fixed content. # -# To build on another base, re-declare this in the gap below rather than editing -# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +# Re-declare BASE_IMAGE in the gap below to build on another base; a later ARG +# wins, and the pins anvil maintains stay current. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header index d6a937b3..d063d715 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header @@ -1 +1,4 @@ # syntax=docker/dockerfile:1 + +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region deleted file mode 100644 index eb403125..00000000 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.header.region +++ /dev/null @@ -1,19 +0,0 @@ -# Copyright (c) Microsoft Corporation. -# Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps -# between them are yours and are preserved. Add to a gap, not inside a region: -# anvil will not overwrite an edit inside one, so editing there silently freezes -# the base digest and the tool pins at that moment. -# -# Which gap, since position decides whether a line works at all: -# -# after base-image re-declare ARG BASE_IMAGE to build on your own base -# after base root CA, proxy, apt mirror -- anything the first download needs -# after tools libraries a catalog tool needs to compile from source -# after setup what your own checks need at run time -# -# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, -# and a region sentinel is a comment. diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region index 4deca8f6..d3306a1e 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.setup.region @@ -1,20 +1,14 @@ -# Install the pinned toolchain and cargo subcommands from the generated catalog. -# The whole recipe tree is copied because `just` has to parse it, and the whole -# tree is hashed into the image tag: `anvil-setup` reaches the install recipes -# through the tier, group and check recipes, so any of them can change what this -# layer installs. +# The whole recipe tree is copied because `just` parses it to reach the install +# recipes. # -# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do -# not compile from source on every pinned toolchain. -# -# The credential files are removed in the same layer as the install. A build +# The credential files are removed in the same layer that used them: a build # secret never lands in a layer, but anything the install *writes* with it is -# ordinary content, and the `chmod` below would publish it world-readable. -# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. +# ordinary content, and the `chmod` below would publish it world-readable. A +# later `RUN` cannot undo that, because the earlier layer keeps them. # -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each, and an engine seeds a new volume from the image path -# it covers. A path that does not exist seeds as root-owned 0755, which the +# `registry` and `git` must exist before the `chmod`. The run mounts a named +# volume over each, and an engine seeds a new volume from the image path it +# covers; a path that does not exist seeds as root-owned 0755, which the # `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles diff --git a/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region index 639b3a71..dd3e315c 100644 --- a/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region +++ b/crates/cargo-anvil/templates/anvil/container/Dockerfile.tools.region @@ -32,10 +32,6 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. RUN curl -fsSLo /tmp/cargo-binstall.tgz \ "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index fcc1e3b1..8f8484a9 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -106,6 +106,33 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() +# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# +# Anvil resolves every path it manages against what the repository already has, +# so a checkout that carries `dockerfile` keeps that name and the regions are +# maintained there. On a case-sensitive filesystem the canonical literal then +# names nothing, and the build fails on a file the generator is maintaining. +# +# Also the one place that asserts the file exists: the tag's directory walk +# cannot, because a missing Dockerfile simply contributes nothing to the hash +# and yields a confident tag for an image that can never be built. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-dockerfile: + $ErrorActionPreference = 'Stop' + $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' + $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) + # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. + $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] + if (-not $match) { + $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + } + if (-not $match) { + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 + } + Write-Output ".anvil/container/$($match.Name)" + # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its @@ -127,7 +154,9 @@ _anvil-container-path host_path: anvil-container-tag: $ErrorActionPreference = 'Stop' $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') @@ -166,14 +195,6 @@ anvil-container-tag: $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } - # The walk cannot assert this on its own: a missing Dockerfile would simply - # contribute nothing and yield a confident tag for an image that can never - # be built. It is the one file under that directory that must exist, so it - # is checked by name. - if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { - Write-Error "anvil: container image input is missing: $dockerfile" - exit 1 - } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, # group and check recipes before it reaches the install recipes in @@ -302,7 +323,9 @@ _anvil-container-image: $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag @@ -571,12 +594,18 @@ anvil-container *command: $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # A linked worktree keeps its real git directory outside the checkout: its - # `.git` is a file naming an absolute host path, which does not exist inside - # the container, so git resolves neither HEAD nor origin/main. Mount the - # common git directory and replace the checkout's `.git` file with one - # naming that mount; the `commondir` file in the worktree's entry is - # relative, so it resolves under it. + # A checkout whose `.git` is a file keeps its real git directory elsewhere: + # a linked worktree points into the main clone, `--separate-git-dir` and a + # submodule point somewhere else again. The path recorded there is a host + # path that does not exist inside the container, so git resolves neither + # HEAD nor origin/main. Mount the common git directory and replace the + # checkout's `.git` with one naming that mount; the `commondir` file in a + # worktree's entry is relative, so it resolves under it. + # + # The predicate is the shape of `.git`, not whether the git directory + # differs from the common one: `--separate-git-dir` redirects without + # differing, and testing for a difference skips it and leaves git pointed at + # a path the container cannot see. # # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE # so that it stays scoped to it. Those variables are ambient: every process @@ -594,23 +623,35 @@ anvil-container *command: if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null - if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon -and + (Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)) { $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path - if ($gitDirAbs -ne $gitCommonAbs) { - $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitCommon = "$engineGitCommon".Trim() - # LF and no trailing newline: git parses this file strictly. - $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" - [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitFile = "$engineGitFile".Trim() - $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + # One mount has to carry both directories, so the git directory must + # sit under the common one. `git worktree` always places it there and + # a redirect without a separate worktree entry makes the two equal; + # anything else cannot be expressed as a single mount, and emitting a + # path that climbs out of it would fail inside the container instead. + if ($rel -eq '.') { + $containerGitDir = '/anvil/gitdir' + } elseif ($rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)) { + Write-Error "anvil: this checkout's git directory ($gitDirAbs) is not inside its common git directory ($gitCommonAbs), so the two cannot be mounted as one tree. Run the container from an ordinary clone or a git worktree checkout." + exit 1 + } else { + $containerGitDir = "/anvil/gitdir/$rel" } + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: $containerGitDir`n") + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } @@ -688,18 +729,41 @@ anvil-container *command: # A dry run has no side effects, and a target that cannot be planned # (a typo, a recipe needing arguments) yields nothing, so the run # fails on its own terms rather than on a missing token. + # + # A plan covers the bodies just runs itself, not the body of a + # recipe that one of them launches as a child process. The unscoped + # tier wrapper launches its tier that way, so planning + # `anvil-scheduled` shows the wrapper and none of the checks + # underneath it. Follow each nested target a plan names, or a + # wrapped tier reads as needing nothing and runs unauthenticated. $plan = '' - # The same executable that launched this tree, for the reason every - # other nested call uses it: a caller invoking `just` by absolute - # path with its directory off PATH would otherwise fail here. That - # failure is silent, because an empty plan reads as "does not need a - # token" -- so anvil-aprz would run unauthenticated in an image with - # no gh of its own and block on the rate limit for up to an hour. - try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | - ForEach-Object { $_.ToString() }) -join "`n" - } catch { - $plan = '' + $targets = [System.Collections.Generic.List[object]]::new() + $targets.Add([string[]]@($argv | Select-Object -Skip 1)) + $planned = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + for ($i = 0; $i -lt $targets.Count; $i++) { + $target = [string[]]$targets[$i] + # A recipe reachable twice is planned once, and a body naming + # itself terminates. + if (-not $planned.Add(($target -join ' '))) { continue } + # The same executable that launched this tree, for the reason + # every other nested call uses it: a caller invoking `just` by + # absolute path with its directory off PATH would otherwise fail + # here. That failure is silent, because an empty plan reads as + # "does not need a token" -- so anvil-aprz would run + # unauthenticated in an image with no gh of its own and block on + # the rate limit for up to an hour. + $step = '' + try { + $step = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @target 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $step = '' + } + $plan = "$plan`n$step" + # A launched recipe appears as a quoted argument to `just`. + foreach ($nested in [regex]::Matches($step, "'(_anvil-[^'\s]+)'")) { + $targets.Add([string[]]@($nested.Groups[1].Value)) + } } $needsToken = $plan -match 'GITHUB_TOKEN' } @@ -719,11 +783,14 @@ anvil-container *command: # The recipe contract's own inputs. These are read by generated checks -- # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its - # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so - # dropping them at the boundary makes the same command mean different things - # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it - # exits 0 with a skip notice, so a title a native run rejects passes in a - # container and the tier still reports green. + # CI equivalents, and `anvil-impact` reads ANVIL_IMPACT to decide whether to + # compute scoping, consume a downloaded cache, or skip -- so dropping them at + # the boundary makes the same command mean different things inside and out. + # anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with a + # skip notice, so a title a native run rejects passes in a container and the + # tier still reports green. ANVIL_IMPACT is the other: a CI group job exports + # `consume`, and a container that did not inherit it would recompute scoping + # from a diff instead of trusting the artifact the group downloaded. # # Forwarded by name and only when set, so an unset variable stays unset # rather than arriving as an empty string, which several of these treat as @@ -731,7 +798,7 @@ anvil-container *command: foreach ($name in @( 'PR_TITLE', 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', - 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + 'ANVIL_IMPACT')) { if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { $forwardedEnv += $name $runArgs += @('-e', $name) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 016d2aa8..d47d167e 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -27,6 +27,7 @@ const TOOLS: &str = include_str!("../templates/justfiles/anvil/tools.just"); const APRZ: &str = include_str!("../templates/justfiles/anvil/checks/aprz.just"); const MUTANTS_DIFF: &str = include_str!("../templates/justfiles/anvil/checks/mutants-diff.just"); const VERSIONS: &str = include_str!("../templates/justfiles/anvil/versions.just"); +const CONTAINER: &str = include_str!("../templates/justfiles/anvil/container.just"); const FAKE_CARGO_PS1: &str = r#" $joined = $args -join ' ' if ($env:FAKE_CARGO_LOG) { @@ -1114,3 +1115,108 @@ fn unscoped_wrapper_exports_impact_off_before_dependencies_run() { String::from_utf8_lossy(&direct.stderr) ); } + +/// `just --dry-run` reports the bodies just runs itself, not the body of a +/// recipe that one of them launches as a child process. The unscoped wrapper +/// launches its tier that way, so a plan of the public tier name reveals the +/// wrapper alone. +/// +/// The container driver decides whether to mint a GitHub token by matching the +/// plan for `GITHUB_TOKEN`, so this is why it has to follow each nested target +/// rather than reading one plan. If this test ever fails because a plan now +/// reaches through the child process, that expansion can be deleted. +#[test] +fn a_wrapped_tier_hides_its_checks_from_a_plan() { + const PROBE: &str = "[private]\n[script(\"pwsh\", \"-NoProfile\")]\n_anvil-probe:\n \ + if (-not $env:GITHUB_TOKEN) { exit 1 }\n\n\ + probe: (_anvil-unscoped \"probe\")\n"; + + if !tools_available() { + return; + } + let tmp = fixture(&[("helpers.just", HELPERS), ("probe.just", PROBE)], &[]); + let root = tmp.path(); + + let wrapped = run_just(root, &["--dry-run", "probe"], &[]); + let wrapped_plan = format!( + "{}{}", + String::from_utf8_lossy(&wrapped.stdout), + String::from_utf8_lossy(&wrapped.stderr) + ); + assert!( + !wrapped_plan.contains("GITHUB_TOKEN"), + "a wrapped tier's plan must not reach the recipe it launches, or the driver's expansion is dead code\n{wrapped_plan}" + ); + assert!( + wrapped_plan.contains("_anvil-probe"), + "the wrapper must still name the recipe it launches, which is what the driver follows\n{wrapped_plan}" + ); + + let direct = run_just(root, &["--dry-run", "_anvil-probe"], &[]); + let direct_plan = format!( + "{}{}", + String::from_utf8_lossy(&direct.stdout), + String::from_utf8_lossy(&direct.stderr) + ); + assert!( + direct_plan.contains("GITHUB_TOKEN"), + "planning the launched recipe directly must reveal the variable, or this test proves nothing\n{direct_plan}" + ); +} + +/// Anvil resolves every path it manages against what the repository already +/// carries, so a checkout holding `dockerfile` keeps that name and the regions +/// are maintained inside it. The container recipes have to agree, or on a +/// case-sensitive filesystem the build names a file that does not exist. +#[test] +fn the_container_dockerfile_resolves_to_its_on_disk_casing() { + if !tools_available() { + return; + } + + for name in ["Dockerfile", "dockerfile", "DOCKERFILE"] { + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join(".anvil/container").join(name), "FROM scratch\n"); + + let output = run_just(root, &["_anvil-container-dockerfile"], &[]); + assert!( + output.status.success(), + "resolving {name} failed\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert_eq!( + String::from_utf8_lossy(&output.stdout).trim(), + format!(".anvil/container/{name}"), + "the resolved path must carry the casing on disk" + ); + } + + // Absent, the tag would hash a directory that contributes nothing for it + // and hand back a confident reference to an image that cannot be built. + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let missing = run_just(tmp.path(), &["_anvil-container-dockerfile"], &[]); + assert_failed(&missing, "resolving an absent Dockerfile"); + assert!( + String::from_utf8_lossy(&missing.stderr).contains("container image input is missing"), + "the failure must name the missing input\nstderr:\n{}", + String::from_utf8_lossy(&missing.stderr) + ); + + // Exact case wins over a fold. Only a case-sensitive filesystem can hold + // both spellings at once, so this is the one assertion that cannot run + // everywhere; on Windows the two names are the same file. + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join(".anvil/container/dockerfile"), "FROM scratch\n"); + write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); + if fs::read_dir(root.join(".anvil/container")).unwrap().count() == 2 { + let output = run_just(root, &["_anvil-container-dockerfile"], &[]); + assert_eq!( + String::from_utf8_lossy(&output.stdout).trim(), + ".anvil/container/Dockerfile", + "an exact-case match must win over a case-insensitive one" + ); + } +} \ No newline at end of file diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 6c47bf3e..762c810c 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -5,36 +5,17 @@ expression: render_tree(tmp.path()) === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps -# between them are yours and are preserved. Add to a gap, not inside a region: -# anvil will not overwrite an edit inside one, so editing there silently freezes -# the base digest and the tool pins at that moment. -# -# Which gap, since position decides whether a line works at all: -# -# after base-image re-declare ARG BASE_IMAGE to build on your own base -# after base root CA, proxy, apt mirror -- anything the first download needs -# after tools libraries a catalog tool needs to compile from source -# after setup what your own checks need at run time -# -# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, -# and a region sentinel is a comment. -# <<< anvil-managed: anvil-container-header # >>> anvil-managed: anvil-container-base-image -# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). -# `anvil-setup binstall` installs prebuilt binaries that need that runner's -# glibc, so this moves when the runner does. Digest-pinned because a floating -# tag can change under an image reference that claims to name fixed content. +# Prebuilt binaries installed by `anvil-setup binstall` link against this +# image's glibc, so it tracks the Linux runner the generated workflows use. +# Digest-pinned: a floating tag moves content under a reference that claims to +# name fixed content. # -# To build on another base, re-declare this in the gap below rather than editing -# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +# Re-declare BASE_IMAGE in the gap below to build on another base; a later ARG +# wins, and the pins anvil maintains stay current. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea # <<< anvil-managed: anvil-container-base-image @@ -92,10 +73,6 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. RUN curl -fsSLo /tmp/cargo-binstall.tgz \ "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ @@ -107,23 +84,17 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated catalog. -# The whole recipe tree is copied because `just` has to parse it, and the whole -# tree is hashed into the image tag: `anvil-setup` reaches the install recipes -# through the tier, group and check recipes, so any of them can change what this -# layer installs. -# -# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do -# not compile from source on every pinned toolchain. +# The whole recipe tree is copied because `just` parses it to reach the install +# recipes. # -# The credential files are removed in the same layer as the install. A build +# The credential files are removed in the same layer that used them: a build # secret never lands in a layer, but anything the install *writes* with it is -# ordinary content, and the `chmod` below would publish it world-readable. -# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. +# ordinary content, and the `chmod` below would publish it world-readable. A +# later `RUN` cannot undo that, because the earlier layer keeps them. # -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each, and an engine seeds a new volume from the image path -# it covers. A path that does not exist seeds as root-owned 0755, which the +# `registry` and `git` must exist before the `chmod`. The run mounts a named +# volume over each, and an engine seeds a new volume from the image path it +# covers; a path that does not exist seeds as root-owned 0755, which the # `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles @@ -3664,6 +3635,33 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() +# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# +# Anvil resolves every path it manages against what the repository already has, +# so a checkout that carries `dockerfile` keeps that name and the regions are +# maintained there. On a case-sensitive filesystem the canonical literal then +# names nothing, and the build fails on a file the generator is maintaining. +# +# Also the one place that asserts the file exists: the tag's directory walk +# cannot, because a missing Dockerfile simply contributes nothing to the hash +# and yields a confident tag for an image that can never be built. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-dockerfile: + $ErrorActionPreference = 'Stop' + $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' + $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) + # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. + $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] + if (-not $match) { + $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + } + if (-not $match) { + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 + } + Write-Output ".anvil/container/$($match.Name)" + # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its @@ -3685,7 +3683,9 @@ _anvil-container-path host_path: anvil-container-tag: $ErrorActionPreference = 'Stop' $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') @@ -3724,14 +3724,6 @@ anvil-container-tag: $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } - # The walk cannot assert this on its own: a missing Dockerfile would simply - # contribute nothing and yield a confident tag for an image that can never - # be built. It is the one file under that directory that must exist, so it - # is checked by name. - if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { - Write-Error "anvil: container image input is missing: $dockerfile" - exit 1 - } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, # group and check recipes before it reaches the install recipes in @@ -3860,7 +3852,9 @@ _anvil-container-image: $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag @@ -4129,12 +4123,18 @@ anvil-container *command: $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # A linked worktree keeps its real git directory outside the checkout: its - # `.git` is a file naming an absolute host path, which does not exist inside - # the container, so git resolves neither HEAD nor origin/main. Mount the - # common git directory and replace the checkout's `.git` file with one - # naming that mount; the `commondir` file in the worktree's entry is - # relative, so it resolves under it. + # A checkout whose `.git` is a file keeps its real git directory elsewhere: + # a linked worktree points into the main clone, `--separate-git-dir` and a + # submodule point somewhere else again. The path recorded there is a host + # path that does not exist inside the container, so git resolves neither + # HEAD nor origin/main. Mount the common git directory and replace the + # checkout's `.git` with one naming that mount; the `commondir` file in a + # worktree's entry is relative, so it resolves under it. + # + # The predicate is the shape of `.git`, not whether the git directory + # differs from the common one: `--separate-git-dir` redirects without + # differing, and testing for a difference skips it and leaves git pointed at + # a path the container cannot see. # # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE # so that it stays scoped to it. Those variables are ambient: every process @@ -4152,23 +4152,35 @@ anvil-container *command: if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null - if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon -and + (Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)) { $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path - if ($gitDirAbs -ne $gitCommonAbs) { - $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitCommon = "$engineGitCommon".Trim() - # LF and no trailing newline: git parses this file strictly. - $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" - [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitFile = "$engineGitFile".Trim() - $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + # One mount has to carry both directories, so the git directory must + # sit under the common one. `git worktree` always places it there and + # a redirect without a separate worktree entry makes the two equal; + # anything else cannot be expressed as a single mount, and emitting a + # path that climbs out of it would fail inside the container instead. + if ($rel -eq '.') { + $containerGitDir = '/anvil/gitdir' + } elseif ($rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)) { + Write-Error "anvil: this checkout's git directory ($gitDirAbs) is not inside its common git directory ($gitCommonAbs), so the two cannot be mounted as one tree. Run the container from an ordinary clone or a git worktree checkout." + exit 1 + } else { + $containerGitDir = "/anvil/gitdir/$rel" } + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: $containerGitDir`n") + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } @@ -4246,18 +4258,41 @@ anvil-container *command: # A dry run has no side effects, and a target that cannot be planned # (a typo, a recipe needing arguments) yields nothing, so the run # fails on its own terms rather than on a missing token. + # + # A plan covers the bodies just runs itself, not the body of a + # recipe that one of them launches as a child process. The unscoped + # tier wrapper launches its tier that way, so planning + # `anvil-scheduled` shows the wrapper and none of the checks + # underneath it. Follow each nested target a plan names, or a + # wrapped tier reads as needing nothing and runs unauthenticated. $plan = '' - # The same executable that launched this tree, for the reason every - # other nested call uses it: a caller invoking `just` by absolute - # path with its directory off PATH would otherwise fail here. That - # failure is silent, because an empty plan reads as "does not need a - # token" -- so anvil-aprz would run unauthenticated in an image with - # no gh of its own and block on the rate limit for up to an hour. - try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | - ForEach-Object { $_.ToString() }) -join "`n" - } catch { - $plan = '' + $targets = [System.Collections.Generic.List[object]]::new() + $targets.Add([string[]]@($argv | Select-Object -Skip 1)) + $planned = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + for ($i = 0; $i -lt $targets.Count; $i++) { + $target = [string[]]$targets[$i] + # A recipe reachable twice is planned once, and a body naming + # itself terminates. + if (-not $planned.Add(($target -join ' '))) { continue } + # The same executable that launched this tree, for the reason + # every other nested call uses it: a caller invoking `just` by + # absolute path with its directory off PATH would otherwise fail + # here. That failure is silent, because an empty plan reads as + # "does not need a token" -- so anvil-aprz would run + # unauthenticated in an image with no gh of its own and block on + # the rate limit for up to an hour. + $step = '' + try { + $step = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @target 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $step = '' + } + $plan = "$plan`n$step" + # A launched recipe appears as a quoted argument to `just`. + foreach ($nested in [regex]::Matches($step, "'(_anvil-[^'\s]+)'")) { + $targets.Add([string[]]@($nested.Groups[1].Value)) + } } $needsToken = $plan -match 'GITHUB_TOKEN' } @@ -4277,11 +4312,14 @@ anvil-container *command: # The recipe contract's own inputs. These are read by generated checks -- # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its - # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so - # dropping them at the boundary makes the same command mean different things - # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it - # exits 0 with a skip notice, so a title a native run rejects passes in a - # container and the tier still reports green. + # CI equivalents, and `anvil-impact` reads ANVIL_IMPACT to decide whether to + # compute scoping, consume a downloaded cache, or skip -- so dropping them at + # the boundary makes the same command mean different things inside and out. + # anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with a + # skip notice, so a title a native run rejects passes in a container and the + # tier still reports green. ANVIL_IMPACT is the other: a CI group job exports + # `consume`, and a container that did not inherit it would recompute scoping + # from a diff instead of trusting the artifact the group downloaded. # # Forwarded by name and only when set, so an unset variable stays unset # rather than arriving as an empty string, which several of these treat as @@ -4289,7 +4327,7 @@ anvil-container *command: foreach ($name in @( 'PR_TITLE', 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', - 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + 'ANVIL_IMPACT')) { if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { $forwardedEnv += $name $runArgs += @('-e', $name) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index ca26814d..3ff6e8c8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -5,36 +5,17 @@ expression: render_tree(tmp.path()) === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps -# between them are yours and are preserved. Add to a gap, not inside a region: -# anvil will not overwrite an edit inside one, so editing there silently freezes -# the base digest and the tool pins at that moment. -# -# Which gap, since position decides whether a line works at all: -# -# after base-image re-declare ARG BASE_IMAGE to build on your own base -# after base root CA, proxy, apt mirror -- anything the first download needs -# after tools libraries a catalog tool needs to compile from source -# after setup what your own checks need at run time -# -# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, -# and a region sentinel is a comment. -# <<< anvil-managed: anvil-container-header # >>> anvil-managed: anvil-container-base-image -# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). -# `anvil-setup binstall` installs prebuilt binaries that need that runner's -# glibc, so this moves when the runner does. Digest-pinned because a floating -# tag can change under an image reference that claims to name fixed content. +# Prebuilt binaries installed by `anvil-setup binstall` link against this +# image's glibc, so it tracks the Linux runner the generated workflows use. +# Digest-pinned: a floating tag moves content under a reference that claims to +# name fixed content. # -# To build on another base, re-declare this in the gap below rather than editing -# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +# Re-declare BASE_IMAGE in the gap below to build on another base; a later ARG +# wins, and the pins anvil maintains stay current. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea # <<< anvil-managed: anvil-container-base-image @@ -92,10 +73,6 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. RUN curl -fsSLo /tmp/cargo-binstall.tgz \ "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ @@ -107,23 +84,17 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated catalog. -# The whole recipe tree is copied because `just` has to parse it, and the whole -# tree is hashed into the image tag: `anvil-setup` reaches the install recipes -# through the tier, group and check recipes, so any of them can change what this -# layer installs. -# -# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do -# not compile from source on every pinned toolchain. +# The whole recipe tree is copied because `just` parses it to reach the install +# recipes. # -# The credential files are removed in the same layer as the install. A build +# The credential files are removed in the same layer that used them: a build # secret never lands in a layer, but anything the install *writes* with it is -# ordinary content, and the `chmod` below would publish it world-readable. -# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. +# ordinary content, and the `chmod` below would publish it world-readable. A +# later `RUN` cannot undo that, because the earlier layer keeps them. # -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each, and an engine seeds a new volume from the image path -# it covers. A path that does not exist seeds as root-owned 0755, which the +# `registry` and `git` must exist before the `chmod`. The run mounts a named +# volume over each, and an engine seeds a new volume from the image path it +# covers; a path that does not exist seeds as root-owned 0755, which the # `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles @@ -3543,6 +3514,33 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() +# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# +# Anvil resolves every path it manages against what the repository already has, +# so a checkout that carries `dockerfile` keeps that name and the regions are +# maintained there. On a case-sensitive filesystem the canonical literal then +# names nothing, and the build fails on a file the generator is maintaining. +# +# Also the one place that asserts the file exists: the tag's directory walk +# cannot, because a missing Dockerfile simply contributes nothing to the hash +# and yields a confident tag for an image that can never be built. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-dockerfile: + $ErrorActionPreference = 'Stop' + $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' + $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) + # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. + $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] + if (-not $match) { + $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + } + if (-not $match) { + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 + } + Write-Output ".anvil/container/$($match.Name)" + # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its @@ -3564,7 +3562,9 @@ _anvil-container-path host_path: anvil-container-tag: $ErrorActionPreference = 'Stop' $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') @@ -3603,14 +3603,6 @@ anvil-container-tag: $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } - # The walk cannot assert this on its own: a missing Dockerfile would simply - # contribute nothing and yield a confident tag for an image that can never - # be built. It is the one file under that directory that must exist, so it - # is checked by name. - if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { - Write-Error "anvil: container image input is missing: $dockerfile" - exit 1 - } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, # group and check recipes before it reaches the install recipes in @@ -3739,7 +3731,9 @@ _anvil-container-image: $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag @@ -4008,12 +4002,18 @@ anvil-container *command: $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # A linked worktree keeps its real git directory outside the checkout: its - # `.git` is a file naming an absolute host path, which does not exist inside - # the container, so git resolves neither HEAD nor origin/main. Mount the - # common git directory and replace the checkout's `.git` file with one - # naming that mount; the `commondir` file in the worktree's entry is - # relative, so it resolves under it. + # A checkout whose `.git` is a file keeps its real git directory elsewhere: + # a linked worktree points into the main clone, `--separate-git-dir` and a + # submodule point somewhere else again. The path recorded there is a host + # path that does not exist inside the container, so git resolves neither + # HEAD nor origin/main. Mount the common git directory and replace the + # checkout's `.git` with one naming that mount; the `commondir` file in a + # worktree's entry is relative, so it resolves under it. + # + # The predicate is the shape of `.git`, not whether the git directory + # differs from the common one: `--separate-git-dir` redirects without + # differing, and testing for a difference skips it and leaves git pointed at + # a path the container cannot see. # # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE # so that it stays scoped to it. Those variables are ambient: every process @@ -4031,23 +4031,35 @@ anvil-container *command: if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null - if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon -and + (Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)) { $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path - if ($gitDirAbs -ne $gitCommonAbs) { - $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitCommon = "$engineGitCommon".Trim() - # LF and no trailing newline: git parses this file strictly. - $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" - [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitFile = "$engineGitFile".Trim() - $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + # One mount has to carry both directories, so the git directory must + # sit under the common one. `git worktree` always places it there and + # a redirect without a separate worktree entry makes the two equal; + # anything else cannot be expressed as a single mount, and emitting a + # path that climbs out of it would fail inside the container instead. + if ($rel -eq '.') { + $containerGitDir = '/anvil/gitdir' + } elseif ($rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)) { + Write-Error "anvil: this checkout's git directory ($gitDirAbs) is not inside its common git directory ($gitCommonAbs), so the two cannot be mounted as one tree. Run the container from an ordinary clone or a git worktree checkout." + exit 1 + } else { + $containerGitDir = "/anvil/gitdir/$rel" } + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: $containerGitDir`n") + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } @@ -4125,18 +4137,41 @@ anvil-container *command: # A dry run has no side effects, and a target that cannot be planned # (a typo, a recipe needing arguments) yields nothing, so the run # fails on its own terms rather than on a missing token. + # + # A plan covers the bodies just runs itself, not the body of a + # recipe that one of them launches as a child process. The unscoped + # tier wrapper launches its tier that way, so planning + # `anvil-scheduled` shows the wrapper and none of the checks + # underneath it. Follow each nested target a plan names, or a + # wrapped tier reads as needing nothing and runs unauthenticated. $plan = '' - # The same executable that launched this tree, for the reason every - # other nested call uses it: a caller invoking `just` by absolute - # path with its directory off PATH would otherwise fail here. That - # failure is silent, because an empty plan reads as "does not need a - # token" -- so anvil-aprz would run unauthenticated in an image with - # no gh of its own and block on the rate limit for up to an hour. - try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | - ForEach-Object { $_.ToString() }) -join "`n" - } catch { - $plan = '' + $targets = [System.Collections.Generic.List[object]]::new() + $targets.Add([string[]]@($argv | Select-Object -Skip 1)) + $planned = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + for ($i = 0; $i -lt $targets.Count; $i++) { + $target = [string[]]$targets[$i] + # A recipe reachable twice is planned once, and a body naming + # itself terminates. + if (-not $planned.Add(($target -join ' '))) { continue } + # The same executable that launched this tree, for the reason + # every other nested call uses it: a caller invoking `just` by + # absolute path with its directory off PATH would otherwise fail + # here. That failure is silent, because an empty plan reads as + # "does not need a token" -- so anvil-aprz would run + # unauthenticated in an image with no gh of its own and block on + # the rate limit for up to an hour. + $step = '' + try { + $step = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @target 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $step = '' + } + $plan = "$plan`n$step" + # A launched recipe appears as a quoted argument to `just`. + foreach ($nested in [regex]::Matches($step, "'(_anvil-[^'\s]+)'")) { + $targets.Add([string[]]@($nested.Groups[1].Value)) + } } $needsToken = $plan -match 'GITHUB_TOKEN' } @@ -4156,11 +4191,14 @@ anvil-container *command: # The recipe contract's own inputs. These are read by generated checks -- # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its - # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so - # dropping them at the boundary makes the same command mean different things - # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it - # exits 0 with a skip notice, so a title a native run rejects passes in a - # container and the tier still reports green. + # CI equivalents, and `anvil-impact` reads ANVIL_IMPACT to decide whether to + # compute scoping, consume a downloaded cache, or skip -- so dropping them at + # the boundary makes the same command mean different things inside and out. + # anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with a + # skip notice, so a title a native run rejects passes in a container and the + # tier still reports green. ANVIL_IMPACT is the other: a CI group job exports + # `consume`, and a container that did not inherit it would recompute scoping + # from a diff instead of trusting the artifact the group downloaded. # # Forwarded by name and only when set, so an unset variable stays unset # rather than arriving as an empty string, which several of these treat as @@ -4168,7 +4206,7 @@ anvil-container *command: foreach ($name in @( 'PR_TITLE', 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', - 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + 'ANVIL_IMPACT')) { if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { $forwardedEnv += $name $runArgs += @('-e', $name) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 78da812c..89406f94 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -5,36 +5,17 @@ expression: render_tree(tmp.path()) === .anvil/container/Dockerfile === # syntax=docker/dockerfile:1 -# >>> anvil-managed: anvil-container-header # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -# -# Anvil execution image. -# -# The `anvil-managed:` regions are cargo-anvil's and are kept current. The gaps -# between them are yours and are preserved. Add to a gap, not inside a region: -# anvil will not overwrite an edit inside one, so editing there silently freezes -# the base digest and the tool pins at that moment. -# -# Which gap, since position decides whether a line works at all: -# -# after base-image re-declare ARG BASE_IMAGE to build on your own base -# after base root CA, proxy, apt mirror -- anything the first download needs -# after tools libraries a catalog tool needs to compile from source -# after setup what your own checks need at run time -# -# `# syntax=` stays on line 1: BuildKit honors it only when nothing precedes it, -# and a region sentinel is a comment. -# <<< anvil-managed: anvil-container-header # >>> anvil-managed: anvil-container-base-image -# Tracks the Linux runner the generated workflows use (`ubuntu-latest`). -# `anvil-setup binstall` installs prebuilt binaries that need that runner's -# glibc, so this moves when the runner does. Digest-pinned because a floating -# tag can change under an image reference that claims to name fixed content. +# Prebuilt binaries installed by `anvil-setup binstall` link against this +# image's glibc, so it tracks the Linux runner the generated workflows use. +# Digest-pinned: a floating tag moves content under a reference that claims to +# name fixed content. # -# To build on another base, re-declare this in the gap below rather than editing -# here: a later ARG wins, and anvil keeps updating the pins you did not touch. +# Re-declare BASE_IMAGE in the gap below to build on another base; a later ARG +# wins, and the pins anvil maintains stay current. ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea # <<< anvil-managed: anvil-container-base-image @@ -92,10 +73,6 @@ RUN curl --proto '=https' --tlsv1.2 -fsSLo /tmp/rustup-init \ && /tmp/rustup-init -y --profile minimal --default-toolchain none --no-modify-path \ && rm /tmp/rustup-init -# Without this, the first `_install-tool` that needs binstall bootstraps it by -# compiling it from source, which costs minutes on every image build and is one -# more source build a toolchain bump can break. CI installs the prebuilt binary -# for the same reason. RUN curl -fsSLo /tmp/cargo-binstall.tgz \ "https://github.com/cargo-bins/cargo-binstall/releases/download/v${CARGO_BINSTALL_VERSION}/cargo-binstall-x86_64-unknown-linux-musl.tgz" \ && echo "${CARGO_BINSTALL_SHA256} /tmp/cargo-binstall.tgz" | sha256sum -c - \ @@ -107,23 +84,17 @@ RUN curl -fsSLo /tmp/cargo-binstall.tgz \ # <<< anvil-managed: anvil-container-tools # >>> anvil-managed: anvil-container-setup -# Install the pinned toolchain and cargo subcommands from the generated catalog. -# The whole recipe tree is copied because `just` has to parse it, and the whole -# tree is hashed into the image tag: `anvil-setup` reaches the install recipes -# through the tier, group and check recipes, so any of them can change what this -# layer installs. -# -# `binstall` matches what CI passes. Not only a speed-up: some catalog tools do -# not compile from source on every pinned toolchain. +# The whole recipe tree is copied because `just` parses it to reach the install +# recipes. # -# The credential files are removed in the same layer as the install. A build +# The credential files are removed in the same layer that used them: a build # secret never lands in a layer, but anything the install *writes* with it is -# ordinary content, and the `chmod` below would publish it world-readable. -# Deleting them in a later `RUN` would not help -- the earlier layer keeps them. +# ordinary content, and the `chmod` below would publish it world-readable. A +# later `RUN` cannot undo that, because the earlier layer keeps them. # -# `registry` and `git` are created before the `chmod` because the run mounts a -# named volume over each, and an engine seeds a new volume from the image path -# it covers. A path that does not exist seeds as root-owned 0755, which the +# `registry` and `git` must exist before the `chmod`. The run mounts a named +# volume over each, and an engine seeds a new volume from the image path it +# covers; a path that does not exist seeds as root-owned 0755, which the # `--user` mapping cannot write, so the first cargo fetch fails with EACCES. WORKDIR /opt/anvil COPY justfiles ./justfiles @@ -2416,6 +2387,33 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() +# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# +# Anvil resolves every path it manages against what the repository already has, +# so a checkout that carries `dockerfile` keeps that name and the regions are +# maintained there. On a case-sensitive filesystem the canonical literal then +# names nothing, and the build fails on a file the generator is maintaining. +# +# Also the one place that asserts the file exists: the tag's directory walk +# cannot, because a missing Dockerfile simply contributes nothing to the hash +# and yields a confident tag for an image that can never be built. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-dockerfile: + $ErrorActionPreference = 'Stop' + $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' + $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) + # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. + $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] + if (-not $match) { + $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + } + if (-not $match) { + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 + } + Write-Output ".anvil/container/$($match.Name)" + # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its @@ -2437,7 +2435,9 @@ _anvil-container-path host_path: anvil-container-tag: $ErrorActionPreference = 'Stop' $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') @@ -2476,14 +2476,6 @@ anvil-container-tag: $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } - # The walk cannot assert this on its own: a missing Dockerfile would simply - # contribute nothing and yield a confident tag for an image that can never - # be built. It is the one file under that directory that must exist, so it - # is checked by name. - if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { - Write-Error "anvil: container image input is missing: $dockerfile" - exit 1 - } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, # group and check recipes before it reaches the install recipes in @@ -2612,7 +2604,9 @@ _anvil-container-image: $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag @@ -2881,12 +2875,18 @@ anvil-container *command: $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # A linked worktree keeps its real git directory outside the checkout: its - # `.git` is a file naming an absolute host path, which does not exist inside - # the container, so git resolves neither HEAD nor origin/main. Mount the - # common git directory and replace the checkout's `.git` file with one - # naming that mount; the `commondir` file in the worktree's entry is - # relative, so it resolves under it. + # A checkout whose `.git` is a file keeps its real git directory elsewhere: + # a linked worktree points into the main clone, `--separate-git-dir` and a + # submodule point somewhere else again. The path recorded there is a host + # path that does not exist inside the container, so git resolves neither + # HEAD nor origin/main. Mount the common git directory and replace the + # checkout's `.git` with one naming that mount; the `commondir` file in a + # worktree's entry is relative, so it resolves under it. + # + # The predicate is the shape of `.git`, not whether the git directory + # differs from the common one: `--separate-git-dir` redirects without + # differing, and testing for a difference skips it and leaves git pointed at + # a path the container cannot see. # # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE # so that it stays scoped to it. Those variables are ambient: every process @@ -2904,23 +2904,35 @@ anvil-container *command: if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null - if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon -and + (Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)) { $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path - if ($gitDirAbs -ne $gitCommonAbs) { - $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitCommon = "$engineGitCommon".Trim() - # LF and no trailing newline: git parses this file strictly. - $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" - [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitFile = "$engineGitFile".Trim() - $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + # One mount has to carry both directories, so the git directory must + # sit under the common one. `git worktree` always places it there and + # a redirect without a separate worktree entry makes the two equal; + # anything else cannot be expressed as a single mount, and emitting a + # path that climbs out of it would fail inside the container instead. + if ($rel -eq '.') { + $containerGitDir = '/anvil/gitdir' + } elseif ($rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)) { + Write-Error "anvil: this checkout's git directory ($gitDirAbs) is not inside its common git directory ($gitCommonAbs), so the two cannot be mounted as one tree. Run the container from an ordinary clone or a git worktree checkout." + exit 1 + } else { + $containerGitDir = "/anvil/gitdir/$rel" } + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: $containerGitDir`n") + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } @@ -2998,18 +3010,41 @@ anvil-container *command: # A dry run has no side effects, and a target that cannot be planned # (a typo, a recipe needing arguments) yields nothing, so the run # fails on its own terms rather than on a missing token. + # + # A plan covers the bodies just runs itself, not the body of a + # recipe that one of them launches as a child process. The unscoped + # tier wrapper launches its tier that way, so planning + # `anvil-scheduled` shows the wrapper and none of the checks + # underneath it. Follow each nested target a plan names, or a + # wrapped tier reads as needing nothing and runs unauthenticated. $plan = '' - # The same executable that launched this tree, for the reason every - # other nested call uses it: a caller invoking `just` by absolute - # path with its directory off PATH would otherwise fail here. That - # failure is silent, because an empty plan reads as "does not need a - # token" -- so anvil-aprz would run unauthenticated in an image with - # no gh of its own and block on the rate limit for up to an hour. - try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | - ForEach-Object { $_.ToString() }) -join "`n" - } catch { - $plan = '' + $targets = [System.Collections.Generic.List[object]]::new() + $targets.Add([string[]]@($argv | Select-Object -Skip 1)) + $planned = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + for ($i = 0; $i -lt $targets.Count; $i++) { + $target = [string[]]$targets[$i] + # A recipe reachable twice is planned once, and a body naming + # itself terminates. + if (-not $planned.Add(($target -join ' '))) { continue } + # The same executable that launched this tree, for the reason + # every other nested call uses it: a caller invoking `just` by + # absolute path with its directory off PATH would otherwise fail + # here. That failure is silent, because an empty plan reads as + # "does not need a token" -- so anvil-aprz would run + # unauthenticated in an image with no gh of its own and block on + # the rate limit for up to an hour. + $step = '' + try { + $step = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @target 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $step = '' + } + $plan = "$plan`n$step" + # A launched recipe appears as a quoted argument to `just`. + foreach ($nested in [regex]::Matches($step, "'(_anvil-[^'\s]+)'")) { + $targets.Add([string[]]@($nested.Groups[1].Value)) + } } $needsToken = $plan -match 'GITHUB_TOKEN' } @@ -3029,11 +3064,14 @@ anvil-container *command: # The recipe contract's own inputs. These are read by generated checks -- # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its - # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so - # dropping them at the boundary makes the same command mean different things - # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it - # exits 0 with a skip notice, so a title a native run rejects passes in a - # container and the tier still reports green. + # CI equivalents, and `anvil-impact` reads ANVIL_IMPACT to decide whether to + # compute scoping, consume a downloaded cache, or skip -- so dropping them at + # the boundary makes the same command mean different things inside and out. + # anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with a + # skip notice, so a title a native run rejects passes in a container and the + # tier still reports green. ANVIL_IMPACT is the other: a CI group job exports + # `consume`, and a container that did not inherit it would recompute scoping + # from a diff instead of trusting the artifact the group downloaded. # # Forwarded by name and only when set, so an unset variable stays unset # rather than arriving as an empty string, which several of these treat as @@ -3041,7 +3079,7 @@ anvil-container *command: foreach ($name in @( 'PR_TITLE', 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', - 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + 'ANVIL_IMPACT')) { if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { $forwardedEnv += $name $runArgs += @('-e', $name) diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index fcc1e3b1..8f8484a9 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -106,6 +106,33 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() +# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# +# Anvil resolves every path it manages against what the repository already has, +# so a checkout that carries `dockerfile` keeps that name and the regions are +# maintained there. On a case-sensitive filesystem the canonical literal then +# names nothing, and the build fails on a file the generator is maintaining. +# +# Also the one place that asserts the file exists: the tag's directory walk +# cannot, because a missing Dockerfile simply contributes nothing to the hash +# and yields a confident tag for an image that can never be built. +[private] +[script("pwsh", "-NoProfile")] +_anvil-container-dockerfile: + $ErrorActionPreference = 'Stop' + $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' + $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) + # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. + $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] + if (-not $match) { + $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + } + if (-not $match) { + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 + } + Write-Output ".anvil/container/$($match.Name)" + # Print the exec image reference for the current inputs, without building it. # # The tag is a SHA-256 over the image's declared inputs: the Dockerfile and its @@ -127,7 +154,9 @@ _anvil-container-path host_path: anvil-container-tag: $ErrorActionPreference = 'Stop' $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') @@ -166,14 +195,6 @@ anvil-container-tag: $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' } } - # The walk cannot assert this on its own: a missing Dockerfile would simply - # contribute nothing and yield a confident tag for an image that can never - # be built. It is the one file under that directory that must exist, so it - # is checked by name. - if (-not (Test-Path -LiteralPath (Join-Path $repoRoot $dockerfile) -PathType Leaf)) { - Write-Error "anvil: container image input is missing: $dockerfile" - exit 1 - } # Every generated recipe file. The image installs its tools by running # `just anvil-setup`, and that dependency chain runs through the tier, # group and check recipes before it reaches the install recipes in @@ -302,7 +323,9 @@ _anvil-container-image: $enginePrefix = @($engineCmd | Select-Object -Skip 1) $repoRoot = '{{ replace(justfile_directory(), "'", "''") }}' - $dockerfile = '.anvil/container/Dockerfile' + $dockerfile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-dockerfile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $dockerfile = "$dockerfile".Trim() $hookRel = '.anvil/container/hooks.ps1' $image = & '{{ replace(just_executable(), "'", "''") }}' anvil-container-tag @@ -571,12 +594,18 @@ anvil-container *command: $runArgs += $interactive ? '-it' : '-i' $runArgs += @('-v', "${engineRoot}:{{anvil_container_workdir}}") - # A linked worktree keeps its real git directory outside the checkout: its - # `.git` is a file naming an absolute host path, which does not exist inside - # the container, so git resolves neither HEAD nor origin/main. Mount the - # common git directory and replace the checkout's `.git` file with one - # naming that mount; the `commondir` file in the worktree's entry is - # relative, so it resolves under it. + # A checkout whose `.git` is a file keeps its real git directory elsewhere: + # a linked worktree points into the main clone, `--separate-git-dir` and a + # submodule point somewhere else again. The path recorded there is a host + # path that does not exist inside the container, so git resolves neither + # HEAD nor origin/main. Mount the common git directory and replace the + # checkout's `.git` with one naming that mount; the `commondir` file in a + # worktree's entry is relative, so it resolves under it. + # + # The predicate is the shape of `.git`, not whether the git directory + # differs from the common one: `--separate-git-dir` redirects without + # differing, and testing for a difference skips it and leaves git pointed at + # a path the container cannot see. # # The redirection lives in the checkout rather than in GIT_DIR/GIT_WORK_TREE # so that it stays scoped to it. Those variables are ambient: every process @@ -594,23 +623,35 @@ anvil-container *command: if (Get-Command git -ErrorAction SilentlyContinue) { $gitDir = & git rev-parse --git-dir 2>$null $gitCommon = & git rev-parse --git-common-dir 2>$null - if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon) { + if ($LASTEXITCODE -eq 0 -and $gitDir -and $gitCommon -and + (Test-Path -LiteralPath (Join-Path $repoRoot '.git') -PathType Leaf)) { $gitDirAbs = (Resolve-Path -LiteralPath $gitDir).Path $gitCommonAbs = (Resolve-Path -LiteralPath $gitCommon).Path - if ($gitDirAbs -ne $gitCommonAbs) { - $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' - $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitCommon = "$engineGitCommon".Trim() - # LF and no trailing newline: git parses this file strictly. - $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" - [System.IO.File]::WriteAllText($gitFile, "gitdir: /anvil/gitdir/$rel`n") - $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile - if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } - $engineGitFile = "$engineGitFile".Trim() - $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") - $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") + $rel = [System.IO.Path]::GetRelativePath($gitCommonAbs, $gitDirAbs) -replace '\\', '/' + # One mount has to carry both directories, so the git directory must + # sit under the common one. `git worktree` always places it there and + # a redirect without a separate worktree entry makes the two equal; + # anything else cannot be expressed as a single mount, and emitting a + # path that climbs out of it would fail inside the container instead. + if ($rel -eq '.') { + $containerGitDir = '/anvil/gitdir' + } elseif ($rel.StartsWith('../') -or [System.IO.Path]::IsPathRooted($rel)) { + Write-Error "anvil: this checkout's git directory ($gitDirAbs) is not inside its common git directory ($gitCommonAbs), so the two cannot be mounted as one tree. Run the container from an ordinary clone or a git worktree checkout." + exit 1 + } else { + $containerGitDir = "/anvil/gitdir/$rel" } + $engineGitCommon = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitCommonAbs + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitCommon = "$engineGitCommon".Trim() + # LF and no trailing newline: git parses this file strictly. + $gitFile = Join-Path ([System.IO.Path]::GetTempPath()) "anvil-gitfile-$([System.Guid]::NewGuid().ToString('N'))" + [System.IO.File]::WriteAllText($gitFile, "gitdir: $containerGitDir`n") + $engineGitFile = & '{{ replace(just_executable(), "'", "''") }}' _anvil-container-path $gitFile + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + $engineGitFile = "$engineGitFile".Trim() + $runArgs += @('-v', "${engineGitCommon}:/anvil/gitdir") + $runArgs += @('-v', "${engineGitFile}:{{anvil_container_workdir}}/.git:ro") } } @@ -688,18 +729,41 @@ anvil-container *command: # A dry run has no side effects, and a target that cannot be planned # (a typo, a recipe needing arguments) yields nothing, so the run # fails on its own terms rather than on a missing token. + # + # A plan covers the bodies just runs itself, not the body of a + # recipe that one of them launches as a child process. The unscoped + # tier wrapper launches its tier that way, so planning + # `anvil-scheduled` shows the wrapper and none of the checks + # underneath it. Follow each nested target a plan names, or a + # wrapped tier reads as needing nothing and runs unauthenticated. $plan = '' - # The same executable that launched this tree, for the reason every - # other nested call uses it: a caller invoking `just` by absolute - # path with its directory off PATH would otherwise fail here. That - # failure is silent, because an empty plan reads as "does not need a - # token" -- so anvil-aprz would run unauthenticated in an image with - # no gh of its own and block on the rate limit for up to an hour. - try { - $plan = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @($argv | Select-Object -Skip 1) 2>&1 | - ForEach-Object { $_.ToString() }) -join "`n" - } catch { - $plan = '' + $targets = [System.Collections.Generic.List[object]]::new() + $targets.Add([string[]]@($argv | Select-Object -Skip 1)) + $planned = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + for ($i = 0; $i -lt $targets.Count; $i++) { + $target = [string[]]$targets[$i] + # A recipe reachable twice is planned once, and a body naming + # itself terminates. + if (-not $planned.Add(($target -join ' '))) { continue } + # The same executable that launched this tree, for the reason + # every other nested call uses it: a caller invoking `just` by + # absolute path with its directory off PATH would otherwise fail + # here. That failure is silent, because an empty plan reads as + # "does not need a token" -- so anvil-aprz would run + # unauthenticated in an image with no gh of its own and block on + # the rate limit for up to an hour. + $step = '' + try { + $step = (& '{{ replace(just_executable(), "'", "''") }}' --dry-run @target 2>&1 | + ForEach-Object { $_.ToString() }) -join "`n" + } catch { + $step = '' + } + $plan = "$plan`n$step" + # A launched recipe appears as a quoted argument to `just`. + foreach ($nested in [regex]::Matches($step, "'(_anvil-[^'\s]+)'")) { + $targets.Add([string[]]@($nested.Groups[1].Value)) + } } $needsToken = $plan -match 'GITHUB_TOKEN' } @@ -719,11 +783,14 @@ anvil-container *command: # The recipe contract's own inputs. These are read by generated checks -- # `anvil-pr-title` reads PR_TITLE, `_anvil-base-ref` reads BASE_REF and its - # CI equivalents, and the impact filters read the ANVIL_INCLUDE_* set -- so - # dropping them at the boundary makes the same command mean different things - # inside and out. anvil-pr-title is the sharp case: with PR_TITLE unset it - # exits 0 with a skip notice, so a title a native run rejects passes in a - # container and the tier still reports green. + # CI equivalents, and `anvil-impact` reads ANVIL_IMPACT to decide whether to + # compute scoping, consume a downloaded cache, or skip -- so dropping them at + # the boundary makes the same command mean different things inside and out. + # anvil-pr-title is the sharp case: with PR_TITLE unset it exits 0 with a + # skip notice, so a title a native run rejects passes in a container and the + # tier still reports green. ANVIL_IMPACT is the other: a CI group job exports + # `consume`, and a container that did not inherit it would recompute scoping + # from a diff instead of trusting the artifact the group downloaded. # # Forwarded by name and only when set, so an unset variable stays unset # rather than arriving as an empty string, which several of these treat as @@ -731,7 +798,7 @@ anvil-container *command: foreach ($name in @( 'PR_TITLE', 'BASE_REF', 'GITHUB_BASE_REF', 'SYSTEM_PULLREQUEST_TARGETBRANCH', - 'ANVIL_INCLUDE_MODIFIED', 'ANVIL_INCLUDE_AFFECTED', 'ANVIL_INCLUDE_REQUIRED')) { + 'ANVIL_IMPACT')) { if ((Test-Path -LiteralPath "Env:$name") -and -not [string]::IsNullOrEmpty((Get-Item -LiteralPath "Env:$name").Value)) { $forwardedEnv += $name $runArgs += @('-e', $name) From 31a46c3a5c4da690a2dbc24a038a152ee2b3aa50 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 16:58:22 +0200 Subject: [PATCH 72/81] fix(cargo-anvil): restore the trailing newline and dictionary entry the last commit dropped The appended tests left the file without a final newline, which rustfmt rejects, and the containment doc comment introduced a word the hunspell dictionary does not carry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .spelling | 1 + crates/cargo-anvil/tests/recipe_contracts.rs | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.spelling b/.spelling index eca6a02f..73e9491c 100644 --- a/.spelling +++ b/.spelling @@ -893,3 +893,4 @@ cryptographic groupable memoization triaging +lexically diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index d47d167e..7ba6547e 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1219,4 +1219,4 @@ fn the_container_dockerfile_resolves_to_its_on_disk_casing() { "an exact-case match must win over a case-insensitive one" ); } -} \ No newline at end of file +} From 1cfd22be3691933a20b041ed9ef4713d9c83bff2 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 17:04:02 +0200 Subject: [PATCH 73/81] fix(cargo-anvil): fold the executable bit into the container image tag COPY carries a file's mode into the image, so a chmod +x with no content change altered what the image contained while the tag kept resolving, and the stale image was reused. The bit comes from git's index rather than the filesystem: Windows has no executable bit, so a filesystem read would make two checkouts of one commit disagree on the tag. A path git cannot account for frames as non-executable, which can cost a cache miss but never a stale reuse. Every image reference changes once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- .../templates/justfiles/anvil/container.just | 28 +++++++++++- crates/cargo-anvil/tests/recipe_contracts.rs | 45 +++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 28 +++++++++++- .../snapshots/snapshots__github_backend.snap | 28 +++++++++++- .../snapshots/snapshots__local_only.snap | 28 +++++++++++- justfiles/anvil/container.just | 28 +++++++++++- 7 files changed, 182 insertions(+), 7 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index c64edf7f..5a47a891 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:844aa0ce3a36a02529782fbef8a4452933ef9f2312dee2ba758634af240df139" +catalog_checksum = "sha256:3e0b5ade87ea56ff49ac40bccd2e0fa9c41fa4e9ef7faee709a5b3471b6e92e8" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:35c801e86d2c2bfb94861a5ff86ecdcb991d2e86c79dfb7b1231ba8821864a54" +checksum = "sha256:b8b8bf3c285f9426f73c077b8fd22fa82394b0788b5b465cdba8837b1d4e70b5" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 8f8484a9..80f70373 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -261,6 +261,31 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + # The executable bit is part of what `COPY` puts in the image, so a `chmod + # +x` that changes no content still changes the image and must rename the + # tag. Git's index is the only source of that bit which answers identically + # on every platform: Windows has no such bit, so reading it from the + # filesystem would make two checkouts of one commit disagree on the tag. + # + # A path git cannot account for -- no git on PATH, or a file that is + # untracked -- frames as non-executable. The cost is a cache miss against a + # host that knows better, never the reuse of an image that no longer matches + # its inputs. + # + # quotePath=false so a non-ASCII path arrives verbatim rather than + # backslash-escaped, which would key the set on a spelling the walk never + # produces. + $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + if (Get-Command git -ErrorAction SilentlyContinue) { + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $staged) { + if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { + [void]$executable.Add($Matches[1]) + } + } + } + } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on # .NET Core 3.1 where the static overload does not exist. Failing there @@ -279,8 +304,9 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } + $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( - 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $content.Length + ' ') + 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) if ($content.Length -gt 0) { [void]$sha.TransformBlock($content, 0, $content.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 7ba6547e..898fbafc 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1220,3 +1220,48 @@ fn the_container_dockerfile_resolves_to_its_on_disk_casing() { ); } } + +/// `COPY` carries a file's executable bit into the image, so a `chmod +x` with +/// no content change still changes what the image contains. The tag has to +/// follow it, or the changed image keeps a reference that already resolves and +/// the stale one is reused. +/// +/// The bit is read from git's index rather than the filesystem, because Windows +/// has no such bit and two checkouts of one commit must agree on the tag. The +/// fixture's stub git is what makes that observable from either platform. +#[test] +fn the_image_tag_follows_the_executable_bit() { + if !tools_available() { + return; + } + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join("rust-toolchain.toml"), "[toolchain]\nchannel = \"stable\"\n"); + write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); + write(&root.join("justfiles/anvil/setup.sh"), "echo hello\n"); + write( + &root.join("fake-bin/git.ps1"), + "if ($args -contains 'ls-files') {\n \ + $mode = if ($env:FAKE_EXECUTABLE -eq '1') { '100755' } else { '100644' }\n \ + Write-Output \"$mode 0000000000000000000000000000000000000000 0`tjustfiles/anvil/setup.sh\"\n}\nexit 0\n", + ); + + let tag = |executable: &str| { + let output = run_just(root, &["anvil-container-tag"], &[("FAKE_EXECUTABLE", OsStr::new(executable))]); + assert!( + output.status.success(), + "computing the tag failed\nstdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + String::from_utf8_lossy(&output.stdout).trim().to_owned() + }; + + let plain = tag("0"); + let executable = tag("1"); + assert_ne!( + plain, executable, + "the executable bit must reach the digest, or a chmod leaves the image unnamed" + ); + assert_eq!(plain, tag("0"), "the tag must depend on the inputs alone"); +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 762c810c..b58196f2 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3790,6 +3790,31 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + # The executable bit is part of what `COPY` puts in the image, so a `chmod + # +x` that changes no content still changes the image and must rename the + # tag. Git's index is the only source of that bit which answers identically + # on every platform: Windows has no such bit, so reading it from the + # filesystem would make two checkouts of one commit disagree on the tag. + # + # A path git cannot account for -- no git on PATH, or a file that is + # untracked -- frames as non-executable. The cost is a cache miss against a + # host that knows better, never the reuse of an image that no longer matches + # its inputs. + # + # quotePath=false so a non-ASCII path arrives verbatim rather than + # backslash-escaped, which would key the set on a spelling the walk never + # produces. + $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + if (Get-Command git -ErrorAction SilentlyContinue) { + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $staged) { + if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { + [void]$executable.Add($Matches[1]) + } + } + } + } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on # .NET Core 3.1 where the static overload does not exist. Failing there @@ -3808,8 +3833,9 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } + $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( - 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $content.Length + ' ') + 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) if ($content.Length -gt 0) { [void]$sha.TransformBlock($content, 0, $content.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 3ff6e8c8..0cf96dfa 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3669,6 +3669,31 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + # The executable bit is part of what `COPY` puts in the image, so a `chmod + # +x` that changes no content still changes the image and must rename the + # tag. Git's index is the only source of that bit which answers identically + # on every platform: Windows has no such bit, so reading it from the + # filesystem would make two checkouts of one commit disagree on the tag. + # + # A path git cannot account for -- no git on PATH, or a file that is + # untracked -- frames as non-executable. The cost is a cache miss against a + # host that knows better, never the reuse of an image that no longer matches + # its inputs. + # + # quotePath=false so a non-ASCII path arrives verbatim rather than + # backslash-escaped, which would key the set on a spelling the walk never + # produces. + $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + if (Get-Command git -ErrorAction SilentlyContinue) { + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $staged) { + if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { + [void]$executable.Add($Matches[1]) + } + } + } + } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on # .NET Core 3.1 where the static overload does not exist. Failing there @@ -3687,8 +3712,9 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } + $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( - 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $content.Length + ' ') + 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) if ($content.Length -gt 0) { [void]$sha.TransformBlock($content, 0, $content.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 89406f94..e9bb7258 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2542,6 +2542,31 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + # The executable bit is part of what `COPY` puts in the image, so a `chmod + # +x` that changes no content still changes the image and must rename the + # tag. Git's index is the only source of that bit which answers identically + # on every platform: Windows has no such bit, so reading it from the + # filesystem would make two checkouts of one commit disagree on the tag. + # + # A path git cannot account for -- no git on PATH, or a file that is + # untracked -- frames as non-executable. The cost is a cache miss against a + # host that knows better, never the reuse of an image that no longer matches + # its inputs. + # + # quotePath=false so a non-ASCII path arrives verbatim rather than + # backslash-escaped, which would key the set on a spelling the walk never + # produces. + $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + if (Get-Command git -ErrorAction SilentlyContinue) { + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $staged) { + if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { + [void]$executable.Add($Matches[1]) + } + } + } + } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on # .NET Core 3.1 where the static overload does not exist. Failing there @@ -2560,8 +2585,9 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } + $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( - 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $content.Length + ' ') + 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) if ($content.Length -gt 0) { [void]$sha.TransformBlock($content, 0, $content.Length, $null, 0) diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 8f8484a9..80f70373 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -261,6 +261,31 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) + # The executable bit is part of what `COPY` puts in the image, so a `chmod + # +x` that changes no content still changes the image and must rename the + # tag. Git's index is the only source of that bit which answers identically + # on every platform: Windows has no such bit, so reading it from the + # filesystem would make two checkouts of one commit disagree on the tag. + # + # A path git cannot account for -- no git on PATH, or a file that is + # untracked -- frames as non-executable. The cost is a cache miss against a + # host that knows better, never the reuse of an image that no longer matches + # its inputs. + # + # quotePath=false so a non-ASCII path arrives verbatim rather than + # backslash-escaped, which would key the set on a spelling the walk never + # produces. + $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + if (Get-Command git -ErrorAction SilentlyContinue) { + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $staged) { + if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { + [void]$executable.Add($Matches[1]) + } + } + } + } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on # .NET Core 3.1 where the static overload does not exist. Failing there @@ -279,8 +304,9 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } + $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( - 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $content.Length + ' ') + 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) if ($content.Length -gt 0) { [void]$sha.TransformBlock($content, 0, $content.Length, $null, 0) From d9c25b442d3db288db31a507a3b1c5df3d5cd54f Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 17:08:44 +0200 Subject: [PATCH 74/81] fix(cargo-anvil): reject a manifest path that names a directory, and drop a duplicated tier comment Path::components folds a trailing '.' away, so 'a/.' satisfied the name-count check while naming a directory, against the contract the function documents. The raw final segment now decides. The full tier carried its summary line twice, once as prose and once as the doc comment just --list reads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 ++-- crates/cargo-anvil/src/manifest.rs | 9 +++++++-- crates/cargo-anvil/templates/justfiles/anvil/tiers.just | 1 - .../tests/snapshots/snapshots__ado_backend.snap | 1 - .../tests/snapshots/snapshots__github_backend.snap | 1 - .../tests/snapshots/snapshots__local_only.snap | 1 - justfiles/anvil/tiers.just | 1 - 7 files changed, 9 insertions(+), 9 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 5a47a891..20f744b1 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:3e0b5ade87ea56ff49ac40bccd2e0fa9c41fa4e9ef7faee709a5b3471b6e92e8" +catalog_checksum = "sha256:4ddb59624d988e688e62dffe8528e25615ece7a40dbb2299d1b2b4d247f436dc" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -217,7 +217,7 @@ checksum = "sha256:2f7f0187f8c45716a1bed85f0c45ffbc71cdf592ed6414c9bc4cd72cf716a [[file]] path = "justfiles/anvil/tiers.just" -checksum = "sha256:b409feb16d9a675221ec2d9202110eeb1e2fa166971f48a45eeaecba2c35017f" +checksum = "sha256:00453a12cbb34811ee6a2c083dade5f6198575e3b0610f49e4743366326cdd18" [[file]] path = "justfiles/anvil/tools.just" diff --git a/crates/cargo-anvil/src/manifest.rs b/crates/cargo-anvil/src/manifest.rs index 2f2f24f9..9f22bbbe 100644 --- a/crates/cargo-anvil/src/manifest.rs +++ b/crates/cargo-anvil/src/manifest.rs @@ -89,7 +89,12 @@ fn ensure_contained(path: &str, context: &str) -> Result<(), AppError> { } } } - if names == 0 { + // `Path::components` folds a trailing `.` away, so `a/.` arrives as a lone + // `Normal("a")` and satisfies the count above while naming a directory. + // Paths are stored `/`-separated, so the raw final segment is what decides + // whether a file is named at all. + let last = path.rsplit('/').next().unwrap_or_default(); + if names == 0 || last.is_empty() || last == "." { bail!("{context} '{path}' must name a file inside the repository"); } Ok(()) @@ -562,7 +567,7 @@ mod tests { #[test] fn rejects_a_path_that_names_no_file() { - for empty in ["", ".", "./"] { + for empty in ["", ".", "./", "a/.", "a/"] { let toml = format!("version = 1\ntool = \"anvil\"\n\n[[file]]\npath = \"{empty}\"\nchecksum = \"sha256:x\"\n"); let err = Manifest::parse(&toml).unwrap_err(); assert!( diff --git a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just index e2230db6..a06e87b4 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/tiers.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/tiers.just @@ -41,7 +41,6 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-runtime-analysis \ anvil-scheduled-exhaustive -# Full tier: PR + scheduled, end-to-end. Useful before tagging a release. # Full-workspace for the same reason as the scheduled tier. # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index b58196f2..6aa47e28 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -5800,7 +5800,6 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-runtime-analysis \ anvil-scheduled-exhaustive -# Full tier: PR + scheduled, end-to-end. Useful before tagging a release. # Full-workspace for the same reason as the scheduled tier. # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 0cf96dfa..b5bd505b 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -5679,7 +5679,6 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-runtime-analysis \ anvil-scheduled-exhaustive -# Full tier: PR + scheduled, end-to-end. Useful before tagging a release. # Full-workspace for the same reason as the scheduled tier. # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index e9bb7258..1b55d3de 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -4552,7 +4552,6 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-runtime-analysis \ anvil-scheduled-exhaustive -# Full tier: PR + scheduled, end-to-end. Useful before tagging a release. # Full-workspace for the same reason as the scheduled tier. # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. diff --git a/justfiles/anvil/tiers.just b/justfiles/anvil/tiers.just index e2230db6..a06e87b4 100644 --- a/justfiles/anvil/tiers.just +++ b/justfiles/anvil/tiers.just @@ -41,7 +41,6 @@ _anvil-scheduled: anvil-scheduled-validate-prereqs \ anvil-scheduled-runtime-analysis \ anvil-scheduled-exhaustive -# Full tier: PR + scheduled, end-to-end. Useful before tagging a release. # Full-workspace for the same reason as the scheduled tier. # Full tier: PR + scheduled, end-to-end. Useful before tagging a release. From 578d44f250f74e0411bfa10d6d38d294f8c0bc30 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 18:00:28 +0200 Subject: [PATCH 75/81] fix(cargo-anvil): reject a backslash in a manifest path Windows treats the backslash as a separator and Unix as an ordinary filename character, so a path carrying one denotes different things on different machines and slips past whichever check is written in terms of the other: 'a\\' counts a component on Windows yet ends no '/' segment, and '..\\x' climbs on Windows while reading as a single filename on Unix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- crates/cargo-anvil/src/manifest.rs | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/crates/cargo-anvil/src/manifest.rs b/crates/cargo-anvil/src/manifest.rs index 9f22bbbe..b7707a56 100644 --- a/crates/cargo-anvil/src/manifest.rs +++ b/crates/cargo-anvil/src/manifest.rs @@ -76,9 +76,19 @@ pub struct RegionKey { /// # Errors /// /// Returns an error naming `context` and the offending path when it is -/// absolute, carries a drive or network-share prefix, contains a `..` -/// component, or names no file at all. +/// absolute, carries a drive or network-share prefix or a backslash, contains +/// a `..` component, or names no file at all. fn ensure_contained(path: &str, context: &str) -> Result<(), AppError> { + // The format is `/`-separated, and the platforms disagree about `\`: + // Windows treats it as a separator, Unix as an ordinary filename character. + // A path carrying one therefore denotes different things on different + // machines and slips past whichever check is written in terms of the other + // -- `a\` counts a component on Windows yet ends no `/` segment, and + // `..\x` climbs on Windows while reading as one filename on Unix. Rejected + // rather than interpreted. + if path.contains('\\') { + bail!("{context} '{path}' must be a relative path inside the repository"); + } let mut names = 0_usize; for component in Path::new(path).components() { match component { @@ -548,8 +558,19 @@ mod tests { } #[test] fn rejects_a_file_path_that_escapes_the_repository() { - for escape in ["../outside.txt", "/etc/passwd", "a/../../b.txt"] { - let toml = format!("version = 1\ntool = \"anvil\"\n\n[[file]]\npath = \"{escape}\"\nchecksum = \"sha256:x\"\n"); + for escape in [ + "../outside.txt", + "/etc/passwd", + "a/../../b.txt", + "a\\", + "..\\outside.txt", + "C:\\x.txt", + ] { + // A TOML basic string treats `\` as an escape, so a path carrying + // one has to arrive doubled or the document itself is malformed and + // the parse fails before the path is ever validated. + let literal = escape.replace('\\', "\\\\"); + let toml = format!("version = 1\ntool = \"anvil\"\n\n[[file]]\npath = \"{literal}\"\nchecksum = \"sha256:x\"\n"); let err = Manifest::parse(&toml).unwrap_err(); assert!( format!("{err}").contains("must be a relative path inside the repository"), From 821aaff827ed1f3887daf2a509bb77c7e57168a0 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 19:16:52 +0200 Subject: [PATCH 76/81] fix(cargo-anvil): keep the ignore file reachable, stop an unstaged chmod naming the wrong image, and reject a drive-qualified manifest path Resolving the Dockerfile to its on-disk casing broke the ignore file. The engine derives that name from the Dockerfile's -- BuildKit reads .dockerignore and takes no flag pointing elsewhere -- while anvil maintains the artifact at a fixed canonical path, so a repository carrying 'dockerfile' would build with no ignore file at all and stream the whole worktree into the context. The same derivation missed the owned file in the text-normalized set, so CRLF and LF checkouts could disagree on the tag. The two names must agree and only one can move, so a case variant is now refused with the reason rather than accommodated. The digest reads the index mode while the build copies the working tree, so an unstaged chmod changed the image without renaming it. Framing the working tree instead would break the property the index was chosen for: Windows has no executable bit, so two checkouts of one commit would disagree and a published image would stop resolving. The disagreement is refused instead, naming the file and the recovery. Path::components reports a drive qualifier as a Prefix on Windows and as an ordinary name on Unix, so 'C:x.txt' was accepted there despite the documented contract. Matched lexically now. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- .../src/anvil/artifacts/container.rs | 21 ++-- crates/cargo-anvil/src/manifest.rs | 12 +- .../templates/justfiles/anvil/container.just | 62 ++++++++--- crates/cargo-anvil/tests/recipe_contracts.rs | 103 ++++++++++++------ .../snapshots/snapshots__ado_backend.snap | 62 ++++++++--- .../snapshots/snapshots__github_backend.snap | 62 ++++++++--- .../snapshots/snapshots__local_only.snap | 62 ++++++++--- justfiles/anvil/container.just | 62 ++++++++--- 9 files changed, 313 insertions(+), 137 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 20f744b1..58ae6ebb 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:4ddb59624d988e688e62dffe8528e25615ece7a40dbb2299d1b2b4d247f436dc" +catalog_checksum = "sha256:53bd8acbb5beed0120b532a7a0a071d449a8eb8b1018a69bf5b933567449e074" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:b8b8bf3c285f9426f73c077b8fd22fa82394b0788b5b465cdba8837b1d4e70b5" +checksum = "sha256:14a065c8ced786be257eccf7023aaf549eaa2b7d55ec542b86cc5b9198f9fa31" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index d4e54a0f..47f179fc 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -960,24 +960,29 @@ mod tests { } #[test] - fn the_dockerfile_is_found_under_the_casing_on_disk() { - // Anvil resolves every path it manages against what the repository - // already has, so a checkout carrying `dockerfile` keeps that name and - // the regions are maintained there. A recipe hard-coding the canonical - // literal then names nothing on a case-sensitive filesystem. + fn the_dockerfile_must_carry_the_canonical_name() { + // The engine reads the ignore file as `.dockerignore` and + // anvil maintains it at `.anvil/container/Dockerfile.dockerignore`, so + // the two names have to agree and only one of them can move. A variant + // is refused, because building from `dockerfile` would find no ignore + // file and stream the whole worktree into the build context. assert!( !RECIPE.contains("$dockerfile = '.anvil/container/Dockerfile'"), - "the path must be resolved, not assumed" + "the path must come from the recipe that checks it" ); assert!(RECIPE.contains("_anvil-container-dockerfile:")); // -ceq, because PowerShell's -eq on strings is case-insensitive and - // would make the exact-match pass indistinguishable from the fallback. + // would make the canonical name indistinguishable from a variant. assert!(RECIPE.contains("$_.Name -ceq 'Dockerfile'")); - assert!(RECIPE.contains("$_.Name -ieq 'Dockerfile'")); + assert!(RECIPE.contains("must be named exactly '.anvil/container/Dockerfile'")); // The one place that asserts the file exists: the tag's directory walk // cannot, because a missing Dockerfile contributes nothing to the hash // and yields a confident tag for an image that can never be built. assert!(RECIPE.contains("anvil: container image input is missing: .anvil/container/Dockerfile")); + // The ignore file is derived from that name, so it lands on the owned + // artifact and is normalized as the text it is -- otherwise a CRLF and + // an LF checkout of one commit disagree on the tag. + assert!(RECIPE.contains(r#"[void]$declaredText.Add("$dockerfile.dockerignore")"#)); } #[test] diff --git a/crates/cargo-anvil/src/manifest.rs b/crates/cargo-anvil/src/manifest.rs index b7707a56..2a2b0d37 100644 --- a/crates/cargo-anvil/src/manifest.rs +++ b/crates/cargo-anvil/src/manifest.rs @@ -86,7 +86,15 @@ fn ensure_contained(path: &str, context: &str) -> Result<(), AppError> { // -- `a\` counts a component on Windows yet ends no `/` segment, and // `..\x` climbs on Windows while reading as one filename on Unix. Rejected // rather than interpreted. - if path.contains('\\') { + // + // A drive qualifier divides the platforms the same way: `Path::components` + // reports `C:x.txt` as a `Prefix` on Windows and as one ordinary name on + // Unix, so the check below cannot see it there. Matched lexically instead. + let drive_qualified = { + let mut chars = path.chars(); + matches!((chars.next(), chars.next()), (Some(letter), Some(':')) if letter.is_ascii_alphabetic()) + }; + if path.contains('\\') || drive_qualified { bail!("{context} '{path}' must be a relative path inside the repository"); } let mut names = 0_usize; @@ -565,6 +573,8 @@ mod tests { "a\\", "..\\outside.txt", "C:\\x.txt", + "C:x.txt", + "z:dir/file.txt", ] { // A TOML basic string treats `\` as an escape, so a path carrying // one has to arrive doubled or the document itself is malformed and diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 80f70373..40fcc8bd 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -106,12 +106,15 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# Verify the composed Dockerfile is present under the name the build uses. # -# Anvil resolves every path it manages against what the repository already has, -# so a checkout that carries `dockerfile` keeps that name and the regions are -# maintained there. On a case-sensitive filesystem the canonical literal then -# names nothing, and the build fails on a file the generator is maintaining. +# The engine derives the ignore file's name from the Dockerfile's: BuildKit +# reads `.dockerignore` and there is no flag to point it elsewhere. +# Anvil owns that artifact at the fixed path `.anvil/container/Dockerfile.dockerignore`, +# so the two names have to agree, and only one of them can move. A case variant +# is therefore refused rather than accommodated: building from `dockerfile` +# would silently use no ignore file at all, streaming the whole worktree into +# the build context and admitting inputs the tag does not cover. # # Also the one place that asserts the file exists: the tag's directory walk # cannot, because a missing Dockerfile simply contributes nothing to the hash @@ -122,16 +125,20 @@ _anvil-container-dockerfile: $ErrorActionPreference = 'Stop' $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) - # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. - $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] - if (-not $match) { - $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + # -ceq because PowerShell's -eq on strings is case-insensitive, which would + # make the exact name indistinguishable from a variant on a filesystem that + # can hold both. + if (@($entries | Where-Object { $_.Name -ceq 'Dockerfile' }).Count -eq 1) { + Write-Output '.anvil/container/Dockerfile' + exit 0 } - if (-not $match) { - Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + $variant = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + if ($variant) { + Write-Error "anvil: the container image input must be named exactly '.anvil/container/Dockerfile', but this repository has '.anvil/container/$($variant.Name)'. The engine reads the ignore file as '.dockerignore', and anvil maintains '.anvil/container/Dockerfile.dockerignore', so a differently-cased name would build with no ignore file. Rename it." exit 1 } - Write-Output ".anvil/container/$($match.Name)" + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 # Print the exec image reference for the current inputs, without building it. # @@ -265,19 +272,24 @@ anvil-container-tag: # +x` that changes no content still changes the image and must rename the # tag. Git's index is the only source of that bit which answers identically # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag. + # filesystem would make two checkouts of one commit disagree on the tag, and + # a published image would stop resolving for half the people who use it. # - # A path git cannot account for -- no git on PATH, or a file that is - # untracked -- frames as non-executable. The cost is a cache miss against a - # host that knows better, never the reuse of an image that no longer matches - # its inputs. + # The index is authoritative only while the working tree agrees with it. A + # mode change that has not been staged would be copied by the build and + # missed by the tag, so it is refused below rather than absorbed. + # + # An untracked file's mode is not an input: it has no committed identity, so + # no other checkout can reproduce it and there is nothing for a shared tag + # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the set on a spelling the walk never # produces. $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { - $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { @@ -285,6 +297,20 @@ anvil-container-tag: } } } + # `--raw` reports the index mode and the working-tree mode as the first + # two fields, so a mode-only change is visible even though the content + # is identical. On Windows core.fileMode is normally false and git + # reports no drift, which is correct: the filesystem has no bit to + # disagree with. + $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $drift) { + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { + Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } + } + } } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 898fbafc..8fb6c06f 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1164,32 +1164,42 @@ fn a_wrapped_tier_hides_its_checks_from_a_plan() { ); } -/// Anvil resolves every path it manages against what the repository already -/// carries, so a checkout holding `dockerfile` keeps that name and the regions -/// are maintained inside it. The container recipes have to agree, or on a -/// case-sensitive filesystem the build names a file that does not exist. +/// The engine derives the ignore file's name from the Dockerfile's, and anvil +/// maintains that artifact at a fixed canonical path, so the two names have to +/// agree. A case variant is refused rather than accommodated: building from +/// `dockerfile` would find no `dockerfile.dockerignore`, silently stream the +/// whole worktree into the build context, and admit inputs the tag does not +/// cover. #[test] -fn the_container_dockerfile_resolves_to_its_on_disk_casing() { +fn a_case_variant_dockerfile_is_refused_rather_than_built_from() { if !tools_available() { return; } - for name in ["Dockerfile", "dockerfile", "DOCKERFILE"] { - let tmp = fixture(&[("container.just", CONTAINER)], &[]); - let root = tmp.path(); - write(&root.join(".anvil/container").join(name), "FROM scratch\n"); + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); + let canonical = run_just(root, &["_anvil-container-dockerfile"], &[]); + assert!( + canonical.status.success(), + "the canonical name must resolve\nstderr:\n{}", + String::from_utf8_lossy(&canonical.stderr) + ); + assert_eq!(String::from_utf8_lossy(&canonical.stdout).trim(), ".anvil/container/Dockerfile"); - let output = run_just(root, &["_anvil-container-dockerfile"], &[]); + // Only a case-sensitive filesystem can hold a variant that is a different + // file, which is exactly where the ignore-file lookup breaks. + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join(".anvil/container/dockerfile"), "FROM scratch\n"); + let holds_variant = !root.join(".anvil/container/Dockerfile").exists(); + if holds_variant { + let variant = run_just(root, &["_anvil-container-dockerfile"], &[]); + assert_failed(&variant, "resolving a case-variant Dockerfile"); + let stderr = String::from_utf8_lossy(&variant.stderr); assert!( - output.status.success(), - "resolving {name} failed\nstdout:\n{}\nstderr:\n{}", - String::from_utf8_lossy(&output.stdout), - String::from_utf8_lossy(&output.stderr) - ); - assert_eq!( - String::from_utf8_lossy(&output.stdout).trim(), - format!(".anvil/container/{name}"), - "the resolved path must carry the casing on disk" + stderr.contains("must be named exactly") && stderr.contains("dockerignore"), + "the refusal must name the rule and the reason\nstderr:\n{stderr}" ); } @@ -1203,24 +1213,7 @@ fn the_container_dockerfile_resolves_to_its_on_disk_casing() { "the failure must name the missing input\nstderr:\n{}", String::from_utf8_lossy(&missing.stderr) ); - - // Exact case wins over a fold. Only a case-sensitive filesystem can hold - // both spellings at once, so this is the one assertion that cannot run - // everywhere; on Windows the two names are the same file. - let tmp = fixture(&[("container.just", CONTAINER)], &[]); - let root = tmp.path(); - write(&root.join(".anvil/container/dockerfile"), "FROM scratch\n"); - write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); - if fs::read_dir(root.join(".anvil/container")).unwrap().count() == 2 { - let output = run_just(root, &["_anvil-container-dockerfile"], &[]); - assert_eq!( - String::from_utf8_lossy(&output.stdout).trim(), - ".anvil/container/Dockerfile", - "an exact-case match must win over a case-insensitive one" - ); - } } - /// `COPY` carries a file's executable bit into the image, so a `chmod +x` with /// no content change still changes what the image contains. The tag has to /// follow it, or the changed image keeps a reference that already resolves and @@ -1265,3 +1258,41 @@ fn the_image_tag_follows_the_executable_bit() { ); assert_eq!(plain, tag("0"), "the tag must depend on the inputs alone"); } + +/// The index is authoritative for the tag only while the working tree agrees +/// with it. A `chmod` that has not been staged is copied by the build and +/// missed by the digest, so the reference would name an image the build does +/// not produce; the run has to stop rather than absorb that. +#[test] +fn an_unstaged_mode_change_stops_the_run() { + if !tools_available() { + return; + } + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join("rust-toolchain.toml"), "[toolchain]\nchannel = \"stable\"\n"); + write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); + write(&root.join("justfiles/anvil/setup.sh"), "echo hello\n"); + write( + &root.join("fake-bin/git.ps1"), + "if ($args -contains 'ls-files') {\n \ + Write-Output \"100644 0000000000000000000000000000000000000000 0`tjustfiles/anvil/setup.sh\"\n}\n\ + if ($args -contains 'diff' -and $env:FAKE_DRIFT -eq '1') {\n \ + Write-Output \":100644 100755 0000000 0000000 M`tjustfiles/anvil/setup.sh\"\n}\nexit 0\n", + ); + + let clean = run_just(root, &["anvil-container-tag"], &[("FAKE_DRIFT", OsStr::new("0"))]); + assert!( + clean.status.success(), + "an agreeing working tree must compute a tag\nstderr:\n{}", + String::from_utf8_lossy(&clean.stderr) + ); + + let drifted = run_just(root, &["anvil-container-tag"], &[("FAKE_DRIFT", OsStr::new("1"))]); + assert_failed(&drifted, "computing a tag against an unstaged mode change"); + let stderr = String::from_utf8_lossy(&drifted.stderr); + assert!( + stderr.contains("working tree") && stderr.contains("git add"), + "the refusal must name the drift and the recovery\nstderr:\n{stderr}" + ); +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 6aa47e28..6f384abd 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3635,12 +3635,15 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# Verify the composed Dockerfile is present under the name the build uses. # -# Anvil resolves every path it manages against what the repository already has, -# so a checkout that carries `dockerfile` keeps that name and the regions are -# maintained there. On a case-sensitive filesystem the canonical literal then -# names nothing, and the build fails on a file the generator is maintaining. +# The engine derives the ignore file's name from the Dockerfile's: BuildKit +# reads `.dockerignore` and there is no flag to point it elsewhere. +# Anvil owns that artifact at the fixed path `.anvil/container/Dockerfile.dockerignore`, +# so the two names have to agree, and only one of them can move. A case variant +# is therefore refused rather than accommodated: building from `dockerfile` +# would silently use no ignore file at all, streaming the whole worktree into +# the build context and admitting inputs the tag does not cover. # # Also the one place that asserts the file exists: the tag's directory walk # cannot, because a missing Dockerfile simply contributes nothing to the hash @@ -3651,16 +3654,20 @@ _anvil-container-dockerfile: $ErrorActionPreference = 'Stop' $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) - # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. - $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] - if (-not $match) { - $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + # -ceq because PowerShell's -eq on strings is case-insensitive, which would + # make the exact name indistinguishable from a variant on a filesystem that + # can hold both. + if (@($entries | Where-Object { $_.Name -ceq 'Dockerfile' }).Count -eq 1) { + Write-Output '.anvil/container/Dockerfile' + exit 0 } - if (-not $match) { - Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + $variant = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + if ($variant) { + Write-Error "anvil: the container image input must be named exactly '.anvil/container/Dockerfile', but this repository has '.anvil/container/$($variant.Name)'. The engine reads the ignore file as '.dockerignore', and anvil maintains '.anvil/container/Dockerfile.dockerignore', so a differently-cased name would build with no ignore file. Rename it." exit 1 } - Write-Output ".anvil/container/$($match.Name)" + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 # Print the exec image reference for the current inputs, without building it. # @@ -3794,19 +3801,24 @@ anvil-container-tag: # +x` that changes no content still changes the image and must rename the # tag. Git's index is the only source of that bit which answers identically # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag. + # filesystem would make two checkouts of one commit disagree on the tag, and + # a published image would stop resolving for half the people who use it. # - # A path git cannot account for -- no git on PATH, or a file that is - # untracked -- frames as non-executable. The cost is a cache miss against a - # host that knows better, never the reuse of an image that no longer matches - # its inputs. + # The index is authoritative only while the working tree agrees with it. A + # mode change that has not been staged would be copied by the build and + # missed by the tag, so it is refused below rather than absorbed. + # + # An untracked file's mode is not an input: it has no committed identity, so + # no other checkout can reproduce it and there is nothing for a shared tag + # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the set on a spelling the walk never # produces. $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { - $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { @@ -3814,6 +3826,20 @@ anvil-container-tag: } } } + # `--raw` reports the index mode and the working-tree mode as the first + # two fields, so a mode-only change is visible even though the content + # is identical. On Windows core.fileMode is normally false and git + # reports no drift, which is correct: the filesystem has no bit to + # disagree with. + $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $drift) { + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { + Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } + } + } } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index b5bd505b..f8d158b7 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3514,12 +3514,15 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# Verify the composed Dockerfile is present under the name the build uses. # -# Anvil resolves every path it manages against what the repository already has, -# so a checkout that carries `dockerfile` keeps that name and the regions are -# maintained there. On a case-sensitive filesystem the canonical literal then -# names nothing, and the build fails on a file the generator is maintaining. +# The engine derives the ignore file's name from the Dockerfile's: BuildKit +# reads `.dockerignore` and there is no flag to point it elsewhere. +# Anvil owns that artifact at the fixed path `.anvil/container/Dockerfile.dockerignore`, +# so the two names have to agree, and only one of them can move. A case variant +# is therefore refused rather than accommodated: building from `dockerfile` +# would silently use no ignore file at all, streaming the whole worktree into +# the build context and admitting inputs the tag does not cover. # # Also the one place that asserts the file exists: the tag's directory walk # cannot, because a missing Dockerfile simply contributes nothing to the hash @@ -3530,16 +3533,20 @@ _anvil-container-dockerfile: $ErrorActionPreference = 'Stop' $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) - # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. - $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] - if (-not $match) { - $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + # -ceq because PowerShell's -eq on strings is case-insensitive, which would + # make the exact name indistinguishable from a variant on a filesystem that + # can hold both. + if (@($entries | Where-Object { $_.Name -ceq 'Dockerfile' }).Count -eq 1) { + Write-Output '.anvil/container/Dockerfile' + exit 0 } - if (-not $match) { - Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + $variant = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + if ($variant) { + Write-Error "anvil: the container image input must be named exactly '.anvil/container/Dockerfile', but this repository has '.anvil/container/$($variant.Name)'. The engine reads the ignore file as '.dockerignore', and anvil maintains '.anvil/container/Dockerfile.dockerignore', so a differently-cased name would build with no ignore file. Rename it." exit 1 } - Write-Output ".anvil/container/$($match.Name)" + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 # Print the exec image reference for the current inputs, without building it. # @@ -3673,19 +3680,24 @@ anvil-container-tag: # +x` that changes no content still changes the image and must rename the # tag. Git's index is the only source of that bit which answers identically # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag. + # filesystem would make two checkouts of one commit disagree on the tag, and + # a published image would stop resolving for half the people who use it. # - # A path git cannot account for -- no git on PATH, or a file that is - # untracked -- frames as non-executable. The cost is a cache miss against a - # host that knows better, never the reuse of an image that no longer matches - # its inputs. + # The index is authoritative only while the working tree agrees with it. A + # mode change that has not been staged would be copied by the build and + # missed by the tag, so it is refused below rather than absorbed. + # + # An untracked file's mode is not an input: it has no committed identity, so + # no other checkout can reproduce it and there is nothing for a shared tag + # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the set on a spelling the walk never # produces. $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { - $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { @@ -3693,6 +3705,20 @@ anvil-container-tag: } } } + # `--raw` reports the index mode and the working-tree mode as the first + # two fields, so a mode-only change is visible even though the content + # is identical. On Windows core.fileMode is normally false and git + # reports no drift, which is correct: the filesystem has no bit to + # disagree with. + $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $drift) { + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { + Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } + } + } } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 1b55d3de..32ec6fa7 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2387,12 +2387,15 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# Verify the composed Dockerfile is present under the name the build uses. # -# Anvil resolves every path it manages against what the repository already has, -# so a checkout that carries `dockerfile` keeps that name and the regions are -# maintained there. On a case-sensitive filesystem the canonical literal then -# names nothing, and the build fails on a file the generator is maintaining. +# The engine derives the ignore file's name from the Dockerfile's: BuildKit +# reads `.dockerignore` and there is no flag to point it elsewhere. +# Anvil owns that artifact at the fixed path `.anvil/container/Dockerfile.dockerignore`, +# so the two names have to agree, and only one of them can move. A case variant +# is therefore refused rather than accommodated: building from `dockerfile` +# would silently use no ignore file at all, streaming the whole worktree into +# the build context and admitting inputs the tag does not cover. # # Also the one place that asserts the file exists: the tag's directory walk # cannot, because a missing Dockerfile simply contributes nothing to the hash @@ -2403,16 +2406,20 @@ _anvil-container-dockerfile: $ErrorActionPreference = 'Stop' $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) - # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. - $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] - if (-not $match) { - $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + # -ceq because PowerShell's -eq on strings is case-insensitive, which would + # make the exact name indistinguishable from a variant on a filesystem that + # can hold both. + if (@($entries | Where-Object { $_.Name -ceq 'Dockerfile' }).Count -eq 1) { + Write-Output '.anvil/container/Dockerfile' + exit 0 } - if (-not $match) { - Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + $variant = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + if ($variant) { + Write-Error "anvil: the container image input must be named exactly '.anvil/container/Dockerfile', but this repository has '.anvil/container/$($variant.Name)'. The engine reads the ignore file as '.dockerignore', and anvil maintains '.anvil/container/Dockerfile.dockerignore', so a differently-cased name would build with no ignore file. Rename it." exit 1 } - Write-Output ".anvil/container/$($match.Name)" + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 # Print the exec image reference for the current inputs, without building it. # @@ -2546,19 +2553,24 @@ anvil-container-tag: # +x` that changes no content still changes the image and must rename the # tag. Git's index is the only source of that bit which answers identically # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag. + # filesystem would make two checkouts of one commit disagree on the tag, and + # a published image would stop resolving for half the people who use it. # - # A path git cannot account for -- no git on PATH, or a file that is - # untracked -- frames as non-executable. The cost is a cache miss against a - # host that knows better, never the reuse of an image that no longer matches - # its inputs. + # The index is authoritative only while the working tree agrees with it. A + # mode change that has not been staged would be copied by the build and + # missed by the tag, so it is refused below rather than absorbed. + # + # An untracked file's mode is not an input: it has no committed identity, so + # no other checkout can reproduce it and there is nothing for a shared tag + # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the set on a spelling the walk never # produces. $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { - $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { @@ -2566,6 +2578,20 @@ anvil-container-tag: } } } + # `--raw` reports the index mode and the working-tree mode as the first + # two fields, so a mode-only change is visible even though the content + # is identical. On Windows core.fileMode is normally false and git + # reports no drift, which is correct: the filesystem has no bit to + # disagree with. + $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $drift) { + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { + Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } + } + } } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 80f70373..40fcc8bd 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -106,12 +106,15 @@ _anvil-container-path host_path: } Write-Output $translated.Trim() -# Print the composed Dockerfile's repository-relative path in its on-disk casing. +# Verify the composed Dockerfile is present under the name the build uses. # -# Anvil resolves every path it manages against what the repository already has, -# so a checkout that carries `dockerfile` keeps that name and the regions are -# maintained there. On a case-sensitive filesystem the canonical literal then -# names nothing, and the build fails on a file the generator is maintaining. +# The engine derives the ignore file's name from the Dockerfile's: BuildKit +# reads `.dockerignore` and there is no flag to point it elsewhere. +# Anvil owns that artifact at the fixed path `.anvil/container/Dockerfile.dockerignore`, +# so the two names have to agree, and only one of them can move. A case variant +# is therefore refused rather than accommodated: building from `dockerfile` +# would silently use no ignore file at all, streaming the whole worktree into +# the build context and admitting inputs the tag does not cover. # # Also the one place that asserts the file exists: the tag's directory walk # cannot, because a missing Dockerfile simply contributes nothing to the hash @@ -122,16 +125,20 @@ _anvil-container-dockerfile: $ErrorActionPreference = 'Stop' $dir = Join-Path '{{ replace(justfile_directory(), "'", "''") }}' '.anvil/container' $entries = @(Get-ChildItem -LiteralPath $dir -File -Force -ErrorAction SilentlyContinue) - # Exact case wins; -ceq because PowerShell's -eq is case-insensitive. - $match = @($entries | Where-Object { $_.Name -ceq 'Dockerfile' })[0] - if (-not $match) { - $match = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + # -ceq because PowerShell's -eq on strings is case-insensitive, which would + # make the exact name indistinguishable from a variant on a filesystem that + # can hold both. + if (@($entries | Where-Object { $_.Name -ceq 'Dockerfile' }).Count -eq 1) { + Write-Output '.anvil/container/Dockerfile' + exit 0 } - if (-not $match) { - Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + $variant = @($entries | Where-Object { $_.Name -ieq 'Dockerfile' })[0] + if ($variant) { + Write-Error "anvil: the container image input must be named exactly '.anvil/container/Dockerfile', but this repository has '.anvil/container/$($variant.Name)'. The engine reads the ignore file as '.dockerignore', and anvil maintains '.anvil/container/Dockerfile.dockerignore', so a differently-cased name would build with no ignore file. Rename it." exit 1 } - Write-Output ".anvil/container/$($match.Name)" + Write-Error 'anvil: container image input is missing: .anvil/container/Dockerfile' + exit 1 # Print the exec image reference for the current inputs, without building it. # @@ -265,19 +272,24 @@ anvil-container-tag: # +x` that changes no content still changes the image and must rename the # tag. Git's index is the only source of that bit which answers identically # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag. + # filesystem would make two checkouts of one commit disagree on the tag, and + # a published image would stop resolving for half the people who use it. # - # A path git cannot account for -- no git on PATH, or a file that is - # untracked -- frames as non-executable. The cost is a cache miss against a - # host that knows better, never the reuse of an image that no longer matches - # its inputs. + # The index is authoritative only while the working tree agrees with it. A + # mode change that has not been staged would be copied by the build and + # missed by the tag, so it is refused below rather than absorbed. + # + # An untracked file's mode is not an input: it has no committed identity, so + # no other checkout can reproduce it and there is nothing for a shared tag + # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the set on a spelling the walk never # produces. $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { - $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- '.anvil/container' 'justfiles' 'rust-toolchain.toml' 2>$null + $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { @@ -285,6 +297,20 @@ anvil-container-tag: } } } + # `--raw` reports the index mode and the working-tree mode as the first + # two fields, so a mode-only change is visible even though the content + # is identical. On Windows core.fileMode is normally false and git + # reports no drift, which is correct: the filesystem has no bit to + # disagree with. + $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null + if ($LASTEXITCODE -eq 0) { + foreach ($entry in $drift) { + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { + Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } + } + } } # ComputeHash rather than the static HashData: the latter arrived in .NET 5, # and the prerequisite check accepts any PowerShell 7, including 7.0 on From 271a0396c06d94be5ca56f7e1ee4e04ea68f3663 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 20:24:26 +0200 Subject: [PATCH 77/81] fix(cargo-anvil): stop the mode-drift guard firing on an absent path git diff --raw encodes an unstaged deletion as a transition to mode 000000 and an intent-to-add as one from it, so the guard added for chmod drift aborted the tag on both. Retiring a managed file and not yet staging the deletion is the ordinary way to reach that. A path missing from the build context is missing from the digest too, so only a transition between two real modes is drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +-- .../templates/justfiles/anvil/container.just | 15 ++++++++-- crates/cargo-anvil/tests/recipe_contracts.rs | 29 +++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 15 ++++++++-- .../snapshots/snapshots__github_backend.snap | 15 ++++++++-- .../snapshots/snapshots__local_only.snap | 15 ++++++++-- justfiles/anvil/container.just | 15 ++++++++-- 7 files changed, 91 insertions(+), 17 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 58ae6ebb..5ad6990e 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:53bd8acbb5beed0120b532a7a0a071d449a8eb8b1018a69bf5b933567449e074" +catalog_checksum = "sha256:1a272c800ded68c3b92b3338c723186a6fe6bd95b5574003626aa6fdd8528e0e" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:14a065c8ced786be257eccf7023aaf549eaa2b7d55ec542b86cc5b9198f9fa31" +checksum = "sha256:9a3f3f88995247c7aca129ae7fd621e78fefa4b6de66ec4ee77bbbbe75551c1b" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 40fcc8bd..28c713de 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -302,12 +302,21 @@ anvil-container-tag: # is identical. On Windows core.fileMode is normally false and git # reports no drift, which is correct: the filesystem has no bit to # disagree with. + # + # A zero mode means the path is absent on that side -- an unstaged + # deletion, or an intent-to-add entry. Neither can make the image + # disagree with its tag, because a path missing from the build context + # is missing from the digest too, so only a transition between two real + # modes is drift. A `chmod` is always one of those. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { - Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." - exit 1 + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] + if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } } } } diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 8fb6c06f..7e854fff 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1295,4 +1295,33 @@ fn an_unstaged_mode_change_stops_the_run() { stderr.contains("working tree") && stderr.contains("git add"), "the refusal must name the drift and the recovery\nstderr:\n{stderr}" ); + + // A zero mode means the path is absent on that side. `git diff --raw` + // encodes an unstaged deletion as `:100644 000000 ... D` and an + // intent-to-add as `:000000 100644 ... A`, and both differ numerically + // without being drift -- a path missing from the build context is missing + // from the digest too. Retiring a managed file and not yet staging the + // deletion is the ordinary way to reach this. + for (kind, raw) in [ + ("an unstaged deletion", ":100644 000000 0000000 0000000 D`tjustfiles/anvil/setup.sh"), + ( + "an intent-to-add entry", + ":000000 100644 0000000 0000000 A`tjustfiles/anvil/setup.sh", + ), + ] { + write( + &root.join("fake-bin/git.ps1"), + &format!( + "if ($args -contains 'ls-files') {{\n \ + Write-Output \"100644 0000000000000000000000000000000000000000 0`tjustfiles/anvil/setup.sh\"\n}}\n\ + if ($args -contains 'diff') {{\n Write-Output \"{raw}\"\n}}\nexit 0\n" + ), + ); + let absent = run_just(root, &["anvil-container-tag"], &[]); + assert!( + absent.status.success(), + "{kind} must not be reported as mode drift\nstderr:\n{}", + String::from_utf8_lossy(&absent.stderr) + ); + } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 6f384abd..8bff73c8 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3831,12 +3831,21 @@ anvil-container-tag: # is identical. On Windows core.fileMode is normally false and git # reports no drift, which is correct: the filesystem has no bit to # disagree with. + # + # A zero mode means the path is absent on that side -- an unstaged + # deletion, or an intent-to-add entry. Neither can make the image + # disagree with its tag, because a path missing from the build context + # is missing from the digest too, so only a transition between two real + # modes is drift. A `chmod` is always one of those. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { - Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." - exit 1 + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] + if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } } } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index f8d158b7..2940aa8e 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3710,12 +3710,21 @@ anvil-container-tag: # is identical. On Windows core.fileMode is normally false and git # reports no drift, which is correct: the filesystem has no bit to # disagree with. + # + # A zero mode means the path is absent on that side -- an unstaged + # deletion, or an intent-to-add entry. Neither can make the image + # disagree with its tag, because a path missing from the build context + # is missing from the digest too, so only a transition between two real + # modes is drift. A `chmod` is always one of those. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { - Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." - exit 1 + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] + if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } } } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index 32ec6fa7..a7b9f3b5 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2583,12 +2583,21 @@ anvil-container-tag: # is identical. On Windows core.fileMode is normally false and git # reports no drift, which is correct: the filesystem has no bit to # disagree with. + # + # A zero mode means the path is absent on that side -- an unstaged + # deletion, or an intent-to-add entry. Neither can make the image + # disagree with its tag, because a path missing from the build context + # is missing from the digest too, so only a transition between two real + # modes is drift. A `chmod` is always one of those. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { - Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." - exit 1 + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] + if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } } } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 40fcc8bd..28c713de 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -302,12 +302,21 @@ anvil-container-tag: # is identical. On Windows core.fileMode is normally false and git # reports no drift, which is correct: the filesystem has no bit to # disagree with. + # + # A zero mode means the path is absent on that side -- an unstaged + # deletion, or an intent-to-add entry. Neither can make the image + # disagree with its tag, because a path missing from the build context + # is missing from the digest too, so only a transition between two real + # modes is drift. A `chmod` is always one of those. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$' -and $Matches[1] -ne $Matches[2]) { - Write-Error "anvil: '$($Matches[3])' has mode $($Matches[2]) in the working tree and $($Matches[1]) in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." - exit 1 + if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] + if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + exit 1 + } } } } From a22da4f7a2030c4b4d0450413731a2774d5a79fd Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 21:30:59 +0200 Subject: [PATCH 78/81] fix(cargo-anvil): compare the working-tree mode against the bit the digest framed The drift guard compared the two mode fields of git diff --raw, but the raw index field is not what the digest uses. An intent-to-add entry reports zero there while ls-files --stage reports a real placeholder mode, so an executable file added with git add -N was exempted although it reaches the build context with its mode, and an ordinary one would have been rejected. Only a zero working-tree mode is exempt now, because that is a deletion and the path is in neither the context nor the digest. Everything else is compared against the framed bit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- .../templates/justfiles/anvil/container.just | 30 +++---- crates/cargo-anvil/tests/recipe_contracts.rs | 83 ++++++++++--------- .../snapshots/snapshots__ado_backend.snap | 30 +++---- .../snapshots/snapshots__github_backend.snap | 30 +++---- .../snapshots/snapshots__local_only.snap | 30 +++---- justfiles/anvil/container.just | 30 +++---- 7 files changed, 124 insertions(+), 113 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 5ad6990e..c121991b 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:1a272c800ded68c3b92b3338c723186a6fe6bd95b5574003626aa6fdd8528e0e" +catalog_checksum = "sha256:8d2288ed5edbcc40cb671ef1e4fd80f3ed5dbac1447791627ba13bec80f33716" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:9a3f3f88995247c7aca129ae7fd621e78fefa4b6de66ec4ee77bbbbe75551c1b" +checksum = "sha256:13230f8617438e3079f53cb4262a7ca11cae5d35f9dde2ee399810204df578a0" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 28c713de..5e2bcfc4 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -297,24 +297,26 @@ anvil-container-tag: } } } - # `--raw` reports the index mode and the working-tree mode as the first - # two fields, so a mode-only change is visible even though the content - # is identical. On Windows core.fileMode is normally false and git - # reports no drift, which is correct: the filesystem has no bit to - # disagree with. + # `--raw` reports the working-tree mode as its second field, so a + # mode-only change is visible even though the content is identical. On + # Windows core.fileMode is normally false and git reports no drift, + # which is correct: the filesystem has no bit to disagree with. # - # A zero mode means the path is absent on that side -- an unstaged - # deletion, or an intent-to-add entry. Neither can make the image - # disagree with its tag, because a path missing from the build context - # is missing from the digest too, so only a transition between two real - # modes is drift. A `chmod` is always one of those. + # A zero working-tree mode is a deletion: the path is in neither the + # build context nor the digest, so there is nothing to disagree about. + # Every other entry is present in the context and is compared against + # the bit the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that bit -- an + # intent-to-add entry reports zero there while `ls-files` reports a real + # placeholder mode -- so comparing the two raw fields would both miss an + # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { - $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] - if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $worktreeMode, $driftPath = $Matches[1], $Matches[2] + if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." exit 1 } } diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 7e854fff..e2c11c84 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1259,12 +1259,17 @@ fn the_image_tag_follows_the_executable_bit() { assert_eq!(plain, tag("0"), "the tag must depend on the inputs alone"); } -/// The index is authoritative for the tag only while the working tree agrees -/// with it. A `chmod` that has not been staged is copied by the build and -/// missed by the digest, so the reference would name an image the build does -/// not produce; the run has to stop rather than absorb that. +/// The tag is computed from the index while the build copies the working tree, +/// so the two have to agree about the executable bit. Where they do not, the +/// reference names an image the build does not produce, and the run stops +/// rather than absorbing it. +/// +/// `git diff --raw` has three shapes here and only one of them is drift, so +/// each is pinned: an ordinary modification, a deletion (absent from both the +/// context and the digest), and an intent-to-add entry, whose raw index mode is +/// zero even though `ls-files --stage` reports a real placeholder mode. #[test] -fn an_unstaged_mode_change_stops_the_run() { +fn a_working_tree_mode_the_tag_did_not_frame_stops_the_run() { if !tools_available() { return; } @@ -1273,55 +1278,51 @@ fn an_unstaged_mode_change_stops_the_run() { write(&root.join("rust-toolchain.toml"), "[toolchain]\nchannel = \"stable\"\n"); write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); write(&root.join("justfiles/anvil/setup.sh"), "echo hello\n"); + // The digest frames this path from `ls-files --stage`, which reports + // 100644 in every case below -- including the intent-to-add ones, where + // the raw index mode is zero but the placeholder is a real mode. write( &root.join("fake-bin/git.ps1"), "if ($args -contains 'ls-files') {\n \ Write-Output \"100644 0000000000000000000000000000000000000000 0`tjustfiles/anvil/setup.sh\"\n}\n\ - if ($args -contains 'diff' -and $env:FAKE_DRIFT -eq '1') {\n \ - Write-Output \":100644 100755 0000000 0000000 M`tjustfiles/anvil/setup.sh\"\n}\nexit 0\n", + if ($args -contains 'diff' -and $env:FAKE_RAW) {\n Write-Output $env:FAKE_RAW\n}\nexit 0\n", ); - let clean = run_just(root, &["anvil-container-tag"], &[("FAKE_DRIFT", OsStr::new("0"))]); - assert!( - clean.status.success(), - "an agreeing working tree must compute a tag\nstderr:\n{}", - String::from_utf8_lossy(&clean.stderr) - ); + let tag = |raw: &str| run_just(root, &["anvil-container-tag"], &[("FAKE_RAW", OsStr::new(raw))]); - let drifted = run_just(root, &["anvil-container-tag"], &[("FAKE_DRIFT", OsStr::new("1"))]); - assert_failed(&drifted, "computing a tag against an unstaged mode change"); - let stderr = String::from_utf8_lossy(&drifted.stderr); - assert!( - stderr.contains("working tree") && stderr.contains("git add"), - "the refusal must name the drift and the recovery\nstderr:\n{stderr}" - ); - - // A zero mode means the path is absent on that side. `git diff --raw` - // encodes an unstaged deletion as `:100644 000000 ... D` and an - // intent-to-add as `:000000 100644 ... A`, and both differ numerically - // without being drift -- a path missing from the build context is missing - // from the digest too. Retiring a managed file and not yet staging the - // deletion is the ordinary way to reach this. for (kind, raw) in [ - ("an unstaged deletion", ":100644 000000 0000000 0000000 D`tjustfiles/anvil/setup.sh"), + ("no working-tree change at all", ""), + ("an unstaged deletion", ":100644 000000 0000000 0000000 D\tjustfiles/anvil/setup.sh"), + ( + "an intent-to-add entry that is not executable", + ":000000 100644 0000000 0000000 A\tjustfiles/anvil/setup.sh", + ), ( - "an intent-to-add entry", - ":000000 100644 0000000 0000000 A`tjustfiles/anvil/setup.sh", + "an edit that leaves the mode alone", + ":100644 100644 0000000 0000000 M\tjustfiles/anvil/setup.sh", ), ] { - write( - &root.join("fake-bin/git.ps1"), - &format!( - "if ($args -contains 'ls-files') {{\n \ - Write-Output \"100644 0000000000000000000000000000000000000000 0`tjustfiles/anvil/setup.sh\"\n}}\n\ - if ($args -contains 'diff') {{\n Write-Output \"{raw}\"\n}}\nexit 0\n" - ), + let output = tag(raw); + assert!( + output.status.success(), + "{kind} must not be reported as drift\nstderr:\n{}", + String::from_utf8_lossy(&output.stderr) ); - let absent = run_just(root, &["anvil-container-tag"], &[]); + } + + for (kind, raw) in [ + ("an unstaged chmod +x", ":100644 100755 0000000 0000000 M\tjustfiles/anvil/setup.sh"), + ( + "an intent-to-add entry that is executable", + ":000000 100755 0000000 0000000 A\tjustfiles/anvil/setup.sh", + ), + ] { + let output = tag(raw); + assert_failed(&output, kind); + let stderr = String::from_utf8_lossy(&output.stderr); assert!( - absent.status.success(), - "{kind} must not be reported as mode drift\nstderr:\n{}", - String::from_utf8_lossy(&absent.stderr) + stderr.contains("working tree") && stderr.contains("git add"), + "{kind} must name the drift and the recovery\nstderr:\n{stderr}" ); } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 8bff73c8..c649dee4 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3826,24 +3826,26 @@ anvil-container-tag: } } } - # `--raw` reports the index mode and the working-tree mode as the first - # two fields, so a mode-only change is visible even though the content - # is identical. On Windows core.fileMode is normally false and git - # reports no drift, which is correct: the filesystem has no bit to - # disagree with. + # `--raw` reports the working-tree mode as its second field, so a + # mode-only change is visible even though the content is identical. On + # Windows core.fileMode is normally false and git reports no drift, + # which is correct: the filesystem has no bit to disagree with. # - # A zero mode means the path is absent on that side -- an unstaged - # deletion, or an intent-to-add entry. Neither can make the image - # disagree with its tag, because a path missing from the build context - # is missing from the digest too, so only a transition between two real - # modes is drift. A `chmod` is always one of those. + # A zero working-tree mode is a deletion: the path is in neither the + # build context nor the digest, so there is nothing to disagree about. + # Every other entry is present in the context and is compared against + # the bit the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that bit -- an + # intent-to-add entry reports zero there while `ls-files` reports a real + # placeholder mode -- so comparing the two raw fields would both miss an + # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { - $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] - if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $worktreeMode, $driftPath = $Matches[1], $Matches[2] + if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." exit 1 } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 2940aa8e..c619f426 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3705,24 +3705,26 @@ anvil-container-tag: } } } - # `--raw` reports the index mode and the working-tree mode as the first - # two fields, so a mode-only change is visible even though the content - # is identical. On Windows core.fileMode is normally false and git - # reports no drift, which is correct: the filesystem has no bit to - # disagree with. + # `--raw` reports the working-tree mode as its second field, so a + # mode-only change is visible even though the content is identical. On + # Windows core.fileMode is normally false and git reports no drift, + # which is correct: the filesystem has no bit to disagree with. # - # A zero mode means the path is absent on that side -- an unstaged - # deletion, or an intent-to-add entry. Neither can make the image - # disagree with its tag, because a path missing from the build context - # is missing from the digest too, so only a transition between two real - # modes is drift. A `chmod` is always one of those. + # A zero working-tree mode is a deletion: the path is in neither the + # build context nor the digest, so there is nothing to disagree about. + # Every other entry is present in the context and is compared against + # the bit the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that bit -- an + # intent-to-add entry reports zero there while `ls-files` reports a real + # placeholder mode -- so comparing the two raw fields would both miss an + # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { - $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] - if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $worktreeMode, $driftPath = $Matches[1], $Matches[2] + if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." exit 1 } } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index a7b9f3b5..f57f92de 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2578,24 +2578,26 @@ anvil-container-tag: } } } - # `--raw` reports the index mode and the working-tree mode as the first - # two fields, so a mode-only change is visible even though the content - # is identical. On Windows core.fileMode is normally false and git - # reports no drift, which is correct: the filesystem has no bit to - # disagree with. + # `--raw` reports the working-tree mode as its second field, so a + # mode-only change is visible even though the content is identical. On + # Windows core.fileMode is normally false and git reports no drift, + # which is correct: the filesystem has no bit to disagree with. # - # A zero mode means the path is absent on that side -- an unstaged - # deletion, or an intent-to-add entry. Neither can make the image - # disagree with its tag, because a path missing from the build context - # is missing from the digest too, so only a transition between two real - # modes is drift. A `chmod` is always one of those. + # A zero working-tree mode is a deletion: the path is in neither the + # build context nor the digest, so there is nothing to disagree about. + # Every other entry is present in the context and is compared against + # the bit the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that bit -- an + # intent-to-add entry reports zero there while `ls-files` reports a real + # placeholder mode -- so comparing the two raw fields would both miss an + # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { - $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] - if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $worktreeMode, $driftPath = $Matches[1], $Matches[2] + if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." exit 1 } } diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 28c713de..5e2bcfc4 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -297,24 +297,26 @@ anvil-container-tag: } } } - # `--raw` reports the index mode and the working-tree mode as the first - # two fields, so a mode-only change is visible even though the content - # is identical. On Windows core.fileMode is normally false and git - # reports no drift, which is correct: the filesystem has no bit to - # disagree with. + # `--raw` reports the working-tree mode as its second field, so a + # mode-only change is visible even though the content is identical. On + # Windows core.fileMode is normally false and git reports no drift, + # which is correct: the filesystem has no bit to disagree with. # - # A zero mode means the path is absent on that side -- an unstaged - # deletion, or an intent-to-add entry. Neither can make the image - # disagree with its tag, because a path missing from the build context - # is missing from the digest too, so only a transition between two real - # modes is drift. A `chmod` is always one of those. + # A zero working-tree mode is a deletion: the path is in neither the + # build context nor the digest, so there is nothing to disagree about. + # Every other entry is present in the context and is compared against + # the bit the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that bit -- an + # intent-to-add entry reports zero there while `ls-files` reports a real + # placeholder mode -- so comparing the two raw fields would both miss an + # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { - if ($entry -match '^:(\d{6}) (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { - $indexMode, $worktreeMode, $driftPath = $Matches[1], $Matches[2], $Matches[3] - if ($indexMode -ne '000000' -and $worktreeMode -ne '000000' -and $indexMode -ne $worktreeMode) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree and $indexMode in the index. The build copies the working tree while the image tag is computed from the index, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { + $worktreeMode, $driftPath = $Matches[1], $Matches[2] + if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." exit 1 } } From 539f290661f54a633405621bd1d7dce5b2e27a50 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Sun, 30 Aug 2026 22:39:57 +0200 Subject: [PATCH 79/81] fix(cargo-anvil): frame the whole git mode in the image tag, not one bit COPY preserves a symlink as a symlink while the digest walk reads through it, so replacing a regular file with a link to identical bytes changed the image object without renaming the tag. The executable bit was the only part of the mode being framed, and the drift guard compared the same single bit, so a 100644 to 120000 transition passed both. The digest now frames the index mode itself and the guard compares against that mode, which subsumes the executable bit and covers every regular-file transition git can record. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- .../templates/justfiles/anvil/container.just | 38 +++++++++++-------- crates/cargo-anvil/tests/recipe_contracts.rs | 9 ++++- .../snapshots/snapshots__ado_backend.snap | 38 +++++++++++-------- .../snapshots/snapshots__github_backend.snap | 38 +++++++++++-------- .../snapshots/snapshots__local_only.snap | 38 +++++++++++-------- justfiles/anvil/container.just | 38 +++++++++++-------- 7 files changed, 120 insertions(+), 83 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index c121991b..be2b8e10 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:8d2288ed5edbcc40cb671ef1e4fd80f3ed5dbac1447791627ba13bec80f33716" +catalog_checksum = "sha256:62817f1937e28f52b6bca24ea5db27ea0dbf2ba282a9a82089f60e2a7a49dd7e" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:13230f8617438e3079f53cb4262a7ca11cae5d35f9dde2ee399810204df578a0" +checksum = "sha256:10efc817c80d0ad15a16a3bb5becb79df4e4d2ec928a643f58f610cab79bbd03" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 5e2bcfc4..8e07ca9b 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -268,12 +268,14 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) - # The executable bit is part of what `COPY` puts in the image, so a `chmod - # +x` that changes no content still changes the image and must rename the - # tag. Git's index is the only source of that bit which answers identically - # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag, and - # a published image would stop resolving for half the people who use it. + # A file's git mode is part of what `COPY` puts in the image -- the + # executable bit, and whether the entry is a regular file or a symlink -- so + # a change that leaves the bytes alone still changes the image and must + # rename the tag. Git's index is the only source of that mode which answers + # identically on every platform: Windows has no executable bit, so reading + # it from the filesystem would make two checkouts of one commit disagree on + # the tag, and a published image would stop resolving for half the people + # who use it. # # The index is authoritative only while the working tree agrees with it. A # mode change that has not been staged would be copied by the build and @@ -284,16 +286,16 @@ anvil-container-tag: # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than - # backslash-escaped, which would key the set on a spelling the walk never + # backslash-escaped, which would key the map on a spelling the walk never # produces. - $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $indexMode = @{} $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { - if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { - [void]$executable.Add($Matches[1]) + if ($entry -match '^(\d{6}) [0-9a-f]+ \d+\t(.+)$') { + $indexMode[$Matches[2]] = $Matches[1] } } } @@ -305,18 +307,18 @@ anvil-container-tag: # A zero working-tree mode is a deletion: the path is in neither the # build context nor the digest, so there is nothing to disagree about. # Every other entry is present in the context and is compared against - # the bit the digest actually framed, which comes from `ls-files - # --stage` above. The raw index-side mode is not that bit -- an + # the mode the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that mode -- an # intent-to-add entry reports zero there while `ls-files` reports a real - # placeholder mode -- so comparing the two raw fields would both miss an + # placeholder -- so comparing the two raw fields would both miss an # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { $worktreeMode, $driftPath = $Matches[1], $Matches[2] - if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($worktreeMode -ne '000000' -and $indexMode.ContainsKey($driftPath) -and $indexMode[$driftPath] -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, but the image tag was computed from mode $($indexMode[$driftPath]). The build copies the working tree, so the image would not match its own reference. Stage the change (git add) and re-run." exit 1 } } @@ -341,7 +343,11 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } - $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } + # The whole git mode, not just the executable bit: `COPY` preserves + # a symlink as a symlink, while the walk above reads through it, so + # replacing a regular file with a link to identical bytes would + # otherwise keep the tag. An untracked path has no framed mode. + $mode = if ($indexMode.ContainsKey($rel)) { $indexMode[$rel] } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index e2c11c84..7edafd34 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1316,12 +1316,19 @@ fn a_working_tree_mode_the_tag_did_not_frame_stops_the_run() { "an intent-to-add entry that is executable", ":000000 100755 0000000 0000000 A\tjustfiles/anvil/setup.sh", ), + ( + "a regular file replaced by a symlink", + ":100644 120000 0000000 0000000 T\tjustfiles/anvil/setup.sh", + ), ] { let output = tag(raw); assert_failed(&output, kind); let stderr = String::from_utf8_lossy(&output.stderr); + // PowerShell wraps an error record and decorates each continuation, so + // a multi-word phrase is not a substring of what reaches stderr. It + // wraps at spaces though, so individual words survive intact. assert!( - stderr.contains("working tree") && stderr.contains("git add"), + stderr.contains("working") && stderr.contains("Stage"), "{kind} must name the drift and the recovery\nstderr:\n{stderr}" ); } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index c649dee4..77c771ad 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3797,12 +3797,14 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) - # The executable bit is part of what `COPY` puts in the image, so a `chmod - # +x` that changes no content still changes the image and must rename the - # tag. Git's index is the only source of that bit which answers identically - # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag, and - # a published image would stop resolving for half the people who use it. + # A file's git mode is part of what `COPY` puts in the image -- the + # executable bit, and whether the entry is a regular file or a symlink -- so + # a change that leaves the bytes alone still changes the image and must + # rename the tag. Git's index is the only source of that mode which answers + # identically on every platform: Windows has no executable bit, so reading + # it from the filesystem would make two checkouts of one commit disagree on + # the tag, and a published image would stop resolving for half the people + # who use it. # # The index is authoritative only while the working tree agrees with it. A # mode change that has not been staged would be copied by the build and @@ -3813,16 +3815,16 @@ anvil-container-tag: # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than - # backslash-escaped, which would key the set on a spelling the walk never + # backslash-escaped, which would key the map on a spelling the walk never # produces. - $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $indexMode = @{} $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { - if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { - [void]$executable.Add($Matches[1]) + if ($entry -match '^(\d{6}) [0-9a-f]+ \d+\t(.+)$') { + $indexMode[$Matches[2]] = $Matches[1] } } } @@ -3834,18 +3836,18 @@ anvil-container-tag: # A zero working-tree mode is a deletion: the path is in neither the # build context nor the digest, so there is nothing to disagree about. # Every other entry is present in the context and is compared against - # the bit the digest actually framed, which comes from `ls-files - # --stage` above. The raw index-side mode is not that bit -- an + # the mode the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that mode -- an # intent-to-add entry reports zero there while `ls-files` reports a real - # placeholder mode -- so comparing the two raw fields would both miss an + # placeholder -- so comparing the two raw fields would both miss an # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { $worktreeMode, $driftPath = $Matches[1], $Matches[2] - if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($worktreeMode -ne '000000' -and $indexMode.ContainsKey($driftPath) -and $indexMode[$driftPath] -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, but the image tag was computed from mode $($indexMode[$driftPath]). The build copies the working tree, so the image would not match its own reference. Stage the change (git add) and re-run." exit 1 } } @@ -3870,7 +3872,11 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } - $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } + # The whole git mode, not just the executable bit: `COPY` preserves + # a symlink as a symlink, while the walk above reads through it, so + # replacing a regular file with a link to identical bytes would + # otherwise keep the tag. An untracked path has no framed mode. + $mode = if ($indexMode.ContainsKey($rel)) { $indexMode[$rel] } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index c619f426..781650c6 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3676,12 +3676,14 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) - # The executable bit is part of what `COPY` puts in the image, so a `chmod - # +x` that changes no content still changes the image and must rename the - # tag. Git's index is the only source of that bit which answers identically - # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag, and - # a published image would stop resolving for half the people who use it. + # A file's git mode is part of what `COPY` puts in the image -- the + # executable bit, and whether the entry is a regular file or a symlink -- so + # a change that leaves the bytes alone still changes the image and must + # rename the tag. Git's index is the only source of that mode which answers + # identically on every platform: Windows has no executable bit, so reading + # it from the filesystem would make two checkouts of one commit disagree on + # the tag, and a published image would stop resolving for half the people + # who use it. # # The index is authoritative only while the working tree agrees with it. A # mode change that has not been staged would be copied by the build and @@ -3692,16 +3694,16 @@ anvil-container-tag: # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than - # backslash-escaped, which would key the set on a spelling the walk never + # backslash-escaped, which would key the map on a spelling the walk never # produces. - $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $indexMode = @{} $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { - if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { - [void]$executable.Add($Matches[1]) + if ($entry -match '^(\d{6}) [0-9a-f]+ \d+\t(.+)$') { + $indexMode[$Matches[2]] = $Matches[1] } } } @@ -3713,18 +3715,18 @@ anvil-container-tag: # A zero working-tree mode is a deletion: the path is in neither the # build context nor the digest, so there is nothing to disagree about. # Every other entry is present in the context and is compared against - # the bit the digest actually framed, which comes from `ls-files - # --stage` above. The raw index-side mode is not that bit -- an + # the mode the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that mode -- an # intent-to-add entry reports zero there while `ls-files` reports a real - # placeholder mode -- so comparing the two raw fields would both miss an + # placeholder -- so comparing the two raw fields would both miss an # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { $worktreeMode, $driftPath = $Matches[1], $Matches[2] - if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($worktreeMode -ne '000000' -and $indexMode.ContainsKey($driftPath) -and $indexMode[$driftPath] -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, but the image tag was computed from mode $($indexMode[$driftPath]). The build copies the working tree, so the image would not match its own reference. Stage the change (git add) and re-run." exit 1 } } @@ -3749,7 +3751,11 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } - $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } + # The whole git mode, not just the executable bit: `COPY` preserves + # a symlink as a symlink, while the walk above reads through it, so + # replacing a regular file with a link to identical bytes would + # otherwise keep the tag. An untracked path has no framed mode. + $mode = if ($indexMode.ContainsKey($rel)) { $indexMode[$rel] } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index f57f92de..aeb5d7e3 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2549,12 +2549,14 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) - # The executable bit is part of what `COPY` puts in the image, so a `chmod - # +x` that changes no content still changes the image and must rename the - # tag. Git's index is the only source of that bit which answers identically - # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag, and - # a published image would stop resolving for half the people who use it. + # A file's git mode is part of what `COPY` puts in the image -- the + # executable bit, and whether the entry is a regular file or a symlink -- so + # a change that leaves the bytes alone still changes the image and must + # rename the tag. Git's index is the only source of that mode which answers + # identically on every platform: Windows has no executable bit, so reading + # it from the filesystem would make two checkouts of one commit disagree on + # the tag, and a published image would stop resolving for half the people + # who use it. # # The index is authoritative only while the working tree agrees with it. A # mode change that has not been staged would be copied by the build and @@ -2565,16 +2567,16 @@ anvil-container-tag: # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than - # backslash-escaped, which would key the set on a spelling the walk never + # backslash-escaped, which would key the map on a spelling the walk never # produces. - $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $indexMode = @{} $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { - if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { - [void]$executable.Add($Matches[1]) + if ($entry -match '^(\d{6}) [0-9a-f]+ \d+\t(.+)$') { + $indexMode[$Matches[2]] = $Matches[1] } } } @@ -2586,18 +2588,18 @@ anvil-container-tag: # A zero working-tree mode is a deletion: the path is in neither the # build context nor the digest, so there is nothing to disagree about. # Every other entry is present in the context and is compared against - # the bit the digest actually framed, which comes from `ls-files - # --stage` above. The raw index-side mode is not that bit -- an + # the mode the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that mode -- an # intent-to-add entry reports zero there while `ls-files` reports a real - # placeholder mode -- so comparing the two raw fields would both miss an + # placeholder -- so comparing the two raw fields would both miss an # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { $worktreeMode, $driftPath = $Matches[1], $Matches[2] - if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($worktreeMode -ne '000000' -and $indexMode.ContainsKey($driftPath) -and $indexMode[$driftPath] -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, but the image tag was computed from mode $($indexMode[$driftPath]). The build copies the working tree, so the image would not match its own reference. Stage the change (git add) and re-run." exit 1 } } @@ -2622,7 +2624,11 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } - $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } + # The whole git mode, not just the executable bit: `COPY` preserves + # a symlink as a symlink, while the walk above reads through it, so + # replacing a regular file with a link to identical bytes would + # otherwise keep the tag. An untracked path has no framed mode. + $mode = if ($indexMode.ContainsKey($rel)) { $indexMode[$rel] } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 5e2bcfc4..8e07ca9b 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -268,12 +268,14 @@ anvil-container-tag: # same collision from the other direction. $ordered = [System.Collections.Generic.SortedSet[string]]::new( [string[]]$inputs, [System.StringComparer]::Ordinal) - # The executable bit is part of what `COPY` puts in the image, so a `chmod - # +x` that changes no content still changes the image and must rename the - # tag. Git's index is the only source of that bit which answers identically - # on every platform: Windows has no such bit, so reading it from the - # filesystem would make two checkouts of one commit disagree on the tag, and - # a published image would stop resolving for half the people who use it. + # A file's git mode is part of what `COPY` puts in the image -- the + # executable bit, and whether the entry is a regular file or a symlink -- so + # a change that leaves the bytes alone still changes the image and must + # rename the tag. Git's index is the only source of that mode which answers + # identically on every platform: Windows has no executable bit, so reading + # it from the filesystem would make two checkouts of one commit disagree on + # the tag, and a published image would stop resolving for half the people + # who use it. # # The index is authoritative only while the working tree agrees with it. A # mode change that has not been staged would be copied by the build and @@ -284,16 +286,16 @@ anvil-container-tag: # to encode. # # quotePath=false so a non-ASCII path arrives verbatim rather than - # backslash-escaped, which would key the set on a spelling the walk never + # backslash-escaped, which would key the map on a spelling the walk never # produces. - $executable = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) + $indexMode = @{} $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $staged) { - if ($entry -match '^100755 [0-9a-f]+ \d+\t(.+)$') { - [void]$executable.Add($Matches[1]) + if ($entry -match '^(\d{6}) [0-9a-f]+ \d+\t(.+)$') { + $indexMode[$Matches[2]] = $Matches[1] } } } @@ -305,18 +307,18 @@ anvil-container-tag: # A zero working-tree mode is a deletion: the path is in neither the # build context nor the digest, so there is nothing to disagree about. # Every other entry is present in the context and is compared against - # the bit the digest actually framed, which comes from `ls-files - # --stage` above. The raw index-side mode is not that bit -- an + # the mode the digest actually framed, which comes from `ls-files + # --stage` above. The raw index-side mode is not that mode -- an # intent-to-add entry reports zero there while `ls-files` reports a real - # placeholder mode -- so comparing the two raw fields would both miss an + # placeholder -- so comparing the two raw fields would both miss an # executable `git add -N` file and reject an ordinary one. $drift = & git -c core.quotePath=false -C $repoRoot diff --no-ext-diff --raw -- @tracked 2>$null if ($LASTEXITCODE -eq 0) { foreach ($entry in $drift) { if ($entry -match '^:\d{6} (\d{6}) [0-9a-f]+ [0-9a-f]+ \S+\t(.+)$') { $worktreeMode, $driftPath = $Matches[1], $Matches[2] - if ($worktreeMode -ne '000000' -and $executable.Contains($driftPath) -ne ($worktreeMode -eq '100755')) { - Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, which is not the mode the image tag was computed from. The build copies the working tree, so the image would not match its own reference. Stage the mode change (git add) and re-run." + if ($worktreeMode -ne '000000' -and $indexMode.ContainsKey($driftPath) -and $indexMode[$driftPath] -ne $worktreeMode) { + Write-Error "anvil: '$driftPath' has mode $worktreeMode in the working tree, but the image tag was computed from mode $($indexMode[$driftPath]). The build copies the working tree, so the image would not match its own reference. Stage the change (git add) and re-run." exit 1 } } @@ -341,7 +343,11 @@ anvil-container-tag: } else { $content = [System.IO.File]::ReadAllBytes($path) } - $mode = if ($executable.Contains($rel)) { 'x' } else { '-' } + # The whole git mode, not just the executable bit: `COPY` preserves + # a symlink as a symlink, while the walk above reads through it, so + # replacing a regular file with a link to identical bytes would + # otherwise keep the tag. An untracked path has no framed mode. + $mode = if ($indexMode.ContainsKey($rel)) { $indexMode[$rel] } else { '-' } $header = [System.Text.Encoding]::UTF8.GetBytes( 'file ' + [System.Text.Encoding]::UTF8.GetByteCount($rel) + ' ' + $rel + ' ' + $mode + ' ' + $content.Length + ' ') [void]$sha.TransformBlock($header, 0, $header.Length, $null, 0) From f99bd39190afc87f80eb65ddcd82d0d5f380ae9c Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Mon, 31 Aug 2026 00:41:02 +0200 Subject: [PATCH 80/81] fix(cargo-anvil): refuse a link among the image inputs, and key the mode map ordinally The digest walk used -File, which hides a link to a directory entirely, so a repository-owned COPY could carry one into the image while the tag never saw it. A link to a file was worse than invisible: it was read through, so the digest hashed the target's bytes while the engine copied the link, and a retarget changed the image without changing anything the walk could see. Framing the link text instead would have to work on Windows, where git materializes a symlink as an ordinary file unless the checkout was privileged, so one commit would digest differently per platform. Anvil creates no link under these trees, so the whole class is refused with the paths named. The mode map was a PowerShell hashtable, which folds case, while every collection around it is ordinal for exactly the reason git records here: two paths differing only in case would collapse and both be framed with whichever mode was stored last. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +- .../src/anvil/artifacts/container.rs | 12 +++- .../templates/justfiles/anvil/container.just | 35 +++++++++-- crates/cargo-anvil/tests/recipe_contracts.rs | 63 +++++++++++++++++++ .../snapshots/snapshots__ado_backend.snap | 35 +++++++++-- .../snapshots/snapshots__github_backend.snap | 35 +++++++++-- .../snapshots/snapshots__local_only.snap | 35 +++++++++-- justfiles/anvil/container.just | 35 +++++++++-- 8 files changed, 224 insertions(+), 30 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index be2b8e10..9f2d27e4 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:62817f1937e28f52b6bca24ea5db27ea0dbf2ba282a9a82089f60e2a7a49dd7e" +catalog_checksum = "sha256:0cdc61db9e974a45898d27d2f2a764599e8e08fa90b78cfeb037f5193204aa2b" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:10efc817c80d0ad15a16a3bb5becb79df4e4d2ec928a643f58f610cab79bbd03" +checksum = "sha256:f2abdfa0853d3e259ce445e7fe3f9e4c5f022124b04406ca0cf9e793ce7e929c" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/src/anvil/artifacts/container.rs b/crates/cargo-anvil/src/anvil/artifacts/container.rs index 47f179fc..b26e7bb0 100644 --- a/crates/cargo-anvil/src/anvil/artifacts/container.rs +++ b/crates/cargo-anvil/src/anvil/artifacts/container.rs @@ -667,7 +667,7 @@ mod tests { .find("$containerRoot = Join-Path $repoRoot '.anvil/container'") .expect("the tag must walk the container directory"); assert!( - RECIPE[walk..].contains("Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force"), + RECIPE[walk..].contains("Get-ChildItem -LiteralPath $containerRoot -Recurse -Force"), "the walk must be recursive and include hidden entries" ); // A missing Dockerfile must still be fatal: the walk alone would let @@ -783,8 +783,14 @@ mod tests { // into the image. Filtering here would let it change the image's // contents without changing its tag. -Force because a dot-prefixed // file is copied like any other and would otherwise be skipped. - assert!(RECIPE.contains("-Recurse -File -Force")); - assert!(!RECIPE.contains("-Recurse -File -Force -Filter '*.just'")); + // Directories are enumerated too, because `-File` hides a link to one + // and a link is refused rather than digested. + assert!(RECIPE.contains("-Recurse -Force")); + assert!(!RECIPE.contains("-Recurse -Force -Filter '*.just'")); + assert!( + !RECIPE.contains("-Recurse -File -Force"), + "a link to a directory is invisible to a file-only walk" + ); // ComputeHash, not the static HashData: the latter needs .NET 5, and // the prerequisite check accepts PowerShell 7.0 on .NET Core 3.1. assert!(!RECIPE.contains("SHA256]::HashData")); diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index 8e07ca9b..f45a49ba 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -167,6 +167,7 @@ anvil-container-tag: $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') + $links = @() # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when @@ -197,9 +198,12 @@ anvil-container-tag: # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } # Every generated recipe file. The image installs its tools by running @@ -231,12 +235,29 @@ anvil-container-tag: # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } + # A symlink is refused rather than digested. The engine copies the link + # itself while any reading of it here follows it, so a retarget changes the + # image without changing a single byte the walk can see, and a link to a + # directory is not enumerated by the walk at all. Framing link text instead + # would have to work on Windows, where git materializes a symlink as an + # ordinary file unless the checkout was privileged, so the same commit would + # digest differently per platform. Anvil never creates one under these + # trees, so refusing costs nothing and closes the whole class. + if ($links.Count -gt 0) { + $named = ($links | ForEach-Object { [System.IO.Path]::GetRelativePath($repoRoot, $_) -replace '\\', '/' }) -join ', ' + Write-Error "anvil: the container image inputs must be regular files, but these are links: $named. The engine copies a link as a link while the image tag is computed from what it points at, so the image would not match its own reference. Replace them with regular files." + exit 1 + } + # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: @@ -288,7 +309,11 @@ anvil-container-tag: # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the map on a spelling the walk never # produces. - $indexMode = @{} + # Ordinal, like the sort and the dedup below: PowerShell's `@{}` folds case, + # so two paths differing only in case -- which git permits and the + # case-sensitive filesystem the image is built on can hold -- would collapse + # to one entry and both would be framed with whichever mode was stored last. + $indexMode = [System.Collections.Generic.Dictionary[string, string]]::new([System.StringComparer]::Ordinal) $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 7edafd34..8d28c7b2 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1333,3 +1333,66 @@ fn a_working_tree_mode_the_tag_did_not_frame_stops_the_run() { ); } } + +/// The engine copies a link as a link, while any read of one here follows it, +/// so a retarget changes the image without changing a byte the walk can see -- +/// and a link to a directory is not enumerated by the walk at all. Framing the +/// link text instead would have to work on Windows, where git materializes a +/// symlink as an ordinary file unless the checkout was privileged, so the same +/// commit would digest differently per platform. Anvil creates no link under +/// these trees, so the whole class is refused. +#[test] +fn a_link_among_the_image_inputs_is_refused() { + if !tools_available() { + return; + } + + for (kind, name, links_a_directory) in [("a file link", "linked.just", false), ("a directory link", "linked", true)] { + let tmp = fixture(&[("container.just", CONTAINER)], &[]); + let root = tmp.path(); + write(&root.join("rust-toolchain.toml"), "[toolchain]\nchannel = \"stable\"\n"); + write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); + write(&root.join("justfiles/anvil/mod.just"), "# recipes\n"); + write(&root.join("elsewhere/target.just"), "# shared\n"); + + let link = root.join("justfiles/anvil").join(name); + let created = if links_a_directory { + symlink_dir(&root.join("elsewhere"), &link) + } else { + symlink_file(&root.join("elsewhere/target.just"), &link) + }; + // Creating a link needs a privilege that not every environment grants. + // Where it is refused there is nothing to assert about. + if created.is_err() { + continue; + } + + let output = run_just(root, &["anvil-container-tag"], &[]); + assert_failed(&output, &format!("computing a tag with {kind} among the inputs")); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains("regular") && stderr.contains(name), + "{kind} must be named in the refusal\nstderr:\n{stderr}" + ); + } +} + +#[cfg(windows)] +fn symlink_file(target: &Path, link: &Path) -> std::io::Result<()> { + std::os::windows::fs::symlink_file(target, link) +} + +#[cfg(windows)] +fn symlink_dir(target: &Path, link: &Path) -> std::io::Result<()> { + std::os::windows::fs::symlink_dir(target, link) +} + +#[cfg(unix)] +fn symlink_file(target: &Path, link: &Path) -> std::io::Result<()> { + std::os::unix::fs::symlink(target, link) +} + +#[cfg(unix)] +fn symlink_dir(target: &Path, link: &Path) -> std::io::Result<()> { + std::os::unix::fs::symlink(target, link) +} diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 77c771ad..5d4b0e6f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3696,6 +3696,7 @@ anvil-container-tag: $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') + $links = @() # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when @@ -3726,9 +3727,12 @@ anvil-container-tag: # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } # Every generated recipe file. The image installs its tools by running @@ -3760,12 +3764,29 @@ anvil-container-tag: # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } + # A symlink is refused rather than digested. The engine copies the link + # itself while any reading of it here follows it, so a retarget changes the + # image without changing a single byte the walk can see, and a link to a + # directory is not enumerated by the walk at all. Framing link text instead + # would have to work on Windows, where git materializes a symlink as an + # ordinary file unless the checkout was privileged, so the same commit would + # digest differently per platform. Anvil never creates one under these + # trees, so refusing costs nothing and closes the whole class. + if ($links.Count -gt 0) { + $named = ($links | ForEach-Object { [System.IO.Path]::GetRelativePath($repoRoot, $_) -replace '\\', '/' }) -join ', ' + Write-Error "anvil: the container image inputs must be regular files, but these are links: $named. The engine copies a link as a link while the image tag is computed from what it points at, so the image would not match its own reference. Replace them with regular files." + exit 1 + } + # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: @@ -3817,7 +3838,11 @@ anvil-container-tag: # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the map on a spelling the walk never # produces. - $indexMode = @{} + # Ordinal, like the sort and the dedup below: PowerShell's `@{}` folds case, + # so two paths differing only in case -- which git permits and the + # case-sensitive filesystem the image is built on can hold -- would collapse + # to one entry and both would be framed with whichever mode was stored last. + $indexMode = [System.Collections.Generic.Dictionary[string, string]]::new([System.StringComparer]::Ordinal) $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 781650c6..075f971f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3575,6 +3575,7 @@ anvil-container-tag: $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') + $links = @() # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when @@ -3605,9 +3606,12 @@ anvil-container-tag: # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } # Every generated recipe file. The image installs its tools by running @@ -3639,12 +3643,29 @@ anvil-container-tag: # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } + # A symlink is refused rather than digested. The engine copies the link + # itself while any reading of it here follows it, so a retarget changes the + # image without changing a single byte the walk can see, and a link to a + # directory is not enumerated by the walk at all. Framing link text instead + # would have to work on Windows, where git materializes a symlink as an + # ordinary file unless the checkout was privileged, so the same commit would + # digest differently per platform. Anvil never creates one under these + # trees, so refusing costs nothing and closes the whole class. + if ($links.Count -gt 0) { + $named = ($links | ForEach-Object { [System.IO.Path]::GetRelativePath($repoRoot, $_) -replace '\\', '/' }) -join ', ' + Write-Error "anvil: the container image inputs must be regular files, but these are links: $named. The engine copies a link as a link while the image tag is computed from what it points at, so the image would not match its own reference. Replace them with regular files." + exit 1 + } + # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: @@ -3696,7 +3717,11 @@ anvil-container-tag: # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the map on a spelling the walk never # produces. - $indexMode = @{} + # Ordinal, like the sort and the dedup below: PowerShell's `@{}` folds case, + # so two paths differing only in case -- which git permits and the + # case-sensitive filesystem the image is built on can hold -- would collapse + # to one entry and both would be framed with whichever mode was stored last. + $indexMode = [System.Collections.Generic.Dictionary[string, string]]::new([System.StringComparer]::Ordinal) $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index aeb5d7e3..f3c56688 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2448,6 +2448,7 @@ anvil-container-tag: $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') + $links = @() # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when @@ -2478,9 +2479,12 @@ anvil-container-tag: # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } # Every generated recipe file. The image installs its tools by running @@ -2512,12 +2516,29 @@ anvil-container-tag: # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } + # A symlink is refused rather than digested. The engine copies the link + # itself while any reading of it here follows it, so a retarget changes the + # image without changing a single byte the walk can see, and a link to a + # directory is not enumerated by the walk at all. Framing link text instead + # would have to work on Windows, where git materializes a symlink as an + # ordinary file unless the checkout was privileged, so the same commit would + # digest differently per platform. Anvil never creates one under these + # trees, so refusing costs nothing and closes the whole class. + if ($links.Count -gt 0) { + $named = ($links | ForEach-Object { [System.IO.Path]::GetRelativePath($repoRoot, $_) -replace '\\', '/' }) -join ', ' + Write-Error "anvil: the container image inputs must be regular files, but these are links: $named. The engine copies a link as a link while the image tag is computed from what it points at, so the image would not match its own reference. Replace them with regular files." + exit 1 + } + # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: @@ -2569,7 +2590,11 @@ anvil-container-tag: # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the map on a spelling the walk never # produces. - $indexMode = @{} + # Ordinal, like the sort and the dedup below: PowerShell's `@{}` folds case, + # so two paths differing only in case -- which git permits and the + # case-sensitive filesystem the image is built on can hold -- would collapse + # to one entry and both would be framed with whichever mode was stored last. + $indexMode = [System.Collections.Generic.Dictionary[string, string]]::new([System.StringComparer]::Ordinal) $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index 8e07ca9b..f45a49ba 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -167,6 +167,7 @@ anvil-container-tag: $hookRel = '.anvil/container/hooks.ps1' $inputs = @('rust-toolchain.toml') + $links = @() # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when @@ -197,9 +198,12 @@ anvil-container-tag: # on the tag and a published image would stop resolving. $containerRoot = Join-Path $repoRoot '.anvil/container' if (Test-Path -LiteralPath $containerRoot) { - foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $containerRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } # Every generated recipe file. The image installs its tools by running @@ -231,12 +235,29 @@ anvil-container-tag: # repository that customizes it gets the proposal written right here. $recipeRoot = Join-Path $repoRoot 'justfiles/anvil' if (Test-Path -LiteralPath $recipeRoot) { - foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -File -Force | + foreach ($file in Get-ChildItem -LiteralPath $recipeRoot -Recurse -Force | Where-Object { -not $_.Name.EndsWith('.anvil-proposed') }) { - $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + if ($file.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { $links += $file.FullName } + elseif (-not $file.PSIsContainer) { + $inputs += [System.IO.Path]::GetRelativePath($repoRoot, $file.FullName) -replace '\\', '/' + } } } + # A symlink is refused rather than digested. The engine copies the link + # itself while any reading of it here follows it, so a retarget changes the + # image without changing a single byte the walk can see, and a link to a + # directory is not enumerated by the walk at all. Framing link text instead + # would have to work on Windows, where git materializes a symlink as an + # ordinary file unless the checkout was privileged, so the same commit would + # digest differently per platform. Anvil never creates one under these + # trees, so refusing costs nothing and closes the whole class. + if ($links.Count -gt 0) { + $named = ($links | ForEach-Object { [System.IO.Path]::GetRelativePath($repoRoot, $_) -replace '\\', '/' }) -join ', ' + Write-Error "anvil: the container image inputs must be regular files, but these are links: $named. The engine copies a link as a link while the image tag is computed from what it points at, so the image would not match its own reference. Replace them with regular files." + exit 1 + } + # Hash a tagged stream rather than raw concatenation, so no rearrangement of # names and contents can collide. Line endings are normalized once, here, so # a CRLF checkout and an LF checkout agree on the tag. Ordinal sort and dedup: @@ -288,7 +309,11 @@ anvil-container-tag: # quotePath=false so a non-ASCII path arrives verbatim rather than # backslash-escaped, which would key the map on a spelling the walk never # produces. - $indexMode = @{} + # Ordinal, like the sort and the dedup below: PowerShell's `@{}` folds case, + # so two paths differing only in case -- which git permits and the + # case-sensitive filesystem the image is built on can hold -- would collapse + # to one entry and both would be framed with whichever mode was stored last. + $indexMode = [System.Collections.Generic.Dictionary[string, string]]::new([System.StringComparer]::Ordinal) $tracked = @('.anvil/container', 'justfiles', 'rust-toolchain.toml') if (Get-Command git -ErrorAction SilentlyContinue) { $staged = & git -c core.quotePath=false -C $repoRoot ls-files --stage -- @tracked 2>$null From 156b021dbbcd6817bff53f92f098947528fe1ed4 Mon Sep 17 00:00:00 2001 From: Martin Havelka Date: Mon, 31 Aug 2026 01:47:10 +0200 Subject: [PATCH 81/81] fix(cargo-anvil): check the declared input and the walk roots for links, not only their descendants A walk reports descendants, so a link that is itself rust-toolchain.toml, .anvil/container or justfiles/anvil was followed and never appeared in its own output. The digest then read through it while the engine copies the link, so retargeting it between byte-identical trees left the framed path, mode and content unchanged and reused the image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8e2a0c6a-d500-4acf-a91b-b443c029e866 --- .anvil.lock | 4 +-- .../templates/justfiles/anvil/container.just | 10 ++++++ crates/cargo-anvil/tests/recipe_contracts.rs | 34 +++++++++++++++---- .../snapshots/snapshots__ado_backend.snap | 10 ++++++ .../snapshots/snapshots__github_backend.snap | 10 ++++++ .../snapshots/snapshots__local_only.snap | 10 ++++++ justfiles/anvil/container.just | 10 ++++++ 7 files changed, 80 insertions(+), 8 deletions(-) diff --git a/.anvil.lock b/.anvil.lock index 9f2d27e4..fc37cc9f 100644 --- a/.anvil.lock +++ b/.anvil.lock @@ -1,7 +1,7 @@ version = 1 tool = "anvil" tool_version = "0.5.0" -catalog_checksum = "sha256:0cdc61db9e974a45898d27d2f2a764599e8e08fa90b78cfeb037f5193204aa2b" +catalog_checksum = "sha256:06b57c3c034a1e91e9e8bac0411d892496a3aeba35aa5848005312c6b192bbb7" [[file]] path = ".anvil/container/Dockerfile.dockerignore" @@ -165,7 +165,7 @@ checksum = "sha256:6efd7378a2cd0f5d86519bd32fd86f2055a60191187dd77a8842b374b8eb7 [[file]] path = "justfiles/anvil/container.just" -checksum = "sha256:f2abdfa0853d3e259ce445e7fe3f9e4c5f022124b04406ca0cf9e793ce7e929c" +checksum = "sha256:c8ad802cf2a7d7630098dacec536284e80dbeb481a4bba813d941c6460c144ac" [[file]] path = "justfiles/anvil/groups/pr-fast.just" diff --git a/crates/cargo-anvil/templates/justfiles/anvil/container.just b/crates/cargo-anvil/templates/justfiles/anvil/container.just index f45a49ba..bdf35146 100644 --- a/crates/cargo-anvil/templates/justfiles/anvil/container.just +++ b/crates/cargo-anvil/templates/justfiles/anvil/container.just @@ -168,6 +168,16 @@ anvil-container-tag: $inputs = @('rust-toolchain.toml') $links = @() + # The declared input and the two walk roots are checked here, because a walk + # only ever reports descendants: a link that *is* the root is traversed or + # read through and never appears in its own output. Same hazard as a link + # below them -- the engine copies the link while everything here follows it. + foreach ($declared in @('rust-toolchain.toml', '.anvil/container', 'justfiles/anvil')) { + $item = Get-Item -LiteralPath (Join-Path $repoRoot $declared) -Force -ErrorAction SilentlyContinue + if ($item -and ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + $links += $item.FullName + } + } # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when diff --git a/crates/cargo-anvil/tests/recipe_contracts.rs b/crates/cargo-anvil/tests/recipe_contracts.rs index 8d28c7b2..22986a4a 100644 --- a/crates/cargo-anvil/tests/recipe_contracts.rs +++ b/crates/cargo-anvil/tests/recipe_contracts.rs @@ -1347,15 +1347,37 @@ fn a_link_among_the_image_inputs_is_refused() { return; } - for (kind, name, links_a_directory) in [("a file link", "linked.just", false), ("a directory link", "linked", true)] { + // A walk only ever reports descendants, so a link that *is* a declared + // input or a walk root is followed and never appears in its own output. + // Both positions are covered. + for (kind, name, links_a_directory) in [ + ("a file link below a walk root", "justfiles/anvil/linked.just", false), + ("a directory link below a walk root", "justfiles/anvil/linked", true), + ("a linked declared input", "rust-toolchain.toml", false), + ("a linked recipe walk root", "justfiles/anvil", true), + ("a linked container walk root", ".anvil/container", true), + ] { let tmp = fixture(&[("container.just", CONTAINER)], &[]); let root = tmp.path(); - write(&root.join("rust-toolchain.toml"), "[toolchain]\nchannel = \"stable\"\n"); - write(&root.join(".anvil/container/Dockerfile"), "FROM scratch\n"); - write(&root.join("justfiles/anvil/mod.just"), "# recipes\n"); write(&root.join("elsewhere/target.just"), "# shared\n"); + write(&root.join("elsewhere/Dockerfile"), "FROM scratch\n"); + // Everything the tag needs, except whatever this case replaces with a + // link. The link stands in for it, so writing it first would defeat the + // case for a walk root and leave nothing to link at all. + for (path, body) in [ + ("rust-toolchain.toml", "[toolchain]\nchannel = \"stable\"\n"), + (".anvil/container/Dockerfile", "FROM scratch\n"), + ("justfiles/anvil/mod.just", "# recipes\n"), + ] { + if !path.starts_with(name) { + write(&root.join(path), body); + } + } - let link = root.join("justfiles/anvil").join(name); + let link = root.join(name); + if let Some(parent) = link.parent() { + fs::create_dir_all(parent).unwrap(); + } let created = if links_a_directory { symlink_dir(&root.join("elsewhere"), &link) } else { @@ -1371,7 +1393,7 @@ fn a_link_among_the_image_inputs_is_refused() { assert_failed(&output, &format!("computing a tag with {kind} among the inputs")); let stderr = String::from_utf8_lossy(&output.stderr); assert!( - stderr.contains("regular") && stderr.contains(name), + stderr.contains("regular") && stderr.contains(name.rsplit('/').next().unwrap()), "{kind} must be named in the refusal\nstderr:\n{stderr}" ); } diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap index 5d4b0e6f..c4575148 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap @@ -3697,6 +3697,16 @@ anvil-container-tag: $inputs = @('rust-toolchain.toml') $links = @() + # The declared input and the two walk roots are checked here, because a walk + # only ever reports descendants: a link that *is* the root is traversed or + # read through and never appears in its own output. Same hazard as a link + # below them -- the engine copies the link while everything here follows it. + foreach ($declared in @('rust-toolchain.toml', '.anvil/container', 'justfiles/anvil')) { + $item = Get-Item -LiteralPath (Join-Path $repoRoot $declared) -Force -ErrorAction SilentlyContinue + if ($item -and ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + $links += $item.FullName + } + } # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap index 075f971f..6d3b5b9f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap @@ -3576,6 +3576,16 @@ anvil-container-tag: $inputs = @('rust-toolchain.toml') $links = @() + # The declared input and the two walk roots are checked here, because a walk + # only ever reports descendants: a link that *is* the root is traversed or + # read through and never appears in its own output. Same hazard as a link + # below them -- the engine copies the link while everything here follows it. + foreach ($declared in @('rust-toolchain.toml', '.anvil/container', 'justfiles/anvil')) { + $item = Get-Item -LiteralPath (Join-Path $repoRoot $declared) -Force -ErrorAction SilentlyContinue + if ($item -and ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + $links += $item.FullName + } + } # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when diff --git a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap index f3c56688..9b098f5f 100644 --- a/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap +++ b/crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap @@ -2449,6 +2449,16 @@ anvil-container-tag: $inputs = @('rust-toolchain.toml') $links = @() + # The declared input and the two walk roots are checked here, because a walk + # only ever reports descendants: a link that *is* the root is traversed or + # read through and never appears in its own output. Same hazard as a link + # below them -- the engine copies the link while everything here follows it. + foreach ($declared in @('rust-toolchain.toml', '.anvil/container', 'justfiles/anvil')) { + $item = Get-Item -LiteralPath (Join-Path $repoRoot $declared) -Force -ErrorAction SilentlyContinue + if ($item -and ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + $links += $item.FullName + } + } # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when diff --git a/justfiles/anvil/container.just b/justfiles/anvil/container.just index f45a49ba..bdf35146 100644 --- a/justfiles/anvil/container.just +++ b/justfiles/anvil/container.just @@ -168,6 +168,16 @@ anvil-container-tag: $inputs = @('rust-toolchain.toml') $links = @() + # The declared input and the two walk roots are checked here, because a walk + # only ever reports descendants: a link that *is* the root is traversed or + # read through and never appears in its own output. Same hazard as a link + # below them -- the engine copies the link while everything here follows it. + foreach ($declared in @('rust-toolchain.toml', '.anvil/container', 'justfiles/anvil')) { + $item = Get-Item -LiteralPath (Join-Path $repoRoot $declared) -Force -ErrorAction SilentlyContinue + if ($item -and ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint)) { + $links += $item.FullName + } + } # The declared inputs are text this tree owns, so their line endings are # normalized before hashing and a CRLF checkout agrees with an LF one. # Everything discovered by walking a directory is treated as text only when