diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 64d2c52..c9a5a2e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -348,6 +348,11 @@ jobs: - name: E2E quarantine lane tests run: python3 scripts/test_e2e_quarantine.py + # The v3 composite actions (actions/): actionlint and lint_callables.py read only .github/workflows, so this is + # the one check that sees them. It runs each step's shipped script, and shellcheck over it. + - name: Composite action tests (v3) + run: python3 scripts/test_actions.py + # Cheap belt-and-braces: actionlint parses YAML itself, but a file it # cannot classify as a workflow is skipped rather than failed, so an # unparseable stray .yml would slip past both checks above. diff --git a/AGENTS.md b/AGENTS.md index 28dc696..c12ac2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,6 +19,10 @@ public; its own CI and every public caller must use GitHub-hosted runners. Private callers retain their existing reusable-workflow access and may use manifest-routed self-hosted capacity where their repository policy permits it. +v3 is four composite actions under [`actions/`](./actions/README.md) (`setup`, +`verify`, `version-proof`, `health`): steps a caller's own job uses, with no +runner input and no `Required` job. `scripts/test_actions.py` is their gate. + The estate-wide operating manual for agents lives in [narduk-enterprises/agent-infrastructure](https://github.com/narduk-enterprises/agent-infrastructure) (its root `AGENTS.md`). On the Macs it is loaded automatically; in a cloud @@ -30,7 +34,7 @@ container nothing loads it for you — read it first, then return here. |---|---| | What is this repo? | The estate's shared reusable Actions workflows, consumed via `workflow_call` | | Estate operating manual | [narduk-enterprises/agent-infrastructure](https://github.com/narduk-enterprises/agent-infrastructure) → `AGENTS.md` | -| Run the checks | `actionlint -no-color -oneline .github/workflows/*.yml`, `python3 scripts/lint_callables.py`, then each `python3 scripts/test_*.py` | +| Run the checks | `actionlint -no-color -oneline .github/workflows/*.yml`, `python3 scripts/lint_callables.py`, then each `python3 scripts/test_*.py` (`test_actions.py` alone covers `actions/`) | | Catalog, consumption, versioning | [`README.md`](./README.md) — what each callable does, how to call it, how tags move | | Structural rules (R1–R13) | [`scripts/lint_callables.py`](./scripts/lint_callables.py) — the rule table at the top | | Cloud-session setup | [narduk-enterprises/agent-infrastructure](https://github.com/narduk-enterprises/agent-infrastructure) → `docs/cloud-sessions.md` | diff --git a/README.md b/README.md index 0a9276d..df73b02 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,10 @@ in their app-class workflow. | `flake-digest.yml` | **New 2026-09-24** (narduk-reboot P3-C2). Weekly, advisory: scans a caller's completed runs of one workflow, counts failures in jobs matching `quarantine-job-pattern` (nuxt-cloudflare.yml's `e2e-quarantine` by default), and files or refreshes one digest issue per ISO week. Never gates `main`, never opens `red-main` | | `taskgraph.yml` | **New 2026-10-29.** Vercel CI's model (beta 2026-10-28) for any `package.json`-script repo: the caller's `turbo.json` tags the CI tasks, each task gets a content key from Git's blob hashes, a task whose key already passed is never run again (GitHub's own record of a green run of the same content, with no secret; per task, a pointer in the R2 CI artifact store that GitHub's API must confirm), `FAST_BASE_REF` scripts run narrowed on PRs keyed on their base, and the rest run side by side after one install per machine, each in its own worktree; a `ci:machine` task gets its own seat. Plan, Tasks legs, Required, and a `Fast` alias of Required's real result for rulesets that still require `ci / Fast`; nine inputs. See [taskgraph.yml](#taskgraphyml) | +**v3 (2026-11-17) is composite actions, not more callables:** `setup`, `verify`, `version-proof` and `health` under +[`actions/`](./actions/README.md). A caller keeps its own jobs and uses them as steps, so there is no `runner` input, no +`Required` job and nothing here to route. New work starts there; the callables above keep their callers. + All eleven shipped callables are `on: workflow_call` only — none of them declare their own triggers, and none declare workflow-level `concurrency:` (see "How to consume" below for job-level cancellation and the caller's workflow group). diff --git a/actions/README.md b/actions/README.md new file mode 100644 index 0000000..27c9bf1 --- /dev/null +++ b/actions/README.md @@ -0,0 +1,76 @@ +# v3: composite actions + +Four steps a repository's own workflow calls. They are composite actions, not reusable workflows, on purpose: + +- **The caller keeps its jobs.** A reusable workflow brings its own jobs, runner choice and a `Required` job that takes a + runner to add up the others. A composite action is steps inside the caller's job: the caller owns `runs-on`, `if`, + `needs`, `permissions`, `concurrency` and the check's name, and no job exists only to report on other jobs. +- **Nothing to route.** No `runner` input, no organization variable, no visibility test. The job that uses the action + already says where it runs. +- **Small enough to read.** Each is one file under 80 lines, or one script; `scripts/test_actions.py` runs the shipped + text of every step. + +They come from PACC·TRAC's CI after it was cut from six jobs per pull request to one (2026-11-17). v1 and v2 (the +reusable workflows in `.github/workflows/`) are unchanged and keep their callers. + +| Action | What it does | Inputs | +| --- | --- | --- | +| [`setup`](./setup/action.yml) | Node from `.node-version`, pnpm from `packageManager`, the pnpm store where the runner can keep it, `pnpm install --frozen-lockfile` | `node-version-file`, `install-args` | +| [`verify`](./verify/action.yml) | Runs the checks `Config/remote-gate.json` names: what the diff selects on a pull request, the whole list otherwise; side by side, one timed line each | `scope`, `alone`, `config`, `base` | +| [`version-proof`](./version-proof/action.yml) | Polls a URL until it answers this commit's SHA; red when it never does | `url`, `expected`, `attempts`, `interval` | +| [`health`](./health/action.yml) | Polls a health URL until it answers 200; red, with the body, when it never does | `url`, `attempts`, `interval`, `means` | + +## A caller + +```yaml +jobs: + verify: # the one required check: on a pull request and on main + runs-on: + timeout-minutes: 20 + permissions: + contents: read + steps: + - uses: actions/checkout@ # v7 + with: + fetch-depth: 2 # a pull request is compared with HEAD^1: that commit must be present, older history need not be + persist-credentials: false + - uses: narduk-enterprises/workflows/actions/setup@ # v3 + - uses: narduk-enterprises/workflows/actions/verify@ # v3 + with: + alone: typecheck-demo build + + deploy-staging: + needs: [verify, build] + steps: + # ... the repository's own deploy ... + - uses: narduk-enterprises/workflows/actions/version-proof@ # v3 + with: + url: https://staging.example.com/version + - uses: narduk-enterprises/workflows/actions/health@ # v3 + with: + url: https://staging.example.com/healthz?deep=1 + means: the database binding or its migrations are wrong +``` + +Pin the full commit SHA `v3` points to, with a `# v3` comment, as for v2. Never `@main`. + +A repository whose checks read history itself (old commits, their trees) needs `fetch-depth: 0`; give it +`filter: blob:none` as well. Full history with every old file's contents was PACC·TRAC's longest step in this job (28 s +at the median, up to 653 s, 2026-11-17); without the contents the same checkout took 4 to 10 s. + +## The gate file + +`verify` reads the same `Config/remote-gate.json` that `remote-gate --changed` reads on a developer's machine +(agent-infrastructure `scripts/remote_gate.py`), so a change that is green there runs the same checks in CI: + +- `checks`: name to shell command. `{files}` becomes the changed files a rule matched; `{base}` the base commit. +- `default`: the names every event but a pull request runs. +- `changed.rules`: `paths` (globs from the repository root) to `checks`; `changed.always`: names every pull request + runs; `changed.full`: paths that swap one check for another (the changed-files lint for the whole lint). + +`alone` is the action's own input, not part of the file: the checks that must not share memory with the rest. + +## What is not here yet + +The build with its byte-exact handoff to the deploy, the deploy itself (migrate, read the ledger back, `wrangler deploy`) +and the browser sweep are still PACC·TRAC's own steps. They move here when a second repository needs them. diff --git a/actions/health/action.yml b/actions/health/action.yml new file mode 100644 index 0000000..1b0e1d8 --- /dev/null +++ b/actions/health/action.yml @@ -0,0 +1,41 @@ +name: Health probe +description: >- + Ask a health URL until it answers 200, and fail with the first 600 bytes of its last answer when it never does. + For the deep probe after a deploy: the code is proven by version-proof, this proves what it is bound to. + +inputs: + url: + description: The URL to probe, e.g. https://staging.example.com/healthz?deep=1. + required: true + attempts: + description: How many times to ask. + default: '10' + interval: + description: Seconds between attempts. + default: '5' + means: + description: What a failure means, for the error line, e.g. "the D1 binding or its migrations are wrong". + default: '' + +runs: + using: composite + steps: + - name: Probe the health URL + shell: bash + env: + HEALTH_URL: ${{ inputs.url }} + HEALTH_ATTEMPTS: ${{ inputs.attempts }} + HEALTH_INTERVAL: ${{ inputs.interval }} + HEALTH_MEANS: ${{ inputs.means }} + run: | + set -uo pipefail + body="$(mktemp)"; trap 'rm -f "$body"' EXIT + for i in $(seq 1 "$HEALTH_ATTEMPTS"); do + code=$(curl --silent --max-time 15 -o "$body" -w '%{http_code}' "$HEALTH_URL") || code=000 + echo "[$i] $HEALTH_URL -> $code" + [ "$code" = 200 ] && { echo "$HEALTH_URL is healthy"; exit 0; } + [ "$i" = "$HEALTH_ATTEMPTS" ] || sleep "$HEALTH_INTERVAL" + done + echo "::error::$HEALTH_URL did not answer 200${HEALTH_MEANS:+: $HEALTH_MEANS}" + head -c 600 "$body" 2>/dev/null; echo + exit 1 diff --git a/actions/setup/action.yml b/actions/setup/action.yml new file mode 100644 index 0000000..2347451 --- /dev/null +++ b/actions/setup/action.yml @@ -0,0 +1,71 @@ +name: Node, pnpm and the dependencies +description: >- + The toolchain the checkout pins (.node-version, packageManager) and `pnpm install --frozen-lockfile`, with the pnpm + store where the runner can keep it. Run it after actions/checkout. + +inputs: + node-version-file: + description: The file that pins Node. + default: .node-version + install-args: + description: Extra arguments for `pnpm install --frozen-lockfile`, e.g. `--ignore-scripts` in a job that holds a deploy token. + default: '' + +runs: + using: composite + steps: + - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0 + with: + run_install: false + + # About the disk, not the provider. pnpm hard-links node_modules out of its store, and a hard link cannot cross + # filesystems. A runner whose workspace sits on its own volume keeps one live store beside it, and no cache is + # restored or saved: a cache archive of a store other jobs are writing is torn. A runner whose workspace is on the + # root filesystem (a hosted VM) has no such store: it gets one inside the workspace, restored by setup-node's cache. + - name: Where the pnpm store lives + id: store + shell: bash + run: | + set -euo pipefail + # POSIX `df -P`: one line a filesystem, the mount point last. GNU's `--output=target` stops a macOS runner here. + # A mount point with a space in it is read short, finds no store, and falls to the last branch. + mount_point=$(df -P "$GITHUB_WORKSPACE" | tail -n 1 | awk '{print $NF}') + store="$mount_point/pnpm-store" + if [ "$mount_point" = "/" ]; then + store="$GITHUB_WORKSPACE/.pnpm-store" + echo "cache=true" >> "$GITHUB_OUTPUT" + echo "npm_config_store_dir=$store" >> "$GITHUB_ENV" + echo "pnpm store -> $store (in the workspace, restored from the Actions cache)" + elif [ -d "$store" ] && [ -w "$store" ]; then + echo "cache=false" >> "$GITHUB_OUTPUT" + echo "npm_config_store_dir=$store" >> "$GITHUB_ENV" + echo "pnpm store -> $store (kept on this runner, same filesystem as $GITHUB_WORKSPACE)" + else + echo "cache=false" >> "$GITHUB_OUTPUT" + echo "no writable store beside the workspace; pnpm's default store, no cache (installs copy instead of linking)" + fi + + - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 + if: steps.store.outputs.cache != 'true' + with: + node-version-file: ${{ inputs.node-version-file }} + package-manager-cache: false + + - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 + if: steps.store.outputs.cache == 'true' + with: + node-version-file: ${{ inputs.node-version-file }} + cache: pnpm + cache-dependency-path: | + pnpm-lock.yaml + package.json + + - name: Install dependencies + shell: bash + env: + INSTALL_ARGS: ${{ inputs.install-args }} + run: | + set -euo pipefail + # Word-split on purpose: the input is a list of flags. + # shellcheck disable=SC2086 + pnpm install --frozen-lockfile $INSTALL_ARGS diff --git a/actions/verify/action.yml b/actions/verify/action.yml new file mode 100644 index 0000000..9ef2a0e --- /dev/null +++ b/actions/verify/action.yml @@ -0,0 +1,39 @@ +name: Verify +description: >- + Run the checks the repository's gate file (Config/remote-gate.json) names: on a pull request the ones its diff + selects, on any other event the whole list. They run side by side in this one job and each prints one timed line. + Run it after actions/checkout (with the base commit present: fetch-depth 2 or more on a pull request) and after the + dependencies are installed. + +inputs: + scope: + description: "`changed` runs what the diff from `base` selects; `all` runs the gate file's `default` list." + default: ${{ github.event_name == 'pull_request' && 'changed' || 'all' }} + alone: + description: Names of checks that run one at a time after the others, space separated, in that order (a build, a second typecheck). + default: '' + config: + description: The gate file, relative to the workspace. + default: Config/remote-gate.json + base: + description: The commit a pull request is compared with. HEAD^1 is the base branch on the merge commit a pull request checks out. + default: HEAD^1 + +runs: + using: composite + steps: + - name: Plan and run the checks + shell: bash + env: + VERIFY_SCOPE: ${{ inputs.scope }} + VERIFY_ALONE: ${{ inputs.alone }} + VERIFY_CONFIG: ${{ inputs.config }} + VERIFY_BASE: ${{ inputs.base }} + run: | + set -euo pipefail + case "$VERIFY_SCOPE" in + changed) python3 "$GITHUB_ACTION_PATH/verify.py" plan ;; + all) python3 "$GITHUB_ACTION_PATH/verify.py" plan --all ;; + *) echo "::error::scope must be changed or all, not '$VERIFY_SCOPE'"; exit 1 ;; + esac + python3 "$GITHUB_ACTION_PATH/verify.py" run diff --git a/actions/verify/verify.py b/actions/verify/verify.py new file mode 100644 index 0000000..953f9c0 --- /dev/null +++ b/actions/verify/verify.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +"""One job that checks the source: choose the checks, run them side by side, print one timed line each. + + plan A pull request. Read the changed files and the repository's gate file, choose the checks, write the + plan to $RUNNER_TEMP/verify-plan.json. + plan --all Any other event. The whole list (`default` in the gate file). + run Run the planned checks side by side, keep going past a red one, print one timed line each. + Exit 1 when any check failed. + select Read changed file names on stdin; print the checks `plan` would choose for them. + +The gate file is `Config/remote-gate.json`, the file `remote-gate --changed` reads on a developer's machine +(agent-infrastructure scripts/remote_gate.py: `glob_match`, `select_changed`). CI and the desk read one mapping, so a +change that is green locally runs the same checks here. The file is the source of truth; this is its second reader, +and the two readings must stay identical. + +Environment: VERIFY_CONFIG (the gate file, relative to the workspace), VERIFY_ALONE (names of checks that run one at +a time after the others, in this order: a build or a second typecheck whose memory must not stack on the rest), +VERIFY_BASE (the commit a pull request is compared with; HEAD^1 is the base branch on a merge commit). +""" +import json +import os +import re +import shlex +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +ROOT = Path(os.environ.get('GITHUB_WORKSPACE') or os.getcwd()) +CONFIG = ROOT / os.environ.get('VERIFY_CONFIG', 'Config/remote-gate.json') +PLAN = Path(os.environ.get('RUNNER_TEMP', tempfile.gettempdir())) / 'verify-plan.json' +BASE = os.environ.get('VERIFY_BASE') or 'HEAD^1' +ALONE = tuple(os.environ.get('VERIFY_ALONE', '').split()) + + +def glob_match(pattern: str, path: str, fold: bool = False) -> bool: + """Repo-root-anchored glob: `*` and `?` stay inside a directory, `**/` spans directories.""" + out, i = '', 0 + while i < len(pattern): + if pattern.startswith('**/', i): + out, i = out + '(?:.*/)?', i + 3 + elif pattern.startswith('**', i): + out, i = out + '.*', i + 2 + elif pattern[i] == '*': + out, i = out + '[^/]*', i + 1 + elif pattern[i] == '?': + out, i = out + '[^/]', i + 1 + else: + out, i = out + re.escape(pattern[i]), i + 1 + return re.fullmatch(out, path, re.IGNORECASE if fold else 0) is not None + + +def changed_files() -> list[str]: + """What this pull request proposes: parent 1 of the merge commit is the base branch. + `-z`: without it git quotes and escapes a name that has a non-ASCII character, a tab or a quote in it + (`"src/caf\\303\\251.ts"`), no rule matches the quoted text, and that file's checks are silently not chosen.""" + out = subprocess.run( + ['git', 'diff', '--no-renames', '--name-only', '-z', BASE, 'HEAD'], + cwd=ROOT, check=True, capture_output=True, text=True, + ).stdout + return [f for f in out.split('\0') if f] + + +def select_changed(spec: dict, checks: dict, files: list[str]) -> list[tuple[str, str]]: + picked: dict[str, list[str]] = {} + for rule in spec['rules']: + hit = [f for f in files if any(glob_match(g, f) for g in rule['paths'])] + for name in rule['checks'] if hit else []: + picked.setdefault(name, []) + picked[name] += [f for f in hit if f not in picked[name]] + full = spec.get('full') + if full and any(glob_match(g, f) for f in files for g in full['paths']): + picked = {full['swap'].get(n, n): fs for n, fs in picked.items()} + for name in spec.get('always', []): + picked.setdefault(name, []) + plan = [] + for name, hit in picked.items(): + command = checks[name] + if '{files}' in command: + # A deleted file is a change, but there is nothing for a linter to open. `./` so a file named `--something` is never read as a flag. + present = [f for f in (hit or files) if (ROOT / f).exists()] + if not present: + continue + command = command.replace('{files}', ' '.join(shlex.quote('./' + f) for f in present)) + plan.append((name, command.replace('{base}', BASE))) + return plan + + +def summary(text: str) -> None: + print(text) + path = os.environ.get('GITHUB_STEP_SUMMARY') + if path: + with open(path, 'a') as fh: + fh.write(text + '\n') + + +def plan(*flags: str) -> int: + config = json.loads(CONFIG.read_text()) + unknown = [name for name in ALONE if name not in config['checks']] + if unknown: + sys.exit(f'verify: `alone` names no check in {CONFIG.name}: {", ".join(unknown)}') + if '--all' in flags: + names = config['default'] + chosen = [(name, config['checks'][name]) for name in names] + summary('Every check (not a pull request); verify runs: ' + ', '.join(names)) + else: + files = changed_files() + chosen = select_changed(config['changed'], config['checks'], files) + summary(f'{len(files)} changed file(s) vs {BASE}; verify runs: ' + ', '.join(n for n, _ in chosen)) + PLAN.write_text(json.dumps({'checks': chosen})) + return 0 + + +def stages(chosen: list) -> list: + """Checks that share nothing run side by side, so the job is as long as its slowest check, not their sum. + The ALONE ones follow, one at a time, in ALONE's order.""" + together = [c for c in chosen if c[0] not in ALONE] + alone = sorted((c for c in chosen if c[0] in ALONE), key=lambda c: ALONE.index(c[0])) + return ([together] if together else []) + [[c] for c in alone] + + +def run_stage(stage: list) -> list: + """[(name, code, seconds)] for one stage, in the plan's order. Each check's log is printed as one group when the + stage ends, and each check's seconds are its own (start of the stage to its own exit), not the slowest one's.""" + started = time.monotonic() + procs = [] + for name, command in stage: + log = tempfile.TemporaryFile(mode='w+') + procs.append((name, command, log, subprocess.Popen(command, cwd=ROOT, shell=True, stdout=log, stderr=subprocess.STDOUT))) + took: dict[str, float] = {} + while len(took) < len(procs): + for name, _command, _log, proc in procs: + if name not in took and proc.poll() is not None: + took[name] = time.monotonic() - started + time.sleep(0.2) + results = [] + for name, command, log, proc in procs: + log.seek(0) + print(f'::group::{name}: {command}', flush=True) + shutil.copyfileobj(log, sys.stdout) + print('::endgroup::', flush=True) + results.append((name, proc.returncode, took[name])) + return results + + +def run() -> int: + chosen = json.loads(PLAN.read_text())['checks'] + began = time.monotonic() + rows, failed = [], [] + for stage in stages(chosen): + for name, code, took in run_stage(stage): + rows.append((name, 'pass' if code == 0 else f'FAIL({code})', took)) + if code != 0: + failed.append(name) + summary('\n'.join(['| check | result | wall |', '| --- | --- | --- |'] + [f'| {n} | {r} | {t:.1f} s |' for n, r, t in rows])) + summary(f'verify checks total: {time.monotonic() - began:.1f} s') + if failed: + print(f'::error::verify failed: {", ".join(failed)}') + return 1 + return 0 + + +def select() -> int: + config = json.loads(CONFIG.read_text()) + files = [f for f in sys.stdin.read().splitlines() if f] + for name, _ in select_changed(config['changed'], config['checks'], files): + print(name) + return 0 + + +COMMANDS = {'plan': plan, 'run': run, 'select': select} + +if __name__ == '__main__': + command, flags = (sys.argv[1] if len(sys.argv) > 1 else ''), sys.argv[2:] + if command not in COMMANDS or (flags and (command != 'plan' or flags != ['--all'])): + sys.exit(f'usage: verify.py {{{"|".join(COMMANDS)}}}; plan takes --all') + sys.exit(COMMANDS[command](*flags)) diff --git a/actions/version-proof/action.yml b/actions/version-proof/action.yml new file mode 100644 index 0000000..6c94f03 --- /dev/null +++ b/actions/version-proof/action.yml @@ -0,0 +1,51 @@ +name: Prove the deployed commit +description: >- + Poll a URL until it answers 200 with a body that is exactly the expected text (by default this commit's SHA), and + fail when it never does. A deploy step that exits 0 says the upload was accepted, not that the new code is serving: + this asks the live host. + +inputs: + url: + description: The URL to read, e.g. https://staging.example.com/version. It must answer with the commit and nothing else. + required: true + expected: + description: The text the URL must answer with. + default: ${{ github.sha }} + attempts: + description: How many times to ask. + default: '60' + interval: + description: Seconds between attempts. + default: '5' + +runs: + using: composite + steps: + - name: Ask the live host which commit it serves + shell: bash + env: + PROOF_URL: ${{ inputs.url }} + PROOF_EXPECTED: ${{ inputs.expected }} + PROOF_ATTEMPTS: ${{ inputs.attempts }} + PROOF_INTERVAL: ${{ inputs.interval }} + run: | + set -uo pipefail + body="$(mktemp)"; trap 'rm -f "$body"' EXIT + live=""; said="" + for i in $(seq 1 "$PROOF_ATTEMPTS"); do + # Only a whole answer with a 200 counts. The commit in the body of a 503, or in the part of a response that + # arrived before the transfer failed, is not a host serving that commit. + if code=$(curl --silent --max-time 10 -o "$body" -w '%{http_code}' "$PROOF_URL"); then + if [ "$code" = 200 ]; then live=$(cat "$body"); said="'$live'"; else live=""; said="HTTP $code"; fi + else + live=""; said="" + fi + echo "[$i] $PROOF_URL=$said expected=$PROOF_EXPECTED" + [ "$live" = "$PROOF_EXPECTED" ] && break + [ "$i" = "$PROOF_ATTEMPTS" ] || sleep "$PROOF_INTERVAL" + done + if [ "$live" != "$PROOF_EXPECTED" ]; then + echo "::error::$PROOF_URL answers $said, not $PROOF_EXPECTED: the deploy shipped nothing, or shipped something else" + exit 1 + fi + echo "$PROOF_URL serves $PROOF_EXPECTED" diff --git a/scripts/test_actions.py b/scripts/test_actions.py new file mode 100644 index 0000000..528be5f --- /dev/null +++ b/scripts/test_actions.py @@ -0,0 +1,357 @@ +#!/usr/bin/env python3 +"""Tests for the composite actions under `actions/` (v3). + +Nothing here tests a copy: each test extracts the shipped `run:` text of a step out of its `action.yml`, or runs the +shipped `verify.py`, and executes it. `df` and `curl` targets are chosen by the test (a shim, a local HTTP server), so +it runs the same on a developer Mac and on the Linux runner. + +What is locked down: + shape every action is composite, every `run:` step names bash and passes shellcheck (actionlint does not + read a composite action, so nothing else would), every `uses:` is a full SHA with a version comment, + and no `run:` text interpolates `${{ }}` (an input reaches a script through `env:`). + setup the three places a pnpm store can live: in the workspace with the cache (workspace on `/`), beside + the workspace without a cache (a persistent runner), and nowhere (no writable store: degrade, never fail); + and the step ends well with this machine's own `df`, GNU or BSD. + version-proof green only when the URL answers 200 with exactly the expected text; red, after every attempt, when it + does not, when the text comes with another status, and when the response is cut short. + health green on a 200; red, with the body's head, on anything else or no answer. + verify the selection from a gate file (rules, `always`, the `full` swap, a deleted file), the whole list, + side-by-side runs that keep going past a red check, `alone` order, and a refused unknown `alone` name. + +Run: python3 scripts/test_actions.py +""" + +from __future__ import annotations + +import http.server +import json +import os +import re +import shutil +import stat +import subprocess +import sys +import tempfile +import threading +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[1] +ACTIONS = ROOT / "actions" +USES = re.compile(r"^\s*-?\s*uses:\s*(\S+)\s*(#.*)?$") +FAILED: list[str] = [] + + +def check(ok: bool, what: str) -> None: + print(("ok " if ok else "FAIL ") + what) + if not ok: + FAILED.append(what) + + +def load(name: str) -> dict: + return yaml.safe_load((ACTIONS / name / "action.yml").read_text()) + + +def step_script(name: str, step_name: str) -> str: + for step in load(name)["runs"]["steps"]: + if step.get("name") == step_name: + return step["run"] + raise SystemExit(f"::error::actions/{name}/action.yml has no step named {step_name!r}") + + +def bash(script: str, env: dict, cwd: Path | None = None) -> subprocess.CompletedProcess: + return subprocess.run(["bash", "-c", script], env={**os.environ, **env}, cwd=cwd, capture_output=True, text=True, timeout=120) + + +# -- shape ------------------------------------------------------------------------------------------------------------ + +def shellcheck(script: str) -> bool: + """True when shellcheck passes the script. Without the binary: a failure in CI, a skip on a desk.""" + if not shutil.which("shellcheck"): + if os.environ.get("CI"): + return False + print(" (shellcheck is not installed: skipped)") + return True + with tempfile.NamedTemporaryFile("w", suffix=".sh") as fh: + fh.write("#!/usr/bin/env bash\n" + script) + fh.flush() + done = subprocess.run(["shellcheck", "-s", "bash", fh.name], capture_output=True, text=True) + if done.returncode != 0: + print(done.stdout) + return done.returncode == 0 + + +def test_shape() -> None: + names = sorted(p.parent.name for p in ACTIONS.glob("*/action.yml")) + check(names == ["health", "setup", "verify", "version-proof"], f"the actions are health, setup, verify, version-proof (found {names})") + for name in names: + doc = load(name) + text = (ACTIONS / name / "action.yml").read_text() + check(doc["runs"]["using"] == "composite", f"{name}: composite") + check(bool(doc.get("name")) and bool(doc.get("description")), f"{name}: has a name and a description") + for step in doc["runs"]["steps"]: + if "run" in step: + check(step.get("shell") == "bash", f"{name}: step {step.get('name')!r} names bash") + check("${{" not in step["run"], f"{name}: step {step.get('name')!r} reads inputs through env, not in the script") + check(shellcheck(step["run"]), f"{name}: step {step.get('name')!r} passes shellcheck") + for line in text.splitlines(): + m = USES.match(line) + if not m: + continue + ref = m.group(1).split("@")[-1] + check(re.fullmatch(r"[0-9a-f]{40}", ref) is not None and re.search(r"#\s*v\d", m.group(2) or "") is not None, + f"{name}: {m.group(1).split('@')[0]} is pinned to a full SHA with its version") + for key, spec in (doc.get("inputs") or {}).items(): + check(bool(spec.get("description")), f"{name}: input {key} is described") + + +# -- setup ------------------------------------------------------------------------------------------------------------ + +def run_store(tmp: Path, mount: str) -> tuple[subprocess.CompletedProcess, str, str]: + bindir = tmp / "bin" + bindir.mkdir(exist_ok=True) + df = bindir / "df" + df.write_text(f"#!/bin/sh\necho 'Mounted on'\necho '{mount}'\n") + df.chmod(df.stat().st_mode | stat.S_IEXEC) + out, env = tmp / "out", tmp / "env" + out.write_text("") + env.write_text("") + workspace = tmp / "volume" / "work" + workspace.mkdir(parents=True, exist_ok=True) + done = bash(step_script("setup", "Where the pnpm store lives"), { + "PATH": f"{bindir}:{os.environ['PATH']}", "GITHUB_WORKSPACE": str(workspace), + "GITHUB_OUTPUT": str(out), "GITHUB_ENV": str(env), + }) + return done, out.read_text(), env.read_text() + + +def test_setup() -> None: + with tempfile.TemporaryDirectory() as d: + tmp = Path(d) + done, out, env = run_store(tmp, "/") + check(done.returncode == 0 and out.strip() == "cache=true" and env.strip() == f"npm_config_store_dir={tmp}/volume/work/.pnpm-store", + "setup: a workspace on / gets a store inside the workspace, with the cache") + with tempfile.TemporaryDirectory() as d: + tmp = Path(d) + (tmp / "volume" / "pnpm-store").mkdir(parents=True) + done, out, env = run_store(tmp, str(tmp / "volume")) + check(done.returncode == 0 and out.strip() == "cache=false" and env.strip() == f"npm_config_store_dir={tmp}/volume/pnpm-store", + "setup: a runner with a store beside the workspace uses it, without the cache") + with tempfile.TemporaryDirectory() as d: + tmp = Path(d) + done, out, env = run_store(tmp, str(tmp / "volume")) + check(done.returncode == 0 and out.strip() == "cache=false" and env.strip() == "", + "setup: no writable store beside the workspace degrades to pnpm's default, and does not fail") + with tempfile.TemporaryDirectory() as d: + # No shim: the `df` of the machine this runs on (BSD on a Mac, GNU on the Linux runner). The step must end well + # and say where the store is, whatever that machine's mount table looks like. + tmp = Path(d) + out, env = tmp / "out", tmp / "env" + out.write_text("") + env.write_text("") + done = bash(step_script("setup", "Where the pnpm store lives"), {"GITHUB_WORKSPACE": str(tmp), "GITHUB_OUTPUT": str(out), "GITHUB_ENV": str(env)}) + check(done.returncode == 0 and out.read_text().strip() in ("cache=true", "cache=false"), + f"setup: the step ends well with this machine's own df ({done.stderr.strip() or out.read_text().strip()})") + doc = load("setup") + nodes = [s for s in doc["runs"]["steps"] if str(s.get("uses", "")).startswith("actions/setup-node@")] + check(len(nodes) == 2 and nodes[0]["if"] == "steps.store.outputs.cache != 'true'" and nodes[1]["if"] == "steps.store.outputs.cache == 'true'" + and nodes[0]["with"].get("package-manager-cache") is False and nodes[1]["with"].get("cache") == "pnpm", + "setup: setup-node restores a cache only where the store is in the workspace") + install = step_script("setup", "Install dependencies") + check("pnpm install --frozen-lockfile $INSTALL_ARGS" in install, "setup: the install is frozen and takes the caller's flags") + + +# -- version-proof and health ----------------------------------------------------------------------------------------- + +class Handler(http.server.BaseHTTPRequestHandler): + routes: dict = {} + hits: dict = {} + + def do_GET(self): # noqa: N802 + path = self.path.split("?")[0] + Handler.hits[path] = Handler.hits.get(path, 0) + 1 + answers = Handler.routes.get(path, [(404, "not here")]) + # (status, body) or (status, body, the Content-Length to declare): a declared length longer than the body is a + # response cut short, which the client sees when the connection closes. + code, body, *declared = answers[min(Handler.hits[path], len(answers)) - 1] + self.send_response(code) + if declared: + self.send_header("Content-Length", str(declared[0])) + self.end_headers() + self.wfile.write(body.encode()) + + def log_message(self, *args): + pass + + +def serve() -> tuple[http.server.ThreadingHTTPServer, str]: + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + return server, f"http://127.0.0.1:{server.server_address[1]}" + + +def test_probes() -> None: + sha = "0123456789abcdef0123456789abcdef01234567" + Handler.routes = { + "/version": [(200, sha)], + "/late": [(200, "old"), (200, "old"), (200, sha)], + "/stale": [(200, "old")], + "/sick": [(503, sha)], + "/cut": [(200, sha, len(sha) + 40)], + "/healthz": [(200, "ok")], + "/warming": [(503, "warming"), (200, "ok")], + "/broken": [(503, "the control database is not migrated")], + } + Handler.hits = {} + server, base = serve() + try: + proof = step_script("version-proof", "Ask the live host which commit it serves") + env = {"PROOF_EXPECTED": sha, "PROOF_ATTEMPTS": "3", "PROOF_INTERVAL": "0"} + done = bash(proof, {**env, "PROOF_URL": f"{base}/version"}) + check(done.returncode == 0 and Handler.hits["/version"] == 1, "version-proof: green at the first answer that is the commit") + done = bash(proof, {**env, "PROOF_URL": f"{base}/late"}) + check(done.returncode == 0 and Handler.hits["/late"] == 3, "version-proof: keeps asking while the old commit answers") + done = bash(proof, {**env, "PROOF_URL": f"{base}/stale"}) + check(done.returncode == 1 and Handler.hits["/stale"] == 3 and "::error::" in done.stdout and "'old'" in done.stdout, + "version-proof: red after every attempt, naming what answered") + done = bash(proof, {**env, "PROOF_URL": "http://127.0.0.1:9/version", "PROOF_ATTEMPTS": "2"}) + check(done.returncode == 1 and "" in done.stdout, "version-proof: an unreachable host is red, not an error in the script") + done = bash(proof, {**env, "PROOF_URL": f"{base}/sick"}) + check(done.returncode == 1 and Handler.hits["/sick"] == 3 and "answers HTTP 503" in done.stdout, + "version-proof: the commit in the body of a 503 is not proof") + done = bash(proof, {**env, "PROOF_URL": f"{base}/cut"}) + check(done.returncode == 1 and Handler.hits["/cut"] == 3 and f"serves {sha}" not in done.stdout, + "version-proof: the commit at the head of a response that was cut short is not proof") + + health = step_script("health", "Probe the health URL") + env = {"HEALTH_ATTEMPTS": "3", "HEALTH_INTERVAL": "0", "HEALTH_MEANS": "the binding is wrong"} + done = bash(health, {**env, "HEALTH_URL": f"{base}/healthz?deep=1"}) + check(done.returncode == 0 and Handler.hits["/healthz"] == 1, "health: green on a 200") + done = bash(health, {**env, "HEALTH_URL": f"{base}/warming"}) + check(done.returncode == 0 and Handler.hits["/warming"] == 2, "health: waits through a 503") + done = bash(health, {**env, "HEALTH_URL": f"{base}/broken"}) + check(done.returncode == 1 and Handler.hits["/broken"] == 3 and "the binding is wrong" in done.stdout + and "the control database is not migrated" in done.stdout, "health: red after every attempt, with the reason and the body") + done = bash(health, {**env, "HEALTH_URL": "http://127.0.0.1:9/healthz", "HEALTH_ATTEMPTS": "1"}) + check(done.returncode == 1 and "-> 000" in done.stdout, "health: no answer is 000 and red") + finally: + server.shutdown() + + +# -- verify ----------------------------------------------------------------------------------------------------------- + +GATE = { + "schemaVersion": 1, + "checks": { + "lint": "echo lint-all", + "lint-files": "echo lint {files}", + "types": "echo types", + "test": "echo test", + "slow": "sleep 0.4 && echo slow", + "red": "echo broken && exit 3", + "build": "echo build", + }, + "default": ["lint", "types", "test"], + "changed": { + "rules": [ + {"paths": ["**/*.ts"], "checks": ["lint-files"]}, + {"paths": ["eslint.config.*", "package.json"], "checks": ["lint"]}, + {"paths": ["src/**", "package.json"], "checks": ["types"]}, + {"paths": ["package.json"], "checks": ["build"]}, + ], + "always": ["test"], + "full": {"paths": ["eslint.config.*", "package.json"], "swap": {"lint-files": "lint"}}, + }, +} +VERIFY = ACTIONS / "verify" / "verify.py" + + +def verify(repo: Path, *args: str, stdin: str = "", alone: str = "", gate: dict | None = None) -> subprocess.CompletedProcess: + (repo / "Config").mkdir(exist_ok=True) + (repo / "Config" / "remote-gate.json").write_text(json.dumps(gate or GATE)) + env = {**os.environ, "GITHUB_WORKSPACE": str(repo), "RUNNER_TEMP": str(repo), "VERIFY_ALONE": alone} + env.pop("GITHUB_STEP_SUMMARY", None) + return subprocess.run([sys.executable, str(VERIFY), *args], env=env, input=stdin, capture_output=True, text=True, timeout=120) + + +def git(repo: Path, *args: str) -> None: + subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", *args], cwd=repo, check=True, capture_output=True) + + +def test_verify() -> None: + with tempfile.TemporaryDirectory() as d: + repo = Path(d) + (repo / "src").mkdir() + (repo / "src" / "a.ts").write_text("a") + (repo / "README.md").write_text("r") + + def select(*files: str) -> list[str]: + return verify(repo, "select", stdin="\n".join(files)).stdout.split() + + check(select("README.md") == ["test"], "verify: a file no rule names still runs what `always` lists") + check(select("src/a.ts") == ["lint-files", "types", "test"], "verify: a source file picks its lint, its typecheck and the tests") + check(select("src/a.ts", "package.json") == ["lint", "types", "build", "test"], "verify: a `full` path swaps the changed-files lint for the whole lint") + check(select("docs/gone.ts") == ["test"], "verify: a changed-files lint with no file left to open is dropped") + + git(repo, "init", "-q") + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "base") + (repo / "src" / "b.ts").write_text("b") + (repo / "src" / "a.ts").unlink() + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "change") + done = verify(repo, "plan") + plan = json.loads((repo / "verify-plan.json").read_text())["checks"] + check(done.returncode == 0 and plan == [["lint-files", "echo lint ./src/b.ts"], ["types", "echo types"], ["test", "echo test"]], + f"verify: a pull request's plan lints the files that still exist, never a deleted one ({plan})") + done = verify(repo, "run") + check(done.returncode == 0 and "lint ./src/b.ts" in done.stdout and "| types | pass |" in done.stdout and "verify checks total" in done.stdout, + "verify: run prints each check's log and one timed line each") + + # git prints such a name quoted and escaped unless asked for NUL-separated names: read line by line, the only + # changed file matches no rule and the pull request runs neither its lint nor its typecheck. + (repo / "docs").mkdir() + (repo / "docs" / "caf\u00e9 menu.ts").write_text("c") + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "a name git would quote") + done = verify(repo, "plan") + plan = json.loads((repo / "verify-plan.json").read_text())["checks"] + check(done.returncode == 0 and plan == [["lint-files", "echo lint './docs/caf\u00e9 menu.ts'"], ["test", "echo test"]], + f"verify: a changed file whose name git would quote is still chosen by its rule ({plan})") + git(repo, "reset", "-q", "--hard", "HEAD^") + + done = verify(repo, "plan", "--all") + plan = json.loads((repo / "verify-plan.json").read_text())["checks"] + check(done.returncode == 0 and [n for n, _ in plan] == ["lint", "types", "test"], "verify: plan --all is the gate file's default list") + + gate = {**GATE, "default": ["slow", "red", "test", "build"]} + verify(repo, "plan", "--all", gate=gate, alone="build") + done = verify(repo, "run", gate=gate, alone="build") + rows = [line for line in done.stdout.splitlines() if line.startswith("| ") and "check | result" not in line and "---" not in line] + check(done.returncode == 1 and "::error::verify failed: red" in done.stdout, "verify: one red check fails the job and is named") + check(len(rows) == 4 and rows[1].startswith("| red | FAIL(3) |") and rows[2].startswith("| test | pass |") and rows[3].startswith("| build | pass |"), + f"verify: the checks after a red one still run, and an `alone` check runs last ({rows})") + took = {row.split("|")[1].strip(): float(row.split("|")[3].split()[0]) for row in rows} + check(took["test"] < took["slow"], "verify: side by side, a quick check is timed as its own, not as the slowest one's") + + done = verify(repo, "plan", "--all", alone="nope") + check(done.returncode != 0 and "nope" in done.stderr, "verify: an `alone` name the gate file does not have is refused") + done = verify(repo, "plan", "--everything") + check(done.returncode != 0 and "usage" in done.stderr, "verify: an unknown flag is refused") + + script = step_script("verify", "Plan and run the checks") + check('verify.py" plan --all' in script and 'verify.py" plan ;;' in script and 'verify.py" run' in script and "$GITHUB_ACTION_PATH" in script, + "verify: the action plans by scope, then runs, from its own directory") + + +if __name__ == "__main__": + test_shape() + test_setup() + test_probes() + test_verify() + if FAILED: + for what in FAILED: + print(f"::error::{what}") + sys.exit(1) + print("actions: all checks passed")