From 0671165bb27e3eecdbc63f4c4900fbf5d932700e Mon Sep 17 00:00:00 2001 From: TheDancingDeveloper Date: Mon, 3 Aug 2026 00:41:26 +0000 Subject: [PATCH 1/3] fix(policy): make runner enforcement actually cover the organization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit audit.yml was named "Organization workflow policy audit" but checks out only this repository and runs the checker against `.`. It therefore audited exactly one repository — itself. Its own workflows are compliant, so it reported green every day while NGMS, agent-harness and nntp-client-bench ran CI on ubuntu-latest. The checker was never wrong; it was never pointed at the org. Add enforce-runner-policy.yml, designed to be attached to every repository via an organization ruleset so it runs in each target repository's context. This needs no cross-repository token, which is why per-repo enforcement is the right mechanism here — the org has no Actions secrets at all. It fetches the checker over HTTPS rather than checking this repo out into the workspace, so the policy repo's own workflows are not conflated with the target tree. Rename audit.yml to "Policy repository self-audit" so its scope is not overstated again, and document both workflows in the README. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/audit.yml | 16 ++++++- .github/workflows/enforce-runner-policy.yml | 49 +++++++++++++++++++++ README.md | 46 ++++++++++++++++++- 3 files changed, 108 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/enforce-runner-policy.yml diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index eeb5da9..d80cc37 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -1,4 +1,16 @@ -name: Organization workflow policy audit +# Self-audit for this repository only. +# +# Renamed 2026-08-03. This workflow was previously called "Organization +# workflow policy audit", but it checks out only this repository and runs the +# audit against `.`, so it has never inspected any other repository. Its own +# workflows are compliant, so it reported green while three repositories in the +# organization ran on GitHub-hosted runners. +# +# Organization-wide enforcement is `enforce-runner-policy.yml`, attached to +# every repository through an organization ruleset so it runs in each target +# repository's own context. Keep this file scoped to this repository. + +name: Policy repository self-audit on: pull_request: @@ -16,5 +28,5 @@ jobs: runs-on: [self-hosted, node-b, linux, x64] steps: - uses: actions/checkout@v4 - - name: Audit workflow runner policy and exceptions + - name: Audit this repository's workflows and exception expiry run: bash scripts/audit-workflows.sh . diff --git a/.github/workflows/enforce-runner-policy.yml b/.github/workflows/enforce-runner-policy.yml new file mode 100644 index 0000000..ff43d7f --- /dev/null +++ b/.github/workflows/enforce-runner-policy.yml @@ -0,0 +1,49 @@ +# Runner policy gate. +# +# This workflow is designed to be attached to every repository in the +# organization through an organization ruleset ("required workflows"), so it +# runs in the *target* repository's context and its check must pass before a +# pull request can merge. +# +# It deliberately fetches the policy script over HTTPS rather than checking out +# this repository into the workspace: a nested checkout would place this repo's +# own workflows inside the scanned tree and conflate the two results. +# +# Fail-closed note: this job requests self-hosted labels. A repository that has +# not been added to the `public-node-b` runner group has no runner that can +# accept it, so the check stays queued and the pull request cannot merge. That +# is intended — it forces runner-group membership to be configured rather than +# letting a repository quietly fall back to GitHub-hosted runners. + +name: Runner policy + +on: + pull_request: + push: + branches: [main] + workflow_dispatch: + +permissions: + contents: read + +jobs: + runner-policy: + runs-on: [self-hosted, node-b, linux, x64] + steps: + - uses: actions/checkout@v5 + + - name: Fetch organization policy + env: + POLICY_RAW: https://raw.githubusercontent.com/TheDancingDeveloper-org/github-policy/main + run: | + set -euo pipefail + mkdir -p "$RUNNER_TEMP/policy" + curl -fsSL -o "$RUNNER_TEMP/policy/audit-workflows.sh" \ + "$POLICY_RAW/scripts/audit-workflows.sh" + curl -fsSL -o "$RUNNER_TEMP/policy/runner-exceptions.json" \ + "$POLICY_RAW/runner-exceptions.json" + + - name: Audit runner selection + env: + RUNNER_EXCEPTION_FILE: ${{ runner.temp }}/policy/runner-exceptions.json + run: bash "$RUNNER_TEMP/policy/audit-workflows.sh" . diff --git a/README.md b/README.md index 5ea446e..31a1219 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,51 @@ This repository is the source of truth for organization-wide CI policy. - Public-hosted runners require an owner-approved, time-bounded exception in `runner-exceptions.json`. - Workflow and policy changes require owner review through `CODEOWNERS`. -- `scripts/audit-workflows.sh` is the shared enforcement entry point. +- `scripts/audit-workflows.sh` is the shared checker used by both workflows + below. + +## How enforcement actually works + +There are two workflows, and the distinction matters: + +| Workflow | Scope | Role | +|---|---|---| +| `enforce-runner-policy.yml` | The repository it runs in | **The gate.** Attached to every repository by an organization ruleset, so it runs in each target repository's context and must pass before a pull request merges. | +| `audit.yml` | This repository only | Self-audit, plus the daily expiry check on `runner-exceptions.json`. | + +`audit.yml` was previously named "Organization workflow policy audit", which +was misleading: it checks out only this repository, so it audited exactly one +repository — itself — and stayed green while three repositories in the +organization ran on `ubuntu-latest`. Nothing about the checker was wrong; it +was never pointed at the organization. Do not re-add organization-wide +ambitions to that file. Per-repository enforcement is the correct mechanism +because it needs no cross-repository token. + +### Fail-closed behaviour + +`enforce-runner-policy.yml` requests self-hosted labels. A repository that has +not been added to the `public-node-b` runner group has no runner able to accept +the job, so the check stays queued and the pull request cannot merge. This is +intentional: it surfaces missing runner-group membership instead of letting a +repository quietly fall back to GitHub-hosted runners, which is exactly how the +2026-08-03 violations arose. + +## Exceptions + +```json +{ "schema_version": 1, + "exceptions": [ + { "repo": "example", "workflow": "ci.yml", "reason": "...", + "expires_on": "2026-09-30" } ] } +``` + +The audit fails on any exception whose `expires_on` has passed, so exceptions +cannot rot silently. + +## Related + +- `~/Working/docs/CI-RUNNER-GATES.md` — the human-facing gate document: + approved labels, new-repository checklist, and known gaps. The destination organization repository must be created with Actions disabled, then protected rules and the self-hosted runner group must be configured before From 636fe912fd2c7c66db7bf354a3411ebf056b461a Mon Sep 17 00:00:00 2001 From: TheDancingDeveloper Date: Mon, 3 Aug 2026 00:45:06 +0000 Subject: [PATCH 2/3] fix(policy): inline the checker; the policy repo is private The first version of the gate fetched scripts/audit-workflows.sh over HTTPS. That cannot work: this repository is private, so raw.githubusercontent.com returns 404 unauthenticated and the target repository's GITHUB_TOKEN has no read access here either. It failed on its own pull request with curl (22) 404. Inline the checker into enforce-runner-policy.yml. The ruleset already delivers that file from this repository, so there is still a single source of truth. scripts/audit-workflows.sh stays as the local checker for audit.yml; the two must be kept in sync. The inlined gate reads no exceptions file because it cannot see one, so it is strict. Owner-approved exceptions are expressed in the ruleset conditions or bypass actors, which is a more auditable record than a JSON entry anyway. Verified locally against a compliant tree (passes, 2 files) and a deliberately violating tree (rejects ubuntu-latest and dynamic runs-on). Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/enforce-runner-policy.yml | 105 +++++++++++++++----- README.md | 25 ++++- 2 files changed, 104 insertions(+), 26 deletions(-) diff --git a/.github/workflows/enforce-runner-policy.yml b/.github/workflows/enforce-runner-policy.yml index ff43d7f..774bd4a 100644 --- a/.github/workflows/enforce-runner-policy.yml +++ b/.github/workflows/enforce-runner-policy.yml @@ -1,19 +1,27 @@ # Runner policy gate. # -# This workflow is designed to be attached to every repository in the -# organization through an organization ruleset ("required workflows"), so it -# runs in the *target* repository's context and its check must pass before a -# pull request can merge. +# Attached to every repository in the organization through an organization +# ruleset ("workflows" rule), so it runs in the *target* repository's context +# and its check must pass before a pull request can merge. # -# It deliberately fetches the policy script over HTTPS rather than checking out -# this repository into the workspace: a nested checkout would place this repo's -# own workflows inside the scanned tree and conflate the two results. +# The checker is inlined deliberately. This repository is private, so a target +# repository cannot fetch scripts/audit-workflows.sh from it: raw.github- +# usercontent.com returns 404 unauthenticated, and the target repo's +# GITHUB_TOKEN has no read access here either. The ruleset already delivers +# this file from this repository, so the policy still has a single source of +# truth — it is this workflow. Keep the logic below in sync with +# scripts/audit-workflows.sh, which remains the local checker for this repo. # -# Fail-closed note: this job requests self-hosted labels. A repository that has -# not been added to the `public-node-b` runner group has no runner that can -# accept it, so the check stays queued and the pull request cannot merge. That -# is intended — it forces runner-group membership to be configured rather than -# letting a repository quietly fall back to GitHub-hosted runners. +# Exceptions: this gate is strict and reads no exceptions file (it cannot see +# one). A genuine, owner-approved exception is expressed by excluding the +# repository in the ruleset's conditions, or by adding a bypass actor — both +# are visible in the ruleset UI and in the org audit log. +# +# Fail-closed note: this job requests self-hosted labels. A repository not yet +# added to the `public-node-b` runner group has no runner able to accept it, so +# the check stays queued and the pull request cannot merge. That is intended — +# it forces runner-group membership to be configured rather than letting a +# repository quietly fall back to GitHub-hosted runners. name: Runner policy @@ -32,18 +40,67 @@ jobs: steps: - uses: actions/checkout@v5 - - name: Fetch organization policy - env: - POLICY_RAW: https://raw.githubusercontent.com/TheDancingDeveloper-org/github-policy/main + - name: Audit runner selection + shell: bash run: | set -euo pipefail - mkdir -p "$RUNNER_TEMP/policy" - curl -fsSL -o "$RUNNER_TEMP/policy/audit-workflows.sh" \ - "$POLICY_RAW/scripts/audit-workflows.sh" - curl -fsSL -o "$RUNNER_TEMP/policy/runner-exceptions.json" \ - "$POLICY_RAW/runner-exceptions.json" + python3 - <<'PY' + import pathlib + import re + import sys - - name: Audit runner selection - env: - RUNNER_EXCEPTION_FILE: ${{ runner.temp }}/policy/runner-exceptions.json - run: bash "$RUNNER_TEMP/policy/audit-workflows.sh" . + def indent(line: str) -> int: + return len(line) - len(line.lstrip(' ')) + + def values(lines: list[str], index: int) -> list[str]: + line = lines[index] + base = indent(line) + value = line.split(':', 1)[1].split('#', 1)[0].strip() + if value: + return [p.strip().strip('"\'') for p in value.strip('[]').split(',') if p.strip()] + result = [] + for child in lines[index + 1:]: + stripped = child.strip() + if not stripped or stripped.startswith('#'): + continue + if indent(child) <= base: + break + match = re.match(r'^-\s*([^#]+)', stripped) + if match: + result.append(match.group(1).strip().strip('"\'')) + return result + + failed = False + workflows = sorted( + p for pattern in ('.github/workflows/*.yml', '.github/workflows/*.yaml') + for p in pathlib.Path('.').glob(pattern) + ) + if not workflows: + print('no workflows found; nothing to audit') + + for path in workflows: + lines = path.read_text(encoding='utf-8').splitlines() + for number, line in enumerate(lines): + if not re.match(r'^\s*runs-on\s*:', line, re.I): + continue + selected = values(lines, number) + if not any(v.lower() == 'self-hosted' for v in selected): + print(f'{path}:{number + 1}: runner selection is not explicitly ' + f'self-hosted -> {selected}', file=sys.stderr) + failed = True + if any('${{' in v for v in selected): + print(f'{path}:{number + 1}: dynamic runner selection requires ' + f'explicit review', file=sys.stderr) + failed = True + + if failed: + print('', file=sys.stderr) + print('Organization policy: every job must select a self-hosted runner ' + 'explicitly, e.g.', file=sys.stderr) + print(' runs-on: [self-hosted, node-b, linux, x64]', file=sys.stderr) + print(' runs-on: [self-hosted, node-b, linux, x64, docker, publish] ' + '# needs Docker', file=sys.stderr) + sys.exit(1) + + print(f'ok: {len(workflows)} workflow file(s) audited, all self-hosted') + PY diff --git a/README.md b/README.md index 31a1219..11b5c06 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,25 @@ intentional: it surfaces missing runner-group membership instead of letting a repository quietly fall back to GitHub-hosted runners, which is exactly how the 2026-08-03 violations arose. +### Why the gate inlines its checker + +This repository is **private**. A target repository cannot fetch +`scripts/audit-workflows.sh` from it — `raw.githubusercontent.com` returns 404 +unauthenticated, and the target repo's `GITHUB_TOKEN` has no read access here. +The first version of the gate tried exactly that and failed with `curl: (22) +404` on its own pull request. + +So `enforce-runner-policy.yml` carries the checker inline. The ruleset already +delivers that file from this repository, so there is still one source of truth +— it is the workflow. `scripts/audit-workflows.sh` remains the local checker +used by `audit.yml`. **Keep the two in sync**; they implement the same two +rules (explicit `self-hosted`, no dynamic `runs-on`). + ## Exceptions +`runner-exceptions.json` is consumed by `audit.yml` only, which fails on any +entry whose `expires_on` has passed so exceptions cannot rot silently: + ```json { "schema_version": 1, "exceptions": [ @@ -44,8 +61,12 @@ repository quietly fall back to GitHub-hosted runners, which is exactly how the "expires_on": "2026-09-30" } ] } ``` -The audit fails on any exception whose `expires_on` has passed, so exceptions -cannot rot silently. +The org-wide gate cannot read that file (see above), so it is strict. A +genuine, owner-approved exception is expressed in the ruleset itself — exclude +the repository in the ruleset conditions, or add a bypass actor. Both are +visible in the ruleset UI and the organization audit log, which is a stronger +record than a JSON entry. Record the reason and expiry in +`runner-exceptions.json` as well so the daily expiry check still surfaces it. ## Related From b96ff2e4523bba6c5ace149a04689c37b72b0885 Mon Sep 17 00:00:00 2001 From: TheDancingDeveloper Date: Mon, 3 Aug 2026 00:48:52 +0000 Subject: [PATCH 3/3] fix(policy): stop the inlined checker breaking workflow parsing The dynamic-runner rule needs to look for a literal GitHub Actions expression delimiter. Writing it literally inside the run block meant Actions tried to evaluate it while parsing the workflow, so the file never became a valid workflow at all -- the run showed up under its raw filename with no resolvable name and failed before executing a step. Assemble the delimiter at runtime instead. Verified the workflow now parses and that the dynamic-runner rule still fires against a violating tree. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/enforce-runner-policy.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/enforce-runner-policy.yml b/.github/workflows/enforce-runner-policy.yml index 774bd4a..06d7191 100644 --- a/.github/workflows/enforce-runner-policy.yml +++ b/.github/workflows/enforce-runner-policy.yml @@ -70,6 +70,13 @@ jobs: result.append(match.group(1).strip().strip('"\'')) return result + # Assembled at runtime on purpose. A literal dollar-brace-brace in + # this file would be parsed as a GitHub Actions expression before the + # script ever runs, which breaks the whole workflow -- the first + # version of this file failed with an unresolvable workflow name for + # exactly that reason. + EXPR = '$' + '{' + '{' + failed = False workflows = sorted( p for pattern in ('.github/workflows/*.yml', '.github/workflows/*.yaml') @@ -88,7 +95,7 @@ jobs: print(f'{path}:{number + 1}: runner selection is not explicitly ' f'self-hosted -> {selected}', file=sys.stderr) failed = True - if any('${{' in v for v in selected): + if any(EXPR in v for v in selected): print(f'{path}:{number + 1}: dynamic runner selection requires ' f'explicit review', file=sys.stderr) failed = True