Skip to content

ci: shared tests and release on merge to stable, version from Cargo.toml - #96

Closed
singlerider wants to merge 25 commits into
blacklanternsecurity:devfrom
singlerider:ci/publish-on-tag
Closed

singlerider wants to merge 25 commits into
blacklanternsecurity:devfrom
singlerider:ci/publish-on-tag

Conversation

@singlerider

@singlerider singlerider commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

What changed

  • Publishing removed from rust-tests.yml and python-tests.yml, which used to publish on every stable push with long-lived tokens. Neither triggers on push: stable any more, so they no longer duplicate the calls from publish.yml.
  • python-tests.yml and rust-tests.yml call CLA's shared workflows. The Python matrix comes from requires-python. ruff format --check now runs, and the extension is built with maturin develop. Rust runs --all-features --locked on the rust-toolchain.toml channel. CODECOV_TOKEN is passed by name. Until the default branch ruleset requires python / passed, a required job reports test (3.11) to (3.15) from the real result.
  • publish.yml runs on a push to stable: CLA's release-check.yml, then rust and python tests, then CLA's maturin-wheels.yml (every interpreter requires-python allows, on Windows and macOS too), then PyPI with OIDC in release, then crates.io through CLA's crates.yml with trusted publishing, then CLA's publish.yml (tag and GitHub Release with SBOM and wheels, created last). No caller passes App credentials.
  • stable-guard.yml: release ready on pull requests into stable refuses a non-MAJOR.MINOR.PATCH version and git or URL dependencies.
  • cla.yml passes APP_ID/APP_PRIVATE_KEY by name. No workflow uses secrets: inherit. benchmark.yml declares contents: read and pull-requests: write and pins its runner.
  • Cargo.toml is the single version source, now 4.3.1: 4.3.0 is on PyPI and crates.io with no v4.3.0 tag.
  • Dead poetry.lock and .flake8 removed. Merges current dev (dependabot grouping, shared auto-merge).
  • Every CLA reference is pinned to CLA#11 at 07e14bcbba5e737e1b29bd9e3f9128fbf2483be9. Re-pin to its merge commit before this merges.

Before promoting to stable

  • release environment with reviewers and a stable-only deployment policy, holding BLS_RELEASE_APP_ID and BLS_RELEASE_PRIVATE_KEY.
  • Trusted publishers on PyPI and crates.io for publish.yml, environment release. Then delete the PyPI and crates.io token secrets.
  • Ruleset protecc (11169658), and the matching .github-config#1/Prune / Defragment branches to save space #2 Terraform: require python / passed, rust / passed and release ready, then drop the required job.

Validation

$ cargo test --all-features --locked
86 passed
$ cargo metadata --locked --no-deps
(lock consistent at 4.3.1)
$ uvx --from actionlint-py actionlint
(no findings)

Related issues

  • Part of blacklanternsecurity/bbot-enterprise#139
  • Part of blacklanternsecurity/bbot-enterprise#140
  • Part of blacklanternsecurity/bbot-enterprise#149
  • Part of blacklanternsecurity/bbot-enterprise#155
  • Part of blacklanternsecurity/bbot-enterprise#156

TheTechromancer and others added 14 commits December 16, 2025 10:23
ubuntu-latest is a label GitHub moves. The image changes under us and
a green pipeline turns red on a morning nobody touched the code.
Pinning to ubuntu-24.04 means the runner moves when we move it.

dtolnay/rust-toolchain was called at @stable, a branch, not a release.
It is pinned to a commit, dated in the trailing comment because that
repo publishes no release tag for it. Version-tagged actions are
untouched.
The lint job ran an unpinned `pipx install ruff`, which picked up 0.16.9
and failed on 8 pre-existing errors (import order, __slots__ sort,
SIM102, SIM201). The Rust job tracked `stable`, whose clippy now denies
for_kv_map in src/node/base.rs. Both blocked unrelated PRs such as #92.

Pin ruff==0.16.9 in the workflow and dev group, pin Rust 1.97.1 with
clippy and rustfmt in rust-toolchain.toml and rust-tests.yml, and fix
the errors in source.
Fork PRs get a read-only GITHUB_TOKEN, so the comment step failed the
job after the benchmark itself passed.
windows-latest and macos-latest were still floating in the wheel
matrix.
rust-tests.yml and python-tests.yml published to crates.io and PyPI on
every push to stable, each with a long-lived token. Publishing moves to
publish.yml, triggered only by v* tags. Its first job refuses tags
outside vMAJOR.MINOR.PATCH or vMAJOR.MINOR.PATCH-rc.N and any tag that
does not equal the Cargo.toml version. Both test workflows run before
any wheel is built, every build must succeed before a registry sees
the release, PyPI uses trusted publishing in the pypi environment,
crates.io runs in the release environment, and the last job creates
the GitHub Release with an SPDX JSON SBOM attached.

actions-rs/toolchain is archived and is replaced by
actions-rust-lang/setup-rust-toolchain reading rust-toolchain.toml.
Every workflow declares least-privilege permissions, every action is
pinned to a commit SHA, and CLA calls the shared reusable workflow.
@codecov-commenter

codecov-commenter commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100%. Comparing base (9fc51ef) to head (3bdc148).
⚠️ Report is 33 commits behind head on dev.

Additional details and impacted files
@@         Coverage Diff         @@
##            dev    #96   +/-   ##
===================================
  Coverage   100%   100%           
===================================
  Files         3      3           
  Lines       604    605    +1     
===================================
+ Hits        604    605    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ruff 0.16 formats Python blocks in Markdown, so ruff format --check
failed on README.md and the lint job only ran ruff check. Formatting
it lets lint run both checks.
pyproject.toml and Cargo.toml both declared 4.2.0, two sources that
publish.yml could only compare against one of. maturin fills a dynamic
[project] version from Cargo.toml, so Cargo.toml is now the single
source for crates.io, PyPI, and the tag check.

poetry.lock and .flake8 are left over from before the move to uv and
ruff. Nothing reads them.
#139 replaces long-lived registry tokens with OIDC. crates.io supports
trusted publishing: rust-lang/crates-io-auth-action exchanges the job's
OIDC token for a short-lived publish token, so CARGO_REGISTRY_TOKEN no
longer needs to exist as a secret.
rust-tests.yml, python-tests.yml, the CLA caller, and the release
steps of publish.yml now use the org workflows in
blacklanternsecurity/CLA at one pinned SHA instead of carrying copies.

- python: python-tests.yml derives the matrix from requires-python,
  replacing the literal 3.9 to 3.14 list, runs ruff check and ruff
  format --check (format was never checked here), and builds the
  extension with maturin develop when maturin is the backend.
- rust: rust-tests.yml with --all-features --locked on the
  rust-toolchain.toml channel.
- publish: release-check.yml validates the tag against Cargo.toml (now
  the single version source). The wheel matrix and the crates.io and
  PyPI uploads stay here because they are radixtarget-specific and PyPI
  trusted publishing does not work in a reusable workflow. publish.yml
  creates the release with the SPDX SBOM and attaches the wheels.
- archive/** is no longer excluded: archive tags are not created.
Picks up secret-env skipping absent secrets on fork PRs and rust-tests
running setup-script before clippy. No caller input changes.
@singlerider singlerider changed the title ci: publish on version tags only ci: shared tests and tag-only publish, version from Cargo.toml Oct 3, 2026
e532142 sits on feat/shared-workflows, which was squash-merged and
can stop resolving once the branch is deleted. 40c6e18 is CLA main and
a superset: it adds optional registry login for image SBOMs to
publish.yml, with defaults that leave existing callers unchanged.

Refs blacklanternsecurity/bbot-enterprise#139

@liquidsec liquidsec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will mean anyone with write access can publish. This needs to be discussed.

Follows .github-config#17 through CLA#11: publish.yml runs on push to
the trunk instead of a v* tag, publishes in the release environment a
bbot-leads member approves, re-runs safely (PyPI skip-existing,
crates.io only when the version is new), and leaves the tag and the
GitHub release to the bls-release App at the end of the run. The
pypi environment and the long-lived registry token are gone.
# Conflicts:
#	.github/workflows/benchmark.yml
#	.github/workflows/python-tests.yml
#	pyproject.toml
#	radixtarget/test/test_rust_radixtarget.py
#	uv.lock
@singlerider

Copy link
Copy Markdown
Contributor Author

@liquidsec reworked to the approved publishing policy (.github-config#17), through CLA#11. This answers "anyone with write access can publish" and the review and pipeline gates:

  • publish.yml runs on a merge to the trunk, not on a tag. A release is a reviewed PR into the trunk that bumps the version.
  • Every publishing job runs in the release environment, which a bbot-leads member other than the run's starter must approve; admins can't skip it.
  • Tests run before anything publishes. Re-runs are safe: PyPI skip-existing, crates.io only if the version is new.
  • The tag and GitHub release come last, from the bls-release App. The tag rulesets in .github-config#1 block everyone else.
  • The pypi environment and the long-lived registry tokens are gone.

@singlerider
singlerider requested a review from liquidsec October 21, 2026 12:55
@singlerider
singlerider requested a review from en0f October 21, 2026 13:06

@pjhartlieb pjhartlieb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@singlerider, I reviewed this at 2e983b3b (based on current dev, 950cd0dd) against the .github-config#16 spec part B, the publishing policy, and CLA#11 at 4894c4ad. The inputs match CLA#11's interface.

The structure is right: push: stable, release-check, then rust and python tests, then the wheel matrix, then crates.io (trusted publishing, existence check) and PyPI (release, OIDC, --skip-existing), then the release last. The old publishers are gone: python-tests.yml (maturin publish --password ${{ secrets.PYPI_TOKEN }} at line 61, and the MATURIN_PYPI_TOKEN upload at line 240) and rust-tests.yml (cargo publish --token at line 58), on both dev and stable.

High

H1. The release App key can't come from the release environment (C1).
publish.yml:219 and :225 pass vars.RELEASE_APP_ID / secrets.RELEASE_APP_KEY from a uses: job, which can't hold environment:. Neither exists, and the spec's names are BLS_RELEASE_APP_ID / BLS_RELEASE_PRIVATE_KEY, held in release. As written, the release fails after crates.io and PyPI have published, or the key moves to repo or org level.
Requested: once CLA#11 reads them inside its environment job, drop release-app-id and the secrets: block.

H2. The required checks on stable will never report.
Ruleset protecc (11169658) requires test (3.11)–test (3.15). This head reports python / test (3.11)…(3.15), python / passed, rust / test and rust / passed.
Requested: change the ruleset (and .github-config#1/#2, which the spec says now lists test (3.11)–(3.15)) to the new contexts before the promotion, or keep the live names until then.

Medium

M1. Windows and macOS wheels drop to one Python version.
publish.yml:111–114 and :139–141 set python-version: 3.x, so --find-interpreter sees one interpreter. dev lists 3.11–3.15 for exactly this reason (its comment reads "maturin --find-interpreter only builds for interpreters on the runner"). The crate isn't abi3 (Cargo.toml:19; pyproject.toml features ["py"]), and 4.3.0 on PyPI has cp311, cp312 and cp313 wheels for win32, win_amd64 and both macOS targets. After this change, users on the other versions would build from the sdist.
Requested: call CLA's maturin-wheels.yml (interpreters come from requires-python), as blastdns#139 does, or restore the version list.

M2. crates.io publishes in parallel with PyPI, not after it.
crates (publish.yml:168–170) and pypi (:195–197) both depend only on the builds. The policy and spec order is PyPI with skip-existing, then crates.io if absent, then the tag.
Requested: crates: needs: [check, pypi]. You could also replace the inline job with CLA's crates.yml (same existence check and OIDC) so there's one copy.

M3. 4.3.0 is already on PyPI and crates.io with no v4.3.0 release (C3).
The first stable merge would tag a different commit as 4.3.0 and attach wheels built from it.
Requested: bump in the promotion PR, or tag 4.3.0's real commit before cut-over.

M4. Stale description and pin (C2, C4, C5).
The description says v* tags, and its admin step registers PyPI against environment pypi.
Requested: change the environment to release, re-pin to the CLA#11 merge commit (the test files and cla.yml are on 40c6e18), and hold the promotion until release is protected.

Low

  • L1. secrets: inherit (publish.yml:27, python-tests.yml:22, cla.yml:16) passes CARGO_REGISTRY_TOKEN, PYPI_TOKEN and the CLA App key to code that needs CODECOV_TOKEN (or APP_ID/APP_PRIVATE_KEY). Pass these by name (C6).
  • L2. Duplicate test runs. python-tests.yml and rust-tests.yml still run on push: stable alongside the calls from publish.yml, in the same groups (C7).
  • L3. benchmark.yml has no permissions: block, so it runs with the repo default. Set contents: read and pull-requests: write.

Verdict

Request changes: H1 (depends on CLA#11), H2 before the promotion, and M1–M4.

Next: we're making these changes ourselves in a follow-up PR, which will be linked here when it opens. This review is the record of what that PR addresses.

@en0f en0f left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one real functional break. The PR is also blocked from merging right now: two reviewers have requested changes. All 14 checks pass, but none of them run publish.yml, so the release path below hasn’t been tested.

Windows and macOS wheels will only be built for one Python version. In .github/workflows/publish.yml, the windows and macos jobs set python-version: 3.x. The old workflow on dev deliberately installed 3.11 through 3.15, with a comment saying --find-interpreter only builds for interpreters that are on the runner. Cargo.toml doesn’t use abi3 builds, so each Python version needs its own wheel. As written, a release would publish Windows and macOS wheels for only the newest Python. Users on 3.11–3.14 would fall back to the sdist, which needs a Rust toolchain to install. Linux and musllinux builds are unaffected because they run in the manylinux containers, which include every interpreter. The fix is to put the old list back in both jobs:

yaml
python-version: |
3.11
3.12
3.13
3.14
3.15

Needs confirming before the first release: PyPI and crates.io must already trust this workflow. The pypi and crates jobs now publish without stored tokens, so both registries need a trusted publisher set up for publish.yml with the release environment. If that isn’t configured, the first push to stable with a new version fails at upload.

@singlerider singlerider added the github_actions Pull requests that update GitHub Actions code label Oct 22, 2026
- publish: build wheels through CLA's maturin-wheels, which installs
  every interpreter requires-python allows, so Windows and macOS ship
  per-version wheels again; publish crates.io through CLA's crates.yml
  after PyPI; pass no App credentials
- python-tests, rust-tests, cla: CODECOV_TOKEN and the CLA App
  credentials by name instead of secrets: inherit; drop the stable push
  triggers that duplicated the publish calls
- python-tests: keep reporting the required test (3.11) to (3.15)
  contexts from the real result until the ruleset moves
- stable-guard: refuse a pull request into stable whose version or
  dependencies cannot release
- benchmark: explicit permissions and a pinned runner
- re-pin every CLA reference to CLA#11
# Conflicts:
#	.github/workflows/benchmark.yml
#	.github/workflows/python-tests.yml
#	.github/workflows/rust-tests.yml
4.3.0 is on PyPI and crates.io with no v4.3.0 tag, so release-check
would refuse it as partly published.
@singlerider singlerider changed the title ci: shared tests and tag-only publish, version from Cargo.toml ci: shared tests and release on merge to stable, version from Cargo.toml Oct 23, 2026

@en0f en0f left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared workflows point at an unmerged branch. Every shared workflow call in the PR, including the dependabot auto-merge one that used to point at CLA main, uses blacklanternsecurity/CLA@07e14bc. That commit is only on CLA#11’s open branch (feat/release-on-stable), not on main. The PR description says to switch these to CLA#11’s merge commit before merging, and that hasn’t happened. If this merges first, dev runs its tests, CLA check and dependabot auto-merge off unreviewed CLA code.

@pjhartlieb pjhartlieb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@singlerider, re-reviewed at 3bdc1486.

From the 2026-10-21 review

Status at 3bdc1486
H1 App credentials Fixed. No App lines in callers; publish.yml:66-76
H2 required checks on stable Fixed. python-tests.yml:26-38 reports test (3.11)–(3.15), all green
M1 Windows/macOS wheels Fixed for CPython: publish.yml:30-34 calls CLA maturin-wheels.yml; all 14 platforms 4.3.0 shipped are built. See 2 below
M2 crates after PyPI Fixed. publish.yml:57
M3 4.3.0 already published Fixed. Cargo.toml:3 4.3.1: no tag, not on PyPI or crates.io. A bump into dev only releases on promotion, and CLA's release-check refuses a partly published version
M4 description, pin Description fixed. Pin: see 1 below
L1 secrets: inherit Fixed. publish.yml:27-28, python-tests.yml:23-24, cla.yml:16-18
L2 duplicate runs Fixed. No test workflow triggers on push: stable
L3 benchmark.yml permissions Fixed. benchmark.yml:6-8

Still open

  1. Pin. Every CLA ref is CLA#11's unmerged head 07e14bcb. Re-pin to the CLA merge commit before this merges, as the description says.
  2. dependabot-auto-merge.yml is still on dev (re-pinned, not removed). Once dev drops to 0 approvals it merges Dependabot PRs on CI alone. Delete it in this PR.
  3. publish.yml:7-8 is contents: read. Make it permissions: {} and grant contents: read on check, rust, python and wheels.
  4. Before cut-over: 4.3.0 also shipped pp311 (10 wheels), cp314t and cp315t. CLA builds released CPython, default variant only (scripts/manifest.py at 07e14bcb), so those users get the sdist and need Rust. Either accept that and say so in the description, or add an interpreter input to maturin-wheels.yml.
  5. Before cut-over: a release prompts for release approval four times: release-check's credentials job, pypi, crates and the release. Put that in the release docs. The README has no Releasing section (blastdns and blasthttp have one).

Reproduce

TMPDIR=<scratch dir> bash verify-radixtarget-96.sh 3bdc148608c331763dc0f1ed4494c27eeef11f04
# expect: SUMMARY repo=radixtarget pr=96 sha=3bdc1486... pass=19 fail=3   (P13, P14, W02 = items 3, 2, 4)
bash verify-radixtarget-96.sh 2e983b3b6fb1e31baba011fe883e2c154f3a888f
# negative control, the head reviewed 2026-10-21: pass=9 fail=13 (P02 P04 P05 P06 P08 P11 P12 P13 P15.stable P16 W00 W01 W02)

It needs git, an authenticated gh (GET calls only), uv and curl. It downloads actionlint 1.7.12 and checks it against the release checksums.

verify-radixtarget-96.sh
#!/usr/bin/env bash
# verify-radixtarget-96.sh <sha>
#
# Read-only re-check of blacklanternsecurity/radixtarget#96 (publish on merge to stable)
# against the 2026-10-21 review findings and go-live plan item A4 (+ R6, R11).
# Clean blobless clone of the repo at <sha>; GitHub reads are `gh api --method GET`
# only; PyPI and crates.io reads are anonymous GETs. Nothing is written anywhere
# except a temp dir under $TMPDIR (removed on exit).
#
# Prints PASS/FAIL/INFO per check and a final SUMMARY line; exits 1 on any FAIL.
# Needs: git, gh (authenticated), uv, curl, shasum or sha256sum.
# Env: CLA_SHA (expected CLA pin, default the CLA#11 head reviewed 2026-10-23).
set -uo pipefail

OWNER=blacklanternsecurity
REPO=radixtarget
PR=96
SHA=${1:?usage: $0 <commit-sha>}
CLA_SHA=${CLA_SHA:-07e14bcbba5e737e1b29bd9e3f9128fbf2483be9}
ACTIONLINT_VERSION=1.7.12

WORK=$(mktemp -d "${TMPDIR:-$PWD}/verify-$REPO-$PR.XXXXXX")
trap 'rm -rf "$WORK"' EXIT

echo "verify $OWNER/$REPO#$PR at $SHA (CLA_SHA=$CLA_SHA) $(date -u +%Y-%m-%dT%H:%M:%SZ)"

git clone --quiet --filter=blob:none --no-checkout "https://github.com/$OWNER/$REPO.git" "$WORK/repo" || { echo "FAIL clone"; echo "SUMMARY repo=$REPO sha=$SHA pass=0 fail=1"; exit 1; }
git -C "$WORK/repo" fetch --quiet origin "$SHA" 2>/dev/null \
  || git -C "$WORK/repo" fetch --quiet origin "+refs/pull/$PR/head:refs/remotes/origin/pr"
git -C "$WORK/repo" -c advice.detachedHead=false checkout --quiet "$SHA" || { echo "FAIL checkout $SHA"; echo "SUMMARY repo=$REPO sha=$SHA pass=0 fail=1"; exit 1; }

# actionlint, downloaded from the release and verified against the release's checksums file
case "$(uname -s)_$(uname -m)" in
  Darwin_arm64) PLAT=darwin_arm64 ;; Darwin_x86_64) PLAT=darwin_amd64 ;;
  Linux_x86_64) PLAT=linux_amd64 ;; Linux_aarch64|Linux_arm64) PLAT=linux_arm64 ;;
  *) PLAT="" ;;
esac
ACTIONLINT=""
if [ -n "$PLAT" ]; then
  base="https://github.com/rhysd/actionlint/releases/download/v$ACTIONLINT_VERSION"
  tgz="actionlint_${ACTIONLINT_VERSION}_${PLAT}.tar.gz"
  sums="actionlint_${ACTIONLINT_VERSION}_checksums.txt"
  mkdir -p "$WORK/al"
  if curl -sSfL -o "$WORK/al/$tgz" "$base/$tgz" && curl -sSfL -o "$WORK/al/$sums" "$base/$sums"; then
    want=$(grep " $tgz\$" "$WORK/al/$sums" | cut -d' ' -f1)
    got=$( (shasum -a 256 "$WORK/al/$tgz" 2>/dev/null || sha256sum "$WORK/al/$tgz") | cut -d' ' -f1)
    if [ -n "$want" ] && [ "$want" = "$got" ]; then
      tar -xzf "$WORK/al/$tgz" -C "$WORK/al" actionlint && ACTIONLINT="$WORK/al/actionlint"
      echo "INFO actionlint $ACTIONLINT_VERSION $PLAT sha256 $got matches the release checksums file"
    else
      echo "INFO actionlint sha256 mismatch (want=$want got=$got); not used"
    fi
  fi
fi

cat > "$WORK/check.py" <<'PY'
import base64, json, os, re, subprocess, sys, tomllib, urllib.request, urllib.error
import yaml

owner, repo, pr, sha, cla_sha, root, actionlint = sys.argv[1:8]
wfdir = os.path.join(root, ".github", "workflows")
results = []

def out(status, cid, desc, detail=""):
    results.append(status)
    print(f"{status} {cid} {desc}" + (f" -- {detail}" if detail else ""))

def check(cid, desc, ok, detail=""):
    out("PASS" if ok else "FAIL", cid, desc, detail)

def info(cid, desc, detail=""):
    print(f"INFO {cid} {desc}" + (f" -- {detail}" if detail else ""))

def gh(path):
    cmd = ["gh", "api", "--method", "GET", path]
    p = subprocess.run(cmd, capture_output=True, text=True)
    return p.returncode, p.stdout

def http_status(url):
    req = urllib.request.Request(url, headers={"User-Agent": f"{owner}-{repo}-pr-review-readonly"})
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            return r.status, r.read()
    except urllib.error.HTTPError as e:
        return e.code, b""

def load(path):
    with open(path) as f:
        return yaml.safe_load(f)

def triggers(wf):
    on = wf.get("on", wf.get(True))
    if isinstance(on, str):
        return {on: None}
    if isinstance(on, list):
        return {k: None for k in on}
    return on or {}

files = sorted(f for f in os.listdir(wfdir) if f.endswith((".yml", ".yaml"))) if os.path.isdir(wfdir) else []
wfs = {f: load(os.path.join(wfdir, f)) for f in files}
texts = {f: open(os.path.join(wfdir, f)).read() for f in files}
pub = wfs.get("publish.yml")
jobs = (pub or {}).get("jobs", {}) or {}

def needs(job):
    n = (jobs.get(job) or {}).get("needs", [])
    return [n] if isinstance(n, str) else list(n)

def job_using(fragment):
    return [j for j, d in jobs.items() if fragment in str((d or {}).get("uses", ""))]

# P01 trigger
t = triggers(pub or {})
push = t.get("push") or {}
check("P01", "publish.yml triggers only on push to stable",
      pub is not None and set(t) == {"push"} and push.get("branches") == ["stable"] and not push.get("tags"),
      f"on={json.dumps(t, default=str)}")

# P02 job order: tests -> wheels -> pypi -> crates -> release
wheels = job_using("maturin-wheels.yml") or [j for j in jobs if j == "wheels"]
crates = job_using("CLA/.github/workflows/crates.yml") or [j for j in jobs if j == "crates"]
rel = job_using("CLA/.github/workflows/publish.yml")
pypi = [j for j, d in jobs.items() if "pypa/gh-action-pypi-publish" in json.dumps(d) or "\"command\": \"upload\"" in json.dumps(d)]
local_tests = [j for j, d in jobs.items() if str((d or {}).get("uses", "")).startswith("./")]
order = []
if wheels and pypi and crates and rel:
    w, p, c, r = wheels[0], pypi[0], crates[0], rel[0]
    order = [set(local_tests) <= set(needs(w)) and bool(local_tests), w in needs(p), p in needs(c), {c, p} <= set(needs(r))]
check("P02", "job order tests -> wheels -> pypi -> crates -> release",
      bool(order) and all(order),
      f"tests={local_tests} wheels.needs={needs(wheels[0]) if wheels else None} pypi.needs={needs(pypi[0]) if pypi else None} "
      f"crates.needs={needs(crates[0]) if crates else None} release.needs={needs(rel[0]) if rel else None}")

# P03 pypi job
pj = jobs.get(pypi[0]) if pypi else {}
steps = json.dumps((pj or {}).get("steps", []))
check("P03", "pypi job: environment release, id-token: write, skip-existing",
      bool(pj) and pj.get("environment") == "release" and (pj.get("permissions") or {}).get("id-token") == "write"
      and re.search(r'"skip-existing": (true|"true")|--skip-existing', steps) is not None,
      f"environment={pj.get('environment') if pj else None} permissions={pj.get('permissions') if pj else None}")

# P04 crates via CLA crates.yml with id-token
cj = jobs.get(crates[0]) if crates else {}
check("P04", "crates job calls CLA crates.yml and grants id-token: write",
      bool(cj) and "CLA/.github/workflows/crates.yml" in str(cj.get("uses", "")) and (cj.get("permissions") or {}).get("id-token") == "write",
      f"permissions={cj.get('permissions') if cj else None}")

alltext = "\n".join(f"{f}:{i+1}:{l}" for f, s in texts.items() for i, l in enumerate(s.splitlines()))

def hits(rx):
    return [l for l in alltext.splitlines() if re.search(rx, l.split(":", 2)[2])]

# P05 no App credential lines in callers
h = hits(r"release-app-|RELEASE_APP_|BLS_RELEASE_")
check("P05", "no release App credential lines in callers (CLA reads them in `release`)", not h, "; ".join(h[:4]))

# P06 no secrets: inherit
h = hits(r"secrets:\s*inherit")
check("P06", "no `secrets: inherit` in any workflow", not h, "; ".join(h[:4]))

# P07 SHA pins
bad = []
for l in alltext.splitlines():
    m = re.search(r"^\s*(?:-\s+)?uses:\s*([^\s#]+)", l.split(":", 2)[2])
    if m and not m.group(1).startswith("./") and not re.search(r"@[0-9a-f]{40}$", m.group(1)):
        bad.append(l.strip())
check("P07", "every `uses:` is local or pinned to a full SHA", not bad, "; ".join(bad[:4]))

# P08 / P09 CLA pins
cla_refs = sorted({m.group(1, 2) for m in re.finditer(r"blacklanternsecurity/CLA/(\.github/workflows/[\w.-]+)@([0-9a-f]{7,40})", alltext)})
pins = sorted({s for _, s in cla_refs})
check("P08", f"every CLA reference pinned to CLA_SHA {cla_sha[:8]}", pins == [cla_sha], f"pins={[p[:8] for p in pins]}")
missing = []
for path, s in cla_refs:
    rc, _ = gh(f"repos/blacklanternsecurity/CLA/contents/{path}?ref={s}")
    if rc != 0:
        missing.append(f"{path}@{s[:8]}")
check("P09", "every referenced CLA workflow exists at its pin", bool(cla_refs) and not missing,
      f"{len(cla_refs)} refs" + (f"; missing {missing}" if missing else ""))

# P10 old publish paths
h = hits(r"PYPI_TOKEN|PYPI_API_TOKEN|MATURIN_PYPI_TOKEN|secrets\.CARGO_REGISTRY_TOKEN|cargo publish .*--token|maturin publish|twine upload|gh release create|git push .*--tags|git tag ")
extra = [f for f in ("release.toml", "release.sh") if os.path.exists(os.path.join(root, f))]
check("P10", "no old publish path in any workflow (stored tokens, cargo publish --token, maturin publish, tag pushes)", not h and not extra, "; ".join(h[:4] + extra))

# P11 no other workflow runs on push to stable (duplicate test runs, C7)
dup = []
for f, wf in wfs.items():
    if f == "publish.yml":
        continue
    tt = triggers(wf)
    if "push" in tt:
        b = (tt.get("push") or {}).get("branches")
        if b is None or "stable" in b or any("*" in x for x in b):
            dup.append(f)
check("P11", "no workflow besides publish.yml triggers on push to stable", not dup, f"{dup}")

# P12 explicit top-level permissions (R6)
noperm = [f for f, wf in wfs.items() if "permissions" not in wf]
check("P12", "every workflow declares top-level permissions (R6)", not noperm, f"{noperm}")

# P13 permissions: {} on publish.yml (A4)
check("P13", "publish.yml top-level `permissions: {}` (A4)", pub is not None and pub.get("permissions") == {},
      f"permissions={pub.get('permissions') if pub else None}")

# P14 dependabot-auto-merge removed from dev (R11)
dam = [f for f in files if "dependabot-auto-merge" in f]
check("P14", "dependabot-auto-merge workflow removed (R11)", not dam, f"{dam}")

# P15 live required checks reported at this SHA
p = subprocess.run(["gh", "api", "--method", "GET", "--paginate", f"repos/{owner}/{repo}/commits/{sha}/check-runs?per_page=100",
                    "--jq", ".check_runs[] | {name, status, conclusion, app: {id: .app.id}}"], capture_output=True, text=True)
runs = [json.loads(l) for l in p.stdout.splitlines() if l.strip()]
reported = {(r["name"], (r.get("app") or {}).get("id")) for r in runs}
for branch in ("dev", "stable"):
    rc, body = gh(f"repos/{owner}/{repo}/rules/branches/{branch}")
    req = []
    for rule in json.loads(body or "[]"):
        if rule.get("type") == "required_status_checks":
            p = rule["parameters"]
            req += [(c["context"], c.get("integration_id"), rule.get("ruleset_id"), p.get("strict_required_status_checks_policy")) for c in p["required_status_checks"]]
    miss = [c for c, i, _, _ in req if not any(n == c and (i is None or a == i) for n, a in reported)]
    strict = sorted({s for *_, s in req})
    check(f"P15.{branch}", f"every live `{branch}` required check is reported at this SHA",
          not miss, f"required={len(req)} strict={strict} missing={miss}")

# P16 version not already published (C3)
cargo = tomllib.load(open(os.path.join(root, "Cargo.toml"), "rb"))
pyproj = tomllib.load(open(os.path.join(root, "pyproject.toml"), "rb"))
ver = cargo["package"]["version"]
crate = cargo["package"]["name"]
pyname = pyproj.get("project", {}).get("name")
pyver = pyproj.get("project", {}).get("version")
rc_tag, _ = gh(f"repos/{owner}/{repo}/git/ref/tags/v{ver}")
s_pypi, _ = http_status(f"https://pypi.org/pypi/{pyname}/{ver}/json")
s_crates, _ = http_status(f"https://crates.io/api/v1/crates/{crate}/{ver}")
check("P16", f"manifest version {ver} is unreleased (no v{ver} tag, not on PyPI or crates.io)",
      rc_tag != 0 and s_pypi == 404 and s_crates == 404 and (pyver in (None, ver)),
      f"tag_exists={rc_tag == 0} pypi={s_pypi} crates={s_crates} pyproject.version={pyver}")

# P17 actionlint
if actionlint:
    p = subprocess.run([actionlint, "-shellcheck=", "-pyflakes=", "-no-color"], cwd=root, capture_output=True, text=True)
    check("P17", "actionlint 1.7.12 clean (shellcheck/pyflakes integration off)", p.returncode == 0, (p.stdout + p.stderr).strip().replace("\n", " | ")[:600])
else:
    out("FAIL", "P17", "actionlint 1.7.12 clean", "actionlint unavailable or checksum mismatch")

# P18 CI at this SHA
bad = sorted({r["name"] for r in runs if r.get("status") == "completed" and r.get("conclusion") not in ("success", "skipped", "neutral")})
pend = sorted({r["name"] for r in runs if r.get("status") != "completed"})
check("P18", "every check run at this SHA concluded success/skipped/neutral", bool(runs) and not bad and not pend,
      f"runs={len(runs)} failing={bad} pending={pend}")

# W: wheel matrix through CLA maturin-wheels.yml, compared with the latest PyPI release
wj = jobs.get(wheels[0]) if wheels else None
via_cla = bool(wj) and "CLA/.github/workflows/maturin-wheels.yml" in str(wj.get("uses", ""))
check("W00", "wheels built by CLA maturin-wheels.yml", via_cla, f"wheels job={wheels[0] if wheels else None}")
s, raw = http_status(f"https://pypi.org/pypi/{pyname}/json")
latest = json.loads(raw) if s == 200 else {}
wheels_pypi = [u["filename"] for u in latest.get("urls", []) if u["filename"].endswith(".whl")]
arch = {"x86_64": "x86_64", "i686": "x86", "aarch64": "aarch64", "armv7l": "armv7", "s390x": "s390x", "ppc64le": "ppc64le",
        "amd64": "x64", "win32": "x86", "arm64": "aarch64"}
def fam(plat):
    a = next((v for k, v in arch.items() if plat.endswith(k)), plat)
    for prefix, family in (("musllinux", "musllinux"), ("manylinux", "linux"), ("linux", "linux"), ("macosx", "macos"), ("win", "windows")):
        if plat.startswith(prefix):
            return family, a
    return "other", plat
shipped = set()
interp = set()
for fn in wheels_pypi:
    parts = fn[:-4].split("-")
    for plat in parts[4].split("."):
        f, a = fam(plat)
        shipped.add((f, a))
    interp.add(f"{parts[2]}-{parts[3]}")
if via_cla:
    m = re.search(r"maturin-wheels\.yml@([0-9a-f]{40})", str(wj["uses"]))
    rc, src = gh(f"repos/blacklanternsecurity/CLA/contents/.github/workflows/maturin-wheels.yml?ref={m.group(1)}") if m else (1, "")
    clawf = yaml.safe_load(base64.b64decode(json.loads(src)["content"]).decode()) if rc == 0 else {}
    default = clawf.get("on", clawf.get(True))["workflow_call"]["inputs"]["targets"]["default"]
    targets = json.loads((wj.get("with") or {}).get("targets") or default)
    built = {(f, a) for f, al in targets.items() for a in al}
    gap = sorted(shipped - built)
    check("W01", f"platforms built cover every platform of {pyname} {latest.get('info', {}).get('version')} on PyPI",
          not gap, f"shipped={len(shipped)} built={len(built)} not_built={gap}")
    rc2, src2 = gh(f"repos/blacklanternsecurity/CLA/contents/scripts/manifest.py?ref={m.group(1)}")
    mp = base64.b64decode(json.loads(src2)["content"]).decode() if rc2 == 0 else ""
    cpython_default_only = 'build["implementation"] != "cpython"' in mp and 'build["variant"] != "default"' in mp
    dropped = sorted(i for i in interp if not re.fullmatch(r"cp3(\d+)-cp3\1", i)) if cpython_default_only else []
    check("W02", "every interpreter ABI on the latest PyPI release is still built (CLA builds released CPython, default variant, only)",
          not dropped, f"pypi_abis={sorted(interp)} not_built={dropped}")
else:
    out("FAIL", "W01", "platform coverage", "depends on W00")
    out("FAIL", "W02", "interpreter coverage", "depends on W00")

# repo-specific checks
extra = os.environ.get("EXTRA_CHECKS", "")
if "blasthttp" in extra:
    rt = wfs.get("rust-tests.yml") or {}
    rtt = texts.get("rust-tests.yml", "")
    unp = [u for u in re.findall(r"(?m)^\s*(?:-\s+)?uses:\s*([^\s#]+)", rtt) if not re.search(r"@[0-9a-f]{40}$", u)]
    tt = triggers(rt)
    check("H01", "rust-tests.yml: SHA-pinned, contents: read, no push trigger, persist-credentials: false",
          bool(rt) and not unp and rt.get("permissions") == {"contents": "read"} and "push" not in tt and "persist-credentials: false" in rtt,
          f"unpinned={unp} permissions={rt.get('permissions')} triggers={list(tt)}")
    ok = "rust-tests" in jobs and "rust-tests" in needs(wheels[0] if wheels else "")
    check("H02", "publish.yml runs rust-tests.yml before the wheels", ok, f"wheels.needs={needs(wheels[0]) if wheels else None}")
    env_tests = "PYO3_USE_ABI3_FORWARD_COMPATIBILITY" in texts.get("tests.yml", "")
    env_wheels = "PYO3_USE_ABI3_FORWARD_COMPATIBILITY" in json.dumps(wj or {})
    check("H03", "PYO3_USE_ABI3_FORWARD_COMPATIBILITY passed to the shared tests and wheels (en0f)", env_tests and env_wheels,
          f"tests={env_tests} wheels={env_wheels}")
    readme = open(os.path.join(root, "README.md")).read()
    links = sorted(set(re.findall(r"https://github\.com/blacklanternsecurity/\.github[\w.-]*/blob/[^\s)]+", readme)))
    st = {l: http_status(l)[0] for l in links}
    check("H04", "README release-policy link resolves for an anonymous reader", bool(links) and all(v == 200 for v in st.values()), f"{st}")

n_pass, n_fail = results.count("PASS"), results.count("FAIL")
print(f"SUMMARY repo={repo} pr={pr} sha={sha} pass={n_pass} fail={n_fail}")
sys.exit(1 if n_fail else 0)
PY

echo "INFO env release environment exists: $(gh api --method GET "repos/$OWNER/$REPO/environments" --jq '[.environments[].name] | index("release") != null' 2>/dev/null || echo unknown)"
EXTRA_CHECKS="" uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.2 python -I "$WORK/check.py" \
  "$OWNER" "$REPO" "$PR" "$SHA" "$CLA_SHA" "$WORK/repo" "$ACTIONLINT"

@singlerider

Copy link
Copy Markdown
Contributor Author

Superseded by #110, moved from a fork to an upstream branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants