ci(deploy): capture a production admission denial receipt for paperclip-api (BLO-33027)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Its production release path (.github/workflows/docker.yml → deploy) is guarded at the apiserver by ValidatingAdmissionPolicy/paperclip-api-image-approval, which permits the paperclip-api Deployment to run only an explicitly approved immutable digest
• That policy is installed and undrifted in production — but as of 2026-09-09, and re-confirmed 2026-09-10, it had never been evaluated there: apiserver_validating_admission_policy_check_total{policy="paperclip-api-image-approval"} has zero samples, while five other policies report into the same metric family (so the family is scraped; this policy simply never fired)
• Enforcement therefore rested on two indirect links: a CEL deny-assertion that runs against a kind cluster in onprem-k8s, and paperclip_admission_policy_drift{object="paperclip-api-image-approval"} = 0, which proves the Binding exists with the expected validationActions. A sound inference chain — but no link is a denial on the production apiserver
• The obvious fixes were both rejected in BLO-20802: granting an agent deployment-write in paperclip breaches its AC-5, and amending the criterion abandons the check. The third route is that the deploy job already holds deployment write there, because that is exactly what helm upgrade needs
• This pull request adds a negative admission probe to the end of that job, executed by the pre-existing deploy identity, which requires the apiserver to deny an unapproved digest and records the denial as the receipt
• The benefit is that image-approval enforcement becomes directly observable in production on every release, instead of inferred — and it costs no new RBAC for anyone

Linked Issues or Issue Description

• Refs BLO-33027 — this PR
• Refs BLO-20802 — closed accepting a CI + drift-zero bridge for its AC-4; this closes the one gap that bridge left. Not a re-litigation: BLO-20802 stays closed either way
• Refs ci(deploy): authorize the release digest at admission time (BLO-19955) #907 — the policy this proves is effective
• Refs ci(deploy): make the abandon-in-flight escape hatch reachable from docker.yml #1636 — the review that established why these guard tests execute the step rather than grep it

What Changed

• .github/workflows/docker.yml — new final step in deploy, Negative admission probe — production denial receipt (BLO-33027). It reads the live paperclip-api Deployment, changes only spec.template.spec.containers[].image to a synthetic unapproved digest in the real repository, and submits it via kubectl replace --dry-run=server. A denial naming both paperclip-api-image-approval and approved immutable digest is the receipt; anything else is a failure or an explicit inconclusive.
• scripts/check-docker-admission-denial-receipt.test.js (new, 16 tests) — extracts the step's real shell between explicit markers and executes it against a stubbed apiserver with real jq.
• .github/workflows/pr.yml — wires that suite in as a step, matching the convention of its check-docker-* neighbours.

Three deliberate design decisions, all pinned by tests:

• It reports; it cannot gate. It runs after the rollout has landed and been verified, so no probe outcome can delay a legitimate release. continue-on-error: true is load-bearing rather than lazy: scheduled-production-deploy.yml decides whether production is behind by scanning for a deploy job with conclusion == "success", so failing the job would make a release that did land look undeployed and re-request human production approval on every schedule tick, forever. The step still genuinely exit 1s — continue-on-error preserves outcome: failure and overrides only conclusion — so a regression surfaces as a failed step plus an ::error:: annotation.
• UPDATE semantics against the live object, not a hand-rolled skeleton. Taking the live Deployment as the base preserves immutable fields such as spec.selector byte-for-byte, which is what keeps the denial attributable to image validation instead of to field is immutable from the wrong source.
• The digest is obviously synthetic (sha256:deadbeef…) yet well-formed against the policy's own ^sha256:[0-9a-f]{64}$ filter, so the denial stays attributable to non-approval rather than malformed input — and no build can ever emit it, so it cannot collide with a real artifact or be approved by accident.

Verification

No RBAC added, widened, or bound — the central claim. The probe runs as the existing KUBECONFIG_PAPERCLIP_CI_DEPLOY identity. This PR contains no Role, RoleBinding, ClusterRole, or ServiceAccount change:

$ git diff --stat origin/master...HEAD
 .github/workflows/docker.yml                       | 221 +++++++++++
 .github/workflows/pr.yml                           |  27 ++
 scripts/check-docker-admission-denial-receipt.test.js | 427 +++++++++++++++

Premise confirmed before writing the step — paperclip-ci-deploy exists in paperclip (age 133d), and the identity demonstrably writes Deployments there today:

$ kubectl -n paperclip get deploy paperclip-api \
    -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}{" gen="}{.metadata.generation}'
Helm gen=577

Policy and Binding both present and undrifted:

paperclip_admission_policy_drift{kind="[paperclip-egress-scrub redacted: high-entropy-assignment]",       object="paperclip-api-image-approval"} 0
paperclip_admission_policy_drift{kind="[paperclip-egress-scrub redacted: high-entropy-assignment]", object="paperclip-api-image-approval"} 0

The gap this closes, measured 2026-09-10 — five policies report, this one has no samples:

$ count by (policy) (apiserver_validating_admission_policy_check_total)
{policy="bc-protected-secret-write"}                     2
{policy="platform-sre-backup-backend-bearer-use"}        2
{policy="penstock-database-release-evidence-gc"}         1
{policy="penstock-database-release-secret-gc"}           1
{policy="penstock-runtime-database-secret-generation"}   1
# paperclip-api-image-approval: absent

Guard suite, 22s wall clock (step allows 120s):

$ node --test scripts/check-docker-admission-denial-receipt.test.js
ℹ tests 16
ℹ pass 16
ℹ fail 0

The tests have teeth — presence-only assertions are the failure mode #1636 found, so each mutation below was applied to the real step and the suite re-run:

Mutation to the probe	Result
Admitted path no longer exit 1s	✖ 1 failed — an ADMITTED unapproved digest fails the step
Drop the approved immutable digest half of attribution	✖ 1 failed — …WITHOUT the message fragment is not a receipt
Credit any non-zero kubectl exit as a denial	✖ 4 failed

docker.yml was confirmed byte-identical to its pre-mutation state afterwards (git diff --quiet clean), so no experiment residue is in this branch.

After merge, the first production release supplies the remaining evidence for the issue's acceptance criteria: the step log with the denial text verbatim, metadata.generation identical either side of the probe, and at least one sample in apiserver_validating_admission_policy_check_total{policy="paperclip-api-image-approval"}.

Risks

Low, and bounded by construction — the probe cannot break a release. It is the last step in deploy, runs only after the rollout has landed and been verified, and continue-on-error: true keeps it out of the job's conclusion, which scheduled-production-deploy.yml reads.

• Side effects: none, by three independent mechanisms. --dry-run=server never persists; a denied request persists nothing by definition; and the step compares metadata.generation either side and fails loudly if it ever moved — a generation change outranks even a good denial verdict (pinned by a test).
• Secret exposure: explicitly guarded. The probe manifest is the live API Deployment and carries the full container env, so it is never printed. A test asserts a representative env value reaches neither the step log nor the job summary.
• False alarm on a real regression is the intended behaviour, and it is a step-level failure with an ::error:: annotation, not a blocked release.
• The honest residual: if the policy is ever legitimately retired or renamed, this step starts failing and must be updated with it. That is the correct coupling — the step exists to notice exactly that class of change.
• Not verified here: whether the deploy identity holds create/update by direct RBAC introspection. My seat cannot read rolebindings or clusterroles (Forbidden) and cannot impersonate, so I confirmed the premise empirically (Helm-managed, generation 577) rather than by reading the grant. The step itself prints kubectl auth whoami and kubectl auth can-i as the real identity on every release, which is stronger evidence than an impersonated --as= check and lands with the first run.

Model Used

• Claude Opus 5 (claude-opus-5), 1M context, extended thinking, agentic tool use — running as the PlatformSREEngineer agent under Paperclip.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — n/a, CI/deploy only
• I have updated relevant documentation to reflect my changes — rationale is inline in both workflows, where the next reader will be
• I have considered and documented any risks above
• All Paperclip CI gates are green — pending first run on this branch
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
• I will address all Greptile and reviewer comments before requesting merge

---

Paperclip issue: https://paperclip.blockcast.net/BLO/issues/BLO-33027

[allyblockcast]

🔗 Paperclip issue: BLO-20802
🔗 Paperclip issue: BLO-33027

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 99b806a

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The production probe uses the live Deployment as its update base, changes only the image, uses --dry-run=server, and checks generation before and after to keep the denial receipt side-effect-free.
• The test suite executes the extracted workflow shell against a stubbed kubectl and covers attribution, admission, retry, inconclusive, secret-redaction, and workflow-ordering paths rather than relying on presence-only assertions.
• The probe is explicitly non-gating and runs after rollout verification, preserving the existing deploy-dispatch semantics while surfacing enforcement regressions through step failures and annotations.
• The latest documentation correction identifies the scoped ci-deploy Role as the authoritative grant instead of relying on the deprecated admin binding.

Recommended Action

1. No Critical or Important issues found; this review is ready for the normal merge process.