feat(ci): detect paperclip-production environment protection drift (BLO-22329)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Production releases of Paperclip itself ship through docker.yml, whose deploy job is bound to the paperclip-production GitHub environment — that environment's required-reviewers rule is the only thing standing between a merged commit and a live helm upgrade
• That protection has been recorded as correct by the board and been untrue three times: never applied (2026-07-31), lapsed (2026-08-04), and widened under an incident override that was never restored (2026-08-08, still live today)
• Each lapse went unnoticed for 2–6 days because a closed approval card was treated as evidence the control existed, and nothing ever re-read the artifact — the decisions were right, the verification was missing
• This pull request adds a scheduled guard that re-reads the live environment and asserts the ratified shape, alerting a human when it drifts
• The benefit is that the gap between "a control lapsed" and "somebody knows" drops from days to hours, without needing anyone to remember to re-probe

Linked Issues or Issue Description

Refs BLO-22329 (this PR is the detector it specifies)
Refs BLO-18577 (the environment being guarded)

Related PRs searched — no duplicates. The nearest prior art is #1340 (hourly master lockfile drift monitor) and #1273; both are dependency-drift, not environment-protection, and neither covers this artifact. See "Risks" for why this PR deliberately does not copy #1340's alerting approach.

What Changed

• .github/scripts/check-production-environment-protection.mjs — reads the live environment and asserts the shape ratified on approval b75f8156: a required_reviewers rule with prevent_self_review: true, can_admins_bypass: false, and deployment_branch_policy.protected_branches: true. Exit codes are load-bearing: 0 compliant, 1 drift, 2 unreadable.
• Reviewer set is compared by MEMBERSHIP, not for non-emptiness. This is the substantive design call. The live drift added kkroo (a repo admin) as a third reviewer alongside can_admins_bypass: true; together those route around prevent_self_review. A "does a required_reviewers rule exist?" check passes that shape — so the obvious implementation would miss the drift that is live right now. Changing the ratified set is deliberately a code change, so the PR is the audit trail the three silent edits lacked.
• .github/scripts/post-environment-protection-alert.mjs — pushes the drift to Alertmanager as severity: critical, which is what the live route tree matches to the slack-relay receiver. Alert delivery failure is fatal and loud.
• .github/workflows/production-environment-protection-guard.yml — twice-daily schedule plus workflow_dispatch; mints the App token via the existing get-bot-token.mjs idiom.
• .github/workflows/pr.yml — wires both test files into the policy job so they actually guard.
• Tests pin the 2026-08-04 lapse shape and the 2026-08-08 widening shape as regression fixtures.

Verification

All run locally 2026-08-14T08:46–08:55Z.

Unit tests — 19/19 pass:

node --test ./.github/scripts/tests/check-production-environment-protection.test.mjs \
            ./.github/scripts/tests/post-environment-protection-alert.test.mjs
# tests 19 | pass 19 | fail 0

Against the LIVE environment — exit 1, both violations named:

DRIFT: Blockcast/paperclip environment 'paperclip-production' no longer matches the board-ratified shape:
  - required_reviewers membership: unexpected ["kkroo"] (ratified set is ["eyad-hussein","MohamedElmdary"])
  - can_admins_bypass: expected false, got true
EXIT=1

Against a compliant fixture — exit 0, violations: [].

Unreadable paths — never reported as compliant:

• invalid token → EXIT=2, summary status: unreadable, 401 surfaced verbatim
• no token at all → EXIT=2

Token choice: the default Actions GITHUB_TOKEN does not carry environments read; the allyblockcast App installation token does — verified live this run against GET /repos/Blockcast/paperclip/environments/paperclip-production.

Risks

Low risk to anything existing — this adds a read-only scheduled job and touches no deploy path. It cannot block a release.

Two things a reviewer should push back on if they disagree:

1. Runner → Alertmanager reachability is unproven. I could not verify from my environment that an ARC runner can reach alertmanager.monitoring.svc.cluster.local:9093. The monitoring namespace has a default-deny-all NetworkPolicy whose only ingress allows are intra-namespace and gateway-system, yet a probe from the paperclip namespace returned 200 — so enforcement does not match the policy text, and I did not want to build on a guess. Mitigated by making delivery failure fatal and loud rather than silent, so the first run reports it honestly instead of pretending to alert. ALERTMANAGER_URL is overridable via repo vars.

2. This deliberately diverges from ci(lockfile): add hourly master lockfile-overrides drift monitor (BLO-24150) #1340's precedent. That monitor alerts "via Actions notifications". BLO-22329 explicitly rules that out — a red scheduled run emails the last committer and is easy to miss for days, which is the exact failure mode being fixed. Hence Alertmanager. If reviewers prefer consistency with ci(lockfile): add hourly master lockfile-overrides drift monitor (BLO-24150) #1340 over reaching a human, that is the trade-off to argue.

Note Alertmanager's repeat_interval is 1h, so a persistent drift re-notifies hourly. Deliberate for a lapsed production gate, but it is a monitoring-side tuning knob, not something this PR sets.

Not in scope: restoring can_admins_bypass: false and removing kkroo needs the repo-admin credential (the App gets 403 on environment writes) and is an owner call while a deploy sits parked in waiting. Tracked on the issue.

On docker.yml also hard-asserting the shape before helm upgrade (the open question BLO-22329 asked me to decide): recommendation is assert-and-report, never assert-and-fail, and in a separate PR. The 2026-08-08 widening was a legitimate, disclosed incident response to a company-wide dispatch deadlock; a hard-fail assert would have blocked it, and a control that blocks incident response gets disabled under pressure — strictly worse than not having it. It is not zero-value though: with can_admins_bypass: true an admin can self-approve, and only a deploy-time check catches that at the moment it matters.

Model Used

Claude Opus 5 (claude-opus-5), 1M context window, extended thinking, with tool use and code execution.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — N/A, no UI surface
• I have updated relevant documentation to reflect my changes — the scripts carry the rationale inline; no separate doc covers this control
• I have considered and documented any risks above
• All Paperclip CI gates are green — in progress at time of writing
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-22329

[allyblockcast]

🔗 Paperclip issue: BLO-22329

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

@ally please review at head 7b84cd3753fc1303b78b717f19a770bd3daa30f1.

This adds a scheduled detector for drift in the paperclip-production GitHub environment protection (BLO-22329). The control has silently lapsed three times; the detector is the thing that makes a lapse loud.

Highest-value review focus, in order:

1. Fail-closed correctness in check-production-environment-protection.mjs. An unreadable environment (403/404/network/invalid token) must exit 2 as unreadable and must never be reported as compliant. This is the property that matters most — a detector that reads "no drift" when it actually got a 403 is worse than no detector at all. Please attack that path specifically.
2. Reviewer-set comparison by membership, not by non-emptiness. The earlier draft only asserted required_reviewers was non-empty, which would have PASSED against the live drift (an extra admin reviewer added). Check the set-difference logic reports both additions and removals correctly.
3. post-environment-protection-alert.mjs delivery failure is deliberately fatal. I could not prove an ARC runner can reach Alertmanager, so a failed POST must fail the run loudly rather than silently "alerting". Please confirm there is no path where a delivery error is swallowed.
4. Workflow wiring in production-environment-protection-guard.yml and the pr.yml test hookup — particularly that the scheduled job cannot pass vacuously (e.g. missing token treated as success).

Note: the two red checks on this head were a single ARC runner eviction in general_tests (The runner has received a shutdown signal), with verify failing only derivatively. Re-running; not a defect in the diff.

[allyblockcast]

@ally please review at head 7b84cd3 — ci: detect paperclip-production environment protection drift. Focus on false-positive drift alerts from benign API shape changes.

Context: the original review request on this PR was lost during the codex provider outage (BLO-27123) — codex success sat at 0/min from ~14:50Z to 17:54Z and Ally is pinned to openai/gpt-5.6-terra on that pool. Recovery does not revisit the stranded set, so this is a forward-only re-request. Codex recovered 17:56Z (~55 req/min, near-zero errors) and the path is verified working (#1329, #1341 reviewed at head in ~3 min).

[allyblockcast]

@ally please review at head 8d06e921ecc87e518988ee7ebda5c041ca02aaa5.

This is the third delivery attempt on this PR (markers at 7b84cd37 on 2026-08-14T17:47Z and 18:06Z, plus a synchronize from the 08-15T07:55Z rebase). None produced a review, while 8 reviews landed on 6 sibling paperclip PRs in the same window — so the earlier markers were live and correctly targeted, and this head is now the only valid target.

Review focus:

1. check-production-environment-protection.mjs — does it fail closed? An unreadable environment (403/404/network) must report a distinguishable error, never a pass. That is the core acceptance criterion.
2. The three drift conditions: required_reviewers non-empty with prevent_self_review: true; can_admins_bypass == false; deployment_branch_policy.protected_branches == true.
3. post-environment-protection-alert.mjs — the alert must reach a human on an owned alerting path, not rely solely on a red workflow run (a red scheduled run emails the last committer and is exactly the miss being fixed).
4. production-environment-protection-guard.yml — schedule, token selection (App token carries environments read; default GITHUB_TOKEN may not), and silence on the compliant path.