feat(cicd): estate census over GraphQL — 4% of a quota nobody was using - #583
Conversation
CICD-SIGNAL-DISCIPLINE says to run the census on a schedule rather than once,
because a one-off audit is archaeology and stale within a week. The REST
implementation made that impractical: ~4 calls per repo for requirements and
~20 more for emissions is roughly 4,000 calls for a 424-repo estate — 80% of
the 5,000/hour CORE budget in a single pass, leaving nothing for the
remediation the census exists to drive. It also died halfway more than once
and silently under-reported.
MEASURED, same estate, same data:
REST ~4,000 calls exhausted core died halfway twice
GraphQL 214 points 4% of graphql 3m50s, complete
GraphQL has a SEPARATE 5,000/hour budget which was sitting entirely unused —
5000/5000 at the moment core hit zero. Requirements cost ONE POINT PER 25
REPOSITORIES; emissions one per repo, and only for repos that require
something, since a repo requiring nothing cannot have a phantom.
IT IS ALSO MORE CORRECT, not merely cheaper:
* Requirements live in EITHER branch protection OR a ruleset. The REST scan
read those through different endpoints, so a repo using branch protection
was reported as "no ruleset required X" and its fix silently skipped —
which is exactly what happened to knot-rider, neurophone and
recon-silly-ation. One query returns both, so they cannot drift apart.
* statusCheckRollup returns CheckRun names AND StatusContext contexts
together. Reading only the Actions API brands every external check
(SonarCloud and friends) a phantom.
* Ruleset TARGET is recorded. Required status checks on a ruleset targeting
TAGS are INERT — tags have no pull request to gate. 88 such requirements
exist across 16 repos, and one repo carries 30+. They neither gate nor
block, which makes them the hardest of the three species to notice.
* Rulesets with enforcement != ACTIVE are skipped: they gate nothing.
Cross-validated against the REST census: both find 131 repositories requiring
something, differing by one repo that is newer than the REST run's repo list.
The GraphQL version reports FEWER phantoms — 56 across 14 repos versus 59
across 21 — because its wider sampling produces fewer false positives.
The known limit is documented in the script header rather than left to be
discovered: a context appearing only on a RARE trigger can be missed by any
sample. Measured case — `Dependabot` appears only on Dependabot pull requests,
so it is absent unless one falls in the window, and would be wrongly reported
as a phantom. "Phantom" is therefore a CANDIDATE requiring confirmation, never
a verdict — the same discipline this document already sets for fake gates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
|
| branchProtectionRules(first: 3) { nodes { requiredStatusCheckContexts } } | ||
| rulesets(first: 5) { | ||
| nodes { | ||
| target enforcement | ||
| rules(first: 20) { | ||
| nodes { type parameters { ... on RequiredStatusChecksParameters { requiredStatusChecks { context } } } } |
There was a problem hiding this comment.
⚠️ Edge Case: Fixed connection limits silently truncate requirements/emissions
Every nested GraphQL connection uses a hardcoded page size with no pagination: branchProtectionRules(first: 3), rulesets(first: 5), rules(first: 20) in REQ_QUERY and contexts(first: 100) in EMIT_QUERY. A repo with more than 5 rulesets, more than 20 rules in a ruleset, or more than 3 branch-protection rules will have its extra requirements dropped, and a busy commit with >100 checks will drop emitted contexts — exactly the kind of silent under-reporting this script was written to eliminate (a dropped requirement looks like it doesn't exist; a dropped emission fabricates a phantom). Either raise the caps well above the estate maximum and assert hasNextPage is false, or paginate these connections.
Was this helpful? React with 👍 / 👎
| RESP=$(gh api graphql -F login="$LOGIN" -F cursor=null -f query="$REQ_QUERY" 2>/dev/null) | ||
| else | ||
| RESP=$(gh api graphql -F login="$LOGIN" -F cursor="$CURSOR" -f query="$REQ_QUERY" 2>/dev/null) |
There was a problem hiding this comment.
💡 Bug: Cursor passed via -F may be coerced from string to non-string
The pagination cursor is passed with -F cursor="$CURSOR", and gh api's -F/--field performs automatic type coercion (numbers, booleans, null). GitHub cursors are base64 strings that can resemble those types, in which case the wrong type is sent for the $cursor: String variable, breaking pagination for that login and silently ending the scan early. Use the raw-field flag for the non-null branch: -f cursor="$CURSOR" (the null branch must stay -F cursor=null).
Send the cursor as a raw string to avoid type coercion.:
if [ -z "$CURSOR" ]; then
RESP=$(gh api graphql -F login="$LOGIN" -F cursor=null -f query="$REQ_QUERY" 2>/dev/null)
else
RESP=$(gh api graphql -F login="$LOGIN" -f cursor="$CURSOR" -f query="$REQ_QUERY" 2>/dev/null)
fi
Was this helpful? React with 👍 / 👎
| while [ $# -gt 0 ]; do | ||
| case "$1" in | ||
| --commits) COMMITS="$2"; shift 2 ;; | ||
| --prs) PRS="$2"; shift 2 ;; | ||
| *) echo "unknown option: $1" >&2; exit 2 ;; | ||
| esac | ||
| done |
There was a problem hiding this comment.
💡 Edge Case: --commits/--prs crash on missing or non-numeric value
Under set -u, invoking --commits or --prs as the final argument makes $2 an unbound variable, aborting the script with a cryptic error rather than a usage message. There is also no validation that the value is numeric, so --commits foo is passed to the Int! GraphQL variable, causing every emission query to error out and return no data. Validate that the option has a numeric argument before assigning.
Reject missing or non-numeric option values with a clear message.:
--commits) [ "${2:-}" -ge 0 ] 2>/dev/null || { echo "--commits needs a number" >&2; exit 2; }; COMMITS="$2"; shift 2 ;;
--prs) [ "${2:-}" -ge 0 ] 2>/dev/null || { echo "--prs needs a number" >&2; exit 2; }; PRS="$2"; shift 2 ;;
Was this helpful? React with 👍 / 👎
|
|
||
| # Emissions only for repos that require something: a repo requiring nothing | ||
| # cannot have a phantom, so scanning it is wasted budget. | ||
| cut -f1 "$OUT" | sort -u > "$OUT.repos" |
There was a problem hiding this comment.
💡 Quality: Temporary $OUT.repos file is never cleaned up
The intermediate repo list written to "$OUT.repos" is left on disk after the run completes. It is harmless but clutters the output directory and can go stale across runs. Remove it at the end (e.g. rm -f "$OUT.repos") or register a trap.
Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Configure merge blocking · Maintainers can dismiss this review.
|
Note Automatic reviews are paused because your trial's included automatic processing has been used for this period. Upgrade now, or comment "Gitar review" to run a review anytime. Code Review
|
| Compact |
|
Important
Your trial ends in 4 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.
Was this helpful? React with 👍 / 👎 | Gitar
…es (#39) The governance and Hypatia workflows here are **`startup_failure` on every run** — not failing, **never starting**. GitHub refuses to begin a reusable-workflow call whose caller grants **less** than the reusable itself declares. The refusal happens before any job exists: **no log, no check run, no row in `gh pr checks`**. ## Why this was misdiagnosed as something else Because the workflow never emits its status context, a perfectly correct branch rule requiring `governance / Validate Hypatia Baseline` **looks like a phantom** — a rule naming something that doesn't exist. It isn't. **The rule is right and the workflow is broken.** Rewriting the rule would have converted *"this check is broken"* into *"this check is not required"* — which is exactly how enforcement quietly evaporates, and it was the outcome I nearly produced before checking workflow health. ## The fix `haec` runs the identical reusables at the identical pin, successfully. It grants: ```yaml permissions: actions: read contents: read ``` The repos stuck at `startup_failure` grant only `contents: read`. This adds the missing `actions: read` **and nothing else**. It deliberately does not widen permissions further — over-granting is how a scanner quietly acquires write access it was never meant to have. Found by the estate CI/CD census (hyperpolymath/standards#583), which pairs what each repository *requires* against what it actually *emits*. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…es (#70) The governance and Hypatia workflows here are **`startup_failure` on every run** — not failing, **never starting**. GitHub refuses to begin a reusable-workflow call whose caller grants **less** than the reusable itself declares. The refusal happens before any job exists: **no log, no check run, no row in `gh pr checks`**. ## Why this was misdiagnosed as something else Because the workflow never emits its status context, a perfectly correct branch rule requiring `governance / Validate Hypatia Baseline` **looks like a phantom** — a rule naming something that doesn't exist. It isn't. **The rule is right and the workflow is broken.** Rewriting the rule would have converted *"this check is broken"* into *"this check is not required"* — which is exactly how enforcement quietly evaporates, and it was the outcome I nearly produced before checking workflow health. ## The fix `haec` runs the identical reusables at the identical pin, successfully. It grants: ```yaml permissions: actions: read contents: read ``` The repos stuck at `startup_failure` grant only `contents: read`. This adds the missing `actions: read` **and nothing else**. It deliberately does not widen permissions further — over-granting is how a scanner quietly acquires write access it was never meant to have. Found by the estate CI/CD census (hyperpolymath/standards#583), which pairs what each repository *requires* against what it actually *emits*. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…es (#63) The governance and Hypatia workflows here are **`startup_failure` on every run** — not failing, **never starting**. GitHub refuses to begin a reusable-workflow call whose caller grants **less** than the reusable itself declares. The refusal happens before any job exists: **no log, no check run, no row in `gh pr checks`**. ## Why this was misdiagnosed as something else Because the workflow never emits its status context, a perfectly correct branch rule requiring `governance / Validate Hypatia Baseline` **looks like a phantom** — a rule naming something that doesn't exist. It isn't. **The rule is right and the workflow is broken.** Rewriting the rule would have converted *"this check is broken"* into *"this check is not required"* — which is exactly how enforcement quietly evaporates, and it was the outcome I nearly produced before checking workflow health. ## The fix `haec` runs the identical reusables at the identical pin, successfully. It grants: ```yaml permissions: actions: read contents: read ``` The repos stuck at `startup_failure` grant only `contents: read`. This adds the missing `actions: read` **and nothing else**. It deliberately does not widen permissions further — over-granting is how a scanner quietly acquires write access it was never meant to have. Found by the estate CI/CD census (hyperpolymath/standards#583), which pairs what each repository *requires* against what it actually *emits*. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every governance and Hypatia run here is **`startup_failure`** — not failing, **never starting**. Two causes, and the second is invisible until the first is fixed. ## 1. No lockfile Workflow-lockfile enforcement is active on this account and this repo had **no `.github/workflows/actions.lock`**, so every workflow was rejected before any job was created — no log, no check run, no row in `gh pr checks`. ## 2. Two stale reusable pins `81dbf2dd` (2026-06-27) across 3 files, and `e9c88887` across 2. Re-pinned to standards `bd0df9ead7fa` — the commit that made the governance check lockfile-aware. ## Why this was misdiagnosed Because the workflow never emitted its status context, the branch rule requiring `governance / Validate Hypatia Baseline` **looked like a phantom**. It isn't — **the rule is correct and the workflow is broken.** Rewriting the rule would have turned *"this check is broken"* into *"this check is not required"*. ## The cure, in order Each step's failure is invisible until the previous one is fixed: 1. **`gh actions-lock`** — generates the lockfile, normalises pins to tags 2. **Hoist SPDX back to line 1** — `actions-lock` puts its banner there and the linter requires SPDX first, so the tool that cures the startup failures would otherwise redden all 20 workflow files 3. **Hand-author `[]` lockfile entries** for the 5 reusable callers — `actions-lock` **skips callers**, so without this they stay `startup_failure` while everything else goes green 4. **Re-pin** the callers **Verified:** all 20 workflows parse, SPDX on line 1, 5 caller entries in the lockfile. **Expect previously-unseen failures.** Nothing here has been checked in a long time; the first green run is a starting point, not a result. Found by the estate CI/CD census (hyperpolymath/standards#583). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ure (#40) All **27 workflows** here return `startup_failure`: not failing, **never starting**. Workflow-lockfile enforcement is active on this account and this repo had **no `.github/workflows/actions.lock`**, so every workflow was rejected before any job was created — no log, no check run, no row in `gh pr checks`. ## It also explains what looked like a separate problem The branch rules here require `CodeQL` and `openssf-compliance`, and neither ever reported — so both appeared to be **phantom** requirements naming checks that do not exist. They are not. **The rules are correct and every workflow was dead.** Rewriting them would have converted *"these checks are broken"* into *"these checks are not required"*. ## The cure, in order Each step's failure is invisible until the previous one is fixed: 1. **`gh actions-lock`** — generates the lockfile, normalises pins to tags (it also flagged **12 actions pinned to a bare SHA with no tag ref**) 2. **Hoist SPDX back to line 1** — `actions-lock` puts its banner there and the linter requires SPDX first, so the cure would otherwise redden all 27 files 3. **Hand-author `[]` lockfile entries** for the 6 reusable callers — `actions-lock` **skips callers**, so without this they stay `startup_failure` while everything else goes green 4. **Re-pin** those callers to standards `bd0df9ead7fa` **Verified:** all 27 workflows parse, SPDX on line 1 of each, 6 caller entries in the lockfile. **Expect previously-unseen failures.** Nothing here has been checked in a long time — the first green run is a starting point, not a result. Found by the estate CI/CD census (hyperpolymath/standards#583). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>



CICD-SIGNAL-DISCIPLINE says to run the census on a schedule, not once — a one-off audit is archaeology and stale within a week. The REST implementation made that impractical.
Measured, same estate, same data
GraphQL has a separate 5,000/hour budget that was sitting entirely unused — it read
5000/5000at the exact moment core hit zero. Requirements cost one point per 25 repositories; emissions one per repo, and only for repos that require something, since a repo requiring nothing cannot have a phantom.It is also more correct, not merely cheaper
Requirements live in either branch protection or a ruleset. The REST scan read those through different endpoints, so a repo using branch protection was reported as "no ruleset required CodeQL" and its fix silently skipped — which is precisely what happened to
knot-rider,neurophoneandrecon-silly-ationduring this campaign. One query returns both, so they cannot drift apart.statusCheckRollupreturns CheckRun names and StatusContext contexts together. Reading only the Actions API brands every external check — SonarCloud and friends — a phantom.Ruleset
targetis recorded. Required status checks on a ruleset targeting tags are inert: tags have no pull request to gate. There are 88 such requirements across 16 repos, one carrying 30+. They neither gate nor block, which makes them the hardest of the three species to notice — a fake gate passes without checking, a phantom blocks without reporting, an inert requirement does nothing in either direction.Disabled rulesets are skipped —
enforcement != ACTIVEgates nothing.Cross-validated, not asserted
Both methods independently find 131 repositories requiring something, differing by one repo that postdates the REST run's repo list. The GraphQL version reports fewer phantoms because its wider sampling (8 commits + 5 PRs, vs 12 runs + 3 commits) produces fewer false positives.
The known limit is in the header, not left to be discovered
A context appearing only on a rare trigger can be missed by any sample. Measured case:
Dependabotappears only on Dependabot pull requests, so it is absent unless one falls in the sampled window — and would be wrongly reported as a phantom.So "phantom" is a candidate requiring confirmation, never a verdict — the same discipline this document already sets for fake gates.
--commitsand--prswiden the window.🤖 Generated with Claude Code