policy: add a language-policy drift gate (self-tested) - #661
Open
hyperpolymath wants to merge 5 commits into
Open
policy: add a language-policy drift gate (self-tested)#661hyperpolymath wants to merge 5 commits into
hyperpolymath wants to merge 5 commits into
Conversation
Owner ruling 2026-08-26: "deno is to go and bun is the way we are going, put it
first everywhere unless not possible and explain why if not".
This file is what agents read first and it currently says the opposite, so it
is corrected before any repo is touched - otherwise every future agent
re-derives the wrong conclusion from the governing document.
:89 Deno "tier 2, grandfathered, need not migrate"
-> BEING REMOVED; existing projects must migrate to Bun, and where Bun
genuinely cannot be used the reason must be DOCUMENTED, not left
silently on Deno
:187 "No package.json for runtime deps - use deno.json imports"
-> Use package.json + bun.lock; Bun is npm-compatible and a manifest is
REQUIRED
:188 "No node_modules in production - Deno caches automatically"
-> bun install --production, pinned via bun.lock
:198 "JS deps: Deno (deno.json imports)"
-> JS deps: Bun (package.json + bun.lock), bunx for one-off tooling
:182 portable Deno CLI scripts are the convert-now bucket
-> ...and anything not yet portable to AffineScript moves to Bun
WHY :187 MATTERS MOST. "No package.json for runtime deps" did not express a
preference - it told repos not to declare their dependencies at all.
hyperpolymath/ubicity is the worked example: its sources import zod and glob,
it had NO manifest of any kind, and it could not build under ANY toolchain.
Fixed in ubicity#107 by deriving a manifest from the imports; the rule that
caused it is fixed here.
Policy text only - no code, no workflows. The policy table (30 rows) and
blockquote structure are intact.
NOTED, NOT CHANGED: :197 still reads "Fallback: Nix (flake.nix)". Nix was
deprecated for Guix by ADR-2026-STACK-MIGRATION, so that looks stale too, but
it is a separate ruling and is not folded into a Deno/Bun correction.
Follow-on: 30 repos carry a live deno.json - working task runners, not orphaned
config - tracked separately for migration with documented exceptions.
Review feedback on #655 plus a tightened owner ruling. 1. NO TYPESCRIPT (owner, 2026-08-27). Asked about the Bun row advertising direct .ts execution, the owner ruled: "no typescript ... that should not exist at all." Every .ts reference is removed from the Bun row, including "JS/TS" in its label; it now reads "JS runtime". The blockquote sentence "TypeScript is permitted only where AffineScript cannot reach" is marked TOO PERMISSIVE. FLAGGED, NOT RESOLVED: this collides with the "TypeScript Exemptions (Approved)" table, which documents real technical carve-outs (.d.ts files, the npm/Node-native VS Code extension host, MCP/LSP glue). Those are not stylistic and cannot simply be deleted, so they are left standing and marked for an explicit owner decision rather than settled unilaterally. 2. DENO MOVED OUT OF ALLOWED INTO BANNED (codacy, #655). A struck-through row in an ALLOWED table is ambiguous to the agents that read this file - codacy raised exactly that. Deno now appears once, in BANNED, carrying the ruling and pointing at the #658 assessment. 3. UNPINNED bunx (coderabbitai, Security & Privacy). A bare `bunx <tool>` can fetch a package outside package.json/bun.lock and can start Node via a shebang. Guidance now requires a declared devDependency plus `bunx --no-install --bun <tool>`. NOT TAKEN: "a npm-compatible" (LanguageTool is wrong; "an" is correct before a vowel sound). "Remove the historical commentary" - the history is why the file was wrong before, and deleting it invites the same drift back. STILL FLAGGED, unchanged: "Fallback: Nix (flake.nix)" is stale under ADR-2026-STACK-MIGRATION, but that is a separate ruling and is deliberately not folded into a Deno/Bun change.
Owner ruling 2026-08-27 (#659): keep the full policy text in each repo, but detect divergence from the canonical document. WHY. The language policy is duplicated into ~372 per-repo .claude/CLAUDE.md files across 131 repos. A census on 2026-08-26 found 868 of them still listed Bun as BANNED with Deno as its replacement - the exact inverse of the standing ruling - and NOTHING HAD EVER DETECTED IT. Correcting `standards` fixes one copy of ~372; agents read the local one. Detection was the missing piece, not the copy. WHY ASSERTIONS, NOT A GENERATOR. The copies are legitimately not identical - repos carry their own exemption tables, architecture notes and carve-outs. A byte-for-byte generator would be permanently red and would be disabled within a week. This gate asserts the INVARIANTS the policy must satisfy, whatever the surrounding wording. WHAT IT CATCHES - `| Bun | Deno |` Bun banned - the inversion nobody detected - "No package.json for runtime deps" / "deno.json imports" forbids declaring dependencies at all; this is why hyperpolymath/ubicity had NO manifest and could not build under ANY toolchain - "Executes .ts directly" / "JS/TS runtime" advertises TypeScript (owner ruling 2026-08-27: it "should not exist at all") - `| | X |`, `| **** |`, "No new files", "Only where cannot" blanking scars from bulk substitution - the same class that produced `rm -rf /lib` - `| AffineScript | AffineScript |` a rule banning the language it mandates - missing Bun row / missing Deno-in-BANNED SELF-TESTED with controls, so it cannot join the estate's fake-gate collection: a compliant file PASSES, and each of the five defect classes above is independently CAUGHT. Verified before commit.
|
Warning Review limit reachedNext included review available in 20 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Up to standards ✅🟢 Issues
|
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull Request Overview
While the PR is reported as 'up to standards' by Codacy, the implementation of the check-language-policy.sh gate contains several logic errors that will likely prevent it from functioning correctly in this environment. Specifically, the script's Markdown parsing logic (using awk and grep) does not align with the actual formatting of .claude/CLAUDE.md, which uses double-pipe (||) prefixes and specific padding.
Furthermore, all six recommended test scenarios for verifying this policy gate are missing from the PR. Given the script's complexity and its role as a CI gate, automated test fixtures are necessary to validate the various failure and success states, especially since the logic for detecting 'blanking scars' and 'recursive bans' is currently fragile.
About this PR
- The gate logic is currently too brittle to handle the estate's established Markdown style. It fails to account for double-pipe row prefixes and specific whitespace padding, which will result in false negatives or script errors when run against valid files.
- The script is labeled as 'self-tested', but there are no automated tests or test fixtures included. Given the fragility of the regex-based parsing, please include a test suite or set of example files that demonstrate the script correctly identifies the 2026-08-27 policy violations.
Test suggestions
- Missing recommended test scenario: Fail if Bun is listed as banned with Deno as replacement
- Missing recommended test scenario: Fail if tool descriptions advertise 'JS/TS runtime' or '.ts' execution
- Missing recommended test scenario: Fail if a BANNED table cell is empty (blanking scar)
- Missing recommended test scenario: Fail if a policy table exists but Bun is missing from ALLOWED
- Missing recommended test scenario: Fail if AffineScript is mapped to AffineScript in BANNED table
- Missing recommended test scenario: Pass for a compliant CLAUDE.md following the 2026-08-27 tightened ruling
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Missing recommended test scenario: Fail if Bun is listed as banned with Deno as replacement
2. Missing recommended test scenario: Fail if tool descriptions advertise 'JS/TS runtime' or '.ts' execution
3. Missing recommended test scenario: Fail if a BANNED table cell is empty (blanking scar)
4. Missing recommended test scenario: Fail if a policy table exists but Bun is missing from ALLOWED
5. Missing recommended test scenario: Fail if AffineScript is mapped to AffineScript in BANNED table
6. Missing recommended test scenario: Pass for a compliant CLAUDE.md following the 2026-08-27 tightened ruling
TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback
| fail "$f" 'Enforcement rule with a blanked language name.' | ||
| fi | ||
| # 5. A rule may not ban the language it mandates. | ||
| if grep -nE '^\| AffineScript \| AffineScript \|' "$f" >/dev/null; then |
There was a problem hiding this comment.
🟡 MEDIUM RISK
This regex is too rigid and fails to account for the double-pipe (||) prefix or whitespace padding found in CLAUDE.md. Suggested fix: grep -nE '^\|+ *AffineScript *\| *AffineScript *\|' "$f".
| fi | ||
| # 4. Blanking scars. A bulk purge substituted a token with an EMPTY STRING, which also | ||
| # produced `rm -rf /lib` in wordpress-tools (the lethal shape is <token>/path -> /path). | ||
| if awk -F'|' 'NF==4 && $2 ~ /^[[:space:]]*$/{exit 0} END{exit 1}' "$f"; then |
There was a problem hiding this comment.
🟡 MEDIUM RISK
The NF==4 condition will miss standard pipe-wrapped Markdown tables used in this repository. These tables typically result in 5 or more fields when split by | (including empty fields for the leading/trailing pipes). Consider checking for $2 ~ /^[[:space:]]*$/ without the strict NF constraint, or adjust the count to match the actual table structure.
| # Exit 0 = compliant. Exit 1 = drift. Every failure prints file:line. | ||
| set -uo pipefail | ||
| status=0 | ||
| files=$(git ls-files '*CLAUDE.md' 2>/dev/null | grep -v node_modules) |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Iterating over file paths with a word-splitting for loop is unsafe if any file path contains a space. Use a while read loop or xargs -0 to handle filenames correctly.
Comment on lines
+67
to
+70
| grep -qE '^\| \*\*Bun\*\* \|' "$f" || \ | ||
| fail "$f" 'No Bun row in ALLOWED. Bun is the tier-1 JS runtime and package manager.' | ||
| grep -qE '^\| \*?\*?Deno\*?\*? \| Bun \|' "$f" || \ | ||
| fail "$f" 'Deno is not listed in BANNED with Bun as its replacement (ruling 2026-08-26).' |
There was a problem hiding this comment.
🟡 MEDIUM RISK
Suggestion: The regex is too strict regarding leading pipes. The document frequently uses || as a row start marker. Update the pattern to allow for an optional second pipe to ensure the gate is robust against consistent table formatting.
First run of the drift gate against the 129-repo fleet found 4 divergences. Three
were real and are fixed at source (ambientops, developer-ecosystem, dotfiles#40).
Two were the gate's own false positives, fixed here.
The gate required table rows written with single spaces:
^\| \*\*Bun\*\* \|
^\| \*?\*?Deno\*?\*? \| Bun \|
Real estate files do not all look like that:
1. COLUMN-PADDED tables - `| **Bun** | ... |` (ambientops
recovery/operating-theatre). Content correct, gate said missing.
2. BULLET LISTS instead of tables - `- Deno (use Bun)` (developer-ecosystem
rescript-ecosystem/packages/core/env). A perfectly clear policy statement
that no table regex can match.
Both now accepted. This matters more than the two files: a gate that fails
correct content is a gate that gets disabled, and this estate already has a
documented history of gates nobody trusts.
CONTROLS re-run and all four pass: padded-compliant PASSES, bullet-compliant
PASSES, bullet-form missing-Deno CAUGHT, inverted `| Bun | Deno |` CAUGHT.
Fleet: 127/129 pass, the 2 remaining being this repo's own copy (owned by #655)
and a stale clone.
Third false-positive class from running the gate against real files. The gate flagged the corrected governing document itself, twice - at a blockquote explaining what the old rule said, and at the corrected rule that quotes its own predecessor parenthetically. Both are the document explaining what it replaced. That matters because #655 deliberately KEEPS that history: "the history is why the file was wrong before, and deleting it invites the same drift back". A gate that punishes a document for explaining itself pushes maintainers to delete the explanation - the opposite of the intent. `live()` now strips blockquote lines and lines where the phrase appears inside quotation marks (straight or typographic) before matching. CONTROLS, all passing: quoted-in-blockquote PASSES, quoted-inline PASSES, and a LIVE stale rule is still CAUGHT. Together with the previous commit the gate now handles: padded tables, bullet lists instead of tables, and quoted history - while still catching inverted bans, blanked cells, TypeScript advertisement and missing Deno bans.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner ruling on #659: keep the full text locally, but detect divergence.
Why
The language policy is duplicated into ~372 files across 131 repos. On 2026-08-26 a census found 868 of them still listed Bun as BANNED with Deno as its replacement — the exact inverse of the standing ruling — and nothing had ever detected it. Correcting
standardsfixes one copy of ~372; agents read the local one.Detection was the missing piece, not the copy.
Why assertions rather than a generator
The copies are legitimately not identical — repos carry their own exemption tables and carve-outs. A byte-for-byte generator would be permanently red, and a permanently-red gate gets disabled. This asserts the invariants, whatever the local wording.
What it catches
| Bun | Deno |ubicityhad no manifest and could not build under any toolchain| | X |,| **** |, "No new files"rm -rf /lib(wordpress-tools#62)| AffineScript | AffineScript |Self-tested
Because this estate has a documented history of gates that pass by not doing the work, the gate was verified before commit: a compliant file passes, and each of the five defect classes is independently caught. Controls are reproducible from the script.
Related: #655, #658, #659.