ci: rosetta-audit without pins -- baseline-classified, refuses to no-op (#2682) - #2699
Merged
Merged
Conversation
…op (#2682) The keyword-rosetta gate was modelled on tri-comparison and then grew two things tri-comparison never had: a second pin (KEYWORD_ROSETTA_REF here, ENGINE_REF there) and a hand-run regeneration inside every corpus PR. The pins pointed at each other, so no merge order satisfied both, and every way the setup broke on 2026-09-02 failed green. Neither repo requires a status check to merge, so both were only ever making an advisory light green before a merge that happened anyway. rosetta-audit.yml now checks out keyword-rosetta at main (no variable), and runs the new tests/tools/rosetta_audit.py: every language against the PR build, then every failing language against a second build of the PR's base branch, classified as a regression this PR introduces (fail) or drift the corpus already carries against main (notice). The rosetta:rebless-owed label turns regressions into warnings for an intentional, corpus-visible change, which then simply merges first; the corpus re-blesses against engine main afterwards. Exit 2 if the audit did not actually run (no galaxyscope, zero languages, a verifier that crashed instead of reporting). Same refuse-to-no-op guard on the other two corpus-backed runners: tri_comparison_chart --ci used to skip every language and print "all OK" with exit 0 when galaxyscope was off PATH (reproduced); a baselined language with no data now fails closed, and zero languages checked is a failure in both it and tree_sitter_accuracy_audit --all. Docs: BUMPING_THE_ROSETTA_PIN.md becomes ROSETTA_AUDIT.md; CONTRIBUTING, ecosystem and CLAUDE.md describe the three-step flow. Companion: keyword-rosetta PR retiring ENGINE_REF and adding bias-history.yml. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
|
Cross-repo companion: squid-protocol/keyword-rosetta#46 (retires |
…ession verdict -- REVERTED next commit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Owner
Author
|
Regression-verdict demo, as promised. Commit 30e989a dropped Run 33811217240 on the throwaway commit: Job failed with an annotation naming go; the 45 other languages stayed clean, so the verdict is per-language, not a whole-corpus red. Side effect worth noting: the 46-gate sweep took 85 s (4 workers) against ~5 min for the old serial loop. |
This reverts the throwaway commit 30e989a. The point was run 33811217240: go classified REGRESSION (fails on this build, passes on the base branch), 46 checked, exit 1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
squid-protocol
added a commit
that referenced
this pull request
Sep 4, 2026
`rosetta:rebless-owed` is what turns an intentional corpus-visible change's regressions into warnings, but `rosetta-audit.yml` only listened for the default `pull_request` types, so the label could only ever be read by the run that OPENED the PR. `gh pr create --label` attaches labels after that payload is built, so #2703 -- the first PR to actually use the flow #2699 introduced -- read zero labels and went red on a regression its author had already declared, with no way to re-trigger short of an empty commit. Adds `labeled`/`unlabeled` to the trigger types and documents the gotcha in ROSETTA_AUDIT.md's decision list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol
added a commit
that referenced
this pull request
Sep 4, 2026
) (#2703) * fix(core-engine): matlab's return channel is not a state mutation (#2654) MATLAB has no `return <value>` statement. A function result IS an assignment to a variable named in the `function [out] = f(...)` signature, so `out = env;` is the exact statement c/go/java write as `return env;`. state_mutation's rule counts every assignment, which made matlab the one language in the registry that pays for returning: on the keyword-rosetta corpus 15 of 19 raw hits were the return convention, and with #2546's per-function x3 flux weighting that is 25 of the cell's 25 -- against a corpus median of 2, the widest deviation in the whole bias table. Excluding every write to an output variable would be wrong the other way. runica.m uses `weights` as the accumulator for a whole ICA loop and assigns it repeatedly; that is genuine working state. The discriminator is not anything on the assignment's own line but whether the enclosing function ever READS the name back: a name whose every occurrence in the body is a bare left-hand side is purely the return channel, while a read, a self-reference (`out = out + i`) or a structured write (`out(i) = x`) makes it ordinary state and keeps all its hits. Implemented as a `matlab_return_channel` entry in #2674's registry-declared `_scope_filters`, so the registry stays data and an unknown filter name can only restore the pre-filter count, never zero the metric. Measured over Prism code streams, which is what coding_analysis actually sees: * keyword-rosetta: 19 hits -> 4, i.e. the cell moves 25 -> 4 (median 2, the same value haskell/m4/perl/dockerfile already sit at). All four files move and no other language does -- rosetta_audit reports 45/46 PASS with matlab the single regression, so this PR carries `rosetta:rebless-owed` and the corpus re-bless follows against engine main. * language-crucible eeglab: 1321 -> 1306 (1.1%). Every one of the 15 drops is a terminal binding (`com = ''`, `varargout = {...}`, `name = dirName;`, `h = uimenu(...)`, `this = class(TMP, 'eegobj')`); runica.m, the file that works its outputs hardest, is byte-identical at 440. Golden master re-blessed in both modes: 72 diffs each, 43 of them the benign corpus-wide topological re-solve and 29 substantive -- all inside matlab/eeglab apart from the two whole-corpus health averages those roll up into. Risk exposures fall slightly for real MATLAB (eeglab cognitive_load 62.99 -> 60.13), which is the intended direction. Suite 7533 green, audit_check clear (ruff/mypy/dead-key/ast), tree-sitter 30/30 and tri-comparison 3/3 OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: re-run rosetta-audit when the rebless-owed label lands (#2654) `rosetta:rebless-owed` is what turns an intentional corpus-visible change's regressions into warnings, but `rosetta-audit.yml` only listened for the default `pull_request` types, so the label could only ever be read by the run that OPENED the PR. `gh pr create --label` attaches labels after that payload is built, so #2703 -- the first PR to actually use the flow #2699 introduced -- read zero labels and went red on a regression its author had already declared, with no way to re-trigger short of an empty commit. Adds `labeled`/`unlabeled` to the trigger types and documents the gotcha in ROSETTA_AUDIT.md's decision list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
27 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #2682. Makes the keyword-rosetta gate behave the way the tri-comparison system already does: a PR-time audit that only measures and is baseline-gated, no pin on either side, and a push-to-main job (in the corpus repo) that regenerates the chart by itself.
Root cause, in one paragraph.
ENGINE_REF(corpus) andKEYWORD_ROSETTA_REF(here) pointed at each other. The corpus PR neededENGINE_REF=pull/N/headto pass, had to be back onmainbefore merging, then failed againstmainbecause the engine PR was not in it yet, while the engine PR waited for a corpus SHA that only existed after that merge. In practice the corpus merged with a knowingly wrong ref and owed a reset nothing enforced (keyword-rosetta#39 was a PR whose only content was that reset). Neither repo requires a status check to merge, so both pins were only ever making an advisory check green before a merge that happened anyway.What changes
.github/workflows/rosetta-audit.ymlmain; no variable. Builds the PR's base branch in a second venv and calls the script below.rosetta:rebless-owedlabel → regressions become warnings.tests/tools/rosetta_audit.py(new)tests/tools/tri_comparison_chart.py--cifails closed when a baselined language has no data (that is the galaxyscope-off-PATH no-op, reproduced before the fix: 3 languages skipped,all OK, exit 0). Zero languages → 1.tests/tools/tree_sitter_accuracy_audit.pytests/tools/test_audit_runners_refuse_empty.py(new)docs/self_scan/ROSETTA_AUDIT.mdBUMPING_THE_ROSETTA_PIN.md. The three outcomes, the label, local repro.CONTRIBUTING.md,docs/ecosystem.md,CLAUDE.mdThe flow after this
rosetta-auditnames the languages it moves; label it if intended.bias-history.ymlregenerates the chart and keeps a "corpus owes a re-bless" issue in sync in between.Verified
rosetta_audit.pyagainst the corpus with the main.venv— 46 checked, 0 regression, 0 pre-existing, 0 broken. With galaxyscope off PATH / no corpus / zero folders → exit 2 each.pytestfrom an unrelated cwd: 21 passed (new file +test_tri_comparison_chart_badge.py).audit_check.py: all clear.rosetta-auditrun exercises the new workflow (the file is in its path filter). A follow-up throwaway commit will deliberately move one corpus count to show the regression classification firing, then be reverted.Cross-repo
KEYWORD_ROSETTA_REFvariable (already unused), create therosetta:rebless-owedlabel, note the retired pin check on Rosetta to green: cause-first batch plan + multi-session coordination (child of #2560) #2669.No engine code, manifests, ledger entries or golden masters change.
🤖 Generated with Claude Code