feat(core-engine): jcl dead_code + jcl/yaml spec_exposure comment-stream rules (#2732) - #2736
Merged
Merged
Conversation
…eam rules (#2732) Fills the three comment-stream rule absences #2732 found: jcl had neither dead_code nor spec_exposure, yaml had no spec_exposure. Both jcl absences are the #2610 leftover -- jcl's comment stream was structurally empty until _strip_jcl_comments, and #2610 added the debt/ownership rules without revisiting these two. jcl/dead_code: commenting a statement out by turning `//` into `//*` is the universal JCL practice. The operand guard (a keyword must be followed by operand-shaped text) is load-bearing rather than decorative: cics-genapp's CICSTS56.jcl carries `//* SET THE RETURN CODE TO CONTROL...` and `//* EXECUTE DUMP UTILITY PROGRAM...` English banners a few lines from four genuinely commented-out `//* DD DSN=CSQ901...` statements. A bare `(?:EXEC|DD|JOB|SET|INCLUDE)\b` would have counted all of them. yaml/spec_exposure departs from the shape the issue proposed, and the departure is the substance of this change. #2732 asked for the generic bracket-tag rule verbatim, reasoning that "spec_exposure never sees the code stream, so YAML's [a, b] flow-sequence syntax cannot FP against it". That premise is wrong: coding_analysis applies every non-underscore rule to the code stream and comment_analysis then adds a second pass over the comments -- it supplements the code-stream pass, it does not replace it. Dropped in verbatim, the generic rule scores spec_exposure=1 on a workflow with no comments at all, off `needs: [audit, lint]` alone. YAML is the one language where a bracket holding bare unquoted words is ordinary syntax rather than a tag, so the absence was never as arbitrary as it looked. Both new spec_exposure rules are therefore anchored to their language's comment marker (`^[ \t]*#`, `^//\*`) -- the same anchoring yaml's own dead_code and jcl's own ownership rule already use -- which makes them structurally comment-only, since prism strips those markers out of the code stream. Second measured fix carried into both: the generic rule's bare `spec` branch has no trailing boundary, so it matches "specified" and "species". 2 of the 3 code-stream hits across the 41,815 pool .yml/.yaml files were exactly that. Added `\b` after the alternation. Differential Scan: 28 differences in both modes, all jcl, all attributable to jcl/dead_code -- 13 files in cics-banking-sample-application-cbsa gaining 16 occurrences (CICSTS56 4, the twelve CREL*/CREDB2L 1 each, matching an independent count over the 443 licensed .jcl/.prc files in the local pool), plus the derived Commented Logic Exposure percentages and the repo average. No coordinate churn, no other language moved, and yaml moved nothing at all -- the anchored rule finds no tag in the corpus, which is why the paired corpus plant below is required rather than optional. The regenerated fixtures also carry four machine-specific metadata lines per file (absolute path, remote URL, timestamp, duration); golden_diff.load_and_sanitize pops all four. Cross-repo: pairs with keyword-rosetta PR planting the jcl commented-out statement and the jcl/yaml [SPEC-n] tags, and retiring ledger entry comment-stream-rules-missing-jcl-yaml. Per docs/GATING.md's n/a semantics, filling a None rule ends the cell's incomparability whether or not the corpus has anything to match, so without that plant these three cells would flip from documented-n/a to a manufactured measured-0. THIS PR MERGES FIRST (rosetta CI checks out engine main); the corpus PR re-blesses against engine main after. Closes #2732 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W
…ment-stream-rules # Conflicts: # tests/golden_master_audit.json # tests/golden_master_zero_dep_audit.json
main moved to 074bbef while this branch was open; #2734 (api-rule orphan surface) rewrote both fixtures wholesale, so the merge conflicted on them. Resolved by taking main's fixtures and regenerating from scratch on the merged engine rather than hand-merging -- the two changes are orthogonal, and the drift against main is again exactly 28 differences, the same jcl-only set as before the merge (13 files' Commented-out Code counts, their derived Commented Logic Exposure percentages, and the repo-level average). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W
Contributor
squid-protocol
added a commit
that referenced
this pull request
Sep 5, 2026
Conflict: both #2736 and this branch re-blessed the golden masters. Resolved to main's fixtures (#2736's bless of the jcl/yaml comment-stream rules); this branch's own two lines are regenerated on top in the next commit rather than text-merged, since a golden master is generated output, not source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol
added a commit
that referenced
this pull request
Sep 5, 2026
Regenerated on top of #2736's fixtures rather than text-merging the conflict. The residual diff against main is exactly this branch's own two lines -- `Class/Entity Declarations` 0 -> 1 on yacc/freebsd/config.y and jailparse.y -- confirming the two changes are orthogonal: measured on the merged tree BEFORE regenerating, the only drift against #2736's fixtures was those same two lines. Both modes PASS after the update. The header churn (absolute corpus path, timestamp, scan duration, remote URL suffix) is machine-specific and sanitized away by tests/golden_diff.py's load_and_sanitize. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol
added a commit
that referenced
this pull request
Sep 5, 2026
…wlist entry that makes it safe (#2644) (#2737) * feat(core-engine): yacc's %union is a real class_start, with the allowlist entry that makes it safe (#2644) `yacc.py` wired `class_start: None` on the reasoning that a grammar file has no object/type concept. True for OOP-style classes, but it misses that Bison's `%union` directive declares a real compound type -- the C union spanning every grammar rule's semantic value (`$$`/`$1`, which `args` already counts). That is the same "non-OOP language's struct/class equivalent" mapping the engine already makes for Fortran's `TYPE ... END TYPE`, COBOL's `PROGRAM-ID` and assembly's `struc` macros, and it is core grammar syntax rather than incidentally-embedded C: `internal_discriminator` already lists `union` among the `%`-directives used to identify a file as yacc in the first place. The rule alone would have made yacc worse. detector.py's named-class extractor consults a language's own `class_start` only for languages in `_CLASS_START_NAMED_EXTRACTION_LANGS`; everyone else falls through to a legacy generic regex (`class|struct|interface|trait|enum`), which reads every `struct foo` declaration in a grammar's embedded C action code as a class -- 17 on config.y, 9 on jailparse.y and 109 across all four real grammar files in the crucible corpus, where the honest answers are 1, 1 and 2. So yacc joins the allowlist in the same change; the two halves only make sense together. Verified by direct source cross-check, the same way abap/cobol/jcl/sqlite were: yacc is tree-sitter-blind, and ctags cannot corroborate classes either (`ctags --list-kinds-full=YACC` exposes exactly one kind, `l`, which is why `CTAGS_CLASS_KINDS["yacc"]` is empty on purpose). `%union` fires exactly once in each grammar that declares one (config.y:1, jailparse.y:45) and zero times in the two that do not (gnucobol's parser.y/scanner.l use `%define api.value.type`) -- 100% precision, and an honest zero where there is no union. Golden masters re-blessed. The diff is exactly two lines in each mode: `Class/Entity Declarations` 0 -> 1 on config.y and jailparse.y. Nothing else moves -- no mass, no risk exposure, no topological re-solve. Tests: the yacc extraction gauntlet's `CLASS_CASES` were empty (the rule was None) and are now populated, including the invalid cases that matter here -- a bare C `union`/`struct` in action code, a mid-line or commented-out occurrence, and `%unionize` against the `\b` guard. Two detector-level tests pin the pair together (one `%union` plus surrounding C structs -> exactly one class; a grammar with no `%union` -> none), plus ReDoS coverage for the new rule. Closes #2644. Part of #2669. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: re-bless golden master after merging main (#2644) Regenerated on top of #2736's fixtures rather than text-merging the conflict. The residual diff against main is exactly this branch's own two lines -- `Class/Entity Declarations` 0 -> 1 on yacc/freebsd/config.y and jailparse.y -- confirming the two changes are orthogonal: measured on the merged tree BEFORE regenerating, the only drift against #2736's fixtures was those same two lines. Both modes PASS after the update. The header churn (absolute corpus path, timestamp, scan duration, remote URL suffix) is machine-specific and sanitized away by tests/golden_diff.py's load_and_sanitize. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
27 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2732.
Fills the three comment-stream rule absences the epic #2560 n/a sweep flagged:
jcl/dead_code,jcl/spec_exposure,yaml/spec_exposure. Both jcl absences are the #2610 leftover — jcl's comment stream was structurally empty until_strip_jcl_comments, and #2610/#2611 added the debt/ownership rules without revisiting these two.jcl/dead_code
Turning a statement's
//into//*is the universal JCL way to comment it out. The operand guard — a keyword must be followed by operand-shaped text, not prose — is load-bearing rather than decorative, and cics-genapp'sCICSTS56.jclis the proof: it carriesas English banners a few lines above four genuinely commented-out statements:
A bare
(?:EXEC|DD|JOB|SET|INCLUDE)\bcounts all six. The guard keeps the four and drops the two (THEis notNAME=;EXECUTEleaves no space afterEXEC).yaml/spec_exposure — where this departs from the issue, and why
This is the substantive part of the review. The issue asked for the generic bracket-tag rule verbatim from python/go/java/js, on the reasoning that
That premise is wrong.
coding_analysisapplies every non-underscore rule to the code stream, andcomment_analysisthen runs the comment-stream rules a second time over the comments — it supplements the code-stream pass, it does not replace it. Measured, with the issue's rule dropped in verbatim, on a workflow containing no comments whatsoever:YAML is the one language where a bracket holding bare unquoted words is ordinary syntax rather than a tag, so the absence was never as arbitrary as it looked. Both new
spec_exposurerules are anchored to their language's comment marker instead (^[ \t]*#,^//\*) — the same anchoringyaml's owndead_codeandjcl's ownownershiprule already use. Since prism strips those markers out of the code stream, the anchor makes the rules structurally comment-only, and it costs no recall: prism re-emits an end-of-line comment on its own#-led line, so trailing tags still count (pinned by a test).Second measured fix, carried into both rules: the generic pattern's bare
specbranch has no trailing boundary, so it matches "specified" and "species". That isn't hypothetical — 2 of the 3 code-stream hits across the 41,815.yml/.yamlfiles in the local pool corpus were[specified\n per-machine](meson's docs) and[species](an elasticsearch fixture). Added\bafter the alternation.Both new patterns are ReDoS-safe by construction: each bounded run excludes the delimiter that must follow it (
[^\n\[]{0,200}before a literal[,[^\]\n]{0,300}before], the name charset before[ \t]+), so every quantifier has exactly one landing site and there is no ambiguous partition to backtrack over. Covered byassert_redos_immunepayloads sized to each quantifier.Differential Scan
28 differences in both modes, every one attributable to
jcl/dead_code. Enumerated in full via the manualgolden_diffroute (thecrucible_check.pyoutput is capped, so scoping from it alone isn't possible):Commented-out Code (Dead Logic)per fileCommented Logic Exposureper filedead_code+Commented Logic Exposureaverage13 files in
cics-banking-sample-application-cbsagain 16 occurrences (CICSTS56 → 4, the twelveCREL*/CREDB2L→ 1 each). That matches an independent count taken over the 443 licensed.jcl/.prcfiles in the local pool, which contains the same source — a useful cross-check that the rule fires on exactly the shapes it was built from.No coordinate churn, no other language moved, and yaml moved nothing at all — the anchored rule finds no tag anywhere in the corpus, which is precisely why the paired corpus plant is required rather than optional.
The regenerated fixtures also carry four machine-specific metadata lines per file (absolute path, remote origin URL, timestamp, scan duration);
golden_diff.load_and_sanitizeexplicitly pops all four as machine/runner-specific noise, so they are inert.Verification
pytest tests/— 7580 passed, 4 skipped, 9 xfailed, 3 xpassedtests/tools/audit_check.py— ruff (70-finding baseline, format clean), mypy (2-error baseline), dead-key, ast-accuracy all cleartests/tools/crucible_check.py— PASS in both full_precision and zero_dependency after the blessdead_code+ 2spec_exposuretable cases for jcl, 3 for yaml; an end-to-end prism→detector test per language proving the counts arrive through the comment stream and that the JES3//*MAINverb stays in code; a dedicated regression pinning the flow-sequence FP the issue's proposed rule would have introduced; word-boundary and ReDoS testsCross-repo
Pairs with squid-protocol/keyword-rosetta#55 (draft), which plants the jcl commented-out statement and the jcl/yaml
[SPEC-n]tags, and retires ledger entrycomment-stream-rules-missing-jcl-yaml(still_reproduces→false).Per keyword-rosetta
docs/GATING.md's n/a semantics, filling aNonerule ends the cell's incomparability whether or not the corpus has anything for it to match — so without that plant these three cells flip from documented-n/a to a manufactured measured-0 scored against the median.Merge order: this PR first. Rosetta CI checks out engine
main, so the corpus PR stays draft until this lands and re-blesses against engine main afterwards.rosetta-auditon this PR will report the three cells moving; that is the expected signal, not a failure.🤖 Generated with Claude Code
https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W