Skip to content

feat(core-engine): jcl dead_code + jcl/yaml spec_exposure comment-stream rules (#2732) - #2736

Merged
squid-protocol merged 3 commits into
mainfrom
fix/2732-jcl-yaml-comment-stream-rules
Sep 5, 2026
Merged

feat(core-engine): jcl dead_code + jcl/yaml spec_exposure comment-stream rules (#2732)#2736
squid-protocol merged 3 commits into
mainfrom
fix/2732-jcl-yaml-comment-stream-rules

Conversation

@squid-protocol

@squid-protocol squid-protocol commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Closes #2732.

Fills the three comment-stream rule absences the epic #2560 n/a sweep flagged: jcl/dead_code, jcl/spec_exposure, yaml/spec_exposure. Both jcl absences are the #2610 leftover — jcl's comment stream was structurally empty until _strip_jcl_comments, and #2610/#2611 added the debt/ownership rules without revisiting these two.

jcl/dead_code

Turning a statement's // into //* is the universal JCL way to comment it out. The operand guard — a keyword must be followed by operand-shaped text, not prose — is load-bearing rather than decorative, and cics-genapp's CICSTS56.jcl is the proof: it carries

//* SET THE RETURN CODE TO CONTROL IF CICS SHOULD BE
//* EXECUTE DUMP UTILITY PROGRAM TO PRINT THE

as English banners a few lines above four genuinely commented-out statements:

//*        DD DSN=CSQ901.SCSQLOAD,DISP=SHR

A bare (?:EXEC|DD|JOB|SET|INCLUDE)\b counts all six. The guard keeps the four and drops the two (THE is not NAME=; EXECUTE leaves no space after EXEC).

yaml/spec_exposure — where this departs from the issue, and why

This is the substantive part of the review. The issue asked for the generic bracket-tag rule verbatim from python/go/java/js, on the reasoning that

because spec_exposure never sees the code stream, YAML's [a, b] flow-sequence syntax cannot FP against it — the usual reason to hesitate doesn't apply.

That premise is wrong. coding_analysis applies every non-underscore rule to the code stream, and comment_analysis then runs the comment-stream rules a second time over the comments — it supplements the code-stream pass, it does not replace it. Measured, with the issue's rule dropped in verbatim, on a workflow containing no comments whatsoever:

jobs:
  audit:
    steps: [...]
  build:
    needs: [audit, lint]     # ← spec_exposure = 1

YAML is the one language where a bracket holding bare unquoted words is ordinary syntax rather than a tag, so the absence was never as arbitrary as it looked. Both new spec_exposure rules are anchored to their language's comment marker instead (^[ \t]*#, ^//\*) — the same anchoring yaml's own dead_code and jcl's own ownership rule already use. Since prism strips those markers out of the code stream, the anchor makes the rules structurally comment-only, and it costs no recall: prism re-emits an end-of-line comment on its own #-led line, so trailing tags still count (pinned by a test).

Second measured fix, carried into both rules: the generic pattern's bare spec branch has no trailing boundary, so it matches "specified" and "species". That isn't hypothetical — 2 of the 3 code-stream hits across the 41,815 .yml/.yaml files in the local pool corpus were [specified\n per-machine] (meson's docs) and [species] (an elasticsearch fixture). Added \b after the alternation.

Both new patterns are ReDoS-safe by construction: each bounded run excludes the delimiter that must follow it ([^\n\[]{0,200} before a literal [, [^\]\n]{0,300} before ], the name charset before [ \t]+), so every quantifier has exactly one landing site and there is no ambiguous partition to backtrack over. Covered by assert_redos_immune payloads sized to each quantifier.

Differential Scan

28 differences in both modes, every one attributable to jcl/dead_code. Enumerated in full via the manual golden_diff route (the crucible_check.py output is capped, so scoping from it alone isn't possible):

what moved count
Commented-out Code (Dead Logic) per file 13
derived Commented Logic Exposure per file 13
repo-level dead_code + Commented Logic Exposure average 2

13 files in cics-banking-sample-application-cbsa gain 16 occurrences (CICSTS56 → 4, the twelve CREL*/CREDB2L → 1 each). That matches an independent count taken over the 443 licensed .jcl/.prc files in the local pool, which contains the same source — a useful cross-check that the rule fires on exactly the shapes it was built from.

No coordinate churn, no other language moved, and yaml moved nothing at all — the anchored rule finds no tag anywhere in the corpus, which is precisely why the paired corpus plant is required rather than optional.

The regenerated fixtures also carry four machine-specific metadata lines per file (absolute path, remote origin URL, timestamp, scan duration); golden_diff.load_and_sanitize explicitly pops all four as machine/runner-specific noise, so they are inert.

Verification

  • pytest tests/ — 7580 passed, 4 skipped, 9 xfailed, 3 xpassed
  • tests/tools/audit_check.py — ruff (70-finding baseline, format clean), mypy (2-error baseline), dead-key, ast-accuracy all clear
  • tests/tools/crucible_check.py — PASS in both full_precision and zero_dependency after the bless
  • New coverage: 5 dead_code + 2 spec_exposure table cases for jcl, 3 for yaml; an end-to-end prism→detector test per language proving the counts arrive through the comment stream and that the JES3 //*MAIN verb stays in code; a dedicated regression pinning the flow-sequence FP the issue's proposed rule would have introduced; word-boundary and ReDoS tests

Cross-repo

Pairs with squid-protocol/keyword-rosetta#55 (draft), which plants the jcl commented-out statement and the jcl/yaml [SPEC-n] tags, and retires ledger entry comment-stream-rules-missing-jcl-yaml (still_reproducesfalse).

Per keyword-rosetta docs/GATING.md's n/a semantics, filling a None rule ends the cell's incomparability whether or not the corpus has anything for it to match — so without that plant these three cells flip from documented-n/a to a manufactured measured-0 scored against the median.

Merge order: this PR first. Rosetta CI checks out engine main, so the corpus PR stays draft until this lands and re-blesses against engine main afterwards. rosetta-audit on this PR will report the three cells moving; that is the expected signal, not a failure.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W

squid-protocol and others added 3 commits September 4, 2026 21:25
…eam rules (#2732)

Fills the three comment-stream rule absences #2732 found: jcl had neither
dead_code nor spec_exposure, yaml had no spec_exposure. Both jcl absences are
the #2610 leftover -- jcl's comment stream was structurally empty until
_strip_jcl_comments, and #2610 added the debt/ownership rules without revisiting
these two.

jcl/dead_code: commenting a statement out by turning `//` into `//*` is the
universal JCL practice. The operand guard (a keyword must be followed by
operand-shaped text) is load-bearing rather than decorative: cics-genapp's
CICSTS56.jcl carries `//* SET THE RETURN CODE TO CONTROL...` and `//* EXECUTE
DUMP UTILITY PROGRAM...` English banners a few lines from four genuinely
commented-out `//*        DD DSN=CSQ901...` statements. A bare
`(?:EXEC|DD|JOB|SET|INCLUDE)\b` would have counted all of them.

yaml/spec_exposure departs from the shape the issue proposed, and the departure
is the substance of this change. #2732 asked for the generic bracket-tag rule
verbatim, reasoning that "spec_exposure never sees the code stream, so YAML's
[a, b] flow-sequence syntax cannot FP against it". That premise is wrong:
coding_analysis applies every non-underscore rule to the code stream and
comment_analysis then adds a second pass over the comments -- it supplements
the code-stream pass, it does not replace it. Dropped in verbatim, the generic
rule scores spec_exposure=1 on a workflow with no comments at all, off
`needs: [audit, lint]` alone. YAML is the one language where a bracket holding
bare unquoted words is ordinary syntax rather than a tag, so the absence was
never as arbitrary as it looked. Both new spec_exposure rules are therefore
anchored to their language's comment marker (`^[ \t]*#`, `^//\*`) -- the same
anchoring yaml's own dead_code and jcl's own ownership rule already use -- which
makes them structurally comment-only, since prism strips those markers out of
the code stream.

Second measured fix carried into both: the generic rule's bare `spec` branch has
no trailing boundary, so it matches "specified" and "species". 2 of the 3
code-stream hits across the 41,815 pool .yml/.yaml files were exactly that.
Added `\b` after the alternation.

Differential Scan: 28 differences in both modes, all jcl, all attributable to
jcl/dead_code -- 13 files in cics-banking-sample-application-cbsa gaining 16
occurrences (CICSTS56 4, the twelve CREL*/CREDB2L 1 each, matching an
independent count over the 443 licensed .jcl/.prc files in the local pool),
plus the derived Commented Logic Exposure percentages and the repo average. No
coordinate churn, no other language moved, and yaml moved nothing at all --
the anchored rule finds no tag in the corpus, which is why the paired corpus
plant below is required rather than optional. The regenerated fixtures also
carry four machine-specific metadata lines per file (absolute path, remote URL,
timestamp, duration); golden_diff.load_and_sanitize pops all four.

Cross-repo: pairs with keyword-rosetta PR planting the jcl commented-out
statement and the jcl/yaml [SPEC-n] tags, and retiring ledger entry
comment-stream-rules-missing-jcl-yaml. Per docs/GATING.md's n/a semantics,
filling a None rule ends the cell's incomparability whether or not the corpus
has anything to match, so without that plant these three cells would flip from
documented-n/a to a manufactured measured-0. THIS PR MERGES FIRST (rosetta CI
checks out engine main); the corpus PR re-blesses against engine main after.

Closes #2732

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W
…ment-stream-rules

# Conflicts:
#	tests/golden_master_audit.json
#	tests/golden_master_zero_dep_audit.json
main moved to 074bbef while this branch was open; #2734 (api-rule orphan
surface) rewrote both fixtures wholesale, so the merge conflicted on them.
Resolved by taking main's fixtures and regenerating from scratch on the merged
engine rather than hand-merging -- the two changes are orthogonal, and the
drift against main is again exactly 28 differences, the same jcl-only set as
before the merge (13 files' Commented-out Code counts, their derived Commented
Logic Exposure percentages, and the repo-level average).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8F1cmMsujEjWnnw4d1e4W
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit 97762b4 into main Sep 5, 2026
31 checks passed
@squid-protocol
squid-protocol deleted the fix/2732-jcl-yaml-comment-stream-rules branch September 5, 2026 02:23
squid-protocol added a commit that referenced this pull request Sep 5, 2026
Conflict: both #2736 and this branch re-blessed the golden masters. Resolved to
main's fixtures (#2736's bless of the jcl/yaml comment-stream rules); this
branch's own two lines are regenerated on top in the next commit rather than
text-merged, since a golden master is generated output, not source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol added a commit that referenced this pull request Sep 5, 2026
Regenerated on top of #2736's fixtures rather than text-merging the conflict.
The residual diff against main is exactly this branch's own two lines --
`Class/Entity Declarations` 0 -> 1 on yacc/freebsd/config.y and jailparse.y --
confirming the two changes are orthogonal: measured on the merged tree BEFORE
regenerating, the only drift against #2736's fixtures was those same two lines.

Both modes PASS after the update. The header churn (absolute corpus path,
timestamp, scan duration, remote URL suffix) is machine-specific and sanitized
away by tests/golden_diff.py's load_and_sanitize.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol added a commit that referenced this pull request Sep 5, 2026
…wlist entry that makes it safe (#2644) (#2737)

* feat(core-engine): yacc's %union is a real class_start, with the allowlist entry that makes it safe (#2644)

`yacc.py` wired `class_start: None` on the reasoning that a grammar file has no
object/type concept. True for OOP-style classes, but it misses that Bison's
`%union` directive declares a real compound type -- the C union spanning every
grammar rule's semantic value (`$$`/`$1`, which `args` already counts). That is
the same "non-OOP language's struct/class equivalent" mapping the engine already
makes for Fortran's `TYPE ... END TYPE`, COBOL's `PROGRAM-ID` and assembly's
`struc` macros, and it is core grammar syntax rather than incidentally-embedded
C: `internal_discriminator` already lists `union` among the `%`-directives used
to identify a file as yacc in the first place.

The rule alone would have made yacc worse. detector.py's named-class extractor
consults a language's own `class_start` only for languages in
`_CLASS_START_NAMED_EXTRACTION_LANGS`; everyone else falls through to a legacy
generic regex (`class|struct|interface|trait|enum`), which reads every `struct
foo` declaration in a grammar's embedded C action code as a class -- 17 on
config.y, 9 on jailparse.y and 109 across all four real grammar files in the
crucible corpus, where the honest answers are 1, 1 and 2. So yacc joins the
allowlist in the same change; the two halves only make sense together.

Verified by direct source cross-check, the same way abap/cobol/jcl/sqlite were:
yacc is tree-sitter-blind, and ctags cannot corroborate classes either (`ctags
--list-kinds-full=YACC` exposes exactly one kind, `l`, which is why
`CTAGS_CLASS_KINDS["yacc"]` is empty on purpose). `%union` fires exactly once in
each grammar that declares one (config.y:1, jailparse.y:45) and zero times in
the two that do not (gnucobol's parser.y/scanner.l use `%define api.value.type`)
-- 100% precision, and an honest zero where there is no union.

Golden masters re-blessed. The diff is exactly two lines in each mode:
`Class/Entity Declarations` 0 -> 1 on config.y and jailparse.y. Nothing else
moves -- no mass, no risk exposure, no topological re-solve.

Tests: the yacc extraction gauntlet's `CLASS_CASES` were empty (the rule was
None) and are now populated, including the invalid cases that matter here -- a
bare C `union`/`struct` in action code, a mid-line or commented-out occurrence,
and `%unionize` against the `\b` guard. Two detector-level tests pin the pair
together (one `%union` plus surrounding C structs -> exactly one class; a
grammar with no `%union` -> none), plus ReDoS coverage for the new rule.

Closes #2644. Part of #2669.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: re-bless golden master after merging main (#2644)

Regenerated on top of #2736's fixtures rather than text-merging the conflict.
The residual diff against main is exactly this branch's own two lines --
`Class/Entity Declarations` 0 -> 1 on yacc/freebsd/config.y and jailparse.y --
confirming the two changes are orthogonal: measured on the merged tree BEFORE
regenerating, the only drift against #2736's fixtures was those same two lines.

Both modes PASS after the update. The header churn (absolute corpus path,
timestamp, scan duration, remote URL suffix) is machine-specific and sanitized
away by tests/golden_diff.py's load_and_sanitize.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jcl + yaml: dead_code and spec_exposure comment-stream rules missing where the comment stream exists (#2610 leftover)

1 participant