Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/api_rule_contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ not a defect:
| `makefile` | `.PHONY`, `export`, the conventional public targets |
| `sqlite` | `CREATE [TEMP] VIEW` / `CREATE VIRTUAL TABLE` |
| `matlab` | a column-0 `function` declaration (a function file's callable surface) |
| `jcl` | `//name PROC` — a cataloged or in-stream procedure, what `EXEC name` in other members invokes ([#2748](https://github.com/squid-protocol/gitgalaxy/issues/2748)) |

`matlab` is the one place where the fallback is knowingly approximate: a `.m` function file
publishes only its *leading* function, and local functions after it are file-private, but nothing
Expand Down Expand Up @@ -91,7 +92,7 @@ untouched.
| `html` | 514 | 5 |
| `java` | 220 | 12 |
| `javascript` | 267 | 12 |
| `jcl` | rule is `None` | n/a |
| `jcl` | `None` → 13 | n/a → 1 (#2748) |
| `kotlin` | 17 → 5 | 12 |
| `livecode` | 607 | 13 |
| `lua` | 288 → 272 | 13 |
Expand Down Expand Up @@ -161,7 +162,7 @@ answer to "why did a real fix produce no diff" for those two — the strict test
(`test_java_api_contract_2730`, `test_csharp_api_contract_2730`) hold the guard in place.

**Already inside the contract (23, untouched):** `apex`, `cpp`, `css`, `dockerfile`,
`embedded_python`, `fortran`, `haskell`, `html`, `javascript`, `jcl` (`None`), `livecode`, `m4`,
`embedded_python`, `fortran`, `haskell`, `html`, `javascript`, `jcl` (`None` at the time; #2748 later gave it the PROC statement, fallback family), `livecode`, `m4`,
`makefile`, `markdown` (`None`), `python`, `rust`, `scala`, `scheme`, `solidity`, `sqlite`, `tcl`,
`yacc`, `zig`.

Expand Down
43 changes: 26 additions & 17 deletions docs/language_status/jcl.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
| **Status** | production |
| **Target Version** | IBM z/OS JCL |
| **Lexical Family** | line_exclusive (with a dedicated `//*` whole-line stripper — see §10) |
| **Rules Wired** | 16 / 27 |
| **Rules Wired** | 21 / 30 |
| **Extraction tests** | 42 |
| **Strict tests** | 65 |
| **Strict tests** | 100 |

## 2. Identification surface

Expand All @@ -28,18 +28,21 @@
- `class_start`: Matches JCL JOB cards.

**Safety & Risk**
- `high_risk_execution`: Matches execution of specific programs via `PGM=`.
- `high_risk_execution`: Matches `PGM=` naming a program whose purpose is to execute **caller-supplied commands** — TSO/E batch (`IKJEFT01`/`IKJEFT1A`/`IKJEFT1B`), the z/OS UNIX launchers (`BPXBATCH`/`BPXBATSL`/`BPXBATA2`/`BPXBATA8`/`AOPBATCH`), the REXX interpreter (`IRXJCL`) and batch `SDSF`. Narrowed by [#2751](https://github.com/squid-protocol/gitgalaxy/issues/2751) from a bare `PGM=<anything>`, which counted every step (188 of the corpus's 376 `EXEC` statements: `IEFBR14` 26, compilers and the linker 26, copy/catalog utilities 28) — running a program is what a step *is*, and no other language's rule counts "runs a command". Now 61 hits in 48 files, all `IKJEFT01` on this corpus.
- `safety`: Matches `COND=` return-code tests (JCL's step error-handling); the bare bypass forms are excluded by lookahead. Added by [#2610](https://github.com/squid-protocol/gitgalaxy/issues/2610).
- `safety_bypasses`: Matches `COND=EVEN` / `COND=ONLY` (run the step despite a prior abend — JCL's native ignore-the-error idiom). The combined form `COND=((4,LT),EVEN)` deliberately counts **both** safety and bypass: it carries a real RC test and a run-after-abend bypass at once. Added by #2610.

**Resource Management**
- `io`: Matches dataset definitions and I/O routing such as `DSN`, `DSNAME`, `SYSOUT`, `SYSPRINT`, `DISP=`.
- `sync_locks`: Matches `DISP=OLD` / `DISP=MOD` — the dispositions that request an **exclusive system ENQ** on a dataset, z/OS's native serialization idiom (`DISP=SHR` requests shared access and `DISP=NEW` allocates, so neither counts). Added by [#2733](https://github.com/squid-protocol/gitgalaxy/issues/2733). Every hit is also an `io` hit, since `io` counts the bare `DISP=` keyword; that overlap is accepted deliberately because the shape is narrow (48 hits in 15 of the corpus's 186 `.jcl`/`.prc`/`.bms` files, ~9% of its 525 `DISP=` occurrences; 429 of the remaining 477 are `DISP=SHR` and the other 48 allocate — `DISP=(NEW,…)`, or an omitted first positional that defaults to it) and the two meanings genuinely differ, the same way `COND=((4,LT),EVEN)` counts both `safety` and `safety_bypasses`. Contrast §4's `cleanup`, where the overlap would have covered essentially every disposition.
- `cleanup`: Matches `DELETE` as a dataset's **normal-termination** disposition — `DISP=(MOD,DELETE,DELETE)`, `DISP=(OLD,DELETE)`, `DISP=(,DELETE)` — JCL's teardown idiom (`IEFBR14` + `DISP=(MOD,DELETE,DELETE)` is how a batch job deletes a dataset). Added by [#2749](https://github.com/squid-protocol/gitgalaxy/issues/2749), reversing #2610's rejection on the terms #2733 set for `sync_locks`: a narrow subset (36 of the corpus's 533 `DISP=` occurrences, 13 of 186 files) with an accepted, test-pinned overlap — every hit is also an `io` hit, and the OLD/MOD forms are `sync_locks` hits too. The abnormal-termination positional of an allocation (`DISP=(NEW,CATLG,DELETE)`, 12 more) is excluded: a conditional disposition on a *create* is not teardown.
- `sync_locks`: Matches `DISP=OLD` / `DISP=MOD` — the dispositions that request an **exclusive system ENQ** on a dataset, z/OS's native serialization idiom (`DISP=SHR` requests shared access and `DISP=NEW` allocates, so neither counts). Added by [#2733](https://github.com/squid-protocol/gitgalaxy/issues/2733). Every hit is also an `io` hit, since `io` counts the bare `DISP=` keyword; that overlap is accepted deliberately because the shape is narrow (48 hits in 15 of the corpus's 186 `.jcl`/`.prc`/`.bms` files, ~9% of its 525 `DISP=` occurrences; 429 of the remaining 477 are `DISP=SHR` and the other 48 allocate — `DISP=(NEW,…)`, or an omitted first positional that defaults to it) and the two meanings genuinely differ, the same way `COND=((4,LT),EVEN)` counts both `safety` and `safety_bypasses`. #2749 later took the same route for `cleanup`'s DELETE subset.

**State Mutation**
- `state_mutation`: Matches JCL symbolic variable assignments via `SET`.
- `globals`: Matches the job-scoped declarations — `//JOBLIB DD` (the program search library for every step; its step-scoped twin `STEPLIB` does not count), a `// SET` symbol (readable by every later statement; its scoped twin is a `PROC` parameter) and `// EXPORT SYMLIST=`. Added by [#2750](https://github.com/squid-protocol/gitgalaxy/issues/2750). `SET` is also `state_mutation`, the same dual dockerfile's `ENV` carries; `JOBLIB` is also an `io` hit and a dependency edge, which is right — a JOBLIB is a dependency of every step. 73 hits in 45 corpus files (JOBLIB 32, SET 39, EXPORT 2).

**Architecture & Domain Sensors**
- `api`: Matches `//name PROC` — a cataloged or in-stream procedure, the callable surface that `EXEC name` / `EXEC PROC=name` in other members invoke (the [api contract](../api_rule_contract.md)'s fallback family; the call site is a reference and does not count). Added by [#2748](https://github.com/squid-protocol/gitgalaxy/issues/2748). 13 declarations in 13 corpus files, invoked by 185 of its 376 `EXEC` statements.
- `import`: Matches JCL includes (`INCLUDE`).
- `_dependency_capture`: Captures the `MEMBER=` name for the dependency graph, as well as dataset names in `DD` statements and `JCLLIB` orders.
- `ownership`: Matches ownership/maintainer comments like `//* Author:` (counted on the comment stream since #2610 — previously it only worked by accident on the code stream, see §10).
Expand All @@ -50,17 +53,7 @@

## 4. What GitGalaxy explicitly does not track

- `api`: None (JCL has no api rule; in practice `api` still appears on scanned JCL via the
orphan-conversion mechanism — see the keyword-rosetta ledger's `api-contextual-baseline-fix`).
- `cleanup`: None — **a deliberate decision, not an oversight** (#2610): the honest JCL cleanup
idiom is `DISP=(...,DELETE)`, but `DISP=` already feeds the `io` rule, so a cleanup rule would
double-count every disposition. Recorded in the keyword-rosetta deviation ledger
(`jcl-2610-rebaseline-residual-morphology`) as intended morphology. #2733 revisited the same
overlap for `sync_locks` and decided the other way — see §3 for why the two dispositions split.
- `globals`: None — JCL has no scoped-vs-global variable distinction (`SET` symbolics are already
`state_mutation`; a `JOBLIB`/`STEPLIB` rule was considered and rejected because those DD
statements would inflate `io` and `dependency_links`).
- `concurrency`: None.
- `concurrency`: None — steps within a job execute strictly sequentially; parallelism lives in JES scheduling, outside the language's own syntax.
- `ui_framework`: None.
- `closures`: None.
- `decorators`: None.
Expand All @@ -69,6 +62,7 @@
- `scientific`: None.
- `reflection_metaprogramming`: None.
- `debug_prints`: None.
- `doc`: None — no structured header-comment idiom is attested in the corpus (0 of its 1,610 `//*` lines carry a `Description:`/`Purpose:`-style key; the structured lines are copyright/SPDX banners). `test`: None — no native testing concept (`TYPRUN=SCAN`/`IEBCOMPR` do not appear in real decks either). Both re-checked 2026-09-05.

## 5. Known limitations (accepted, not fixed)

Expand All @@ -80,10 +74,12 @@ None currently. ([#2415](https://github.com/squid-protocol/gitgalaxy/issues/2415
## 6. Test depth

- **Extraction-gauntlet tests**: 42 cases in `tests/extraction/languages/test_jcl.py`
- **Strict-signature tests**: 80 cases in `tests/extraction/languages/test_jcl_strict.py`
- **Strict-signature tests**: 100 cases in `tests/extraction/languages/test_jcl_strict.py`
(51 → 65 with #2610's COND-partition semantics, JES3-guard and ReDoS detonation cases;
65 → 74 with #2732's `dead_code`/`spec_exposure` rules; 74 → 80 with #2733's `sync_locks`
disposition partition, io-overlap pin and ReDoS case)
disposition partition, io-overlap pin and ReDoS case; 80 → 100 with #2748–#2751's `api`,
`cleanup`, `globals` and narrowed `high_risk_execution` — per-rule partitions, overlap pins,
one end-to-end deck through prism + splice, and ReDoS detonations)

## 7. Relevant closed work

Expand Down Expand Up @@ -256,6 +252,19 @@ decomposed into exactly five causes, each with a different fix path — recorded
and the ×3 flux weighting ([#2546](https://github.com/squid-protocol/gitgalaxy/issues/2546))
inflating the cross-language median. No JCL action; re-baselines when those land.

**2026-09-05 revisit (#2748–#2751).** The second-look pass over every non-green rosetta cell
re-read bucket 4's zeros against the live rules and the crucible corpus and found that three of
them were rule gaps after all, plus one over-count: `api` (`//name PROC` — 185 of the corpus's
376 EXEC steps invoke a procedure), `cleanup` (`DISP=(…,DELETE)` — #2610's io-overlap rejection
no longer held once #2733 accepted the same overlap for `sync_locks`), `globals` (JOBLIB vs
STEPLIB, SET vs PROC parameters, EXPORT SYMLIST — the scoped-vs-global distinction bucket 4 said
did not exist) and `high_risk_execution` (a bare `PGM=` counted every step, so the corpus had
been dodging the rule with `EXEC <proc>` steps rather than measuring it). §3/§4 carry the rules
and the reasoning; `doc` and `test` were re-checked and stay `None` with corpus evidence. The
corpus pairing (a PROC wrapper, a `DISP=(OLD,DELETE)` DD that also pays #2742's owed
`DISP=OLD` plant, no new plant for `globals` since `b.jcl`'s two `SET` lines already read 2,
and `PGM=IKJEFT01`/`BPXBATCH` in place of `IEFBR14`) is keyword-rosetta#59.

**Remaining out-of-band** (all accounted, none actionable at the JCL level): the bucket-4
morphology zeros (final disposition is the epic's scoring-side "absent morphology = incomparable"
work), the bucket-5 metrics (blocked on #2545/#2546), and the §3 risk-score consequences
Expand Down
100 changes: 96 additions & 4 deletions gitgalaxy/standards/language_standards/languages/jcl.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,34 @@
"func_start": re.compile(r"^[ \t]*//([A-Za-z0-9_#$@]*)[ \t]+EXEC\b", re.M | re.I),
# Classes/Entities (JOB cards). Same `\s+` -> `[ \t]+` cross-line fix as above.
"class_start": re.compile(r"^[ \t]*//([A-Za-z0-9_#$@]+)[ \t]+JOB\b", re.M | re.I),
# Danger (Execution of arbitrary programs)
"high_risk_execution": re.compile(r"\bPGM=[A-Za-z0-9_#$@]+\b", re.I),
# Danger (execution of arbitrary programs). #2751: the rule used to be a
# bare `PGM=<anything>`, which counted every step -- running a program
# is what a JCL step IS, so the metric read "how many steps name a
# program" rather than "how many steps execute something arbitrary".
# On the language-crucible corpus that was 188 hits over 376 EXEC
# statements: IKJEFT01 60, IEFBR14 26 (a program that does NOTHING --
# run only for its DD allocation/deletion side effects), IDCAMS 16,
# compilers and the linker 26, copy utilities 12, application programs
# 38. A compile-link-go job scored three high-risk executions for
# compiling. No other language's high_risk_execution counts "runs a
# command" (shell/dockerfile/python count eval/exec/os.system, not every
# command line), so the same planted intent read differently in JCL.
# Narrowed to the programs whose purpose is to execute CALLER-SUPPLIED
# commands: TSO/E batch (IKJEFT01 and its IKJEFT1A/IKJEFT1B variants,
# which run whatever SYSTSIN carries -- TSO commands, CLISTs, REXX, DB2
# DSN RUN PROGRAM), the z/OS UNIX shell and program launchers
# (BPXBATCH/BPXBATSL/BPXBATA2/BPXBATA8, AOPBATCH), the REXX
# interpreter (IRXJCL) and batch SDSF (its ISFIN stream issues MVS
# operator commands). Compilers, copy/catalog utilities and IEFBR14 are
# steps, not arbitrary execution; IDCAMS/IEHPROGM/ADRDSSU are
# destructive-capable utilities but execute a fixed command language,
# so they are left out on the same reasoning that keeps `rm` (not
# `rm -rf /`) out of shell's rule. Unanchored like the other operand
# rules: PGM= can sit on a `//` continuation line (the #2482 shape).
"high_risk_execution": re.compile(
r"\bPGM=(?:IKJEFT01|IKJEFT1[AB]|BPXBATCH|BPXBATSL|BPXBATA[28]|AOPBATCH|IRXJCL|SDSF)\b",
re.I,
),
# I/O (Data Set Names and Sysouts)
"io": re.compile(r"\b(DSN|DSNAME|SYSOUT|SYSPRINT|DISP=)\b", re.I),
# #2610: JCL's error handling is the COND= operand -- a return-code
Expand All @@ -93,7 +119,21 @@
# deliberately counts BOTH -- it carries a real RC test and a run-even-
# after-abend bypass at once.
"safety": re.compile(r"\bCOND=(?!(?:EVEN|ONLY)\b)", re.I),
"api": None,
# #2748: a cataloged or in-stream procedure is JCL's callable surface --
# `//name PROC` declares what `EXEC name` / `EXEC PROC=name` in other
# members invoke, which is the api contract's definition (a declaration
# that makes a named unit visible outside the file it is declared in;
# docs/api_rule_contract.md, fallback family). On the language-crucible
# corpus 185 of the 376 EXEC statements invoke a procedure and 13
# members declare one (10 of them in the member's first seven lines --
# the member IS the procedure). Corollary 1 keeps the call site out:
# `EXEC name` is a reference. Name optional like the other statement
# rules (a cataloged PROC statement may be unnamed); `PEND` only closes
# an in-stream proc and is not a second declaration. A PROC that carries
# parameter defaults (`//BATCH PROC MEMBER=`) also matches args' `PROC`
# alternative -- a declaration with its parameter list, the same
# api+args pair every `def f(x)` produces.
"api": re.compile(r"^[ \t]*//([A-Za-z0-9_#$@]*)[ \t]+PROC\b", re.M | re.I),
# #2610: COND=EVEN ("run even if a prior step abended") and COND=ONLY
# ("run only after an abend") execute a step in spite of upstream
# failure -- JCL's native ignore-the-error idiom. Two alternatives:
Expand Down Expand Up @@ -146,9 +186,61 @@
# than the line carrying the ddname, the same real-corpus shape the
# args rule's #2482 note documents.
"sync_locks": re.compile(r"\bDISP=\(?(?:OLD|MOD)\b", re.I),
# #2749: DELETE as a dataset's normal-termination disposition is JCL's
# teardown idiom -- `//S EXEC PGM=IEFBR14` + `DD DSN=X,DISP=(MOD,DELETE,
# DELETE)` is THE way a batch job deletes a dataset (a no-op program run
# purely for the allocation side effect), and `DISP=(OLD,DELETE)` on a
# `&&TEMP` dataset frees it once the step is done. That is what cleanup
# measures through each language's own idiom elsewhere (shell's `rm -f`
# / `trap ... EXIT`, dockerfile's `apt-get clean`, yaml's `docker
# compose down`, #2647).
# #2610 declined this for the `io` overlap; #2733 (PR #2742) reversed
# that posture for the same operand's OLD/MOD subset, and the same
# measurement applies here: on the language-crucible corpus 36 of 533
# `DISP=` occurrences (13 of 186 files) carry DELETE in the
# normal-termination position. Narrowed to that position deliberately:
# the abnormal-termination positional (`DISP=(NEW,CATLG,DELETE)`, 12
# more) is a conditional disposition on an ALLOCATION -- the step's
# intent is to create the dataset and keep it, and only discard it if
# the step abends -- so it is excluded, the way `DISP=SHR`/`NEW` stay
# out of sync_locks. `[^,()\n]*` for the status positional accepts the
# omitted form `DISP=(,DELETE)` (a scratch dataset created and dropped
# in one step) and cannot cross a newline or a paren, so the scan is
# bounded to one operand (ReDoS-safe: the class excludes `,`, so the
# comma partitions at exactly one position).
# The overlap is accepted, not avoided, on #2742's terms: every hit is
# also an `io` hit (the DD's `DSN=`; `DISP=(` itself does not match
# io's `\bDISP=\b` -- no word boundary between `=` and `(`), and the
# `(OLD,DELETE)` / `(MOD,DELETE,DELETE)` forms are also sync_locks hits,
# because they hold an exclusive ENQ on the dataset they then drop.
# Pinned by test_jcl_cleanup_overlaps_io_and_sync_locks_by_design.
"cleanup": re.compile(r"\bDISP=\([^,()\n]*,[ \t]*DELETE\b", re.I),
"ui_framework": None,
"closures": None,
"globals": None,
# #2750: JCL does have a scoped-vs-global distinction, in three places:
# `//JOBLIB DD` is the program search library for EVERY step of the job
# (its step-scoped twin, `//STEPLIB DD`, applies to one step and is not
# counted -- crucible: JOBLIB in 32 files, STEPLIB 70 lines in 59); a
# `// SET SYM=value` symbol is readable by every later statement in the
# job (its scoped twin is a `PROC` parameter -- SET 39 lines in 12
# files); `// EXPORT SYMLIST=` makes symbols visible inside in-stream
# data (2). That is the global-vs-local pair globals measures elsewhere
# (a python module-level name vs a def-local, dockerfile's image-wide
# `ENV` vs a RUN-local export, yaml's `${{ env.X }}`).
# SET is also `state_mutation`, and that is dockerfile's `ENV` shape
# exactly -- dual globals + state_mutation, ledgered in the rosetta
# corpus as batch4-dual-keyword-overlaps: a declaration that creates a
# job-wide symbol is both the creation of global state and a mutation
# of it. JOBLIB's DD line is also an `io` hit (its DSN) and a
# `_dependency_capture` edge, which is right -- a JOBLIB is a dependency
# of every step. Same anchoring as state_mutation (a real statement
# line, never inline SYSIN payload); the JOBLIB alternative is named
# exactly so a `//JOBLIBX` or `//SYSPROC` ddname cannot match.
"globals": re.compile(
r"^[ \t]*//JOBLIB[ \t]+DD\b"
r"|^[ \t]*//[A-Za-z0-9_#$@]*[ \t]+(?:SET[ \t]+[A-Za-z0-9_#$@]+=|EXPORT[ \t]+SYMLIST\b)",
re.M | re.I,
),
"decorators": None,
"generics": None,
"comprehensions": None,
Expand Down
Loading
Loading