Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions docs/language_status/jcl.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@

**Resource Management**
- `io`: Matches dataset definitions and I/O routing such as `DSN`, `DSNAME`, `SYSOUT`, `SYSPRINT`, `DISP=`.
- `sync_locks`: Matches `DISP=OLD` / `DISP=MOD` — the dispositions that request an **exclusive system ENQ** on a dataset, z/OS's native serialization idiom (`DISP=SHR` requests shared access and `DISP=NEW` allocates, so neither counts). Added by [#2733](https://github.com/squid-protocol/gitgalaxy/issues/2733). Every hit is also an `io` hit, since `io` counts the bare `DISP=` keyword; that overlap is accepted deliberately because the shape is narrow (48 hits in 15 of the corpus's 186 `.jcl`/`.prc`/`.bms` files, ~9% of its 525 `DISP=` occurrences; 429 of the remaining 477 are `DISP=SHR` and the other 48 allocate — `DISP=(NEW,…)`, or an omitted first positional that defaults to it) and the two meanings genuinely differ, the same way `COND=((4,LT),EVEN)` counts both `safety` and `safety_bypasses`. Contrast §4's `cleanup`, where the overlap would have covered essentially every disposition.

**State Mutation**
- `state_mutation`: Matches JCL symbolic variable assignments via `SET`.
Expand All @@ -44,6 +45,8 @@
- `ownership`: Matches ownership/maintainer comments like `//* Author:` (counted on the comment stream since #2610 — previously it only worked by accident on the code stream, see §10).
- `telemetry`: Matches `MSGLEVEL=` / `MSGCLASS=` (job-log verbosity and routing — JCL's observability dials). Added by #2610.
- `planned_debt` / `fragile_debt`: The shared `GLOBAL_PLANNED_DEBT` / `GLOBAL_FRAGILE_DEBT` comment-anchored patterns (same wiring as cobol) — a `//* TODO ...` / `//* HACK ...` banner in a job deck now counts. Added by #2610; structurally dead before it because JCL's comment stream was always empty (§10).
- `dead_code`: Matches a JCL statement commented out by turning `//` into `//*` (`//*STEP1 EXEC PGM=IEFBR14`), distinguished from an ordinary prose banner that merely contains the keyword. Added by [#2732](https://github.com/squid-protocol/gitgalaxy/issues/2732).
- `spec_exposure`: Matches the generic traceability tag in a `//*` comment (`//* [SPEC-4412] ...`). Added by #2732.

## 4. What GitGalaxy explicitly does not track

Expand All @@ -52,7 +55,8 @@
- `cleanup`: None — **a deliberate decision, not an oversight** (#2610): the honest JCL cleanup
idiom is `DISP=(...,DELETE)`, but `DISP=` already feeds the `io` rule, so a cleanup rule would
double-count every disposition. Recorded in the keyword-rosetta deviation ledger
(`jcl-2610-rebaseline-residual-morphology`) as intended morphology.
(`jcl-2610-rebaseline-residual-morphology`) as intended morphology. #2733 revisited the same
overlap for `sync_locks` and decided the other way — see §3 for why the two dispositions split.
- `globals`: None — JCL has no scoped-vs-global variable distinction (`SET` symbolics are already
`state_mutation`; a `JOBLIB`/`STEPLIB` rule was considered and rejected because those DD
statements would inflate `io` and `dependency_links`).
Expand All @@ -76,8 +80,10 @@ None currently. ([#2415](https://github.com/squid-protocol/gitgalaxy/issues/2415
## 6. Test depth

- **Extraction-gauntlet tests**: 42 cases in `tests/extraction/languages/test_jcl.py`
- **Strict-signature tests**: 65 cases in `tests/extraction/languages/test_jcl_strict.py`
(grew 51 → 65 with #2610's COND-partition semantics, JES3-guard, and ReDoS detonation cases)
- **Strict-signature tests**: 80 cases in `tests/extraction/languages/test_jcl_strict.py`
(51 → 65 with #2610's COND-partition semantics, JES3-guard and ReDoS detonation cases;
65 → 74 with #2732's `dead_code`/`spec_exposure` rules; 74 → 80 with #2733's `sync_locks`
disposition partition, io-overlap pin and ReDoS case)

## 7. Relevant closed work

Expand Down
30 changes: 30 additions & 0 deletions gitgalaxy/standards/language_standards/languages/jcl.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,36 @@
# both `\s+` gaps bounded to `[ \t]+` for the same cross-line reason.
"state_mutation": re.compile(r"^[ \t]*//[A-Za-z0-9_#$@]*[ \t]+SET[ \t]+[A-Za-z0-9_#$@]+=", re.M | re.I),
"concurrency": None,
# #2733: dataset disposition IS z/OS's serialization primitive.
# `DISP=OLD` and `DISP=MOD` request an exclusive system ENQ on the
# dataset; `DISP=SHR` requests shared access. In a batch shop that is
# the construct engineers reason about when two jobs contend for a
# resource -- a lock acquisition declared in the job deck -- which is
# what sync_locks measures through each language's own idiom elsewhere
# (cobol's `EXEC CICS ENQ`, abap's `ENQUEUE_`/`DEQUEUE_`, solidity's
# `nonReentrant`).
# Narrowed to OLD/MOD deliberately: SHR is the shared-access default
# every job asks for, and NEW is an allocation rather than contention
# over an already-existing resource -- neither declares a
# serialization decision. On the language-crucible corpus that is 48
# hits across 15 of 186 files, ~9% of the 525 `DISP=` occurrences; of
# the remaining 477, 429 are `DISP=SHR` and 48 allocate (`DISP=(NEW,`,
# or an omitted first positional -- `DISP=(,PASS)` -- that defaults to
# it).
# The residual overlap with `io` (which counts the bare `DISP=`
# keyword) is accepted, not avoided. #2610 rejected a `cleanup` rule on
# `DISP=(...,DELETE/CATLG)` because a second-positional disposition
# rides along on essentially every `DISP=`, so that rule would have
# re-counted the whole operand; this shape is a small, semantically
# distinct subset instead. The engine already tolerates deliberate
# overlaps where the meanings genuinely differ -- jcl's own
# `COND=((4,LT),EVEN)` counts safety AND safety_bypasses, and haskell's
# `finally` counts cleanup and safety.
# Unanchored like the other operand rules (io/safety/telemetry): a DD
# statement's DISP= routinely sits on a `//` continuation line rather
# than the line carrying the ddname, the same real-corpus shape the
# args rule's #2482 note documents.
"sync_locks": re.compile(r"\bDISP=\(?(?:OLD|MOD)\b", re.I),
"ui_framework": None,
"closures": None,
"globals": None,
Expand Down
75 changes: 75 additions & 0 deletions tests/extraction/languages/test_jcl_strict.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,11 @@
("dead_code", "//*CREL005 JOB ,,CLASS=A,MSGCLASS=H,", "//* PROC statements are documented in the runbook"),
("dead_code", "//* SET COUNTER=1", "//* JOB scheduling notes live in the runbook"),
("dead_code", "//* INCLUDE MEMBER=OLDPROC", "//* INCLUDE the operations team on any change"),
# #2733: sync_locks = the exclusive-ENQ dispositions. DISP=SHR (shared
# access, the default request) and DISP=NEW (allocation) are excluded.
("sync_locks", "//SYSLIN DD DISP=OLD,DSN=HLQ.SAMPLE.OBJ(SAM1)", "//STEPLIB DD DSN=SYS1.LINKLIB,DISP=SHR"),
("sync_locks", "//DD1 DD DSN=HLQ.CUSTRPT,DISP=(MOD,DELETE,DELETE),", "//SYSUT2 DD DISP=(NEW,CATLG),DSN=HLQ.OUT"),
("sync_locks", "//SYSLIN DD DSNAME=&&LOADSET,DISP=(OLD,DELETE)", "//S1 EXEC PGM=IEBGENER,PARM='OLDMODE'"),
# #2732: spec_exposure = the generic traceability tag, `//*`-anchored
("spec_exposure", "//* [SPEC-4412] see the change request", "//* nothing traceable here"),
("spec_exposure", "//* raised under [audit] last quarter", "//* the behaviour is [specified] upstream"),
Expand Down Expand Up @@ -363,6 +368,76 @@ def test_jcl_cond_bypass_redos_immunity():
assert_redos_immune(pattern, "//X EXEC PGM=Y,COND=(" + "A" * 100000, timeout_sec=3.0)


def test_jcl_sync_locks_only_the_exclusive_enq_dispositions():
"""
#2733: DISP=OLD/MOD request an exclusive system ENQ on the dataset; DISP=SHR
and DISP=NEW do not declare contention over an existing resource and must
stay out of the lock signal. The rule is a deliberately narrow subset of the
`DISP=` operand `io` already counts (~9% of corpus occurrences), so the
exclusions are the substance of the design -- assert them directly rather
than trusting the positive cases alone.
"""
sync_locks = JCL_RULES["sync_locks"]

for exclusive in (
"//STEPLIB DD DSN=SYS1.LINKLIB,DISP=OLD",
"//SYSLIN DD DISP=(OLD,DELETE),DSN=&&LOADSET",
"//DD1 DD DSN=HLQ.CUSTRPT,DISP=(MOD,DELETE,DELETE),",
"//SYSUT1 DD DISP=(MOD,PASS),SPACE=(CYL,(1,1))",
):
assert sync_locks.search(exclusive), f"missed an exclusive-ENQ disposition: {exclusive!r}"

for shared_or_new in (
"//STEPLIB DD DSN=SYS1.LINKLIB,DISP=SHR",
"//SYSUT2 DD DISP=(NEW,CATLG,DELETE),DSN=HLQ.OUT",
"//SYSUT3 DD DISP=(,PASS),UNIT=SYSDA",
):
assert not sync_locks.search(shared_or_new), f"counted a non-exclusive disposition: {shared_or_new!r}"

# The disposition keyword itself is required -- a bare OLD/MOD token
# elsewhere on the statement (a PARM value, a dataset name) is not a lock.
for not_a_disposition in (
"//S1 EXEC PGM=IEBGENER,PARM='OLDMODE'",
"//SYSUT1 DD DSN=HLQ.OLD.BACKUP,DISP=SHR",
"//SYSUT1 DD DISP=OLDER",
"//SYSUT1 DD DISPOSITION=OLD",
):
assert not sync_locks.search(not_a_disposition), f"false positive: {not_a_disposition!r}"

# Operand-anchored, not line-anchored: DISP= routinely sits on a `//`
# continuation line rather than the line carrying the ddname (the #2482
# shape), exactly as io/safety/telemetry already assume.
continuation = "//SYSUT1 DD DSN=HLQ.WORK,\n// DISP=(MOD,DELETE,DELETE),\n// UNIT=SYSDA"
assert sync_locks.search(continuation)


def test_jcl_sync_locks_overlaps_io_by_design():
"""
#2733: every sync_locks hit is also an `io` hit, because `io` counts the
bare `DISP=` keyword. That overlap is the design decision the issue records
(accepted for a narrow OLD/MOD subset, unlike the broad `cleanup` rule
#2610 rejected), so pin it as intended behaviour -- if a later change makes
the two rules disjoint, that is a decision to re-make, not a silent drift.
"""
sync_locks = JCL_RULES["sync_locks"]
io = JCL_RULES["io"]

exclusive = "//SYSLIN DD DISP=(OLD,DELETE),DSN=&&LOADSET"
assert sync_locks.search(exclusive) and io.search(exclusive)

# ...but the converse does not hold: the overwhelming majority of DISP=
# occurrences (427 of 525 in the corpus) are DISP=SHR, io-only.
shared = "//STEPLIB DD DSN=SYS1.LINKLIB,DISP=SHR"
assert io.search(shared) and not sync_locks.search(shared)


def test_jcl_sync_locks_redos_immunity():
"""#2733: no nesting and no unbounded repetition, but hold the line on it."""
pattern = JCL_RULES["sync_locks"]
assert_redos_immune(pattern, "//X DD " + "DISP=(" * 20000, timeout_sec=3.0)
assert_redos_immune(pattern, "//X DD DISP=(" + "OLD" * 50000, timeout_sec=3.0)


def test_jcl_dead_code_counts_through_the_real_comment_stream():
"""
#2732: end-to-end proof that jcl's two new comment-stream rules actually
Expand Down
Loading
Loading