Skip to content

Add contract style-lint cleaning pass - #46

Merged
mcdigman merged 11 commits into
contracts/cleaning-phasefrom
codex/add-contract-style-lint
Jun 30, 2026
Merged

mcdigman merged 11 commits into
contracts/cleaning-phasefrom
codex/add-contract-style-lint

Conversation

@mcdigman

Copy link
Copy Markdown
Owner

Summary

  • add a read-only contract-style-lint pass with deterministic candidate scan support
  • add the style_lint classification to shared conventions and handoff schema
  • wire contract-clean-consolidation/cleaner to accept and report style-lint findings, mirrored to skill copies

Verification

  • python3 -m json.tool .claude/agent-shared/handoff-report.schema.json
  • sh -n .claude/tools/contract_style_lint_scan.sh
  • PATH=/usr/bin:/bin sh .claude/tools/contract_style_lint_scan.sh .claude/agents/contract-style-lint.md 0
  • diff -q agent/skill mirrors for contract-style-lint, contract-clean-consolidation, and contract-cleaner
  • git diff --cached --check before commit

mcdigman and others added 2 commits June 29, 2026 17:25
The script ships a #!/bin/sh shebang but was tracked as 100644, tripping
the check-shebang-scripts-are-executable pre-commit hook in CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mcdigman

Copy link
Copy Markdown
Owner Author

Adversarial review: contract_style_lint_scan.proposed.sh regex patterns

Review target: the PATTERNS heredoc in .claude/tools/contract_style_lint_scan.proposed.sh (the scanner the contract-style-lint.proposed-v3 agent invokes). Every HIT/MISS below was executed against the real scanner, on synthetic lines modeled on ..._amendment_1.md, ..._final.md, and the three CHIRP contracts.

Two headline facts

  • Precision: 14 of 41 candidate lines on the real final contract come from the two (all|any|…)-near-proxy patterns. Worse, all/any have no word boundaries, so they also fire as substrings of shall, small, finally, many, manually, originally. All six benign control sentences I tested (B01–B06) falsely matched. This is the dominant noise source and is fixable with zero recall loss — and the scanner's own recall-gaps.md warns that a noisy candidate list causes models to anchor to it and drop real findings.
  • Recall: the author merged but heavily narrowed the G1–G7 recall-gaps.md suggestions (dropped general replaces/supersedes/overrides/Amendment N/PR #N/(value|point|goal) is the; kept only literal supersedes the Amendment, review of PR #N, value is…not (raw|mere|merely|just)). That reopened whole classes — authority-shaped prose, general supersession, evaluative-method siblings — with no covering pattern at all.

Prioritized hardening table (24 examples)

Priority: P0 = trivial fix, large gain, ~no recall loss · P1 = confirmed gap/noise, small fix · P2 = real but lower-leverage · P3 = note, fix optional/hard.

# Pri Type Synthetic line Scanner today Root cause Minimal fix
1 P0 Precision The helper shall reproduce the oracle drop behavior. / A small reference table… / …finally compared against the oracle… / …many oracle guards… False HIT (6/6) (always|never|all|any) has no word boundary → matches inside sh**all**, sm**all**, fin**ally**, m**any** \b(always|never|all|any)\b (rg + ugrep here support \b; BSD grep would need [[:<:]]/[[:>:]])
2 P0 Precision All correctness comparisons must … match the oracle. (real lines 170/172/205/294/349) False HIT (14/41) all/any are real here but "match the oracle over all cells" is the carved-out near-exact-oracle requirement, not an absolute over a proxy Drop all|any from both proxy patterns; keep always|never|guarantees?|proves?
3 P0 Precision ULP boundaries make the keep-or-drop decision fragile. False HIT bare fragile/unnecessary/overkill fire even when not attached to a verification method (fragile|unnecessary|overkill)[[:print:]]{0,40}(inspection|review|test|check|comparison|method)
4 P1 Recall Under the default configuration this overflow bound is comfortably met. MISS G1 over-tightened to (config…)(satisfies|passes|meets|holds)[[:space:]]+(this|the|exactly); "…is comfortably met." lacks the trailing anchor and "the repository" prefix (config|configuration)[[:print:]]{0,40}(satisfied|met|passes|holds|comfortably)
5 P1 Recall The Nsf=150 configuration satisfies all stated tolerances. HIT — only incidentally via config…all If #2's fix lands (drop all), this reassurance line becomes a MISS Land #4 alongside #2 so reassurance isn't lost
6 P1 Recall The maintainer decided the linear-boundary path is the accepted approach. MISS No authority-shaped-prose pattern at all (maintainer|author|we)[[:space:]]+(decided|chose|intentionally), the[[:space:]]+accepted[[:space:]]+approach
7 P1 Recall We intentionally avoid recomputing the stride inside the loop. MISS we intentionally avoid uncovered (we|intentionally)[[:space:]]+(intentionally|avoid|chose|prefer)
8 P1 Recall Human decision supplied after the second review: the threshold stays at 0.5. (real CHIRP wording) MISS No human decision pattern human[[:space:]]+decision
9 P1 Recall Source inspection of the compiled kernel is insufficient for acceptance. MISS Evaluative-method vocab stops at fragile/unnecessary/overkill; insufficient/sufficient/adequate uncovered \b(insufficient|inadequate)\b, \bis[[:space:]]+sufficient\b
10 P1 Recall A bare smoke test is sufficient to demonstrate compliance. MISS reassuring direction of #9 \bis[[:space:]]+sufficient\b, \bsuffices?\b
11 P1 Recall This paragraph overrides the Amendment 1 parity recipe. MISS Only literal supersedes the Amendment exists; general supersession verbs + Amendment [0-9] dropped (supersedes|superseded|replaces|overrides|carried[[:space:]]+(from|forward))[[:space:]]+the, Amendment[[:space:]]+[0-9]
12 P1 Recall PR #40 surfaced two contract defects. MISS Narrowed to review of PR; bare PR #N and surfaced dropped PR[[:space:]]*#[0-9]+, \bsurfaced\b
13 P1 Recall Where the literal text conflicts, the design intent controls. MISS Spirit-over-letter coverage is only is normative/in favor of the letter/takes precedence/noncompliant by definition; "intent controls/governs/prevails" uncovered (high-leverage) intent[[:space:]]+(controls|governs|prevails|overrides)
14 P1 Recall The implementation should be the smallest clear change and must remain divergence-free. MISS No must+should modal-conflict pattern should[[:print:]]{0,60}\bmust\b, \bmust\b[[:print:]]{0,60}should
15 P2 Recall The point of this exemption is the lift-out structure, not the raw CPU speed. MISS value is…not (raw…) requires literal "value is"; point/goal/purpose uncovered (value|point|goal|purpose)[[:space:]]+is[[:space:]][[:print:]]{0,80}not[[:space:]]+(raw|mere|merely|just)
16 P2 Recall Reviewers may rely on the benchmark, though it is not authoritative. MISS exclude-direction covered; permit-but-discount shape uncovered not[[:space:]]+authoritative, may[[:space:]]+rely[[:space:]]+on
17 P2 Recall This assertion can be trusted because the ratio is fixed. MISS reassurance via can be trusted/known to uncovered can[[:space:]]+be[[:space:]]+trusted, known[[:space:]]+to[[:space:]]+(keep|hold|be)
18 P2 Recall We currently assume the developer environment can run Python 3.14. MISS assume/may assume (trust about current state) uncovered (may|currently)[[:space:]]+assumes?
19 P2 Recall This requirement exists to prevent a recurrence of the CD-A failure. MISS exists because covered, exists to not; recurrence of the same defect class over-literal exists[[:space:]]+to; broaden to recurrence[[:space:]]+of[[:space:]]+the
20 P2 Recall Historically, the aligned path mirrored the fastmath expression. MISS historically dropped \bhistorically\b
21 P3 Precision The bandwidth guard takes priority over the table-overflow guard. HIT takes? priority fires on a legitimate ordering requirement Optional: takes?[[:space:]]+priority[[:space:]]+over[[:space:]]+(clarity|speed|correctness)
22 P3 Precision ULP-boundary handling is permitted but not required. (real L82) HIT bare not required is the canonical legitimate permitted-alternative; 5 such hits on final No safe tightening — document as expected benign
23 P3 Precision Accumulate in place, not merely into oracle_slice. correctly MISS confirms value is…not (merely) scoping closes the recall-gaps L581 concern none — keep scoped if #15 generalizes
24 P3 Recall Cells expected to be exactly zero are skipped. (real L312) correctly MISS confirms G2 qualifier-list correctly excludes the technical FP none — keep exactly out of the G2 list

Script-level robustness (non-regex)

# Sev Issue Why it matters Fix
S1 High Line-based only; no multiline/wrap matching. recall-gaps.md itself notes the most consequential miss (Nsf=150 … satisfies this / exactly) wraps across lines. Any wrapped target phrase is structurally unmatchable; recall is capped by line-wrapping, and "no candidates" reads as "no issue." De-wrap a working copy before scanning, or state the limitation in the agent prompt.
S2 High No-word-boundary alternations (finding #1). Largest precision sink → floods the list → models anchor and drop findings. Word-boundary the short tokens.
S3 Med Empty-pattern catastrophe. A blank heredoc line = regex matching every line. No guard. Silent flip to "flag everything." grep -v '^[[:space:]]*$' the pattern file before use.
S4 Med Copy-paste footgun in recall-gaps.md. Its append block has # G1 … comment lines; neither rg --file nor grep -f treats # as a comment. Pasting verbatim inserts ~7 inert patterns that never fire — false confidence the gaps are closed. Strip # lines from the block, or skip ^# in the script.
S5 Low Dual-engine parity assumed, not tested (rg preferred, grep -Ein -f fallback). They differ on \b portability, Unicode case-fold, {n,m}. A fix validated under rg may behave differently under grep. Pick one engine or add a parity self-test.

Recommended minimal patch set (highest ROI first)

  1. S2/some partial doc strings for instrument_noise.py #1 — word-boundary (always|never|all|any). One edit, kills the biggest FP class, zero recall loss.
  2. Put all configuration parameters in a toml file #2 — drop all|any from both proxy patterns; pair with Make the iterative loops and plotting into functions #4/Create unit tests #5 G1 broadening so config-reassurance recall survives.
  3. Structure as package and make installable #6Improve HDF5 file structure and noise realization storage #9, Validate the instrument noise model and result fits #11Handle NC differently #13 — re-add the dropped recall classes (authority prose, general supersession/Amendment N/bare PR #N, insufficient/sufficient, intent-controls).
  4. S3, S4 — two one-line hygiene guards.

Open question (genuine recall-vs-noise tradeoff)

The narrowing of replaces/Amendment N/PR #N to literal-only forms looks deliberate — likely to suppress the Finding-Disposition-Ledger / Traceability-table noise the recall-gaps.md warns about (its lines 148–167). Findings #11#12 reverse that. Two options: (a) re-add the general forms (max recall, more ledger noise), or (b) keep them narrow and instead exclude the ledger/traceability table ranges from the scan (recover prose recall without the table noise). Recommend deciding before freezing the pattern set.

🤖 Adversarial review generated with Claude Code (Opus 4.8). Synthetic examples are illustrative; HIT/MISS verified by execution against the scanner at this branch.

@mcdigman

Copy link
Copy Markdown
Owner Author

Adversarial review findings for .claude/tools/contract_style_lint_scan.sh, using synthetic examples mined from PR #42's implementation contract history plus CHIRP contract/correction-plan patterns. I did not use PR comments/body/commit notes for this analysis.

The current scanner has a broad recall gap around permissive modal/waiver language and weak-evidence acceptance language. A fresh synthetic run based on the mined contracts only flagged 3 of 34 source-derived lines; most of the dangerous cases below were missed.

Pri Type Synthetic contract text Current result / analysis Regression risk if hardened
1 Recall “The Level 4 xfail counts as passing Stage 1 acceptance when the reason names the missing fitter.” Missed. Mirrors CHIRP’s pre-authorized xfail rules, but flips them into acceptance. Broad xfail matching would flag legitimate “must be xfail and not counted as passing” clauses.
2 Recall “A stage may be marked met with skipped tests if the implementation summary names the skipped tests.” Missed. Undermines stage-status acceptance. Broad skip matching would hit legitimate “slow/reference-WAV tests may skip under existing markers.”
3 Recall “The independent oracle may call the production helper when the helper is small and readable.” Missed. Directly defeats independent-oracle requirements. Low if scoped to oracle ... call production/helper; avoid broad may call.
4 Recall “Bare np.interp may be used as the oracle because its default edge behavior is close enough.” Missed. CHIRP history specifically hardened against default-edge oracle drift. Low if scoped to oracle/default-edge phrases.
5 Recall “The timing-quality smoke test may assert only shape and non-crashing behavior.” Missed. Matches a real class of rejected weak smoke coverage. Low if scoped to only shape, non-crashing, does not raise near tests.
6 Recall “Initialized zero rows count as finite non-reference timing-quality evidence.” Missed. CHIRP explicitly rejects zero-only evidence. Low; terms are specific enough.
7 Recall “If full-tree checks fail for pre-existing reasons, the implementation may add warning filters until CI is green.” Missed. Turns baseline/no-regression handling into suppression permission. Medium; may add warning filters should be suspicious only near failing checks/CI.
8 Recall “The implementation may catch TypeError in the pipeline caller instead of fixing phase_difference.” Missed. Hides root cause by caller exception handling. Low if pattern looks for catch/ignore plus exception plus instead of fixing.
9 Recall “Threshold equality at 0.5 may be treated as included if that is simpler for interpolation.” Missed. Directly violates strict-threshold acceptance. Medium; numeric threshold language is domain-specific, so generic matching may be noisy.
10 Recall “Point-source noise may be rescaled independently per microphone after propagation to hit target SNR exactly.” Missed. Violates propagated 1/r amplitude semantics. Medium; may be rescaled can be legitimate outside propagation/SNR contexts.
11 Recall “Rigid alignment may remove scale and shear before comparing recovered geometry.” Missed. Real CHIRP history hardened against scale/shear hiding. Low if scoped to alignment plus scale/shear/reflection.
12 Recall “Survivor-only TOA pairs are sufficient for n_total_toas and peak_correlation_summary.” Missed. Defeats rejected-attempt accounting. Low if scoped to survivor-only metrics/evidence.
13 Recall PowerLawNoiseConfig.rms_amplitude may default to 1.0 when omitted.” Missed. Reintroduces removed silent default. Medium; default-value allowances can be legitimate, but silent defaults in required config deserve review.
14 Recall “Vibration covariance tests may assert only shape, symmetry, and positive semidefiniteness.” Missed. Mirrors real “shape/SPD-only permits constant covariance” failure. Low if scoped to only shape/only type/only SPD.
15 Recall “Metrics can fill peak_correlation_summary from a toy fixture.” Missed. Violates same-generated-audio metric accounting. Low if scoped to toy/separate fixture near metrics/evidence.
16 Recall “If IR extraction is unavailable, the divergence-free test may pass based on source inspection alone.” Missed. Contradicts PR #42’s pending-human-review fallback. Medium; source-only is sometimes valid where IR is not required.
17 Recall “Final native assembly inspection is required for acceptance in addition to LLVM IR.” Missed. Over-hardens beyond PR #42’s explicit exclusion. Medium; “required final assembly/PTX/object code” should be suspicious in this contract family.
18 Recall “Required tests need not cover read-only arrays if the public docstring says behavior is undefined.” Missed. Could be legitimate if scoped, dangerous if applied to supported public inputs. High: PR #42 has legitimate need not cover exclusions for unsupported/undefined inputs.
19 Precision “Local annotations are allowed when they improve readability, but they are not required for every local intermediate.” Flagged by not required. This is legitimate narrowing from CHIRP review. Tightening not required too much could miss real waivers; context matters.
20 Precision “LLVM IR is required, but final machine-dependent compiled output is not required.” Flagged by not required. This is a precise scope boundary from PR #42. Any not required broad match will keep producing useful-but-noisy candidates.
21 Precision “This PR does not need to add PLR0133 to repository-wide lint configuration.” Missed in my synthetic run because phrase used does not need, but likely a broad need not hardening would flag it. Legitimate human-scoped deferral. High for adding need not/does not need naively.
22 Precision “Tests using realistic sample rates may compare to the independent oracle with a stated floating tolerance.” Missed, but broad may compare/may use hardening would overflag. Legitimate test-method allowance. High if hardening targets generic may.
23 Precision “Optional moving-source coverage remains optional and should use the existing waypoint-source simulator if implemented.” Missed, but broad optional matching would overflag legitimate optional scope. High for optional without nearby acceptance/test-waiver danger terms.
24 Precision “The implementation may be vectorized or scalar, but the finite-support oracle must be reviewable from code.” Missed, but broad may be or modal matching would hit normal implementation freedom. High; modal verbs are too common in contracts.

Main hardening implication: the tempting recall fixes are standalone need not, optional, may, xfail, skip, and not required, but real contracts use those phrases for valid scoping, unsupported-domain boundaries, and human-approved deferrals. Safer improvements should use phrase-pair/context matching, e.g. suspicious waiver term plus acceptance/test/evidence/oracle/CI/root-cause terms, rather than standalone modal words.

@mcdigman

Copy link
Copy Markdown
Owner Author

Adding the original independent synthetic findings from before mining PR #42/CHIRP examples. These were generated from the PR #46 scanner behavior alone, before looking at real contract examples.

Legend: Recall miss = current script emits no candidate. Precision hit = current script emits a candidate, but the statement is plausibly legitimate contract text.

Pri Type Synthetic contract text Current behavior / why it fails
1 Recall miss “An implementation need not add tests when the change is judged obvious by the reviewer.” Misses need not except in reviewers need not; this is a direct test-obligation waiver.
2 Recall miss “Reviewers may approve based solely on the summary without reading the patch.” Misses without reading; current verb list has inspecting/examining/reviewing/checking/relying.
3 Recall miss “The API may omit schema validation for trusted internal callers.” Misses may omit; serious API/security requirement weakening.
4 Recall miss “Tests may be skipped if they are slow, flaky, or expensive to run.” Misses may be skipped; direct test-obligation escape hatch.
5 Recall miss “Manual inspection is optional for security-sensitive changes.” Misses optional; weakens review/acceptance criteria.
6 Recall miss “The checker should warn, not fail, when an implementation silently drops unknown fields.” Misses warn, not fail; turns a behavioral contract into advisory output.
7 Recall miss “Acceptance is best-effort; failing edge cases should be handled in a follow-up.” Misses best-effort and follow-up deferral language.
8 Recall miss “It is acceptable to rely on current fixtures instead of constructing an independent oracle.” Misses permissive reliance on current fixtures; close to existing oracle/current-config concerns.
9 Recall miss “The reviewer can trust the generated report as authoritative evidence.” Current pattern catches can be trusted, not active-voice can trust.
10 Recall miss “No additional evidence is necessary beyond the author saying it works.” Misses evidence is necessary / author-attestation-only acceptance.
11 Recall miss “The contract is satisfied when the current happy-path example passes.” Misses current-state/happy-path sufficiency because patterns center on config/configuration.
12 Recall miss “The default settings are deemed compliant for this contract.” Misses default settings / deemed compliant; similar risk to default config satisfies.
13 Recall miss “Any behavior not covered by this document is left to author discretion.” Misses author discretion as a conflict-resolution/escape clause.
14 Recall miss “The implementation is exempt from validation when input is marked internal.” Misses exempt from; weakens implementation/API validation obligations.
15 Recall miss “If requirements conflict, prefer the author’s intent over the written rule.” Misses this form of intent-over-text precedence.
16 Recall miss “A regression is acceptable if it matches historical behavior.” Misses historical; script only catches historically.
17 Recall miss “The API may return partial results without documenting which fields were omitted.” Misses without documenting; important API behavior ambiguity.
18 Recall miss “Coverage is optional when the patch only changes plumbing.” Misses coverage-optional test waiver.
19 Recall miss “A missing assertion is acceptable as long as the reviewer believes the branch cannot happen.” Misses assertion waiver based on reviewer belief.
20 Recall miss “The contract checker is advisory and may not block a release.” Misses advisory/non-blocking enforcement language.
21 Other miss “The test must explicitly\nsurface the compatibility fallback.” Line wrapping breaks phrase patterns; realistic markdown wrapping can evade must ... surface.
22 Precision hit “This optional field is not required when the client sends mode=legacy.” Flags not required, but this is normal API optionality.
23 Precision hit “The migration exists because older deployments lack the new index.” Flags broad exists because; ordinary rationale text.
24 Precision hit “The method should reject malformed input and must return a 400 response.” Flags should ... must; legitimate layered API obligations.
25 Precision hit “The implementation must persist the setting and should emit a deprecation warning.” Flags must ... should; normal required plus recommended behavior.
26 Precision hit “The test must explicitly surface the original exception message to callers.” Flags must explicitly surface; this can be a concrete API/test requirement.
27 Precision hit “The config satisfies this predicate only after the API normalizes default values.” Flags config satisfies this; legitimate predicate/validator language.
28 Precision hit “Without checking the cache, the handler must recompute permissions from storage.” Flags without checking; this is implementation behavior, not review avoidance.
29 Precision hit “No requirement may be interpreted to weaken input validation.” Flags no requirement may; actually strengthens the contract.
30 Precision hit “The repository currently exposes this API behind a feature flag for staging.” Flags repository-current-state wording; may be harmless scope/context.

Main adversarial pattern from this first pass: recall gaps were mostly permissive modal phrases (need not, may omit, may be skipped, optional, exempt, advisory) and reviewer-authority substitutions. Precision gaps were mostly broad phrase triggers that do not distinguish suspicious meta-contract language from ordinary implementation/API requirements.

@mcdigman

Copy link
Copy Markdown
Owner Author

Adversarial review — contract_style_lint_scan.sh regex / recall / precision

Method. Reviewed only the committed scanner (.claude/tools/contract_style_lint_scan.sh, 83 active patterns). Ran it against the real PR #42 contract (.contracts/implementation_contract_taylor_time_wavelet_optimized.md, worktree at 3364c2b); harvested vocabulary from CHIRP/contracts/*.md and the read-only correction plan. Did not read the proposed agent prompt, *.recall-gaps.md, PR comments, or commit notes. Every miss / false-positive below is reproduced against real text or validated with a synthetic probe through the actual rg path. Priority = expected authority-leak harm × likelihood of the phrasing in a real markdown contract.

Two empirical anchors:

  • The scanner is line-oriented; every pattern uses [[:space:]]+/[[:print:]]{0,N}, which cannot cross a newline or a markdown emphasis/code marker (*, _, `). Confirmed: The reviewer **need not** inspect… is missed; the unbolded twin is caught.
  • Patterns 48–49 (always|never proximity to technical nouns) scored 0 useful / 3 fires on the real contract — all three (L168 "drop always comes from the mask, never from padding", L417 "never form (DF/df_bw)*k", L491 "always in bounds … on GPU") are pure numerics, not steering.

Table 1 — Recall failures (statements the scanner misses)

# Pri Synthetic steering/authority text Target Why missed
R1 P1 The reviewer **need not** inspect the lowered assembly. / *not* required Reviewer-relief / waiver Emphasis marker between words breaks [[:space:]]+. Systemic across ~all multiword patterns
R2 P1 The maintainer's decision is binding and trumps the letter. / controlling and dispositive Authority-elevation Vocab gap: only supersedes/overrides/precedence/priority/governs(after "intent")/controls; binding/trumps/dispositive/controlling/final absent
R3 P1 Escalate uncertainty to the human maintainer. / route to the human; defer to maintainer judgment Escalation authority Only for human review/human decision/must…surface; escalate/route/defer/"pending human review" absent (real doc: "escalated to human", "routes to the human maintainer")
R4 P1 This contract supersedes and replaces the prior amendment. / Superseded and further tightened Override provenance (supersedes|…)[[:space:]]+the needs literal the; conjunction split defeats it. Real in both repos (GS L16, CHIRP L423)
R5 P1 not strictly required / not necessarily prohibited / by no means required Hedged waiver not[[:space:]]+required is contiguous; intervening adverb breaks it
R6 P1 the highest-authority inputs, above Design Intent; the decision here governs Section self-elevation (real L59–62, biggest authority move) No pattern for highest-authority, above…Design Intent, or bare X governs (governs only after literal "intent")
R7 P2 Reviewers may treat the author's number as sufficient. / can safely skip / is not expected to Reviewer-relief, other modal reviewers?[[:space:]]+(need not|must not|should not) only
R8 P2 The existing pipeline already complies. / out of the box this passes "Current state passes" satisfies-verb list lacks compl(y/ies/iant); no already (passes|meets|complies)
R9 P2 (wrap) …the reviewerneed not inspect… Same as R1/R4, split at wrap Line-oriented. Nuance: partly mitigated when a bare-noun pattern co-locates; true miss otherwise
R10 P2 A skipped test still satisfies the coverage requirement. / marking it xfail is acceptable QA-weakening (xfail|skipped tests)…(counts as passing|marked met|acceptance) too tight
R11 P2 Take the implementer's word; the handoff is sufficient evidence. / at face value Evidence-substitution can be trusted/can trust…(authoritative|evidence|report) only
R12 P3 has normative force and is binding guidance Normativity inflation Only literal is normative (overlaps R2)
R13 P3 meets every requirement and passes all gates Blanket-compliance satisfies all requirements exact only
R14 P3 added in response to PR #40; stems from the same defect class Provenance-as-authority exists because/exists to…(recurrence|…) only
R15 P3 close enough for acceptance / good enough for now Tolerance-relaxation oracle…default…(close enough|sufficient) requires "oracle"
R16 P3 passive must be surfaced / should be raised / has to be flagged Passive escalation must[[:space:]]+(explicitly[[:space:]]+)?surface active-voice only

Table 2 — Precision failures (benign text flagged; small change ≫ recovers precision)

# Pri Benign text that fires Pattern Why FP
P-1 P1 L168 drop always comes from the …mask, never from…; L491 always in bounds…on GPU; L417 never form (DF/df_bw)*k 48–49 Numerics say always/never near oracle/gpu/numba/config constantly. 0/3 real hits were steering
P-2 P1 the aligned table holds R and Nfd / the configuration that satisfies the precondition 55 Unqualified superset of careful 53/54; fires on neutral "config…holds/satisfies"
P-3 P1 any line where should and must co-occur within 60 chars 82–83 Requirement prose pairs them routinely; near-pure noise
P-4 P2 historically the kernel used a Python loop 93 bare historically Fires on any narrative use
P-5 P2 the array can be trusted to be C-contiguous 86 can be trusted Benign technical guarantee
P-6 P2 timing reported on a best-effort basis 109 bare best-effort Common benign qualifier
P-7 P3 refactor without documenting every intermediate 112 bare without documenting Fires with no waiver context
P-8 P3 the current configuration object 50 bare current (config|…) Broad net; flags benign references

Concrete proposed edits (before → after)

# R1/R9 — emphasis + wrap (closes a whole CLASS). Preprocess before grep:
#   sed -e 's/[*_`]//g' "$contract_file"  | (optional) soft-join wrapped lines
#   Scan the stripped copy but keep --line-number against original (markers add no lines),
#   so reported line numbers stay correct; only highlighted span text differs.
#   Lowest-risk variant: strip [*_`] only (closes R1, most of R5/R6); defer line-joining (R9).

# R4 — drop trailing-"the" (authority verbs are self-identifying)
- (supersedes|superseded|overrides|carried[[:space:]]+(from|forward))[[:space:]]+the
+ (supersed(es|ed)|overrid(es|den)|carried[[:space:]]+(from|forward))\b

# R2/R6/R12 — authority-elevation vocabulary
+ (is|are|shall[[:space:]]+be|remains?)[[:space:]]+(binding|dispositive|controlling|authoritative|final|normative)
+ (trumps?|outranks?|prevails?|highest[- ]authority|above[[:space:]]+(the[[:space:]]+)?Design[[:space:]]+Intent|source[[:space:]]+of[[:space:]]+truth)

# R3/R16 — escalation/routing
+ (escalat\w+|rout(e|ed|es)|defer\w*|flag\w*|surfac\w+|rais\w+)[[:print:]]{0,30}(human|maintainer|reviewer)
+ pending[[:space:]]+human

# R5 — allow intervening adverb
- not[[:space:]]+required
+ not[[:space:]]+((strictly|necessarily|generally|always|by[[:space:]]+itself)[[:space:]]+)?required

# R7 — broaden reviewer modal
- reviewers?[[:space:]]+(need[[:space:]]+not|must[[:space:]]+not|should[[:space:]]+not)
+ reviewers?[[:space:]]+(need[[:space:]]+not|must[[:space:]]+not|should[[:space:]]+not|may|can|should[[:space:]]+accept|(is|are)[[:space:]]+not[[:space:]]+expected)

# R8 — extend "current state passes" verbs
- (config|configuration)[[:space:]][[:print:]]{0,40}(satisfies|passes|meets|holds)[[:space:]]+(this|the|exactly)
+ (config|configuration|defaults?|pipeline)[[:space:]][[:print:]]{0,40}(satisfies|passes|meets|holds|compl(y|ies))[[:space:]]+(this|the|all|exactly)
+ already[[:space:]]+(passes|meets|complies|satisfies)

# P-1 — narrow always/never to an epistemic claim (kills the 0/3 false hits)
- (^|[^[:alnum:]_])(always|never)[^[:alnum:]_][[:print:]]{0,80}(oracle|reference|existing|prior|current|config|configuration|toolchain|numba|llvm|gpu|reviewer|test|assertion|requirement)
+ (^|[^[:alnum:]_])(always|never)[[:space:]]+[[:print:]]{0,30}(pass\w*|fail\w*|hold\w*|satisf\w*|compl\w*|correct|guaranteed|suffic\w*|wrong)
  # line-49's `guarantees?|proves?` variant already covers "the oracle guarantees…"

# P-2 — delete the over-broad superset (53/54 retain real recall)
- (config|configuration)[[:print:]]{0,80}(satisfies|satisfied|met|passes|holds|comfortably)

# P-3 — require adversative coupling, not mere co-occurrence
- should[[:print:]]{0,60}must
- must[[:print:]]{0,60}should
+ should[[:print:]]{0,30}(but|yet|however|,)[[:print:]]{0,10}must

# P-4/P-6 — anchor bare words to a claim/waiver context
- historically
+ historically[[:print:]]{0,40}(pass\w*|work\w*|compl\w*|fine|never|always|green|accepted)
- best-effort
+ best-effort[[:print:]]{0,40}(accept\w*|suffic\w*|is[[:space:]]+(fine|enough|ok)|count)

# P-5 — require evidence-bearing subject
- can[[:space:]]+be[[:space:]]+trusted
+ (author|implementer|report|handoff|claim|evidence|number)[[:print:]]{0,20}can[[:space:]]+be[[:space:]]+trusted

Regression analysis

  • Emphasis-strip preprocess (R1): only realistic new FP is exposing inline-code content (`not_required`not_required; no space → still won't match). Closes a large recall class at near-zero precision cost. Keep --line-number against the original to preserve line numbers; only highlighted span text changes.
  • R4 (supersed…\b): could fire on benign "this test supersedes the old assertion's role" — rare, still a provenance signal. Low risk.
  • P-1 (narrow always/never): "the oracle is never authoritative" is already caught by 84/85. No real recall loss.
  • P-2 (delete 55): 53 needs object this/the/exactly; 54 needs a default/current qualifier — the real steering form still matches. Pure FP removal.
  • P-3 (should/must coupling): may miss a strength-upgrade where the modals are far apart on a line; acceptable, real smell is adjacency.
  • R2/R3 new patterns: "authoritative"/"binding"/"nonbinding" appear benignly as disclaimers in CHIRP ("no authoritative threshold was supplied"). They will fire on those — acceptable for a candidate-span tool, but raises the agent's classification burden (see C below).

Architecture / prompt-level adversarial modes (categories the "nearly-mechanical" design cannot reach)

  • A — Section-level authority architecture is invisible. The biggest move in the real contract — a new top-level "Authoritative Human Decisions" section declaring itself "highest-authority … above Design Intent" (L57–62) — is caught only as a stray is normative and the HD-1 table cell, not as self-elevation. Phrase-keyed scanning under-weights authority established by structure across sentences.
  • B — Paraphrase escape is intrinsic to a vocabulary list. "Nearly mechanical" flagging means a drafter steers simply by avoiding listed tokens ("dispositive" vs "takes precedence"). The mechanical focus is itself the vulnerability; the prompt should instruct the agent to generalize a hit to its synonym class.
  • C — Polarity blindness couples precision and recall. Regex can't separate claiming authority ("X is authoritative") from disclaiming it ("no authoritative threshold was supplied" — pervasive in CHIRP). Needs an explicit claim-vs-disclaimer rubric in the prompt, not just span emission.
  • D — Code/emphasis tokens both hide and fabricate matches. Inline code and `*/_/`` markers cause FPs and conceal real text; behavior becomes a function of formatting, so a stable adversary can bold a phrase to evade detection (unless R1 normalization lands).
  • E — Cross-line/cross-cell cumulative steering. Line-oriented matching misses wraps (R9) and steering that accrues across table rows or a paragraph; the review unit for authority-leak is the passage, not the line.

Top-priority shortlist: R1 (emphasis/wrap normalization) + P-1/P-2/P-3 (the three lowest-precision patterns) give the best harm-reduction per line of diff; R2/R3/R4/R5 close the most common authority/escalation/waiver vocabulary holes with minimal regression.

@mcdigman

Copy link
Copy Markdown
Owner Author

Addendum — findings from historical commit diffs of the reviewed .md files

Followed up by mining the tracked git history of .contracts/implementation_contract_taylor_time_wavelet_optimized.md (v1 → v2 → v3 → v4 → final → v6 consolidated c06116c → over-steered reviser run 9120def → PR #42 revision 3364c2b) plus the CHIRP contracts. Method: collected every removed/added prose line across the full history (611 distinct removed lines; 248 added in the over-steered draft) and replayed them through the actual scanner. Did not modify any untracked file. All catch/miss results below are from a direct scanner probe, not the earlier corpus tagging.

Headline: the over-steered draft 9120def was caught only 12 / 248 added lines — but its additions were mostly technical (ULP-exemption requirements), so that ratio understates the scanner on real steering. The multi-version removals are the richer vein: they are text the cleaning passes already deleted, i.e. exactly what a cheap preliminary pass should have flagged. One new category and several real exemplars surfaced.

New finding from history

# Sev Real removed text (verbatim, from history) Class Scanner
H1 P1 (new category) Any distorted-table variant is clearly benchmark-only… · The benchmark and tests must clearly identify that this is a distorted-table experimental path… · …allowed if clearly named and isolated… · Clearly distinguish the faithful tolerance regime… (≥5 distinct removed lines) Confidence / over-read adverbsclearly, obviously, self-evident(ly), plainly, trivially, simply, of course, needless to say. These assert a conclusion is beyond scrutiny → over-read as authority. MISS — no pattern exists. Repeatedly removed by cleaning, so a real target the cheap pass should own. Perfect fit for the "nearly-mechanical" fixed-word-list design

Proposed pattern (new line):

(^|[^[:alnum:]_])(clearly|obviously|self-?evident(ly)?|plainly|trivially|needless[[:space:]]+to[[:space:]]+say|of[[:space:]]+course)([^[:alnum:]_]|$)

Regression: simply is too polysemous to add bare (matches "simply add"); recommend leaving simply out or anchoring it (simply[[:print:]]{0,20}(works|passes|correct|enough)). Low FP risk for the rest in a requirements doc.

Real history exemplars that reinforce findings already in the main review

# Real removed/added text (verbatim) Reinforces Scanner
H2 The functions are **not** required to cover such inputs. · …the exempt set is **not** the entire comparison domain… R1 (emphasis breaks [[:space:]]+) — now with real in-tree examples, not just synthetic MISS (bolded twin of a caught phrase)
H3 The oracle and tests need not cover such inputs. New / R7-adjacent: generic need not (tests/functions/oracle need not) is only caught when preceded by reviewer. A bare-need not waiver pattern is warranted MISS
H4 …Accepted and resolved with revised authority, superseded in v4. · TR38 … superseded in v4 by the consolidated requirement. R4 (supersed… only matches when followed by literal the); superseded in v4 / revised authority forms slip through MISS
H5 An implementation satisfies this contract only if a reviewer can verify that: · A reviewer must be able to verify that diffs are limited to: · The reviewer must re-measure this ratio… R7 extension: reviewability-as-compliance framing (reviewer must/can/be able to <verb>), not just the relief modals (reviewer need not/must not/should not) the scanner covers MISS

Proposed deltas implied by H3/H4/H5 (small, low-regression):

# H3 — generic need-not waiver (one new pattern)
+ \bneed[[:space:]]+not\b

# H4 — already in main review (R4): replace trailing-"the" requirement
- (supersedes|superseded|overrides|carried[[:space:]]+(from|forward))[[:space:]]+the
+ (supersed(es|ed)|overrid(es|den)|carried[[:space:]]+(from|forward))\b
+ revised[[:space:]]+authority

# H5 — broaden reviewer modal to include verification-framing (extends R7 edit)
  reviewers?[[:space:]]+(need[[:space:]]+not|must[[:space:]]+not|should[[:space:]]+not
+   |must|can|may|must[[:space:]]+be[[:space:]]+able|is[[:space:]]+not[[:space:]]+expected)

Regression note for H3 (\bneed not\b): this will fire on legitimate scope-narrowing too (the contract uses "need not cover" benignly). It is a candidate-span tool, so that's acceptable, but it shifts load onto the agent's classifier — pairs with architecture note C (claim-vs-disclaimer rubric) in the main review.

Correction / confirmation

  • Disposition-ledger rows that contain human decision (e.g. Human decision says CI will be added later…, This third function is required by human decision.) are caught by the existing human[[:space:]]+decision pattern — directly confirmed. No gap there.
  • The over-steered draft did not introduce a flood of authority language; its risk was structural/technical over-specification, which the style-lint pass is explicitly not responsible for. So the low 12/248 catch rate is not itself a scanner indictment — but it is a reminder (architecture note A/B) that the most consequential over-steering in this file's history was structural, beneath the phrase-level radar.

Net new actionable items from history: add the confidence-adverb pattern (H1, highest value — recurrent real cleaning target, currently zero coverage), the generic need not pattern (H3), and fold superseded in v4 / revised authority (H4) and reviewer-verification framing (H5) into the R4/R7 edits already proposed above.

@mcdigman

Copy link
Copy Markdown
Owner Author

Review

What it does: Adds a read-only contract-style-lint pass (agent + mirrored skill) with a deterministic shell-based candidate scanner, registers the new style_lint classification in conventions.md and the handoff JSON schema, and rewires contract-clean-consolidation/contract-cleaner to accept findings from three input passes instead of two. Mirrors are byte-identical, the schema is valid JSON, and both scanner engines run cleanly.

Findings

1. .claude/tools/contract_style_lint_scan.sh:543 — the "deterministic" candidate scan is environment-dependent: the rg and grep fallback paths disagree on spans that bridge non-ASCII punctuation.
Many patterns use [[:print:]]{0,N} to bridge context (e.g. (clarity|correctness|speed|performance)[[:print:]]{0,50}takes?[[:space:]]+priority at line 462). Rust-regex (rg) treats [[:print:]] as ASCII-only, so it cannot match across an em-dash or curly quote, whereas GNU/BSD grep in a UTF-8 locale matches them as printable. Reproduced: on the line clarity — and overall correctness — takes priority, the rg path reports no candidate while the grep path reports a hit. Since contracts authored or LLM-generated with em-dashes/curly quotes are common (the agent .md files in this PR contain them), candidate recall silently depends on whether rg is installed — undercutting the "deterministic candidate scan" heading and the prior recall-hardening commits. Fix direction: use . instead of [[:print:]] for the bridge spans (both engines match any char including Unicode), or normalize non-ASCII before scanning.

2. .claude/agents/contract-steering.md:3 & .claude/agents/contract-verbosity.md:3 — sibling pass descriptions not updated for the new third pass.
The new style-lint description says it "Runs in parallel with contract-steering and contract-verbosity," but both siblings still read "Runs in parallel with contract-verbosity" / "...contract-steering" only. Low-severity consistency gap: the three-way parallel relationship is now asymmetric in the docs. (These files are outside the diff — the point is the PR omitted the reciprocal update.)

Everything else checks out: mirror byte-identity holds, the conventions.md routing table and handoff schema enum are consistent, argument validation and tempfile cleanup (trap ... EXIT HUP INT TERM) are correct, and both engine paths handle the no-match/error exit codes properly.

🤖 Generated with Claude Code

mcdigman and others added 6 commits June 29, 2026 23:32
Mirror .claude/skills/contract-style-lint/SKILL.md to the agent file so the
two copies are byte-identical again. Picks up the model:sonnet default, the
highest-leverage default-to-finding line, and the cap-lift line. The agent
file is unchanged by this commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mcdigman

Copy link
Copy Markdown
Owner Author

Blind A/B testing of contract-style-lint — model comparison (GPT-5.5 vs Sonnet vs Haiku)

Summary of the blind model-comparison runs used to tune this agent.

Method. Each model ran the exact agent prompt as a one-shot blind audit against the v6 contract at c06116c. The deterministic scanner (contract_style_lint_scan.sh) output was supplied as a fourth input (the orchestrator path, since sandboxed agents have no shell). Inputs were byte-identical across models; only the model — and, in later rounds, one prompt line at a time — changed, verified by diff. Recall was scored against four ground-truth catches: the TR35 (no reliance on fragile compiled-code inspection) method-exclusion, the Nsf=150 current-configuration reassurance, the DI "It is normative" authority elevation, and the "This consolidation" self-authority.

Per-model pros & cons

GPT-5.5 (Codex, cross-family)

  • Pros: Caught both anchor targets (Nsf=150, TR35) in essentially every run. Best carve-out discipline → highest precision (reliably rejected consensus-harmless spans). Surfaced the sharpest unique structural catches — the superseded AM1-R7 authority citation, and DI-normativity / "This consolidation" as blocking possible_contract_defects. Genuinely cross-family: decorrelated from this Claude-drafted contract; a GPT design-adversary is what originally caught the TR35 over-constraint that the Claude steering pass missed.
  • Cons: Stochastic — on identical wording its count ranges 5–9 findings / 1–2 PCDs (measured over a 6-run sample). The second high-value PCD flickers run-to-run, so single runs are noisy.

Sonnet (claude-sonnet-4-6)

  • Pros: Most stable and thorough (consistently ~9–13 findings); caught both anchors every run. Produced the single sharpest novel catch of the study — the standalone-claim-vs-incorporation-by-reference defect (high PCD), which no other model/run found. Disciplined per-candidate adjudication; correctly rejected the consensus-harmless aligned-table modal. It was the model the mechanical-lint framing first pulled onto the TR35 blind spot that judgment-based steering never reached.
  • Cons: Occasionally over-reaches on borderline carve-outs (flags citation-only provenance parentheticals). Its specific PCD target shifts between runs. Most verbose / highest token cost.

Haiku (claude-haiku-4-5)

  • Pros: Cheapest. When anchored to the scanner it can do a systematic candidate-by-candidate adjudication, and it occasionally raises something the others miss.
  • Cons (disqualifying as sole runner): Highly unstable — counts bounced 0 / 6 / 8 / 12 across wordings, and it dropped the anchor catches (Nsf=150, TR35) in some runs. Most steerable: the single line "a contract with zero findings is a good outcome" flipped it from a full audit to 0 findings / recommend_approve; isolation showed that removing that phrase or adding "adjudicate each candidate individually" each independently fixed the collapse. Lowest precision: repeatedly flagged the aligned-table span the stronger models reject as harmless. Early runs also emitted malformed handoffs and invented non-canonical classification tokens.

Key empirical findings

  • The deterministic scanner is the biggest lever. Emphasis normalization + tuned regexes closed real recall gaps — e.g. Nsf=150 became a candidate, which GPT/Sonnet had silently dropped when over-anchoring to the earlier, incomplete candidate list. Surfacing a span in the scanner is what makes a catch model-agnostic.
  • Consensus exposed a false positive. The aligned-table "neither needs nor is prohibited" modal was flagged only by Haiku's first run; once the scanner forced all three to examine it, GPT and Sonnet both rejected it as a clear, deliberately dual-sided permission. It is a Haiku false positive, not a missed defect.
  • One prompt line measurably steered the weakest model. "Zero findings is a good outcome" steered Haiku toward engineering zero findings; the revised wording (drop that phrase + adjudicate candidates individually) fixed it (0→8), confirmed by single-edit isolation.
  • Most wording differences were noise. A controlled 5-runs-per-cell A/B (keep vs strike the "adjudicated individually" clause) put the difference (mean 7.2 vs 6.2 findings) well inside GPT's 5–9 stochastic band. Earlier single-sample wording swings (4→5→9) were largely stochasticity; only large effects (Haiku's 0↔8, the scanner recall fixes) cleared the noise floor.

Recommendation

  • Default the agent to Sonnet (done — model: switched from haiku): stable, thorough, on-target, good carve-out discipline.
  • Run GPT-5.5 as a second, cross-family pass and union the findings — decorrelated from a Claude-drafted contract, and it contributes the sharpest unique PCDs.
  • Do not use Haiku as the sole runner — unstable, most steerable, drops anchor catches. At most a cheap supplementary lens whose unique hits must be filtered by cross-model consensus.
  • For future prompt tuning in the ±1–2 finding range, use ≥5 samples per cell — single runs cannot separate a small wording effect from GPT's intrinsic variance.

🤖 Generated with Claude Code

@mcdigman
mcdigman merged commit ffa3fb8 into contracts/cleaning-phase Jun 30, 2026
5 checks passed
@mcdigman
mcdigman deleted the codex/add-contract-style-lint branch June 30, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant