Improvments

[alxxjohn]

Execute the checks-improvement plan: precision, engine, agent-readiness, and the tree-sitter TypeScript engine

272 files changed, +10,264 / −295 · tests 505 → 632 (46 packages) · golangci-lint 0 issues · -race clean · self-scan exit 0

Why

This PR executes a first-principles improvement plan for the check engine. The premise: a check's value is (real bugs caught × cost avoided) − (false positives × attention burned), and the primary consumer of findings is shifting from humans to AI agents. That drove six workstreams: measure and fix precision, stabilize finding identity, make every finding machine-actionable, check codebase legibility itself, close OWASP gaps, and instrument rule health.

1. Detection precision (Tier 1)

AST-based Go security checks. security.insecure-tls and security.shell-execution were literal substring matches on raw source — os/exec in a comment fired, InsecureSkipVerify:true (no space) didn't. Both now walk the cached go/ast (import-alias aware, call-site precise); unparseable files fall back to a masked-line regex scan. A new Go source masker also hardens the remaining line-scan rules against comments/strings.

Ground-truth precision corpus (tests/corpus/). 96+ fixtures across 24 security rules — mutation variants (whitespace, renamed vars) and adversarial cleans (comments, docstrings, placeholders, import-only files) — scored per-rule against expectations.yaml. Known gaps are tracked assertions: the harness fails when a documented gap is unexpectedly fixed so it gets promoted, and fails on regression. The corpus mechanically confirmed the AST rework: precision 0.878 → 0.926, recall 0.956 → 0.969, with insecure-tls/shell-execution at 1.000/1.000. Six remaining gaps are documented in the manifest as the next targets.

Finding confidence. Every finding carries confidence: high|medium|low (AST/manifest-backed = high; name-based and entropy heuristics = low), surfaced in JSON, SARIF property bags, and text. Secret findings in fixture paths (testdata/, *_test.*, *.spec.ts, …) demote fail→warn with low confidence behind checks.security_rules.demote_fixture_findings (default on) — the top secret-rule FP source, still reported, never silent.

2. Engine performance & stability (Tier 2)

• Intra-section file parallelism: ScanTargetFiles fans out to a bounded worker pool (min(4, NumCPU)) with deterministic output order and panic propagation; NL custom rules and the Python design graph stay sequential by design. ~3× faster on a 64-file section benchmark. Also fixes a latent warm-cache bug that left the Python import graph empty on cache-hit scans.
• Caller context threading: ctx now flows into every git subprocess (diff scope, changed files, patch materialization, verified-fix) with the 2-minute timeout as an upper bound; MCP cancellation propagates.
• Line-shift-resilient fingerprints: findings gain a context fingerprint (sha256(rule|path|normalized ±2-line snippet)); baselines match on either the legacy or context fingerprint, so unrelated edits that shift line numbers no longer break suppression. Old baseline files keep working. SARIF results emit partialFingerprints — what GitHub code scanning uses for cross-commit alert dedup.

3. Agent-actionable output (Tier 3)

Every one of the 141 catalog rules now ships a FixTemplate (119 backfilled), typed with kind: deterministic|guided (19 deterministic). Exposed through explain -format agent (new fix_template_kind) and the MCP explain tool. A catalog test enforces the invariant for all future rules.

4. New context.* family — agent-legibility checks (Tier 4)

A new check section, "Agent Context", scoring how workable a repo is for AI agents:

• context.agent-docs-missing — no CLAUDE.md / AGENTS.md / .cursorrules / copilot-instructions
• context.agent-docs-drift / context.readme-drift — docs referencing paths, make targets, or npm scripts that provably don't exist (conservative resolver; skips URLs, globs, placeholders; zero FPs on this repo's own README)
• context.oversized-context-unit — files over an agent-context budget (default 1500 lines; skips generated/vendored)
• context.ambiguous-symbol — duplicated basenames (utils.ts ×9) that defeat grep navigation
• repo_legibility artifact — explainable 0–100 score with per-component breakdown, slop-score style

Enabled in full scans, skipped in diff scans by default (mirrors contracts) so repo-level findings don't nag every PR. All warn-level; the section honestly flags this repo's own gaps without failing CI.

5. OWASP A09 + coverage hygiene (Tier 5)

• security.log-secret-exposure — secret-bearing identifiers/keys/concatenations inside logging call argument lists (Go/Python/TS/JS, masked-source, format-verb aware: "token count: %d" never fires)
• security.unsanitized-error-response — raw errors written into HTTP responses (http.Error(w, err.Error(), …), res.send(err), return str(e) in except blocks)
• codeguard owasp coverage: 8/10 → 9/10 categories (A04 Insecure Design remains, inherently non-static)
• Label fix: Python taint/SSRF rules were mislabeled go-native; now language-agnostic

6. Rule-health telemetry (Tier 6)

Scans record per-rule emitted vs baseline/waiver/inline-suppressed counts into a rule_stats artifact plus a capped history file. doctor now flags dead waivers (matching no rule), expired waivers, and rules suppressed >50% of the time ("consider tuning or disabling") — a suppressed-to-death rule is a broken rule, and now it's visible.

7. Tree-sitter TypeScript engine (spike → shipped behind a flag)

docs/treesitter-spike.md documents the full evaluation (options matrix incl. CGo/WASM routes, benchmarks, supply-chain assessment, phased plan). Outcome: conditional GO on a pure-Go runtime — every CGo option breaks the CGO_ENABLED=0 goreleaser matrix and go install distribution of the Action.

Shipped in this PR, default off:

• First root dependency: github.com/odvcencio/gotreesitter pinned exactly at v0.20.8 (MIT). This is the deliberate, documented end of the zero-dep stance; mitigations per spike §8 — exact pin, go.sum enforcement, differential CI vs the reference C runtime, grammar-subset embedding. Vendor-vs-fork is the flagged follow-up before default-on.
• parsers.treesitter: off|auto (default off — byte-for-byte legacy behavior, verified by a toggle test).
• ParseScriptFile seam (SyntaxTree/CompiledQuery/QueryHit in checks/support), cached per scan in the corpus like Go ASTs. Per-file regex fallback on parse failure, >25% error-node bytes, or files >256KiB (bounds parse heap).
• Four rules migrated to query-based detection: quality.typescript.explicit-any, non-null-assertion (incl. definite-assignment !), double-assertion, security.typescript.unsafe-html-sink + its JS mirror. Same IDs/levels/messages; tree findings carry confidence: high.
• Differential results (EXPECT-marker corpora: comments, template literals, regex literals, generics, satisfies, formatter-split expressions, JSX): regex baseline 63.6% precision / 53.8% recall → tree 100%/100%, with pure-Go vs CGo node-level parity.
• Release builds embed only TS/TSX/JS grammars via build tags (binary 13.6MB → 18.9MB; Makefile + goreleaser wired).

Compatibility

• No breaking changes to config, CLI, SDK, or output formats; all new config keys have safe defaults (parsers.treesitter: off, fixture demotion on, context family off in diff mode).
• RuleMetadata.FixTemplate changed from string to a typed struct in the SDK surface; explain output keeps fix_template as a string and adds fix_template_kind.
• Scan-cache version bumped: first scan after upgrade recomputes (by design — cached findings predate context fingerprints/confidence).
• One SARIF addition (partialFingerprints, properties.confidence) — additive, spec-compliant.

Testing

505 → 632 tests. New: precision corpus with per-rule TP/FN/FP scoring, tree-sitter differential corpora (auto == ground truth AND off == pinned regex), off-vs-auto byte-equality, fallback paths (oversize/garbage/error-heavy), concurrency determinism + -race, baseline line-shift resilience, doctor rule-health, context-family fixtures, A09 adversarial negatives. The repo self-scan stays clean, and the corpus harness now gates every future detector change.

Follow-ups (tracked in docs/treesitter-spike.md §9)

1. Vendor-or-fork gotreesitter before default-on; large-corpus burn-in (parity ≥99.99% of nodes).
2. Flip parsers.treesitter default once rule_stats telemetry shows strictly fewer suppressions in the field.
3. Migrate Python to the tree-sitter path; then Rust/Java; retire hand-rolled parsers per language after two stable releases.
4. Close the six documented corpus gaps (two-arg Header.Set bearer tokens, dbPass := naming miss, comment FPs in raw-line rules).

🤖 Generated with Claude Code