perf(scan): parse/read files once, parallelize sections, scope cache

.claude/knowledge/architecture-boundaries.md

@@ -15,6 +15,10 @@ Key architectural decisions, service boundaries, data flow, integration points,
 - **The secret scanner exposes a pure API for reuse: `security.BuildScanner(cfg) Scanner` + `Scanner.ScanContent(content) []Match` + `Scanner.SkipPath(file)`.** `secretFindingsForFile` wraps `ScanContent` with `env.NewFinding`. Tiers in `scanLine` (one match per line, priority order): private-key → `credentialPatterns` (known formats + DB/Bearer/Azure, fail) → config `custom_patterns` → name-based (warn) → opt-in entropy (`security.high-entropy-string`, warn). Values are masked in messages via `maskSecret` (first4…last4). Entropy = `shannonEntropy` (bits/char) over whitespace-free quoted literals, gated by `secrets.entropy` (off by default; min_length 20, threshold 4.5). GitHub-token CRC32 checksum validation was deliberately NOT added — getting the algorithm slightly wrong yields false negatives (worse than a false positive for a security tool).
 - **Git-history secret scan lives in its own top-level package `internal/codeguard/history` (shells `git log -p -U0`), exposed via `pkg/codeguard.ScanGitHistory` and the `scan-history` CLI command.** It MUST be a separate package, not `runner/support`: `checks/security` imports `runner/support` (for `MatchPattern`), so `runner/support` importing `checks/security` would cycle. `history` imports `checks/security` (above it) and is consumed by `pkg` + CLI (both top-level). The parser tracks `+++ b/<path>`, hunk headers (`@@ … +c`), and `+`/`-` line prefixes to attribute added lines to commits; dedupes by rule+path+masked-message keeping the newest commit (log is newest-first).
 - **GOTCHA: `support.ScanTargetFiles` caches per-file findings keyed by `(sectionID, target.Path, rel)`.** If two passes scan the same files with the same `sectionID`, the second pass gets a cache hit and returns the FIRST pass's findings. The secret pass therefore uses a distinct cache id `"security-secrets"` (not `"security"`) so it doesn't collide with the language pass. `sectionID` here only affects the cache key — findings still land in the real section via their RuleID/`FinalizeSection`. When adding any additional `ScanTargetFiles` call to an existing section, give it a unique cache id.
+- **GOTCHA: `ScanTargetFiles`'s per-file cache only works for PURE evaluators** (evaluators whose sole output is the returned `[]core.Finding`). On a cache hit the evaluator is NOT called, so any evaluator that builds state via a captured closure and returns `nil` will silently produce nothing once a warm cache exists. Clone detection hit exactly this (all `quality.duplicate-code` findings vanished on the second scan). Cross-file/stateful passes MUST use `VisitTargetFiles` (bypasses the findings cache, visits every file) instead of `ScanTargetFiles`. See `checks/quality/quality_clone.go:cloneDocumentsForTarget`.
+- **PERF: the scan is parallelized at the section level** (`runner/checks/checks.go:Build`) on a `runtime.NumCPU()`-bounded worker pool; results are written into position-indexed slots so output order stays deterministic. This required making the shared mutable state concurrency-safe: `ScanCache` (mutex in `cache_types.go`), `ArtifactStore` (mutex in `artifacts.go`), and the per-scan file corpus. `OnSectionComplete` is wrapped in a mutex (`withSynchronizedSectionCallback`) since sections now finish concurrently. File-level parallelism was deliberately NOT used — some evaluators mutate closure state (clone), which is only safe when a section runs serially.
+- **PERF: files are walked, read, and Go-parsed once per scan via a shared `fileCorpus`** (`runner/support/corpus.go`), held by pointer on the by-value `Context` so all sections share it. `ScanTargetFiles`/`VisitTargetFiles` read bytes through it; `support.ParseGoSource(env, file, data)` returns a shared `*ast.File`/`*token.FileSet` (parsed with `ParseComments`) so quality and design no longer re-parse the same file. Cached ASTs are read-only — never mutate them. Tradeoff: peak memory is higher (the corpus retains all read bytes + ASTs for the scan), but wall-clock improves ~1.2x (small repos) to ~1.7x (large repos). Taint/graph/perf parsers keep their own cheaper parse modes (`SkipObjectResolution`/`ImportsOnly`) and are intentionally not routed through the shared cache.
+- **The per-file cache fingerprint is per-config-family, not global** (`SectionConfigHashes` in `cache_helpers.go`; cache version bumped to 7). Each family (`quality`/`design`/`security`/`prompts`/`ci`/`contracts`) hashes only the config that can change its findings (quality also hashes `DesignRules` + `AI`; the rule catalog is in every family since a metadata override changes `NewFinding` output). `sectionConfigFamily` maps a cache section id to its family by prefix; unknown ids fall back to the `""` all-checks hash so a new section can never silently serve stale entries. Editing an unrelated field (output format, target names, exclude list) no longer invalidates the whole cache.
 - **The MCP server is hand-rolled (no SDK) and transport-agnostic.** Shared JSON-RPC message builders + the synchronous method router live in `internal/cli/mcp_dispatch.go` (`buildResultMessage`/`buildErrorMessage`/`buildProgressMessage`, `serverCapabilities`, `dispatchSyncMethod`). Both transports reuse them. stdio (`mcp_run.go`/`mcp_requests.go`) keeps an async goroutine + cancel-map for `tools/call` (cancellation via `notifications/cancelled`); HTTP (`mcp_http.go`) runs `tools/call` synchronously per-request and streams progress over SSE, cancellation driven by the request context. When adding a capability, wire it into `dispatchSyncMethod` (HTTP picks it up automatically) AND the stdio method switch in `mcp_run.go`.
 - **MCP server→client requests (sampling/roots)** flow through `internal/cli/mcp_client.go`: `clientCaller`/`clientBridge` + a `serverRequester` (pending-id→chan registry). The transport supplies a `send` closure and feeds inbound responses (classified by `isResponseMessage`: id present, no method) back via `serverRequester.deliver`. stdio demuxes responses in the read loop (`mcp_run.go handleLine`); HTTP needs the client to open the `GET {mcp-path}` SSE stream — server-initiated requests are written there and answered on later POSTs, correlated per session in `internal/cli/mcp_http_session.go`. Client capabilities are captured at `initialize` (`parseClientCapabilities`). The `clientCaller`/progress emitter reach tools via context (`withClientCaller`/`withProgress`), not signatures.
 - **MCP fix tools** (`internal/cli/mcp_fix.go`): `verify_fix` wraps `service.VerifyFix` (caller diff); `propose_fix` wraps `service.GenerateVerifiedFix` with a generator resolved as sampling-first (`samplingGenerator` calls the client LLM) then `internalfix.NewAIGenerator(cfg.AI)`; `apply_fix` verifies then writes the tree via `runnersupport.ApplyUnifiedDiff` (the one `destructiveHint` tool), confirming via `clientCaller.elicit` (`elicitation/create`) when supported. Verification test execution is trust-gated, so it needs `CODEGUARD_ALLOW_CONFIG_COMMANDS=1` or it fails closed. Fix failures return `toolErrorResultData` (isError + structuredContent with attempted diff + remaining findings).

.codeguard/codeguard.yaml

@@ -15,6 +15,9 @@ waivers:
   - rule: ci.test-without-assertion
     path: tests/**/trust_main_test.go
     reason: Go TestMain functions bootstrap package-wide trust policy and are not assertion-bearing tests
+  - rule: quality.unbounded-goroutines-in-loop
+    path: internal/codeguard/runner/checks/checks.go
+    reason: section workers are bounded by a NumCPU-sized semaphore before each goroutine is launched
 targets:
   - name: repository
     path: ..

internal/codeguard/checks/design/design_go.go

@@ -3,7 +3,6 @@ package design
 import (
 	"fmt"
 	"go/ast"
-	"go/parser"
 	"go/token"
 	"path/filepath"
 	"strings"
@@ -21,8 +20,7 @@ func goTargetFindings(env support.Context, target core.TargetConfig) []core.Find
 }
 
 func goFindingsForFile(env support.Context, file string, data []byte) []core.Finding {
-	fset := token.NewFileSet()
-	parsed, err := parser.ParseFile(fset, file, data, parser.ParseComments)
+	fset, parsed, err := support.ParseGoSource(env, file, data)
 	if err != nil {
 		return nil
 	}

internal/codeguard/checks/quality/quality_clone.go

@@ -50,14 +50,17 @@ func cloneFindingsForTarget(env support.Context, target core.TargetConfig) []cor
 func cloneDocumentsForTarget(env support.Context, target core.TargetConfig) []cloneDocument {
 	docs := make([]cloneDocument, 0)
 	include := cloneIncludeForLanguage(target.Language)
-	env.ScanTargetFiles(target, "quality-clone", func(rel string) bool {
+	// Clone detection builds cross-file state (the document list) rather than
+	// per-file findings, so it must visit every file directly. Routing it
+	// through the per-file findings cache would skip the tokenizer on a cache
+	// hit and silently drop every clone once a warm cache exists.
+	env.VisitTargetFiles(target, func(rel string) bool {
 		return include(rel) && !cloneExcludedPath(target.Language, rel)
-	}, func(file string, data []byte) []core.Finding {
+	}, func(file string, data []byte) {
 		tokens := tokenizeNormalizedCloneText(string(data))
 		if len(tokens) > 0 {
 			docs = append(docs, cloneDocument{Path: file, Tokens: tokens})
 		}
-		return nil
 	})
 	return docs
 }
@@ -81,24 +84,39 @@ func cloneWindowIndex(docs []cloneDocument, threshold int) cloneIndex {
 }
 
 func collectCloneCandidates(index cloneIndex, docs []cloneDocument, threshold int) []cloneCandidate {
-	candidates := make([]cloneCandidate, 0)
+	// Candidates are partitioned by their (LeftDoc, RightDoc) pair. The overlap
+	// merge only ever compares candidates that share a pair, so bucketing by
+	// pair turns the merge from a linear scan of every candidate found so far
+	// into a scan of just the (usually tiny) bucket for that file pair.
+	byPair := make(map[[2]int][]cloneCandidate)
 	for _, occurrences := range index {
-		candidates = appendClonePairs(candidates, occurrences, docs, threshold)
+		addClonePairs(byPair, occurrences, docs, threshold)
 	}
-	return candidates
+	return flattenCloneCandidates(byPair)
 }
 
-func appendClonePairs(candidates []cloneCandidate, occurrences []cloneOccurrence, docs []cloneDocument, threshold int) []cloneCandidate {
+func addClonePairs(byPair map[[2]int][]cloneCandidate, occurrences []cloneOccurrence, docs []cloneDocument, threshold int) {
 	if len(occurrences) < 2 {
-		return candidates
+		return
 	}
 	for i := 0; i < len(occurrences); i++ {
 		for j := i + 1; j < len(occurrences); j++ {
 			if next, ok := cloneCandidateForPair(occurrences[i], occurrences[j], docs, threshold); ok {
-				candidates = appendOrMergeCloneCandidate(candidates, next)
+				addOrMergeCloneCandidate(byPair, next)
 			}
 		}
 	}
+}
+
+func flattenCloneCandidates(byPair map[[2]int][]cloneCandidate) []cloneCandidate {
+	total := 0
+	for _, bucket := range byPair {
+		total += len(bucket)
+	}
+	candidates := make([]cloneCandidate, 0, total)
+	for _, bucket := range byPair {
+		candidates = append(candidates, bucket...)
+	}
 	return candidates
 }
 
@@ -120,40 +138,45 @@ func cloneCandidateForPair(left cloneOccurrence, right cloneOccurrence, docs []c
 }
 
 func sortCloneCandidates(candidates []cloneCandidate, docs []cloneDocument) {
+	// Normalize each document path once up front rather than 2-4 times inside
+	// every comparator call.
+	slashPaths := make([]string, len(docs))
+	for i := range docs {
+		slashPaths[i] = filepath.ToSlash(docs[i].Path)
+	}
 	sort.Slice(candidates, func(i, j int) bool {
-		leftDoc := filepath.ToSlash(docs[candidates[i].LeftDoc].Path)
-		rightDoc := filepath.ToSlash(docs[candidates[j].LeftDoc].Path)
+		leftDoc := slashPaths[candidates[i].LeftDoc]
+		rightDoc := slashPaths[candidates[j].LeftDoc]
 		if leftDoc != rightDoc {
 			return leftDoc < rightDoc
 		}
 		if candidates[i].LeftStart != candidates[j].LeftStart {
 			return candidates[i].LeftStart < candidates[j].LeftStart
 		}
-		otherLeft := filepath.ToSlash(docs[candidates[i].RightDoc].Path)
-		otherRight := filepath.ToSlash(docs[candidates[j].RightDoc].Path)
+		otherLeft := slashPaths[candidates[i].RightDoc]
+		otherRight := slashPaths[candidates[j].RightDoc]
 		if otherLeft != otherRight {
 			return otherLeft < otherRight
 		}
 		return candidates[i].RightStart < candidates[j].RightStart
 	})
 }
 
-func appendOrMergeCloneCandidate(candidates []cloneCandidate, next cloneCandidate) []cloneCandidate {
+func addOrMergeCloneCandidate(byPair map[[2]int][]cloneCandidate, next cloneCandidate) {
 	if next.LeftDoc > next.RightDoc {
 		next.LeftDoc, next.RightDoc = next.RightDoc, next.LeftDoc
 		next.LeftStart, next.RightStart = next.RightStart, next.LeftStart
 	}
-	for idx, existing := range candidates {
-		if existing.LeftDoc != next.LeftDoc || existing.RightDoc != next.RightDoc {
-			continue
-		}
+	key := [2]int{next.LeftDoc, next.RightDoc}
+	bucket := byPair[key]
+	for idx, existing := range bucket {
 		if cloneRangesOverlap(existing.LeftStart, existing.Length, next.LeftStart, next.Length) &&
 			cloneRangesOverlap(existing.RightStart, existing.Length, next.RightStart, next.Length) {
 			if next.Length > existing.Length {
-				candidates[idx] = next
+				bucket[idx] = next
 			}
-			return candidates
+			return
 		}
 	}
-	return append(candidates, next)
+	byPair[key] = append(bucket, next)
 }

internal/codeguard/checks/quality/quality_go.go

@@ -4,7 +4,6 @@ import (
 	"fmt"
 	"go/ast"
 	"go/format"
-	"go/parser"
 	"go/token"
 	"path/filepath"
 	"strings"
@@ -39,8 +38,7 @@ func goFindingsForFile(env support.Context, file string, data []byte) []core.Fin
 		}))
 	}
 
-	fset := token.NewFileSet()
-	parsed, err := parser.ParseFile(fset, file, data, parser.ParseComments)
+	fset, parsed, err := support.ParseGoSource(env, file, data)
 	if err != nil {
 		findings = append(findings, env.NewFinding(support.FindingInput{
 			RuleID:  "quality.parse-error",

internal/codeguard/checks/security/security_regex_cache.go

@@ -0,0 +1,25 @@
+package security
+
+import (
+	"regexp"
+	"sync"
+)
+
+// dynamicPatternCache memoizes regexes compiled from runtime-derived text such
+// as import aliases, namespaces, and module names. The same handful of aliases
+// and modules recur across many files in a project, so compiling each distinct
+// pattern once and reusing it avoids recompiling identical regexes per file.
+var dynamicPatternCache sync.Map // map[string]*regexp.Regexp
+
+// compileDynamicPattern returns the compiled form of expr, reusing a previously
+// compiled instance when one exists. The expressions passed here are always
+// valid (fixed fragments plus regexp.QuoteMeta-escaped input), so it mirrors the
+// regexp.MustCompile contract and panics on a genuinely malformed pattern.
+func compileDynamicPattern(expr string) *regexp.Regexp {
+	if cached, ok := dynamicPatternCache.Load(expr); ok {
+		return cached.(*regexp.Regexp)
+	}
+	compiled := regexp.MustCompile(expr)
+	actual, _ := dynamicPatternCache.LoadOrStore(expr, compiled)
+	return actual.(*regexp.Regexp)
+}

internal/codeguard/checks/security/security_typescript.go

@@ -63,7 +63,7 @@ func typeScriptAliasedShellFindings(ctx typeScriptScanContext) []core.Finding {
 	message := support.ScriptLabelForPath(ctx.file) + " shell execution primitive should be reviewed"
 
 	for alias := range execAliases {
-		pattern := regexp.MustCompile(`\b` + regexp.QuoteMeta(alias) + `\s*\(`)
+		pattern := compileDynamicPattern(`\b` + regexp.QuoteMeta(alias) + `\s*\(`)
 		findings = append(findings, regexTypeScriptSecurityFindings(ctx, support.ScriptRegexSpec{Pattern: pattern, RuleID: ruleID, Level: "warn", Message: "shell execution primitive should be reviewed"})...)
 	}
 	for alias := range spawnAliases {
@@ -72,7 +72,7 @@ func typeScriptAliasedShellFindings(ctx typeScriptScanContext) []core.Finding {
 		}
 	}
 	for namespace := range childProcessNamespaces {
-		pattern := regexp.MustCompile(`\b` + regexp.QuoteMeta(namespace) + `\s*\.\s*(?:exec|execSync)\s*\(`)
+		pattern := compileDynamicPattern(`\b` + regexp.QuoteMeta(namespace) + `\s*\.\s*(?:exec|execSync)\s*\(`)
 		findings = append(findings, regexTypeScriptSecurityFindings(ctx, support.ScriptRegexSpec{Pattern: pattern, RuleID: ruleID, Level: "warn", Message: "shell execution primitive should be reviewed"})...)
 		for _, line := range typeScriptCallLinesWithShellOption(ctx, namespace, true) {
 			findings = append(findings, newTypeScriptSecurityFinding(ctx, ruleID, line, message))
@@ -88,15 +88,15 @@ func typeScriptVMFindings(ctx typeScriptScanContext) []core.Finding {
 	vmNamespaces := collectTypeScriptNamespaceBindings(ctx.source, "vm")
 
 	for alias, original := range directAliases {
-		pattern := regexp.MustCompile(`\b` + regexp.QuoteMeta(alias) + `\s*\(`)
+		pattern := compileDynamicPattern(`\b` + regexp.QuoteMeta(alias) + `\s*\(`)
 		if original == "Script" {
-			pattern = regexp.MustCompile(`\bnew\s+` + regexp.QuoteMeta(alias) + `\s*\(`)
+			pattern = compileDynamicPattern(`\bnew\s+` + regexp.QuoteMeta(alias) + `\s*\(`)
 		}
 		findings = append(findings, regexTypeScriptSecurityFindings(ctx, support.ScriptRegexSpec{Pattern: pattern, RuleID: securityRuleID(ctx.file, "vm-dynamic-code"), Level: "warn", Message: "Node vm dynamic code execution should be reviewed"})...)
 	}
 	for namespace := range vmNamespaces {
-		methodPattern := regexp.MustCompile(`\b` + regexp.QuoteMeta(namespace) + `\s*\.\s*(?:runInContext|runInNewContext|runInThisContext|compileFunction)\s*\(`)
-		scriptPattern := regexp.MustCompile(`\bnew\s+` + regexp.QuoteMeta(namespace) + `\s*\.\s*Script\s*\(`)
+		methodPattern := compileDynamicPattern(`\b` + regexp.QuoteMeta(namespace) + `\s*\.\s*(?:runInContext|runInNewContext|runInThisContext|compileFunction)\s*\(`)
+		scriptPattern := compileDynamicPattern(`\bnew\s+` + regexp.QuoteMeta(namespace) + `\s*\.\s*Script\s*\(`)
 		findings = append(findings, regexTypeScriptSecurityFindings(ctx, support.ScriptRegexSpec{Pattern: methodPattern, RuleID: securityRuleID(ctx.file, "vm-dynamic-code"), Level: "warn", Message: "Node vm dynamic code execution should be reviewed"})...)
 		findings = append(findings, regexTypeScriptSecurityFindings(ctx, support.ScriptRegexSpec{Pattern: scriptPattern, RuleID: securityRuleID(ctx.file, "vm-dynamic-code"), Level: "warn", Message: "Node vm dynamic code execution should be reviewed"})...)
 	}

internal/codeguard/checks/security/security_typescript_bindings.go

@@ -33,7 +33,7 @@ func collectTypeScriptNamedModuleBindings(source string, module string, allowed
 func collectTypeScriptNamespaceBindings(source string, module string) map[string]struct{} {
 	namespaces := make(map[string]struct{})
 	for _, pattern := range []*regexp.Regexp{tsNamespaceImportPattern, tsDefaultImportPattern, tsNamespaceRequirePattern} {
-		re := regexp.MustCompile(strings.ReplaceAll(pattern.String(), "%s", regexp.QuoteMeta(module)))
+		re := compileDynamicPattern(strings.ReplaceAll(pattern.String(), "%s", regexp.QuoteMeta(module)))
 		for _, match := range re.FindAllStringSubmatch(source, -1) {
 			if len(match) > 1 {
 				namespaces[match[1]] = struct{}{}
@@ -46,7 +46,7 @@ func collectTypeScriptNamespaceBindings(source string, module string) map[string
 func collectTypeScriptBindingSpecs(source string, module string, patterns ...*regexp.Regexp) []string {
 	specs := make([]string, 0)
 	for _, pattern := range patterns {
-		re := regexp.MustCompile(strings.ReplaceAll(pattern.String(), "%s", regexp.QuoteMeta(module)))
+		re := compileDynamicPattern(strings.ReplaceAll(pattern.String(), "%s", regexp.QuoteMeta(module)))
 		for _, match := range re.FindAllStringSubmatch(source, -1) {
 			if len(match) > 1 {
 				specs = append(specs, splitTypeScriptBindingSpecs(match[1])...)
@@ -88,7 +88,7 @@ func typeScriptCallLinesWithShellOption(ctx typeScriptScanContext, alias string,
 	} else {
 		patternText += `\s*\(`
 	}
-	pattern := regexp.MustCompile(patternText)
+	pattern := compileDynamicPattern(patternText)
 	for _, call := range support.FindScriptCalls(ctx.source, ctx.code, pattern) {
 		if !scriptCallHasShellTrue(call.Args) {
 			continue