feat(security): hardcoded secret/credential detection + git-history scan

.claude/knowledge/architecture-boundaries.md

@@ -8,6 +8,13 @@ Key architectural decisions, service boundaries, data flow, integration points,
 - **OWASP categories are baked into the rule catalog at var-init time**, not via `init()`. The merged `catalog` var calls `withSecurityOWASP(mergeRuleCatalogs(...))` in `rules/catalog.go`; an `init()` would run *after* the `catalog` var initializer copied stale entries. The mapping lives in `rules/catalog_security_owasp.go`. All read paths (`Catalog`, `RuleCatalogForConfig`, SARIF, CLI) flow through `rules.Catalog()`.
 - Config-controlled artifact paths (`baseline.path`, `cache.path`, `ai.cache.path`) are resolved relative to the config dir and contained within it by `config.containConfigArtifactPaths` (in `config/io.go`, `LoadFile`). Paths escaping the config directory are rejected.
 - The language-agnostic OWASP-gap line rules (CORS, debug, bind-all, weak-hash/cipher, deserialization, Dockerfile root) live in `checks/security/security_owasp_extra.go` and run per-line via `findingsForFile`. NOTE: `findingsForFile` is the **non-TypeScript** path — `securityTargetFindings` routes TS/JS targets to a separate `typeScriptTargetFindings` pipeline, so these rules do not run on TS/JS targets. String-literal signals (CORS, bind-all) match the raw line; call/identifier signals match the masked line.
+- **Secret/credential scanning (`checks/security/security_secrets.go`) runs as its own repository-wide pass in `securityTargetFindings`, unconditionally for BOTH the TS and non-TS branches** — precisely so it covers TS/JS targets that bypass `findingsForFile`. Tiers: `security.hardcoded-credential` (fail; known provider formats + `security_rules.secrets.custom_patterns`), `security.private-key` (fail), `security.hardcoded-secret` (warn; lower-confidence name-based heuristic — was fail before, deliberately downgraded). Patterns match the **raw** line (the token lives in a string literal that masking would blank). Placeholder/allowlist filtering is done in Go after the regex match because RE2 has no lookahead (mirrors `isInsecureDeserialization`). Config: `security_rules.secrets` (`enabled`/`allow_paths`/`allow_patterns`/`custom_patterns`), defaulted in `applySecurityDefaults`, validated in `validateSecretsRules`.
+- **PERF: the secret scanner is gated by a cheap literal pre-check (`builtinGatePasses`), not a combined regex.** Profiling showed `strings.Contains`/byte scans are ~85x faster than RE2 alternation, and that a leading `\b` or `(?i)` defeats RE2's literal-prefix fast path (those patterns ran ~1.8–7ms per 2000 lines; literal-prefix patterns like the Slack webhook ran ~50µs). The gate is `gateLiterals` (case-sensitive `strings.Contains`) + `gateFoldLiterals` (lowercase, via alloc-free `asciiContainsFold`); when a line contains no marker, the ~17 built-in regexes are skipped. Net: default scan went 3.2→99 MB/s, entropy mode 3.1→47 MB/s. **INVARIANT: the gate must be a superset of every built-in pattern — adding a credential pattern requires adding its required literal to the gate.** `TestBuiltinGateCoversPatterns` enforces this; custom config patterns have arbitrary markers so they bypass the gate and always run.
+- **CodeQL `go/regex/missing-regexp-anchor` (High) fires on unanchored regexes containing a host literal (e.g. `hooks\.slack\.com`), even for detection patterns.** The repo's accepted fix (see `quality_ai_style_drift.go:17` and the Slack webhook pattern in `security_secret_patterns.go`) is to bound the host with `(?:^|[^A-Za-z0-9_])(...)(?:$|[^A-Za-z0-9_])` — start/non-host-char on both sides — which adds the capture group as the matched value. CodeQL accepts this form; a bare `\b` or `(?:^|...)` left-only boundary may not. Apply this whenever adding a regex with an embedded domain literal.
+- **HARDENING (codeguard scans untrusted PR content): the scanner bounds its work** — skips files > `maxScanFileBytes` (5 MiB), scans only the first `maxScanLineBytes` (64 KiB) of any line, and skips binary files (`looksBinary` = NUL byte in first 8 KiB). The git-history parser uses `bufio.Reader.ReadString` (grows for any line length), NOT `bufio.Scanner` — Scanner silently stops on an over-long token, which for a security scan would skip the rest of history undetected.
+- **The secret scanner exposes a pure API for reuse: `security.BuildScanner(cfg) Scanner` + `Scanner.ScanContent(content) []Match` + `Scanner.SkipPath(file)`.** `secretFindingsForFile` wraps `ScanContent` with `env.NewFinding`. Tiers in `scanLine` (one match per line, priority order): private-key → `credentialPatterns` (known formats + DB/Bearer/Azure, fail) → config `custom_patterns` → name-based (warn) → opt-in entropy (`security.high-entropy-string`, warn). Values are masked in messages via `maskSecret` (first4…last4). Entropy = `shannonEntropy` (bits/char) over whitespace-free quoted literals, gated by `secrets.entropy` (off by default; min_length 20, threshold 4.5). GitHub-token CRC32 checksum validation was deliberately NOT added — getting the algorithm slightly wrong yields false negatives (worse than a false positive for a security tool).
+- **Git-history secret scan lives in its own top-level package `internal/codeguard/history` (shells `git log -p -U0`), exposed via `pkg/codeguard.ScanGitHistory` and the `scan-history` CLI command.** It MUST be a separate package, not `runner/support`: `checks/security` imports `runner/support` (for `MatchPattern`), so `runner/support` importing `checks/security` would cycle. `history` imports `checks/security` (above it) and is consumed by `pkg` + CLI (both top-level). The parser tracks `+++ b/<path>`, hunk headers (`@@ … +c`), and `+`/`-` line prefixes to attribute added lines to commits; dedupes by rule+path+masked-message keeping the newest commit (log is newest-first).
+- **GOTCHA: `support.ScanTargetFiles` caches per-file findings keyed by `(sectionID, target.Path, rel)`.** If two passes scan the same files with the same `sectionID`, the second pass gets a cache hit and returns the FIRST pass's findings. The secret pass therefore uses a distinct cache id `"security-secrets"` (not `"security"`) so it doesn't collide with the language pass. `sectionID` here only affects the cache key — findings still land in the real section via their RuleID/`FinalizeSection`. When adding any additional `ScanTargetFiles` call to an existing section, give it a unique cache id.
 - **The MCP server is hand-rolled (no SDK) and transport-agnostic.** Shared JSON-RPC message builders + the synchronous method router live in `internal/cli/mcp_dispatch.go` (`buildResultMessage`/`buildErrorMessage`/`buildProgressMessage`, `serverCapabilities`, `dispatchSyncMethod`). Both transports reuse them. stdio (`mcp_run.go`/`mcp_requests.go`) keeps an async goroutine + cancel-map for `tools/call` (cancellation via `notifications/cancelled`); HTTP (`mcp_http.go`) runs `tools/call` synchronously per-request and streams progress over SSE, cancellation driven by the request context. When adding a capability, wire it into `dispatchSyncMethod` (HTTP picks it up automatically) AND the stdio method switch in `mcp_run.go`.
 - **MCP server→client requests (sampling/roots)** flow through `internal/cli/mcp_client.go`: `clientCaller`/`clientBridge` + a `serverRequester` (pending-id→chan registry). The transport supplies a `send` closure and feeds inbound responses (classified by `isResponseMessage`: id present, no method) back via `serverRequester.deliver`. stdio demuxes responses in the read loop (`mcp_run.go handleLine`); HTTP needs the client to open the `GET {mcp-path}` SSE stream — server-initiated requests are written there and answered on later POSTs, correlated per session in `internal/cli/mcp_http_session.go`. Client capabilities are captured at `initialize` (`parseClientCapabilities`). The `clientCaller`/progress emitter reach tools via context (`withClientCaller`/`withProgress`), not signatures.
 - **MCP fix tools** (`internal/cli/mcp_fix.go`): `verify_fix` wraps `service.VerifyFix` (caller diff); `propose_fix` wraps `service.GenerateVerifiedFix` with a generator resolved as sampling-first (`samplingGenerator` calls the client LLM) then `internalfix.NewAIGenerator(cfg.AI)`; `apply_fix` verifies then writes the tree via `runnersupport.ApplyUnifiedDiff` (the one `destructiveHint` tool), confirming via `clientCaller.elicit` (`elicitation/create`) when supported. Verification test execution is trust-gated, so it needs `CODEGUARD_ALLOW_CONFIG_COMMANDS=1` or it fails closed. Fix failures return `toolErrorResultData` (isError + structuredContent with attempted diff + remaining findings).

.claude/knowledge/testing-patterns.md

@@ -4,6 +4,7 @@ Testing strategies, test infrastructure quirks, how to run/debug specific test s
 
 - The trust policy (`internal/codeguard/trust`) is a process-global. Tests that exercise command-driven checks or local (127.0.0.1) AI endpoints must enable it. `tests/checks` and `tests/codeguard` each have a `TestMain` (`trust_main_test.go`) that sets `Policy{AllowConfigCommands: true, AllowConfigAIEndpoints: true}`. If you add a new test package that runs config commands or hits a loopback test server, add a similar `TestMain` or the tests will fail with "refusing to run config-supplied command" / "ssrf guard" errors.
 - `tests/security` deliberately has NO trust `TestMain`: it verifies the secure *default*. Tests there set/restore the policy locally with `trust.Set(...)` + `t.Cleanup(trust.ResetFromEnv)`. Do not add a package-wide opt-in there.
-- All tests live under `tests/` as external (`_test`) packages; there are no white-box tests inside `internal/`/`pkg/`. Internal packages are still importable from `tests/` because they are within the module.
+- All tests live under `tests/` as external (`_test`) packages; there are no white-box tests inside `internal/`/`pkg/`. Internal packages are still importable from `tests/` because they are within the module. This is enforced by `ci.test-file-location` (the repo's own scan fails on `_test.go` files outside `tests/**`), so do not add tests next to the code even for unexported access — import the internal package from `tests/` and test via its exported API instead.
+- **Never commit a contiguous real-format secret as a test fixture.** GitHub push protection rejects the push (it flags GitLab/Stripe/SendGrid/Twilio/etc. tokens even in `_test.go` files), and it is poor practice in a secret-detection repo. Assemble fixtures at runtime from a prefix + body via the `cred(prefix, body)` helper in `tests/checks/test_helpers_test.go` (e.g. `cred("AKIA", "1234567890ABCDEF")`), splitting the recognizable provider prefix from the rest so no full token literal appears in source. The reconstructed value still exercises the scanner identically. `goConst(value)` wraps a value as a minimal Go source fixture.
 - The MCP server smoke tests (`tests/mcp/`) drive the **real binary in a subprocess**, not the in-process handler, to keep tests external. Pattern: a `Test...HelperProcess` func gated by an env var (`GO_WANT_MCP_HELPER_PROCESS` for stdio, `GO_WANT_MCP_HTTP_HELPER_PROCESS` for HTTP) re-execs `os.Args[0]` and calls `cli.Run(...)`. stdio replays NDJSON transcripts over stdin; HTTP reserves a free `127.0.0.1:0` port, launches `serve --mcp --http`, polls `/healthz` for readiness, then issues real HTTP requests. Add new MCP behavior as a transcript + assertion (stdio) or a subtest in `http_test.go` (HTTP).
 - **Bidirectional (server→client) MCP tests** live in `tests/mcp/sampling_test.go`: the test acts as the MCP client, advertises `sampling`/`roots` at `initialize`, and answers the server's server-initiated requests. stdio uses interactive `StdinPipe`/`StdoutPipe` (not the replay harness). HTTP opens the `GET /mcp` SSE stream (waits for the `: ready` comment to avoid the attach race), reads the request off the stream, and POSTs the response with the matching `Mcp-Session-Id`. propose_fix verification is expected to fail on the throwaway diff — assert the round trip fired, not a verified patch. The HTTP helper passes `-config` via `CODEGUARD_TEST_HTTP_CONFIG`.

README.md

@@ -71,9 +71,10 @@ codeguard init
 codeguard validate -config codeguard.yaml
 codeguard doctor -config codeguard.yaml
 codeguard scan -config codeguard.yaml
+codeguard scan-history
 codeguard rules
 codeguard profiles
-codeguard explain security.hardcoded-secret
+codeguard explain security.hardcoded-credential
 codeguard baseline -config codeguard.yaml -output codeguard-baseline.json
 ```
 

docs/checks.md

@@ -430,12 +430,103 @@ Language command example:
 ## Security
 
 Purpose:
-- Hardcoded secret detection
+- Hardcoded credential detection (known provider formats)
+- Hardcoded secret detection (name-based heuristic)
 - Private key detection
 - Insecure TLS detection
 - Shell execution review markers
 - Optional `govulncheck`
 
+### Secret & credential scanning
+
+The secret scan runs **repository-wide for every target language** (including
+TypeScript/JavaScript) and reports in both full scans and `-mode diff` scans, so a
+hardcoded credential introduced in a PR fails the changed-lines diff check as well as a
+full scan. It has two confidence tiers:
+
+- `security.hardcoded-credential` (**fail**) — a value matching a known provider format:
+  AWS access keys, GitHub/GitLab tokens, Slack tokens and webhook URLs, Stripe live keys,
+  Google API keys, npm/PyPI/Docker Hub tokens, SendGrid and Twilio keys, Azure storage
+  account keys, database connection strings with embedded passwords
+  (`postgres://user:pass@…`), `Authorization: Bearer …` tokens,
+  `aws_secret_access_key`/`client_secret`/`private_token` assignments, plus any configured
+  `custom_patterns`.
+- `security.hardcoded-secret` (**warn**) — the lower-confidence name-based heuristic: a
+  `secret`/`token`/`api_key`/`password` identifier assigned a quoted literal.
+- `security.high-entropy-string` (**warn**, opt-in) — a high-entropy string literal that
+  may be an unknown/random secret matching no known format. Enabled via
+  `secrets.entropy.enabled`.
+
+Findings report the value **masked** (`AKIA…CDEF`) so the message itself never reprints
+the secret. Obvious placeholders are skipped automatically (`REDACTED`, `xxxx…`,
+`example`, `your-…`, `${ENV}` / `{{ }}` / `<…>` interpolations, `$(...)` command
+substitutions, `op://` / `vault://` secret references, all-same-character fillers,
+`process.env.*` / `os.environ[...]` references).
+
+Config keys (under `checks.security_rules.secrets`):
+
+```json
+{
+  "checks": {
+    "security": true,
+    "security_rules": {
+      "secrets": {
+        "enabled": true,
+        "allow_paths": ["testdata/**", "**/testdata/**"],
+        "allow_patterns": ["EXAMPLE"],
+        "custom_patterns": [
+          {
+            "id": "security.acme-token",
+            "regex": "\\bacme_live_[0-9a-f]{16}\\b",
+            "message": "Acme live tokens must not be committed",
+            "level": "fail"
+          }
+        ],
+        "entropy": {
+          "enabled": false,
+          "min_length": 20,
+          "threshold": 4.5,
+          "level": "warn"
+        }
+      }
+    }
+  }
+}
+```
+
+- `enabled` toggles the whole scan (default `true`).
+- `allow_paths` are globs whose files are skipped (e.g. fixtures under `testdata/`).
+- `allow_patterns` are regexes; a line matching any of them is never reported.
+- `custom_patterns` add repo-specific credential formats; `level` defaults to `fail`.
+- `entropy` enables the high-entropy heuristic (off by default); tune `min_length`
+  (default 20), `threshold` in bits/char (default 4.5), and `level` (default `warn`).
+
+Existing `exclude`, `waivers`, `baseline`, and inline `codeguard:ignore` suppressions
+also apply to these findings.
+
+For performance and resistance to pathological/untrusted input, the scan skips binary
+files and files larger than 5 MiB, and scans only the first 64 KiB of any single line. A
+cheap literal pre-filter skips the per-pattern regexes on lines that contain no credential
+marker, keeping a full-repo scan fast.
+
+### Git-history secret scan
+
+Working-tree and `-mode diff` scans only see the current state. A secret that was
+committed and later removed is still leaked and must be **rotated**, not just deleted.
+`codeguard scan-history` walks added lines across git history and reports any that match
+the secret/credential patterns (using the same `secrets` config — allowlist, custom
+patterns, entropy):
+
+```bash
+codeguard scan-history                       # HEAD history, text output
+codeguard scan-history -all -format json     # every ref, machine-readable
+codeguard scan-history -max-commits 500       # bound the walk on large repos
+```
+
+Findings are deduplicated by rule, path, and masked value, reporting the most recent
+commit that introduced each. The command exits non-zero when any `fail`-level credential
+is found, so it can gate CI.
+
 Current behavior:
 - repository-wide secret and private-key scans apply regardless of target language
 - Go targets include insecure TLS review, shell execution review, and optional `govulncheck`

docs/security.md

@@ -90,6 +90,8 @@ taint engine and default to `fail`.
 
 | Rule | OWASP | Detects |
 | --- | --- | --- |
+| `security.hardcoded-credential` | A07 | values matching known credential formats (AWS, GitHub, GitLab, Slack, Stripe, Google, npm, PyPI, Docker, SendGrid, Twilio, Azure, DB connection strings, Bearer tokens) or a configured custom pattern; **fail** |
+| `security.high-entropy-string` | A07 | opt-in Shannon-entropy heuristic for unknown/random secrets; **warn** |
 | `security.cors-wildcard` | A05 | `Access-Control-Allow-Origin: *` |
 | `security.debug-enabled` | A05 | framework debug flag enabled (`debug=True`) |
 | `security.bind-all-interfaces` | A05 | binding to `0.0.0.0` |

examples/codeguard.json

@@ -94,6 +94,25 @@
     "security_rules": {
       "govulncheck_mode": "auto",
       "govulncheck_command": "govulncheck",
+      "secrets": {
+        "enabled": true,
+        "allow_paths": ["testdata/**", "**/testdata/**"],
+        "allow_patterns": ["EXAMPLE"],
+        "entropy": {
+          "enabled": false,
+          "min_length": 20,
+          "threshold": 4.5,
+          "level": "warn"
+        },
+        "custom_patterns": [
+          {
+            "id": "security.acme-token",
+            "regex": "\\bacme_live_[0-9a-f]{16}\\b",
+            "message": "Acme live tokens must not be committed",
+            "level": "fail"
+          }
+        ]
+      },
       "language_commands": {
         "typescript": [
           {

internal/cli/helpers.go

@@ -32,6 +32,7 @@ Usage:
   codeguard validate [-config codeguard.yaml] [-profile startup|strict|enterprise|ai-safe]
   codeguard validate-patch [-config codeguard.yaml] [-format text|json|sarif|github] [-profile startup|strict|enterprise|ai-safe] [-ai] < patch.diff
   codeguard scan [-config codeguard.yaml] [-mode full|diff] [-base-ref main] [-format text|json|sarif|github] [-interactive] [-profile startup|strict|enterprise|ai-safe] [-ai] [-allow-config-commands] [-allow-config-ai-endpoints]
+  codeguard scan-history [-config codeguard.yaml] [-path .] [-max-commits N] [-all] [-format text|json]   # scan git history for committed secrets
   codeguard fix [-config codeguard.yaml] [-mode full|diff] [-base-ref main] [-profile startup|strict|enterprise|ai-safe] [-rule rule.id] [-path rel/path] [-line N] -ai
   codeguard baseline [-config codeguard.yaml] [-output codeguard-baseline.json] [-mode full|diff] [-base-ref main] [-profile startup|strict|enterprise|ai-safe]
   codeguard report -slop-history [-config codeguard.yaml] [-limit N] [-profile startup|strict|enterprise|ai-safe]

internal/cli/run.go

@@ -20,6 +20,7 @@ var commandCatalog = map[string]commandRunner{
 	"report":         withoutStdin(runReport),
 	"rules":          withoutStdin(runRules),
 	"scan":           runScan,
+	"scan-history":   withoutStdin(runScanHistory),
 	"serve":          runServe,
 	"validate":       withoutStdin(runValidate),
 	"validate-patch": runValidatePatch,