Skip to content

feat(security): hardcoded secret/credential detection + git-history scan#17

Merged
alxxjohn merged 4 commits into
mainfrom
var-secrets
Jun 30, 2026
Merged

feat(security): hardcoded secret/credential detection + git-history scan#17
alxxjohn merged 4 commits into
mainfrom
var-secrets

Conversation

@alxxjohn

Copy link
Copy Markdown
Contributor

Summary

Adds a repository-wide hardcoded secret/credential scanner to the security suite that runs in both full scans and diff (PR) scans, plus a new codeguard scan-history command. Replaces the previous single name-based regex.

Detection tiers

  • security.hardcoded-credential (fail) — known provider formats: AWS, GitHub, GitLab, Slack (token + webhook), Stripe, Google, npm, PyPI, Docker Hub, SendGrid, Twilio, Azure storage keys, DB connection strings with embedded passwords, and Authorization: Bearer tokens — plus config-defined custom_patterns.
  • security.hardcoded-secret (warn) — lower-confidence name-based heuristic. Reclassified from fail → warn (high-confidence detection now lives in the credential rule).
  • security.high-entropy-string (warn, opt-in) — Shannon-entropy detection of unknown/random secrets matching no known format.

Config (security_rules.secrets)

enabled, allow_paths (globs), allow_patterns (regex), custom_patterns, and entropy (enabled/min_length/threshold/level). Findings show the value masked (AKIA…CDEF); placeholders and secret references (op://, vault://, $(...), ${ENV}) are skipped automatically.

codeguard scan-history

Walks git history for secrets that were committed and later removed (still leaked — need rotation), reusing the same scanner/allowlist. Supports -all, -max-commits, -format json; exits non-zero on fail-level hits.

Performance & hardening

  • A fast literal pre-filter gates the per-pattern regexes — ~31× faster default scan (3.2 → 99 MB/s), ~15× with entropy. (strings.Contains proved ~85× faster than the equivalent regex alternation; a leading \b/(?i) defeats RE2's literal-prefix fast path.)
  • Bounds for untrusted PR input: skips binary files and files >5 MiB; scans only the first 64 KiB of any line.
  • History parsing uses bufio.Reader (not Scanner) to avoid silent truncation on over-long lines.
  • Runs across all target languages, including TypeScript/JavaScript (which previously bypassed the secret pass).

Testing

  • go test ./... green (7 packages); go vet and gofmt clean.
  • New table tests per provider format, placeholder/allowlist/custom-pattern/entropy/masking/binary/long-line cases, a gate-superset guard, and git-history integration tests.
  • Test fixtures assemble tokens at runtime (cred(prefix, body)) so no contiguous real-format secret is committed — required for GitHub push protection and good practice in a secret-detection suite.

Self-scan

codeguard (full + diff vs main), validate, doctor, and make check all pass on this branch: 0 fail findings.

Knowledge captured

.claude/knowledge/architecture-boundaries.md updated with: diff-vs-full is centralized in FinalizeSection; TS/JS bypass findingsForFile (repo-wide rules need a separate pass); the literal-gate perf insight and gate-superset invariant; the ScanTargetFiles per-section cache-key gotcha; and the history package layering (must be top-level to avoid an import cycle).

🤖 Generated with Claude Code

alxxjohn and others added 2 commits June 30, 2026 13:38
Add a repository-wide secret/credential scanner to the security suite that
runs in both full scans and diff (PR) scans, and a command to scan git history.

Detection tiers:
- security.hardcoded-credential (fail): known provider formats — AWS, GitHub,
  GitLab, Slack (token + webhook), Stripe, Google, npm, PyPI, Docker Hub,
  SendGrid, Twilio, Azure, DB connection strings with embedded passwords, and
  Bearer tokens — plus config-defined custom_patterns
- security.hardcoded-secret (warn): lower-confidence name-based heuristic
  (reclassified from fail)
- security.high-entropy-string (warn, opt-in): Shannon-entropy detection of
  unknown/random secrets

Config (security_rules.secrets): enabled, allow_paths, allow_patterns,
custom_patterns, entropy. Values are masked in findings; placeholders and
secret references (op://, vault://, $(...)) are skipped automatically.

New `codeguard scan-history` walks git history for secrets that were committed
and later removed (still leaked, need rotation), reusing the same scanner.

Performance/hardening: a fast literal pre-filter gates the per-pattern regexes
(~31x faster default scan, ~15x with entropy); binary/file-size/line-length
caps bound work on untrusted PR input; history parsing avoids silent truncation
on over-long lines.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ssembly

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread internal/codeguard/checks/security/security_secret_patterns.go Fixed
alxxjohn and others added 2 commits June 30, 2026 13:48
Bound the hooks.slack.com webhook pattern with start/non-host-char on both
sides (matching the repo's existing fix for go/regex/missing-regexp-anchor in
quality_ai_style_drift.go) so an arbitrary host cannot precede or follow the
match. The captured group is the webhook URL; detection behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alxxjohn alxxjohn merged commit c1afab7 into main Jun 30, 2026
13 checks passed
@alxxjohn alxxjohn deleted the var-secrets branch June 30, 2026 18:01
alxxjohn added a commit that referenced this pull request Jun 30, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.6.0](v0.5.0...v0.6.0)
(2026-06-30)


### Features

* **security:** hardcoded secret/credential detection + git-history scan
([f2f6c61](f2f6c61))
* **security:** hardcoded secret/credential detection + git-history scan
([#17](#17))
([c1afab7](c1afab7))


### Bug Fixes

* **security:** anchor Slack webhook regex to satisfy CodeQL
([52de05a](52de05a))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants