Repository navigation
Fix partial redaction of Slack and Stripe tokens, plus detection and CLI bugs - #21
Conversation
The slack-access-token rule stopped at the first dash. Slack tokens are dash-separated, with the secret in the last segment, so Redact turned `xoxb-<team id>-<bot id>-<secret>` into `[REDACTED]-<bot id>-<secret>`. Match the numeric IDs and the secret segment that follows them, and keep the single-segment shape as a fallback. The match ends with the secret, so text hyphenated onto a token is left alone, and carries on only over another token glued straight onto it.
The sk_/rk_ rules capped the key body at 32 characters, but current Stripe keys carry 99, so Redact left the last 67 characters of the key in the output and Find reported a truncated value. Raise the ceiling to 247, as TruffleHog does. A body also takes the `sk` of a key glued straight onto it, as the old one already did for 24-character keys, leaving the rest of that key in clear. Carry the match on over it.
A bot token starts with the base64 of the bot's numeric ID, so the ID's leading digit fixes the first two characters. The keyword list had six of the nine possibilities: tokens starting with Mj, Nj or OT (IDs starting with 2, 6 or 9) never reached the regex. The regex took any second character, relying on the keywords to be found somewhere in the input. Make it require one of the nine prefixes itself, so the longer keyword list can't turn more dotted identifiers into tokens.
The password class of the connection-string rules is broad enough to match "[REDACTED]", so Contains(Redact(x)) was true and Find returned the marker for every redacted mongodb / postgres / mysql / redis / amqp URL. Skip spans that are exactly the marker. Contains keeps its allocation-free MatchString path on marker-free input; otherwise it looks at one match more than there are markers.
The reader trusted the stat size, which is 0 for a pipe, so `portcullis-scan <(git diff)` read nothing and exited 0 whatever the content. Read to EOF instead, and enforce -max-size on what was actually read. The stat size still sizes the buffer for small files.
The leading "/" was stripped before deciding whether a pattern is anchored, so `-ignore /secrets` turned into a basename rule and dropped every "secrets" directory in the tree from the scan, not just the one at the root.
The fixture was `ABSK` plus filler, which the Bedrock validator rejects; the case only passed because `SK` + 32 hex characters trips the Twilio rule. Use a body that decodes to a Bedrock envelope.
The card, SSN and IBAN prefilters each scanned the whole input, which on clean text cost five times the Aho-Corasick scan itself. They all start at a digit, so one pass can look for the three shapes at once. Redact and Contains on a 9 kB clean input: 17µs -> 6µs.
`(?i)[A-Z]` also matches `ſ` and the Kelvin sign, which the gate does not take for letters. An IBAN spelled with `ſ` was therefore redacted or not depending on whether the input held another letter-letter- digit-digit run somewhere else. Spell out both cases instead, and check that every shape is implied by its rule's regex.
The README and package doc still described the API as Contains and Redact only, and never mentioned the CLI. The performance section predated the checksum validators and PII rules: refresh its numbers and drop the claims that no longer hold.
The snippet at the top of the README shows a GitHub token with an invalid checksum: since tokens are checksum-validated, Redact returns it untouched and Contains reports false. Show an AWS access key ID instead, and pin the snippet's inputs and results in a test so it can't go stale again.
It sat on compiledMatch, the type declared right above, and still said a clean input runs the PII expressions, which their shape gate now prevents.
A token of 132 characters always has a length that is a multiple of 4.
Line breaks were collapsed in the matched value but not in the path, so a file name holding one split its match over two output lines, the second free to look like a match of its own.
There was a problem hiding this comment.
🔵 Validation recommended
2 open issues
What changed in this PR
- Updates secret detection, stream scanning, ignore patterns, and CLI output.
1 recommended validation
-
Check stream limits after the sniff window
What to check:
Extend the FIFO test with a stream larger than 8 KiB and a secret after the sniff window. Check that a stream at the size limit reports the secret and one byte over the limit is skipped.
Why:
The FIFO cases in
stream_unix_test.gouse short content, whileTestRunSkipsFilesAboveMaxSizeuses a regular file rejected by its stat size. Neither exercises the new read-to-EOF path after the sniff window.
Comment @docker-gordon help to see review commands.
Finding IDs:
f-50f951fac74c1297ec9ff433(slack-numeric-body-swallows-suffix)f-65cb5146f414076b82a78dfa(sanitized-path-collision)
| // leaves the secret itself in clear. It stops there too, so | ||
| // whatever is hyphenated onto a token stays out of it. The | ||
| // second alternative is the original single-segment shape. | ||
| expression: asSecretGroup(`?P<secret>` + chain(`xox[baprs]-`, `xox[baprs]-`, `(?:(?:[0-9]{8,14}-){1,3}[0-9a-zA-Z]{18,128}|[0-9a-zA-Z]{10,48})`)), |
There was a problem hiding this comment.
For a single-segment token such as xoxa-1234567890 followed by -retrycountlonglabel, the first alternative treats the numeric token body as an ID and the label as the final token segment. Redact now removes the label too; previously it stopped at the first hyphen. Preserve adjacent text for this legacy token shape, or define a way to distinguish it from a multi-segment token.
Comment @docker-gordon help to see review commands.
Finding f-50f951fac74c1297ec9ff433
| if rel, relErr := filepath.Rel(root, path); relErr == nil && rel != "." { | ||
| display = rel | ||
| } | ||
| display = sanitizeValue(display) |
There was a problem hiding this comment.
Two files named a\nb and a b produce the same displayed path after this replacement. If both contain a secret at the same position, the output no longer identifies which file needs attention. Escape line breaks in paths instead of replacing them with spaces, and escape the escape character so the original path remains recoverable.
Comment @docker-gordon help to see review commands.
Finding f-65cb5146f414076b82a78dfa

Summary
Known limits
Glued Slack or Stripe tokens are covered, but a malformed, too-short token can still leave a following token's body in clear. Pre-existing: Redact is not fully idempotent when a secret is immediately followed by digits forming an SSN; dates or hashes can still trigger the SSN or IBAN regexes.
Validation
go test ./...,golangci-lint run, andgo mod tidy --diff.go test -race ./....