Skip to content

Fix partial redaction of Slack and Stripe tokens, plus detection and CLI bugs - #21

Merged
dgageot merged 14 commits into
mainfrom
worktree-board-999f855d66a83221
Nov 20, 2026
Merged

dgageot merged 14 commits into
mainfrom
worktree-board-999f855d66a83221

Conversation

@dgageot

@dgageot dgageot commented Nov 20, 2026

Copy link
Copy Markdown
Member

Summary

  • Redact complete Slack access tokens and current-length Stripe keys; detect all nine Discord bot token prefixes, exclude the redaction marker as a secret, and restrict IBAN letters to ASCII.
  • Scan pipes and FIFOs to EOF while enforcing the size limit, anchor leading-slash ignore patterns to the scan root, and keep paths with line breaks on one output line.
  • Merge the three PII prefilters into one pass (about 17 µs to 6 µs for Redact on 9 kB of clean input), with differential tests and a fuzz target. Fix the README example, expand API and CLI documentation, and correct the Bedrock test.

Known limits

Glued Slack or Stripe tokens are covered, but a malformed, too-short token can still leave a following token's body in clear. Pre-existing: Redact is not fully idempotent when a secret is immediately followed by digits forming an SSN; dates or hashes can still trigger the SSN or IBAN regexes.

Validation

  • Each commit: build, go test ./..., golangci-lint run, and go mod tidy --diff.
  • HEAD: go test -race ./....

The slack-access-token rule stopped at the first dash. Slack tokens are
dash-separated, with the secret in the last segment, so Redact turned
`xoxb-<team id>-<bot id>-<secret>` into `[REDACTED]-<bot id>-<secret>`.

Match the numeric IDs and the secret segment that follows them, and
keep the single-segment shape as a fallback. The match ends with the
secret, so text hyphenated onto a token is left alone, and carries on
only over another token glued straight onto it.
The sk_/rk_ rules capped the key body at 32 characters, but current
Stripe keys carry 99, so Redact left the last 67 characters of the key
in the output and Find reported a truncated value. Raise the ceiling
to 247, as TruffleHog does.

A body also takes the `sk` of a key glued straight onto it, as the old
one already did for 24-character keys, leaving the rest of that key in
clear. Carry the match on over it.
A bot token starts with the base64 of the bot's numeric ID, so the ID's
leading digit fixes the first two characters. The keyword list had six
of the nine possibilities: tokens starting with Mj, Nj or OT (IDs
starting with 2, 6 or 9) never reached the regex.

The regex took any second character, relying on the keywords to be
found somewhere in the input. Make it require one of the nine prefixes
itself, so the longer keyword list can't turn more dotted identifiers
into tokens.
The password class of the connection-string rules is broad enough to
match "[REDACTED]", so Contains(Redact(x)) was true and Find returned
the marker for every redacted mongodb / postgres / mysql / redis / amqp
URL. Skip spans that are exactly the marker.

Contains keeps its allocation-free MatchString path on marker-free
input; otherwise it looks at one match more than there are markers.
The reader trusted the stat size, which is 0 for a pipe, so
`portcullis-scan <(git diff)` read nothing and exited 0 whatever the
content. Read to EOF instead, and enforce -max-size on what was
actually read. The stat size still sizes the buffer for small files.
The leading "/" was stripped before deciding whether a pattern is
anchored, so `-ignore /secrets` turned into a basename rule and
dropped every "secrets" directory in the tree from the scan, not just
the one at the root.
The fixture was `ABSK` plus filler, which the Bedrock validator
rejects; the case only passed because `SK` + 32 hex characters trips
the Twilio rule. Use a body that decodes to a Bedrock envelope.
The card, SSN and IBAN prefilters each scanned the whole input, which
on clean text cost five times the Aho-Corasick scan itself. They all
start at a digit, so one pass can look for the three shapes at once.

Redact and Contains on a 9 kB clean input: 17µs -> 6µs.
`(?i)[A-Z]` also matches `ſ` and the Kelvin sign, which the gate does
not take for letters. An IBAN spelled with `ſ` was therefore redacted
or not depending on whether the input held another letter-letter-
digit-digit run somewhere else. Spell out both cases instead, and
check that every shape is implied by its rule's regex.
The README and package doc still described the API as Contains and
Redact only, and never mentioned the CLI. The performance section
predated the checksum validators and PII rules: refresh its numbers
and drop the claims that no longer hold.
The snippet at the top of the README shows a GitHub token with an
invalid checksum: since tokens are checksum-validated, Redact returns
it untouched and Contains reports false. Show an AWS access key ID
instead, and pin the snippet's inputs and results in a test so it
can't go stale again.
It sat on compiledMatch, the type declared right above, and still said
a clean input runs the PII expressions, which their shape gate now
prevents.
A token of 132 characters always has a length that is a multiple of 4.
Line breaks were collapsed in the matched value but not in the path,
so a file name holding one split its match over two output lines, the
second free to look like a match of its own.
@dgageot
dgageot requested a review from a team as a code owner November 20, 2026 17:51

@docker-gordon docker-gordon Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Validation recommended

2 open issues
What changed in this PR
  • Updates secret detection, stream scanning, ignore patterns, and CLI output.
1 recommended validation
  • Check stream limits after the sniff window

    What to check:

    Extend the FIFO test with a stream larger than 8 KiB and a secret after the sniff window. Check that a stream at the size limit reports the secret and one byte over the limit is skipped.

    Why:

    The FIFO cases in stream_unix_test.go use short content, while TestRunSkipsFilesAboveMaxSize uses a regular file rejected by its stat size. Neither exercises the new read-to-EOF path after the sniff window.

Comment @docker-gordon help to see review commands.

Finding IDs:
  • f-50f951fac74c1297ec9ff433 (slack-numeric-body-swallows-suffix)
  • f-65cb5146f414076b82a78dfa (sanitized-path-collision)

Comment thread rules.go
// leaves the secret itself in clear. It stops there too, so
// whatever is hyphenated onto a token stays out of it. The
// second alternative is the original single-segment shape.
expression: asSecretGroup(`?P<secret>` + chain(`xox[baprs]-`, `xox[baprs]-`, `(?:(?:[0-9]{8,14}-){1,3}[0-9a-zA-Z]{18,128}|[0-9a-zA-Z]{10,48})`)),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low severity Numeric single-segment Slack tokens can cause adjacent text to be redacted.

For a single-segment token such as xoxa-1234567890 followed by -retrycountlonglabel, the first alternative treats the numeric token body as an ID and the label as the final token segment. Redact now removes the label too; previously it stopped at the first hyphen. Preserve adjacent text for this legacy token shape, or define a way to distinguish it from a multi-segment token.

Comment @docker-gordon help to see review commands.

Finding f-50f951fac74c1297ec9ff433

if rel, relErr := filepath.Rel(root, path); relErr == nil && rel != "." {
display = rel
}
display = sanitizeValue(display)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low severity Sanitizing paths can make distinct files indistinguishable.

Two files named a\nb and a b produce the same displayed path after this replacement. If both contain a secret at the same position, the output no longer identifies which file needs attention. Escape line breaks in paths instead of replacing them with spaces, and escape the escape character so the original path remains recoverable.

Comment @docker-gordon help to see review commands.

Finding f-65cb5146f414076b82a78dfa

@dgageot
dgageot merged commit a805c41 into main Nov 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant