Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 50 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ credentials, and other secret material in arbitrary text.
```go
import "github.com/docker/portcullis"

clean := portcullis.Redact("Run this with token=ghp_1234567890abcdef1234567890abcdef1234 please.")
// → "Run this with token=[REDACTED] please."
clean := portcullis.Redact("Run this with AWS_ACCESS_KEY_ID=AKIARZPUZDIKQEXAMPLE please.")
// → "Run this with AWS_ACCESS_KEY_ID=[REDACTED] please."

portcullis.Contains("not a secret") // false
portcullis.Contains("token=ghp_1234567890abcdef1234567890abcdef1234") // true
portcullis.Contains("not a secret") // false
portcullis.Contains("AWS_ACCESS_KEY_ID=AKIARZPUZDIKQEXAMPLE") // true
```

## Why
Expand Down Expand Up @@ -47,12 +47,40 @@ const Marker = "[REDACTED]"

func Contains(text string) bool // detect
func Redact(text string) string // scrub
func Find(text string) []Match // locate

// Same as above for callers holding a []byte, without copying it.
// Each Match.Value then aliases b: leave b alone while they are in use.
func ContainsBytes(b []byte) bool
func FindBytes(b []byte) []Match

type Match struct {
Start, End int // byte offsets into the input
Value string // the input between them
}
```

Both functions are safe for concurrent use. The compiled rule set and
All of them are safe for concurrent use. The compiled rule set and
its Aho–Corasick automaton are built once on first call and shared
across goroutines.

## Command line

`portcullis-scan` runs the same detection over a file or a directory
tree and prints one `path:line:col: value` line per secret:

```sh
go install github.com/docker/portcullis/cmd/portcullis-scan@latest

portcullis-scan -ignore 'vendor/**' -ignore '*_test.go' .
```

It exits with 0 when nothing was found, 1 when a secret was found and
2 on a usage error or an unreadable file, so it can gate a CI job.
Its output holds the secrets themselves: where logs are kept, discard
it and go by the exit status. Binary files and files over 10 MiB are
skipped unless `-binary` and `-max-size` say otherwise.

## What it detects

The built-in catalogue covers ~240 patterns spanning:
Expand Down Expand Up @@ -126,28 +154,30 @@ Go version specified in go.mod (Go 1.27.4) on every PR

## Performance

On a typical clean input the cost is dominated by a single linear scan
over the bytes; the regex engine is never invoked. With a secret, only
the rules whose keywords are present run. Both functions allocate
only when text actually changes.
An input with none of the rules' keywords and none of the shapes the
PII rules start from (nine or more digits in a row, single dashes or
spaces aside, or two letters right before two digits) never reaches a
regex and, once the rule set is built, allocates nothing. Its cost is
that of scanning the bytes for keywords (Aho–Corasick), then for
those shapes. Otherwise only the rules whose keyword or shape is
present run their regex.

On an Apple M4 Max scrubbing a 9000-byte clean payload and a
secret-bearing 1.5 KB payload:

```
BenchmarkAhoScanCleanInput-16 257198 4823 ns/op 1865.95 MB/s 0 B/op 0 allocs/op
BenchmarkAhoScanWithKeyword-16 1251084 956.0 ns/op 1636.96 MB/s 0 B/op 0 allocs/op
BenchmarkRedactCleanInput-16 228036 4803 ns/op 0 B/op 0 allocs/op
BenchmarkContainsCleanInput-16 233336 4782 ns/op 0 B/op 0 allocs/op
BenchmarkRedactWithSecret-16 565590 2094 ns/op 1585 B/op 2 allocs/op
BenchmarkContainsWithSecret-16 912943 1312 ns/op 0 B/op 0 allocs/op
BenchmarkAhoScanCleanInput-16 427100 2832 ns/op 3177.96 MB/s 0 B/op 0 allocs/op
BenchmarkAhoScanWithKeyword-16 1760301 683.8 ns/op 2288.76 MB/s 0 B/op 0 allocs/op
BenchmarkRedactCleanInput-16 200227 6089 ns/op 0 B/op 0 allocs/op
BenchmarkContainsCleanInput-16 203072 6016 ns/op 0 B/op 0 allocs/op
BenchmarkRedactWithSecret-16 418003 2835 ns/op 1717 B/op 7 allocs/op
BenchmarkContainsWithSecret-16 745269 1609 ns/op 129 B/op 5 allocs/op
```

The AC scan dominates the clean-input path — `Redact` and `Contains`
add no measurable overhead on top of it because the rule loop
short-circuits on an empty keyword mask. `Contains` allocates zero
bytes even on a secret-bearing input: it only needs the regex
`MatchString`, which uses pooled state machines internally.
`Contains` returns on the first rule that matches, so on a
secret-bearing input it usually skips the shape scan and the rest of
the catalogue. `Redact` has to visit every rule, and allocates for
the matches it finds and the redacted copy.

## Provenance

Expand Down
2 changes: 1 addition & 1 deletion aho.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ package portcullis
// which patterns occurred in a scanned input and which keywords each
// rule subscribes to. Storing it as a small array keeps every test
// branch-free; the cap of len(kwMask)*64 indices (currently 320)
// accommodates today's catalogue of ~285 unique keywords with limited
// accommodates today's catalogue of ~295 unique keywords with limited
// remaining headroom for future rules (an overflow trips a
// deterministic panic in [buildAhoCorasick]).
type kwMask [5]uint64
Expand Down
17 changes: 10 additions & 7 deletions cmd/portcullis-scan/ignore.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ import (
// - "**" matches any run of characters, including separators
// - "[...]" character class (passed through to the regexp engine)
//
// A pattern containing "/" is anchored to the scan root and matched
// against the slash-separated relative path; otherwise it matches
// the basename of every entry. A trailing "/" restricts the rule to
// directories. A "X/**" rule also prunes "X" itself so its subtree
// is not walked.
// A pattern with a "/" at its start or in its middle is anchored to
// the scan root and matched against the slash-separated relative
// path; otherwise it matches the basename of every entry. A trailing
// "/" restricts the rule to directories. A "X/**" rule also prunes
// "X" itself so its subtree is not walked.
type ignoreMatcher struct {
rules []ignoreRule
}
Expand All @@ -39,11 +39,14 @@ func newIgnoreMatcher(patterns []string) (*ignoreMatcher, error) {
}
dirOnly := strings.HasSuffix(p, "/")
p = strings.TrimSuffix(p, "/")
p = strings.TrimPrefix(p, "/")
// A leading "/" anchors a pattern even when it has no other
// separator: "/build" is the root's build, not every build
// in the tree.
p, anchored := strings.CutPrefix(p, "/")
if p == "" {
continue
}
pathMode := strings.Contains(p, "/")
pathMode := anchored || strings.Contains(p, "/")
re, err := compileGlob(p)
if err != nil {
return nil, fmt.Errorf("invalid ignore pattern %q: %w", raw, err)
Expand Down
20 changes: 14 additions & 6 deletions cmd/portcullis-scan/ignore_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ func TestRunIgnoreFlagIsRepeatable(t *testing.T) {
assert.NotContains(t, out, "leaky_test.go")
}

// TestRunIgnoresAnchoredPathPattern: as in gitignore, a "/" at the
// start or in the middle of a pattern anchors it to the scan root.
// Without the anchor "/secrets" would silently drop every "secrets"
// directory in the tree from the scan.
func TestRunIgnoresAnchoredPathPattern(t *testing.T) {
t.Parallel()

Expand All @@ -83,13 +87,13 @@ func TestRunIgnoresAnchoredPathPattern(t *testing.T) {
require.NoError(t, os.WriteFile(filepath.Join(root, "secrets", "g.txt"),
[]byte(githubPAT), 0o644))

var stdout, stderr bytes.Buffer
code := run([]string{"-ignore", "secrets/**", root}, &stdout, &stderr)
for _, pattern := range []string{"secrets/**", "/secrets/**", "/secrets", "/secrets/"} {
var stdout, stderr bytes.Buffer
code := run([]string{"-ignore", pattern, root}, &stdout, &stderr)

assert.Equal(t, exitFound, code)
out := stdout.String()
assert.Contains(t, out, filepath.Join("a", "secrets", "f.txt"))
assert.NotContains(t, out, filepath.Join("secrets", "g.txt"))
assert.Equal(t, exitFound, code, pattern)
assert.Equal(t, filepath.Join("a", "secrets", "f.txt")+":1:1: "+githubPAT+"\n", stdout.String(), pattern)
}
}

func TestRunRejectsInvalidIgnorePattern(t *testing.T) {
Expand Down Expand Up @@ -125,6 +129,10 @@ func TestIgnoreMatcher(t *testing.T) {
{"double star prefix at root", []string{"**/foo.txt"}, "foo.txt", "foo.txt", false, true},
{"single star is single segment", []string{"a/*.go"}, "a/b/c.go", "c.go", false, false},
{"question mark", []string{"?.go"}, "a.go", "a.go", false, true},
{"leading slash at root", []string{"/build"}, "build", "build", true, true},
{"leading slash is anchored", []string{"/build"}, "src/build", "build", true, false},
{"leading slash with star", []string{"/*.env"}, "sub/prod.env", "prod.env", false, false},
{"leading slash dir-only on file", []string{"/build/"}, "build", "build", false, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
Expand Down
76 changes: 48 additions & 28 deletions cmd/portcullis-scan/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
// <path>:<line>:<col>: <value>
//
// Paths are relative to the scan root. Newlines and carriage returns
// inside a matched value are collapsed to spaces so each match
// remains on a single line; this only affects display, not detection.
// inside a path or a matched value are collapsed to spaces so each
// match remains on a single line; this only affects display, not
// detection.
// Files are scanned in parallel; output order matches the walker's
// lexical order, so two runs over the same tree produce identical
// output regardless of worker count or scheduling.
Expand All @@ -24,12 +25,13 @@
// -binary also scan binary files (default: skipped, like ripgrep).
// A file is considered binary when its first 8 KiB
// contains a NUL byte.
// -ignore glob pattern to skip; repeatable. Patterns containing
// '/' are anchored to the scan root, others match
// basenames. '**' matches any run of characters,
// including separators. Trailing '/' restricts the
// rule to directories. Examples: -ignore '*_test.go'
// -ignore 'vendor/**' -ignore 'node_modules/'
// -ignore glob pattern to skip; repeatable. Patterns with a
// '/' at the start or in the middle are anchored to
// the scan root, others match basenames. '**' matches
// any run of characters, including separators.
// Trailing '/' restricts the rule to directories.
// Examples: -ignore '*_test.go' -ignore 'vendor/**'
// -ignore 'node_modules/' -ignore '/build'
//
// Exit codes:
//
Expand All @@ -41,10 +43,12 @@ package main

import (
"bytes"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"math"
"os"
"path/filepath"
"runtime"
Expand Down Expand Up @@ -327,6 +331,7 @@ func scanFileBytes(path, root string, maxSize int64, scanBinary bool) ([]byte, e
if rel, relErr := filepath.Rel(root, path); relErr == nil && rel != "." {
display = rel
}
display = sanitizeValue(display)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low severity Sanitizing paths can make distinct files indistinguishable.

Two files named a\nb and a b produce the same displayed path after this replacement. If both contain a secret at the same position, the output no longer identifies which file needs attention. Escape line breaks in paths instead of replacing them with spaces, and escape the escape character so the original path remains recoverable.

Comment @docker-gordon help to see review commands.

Finding f-65cb5146f414076b82a78dfa

var b strings.Builder
// Matches arrive in ascending Start order, so line/column are
// tracked with one cursor instead of rescanning data from the
Expand Down Expand Up @@ -361,44 +366,59 @@ func readIfScannable(path string, maxSize int64, scanBinary bool) (data []byte,
if err != nil {
return nil, false, err
}
size := st.Size()
if maxSize > 0 && size > maxSize {
if maxSize > 0 && st.Size() > maxSize {
return nil, false, nil
}

if scanBinary {
data, err = io.ReadAll(f)
if err != nil {
return nil, false, err
}
return data, true, nil
// That size is only a hint: a pipe (`<(git diff)`, a FIFO) reports
// 0 and a file being written to can outgrow it. So read to EOF
// rather than to that size, stopping one byte past maxSize so that
// content over the limit is still skipped.
var r io.Reader = f
if maxSize > 0 && maxSize < math.MaxInt64 {
r = io.LimitReader(f, maxSize+1)
}

// Sniff first 8 KiB for a NUL byte before committing to read
// the rest. On large binaries (git packs, archives, prebuilt
// executables) this avoids both the IO and the regex pass.
sniffN := min(int64(binarySniffSize), size)
sniff := make([]byte, sniffN)
if _, err = io.ReadFull(f, sniff); err != nil {
//
// The buffer starts one byte past the stat size, enough to see a
// small file end where it should, and only widens to the full
// window if there turns out to be more.
data = make([]byte, min(st.Size(), binarySniffSize-1)+1)
n, err := io.ReadFull(r, data)
if err == nil && n < binarySniffSize {
data = append(data, make([]byte, binarySniffSize-n)...)
var more int
more, err = io.ReadFull(r, data[n:])
n += more
}
atEOF := errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF)
if err != nil && !atEOF {
return nil, false, err
}
if bytes.IndexByte(sniff, 0) >= 0 {
data = data[:n]
if !scanBinary && bytes.IndexByte(data, 0) >= 0 {
return nil, false, nil
}
if size == sniffN {
return sniff, true, nil
if !atEOF {
rest, err := io.ReadAll(r)
if err != nil {
return nil, false, err
}
data = append(data, rest...)
}
rest, err := io.ReadAll(f)
if err != nil {
return nil, false, err
if maxSize > 0 && int64(len(data)) > maxSize {
return nil, false, nil
}
return append(sniff, rest...), true, nil
return data, true, nil
}

var valueSanitizer = strings.NewReplacer("\r\n", " ", "\n", " ", "\r", " ")

// sanitizeValue collapses CR / LF in v so a multi-line match (e.g. a
// PEM block) stays on a single output line.
// sanitizeValue collapses CR / LF in v, a matched value or a path, so
// a multi-line match (e.g. a PEM block) stays on a single output line.
func sanitizeValue(v string) string {
if !strings.ContainsAny(v, "\r\n") {
return v
Expand Down
21 changes: 21 additions & 0 deletions cmd/portcullis-scan/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"errors"
"os"
"path/filepath"
"runtime"
"strings"
"testing"

Expand Down Expand Up @@ -216,6 +217,26 @@ func TestRunSanitisesMultilineSecrets(t *testing.T) {
assert.NotContains(t, strings.TrimSuffix(out, "\n"), "\n")
}

// TestRunKeepsPathOnOneLine: a file name can hold a line break too,
// and could otherwise pass the rest of its name off as an output line
// of its own.
func TestRunKeepsPathOnOneLine(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("file names cannot contain line breaks")
}

root := t.TempDir()
name := "a\nb.env:1:1: forged"
require.NoError(t, os.WriteFile(filepath.Join(root, name), []byte("TOKEN="+githubPAT+"\n"), 0o644))

var stdout, stderr bytes.Buffer
code := run([]string{root}, &stdout, &stderr)

assert.Equal(t, exitFound, code)
assert.Equal(t, "a b.env:1:1: forged:1:7: "+githubPAT+"\n", stdout.String())
}

func TestRunSkipsBinaryFiles(t *testing.T) {
t.Parallel()

Expand Down
Loading