Skip to content

Commit 3a68ccb

Browse files
TMHSDigitalclaude
andcommitted
Add fuzz targets and skip over-long Wayback lines (#259)
A single CDX line over 1 MiB failed bufio.Scanner and with it the whole Wayback answer. Read lines with a bufio.Reader instead and skip an over-long one (it still counts toward the URL limit). Add tests for userinfo, IPv6 literals, wildcards, malformed URLs, an uppercase domain, a missing final newline and a dropped connection. Add fuzz targets for the Wayback parser (FuzzHostNames), labzone.Parse, validate.NormalizeDomain, wordlist.Normalize and an MCP request line; the AXFR and denial parsers already had them. Fuzzing NormalizeDomain found #281, filed separately. CONTRIBUTING says how to run them. Fixes #259 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 69ab79a commit 3a68ccb

8 files changed

Lines changed: 259 additions & 24 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).
3737
- Passive sources (`-sources`, `-ct`, `Config.Sources`) retry a busy answer (429, 502, 503, 504) or a dropped connection twice, after 2 and 5 seconds or the server's `Retry-After` (at most 30 seconds), within each source's 60-second limit; crt.sh often answers 502 or 503 under load, which used to cost the scan all of its names. A target's sources are now asked at the same time instead of one after another (#271).
3838
- When the Wayback Machine's answer reaches the 200,000-URL or 64 MiB limit, a note says its results were truncated and some hosts may be missing, the report's `sources` entry (and the MCP report's) is marked `truncated`, `explain_stats` mentions it, and the library sends a `NoticeSource` event; the names it did return are still scanned. It used to stop silently (#272).
39+
- A Wayback Machine answer with one archived URL over 1 MiB skips that line instead of failing and losing every Wayback name (#259).
3940
- crt.sh's answer is read one entry at a time instead of decoded whole, so memory follows the names rather than the body. An answer that reaches the 64 MiB limit keeps the names read before the cut and is reported as truncated, like Wayback's, instead of failing and losing every CT name (#251).
4041
- The TUI runs on Bubble Tea v2: `charm.land/bubbletea/v2` v2.1.1, `bubbles/v2` v2.2.1 and `lipgloss/v2` v2.0.6 replace the v1 modules, dropping v1-only helpers that had not had a release in years. Keys, screens and the form work as before; the `go 1.26.0` minimum is unchanged (#188).
4142
- With text output (the default) and no `-show-records` or `-diff`, the CLI keeps only the names of the current target's results for `-permute` and the `-resume` state, plus the few results with a takeover hint, an internal address or a permutation, instead of every result with its records. A simulated 1M-name scan at 100% hits peaks at about 240 MB instead of 750 MB, against 233 MB at 1% hits. A resumed run re-emits the same text. Structured formats still keep the records a resumed run writes back (#183).

‎docs/CONTRIBUTING.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,4 +190,16 @@ Do not hit public resolvers. Use `-simulate` or the in-process DNS server in
190190
NODATA, delay, drop, truncation) and counts every query per name and type.
191191
`internal/dns` tests can also use the table-driven `startTestDNS` wrapper in
192192
`internal/dns/testdns_test.go`. The optional live smoke test is gated on
193-
`SUBENUM_NETWORK_TESTS=1`.
193+
`SUBENUM_NETWORK_TESTS=1`.
194+
195+
Parsers of untrusted input have fuzz targets (`grep -rn "^func Fuzz" .`): the
196+
Wayback answer, lab scenarios, domain and wordlist normalisation, MCP request
197+
lines, and zone-transfer and DNSSEC denial messages. `go test` runs their seed
198+
inputs; to fuzz one, name it and its package:
199+
200+
```bash
201+
go test -run '^$' -fuzz '^FuzzHostNames$' -fuzztime 1m ./internal/passive/
202+
```
203+
204+
A failing input is saved under the package's `testdata/fuzz/`; commit it with
205+
the fix so it stays a regression test.

‎internal/labzone/fuzz_test.go‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
package labzone
2+
3+
import (
4+
"os"
5+
"path/filepath"
6+
"strings"
7+
"testing"
8+
)
9+
10+
// FuzzParse: any scenario text parses or fails without panicking, and a
11+
// zone that parses answers any name (#259).
12+
func FuzzParse(f *testing.F) {
13+
labs, _ := filepath.Glob(filepath.Join("..", "..", "examples", "labs", "*.zone"))
14+
for _, p := range labs {
15+
if b, err := os.ReadFile(p); err == nil {
16+
f.Add(string(b))
17+
}
18+
}
19+
f.Add("$DELAY 20ms\n$REFUSE-ABOVE 50\n@ A 192.0.2.1\nmail MX 10 mx1\n@ TXT \"v=spf1 -all\"\nslow TIMEOUT\n*.dev A 192.0.2.9\n")
20+
f.Add("www CNAME www\nx NS ns1.gone.example.\n$DELAY -1s\nbad AAAA 192.0.2.1\n")
21+
f.Fuzz(func(_ *testing.T, text string) {
22+
z, err := Parse(strings.NewReader(text))
23+
if err != nil {
24+
return
25+
}
26+
for _, name := range []string{"", "www", "a.dev", "slow", "mail", "nope.x"} {
27+
_ = z.Answer(name, "lab.example")
28+
}
29+
})
30+
}

‎internal/mcp/fuzz_test.go‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
package mcp
2+
3+
import (
4+
"bytes"
5+
"context"
6+
"encoding/json"
7+
"strings"
8+
"testing"
9+
"time"
10+
)
11+
12+
// FuzzDispatch: any input line gets at most a JSON reply per request, and
13+
// never panics or hangs the session (#259).
14+
func FuzzDispatch(f *testing.F) {
15+
for _, s := range []string{
16+
`{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18"}}`,
17+
`{"jsonrpc":"2.0","id":"a","method":"tools/call","params":{"name":"echo","arguments":{"x":1}}}`,
18+
`{"jsonrpc":"2.0","method":"notifications/cancelled","params":{"requestId":1}}`,
19+
`{"jsonrpc":"2.0","id":2,"method":"tools/list"}`,
20+
`[{"jsonrpc":"2.0","id":3,"method":"ping"}]`,
21+
`{"id":null}`, `{`, `"x"`, ``,
22+
} {
23+
f.Add(s)
24+
}
25+
f.Fuzz(func(t *testing.T, line string) {
26+
if strings.ContainsAny(line, "\r\n") {
27+
return // one line per call
28+
}
29+
var out bytes.Buffer
30+
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
31+
defer cancel()
32+
_ = (&Server{Tools: []Tool{echoTool()}}).Serve(ctx, strings.NewReader(line+"\n"), &out)
33+
if ctx.Err() != nil {
34+
t.Fatalf("session did not end after input %q", line)
35+
}
36+
for _, reply := range strings.Split(strings.TrimSpace(out.String()), "\n") {
37+
if reply != "" && !json.Valid([]byte(reply)) {
38+
t.Errorf("input %q: reply %q is not JSON", line, reply)
39+
}
40+
}
41+
})
42+
}

‎internal/passive/passive_test.go‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import (
1111
"time"
1212

1313
"github.com/TMHSDigital/subenum/internal/retryhttp"
14+
"github.com/TMHSDigital/subenum/internal/wordlist"
1415
)
1516

1617
// Failing test servers are retried at once, not after a real backoff.
@@ -157,3 +158,63 @@ func TestWaybackError(t *testing.T) {
157158
t.Errorf("err = %v, want the 503", err)
158159
}
159160
}
161+
162+
// TestWaybackEdgeCases covers #259: an over-long line is skipped rather than
163+
// failing the answer, and odd hosts are handled.
164+
func TestWaybackEdgeCases(t *testing.T) {
165+
body := strings.Join([]string{
166+
"http://" + strings.Repeat("a", 2<<20) + ".example.com/", // 2 MiB: skipped
167+
"http://www.example.com/",
168+
"http://user:pw@login.example.com/", // userinfo
169+
"http://[2001:db8::1]/x", // IPv6 literal
170+
"http://*.wild.example.com/", // wildcard: not a name
171+
"http://%zz.example.com/", // malformed URL
172+
"https://Shop.EXAMPLE.com.:443/cart", // case and trailing dot
173+
"http://evil.example.com.attacker.test/", // a lookalike outside the domain
174+
"no-newline.example.com", // last line without a newline
175+
}, "\n")
176+
names, lines, err := hostNames(strings.NewReader(body), "EXAMPLE.com.")
177+
if err != nil {
178+
t.Fatal(err)
179+
}
180+
if want := []string{"login", "no-newline", "shop", "www"}; !slices.Equal(names, want) {
181+
t.Errorf("names = %q, want %q", names, want)
182+
}
183+
if lines != 9 {
184+
t.Errorf("lines = %d, want 9 (the skipped line counts toward the URL limit)", lines)
185+
}
186+
}
187+
188+
// TestWaybackTransportError: a connection dropped mid-answer is reported,
189+
// not taken for an empty index.
190+
func TestWaybackTransportError(t *testing.T) {
191+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
192+
w.Header().Set("Content-Length", "1000")
193+
_, _ = w.Write([]byte("http://www.example.com/\n"))
194+
}))
195+
defer srv.Close()
196+
if names, err := (Wayback{BaseURL: srv.URL}).Names(context.Background(), "example.com"); err == nil {
197+
t.Errorf("a short body was accepted: names %q", names)
198+
}
199+
}
200+
201+
// FuzzHostNames: any answer parses without panicking, and every name it
202+
// gives is a valid entry under the domain (#259).
203+
func FuzzHostNames(f *testing.F) {
204+
f.Add("http://www.example.com/\nhttps://u@API.example.com:8443/x\n", "example.com")
205+
f.Add("[::1]\n*.example.com\n\n\r\n%zz", "Example.COM.")
206+
f.Fuzz(func(t *testing.T, body, domain string) {
207+
names, _, err := hostNames(strings.NewReader(body), domain)
208+
if err != nil {
209+
t.Fatalf("reading from memory failed: %v", err)
210+
}
211+
if !slices.IsSorted(names) {
212+
t.Errorf("names not sorted: %q", names)
213+
}
214+
for _, n := range names {
215+
if norm, ok := wordlist.Normalize(n); !ok || norm != n {
216+
t.Errorf("name %q is not a normalised entry", n)
217+
}
218+
}
219+
})
220+
}

‎internal/passive/wayback.go‎

Lines changed: 58 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package passive
33
import (
44
"bufio"
55
"context"
6+
"errors"
67
"fmt"
78
"io"
89
"net/http"
@@ -93,38 +94,32 @@ func (w Wayback) Names(ctx context.Context, domain string) ([]string, error) {
9394
return names, nil
9495
}
9596

97+
// waybackMaxLine bounds one archived URL. A longer line is skipped, not
98+
// fatal: it used to fail the whole answer and lose every name (#259).
99+
const waybackMaxLine = 1 << 20
100+
96101
// hostNames reads one archived URL per line and returns the hosts under
97102
// domain, relative to it, and how many lines it read.
98103
func hostNames(r io.Reader, domain string) ([]string, int, error) {
99104
domain = strings.ToLower(strings.TrimSuffix(domain, "."))
100105
set := map[string]bool{}
101-
sc := bufio.NewScanner(r)
102-
sc.Buffer(make([]byte, 64<<10), 1<<20)
106+
br := bufio.NewReaderSize(r, 64<<10)
103107
lines := 0
104-
for sc.Scan() {
105-
line := strings.TrimSpace(sc.Text())
106-
if line == "" {
107-
continue
108-
}
109-
lines++
110-
if !strings.Contains(line, "://") {
111-
line = "http://" + line
108+
for {
109+
raw, tooLong, err := readURLLine(br)
110+
if err != nil && !errors.Is(err, io.EOF) {
111+
return nil, lines, err
112112
}
113-
u, err := url.Parse(line)
114-
if err != nil {
115-
continue
113+
line := strings.TrimSpace(string(raw))
114+
if tooLong {
115+
lines++
116+
} else if line != "" {
117+
lines++
118+
addHost(set, line, domain)
116119
}
117-
host := strings.ToLower(strings.TrimSuffix(u.Hostname(), "."))
118-
rel, ok := strings.CutSuffix(host, "."+domain)
119-
if !ok || rel == "" {
120-
continue
120+
if err != nil { // io.EOF, after the last line
121+
break
121122
}
122-
if norm, ok := wordlist.Normalize(rel); ok {
123-
set[norm] = true
124-
}
125-
}
126-
if err := sc.Err(); err != nil {
127-
return nil, lines, err
128123
}
129124
out := make([]string, 0, len(set))
130125
for n := range set {
@@ -133,3 +128,43 @@ func hostNames(r io.Reader, domain string) ([]string, int, error) {
133128
sort.Strings(out)
134129
return out, lines, nil
135130
}
131+
132+
// readURLLine reads one line, without its newline. A line over
133+
// waybackMaxLine is read to its end and returned empty with tooLong set. The
134+
// last line may lack a newline: it comes with io.EOF.
135+
func readURLLine(br *bufio.Reader) (line []byte, tooLong bool, err error) {
136+
for {
137+
chunk, err := br.ReadSlice('\n')
138+
if !tooLong {
139+
if len(line)+len(chunk) > waybackMaxLine+1 { // +1: the newline
140+
line, tooLong = nil, true
141+
} else {
142+
line = append(line, chunk...)
143+
}
144+
}
145+
if errors.Is(err, bufio.ErrBufferFull) {
146+
continue
147+
}
148+
return line, tooLong, err
149+
}
150+
}
151+
152+
// addHost adds the host of one archived URL to set, relative to domain, if
153+
// it is a valid name under it.
154+
func addHost(set map[string]bool, line, domain string) {
155+
if !strings.Contains(line, "://") {
156+
line = "http://" + line
157+
}
158+
u, err := url.Parse(line)
159+
if err != nil {
160+
return
161+
}
162+
host := strings.ToLower(strings.TrimSuffix(u.Hostname(), "."))
163+
rel, ok := strings.CutSuffix(host, "."+domain)
164+
if !ok || rel == "" {
165+
return
166+
}
167+
if norm, ok := wordlist.Normalize(rel); ok {
168+
set[norm] = true
169+
}
170+
}

‎internal/validate/fuzz_test.go‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
package validate
2+
3+
import "testing"
4+
5+
// FuzzNormalizeDomain: any input normalises or fails without panicking, and
6+
// an accepted domain passes Domain and normalises to itself (#259). Fuzzing
7+
// it finds #281 (a kept trailing dot) until that is fixed.
8+
func FuzzNormalizeDomain(f *testing.F) {
9+
for _, s := range []string{"example.com", " HTTPS://Example.COM:8443/path?q ", "*.example.com.", "bücher.example", "user@example.com", "[::1]:53", "a..b", "xn--bcher-kva.example", "example.com/#x"} {
10+
f.Add(s)
11+
}
12+
f.Fuzz(func(t *testing.T, input string) {
13+
domain, _, err := NormalizeDomain(input)
14+
if err != nil {
15+
return
16+
}
17+
if derr := Domain(domain); derr != nil {
18+
t.Errorf("NormalizeDomain(%q) = %q, which Domain rejects: %v", input, domain, derr)
19+
}
20+
if again, _, err := NormalizeDomain(domain); err != nil || again != domain {
21+
t.Errorf("NormalizeDomain(%q) = %q, %v; want it to be stable", domain, again, err)
22+
}
23+
})
24+
}

‎internal/wordlist/fuzz_test.go‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
package wordlist
2+
3+
import (
4+
"strings"
5+
"testing"
6+
)
7+
8+
// FuzzNormalize: any line normalises without panicking, and an accepted
9+
// entry is lowercase, has no surrounding space or trailing dot, and
10+
// normalises to itself (#259).
11+
func FuzzNormalize(f *testing.F) {
12+
for _, s := range []string{"www", " API. ", "dev.api", "#comment", "*.x", "a..b", strings.Repeat("a", 64), "xn--bcher-kva", "\xef\xbb\xbfwww"} {
13+
f.Add(s)
14+
}
15+
f.Fuzz(func(t *testing.T, line string) {
16+
entry, ok := Normalize(line)
17+
if !ok {
18+
if entry != "" {
19+
t.Errorf("Normalize(%q) rejected the line but returned %q", line, entry)
20+
}
21+
return
22+
}
23+
if entry != strings.ToLower(entry) || entry != strings.TrimSpace(entry) || strings.HasSuffix(entry, ".") || entry == "" {
24+
t.Errorf("Normalize(%q) = %q, not a canonical entry", line, entry)
25+
}
26+
if again, ok := Normalize(entry); !ok || again != entry {
27+
t.Errorf("Normalize(%q) = %q, %v; want it to be stable", entry, again, ok)
28+
}
29+
})
30+
}

0 commit comments

Comments
 (0)