Skip to content

Commit 87cc6b6

Browse files
committed
Cut three allocations that grow with large wordlists (#321)
Merging passive-source, zone-transfer or permutation names built a map holding every wordlist entry, per target; the lab DNS server kept a per-name counter that only tests read; and each CNAME query allocated a 64 KB read buffer. wordlist.NotIn now puts the few new names in the set and makes one pass over the entries, and passive.Merge, the CLI, the TUI and pkg/subenum use it. Per-name counting in internal/dnsserver is opt-in (Options.CountNames, set by internal/dnstest). A CNAME reply over UDP is read into a buffer the size of the advertised EDNS payload, and over TCP into exactly the announced length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Fixes #321
1 parent 4601644 commit 87cc6b6

18 files changed

Lines changed: 241 additions & 73 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6161
- The wordlist is normalized once per run instead of once per `-dL` target: `wordlist.Prepare` normalizes the lines and `Prepared.For` applies only each target's 253-character limit. On a 1M-line list, normalizing takes about 0.5 s instead of 0.8 s with almost no allocations, and each further target about 0.5 ms. Results and counts are unchanged (#182).
6262
- Tests no longer flake under load: lab end-to-end runs use a 5 s timeout with 3 attempts and fail with a clear message if the lab 2 wildcard probe times out, the resolver-pool spread test counts lookups per member instead of timing the scan (the timing view is now `BenchmarkPoolSpreadsLoad`), and CI runs tests in shuffled order (#167).
6363
- The names whose lookup timed out, was refused or failed are kept only when something reads them: for the `-inconclusive` file, and until the `-retry-inconclusive` pass has run. A scan without either only counts them, so under `-no-abort` or `-auto-rate` against a failing resolver memory no longer grows with every failed lookup (about 1 GB at 10M failures). When they are kept, a wordlist entry costs its index rather than its name. Counts, the report and `pkg/subenum`'s `Stats.Inconclusive` are unchanged; the `-resume` state lists the names only with `-inconclusive` (#300).
64+
- Less memory on large wordlists and busy scans: `-sources`, `-ct`, `-axfr` and `-permute` check their names against the wordlist without building a set as large as it (about 200 MB per feature and target on a 5M-line list), a `-simulate-zone` lab scan no longer keeps a count per queried name (about 170 MB on a 1M-line list), and a `CNAME` lookup reads a UDP answer into a buffer sized for the 1232-byte EDNS payload it advertises instead of 64 KB. Results are unchanged (#321).
6465

6566
### Fixed
6667
- With `-format csv`, each row reaches stdout as its result is found, as with the other formats, instead of when 4 KB had built up or the run ended; a pipe sees live rows and a force quit loses none (#319).

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ internal/dnstest/dnstest.go - Start, StartWith, StartTLS: test helpers around
107107
internal/validate/validate.go - DNSServer, Domain, NormalizeDomain, DefaultDNSServer
108108
internal/validate/punycode.go - IDN to punycode (A-label) conversion
109109
internal/cliargs/cliargs.go - Parse: flags anywhere on the command line, "--" terminator
110-
internal/wordlist/reader.go - ReadLines, ReadLinesFrom, Normalize, Build, Prepare / Prepared.For, LoadWordlist
110+
internal/wordlist/reader.go - ReadLines, ReadLinesFrom, Normalize, Build, Prepare / Prepared.For, NotIn, LoadWordlist
111111
internal/tui/model.go - Root Bubble Tea model (form -> scan state machine)
112112
internal/tui/form.go - Config form screen (textinput fields + toggles)
113113
internal/tui/scan_view.go - Live results screen (viewport + progress bar)

‎internal/cli/ct.go‎

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -105,14 +105,12 @@ func sourceByName(sources []targetSource, name string) targetSource {
105105

106106
// candidates gathers a target's candidate entries and -permute seeds as the
107107
// passive sources and the zone transfer add names to the wordlist's (#248).
108-
// The set of entries is built once, at the first names, however many
109-
// sources follow; it used to be rebuilt per source, a map as large as the
110-
// wordlist each time.
108+
// No map as large as the wordlist is built: each source's names are checked
109+
// against the entries in one pass (#321).
111110
type candidates struct {
112111
target string
113112
entries []string // the wordlist's, then the new names in order
114113
seeds map[string]struct{} // -seeds names plus every added name, full
115-
have map[string]bool // entries, once a source has names
116114
owned bool // seeds is this target's copy, not the shared -seeds map
117115

118116
// Provenance (#273): from has every name a source proposed, as a bit per
@@ -142,27 +140,27 @@ func (c *candidates) add(source string, names []string) int {
142140
if len(names) == 0 {
143141
return 0
144142
}
145-
if c.have == nil {
146-
c.have = make(map[string]bool, len(c.entries)+len(names))
147-
for _, e := range c.entries {
148-
c.have[e] = true
149-
}
143+
if c.from == nil {
150144
c.labels, c.from = []string{sourceWordlist}, map[string]uint8{}
151145
}
152-
// Before the merge, while have still tells a wordlist entry from a name
153-
// only an earlier source had (those are in from).
146+
var added []string
147+
c.entries, added = passive.Merge(c.entries, names, c.target)
148+
fresh := make(map[string]struct{}, len(added))
149+
for _, n := range added {
150+
fresh[n] = struct{}{}
151+
}
152+
// A proposed name that was already an entry and that no earlier source
153+
// had (those are in from) is a wordlist entry.
154154
bit := uint8(1) << len(c.labels)
155155
c.labels = append(c.labels, source)
156156
proposed, _, _ := wordlist.Build(names, c.target)
157157
for _, n := range proposed {
158158
mask, seen := c.from[n]
159-
if !seen && c.have[n] {
159+
if _, isNew := fresh[n]; !seen && !isNew {
160160
mask = 1
161161
}
162162
c.from[n] = mask | bit
163163
}
164-
var added int
165-
c.entries, added = passive.Merge(c.entries, c.have, names, c.target)
166164
if !c.owned {
167165
own := make(map[string]struct{}, len(c.seeds)+len(names))
168166
for s := range c.seeds {
@@ -173,7 +171,7 @@ func (c *candidates) add(source string, names []string) int {
173171
for _, n := range names {
174172
c.seeds[n+"."+c.target] = struct{}{}
175173
}
176-
return added
174+
return len(added)
177175
}
178176

179177
// sourcesOf names what proposed a main-pass result (#273). A name no source

‎internal/cli/ct_test.go‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,13 @@ package cli
33
import (
44
"encoding/csv"
55
"encoding/json"
6+
"fmt"
67
"maps"
78
"net/http"
89
"net/http/httptest"
910
"os"
1011
"path/filepath"
12+
"runtime"
1113
"slices"
1214
"strings"
1315
"testing"
@@ -220,6 +222,45 @@ func TestCandidates(t *testing.T) {
220222
}
221223
}
222224

225+
// TestCandidatesNoWordlistSizedMap covers #321: merging a few source names
226+
// or permutations into a large wordlist checks them against the entries
227+
// without building a map as large as the wordlist.
228+
func TestCandidatesNoWordlistSizedMap(t *testing.T) {
229+
const size = 200_000 // a map of these would take several MB
230+
entries := make([]string, size, size+64)
231+
for i := range entries {
232+
entries[i] = fmt.Sprintf("w%d", i)
233+
}
234+
allocated := func(fn func()) uint64 {
235+
var before, after runtime.MemStats
236+
runtime.ReadMemStats(&before)
237+
fn()
238+
runtime.ReadMemStats(&after)
239+
return after.TotalAlloc - before.TotalAlloc
240+
}
241+
const limit = 1 << 20
242+
243+
c := &candidates{target: "example.com", entries: entries}
244+
var added int
245+
if b := allocated(func() { added = c.add("ct", []string{"api", "w7", "dev"}) }); b > limit {
246+
t.Errorf("candidates.add allocated %d bytes, want at most %d", b, limit)
247+
}
248+
if added != 2 || len(c.entries) != size+2 {
249+
t.Errorf("added %d (%d entries), want 2 (%d)", added, len(c.entries), size+2)
250+
}
251+
if got := c.sourcesOf("w7.example.com"); !slices.Equal(got, []string{"wordlist", "ct"}) {
252+
t.Errorf("sources of w7 = %v, want [wordlist ct]", got)
253+
}
254+
255+
var perms []string
256+
if b := allocated(func() { perms, _ = permutationCandidates("example.com", entries, []string{"api"}) }); b > limit {
257+
t.Errorf("permutationCandidates allocated %d bytes, want at most %d", b, limit)
258+
}
259+
if len(perms) == 0 || slices.Contains(perms, "api") {
260+
t.Errorf("permutations = %v", perms)
261+
}
262+
}
263+
223264
// TestE2ELabRefusesSources: passive sources reach external services, so lab
224265
// mode refuses them, and simulation ignores them with a note.
225266
func TestE2ELabRefusesSources(t *testing.T) {

‎internal/cli/dryrun_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ func TestDryRunPreflight(t *testing.T) {
8181
{"servfail", dnsserver.ServFail, exitFailure, "error", false},
8282
} {
8383
t.Run(tc.name, func(t *testing.T) {
84-
srv, err := dnsserver.Listen(dnsserver.Options{}, func(dnsserver.Query) dnsserver.Reply { return tc.reply })
84+
srv, err := dnsserver.Listen(dnsserver.Options{CountNames: true}, func(dnsserver.Query) dnsserver.Reply { return tc.reply })
8585
if err != nil {
8686
t.Fatal(err)
8787
}

‎internal/cli/permute.go‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,16 +69,15 @@ func mainPassNames(results []output.Result) []string {
6969
// to target) that are not wordlist entries or seeds themselves, in a stable
7070
// order. Candidates whose full name would pass the 253-character limit are
7171
// dropped and counted in tooLong: they would otherwise fail as "other" and
72-
// count against the reliability guard (#111).
72+
// count against the reliability guard (#111). The permutations are checked
73+
// against entries in one pass, with no map as large as the wordlist (#321).
7374
func permutationCandidates(target string, entries, rel []string) (candidates []string, tooLong int) {
74-
skip := make(map[string]bool, len(entries)+len(rel))
75-
for _, e := range entries {
76-
skip[e] = true
77-
}
75+
skip := make(map[string]bool, len(rel))
7876
for _, s := range rel {
7977
skip[s] = true
8078
}
81-
candidates, _, tooLong = wordlist.Build(permute.Generate(rel, permute.Words, skip), target)
79+
generated := wordlist.NotIn(permute.Generate(rel, permute.Words, skip), entries)
80+
candidates, _, tooLong = wordlist.Build(generated, target)
8281
return candidates, tooLong
8382
}
8483

‎internal/dns/cname.go‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,13 @@ func cnameExchange(ctx context.Context, resolver *net.Resolver, network string,
108108
if _, err := conn.Write(query); err != nil {
109109
return nil, err
110110
}
111-
buf := make([]byte, 65535)
111+
// A UDP answer is at most the payload the query advertises, as in Go's
112+
// resolver; a stream answer gets exactly the length its prefix gives. No
113+
// 64 KiB buffer per lookup (#321).
114+
var buf []byte
115+
if packet {
116+
buf = make([]byte, ednsPayload)
117+
}
112118
for {
113119
var reply []byte
114120
if packet {
@@ -122,7 +128,7 @@ func cnameExchange(ctx context.Context, resolver *net.Resolver, network string,
122128
if _, err := io.ReadFull(conn, lenb[:]); err != nil {
123129
return nil, err
124130
}
125-
reply = buf[:binary.BigEndian.Uint16(lenb[:])]
131+
reply = make([]byte, binary.BigEndian.Uint16(lenb[:]))
126132
if _, err := io.ReadFull(conn, reply); err != nil {
127133
return nil, err
128134
}
@@ -145,6 +151,9 @@ func cnameExchange(ctx context.Context, resolver *net.Resolver, network string,
145151
}
146152
}
147153

154+
// ednsPayload is the UDP answer size cnameQuery advertises, Go's resolver's.
155+
const ednsPayload = 1232
156+
148157
// cnameQuery builds a recursive CNAME query with an EDNS(0) OPT record
149158
// advertising 1232-byte UDP answers, as Go's resolver sends.
150159
func cnameQuery(id uint16, qname dnsmessage.Name) ([]byte, error) {
@@ -160,7 +169,7 @@ func cnameQuery(id uint16, qname dnsmessage.Name) ([]byte, error) {
160169
return nil, err
161170
}
162171
var opt dnsmessage.ResourceHeader
163-
if err := opt.SetEDNS0(1232, dnsmessage.RCodeSuccess, false); err != nil {
172+
if err := opt.SetEDNS0(ednsPayload, dnsmessage.RCodeSuccess, false); err != nil {
164173
return nil, err
165174
}
166175
if err := b.OPTResource(opt, dnsmessage.OPTResource{}); err != nil {

‎internal/dns/cname_test.go‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"fmt"
77
"io"
88
"net"
9+
"runtime"
910
"strings"
1011
"sync/atomic"
1112
"testing"
@@ -403,6 +404,39 @@ func TestCNAMEInvalidNameSendsNoQuery(t *testing.T) {
403404
}
404405
}
405406

407+
// TestCNAMEReadBuffer covers #321: a CNAME lookup reads a UDP answer into a
408+
// buffer sized for the advertised EDNS(0) payload and a TCP answer into its
409+
// announced length, not a 64 KiB buffer per lookup.
410+
func TestCNAMEReadBuffer(t *testing.T) {
411+
srv := dnstest.Start(t, func(q dnstest.Query) dnstest.Reply {
412+
if q.Name == "big.example.com" {
413+
return dnstest.Reply{CNAME: "edge.example.net", Truncate: true} // UDP, then TCP
414+
}
415+
return dnstest.Reply{CNAME: "ghost.example.net"}
416+
})
417+
r := NewResolver(time.Second, srv.Addr)
418+
for _, name := range []string{"blog.example.com.", "big.example.com."} {
419+
lookup := func() {
420+
if _, err := lookupCNAME(contextWithTimeout(t, 5*time.Second), r, name); err != nil {
421+
t.Fatalf("%s: %v", name, err)
422+
}
423+
}
424+
lookup() // warm up
425+
const runs = 20
426+
var before, after runtime.MemStats
427+
runtime.ReadMemStats(&before)
428+
for range runs {
429+
lookup()
430+
}
431+
runtime.ReadMemStats(&after)
432+
// The client and the in-process server together; a 64 KiB read
433+
// buffer alone would be far over this.
434+
if per := (after.TotalAlloc - before.TotalAlloc) / runs; per > 16<<10 {
435+
t.Errorf("%s: %d bytes allocated per lookup, want at most 16 KiB", name, per)
436+
}
437+
}
438+
}
439+
406440
// writeCountingConn counts the writes on a connection.
407441
type writeCountingConn struct {
408442
net.Conn

‎internal/dnsserver/server.go‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,10 @@ type Options struct {
8484
Serial bool
8585
// IPv6 listens on [::1] instead of 127.0.0.1.
8686
IPv6 bool
87+
// CountNames keeps a count per name and type for QueriesFor. Tests set
88+
// it (internal/dnstest does); a lab scan leaves it off, since the map
89+
// would grow with every candidate (#321).
90+
CountNames bool
8791
}
8892

8993
// Server is a running DNS server.
@@ -99,7 +103,7 @@ type Server struct {
99103

100104
total atomic.Int64
101105
mu sync.Mutex
102-
counts map[countKey]int64
106+
counts map[countKey]int64 // nil unless Options.CountNames
103107

104108
closers []io.Closer
105109
// wg covers every goroutine the server starts: listeners, stream
@@ -141,7 +145,10 @@ func Listen(opts Options, h Handler) (*Server, error) {
141145
}
142146
return nil, err
143147
}
144-
s := &Server{Addr: ln.Addr().String(), counts: map[countKey]int64{}, closers: []io.Closer{ln, pc}, conns: map[net.Conn]struct{}{}}
148+
s := &Server{Addr: ln.Addr().String(), closers: []io.Closer{ln, pc}, conns: map[net.Conn]struct{}{}}
149+
if opts.CountNames {
150+
s.counts = map[countKey]int64{}
151+
}
145152

146153
s.wg.Add(2)
147154
go func() {
@@ -247,7 +254,11 @@ func (s *Server) Queries() int64 { return s.total.Load() }
247254

248255
// QueriesFor returns how many queries for name and qtype the server has
249256
// received. name is matched case-insensitively, with or without a trailing dot.
257+
// It panics unless the server was started with Options.CountNames.
250258
func (s *Server) QueriesFor(name string, qtype dnsmessage.Type) int64 {
259+
if s.counts == nil {
260+
panic("dnsserver: QueriesFor needs Options.CountNames")
261+
}
251262
s.mu.Lock()
252263
defer s.mu.Unlock()
253264
return s.counts[countKey{normalize(name), qtype}]
@@ -422,9 +433,11 @@ func (s *Server) handle(msg []byte, tcp bool, h Handler) []byte {
422433
}
423434
q := Query{Name: normalize(question.Name.String()), Type: question.Type, TCP: tcp}
424435
s.total.Add(1)
425-
s.mu.Lock()
426-
s.counts[countKey{q.Name, q.Type}]++
427-
s.mu.Unlock()
436+
if s.counts != nil {
437+
s.mu.Lock()
438+
s.counts[countKey{q.Name, q.Type}]++
439+
s.mu.Unlock()
440+
}
428441

429442
r := h(q)
430443
if r.Delay > 0 {

‎internal/dnsserver/server_test.go‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,44 @@ func TestCloseWaitsForHandlers(t *testing.T) {
7979
}
8080
}
8181

82+
// TestCountNamesOptIn covers #321: a server keeps a count per name only when
83+
// Options.CountNames asks for it, so a lab scan over a large wordlist does
84+
// not hold one entry per candidate. The total is always kept.
85+
func TestCountNamesOptIn(t *testing.T) {
86+
for _, count := range []bool{false, true} {
87+
srv, err := Listen(Options{CountNames: count}, func(Query) Reply { return Reply{A: []string{"192.0.2.1"}} })
88+
if err != nil {
89+
t.Fatal(err)
90+
}
91+
var d net.Dialer
92+
c, err := d.DialContext(context.Background(), "udp", srv.Addr)
93+
if err != nil {
94+
srv.Close()
95+
t.Fatal(err)
96+
}
97+
_ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
98+
if _, err := c.Write(query(t, "www.example.com.")); err != nil {
99+
t.Fatal(err)
100+
}
101+
if _, err := c.Read(make([]byte, 512)); err != nil {
102+
t.Fatalf("CountNames %v: no answer: %v", count, err)
103+
}
104+
_ = c.Close()
105+
srv.Close()
106+
if n := srv.Queries(); n != 1 {
107+
t.Errorf("CountNames %v: Queries = %d, want 1", count, n)
108+
}
109+
if got := len(srv.counts); count && got != 1 || !count && got != 0 {
110+
t.Errorf("CountNames %v: %d names counted", count, got)
111+
}
112+
if count {
113+
if n := srv.QueriesFor("WWW.example.com.", dnsmessage.TypeA); n != 1 {
114+
t.Errorf("QueriesFor = %d, want 1", n)
115+
}
116+
}
117+
}
118+
}
119+
82120
// TestCloseStreamsAfter: a server can close each stream connection after n
83121
// answers, like servers with a per-connection query limit.
84122
func TestCloseStreamsAfter(t *testing.T) {

0 commit comments

Comments
 (0)