Skip to content

Commit 80acb01

Browse files
TMHSDigitalclaude
andcommitted
Keep only collidable names in the recursive visited set (#359)
One multi-label entry, from the wordlist or a nested CT, Wayback or AXFR name, turned the recursive visited set back on for every candidate. Two (entry, parent) pairs can only produce the same name when a multi-label entry is a prefix of it, and each parent is already expanded once, so the set now records only those names. Both orders of reaching such a name are still deduplicated, in next() and in stopAdmitting(). BenchmarkDispatcherRecursiveDotted (100,000 entries plus "dev.api", root and ten found parents, 1.1M candidates): before: 1,100,011 visited, 134.8 MB/op, ~470 ms/op after: 11 visited, 26.5 MB/op, ~125 ms/op Fixes #359 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0507cbe commit 80acb01

4 files changed

Lines changed: 130 additions & 34 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4646
- `pkg/subenum` exports `ProbeZone(ctx, domain, ZoneOptions)`, which tries a zone transfer (AXFR) from the domain's name servers with `-axfr`'s limits (8 servers, 2 addresses each) and reports the names, the servers that allowed it and whether the zone uses plain NSEC (`ZoneInfo`), and `ZoneTransfer(ZoneOptions)`, a `Source` for `Config.Sources` that scans the transferred names. Its queries are not paced by `Rate`, as `-axfr`'s are not. The CLI and the library share the attempt in a new `internal/axfr` package (#314).
4747

4848
### Changed
49-
- A `-recursive` scan over a wordlist of single labels no longer remembers every candidate it generated (about 75 MB per million), and one that reaches `-max-queries` or `-max-duration` counts the untested candidates without building each name, so it stops on time. A scan whose candidates include multi-label names (a wordlist entry with a dot, or names from `-sources`) still keeps that record, since only there can two parents reach the same name (#301).
49+
- A `-recursive` scan over a wordlist of single labels no longer remembers every candidate it generated (about 75 MB per million), and one that reaches `-max-queries` or `-max-duration` counts the untested candidates without building each name, so it stops on time (#301).
50+
- A `-recursive` scan whose candidates include a multi-label name (a wordlist entry with a dot, or a nested name from `-sources` or `-axfr`) no longer remembers every candidate it generated either: it records only the names a multi-label entry begins, the only ones two parents can reach, so one nested name from CT no longer brings the per-candidate record back. In a benchmark of 100,000 entries plus one with a dot and ten found parents, the record drops from 1.1 million names to 11 and the scan allocates 26 MB instead of 135 MB (#359).
5051
- A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves its resume state to the `-state` file like an interrupted one and prints `continue with: subenum -resume <file>`, so a time-boxed job continues where the last one stopped instead of testing the same first names again. The exit code is unchanged (0 for a spent budget). A resumed run that stops the same way saves its state again, and one that finishes no longer counts the names an earlier part skipped, so it can end `complete` (#294).
5152
- The command's source moved out of the repository root: the `main_*.go` files are now `internal/cli/*.go`, and the root `main.go` only calls `cli.Main`. `go install github.com/TMHSDigital/subenum@latest`, the Action, the Docker image and `-X main.Version` builds are unchanged. The lab scenarios are embedded through a small `examples/labs` package, and `SECURITY.md` is now `.github/SECURITY.md`.
5253
- A `CNAME` lookup sends one CNAME query instead of going through Go's `LookupCNAME`, which also sent A and AAAA queries and threw their answers away. `-type A,AAAA,CNAME` now costs three queries a name instead of five, so the same `-rate` tests two-thirds more names per second, and `-dry-run`, `-max-queries` help and the `-rate-per-resolver` reservation count it that way. The query goes through the same transports (UDP with TCP fallback, DoT, DoH), rate limiter and query counter as the other types (#249).

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ tools/licenses/main.go - THIRD_PARTY_LICENSES for release archives and t
186186
* **`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings: it embeds the `scan.Config` the engine runs on, plus `MaxDuration` and `RatePerResolver`, which are only checked there (#331). Its `Validate` method checks ranges and the resolver address, and its `Config` field is what `scan.Run` takes. The CLI, the TUI and `pkg/subenum` each fill in an `Options`, so they cannot drift apart in what they accept, and each has a test that walks `Options.Fields()` and fails when a setting is left unset without a listed reason. `scan.DefaultConcurrency`, `DefaultTimeoutMs` and `DefaultHitRate` are the defaults all three use.
187187
* **`scan.Event` / `scan.EventKind`**: Typed events emitted on the channel: `EventResult` (a `scan.Result`: subdomain, records, simulated flag, takeover hint and internal address scope), `EventProgress`, `EventNotice`, `EventError`, and `EventDone`. An `EventNotice` carries a `scan.NoticeKind` (`NoticeWildcard`, `NoticeCap`, `NoticeCeiling`, `NoticeSkip`, `NoticeRate`) so consumers can style or filter notices without parsing the message. `EventDone` carries a `scan.Stats` snapshot with per-outcome counters (found, nxdomain, timeout, refused, other, wildcard-filtered).
188188
* **Preflight and wildcard check**: Before any candidate is dispatched, `scan.Run` resolves the apex (with retries) and runs `Lookup.FingerprintWildcard`; see the Data Flow section.
189-
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), the set of parents already queued (recursive scans only), and a pending-work counter. A set of generated candidates is kept only for a recursive scan whose entries include a multi-label one, the only case where two parents can reach the same name. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.
189+
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), the set of parents already queued (recursive scans only), and a pending-work counter. A set of generated candidates is kept only for a recursive scan whose entries include a multi-label one, the only case where two parents can reach the same name, and it holds only the names a multi-label entry begins (`a.b.c` for the entry `a.b`), since no other name can be reached twice. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.
190190
* **Recursion ceiling**: When `-recursive` is set, `scan.RecursionCeiling` computes the theoretical job count (`sum n^d`). Above 1e7 the scan refuses to start unless `-max-queries` or `-force` is set; otherwise a `NoticeCeiling` warning is emitted.
191191
* **Worker goroutines**: The pool size is `Config.Concurrency`, capped at the most jobs the scan can produce. Each worker takes a chunk and, for every job in it (which already holds the full domain), waits on the rate limiter if the scan is simulated, resolves it, classifies the outcome, and checks the result against the wildcard fingerprint.
192192
* **Near-miss revalidation**: A result whose records are all in the fingerprint is a wildcard answer and is counted as wildcard-filtered. A result that shares *some* records with it is a near miss, typical of a rotating wildcard pool: two fresh random labels under the same parent are probed, their answers join the fingerprint, and the result is checked again. Results sharing nothing with the fingerprint cost no extra queries.

‎internal/scan/dispatcher.go‎

Lines changed: 69 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,12 @@ type dispatcher struct {
2929
// then has one (entry, parent) pair. Only multi-label entries can reach
3030
// one name through two parents ("a.b" under the root, "a" under
3131
// "b.root"), and only then is visited, the set of generated candidates,
32-
// kept (#301). Both are nil for a non-recursive scan.
32+
// kept (#301). Even then it holds only the names one of those entries
33+
// begins (see collidable), not every candidate (#359). multi is the set
34+
// of multi-label entries. All three are nil for a non-recursive scan.
3335
expanded map[string]struct{}
3436
visited map[string]struct{}
37+
multi map[string]struct{}
3538
frontier []expansion
3639
// multiLabel is set when an entry holds more than one label ("dev.api"),
3740
// which countUnbuilt cannot place under an exclusion (#361).
@@ -59,13 +62,59 @@ type dispatcher struct {
5962
rootOutcome []uint8
6063
}
6164

62-
// visitedSet is the dispatcher's set of generated candidates: a map for a
63-
// recursive scan whose entries include a multi-label one, nil otherwise.
64-
func visitedSet(recursive, multiLabel bool, n int) map[string]struct{} {
65+
// multiLabelEntries is the set of entries that hold more than one label, for
66+
// a recursive scan that has any; nil otherwise.
67+
func multiLabelEntries(recursive, multiLabel bool, entries []string) map[string]struct{} {
6568
if !recursive || !multiLabel {
6669
return nil
6770
}
68-
return make(map[string]struct{}, n)
71+
multi := make(map[string]struct{})
72+
for _, e := range entries {
73+
if strings.Contains(e, ".") {
74+
multi[e] = struct{}{}
75+
}
76+
}
77+
return multi
78+
}
79+
80+
// collidable reports whether a candidate, named rel relative to the target,
81+
// may be generated by two (entry, parent) pairs. A pair splits rel at a
82+
// label boundary into an entry and the parent below it, so two pairs need
83+
// two entries that are prefixes of rel, and the longer of them holds more
84+
// than one label. A name no multi-label entry begins has one pair, and its
85+
// parent is expanded once, so it is generated once without being recorded.
86+
func (d *dispatcher) collidable(rel string) bool {
87+
for i := 0; i < len(rel); i++ {
88+
if rel[i] == '.' {
89+
if _, ok := d.multi[rel[:i]]; ok {
90+
return true
91+
}
92+
}
93+
}
94+
_, ok := d.multi[rel]
95+
return ok
96+
}
97+
98+
// firstVisit reports whether the candidate entry.parent at depth is new, and
99+
// records it when it could be generated again (#301, #359).
100+
func (d *dispatcher) firstVisit(entry, name string, depth int) bool {
101+
if d.visited == nil {
102+
return true
103+
}
104+
// Key by the name relative to the root domain. For depth 1 that is the
105+
// entry itself, so no extra string is retained.
106+
key := entry
107+
if depth > 1 {
108+
key = name[:len(name)-len(d.cfg.Domain)-1]
109+
}
110+
if !d.collidable(key) {
111+
return true
112+
}
113+
if _, dup := d.visited[key]; dup {
114+
return false
115+
}
116+
d.visited[key] = struct{}{}
117+
return true
69118
}
70119

71120
// hasMultiLabel reports whether any entry holds more than one label.
@@ -87,17 +136,21 @@ func newDispatcher(cfg Config, maxDepth int) *dispatcher {
87136
n: n,
88137
// Entries are unique (every caller deduplicates them), so only a
89138
// recursive scan with multi-label entries, whose levels can reach
90-
// one name twice, needs to remember what it visited. Without it a
91-
// multi-million-entry list skips a map as large as the list (#248),
92-
// and a recursive scan one that grows with every candidate (#301).
93-
visited: visitedSet(cfg.Recursive, multiLabel, n),
139+
// one name twice, needs to remember what it visited, and only the
140+
// names those entries begin. Without it a multi-million-entry list
141+
// skips a map as large as the list (#248), and a recursive scan one
142+
// that grows with every candidate (#301, #359).
143+
multi: multiLabelEntries(cfg.Recursive, multiLabel, cfg.Entries),
94144
multiLabel: multiLabel,
95145
frontier: []expansion{{parent: cfg.Domain, depth: 1, next: start}},
96146
remaining: n - start,
97147
start: start,
98148
rootDone: make([]bool, n-start),
99149
watermark: start,
100150
}
151+
if d.multi != nil {
152+
d.visited = make(map[string]struct{})
153+
}
101154
// Resumed recursive scans re-expand the names found before the
102155
// interrupt, whose children may not all have been tested (#73). Run
103156
// passes only the ones a live scan would expand (see gateResumeParents).
@@ -228,8 +281,8 @@ func (d *dispatcher) settle(i int) {
228281
// exclusions a cap stays O(1), and with them no candidate is built (see
229282
// countUnbuilt). Multi-label entries need every remaining candidate walked:
230283
// countUnbuilt cannot place them under a pattern (#361), and a visited set
231-
// must drop the names generated twice (#301), so each walked name joins it
232-
// and is counted once.
284+
// must drop the names generated twice (#301), so each walked name that could
285+
// repeat joins it and is counted once.
233286
func (d *dispatcher) stopAdmitting() {
234287
switch {
235288
case d.visited == nil && d.cfg.scope.empty():
@@ -241,15 +294,8 @@ func (d *dispatcher) stopAdmitting() {
241294
for i := e.next; i < d.n; i++ {
242295
entry := d.cfg.Entries[i]
243296
name := entry + "." + e.parent
244-
if d.visited != nil {
245-
key := entry
246-
if e.depth > 1 {
247-
key = name[:len(name)-len(d.cfg.Domain)-1]
248-
}
249-
if _, dup := d.visited[key]; dup {
250-
continue
251-
}
252-
d.visited[key] = struct{}{}
297+
if !d.firstVisit(entry, name, e.depth) {
298+
continue
253299
}
254300
if d.cfg.scope.excluded(name) {
255301
d.excluded++
@@ -347,18 +393,9 @@ func (d *dispatcher) next() (job, bool) {
347393
if e.depth == 1 {
348394
root = e.next - 1
349395
}
350-
if d.visited != nil {
351-
// Key by the name relative to the root domain. For depth 1 that
352-
// is the entry itself, so no extra string is retained.
353-
key := entry
354-
if e.depth > 1 {
355-
key = name[:len(name)-len(d.cfg.Domain)-1]
356-
}
357-
if _, dup := d.visited[key]; dup {
358-
d.settle(root)
359-
continue
360-
}
361-
d.visited[key] = struct{}{}
396+
if !d.firstVisit(entry, name, e.depth) {
397+
d.settle(root)
398+
continue
362399
}
363400
if d.cfg.scope.excluded(name) {
364401
d.excluded++

‎internal/scan/dispatcher_test.go‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package scan
22

33
import (
44
"slices"
5+
"strings"
56
"testing"
67
"time"
78
)
@@ -108,6 +109,63 @@ func TestDispatcherVisitedSetOnlyForMultiLabelEntries(t *testing.T) {
108109
}
109110
}
110111

112+
// TestDispatcherVisitedHoldsOnlyCollidableNames covers #359: one multi-label
113+
// entry no longer makes the set remember every candidate. Only a name with a
114+
// multi-label entry as a prefix ("a.b" in "a.b.c") can be reached twice, so
115+
// only those are recorded, and the duplicate is dropped whichever path
116+
// generates the name first.
117+
func TestDispatcherVisitedHoldsOnlyCollidableNames(t *testing.T) {
118+
cfg := Config{Entries: []string{"a.b", "a", "b", "c"}, Recursive: true}
119+
d := testDispatcher(cfg, 3)
120+
drain(d)
121+
d.expand(job{domain: "c.example.com", depth: 1})
122+
drain(d)
123+
// a.b, then a.b.c from c's children; a, b, c, a.c, b.c, c.c are not kept.
124+
if len(d.visited) != 2 {
125+
t.Errorf("visited holds %d names, want 2 (a.b and a.b.c)", len(d.visited))
126+
}
127+
128+
// x.a.b.example.com is "x.a" under b.example.com and "x" under
129+
// a.b.example.com; either expansion may run first.
130+
entries := []string{"x.a", "x", "a", "b"}
131+
for _, order := range [][]string{
132+
{"b.example.com", "a.b.example.com"},
133+
{"a.b.example.com", "b.example.com"},
134+
} {
135+
d := testDispatcher(Config{Entries: entries, Recursive: true}, 3)
136+
drain(d)
137+
for _, p := range order {
138+
d.expand(job{domain: p, depth: strings.Count(p, ".") - 1})
139+
}
140+
got := drain(d)
141+
if n := slices.Index(got, "x.a.b.example.com"); n < 0 || slices.Contains(got[n+1:], "x.a.b.example.com") {
142+
t.Errorf("order %v: x.a.b.example.com generated %d times in %v, want once",
143+
order, strings.Count(strings.Join(got, " "), "x.a.b.example.com"), got)
144+
}
145+
}
146+
}
147+
148+
// BenchmarkDispatcherRecursiveDotted measures a recursive scan's memory when
149+
// its 100,000 entries include one with a dot, as a CT or Wayback name does
150+
// (#359): the root and ten found parents, 1.1 million candidates.
151+
func BenchmarkDispatcherRecursiveDotted(b *testing.B) {
152+
entries := append(makeEntries(100_000), "dev.api")
153+
b.ReportAllocs()
154+
var d *dispatcher
155+
for b.Loop() {
156+
d = testDispatcher(Config{Entries: entries, Recursive: true}, 2)
157+
for i := range 10 {
158+
d.expand(job{domain: entries[i] + ".example.com", depth: 1})
159+
}
160+
for {
161+
if _, ok := d.next(); !ok {
162+
break
163+
}
164+
}
165+
}
166+
b.ReportMetric(float64(len(d.visited)), "visited")
167+
}
168+
111169
// Jobs settle as they complete, in any order, and the watermark is the first
112170
// entry not yet done.
113171
func TestDispatcherSettlesOutOfOrder(t *testing.T) {

0 commit comments

Comments
 (0)