Skip to content

Commit 654f974

Browse files
committed
Keep the inconclusive-name list only when something reads it (#300)
Every lookup that timed out, was refused or failed kept its name for the whole scan, even when neither -inconclusive nor -retry-inconclusive was set, so a scan against a failing resolver held one string per name. scan.Config and scan.Options gain ListInconclusive. The engine collects the names only for that or the retry pass, stores a wordlist entry as its index, and fills Stats.Inconclusive only when the list was asked for. The CLI asks for it with -inconclusive; pkg/subenum always does, so library callers see no change. Counts are unaffected. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com> Fixes #300
1 parent d826821 commit 654f974

11 files changed

Lines changed: 203 additions & 29 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
5050
- `-format json` streams its array instead of holding every result and then encoding the whole document, so its peak memory matches `-format jsonl` (a 300k-result run: about 200 MB instead of 420 MB). The output is byte for byte the same (#181).
5151
- The wordlist is normalized once per run instead of once per `-dL` target: `wordlist.Prepare` normalizes the lines and `Prepared.For` applies only each target's 253-character limit. On a 1M-line list, normalizing takes about 0.5 s instead of 0.8 s with almost no allocations, and each further target about 0.5 ms. Results and counts are unchanged (#182).
5252
- Tests no longer flake under load: lab end-to-end runs use a 5 s timeout with 3 attempts and fail with a clear message if the lab 2 wildcard probe times out, the resolver-pool spread test counts lookups per member instead of timing the scan (the timing view is now `BenchmarkPoolSpreadsLoad`), and CI runs tests in shuffled order (#167).
53+
- The names whose lookup timed out, was refused or failed are kept only when something reads them: for the `-inconclusive` file, and until the `-retry-inconclusive` pass has run. A scan without either only counts them, so under `-no-abort` or `-auto-rate` against a failing resolver memory no longer grows with every failed lookup (about 1 GB at 10M failures). When they are kept, a wordlist entry costs its index rather than its name. Counts, the report and `pkg/subenum`'s `Stats.Inconclusive` are unchanged; the `-resume` state lists the names only with `-inconclusive` (#300).
5354

5455
### Fixed
5556
- The TUI configuration form fits a terminal shorter than its 31 lines, such as the standard 80×24: the logo is dropped, then the field rows scroll to keep the focused field in view, with `↑ N more fields` and `↓ N more fields` lines for the rest. The validation error and the key hints are always shown, so a rejected ctrl+r no longer fails without a visible reason (#287).

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
184184
* **Recursive enumeration** (optional): when `Config.Recursive` is set and a job below the depth cap resolves, the worker first checks the new parent with `Lookup.CheckWildcard` (cached per parent; inconclusive results are not cached). A wildcard or inconclusive parent is not expanded and a `NoticeSkip` notice is emitted. Otherwise the parent is handed to the dispatcher, whose visited set deduplicates the children (loop and duplicate protection); the progress total grows as new work is admitted.
185185
* **Progress ticker**: A separate goroutine emits `EventProgress` once per second. The total is read atomically since recursion can expand it mid-scan.
186186
* **Rate limiter** (optional): one `dns.RateLimiter` paces the whole scan. Live scans attach it to the context with `dns.WithLimiter`, so every wire query takes a slot (see 2.3). Simulated scans have no wire traffic and take one slot per job. `0` means unlimited.
187-
* **Retry pass (`-retry-inconclusive`)**: workers record each name whose lookup timed out, was refused or failed. Once every worker has exited, `retryPending` looks each of them up once more against the trusted resolver, with up to `Concurrency` goroutines. A name that now resolves is reported like any other result but not expanded recursively. The pass is skipped after a guard abort, an interrupt or a spent `-max-duration`. `Stats.Retried` and `Stats.Recovered` count it, and the names still without an answer are listed in `Stats.Inconclusive` whether or not the pass ran.
187+
* **Retry pass (`-retry-inconclusive`)**: workers record each name whose lookup timed out, was refused or failed. Once every worker has exited, `retryPending` looks each of them up once more against the trusted resolver, with up to `Concurrency` goroutines. A name that now resolves is reported like any other result but not expanded recursively. The pass is skipped after a guard abort, an interrupt or a spent `-max-duration`. `Stats.Retried` and `Stats.Recovered` count it, and the names still without an answer are listed in `Stats.Inconclusive` whether or not the pass ran, when `Config.ListInconclusive` is set: the CLI sets it for `-inconclusive` and `pkg/subenum` always does. Without it and without the pass the names are only counted, and the pass alone drops them once it has run, so failed lookups do not hold memory for a list nobody reads. A wordlist entry is kept as its index (`pendingLookup`); only a recursive child keeps its name.
188188
* **Completion**: a `sync.WaitGroup` waits for all workers (after the dispatcher closes `jobs`), the optional retry pass runs, then the progress ticker is stopped and `EventDone` is emitted.
189189
* **Interactions**: `scan.Run` is the single entry point for scanning used by both the CLI output pipeline and the Bubble Tea TUI. It decouples the scan engine from any specific display layer.
190190

‎internal/cli/cli.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -683,6 +683,8 @@ func scanOptions(f cliFlags, in *runtimeInputs, p scanPass, logf dns.Logf) scan.
683683
ResumeFrom: p.resumeFrom,
684684
// The retry pass belongs to the scan, so a resumed scan runs it too.
685685
RetryInconclusive: f.retryIncon,
686+
// Only the -inconclusive file reads the names (#300).
687+
ListInconclusive: f.inconFile != "",
686688
TakeoverProviders: in.providers,
687689
// Names found before an interrupt still need their children expanded.
688690
ResumeParents: p.resumeNames,

‎internal/cli/options_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ func TestScanOptionsSetsEveryField(t *testing.T) {
2222
f := cliFlags{
2323
concurrency: 7, timeoutMs: 700, dnsServer: "192.0.2.53:53", testMode: true, testHitRate: 40,
2424
seed: 9, force: true, recursive: true, depth: 3, rate: 50, autoRate: true,
25-
maxDuration: time.Minute, retryIncon: true, ratePerRes: 5, noAbort: true, verbose: true,
25+
maxDuration: time.Minute, retryIncon: true, inconFile: "left.txt", ratePerRes: 5, noAbort: true, verbose: true,
2626
}
2727
in := &runtimeInputs{
2828
maxAttempts: 2, recordTypes: []string{"A"}, excludes: []string{"vpn.example.com"},
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
package scan
2+
3+
import (
4+
"context"
5+
"runtime"
6+
"slices"
7+
"sync"
8+
"testing"
9+
"time"
10+
11+
"github.com/TMHSDigital/subenum/internal/dns"
12+
)
13+
14+
// The inconclusive names are collected only for a reader (#300).
15+
16+
// TestInconclusiveListOnlyWhenAsked: without ListInconclusive the names are
17+
// not listed, with or without the retry pass, and every count is what it is
18+
// with the list.
19+
func TestInconclusiveListOnlyWhenAsked(t *testing.T) {
20+
entries := []string{"ok1", "gone1", "flaky-ok1", "flaky-ok2", "flaky-gone1", "flaky-down1", "flaky-down2"}
21+
run := func(retry, list bool) Stats {
22+
cfg := Config{
23+
Domain: "example.com", Entries: entries, Concurrency: 3, Timeout: time.Second,
24+
Simulate: true, Attempts: 1, RetryInconclusive: retry, ListInconclusive: list,
25+
resolveHook: retryHook(&sync.Map{}),
26+
}
27+
events := make(chan Event, 64)
28+
go Run(context.Background(), cfg, events)
29+
done, errs, _ := collect(events)
30+
if len(errs) != 0 || done == nil {
31+
t.Fatalf("retry %v list %v: errors %v, done %v", retry, list, errs, done)
32+
}
33+
return done.Stats
34+
}
35+
for _, retry := range []bool{false, true} {
36+
with, without := run(retry, true), run(retry, false)
37+
if without.Inconclusive != nil {
38+
t.Errorf("retry %v: Inconclusive = %v without ListInconclusive, want nil", retry, without.Inconclusive)
39+
}
40+
if int64(len(with.Inconclusive)) != with.Timeout {
41+
t.Errorf("retry %v: Inconclusive lists %d names, want the %d timeouts", retry, len(with.Inconclusive), with.Timeout)
42+
}
43+
with.Inconclusive = nil
44+
if without.Sum() != int64(len(entries)) || without.Found != with.Found || without.NXDomain != with.NXDomain ||
45+
without.Timeout != with.Timeout || without.Retried != with.Retried || without.Recovered != with.Recovered {
46+
t.Errorf("retry %v: counts without the list %+v, with it %+v; want the same", retry, without, with)
47+
}
48+
}
49+
}
50+
51+
// TestInconclusiveListNamesRecursiveChildren: the list rebuilds a wordlist
52+
// entry's name from its index and keeps a recursive child's name as it was.
53+
func TestInconclusiveListNamesRecursiveChildren(t *testing.T) {
54+
cfg := Config{
55+
Domain: "example.com", Entries: []string{"ok", "down"}, Concurrency: 2, Timeout: time.Second,
56+
Simulate: true, Attempts: 1, Recursive: true, Depth: 2, ListInconclusive: true,
57+
resolveHook: func(_ context.Context, name string) ([]dns.Record, dns.Outcome) {
58+
if name == "ok.example.com" {
59+
return []dns.Record{{Type: "A", Value: "192.0.2.1"}}, dns.OutcomeFound
60+
}
61+
return nil, dns.OutcomeTimeout
62+
},
63+
}
64+
events := make(chan Event, 64)
65+
go Run(context.Background(), cfg, events)
66+
done, errs, _ := collect(events)
67+
if len(errs) != 0 || done == nil {
68+
t.Fatalf("errors %v, done %v", errs, done)
69+
}
70+
want := []string{"down.example.com", "down.ok.example.com", "ok.ok.example.com"}
71+
if !slices.Equal(done.Stats.Inconclusive, want) {
72+
t.Errorf("Inconclusive = %v, want %v", done.Stats.Inconclusive, want)
73+
}
74+
}
75+
76+
// TestFailedLookupsAreNotRetainedWithoutTheList: with no reader for the
77+
// names, a scan whose every lookup is refused holds no more memory at its
78+
// last lookup than one whose every lookup is answered.
79+
func TestFailedLookupsAreNotRetainedWithoutTheList(t *testing.T) {
80+
const n = 100_000
81+
entries := make([]string, n)
82+
for i := range entries {
83+
entries[i] = "host-" + itoa(i)
84+
}
85+
last := entries[n-1] + ".example.com"
86+
heapAtEnd := func(o dns.Outcome) uint64 {
87+
var heap uint64
88+
cfg := Config{
89+
Domain: "example.com", Entries: entries, Concurrency: 1, Timeout: time.Second,
90+
Simulate: true, Attempts: 1, NoAbort: true,
91+
resolveHook: func(_ context.Context, name string) ([]dns.Record, dns.Outcome) {
92+
if name == last {
93+
var m runtime.MemStats
94+
runtime.GC()
95+
runtime.ReadMemStats(&m)
96+
heap = m.HeapAlloc
97+
}
98+
return nil, o
99+
},
100+
}
101+
events := make(chan Event, 64)
102+
go Run(context.Background(), cfg, events)
103+
done, _, _ := collect(events)
104+
if done == nil || done.Stats.Sum() != n {
105+
t.Fatalf("outcome %v: done %+v, want %d lookups", o, done, n)
106+
}
107+
if heap == 0 {
108+
t.Fatalf("outcome %v: the last name was never looked up", o)
109+
}
110+
return heap
111+
}
112+
answered, refused := heapAtEnd(dns.OutcomeNXDomain), heapAtEnd(dns.OutcomeRefused)
113+
// A retained entry costs at least 32 bytes, 3.2 MB here; allow 1 MB of noise.
114+
if refused > answered+1<<20 {
115+
t.Errorf("heap at the last lookup: %d bytes with every lookup refused, %d with every lookup answered; failed lookups are retained",
116+
refused, answered)
117+
}
118+
}

‎internal/scan/options.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ type Options struct {
4747
Exclude []string
4848
// RetryInconclusive retries inconclusive lookups once at the end (#217).
4949
RetryInconclusive bool
50+
// ListInconclusive fills Stats.Inconclusive; set it only when something
51+
// reads the list, which costs memory per failed lookup (#300).
52+
ListInconclusive bool
5053
// TakeoverProviders replaces the built-in takeover provider list when
5154
// non-nil (#219).
5255
TakeoverProviders dns.TakeoverProviders
@@ -121,6 +124,7 @@ func (o Options) Config() Config {
121124
Pool: o.Pool,
122125

123126
RetryInconclusive: o.RetryInconclusive,
127+
ListInconclusive: o.ListInconclusive,
124128
TakeoverProviders: o.TakeoverProviders,
125129

126130
ResumeFrom: o.ResumeFrom,

‎internal/scan/retry_edges_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ func TestRetryCancelledKeepsNamesPending(t *testing.T) {
6363
var calls sync.Map
6464
cfg := Config{
6565
Domain: "example.com", Entries: []string{"a", "b", "c", "d"}, Concurrency: 1,
66-
Timeout: time.Second, Simulate: true, Attempts: 1, RetryInconclusive: true,
66+
Timeout: time.Second, Simulate: true, Attempts: 1, RetryInconclusive: true, ListInconclusive: true,
6767
resolveHook: func(ctx context.Context, name string) ([]dns.Record, dns.Outcome) {
6868
n, _ := calls.LoadOrStore(name, new(atomic.Int32))
6969
if n.(*atomic.Int32).Add(1) == 2 {

‎internal/scan/runner.go‎

Lines changed: 71 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ type Config struct {
6060
// The retry goes to the trusted resolver, never the pool. A name it
6161
// resolves is reported like any other, but not expanded by Recursive.
6262
RetryInconclusive bool
63+
// ListInconclusive fills Stats.Inconclusive. Without it the names are
64+
// only counted, and kept no longer than the retry pass needs them, so a
65+
// scan against a failing resolver does not hold a name per failed
66+
// lookup for a list nobody reads (#300).
67+
ListInconclusive bool
6368
// TakeoverProviders are the takeover-prone services CNAME targets are
6469
// matched against; nil is the built-in list (#219).
6570
TakeoverProviders dns.TakeoverProviders
@@ -163,7 +168,8 @@ type Stats struct {
163168
// then got an answer: their first outcome (timeout, refused, other)
164169
// moves to the new one, so the outcome counters still add up to the
165170
// names tested. Inconclusive lists, sorted, the names whose lookup is
166-
// still a timeout, refusal or failure at the end of the scan.
171+
// still a timeout, refusal or failure at the end of the scan; it is nil
172+
// without Config.ListInconclusive (#300).
167173
Retried int64
168174
Recovered int64
169175
Inconclusive []string
@@ -229,42 +235,72 @@ func inconclusive(o dns.Outcome) bool {
229235
return o == dns.OutcomeTimeout || o == dns.OutcomeRefused || o == dns.OutcomeOther
230236
}
231237

232-
// pendingLookup is a job whose lookup was inconclusive, and that outcome.
238+
// pendingLookup is a name whose lookup was inconclusive, and that outcome.
239+
// A depth-1 name is kept as its index into cfg.Entries and rebuilt on
240+
// demand, so the list does not keep the job's name string alive (#300);
241+
// only a recursive child, which has no such index, keeps its name.
233242
type pendingLookup struct {
234-
j job
243+
name string // a recursive child's full name; "" for a depth-1 name
244+
root int // index into cfg.Entries for a depth-1 name, else -1
235245
outcome dns.Outcome
236246
}
237247

248+
// pendingOf is the list entry for job j, whose lookup ended in o.
249+
func pendingOf(j job, o dns.Outcome) pendingLookup {
250+
if j.root >= 0 {
251+
return pendingLookup{root: j.root, outcome: o}
252+
}
253+
return pendingLookup{name: j.domain, root: -1, outcome: o}
254+
}
255+
256+
// domain is the name that was looked up.
257+
func (p pendingLookup) domain(cfg *Config) string {
258+
if p.root >= 0 {
259+
return cfg.Entries[p.root] + "." + cfg.Domain
260+
}
261+
return p.name
262+
}
263+
238264
// pendingList collects the inconclusive lookups of a scan, for the retry
239-
// pass and Stats.Inconclusive (#217). Workers add to it concurrently.
265+
// pass and Stats.Inconclusive (#217). Workers add to it concurrently. A nil
266+
// list collects nothing: the scan has no reader for the names (#300).
240267
type pendingList struct {
241268
mu sync.Mutex
242269
items []pendingLookup
243270
}
244271

245-
func (l *pendingList) add(j job, o dns.Outcome) {
272+
func (l *pendingList) add(it pendingLookup) {
273+
if l == nil {
274+
return
275+
}
246276
l.mu.Lock()
247-
l.items = append(l.items, pendingLookup{j: j, outcome: o})
277+
l.items = append(l.items, it)
248278
l.mu.Unlock()
249279
}
250280

251281
// take empties the list and returns what it held.
252282
func (l *pendingList) take() []pendingLookup {
283+
if l == nil {
284+
return nil
285+
}
253286
l.mu.Lock()
254287
defer l.mu.Unlock()
255288
items := l.items
256289
l.items = nil
257290
return items
258291
}
259292

260-
// names returns the listed names, sorted.
261-
func (l *pendingList) names() []string {
262-
l.mu.Lock()
263-
out := make([]string, 0, len(l.items))
264-
for _, it := range l.items {
265-
out = append(out, it.j.domain)
293+
// names empties the list and returns the names it held, sorted; nil for a
294+
// nil list.
295+
func (l *pendingList) names(cfg *Config) []string {
296+
if l == nil {
297+
return nil
298+
}
299+
items := l.take()
300+
out := make([]string, 0, len(items))
301+
for _, it := range items {
302+
out = append(out, it.domain(cfg))
266303
}
267-
l.mu.Unlock()
268304
sort.Strings(out)
269305
return out
270306
}
@@ -605,7 +641,10 @@ func Run(ctx context.Context, cfg Config, events chan<- Event) {
605641

606642
st := &scanState{
607643
cfg: cfg, maxDepth: maxDepth, limiter: jobLimiter, events: events,
608-
cancel: cancel, fp: fingerprint, wc: &wildcardCache{known: make(map[string]bool)}, pending: &pendingList{},
644+
cancel: cancel, fp: fingerprint, wc: &wildcardCache{known: make(map[string]bool)},
645+
}
646+
if cfg.ListInconclusive || cfg.RetryInconclusive {
647+
st.pending = &pendingList{}
609648
}
610649
st.total.Store(int64(len(cfg.Entries)))
611650
if cfg.AutoRate {
@@ -649,7 +688,9 @@ func Run(ctx context.Context, cfg Config, events chan<- Event) {
649688
snap.QueriesSent = queriesSent.Load()
650689
snap.Wildcard = wildcardRoot
651690
snap.Fingerprint = fingerprint.records()
652-
snap.Inconclusive = st.pending.names()
691+
if cfg.ListInconclusive {
692+
snap.Inconclusive = st.pending.names(&cfg)
693+
}
653694
if st.auto != nil {
654695
snap.RateBackoffs, snap.FinalRate = st.auto.result()
655696
}
@@ -833,7 +874,7 @@ type scanState struct {
833874
guardOnce sync.Once
834875
fp *fingerprint
835876
wc *wildcardCache
836-
pending *pendingList // inconclusive lookups (#217)
877+
pending *pendingList // inconclusive lookups (#217); nil when nothing reads them (#300)
837878
auto *autoRate // the AutoRate controller; nil without AutoRate (#220)
838879
}
839880

@@ -886,7 +927,7 @@ func (s *scanState) processJob(ctx context.Context, j job) (finished bool) {
886927

887928
st.add(outcome)
888929
if inconclusive(outcome) {
889-
s.pending.add(j, outcome)
930+
s.pending.add(pendingOf(j, outcome))
890931
}
891932
s.observe(inconclusive(outcome))
892933
s.checkReliability(s.processed.Add(1))
@@ -1007,6 +1048,11 @@ func (s *scanState) retryPending(ctx context.Context, workers int) {
10071048
return
10081049
}
10091050
items := s.pending.take()
1051+
if !s.cfg.ListInconclusive {
1052+
// Nothing lists the names the pass leaves unanswered. No worker is
1053+
// running here, so the field can be cleared without a lock.
1054+
s.pending = nil
1055+
}
10101056
ch := make(chan pendingLookup)
10111057
var wg sync.WaitGroup
10121058
for range min(workers, len(items)) {
@@ -1021,7 +1067,7 @@ func (s *scanState) retryPending(ctx context.Context, workers int) {
10211067
for i, it := range items {
10221068
if ctx.Err() != nil || s.cfg.pastDeadline() {
10231069
for _, rest := range items[i:] {
1024-
s.pending.add(rest.j, rest.outcome)
1070+
s.pending.add(rest)
10251071
}
10261072
break
10271073
}
@@ -1038,28 +1084,29 @@ func (s *scanState) retryPending(ctx context.Context, workers int) {
10381084
func (s *scanState) retry(ctx context.Context, it pendingLookup) {
10391085
cfg, st := s.cfg, &s.stats
10401086
if err := s.limiter.Wait(ctx); err != nil {
1041-
s.pending.add(it.j, it.outcome)
1087+
s.pending.add(it)
10421088
return
10431089
}
1044-
records, outcome := cfg.resolve(ctx, it.j.domain, cfg.logf())
1090+
name := it.domain(&s.cfg)
1091+
records, outcome := cfg.resolve(ctx, name, cfg.logf())
10451092
if outcome = settle(ctx, outcome); outcome == dns.OutcomeCanceled {
1046-
s.pending.add(it.j, it.outcome)
1093+
s.pending.add(it)
10471094
return
10481095
}
10491096
st.retried.Add(1)
10501097
if inconclusive(outcome) {
1051-
s.pending.add(it.j, it.outcome)
1098+
s.pending.add(it)
10521099
return
10531100
}
10541101
st.recovered.Add(1)
10551102
st.counter(it.outcome).Add(-1)
1056-
if outcome == dns.OutcomeFound && isWildcardAnswer(ctx, cfg, s.fp, it.j.domain, records) {
1103+
if outcome == dns.OutcomeFound && isWildcardAnswer(ctx, cfg, s.fp, name, records) {
10571104
st.wildcardFiltered.Add(1)
10581105
return
10591106
}
10601107
st.add(outcome)
10611108
if outcome == dns.OutcomeFound {
1062-
s.report(ctx, it.j.domain, records)
1109+
s.report(ctx, name, records)
10631110
}
10641111
}
10651112

0 commit comments

Comments
 (0)