Skip to content

Commit 3e0ed5b

Browse files
TMHSDigitalclaude
andcommitted
Embed scan.Config in scan.Options instead of copying it (#331)
scan.Options and scan.Config declared the same settings, and Options.Config() copied about 25 fields one by one, so every engine option took three edits in internal/scan before a front end saw it. The only differences were the timeout's unit (TimeoutMs against a Duration) and the two settings that are checked but not passed on. Options now embeds Config and keeps only MaxDuration and RatePerResolver beside it; the copy and TestOptionsConfig are gone, and the front ends pass opts.Config to scan.Run. The timeout is a Duration everywhere: the CLI and the TUI convert their milliseconds, the library keeps rounding up to whole milliseconds, and Validate keeps its "at least 1 ms" message. The three front-end drift tests now walk the new Options.Fields(), the embedded Config's exported fields and Options' own, so they still check every setting; walking Options' top-level fields alone would have seen one non-zero Config and passed whatever was left unset. Resolver, which scan.Run builds from DNSServer, is listed as unmapped in each. Fixes #331 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 7d64244 commit 3e0ed5b

15 files changed

Lines changed: 161 additions & 199 deletions

‎docs/ARCHITECTURE.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ internal/cli/results.go - resultLog: a target's results, kept for -permut
7171
internal/cli/summary.go - assemble the run-quality report (-stats, JSONL summary line)
7272
internal/cli/explain.go - parseSummary, explainSummary: a run-quality report in plain language (explain_stats)
7373
internal/cli/wordlist.go - prepareWordlist: the -w file, or the bundled list when -w is not given
74-
internal/scan/options.go - Options: user-facing settings shared by CLI and TUI (Validate, Config)
74+
internal/scan/options.go - Options: Config plus the settings only checked, shared by CLI, TUI and library (Validate, Fields)
7575
internal/scan/runner.go - Scan engine: Config, Event/EventKind/NoticeKind, Stats, Run
7676
internal/scan/dispatcher.go - dispatcher: lazy candidate generation in chunks, budgets, resume watermark
7777
internal/scan/autorate.go - autoRate: the -auto-rate back-off controller
@@ -183,7 +183,7 @@ tools/licenses/main.go - THIRD_PARTY_LICENSES for release archives and t
183183

184184
* **Purpose**: To efficiently perform DNS lookups for a large number of potential subdomains, `subenum` employs a worker pool pattern. This allows multiple DNS queries to be in flight concurrently, significantly speeding up the enumeration process compared to sequential lookups.
185185
* **Implementation**: The scan engine lives in `internal/scan/runner.go` as `scan.Run`, which takes a context, a `scan.Config`, and an events channel. Both the CLI (`scanTarget` in `internal/cli/cli.go`) and the TUI call it.
186-
* **`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings (timeout in milliseconds, attempts, rate, and so on). Its `Validate` method checks ranges and the resolver address, and its `Config` method builds the `scan.Config` the engine runs on. The CLI, the TUI and `pkg/subenum` each fill in an `Options`, so they cannot drift apart in what they accept, and each has a test that fails when an `Options` field is left unset without a listed reason. `scan.DefaultConcurrency`, `DefaultTimeoutMs` and `DefaultHitRate` are the defaults all three use.
186+
* **`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings: it embeds the `scan.Config` the engine runs on, plus `MaxDuration` and `RatePerResolver`, which are only checked there (#331). Its `Validate` method checks ranges and the resolver address, and its `Config` field is what `scan.Run` takes. The CLI, the TUI and `pkg/subenum` each fill in an `Options`, so they cannot drift apart in what they accept, and each has a test that walks `Options.Fields()` and fails when a setting is left unset without a listed reason. `scan.DefaultConcurrency`, `DefaultTimeoutMs` and `DefaultHitRate` are the defaults all three use.
187187
* **`scan.Event` / `scan.EventKind`**: Typed events emitted on the channel: `EventResult` (a `scan.Result`: subdomain, records, simulated flag, takeover hint and internal address scope), `EventProgress`, `EventNotice`, `EventError`, and `EventDone`. An `EventNotice` carries a `scan.NoticeKind` (`NoticeWildcard`, `NoticeCap`, `NoticeCeiling`, `NoticeSkip`, `NoticeRate`) so consumers can style or filter notices without parsing the message. `EventDone` carries a `scan.Stats` snapshot with per-outcome counters (found, nxdomain, timeout, refused, other, wildcard-filtered).
188188
* **Preflight and wildcard check**: Before any candidate is dispatched, `scan.Run` resolves the apex (with retries) and runs `Lookup.FingerprintWildcard`; see the Data Flow section.
189189
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), the set of parents already queued (recursive scans only), and a pending-work counter. A set of generated candidates is kept only for a recursive scan whose entries include a multi-label one, the only case where two parents can reach the same name. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.

‎docs/DEVELOPER_GUIDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ subenum/
139139
│ ├── permute/ # -permute candidate generation
140140
│ ├── report/ # Run-quality report built from finished scans
141141
│ ├── scan/
142-
│ │ ├── options.go # Options: settings shared by CLI and TUI (Validate, Config)
142+
│ │ ├── options.go # Options: Config plus the settings only checked (Validate, Fields)
143143
│ │ ├── runner.go # Scan engine: Config, Event/NoticeKind, Stats, Run
144144
│ │ ├── dispatcher.go # Lazy candidate generation in chunks, budgets, resume watermark
145145
│ │ ├── autorate.go # -auto-rate back-off controller

‎internal/cli/cli.go‎

Lines changed: 37 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -736,40 +736,43 @@ func scanOptions(f cliFlags, in *runtimeInputs, p scanPass, logf dns.Logf) scan.
736736
recursive, depth = false, 1
737737
}
738738
return scan.Options{
739-
Domain: p.domain,
740-
Entries: p.entries,
741-
Concurrency: f.concurrency,
742-
TimeoutMs: f.timeoutMs,
743-
DNSServer: f.dnsServer,
744-
Simulate: f.testMode,
745-
HitRate: f.testHitRate,
746-
Seed: f.seed,
747-
Attempts: in.maxAttempts,
748-
Force: f.force,
749-
Types: in.recordTypes,
750-
Recursive: recursive,
751-
Depth: depth,
752-
Rate: f.rate,
753-
AutoRate: f.autoRate,
754-
MaxQueries: p.maxQueries,
755-
MaxDuration: f.maxDuration,
756-
Deadline: in.deadline,
757-
Exclude: in.excludes,
758-
ResumeFrom: p.resumeFrom,
759-
// The retry pass belongs to the scan, so a resumed scan runs it too,
760-
// for the names the stopped run left it as well (#315).
761-
RetryInconclusive: f.retryIncon,
762-
RetryPrior: p.retryPrior,
763-
// Only the -inconclusive file reads the names (#300).
764-
ListInconclusive: f.inconFile != "",
765-
TakeoverProviders: in.providers,
766-
// Names found before an interrupt still need their children expanded.
767-
ResumeParents: p.resumeNames,
768-
Pool: in.pool,
739+
Config: scan.Config{
740+
Domain: p.domain,
741+
Entries: p.entries,
742+
Concurrency: f.concurrency,
743+
Timeout: time.Duration(f.timeoutMs) * time.Millisecond,
744+
DNSServer: f.dnsServer,
745+
Simulate: f.testMode,
746+
HitRate: f.testHitRate,
747+
Seed: f.seed,
748+
Attempts: in.maxAttempts,
749+
Force: f.force,
750+
Types: in.recordTypes,
751+
Recursive: recursive,
752+
Depth: depth,
753+
Rate: f.rate,
754+
AutoRate: f.autoRate,
755+
MaxQueries: p.maxQueries,
756+
Deadline: in.deadline,
757+
Exclude: in.excludes,
758+
ResumeFrom: p.resumeFrom,
759+
// The retry pass belongs to the scan, so a resumed scan runs it
760+
// too, for the names the stopped run left it as well (#315).
761+
RetryInconclusive: f.retryIncon,
762+
RetryPrior: p.retryPrior,
763+
// Only the -inconclusive file reads the names (#300).
764+
ListInconclusive: f.inconFile != "",
765+
TakeoverProviders: in.providers,
766+
// Names found before an interrupt still need their children
767+
// expanded.
768+
ResumeParents: p.resumeNames,
769+
Pool: in.pool,
770+
NoAbort: f.noAbort,
771+
Verbose: f.verbose,
772+
Logf: logf,
773+
},
774+
MaxDuration: f.maxDuration,
769775
RatePerResolver: f.ratePerRes,
770-
NoAbort: f.noAbort,
771-
Verbose: f.verbose,
772-
Logf: logf,
773776
}
774777
}
775778

@@ -797,7 +800,7 @@ func (r targetResult) unfinished() bool {
797800
func scanTarget(ctx context.Context, s *session, p scanPass, emit func(scan.Event)) (res targetResult) {
798801
f, out := s.f, s.out
799802
// Logf is serialized with the progress line (#36).
800-
cfg := scanOptions(f, &s.in, p, out.Verbose).Config()
803+
cfg := scanOptions(f, &s.in, p, out.Verbose).Config
801804

802805
events := make(chan scan.Event, 64)
803806
go scan.Run(ctx, cfg, events)

‎internal/cli/cli_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ func TestCLIBuildsSharedConfig(t *testing.T) {
247247
t.Fatal(err)
248248
}
249249
in := &runtimeInputs{maxAttempts: f.attempts, recordTypes: types}
250-
got := scanOptions(f, in, scanPass{domain: pos[0], entries: []string{"www", "mail"}, maxQueries: f.maxQueries}, nil).Config()
250+
got := scanOptions(f, in, scanPass{domain: pos[0], entries: []string{"www", "mail"}, maxQueries: f.maxQueries}, nil).Config
251251
want := scan.Config{
252252
Domain: "example.com", Entries: []string{"www", "mail"}, Concurrency: 50,
253253
Timeout: 800 * time.Millisecond, DNSServer: "1.1.1.1:53", Simulate: true, HitRate: 30,

‎internal/cli/diff.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -357,7 +357,7 @@ func emitRemoved(ctx context.Context, s *session, summary *output.Summary) int64
357357
if budget > 0 && budget < len(missing) {
358358
out.Info("Note: the remaining -max-queries budget covers %d of them", budget)
359359
}
360-
cfg := scanOptions(s.f, &s.in, scanPass{maxQueries: budget}, nil).Config()
360+
cfg := scanOptions(s.f, &s.in, scanPass{maxQueries: budget}, nil).Config
361361
cfg.Pool = nil // the trusted resolver decides
362362
outcomes, sent := scan.Verify(ctx, cfg, missing)
363363
for _, name := range missing {

‎internal/cli/options_test.go‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ import (
1717
// scan.Options fails here until the CLI maps it or lists it in cliUnmapped
1818
// with a reason.
1919
func TestScanOptionsSetsEveryField(t *testing.T) {
20-
cliUnmapped := map[string]string{}
20+
cliUnmapped := map[string]string{
21+
"Resolver": "scan.Run builds the resolver from DNSServer",
22+
}
2123

2224
f := cliFlags{
2325
concurrency: 7, timeoutMs: 700, dnsServer: "192.0.2.53:53", testMode: true, testHitRate: 40,
@@ -61,19 +63,17 @@ func TestRangeErrorsNameFlags(t *testing.T) {
6163
// field that is set after all.
6264
func checkEveryOptionSet(t *testing.T, opts scan.Options, unmapped map[string]string) {
6365
t.Helper()
64-
v := reflect.ValueOf(opts)
65-
for i := range v.NumField() {
66-
name := v.Type().Field(i).Name
66+
for name, field := range opts.Fields() {
6767
_, listed := unmapped[name]
68-
switch zero := v.Field(i).IsZero(); {
68+
switch zero := field.IsZero(); {
6969
case zero && !listed:
7070
t.Errorf("scan.Options.%s is never set; map it or list it as unmapped with a reason", name)
7171
case !zero && listed:
7272
t.Errorf("scan.Options.%s is set but listed as unmapped", name)
7373
}
7474
}
7575
for name := range unmapped {
76-
if _, ok := v.Type().FieldByName(name); !ok {
76+
if _, ok := reflect.TypeFor[scan.Options]().FieldByName(name); !ok {
7777
t.Errorf("unmapped lists %s, which scan.Options does not have", name)
7878
}
7979
}

‎internal/scan/options.go‎

Lines changed: 34 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@ package scan
33
import (
44
"context"
55
"fmt"
6+
"iter"
7+
"reflect"
68
"time"
79

8-
"github.com/TMHSDigital/subenum/internal/dns"
910
"github.com/TMHSDigital/subenum/internal/validate"
1011
)
1112

@@ -23,48 +24,44 @@ const (
2324
// and Config, so the front ends cannot drift apart in what they accept or
2425
// how they scan (#80). Each front end has a test that fails when a field
2526
// here is left unset without a stated reason (#262).
27+
//
28+
// They are the engine's Config, documented there, plus the two settings
29+
// that are only checked here (#331).
2630
type Options struct {
27-
Domain string
28-
Entries []string
29-
Concurrency int
30-
TimeoutMs int
31-
DNSServer string
32-
Simulate bool
33-
HitRate int
34-
Seed uint64
35-
Attempts int
36-
Force bool
37-
Types []string
38-
Recursive bool
39-
Depth int
40-
Rate int
41-
AutoRate bool // back off instead of aborting when lookups fail (#220)
42-
MaxQueries int
31+
Config
4332
// MaxDuration is the run's time budget, only checked here: the front end
4433
// turns it into Deadline when the run starts, since one budget covers
4534
// every target of a run (#190).
4635
MaxDuration time.Duration
47-
Deadline time.Time // MaxDuration's end; zero = none (#190)
48-
Exclude []string
49-
// RetryInconclusive retries inconclusive lookups once at the end (#217).
50-
RetryInconclusive bool
51-
// ListInconclusive fills Stats.Inconclusive; set it only when something
52-
// reads the list, which costs memory per failed lookup (#300).
53-
ListInconclusive bool
54-
// TakeoverProviders replaces the built-in takeover provider list when
55-
// non-nil (#219).
56-
TakeoverProviders dns.TakeoverProviders
57-
Pool *dns.Pool // -r resolver pool; DNSServer is then the trusted resolver
5836
// RatePerResolver is the pool's per-member rate cap, only checked here:
5937
// the front end applies it when it builds Pool (dns.NewPoolFromLines).
6038
RatePerResolver int
61-
// Resume point of an interrupted scan (#73); see Config.
62-
ResumeFrom int
63-
ResumeParents []string
64-
RetryPrior []Pending // see Config (#315)
65-
NoAbort bool
66-
Verbose bool
67-
Logf dns.Logf
39+
}
40+
41+
// Fields yields each setting of o by name: the embedded Config's exported
42+
// fields, then Options' own. The front ends' drift tests walk it to check
43+
// they set every setting (#262, #331).
44+
func (o Options) Fields() iter.Seq2[string, reflect.Value] {
45+
return func(yield func(string, reflect.Value) bool) {
46+
var walk func(v reflect.Value) bool
47+
walk = func(v reflect.Value) bool {
48+
for i := range v.NumField() {
49+
f := v.Type().Field(i)
50+
switch {
51+
case f.Anonymous && f.Type.Kind() == reflect.Struct:
52+
if !walk(v.Field(i)) {
53+
return false
54+
}
55+
case f.IsExported():
56+
if !yield(f.Name, v.Field(i)) {
57+
return false
58+
}
59+
}
60+
}
61+
return true
62+
}
63+
walk(reflect.ValueOf(o))
64+
}
6865
}
6966

7067
type noInconclusiveListKey struct{}
@@ -91,8 +88,8 @@ func (o Options) Validate() error {
9188
switch {
9289
case o.Concurrency < 1:
9390
return fmt.Errorf("concurrency ({concurrency}) must be at least 1, got %d", o.Concurrency)
94-
case o.TimeoutMs < 1:
95-
return fmt.Errorf("timeout ({timeout}) must be at least 1 ms, got %d", o.TimeoutMs)
91+
case o.Timeout < time.Millisecond:
92+
return fmt.Errorf("timeout ({timeout}) must be at least 1 ms, got %d", o.Timeout.Milliseconds())
9693
case o.Attempts < 1:
9794
return fmt.Errorf("attempts ({attempts}) must be at least 1, got %d", o.Attempts)
9895
case o.Simulate && (o.HitRate < 1 || o.HitRate > 100):
@@ -115,39 +112,3 @@ func (o Options) Validate() error {
115112
}
116113
return ValidateExcludes(o.Exclude)
117114
}
118-
119-
// Config builds the scan.Config for these options.
120-
func (o Options) Config() Config {
121-
return Config{
122-
Domain: o.Domain,
123-
Entries: o.Entries,
124-
Concurrency: o.Concurrency,
125-
Timeout: time.Duration(o.TimeoutMs) * time.Millisecond,
126-
DNSServer: o.DNSServer,
127-
Simulate: o.Simulate,
128-
HitRate: o.HitRate,
129-
Seed: o.Seed,
130-
Attempts: o.Attempts,
131-
Force: o.Force,
132-
Verbose: o.Verbose,
133-
Logf: o.Logf,
134-
Rate: o.Rate,
135-
AutoRate: o.AutoRate,
136-
Types: o.Types,
137-
Recursive: o.Recursive,
138-
Depth: o.Depth,
139-
NoAbort: o.NoAbort,
140-
MaxQueries: o.MaxQueries,
141-
Deadline: o.Deadline,
142-
Exclude: o.Exclude,
143-
Pool: o.Pool,
144-
145-
RetryInconclusive: o.RetryInconclusive,
146-
ListInconclusive: o.ListInconclusive,
147-
TakeoverProviders: o.TakeoverProviders,
148-
149-
ResumeFrom: o.ResumeFrom,
150-
ResumeParents: o.ResumeParents,
151-
RetryPrior: o.RetryPrior,
152-
}
153-
}

‎internal/scan/options_test.go‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,10 @@ import (
77
)
88

99
func validOptions() Options {
10-
return Options{
11-
Domain: "example.com", Concurrency: 10, TimeoutMs: 500, DNSServer: "1.1.1.1:53",
10+
return Options{Config: Config{
11+
Domain: "example.com", Concurrency: 10, Timeout: 500 * time.Millisecond, DNSServer: "1.1.1.1:53",
1212
HitRate: 15, Attempts: 1, Depth: 1,
13-
}
13+
}}
1414
}
1515

1616
// TestOptionsValidate covers the shared validator both front ends call (#80).
@@ -22,7 +22,8 @@ func TestOptionsValidate(t *testing.T) {
2222
}{
2323
{"valid", func(*Options) {}, ""},
2424
{"concurrency", func(o *Options) { o.Concurrency = 0 }, "concurrency"},
25-
{"timeout", func(o *Options) { o.TimeoutMs = 0 }, "timeout"},
25+
{"timeout", func(o *Options) { o.Timeout = 0 }, "timeout ({timeout}) must be at least 1 ms, got 0"},
26+
{"timeout under 1 ms", func(o *Options) { o.Timeout = 500 * time.Microsecond }, "timeout ({timeout}) must be at least 1 ms"},
2627
{"attempts", func(o *Options) { o.Attempts = 0 }, "attempts"},
2728
{"depth", func(o *Options) { o.Depth = 0 }, "depth"},
2829
{"rate", func(o *Options) { o.Rate = -1 }, "rate"},
@@ -51,11 +52,3 @@ func TestOptionsValidate(t *testing.T) {
5152
})
5253
}
5354
}
54-
55-
func TestOptionsConfig(t *testing.T) {
56-
o := validOptions()
57-
o.TimeoutMs = 750
58-
if cfg := o.Config(); cfg.Timeout != 750*time.Millisecond || cfg.Domain != o.Domain || cfg.Concurrency != 10 {
59-
t.Fatalf("Config() = %+v", cfg)
60-
}
61-
}

‎internal/tui/form.go‎

Lines changed: 22 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -618,26 +618,28 @@ func (m *formModel) validate() (formValues, string) {
618618

619619
v := formValues{
620620
Options: scan.Options{
621-
Domain: domain,
622-
Concurrency: concurrency,
623-
TimeoutMs: timeout,
624-
DNSServer: dnsServer,
625-
Simulate: m.toggles[fieldSimulate],
626-
HitRate: hitRate,
627-
Attempts: attempts,
628-
Force: m.toggles[fieldForce],
629-
Types: recordTypes,
630-
Recursive: m.toggles[fieldRecursive],
631-
Depth: depth,
632-
Rate: rate,
633-
AutoRate: m.toggles[fieldAutoRate],
634-
MaxQueries: maxQueries,
635-
MaxDuration: maxDuration,
636-
Exclude: exclude,
637-
NoAbort: m.toggles[fieldNoAbort],
638-
639-
RatePerResolver: ratePerResolver,
640-
RetryInconclusive: m.toggles[fieldRetryIncon],
621+
Config: scan.Config{
622+
Domain: domain,
623+
Concurrency: concurrency,
624+
Timeout: time.Duration(timeout) * time.Millisecond,
625+
DNSServer: dnsServer,
626+
Simulate: m.toggles[fieldSimulate],
627+
HitRate: hitRate,
628+
Attempts: attempts,
629+
Force: m.toggles[fieldForce],
630+
Types: recordTypes,
631+
Recursive: m.toggles[fieldRecursive],
632+
Depth: depth,
633+
Rate: rate,
634+
AutoRate: m.toggles[fieldAutoRate],
635+
MaxQueries: maxQueries,
636+
Exclude: exclude,
637+
NoAbort: m.toggles[fieldNoAbort],
638+
639+
RetryInconclusive: m.toggles[fieldRetryIncon],
640+
},
641+
MaxDuration: maxDuration,
642+
RatePerResolver: ratePerResolver,
641643
},
642644
wordlist: wl,
643645
sources: sources,

0 commit comments

Comments
 (0)