Skip to content

Commit 4b22c95

Browse files
TMHSDigitalclaude
andcommitted
Share scan option checks, pool building and defaults (#262)
The CLI, TUI and pkg/subenum each built scan.Options by hand, with some range checks, the resolver pool and the defaults repeated per front end. - scan.Options gains RatePerResolver and MaxDuration, and Validate checks both (new {rate_per_resolver} setting), replacing the separate checks in main.go, the TUI form and pkg/subenum. - dns.NewPoolFromLines parses a resolver list and builds the capped pool; all three front ends use it. - scan.DefaultConcurrency, DefaultTimeoutMs and DefaultHitRate feed the CLI flag defaults and the TUI form; pkg/subenum keeps its documented literals, now pinned to them by TestDefaultsMatchTheCLI. - Each front end has a drift-guard test: with every input set, every scan.Options field must be set or listed as unmapped with a reason. The TUI's list records the gaps #274 is about (AutoRate, RetryInconclusive, MaxDuration). - Tests pin the new messages for a negative rate per resolver and max duration in each front end. Fixes #262 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent dddb5eb commit 4b22c95

12 files changed

Lines changed: 274 additions & 53 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
147147
* `https://host/path` - DNS over HTTPS (RFC 8484), each query sent as an HTTP POST of `application/dns-message`.
148148

149149
`scan.Run` builds one resolver per scan (unless `Config.Resolver` is set) and reuses it for every lookup; `dns.CloseIdleConnections` releases its pooled connections when the scan ends.
150-
* **Resolver pool (`-r`)**: `dns.ParseResolverList` reads the `-r` file (`ip` or `ip:port`, port 53 by default) and `dns.NewPool` builds a `dns.Pool` over it. Candidate lookups go round-robin over the healthy members; the `-dns-server` resolver stays the *trusted* resolver and runs the preflight and wildcard probes. Safeguards:
150+
* **Resolver pool (`-r`)**: `dns.NewPoolFromLines` parses the `-r` file (`ip` or `ip:port`, port 53 by default) and builds a `dns.Pool` over it with the `-rate-per-resolver` cap; the TUI and `pkg/subenum` build their pools with it too. Candidate lookups go round-robin over the healthy members; the `-dns-server` resolver stays the *trusted* resolver and runs the preflight and wildcard probes. Safeguards:
151151
* **Health benching**: over each window of 20 lookups, a member that failed (timeout, refused, other) more than half of them is benched for 30 seconds, then tried again.
152152
* **Per-resolver rate cap (`-rate-per-resolver`)**: `Pool.SetRatePerResolver` spaces the queries sent to each member like `-rate` does globally. A lookup prefers a member with room, waits for the one that frees up soonest when every cap is spent (before its timeout starts), and books one slot per record type; canaries and second opinions book slots too. Each spent cap a lookup runs into counts as `throttled`. The global `-rate` limiter still applies, so the stricter of the two wins.
153153
* **Hit re-validation**: every pool hit is resolved again on the trusted resolver, and the trusted answer is the one reported (`resolveViaPool` in `internal/scan/runner.go`), so a poisoned member cannot inject names.
@@ -170,7 +170,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
170170

171171
* **Purpose**: To efficiently perform DNS lookups for a large number of potential subdomains, `subenum` employs a worker pool pattern. This allows multiple DNS queries to be in flight concurrently, significantly speeding up the enumeration process compared to sequential lookups.
172172
* **Implementation**: The scan engine lives in `internal/scan/runner.go` as `scan.Run`, which takes a context, a `scan.Config`, and an events channel. Both the CLI (`scanTarget` in `main.go`) and the TUI call it.
173-
* **`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings (timeout in milliseconds, attempts, rate, and so on). Its `Validate` method checks ranges and the resolver address, and its `Config` method builds the `scan.Config` the engine runs on. The CLI and the TUI each fill in an `Options`, so they cannot drift apart in what they accept.
173+
* **`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings (timeout in milliseconds, attempts, rate, and so on). Its `Validate` method checks ranges and the resolver address, and its `Config` method builds the `scan.Config` the engine runs on. The CLI, the TUI and `pkg/subenum` each fill in an `Options`, so they cannot drift apart in what they accept, and each has a test that fails when an `Options` field is left unset without a listed reason. `scan.DefaultConcurrency`, `DefaultTimeoutMs` and `DefaultHitRate` are the defaults all three use.
174174
* **`scan.Event` / `scan.EventKind`**: Typed events emitted on the channel: `EventResult` (a `scan.Result`: subdomain, records, simulated flag, takeover hint and internal address scope), `EventProgress`, `EventNotice`, `EventError`, and `EventDone`. An `EventNotice` carries a `scan.NoticeKind` (`NoticeWildcard`, `NoticeCap`, `NoticeCeiling`, `NoticeSkip`, `NoticeRate`) so consumers can style or filter notices without parsing the message. `EventDone` carries a `scan.Stats` snapshot with per-outcome counters (found, nxdomain, timeout, refused, other, wildcard-filtered).
175175
* **Preflight and wildcard check**: Before any candidate is dispatched, `scan.Run` resolves the apex (with retries) and runs `Lookup.FingerprintWildcard`; see the Data Flow section.
176176
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), a visited set, and a pending-work counter. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.
@@ -249,7 +249,7 @@ Visually, this can be seen as:
249249

250250
* **Missing Required Arguments**: When `-w` or a target (domain or `-dL`) is missing, the tool says what is missing, prints the usage message and flag list, and exits with status 2.
251251
* **Validation** (status 2 on failure): the tool validates:
252-
* Concurrency, timeout, attempts and depth must be at least 1; `-rate` and `-max-queries` must be 0 (unlimited) or positive. These checks live in `scan.Options.Validate`.
252+
* Concurrency, timeout, attempts and depth must be at least 1; `-rate`, `-rate-per-resolver`, `-max-queries` and `-max-duration` must be 0 (unlimited) or positive. These checks live in `scan.Options.Validate`.
253253
* The DNS server must be a valid `ip:port` with an IP host and a port in 1-65535 (`validate.DNSServer`; not checked in simulation mode).
254254
* The target domain must conform to DNS naming rules after normalization (`validate.NormalizeDomain`, `validate.Domain`). In a `-dL` list, invalid domains are skipped with a message instead.
255255
* Hit rate (simulation mode) must be 1-100.

‎internal/dns/pool.go‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package dns
22

33
import (
44
"context"
5+
"errors"
56
"fmt"
67
"net"
78
"strings"
@@ -132,6 +133,27 @@ func NewPool(addrs []string, timeout time.Duration) *Pool {
132133
return p
133134
}
134135

136+
// ErrNoResolvers is NewPoolFromLines' error for a list that names no
137+
// resolver; callers say which list it was.
138+
var ErrNoResolvers = errors.New("lists no resolvers")
139+
140+
// NewPoolFromLines parses a resolver list (see ParseResolverList) and builds
141+
// its pool with the per-member rate cap perSecond (see SetRatePerResolver).
142+
// The CLI's -r, the TUI's Resolver Pool and pkg/subenum's ResolverPool all
143+
// build their pool here (#262).
144+
func NewPoolFromLines(lines []string, timeout time.Duration, perSecond int) (*Pool, error) {
145+
addrs, err := ParseResolverList(lines)
146+
if err != nil {
147+
return nil, err
148+
}
149+
if len(addrs) == 0 {
150+
return nil, ErrNoResolvers
151+
}
152+
p := NewPool(addrs, timeout)
153+
p.SetRatePerResolver(perSecond)
154+
return p, nil
155+
}
156+
135157
// CloseIdleConnections closes the sockets every member keeps for reuse, as
136158
// CloseIdleConnections does for one resolver. A scan calls it when it ends.
137159
func (p *Pool) CloseIdleConnections() {

‎internal/scan/options.go‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,20 @@ import (
88
"github.com/TMHSDigital/subenum/internal/validate"
99
)
1010

11-
// Options are the user-facing scan settings. The CLI and the TUI each fill
12-
// one in from their own inputs, then share Validate and Config, so the two
13-
// front ends cannot drift apart in what they accept or how they scan (#80).
11+
// Defaults every front end gives a setting that is not set (#262): the CLI's
12+
// flag defaults, the TUI form's and pkg/subenum's Default constants, which
13+
// tests keep equal to these.
14+
const (
15+
DefaultConcurrency = 100
16+
DefaultTimeoutMs = 1000
17+
DefaultHitRate = 15
18+
)
19+
20+
// Options are the user-facing scan settings. The CLI, the TUI and
21+
// pkg/subenum each fill one in from their own inputs, then share Validate
22+
// and Config, so the front ends cannot drift apart in what they accept or
23+
// how they scan (#80). Each front end has a test that fails when a field
24+
// here is left unset without a stated reason (#262).
1425
type Options struct {
1526
Domain string
1627
Entries []string
@@ -28,14 +39,21 @@ type Options struct {
2839
Rate int
2940
AutoRate bool // back off instead of aborting when lookups fail (#220)
3041
MaxQueries int
31-
Deadline time.Time // -max-duration budget's end; zero = none (#190)
42+
// MaxDuration is the run's time budget, only checked here: the front end
43+
// turns it into Deadline when the run starts, since one budget covers
44+
// every target of a run (#190).
45+
MaxDuration time.Duration
46+
Deadline time.Time // MaxDuration's end; zero = none (#190)
3247
Exclude []string
3348
// RetryInconclusive retries inconclusive lookups once at the end (#217).
3449
RetryInconclusive bool
3550
// TakeoverProviders replaces the built-in takeover provider list when
3651
// non-nil (#219).
3752
TakeoverProviders dns.TakeoverProviders
3853
Pool *dns.Pool // -r resolver pool; DNSServer is then the trusted resolver
54+
// RatePerResolver is the pool's per-member rate cap, only checked here:
55+
// the front end applies it when it builds Pool (dns.NewPoolFromLines).
56+
RatePerResolver int
3957
// Resume point of an interrupted scan (#73); see Config.
4058
ResumeFrom int
4159
ResumeParents []string
@@ -63,6 +81,10 @@ func (o Options) Validate() error {
6381
return fmt.Errorf("rate ({rate}) must be 0 (unlimited) or a positive integer, got %d", o.Rate)
6482
case o.MaxQueries < 0:
6583
return fmt.Errorf("max queries ({max_queries}) must be 0 (unlimited) or a positive integer, got %d", o.MaxQueries)
84+
case o.MaxDuration < 0:
85+
return fmt.Errorf("max duration ({max_duration}) must be 0 (unlimited) or positive, got %s", o.MaxDuration)
86+
case o.RatePerResolver < 0:
87+
return fmt.Errorf("rate per resolver ({rate_per_resolver}) must be 0 (unlimited) or a positive integer, got %d", o.RatePerResolver)
6688
}
6789
if !o.Simulate {
6890
if err := validate.DNSServer(o.DNSServer); err != nil {

‎internal/scan/options_test.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ func TestOptionsValidate(t *testing.T) {
2727
{"depth", func(o *Options) { o.Depth = 0 }, "depth"},
2828
{"rate", func(o *Options) { o.Rate = -1 }, "rate"},
2929
{"max queries", func(o *Options) { o.MaxQueries = -1 }, "max queries"},
30+
// Checked once here for every front end, not by each one (#262).
31+
{"max duration", func(o *Options) { o.MaxDuration = -time.Second }, "max duration ({max_duration}) must be 0 (unlimited) or positive, got -1s"},
32+
{"rate per resolver", func(o *Options) { o.RatePerResolver = -1 }, "rate per resolver ({rate_per_resolver}) must be 0 (unlimited) or a positive integer, got -1"},
3033
{"bad resolver live", func(o *Options) { o.DNSServer = "nope" }, "DNS server"},
3134
{"bad resolver ignored when simulating", func(o *Options) { o.DNSServer = "nope"; o.Simulate = true }, ""},
3235
{"hit rate checked when simulating", func(o *Options) { o.Simulate = true; o.HitRate = 0 }, "hit rate"},

‎internal/scan/settings.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ const (
2121
SettingHitRate Setting = "hit_rate"
2222
SettingDepth Setting = "depth"
2323
SettingRate Setting = "rate"
24+
SettingRatePerRes Setting = "rate_per_resolver"
2425
SettingMaxQueries Setting = "max_queries"
2526
SettingMaxDuration Setting = "max_duration"
2627
SettingExclude Setting = "exclude"
@@ -48,6 +49,7 @@ var settingTable = []struct {
4849
{SettingHitRate, SettingNames{"hit-rate", "HitRate", "Hit Rate (%)"}},
4950
{SettingDepth, SettingNames{"depth", "Depth", "Depth"}},
5051
{SettingRate, SettingNames{"rate", "Rate", "Rate (qps)"}},
52+
{SettingRatePerRes, SettingNames{"rate-per-resolver", "RatePerResolver", "Rate/Resolver"}},
5153
{SettingMaxQueries, SettingNames{"max-queries", "MaxQueries", "Max Names"}},
5254
{SettingMaxDuration, SettingNames{"max-duration", "MaxDuration", "Max Duration"}},
5355
{SettingExclude, SettingNames{"exclude", "Exclude", "Exclude"}},

‎internal/tui/form.go‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -80,17 +80,17 @@ var formFields = [fieldCount]formField{
8080
saved: func(c *savedConfig) any { return &c.Wordlist }},
8181
fieldSimulate: {label: "Simulate", flag: "simulate", kind: toggleField,
8282
saved: func(c *savedConfig) any { return &c.Simulate }},
83-
fieldHitRate: {setting: scan.SettingHitRate, kind: textField, placeholder: "1–100", def: "15", gate: fieldSimulate,
83+
fieldHitRate: {setting: scan.SettingHitRate, kind: textField, placeholder: "1–100", def: strconv.Itoa(scan.DefaultHitRate), gate: fieldSimulate,
8484
saved: func(c *savedConfig) any { return &c.HitRate }},
8585
fieldDNSServer: {label: "DNS Server", flag: "dns-server", kind: textField, placeholder: "e.g. 8.8.8.8:53", def: validate.DefaultDNSServer,
8686
saved: func(c *savedConfig) any { return &c.DNSServer }},
8787
fieldResolvers: {label: "Resolver Pool", flag: "r", kind: textField, placeholder: "optional, file of resolver addresses",
8888
saved: func(c *savedConfig) any { return &c.Resolvers }},
89-
fieldRatePerResolver: {label: "Rate/Resolver", flag: "rate-per-resolver", kind: textField, placeholder: "0 = unlimited", def: "0",
89+
fieldRatePerResolver: {setting: scan.SettingRatePerRes, kind: textField, placeholder: "0 = unlimited", def: "0",
9090
saved: func(c *savedConfig) any { return &c.RatePerResolver }},
91-
fieldConcurrency: {setting: scan.SettingConcurrency, kind: textField, placeholder: "e.g. 100", def: "100",
91+
fieldConcurrency: {setting: scan.SettingConcurrency, kind: textField, placeholder: "e.g. 100", def: strconv.Itoa(scan.DefaultConcurrency),
9292
saved: func(c *savedConfig) any { return &c.Concurrency }},
93-
fieldTimeout: {setting: scan.SettingTimeout, kind: textField, placeholder: "e.g. 1000", def: "1000",
93+
fieldTimeout: {setting: scan.SettingTimeout, kind: textField, placeholder: "e.g. 1000", def: strconv.Itoa(scan.DefaultTimeoutMs),
9494
saved: func(c *savedConfig) any { return &c.TimeoutMs }},
9595
fieldAttempts: {setting: scan.SettingAttempts, kind: textField, placeholder: "e.g. 1", def: "1",
9696
saved: func(c *savedConfig) any { return &c.Attempts }},
@@ -411,7 +411,7 @@ func (m *formModel) validate() (formValues, string) {
411411
attempts := num(fieldAttempts, "Attempts", 0)
412412
// Hit rate only matters in simulation mode, and depth only when
413413
// recursive is on; otherwise their fields must not block the scan.
414-
hitRate := 15
414+
hitRate := scan.DefaultHitRate
415415
if m.shown(fieldHitRate) {
416416
hitRate = num(fieldHitRate, "Hit rate", 0)
417417
}
@@ -422,9 +422,6 @@ func (m *formModel) validate() (formValues, string) {
422422
rate := num(fieldRate, "Rate", 0) // blank means unlimited
423423
maxQueries := num(fieldMaxQueries, "Max queries", 0) // blank means unlimited
424424
ratePerResolver := num(fieldRatePerResolver, formFields[fieldRatePerResolver].name(), 0)
425-
if err == nil && ratePerResolver < 0 {
426-
err = fmt.Errorf("%s must be 0 (unlimited) or positive, got %d", formFields[fieldRatePerResolver].name(), ratePerResolver)
427-
}
428425
var exclude []string // out-of-scope names, never queried (#87)
429426
for _, p := range strings.Split(text(fieldExclude), ",") {
430427
if p = strings.TrimSpace(p); p != "" {
@@ -502,6 +499,8 @@ func (v formValues) options(entries []string, seed uint64) scan.Options {
502499
MaxQueries: v.maxQueries,
503500
Exclude: v.exclude,
504501
NoAbort: v.noAbort,
502+
503+
RatePerResolver: v.ratePerResolver,
505504
}
506505
}
507506

‎internal/tui/form_test.go‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -391,7 +391,8 @@ func TestResolverPoolField(t *testing.T) {
391391
}
392392

393393
m.inputs[fieldRatePerResolver].SetValue("-1")
394-
if _, errStr := m.validate(); !strings.Contains(errStr, "Rate/Resolver must be 0") {
394+
// The range check is scan.Options.Validate's, rendered as the form label (#262).
395+
if _, errStr := m.validate(); errStr != "rate per resolver (Rate/Resolver) must be 0 (unlimited) or a positive integer, got -1" {
395396
t.Errorf("negative Rate/Resolver: got %q", errStr)
396397
}
397398
m.inputs[fieldRatePerResolver].SetValue("0")
@@ -423,3 +424,44 @@ func TestWithSettingsSeedsEveryFlaggedField(t *testing.T) {
423424
}
424425
}
425426
}
427+
428+
// TestOptionsSetsEveryField is the TUI's drift guard of #262: with every form
429+
// value set, options fills in every scan.Options field except the ones listed
430+
// in tuiUnmapped, each with the reason. A field added to scan.Options fails
431+
// here until the form maps it or the list says why it does not.
432+
func TestOptionsSetsEveryField(t *testing.T) {
433+
tuiUnmapped := map[string]string{
434+
"Pool": "beginScan sets it from the pool loadPool built",
435+
"AutoRate": "no form field yet (#274)",
436+
"RetryInconclusive": "no form field yet (#274)",
437+
"MaxDuration": "no form field yet (#274)",
438+
"Deadline": "no form field yet (#274)",
439+
"TakeoverProviders": "the form has no takeover provider file; the built-in list applies",
440+
"ResumeFrom": "the TUI does not resume interrupted scans",
441+
"ResumeParents": "the TUI does not resume interrupted scans",
442+
"Verbose": "the scan screen shows events itself; there is no debug log",
443+
"Logf": "the scan screen shows events itself; there is no debug log",
444+
}
445+
v := formValues{
446+
domain: "example.com", wordlist: "words.txt", dnsServer: "192.0.2.53:53", resolvers: "pool.txt",
447+
ratePerResolver: 5, concurrency: 7, timeoutMs: 700, attempts: 2, hitRate: 40, recordTypes: []string{"A"},
448+
recursive: true, depth: 3, rate: 50, maxQueries: 10, exclude: []string{"vpn.example.com"},
449+
noAbort: true, simulate: true, force: true,
450+
}
451+
opts := reflect.ValueOf(v.options([]string{"www"}, 9))
452+
for i := range opts.NumField() {
453+
name := opts.Type().Field(i).Name
454+
_, listed := tuiUnmapped[name]
455+
switch zero := opts.Field(i).IsZero(); {
456+
case zero && !listed:
457+
t.Errorf("scan.Options.%s is never set by the form; map it or list it in tuiUnmapped with a reason", name)
458+
case !zero && listed:
459+
t.Errorf("scan.Options.%s is set by the form but listed in tuiUnmapped", name)
460+
}
461+
}
462+
for name := range tuiUnmapped {
463+
if _, ok := opts.Type().FieldByName(name); !ok {
464+
t.Errorf("tuiUnmapped lists %s, which scan.Options does not have", name)
465+
}
466+
}
467+
}

‎internal/tui/model.go‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package tui
33

44
import (
55
"context"
6+
"errors"
67
"fmt"
78
"math/rand/v2"
89
"time"
@@ -185,15 +186,13 @@ func loadPool(vals formValues) (*dns.Pool, error) {
185186
if err != nil {
186187
return nil, fmt.Errorf("cannot read resolver pool: %w", err)
187188
}
188-
addrs, err := dns.ParseResolverList(lines)
189-
if err == nil && len(addrs) == 0 {
190-
err = fmt.Errorf("%s lists no resolvers", vals.resolvers)
189+
pool, err := dns.NewPoolFromLines(lines, time.Duration(vals.timeoutMs)*time.Millisecond, vals.ratePerResolver)
190+
if errors.Is(err, dns.ErrNoResolvers) {
191+
err = fmt.Errorf("%s %w", vals.resolvers, err)
191192
}
192193
if err != nil {
193194
return nil, fmt.Errorf("resolver pool: %w", err)
194195
}
195-
pool := dns.NewPool(addrs, time.Duration(vals.timeoutMs)*time.Millisecond)
196-
pool.SetRatePerResolver(vals.ratePerResolver)
197196
return pool, nil
198197
}
199198

0 commit comments

Comments
 (0)