Skip to content

Commit 9f2222c

Browse files
TMHSDigitalclaude
andcommitted
feat: resolver pool with trusted re-validation (-r)
-r resolvers.txt spreads candidate lookups over a pool of resolvers, round-robin over the healthy ones. A resolver failing more than half of its last 20+ lookups is benched for 30s; retries go to a different resolver. Every pool hit is re-validated against -dns-server, whose answer is the one reported and wildcard-filtered, so a poisoned or hijacking pool resolver cannot inject results. Preflight and wildcard checks use the trusted resolver. Accounting: per-resolver lookups/found/nxdomain/failed/benched in a stderr table and in the run-quality report, plus pool_hits, confirmed and unconfirmed per target. Hermetic test: 20 rate-limited resolvers ran the same scan 14x faster than one. Fixes #69 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 91be78a commit 9f2222c

12 files changed

Lines changed: 572 additions & 3 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- Resolver pool: `-r resolvers.txt` spreads lookups over many resolvers, round-robin over the healthy ones (a resolver failing over half its recent lookups is benched for 30s), with retries sent to a different resolver. Every pool hit is re-validated against `-dns-server`, whose answer is reported, so a lying pool resolver cannot inject results. Per-resolver accounting appears in a stderr table and in the run-quality report, with `pool_hits` / `confirmed` / `unconfirmed` per target. In a hermetic test, 20 rate-limited resolvers ran the same scan 14x faster than one (#69).
1112
- Scope control: `-exclude 'vpn.example.com,*.corp.example.com'` and `-exclude-file` keep out-of-scope names out of a scan. Excluded names are never queried, wildcard-probed or expanded recursively (a found parent whose whole subtree is excluded costs no queries), they are counted as `excluded` in the breakdown and the run-quality report, and a target that is itself out of scope is refused. The TUI has a matching Exclude field (#87).
1213
- Run-quality report: `-stats <file>` writes a schema-versioned JSON summary of a run (outcomes per target, DNS queries actually sent, achieved query rate, wildcard fingerprint, candidates skipped by `-max-queries`) with a `complete` / `degraded` / `unreliable` verdict and its reason. `-format jsonl` ends with the same object as its last line. The README documents the schema and how to gate CI on the verdict (#70).
1314

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,7 @@ make help # list all targets
255255
| `-simulate` | `false` | Simulation mode: no real DNS queries |
256256
| `-hit-rate <n>` | `15` | Simulated resolution rate, percent (1-100), applied uniformly to every name |
257257
| `-seed <n>` | `0` | Simulation seed; the same seed reproduces the same results (line order can vary unless `-t 1`). `0` picks a random seed and prints it |
258+
| `-r <file>` | n/a | Resolver pool: one `ip` or `ip:port` per line. Lookups rotate over the healthy resolvers (a resolver failing over half its recent lookups is benched for 30s), retries go to a different resolver, and every hit is re-validated against `-dns-server`, whose answer is the one reported. Preflight and wildcard checks use `-dns-server` |
258259
| `-exclude <list>` | n/a | Comma-separated out-of-scope names: exact (`vpn.example.com`) or `*.parent` (every name below `parent`, not `parent` itself). Excluded names are never queried, probed or expanded, and are counted as `excluded` |
259260
| `-exclude-file <file>` | n/a | Same patterns, one per line (`#` comments allowed), e.g. a bug-bounty program's out-of-scope list |
260261
| `-stats <file>` | n/a | Write a JSON run-quality report (outcomes, queries sent, verdict); see [Run-quality report](#run-quality-report) |
@@ -376,7 +377,7 @@ Every lookup is accounted for, and subenum says how far the results can be trust
376377
| `degraded` | Usable, but some names are unknown: 1-20% of lookups failed, `-max-queries` skipped candidates, or the run was interrupted |
377378
| `unreliable` | Over 20% of lookups failed (the reliability guard's threshold), the guard aborted the scan, or the target could not be scanned |
378379

379-
The overall `verdict` is the worst target verdict. `queries_sent` counts DNS messages actually dialed, including retries; it is 0 in simulation mode. SERVFAIL and other errors Go's resolver does not distinguish are counted under `other`. `excluded` counts candidates `-exclude` kept out of scope; they are never queried. `status` is one of `ok`, `failed`, `interrupted`, `skipped` or `not_run`. The `schema` number changes only when a field is renamed, removed or changes meaning; new fields can appear at any time.
380+
The overall `verdict` is the worst target verdict. `queries_sent` counts DNS messages actually dialed, including retries; it is 0 in simulation mode. SERVFAIL and other errors Go's resolver does not distinguish are counted under `other`. With `-r`, each target also reports `pool_hits`, `confirmed` and `unconfirmed` (hits the trusted resolver denied, so they were dropped), and the top level gains `resolvers`: per-resolver `lookups`, `found`, `nxdomain`, `failed` and `benched` counts; `resolver` is then the trusted resolver. `excluded` counts candidates `-exclude` kept out of scope; they are never queried. `status` is one of `ok`, `failed`, `interrupted`, `skipped` or `not_run`. The `schema` number changes only when a field is renamed, removed or changes meaning; new fields can appear at any time.
380381

381382
Gate a CI job on the verdict with `jq`:
382383

‎docs/ROADMAP.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@ Work in progress or next up.
2525

2626
Planned features that make results more trustworthy and the tool easier to get.
2727

28-
- **Resolver pool with trusted re-validation** ([#69](https://github.com/TMHSDigital/subenum/issues/69)), `-r resolvers.txt`.
2928
- **Monitoring mode** ([#84](https://github.com/TMHSDigital/subenum/issues/84)):
3029
diff against a previous run, plus a GitHub Action for scheduled checks.
3130
- **Release pipeline** ([#61](https://github.com/TMHSDigital/subenum/issues/61)):

‎internal/dns/pool.go‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
package dns
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"net"
7+
"strings"
8+
"sync"
9+
"time"
10+
)
11+
12+
// Pool health thresholds (#69).
13+
const (
14+
poolHealthMinLookups = 20 // lookups before a resolver can be benched
15+
poolHealthFailPct = 50 // failure share that benches a resolver
16+
poolBenchFor = 30 * time.Second // how long a benched resolver sits out
17+
)
18+
19+
// PoolResolver is one member of a Pool.
20+
type PoolResolver struct {
21+
Addr string
22+
resolver *net.Resolver
23+
24+
mu sync.Mutex
25+
stats ResolverStats
26+
window int // lookups since the last health reset
27+
windowKO int // failures in that window
28+
until time.Time
29+
}
30+
31+
// ResolverStats is one pool member's accounting for the run-quality report.
32+
type ResolverStats struct {
33+
Addr string `json:"address"`
34+
Lookups int64 `json:"lookups"`
35+
Found int64 `json:"found"`
36+
NXDomain int64 `json:"nxdomain"`
37+
Failed int64 `json:"failed"` // timeout, refused or other
38+
Benched int64 `json:"benched"` // times it was taken out of rotation
39+
}
40+
41+
// Pool spreads lookups over several resolvers, round-robin over the healthy
42+
// ones. A resolver whose recent failure rate passes poolHealthFailPct is
43+
// benched for poolBenchFor, then given another chance.
44+
type Pool struct {
45+
members []*PoolResolver
46+
mu sync.Mutex
47+
next int
48+
now func() time.Time
49+
}
50+
51+
// ParseResolverList parses resolver addresses, one per line or entry, as
52+
// ip or ip:port (port 53 by default). Blank lines and # comments are skipped.
53+
func ParseResolverList(lines []string) ([]string, error) {
54+
var out []string
55+
seen := map[string]bool{}
56+
for _, line := range lines {
57+
s := strings.TrimSpace(line)
58+
if s == "" || strings.HasPrefix(s, "#") {
59+
continue
60+
}
61+
if net.ParseIP(s) != nil {
62+
s = net.JoinHostPort(s, "53")
63+
}
64+
host, port, err := net.SplitHostPort(s)
65+
if err != nil || net.ParseIP(host) == nil || port == "" {
66+
return nil, fmt.Errorf("invalid resolver %q: want ip or ip:port", line)
67+
}
68+
if !seen[s] {
69+
seen[s] = true
70+
out = append(out, s)
71+
}
72+
}
73+
return out, nil
74+
}
75+
76+
// NewPool builds a pool of resolvers from NewResolver.
77+
func NewPool(addrs []string, timeout time.Duration) *Pool {
78+
p := &Pool{now: time.Now}
79+
for _, a := range addrs {
80+
p.members = append(p.members, &PoolResolver{Addr: a, resolver: NewResolver(timeout, a), stats: ResolverStats{Addr: a}})
81+
}
82+
return p
83+
}
84+
85+
// Size returns the number of resolvers in the pool.
86+
func (p *Pool) Size() int { return len(p.members) }
87+
88+
// pick returns the next healthy resolver, skipping avoid when possible. When
89+
// every resolver is benched, the one whose bench ends soonest is used.
90+
func (p *Pool) pick(avoid *PoolResolver) *PoolResolver {
91+
p.mu.Lock()
92+
defer p.mu.Unlock()
93+
now := p.now()
94+
n := len(p.members)
95+
var fallback *PoolResolver
96+
var fallbackUntil time.Time
97+
for i := 0; i < n; i++ {
98+
m := p.members[(p.next+i)%n]
99+
m.mu.Lock()
100+
until := m.until
101+
m.mu.Unlock()
102+
if now.Before(until) {
103+
if fallback == nil || until.Before(fallbackUntil) {
104+
fallback, fallbackUntil = m, until
105+
}
106+
continue
107+
}
108+
if m == avoid && n > 1 {
109+
continue
110+
}
111+
p.next = (p.next + i + 1) % n
112+
return m
113+
}
114+
if fallback != nil {
115+
return fallback
116+
}
117+
return avoid
118+
}
119+
120+
// report records a lookup's outcome and benches the resolver if it is failing.
121+
func (p *Pool) report(m *PoolResolver, o Outcome) {
122+
if o == OutcomeCanceled {
123+
return
124+
}
125+
m.mu.Lock()
126+
defer m.mu.Unlock()
127+
m.stats.Lookups++
128+
m.window++
129+
switch o {
130+
case OutcomeFound:
131+
m.stats.Found++
132+
case OutcomeNXDomain:
133+
m.stats.NXDomain++
134+
default:
135+
m.stats.Failed++
136+
m.windowKO++
137+
}
138+
if m.window >= poolHealthMinLookups {
139+
if m.windowKO*100 > m.window*poolHealthFailPct {
140+
m.until = p.now().Add(poolBenchFor)
141+
m.stats.Benched++
142+
}
143+
m.window, m.windowKO = 0, 0
144+
}
145+
}
146+
147+
// Resolve looks domain up through the pool, up to attempts times, each
148+
// attempt on a different healthy resolver where possible. It returns the
149+
// records, the outcome of the last attempt, and the resolver that answered.
150+
func (p *Pool) Resolve(ctx context.Context, domain string, timeout time.Duration, logf Logf, attempts int, types []string) ([]Record, Outcome, string) {
151+
var last *PoolResolver
152+
outcome := OutcomeOther
153+
for attempt := 0; attempt < max(attempts, 1); attempt++ {
154+
if ctx.Err() != nil {
155+
return nil, OutcomeCanceled, ""
156+
}
157+
m := p.pick(last)
158+
last = m
159+
var recs []Record
160+
recs, outcome = ResolveDomainWithRetry(ctx, m.resolver, domain, timeout, logf, 1, types)
161+
p.report(m, outcome)
162+
switch outcome {
163+
case OutcomeFound:
164+
return recs, outcome, m.Addr
165+
case OutcomeNXDomain, OutcomeCanceled:
166+
return nil, outcome, m.Addr
167+
}
168+
}
169+
return nil, outcome, ""
170+
}
171+
172+
// Stats returns a snapshot of every member's accounting, in pool order.
173+
func (p *Pool) Stats() []ResolverStats {
174+
out := make([]ResolverStats, 0, len(p.members))
175+
for _, m := range p.members {
176+
m.mu.Lock()
177+
out = append(out, m.stats)
178+
m.mu.Unlock()
179+
}
180+
return out
181+
}

‎internal/dns/pool_test.go‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
package dns
2+
3+
import (
4+
"reflect"
5+
"testing"
6+
"time"
7+
)
8+
9+
func TestParseResolverList(t *testing.T) {
10+
got, err := ParseResolverList([]string{"# public", "1.1.1.1", " 8.8.8.8:5353 ", "", "1.1.1.1:53", "2001:db8::1"})
11+
if err != nil {
12+
t.Fatal(err)
13+
}
14+
want := []string{"1.1.1.1:53", "8.8.8.8:5353", "[2001:db8::1]:53"}
15+
if !reflect.DeepEqual(got, want) {
16+
t.Fatalf("got %v, want %v", got, want)
17+
}
18+
if _, err := ParseResolverList([]string{"resolver.example.com"}); err == nil {
19+
t.Fatal("hostname accepted; resolvers must be IP addresses")
20+
}
21+
}
22+
23+
func TestPoolRotatesAndBenches(t *testing.T) {
24+
p := NewPool([]string{"192.0.2.1:53", "192.0.2.2:53", "192.0.2.3:53"}, time.Second)
25+
now := time.Unix(0, 0)
26+
p.now = func() time.Time { return now }
27+
28+
seen := map[string]int{}
29+
for i := 0; i < 6; i++ {
30+
seen[p.pick(nil).Addr]++
31+
}
32+
for _, m := range p.members {
33+
if seen[m.Addr] != 2 {
34+
t.Fatalf("round-robin uneven: %v", seen)
35+
}
36+
}
37+
if m := p.pick(p.members[0]); m == p.members[0] {
38+
t.Fatal("pick returned the resolver it was asked to avoid")
39+
}
40+
41+
// The first resolver fails every lookup and is benched after the window.
42+
bad := p.members[0]
43+
for i := 0; i < poolHealthMinLookups; i++ {
44+
p.report(bad, OutcomeTimeout)
45+
}
46+
if st := p.Stats()[0]; st.Benched != 1 || st.Failed != poolHealthMinLookups {
47+
t.Fatalf("stats after failures = %+v", st)
48+
}
49+
for i := 0; i < 10; i++ {
50+
if p.pick(nil) == bad {
51+
t.Fatal("benched resolver picked while others are healthy")
52+
}
53+
}
54+
// Once the bench expires it is back in rotation.
55+
now = now.Add(poolBenchFor + time.Second)
56+
back := false
57+
for i := 0; i < 3; i++ {
58+
back = back || p.pick(nil) == bad
59+
}
60+
if !back {
61+
t.Fatal("resolver never returned after its bench")
62+
}
63+
}

‎internal/output/summary.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@ type Summary struct {
3232
Verdict string `json:"verdict"`
3333
Reason string `json:"reason"`
3434
Targets []TargetSummary `json:"targets"`
35+
// Resolvers is the -r pool's per-resolver accounting; Resolver is then
36+
// the trusted resolver every pool hit was re-validated against.
37+
Resolvers []dns.ResolverStats `json:"resolvers,omitempty"`
3538
}
3639

3740
// TargetSummary is one scanned domain's part of a Summary.
@@ -45,6 +48,9 @@ type TargetSummary struct {
4548
Outcomes Outcomes `json:"outcomes"`
4649
QueriesSent int64 `json:"queries_sent"`
4750
SkippedCap int64 `json:"skipped_by_cap"`
51+
PoolHits int64 `json:"pool_hits,omitempty"` // -r: names a pool resolver said exist
52+
Confirmed int64 `json:"confirmed,omitempty"` // pool hits the trusted resolver confirmed
53+
Unconfirmed int64 `json:"unconfirmed,omitempty"` // pool hits the trusted resolver denied
4854
Wildcard bool `json:"wildcard"`
4955
Fingerprint []dns.Record `json:"wildcard_fingerprint,omitempty"`
5056
Verdict string `json:"verdict"`

‎internal/scan/options.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ type Options struct {
2828
Rate int
2929
MaxQueries int
3030
Exclude []string
31+
Pool *dns.Pool // -r resolver pool; DNSServer is then the trusted resolver
3132
NoAbort bool
3233
Verbose bool
3334
Logf dns.Logf
@@ -83,5 +84,6 @@ func (o Options) Config() Config {
8384
NoAbort: o.NoAbort,
8485
MaxQueries: o.MaxQueries,
8586
Exclude: o.Exclude,
87+
Pool: o.Pool,
8688
}
8789
}

0 commit comments

Comments
 (0)