Skip to content

Commit d1da60f

Browse files
TMHSDigitalclaude
andcommitted
Share one idle UDP socket budget across pool members (#252)
Each -r pool member had its own UDP dialer with up to 512 idle sockets, and nothing closed them before the run ended, so a long resolver list held about one socket per member and could hit file-descriptor limits. NewPool now gives its members one shared idle budget (512), counted as sockets enter and leave the idle sets, and Run closes the pool's idle sockets when the scan ends via the new Pool.CloseIdleConnections. Fixes #252 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 3a68ccb commit d1da60f

7 files changed

Lines changed: 129 additions & 11 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).
3737
- Passive sources (`-sources`, `-ct`, `Config.Sources`) retry a busy answer (429, 502, 503, 504) or a dropped connection twice, after 2 and 5 seconds or the server's `Retry-After` (at most 30 seconds), within each source's 60-second limit; crt.sh often answers 502 or 503 under load, which used to cost the scan all of its names. A target's sources are now asked at the same time instead of one after another (#271).
3838
- When the Wayback Machine's answer reaches the 200,000-URL or 64 MiB limit, a note says its results were truncated and some hosts may be missing, the report's `sources` entry (and the MCP report's) is marked `truncated`, `explain_stats` mentions it, and the library sends a `NoticeSource` event; the names it did return are still scanned. It used to stop silently (#272).
39+
- The members of an `-r` resolver pool share one budget of 512 idle UDP sockets, and a scan closes them when it ends, instead of each member keeping its own sockets for the whole run; a 10k-resolver list could hold 10k sockets open and hit file-descriptor limits (#252).
3940
- A Wayback Machine answer with one archived URL over 1 MiB skips that line instead of failing and losing every Wayback name (#259).
4041
- crt.sh's answer is read one entry at a time instead of decoded whole, so memory follows the names rather than the body. An answer that reaches the 64 MiB limit keeps the names read before the cut and is reported as truncated, like Wayback's, instead of failing and losing every CT name (#251).
4142
- The TUI runs on Bubble Tea v2: `charm.land/bubbletea/v2` v2.1.1, `bubbles/v2` v2.2.1 and `lipgloss/v2` v2.0.6 replace the v1 modules, dropping v1-only helpers that had not had a release in years. Keys, screens and the form work as before; the `go 1.26.0` minimum is unchanged (#188).

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
141141
* **Purpose**: This is the core component responsible for performing the DNS lookups for each candidate name and classifying what happened. It also provides wildcard detection and wire-level rate limiting.
142142
* **Implementation**:
143143
* **Resolver and transports**: `dns.NewResolver` builds a `net.Resolver` with `PreferGo: true` whose `Dial` hook always connects to the configured server through a transport chosen from the `-dns-server` value (`dns.Transport`, `internal/dns/transport.go`):
144-
* `ip:port` - UDP, with UDP sockets reused between queries; a truncated answer (TC=1) falls back to TCP, which dials per query.
144+
* `ip:port` - UDP, with UDP sockets reused between queries (at most 512 idle, a budget an `-r` pool shares across its members); a truncated answer (TC=1) falls back to TCP, which dials per query.
145145
* `tls://host[:853]` - DNS over TLS (RFC 7858), over a small pool of reused TLS connections.
146146
* `https://host/path` - DNS over HTTPS (RFC 8484), each query sent as an HTTP POST of `application/dns-message`.
147147

‎internal/dns/pool.go‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,15 +120,29 @@ func ParseResolverList(lines []string) ([]string, error) {
120120
return out, nil
121121
}
122122

123-
// NewPool builds a pool of resolvers from NewResolver.
123+
// NewPool builds a pool of resolvers from NewResolver. Their idle UDP
124+
// sockets share one budget, so a long -r list does not hold a socket open
125+
// per member for the whole run (#252).
124126
func NewPool(addrs []string, timeout time.Duration) *Pool {
125127
p := &Pool{now: time.Now, sleep: sleepCtx, canarySeen: map[string]bool{}}
128+
budget := newIdleBudget()
126129
for _, a := range addrs {
127-
p.members = append(p.members, &PoolResolver{Addr: a, resolver: NewResolver(timeout, a), stats: ResolverStats{Addr: a}})
130+
p.members = append(p.members, &PoolResolver{Addr: a, resolver: newResolver(timeout, a, budget), stats: ResolverStats{Addr: a}})
128131
}
129132
return p
130133
}
131134

135+
// CloseIdleConnections closes the sockets every member keeps for reuse, as
136+
// CloseIdleConnections does for one resolver. A scan calls it when it ends.
137+
func (p *Pool) CloseIdleConnections() {
138+
if p == nil {
139+
return
140+
}
141+
for _, m := range p.members {
142+
CloseIdleConnections(m.resolver)
143+
}
144+
}
145+
132146
// SetRatePerResolver caps the DNS queries sent to each member at perSecond,
133147
// spaced like RateLimiter; perSecond <= 0 removes the cap. Lookups prefer a
134148
// member whose cap has room and wait only when every member's is spent. A

‎internal/dns/resolver.go‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,14 @@ func ParseTypes(s string) ([]string, error) {
141141
// ResolveTypes, a dial beyond the slots reserved up front takes another rate
142142
// slot, which keeps -rate exact however many queries Go decides to send (#50).
143143
func NewResolver(timeout time.Duration, dnsServer string) *net.Resolver {
144-
dl := newDialer(timeout, dnsServer) // UDP/TCP, DoT or DoH (#86)
144+
return newResolver(timeout, dnsServer, nil)
145+
}
146+
147+
// newResolver is NewResolver with its idle UDP sockets counted against
148+
// budget, which a pool shares across its members (#252); nil is a budget of
149+
// the resolver's own.
150+
func newResolver(timeout time.Duration, dnsServer string, budget *idleBudget) *net.Resolver {
151+
dl := newDialer(timeout, dnsServer, budget) // UDP/TCP, DoT or DoH (#86)
145152
r := &net.Resolver{
146153
PreferGo: true,
147154
Dial: func(dialCtx context.Context, network, _ string) (net.Conn, error) {

‎internal/dns/transport.go‎

Lines changed: 40 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import (
1515
"runtime"
1616
"strings"
1717
"sync"
18+
"sync/atomic"
1819
"time"
1920
"weak"
2021

@@ -93,8 +94,10 @@ func CloseIdleConnections(r *net.Resolver) {
9394
const dotIdleConns = 128
9495

9596
// newDialer returns the dialer for a -dns-server value. It is validated
96-
// earlier (validate.DNSServer), so errors here fall back to UDP.
97-
func newDialer(timeout time.Duration, server string) dialer {
97+
// earlier (validate.DNSServer), so errors here fall back to UDP. A UDP
98+
// dialer keeps its idle sockets within budget, which a pool shares across
99+
// its members; nil gives it a budget of its own.
100+
func newDialer(timeout time.Duration, server string, budget *idleBudget) dialer {
98101
switch Transport(server) {
99102
case TransportTLS:
100103
addr, name, err := validate.DoTAddress(server)
@@ -126,13 +129,40 @@ func newDialer(timeout time.Duration, server string) dialer {
126129
},
127130
}
128131
}
129-
return newUDPDialer(server, timeout)
132+
d := newUDPDialer(server, timeout)
133+
if budget != nil {
134+
d.budget = budget
135+
}
136+
return d
130137
}
131138

132139
// udpIdleConns bounds the UDP sockets kept for reuse; it covers a busy -t
133-
// without holding thousands of sockets open.
140+
// without holding thousands of sockets open. A resolver pool shares one
141+
// such budget across its members, so a 10k-resolver -r list keeps at most
142+
// this many idle sockets, not one or more per member (#252).
134143
const udpIdleConns = 512
135144

145+
// idleBudget counts idle UDP sockets against a cap shared by one or more
146+
// dialers.
147+
type idleBudget struct {
148+
n atomic.Int64
149+
max int64
150+
}
151+
152+
func newIdleBudget() *idleBudget { return &idleBudget{max: udpIdleConns} }
153+
154+
// take reserves room for one more idle socket, if there is any.
155+
func (b *idleBudget) take() bool {
156+
if b.n.Add(1) > b.max {
157+
b.n.Add(-1)
158+
return false
159+
}
160+
return true
161+
}
162+
163+
// release gives back the room of a socket that left the idle set.
164+
func (b *idleBudget) release() { b.n.Add(-1) }
165+
136166
// udpDialer reuses UDP sockets between queries (#180). Opening and closing
137167
// a socket per query capped live scans near 10k queries per second, nearly
138168
// all of it in socket teardown. Reuse is safe because Go's resolver reads
@@ -143,16 +173,18 @@ type udpDialer struct {
143173
addr string
144174
timeout time.Duration
145175
idle chan *net.UDPConn
176+
budget *idleBudget // room for idle sockets, possibly shared
146177
}
147178

148179
func newUDPDialer(addr string, timeout time.Duration) *udpDialer {
149-
return &udpDialer{addr: addr, timeout: timeout, idle: make(chan *net.UDPConn, udpIdleConns)}
180+
return &udpDialer{addr: addr, timeout: timeout, idle: make(chan *net.UDPConn, udpIdleConns), budget: newIdleBudget()}
150181
}
151182

152183
func (d *udpDialer) dial(ctx context.Context, network string) (net.Conn, error) {
153184
if network == "udp" {
154185
select {
155186
case u := <-d.idle:
187+
d.budget.release()
156188
return &pooledUDPConn{UDPConn: u, pool: d}, nil
157189
default:
158190
}
@@ -173,6 +205,7 @@ func (d *udpDialer) closeIdle() {
173205
for {
174206
select {
175207
case u := <-d.idle:
208+
d.budget.release()
176209
_ = u.Close()
177210
default:
178211
return
@@ -211,13 +244,14 @@ func (c *pooledUDPConn) Write(p []byte) (int, error) {
211244
}
212245

213246
func (c *pooledUDPConn) Close() error {
214-
if c.bad || c.SetDeadline(time.Time{}) != nil {
247+
if c.bad || c.SetDeadline(time.Time{}) != nil || !c.pool.budget.take() {
215248
return c.UDPConn.Close()
216249
}
217250
select {
218251
case c.pool.idle <- c.UDPConn:
219252
return nil
220253
default:
254+
c.pool.budget.release()
221255
return c.UDPConn.Close()
222256
}
223257
}

‎internal/dns/udppool_test.go‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,3 +141,63 @@ func TestPooledSocketSkipsStaleReply(t *testing.T) {
141141
t.Errorf("SERVFAIL after a stale REFUSED: %v, want other (not refused)", o)
142142
}
143143
}
144+
145+
// TestPoolSharesIdleSocketBudget covers #252: a pool's members keep their
146+
// idle UDP sockets within one shared budget, however many members there
147+
// are, and the pool closes them all at the end of a scan.
148+
func TestPoolSharesIdleSocketBudget(t *testing.T) {
149+
srv := dnstest.Start(t, func(dnstest.Query) dnstest.Reply { return dnstest.NXDomain })
150+
addrs := make([]string, 6)
151+
for i := range addrs {
152+
addrs[i] = srv.Addr // distinct members, same server
153+
}
154+
p := NewPool(addrs, time.Second)
155+
budget := dialerOf(t, p.members[0].resolver).budget
156+
budget.max = 4
157+
idle := func() int {
158+
n := 0
159+
for _, m := range p.members {
160+
d := dialerOf(t, m.resolver)
161+
if d.budget != budget {
162+
t.Fatal("pool members do not share one idle budget")
163+
}
164+
n += len(d.idle)
165+
}
166+
return n
167+
}
168+
for _, m := range p.members {
169+
// Two sockets open at once per member, then both closed.
170+
d := dialerOf(t, m.resolver)
171+
c1, err := d.dial(context.Background(), "udp")
172+
if err != nil {
173+
t.Fatal(err)
174+
}
175+
c2, err := d.dial(context.Background(), "udp")
176+
if err != nil {
177+
t.Fatal(err)
178+
}
179+
_ = c1.Close()
180+
_ = c2.Close()
181+
}
182+
if n := idle(); n != 4 || budget.n.Load() != 4 {
183+
t.Errorf("%d idle sockets (budget counts %d) across %d members, want the shared cap of 4", n, budget.n.Load(), len(p.members))
184+
}
185+
p.CloseIdleConnections()
186+
if n := idle(); n != 0 || budget.n.Load() != 0 {
187+
t.Errorf("after CloseIdleConnections: %d idle sockets, budget counts %d; want 0", n, budget.n.Load())
188+
}
189+
}
190+
191+
// dialerOf returns the UDP dialer behind a resolver from NewResolver.
192+
func dialerOf(t *testing.T, r *net.Resolver) *udpDialer {
193+
t.Helper()
194+
v, ok := pooledDialers.Load(weak.Make(r))
195+
if !ok {
196+
t.Fatal("resolver has no registered dialer")
197+
}
198+
d, ok := v.(*udpDialer)
199+
if !ok {
200+
t.Fatalf("dialer is %T, want *udpDialer", v)
201+
}
202+
return d
203+
}

‎internal/scan/runner.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -584,8 +584,10 @@ func Run(ctx context.Context, cfg Config, events chan<- Event) {
584584
if cfg.Resolver == nil {
585585
cfg.Resolver = dns.NewResolver(cfg.Timeout, cfg.DNSServer)
586586
}
587-
// DoT and DoH connections kept for reuse are not held past the scan (#113).
587+
// DoT and DoH connections kept for reuse are not held past the scan
588+
// (#113), nor are the -r pool members' sockets (#252).
588589
defer dns.CloseIdleConnections(cfg.Resolver)
590+
defer cfg.Pool.CloseIdleConnections()
589591
var jobLimiter *dns.RateLimiter
590592
if cfg.Simulate || cfg.resolveHook != nil {
591593
jobLimiter = limiter

0 commit comments

Comments
 (0)