Skip to content

Commit 661c7fd

Browse files
TMHSDigitalclaude
andcommitted
perf(cli): keep only names of past results when output shows no records (#183)
The CLI kept every output.Result of the current target, records included, for the -permute seeds and the -resume state, and every finished target's results for that state too. A resumed run re-emits them, but needs their records only when the output shows them. resultLog keeps whole results only for structured formats, -show-records or -diff. Otherwise it keeps the names, plus whole copies of the few results carrying a takeover hint, an internal scope or a permutation flag, and rebuilds the rest without records for the state file, so a resumed run writes the same text. A simulated 1M-name scan at 100% hits (text, -o) now peaks at 240 MB instead of 750 MB, against 233 MB at 1% hits (Windows peak working set). Structured formats are unchanged. Fixes #183 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0998773 commit 661c7fd

6 files changed

Lines changed: 167 additions & 11 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2727
- `-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).
2828

2929
### Changed
30+
- With text output (the default) and no `-show-records` or `-diff`, the CLI keeps only the names of the current target's results for `-permute` and the `-resume` state, plus the few results with a takeover hint, an internal address or a permutation, instead of every result with its records. A simulated 1M-name scan at 100% hits peaks at about 240 MB instead of 750 MB, against 233 MB at 1% hits. A resumed run re-emits the same text. Structured formats still keep the records a resumed run writes back (#183).
3031
- The scan dispatcher hands jobs to workers in small chunks (up to 8, fewer when there are few names per worker) over buffered channels, and each worker reports a chunk's jobs at once, instead of two unbuffered hand-offs per job. A 1M-name simulated scan at 50 workers (`make bench`) takes about 0.4 s instead of 1.6 s. A resumed scan may repeat up to one chunk per worker instead of one lookup per worker (#184).
3132
- Live UDP scans reuse sockets instead of opening and closing one per query, which capped throughput near 10k queries per second with nearly all CPU in socket teardown. A 100k-name scan against a loopback server now takes about 2 s instead of 12 s at `-t 200`. A late answer to an earlier query on a reused socket is ignored, as before, and cannot make a lookup count as refused (#180).
3233
- `-format json` streams its array instead of holding every result and then encoding the whole document, so its peak memory matches `-format jsonl` (a 300k-result run: about 200 MB instead of 420 MB). The output is byte for byte the same (#181).

‎main_results.go‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
package main
2+
3+
import "github.com/TMHSDigital/subenum/internal/output"
4+
5+
// resultLog keeps one target's results for the -permute seeds and, after an
6+
// interrupt, the -resume state, which re-emits them so the -o file and the
7+
// structured output hold the whole result set (#183).
8+
//
9+
// Re-emitting needs a result's records only when the output shows them, so
10+
// unless full is set the log keeps just the names, plus whole copies of the
11+
// few results that carry more than a name (a takeover hint, an internal
12+
// address scope, a -permute hit). On a run with a million hits that is tens
13+
// of megabytes instead of hundreds.
14+
type resultLog struct {
15+
full bool // keep every result whole, records included
16+
17+
all []output.Result // full: every result, in order
18+
19+
// Compact: every name in order, the whole result for the indexes that
20+
// are more than a name, and the Simulated flag every result shares.
21+
names []string
22+
extra map[int]output.Result
23+
simulated bool
24+
}
25+
26+
// add records one result.
27+
func (l *resultLog) add(r output.Result) {
28+
if l.full {
29+
l.all = append(l.all, r)
30+
return
31+
}
32+
if len(l.names) == 0 {
33+
l.simulated = r.Simulated
34+
}
35+
if r.Takeover != "" || r.Internal != "" || r.Permutation || r.Simulated != l.simulated {
36+
if l.extra == nil {
37+
l.extra = map[int]output.Result{}
38+
}
39+
l.extra[len(l.names)] = r
40+
}
41+
l.names = append(l.names, r.Subdomain)
42+
}
43+
44+
// results returns the logged results, in order. A compact log rebuilds them
45+
// without records.
46+
func (l *resultLog) results() []output.Result {
47+
if l.full {
48+
return l.all
49+
}
50+
if len(l.names) == 0 {
51+
return nil
52+
}
53+
out := make([]output.Result, len(l.names))
54+
for i, name := range l.names {
55+
if r, ok := l.extra[i]; ok {
56+
out[i] = r
57+
continue
58+
}
59+
out[i].Subdomain, out[i].Simulated = name, l.simulated
60+
}
61+
return out
62+
}
63+
64+
// mainPassNames returns the names the main pass found, leaving out
65+
// permutation hits: they are the -permute seeds.
66+
func (l *resultLog) mainPassNames() []string {
67+
if l.full {
68+
return mainPassNames(l.all)
69+
}
70+
names := make([]string, 0, len(l.names))
71+
for i, name := range l.names {
72+
if r, ok := l.extra[i]; ok && r.Permutation {
73+
continue
74+
}
75+
names = append(names, name)
76+
}
77+
return names
78+
}

‎main_results_test.go‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
package main
2+
3+
import (
4+
"reflect"
5+
"slices"
6+
"testing"
7+
8+
"github.com/TMHSDigital/subenum/internal/dns"
9+
"github.com/TMHSDigital/subenum/internal/output"
10+
"github.com/TMHSDigital/subenum/internal/scan"
11+
)
12+
13+
// TestResultLogCompact covers #183: without full, records are dropped but
14+
// everything else a resumed run re-emits survives, in order.
15+
func TestResultLogCompact(t *testing.T) {
16+
a := []dns.Record{{Type: "A", Value: "192.0.2.1"}}
17+
in := []output.Result{
18+
{Result: scan.Result{Subdomain: "www.example.com", Records: a, Simulated: true}},
19+
{Result: scan.Result{Subdomain: "blog.example.com", Records: a, Simulated: true, Takeover: "dangling"}},
20+
{Result: scan.Result{Subdomain: "intra.example.com", Records: a, Simulated: true, Internal: "private"}},
21+
{Result: scan.Result{Subdomain: "api-dev.example.com", Records: a, Simulated: true}, Permutation: true},
22+
{Result: scan.Result{Subdomain: "mail.example.com", Records: a, Simulated: true}},
23+
}
24+
var l resultLog
25+
for _, r := range in {
26+
l.add(r)
27+
}
28+
got := l.results()
29+
if len(got) != len(in) {
30+
t.Fatalf("results() returned %d results, want %d", len(got), len(in))
31+
}
32+
for i, r := range got {
33+
want := in[i]
34+
if want.Takeover == "" && want.Internal == "" && !want.Permutation {
35+
want.Records = nil // plain results lose only their records
36+
}
37+
if !reflect.DeepEqual(r, want) {
38+
t.Errorf("result %d = %+v, want %+v", i, r, want)
39+
}
40+
}
41+
if names := l.mainPassNames(); !slices.Equal(names, []string{"www.example.com", "blog.example.com", "intra.example.com", "mail.example.com"}) {
42+
t.Errorf("mainPassNames() = %q, want every name but the permutation hit", names)
43+
}
44+
if len(l.extra) != 3 {
45+
t.Errorf("compact log kept %d whole results, want 3 (takeover, internal, permutation)", len(l.extra))
46+
}
47+
}
48+
49+
// TestResultLogFull keeps every result whole, records included.
50+
func TestResultLogFull(t *testing.T) {
51+
in := []output.Result{
52+
{Result: scan.Result{Subdomain: "www.example.com", Records: []dns.Record{{Type: "A", Value: "192.0.2.1"}}}},
53+
{Result: scan.Result{Subdomain: "x.example.com"}, Permutation: true},
54+
}
55+
l := resultLog{full: true}
56+
for _, r := range in {
57+
l.add(r)
58+
}
59+
if !reflect.DeepEqual(l.results(), in) {
60+
t.Errorf("results() = %+v, want %+v", l.results(), in)
61+
}
62+
if names := l.mainPassNames(); !slices.Equal(names, []string{"www.example.com"}) {
63+
t.Errorf("mainPassNames() = %q", names)
64+
}
65+
if l.names != nil {
66+
t.Error("a full log should not also keep names")
67+
}
68+
}
69+
70+
func TestResultLogEmpty(t *testing.T) {
71+
var l resultLog
72+
if l.results() != nil || len(l.mainPassNames()) != 0 {
73+
t.Error("an empty log should have no results")
74+
}
75+
}

‎main_resume.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,7 @@ func saveResumeState(s *session) {
334334
continue
335335
}
336336
res := s.results[i]
337-
t := resumeTarget{Domain: domain, Results: s.runs[i], Entries: s.entryDigests[i], CT: res.ct.names, CTFetched: res.ct.fetched}
337+
t := resumeTarget{Domain: domain, Results: s.runs[i].results(), Entries: s.entryDigests[i], CT: res.ct.names, CTFetched: res.ct.fetched}
338338
switch {
339339
case res.done:
340340
t.Stats, t.Processed, t.Total = res.final.Stats, res.final.Processed, res.final.Total

‎main_resume_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ func TestSaveResumeStateKeepsEarlierPosition(t *testing.T) {
247247
status: []string{"interrupted", ""},
248248
entryDigests: []string{"d1", ""},
249249
results: []targetResult{{}, {}},
250-
runs: [][]output.Result{{{Result: scan.Result{Subdomain: "x.a.example"}}}, nil},
250+
runs: []resultLog{{names: []string{"x.a.example"}}, {}},
251251
})
252252
data, err := os.ReadFile(f.stateFile)
253253
if err != nil {
@@ -288,7 +288,7 @@ func TestSaveResumeStateRecordsFinishedPermutePass(t *testing.T) {
288288
final: scan.Event{Kind: scan.EventDone, Stats: scan.Stats{RootDone: 40}, Processed: 40, Total: 40},
289289
permutation: &permutationResult{candidates: 12, pass: targetResult{done: true, final: final}},
290290
}},
291-
runs: [][]output.Result{nil},
291+
runs: []resultLog{{}},
292292
})
293293
data, err := os.ReadFile(f.stateFile)
294294
if err != nil {

‎main_session.go‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -49,15 +49,15 @@ type session struct {
4949
outWriter *bufio.Writer // the -o file's buffer; nil without -o
5050
diff *differ
5151

52-
targetResults []output.Result // the current target's results: -permute seeds, saved on interrupt
52+
targetResults resultLog // the current target's results: -permute seeds, saved on interrupt
5353
takeovers []output.Result // every target's takeover candidates, for -sarif
5454
restored map[string]bool // names re-emitted from a -resume state
5555
refound int // restored names a resumed pass found again (#156)
5656

5757
status []string
5858
results []targetResult
59-
runs [][]output.Result // results per target, for -resume state
60-
entryDigests []string // per target, for -resume state
59+
runs []resultLog // results per target, for -resume state
60+
entryDigests []string // per target, for -resume state
6161
failedTargets int
6262
anyDone bool
6363
anyFailed bool
@@ -517,7 +517,7 @@ func (s *session) emitResult(ev scan.Event, permutation bool) {
517517
return
518518
}
519519
res := output.Result{Result: ev.Result, Permutation: permutation}
520-
s.targetResults = append(s.targetResults, res)
520+
s.targetResults.add(res)
521521
if res.Takeover != "" {
522522
s.takeovers = append(s.takeovers, res)
523523
}
@@ -545,7 +545,7 @@ func (s *session) scanAll(ctx context.Context) {
545545
n := len(s.targets)
546546
s.status = make([]string, n)
547547
s.results = make([]targetResult, n)
548-
s.runs = make([][]output.Result, n)
548+
s.runs = make([]resultLog, n)
549549
s.entryDigests = make([]string, n)
550550
s.started = time.Now()
551551
if s.f.maxDuration > 0 {
@@ -599,7 +599,9 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
599599
s.failedTargets++
600600
return true
601601
}
602-
s.targetResults = nil
602+
// A resumed run re-emits these results; their records are needed only
603+
// when the output shows them or -diff compares them (#183).
604+
s.targetResults = resultLog{full: s.format != output.FormatText || s.f.showRecords || s.diff != nil}
603605
s.entryDigests[i] = entryDigest(entries)
604606
saved := s.resume.target(target)
605607
var ctInfo targetCT
@@ -626,7 +628,7 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
626628
s.status[i] = "ok"
627629
return true
628630
}
629-
pass.resumeFrom, pass.resumeNames = saved.RootDone, mainPassNames(s.targetResults)
631+
pass.resumeFrom, pass.resumeNames = saved.RootDone, s.targetResults.mainPassNames()
630632
out.Info("Resuming %s at wordlist entry %d of %d (%d results restored)", target, saved.RootDone, len(entries), len(saved.Results))
631633
}
632634
res := s.mainPass(ctx, pass, mainSaved)
@@ -635,7 +637,7 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
635637
// main pass's names only, so a resumed pass rebuilds the same
636638
// candidate list, and it resumes from its own position (#107).
637639
if s.f.permute && res.done && !res.resolverAbort && s.signalCode.Load() == 0 {
638-
p := permuteInput{target: target, entries: entries, found: mainPassNames(s.targetResults), seeds: seeds, restoredHits: restoredHits}
640+
p := permuteInput{target: target, entries: entries, found: s.targetResults.mainPassNames(), seeds: seeds, restoredHits: restoredHits}
639641
if saved != nil {
640642
p.prev = saved.Permute
641643
}

0 commit comments

Comments
 (0)