Skip to content

Commit 826e908

Browse files
committed
Record which source proposed each name found (#273)
Results did not say whether a name came from the wordlist, a passive source, a zone transfer or the permutation pass, so the report could not show what each source contributed. JSON and JSONL results gain a sources list (wordlist, ct, wayback, axfr, or permute alone), CSV gains a last sources column joined with ";", and the report gains found per passive source and axfr_found per target. A resumed run recomputes sources from the saved source names. Text output is unchanged. The TUI's files, subenum mcp reports and pkg/subenum results do not carry sources yet. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com> Refs #273
1 parent 665e33c commit 826e908

12 files changed

Lines changed: 197 additions & 21 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- Each result says what proposed it: a `sources` list in JSON and JSONL and a last `sources` column in CSV (joined with `;`), holding `wordlist`, `ct`, `wayback` and `axfr`, every one that had the name, or `permute` alone for a `-permute` hit. A name found by `-recursive` below another is the wordlist's. The run-quality report counts the results each passive source had proposed (`found` in its `sources` entry) and a zone transfer's (`axfr_found`). Text output, the TUI's files and `subenum mcp` reports are unchanged (#273).
1112
- `subenum mcp`: `scan` and `lab_scan` accept an `offset` that skips that many candidates at the start of the word list (after dedupe), and a report whose `max_queries` left words untested carries `next_offset`, named in its `reason`. An agent can now work through the bundled 5,000-label list in several calls instead of testing the same first `-max-queries` names each time; every call stays within `-max-queries`, `-max-duration` and `-max-calls` (#313).
1213
- The TUI configuration form includes **Auto Rate**, **Max Duration** and **Retry Inconclusive** fields for `-auto-rate`, `-max-duration` and `-retry-inconclusive`. Max Duration accepts a Go duration such as `20m` or `1h30m`; leave it blank for no time limit (#274).
1314
- `subenum mcp` serves subdomain enumeration to AI agents over the Model Context Protocol (stdio). Tools: `lab_scan`, which scans the compiled-in lab scenarios on a loopback DNS server; `scan`, offered only with `-allow` and refusing any domain outside it; and `explain_stats`, which explains a `-stats` report or `-format jsonl` output. The operator sets the resolver, rate, timeout and budgets (`-dns-server`, `-rate`, `-timeout`, `-max-queries`, `-max-duration`, which cannot be turned off). A call can lower `max_queries`, never raise it; unknown arguments are refused, and calls run one at a time. Each scan returns its results with a verdict (`complete`, `degraded`, `unreliable` or `not_run`), and sends progress notifications about once a second when the call carries a `progressToken` (#267). `-sources ct,wayback` lets `scan` calls seed candidates from passive sources through a `sources` argument; it is off by default, and the report counts each source's names or names its error (#268). `scan` and `lab_scan` accept `retry_inconclusive`, `auto_rate`, `exclude` and `depth` (up to 3), all within the operator's caps, so an agent can act on a verdict that is not complete; the report adds `retried`, `recovered`, `rate_backoffs` and `excluded` (#269). Their report is also declared as an `outputSchema` and sent as `structuredContent`, beside the same JSON as text (#270). `-max-calls` (default 100) caps the `scan` calls one session may make, and a client that stops reading replies ends the session, cancelling queued calls instead of scanning for nobody (#240). A request line over 16 MiB gets a JSON-RPC parse error instead of ending the session, a request reusing the ID of one still running is refused, and a scan cancelled before testing any name reports `not_run` rather than `degraded` (#241). The JSON-RPC layer is a small hand-written `internal/mcp` package, so there is no new dependency (#131). `lab_scan`'s `scenario` argument describes what each lab teaches, taken from the opening comment of its zone file (#275).

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,12 @@ subenum -sources ct,wayback example.com
459459

460460
The sources of a target are asked at the same time, and a busy answer (429, 502, 503, 504) or a dropped connection is retried twice with a short backoff, honouring `Retry-After`, within the source's 60-second limit. The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. The Wayback Machine is asked for at most 200,000 URLs and 64 MiB, and crt.sh's answer is read up to 64 MiB; when a source reaches a limit its names are still used, a note says some hosts may be missing, and its entry is marked `truncated`. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
461461

462+
Each result says what proposed it: a `sources` list in JSON and JSONL, and a `sources` column in CSV (joined with `;`). The values are `wordlist`, `ct`, `wayback` and `axfr`, every one that had the name, in that order, or `permute` alone for a `-permute` hit; text output is unchanged. The report counts the results each source had proposed as `found` in its `sources` entry, and `axfr_found` for a zone transfer, so you can see whether a source earns its request:
463+
464+
```bash
465+
subenum -sources ct,wayback -format jsonl example.com | jq -r 'select(.sources == ["wayback"]) | .subdomain'
466+
```
467+
462468
### Zone transfers: the whole zone in one query
463469

464470
A name server that allows anyone a zone transfer (AXFR) lists every name in the zone, and finding one is a misconfiguration worth reporting in itself. `-axfr` looks up the target's name servers through `-dns-server` and asks each of them for a transfer over TCP before the scan. Most refuse, which is correct and stays quiet (`-v` shows each refusal). When one allows it, subenum says so on stderr, names the server in the report (`axfr_allowed`, with `axfr_names`), and adds the names as candidates after the wordlist and as `-permute` seeds. Like `-ct` names they are only candidates: each is still looked up, so a stale entry is never reported.

‎examples/advanced_usage.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -271,11 +271,14 @@ Emits a single JSON array of objects, each with the subdomain and its resolved r
271271
[
272272
{
273273
"subdomain": "www.example.com",
274-
"records": [{ "type": "A", "value": "93.184.216.34" }]
274+
"records": [{ "type": "A", "value": "93.184.216.34" }],
275+
"sources": ["wordlist"]
275276
}
276277
]
277278
```
278279

280+
`sources` names what proposed the name: `wordlist`, `ct`, `wayback` and `axfr`, every one that had it, or `permute` alone for a `-permute` hit.
281+
279282
JSON is streamed: each element is written as it is found, and the array is closed when the scan ends, so a run interrupted with Ctrl+C still leaves a valid document.
280283

281284
### CSV
@@ -287,17 +290,18 @@ Streams a header followed by one row per record. The first three columns are alw
287290
| `simulated` | `-simulate` (every row is `true`) |
288291
| `takeover_candidate` | `-type` includes `CNAME` or `NS` (see takeover hints in the README) |
289292
| `change` | `-diff` (`added`, `removed` or, with `-diff-records`, `changed`) |
293+
| `sources` | always, last: what proposed the name, joined with `;` (empty for a `removed` row) |
290294

291295
```bash
292296
./subenum -w wordlist.txt -format csv -o results.csv example.com
293297
```
294298

295299
```csv
296-
subdomain,type,value
297-
www.example.com,A,93.184.216.34
300+
subdomain,type,value,sources
301+
www.example.com,A,93.184.216.34,wordlist
298302
```
299303

300-
With `-type A,CNAME` the header is `subdomain,type,value,takeover_candidate`. Read columns by header name rather than position.
304+
With `-type A,CNAME` the header is `subdomain,type,value,takeover_candidate,sources`. Read columns by header name rather than position.
301305

302306
## Continuous Integration / Automated Testing
303307

‎internal/cli/axfr_test.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,9 @@ func TestE2EZoneTransfer(t *testing.T) {
9090
if !slices.Equal(tg.AXFRAllowed, []string{"ns1.example.test"}) || tg.AXFRNames != 3 {
9191
t.Errorf("report axfr_allowed = %q, axfr_names = %d; want ns1.example.test and 3", tg.AXFRAllowed, tg.AXFRNames)
9292
}
93+
if tg.AXFRFound != 2 { // www and admin-legacy resolve, gone does not (#273)
94+
t.Errorf("report axfr_found = %d, want 2", tg.AXFRFound)
95+
}
9396
}
9497

9598
// TestE2EZoneTransferRefused: a refusal, the correct configuration, is not

‎internal/cli/cli.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -706,6 +706,7 @@ type targetResult struct {
706706
permutation *permutationResult
707707
sources []targetSource // -ct and -sources names, kept for the report and resume state
708708
axfr targetAXFR // -axfr zone transfer, kept for the report and resume state
709+
foundBy map[string]int // results per source that proposed them, for the report (#273)
709710
}
710711

711712
// unfinished reports whether the pass stopped with candidates untested, on

‎internal/cli/ct.go‎

Lines changed: 65 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,12 @@ package cli
33
import (
44
"context"
55
"errors"
6+
"strings"
67
"sync"
78

89
"github.com/TMHSDigital/subenum/internal/output"
910
"github.com/TMHSDigital/subenum/internal/passive"
11+
"github.com/TMHSDigital/subenum/internal/wordlist"
1012
)
1113

1214
// targetSource is one passive source's lookup for a target (#130, #193).
@@ -107,12 +109,31 @@ type candidates struct {
107109
seeds map[string]struct{} // -seeds names plus every added name, full
108110
have map[string]bool // entries, once a source has names
109111
owned bool // seeds is this target's copy, not the shared -seeds map
112+
113+
// Provenance (#273): from has every name a source proposed, as a bit per
114+
// label of who had it (bit 0 is the wordlist). Names missing from it are
115+
// the wordlist's alone, so a scan without sources keeps no map.
116+
labels []string
117+
from map[string]uint8
118+
lists map[uint8][]string // the labels of each bit set in use, shared by its results
110119
}
111120

112-
// add appends the names that are not already entries, after them, so the
113-
// wordlist's own resume positions stay put, and returns how many were new.
114-
// Names too long for the target are dropped by wordlist.Build.
115-
func (c *candidates) add(names []string) int {
121+
// Result sources that are not a passive source's -sources name (#273).
122+
const (
123+
sourceWordlist = "wordlist"
124+
sourceAXFR = "axfr"
125+
sourcePermute = "permute"
126+
)
127+
128+
var (
129+
fromWordlist = []string{sourceWordlist}
130+
fromPermute = []string{sourcePermute}
131+
)
132+
133+
// add appends source's names that are not already entries, after them, so
134+
// the wordlist's own resume positions stay put, and returns how many were
135+
// new. Names too long for the target are dropped by wordlist.Build.
136+
func (c *candidates) add(source string, names []string) int {
116137
if len(names) == 0 {
117138
return 0
118139
}
@@ -121,6 +142,19 @@ func (c *candidates) add(names []string) int {
121142
for _, e := range c.entries {
122143
c.have[e] = true
123144
}
145+
c.labels, c.from = []string{sourceWordlist}, map[string]uint8{}
146+
}
147+
// Before the merge, while have still tells a wordlist entry from a name
148+
// only an earlier source had (those are in from).
149+
bit := uint8(1) << len(c.labels)
150+
c.labels = append(c.labels, source)
151+
proposed, _, _ := wordlist.Build(names, c.target)
152+
for _, n := range proposed {
153+
mask, seen := c.from[n]
154+
if !seen && c.have[n] {
155+
mask = 1
156+
}
157+
c.from[n] = mask | bit
124158
}
125159
var added int
126160
c.entries, added = passive.Merge(c.entries, c.have, names, c.target)
@@ -136,3 +170,30 @@ func (c *candidates) add(names []string) int {
136170
}
137171
return added
138172
}
173+
174+
// sourcesOf names what proposed a main-pass result (#273). A name no source
175+
// gave is the wordlist's: an entry, or under -recursive an entry below a
176+
// found name.
177+
func (c *candidates) sourcesOf(name string) []string {
178+
if c == nil {
179+
return fromWordlist
180+
}
181+
mask, ok := c.from[strings.TrimSuffix(name, "."+c.target)]
182+
if !ok {
183+
return fromWordlist
184+
}
185+
if list, ok := c.lists[mask]; ok {
186+
return list
187+
}
188+
var list []string
189+
for i, label := range c.labels {
190+
if mask&(1<<i) != 0 {
191+
list = append(list, label)
192+
}
193+
}
194+
if c.lists == nil {
195+
c.lists = map[uint8][]string{}
196+
}
197+
c.lists[mask] = list
198+
return list
199+
}

‎internal/cli/ct_test.go‎

Lines changed: 65 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
package cli
22

33
import (
4+
"encoding/csv"
45
"encoding/json"
6+
"maps"
57
"net/http"
68
"net/http/httptest"
79
"os"
@@ -116,13 +118,55 @@ func TestE2EPassiveSources(t *testing.T) {
116118
t.Fatalf("exit %d, results %q: want the CT and the Wayback name", code, out)
117119
}
118120
tg := report()
119-
if len(tg.Sources) != 2 || tg.Sources[0] != (output.SourceSummary{Name: "ct", Names: 1}) || tg.Sources[1] != (output.SourceSummary{Name: "wayback", Names: 2}) {
120-
t.Errorf("report sources = %+v, want ct 1 and wayback 2", tg.Sources)
121+
if len(tg.Sources) != 2 || tg.Sources[0] != (output.SourceSummary{Name: "ct", Names: 1, Found: 1}) || tg.Sources[1] != (output.SourceSummary{Name: "wayback", Names: 2, Found: 2}) {
122+
t.Errorf("report sources = %+v, want ct 1 (1 found) and wayback 2 (2 found)", tg.Sources)
121123
}
122124
if tg.CTNames != 1 {
123125
t.Errorf("ct_names = %d, want 1 (the ct entry, for compatibility)", tg.CTNames)
124126
}
125127

128+
// #273: each result names every source that proposed it, in JSONL and in
129+
// a CSV column.
130+
want := map[string]string{
131+
"www.example.com": "wordlist;wayback",
132+
"legacy-billing.example.com": "ct",
133+
"old-shop.example.com": "wayback",
134+
}
135+
sourceArgs := runArgs{sourceURLs: map[string]string{"ct": ctUp.URL + "/", "wayback": wayback.URL}}
136+
scanArgs := []string{"-sources", "ct,wayback", "-dns-server", srv.Addr, "-type", "A", "-progress=false", "-w", wl, "example.com"}
137+
sourceArgs.args = append([]string{"-format", "jsonl"}, scanArgs...)
138+
code, out = runCLIWith(t, sourceArgs)
139+
got := map[string]string{}
140+
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
141+
var r struct {
142+
Type string `json:"type"`
143+
Subdomain string `json:"subdomain"`
144+
Sources []string `json:"sources"`
145+
}
146+
if err := json.Unmarshal([]byte(line), &r); err != nil {
147+
t.Fatalf("bad line %q: %v", line, err)
148+
}
149+
if r.Type != "summary" {
150+
got[r.Subdomain] = strings.Join(r.Sources, ";")
151+
}
152+
}
153+
if code != 0 || !maps.Equal(got, want) {
154+
t.Errorf("jsonl: exit %d, sources = %v, want %v", code, got, want)
155+
}
156+
sourceArgs.args = append([]string{"-format", "csv"}, scanArgs...)
157+
code, out = runCLIWith(t, sourceArgs)
158+
rows, err := csv.NewReader(strings.NewReader(out)).ReadAll()
159+
if err != nil || len(rows) == 0 || !slices.Equal(rows[0], []string{"subdomain", "type", "value", "sources"}) {
160+
t.Fatalf("csv: exit %d, err %v, rows %q; want a sources column", code, err, rows)
161+
}
162+
got = map[string]string{}
163+
for _, row := range rows[1:] {
164+
got[row[0]] = row[3]
165+
}
166+
if !maps.Equal(got, want) {
167+
t.Errorf("csv: sources = %v, want %v", got, want)
168+
}
169+
126170
code, out = runCLIWith(t, runArgs{sourceURLs: map[string]string{"ct": ctUp.URL + "/", "wayback": down.URL},
127171
args: []string{"-sources", "wayback,ct", "-dns-server", srv.Addr, "-type", "A", "-progress=false", "-stats", stats, "-w", wl, "example.com"}})
128172
if code != 0 || !strings.Contains(out, "legacy-billing.example.com") || strings.Contains(out, "old-shop") {
@@ -143,15 +187,31 @@ func TestE2EPassiveSources(t *testing.T) {
143187
func TestCandidates(t *testing.T) {
144188
shared := map[string]struct{}{"seed.example.com": {}}
145189
c := &candidates{target: "example.com", entries: []string{"www", "mail"}, seeds: shared}
146-
if n := c.add(nil); n != 0 || len(c.seeds) != 1 {
190+
if n := c.add("ct", nil); n != 0 || len(c.seeds) != 1 {
147191
t.Fatalf("no names: added %d, seeds %v", n, c.seeds)
148192
}
149-
if n := c.add([]string{"api", "www", "bad name!"}); n != 1 {
193+
if got := c.sourcesOf("www.example.com"); !slices.Equal(got, []string{"wordlist"}) {
194+
t.Errorf("no source names: sources = %v, want wordlist", got)
195+
}
196+
if n := c.add("wayback", []string{"api", "www", "bad name!"}); n != 1 {
150197
t.Errorf("first source added %d, want 1 (api)", n)
151198
}
152-
if n := c.add([]string{"api", "old", "OLD"}); n != 1 {
199+
if n := c.add("axfr", []string{"api", "old", "OLD"}); n != 1 {
153200
t.Errorf("second source added %d, want 1 (old)", n)
154201
}
202+
// #273: every source that proposed a name, the wordlist first; a name no
203+
// source gave (an entry, or a -recursive child) is the wordlist's.
204+
for name, want := range map[string][]string{
205+
"www.example.com": {"wordlist", "wayback"},
206+
"mail.example.com": {"wordlist"},
207+
"api.example.com": {"wayback", "axfr"},
208+
"old.example.com": {"axfr"},
209+
"mail.api.example.com": {"wordlist"},
210+
} {
211+
if got := c.sourcesOf(name); !slices.Equal(got, want) {
212+
t.Errorf("sources of %s = %v, want %v", name, got, want)
213+
}
214+
}
155215
if want := []string{"www", "mail", "api", "old"}; !slices.Equal(c.entries, want) {
156216
t.Errorf("entries = %v, want %v", c.entries, want)
157217
}

‎internal/cli/e2e_test.go‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import (
1010
"net"
1111
"os"
1212
"path/filepath"
13+
"slices"
1314
"sort"
1415
"strings"
1516
"sync"
@@ -675,20 +676,29 @@ func TestE2EPermute(t *testing.T) {
675676
if code != 0 {
676677
t.Fatalf("exit %d", code)
677678
}
678-
results := map[string]bool{} // name -> permutation
679+
results := map[string]bool{} // name -> permutation
680+
sources := map[string][]string{} // name -> sources (#273)
679681
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
680682
var r struct {
681-
Type string `json:"type"`
682-
Subdomain string `json:"subdomain"`
683-
Permutation bool `json:"permutation"`
683+
Type string `json:"type"`
684+
Subdomain string `json:"subdomain"`
685+
Permutation bool `json:"permutation"`
686+
Sources []string `json:"sources"`
684687
}
685688
if err := json.Unmarshal([]byte(line), &r); err != nil {
686689
t.Fatalf("bad line %q: %v", line, err)
687690
}
688691
if r.Type != "summary" {
689692
results[r.Subdomain] = r.Permutation
693+
sources[r.Subdomain] = r.Sources
690694
}
691695
}
696+
if got := sources["api-dev.example.com"]; !slices.Equal(got, []string{"permute"}) {
697+
t.Errorf("api-dev.example.com sources = %v, want permute", got)
698+
}
699+
if got := sources["api.example.com"]; !slices.Equal(got, []string{"wordlist"}) {
700+
t.Errorf("api.example.com sources = %v, want wordlist", got)
701+
}
692702
perm, ok := results["api-dev.example.com"]
693703
if !ok || !perm {
694704
t.Fatalf("results = %v, want api-dev.example.com found by the permutation pass", results)

‎internal/cli/session.go‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,8 @@ type session struct {
5151

5252
targetResults resultLog // the current target's results: -permute seeds, saved on interrupt
5353
takeovers []output.Result // every target's takeover candidates, for -sarif
54+
cands *candidates // the current target's candidates: what proposed each name (#273)
55+
foundBy map[string]int // the current target's results per source (#273)
5456
restored map[string]bool // names re-emitted from a -resume state
5557
refound int // restored names a resumed pass found again (#156)
5658

@@ -462,6 +464,7 @@ func (s *session) openOutputs() bool {
462464
w.SetShowRecords(f.showRecords)
463465
w.SetSeed(f.seed)
464466
w.SetTakeoverColumn(dns.HintsTakeover(s.in.recordTypes))
467+
w.SetSourcesColumn(true)
465468
}
466469
return true
467470
}
@@ -541,7 +544,13 @@ func (s *session) emitResult(ev scan.Event, permutation bool) {
541544
s.refound++
542545
return
543546
}
544-
res := output.Result{Result: ev.Result, Permutation: permutation}
547+
res := output.Result{Result: ev.Result, Permutation: permutation, Sources: fromPermute}
548+
if !permutation {
549+
res.Sources = s.cands.sourcesOf(res.Subdomain)
550+
}
551+
for _, src := range res.Sources {
552+
s.foundBy[src]++
553+
}
545554
s.targetResults.add(res)
546555
if res.Takeover != "" {
547556
s.takeovers = append(s.takeovers, res)
@@ -633,9 +642,10 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
633642
// like a wordlist entry (#130, #193). One set of the entries serves
634643
// every source and the zone transfer (#248).
635644
cands := &candidates{target: target, entries: entries, seeds: s.seeds}
645+
s.cands, s.foundBy = cands, map[string]int{}
636646
sourceInfo := passiveNames(ctx, s.in.sources, target, saved, out, s.a.sourceURLs)
637647
for _, src := range sourceInfo {
638-
added := cands.add(src.names)
648+
added := cands.add(src.source, src.names)
639649
if src.fetched {
640650
out.Info("%s: %d names under %s, %d of them new candidates", passive.Labels[src.source], len(src.names), target, added)
641651
}
@@ -645,7 +655,7 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
645655
// Transferred names are candidates like -ct's: each is still looked
646656
// up, so only names that resolve are reported (#85).
647657
axfrInfo = zoneTransfer(ctx, s.f, s.a.transferPort(), target, saved, out)
648-
added := cands.add(axfrInfo.names)
658+
added := cands.add(sourceAXFR, axfrInfo.names)
649659
if len(axfrInfo.allowed) > 0 {
650660
out.Info("Zone transfer: %d names under %s, %d of them new candidates", len(axfrInfo.names), target, added)
651661
}
@@ -691,6 +701,7 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
691701
}
692702
res.sources = sourceInfo
693703
res.axfr = axfrInfo
704+
res.foundBy = s.foundBy
694705
return s.recordTarget(i, res)
695706
}
696707

0 commit comments

Comments
 (0)