Skip to content

Commit bb3843a

Browse files
committed
Add subenum diff to compare two results files offline (#310)
Comparing two saved results needed a new scan with -diff, which sends queries and needs the network. subenum diff [-records] [-type ...] [-format ...] <previous> <current> reads both files and prints added (+), missing (-) and, with -records, changed (~) names, without a DNS query. It exits 4 when the files differ and 0 when they match, like -diff. It is in the shell completions, the man page, the CLI guide and the README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Fixes #310
1 parent 19a3bb5 commit bb3843a

10 files changed

Lines changed: 213 additions & 11 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `subenum diff [-records] [-type A,AAAA,CNAME] [-format text|json|jsonl|csv] <previous> <current>` compares two saved results files, in any format subenum writes, without sending a DNS query: `+` for names only in the second file, `-` for names only in the first, and with `-records` `~` for names whose records differ, compared as `-diff-records` does. In JSON, JSONL and CSV the `change` is `added`, `missing` or `changed`; a name is `missing` rather than `removed` because nothing was looked up to show it is gone. It exits `4` when the files differ, and is in the shell completions and the man page (#310).
1112
- `-resume` accepts the pacing flags `-rate`, `-rate-per-resolver`, `-t`, `-timeout`, `-attempts`, `-auto-rate`, `-no-abort` and `-max-duration` next to the state file, so a scan stopped because timeouts piled up can continue at a gentler rate instead of the one that caused them. They apply over the saved command line, a changed value of one is no longer a reason to refuse the resume, and the next state file records them. Flags that change the candidates or the wildcard check stay refused (#309).
1213
- Library: a `KindError` event has `Fatal` set when it ends the scan (the reliability guard's warning under `NoAbort` is the one error without it), and `KindDone` has `Failed` set when the scan failed. The CLI, the TUI, `Scan` and `subenum mcp` now read these instead of each working it out from the errors seen and the no-abort setting; exit codes, `Scan` errors and verdicts are unchanged (#308).
1314
- The TUI configuration form includes **Sources** and **Permute** fields for `-sources` and `-permute`. The passive sources named in Sources (`ct`, `wayback`) are asked while the wordlist loads and their names are scanned with the wordlist's; a source that fails is a warning on the scan screen, and none is asked with Simulate on. With Permute on, a permutation pass follows the scan on the same screen, and the finished screen's verdict and counters cover both passes. With either field set, the results file names what proposed each result (`sources`, and `permutation` for a permutation hit) and its summary has the `sources` and `permutation` entries the CLI's report has (#274).

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,7 @@ The resumed run refuses to start if an input file, the wordlist (including the b
392392
| `1` | A scan, the wordlist, the domain list or the output file failed |
393393
| `2` | Invalid flags or arguments |
394394
| `3` | `-dL`: some targets failed while others completed |
395-
| `4` | `-diff`: names were added or removed (or, with `-diff-records`, changed) since the previous run |
395+
| `4` | `-diff`: names were added or removed (or, with `-diff-records`, changed) since the previous run; `subenum diff`: the two files differ |
396396
| `130` | Interrupted (SIGINT, `Ctrl+C`); partial results are kept |
397397
| `143` | Terminated (SIGTERM); partial results are kept |
398398

@@ -497,6 +497,8 @@ subenum -permute -format jsonl -o results.jsonl example.com
497497
subenum -w wordlist.txt -format jsonl -diff previous.jsonl -o current.jsonl example.com
498498
```
499499

500+
To compare two results files you already have, without scanning, use `subenum diff previous.jsonl current.jsonl`: `+` for names only in the second file, `-` for names only in the first (missing, not proven gone, since nothing is looked up), and with `-records` `~` for names whose records differ. `-type` limits the record comparison and `-format` picks the output format; the exit code is `4` when the files differ.
501+
500502
### Subdomain takeover hints
501503

502504
When a result has a CNAME record, subenum resolves the CNAME target with the trusted resolver and marks the result as a takeover candidate:

‎docs/ARCHITECTURE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,11 @@ Files, by role:
5252
main.go - Entry point: passes the injected Version to cli.Main
5353
internal/cli/cli.go - CLI entry point: run, flag parsing and validation, scanTarget, exit codes
5454
internal/cli/session.go - runWith and the session: per-run state, the -dL target loop, -tui dispatch
55-
internal/cli/completion.go - completion and man subcommands
55+
internal/cli/completion.go - completion and man subcommands; dispatches diff
5656
internal/cli/config.go - config file and SUBENUM_* environment defaults, -print-config
5757
internal/cli/ct.go - -ct / -sources: passive source names as extra candidates
5858
internal/cli/axfr.go - -axfr: zone transfer names as extra candidates
59-
internal/cli/diff.go - -diff / -diff-records: compare against a previous results file
59+
internal/cli/diff.go - -diff / -diff-records: compare against a previous results file; runDiff: the diff subcommand
6060
internal/cli/dryrun.go - -dry-run: print the scan plan and its cost without sending queries
6161
internal/cli/lab.go - -simulate-zone: serve a lab scenario on a loopback DNS server
6262
internal/cli/mcp.go - subenum mcp: the lab_scan, scan and explain_stats tools, the allow-list and caps
@@ -283,7 +283,7 @@ The tool listens for `SIGINT` and `SIGTERM`. On the first signal it cancels the
283283
* `1` - a scan, the wordlist, the domain list or the output file failed.
284284
* `2` - invalid flags or arguments.
285285
* `3` - `-dL`: some targets failed while others completed.
286-
* `4` - `-diff`: names were added or removed (or, with `-diff-records`, changed) since the previous run.
286+
* `4` - `-diff`: names were added or removed (or, with `-diff-records`, changed) since the previous run; `subenum diff`: the two files differ.
287287
* `130` - interrupted (SIGINT, Ctrl+C); partial results are kept.
288288
* `143` - terminated (SIGTERM); partial results are kept.
289289

‎docs/_data/unreleased.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
# subcommands (#245)
77
commands:
88
- mcp
9+
- diff
910
flags:
1011
- auto-rate
1112
- axfr

‎docs/cli.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,16 @@ subenum [flags] <domain>
1111
subenum [flags] -dL domains.txt
1212
subenum completion bash|zsh|fish|powershell
1313
subenum man
14+
subenum diff [-records] [-type A,AAAA,CNAME] [-format text|json|jsonl|csv] <previous> <current>
1415
subenum mcp [-allow domains] [flags]
1516
```
1617

1718
Flags may appear before or after the domain. The domain is normalized to its host name before scanning: a URL's scheme and path, a `:port` and a leading `*.` are removed and an internationalized name becomes punycode, so `https://app.example.com/login` scans `app.example.com`, not `example.com`.
1819

1920
`completion` prints a shell completion script and `man` prints the man page, both generated from the same flags; release archives ship them under `completions/` and `man/`. `mcp`{% include unreleased.html command="mcp" %} serves subenum to AI agents over the Model Context Protocol, within the domains, resolver and budgets you give it; see [AI agents (MCP)](mcp.html) for its flags.
2021

22+
`diff`{% include unreleased.html command="diff" %} compares two saved results files, in any format subenum writes, without sending a DNS query. It prints `+ name` for names only in `<current>` and `- name` for names only in `<previous>` (`change` is `added` or `missing` in JSON, JSONL and CSV). A missing name is not proven gone, since nothing is looked up; `-diff` during a scan checks each one with a fresh lookup first. `-records` also prints `~ name` (`changed`) for names in both files whose records differ, compared as `-diff-records` does over the `-type` record types (every type by default); it needs JSON, JSONL or CSV files, and compares names only otherwise. The exit code is `4` when the files differ.
23+
2124
## Flags
2225

2326
{% include unreleased-note.html %}
@@ -32,7 +35,7 @@ Flags may appear before or after the domain. The domain is normalized to its hos
3235
| `1` | Failure |
3336
| `2` | Invalid arguments or configuration |
3437
| `3` | Some `-dL` targets failed |
35-
| `4` | `-diff` found added or removed names (or, with `-diff-records`, changed records) |
38+
| `4` | `-diff` found added or removed names (or, with `-diff-records`, changed records); `subenum diff` found differences |
3639
| `130` | Interrupted (Ctrl+C) |
3740
| `143` | Terminated (SIGTERM) |
3841

‎internal/cli/cli.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,15 +58,15 @@ const (
5858
exitFailure = 1 // a scan, the wordlist, the domain list or the output file failed
5959
exitUsage = 2 // invalid flags or arguments
6060
exitPartial = 3 // -dL: some targets failed while others completed
61-
exitChanges = 4 // -diff: names were added or removed (or, with -diff-records, changed) since the previous run
61+
exitChanges = 4 // -diff: names were added or removed (or, with -diff-records, changed) since the previous run; subenum diff: the files differ
6262
)
6363

6464
const exitCodesHelp = `Exit codes:
6565
0 success
6666
1 a scan, the wordlist, the domain list or the output file failed
6767
2 invalid flags or arguments
6868
3 -dL: some targets failed while others completed
69-
4 -diff: names were added or removed (or, with -diff-records, changed) since the previous run
69+
4 -diff: names were added or removed (or, with -diff-records, changed) since the previous run; subenum diff: the files differ
7070
130 interrupted (SIGINT, Ctrl+C); partial results are kept
7171
143 terminated (SIGTERM); partial results are kept
7272
`

‎internal/cli/completion.go‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ var valueChoices = map[string][]string{
3737
}
3838

3939
// subcommands are the non-scan words subenum accepts as its first argument.
40-
var subcommands = []string{"completion", "man", "mcp"}
40+
var subcommands = []string{"completion", "diff", "man", "mcp"}
4141

4242
type flagInfo struct {
4343
name, arg, usage string
@@ -55,13 +55,15 @@ func flagInfos() []flagInfo {
5555
return out
5656
}
5757

58-
// runSubcommand handles `completion`, `man` and the hidden `generate-assets`.
59-
// ok is false when args[0] is not a subcommand.
58+
// runSubcommand handles `completion`, `diff`, `man` and the hidden
59+
// `generate-assets`. ok is false when args[0] is not a subcommand.
6060
func runSubcommand(args []string, stdout, stderr io.Writer) (code int, ok bool) {
6161
if len(args) == 0 {
6262
return 0, false
6363
}
6464
switch args[0] {
65+
case "diff":
66+
return runDiff(args[1:], stdout, stderr), true
6567
case "completion":
6668
if len(args) != 2 {
6769
_, _ = fmt.Fprintln(stderr, "Usage: subenum completion bash|zsh|fish|powershell")
@@ -250,10 +252,12 @@ func writeManPage(w io.Writer) {
250252
p(".SH NAME\nsubenum \\- subdomain enumeration you can trust\n")
251253
p(".SH SYNOPSIS\n.B subenum\n[\\fIoptions\\fR] \\fIdomain\\fR\n.br\n.B subenum\n[\\fIoptions\\fR] \\fB\\-dL\\fR \\fIdomains_file\\fR\n")
252254
p(".br\n.B subenum completion\n\\fBbash\\fR|\\fBzsh\\fR|\\fBfish\\fR|\\fBpowershell\\fR\n.br\n.B subenum man\n")
255+
p(".br\n.B subenum diff\n[\\fB\\-records\\fR] [\\fB\\-type\\fR \\fItypes\\fR] [\\fB\\-format\\fR \\fIformat\\fR] \\fIprevious\\fR \\fIcurrent\\fR\n")
253256
p(".br\n.B subenum mcp\n[\\fB\\-allow\\fR \\fIdomains\\fR] [\\fIflags\\fR]\n")
254257
p(".SH DESCRIPTION\n%s\n.PP\n%s\n",
255258
line("subenum brute-forces subdomains by resolving a wordlist against a target domain with a concurrent worker pool. Resolved names go to stdout; progress, diagnostics and a per-query breakdown go to stderr. Without -w it uses a built-in list of the 5,000 most common subdomain labels."),
256259
line("Only scan domains you own or have explicit written permission to test."))
260+
p(".PP\n%s\n", line("subenum diff compares two saved results files in any output format without sending DNS queries: + for names only in current, - for names only in previous (missing, not proven removed), and with -records ~ for names whose records differ. It exits 4 when anything differs."))
257261
p(".SH OPTIONS\n")
258262
for _, f := range flagInfos() {
259263
if f.arg != "" {

‎internal/cli/diff.go‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ import (
44
"context"
55
"encoding/csv"
66
"encoding/json"
7+
"errors"
8+
"flag"
9+
"fmt"
10+
"io"
711
"os"
812
"sort"
913
"strings"
@@ -128,6 +132,118 @@ func loadBaseline(path string) (baseline, error) {
128132
return b, nil
129133
}
130134

135+
const diffUsage = "Usage: subenum diff [-records] [-type A,AAAA,CNAME] [-format text|json|jsonl|csv] <previous> <current>"
136+
137+
// runDiff is `subenum diff` (#310): it compares two saved results files, in
138+
// any output format, without sending a DNS query. Names only in current are
139+
// "added", names only in previous "missing" (not "removed": nothing was
140+
// looked up to show they are gone), and with -records names in both whose
141+
// records differ are "changed", compared as -diff-records does. It exits
142+
// exitChanges when anything differs.
143+
func runDiff(args []string, stdout, stderr io.Writer) int {
144+
fs := flag.NewFlagSet("diff", flag.ContinueOnError)
145+
fs.SetOutput(stderr)
146+
records := fs.Bool("records", false, "Also report names in both files whose records differ (needs JSON, JSONL or CSV files)")
147+
typeList := fs.String("type", "", "With -records, compare only these record types (default: every type)")
148+
formatStr := fs.String("format", "text", "Output format: text, json, jsonl or csv")
149+
fs.Usage = func() {
150+
_, _ = fmt.Fprintln(stderr, diffUsage)
151+
fs.PrintDefaults()
152+
}
153+
if err := fs.Parse(args); err != nil {
154+
if errors.Is(err, flag.ErrHelp) {
155+
return exitOK
156+
}
157+
return exitUsage
158+
}
159+
if fs.NArg() != 2 {
160+
_, _ = fmt.Fprintln(stderr, diffUsage)
161+
return exitUsage
162+
}
163+
format, err := output.ParseFormat(*formatStr)
164+
if err != nil {
165+
_, _ = fmt.Fprintf(stderr, "Error: %v\n", err)
166+
return exitUsage
167+
}
168+
var types []string // every type
169+
if strings.TrimSpace(*typeList) != "" {
170+
if types, err = dns.ParseTypes(*typeList); err != nil {
171+
_, _ = fmt.Fprintf(stderr, "Error: %v\n", err)
172+
return exitUsage
173+
}
174+
}
175+
var files [2]baseline
176+
for i, path := range fs.Args() {
177+
if files[i], err = loadBaseline(path); err != nil {
178+
_, _ = fmt.Fprintf(stderr, "Error: reading %s: %v\n", path, err)
179+
return exitFailure
180+
}
181+
}
182+
prev, cur := files[0], files[1]
183+
184+
out := output.New(nil, false, format)
185+
out.SetStdout(stdout)
186+
out.SetStderr(stderr)
187+
out.SetPlain(true)
188+
out.SetDiff(true)
189+
out.SetShowRecords(*records)
190+
compare := *records
191+
if compare && (!prev.hasRecords || !cur.hasRecords) {
192+
out.Info("Note: -records needs JSON, JSONL or CSV files; comparing names only")
193+
compare = false
194+
}
195+
196+
sorted := func(m map[string]struct{}) []string {
197+
names := make([]string, 0, len(m))
198+
for name := range m {
199+
names = append(names, name)
200+
}
201+
sort.Strings(names)
202+
return names
203+
}
204+
emit := func(name, change string, recs, previous []dns.Record) {
205+
if recs == nil {
206+
recs = []dns.Record{}
207+
}
208+
out.Emit(output.Result{Result: scan.Result{Subdomain: name, Records: recs}, Change: change, PreviousRecords: previous})
209+
}
210+
var added, missing, changed int
211+
for _, name := range sorted(cur.names) {
212+
if _, ok := prev.names[name]; !ok {
213+
added++
214+
emit(name, "added", cur.records[name], nil)
215+
}
216+
}
217+
if compare {
218+
for _, name := range sorted(cur.names) {
219+
old, ok := prev.records[name]
220+
if _, seen := prev.names[name]; seen && ok && diff.RecordsChanged(old, cur.records[name], types) {
221+
changed++
222+
emit(name, "changed", cur.records[name], old)
223+
}
224+
}
225+
}
226+
for _, name := range sorted(prev.names) {
227+
if _, ok := cur.names[name]; !ok {
228+
missing++
229+
emit(name, "missing", prev.records[name], nil)
230+
}
231+
}
232+
if err := out.Finish(); err != nil {
233+
out.Error("writing output: %v", err)
234+
return exitFailure
235+
}
236+
summary := fmt.Sprintf("%d added, %d missing", added, missing)
237+
if compare {
238+
summary += fmt.Sprintf(", %d changed", changed)
239+
}
240+
out.Info("%s", summary)
241+
if added+missing+changed > 0 {
242+
return exitChanges
243+
}
244+
return exitOK
245+
}
246+
131247
// observe records a found name and says how it differs from the baseline:
132248
// "added", "changed" (with the baseline's records), or "" for unchanged.
133249
func (d *differ) observe(name string, records []dns.Record) (change string, previous []dns.Record) {

‎internal/cli/diff_test.go‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,82 @@
11
package cli
22

33
import (
4+
"bytes"
5+
"encoding/json"
6+
"strings"
47
"testing"
58
)
69

10+
// TestDiffSubcommand: `subenum diff` compares two saved files offline (#310):
11+
// added, missing and (with -records) changed names, exit 4 on differences.
12+
func TestDiffSubcommand(t *testing.T) {
13+
prev := writeFile(t, "prev.jsonl", strings.Join([]string{
14+
`{"subdomain":"www.example.com","records":[{"type":"A","value":"192.0.2.1"}]}`,
15+
`{"subdomain":"old.example.com","records":[{"type":"A","value":"192.0.2.2"}]}`,
16+
`{"subdomain":"cdn.example.com","records":[{"type":"CNAME","value":"a.cdn.test"},{"type":"A","value":"192.0.2.3"}]}`,
17+
`{"type":"summary","verdict":"complete"}`,
18+
}, "\n")+"\n")
19+
cur := writeFile(t, "cur.csv", "subdomain,type,value\n"+
20+
"WWW.example.com.,A,192.0.2.9\n"+
21+
"new.example.com,A,192.0.2.4\n"+
22+
"cdn.example.com,CNAME,b.cdn.test\n")
23+
run := func(args ...string) (int, string, string) {
24+
var stdout, stderr bytes.Buffer
25+
code := runWith(runArgs{args: append([]string{"diff"}, args...), stdin: strings.NewReader(""), stdout: &stdout, stderr: &stderr})
26+
return code, stdout.String(), stderr.String()
27+
}
28+
29+
code, out, errOut := run(prev, cur)
30+
if want := "+ new.example.com\n- old.example.com\n"; code != exitChanges || out != want {
31+
t.Fatalf("names only: code %d, stdout %q (stderr %q); want %d, %q", code, out, errOut, exitChanges, want)
32+
}
33+
34+
code, out, _ = run("-records", prev, cur)
35+
if want := "+ new.example.com A=192.0.2.4\n~ cdn.example.com CNAME=b.cdn.test (was CNAME=a.cdn.test A=192.0.2.3)\n~ www.example.com A=192.0.2.9 (was A=192.0.2.1)\n- old.example.com A=192.0.2.2\n"; code != exitChanges || out != want {
36+
t.Fatalf("-records: code %d, stdout %q; want %q", code, out, want)
37+
}
38+
39+
// -type CNAME leaves www's moved address out of the comparison.
40+
code, out, _ = run("-records", "-type", "CNAME", "-format", "jsonl", prev, cur)
41+
changes := map[string]string{}
42+
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
43+
var r struct{ Subdomain, Change string }
44+
if err := json.Unmarshal([]byte(line), &r); err != nil {
45+
t.Fatalf("jsonl line %q: %v", line, err)
46+
}
47+
changes[r.Subdomain] = r.Change
48+
}
49+
want := map[string]string{"new.example.com": "added", "old.example.com": "missing", "cdn.example.com": "changed"}
50+
if code != exitChanges || len(changes) != len(want) {
51+
t.Fatalf("-type CNAME jsonl: code %d, changes %v; want %v", code, changes, want)
52+
}
53+
for name, c := range want {
54+
if changes[name] != c {
55+
t.Errorf("-type CNAME jsonl: %s is %q, want %q", name, changes[name], c)
56+
}
57+
}
58+
59+
// The same file on both sides: no output, exit 0.
60+
if code, out, _ := run("-records", prev, prev); code != exitOK || out != "" {
61+
t.Errorf("identical files: code %d, stdout %q; want 0 and nothing", code, out)
62+
}
63+
64+
// -records with a text file falls back to names, and says so.
65+
text := writeFile(t, "cur.txt", "www.example.com\ncdn.example.com\nold.example.com\n")
66+
if code, out, errOut := run("-records", prev, text); code != exitOK || out != "" || !strings.Contains(errOut, "comparing names only") {
67+
t.Errorf("text file: code %d, stdout %q, stderr %q", code, out, errOut)
68+
}
69+
70+
for _, args := range [][]string{{prev}, {prev, cur, cur}, {"-format", "xml", prev, cur}, {"-type", "SRV", prev, cur}} {
71+
if code, _, _ := run(args...); code != exitUsage {
72+
t.Errorf("diff %v: code %d, want %d", args, code, exitUsage)
73+
}
74+
}
75+
if code, _, _ := run(prev, writeFile(t, "dir", "")+"-does-not-exist"); code != exitFailure {
76+
t.Errorf("missing file: code %d, want %d", code, exitFailure)
77+
}
78+
}
79+
780
// TestLoadBaselineRecords: structured formats carry records, text does not.
881
func TestLoadBaselineRecords(t *testing.T) {
982
for _, c := range []struct {

‎internal/output/writer.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ type Result struct {
6161
scan.Result
6262
// Change is "added" or "removed" in -diff mode (#84), or "changed" with
6363
// -diff-records when a name's records differ from the baseline (#189).
64+
// `subenum diff` says "missing" instead of "removed": it looks nothing up
65+
// (#310).
6466
Change string `json:"change,omitempty"`
6567
// PreviousRecords are the baseline's records of a "changed" name.
6668
PreviousRecords []dns.Record `json:"previous_records,omitempty"`
@@ -157,7 +159,7 @@ func (w *Writer) SetDiff(on bool) { w.diff = on }
157159

158160
func diffPrefix(change string) string {
159161
switch change {
160-
case "removed":
162+
case "removed", "missing":
161163
return "- "
162164
case "changed":
163165
return "~ "

0 commit comments

Comments
 (0)