Skip to content

Commit 05d51b5

Browse files
TMHSDigitalclaude
andcommitted
feat: permutation pass (-permute, -seeds)
internal/permute generates candidates from found names: built-in environment and role words joined with - on either side or as a new level (api-dev, dev-api, dev.api), and number increments (api2 -> api1, api3, api4; widths kept). -permute runs a second, fully guarded scan per target over them (preflight, wildcard filtering, -rate, -max-queries, -exclude), skipping wordlist entries; -seeds adds seeds from an earlier results file in any format and implies -permute. Permutation hits carry "permutation": true in JSON/JSONL, and the run-quality report gains per-target seeds, candidates, hits and the pass's verdict. E2E: a found api.example.com finds a planted api-dev.example.com that is not in the wordlist. Fixes #72 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 0c4d292 commit 05d51b5

12 files changed

Lines changed: 350 additions & 22 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- Permutation mode: `-permute` follows each target's scan with a second pass over permutations of the names found (word joins such as `api-dev`, `dev-api` and `dev.api`, and number increments), and `-seeds <file>` adds seeds from an earlier run in any format. The pass is fully guarded (preflight, wildcard filtering, `-rate`, `-max-queries`, `-exclude`), skips wordlist entries, marks its hits `"permutation": true`, and reports seeds, candidates and hits in the run-quality report (#72).
1112
- Defaults from a config file and the environment: `config.json` in the user config directory (or `$SUBENUM_CONFIG`) and `SUBENUM_<FLAG>` variables, with precedence flag > env > config > built-in. `-print-config` shows each effective setting and its source. `subenum completion bash|zsh|fish|powershell` and `subenum man` generate shell completions and a man page from the flags, and release archives ship them (#88).
1213
- A useful default wordlist: `-w` is now optional. Without it, subenum uses an embedded list of the 5,000 most common subdomain labels (SecLists `subdomains-top1million-5000.txt`, MIT; the license notice is kept in `data/subdomains-5k.txt`) and says so on stderr. The README gains a "Choosing a wordlist" section on larger lists and their query cost (#63).
1314
- Monitoring mode: `-diff previous` compares with an earlier results file (any format) and prints only names added (`+` / `"change": "added"`) or removed (`-` / `"removed"`), with a `change` CSV column; exit code `4` signals changes. The `-o` file keeps the full current results for the next run. Removals are reported only when the run's verdict is `complete`, and never for `-exclude`d names. The run-quality report gains a `diff` object, and `docs/monitoring.md` has a scheduled GitHub Actions workflow that opens an issue for new names (#84).

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,16 @@ Results can only be as good as the wordlist. Without `-w`, subenum uses its bund
392392

393393
List size sets the query volume: each entry costs one query per record type (two with the default `A,AAAA`), per attempt, per recursion level. A 1M-entry list at `-rate 500` is about 70 minutes of wire time per target, so pair big lists with `-rate` (and `-r` for a resolver pool), and use `-max-queries` as a budget. `tools/wordlist-gen` builds a small target-specific list from a domain's own terms.
394394

395+
### Permutations: a second pass
396+
397+
Once `api.example.com` and `dev.example.com` exist, the next names most likely to exist are `api-dev`, `dev-api`, `dev.api` and `api2`. `-permute` runs a second scan per target over permutations of the names the first pass found: built-in environment and role words (`dev`, `staging`, `prod`, `qa`, `internal`, `v1`, ...) joined with `-` on either side or as a new level, plus number increments (`api2` -> `api1`, `api3`, `api4`). `-seeds results.jsonl` adds names from an earlier run (any format) and implies `-permute`.
398+
399+
The pass is a normal scan with the same preflight, wildcard filtering, `-rate`, `-max-queries` and `-exclude`; candidates already in the wordlist are skipped. Its hits carry `"permutation": true` in JSON/JSONL, and the run-quality report shows the seeds, candidates and hits per target.
400+
401+
```bash
402+
subenum -permute -format jsonl -o results.jsonl example.com
403+
```
404+
395405
### Monitoring for new subdomains
396406

397407
`-diff previous.jsonl` reports only what changed since an earlier run: `+ name` for new subdomains and `- name` for ones that disappeared (only when the run's verdict is `complete`, so failed lookups are never mistaken for removals). The `-o` file keeps the full current results for the next comparison, and the exit code is `4` when anything changed. [docs/monitoring.md](docs/monitoring.md) has a ready-to-use scheduled GitHub Actions workflow that opens an issue when new names appear.

‎docs/ROADMAP.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,6 @@ Planned features that make results more trustworthy and the tool easier to get.
3232

3333
Ideas we want, with no date attached.
3434

35-
- Permutation / alteration mode for second-stage brute-forcing ([#72](https://github.com/TMHSDigital/subenum/issues/72)).
3635
- A stable Go library API, `pkg/subenum` ([#77](https://github.com/TMHSDigital/subenum/issues/77)).
3736
- Resume interrupted scans with `-resume` ([#73](https://github.com/TMHSDigital/subenum/issues/73)).
3837
- Try AXFR and detect NSEC-walkable zones before brute-forcing ([#85](https://github.com/TMHSDigital/subenum/issues/85)).

‎docs/_includes/flags.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,15 @@
1515
| `-max-queries <int>` | - | Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt |
1616
| `-no-abort` | - | Do not abort when the resolver failure rate exceeds 20% (warning is still emitted) |
1717
| `-o <string>` | - | Write results to file (in addition to stdout) |
18+
| `-permute` | - | After each scan, scan permutations of the names found (api -&gt; api-dev, dev-api, dev.api, api2, ...) |
1819
| `-print-config` | - | Print every setting's effective value and its source (flag, env, config or default), then exit |
1920
| `-progress` | true | Show progress during scanning |
2021
| `-r <string>` | - | File of resolvers (ip or ip:port, one per line) to spread queries over; every hit is re-validated against -dns-server |
2122
| `-rate <int>` | - | Max DNS queries per second on the wire, all workers combined; counts every record type, retry and wildcard probe (0 = unlimited) |
2223
| `-recursive` | - | Recursively enumerate subdomains of discovered subdomains |
2324
| `-retries <int>` | - | Deprecated: use -attempts instead |
2425
| `-seed <uint>` | - | In simulation mode, seed for reproducible results (0 = random; the seed used is printed) |
26+
| `-seeds <string>` | - | Results file (any -format) whose names also seed -permute; implies -permute |
2527
| `-show-records` | - | In text format, append each result's records (TYPE=value) |
2628
| `-simulate` | - | Run in simulation mode without actual DNS queries (for testing) |
2729
| `-stats <string>` | - | Write a JSON run-quality report (outcomes, queries sent, verdict) to this file |

‎internal/output/summary.go‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -49,23 +49,33 @@ type DiffSummary struct {
4949

5050
// TargetSummary is one scanned domain's part of a Summary.
5151
type TargetSummary struct {
52-
Domain string `json:"domain"`
53-
Status string `json:"status"` // ok, failed, skipped, interrupted, not_run
54-
Error string `json:"error,omitempty"`
55-
Processed int64 `json:"processed"`
56-
Total int64 `json:"total"`
57-
Found int64 `json:"found"`
58-
Outcomes Outcomes `json:"outcomes"`
59-
QueriesSent int64 `json:"queries_sent"`
60-
SkippedCap int64 `json:"skipped_by_cap"`
61-
Takeover int64 `json:"takeover_candidates,omitempty"` // results with a takeover hint
62-
PoolHits int64 `json:"pool_hits,omitempty"` // -r: names a pool resolver said exist
63-
Confirmed int64 `json:"confirmed,omitempty"` // pool hits the trusted resolver confirmed
64-
Unconfirmed int64 `json:"unconfirmed,omitempty"` // pool hits the trusted resolver denied
65-
Wildcard bool `json:"wildcard"`
66-
Fingerprint []dns.Record `json:"wildcard_fingerprint,omitempty"`
67-
Verdict string `json:"verdict"`
68-
Reason string `json:"reason"`
52+
Domain string `json:"domain"`
53+
Status string `json:"status"` // ok, failed, skipped, interrupted, not_run
54+
Error string `json:"error,omitempty"`
55+
Processed int64 `json:"processed"`
56+
Total int64 `json:"total"`
57+
Found int64 `json:"found"`
58+
Outcomes Outcomes `json:"outcomes"`
59+
QueriesSent int64 `json:"queries_sent"`
60+
SkippedCap int64 `json:"skipped_by_cap"`
61+
Takeover int64 `json:"takeover_candidates,omitempty"` // results with a takeover hint
62+
// Permutation is the -permute pass, when one ran (#72).
63+
Permutation *PermutationSummary `json:"permutation,omitempty"`
64+
PoolHits int64 `json:"pool_hits,omitempty"` // -r: names a pool resolver said exist
65+
Confirmed int64 `json:"confirmed,omitempty"` // pool hits the trusted resolver confirmed
66+
Unconfirmed int64 `json:"unconfirmed,omitempty"` // pool hits the trusted resolver denied
67+
Wildcard bool `json:"wildcard"`
68+
Fingerprint []dns.Record `json:"wildcard_fingerprint,omitempty"`
69+
Verdict string `json:"verdict"`
70+
Reason string `json:"reason"`
71+
}
72+
73+
// PermutationSummary counts a target's -permute pass.
74+
type PermutationSummary struct {
75+
Seeds int `json:"seeds"`
76+
Candidates int `json:"candidates"`
77+
Found int `json:"found"`
78+
Verdict string `json:"verdict,omitempty"`
6979
}
7080

7181
// Outcomes counts how each lookup ended. SERVFAIL and other errors Go's

‎internal/output/writer.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ type Result struct {
5959
Takeover string `json:"takeover_candidate,omitempty"`
6060
// Change is "added" or "removed" in -diff mode (#84).
6161
Change string `json:"change,omitempty"`
62+
// Permutation marks a name found by the -permute pass (#72).
63+
Permutation bool `json:"permutation,omitempty"`
6264
}
6365

6466
// Writer synchronises all output. Results go to stdout (and optionally a file);

‎internal/permute/permute.go‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
// Package permute generates second-pass candidates from names a scan has
2+
// already found (#72): api.example.com suggests api-dev, dev-api, dev.api,
3+
// api2 and so on, the names most likely to exist next to a known one.
4+
package permute
5+
6+
import (
7+
"sort"
8+
"strconv"
9+
"strings"
10+
11+
"github.com/TMHSDigital/subenum/internal/wordlist"
12+
)
13+
14+
// Words are combined with each seed label. They are the environment, version
15+
// and role names that most often sit next to an existing host.
16+
var Words = []string{
17+
"dev", "development", "staging", "stage", "stg", "prod", "production",
18+
"test", "testing", "qa", "uat", "demo", "beta", "alpha", "preview",
19+
"internal", "int", "ext", "admin", "api", "app", "web", "old", "new",
20+
"backup", "v1", "v2", "v3", "us", "eu",
21+
}
22+
23+
// Generate returns permutation prefixes (relative to the apex, like wordlist
24+
// entries) for the seed names. A seed is a name relative to the apex:
25+
// "api" for api.example.com, "api.us" for api.us.example.com; its leftmost
26+
// label is permuted and the rest kept. Candidates in skip, and invalid DNS
27+
// labels, are left out. The result is sorted and duplicate-free.
28+
func Generate(seeds []string, words []string, skip map[string]bool) []string {
29+
out := map[string]bool{}
30+
add := func(candidate string) {
31+
entry, ok := wordlist.Normalize(candidate)
32+
if ok && !skip[entry] {
33+
out[entry] = true
34+
}
35+
}
36+
for _, seed := range seeds {
37+
seed = strings.ToLower(strings.Trim(seed, "."))
38+
if seed == "" {
39+
continue
40+
}
41+
label, rest, _ := strings.Cut(seed, ".")
42+
join := func(l string) string {
43+
if rest == "" {
44+
return l
45+
}
46+
return l + "." + rest
47+
}
48+
for _, w := range words {
49+
if w == label {
50+
continue
51+
}
52+
add(join(w + "-" + label))
53+
add(join(label + "-" + w))
54+
add(w + "." + seed) // a new level under the known name
55+
}
56+
for _, n := range numbered(label) {
57+
add(join(n))
58+
}
59+
}
60+
result := make([]string, 0, len(out))
61+
for c := range out {
62+
result = append(result, c)
63+
}
64+
sort.Strings(result)
65+
return result
66+
}
67+
68+
// numbered returns number variants of label: api -> api1, api2, api3;
69+
// api2 -> api1, api3, api4; web-01 -> web-00, web-02, web-03.
70+
func numbered(label string) []string {
71+
i := len(label)
72+
for i > 0 && label[i-1] >= '0' && label[i-1] <= '9' {
73+
i--
74+
}
75+
base, digits := label[:i], label[i:]
76+
if digits == "" {
77+
return []string{label + "1", label + "2", label + "3"}
78+
}
79+
n, err := strconv.Atoi(digits)
80+
if err != nil {
81+
return nil
82+
}
83+
pad := func(v int) string {
84+
s := strconv.Itoa(v)
85+
for len(s) < len(digits) {
86+
s = "0" + s
87+
}
88+
return base + s
89+
}
90+
var out []string
91+
if n > 0 {
92+
out = append(out, pad(n-1))
93+
}
94+
return append(out, pad(n+1), pad(n+2))
95+
}

‎internal/permute/permute_test.go‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
package permute
2+
3+
import (
4+
"slices"
5+
"testing"
6+
)
7+
8+
func TestGenerate(t *testing.T) {
9+
got := Generate([]string{"api", "web02.us"}, []string{"dev", "api"}, map[string]bool{"api-dev": false, "dev.api": true})
10+
for _, want := range []string{
11+
"api-dev", "dev-api", // word joins
12+
"api1", "api2", "api3", // numbering a bare label
13+
"web01.us", "web03.us", "web04.us", // numbering keeps width and the rest of the name
14+
"dev-web02.us", "web02-dev.us", "dev.web02.us", "api-web02.us",
15+
} {
16+
if !slices.Contains(got, want) {
17+
t.Errorf("missing %q in %v", want, got)
18+
}
19+
}
20+
for _, unwanted := range []string{"dev.api", "api-api", "api"} {
21+
if slices.Contains(got, unwanted) {
22+
t.Errorf("%q should not be generated (skipped, self-join or the seed itself)", unwanted)
23+
}
24+
}
25+
if !slices.IsSorted(got) {
26+
t.Error("output is not sorted")
27+
}
28+
}
29+
30+
func TestNumbered(t *testing.T) {
31+
for in, want := range map[string][]string{
32+
"api": {"api1", "api2", "api3"},
33+
"api0": {"api1", "api2"},
34+
"web-09": {"web-08", "web-10", "web-11"},
35+
} {
36+
if got := numbered(in); !slices.Equal(got, want) {
37+
t.Errorf("numbered(%q) = %v, want %v", in, got, want)
38+
}
39+
}
40+
}

‎main.go‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,8 @@ type cliFlags struct {
141141
excludeFile string
142142
resolvers string // -r: file of resolver addresses for a pool
143143
diff string // -diff: previous results file to compare against
144+
permute bool // -permute: second pass over permutations of found names
145+
seedsFile string // -seeds: extra permutation seeds (any results format)
144146
pool *dns.Pool // built from -r; nil means the single -dns-server
145147
excludes []string // -exclude and -exclude-file patterns, merged
146148
}
@@ -185,6 +187,8 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error) (cli
185187
fs.StringVar(&f.statsFile, "stats", "", "Write a JSON run-quality report (outcomes, queries sent, verdict) to this file")
186188
fs.StringVar(&f.exclude, "exclude", "", "Comma-separated out-of-scope names and *.parent patterns; never queried or expanded")
187189
fs.StringVar(&f.excludeFile, "exclude-file", "", "File of out-of-scope names and *.parent patterns, one per line (# comments allowed)")
190+
fs.BoolVar(&f.permute, "permute", false, "After each scan, scan permutations of the names found (api -> api-dev, dev-api, dev.api, api2, ...)")
191+
fs.StringVar(&f.seedsFile, "seeds", "", "Results file (any -format) whose names also seed -permute; implies -permute")
188192
fs.StringVar(&f.diff, "diff", "", "Previous results file (any -format); report only names added or removed since then, and exit 4 when there are changes")
189193
fs.StringVar(&f.resolvers, "r", "", "File of resolvers (ip or ip:port, one per line) to spread queries over; every hit is re-validated against -dns-server")
190194
fs.Usage = func() {
@@ -634,16 +638,31 @@ func run() (code int) {
634638
w.SetSeed(f.seed)
635639
w.SetTakeoverColumn(slices.Contains(recordTypes, "CNAME"))
636640
}
637-
emit := func(ev scan.Event) {
641+
var targetFound []string // the current target's hits, seeds for -permute
642+
emitResult := func(ev scan.Event, permutation bool) {
643+
targetFound = append(targetFound, ev.Domain)
644+
res := output.Result{Subdomain: ev.Domain, Records: ev.Records, Takeover: ev.Takeover, Permutation: permutation}
638645
if diff == nil {
639-
out.ResultWithHint(ev.Domain, ev.Records, ev.Takeover)
646+
out.Emit(res)
640647
return
641648
}
642649
if fileOut != nil {
643-
fileOut.ResultWithHint(ev.Domain, ev.Records, ev.Takeover)
650+
fileOut.Emit(res)
644651
}
645652
if diff.observe(ev.Domain) {
646-
out.Emit(output.Result{Subdomain: ev.Domain, Records: ev.Records, Takeover: ev.Takeover, Change: "added"})
653+
res.Change = "added"
654+
out.Emit(res)
655+
}
656+
}
657+
emit := func(ev scan.Event) { emitResult(ev, false) }
658+
emitPermutation := func(ev scan.Event) { emitResult(ev, true) }
659+
660+
var seeds map[string]struct{}
661+
if f.seedsFile != "" {
662+
f.permute = true
663+
if seeds, err = loadPrevious(f.seedsFile); err != nil {
664+
out.Error("reading -seeds file: %v", err)
665+
return exitFailure
647666
}
648667
}
649668
// A carriage-return progress line only makes sense on a terminal. When
@@ -738,7 +757,17 @@ func run() (code int) {
738757
failedTargets++
739758
continue
740759
}
760+
targetFound = targetFound[:0]
741761
res := scanTarget(ctx, f, target, entries, maxAttempts, recordTypes, out, outWriter, emit)
762+
// -permute: a second pass over permutations of what was found (#72),
763+
// skipped after an interrupt or a resolver abort.
764+
if f.permute && res.done && !res.resolverAbort && signalCode.Load() == 0 {
765+
found := append([]string(nil), targetFound...)
766+
res.permutation = runPermutationPass(ctx, f, target, entries, found, seeds, maxAttempts, recordTypes, out, outWriter, emitPermutation)
767+
if p := res.permutation; p != nil && p.pass.failed {
768+
res.failed = true
769+
}
770+
}
742771
results[i] = res
743772
anyDone = anyDone || res.done
744773
anyFailed = anyFailed || res.failed
@@ -862,6 +891,7 @@ type targetResult struct {
862891
resolverAbort bool // the reliability guard aborted it
863892
err string // first error message, if any
864893
final scan.Event // the EventDone, when done
894+
permutation *permutationResult
865895
}
866896

867897
// scanTarget runs one domain's scan and streams its events to out.

‎main_e2e_test.go‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ import (
1111
"sort"
1212
"strings"
1313
"testing"
14+
15+
"golang.org/x/net/dns/dnsmessage"
16+
17+
"github.com/TMHSDigital/subenum/internal/dnstest"
1418
)
1519

1620
// runCLI invokes run() as the binary would, with args after the program name,
@@ -550,3 +554,57 @@ func TestE2EDiff(t *testing.T) {
550554
t.Fatalf("degraded run reported removals: %q", out)
551555
}
552556
}
557+
558+
// TestE2EPermute is #72's done-when: a found api.example.com seeds a second
559+
// pass that finds a planted api-dev.example.com absent from the wordlist.
560+
func TestE2EPermute(t *testing.T) {
561+
srv := dnstest.Start(t, func(q dnstest.Query) dnstest.Reply {
562+
switch q.Name {
563+
case "example.com", "api.example.com", "api-dev.example.com":
564+
if q.Type == dnsmessage.TypeA {
565+
return dnstest.Reply{A: []string{"192.0.2.10"}}
566+
}
567+
return dnstest.Reply{}
568+
}
569+
return dnstest.NXDomain
570+
})
571+
wl := writeFile(t, "wl.txt", "api\nwww\n")
572+
stats := filepath.Join(t.TempDir(), "run.json")
573+
code, out := runCLI(t, "", "-dns-server", srv.Addr, "-type", "A", "-progress=false", "-permute",
574+
"-format", "jsonl", "-stats", stats, "-w", wl, "example.com")
575+
if code != 0 {
576+
t.Fatalf("exit %d", code)
577+
}
578+
results := map[string]bool{} // name -> permutation
579+
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
580+
var r struct {
581+
Type string `json:"type"`
582+
Subdomain string `json:"subdomain"`
583+
Permutation bool `json:"permutation"`
584+
}
585+
if err := json.Unmarshal([]byte(line), &r); err != nil {
586+
t.Fatalf("bad line %q: %v", line, err)
587+
}
588+
if r.Type != "summary" {
589+
results[r.Subdomain] = r.Permutation
590+
}
591+
}
592+
perm, ok := results["api-dev.example.com"]
593+
if !ok || !perm {
594+
t.Fatalf("results = %v, want api-dev.example.com found by the permutation pass", results)
595+
}
596+
if results["api.example.com"] {
597+
t.Error("api.example.com came from the wordlist, not the permutation pass")
598+
}
599+
data, _ := os.ReadFile(stats)
600+
var sum struct {
601+
Targets []struct {
602+
Permutation struct {
603+
Seeds, Candidates, Found int
604+
} `json:"permutation"`
605+
} `json:"targets"`
606+
}
607+
if err := json.Unmarshal(data, &sum); err != nil || sum.Targets[0].Permutation.Found != 1 || sum.Targets[0].Permutation.Seeds != 1 {
608+
t.Fatalf("summary permutation = %+v (err %v)", sum.Targets, err)
609+
}
610+
}

0 commit comments

Comments
 (0)