Skip to content

Commit bb046fe

Browse files
author
nova-loop
committed
feat(cli): allow repeating -w to combine wordlists
-w is now a repeatable flag. The files are read in order and their lines concatenated before Prepare, so an entry in more than one list is scanned once. '-' stays valid as one entry (once). A config-file or SUBENUM_W value is a default that the command line replaces rather than extends. Closes #199 Signed-off-by: nova-loop <yearheavyyy081@outlook.com>
1 parent 2131669 commit bb046fe

12 files changed

Lines changed: 196 additions & 20 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `-w` can be repeated to layer wordlists, for example a general list plus a target-specific one: `-w general.txt -w target.txt` reads both in order and scans an entry found in both only once (counted as a duplicate, as within one file). `-` stays valid as one of the entries. A `w` value from the config file or `SUBENUM_W` is replaced, not extended, by `-w` on the command line (#199).
1112
- `-max-duration 20m` gives a run a time budget. When it runs out no new names are tested, lookups already under way finish, and the run ends normally: results, the `-stats` report (`stopped_by: "duration"`, untested names in `skipped_by_cap`) and a `degraded` verdict. Under `-dL` one budget covers every target. The GitHub Action gains a `max-duration` input, so a scheduled scan can stop before the job's `timeout-minutes` kills it without a report, and `pkg/subenum` gains `Config.MaxDuration` and `Stats.Expired`. With `-max-queries`, the report now says `stopped_by: "queries"` (#190).
1213
- The progress line and the TUI status bar show an estimated time left (`| ETA 4m12s`), from the rate over the last 30 seconds, once a scan has run for 5 seconds; it disappears at 100% (#191).
1314
- `-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ make help # list all targets
257257

258258
| Flag | Default | Description |
259259
| :--- | :---: | :--- |
260-
| `-w <file>` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) |
260+
| `-w <file>` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Repeat it to combine lists (`-w general.txt -w target.txt`); entries found in more than one are scanned once. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) |
261261
| `-dL <file>` | n/a | File of apex domains to scan, one per line (`-` reads stdin), instead of a `<domain>` argument. Each domain is an independent scan; results share one output |
262262
| `-t <n>` | `100` | Concurrent worker goroutines |
263263
| `-timeout <ms>` | `1000` | DNS timeout in milliseconds, applied to each record-type lookup separately |

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
100100
* **Implementation**: `run()` only calls `runWith(runArgs)`, which takes the arguments, stdin, stdout and stderr as injected values so tests can drive the whole CLI in-process. Per-run state lives in a `session` (`main_session.go`) whose methods are the run's phases: `parse`, `configure`, `loadInputs`, `openOutputs`, `scanAll` (which calls `scanOneTarget` per target) and `finish`. `cliFlags` holds only parsed flag values; what the run derives from them (attempts, record types, excludes, the resolver pool, the `-max-duration` deadline) is in `runtimeInputs`, and each pass over a target is a `scanPass`. `runWith` first handles the `completion` and `man` subcommands and `-resume`, then builds a `flag.FlagSet` with `parseFlagsWith`, which binds every flag into a `cliFlags` struct and applies config-file and environment defaults (see section 5). The standard `flag` package stops at the first non-flag argument, so `parseInterspersed` re-parses around each positional; flags written after the domain are honored, and everything after a `--` terminator is positional. `runWith` then:
101101
* handles `-tui` (launches `tui.Start`, ignoring other flags), `-print-config` and `-version` first;
102102
* parses `-format` (`output.ParseFormat`), `-type` (`dns.ParseTypes`), and merges `-attempts` with the deprecated `-retries` (`effectiveAttempts`);
103-
* calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin. Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
103+
* calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
104104
* **Domain normalization**: `validate.NormalizeDomain` trims and lowercases the input, strips a URL scheme and path, a `:port`, a leading `*.` and a trailing dot (returning a note for each change), converts IDNs to punycode, and checks the result with `validate.Domain`. `validate.DNSServer` checks the resolver: `ip:port`, `tls://host[:port]` or `https://host/path`. Both are shared with the TUI.
105105
* **Target list (`-dL`)**: `loadTargets` returns either the single positional domain or the entries of the `-dL` file (`-` for stdin). Blank lines, `#` comments and duplicates are skipped; invalid domains are reported and skipped. An empty result is an error.
106106
* **Interactions**: `scanOptions` maps the parsed flags onto a `scan.Options`; its `Config` method produces the `scan.Config` passed to the scan engine.

‎docs/_includes/flags.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,4 +41,4 @@
4141
| `-type <string>` | A,AAAA | Comma-separated DNS record types to look up: A, AAAA, CNAME |
4242
| `-v` | - | Enable verbose output |
4343
| `-version` | - | Show version information |
44-
| `-w <string>` | - | Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list |
44+
| `-w <file>` | - | Path to a wordlist file (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list |

‎main.go‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ func main() {
111111
type cliFlags struct {
112112
tui bool
113113
printConfig bool
114-
wordlistFile string
114+
wordlists wordlistPaths // -w, repeatable (#199)
115115
domainList string
116116
concurrency int
117117
timeoutMs int
@@ -199,7 +199,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
199199
}
200200
fs.BoolVar(&f.tui, "tui", false, "Launch the interactive terminal UI (all other flags are ignored)")
201201
fs.BoolVar(&f.printConfig, "print-config", false, "Print every setting's effective value and its source (flag, env, config or default), then exit")
202-
fs.StringVar(&f.wordlistFile, "w", "", "Path to the wordlist file (- for stdin); omitted: the bundled top-5000 list")
202+
fs.Var(&f.wordlists, "w", "Path to a wordlist `file` (- for stdin); repeat to combine several, deduplicated; omitted: the bundled top-5000 list")
203203
fs.StringVar(&f.domainList, "dL", "", "File of apex domains to scan, one per line (- for stdin); replaces the <domain> argument")
204204
fs.IntVar(&f.concurrency, "t", 100, "Number of concurrent workers")
205205
fs.IntVar(&f.timeoutMs, "timeout", 1000, "DNS lookup timeout in milliseconds")
@@ -329,10 +329,14 @@ func validateFlags(f cliFlags, in *runtimeInputs, positionals []string, fs *flag
329329
if !validateArgs(f, positionals, fs, out) {
330330
return "", false
331331
}
332-
if f.wordlistFile == wordlist.Stdin && f.domainList == wordlist.Stdin {
332+
if f.wordlists.readsStdin() && f.domainList == wordlist.Stdin {
333333
out.Error("-w and -dL cannot both read standard input")
334334
return "", false
335335
}
336+
if n := countStdin(f.wordlists.paths); n > 1 {
337+
out.Error("-w - is given %d times; standard input can only be read once", n)
338+
return "", false
339+
}
336340
if f.simZone != "" {
337341
// Effective values, not just command-line flags: a resolver list or
338342
// simulation mode from the config file or SUBENUM_* would otherwise
@@ -527,10 +531,10 @@ func logVerboseStart(f cliFlags, domain string, maxAttempts int, out *output.Wri
527531
out.Info("Mode: LIVE DNS RESOLUTION")
528532
}
529533
out.Info("Target domain: %s", domain)
530-
if f.wordlistFile == "" {
534+
if len(f.wordlists.paths) == 0 {
531535
out.Info("Wordlist: %s", bundledWordlistName)
532536
} else {
533-
out.Info("Wordlist: %s", f.wordlistFile)
537+
out.Info("Wordlist: %s", f.wordlists.describe())
534538
}
535539
out.Info("Concurrency: %d workers", f.concurrency)
536540
out.Info("Timeout: %d ms", f.timeoutMs)

‎main_config.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,9 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error {
111111
errs = append(errs, fmt.Errorf("%w: %s value %q for %s: %w", errConfig, source, value, name, err))
112112
return false
113113
}
114+
if v, ok := fs.Lookup(name).Value.(interface{ markDefault() }); ok {
115+
v.markDefault()
116+
}
114117
d.values[name] = value
115118
return true
116119
}

‎main_resume.go‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -226,7 +226,8 @@ func inputDigests(f cliFlags) (map[string]string, error) {
226226
if diff == f.outputFile {
227227
diff = ""
228228
}
229-
for _, p := range []string{f.wordlistFile, f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff} {
229+
paths := append(slices.Clone(f.wordlists.paths), f.domainList, f.excludeFile, f.resolvers, f.simZone, f.seedsFile, diff)
230+
for _, p := range paths {
230231
d, err := fileDigest(p)
231232
if err != nil {
232233
return nil, err
@@ -308,7 +309,7 @@ func mergeStats(a, b scan.Stats) scan.Stats {
308309
// run never reached keeps its saved state unchanged.
309310
func saveResumeState(s *session) {
310311
f, out, prev := s.f, s.out, s.resume
311-
if f.wordlistFile == "-" || f.domainList == "-" {
312+
if f.wordlists.readsStdin() || f.domainList == "-" {
312313
out.Info("Not saving resume state: the wordlist or domain list came from stdin")
313314
return
314315
}

‎main_resume_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,7 @@ func TestSaveResumeStateKeepsEarlierPosition(t *testing.T) {
239239
{Domain: "a.example", RootDone: 40, Processed: 40, Stats: scan.Stats{NXDomain: 38}, Results: []output.Result{{Result: scan.Result{Subdomain: "x.a.example"}}}},
240240
{Domain: "b.example", RootDone: 7, Processed: 7, Permute: &resumePass{RootDone: 3}},
241241
}}
242-
f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json"), wordlistFile: ""}
242+
f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json")}
243243
out := output.New(nil, false, output.FormatText)
244244
saveResumeState(&session{
245245
f: f, out: out, resume: prev,

‎main_session.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -273,7 +273,7 @@ func (s *session) loadInputs() bool {
273273
f, out := s.f, s.out
274274
// The wordlist is read once (stdin can only be read once) and normalized
275275
// per target, since the name-length limit depends on the domain.
276-
if f.wordlistFile == "" {
276+
if len(f.wordlists.paths) == 0 {
277277
out.Info("No -w given: using the %s. Pass -w for a larger or target-specific list.", bundledWordlistName)
278278
}
279279
s.settings = effectiveSettings(s.fs)
@@ -297,7 +297,7 @@ func (s *session) loadInputs() bool {
297297
// limit differs (#182).
298298
s.words = wordlist.Prepare(wordLines)
299299
if probe, _, _ := s.words.For(""); len(probe) == 0 {
300-
out.Error("wordlist %s has no valid entries", f.wordlistFile)
300+
out.Error("wordlist %s has no valid entries", f.wordlists.describe())
301301
return false
302302
}
303303
if s.resume != nil {

‎main_test.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ func TestParseFlagsInterspersed(t *testing.T) {
4545
if f.testMode != tc.wantSim {
4646
t.Errorf("simulate = %v, want %v", f.testMode, tc.wantSim)
4747
}
48-
if f.wordlistFile != "wl.txt" {
49-
t.Errorf("wordlist = %q, want wl.txt", f.wordlistFile)
48+
if got := f.wordlists.paths; len(got) != 1 || got[0] != "wl.txt" {
49+
t.Errorf("wordlist = %q, want [wl.txt]", got)
5050
}
5151
})
5252
}
@@ -95,7 +95,7 @@ func TestValidateFlagsDomainListConflicts(t *testing.T) {
9595
if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, []string{"example.com"}, fs, out); ok {
9696
t.Error("domain argument plus -dL should be rejected")
9797
}
98-
f.wordlistFile, f.domainList = "-", "-"
98+
f.wordlists.paths, f.domainList = []string{"-"}, "-"
9999
if _, ok := validateFlags(f, &runtimeInputs{maxAttempts: 1}, nil, fs, out); ok {
100100
t.Error("-w - with -dL - should be rejected")
101101
}

0 commit comments

Comments
 (0)