Skip to content

Commit 19a3bb5

Browse files
committed
Let -resume change the scan's pacing (#309)
-resume took no other arguments, so a scan stopped by a throttling resolver or a spent time budget could only continue at the same rate, timeout and budget. -resume now accepts the pacing flags (-rate, -rate-per-resolver, -t, -timeout, -attempts, -auto-rate, -no-abort, -max-duration) after the state file and applies them over the saved arguments; any other flag is still refused. The next state file records the new values, and a changed SUBENUM_* or config value for a pacing flag no longer blocks a resume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Fixes #309
1 parent 55c554a commit 19a3bb5

7 files changed

Lines changed: 161 additions & 12 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `-resume` accepts the pacing flags `-rate`, `-rate-per-resolver`, `-t`, `-timeout`, `-attempts`, `-auto-rate`, `-no-abort` and `-max-duration` next to the state file, so a scan stopped because timeouts piled up can continue at a gentler rate instead of the one that caused them. They apply over the saved command line, a changed value of one is no longer a reason to refuse the resume, and the next state file records them. Flags that change the candidates or the wildcard check stay refused (#309).
1112
- Library: a `KindError` event has `Fatal` set when it ends the scan (the reliability guard's warning under `NoAbort` is the one error without it), and `KindDone` has `Failed` set when the scan failed. The CLI, the TUI, `Scan` and `subenum mcp` now read these instead of each working it out from the errors seen and the no-abort setting; exit codes, `Scan` errors and verdicts are unchanged (#308).
1213
- The TUI configuration form includes **Sources** and **Permute** fields for `-sources` and `-permute`. The passive sources named in Sources (`ct`, `wayback`) are asked while the wordlist loads and their names are scanned with the wordlist's; a source that fails is a warning on the scan screen, and none is asked with Simulate on. With Permute on, a permutation pass follows the scan on the same screen, and the finished screen's verdict and counters cover both passes. With either field set, the results file names what proposed each result (`sources`, and `permutation` for a permutation hit) and its summary has the `sources` and `permutation` entries the CLI's report has (#274).
1314
- Each result says what proposed it: a `sources` list in JSON and JSONL and a last `sources` column in CSV (joined with `;`), holding `wordlist`, `ct`, `wayback` and `axfr`, every one that had the name, or `permute` alone for a `-permute` hit. A name found by `-recursive` below another is the wordlist's. The run-quality report counts the results each passive source had proposed (`found` in its `sources` entry) and a zone transfer's (`axfr_found`). Text output, the TUI's files and `subenum mcp` reports are unchanged (#273).

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -307,7 +307,7 @@ This table describes `main`, which can be ahead of the latest release: a release
307307
| `-dry-run` | `false` | Print the plan without scanning: candidates after dedupe and `-exclude`, query and duration estimates, and the resolver preflight (its only DNS query), then exit; JSON with `-format json` or `jsonl` |
308308
| `-version` | n/a | Print version and exit |
309309
| `-retries <n>` | n/a | **Deprecated** - alias for `-attempts`, prints a warning |
310-
| `-resume <file>` | n/a | Resume a run that was interrupted, ran out of `-max-duration` or was aborted, from its state file; takes no other arguments |
310+
| `-resume <file>` | n/a | Resume a run that was interrupted, ran out of `-max-duration` or was aborted, from its state file; takes no other arguments but the pacing flags (`-rate`, `-t`, `-timeout`, ...) |
311311
| `-seeds <file>` | n/a | Results file (any `-format`) whose names also seed `-permute`; implies `-permute` |
312312
| `-state <file>` | `subenum-resume.json` | Where a run that was interrupted, ran out of `-max-duration` or was aborted saves its state for `-resume` |
313313

@@ -380,6 +380,8 @@ Press `Ctrl+C` at any time to abort. In-flight queries drain, partial results ar
380380
subenum -resume subenum-resume.json
381381
```
382382

383+
The only flags `-resume` accepts next to the state file are the pacing flags `-attempts`, `-auto-rate`, `-max-duration`, `-no-abort`, `-rate`, `-rate-per-resolver`, `-t` and `-timeout`: for example `subenum -resume subenum-resume.json -rate 20 -auto-rate` continues a scan that timeouts slowed down at a gentler rate. They apply over the saved command line and are recorded in the next state file. Flags that change which names are queried or how wildcards are detected (`-w`, `-type`, `-dns-server`, `-depth` and the rest) cannot change on a resume.
384+
383385
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves the same state (its exit code does not change), so `-resume` continues it too: a scheduled job with a time budget can work through a large wordlist over several runs. When `-diff` and `-o` name the same file, a run that saves a resume state leaves that file unchanged, so the resumed run compares against the same baseline and then writes the full results. Runs that read the wordlist or domain list from stdin cannot be resumed.
384386

385387
**Exit codes**

‎docs/_includes/flags.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
| `-rate <int>` | - | Max DNS queries per second on the wire, all workers combined; counts every record type, retry and wildcard probe (0 = unlimited) |
3131
| `-rate-per-resolver <int>` {% include unreleased.html flag="rate-per-resolver" %} | - | With -r, max DNS queries per second sent to each pool resolver; -rate still caps the total (0 = unlimited) |
3232
| `-recursive` | - | Recursively enumerate subdomains of discovered subdomains |
33-
| `-resume <string>` | - | Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments) |
33+
| `-resume <string>` | - | Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments but the pacing flags -rate, -rate-per-resolver, -t, -timeout, -attempts, -auto-rate, -no-abort and -max-duration) |
3434
| `-retries <int>` | - | Deprecated: use -attempts instead |
3535
| `-retry-inconclusive` {% include unreleased.html flag="retry-inconclusive" %} | - | After each scan, look up once more (against -dns-server) every name that timed out, was refused or failed; names it resolves are reported, but not expanded by -recursive |
3636
| `-sarif <string>` | - | Write takeover candidates to this file as SARIF 2.1.0, for GitHub code scanning (needs CNAME or NS in -type) |

‎internal/cli/cli.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -258,7 +258,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
258258
fs.BoolVar(&f.ct, "ct", false, "Add names from Certificate Transparency logs (crt.sh, one HTTPS request per target, no API key) as candidates and -permute seeds")
259259
fs.StringVar(&f.seedsFile, "seeds", "", "Results file (any -format) whose names also seed -permute; implies -permute")
260260
fs.StringVar(&f.stateFile, "state", defaultStateFile, "Where a run that was interrupted, ran out of -max-duration or was aborted saves its state for -resume")
261-
fs.String("resume", "", "Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments)")
261+
fs.String("resume", "", "Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments but the pacing flags -rate, -rate-per-resolver, -t, -timeout, -attempts, -auto-rate, -no-abort and -max-duration)")
262262
fs.StringVar(&f.diff, "diff", "", "Previous results file (any -format); report only names added or removed since then, and exit 4 when there are changes")
263263
fs.BoolVar(&f.diffRecords, "diff-records", false, "With -diff, also report names whose records changed (a re-pointed CNAME, moved addresses), compared over the -type record types; needs a json, jsonl or csv -diff file")
264264
fs.StringVar(&f.resolvers, "r", "", "File of resolvers (ip or ip:port, one per line) to spread queries over; every hit is re-validated against -dns-server")

‎internal/cli/e2e_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -956,8 +956,8 @@ func TestE2EResume(t *testing.T) {
956956
t.Fatalf("-o after resume does not hold the full result set (err %v)", err)
957957
}
958958

959-
// Resuming takes no other arguments.
960-
if code, _ := runCLI(t, "", "-resume", statePath, "-t", "5"); code != 2 {
959+
// Resuming takes no arguments but the pacing flags (#309).
960+
if code, _ := runCLI(t, "", "-resume", statePath, "-type", "A"); code != 2 {
961961
t.Errorf("-resume with extra arguments: exit %d, want 2", code)
962962
}
963963
}

‎internal/cli/resume.go‎

Lines changed: 72 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,25 @@ func entryDigest(entries []string) string {
134134
// only change what is printed.
135135
var unsavedSettings = map[string]bool{"seed": true, "state": true, "v": true, "progress": true, "max-duration": true}
136136

137+
// pacingFlags may be given next to -resume to change how fast the rest of
138+
// the scan goes, for example after timeouts piled up (#309). They change
139+
// neither the candidate list nor the wildcard fingerprint, so the saved
140+
// position keeps its meaning; the value says whether the flag is a boolean.
141+
var pacingFlags = map[string]bool{
142+
"rate": false, "rate-per-resolver": false, "t": false, "timeout": false,
143+
"attempts": false, "auto-rate": true, "no-abort": true, "max-duration": false,
144+
}
145+
146+
// pacingFlagList names the pacing flags for messages, in a fixed order.
147+
func pacingFlagList() string {
148+
names := make([]string, 0, len(pacingFlags))
149+
for name := range pacingFlags {
150+
names = append(names, "-"+name)
151+
}
152+
sort.Strings(names)
153+
return strings.Join(names, ", ")
154+
}
155+
137156
// effectiveSettings returns every setting's effective value.
138157
func effectiveSettings(fs *flag.FlagSet) map[string]string {
139158
m := map[string]string{}
@@ -147,12 +166,15 @@ func effectiveSettings(fs *flag.FlagSet) map[string]string {
147166

148167
// checkSettings refuses to resume when a setting differs from the
149168
// interrupted run, for example because SUBENUM_DEPTH or the config file
150-
// changed in between: the saved position would mean something else.
169+
// changed in between: the saved position would mean something else. The
170+
// pacing flags may differ (#309).
151171
func (st *resumeState) checkSettings(fs *flag.FlagSet, d *defaultsLoader) error {
152172
now := effectiveSettings(fs)
153173
names := make([]string, 0, len(st.Settings))
154174
for name := range st.Settings {
155-
names = append(names, name)
175+
if _, pacing := pacingFlags[name]; !pacing {
176+
names = append(names, name)
177+
}
156178
}
157179
sort.Strings(names)
158180
for _, name := range names {
@@ -163,15 +185,18 @@ func (st *resumeState) checkSettings(fs *flag.FlagSet, d *defaultsLoader) error
163185
return nil
164186
}
165187

166-
// resumeArgs looks for -resume in args. When present it must be the only
167-
// flag; the state file's original arguments are returned in its place.
188+
// resumeArgs looks for -resume in args. When present the only other flags
189+
// allowed are the pacing flags; the state file's original arguments are
190+
// returned in its place, with the pacing flags given applied over them (#309).
168191
func resumeArgs(args []string) ([]string, *resumeState, error) {
169192
idx := -1
170193
path := ""
194+
var others []string // every argument but -resume and its state file
171195
scan:
172196
for i := 0; i < len(args); i++ {
173197
a := args[i]
174198
if a == "--" {
199+
others = append(others, args[i:]...)
175200
break scan // everything after is a positional argument (#166)
176201
}
177202
if a == "-resume" || a == "--resume" {
@@ -189,16 +214,19 @@ scan:
189214
}
190215
if ok {
191216
idx, path = i, v
217+
continue
192218
}
219+
others = append(others, a)
193220
}
194221
if idx < 0 {
195222
return args, nil, nil
196223
}
197224
if path == "" {
198225
return nil, nil, fmt.Errorf("-resume needs a state file")
199226
}
200-
if len(args) > 2 || (len(args) == 2 && path != args[1]) {
201-
return nil, nil, fmt.Errorf("-resume takes no other arguments; the state file holds the original ones")
227+
overrides, err := pacingOverrides(others)
228+
if err != nil {
229+
return nil, nil, err
202230
}
203231
data, err := os.ReadFile(path) //nolint:gosec // the user names their own state file
204232
if err != nil {
@@ -212,7 +240,44 @@ scan:
212240
return nil, nil, fmt.Errorf("-resume %s: unsupported state version %d", path, st.Version)
213241
}
214242
st.path = path
215-
return st.Args, &st, nil
243+
return withOverrides(st.Args, overrides), &st, nil
244+
}
245+
246+
// pacingOverrides checks that the arguments given next to -resume are all
247+
// pacing flags, and returns them with each value joined to its flag.
248+
func pacingOverrides(args []string) ([]string, error) {
249+
var out []string
250+
for i := 0; i < len(args); i++ {
251+
a := args[i]
252+
name, _, hasValue := strings.Cut(strings.TrimPrefix(strings.TrimPrefix(a, "-"), "-"), "=")
253+
isBool, ok := pacingFlags[name]
254+
if !ok || !strings.HasPrefix(a, "-") {
255+
return nil, fmt.Errorf("-resume takes only the pacing flags %s next to the state file, which holds the other arguments; got %q", pacingFlagList(), a)
256+
}
257+
if !isBool && !hasValue {
258+
if i+1 == len(args) {
259+
return nil, fmt.Errorf("flag needs an argument: -%s", name)
260+
}
261+
i++
262+
a = "-" + name + "=" + args[i]
263+
}
264+
out = append(out, a)
265+
}
266+
return out, nil
267+
}
268+
269+
// withOverrides applies the pacing flags given next to -resume over the
270+
// replayed arguments: placed after them, they win, and the next state file
271+
// records them. They go before a "--", after which they would be positional.
272+
func withOverrides(args, overrides []string) []string {
273+
if len(overrides) == 0 {
274+
return args
275+
}
276+
i := slices.Index(args, "--")
277+
if i < 0 {
278+
i = len(args)
279+
}
280+
return slices.Concat(args[:i:i], overrides, args[i:])
216281
}
217282

218283
// fileDigest returns the sha256 of a file, or "" for none or stdin.

‎internal/cli/resume_test.go‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,87 @@ func TestResumeArgsParsing(t *testing.T) {
166166
}
167167
}
168168

169+
// TestResumeArgsPacing covers #309: the pacing flags may be given next to
170+
// -resume and are applied over the replayed arguments, before any "--";
171+
// every other argument is refused.
172+
func TestResumeArgsPacing(t *testing.T) {
173+
path := writeState(t, resumeState{Args: []string{"-simulate", "-rate", "20", "--", "example.com"}})
174+
got, _, err := resumeArgs([]string{"-t", "4", "-resume", path, "--rate=5", "-auto-rate", "-max-duration", "1h"})
175+
want := []string{"-simulate", "-rate", "20", "-t=4", "--rate=5", "-auto-rate", "-max-duration=1h", "--", "example.com"}
176+
if err != nil || !slices.Equal(got, want) {
177+
t.Errorf("got %q, err %v; want %q", got, err, want)
178+
}
179+
for _, extra := range [][]string{{"-type", "A"}, {"-w", "words.txt"}, {"example.com"}, {"-rate"}, {"--", "-rate", "5"}} {
180+
if _, _, err := resumeArgs(append([]string{"-resume", path}, extra...)); err == nil {
181+
t.Errorf("-resume with %q: no error", extra)
182+
}
183+
}
184+
}
185+
186+
// TestE2EResumeChangesPacing covers #309: a scan stopped while slow resumes
187+
// at another rate; the override is checked against nothing frozen, recorded
188+
// in the next state file, and the scan still finishes complete. Flags that
189+
// change the candidates stay refused.
190+
func TestE2EResumeChangesPacing(t *testing.T) {
191+
t.Parallel()
192+
dir := t.TempDir()
193+
state := filepath.Join(dir, "state.json")
194+
const n = 120
195+
var words strings.Builder
196+
for i := 0; i < n; i++ {
197+
fmt.Fprintf(&words, "host%d\n", i)
198+
}
199+
wl := writeFile(t, "wl.txt", words.String())
200+
base := []string{"-simulate", "-seed", "3", "-hit-rate", "50", "-progress=false", "-format", "jsonl", "-w", wl}
201+
names := func(out string) map[string]bool {
202+
results, _ := jsonlRecords(t, out)
203+
return resultNames(results)
204+
}
205+
_, ref := runCLI(t, "", append(slices.Clone(base), "example.com")...)
206+
want := names(ref)
207+
208+
// -rate 20 with 120 words needs about 6 s; the budget is 1 s.
209+
code, out := runCLIMerged(t, append(slices.Clone(base), "-t", "1", "-rate", "20", "-max-duration", "1s", "-state", state, "example.com")...)
210+
if code != 0 {
211+
t.Fatalf("first run: exit %d\n%s", code, out)
212+
}
213+
if _, err := os.Stat(state); err != nil {
214+
t.Fatalf("first run saved no state: %v", err)
215+
}
216+
// Flags that change the candidate list stay frozen.
217+
if code, out := runCLIMerged(t, "-resume", state, "-type", "A"); code != exitUsage || !strings.Contains(out, "pacing flags") {
218+
t.Fatalf("-resume -type A: exit %d, want %d\n%s", code, exitUsage, out)
219+
}
220+
221+
// A resumed run at another, still slow rate stops again and records it.
222+
if code, out := runCLIMerged(t, "-resume", state, "-rate", "40"); code != 0 {
223+
t.Fatalf("resume at -rate 40: exit %d\n%s", code, out)
224+
}
225+
data, err := os.ReadFile(state)
226+
if err != nil {
227+
t.Fatalf("resume at -rate 40 saved no state: %v", err)
228+
}
229+
var st resumeState
230+
if err := json.Unmarshal(data, &st); err != nil {
231+
t.Fatal(err)
232+
}
233+
if !slices.Contains(st.Args, "-rate=40") || st.Settings["rate"] != "40" {
234+
t.Fatalf("saved args %q, rate setting %q; want the -rate 40 override recorded", st.Args, st.Settings["rate"])
235+
}
236+
237+
// Unthrottled and with a budget it cannot spend, the scan finishes.
238+
code, out = runCLI(t, "", "-resume", state, "-rate", "0", "-t", "8", "-max-duration", "1h")
239+
if code != 0 {
240+
t.Fatalf("unthrottled resume: exit %d", code)
241+
}
242+
if _, err := os.Stat(state); !os.IsNotExist(err) {
243+
t.Errorf("the finished run's state file was not removed (err %v)", err)
244+
}
245+
if got := names(out); !maps.Equal(got, want) {
246+
t.Errorf("resumed scan reported %d names, uninterrupted %d", len(got), len(want))
247+
}
248+
}
249+
169250
// TestE2EResumeCountsRefoundNamesOnce covers #156: the resume point lags the
170251
// saved results by up to a worker pool's worth of names, so a resumed scan
171252
// finds some of them again; they are counted once.

0 commit comments

Comments
 (0)