Skip to content

Commit 274df44

Browse files
committed
Save resume state when -max-duration or the guard stops a run (#294)
A run that ran out of -max-duration with names untested, or that the reliability guard aborted, now saves its state to the -state file and prints the "continue with: subenum -resume" line, as an interrupted run does. Exit codes are unchanged. A target stopped this way is saved as unfinished; a resumed run that expires again saves its state again. mergeStats takes Aborted and Expired from the resumed part and drops the saved part's skipped count, so a scan finished by a resume reports complete. The explain_stats advice for a -max-duration stop mentions -resume for command-line runs. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com> Fixes #294
1 parent 361895a commit 274df44

11 files changed

Lines changed: 210 additions & 31 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3333
- `-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).
3434

3535
### Changed
36+
- A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves its resume state to the `-state` file like an interrupted one and prints `continue with: subenum -resume <file>`, so a time-boxed job continues where the last one stopped instead of testing the same first names again. The exit code is unchanged (0 for a spent budget). A resumed run that stops the same way saves its state again, and one that finishes no longer counts the names an earlier part skipped, so it can end `complete` (#294).
3637
- The command's source moved out of the repository root: the `main_*.go` files are now `internal/cli/*.go`, and the root `main.go` only calls `cli.Main`. `go install github.com/TMHSDigital/subenum@latest`, the Action, the Docker image and `-X main.Version` builds are unchanged. The lab scenarios are embedded through a small `examples/labs` package, and `SECURITY.md` is now `.github/SECURITY.md`.
3738
- A `CNAME` lookup sends one CNAME query instead of going through Go's `LookupCNAME`, which also sent A and AAAA queries and threw their answers away. `-type A,AAAA,CNAME` now costs three queries a name instead of five, so the same `-rate` tests two-thirds more names per second, and `-dry-run`, `-max-queries` help and the `-rate-per-resolver` reservation count it that way. The query goes through the same transports (UDP with TCP fallback, DoT, DoH), rate limiter and query counter as the other types (#249).
3839
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).

‎README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ make help # list all targets
271271
| `-no-abort` | `false` | Keep scanning after the 20% resolver failure-rate abort (warning is still emitted) |
272272
| `-auto-rate` | `false` | When more than 20% of the last 200 lookups failed, halve the query rate (from the rate reached so far when `-rate` is 0) and keep going, with a `rate` notice; speed back up by a quarter at a time, to the rate it started from, once a window of lookups is healthy. Only after 6 back-offs, or at 5 queries/s, does the reliability guard abort. The report counts `rate_backoffs` and the `final_rate`. Pairs well with `-retry-inconclusive`, which retries the names that failed while the resolver was overloaded |
273273
| `-max-queries <n>` | `0` | Max candidate names to test (0 = unlimited). Each name sends one query per record type per attempt |
274-
| `-max-duration <d>` | `0` | Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict. Under `-dL` it covers all targets |
274+
| `-max-duration <d>` | `0` | Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict, and saves a `-state` file so `-resume` tests the rest. Under `-dL` it covers all targets |
275275
| `-retry-inconclusive` | `false` | After each scan, look up once more (against `-dns-server`, never the `-r` pool) every name that timed out, was refused or failed. An answer replaces the first outcome in the counts; a name that resolves is reported, but not expanded by `-recursive`. Skipped after an abort, an interrupt or once `-max-duration` runs out. The report counts `retried` and `recovered` |
276276
| `-inconclusive <file>` | n/a | Write the names still without an answer (timeout, refused, other) to this file, one per line, sorted, so a later run can retry them with `-w <file>` |
277277
| `-o <file>` | n/a | Write results to file in addition to stdout |
@@ -305,9 +305,9 @@ make help # list all targets
305305
| `-dry-run` | `false` | Print the plan without scanning: candidates after dedupe and `-exclude`, query and duration estimates, and the resolver preflight (its only DNS query), then exit; JSON with `-format json` or `jsonl` |
306306
| `-version` | n/a | Print version and exit |
307307
| `-retries <n>` | n/a | **Deprecated** - alias for `-attempts`, prints a warning |
308-
| `-resume <file>` | n/a | Resume an interrupted run from its state file; takes no other arguments |
308+
| `-resume <file>` | n/a | Resume a run that was interrupted, ran out of `-max-duration` or was aborted, from its state file; takes no other arguments |
309309
| `-seeds <file>` | n/a | Results file (any `-format`) whose names also seed `-permute`; implies `-permute` |
310-
| `-state <file>` | `subenum-resume.json` | Where an interrupted run saves its state for `-resume` |
310+
| `-state <file>` | `subenum-resume.json` | Where a run that was interrupted, ran out of `-max-duration` or was aborted saves its state for `-resume` |
311311

312312
<br>
313313

@@ -378,7 +378,7 @@ Press `Ctrl+C` at any time to abort. In-flight queries drain, partial results ar
378378
subenum -resume subenum-resume.json
379379
```
380380

381-
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. Runs that read the wordlist or domain list from stdin cannot be resumed.
381+
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves the same state (its exit code does not change), so `-resume` continues it too: a scheduled job with a time budget can work through a large wordlist over several runs. Runs that read the wordlist or domain list from stdin cannot be resumed.
382382

383383
**Exit codes**
384384

‎docs/_includes/flags.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
| `-hit-rate <int>` | 15 | In simulation mode, percentage of names that resolve (1-100) |
2020
| `-inconclusive <string>` {% include unreleased.html flag="inconclusive" %} | - | Write the names still without an answer (timeout, refused, other) to this file, one per line, to rescan with -w |
2121
| `-log-format <string>` {% include unreleased.html flag="log-format" %} | text | Diagnostics format on stderr: text, or json for one JSON object per line (level, kind, target, message); results on stdout are unchanged |
22-
| `-max-duration <duration>` {% include unreleased.html flag="max-duration" %} | 0s | Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report and a degraded verdict |
22+
| `-max-duration <duration>` {% include unreleased.html flag="max-duration" %} | 0s | Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report, a degraded verdict and saved state for -resume to continue from |
2323
| `-max-queries <int>` | - | Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt |
2424
| `-no-abort` | - | Do not abort when the resolver failure rate exceeds 20% (warning is still emitted) |
2525
| `-o <string>` | - | Write results to file (in addition to stdout) |
@@ -30,7 +30,7 @@
3030
| `-rate <int>` | - | Max DNS queries per second on the wire, all workers combined; counts every record type, retry and wildcard probe (0 = unlimited) |
3131
| `-rate-per-resolver <int>` {% include unreleased.html flag="rate-per-resolver" %} | - | With -r, max DNS queries per second sent to each pool resolver; -rate still caps the total (0 = unlimited) |
3232
| `-recursive` | - | Recursively enumerate subdomains of discovered subdomains |
33-
| `-resume <string>` | - | Resume an interrupted run from its state file (no other arguments) |
33+
| `-resume <string>` | - | Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments) |
3434
| `-retries <int>` | - | Deprecated: use -attempts instead |
3535
| `-retry-inconclusive` {% include unreleased.html flag="retry-inconclusive" %} | - | After each scan, look up once more (against -dns-server) every name that timed out, was refused or failed; names it resolves are reported, but not expanded by -recursive |
3636
| `-sarif <string>` | - | Write takeover candidates to this file as SARIF 2.1.0, for GitHub code scanning (needs CNAME or NS in -type) |
@@ -41,7 +41,7 @@
4141
| `-simulate` | - | Run in simulation mode without actual DNS queries (for testing) |
4242
| `-simulate-zone <string>` | - | Lab mode: answer every query from this scenario file via a local DNS server, so no traffic leaves the machine (see the Labs page) |
4343
| `-sources <string>` {% include unreleased.html flag="sources" %} | - | Comma-separated keyless passive sources whose names become candidates and -permute seeds: ct (Certificate Transparency, crt.sh), wayback (Wayback Machine); one HTTPS request per source and target |
44-
| `-state <string>` | subenum-resume.json | Where an interrupted run saves its state for -resume |
44+
| `-state <string>` | subenum-resume.json | Where a run that was interrupted, ran out of -max-duration or was aborted saves its state for -resume |
4545
| `-stats <string>` | - | Write a JSON run-quality report (outcomes, queries sent, verdict) to this file |
4646
| `-t <int>` | 100 | Number of concurrent workers |
4747
| `-takeover-providers <string>` {% include unreleased.html flag="takeover-providers" %} | - | File of takeover-prone services, one "&lt;suffix&gt; &lt;name&gt;" per line (.example-cdn.net example-cdn; # comments allowed; name - removes a suffix), added to the built-in list for takeover hints |

‎docs/monitoring.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ jobs:
116116
| `previous` | | A previous results file to diff against (add `-diff-records` to `args` to also catch record changes) |
117117
| `baseline-artifact` | | Artifact name for the baseline: downloaded before, replaced after a complete run |
118118
| `sarif` | | Write takeover candidates as SARIF |
119-
| `max-duration`{% include unreleased.html action="max-duration" %} | | Time budget, e.g. `20m` (`-max-duration`). Keep it under the job's `timeout-minutes`: a job killed by its timeout leaves no report, while a scan that reaches its budget stops testing new names and still writes results and a `degraded` report |
119+
| `max-duration`{% include unreleased.html action="max-duration" %} | | Time budget, e.g. `20m` (`-max-duration`). Keep it under the job's `timeout-minutes`: a job killed by its timeout leaves no report, while a scan that reaches its budget stops testing new names and still writes results and a `degraded` report. The CLI also saves a resume state then (`subenum-resume.json`, or the `-state` path), and `subenum -resume <file>` tests the names the budget left; the action does not carry that file to the next run |
120120
| `fail-on` | `unreliable` | Fail on this verdict or worse: `unreliable`, `degraded` or `never` |
121121
| `version` | `latest` | Release to install, or `source` to build the action's checkout |
122122
| `token` | `github.token` | Token for downloading the release and the baseline artifact |

‎internal/cli/cli.go‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
243243
fs.IntVar(&f.depth, "depth", 1, "Max recursion depth when -recursive is set (1 = no recursion)")
244244
fs.BoolVar(&f.noAbort, "no-abort", false, "Do not abort when the resolver failure rate exceeds 20% (warning is still emitted)")
245245
fs.IntVar(&f.maxQueries, "max-queries", 0, "Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt")
246-
fs.DurationVar(&f.maxDuration, "max-duration", 0, "Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report and a degraded verdict")
246+
fs.DurationVar(&f.maxDuration, "max-duration", 0, "Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report, a degraded verdict and saved state for -resume to continue from")
247247
fs.BoolVar(&f.retryIncon, "retry-inconclusive", false, "After each scan, look up once more (against -dns-server) every name that timed out, was refused or failed; names it resolves are reported, but not expanded by -recursive")
248248
fs.StringVar(&f.inconFile, "inconclusive", "", "Write the names still without an answer (timeout, refused, other) to this file, one per line, to rescan with -w")
249249
fs.StringVar(&f.statsFile, "stats", "", "Write a JSON run-quality report (outcomes, queries sent, verdict) to this file")
@@ -257,8 +257,8 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
257257
fs.StringVar(&f.sources, "sources", "", "Comma-separated keyless passive sources whose names become candidates and -permute seeds: ct (Certificate Transparency, crt.sh), wayback (Wayback Machine); one HTTPS request per source and target")
258258
fs.BoolVar(&f.ct, "ct", false, "Add names from Certificate Transparency logs (crt.sh, one HTTPS request per target, no API key) as candidates and -permute seeds")
259259
fs.StringVar(&f.seedsFile, "seeds", "", "Results file (any -format) whose names also seed -permute; implies -permute")
260-
fs.StringVar(&f.stateFile, "state", defaultStateFile, "Where an interrupted run saves its state for -resume")
261-
fs.String("resume", "", "Resume an interrupted run from its state file (no other arguments)")
260+
fs.StringVar(&f.stateFile, "state", defaultStateFile, "Where a run that was interrupted, ran out of -max-duration or was aborted saves its state for -resume")
261+
fs.String("resume", "", "Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments)")
262262
fs.StringVar(&f.diff, "diff", "", "Previous results file (any -format); report only names added or removed since then, and exit 4 when there are changes")
263263
fs.BoolVar(&f.diffRecords, "diff-records", false, "With -diff, also report names whose records changed (a re-pointed CNAME, moved addresses), compared over the -type record types; needs a json, jsonl or csv -diff file")
264264
fs.StringVar(&f.resolvers, "r", "", "File of resolvers (ip or ip:port, one per line) to spread queries over; every hit is re-validated against -dns-server")
@@ -706,6 +706,12 @@ type targetResult struct {
706706
axfr targetAXFR // -axfr zone transfer, kept for the report and resume state
707707
}
708708

709+
// unfinished reports whether the pass stopped with candidates untested, on
710+
// the -max-duration budget or a reliability abort (#294).
711+
func (r targetResult) unfinished() bool {
712+
return r.resolverAbort || (r.done && r.final.Stats.Expired)
713+
}
714+
709715
// scanTarget runs one pass over a domain and streams its events to the
710716
// session's writer, and each result to emit.
711717
func scanTarget(ctx context.Context, s *session, p scanPass, emit func(scan.Event)) (res targetResult) {

‎internal/cli/lab_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ func TestE2EMaxDuration(t *testing.T) {
133133
stats := filepath.Join(t.TempDir(), "run.json")
134134
// -rate 10 with 30 words needs about 3 s; the budget is a third of that.
135135
code, out := runCLI(t, "", "-progress=false", "-w", "examples/labs/words.txt", "-t", "2", "-rate", "10",
136-
"-max-duration", "1s", "-stats", stats, "-simulate-zone", filepath.Join("examples", "labs", "lab1-first-scan.zone"), "lab.example")
136+
"-max-duration", "1s", "-stats", stats, "-state", filepath.Join(t.TempDir(), "state.json"), "-simulate-zone", filepath.Join("examples", "labs", "lab1-first-scan.zone"), "lab.example")
137137
if code != 0 {
138138
t.Fatalf("exit %d\n%s", code, out)
139139
}

‎internal/cli/mcp.go‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -881,13 +881,14 @@ func explainTarget(p func(string, ...any), t output.TargetSummary) {
881881
p(" %d names got no answer; -retry-inconclusive looks them up again at the end of the scan.", failed)
882882
}
883883
if t.SkippedCap > 0 {
884-
budget := "-max-queries"
884+
// A run that ends on -max-queries writes no resume state, so -resume
885+
// is not offered for it (#320). The command line saves one when
886+
// -max-duration runs out (#294); this server's scans never do.
885887
if t.StoppedBy == "duration" {
886-
budget = "-max-duration"
888+
p(" %d names were not tested because -max-duration ran out; raise it and run the scan again, or continue a command-line run with subenum -resume and the state file it saved.", t.SkippedCap)
889+
} else {
890+
p(" %d names were not tested because -max-queries ran out; raise it and run the scan again.", t.SkippedCap)
887891
}
888-
// No resume state is written for a run that ends on a budget (only an
889-
// interrupted run saves one), so -resume is not offered here (#320).
890-
p(" %d names were not tested because %s ran out; raise it and run the scan again.", t.SkippedCap, budget)
891892
}
892893
if t.Wildcard {
893894
p(" The domain answers every name (a wildcard), so subenum dropped %d answers matching the wildcard; a real name that answers the same way is dropped too.", o.WildcardFiltered)

‎internal/cli/mcp_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,7 @@ func TestMCPExplainStatsBranches(t *testing.T) {
343343
}
344344
for _, want := range []string{
345345
"Overall verdict: unreliable", "too many lookups failed", "lab run", "a limit of 50 per second",
346-
"Error: resolver stopped answering", "Next time: lower -rate", "because -max-duration ran out",
346+
"Error: resolver stopped answering", "Next time: lower -rate", "because -max-duration ran out", "subenum -resume",
347347
"(a wildcard)", "2 results have a subdomain-takeover hint", "1 public names answer with internal",
348348
"Zone transfer (AXFR) was allowed by ns1.lab.example", "plain NSEC", "Passive source wayback failed (503)",
349349
"Permutation pass: 40 candidates from 4 seeds, 1 found", "Compared with prev.jsonl: 1 added, 2 removed",

0 commit comments

Comments
 (0)