You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 274df44
Browse filesBrowse the repository at this point in the historyBrowse files
Save resume state when -max-duration or the guard stops a run (#294)
A run that ran out of -max-duration with names untested, or that the
reliability guard aborted, now saves its state to the -state file and
prints the "continue with: subenum -resume" line, as an interrupted run
does. Exit codes are unchanged.
A target stopped this way is saved as unfinished; a resumed run that
expires again saves its state again. mergeStats takes Aborted and Expired
from the resumed part and drops the saved part's skipped count, so a scan
finished by a resume reports complete. The explain_stats advice for a
-max-duration stop mentions -resume for command-line runs.
Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com>
Fixes#294
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,6 +33,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
33
33
-`-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).
34
34
35
35
### Changed
36
+
- A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves its resume state to the `-state` file like an interrupted one and prints `continue with: subenum -resume <file>`, so a time-boxed job continues where the last one stopped instead of testing the same first names again. The exit code is unchanged (0 for a spent budget). A resumed run that stops the same way saves its state again, and one that finishes no longer counts the names an earlier part skipped, so it can end `complete` (#294).
36
37
- The command's source moved out of the repository root: the `main_*.go` files are now `internal/cli/*.go`, and the root `main.go` only calls `cli.Main`. `go install github.com/TMHSDigital/subenum@latest`, the Action, the Docker image and `-X main.Version` builds are unchanged. The lab scenarios are embedded through a small `examples/labs` package, and `SECURITY.md` is now `.github/SECURITY.md`.
37
38
- A `CNAME` lookup sends one CNAME query instead of going through Go's `LookupCNAME`, which also sent A and AAAA queries and threw their answers away. `-type A,AAAA,CNAME` now costs three queries a name instead of five, so the same `-rate` tests two-thirds more names per second, and `-dry-run`, `-max-queries` help and the `-rate-per-resolver` reservation count it that way. The query goes through the same transports (UDP with TCP fallback, DoT, DoH), rate limiter and query counter as the other types (#249).
38
39
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).
Copy file name to clipboardExpand all lines: README.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -271,7 +271,7 @@ make help # list all targets
271
271
|`-no-abort`|`false`| Keep scanning after the 20% resolver failure-rate abort (warning is still emitted) |
272
272
|`-auto-rate`|`false`| When more than 20% of the last 200 lookups failed, halve the query rate (from the rate reached so far when `-rate` is 0) and keep going, with a `rate` notice; speed back up by a quarter at a time, to the rate it started from, once a window of lookups is healthy. Only after 6 back-offs, or at 5 queries/s, does the reliability guard abort. The report counts `rate_backoffs` and the `final_rate`. Pairs well with `-retry-inconclusive`, which retries the names that failed while the resolver was overloaded |
273
273
|`-max-queries <n>`|`0`| Max candidate names to test (0 = unlimited). Each name sends one query per record type per attempt |
274
-
|`-max-duration <d>`|`0`| Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict. Under `-dL` it covers all targets |
274
+
|`-max-duration <d>`|`0`| Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict, and saves a `-state` file so `-resume` tests the rest. Under `-dL` it covers all targets |
275
275
|`-retry-inconclusive`|`false`| After each scan, look up once more (against `-dns-server`, never the `-r` pool) every name that timed out, was refused or failed. An answer replaces the first outcome in the counts; a name that resolves is reported, but not expanded by `-recursive`. Skipped after an abort, an interrupt or once `-max-duration` runs out. The report counts `retried` and `recovered`|
276
276
|`-inconclusive <file>`| n/a | Write the names still without an answer (timeout, refused, other) to this file, one per line, sorted, so a later run can retry them with `-w <file>`|
277
277
|`-o <file>`| n/a | Write results to file in addition to stdout |
@@ -305,9 +305,9 @@ make help # list all targets
305
305
|`-dry-run`|`false`| Print the plan without scanning: candidates after dedupe and `-exclude`, query and duration estimates, and the resolver preflight (its only DNS query), then exit; JSON with `-format json` or `jsonl`|
306
306
|`-version`| n/a | Print version and exit |
307
307
|`-retries <n>`| n/a |**Deprecated** - alias for `-attempts`, prints a warning |
308
-
|`-resume <file>`| n/a | Resume an interrupted run from its state file; takes no other arguments |
308
+
|`-resume <file>`| n/a | Resume a run that was interrupted, ran out of `-max-duration` or was aborted, from its state file; takes no other arguments |
|`-state <file>`|`subenum-resume.json`| Where an interrupted run saves its state for `-resume`|
310
+
|`-state <file>`|`subenum-resume.json`| Where a run that was interrupted, ran out of `-max-duration` or was aborted saves its state for `-resume`|
311
311
312
312
<br>
313
313
@@ -378,7 +378,7 @@ Press `Ctrl+C` at any time to abort. In-flight queries drain, partial results ar
378
378
subenum -resume subenum-resume.json
379
379
```
380
380
381
-
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. Runs that read the wordlist or domain list from stdin cannot be resumed.
381
+
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves the same state (its exit code does not change), so `-resume` continues it too: a scheduled job with a time budget can work through a large wordlist over several runs. Runs that read the wordlist or domain list from stdin cannot be resumed.
Copy file name to clipboardExpand all lines: docs/_includes/flags.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,7 +19,7 @@
19
19
|`-hit-rate <int>`| 15 | In simulation mode, percentage of names that resolve (1-100) |
20
20
|`-inconclusive <string>` {% include unreleased.html flag="inconclusive" %} | - | Write the names still without an answer (timeout, refused, other) to this file, one per line, to rescan with -w |
21
21
|`-log-format <string>` {% include unreleased.html flag="log-format" %} | text | Diagnostics format on stderr: text, or json for one JSON object per line (level, kind, target, message); results on stdout are unchanged |
22
-
|`-max-duration <duration>` {% include unreleased.html flag="max-duration" %} | 0s | Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report and a degraded verdict |
22
+
|`-max-duration <duration>` {% include unreleased.html flag="max-duration" %} | 0s | Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report, a degraded verdict and saved state for -resume to continue from|
23
23
|`-max-queries <int>`| - | Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt |
24
24
|`-no-abort`| - | Do not abort when the resolver failure rate exceeds 20% (warning is still emitted) |
25
25
|`-o <string>`| - | Write results to file (in addition to stdout) |
@@ -30,7 +30,7 @@
30
30
|`-rate <int>`| - | Max DNS queries per second on the wire, all workers combined; counts every record type, retry and wildcard probe (0 = unlimited) |
31
31
|`-rate-per-resolver <int>` {% include unreleased.html flag="rate-per-resolver" %} | - | With -r, max DNS queries per second sent to each pool resolver; -rate still caps the total (0 = unlimited) |
32
32
|`-recursive`| - | Recursively enumerate subdomains of discovered subdomains |
33
-
|`-resume <string>`| - | Resume an interrupted run from its state file (no other arguments) |
33
+
|`-resume <string>`| - | Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments) |
34
34
|`-retries <int>`| - | Deprecated: use -attempts instead |
35
35
|`-retry-inconclusive` {% include unreleased.html flag="retry-inconclusive" %} | - | After each scan, look up once more (against -dns-server) every name that timed out, was refused or failed; names it resolves are reported, but not expanded by -recursive |
36
36
|`-sarif <string>`| - | Write takeover candidates to this file as SARIF 2.1.0, for GitHub code scanning (needs CNAME or NS in -type) |
@@ -41,7 +41,7 @@
41
41
|`-simulate`| - | Run in simulation mode without actual DNS queries (for testing) |
42
42
|`-simulate-zone <string>`| - | Lab mode: answer every query from this scenario file via a local DNS server, so no traffic leaves the machine (see the Labs page) |
43
43
|`-sources <string>` {% include unreleased.html flag="sources" %} | - | Comma-separated keyless passive sources whose names become candidates and -permute seeds: ct (Certificate Transparency, crt.sh), wayback (Wayback Machine); one HTTPS request per source and target |
44
-
|`-state <string>`| subenum-resume.json | Where an interrupted run saves its state for -resume |
44
+
|`-state <string>`| subenum-resume.json | Where a run that was interrupted, ran out of -max-duration or was aborted saves its state for -resume |
45
45
|`-stats <string>`| - | Write a JSON run-quality report (outcomes, queries sent, verdict) to this file |
46
46
|`-t <int>`| 100 | Number of concurrent workers |
47
47
|`-takeover-providers <string>` {% include unreleased.html flag="takeover-providers" %} | - | File of takeover-prone services, one "<suffix><name>" per line (.example-cdn.net example-cdn; # comments allowed; name - removes a suffix), added to the built-in list for takeover hints |
Copy file name to clipboardExpand all lines: docs/monitoring.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -116,7 +116,7 @@ jobs:
116
116
| `previous` | | A previous results file to diff against (add `-diff-records` to `args` to also catch record changes) |
117
117
| `baseline-artifact` | | Artifact name for the baseline: downloaded before, replaced after a complete run |
118
118
| `sarif` | | Write takeover candidates as SARIF |
119
-
| `max-duration`{% include unreleased.html action="max-duration" %} | | Time budget, e.g. `20m` (`-max-duration`). Keep it under the job's `timeout-minutes`: a job killed by its timeout leaves no report, while a scan that reaches its budget stops testing new names and still writes results and a `degraded` report |
119
+
| `max-duration`{% include unreleased.html action="max-duration" %} | | Time budget, e.g. `20m` (`-max-duration`). Keep it under the job's `timeout-minutes`: a job killed by its timeout leaves no report, while a scan that reaches its budget stops testing new names and still writes results and a `degraded` report. The CLI also saves a resume state then (`subenum-resume.json`, or the `-state` path), and `subenum -resume <file>` tests the names the budget left; the action does not carry that file to the next run |
120
120
| `fail-on` | `unreliable` | Fail on this verdict or worse: `unreliable`, `degraded` or `never` |
121
121
| `version` | `latest` | Release to install, or `source` to build the action's checkout |
122
122
| `token` | `github.token` | Token for downloading the release and the baseline artifact |
fs.IntVar(&f.depth, "depth", 1, "Max recursion depth when -recursive is set (1 = no recursion)")
244
244
fs.BoolVar(&f.noAbort, "no-abort", false, "Do not abort when the resolver failure rate exceeds 20% (warning is still emitted)")
245
245
fs.IntVar(&f.maxQueries, "max-queries", 0, "Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt")
246
-
fs.DurationVar(&f.maxDuration, "max-duration", 0, "Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report and a degraded verdict")
246
+
fs.DurationVar(&f.maxDuration, "max-duration", 0, "Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report, a degraded verdict and saved state for -resume to continue from")
247
247
fs.BoolVar(&f.retryIncon, "retry-inconclusive", false, "After each scan, look up once more (against -dns-server) every name that timed out, was refused or failed; names it resolves are reported, but not expanded by -recursive")
248
248
fs.StringVar(&f.inconFile, "inconclusive", "", "Write the names still without an answer (timeout, refused, other) to this file, one per line, to rescan with -w")
249
249
fs.StringVar(&f.statsFile, "stats", "", "Write a JSON run-quality report (outcomes, queries sent, verdict) to this file")
fs.StringVar(&f.sources, "sources", "", "Comma-separated keyless passive sources whose names become candidates and -permute seeds: ct (Certificate Transparency, crt.sh), wayback (Wayback Machine); one HTTPS request per source and target")
258
258
fs.BoolVar(&f.ct, "ct", false, "Add names from Certificate Transparency logs (crt.sh, one HTTPS request per target, no API key) as candidates and -permute seeds")
fs.StringVar(&f.stateFile, "state", defaultStateFile, "Where an interrupted run saves its state for -resume")
261
-
fs.String("resume", "", "Resume an interrupted run from its state file (no other arguments)")
260
+
fs.StringVar(&f.stateFile, "state", defaultStateFile, "Where a run that was interrupted, ran out of -max-duration or was aborted saves its state for -resume")
261
+
fs.String("resume", "", "Resume a run that was interrupted, ran out of -max-duration or was aborted, from its state file (no other arguments)")
262
262
fs.StringVar(&f.diff, "diff", "", "Previous results file (any -format); report only names added or removed since then, and exit 4 when there are changes")
263
263
fs.BoolVar(&f.diffRecords, "diff-records", false, "With -diff, also report names whose records changed (a re-pointed CNAME, moved addresses), compared over the -type record types; needs a json, jsonl or csv -diff file")
264
264
fs.StringVar(&f.resolvers, "r", "", "File of resolvers (ip or ip:port, one per line) to spread queries over; every hit is re-validated against -dns-server")
@@ -706,6 +706,12 @@ type targetResult struct {
706
706
axfrtargetAXFR// -axfr zone transfer, kept for the report and resume state
707
707
}
708
708
709
+
// unfinished reports whether the pass stopped with candidates untested, on
710
+
// the -max-duration budget or a reliability abort (#294).
p(" %d names got no answer; -retry-inconclusive looks them up again at the end of the scan.", failed)
882
882
}
883
883
ift.SkippedCap>0 {
884
-
budget:="-max-queries"
884
+
// A run that ends on -max-queries writes no resume state, so -resume
885
+
// is not offered for it (#320). The command line saves one when
886
+
// -max-duration runs out (#294); this server's scans never do.
885
887
ift.StoppedBy=="duration" {
886
-
budget="-max-duration"
888
+
p(" %d names were not tested because -max-duration ran out; raise it and run the scan again, or continue a command-line run with subenum -resume and the state file it saved.", t.SkippedCap)
889
+
} else {
890
+
p(" %d names were not tested because -max-queries ran out; raise it and run the scan again.", t.SkippedCap)
887
891
}
888
-
// No resume state is written for a run that ends on a budget (only an
889
-
// interrupted run saves one), so -resume is not offered here (#320).
890
-
p(" %d names were not tested because %s ran out; raise it and run the scan again.", t.SkippedCap, budget)
891
892
}
892
893
ift.Wildcard {
893
894
p(" The domain answers every name (a wildcard), so subenum dropped %d answers matching the wildcard; a real name that answers the same way is dropped too.", o.WildcardFiltered)
0 commit comments