Skip to content

Commit 96a04d5

Browse files
TMHSDigitalclaude
andcommitted
Don't spend -max-duration on pre-scan probes (#340)
scan.Run now moves the deadline forward by the time the resolver preflight and wildcard probes took, as long as the budget had not run out before the scan started (a later -dL target keeps a spent budget). Under a low -rate those probes could take all of a short budget, so a run tested no names and every -resume probed again without progress. The time-budget notice says when the budget ran out before any name was tested. TestE2EMaxDuration now requires names tested, and a -resume that skips fewer names than the first run. Fixes #340 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 9ae47e2 commit 96a04d5

6 files changed

Lines changed: 50 additions & 21 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6767
- Less memory on large wordlists and busy scans: `-sources`, `-ct`, `-axfr` and `-permute` check their names against the wordlist without building a set as large as it (about 200 MB per feature and target on a 5M-line list), a `-simulate-zone` lab scan no longer keeps a count per queried name (about 170 MB on a 1M-line list), and a `CNAME` lookup reads a UDP answer into a buffer sized for the 1232-byte EDNS payload it advertises instead of 64 KB. Results are unchanged (#321).
6868

6969
### Fixed
70+
- The resolver preflight and wildcard probes made before a target's first name no longer use up `-max-duration`. Under a low `-rate` they could take all of a short budget, so the run tested no names and each `-resume` probed again and made no progress. A budget that runs out before any name was tested now says so (#340).
7071
- Release archives and the container image carry the licence notices of the Go standard library and every module linked into the binary, in a `THIRD_PARTY_LICENSES` file generated at release time by `tools/licenses` from the module cache. The image also carries the project's `LICENSE`; both are in `/usr/share/doc/subenum/` (#326).
7172
- With `-format csv`, each row reaches stdout as its result is found, as with the other formats, instead of when 4 KB had built up or the run ended; a pipe sees live rows and a force quit loses none (#319).
7273
- With both `-max-queries` and `-max-duration`, a deadline that fires after the query cap no longer loses the chunk of names admitted but not yet handed out: they count as skipped, so tested plus skipped adds up to every candidate and progress reaches 100%, and the run is marked as stopped by the time budget. Children of a name found after a budget stop whose whole branch is excluded count as excluded, not skipped, so they cannot make a complete scan look incomplete (#317).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -273,7 +273,7 @@ This table describes `main`, which can be ahead of the latest release: a release
273273
| `-no-abort` | `false` | Keep scanning after the 20% resolver failure-rate abort (warning is still emitted) |
274274
| `-auto-rate` | `false` | When more than 20% of the last 200 lookups failed, halve the query rate (from the rate reached so far when `-rate` is 0) and keep going, with a `rate` notice; speed back up by a quarter at a time, to the rate it started from, once a window of lookups is healthy. Only after 6 back-offs, or at 5 queries/s, does the reliability guard abort. The report counts `rate_backoffs` and the `final_rate`. Pairs well with `-retry-inconclusive`, which retries the names that failed while the resolver was overloaded |
275275
| `-max-queries <n>` | `0` | Max candidate names to test (0 = unlimited). Each name sends one query per record type per attempt |
276-
| `-max-duration <d>` | `0` | Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict, and saves a `-state` file so `-resume` tests the rest. Under `-dL` it covers all targets |
276+
| `-max-duration <d>` | `0` | Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict, and saves a `-state` file so `-resume` tests the rest. Under `-dL` it covers all targets. The resolver preflight and wildcard probes made before a target's first name are not counted |
277277
| `-retry-inconclusive` | `false` | After each scan, look up once more (against `-dns-server`, never the `-r` pool) every name that timed out, was refused or failed. An answer replaces the first outcome in the counts; a name that resolves is reported, but not expanded by `-recursive`. Skipped after an abort, an interrupt or once `-max-duration` runs out. The report counts `retried` and `recovered` |
278278
| `-inconclusive <file>` | n/a | Write the names still without an answer (timeout, refused, other) to this file, one per line, sorted, so a later run can retry them with `-w <file>` |
279279
| `-o <file>` | n/a | Write results to file in addition to stdout; repeat for several files, each in `-format` or in the format named as a prefix (`-o names.txt -o jsonl:results.jsonl`) |

‎internal/cli/lab_test.go‎

Lines changed: 33 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -137,36 +137,50 @@ func TestE2ELabVerdicts(t *testing.T) {
137137

138138
// TestE2EMaxDuration covers #190: a run that hits its time budget still ends
139139
// with a report, a degraded verdict and stopped_by "duration", and a negative
140-
// budget is a usage error.
140+
// budget is a usage error. The checks before the first name do not use up
141+
// the budget, so the run and each -resume of it test names (#340).
141142
func TestE2EMaxDuration(t *testing.T) {
142143
t.Parallel()
143144
stats := filepath.Join(t.TempDir(), "run.json")
144-
// -rate 10 with 30 words needs about 3 s; the budget is a third of that.
145-
code, out := runCLI(t, "", append(localArgs("-progress=false", "-w", "examples/labs/words.txt", "-t", "2", "-rate", "10",
146-
"-max-duration", "1s", "-stats", stats, "-state", filepath.Join(t.TempDir(), "state.json"), "-simulate-zone", filepath.Join("examples", "labs", "lab1-first-scan.zone")), "lab.example")...)
147-
if code != 0 {
148-
t.Fatalf("exit %d\n%s", code, out)
149-
}
150-
data, err := os.ReadFile(stats)
151-
if err != nil {
152-
t.Fatal(err)
153-
}
154-
var sum struct {
145+
state := filepath.Join(t.TempDir(), "state.json")
146+
type summary struct {
155147
Verdict string `json:"verdict"`
156148
Reason string `json:"reason"`
157149
Targets []struct {
150+
Processed int64 `json:"processed"`
158151
Skipped int64 `json:"skipped_by_cap"`
159152
StoppedBy string `json:"stopped_by"`
160153
} `json:"targets"`
161154
}
162-
if err := json.Unmarshal(data, &sum); err != nil {
163-
t.Fatal(err)
164-
}
165-
if sum.Verdict != "degraded" || !strings.Contains(sum.Reason, "-max-duration") {
166-
t.Errorf("verdict %q (%s), want degraded naming -max-duration", sum.Verdict, sum.Reason)
155+
check := func(run string, code int, out string) summary {
156+
t.Helper()
157+
if code != 0 {
158+
t.Fatalf("%s: exit %d\n%s", run, code, out)
159+
}
160+
data, err := os.ReadFile(stats)
161+
if err != nil {
162+
t.Fatal(err)
163+
}
164+
var sum summary
165+
if err := json.Unmarshal(data, &sum); err != nil {
166+
t.Fatal(err)
167+
}
168+
if sum.Verdict != "degraded" || !strings.Contains(sum.Reason, "-max-duration") {
169+
t.Errorf("%s: verdict %q (%s), want degraded naming -max-duration", run, sum.Verdict, sum.Reason)
170+
}
171+
if len(sum.Targets) != 1 || sum.Targets[0].StoppedBy != "duration" || sum.Targets[0].Skipped == 0 || sum.Targets[0].Processed == 0 {
172+
t.Fatalf("%s: targets %+v, want one stopped by duration with names tested and names skipped", run, sum.Targets)
173+
}
174+
return sum
167175
}
168-
if len(sum.Targets) != 1 || sum.Targets[0].StoppedBy != "duration" || sum.Targets[0].Skipped == 0 {
169-
t.Errorf("targets %+v, want one stopped by duration with names skipped", sum.Targets)
176+
// -rate 10 with 30 words needs about 3 s; the budget is a third of that,
177+
// and the preflight and wildcard probes take about half of it.
178+
code, out := runCLI(t, "", append(localArgs("-progress=false", "-w", "examples/labs/words.txt", "-t", "2", "-rate", "10",
179+
"-max-duration", "1s", "-stats", stats, "-state", state, "-simulate-zone", filepath.Join("examples", "labs", "lab1-first-scan.zone")), "lab.example")...)
180+
first := check("first run", code, out)
181+
code, out = runCLI(t, "", "-resume", state, "-rate", "10", "-max-duration", "1s")
182+
if resumed := check("resumed run", code, out); resumed.Targets[0].Skipped >= first.Targets[0].Skipped {
183+
t.Errorf("resumed run skipped %d names, the first run %d: the resume point did not advance", resumed.Targets[0].Skipped, first.Targets[0].Skipped)
170184
}
171185

172186
if code, _ := runCLI(t, "", "-simulate", "-max-duration", "-1s", "example.com"); code != exitUsage {

‎internal/scan/dispatcher.go‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -448,7 +448,10 @@ func (d *dispatcher) finish(ctx context.Context, stats *counters, events chan<-
448448
return
449449
}
450450
msg := fmt.Sprintf("query cap reached ({max_queries} %d); skipped %d additional jobs", d.cfg.MaxQueries, d.skipped)
451-
if d.expired {
451+
switch {
452+
case d.expired && d.admitted == 0:
453+
msg = fmt.Sprintf("time budget reached ({max_duration}) before any name was tested; skipped %d jobs. Raise {max_duration}", d.skipped)
454+
case d.expired:
452455
msg = fmt.Sprintf("time budget reached ({max_duration}); skipped %d additional jobs", d.skipped)
453456
}
454457
select {

‎internal/scan/runner.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -657,11 +657,19 @@ func Run(ctx context.Context, cfg Config, events chan<- Event) {
657657

658658
cfg.scope = newScope(cfg.Exclude)
659659
cfg.takeover = &dns.TakeoverCache{Providers: cfg.TakeoverProviders, Skip: cfg.scope.excluded}
660+
checksStarted := time.Now()
660661
wildcardRoot, fingerprint, ok, fatal := preflight(ctx, &cfg, maxDepth, events)
661662
if !ok {
662663
failed = fatal
663664
return
664665
}
666+
if !cfg.Deadline.IsZero() && checksStarted.Before(cfg.Deadline) {
667+
// The preflight and wildcard probes do not use up the budget: under a
668+
// low -rate they can take all of a short one, and a -resume probes
669+
// again, so it would never test a name (#340). A budget already spent
670+
// when the scan started (a later -dL target) stays spent.
671+
cfg.Deadline = cfg.Deadline.Add(time.Since(checksStarted))
672+
}
665673
cfg.resumeDepth = resumeDepths(cfg)
666674

667675
st := &scanState{

‎internal/scan/runner_test.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -628,6 +628,9 @@ func TestRunDeadlineStopsAdmission(t *testing.T) {
628628
if !strings.Contains(notice, "time budget reached ({max_duration})") {
629629
t.Errorf("notice %q, want the time budget named", notice)
630630
}
631+
if untested := strings.Contains(notice, "before any name was tested"); untested != (c.wantProcessed == 0) {
632+
t.Errorf("notice %q: says no name was tested %v, want %v (#340)", notice, untested, c.wantProcessed == 0)
633+
}
631634
if v, reason := done.Stats.Verdict(false); v != VerdictDegraded || !strings.Contains(reason, "{max_duration}") {
632635
t.Errorf("verdict %s (%s), want degraded naming the time budget", v, reason)
633636
}

0 commit comments

Comments
 (0)