Skip to content

Commit dd0e2a5

Browse files
committed
Count held jobs when the deadline follows the query cap (#317)
With both -max-queries and -max-duration, a deadline that fired after the cap returned from expire before accounting for the chunk it held, so those jobs were neither tested nor skipped and tested plus skipped fell short of the candidate count; the run was also not marked expired. expire now moves held jobs to skipped whether or not the cap came first, and the scan is marked expired whenever the deadline skipped admitted work. A capped expand of a parent whose whole subtree is excluded now counts its children as excluded, not skipped, as the budget stop already does for queued expansions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Fixes #317
1 parent 77ceb5b commit dd0e2a5

3 files changed

Lines changed: 63 additions & 7 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6363
- The names whose lookup timed out, was refused or failed are kept only when something reads them: for the `-inconclusive` file, and until the `-retry-inconclusive` pass has run. A scan without either only counts them, so under `-no-abort` or `-auto-rate` against a failing resolver memory no longer grows with every failed lookup (about 1 GB at 10M failures). When they are kept, a wordlist entry costs its index rather than its name. Counts, the report and `pkg/subenum`'s `Stats.Inconclusive` are unchanged; the `-resume` state lists the names only with `-inconclusive` (#300).
6464

6565
### Fixed
66+
- With both `-max-queries` and `-max-duration`, a deadline that fires after the query cap no longer loses the chunk of names admitted but not yet handed out: they count as skipped, so tested plus skipped adds up to every candidate and progress reaches 100%, and the run is marked as stopped by the time budget. Children of a name found after a budget stop whose whole branch is excluded count as excluded, not skipped, so they cannot make a complete scan look incomplete (#317).
6667
- A `-r` resolver list line with a malformed or out-of-range port (`9.9.9.9:abc`, `:99999`, `:0`) is rejected when the file is read, naming the line, instead of failing every lookup at dial time; an explicit `-attempts 0` (flag, `SUBENUM_ATTEMPTS` or config) is rejected like any value below 1 instead of running as 1, and `-attempts 0 -retries 3` reports that both cannot be used (#318, thanks @bfs2021).
6768
- Takeover hints no longer look up a CNAME target or name server that `-exclude` puts out of scope, so excluded names are never queried on this path either. Such a target can still be marked `provider:<name>`, which needs no query, but never `dangling` or `dangling-ns` (#316).
6869
- The TUI configuration form fits a terminal shorter than its 31 lines, such as the standard 80×24: the logo is dropped, then the field rows scroll to keep the focused field in view, with `↑ N more fields` and `↓ N more fields` lines for the rest. The validation error and the key hints are always shown, so a rejected ctrl+r no longer fails without a visible reason (#287).

‎internal/scan/dispatcher.go‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -361,25 +361,32 @@ func (d *dispatcher) expand(parent job) {
361361
return
362362
}
363363
switch {
364-
case d.capped:
365-
d.skipped += d.n
364+
// An excluded subtree is excluded whether or not a budget has run out,
365+
// as countUnbuilt counts it, so it cannot lower the verdict (#238).
366366
case d.cfg.scope.subtreeExcluded(parent.domain):
367367
d.excluded += d.n
368+
case d.capped:
369+
d.skipped += d.n
368370
default:
369371
d.frontier = append(d.frontier, expansion{parent: parent.domain, depth: parent.depth + 1})
370372
d.remaining += d.n
371373
}
372374
}
373375

374376
// expire applies the deadline when its timer fires (#190). held is how many
375-
// jobs were admitted but not yet handed to a worker; they are skipped.
377+
// jobs were admitted but not yet handed to a worker; they are skipped, even
378+
// when -max-queries stopped admission first, so tested plus skipped still
379+
// adds up to every candidate (#317).
376380
func (d *dispatcher) expire(held int) {
381+
d.admitted -= held
382+
d.skipped += held
383+
if held > 0 {
384+
d.expired = true
385+
}
377386
if d.capped {
378387
return
379388
}
380389
d.expired = true
381-
d.admitted -= held
382-
d.skipped += held
383390
d.stopAdmitting()
384391
}
385392

@@ -407,11 +414,12 @@ func (d *dispatcher) nextChunk(workers int) []job {
407414
func (d *dispatcher) complete(c completion) {
408415
d.pending--
409416
if c.skipped {
410-
// Taken after the deadline and never looked up.
417+
// Taken after the deadline and never looked up: the time budget
418+
// skipped it, even when -max-queries had already stopped admission.
411419
d.skipped++
412420
d.admitted--
421+
d.expired = true
413422
if !d.capped {
414-
d.expired = true
415423
d.stopAdmitting()
416424
}
417425
}

‎internal/scan/dispatcher_test.go‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -309,3 +309,50 @@ func TestDispatcherDeadline(t *testing.T) {
309309
t.Errorf("expired %v skipped %d pending %d watermark %d, want true 3 0 0", d.expired, d.skipped, d.pending, d.watermark)
310310
}
311311
}
312+
313+
// TestDispatcherCapThenDeadline covers #317: when -max-queries stops
314+
// admission and the deadline then fires with a chunk still held, the held
315+
// jobs are skipped, so tested plus skipped is every candidate, and the scan
316+
// is marked expired because the time budget skipped admitted work.
317+
func TestDispatcherCapThenDeadline(t *testing.T) {
318+
d := testDispatcher(Config{Entries: makeEntries(10), MaxQueries: 4}, 1)
319+
held := drain(d) // the cap admits 4 and skips the other 6
320+
if len(held) != 4 || !d.capped || d.skipped != 6 {
321+
t.Fatalf("admitted %d capped %v skipped %d, want 4 true 6", len(held), d.capped, d.skipped)
322+
}
323+
d.expire(len(held))
324+
if d.admitted != 0 || d.skipped != 10 || !d.expired || d.total() != 0 {
325+
t.Errorf("admitted %d skipped %d expired %v total %d, want 0 10 true 0", d.admitted, d.skipped, d.expired, d.total())
326+
}
327+
328+
// A deadline with nothing held after the cap changes nothing.
329+
d = testDispatcher(Config{Entries: makeEntries(10), MaxQueries: 4}, 1)
330+
drain(d)
331+
d.expire(0)
332+
if d.expired || d.skipped != 6 {
333+
t.Errorf("empty expire after the cap: expired %v skipped %d, want false 6", d.expired, d.skipped)
334+
}
335+
336+
// A worker skips a job after the cap: the time budget skipped it.
337+
d = testDispatcher(Config{Entries: makeEntries(10), MaxQueries: 4}, 1)
338+
drain(d)
339+
d.pending = 1
340+
d.complete(completion{root: 0, skipped: true})
341+
if !d.expired || d.skipped != 7 || d.admitted != 3 {
342+
t.Errorf("skipped job after the cap: expired %v skipped %d admitted %d, want true 7 3", d.expired, d.skipped, d.admitted)
343+
}
344+
}
345+
346+
// A parent found after a budget stop whose whole subtree is excluded has its
347+
// children counted as excluded, not skipped, as countUnbuilt does, so they
348+
// cannot make a complete scan look incomplete (#238).
349+
func TestCappedExpandOfExcludedSubtree(t *testing.T) {
350+
d := testDispatcher(Config{Entries: makeEntries(5), MaxQueries: 2, Recursive: true,
351+
Exclude: []string{"*.p0.example.com"}}, 3)
352+
drain(d)
353+
skipped := d.skipped
354+
d.expand(job{domain: "p0.example.com", depth: 1})
355+
if d.excluded != 5 || d.skipped != skipped {
356+
t.Errorf("excluded %d skipped %d, want the 5 children excluded and skipped unchanged at %d", d.excluded, d.skipped, skipped)
357+
}
358+
}

0 commit comments

Comments
 (0)