Skip to content

Commit 1bddf7d

Browse files
TMHSDigitalclaude
andcommitted
Skip a finished main pass when resuming a -permute pass (#358)
A target stopped in its -permute pass was saved as not done, so the resumed run scanned its main pass again from the end. Under -recursive that re-expanded every saved parent, and a name the repeat found anew changed the permutation candidate list the saved position indexes. The resume state now records main_done for such a target; the resumed run takes the saved main pass as its final event, as it does for a finished target, and goes straight to the permutation pass with the saved names as its seeds. The saved main pass is kept whole, without the repeat discount a re-scanned pass gets. Fixes #358 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 87db011 commit 1bddf7d

5 files changed

Lines changed: 113 additions & 8 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6868
- Less memory on large wordlists and busy scans: `-sources`, `-ct`, `-axfr` and `-permute` check their names against the wordlist without building a set as large as it (about 200 MB per feature and target on a 5M-line list), a `-simulate-zone` lab scan no longer keeps a count per queried name (about 170 MB on a 1M-line list), and a `CNAME` lookup reads a UDP answer into a buffer sized for the 1232-byte EDNS payload it advertises instead of 64 KB. Results are unchanged (#321).
6969

7070
### Fixed
71+
- Resuming a scan stopped in its `-permute` pass (Ctrl+C or `-max-duration`) no longer scans the finished main pass again. Under `-recursive` that repeated every lookup below the saved names, and a name the repeat found anew changed the permutation candidate list, so the resumed pass started at the wrong place in a different list, skipping some candidates and testing others twice. The resume state now records that the main pass finished, and the resumed run goes straight to the permutation pass with the saved names as its seeds (#358).
7172
- When `-max-queries` or `-max-duration` stops a scan whose wordlist holds multi-label entries (such as `dev.api`), the untested names are counted as they would have been: ones `-exclude` would remove count as excluded rather than skipped, so a run whose only untested names were out of scope stays complete, and under `-recursive` a name the rest of the scan would have reached through two parents is counted as skipped once, so tested, skipped and excluded add up to the candidate total (#361).
7273
- A CNAME target or name server that exists but has no A or AAAA record (NODATA, such as a name with only MX or TXT records) is no longer marked `dangling` or `dangling-ns`: Go reports it as not-found like a name that does not exist, so a target whose address lookup comes back empty now gets one more A query, and only an NXDOMAIN answer marks it. A probe that times out or fails marks nothing (#362).
7374
- A wordlist entry that already ends in the target, such as a full name pasted from another tool, is scanned as the prefix in front of it: `www.example.com` under `example.com` is `www`, not `www.example.com.example.com`, and one that repeats an entry already in the list counts as a duplicate. The scan notes how many entries it shortened (#351).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -408,7 +408,7 @@ The only flags `-resume` accepts next to the state file are the pacing flags `-a
408408

409409
<br>
410410

411-
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned. A resumed run that finishes deletes its state file. A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves the same state (its exit code does not change), so `-resume` continues it too: a scheduled job with a time budget can work through a large wordlist over several runs. When `-diff` and `-o` name the same file, a run that saves a resume state leaves that file unchanged, so the resumed run compares against the same baseline and then writes the full results. Runs that read the wordlist or domain list from stdin cannot be resumed.
411+
The resumed run refuses to start if an input file, the wordlist (including the bundled one, which changes with the binary) or an effective setting (for example a `SUBENUM_*` variable or the config file) changed. It re-emits the earlier results (so `-o` and JSON output end up complete), skips targets that had finished, and continues the interrupted one from the first wordlist entry not yet fully looked up; a `-permute` pass resumes from its own position. At most about one worker pool's worth (`-t`) of lookups is repeated, and `-max-queries` covers the whole scan, so a resumed run only gets what is left of it. In recursive mode, names found before the interrupt have their children re-scanned; a run stopped in its `-permute` pass resumes in that pass and does not scan the finished main pass again. A resumed run that finishes deletes its state file. A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves the same state (its exit code does not change), so `-resume` continues it too: a scheduled job with a time budget can work through a large wordlist over several runs. When `-diff` and `-o` name the same file, a run that saves a resume state leaves that file unchanged, so the resumed run compares against the same baseline and then writes the full results. Runs that read the wordlist or domain list from stdin cannot be resumed.
412412

413413
</details>
414414

‎internal/cli/resume.go‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,9 @@ type resumeState struct {
4444

4545
type resumeTarget struct {
4646
Domain string `json:"domain"`
47-
Done bool `json:"done"` // fully scanned before the interrupt
48-
RootDone int `json:"root_done"` // wordlist entries settled, for the interrupted target
47+
Done bool `json:"done"` // fully scanned before the interrupt
48+
RootDone int `json:"root_done"` // wordlist entries settled, for the interrupted target
49+
MainDone bool `json:"main_done,omitempty"` // only the -permute pass was left; not scanned again (#358)
4950
Stats scan.Stats `json:"stats"`
5051
Processed int64 `json:"processed"`
5152
Total int64 `json:"total"`
@@ -542,17 +543,20 @@ func saveResumeState(s *session) bool {
542543
// completed before the interrupt, with nothing left by -max-duration.
543544
t.Done = s.status[i] == report.StatusOK && res.done && !res.unfinished() &&
544545
(res.permutation == nil || res.permutation.candidates == 0 || (res.permutation.pass.done && !res.permutation.pass.unfinished()))
546+
// The -permute pass starts only once the main pass completed, so a
547+
// target with one left resumes in it, without the main pass (#358).
548+
t.MainDone = !t.Done && ((res.done && !res.unfinished() && res.permutation != nil) || (saved != nil && saved.MainDone))
545549
switch {
546550
case res.done:
547551
final := res.final
548-
if !t.Done {
552+
if !t.Done && !t.MainDone {
549553
// The resumed run scans the main pass again from RootDone,
550554
// and re-expands every saved parent (#341).
551555
final = withoutRepeat(final)
552556
}
553557
t.Stats, t.Processed, t.Total = final.Stats, final.Processed, final.Total
554558
t.RootDone = int(final.Stats.RootDone)
555-
if !t.Done {
559+
if !t.Done && !t.MainDone {
556560
t.Pending = savePending(final.Stats.Pending)
557561
}
558562
case saved != nil:

‎internal/cli/resume_test.go‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1015,6 +1015,94 @@ func TestSaveResumeStateRecordsFinishedPermutePass(t *testing.T) {
10151015
}
10161016
}
10171017

1018+
// TestSaveResumeStateMarksFinishedMainPass covers #358: a target stopped in
1019+
// its -permute pass is saved with its main pass finished, counted in full,
1020+
// since the resumed run does not scan it again.
1021+
func TestSaveResumeStateMarksFinishedMainPass(t *testing.T) {
1022+
main := scan.Event{Kind: scan.EventDone, Stats: scan.Stats{RootDone: 40, NXDomain: 30, Repeat: scan.Repeat{NXDomain: 10, Processed: 10}},
1023+
Processed: 40, Total: 40}
1024+
f := cliFlags{stateFile: filepath.Join(t.TempDir(), "state.json")}
1025+
saveResumeState(&session{
1026+
f: f, out: output.New(nil, false, output.FormatText),
1027+
targets: []string{"a.example"},
1028+
status: []report.Status{report.StatusInterrupted},
1029+
entryDigests: []string{"d1"},
1030+
results: []targetResult{{done: true, final: main, permutation: &permutationResult{candidates: 12}}},
1031+
runs: []resultLog{{}},
1032+
})
1033+
data, err := os.ReadFile(f.stateFile)
1034+
if err != nil {
1035+
t.Fatal(err)
1036+
}
1037+
var st resumeState
1038+
if err := json.Unmarshal(data, &st); err != nil {
1039+
t.Fatal(err)
1040+
}
1041+
got := st.Targets[0]
1042+
if got.Done || !got.MainDone {
1043+
t.Errorf("done %v, main_done %v; want only the main pass finished", got.Done, got.MainDone)
1044+
}
1045+
if got.Processed != 40 || got.Stats.NXDomain != 30 || got.RootDone != 40 {
1046+
t.Errorf("main pass saved as %d processed, %d NXDOMAIN, root_done %d; want it whole (40, 30, 40)", got.Processed, got.Stats.NXDomain, got.RootDone)
1047+
}
1048+
}
1049+
1050+
// TestE2EResumePermuteSkipsFinishedMainPass covers #358: resuming a
1051+
// -recursive -permute scan stopped in its permutation pass does not scan the
1052+
// finished main pass again, which would re-expand every saved parent, and
1053+
// resumes in the same candidate list.
1054+
func TestE2EResumePermuteSkipsFinishedMainPass(t *testing.T) {
1055+
t.Parallel()
1056+
words := "api\ndev\nwww\nmail\nshop\nblog\n"
1057+
wl := writeFile(t, "wl.txt", words)
1058+
args := []string{"-simulate", "-seed", "5", "-hit-rate", "50", "-progress=false", "-format", "jsonl",
1059+
"-permute", "-recursive", "-depth", "2", "-w", wl, "example.com"}
1060+
code, ref := runCLI(t, "", args...)
1061+
if code != 0 {
1062+
t.Fatalf("reference run: exit %d", code)
1063+
}
1064+
refResults, _ := jsonlRecords(t, ref)
1065+
// Leave one depth-2 name out of the saved main pass: re-expanding its
1066+
// saved parent would find it again.
1067+
var mainResults []output.Result
1068+
child := ""
1069+
for _, r := range refResults {
1070+
if r.Permutation {
1071+
continue
1072+
}
1073+
if child == "" && strings.Count(strings.TrimSuffix(r.Subdomain, ".example.com"), ".") == 1 {
1074+
child = r.Subdomain
1075+
continue
1076+
}
1077+
mainResults = append(mainResults, r)
1078+
}
1079+
if child == "" {
1080+
t.Fatal("reference run found no depth-2 name; adjust the seed")
1081+
}
1082+
entries, _, _ := wordlist.Build(strings.Split(words, "\n"), "example.com")
1083+
perms := candidatesFor("example.com", entries, mainPassNames(mainResults))
1084+
if len(perms) < 2 {
1085+
t.Fatalf("%d permutation candidates; adjust the seed", len(perms))
1086+
}
1087+
half := len(perms) / 2
1088+
stopped := resumeTarget{Domain: "example.com", MainDone: true, RootDone: len(entries), Processed: int64(len(entries)),
1089+
Results: mainResults, Permute: &resumePass{RootDone: half, Processed: int64(half)}}
1090+
path := writeState(t, resumeState{Args: args, Targets: []resumeTarget{stopped}})
1091+
code, stdout, stderr := runCLIStreams(t, "-resume", path)
1092+
if code != 0 {
1093+
t.Fatalf("resume: exit %d\n%s", code, stderr)
1094+
}
1095+
if strings.Contains(stderr, "at wordlist entry") {
1096+
t.Errorf("the finished main pass was scanned again:\n%s", stderr)
1097+
}
1098+
if want := fmt.Sprintf("at candidate %d of %d", half, len(perms)); !strings.Contains(stderr, want) {
1099+
t.Errorf("permutation pass not resumed %q:\n%s", want, stderr)
1100+
}
1101+
if got, _ := jsonlRecords(t, stdout); resultNames(got)[child] {
1102+
t.Errorf("%s was looked up again by the main pass", child)
1103+
}
1104+
}
1105+
10181106
// TestE2EPermuteRecursive covers #111: -permute with -recursive runs the
10191107
// permutation pass one level deep, so no name is built from a candidate
10201108
// under another candidate and the pass is not refused.

‎internal/cli/session.go‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -720,10 +720,22 @@ func (s *session) scanOneTarget(ctx context.Context, i int, target string) (next
720720
s.status[i] = report.StatusOK
721721
return true
722722
}
723-
pass.resumeFrom, pass.resumeNames = saved.RootDone, s.targetResults.mainPassNames()
724-
out.Info("Resuming %s at wordlist entry %d of %d (%d results restored)", target, saved.RootDone, len(entries), len(saved.Results))
723+
if saved.MainDone {
724+
out.Info("%s finished its main pass before the run stopped; %d results restored", target, len(saved.Results))
725+
} else {
726+
pass.resumeFrom, pass.resumeNames = saved.RootDone, s.targetResults.mainPassNames()
727+
out.Info("Resuming %s at wordlist entry %d of %d (%d results restored)", target, saved.RootDone, len(entries), len(saved.Results))
728+
}
729+
}
730+
var res targetResult
731+
if saved != nil && saved.MainDone {
732+
// Scanning it again would re-expand every saved parent under
733+
// -recursive, and a name it found anew would change the permutation
734+
// candidates the saved position indexes (#358).
735+
res = targetResult{done: true, final: mainSaved.event()}
736+
} else {
737+
res = s.mainPass(ctx, pass, mainSaved)
725738
}
726-
res := s.mainPass(ctx, pass, mainSaved)
727739
// -permute: a second pass over permutations of what was found (#72),
728740
// skipped after an interrupt, a resolver abort or a spent time budget. Its seeds are the
729741
// main pass's names only, so a resumed pass rebuilds the same

0 commit comments

Comments
 (0)