Skip to content

Commit 0507cbe

Browse files
TMHSDigitalclaude
andcommitted
Flag and retry lookups with one failed record type (#357)
Resolve returned a plain Found as soon as any requested type answered, dropping the error of a type that timed out, was refused or failed. The failure was neither retried nor counted, so -diff-records could report a name as changed while the run's verdict stayed complete. Lookup.ResolvePartial now retries a partial answer like a failed one, up to Attempts, and returns the records flagged partial when the failed type never answers. The scan marks such results (Result.Partial, a "partial" JSON field, PARTIAL in text with records), counts them in Stats.Partial, lowers the verdict as failed lookups do, and -diff-records no longer reports a partial name as changed. Fixes #357 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 1bddf7d commit 0507cbe

18 files changed

Lines changed: 209 additions & 36 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6868
- Less memory on large wordlists and busy scans: `-sources`, `-ct`, `-axfr` and `-permute` check their names against the wordlist without building a set as large as it (about 200 MB per feature and target on a 5M-line list), a `-simulate-zone` lab scan no longer keeps a count per queried name (about 170 MB on a 1M-line list), and a `CNAME` lookup reads a UDP answer into a buffer sized for the 1232-byte EDNS payload it advertises instead of 64 KB. Results are unchanged (#321).
6969

7070
### Fixed
71+
- A name whose lookup of one requested type timed out, was refused or failed while another type answered is no longer a plain found result with the records that came back. The whole lookup is tried again up to `-attempts`, and when the failed type never answers the result is marked partial: a `partial` field in JSON and JSONL, `PARTIAL` in text with `-show-records`, a `partial answers` line in the breakdown, and `Stats.Partial` in the library. Partial answers lower the run's verdict as failed lookups do (degraded at 1%), and `-diff-records` does not report a partial name as changed, since its missing records may only be a lookup that failed (#357).
7172
- Resuming a scan stopped in its `-permute` pass (Ctrl+C or `-max-duration`) no longer scans the finished main pass again. Under `-recursive` that repeated every lookup below the saved names, and a name the repeat found anew changed the permutation candidate list, so the resumed pass started at the wrong place in a different list, skipping some candidates and testing others twice. The resume state now records that the main pass finished, and the resumed run goes straight to the permutation pass with the saved names as its seeds (#358).
7273
- When `-max-queries` or `-max-duration` stops a scan whose wordlist holds multi-label entries (such as `dev.api`), the untested names are counted as they would have been: ones `-exclude` would remove count as excluded rather than skipped, so a run whose only untested names were out of scope stays complete, and under `-recursive` a name the rest of the scan would have reached through two parents is counted as skipped once, so tested, skipped and excluded add up to the candidate total (#361).
7374
- A CNAME target or name server that exists but has no A or AAAA record (NODATA, such as a name with only MX or TXT records) is no longer marked `dangling` or `dangling-ns`: Go reports it as not-found like a name that does not exist, so a target whose address lookup comes back empty now gets one more A query, and only an NXDOMAIN answer marks it. A probe that times out or fails marks nothing (#362).

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -634,6 +634,8 @@ A public name that answers with an internal address leaks how the network behind
634634

635635
The scope is an `internal_address` field in JSON and JSONL, `INTERNAL=<scope>` in text with `-show-records`, and a count in the breakdown and the run-quality report (`internal_addresses`). CSV is unchanged: each row already holds one address in its `value` column. Simulated results are never marked, since their addresses are invented.
636636

637+
A name that answered for one `-type` while another type's lookup still timed out, was refused or failed after `-attempts` is reported with the records that came back and marked partial: a `"partial": true` field in JSON and JSONL, `PARTIAL` in text with `-show-records`, and a `partial answers` count in the breakdown. Partial answers lower the run's verdict as failed lookups do, and `-diff-records` never reports a partial name as changed.
638+
637639
### Monitoring for new subdomains
638640

639641
`-diff previous.jsonl` reports only what changed since an earlier run: `+ name` for new subdomains and `- name` for ones that disappeared (only when the run's verdict is `complete`, and only after a fresh lookup of the name returns NXDOMAIN, so names this run did not test and failed lookups are never mistaken for removals). The `-o` file keeps the full current results for the next comparison, and the exit code is `4` when anything changed. The repository is also a GitHub Action (`uses: TMHSDigital/subenum@<release>`) that installs a verified release, keeps the baseline as an artifact, writes a job summary and fails on a bad verdict; [docs/monitoring.md](docs/monitoring.md) documents it and a hand-written scheduled workflow that opens an issue when new names appear.

‎docs/monitoring.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ subenum -w wordlist.txt -format jsonl -type A,AAAA,CNAME -diff previous.jsonl -d
6969
every TTL and would otherwise flag every CDN host.
7070
- The previous file must carry records: JSON, JSONL or CSV. A text file has
7171
none, so the comparison falls back to names with a note.
72+
- A name whose lookup of one type still failed after `-attempts` while another
73+
type answered is marked `"partial": true` and is never reported as changed:
74+
its missing records may only be a lookup that timed out. Partial answers
75+
lower the run's verdict as failed lookups do.
7276

7377
## The subenum GitHub Action
7478

‎internal/cli/cli.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -655,6 +655,9 @@ func logScanBreakdown(domain string, ev scan.Event, out *output.Writer) {
655655
if s.Internal > 0 {
656656
out.Info(" internal addresses: %d (public names answering with loopback, private, link-local or CGNAT addresses)", s.Internal)
657657
}
658+
if s.Partial > 0 {
659+
out.Info(" partial answers: %d (found names a requested record type got no answer for; records may be missing)", s.Partial)
660+
}
658661
if s.RateBackoffs > 0 {
659662
out.Info(" rate back-offs: %d (ended at %d queries/s)", s.RateBackoffs, s.FinalRate)
660663
}

‎internal/cli/diff.go‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -265,8 +265,10 @@ func compareBaselines(prev, cur baseline, records bool, types []string) baseline
265265
}
266266

267267
// observe records a found name and says how it differs from the baseline:
268-
// "added", "changed" (with the baseline's records), or "" for unchanged.
269-
func (d *differ) observe(name string, records []dns.Record) (change string, previous []dns.Record) {
268+
// "added", "changed" (with the baseline's records), or "" for unchanged. A
269+
// partial answer, where a requested type got no answer (#357), is never
270+
// "changed": its missing records say nothing about the name.
271+
func (d *differ) observe(name string, records []dns.Record, partial bool) (change string, previous []dns.Record) {
270272
d.mu.Lock()
271273
defer d.mu.Unlock()
272274
name = strings.ToLower(name)
@@ -277,7 +279,7 @@ func (d *differ) observe(name string, records []dns.Record) (change string, prev
277279
}
278280
// d.types is the run's record types, never empty, so only the types
279281
// this run looked up are compared.
280-
if prev, ok := d.records[name]; ok && diff.RecordsChanged(prev, records, d.types) {
282+
if prev, ok := d.records[name]; ok && !partial && diff.RecordsChanged(prev, records, d.types) {
281283
d.changed++
282284
return "changed", prev
283285
}

‎internal/cli/lab_test.go‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,10 @@ import (
1111
"strings"
1212
"testing"
1313

14+
"golang.org/x/net/dns/dnsmessage"
15+
1416
"github.com/TMHSDigital/subenum/internal/dns"
17+
"github.com/TMHSDigital/subenum/internal/dnstest"
1518
)
1619

1720
// labTimingArgs gives lab runs, and any other run against an in-process DNS
@@ -299,6 +302,52 @@ func TestE2EDiffRecords(t *testing.T) {
299302
}
300303
}
301304

305+
// TestE2EDiffRecordsPartial covers #357: a name whose A lookup answers while
306+
// its AAAA lookup times out on every attempt is flagged partial, lowers the
307+
// verdict, and is not reported as changed by -diff-records.
308+
func TestE2EDiffRecordsPartial(t *testing.T) {
309+
t.Parallel()
310+
srv := dnstest.Start(t, func(q dnstest.Query) dnstest.Reply {
311+
if !strings.EqualFold(strings.TrimSuffix(q.Name, "."), "www.example.com") {
312+
return dnstest.NXDomain
313+
}
314+
if q.Type == dnsmessage.TypeAAAA {
315+
return dnstest.Dropped
316+
}
317+
return dnstest.Reply{A: []string{"192.0.2.10"}}
318+
})
319+
baseline := writeFile(t, "previous.jsonl",
320+
`{"subdomain":"www.example.com","records":[{"type":"A","value":"192.0.2.10"},{"type":"AAAA","value":"2001:db8::10"}]}`+"\n")
321+
args := append([]string{"-progress=false", "-format", "jsonl", "-dns-server", srv.Addr, "-type", "A,AAAA"}, labSlowTimingArgs...)
322+
code, out := runCLI(t, "", append(args, "-w", writeFile(t, "wl.txt", "www\n"),
323+
"-diff", baseline, "-diff-records", "example.com")...)
324+
if code != exitOK {
325+
t.Fatalf("exit %d, want %d (a partial answer is not a change)\n%s", code, exitOK, out)
326+
}
327+
var summary struct {
328+
Verdict string `json:"verdict"`
329+
Diff *struct {
330+
Changed int `json:"changed"`
331+
} `json:"diff"`
332+
}
333+
for _, raw := range strings.Split(strings.TrimSpace(out), "\n") {
334+
if strings.Contains(raw, `"change"`) {
335+
t.Errorf("unexpected diff line: %s", raw)
336+
}
337+
if strings.Contains(raw, `"type":"summary"`) {
338+
if err := json.Unmarshal([]byte(raw), &summary); err != nil {
339+
t.Fatalf("%v: %s", err, raw)
340+
}
341+
}
342+
}
343+
if summary.Verdict != "degraded" {
344+
t.Errorf("verdict %q, want degraded\n%s", summary.Verdict, out)
345+
}
346+
if summary.Diff == nil || summary.Diff.Changed != 0 {
347+
t.Errorf("summary diff %+v, want changed 0", summary.Diff)
348+
}
349+
}
350+
302351
// TestE2ELabTakeoverHint checks lab 1's dangling CNAME: the external target
303352
// does not exist in the lab, so the result is flagged.
304353
func TestE2ELabTakeoverHint(t *testing.T) {

‎internal/cli/results.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ func (l *resultLog) add(r output.Result) {
3232
if len(l.names) == 0 {
3333
l.simulated = r.Simulated
3434
}
35-
if r.Takeover != "" || r.Internal != "" || r.Permutation || r.Simulated != l.simulated {
35+
if r.Takeover != "" || r.Internal != "" || r.Partial || r.Permutation || r.Simulated != l.simulated {
3636
if l.extra == nil {
3737
l.extra = map[int]output.Result{}
3838
}

‎internal/cli/resume.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -448,6 +448,7 @@ func mergeStats(a, b scan.Stats) scan.Stats {
448448
a.Unconfirmed += b.Unconfirmed
449449
a.Takeover += b.Takeover
450450
a.Internal += b.Internal
451+
a.Partial += b.Partial
451452
a.Retried += b.Retried
452453
a.Recovered += b.Recovered
453454
a.Inconclusive = append(append([]string(nil), a.Inconclusive...), b.Inconclusive...)

‎internal/cli/resume_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1234,7 +1234,7 @@ func TestResumeStateKeys(t *testing.T) {
12341234
targets.results.simulated targets.results.sources targets.results.subdomain targets.results.takeover_candidate`)
12351235
// scan.Stats is stored twice, for the target and for its -permute pass.
12361236
statsKeys := strings.Fields(`Aborted Confirmed Excluded Expired FinalRate Fingerprint Fingerprint.type
1237-
Fingerprint.value Found Inconclusive Internal NXDomain Other PoolHits QueriesSent RateBackoffs
1237+
Fingerprint.value Found Inconclusive Internal NXDomain Other Partial PoolHits QueriesSent RateBackoffs
12381238
Recovered Refused Retried RootDone Skipped Takeover Timeout Unconfirmed Wildcard WildcardFiltered`)
12391239
for _, prefix := range []string{"targets.stats.", "targets.permute.stats."} {
12401240
for _, k := range statsKeys {

‎internal/cli/session.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -603,7 +603,7 @@ func (s *session) emitResult(ev scan.Event, permutation bool) {
603603
s.out.Emit(res)
604604
return
605605
}
606-
if change, previous := s.diff.observe(res.Subdomain, res.Records); change != "" {
606+
if change, previous := s.diff.observe(res.Subdomain, res.Records, res.Partial); change != "" {
607607
res.Change, res.PreviousRecords = change, previous
608608
s.out.Emit(res)
609609
}

0 commit comments

Comments
 (0)