You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 80acb01
Browse filesBrowse the repository at this point in the historyBrowse files
Keep only collidable names in the recursive visited set (#359)
One multi-label entry, from the wordlist or a nested CT, Wayback or AXFR
name, turned the recursive visited set back on for every candidate. Two
(entry, parent) pairs can only produce the same name when a multi-label
entry is a prefix of it, and each parent is already expanded once, so
the set now records only those names. Both orders of reaching such a
name are still deduplicated, in next() and in stopAdmitting().
BenchmarkDispatcherRecursiveDotted (100,000 entries plus "dev.api",
root and ten found parents, 1.1M candidates):
before: 1,100,011 visited, 134.8 MB/op, ~470 ms/op
after: 11 visited, 26.5 MB/op, ~125 ms/op
Fixes#359
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,7 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
46
46
-`pkg/subenum` exports `ProbeZone(ctx, domain, ZoneOptions)`, which tries a zone transfer (AXFR) from the domain's name servers with `-axfr`'s limits (8 servers, 2 addresses each) and reports the names, the servers that allowed it and whether the zone uses plain NSEC (`ZoneInfo`), and `ZoneTransfer(ZoneOptions)`, a `Source` for `Config.Sources` that scans the transferred names. Its queries are not paced by `Rate`, as `-axfr`'s are not. The CLI and the library share the attempt in a new `internal/axfr` package (#314).
47
47
48
48
### Changed
49
-
- A `-recursive` scan over a wordlist of single labels no longer remembers every candidate it generated (about 75 MB per million), and one that reaches `-max-queries` or `-max-duration` counts the untested candidates without building each name, so it stops on time. A scan whose candidates include multi-label names (a wordlist entry with a dot, or names from `-sources`) still keeps that record, since only there can two parents reach the same name (#301).
49
+
- A `-recursive` scan over a wordlist of single labels no longer remembers every candidate it generated (about 75 MB per million), and one that reaches `-max-queries` or `-max-duration` counts the untested candidates without building each name, so it stops on time (#301).
50
+
- A `-recursive` scan whose candidates include a multi-label name (a wordlist entry with a dot, or a nested name from `-sources` or `-axfr`) no longer remembers every candidate it generated either: it records only the names a multi-label entry begins, the only ones two parents can reach, so one nested name from CT no longer brings the per-candidate record back. In a benchmark of 100,000 entries plus one with a dot and ten found parents, the record drops from 1.1 million names to 11 and the scan allocates 26 MB instead of 135 MB (#359).
50
51
- A run that `-max-duration` stopped with names untested, or that the reliability guard aborted, saves its resume state to the `-state` file like an interrupted one and prints `continue with: subenum -resume <file>`, so a time-boxed job continues where the last one stopped instead of testing the same first names again. The exit code is unchanged (0 for a spent budget). A resumed run that stops the same way saves its state again, and one that finishes no longer counts the names an earlier part skipped, so it can end `complete` (#294).
51
52
- The command's source moved out of the repository root: the `main_*.go` files are now `internal/cli/*.go`, and the root `main.go` only calls `cli.Main`. `go install github.com/TMHSDigital/subenum@latest`, the Action, the Docker image and `-X main.Version` builds are unchanged. The lab scenarios are embedded through a small `examples/labs` package, and `SECURITY.md` is now `.github/SECURITY.md`.
52
53
- A `CNAME` lookup sends one CNAME query instead of going through Go's `LookupCNAME`, which also sent A and AAAA queries and threw their answers away. `-type A,AAAA,CNAME` now costs three queries a name instead of five, so the same `-rate` tests two-thirds more names per second, and `-dry-run`, `-max-queries` help and the `-rate-per-resolver` reservation count it that way. The query goes through the same transports (UDP with TCP fallback, DoT, DoH), rate limiter and query counter as the other types (#249).
Copy file name to clipboardExpand all lines: docs/ARCHITECTURE.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -186,7 +186,7 @@ tools/licenses/main.go - THIRD_PARTY_LICENSES for release archives and t
186
186
***`scan.Options` / `scan.Config`**: `Options` holds the user-facing settings: it embeds the `scan.Config` the engine runs on, plus `MaxDuration` and `RatePerResolver`, which are only checked there (#331). Its `Validate` method checks ranges and the resolver address, and its `Config` field is what `scan.Run` takes. The CLI, the TUI and `pkg/subenum` each fill in an `Options`, so they cannot drift apart in what they accept, and each has a test that walks `Options.Fields()` and fails when a setting is left unset without a listed reason. `scan.DefaultConcurrency`, `DefaultTimeoutMs` and `DefaultHitRate` are the defaults all three use.
187
187
***`scan.Event` / `scan.EventKind`**: Typed events emitted on the channel: `EventResult` (a `scan.Result`: subdomain, records, simulated flag, takeover hint and internal address scope), `EventProgress`, `EventNotice`, `EventError`, and `EventDone`. An `EventNotice` carries a `scan.NoticeKind` (`NoticeWildcard`, `NoticeCap`, `NoticeCeiling`, `NoticeSkip`, `NoticeRate`) so consumers can style or filter notices without parsing the message. `EventDone` carries a `scan.Stats` snapshot with per-outcome counters (found, nxdomain, timeout, refused, other, wildcard-filtered).
188
188
***Preflight and wildcard check**: Before any candidate is dispatched, `scan.Run` resolves the apex (with retries) and runs `Lookup.FingerprintWildcard`; see the Data Flow section.
189
-
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), the set of parents already queued (recursive scans only), and a pending-work counter. A set of generated candidates is kept only for a recursive scan whose entries include a multi-label one, the only case where two parents can reach the same name. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.
189
+
* **Dispatcher and lazy candidate generation**: A dispatcher goroutine owns the internal `jobs` channel, which carries chunks of jobs rather than single jobs: `nextChunk` hands each send up to one worker's share of what is left, at most `maxChunk` (8), so a large scan pays one channel operation per chunk and the tail still spreads over every worker. `jobs` and `completed` are buffered (`handoffPerWorker` = 4 sends per worker), so workers and the dispatcher rarely wait on each other. The dispatcher also keeps a queue of *expansions* (a parent name plus a cursor into `Config.Entries`), the set of parents already queued (recursive scans only), and a pending-work counter. A set of generated candidates is kept only for a recursive scan whose entries include a multi-label one, the only case where two parents can reach the same name, and it holds only the names a multi-label entry begins (`a.b.c` for the entry `a.b`), since no other name can be reached twice. Candidate names are built only as they are dispatched, so a found parent costs one queue entry rather than one per wordlist entry, and the queue is compacted as it is consumed. `-max-queries` stops admission once the cap is reached; the remaining candidates are counted, never built, and reported in a `NoticeCap` notice.
190
190
***Recursion ceiling**: When `-recursive` is set, `scan.RecursionCeiling` computes the theoretical job count (`sum n^d`). Above 1e7 the scan refuses to start unless `-max-queries` or `-force` is set; otherwise a `NoticeCeiling` warning is emitted.
191
191
***Worker goroutines**: The pool size is `Config.Concurrency`, capped at the most jobs the scan can produce. Each worker takes a chunk and, for every job in it (which already holds the full domain), waits on the rate limiter if the scan is simulated, resolves it, classifies the outcome, and checks the result against the wildcard fingerprint.
192
192
***Near-miss revalidation**: A result whose records are all in the fingerprint is a wildcard answer and is counted as wildcard-filtered. A result that shares *some* records with it is a near miss, typical of a rotating wildcard pool: two fresh random labels under the same parent are probed, their answers join the fingerprint, and the result is checked again. Results sharing nothing with the fingerprint cost no extra queries.
0 commit comments