Skip to content

Commit 0160819

Browse files
TMHSDigitalclaude
andcommitted
feat(sources): pluggable keyless passive sources, with the Wayback Machine (#193)
Seed-then-resolve was built once, for crt.sh. internal/passive now holds a Source interface (Name, Names(ctx, domain)), a registry (Known, ParseList, New) and a second source: Wayback, the hostnames in the Wayback Machine's CDX index of archived URLs. It is one HTTPS request per target, with no API key, bounded to 200k URLs and 64 MB. ct.Client implements Source unchanged. CLI: - -sources ct,wayback picks the sources; -ct stays as shorthand for -sources ct. Unknown names are a usage error. - Every source's names are candidates and -permute seeds through the same merge path as before, each still verified by a lookup. A failed source is a warning, and the scan goes on with the others. - The report gains sources: [{name, names, error}]; ct_names and ct_error are kept. - The resume state keeps each source's names in sources{}, and state files with only CT/CTFetched still resume. - Simulation ignores -sources, lab mode refuses it, and dry-run notes name each source. Library: Config.Sources, the Source interface, CertificateTransparency(), WaybackMachine() and NoticeSource. Sources are asked in Run's goroutine before the scan, so Run still returns at once, and they are ignored with Simulate. Tests: internal/passive unit tests (parsing, Wayback hosts and errors), CLI e2e tests (ct plus wayback, a wayback outage, an unknown source, lab and simulate), and library tests with a custom and a failing source. go test -race ./... and golangci-lint pass. Fixes #193 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 76c0174 commit 0160819

23 files changed

Lines changed: 699 additions & 52 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `-sources ct,wayback` picks keyless passive sources whose names become candidates and `-permute` seeds, each still verified by a lookup. `ct` is crt.sh, as `-ct`, which stays as shorthand. `wayback` is new: the hostnames in the Wayback Machine's index of archived URLs, one HTTPS request per target. The report gains a `sources` entry per source (`names`, or the `error` it hit; `ct_names` and `ct_error` are kept), a failed source is a warning, the resume state keeps each source's names, simulation ignores `-sources` and lab mode refuses it. Sources share one interface in a new `internal/passive` package. `pkg/subenum` gains `Config.Sources`, the `Source` interface, `CertificateTransparency()`, `WaybackMachine()` and `NoticeSource` (#193).
1112
- `-axfr` tries a zone transfer (AXFR) of each target from its name servers before the scan: the NS records come from `-dns-server`, then each server (up to 8, two addresses each) is asked over TCP. A server that allows it is reported on stderr and in the run-quality report (`axfr_allowed`, `axfr_names`), and the zone's names become candidates and `-permute` seeds, each still verified by a lookup. Refusals, the correct configuration, stay quiet unless `-v`. The NS and address lookups and the transfers count in `queries_sent`; simulation ignores `-axfr`, lab mode refuses it, and a resumed scan reuses the transferred names. The test DNS server can answer AXFR (`Reply.Transfer`) (#85).
1213
- `-axfr` also sends one DNSSEC probe per target to an authoritative server: a query for a name that does not exist, with the DNSSEC OK bit set. A zone that proves the absence with plain NSEC can be walked to list every name, so it is reported as `Zone walkable` on stderr. The report's `zone_denial` says `nsec`, `nsec3` or `none` (unsigned). subenum does not walk the zone. The test DNS server can sign negative answers (`Reply.Denial`) (#234).
1314
- `pkg/subenum` exports `Diff(previous, current []Result, types []string) Changes`: the names added and missing and the names whose records changed, compared the way `-diff-records` does (as sets over the given types, CNAME targets only when an alias is involved). It sends no DNS queries, so `Removed` lists names to look up again rather than names known to be gone. The CLI and the library share the comparison in a new `internal/diff` package (#197).

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,6 +265,7 @@ make help # list all targets
265265
| `-dns-server <addr>` | `8.8.8.8:53` | DNS server: `ip:port` (UDP, TCP fallback), `tls://host[:853]` (DNS over TLS) or `https://host/path` (DNS over HTTPS); validated on startup |
266266
| `-attempts <n>` | `1` | DNS resolution attempts per subdomain (1 = no retry) |
267267
| `-axfr` | `false` | Try a zone transfer (AXFR) from each of the target's name servers first; names it gives become candidates and `-permute` seeds. Also reports a zone that DNSSEC NSEC lets anyone walk |
268+
| `-sources <list>` | n/a | Keyless passive sources whose names become candidates and `-permute` seeds: `ct` (crt.sh), `wayback` (Wayback Machine). One HTTPS request per source and target |
268269
| `-ct` | `false` | Add names from Certificate Transparency logs (crt.sh; one HTTPS request per target, no API key) as candidates and `-permute` seeds |
269270
| `-force` | `false` | Continue scanning even if wildcard DNS is detected |
270271
| `-no-abort` | `false` | Keep scanning after the 20% resolver failure-rate abort (warning is still emitted) |
@@ -448,6 +449,14 @@ subenum -ct -permute example.com
448449

449450
The run-quality report counts them per target (`ct_names`). If crt.sh is down or slow, the scan warns, records `ct_error` and carries on with the wordlist. The CT request goes to crt.sh, not to the target and not through `-rate`; lab mode (`-simulate-zone`) refuses `-ct`, and a resumed scan reuses the names the interrupted run fetched.
450451

452+
`-ct` is one of several keyless passive sources, chosen with `-sources`: `ct` (as `-ct`) and `wayback`, the hostnames in the Wayback Machine's index of archived URLs, which often remembers hosts long gone from DNS and certificates. Each is one HTTPS request per target, and each name is still looked up.
453+
454+
```bash
455+
subenum -sources ct,wayback example.com
456+
```
457+
458+
The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
459+
451460
### Zone transfers: the whole zone in one query
452461

453462
A name server that allows anyone a zone transfer (AXFR) lists every name in the zone, and finding one is a misconfiguration worth reporting in itself. `-axfr` looks up the target's name servers through `-dns-server` and asks each of them for a transfer over TCP before the scan. Most refuse, which is correct and stays quiet (`-v` shows each refusal). When one allows it, subenum says so on stderr, names the server in the report (`axfr_allowed`, with `axfr_names`), and adds the names as candidates after the wordlist and as `-permute` seeds. Like `-ct` names they are only candidates: each is still looked up, so a stale entry is never reported.
@@ -688,6 +697,7 @@ subenum/
688697
│ ├── dnstest/ # Runs dnsserver for hermetic tests
689698
│ ├── labzone/ # Lab scenario file parser
690699
│ ├── output/ # Thread-safe Writer, results file, SARIF and summary writers
700+
│ ├── passive/ # -sources: keyless passive name sources (Wayback Machine; crt.sh is in ct/)
691701
│ ├── permute/ # -permute candidate generation
692702
│ ├── report/ # Run-quality report built from finished scans
693703
│ ├── scan/ # Scan engine: Config, Event types, Run()

‎docs/ARCHITECTURE.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ Every package under `internal/` and `pkg/`, summarized from its `// Package` com
3737
| `internal/diff` | Compares two runs' results: new and missing names, changed records (shared by `-diff` and `pkg/subenum`'s `Diff`) |
3838
| `internal/cliargs` | Parses command lines whose flags may follow positional arguments, shared by subenum and wordlist-gen |
3939
| `internal/ct` | Looks up subdomain names in Certificate Transparency logs through crt.sh |
40+
| `internal/passive` | The `Source` interface behind `-sources`, the source registry, and the Wayback Machine source |
4041
| `internal/labzone` | Parses `-simulate-zone` scenario files for lab mode |
4142
| `internal/dnsserver` | Scriptable local DNS server backing lab mode and the hermetic tests |
4243
| `internal/dnstest` | Runs `internal/dnsserver` for tests: stopped on cleanup, listen failures fail or skip the test |
@@ -50,7 +51,7 @@ main.go - CLI entry point: run, flag parsing and validati
5051
main_session.go - runWith and the session: per-run state, the -dL target loop, -tui dispatch
5152
main_completion.go - completion and man subcommands
5253
main_config.go - config file and SUBENUM_* environment defaults, -print-config
53-
main_ct.go - -ct: Certificate Transparency names as extra candidates
54+
main_ct.go - -ct / -sources: passive source names as extra candidates
5455
main_axfr.go - -axfr: zone transfer names as extra candidates
5556
main_diff.go - -diff / -diff-records: compare against a previous results file
5657
main_dryrun.go - -dry-run: print the scan plan and its cost without sending queries
@@ -87,6 +88,9 @@ internal/report/report.go - New, Target, Finish: the run-quality report
8788
internal/permute/permute.go - Generate: permutations of found names
8889
internal/diff/diff.go - Name, RecordsChanged: the comparison behind -diff-records and subenum.Diff
8990
internal/ct/ct.go - Client.Names: crt.sh lookup
91+
internal/passive/passive.go - Source, Known, ParseList, New: the -sources registry
92+
internal/passive/wayback.go - Wayback.Names: hostnames from the Wayback Machine's CDX index
93+
pkg/subenum/sources.go - Source, CertificateTransparency, WaybackMachine: Config.Sources
9094
internal/labzone/labzone.go - Load, Parse: lab scenario files
9195
internal/dnsserver/server.go - Listen: scriptable loopback DNS server
9296
internal/dnstest/dnstest.go - Start, StartWith, StartTLS: test helpers around dnsserver

‎docs/DEVELOPER_GUIDE.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ subenum/
131131
│ │ ├── eta.go # Progress ETA estimate
132132
│ │ ├── log.go # -log-format: text or JSON diagnostics on stderr
133133
│ │ └── *_test.go # Writer, file, ETA tests
134+
│ ├── passive/ # -sources: Source interface, registry, Wayback Machine source
134135
│ ├── permute/ # -permute candidate generation
135136
│ ├── report/ # Run-quality report built from finished scans
136137
│ ├── scan/
@@ -171,7 +172,7 @@ subenum/
171172
├── main_session.go # runWith and the per-run session: the -dL target loop
172173
├── main_completion.go # completion and man subcommands
173174
├── main_config.go # Config file and SUBENUM_* environment defaults, -print-config
174-
├── main_ct.go # -ct: Certificate Transparency names as candidates
175+
├── main_ct.go # -ct / -sources: passive source names as candidates
175176
├── main_axfr.go # -axfr: zone transfer names as candidates
176177
├── main_diff.go # -diff / -diff-records against a previous results file
177178
├── main_dryrun.go # -dry-run: the scan plan and its cost, no queries sent

‎docs/_data/unreleased.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ flags:
1313
- max-duration
1414
- rate-per-resolver
1515
- retry-inconclusive
16+
- sources
1617
- takeover-providers
1718
- takeover-providers-only
1819
# action.yml inputs and outputs
@@ -27,5 +28,6 @@ library:
2728
- MaxDuration
2829
- Permutations
2930
- RatePerResolver
31+
- Sources
3032
- RetryInconclusive
3133
- TakeoverProviders

‎docs/_includes/flags.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
| `-silent` | - | Pipeline mode: bare result names on stdout, and only errors (and a simulation warning) on stderr |
4141
| `-simulate` | - | Run in simulation mode without actual DNS queries (for testing) |
4242
| `-simulate-zone <string>` | - | Lab mode: answer every query from this scenario file via a local DNS server, so no traffic leaves the machine (see the Labs page) |
43+
| `-sources <string>` {% include unreleased.html flag="sources" %} | - | Comma-separated keyless passive sources whose names become candidates and -permute seeds: ct (Certificate Transparency, crt.sh), wayback (Wayback Machine); one HTTPS request per source and target |
4344
| `-state <string>` | subenum-resume.json | Where an interrupted run saves its state for -resume |
4445
| `-stats <string>` | - | Write a JSON run-quality report (outcomes, queries sent, verdict) to this file |
4546
| `-t <int>` | 100 | Number of concurrent workers |

‎docs/library.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ for _, c := range changes.Changed {
108108
| `AutoRate`{% include unreleased.html library="AutoRate" %} | false | When over a fifth of the last 200 lookups fail, halve the rate and keep going (a `NoticeRate` event says so), and raise it again once lookups succeed. It halves up to six times and not below 5 per second; only when it can slow down no further does the reliability guard apply. `Stats.RateBackoffs` and `Stats.FinalRate` report it. |
109109
| `TakeoverProviders`{% include unreleased.html library="TakeoverProviders" %} | built-in list | Extra takeover-prone services (`{Suffix: ".example-cdn.net", Name: "example-cdn"}`) for `provider:` hints, added to the built-in list; a `Name` of `-` removes a suffix. `TakeoverProvidersOnly` uses them alone. |
110110
| `RetryInconclusive`{% include unreleased.html library="RetryInconclusive" %} | false | Once every candidate is tried, look up each name that timed out, was refused or failed once more against `Resolver`. Names it resolves are reported but not expanded by `Recursive`. `Stats.Retried` and `Stats.Recovered` count the pass; `Stats.Inconclusive` lists the names still without an answer. |
111+
| `Sources`{% include unreleased.html library="Sources" %} | none | Passive sources whose names are added after `Words` and looked up like them: `CertificateTransparency()`, `WaybackMachine()`, or your own `Source` (one `Names(ctx, domain)` method). A source that fails sends a `NoticeSource` event and the scan goes on. Ignored with `Simulate`. |
111112
| `Exclude` | none | Out-of-scope names and `*.parent` branches, never queried. |
112113
| `Force` | off | Scan a wildcard zone, dropping answers that match its fingerprint. |
113114
| `NoAbort` | off | Keep going when more than a fifth of lookups fail. `Run` still sends the warning as a `KindError` event; `Scan` returns no error for a scan that finished. |

‎internal/ct/ct.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ type Client struct {
3434
Timeout time.Duration // per lookup; 0 means 60s
3535
}
3636

37+
// Name is the source's -sources name.
38+
func (Client) Name() string { return "ct" }
39+
3740
type entry struct {
3841
NameValue string `json:"name_value"`
3942
}

‎internal/output/summary.go‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,10 @@ type TargetSummary struct {
8989
// candidates and seeds; CTError says why the lookup failed (#130).
9090
CTNames int `json:"ct_names,omitempty"`
9191
CTError string `json:"ct_error,omitempty"`
92+
// Sources has one entry per passive source asked (-ct, -sources): the
93+
// names it added as candidates and seeds, or why its lookup failed
94+
// (#193). ct_names and ct_error repeat the ct entry.
95+
Sources []SourceSummary `json:"sources,omitempty"`
9296
// AXFRAllowed lists the name servers that allowed an -axfr zone
9397
// transfer (a misconfiguration), AXFRNames counts the names it added as
9498
// candidates and seeds, and AXFRError says why no transfer could be
@@ -109,6 +113,13 @@ type TargetSummary struct {
109113
Reason string `json:"reason"`
110114
}
111115

116+
// SourceSummary is one passive source's contribution to a target (#193).
117+
type SourceSummary struct {
118+
Name string `json:"name"` // -sources name: ct, wayback
119+
Names int `json:"names"` // names it added as candidates and seeds
120+
Error string `json:"error,omitempty"` // why the lookup failed; the scan went on without it
121+
}
122+
112123
// PermutationSummary counts a target's -permute pass.
113124
type PermutationSummary struct {
114125
Seeds int `json:"seeds"`

‎internal/passive/passive.go‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
// Package passive names the keyless passive sources whose names seed a scan
2+
// (#193): Certificate Transparency through crt.sh, and the Wayback Machine's
3+
// URL index. A source only proposes candidates; the scan still resolves each
4+
// one, so a stale name costs a query, never a false result.
5+
package passive
6+
7+
import (
8+
"context"
9+
"fmt"
10+
"slices"
11+
"strings"
12+
13+
"github.com/TMHSDigital/subenum/internal/ct"
14+
)
15+
16+
// Source finds names under a domain without querying the domain's DNS.
17+
type Source interface {
18+
// Name is the source's short name, as -sources takes it.
19+
Name() string
20+
// Names returns names under domain relative to it ("api", "dev.api"),
21+
// sorted and deduplicated.
22+
Names(ctx context.Context, domain string) ([]string, error)
23+
}
24+
25+
// Known lists the sources -sources accepts, in the order they run.
26+
var Known = []string{"ct", "wayback"}
27+
28+
// Labels are human-readable names for messages.
29+
var Labels = map[string]string{"ct": "Certificate Transparency", "wayback": "Wayback Machine"}
30+
31+
// Options configure the built-in sources.
32+
type Options struct {
33+
// UserAgent identifies the client to each service.
34+
UserAgent string
35+
// BaseURLs replace a source's endpoint, by source name; tests point them
36+
// at local servers.
37+
BaseURLs map[string]string
38+
}
39+
40+
// ParseList parses a comma-separated -sources value: known names only,
41+
// lowercase, deduplicated, in Known order. Empty is no sources.
42+
func ParseList(s string) ([]string, error) {
43+
want := map[string]bool{}
44+
for _, part := range strings.Split(s, ",") {
45+
name := strings.ToLower(strings.TrimSpace(part))
46+
if name == "" {
47+
continue
48+
}
49+
if !slices.Contains(Known, name) {
50+
return nil, fmt.Errorf("unknown source %q (want %s)", part, strings.Join(Known, ", "))
51+
}
52+
want[name] = true
53+
}
54+
var out []string
55+
for _, name := range Known {
56+
if want[name] {
57+
out = append(out, name)
58+
}
59+
}
60+
return out, nil
61+
}
62+
63+
// New returns the built-in source called name.
64+
func New(name string, opts Options) (Source, error) {
65+
switch name {
66+
case "ct":
67+
return ct.Client{BaseURL: opts.BaseURLs["ct"], UserAgent: opts.UserAgent}, nil
68+
case "wayback":
69+
return Wayback{BaseURL: opts.BaseURLs["wayback"], UserAgent: opts.UserAgent}, nil
70+
}
71+
return nil, fmt.Errorf("unknown source %q", name)
72+
}

0 commit comments

Comments
 (0)