Skip to content

Commit c39dc0e

Browse files
TMHSDigitalclaude
andcommitted
feat: embed a 5k default wordlist; -w is now optional
A first scan with the old 26-49 entry lists found almost nothing. -w is now optional: without it subenum uses data/subdomains-5k.txt, the 5,000 most common subdomain labels from SecLists' subdomains-top1million-5000.txt (MIT; the full license notice is kept in the file header), embedded with go:embed, and says so on stderr. The Dockerfile copies it into the build context. README: Quick Start works with just a domain, and a new "Choosing a wordlist" section covers SecLists, Assetnote, n0kovo and trickest and the trade-off between list size, query volume and -rate. Fixes #63 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 84b9a2f commit c39dc0e

9 files changed

Lines changed: 5102 additions & 19 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- A useful default wordlist: `-w` is now optional. Without it, subenum uses an embedded list of the 5,000 most common subdomain labels (SecLists `subdomains-top1million-5000.txt`, MIT; the license notice is kept in `data/subdomains-5k.txt`) and says so on stderr. The README gains a "Choosing a wordlist" section on larger lists and their query cost (#63).
1112
- Monitoring mode: `-diff previous` compares with an earlier results file (any format) and prints only names added (`+` / `"change": "added"`) or removed (`-` / `"removed"`), with a `change` CSV column; exit code `4` signals changes. The `-o` file keeps the full current results for the next run. Removals are reported only when the run's verdict is `complete`, and never for `-exclude`d names. The run-quality report gains a `diff` object, and `docs/monitoring.md` has a scheduled GitHub Actions workflow that opens an issue for new names (#84).
1213
- Subdomain takeover hints: results with a CNAME are marked `dangling` (target is NXDOMAIN), `provider:<name>` (target at a takeover-prone service such as S3, GitHub Pages, Heroku or Azure) or `dangling:<name>`. The marker appears in JSON/JSONL (`takeover_candidate`), as a CSV column when CNAME records are requested, in text with `-show-records`, in the TUI, and as a count in the breakdown and report. DNS-only; no HTTP probing (#71).
1314
- Resolver pool: `-r resolvers.txt` spreads lookups over many resolvers, round-robin over the healthy ones (a resolver failing over half its recent lookups is benched for 30s), with retries sent to a different resolver. Every pool hit is re-validated against `-dns-server`, whose answer is reported, so a lying pool resolver cannot inject results. Per-resolver accounting appears in a stderr table and in the run-quality report, with `pool_hits` / `confirmed` / `unconfirmed` per target. In a hermetic test, 20 rate-limited resolvers ran the same scan 14x faster than one (#69).

‎Dockerfile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ RUN go mod download
2222
# main_summary.go, ...); _test.go files are ignored by go build.
2323
COPY *.go ./
2424
COPY internal/ ./internal/
25+
# The default wordlist is embedded in the binary (#63).
26+
COPY data/subdomains-5k.txt ./data/
2527

2628
# Build the binary with optimizations
2729
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -o subenum -ldflags="-w -s -X main.Version=${VERSION}" .

‎README.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,11 @@
4242

4343
```bash
4444
go install github.com/TMHSDigital/subenum@latest
45+
subenum yourdomain.com # a domain you own or are authorized to test
4546
```
4647

48+
With no `-w`, subenum uses its built-in list of the 5,000 most common subdomain labels (from SecLists, MIT-licensed), so a first scan finds real results without hunting for a wordlist. See [Choosing a wordlist](#choosing-a-wordlist) for larger lists.
49+
4750
Or build from a clone, which also gives you the sample wordlists used below:
4851

4952
```bash
@@ -234,7 +237,7 @@ make help # list all targets
234237

235238
| Flag | Default | Description |
236239
| :--- | :---: | :--- |
237-
| `-w <file>` | n/a | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin **(required)** |
240+
| `-w <file>` | bundled top-5000 | Wordlist file, one prefix per line; `#` comments allowed; `-` reads stdin. Omit it to use the built-in list (see [Choosing a wordlist](#choosing-a-wordlist)) |
238241
| `-dL <file>` | n/a | File of apex domains to scan, one per line (`-` reads stdin), instead of a `<domain>` argument. Each domain is an independent scan; results share one output |
239242
| `-t <n>` | `100` | Concurrent worker goroutines |
240243
| `-timeout <ms>` | `1000` | DNS timeout in milliseconds, applied to each record-type lookup separately |
@@ -338,6 +341,19 @@ Press `Ctrl+C` at any time to abort. In-flight queries drain, partial results ar
338341

339342
With `-dL`, each domain is scanned in turn as an independent scan: `-max-queries`, `-rate` and the reliability guard apply per target. A failed target does not stop the others, but a reliability abort (the resolver looks overloaded) skips the remaining targets unless `-no-abort` is set. A per-target status list is printed at the end.
340343

344+
### Choosing a wordlist
345+
346+
Results can only be as good as the wordlist. Without `-w`, subenum uses its bundled top-5000 list (`data/subdomains-5k.txt`, from SecLists' `subdomains-top1million-5000.txt`), a good first pass that takes seconds. For deeper coverage:
347+
348+
| List | Size | Notes |
349+
|------|------|-------|
350+
| [SecLists `Discovery/DNS`](https://github.com/danielmiessler/SecLists/tree/master/Discovery/DNS) | 5k to 1M+ | `subdomains-top1million-20000.txt` and `-110000.txt` are the usual next steps |
351+
| [Assetnote wordlists](https://wordlists.assetnote.io/) (`best-dns-wordlist.txt`) | ~9M | Built from real DNS data; for long, thorough scans |
352+
| [n0kovo subdomains](https://github.com/n0kovo/n0kovo_subdomains) | 50k to 3M | Tiered lists from scraped certificates |
353+
| [trickest wordlists](https://github.com/trickest/wordlists) | varies | Also maintains a public resolver list for `-r` |
354+
355+
List size sets the query volume: each entry costs one query per record type (two with the default `A,AAAA`), per attempt, per recursion level. A 1M-entry list at `-rate 500` is about 70 minutes of wire time per target, so pair big lists with `-rate` (and `-r` for a resolver pool), and use `-max-queries` as a budget. `tools/wordlist-gen` builds a small target-specific list from a domain's own terms.
356+
341357
### Monitoring for new subdomains
342358

343359
`-diff previous.jsonl` reports only what changed since an earlier run: `+ name` for new subdomains and `- name` for ones that disappeared (only when the run's verdict is `complete`, so failed lookups are never mistaken for removals). The `-o` file keeps the full current results for the next comparison, and the exit code is `4` when anything changed. [docs/monitoring.md](docs/monitoring.md) has a ready-to-use scheduled GitHub Actions workflow that opens an issue when new names appear.

0 commit comments

Comments
 (0)