Skip to content

Commit 1cb4477

Browse files
committed
Let -o repeat, each file with its own format (#334)
One run could write only one results file in one format, so a pipeline that wanted plain names and JSONL had to scan twice or convert. -o can be given more than once, and a file can name its format with a prefix (jsonl:results.jsonl, csv:, text:); without one it uses -format. Each file gets the full results and its own summary, is written atomically, and a failure to open a later file discards the earlier ones. Stdout keeps its own writer. With -diff -o X on a stopped run, only X is left untouched. -tui takes the first -o. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Fixes #334
1 parent daf571a commit 1cb4477

13 files changed

Lines changed: 303 additions & 83 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `-o` can be given more than once, and a format prefix gives a file its own format: `-o names.txt -o jsonl:results.jsonl` writes bare names for the next tool and JSONL with the run-quality report in one run, where a pipeline used to scan twice or extract names with `jq`. A file without a prefix uses `-format`, as before. Every file gets the full current results, also under `-diff`; on a run stopped with state for `-resume`, only an `-o` file that is also the `-diff` file is left unchanged. The same path given twice is a usage error, and `-tui` takes the first file (#334).
1112
- The TUI's finished screen can save the results of a scan run without an Output file: `s` asks for a path (starting from the Output file, or `subenum-<domain>` with the Format's extension), and the file is written as the Output file would have been, in the format its extension names (`.txt`, `.json`, `.jsonl`, `.csv`, otherwise the form's Format). With a filter on, only the results it shows are saved. `v` toggles each result's records under it, during or after the scan, and a text file saved while they are shown carries them, as with `-show-records`. Results past the view's 10,000 are not kept and cannot be saved (#332).
1213
- `subenum mcp` offers a read-only `diff` tool that compares two result sets without sending a DNS query: `previous` and `current` each take a `scan` or `lab_scan` report (or its `results` array) or the contents of a results file in any format, and an optional `types` limits the record types compared. It returns `added`, `missing` and `changed` names with their records, as `structuredContent` too, compared as `subenum diff -records` does, so CDN address rotation behind an unchanged CNAME is not a change; missing names are marked unverified. Its input is bounded by the server's 16 MiB message limit, as `explain_stats`'s is (#333).
1314
- `subenum diff [-records] [-type A,AAAA,CNAME] [-format text|json|jsonl|csv] <previous> <current>` compares two saved results files, in any format subenum writes, without sending a DNS query: `+` for names only in the second file, `-` for names only in the first, and with `-records` `~` for names whose records differ, compared as `-diff-records` does. In JSON, JSONL and CSV the `change` is `added`, `missing` or `changed`; a name is `missing` rather than `removed` because nothing was looked up to show it is gone. It exits `4` when the files differ, and is in the shell completions and the man page (#310).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -276,7 +276,7 @@ This table describes `main`, which can be ahead of the latest release: a release
276276
| `-max-duration <d>` | `0` | Time budget for the whole run, e.g. `20m` (0 = unlimited). Then no new names are tested, lookups already under way finish, and the run ends normally with its results, report (`stopped_by: "duration"`) and a `degraded` verdict, and saves a `-state` file so `-resume` tests the rest. Under `-dL` it covers all targets |
277277
| `-retry-inconclusive` | `false` | After each scan, look up once more (against `-dns-server`, never the `-r` pool) every name that timed out, was refused or failed. An answer replaces the first outcome in the counts; a name that resolves is reported, but not expanded by `-recursive`. Skipped after an abort, an interrupt or once `-max-duration` runs out. The report counts `retried` and `recovered` |
278278
| `-inconclusive <file>` | n/a | Write the names still without an answer (timeout, refused, other) to this file, one per line, sorted, so a later run can retry them with `-w <file>` |
279-
| `-o <file>` | n/a | Write results to file in addition to stdout |
279+
| `-o <file>` | n/a | Write results to file in addition to stdout; repeat for several files, each in `-format` or in the format named as a prefix (`-o names.txt -o jsonl:results.jsonl`) |
280280
| `-format <fmt>` | `text` | Output format: `text`, `json`, `jsonl` (one object per line, streamed), or `csv` |
281281
| `-log-format <fmt>` | `text` | Diagnostics on stderr: `text`, or `json` for one JSON object per line (`time`, `level`, `kind`, `target`, `message`, and `settings` codes on engine notices). Progress is then an event, only with an explicit `-progress`. Results on stdout are unchanged |
282282
| `-show-records` | `false` | In `text` format, append each result's records as `TYPE=value` |

‎docs/ARCHITECTURE.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -240,11 +240,11 @@ The flow of data through the `subenum` application can be summarized as follows:
240240
5. **Wildcard detection**: `Lookup.FingerprintWildcard` probes random labels under the target (skipped in simulation mode). If the check is inconclusive the scan stops, unless `-force` is set, in which case it scans without filtering and says so in a `NoticeWildcard` notice. If a wildcard is detected, a `NoticeWildcard` notice is emitted and the scan stops unless `-force` is set; with `-force`, later results matching the fingerprint are filtered.
241241
6. **Dispatch**: The dispatcher generates `prefix.target` jobs lazily from the root expansion (a recursive scan expands each found parent once) and feeds them to the buffered `jobs` channel in chunks of up to eight. It does not close `jobs` until no candidates remain and the pending counter drains to zero, or the context is cancelled.
242242
7. **Resolution**: Worker goroutines read jobs and call `Lookup.Resolve` (or `dns.Simulation.Resolve`) for the requested record types; every wire query waits on the rate limiter.
243-
8. **Result Emission**: A found result that is not a wildcard answer increments the found counter and is emitted as an `EventResult` with its typed records. Every completed lookup, success or failure, increments the matching `Stats` counter. The CLI `Writer` routes results to stdout and any `-o` file in the selected `-format`; the TUI renders them in its viewport. After `EventDone` the CLI prints the per-outcome breakdown to stderr and the TUI summary line shows the same counters under the run-quality verdict and its reason.
243+
8. **Result Emission**: A found result that is not a wildcard answer increments the found counter and is emitted as an `EventResult` with its typed records. Every completed lookup, success or failure, increments the matching `Stats` counter. The CLI routes results to its stdout `Writer` in the selected `-format` and to one file-only `Writer` per `-o` file, each in its own format (`parseOutputs`, #334); the TUI renders them in its viewport. After `EventDone` the CLI prints the per-outcome breakdown to stderr and the TUI summary line shows the same counters under the run-quality verdict and its reason.
244244
9. **Recursive Expansion** (optional): when `-recursive` is set and a resolved, non-wildcard job is below the `-depth` cap, the worker hands it back to the dispatcher over the `enqueue` channel, which adds an expansion for its children and grows the progress total.
245245
10. **Progress Tracking**: Workers update atomic counters; the ticker goroutine emits `EventProgress` once per second.
246246
11. **Termination**: After finishing a chunk, a worker reports the completion of every job in it in one send on the buffered `completed` channel; the dispatcher uses these reports for its pending count and the resume watermark, so a resumed run repeats at most one chunk per worker. When no work remains the dispatcher closes `jobs`, the workers exit, and `scan.Run` waits on the `sync.WaitGroup`, runs the `-retry-inconclusive` pass if set, stops the progress ticker, emits `EventDone`, and closes the events channel. On `SIGINT`/`SIGTERM` the context is cancelled, the dispatcher closes `jobs` early, and the same drain-and-finish path runs with partial counts.
247-
12. **Finalization**: If at least one scan finished, the CLI calls `Writer.Finish` and commits the `-o` file, then exits with the code described in 4.5.
247+
12. **Finalization**: If at least one scan finished, the CLI calls `Writer.Finish` and commits the `-o` files, then exits with the code described in 4.5.
248248

249249
Visually, this can be seen as:
250250

@@ -297,7 +297,7 @@ With `-dL`, a reliability-guard abort means the resolver is overloaded, so the r
297297

298298
### 4.6. Output File Support
299299

300-
When the `-o` flag is provided, results are written to the specified file in the selected `-format` in addition to stdout. The file is replaced atomically (see 2.5), and a mutex in the `Writer` protects concurrent writes to both stdout and the file.
300+
When the `-o` flag is provided, results are written to the specified file in addition to stdout, in the selected `-format` or in the format a `format:` prefix names. `-o` is repeatable (`outputSpecs`, #334): each file gets its own `output.NewFile` writer and the full current results, also under `-diff`, where stdout carries only the changes. Each file is replaced atomically (see 2.5), and a mutex in each `Writer` protects its concurrent writes.
301301

302302
### 4.7. Retry Mechanism
303303

‎docs/_data/unreleased.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ values:
2525
- type=NS
2626
- type=MX
2727
- type=TXT
28+
- o=format:path
2829
# action.yml inputs and outputs
2930
action:
3031
- max-duration

‎docs/_includes/flags.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
| `-max-duration <duration>` {% include unreleased.html flag="max-duration" %} | 0s | Time budget for the whole run, e.g. 20m (0 = unlimited); then no new names are tested, lookups in flight finish, and the run ends with its report, a degraded verdict and saved state for -resume to continue from |
2323
| `-max-queries <int>` | - | Max candidate names to test (0 = unlimited); each name sends one query per record type, per attempt |
2424
| `-no-abort` | - | Do not abort when the resolver failure rate exceeds 20% (warning is still emitted) |
25-
| `-o <string>` | - | Write results to file (in addition to stdout) |
25+
| `-o <file>` | - | Write results to file (in addition to stdout); repeat for several files, each in -format or in the format named as a prefix, format:path (jsonl:results.jsonl) format:path: {% include unreleased.html value="o=format:path" %} |
2626
| `-permute` | - | After each scan, scan permutations of the names found (api -&gt; api-dev, dev-api, dev.api, api2, ...) |
2727
| `-print-config` | - | Print every setting's effective value and its source (flag, env, config or default), then exit |
2828
| `-progress` | true | Show progress during scanning |

‎docs/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,4 +59,4 @@ Settings you repeat on every run can live in the environment or a config file. T
5959

6060
## Output streams
6161

62-
Resolved names go to stdout and nothing else does, so `subenum example.com | httpx` works as expected. Progress, warnings and the per-outcome breakdown go to stderr. `-o` writes the same results to a file in the chosen `-format`; results files are replaced atomically.
62+
Resolved names go to stdout and nothing else does, so `subenum example.com | httpx` works as expected. Progress, warnings and the per-outcome breakdown go to stderr. `-o` writes the same results to a file in the chosen `-format`; results files are replaced atomically. Repeat `-o` to write several files; a file without a prefix uses `-format`, and a prefix names its own format: `-o text:names.txt -o jsonl:results.jsonl`{% include unreleased.html value="o=format:path" %} writes bare names and JSONL in one run, whatever `-format` stdout uses.

‎docs/monitoring.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ subenum -w wordlist.txt -format jsonl -diff previous.jsonl -o current.jsonl exam
2525
(removed); JSON and JSONL results carry `"change": "added"` or
2626
`"change": "removed"`; CSV gains a `change` column.
2727
- The `-o` file still receives the **full** current results, so it can be the
28-
next run's `-diff` input.
28+
next run's `-diff` input. With several `-o` files, each one does, in its own
29+
format. If a stopped run saves state for `-resume`, an `-o` file that is also
30+
the `-diff` file is left unchanged; the others get the partial results.
2931
- The exit code is `4` when anything was added or removed, `0` when nothing
3032
changed. A failed scan still exits `1`, `2` or `3`.
3133
- Names are only reported as removed when the run's

‎docs/pipelines.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,12 +49,15 @@ httpx -silent -l names.txt
4949
```
5050

5151
With `-format jsonl`, the same report is the last line of the output, so one
52-
file holds both:
52+
file holds both. `-o` can be given more than once, and a format prefix
53+
(`jsonl:results.jsonl`){% include unreleased.html value="o=format:path" %}
54+
gives a file its own format, so one run saves bare names for the next tool and
55+
JSONL with the report (and a future `-diff` baseline):
5356

5457
```bash
55-
subenum -silent -format jsonl -o results.jsonl example.com > /dev/null
58+
subenum -silent -o names.txt -o jsonl:results.jsonl example.com > /dev/null
5659
tail -n 1 results.jsonl | jq -e '.verdict == "complete"' > /dev/null || exit 1
57-
jq -r 'select(.type != "summary") | .subdomain' results.jsonl | httpx -silent
60+
httpx -silent -l names.txt
5861
```
5962

6063
## Takeover candidates first

‎internal/cli/cli.go‎

Lines changed: 90 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -23,14 +23,15 @@
2323
package cli
2424

2525
import (
26-
"bufio"
2726
"context"
2827
"errors"
2928
"flag"
3029
"fmt"
3130
"io"
3231
"os"
32+
"path/filepath"
3333
"runtime/debug"
34+
"slices"
3435
"strings"
3536
"time"
3637

@@ -130,7 +131,7 @@ type cliFlags struct {
130131
testHitRate int
131132
seed uint64
132133
simZone string // -simulate-zone: lab scenario served by a local DNS server (#76)
133-
outputFile string
134+
outputs outputSpecs
134135
attempts int
135136
retries int
136137
force bool
@@ -229,7 +230,7 @@ func parseFlagsWith(args []string, applyDefaults func(*flag.FlagSet) error, stde
229230
fs.IntVar(&f.testHitRate, "hit-rate", scan.DefaultHitRate, "In simulation mode, percentage of names that resolve (1-100)")
230231
fs.Uint64Var(&f.seed, "seed", 0, "In simulation mode, seed for reproducible results (0 = random; the seed used is printed)")
231232
fs.StringVar(&f.simZone, "simulate-zone", "", "Lab mode: answer every query from this scenario file via a local DNS server, so no traffic leaves the machine (see the Labs page)")
232-
fs.StringVar(&f.outputFile, "o", "", "Write results to file (in addition to stdout)")
233+
fs.Var(&f.outputs, "o", "Write results to `file` (in addition to stdout); repeat for several files, each in -format or in the format named as a prefix, format:path (jsonl:results.jsonl)")
233234
fs.IntVar(&f.attempts, "attempts", 0, "Total DNS resolution attempts per subdomain, 1 = no retry (default 1)")
234235
fs.IntVar(&f.retries, "retries", 0, "Deprecated: use -attempts instead")
235236
fs.BoolVar(&f.force, "force", false, "Continue scanning even if wildcard DNS is detected")
@@ -492,16 +493,88 @@ func loadTargets(f cliFlags, in *runtimeInputs, domain string, out *output.Write
492493
return targets, true
493494
}
494495

495-
func openOutputFile(path string, testMode bool, format output.Format, out *output.Writer) (*output.Writer, *output.File, bool) {
496-
if path == "" {
497-
return out, nil, true
496+
// outputSpecs is the repeatable -o flag (#334). Each -o adds a results file,
497+
// written in -format or in the format its "format:" prefix names, so one run
498+
// can save bare names for the next tool and JSONL for the next -diff.
499+
//
500+
// A config-file or SUBENUM_O value is a default: it is replaced, not added
501+
// to, by a later source or by the first -o on the command line.
502+
type outputSpecs struct {
503+
specs []string
504+
replace bool // the current specs are a default; the next Set replaces them
505+
}
506+
507+
func (o *outputSpecs) String() string {
508+
if o == nil {
509+
return ""
498510
}
499-
f, err := output.CreateFile(path)
500-
if err != nil {
501-
out.Error("creating output file: %v", err)
502-
return out, nil, false
511+
return strings.Join(o.specs, ",")
512+
}
513+
514+
// Set adds a file. An empty value adds nothing, so -o "" still means none.
515+
func (o *outputSpecs) Set(v string) error {
516+
if o.replace {
517+
o.specs, o.replace = nil, false
518+
}
519+
if v != "" {
520+
o.specs = append(o.specs, v)
521+
}
522+
return nil
523+
}
524+
525+
// markDefault records that the current specs came from the config file or
526+
// the environment rather than the command line.
527+
func (o *outputSpecs) markDefault() { o.replace = true }
528+
529+
// describe names the -o files for notices: the paths, comma-separated.
530+
func (o *outputSpecs) describe() string {
531+
paths := make([]string, len(o.specs))
532+
for i, spec := range o.specs {
533+
_, paths[i] = splitOutputSpec(spec)
534+
}
535+
return strings.Join(paths, ", ")
536+
}
537+
538+
// splitOutputSpec splits an -o value into its format prefix, "" for none,
539+
// and its path. Only a format name counts as a prefix, so a Windows drive
540+
// letter (C:\results.txt) is part of the path.
541+
func splitOutputSpec(spec string) (format, path string) {
542+
prefix, rest, ok := strings.Cut(spec, ":")
543+
if ok && rest != "" && slices.Contains(append(output.Formats(), "ndjson"), strings.ToLower(prefix)) {
544+
return prefix, rest
545+
}
546+
return "", spec
547+
}
548+
549+
// outputTarget is one -o file and the format it is written in.
550+
type outputTarget struct {
551+
path string
552+
format output.Format
553+
}
554+
555+
// parseOutputs resolves the -o values; a file without a prefix takes def,
556+
// the -format of stdout. The same path given twice is refused: both would
557+
// replace one file.
558+
func parseOutputs(specs []string, def output.Format) ([]outputTarget, error) {
559+
targets := make([]outputTarget, 0, len(specs))
560+
seen := map[string]bool{}
561+
for _, spec := range specs {
562+
name, path := splitOutputSpec(spec)
563+
format := def
564+
if name != "" {
565+
var err error
566+
if format, err = output.ParseFormat(name); err != nil {
567+
return nil, err
568+
}
569+
}
570+
key := filepath.Clean(path)
571+
if seen[key] {
572+
return nil, fmt.Errorf("-o %s is given more than once", path)
573+
}
574+
seen[key] = true
575+
targets = append(targets, outputTarget{path: path, format: format})
503576
}
504-
return output.New(f.Writer, testMode, format), f, true
577+
return targets, nil
505578
}
506579

507580
// isTerminal reports whether w is a character device (a terminal) rather than
@@ -542,16 +615,16 @@ func logVerboseStart(f cliFlags, domain string, maxAttempts int, out *output.Wri
542615
if !f.testMode {
543616
out.Info("DNS Server: %s", f.dnsServer)
544617
}
545-
if f.outputFile != "" {
546-
out.Info("Output file: %s", f.outputFile)
618+
if len(f.outputs.specs) > 0 {
619+
out.Info("Output file: %s", f.outputs.describe())
547620
}
548621
out.Info("---")
549622
}
550623

551-
func logVerboseDone(ev scan.Event, f cliFlags, outWriter *bufio.Writer, out *output.Writer) {
624+
func logVerboseDone(ev scan.Event, f cliFlags, out *output.Writer) {
552625
out.Info("Processed %d subdomain prefixes", ev.Processed)
553-
if outWriter != nil {
554-
out.Info("Results written to: %s", f.outputFile)
626+
if len(f.outputs.specs) > 0 {
627+
out.Info("Results written to: %s", f.outputs.describe())
555628
}
556629
if f.testMode {
557630
out.Info("\nNOTE: Results were simulated and no actual DNS queries were performed.")
@@ -768,7 +841,7 @@ func scanTarget(ctx context.Context, s *session, p scanPass, emit func(scan.Even
768841
res.resolverAbort = ev.Stats.Aborted
769842
logScanBreakdown(p.domain, ev, out)
770843
if f.verbose {
771-
logVerboseDone(ev, f, s.outWriter, out)
844+
logVerboseDone(ev, f, out)
772845
}
773846
}
774847
}

0 commit comments

Comments
 (0)