Skip to content

Commit c5e4454

Browse files
TMHSDigitalclaude
andcommitted
Let config and SUBENUM_* set several -w or -o files (#353)
-w and -o are repeatable on the command line, but the config file rejected a JSON array for them and a string became one path with commas in it. The config file now takes an array for w and o (each entry is one value, set in order, commas kept) and for sources, as it already did for exclude and type. To match the other list settings, a w or o string in the config file or SUBENUM_W/SUBENUM_O is a comma-separated list; a path with a comma goes in an array entry or on the command line. -print-config shows w and o as a JSON array so ["a,b"] and ["a","b"] are told apart. Fixes #353 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 70a023f commit c5e4454

7 files changed

Lines changed: 118 additions & 23 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7070
- Less memory on large wordlists and busy scans: `-sources`, `-ct`, `-axfr` and `-permute` check their names against the wordlist without building a set as large as it (about 200 MB per feature and target on a 5M-line list), a `-simulate-zone` lab scan no longer keeps a count per queried name (about 170 MB on a 1M-line list), and a `CNAME` lookup reads a UDP answer into a buffer sized for the 1232-byte EDNS payload it advertises instead of 64 KB. Results are unchanged (#321).
7171

7272
### Fixed
73+
- The config file and `SUBENUM_W`/`SUBENUM_O` can set several `-w` wordlists or `-o` files, as the command line can. The config file takes a JSON array for `w` and `o`, one file per entry (`"o": ["names.txt", "jsonl:results.jsonl"]`), and for `sources`, as it already did for `exclude` and `type`; a string for `w` or `o` there or in the environment is a comma-separated list, as it is for the other list settings, where it used to become one path with commas in it. `-print-config` shows `w` and `o` as a JSON array, so one file whose name has a comma is told apart from two (#353).
7374
- A name whose lookup of one requested type timed out, was refused or failed while another type answered is no longer a plain found result with the records that came back. The whole lookup is tried again up to `-attempts`, and when the failed type never answers the result is marked partial: a `partial` field in JSON and JSONL, `PARTIAL` in text with `-show-records`, a `partial answers` line in the breakdown, and `Stats.Partial` in the library. Partial answers lower the run's verdict as failed lookups do (degraded at 1%), and `-diff-records` does not report a partial name as changed, since its missing records may only be a lookup that failed (#357).
7475
- Resuming a scan stopped in its `-permute` pass (Ctrl+C or `-max-duration`) no longer scans the finished main pass again. Under `-recursive` that repeated every lookup below the saved names, and a name the repeat found anew changed the permutation candidate list, so the resumed pass started at the wrong place in a different list, skipping some candidates and testing others twice. The resume state now records that the main pass finished, and the resumed run goes straight to the permutation pass with the saved names as its seeds (#358).
7576
- When `-max-queries` or `-max-duration` stops a scan whose wordlist holds multi-label entries (such as `dev.api`), the untested names are counted as they would have been: ones `-exclude` would remove count as excluded rather than skipped, so a run whose only untested names were out of scope stays complete, and under `-recursive` a name the rest of the scan would have reached through two parents is counted as skipped once, so tested, skipped and excluded add up to the candidate total (#361).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,7 @@ Settings you repeat on every run can live in a config file or the environment. P
313313
{ "dns-server": "1.1.1.1:53", "rate": 200, "t": 50, "r": "/home/me/resolvers.txt" }
314314
```
315315

316-
- The config file is `config.json` in your user config directory (`~/.config/subenum/` on Linux, `~/Library/Application Support/subenum/` on macOS, `%AppData%\subenum\` on Windows), or the path in `SUBENUM_CONFIG`. Keys are flag names; values are strings, numbers or booleans, and `exclude` and `type` also take a list of strings.
316+
- The config file is `config.json` in your user config directory (`~/.config/subenum/` on Linux, `~/Library/Application Support/subenum/` on macOS, `%AppData%\subenum\` on Windows), or the path in `SUBENUM_CONFIG`. Keys are flag names; values are strings, numbers or booleans, and `exclude`, `type` and `sources` also take a list of strings. The repeatable `w` and `o` take a list too, one file per entry (`"o": ["names.txt", "jsonl:results.jsonl"]`); a string for them, in the config file or in `SUBENUM_W`/`SUBENUM_O`, is a comma-separated list, so name a file whose path has a comma in a list entry or on the command line.
317317
- Each flag also reads `SUBENUM_<FLAG>`, upper-cased with dashes as underscores: `SUBENUM_DNS_SERVER`, `SUBENUM_RATE`, `SUBENUM_T`. Mode flags (`tui`, `version`, `print-config`, `dry-run`, `resume`) are command-line only.
318318
- The TUI (`-tui`) starts its form from these values too, and from flags and the domain given with `-tui` on the command line, over the values it remembers from its last session.
319319
- `subenum -print-config` shows every effective setting and where it came from (`flag`, `env`, `config` or `default`). It, `-h` and `-version` still work when a value is bad, so you can find it; a scan refuses to start and names every bad value.

‎docs/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ Flags may appear before or after the domain. The domain is normalized to its hos
4444
Settings you repeat on every run can live in the environment or a config file. They apply to scans (and `-tui`), not to the `mcp` subcommand, which takes its settings from its own flags only. Precedence, highest first: **command-line flag, then `SUBENUM_*` environment variable, then config file, then built-in default.**
4545

4646
- Each flag reads `SUBENUM_<FLAG>`, upper-cased with dashes as underscores: `SUBENUM_DNS_SERVER`, `SUBENUM_RATE`, `SUBENUM_T`.
47-
- The config file is `config.json` in your user config directory (`~/.config/subenum/` on Linux, `~/Library/Application Support/subenum/` on macOS, `%AppData%\subenum\` on Windows), or the path in `SUBENUM_CONFIG`. Keys are flag names; values are strings, numbers or booleans, and `exclude` and `type` also take a list of strings. Mode flags (`tui`, `version`, `print-config`, `dry-run`, `resume`) are command-line only.
47+
- The config file is `config.json` in your user config directory (`~/.config/subenum/` on Linux, `~/Library/Application Support/subenum/` on macOS, `%AppData%\subenum\` on Windows), or the path in `SUBENUM_CONFIG`. Keys are flag names; values are strings, numbers or booleans, and `exclude`, `type` and `sources` also take a list of strings. The repeatable `w` and `o` take a list too, one file per entry (`"o": ["names.txt", "jsonl:results.jsonl"]`); a string for them, in the config file or in `SUBENUM_W`/`SUBENUM_O`, is a comma-separated list, so name a file whose path has a comma in a list entry or on the command line. Mode flags (`tui`, `version`, `print-config`, `dry-run`, `resume`) are command-line only.
4848
- The TUI (`-tui`) starts its form from these values too, and from flags and the domain given with `-tui` on the command line, over the values it remembers from its last session.
4949
- `subenum -print-config` prints every effective setting and where it came from: `flag`, `env`, `config` or `default`. With a bad value, `-h`, `-version` and `-print-config` still run and warn; a scan refuses to start and names every bad value.
5050
- When `-attempts` and the deprecated `-retries` both have a value, the one from the higher-precedence source wins; both on the command line (or both from the same file) is an error.

‎internal/cli/cli.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -528,6 +528,9 @@ func (o *outputSpecs) Set(v string) error {
528528
// the environment rather than the command line.
529529
func (o *outputSpecs) markDefault() { o.replace = true }
530530

531+
// list returns the specs, for -print-config.
532+
func (o *outputSpecs) list() []string { return o.specs }
533+
531534
// describe names the -o files for notices: the paths, comma-separated.
532535
func (o *outputSpecs) describe() string {
533536
paths := make([]string, len(o.specs))

‎internal/cli/config.go‎

Lines changed: 57 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -40,37 +40,64 @@ var noDefaultFlags = map[string]bool{"tui": true, "version": true, "print-config
4040

4141
// listFlags take comma-separated values, so the config file may also give
4242
// them as a JSON array of strings.
43-
var listFlags = map[string]bool{"exclude": true, "type": true}
43+
var listFlags = map[string]bool{"exclude": true, "type": true, "sources": true}
44+
45+
// repeatFlags may be given more than once on the command line (-w, -o). Like
46+
// the listFlags, a config-file or SUBENUM_* string is a comma-separated list
47+
// and the config file may give a JSON array; each array entry is one value,
48+
// commas and all (#353).
49+
var repeatFlags = map[string]bool{"w": true, "o": true}
50+
51+
// repeatValue is the Value of a repeatFlag: the values given, in order.
52+
type repeatValue interface{ list() []string }
4453

4554
// sourceRank orders sources by precedence, highest last.
4655
var sourceRank = map[string]int{sourceDefault: 0, sourceConfig: 1, sourceEnv: 2, sourceFlag: 3}
4756

48-
// configValue turns one config-file value into a flag value. Only strings,
49-
// numbers and booleans are settings; fmt.Sprint of null or an object would
50-
// silently become a bogus value such as "<nil>".
51-
func configValue(name string, v any) (string, error) {
57+
// splitRepeat splits a config-file or environment string for a repeatFlag
58+
// into the values to Set, one per comma-separated entry.
59+
func splitRepeat(name, v string) []string {
60+
if !repeatFlags[name] {
61+
return []string{v}
62+
}
63+
parts := strings.Split(v, ",")
64+
for i, p := range parts {
65+
parts[i] = strings.TrimSpace(p)
66+
}
67+
return parts
68+
}
69+
70+
// configValues turns one config-file value into the flag values to Set: one
71+
// value, or one per entry for a repeatFlag. Only strings, numbers and
72+
// booleans are settings; fmt.Sprint of null or an object would silently
73+
// become a bogus value such as "<nil>".
74+
func configValues(name string, v any) ([]string, error) {
75+
list := listFlags[name] || repeatFlags[name]
5276
switch v := v.(type) {
5377
case string:
54-
return v, nil
78+
return splitRepeat(name, v), nil
5579
case json.Number, bool:
56-
return fmt.Sprint(v), nil
80+
return []string{fmt.Sprint(v)}, nil
5781
case []any:
58-
if listFlags[name] {
82+
if list {
5983
parts := make([]string, len(v))
6084
for i, p := range v {
6185
s, ok := p.(string)
6286
if !ok {
63-
return "", fmt.Errorf("list entries must be strings")
87+
return nil, fmt.Errorf("list entries must be strings")
6488
}
6589
parts[i] = s
6690
}
67-
return strings.Join(parts, ","), nil
91+
if repeatFlags[name] {
92+
return parts, nil
93+
}
94+
return []string{strings.Join(parts, ",")}, nil
6895
}
6996
}
70-
if listFlags[name] {
71-
return "", fmt.Errorf("must be a string or a list of strings")
97+
if list {
98+
return nil, fmt.Errorf("must be a string or a list of strings")
7299
}
73-
return "", fmt.Errorf("must be a string, number or boolean")
100+
return nil, fmt.Errorf("must be a string, number or boolean")
74101
}
75102

76103
// defaultsLoader applies config-file and environment defaults to a FlagSet
@@ -106,15 +133,18 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error {
106133
d.sources = map[string]string{}
107134
d.values = map[string]string{}
108135
var errs []error
109-
set := func(name, value, source string) bool {
110-
if err := fs.Lookup(name).Value.Set(value); err != nil {
111-
errs = append(errs, fmt.Errorf("%w: %s value %q for %s: %w", errConfig, source, value, name, err))
112-
return false
136+
// set applies values in order: one, or each entry of a repeatFlag list.
137+
set := func(name string, values []string, source string) bool {
138+
for _, value := range values {
139+
if err := fs.Lookup(name).Value.Set(value); err != nil {
140+
errs = append(errs, fmt.Errorf("%w: %s value %q for %s: %w", errConfig, source, value, name, err))
141+
return false
142+
}
113143
}
114144
if v, ok := fs.Lookup(name).Value.(interface{ markDefault() }); ok {
115145
v.markDefault()
116146
}
117-
d.values[name] = value
147+
d.values[name] = strings.Join(values, ",")
118148
return true
119149
}
120150

@@ -148,12 +178,12 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error {
148178
errs = append(errs, fmt.Errorf("%w: %s sets unknown setting %q", errConfig, source, name))
149179
continue
150180
}
151-
value, err := configValue(name, cfg[name])
181+
values, err := configValues(name, cfg[name])
152182
if err != nil {
153183
errs = append(errs, fmt.Errorf("%w: %s value for %s %w", errConfig, source, name, err))
154184
continue
155185
}
156-
if set(name, value, source) {
186+
if set(name, values, source) {
157187
d.sources[name] = sourceConfig
158188
}
159189
}
@@ -164,7 +194,7 @@ func (d *defaultsLoader) apply(fs *flag.FlagSet) error {
164194
if noDefaultFlags[fl.Name] {
165195
return
166196
}
167-
if v := d.getenv(envName(fl.Name)); v != "" && set(fl.Name, v, envName(fl.Name)) {
197+
if v := d.getenv(envName(fl.Name)); v != "" && set(fl.Name, splitRepeat(fl.Name, v), envName(fl.Name)) {
168198
d.sources[fl.Name] = sourceEnv
169199
}
170200
})
@@ -216,6 +246,12 @@ func (d *defaultsLoader) printConfig(w io.Writer, fset *flag.FlagSet) {
216246
if given[fl.Name] {
217247
source = sourceFlag
218248
}
249+
if rv, ok := fl.Value.(repeatValue); ok {
250+
// A JSON array, so ["a,b"] (one file) and ["a","b"] differ (#353).
251+
list, _ := json.Marshal(append([]string{}, rv.list()...))
252+
_, _ = fmt.Fprintf(w, "%-14s = %-24s (%s)\n", fl.Name, list, source)
253+
return
254+
}
219255
value := fl.Value.String()
220256
if fl.Name == "attempts" {
221257
value, source = d.effectiveAttempts(fset, value, source)

‎internal/cli/config_test.go‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,58 @@ func TestDefaultsPrecedence(t *testing.T) {
8282
}
8383
}
8484

85+
// TestRepeatFlagDefaults covers #353: the config file sets several -w or -o
86+
// values as a JSON array (one value per entry, commas kept), a config or
87+
// SUBENUM_* string is a comma-separated list, and -print-config shows them as
88+
// a JSON array. -sources takes an array like -exclude and -type.
89+
func TestRepeatFlagDefaults(t *testing.T) {
90+
cfg := filepath.Join(t.TempDir(), "config.json")
91+
if err := os.WriteFile(cfg, []byte(`{"o": ["a.txt", "jsonl:b,c.jsonl"], "w": ["x.txt", "y.txt"], "sources": ["ct", "wayback"]}`), 0o600); err != nil {
92+
t.Fatal(err)
93+
}
94+
cases := []struct {
95+
name string
96+
env map[string]string
97+
wantW []string
98+
wantO []string
99+
printWant []string
100+
}{
101+
{"config arrays", nil, []string{"x.txt", "y.txt"}, []string{"a.txt", "jsonl:b,c.jsonl"},
102+
[]string{`w = ["x.txt","y.txt"]`, `o = ["a.txt","jsonl:b,c.jsonl"]`, `sources = "ct,wayback"`}},
103+
{"env lists replace config", map[string]string{"SUBENUM_W": "w1.txt, w2.txt", "SUBENUM_O": "out.txt,jsonl:out.jsonl"},
104+
[]string{"w1.txt", "w2.txt"}, []string{"out.txt", "jsonl:out.jsonl"},
105+
[]string{`w = ["w1.txt","w2.txt"]`, `o = ["out.txt","jsonl:out.jsonl"] (env)`}},
106+
}
107+
for _, tc := range cases {
108+
t.Run(tc.name, func(t *testing.T) {
109+
d := &defaultsLoader{path: cfg, getenv: func(k string) string { return tc.env[k] }}
110+
f, _, fs, err := parseFlagsWith([]string{"example.com"}, d.apply, nil)
111+
if err != nil {
112+
t.Fatal(err)
113+
}
114+
if !slices.Equal(f.wordlists.paths, tc.wantW) || !slices.Equal(f.outputs.specs, tc.wantO) || f.sources != "ct,wayback" {
115+
t.Errorf("w %q, o %q, sources %q; want %q, %q, ct,wayback", f.wordlists.paths, f.outputs.specs, f.sources, tc.wantW, tc.wantO)
116+
}
117+
var out bytes.Buffer
118+
d.printConfig(&out, fs)
119+
for _, want := range tc.printWant {
120+
if !strings.Contains(out.String(), want) {
121+
t.Errorf("-print-config lacks %q:\n%s", want, out.String())
122+
}
123+
}
124+
})
125+
}
126+
127+
bad := filepath.Join(t.TempDir(), "bad.json")
128+
if err := os.WriteFile(bad, []byte(`{"w": ["x.txt", 1]}`), 0o600); err != nil {
129+
t.Fatal(err)
130+
}
131+
d := &defaultsLoader{path: bad, getenv: func(string) string { return "" }}
132+
if _, _, _, err := parseFlagsWith(nil, d.apply, nil); err == nil || !strings.Contains(err.Error(), "list entries must be strings") {
133+
t.Errorf("non-string -w entry: err = %v", err)
134+
}
135+
}
136+
85137
func TestDefaultsErrors(t *testing.T) {
86138
dir := t.TempDir()
87139
for name, content := range map[string]string{

‎internal/cli/wordlist.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,9 @@ func (w *wordlistPaths) Set(v string) error {
7272
// the environment rather than the command line.
7373
func (w *wordlistPaths) markDefault() { w.replace = true }
7474

75+
// list returns the paths, for -print-config.
76+
func (w *wordlistPaths) list() []string { return w.paths }
77+
7578
// readsStdin reports whether any -w entry is "-".
7679
func (w *wordlistPaths) readsStdin() bool { return countStdin(w.paths) > 0 }
7780

0 commit comments

Comments
 (0)