Skip to content

Commit 665e33c

Browse files
committed
MCP: let a call continue the word list with an offset (#313)
An MCP scan tests at most max_queries names, and a call could not say where to start, so an agent using the bundled list could only ever test its first part. scan and lab_scan accept an integer offset that skips that many words of the deduplicated list, and the report gives next_offset (with a hint in the reason) when the query cap left words untested and every admitted name was tested. Each call still goes through the operator's query, time and call budgets. Co-Authored-By: Claude Mythos 5.2 <noreply@anthropic.com> Fixes #313
1 parent c1eb970 commit 665e33c

4 files changed

Lines changed: 227 additions & 11 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
88
## [Unreleased]
99

1010
### Added
11+
- `subenum mcp`: `scan` and `lab_scan` accept an `offset` that skips that many candidates at the start of the word list (after dedupe), and a report whose `max_queries` left words untested carries `next_offset`, named in its `reason`. An agent can now work through the bundled 5,000-label list in several calls instead of testing the same first `-max-queries` names each time; every call stays within `-max-queries`, `-max-duration` and `-max-calls` (#313).
1112
- The TUI configuration form includes **Auto Rate**, **Max Duration** and **Retry Inconclusive** fields for `-auto-rate`, `-max-duration` and `-retry-inconclusive`. Max Duration accepts a Go duration such as `20m` or `1h30m`; leave it blank for no time limit (#274).
1213
- `subenum mcp` serves subdomain enumeration to AI agents over the Model Context Protocol (stdio). Tools: `lab_scan`, which scans the compiled-in lab scenarios on a loopback DNS server; `scan`, offered only with `-allow` and refusing any domain outside it; and `explain_stats`, which explains a `-stats` report or `-format jsonl` output. The operator sets the resolver, rate, timeout and budgets (`-dns-server`, `-rate`, `-timeout`, `-max-queries`, `-max-duration`, which cannot be turned off). A call can lower `max_queries`, never raise it; unknown arguments are refused, and calls run one at a time. Each scan returns its results with a verdict (`complete`, `degraded`, `unreliable` or `not_run`), and sends progress notifications about once a second when the call carries a `progressToken` (#267). `-sources ct,wayback` lets `scan` calls seed candidates from passive sources through a `sources` argument; it is off by default, and the report counts each source's names or names its error (#268). `scan` and `lab_scan` accept `retry_inconclusive`, `auto_rate`, `exclude` and `depth` (up to 3), all within the operator's caps, so an agent can act on a verdict that is not complete; the report adds `retried`, `recovered`, `rate_backoffs` and `excluded` (#269). Their report is also declared as an `outputSchema` and sent as `structuredContent`, beside the same JSON as text (#270). `-max-calls` (default 100) caps the `scan` calls one session may make, and a client that stops reading replies ends the session, cancelling queued calls instead of scanning for nobody (#240). A request line over 16 MiB gets a JSON-RPC parse error instead of ending the session, a request reusing the ID of one still running is refused, and a scan cancelled before testing any name reports `not_run` rather than `degraded` (#241). The JSON-RPC layer is a small hand-written `internal/mcp` package, so there is no new dependency (#131). `lab_scan`'s `scenario` argument describes what each lab teaches, taken from the opening comment of its zone file (#275).
1314
- `-sources ct,wayback` picks keyless passive sources whose names become candidates and `-permute` seeds, each still verified by a lookup. `ct` is crt.sh, as `-ct`, which stays as shorthand. `wayback` is new: the hostnames in the Wayback Machine's index of archived URLs, one HTTPS request per target. The report gains a `sources` entry per source (`names`, or the `error` it hit; `ct_names` and `ct_error` are kept), a failed source is a warning, the resume state keeps each source's names, simulation ignores `-sources` and lab mode refuses it. Sources share one interface in a new `internal/passive` package. `pkg/subenum` gains `Config.Sources`, the `Source` interface, `CertificateTransparency()`, `WaybackMachine()` and `NoticeSource` (#193).

‎docs/mcp.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ safety. The model chooses only what to scan within those limits.
1818
| :--- | :--- |
1919
| Which domains a live scan may target (`-allow`) | A domain within the allow-list |
2020
| The resolver (`-dns-server`) and the lookup timeout (`-timeout`) | The words to try, and the record types |
21-
| The query rate (`-rate`) | A smaller `max_queries`, never a larger one |
21+
| The query rate (`-rate`) | A smaller `max_queries`, never a larger one, and an `offset` into the word list |
2222
| The most names one call may test (`-max-queries`) and its time budget (`-max-duration`) | For lab scans only: the scenario, a rate, and `force` |
2323

2424
Tool calls run one at a time, so a model cannot multiply the budget by calling
@@ -106,8 +106,17 @@ apply to it, so `SUBENUM_DNS_SERVER` does not change the agent's resolver: pass
106106

107107
A `scan` call without `words` uses the bundled list of 5,000 common labels, but
108108
tests no more than `-max-queries` of them (1,000 by default), so its verdict is
109-
`degraded` with the rest counted as skipped. Raise `-max-queries` to cover the
110-
whole list, or have the agent pass a shorter `words` list.
109+
`degraded` with the rest counted as skipped. The report then carries a
110+
`next_offset`, and its `reason` says to call again with it: `scan` and
111+
`lab_scan` take an `offset` argument that skips that many candidates at the
112+
start of the word list (after dedupe), so the agent can work through the
113+
whole list in five calls (at the default `-max-queries` of 1000) with the same `words`. Every call stays within
114+
`-max-queries`, `-max-duration` and `-max-calls`. `next_offset` is given only
115+
when `max_queries` was what stopped the scan among the words: a call the time
116+
budget ended has gaps among the names it tested, so repeat that one with a
117+
smaller `max_queries`. Names from `sources` come after the words and are not
118+
counted by `offset`. To cover the list in one call instead, raise
119+
`-max-queries`.
111120

112121
In a container, keep standard input open with `-i`, since the protocol runs over
113122
it:

‎internal/cli/mcp.go‎

Lines changed: 73 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,11 @@ func wordsSchema(description string) map[string]any {
358358
return map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "maxItems": maxMCPWords, "description": description}
359359
}
360360

361+
// offsetSchema is the offset argument (#313): with the report's next_offset
362+
// it lets an agent work through a word list longer than max_queries, one
363+
// call at a time, each within the operator's caps.
364+
var offsetSchema = schemaField{"offset", map[string]any{"type": "integer", "minimum": 0, "description": "Skip this many candidates at the start of the word list (after dedupe); default 0. When max_queries leaves words untested the report gives next_offset: call again with the same words and that offset to test the following ones."}}
365+
361366
// remediationProps are the scan settings an agent can change after a verdict
362367
// that is not complete (#269). None can raise the operator's caps.
363368
var remediationProps = schemaObject{
@@ -405,6 +410,7 @@ func labScanTool(cfg mcpConfig) mcp.Tool {
405410
{"types", recordTypeSchema},
406411
{"rate", map[string]any{"type": "integer", "minimum": 0, "maximum": 1000, "description": "Queries per second; 0 or absent is unlimited. Lab traffic stays on this machine."}},
407412
{"max_queries", map[string]any{"type": "integer", "minimum": 1, "maximum": cfg.maxQueries, "description": "Most names to test."}},
413+
offsetSchema,
408414
{"force", map[string]any{"type": "boolean", "description": "Scan a wildcard zone anyway, filtering answers that match the wildcard."}},
409415
}, remediationProps...))
410416
return mcp.Tool{
@@ -421,6 +427,7 @@ func labScanTool(cfg mcpConfig) mcp.Tool {
421427
Types []string `json:"types"`
422428
Rate int `json:"rate"`
423429
MaxQueries int `json:"max_queries"`
430+
Offset int `json:"offset"`
424431
Force bool `json:"force"`
425432
remediation
426433
}
@@ -440,7 +447,7 @@ func labScanTool(cfg mcpConfig) mcp.Tool {
440447
if err != nil {
441448
return mcp.Errorf("%v", err)
442449
}
443-
words, err := callWords(args.Words, labWords())
450+
words, err := callWords(args.Words, labWords(), labDomain, args.Offset)
444451
if err != nil {
445452
return mcp.Errorf("%v", err)
446453
}
@@ -466,7 +473,7 @@ func labScanTool(cfg mcpConfig) mcp.Tool {
466473
}
467474
defer srv.Close()
468475
scanCfg.Resolver = srv.Addr
469-
report := runMCPScan(ctx, scanCfg)
476+
report := runMCPScan(ctx, scanCfg, args.Offset)
470477
report.Scenario = args.Scenario
471478
return report.result()
472479
},
@@ -479,6 +486,7 @@ func scanTool(cfg mcpConfig) mcp.Tool {
479486
{"words", wordsSchema("Subdomain labels to try; default subenum's bundled list of the most common ones.")},
480487
{"types", recordTypeSchema},
481488
{"max_queries", map[string]any{"type": "integer", "minimum": 1, "maximum": cfg.maxQueries, "description": fmt.Sprintf("Most names to test; default and ceiling %d.", cfg.maxQueries)}},
489+
offsetSchema,
482490
}
483491
if len(cfg.sources) > 0 {
484492
props = append(props, schemaField{"sources", map[string]any{"type": "array", "items": map[string]any{"enum": cfg.sources}, "description": "Passive sources to seed candidates from before the scan; default none. Their names are added to words after dedupe, count against max_queries, and are each verified by a lookup. One HTTPS request per source."}})
@@ -499,6 +507,7 @@ func scanTool(cfg mcpConfig) mcp.Tool {
499507
Words []string `json:"words"`
500508
Types []string `json:"types"`
501509
MaxQueries int `json:"max_queries"`
510+
Offset int `json:"offset"`
502511
Sources []string `json:"sources"`
503512
remediation
504513
}
@@ -529,7 +538,7 @@ func scanTool(cfg mcpConfig) mcp.Tool {
529538
if err != nil {
530539
return mcp.Errorf("%v", err)
531540
}
532-
words, err := callWords(args.Words, nil)
541+
words, err := callWords(args.Words, nil, domain, args.Offset)
533542
if err != nil {
534543
return mcp.Errorf("%v", err)
535544
}
@@ -554,7 +563,7 @@ func scanTool(cfg mcpConfig) mcp.Tool {
554563
if calls.Add(1) > int64(cfg.maxCalls) {
555564
return mcp.Errorf("session budget spent: the operator allows %d scan calls per session (-max-calls)", cfg.maxCalls)
556565
}
557-
report := runMCPScan(ctx, scanCfg)
566+
report := runMCPScan(ctx, scanCfg, args.Offset)
558567
for _, c := range counted {
559568
s := mcpSource{Name: c.name, Names: c.names}
560569
var tr *passive.Truncated
@@ -595,14 +604,63 @@ func callBudget(cfg mcpConfig, asked int) (int, error) {
595604
return asked, nil
596605
}
597606

598-
func callWords(words, fallback []string) ([]string, error) {
607+
// callWords is the word list a call scans: its words, or fallback (nil is
608+
// the bundled list) without any, as a scan of domain builds it (valid
609+
// labels, deduplicated, in order) and without its first offset candidates,
610+
// so a call can go on where max_queries stopped an earlier one (#313).
611+
func callWords(words, fallback []string, domain string, offset int) ([]string, error) {
599612
if len(words) > maxMCPWords {
600613
return nil, fmt.Errorf("words has %d entries; at most %d", len(words), maxMCPWords)
601614
}
615+
if offset < 0 {
616+
return nil, fmt.Errorf("offset must be 0 or more, got %d", offset)
617+
}
618+
if words == nil {
619+
words = fallback
620+
}
602621
if words == nil {
603-
return fallback, nil
622+
words = subenum.DefaultWords()
623+
}
624+
entries, _, _ := wordlist.Build(words, domain)
625+
switch {
626+
case offset == 0 && len(entries) == 0:
627+
return words, nil // the scan says why it cannot start
628+
case offset >= len(entries):
629+
return nil, fmt.Errorf("offset %d is past the end of the word list, which has %d candidates", offset, len(entries))
630+
}
631+
return entries[offset:], nil
632+
}
633+
634+
// nextOffset is the offset of the first candidate a scan left untested
635+
// because max_queries was reached, for a call that continues it, or 0 when
636+
// there is nothing to continue from: no candidate was skipped, the words
637+
// were all tested (the skipped ones came from recursion or a source), or
638+
// the scan stopped for another reason (the time budget, an abort, a
639+
// cancellation), which leaves gaps among the names it tested. offset is the
640+
// one the scan itself started at.
641+
func nextOffset(cfg subenum.Config, stats subenum.Stats, offset int) int {
642+
if stats.Skipped == 0 || cfg.MaxQueries == 0 || stats.Total() != int64(cfg.MaxQueries) {
643+
return 0
644+
}
645+
// Candidates are admitted in order, the words before anything else, so
646+
// the names tested are the first MaxQueries the exclusions let through.
647+
if len(cfg.Exclude) == 0 {
648+
if cfg.MaxQueries < len(cfg.Words) {
649+
return offset + cfg.MaxQueries
650+
}
651+
return 0
652+
}
653+
tested := 0
654+
for i, word := range cfg.Words {
655+
if scan.IsExcluded(cfg.Exclude, word+"."+cfg.Domain) {
656+
continue
657+
}
658+
if tested == cfg.MaxQueries {
659+
return offset + i
660+
}
661+
tested++
604662
}
605-
return words, nil
663+
return 0
606664
}
607665

608666
// strictUnmarshal decodes tool arguments, rejecting unknown ones so a
@@ -625,6 +683,7 @@ type mcpReport struct {
625683
Results []mcpResult `json:"results"`
626684
Outcomes mcpOutcomes `json:"outcomes"`
627685
Skipped int64 `json:"skipped_by_cap,omitempty"`
686+
NextOffset int `json:"next_offset,omitempty"` // the offset that continues a scan max_queries cut short
628687
Excluded int64 `json:"excluded,omitempty"`
629688
Retried int64 `json:"retried,omitempty"` // retry_inconclusive looked these up again
630689
Recovered int64 `json:"recovered,omitempty"` // and these then got an answer
@@ -659,7 +718,9 @@ type mcpOutcomes struct {
659718
WildcardFiltered int64 `json:"wildcard_filtered"`
660719
}
661720

662-
func runMCPScan(ctx context.Context, cfg subenum.Config) *mcpReport {
721+
// runMCPScan runs one scan and reports it. offset is where cfg.Words starts
722+
// in the call's word list, for the report's next_offset.
723+
func runMCPScan(ctx context.Context, cfg subenum.Config, offset int) *mcpReport {
663724
report := &mcpReport{Domain: cfg.Domain, Results: []mcpResult{}, MaxQueries: cfg.MaxQueries, Rate: cfg.Rate}
664725
cfg.Logf = func(string, ...any) {} // standard output carries the protocol
665726
events, err := subenum.Run(ctx, cfg)
@@ -695,6 +756,9 @@ func runMCPScan(ctx context.Context, cfg subenum.Config) *mcpReport {
695756
report.Excluded, report.Retried, report.Recovered, report.Backoffs = stats.Excluded, stats.Retried, stats.Recovered, stats.RateBackoffs
696757
report.Wildcard = stats.Wildcard
697758
report.Verdict, report.Reason = mcpVerdict(stats, report.Error, ctx.Err() != nil)
759+
if report.NextOffset = nextOffset(cfg, stats, offset); report.NextOffset > 0 {
760+
report.Reason += fmt.Sprintf("; call again with offset %d to test the words after these", report.NextOffset)
761+
}
698762
return report
699763
}
700764

@@ -760,6 +824,7 @@ var mcpReportSchema = json.RawMessage(`{
760824
"wildcard_filtered": {"type": "integer"}
761825
}},
762826
"skipped_by_cap": {"type": "integer"},
827+
"next_offset": {"type": "integer", "description": "max_queries left words untested: call again with the same words and this offset to test the following ones"},
763828
"excluded": {"type": "integer"},
764829
"retried": {"type": "integer"},
765830
"recovered": {"type": "integer"},

0 commit comments

Comments
 (0)