Skip to content

Commit bb98c4c

Browse files
TMHSDigitalclaude
andcommitted
Stream crt.sh answers and keep names before the read limit (#251)
The crt.sh response was decoded as one JSON array, so memory scaled with the whole body and an answer that hit the 64 MiB cap failed with unexpected EOF, losing every CT name. Decode the array one entry at a time into the name set instead. When the limit cuts the body short, return the names read so far with ct.ErrTruncated; the passive registry turns that into a Truncated error, so the CLI, report and library treat it like a truncated Wayback answer. pkg's CertificateTransparency now builds its source through passive.New to get the same handling. Fixes #251 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 2c771ba commit bb98c4c

7 files changed

Lines changed: 136 additions & 27 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3535
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).
3636
- Passive sources (`-sources`, `-ct`, `Config.Sources`) retry a busy answer (429, 502, 503, 504) or a dropped connection twice, after 2 and 5 seconds or the server's `Retry-After` (at most 30 seconds), within each source's 60-second limit; crt.sh often answers 502 or 503 under load, which used to cost the scan all of its names. A target's sources are now asked at the same time instead of one after another (#271).
3737
- When the Wayback Machine's answer reaches the 200,000-URL or 64 MiB limit, a note says its results were truncated and some hosts may be missing, the report's `sources` entry (and the MCP report's) is marked `truncated`, `explain_stats` mentions it, and the library sends a `NoticeSource` event; the names it did return are still scanned. It used to stop silently (#272).
38+
- crt.sh's answer is read one entry at a time instead of decoded whole, so memory follows the names rather than the body. An answer that reaches the 64 MiB limit keeps the names read before the cut and is reported as truncated, like Wayback's, instead of failing and losing every CT name (#251).
3839
- The TUI runs on Bubble Tea v2: `charm.land/bubbletea/v2` v2.1.1, `bubbles/v2` v2.2.1 and `lipgloss/v2` v2.0.6 replace the v1 modules, dropping v1-only helpers that had not had a release in years. Keys, screens and the form work as before; the `go 1.26.0` minimum is unchanged (#188).
3940
- With text output (the default) and no `-show-records` or `-diff`, the CLI keeps only the names of the current target's results for `-permute` and the `-resume` state, plus the few results with a takeover hint, an internal address or a permutation, instead of every result with its records. A simulated 1M-name scan at 100% hits peaks at about 240 MB instead of 750 MB, against 233 MB at 1% hits. A resumed run re-emits the same text. Structured formats still keep the records a resumed run writes back (#183).
4041
- The scan dispatcher hands jobs to workers in small chunks (up to 8, fewer when there are few names per worker) over buffered channels, and each worker reports a chunk's jobs at once, instead of two unbuffered hand-offs per job. A 1M-name simulated scan at 50 workers (`make bench`) takes about 0.4 s instead of 1.6 s. A resumed scan may repeat up to one chunk per worker instead of one lookup per worker (#184).

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -455,7 +455,7 @@ The run-quality report counts them per target (`ct_names`). If crt.sh is down or
455455
subenum -sources ct,wayback example.com
456456
```
457457

458-
The sources of a target are asked at the same time, and a busy answer (429, 502, 503, 504) or a dropped connection is retried twice with a short backoff, honouring `Retry-After`, within the source's 60-second limit. The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. The Wayback Machine is asked for at most 200,000 URLs and 64 MiB; when it reaches either limit its names are still used, a note says some hosts may be missing, and its entry is marked `truncated`. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
458+
The sources of a target are asked at the same time, and a busy answer (429, 502, 503, 504) or a dropped connection is retried twice with a short backoff, honouring `Retry-After`, within the source's 60-second limit. The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. The Wayback Machine is asked for at most 200,000 URLs and 64 MiB, and crt.sh's answer is read up to 64 MiB; when a source reaches a limit its names are still used, a note says some hosts may be missing, and its entry is marked `truncated`. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
459459

460460
### Zone transfers: the whole zone in one query
461461

‎internal/ct/ct.go‎

Lines changed: 71 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ package ct
77
import (
88
"context"
99
"encoding/json"
10+
"errors"
1011
"fmt"
1112
"io"
1213
"net/http"
@@ -78,29 +79,83 @@ func (c Client) Names(ctx context.Context, domain string) ([]string, error) {
7879
return nil, err
7980
}
8081
defer func() { _ = resp.Body.Close() }()
81-
var entries []entry
82-
if err := json.NewDecoder(io.LimitReader(resp.Body, maxBody)).Decode(&entries); err != nil {
82+
body := &limitReader{r: resp.Body, n: maxBody}
83+
set := map[string]bool{}
84+
if err := decodeEntries(json.NewDecoder(body), set, domain); err != nil {
85+
if body.hit {
86+
// The array is cut off at the limit: keep the entries before the
87+
// cut instead of losing them all (#251).
88+
return sortedNames(set), fmt.Errorf("%w: the answer reached the %d MiB read limit", ErrTruncated, maxBody>>20)
89+
}
8390
return nil, fmt.Errorf("certificate transparency lookup: reading %s's answer: %w", resp.Request.URL.Host, err)
8491
}
85-
return relativeNames(entries, domain), nil
92+
return sortedNames(set), nil
8693
}
8794

88-
func relativeNames(entries []entry, domain string) []string {
95+
// ErrTruncated is returned, wrapped, with the names read before the answer
96+
// reached its read limit: they are good, but some hosts may be missing.
97+
var ErrTruncated = errors.New("certificate transparency answer truncated")
98+
99+
// decodeEntries streams crt.sh's JSON array one entry at a time, adding each
100+
// entry's names to set, so memory follows the names, not the body (#251).
101+
func decodeEntries(dec *json.Decoder, set map[string]bool, domain string) error {
102+
tok, err := dec.Token()
103+
if err != nil {
104+
return err
105+
}
106+
if d, ok := tok.(json.Delim); !ok || d != '[' {
107+
return fmt.Errorf("want a JSON array, got %v", tok)
108+
}
89109
domain = strings.ToLower(strings.TrimSuffix(domain, "."))
90-
set := map[string]bool{}
91-
for _, e := range entries {
92-
for _, name := range strings.Split(e.NameValue, "\n") {
93-
name = strings.ToLower(strings.TrimSuffix(strings.TrimSpace(name), "."))
94-
name = strings.TrimPrefix(name, "*.")
95-
rel, ok := strings.CutSuffix(name, "."+domain)
96-
if !ok || rel == "" {
97-
continue
98-
}
99-
if norm, ok := wordlist.Normalize(rel); ok {
100-
set[norm] = true
101-
}
110+
for dec.More() {
111+
var e entry
112+
if err := dec.Decode(&e); err != nil {
113+
return err
114+
}
115+
addNames(set, e, domain)
116+
}
117+
_, err = dec.Token() // the closing ']'
118+
return err
119+
}
120+
121+
// limitReader is io.LimitReader that records whether the limit cut the
122+
// body short.
123+
type limitReader struct {
124+
r io.Reader
125+
n int64
126+
hit bool
127+
}
128+
129+
func (l *limitReader) Read(p []byte) (int, error) {
130+
if l.n <= 0 {
131+
l.hit = true
132+
return 0, io.EOF
133+
}
134+
if int64(len(p)) > l.n {
135+
p = p[:l.n]
136+
}
137+
n, err := l.r.Read(p)
138+
l.n -= int64(n)
139+
return n, err
140+
}
141+
142+
// addNames adds e's names that fall under domain to set, relative to it.
143+
// Wildcard entries give their parent ("*.dev" gives "dev").
144+
func addNames(set map[string]bool, e entry, domain string) {
145+
for _, name := range strings.Split(e.NameValue, "\n") {
146+
name = strings.ToLower(strings.TrimSuffix(strings.TrimSpace(name), "."))
147+
name = strings.TrimPrefix(name, "*.")
148+
rel, ok := strings.CutSuffix(name, "."+domain)
149+
if !ok || rel == "" {
150+
continue
151+
}
152+
if norm, ok := wordlist.Normalize(rel); ok {
153+
set[norm] = true
102154
}
103155
}
156+
}
157+
158+
func sortedNames(set map[string]bool) []string {
104159
out := make([]string, 0, len(set))
105160
for n := range set {
106161
out = append(out, n)

‎internal/ct/ct_test.go‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -123,10 +123,11 @@ func TestNamesMalformedJSON(t *testing.T) {
123123
}
124124

125125
func TestNamesOversizedBody(t *testing.T) {
126-
// A valid JSON array longer than maxBody: the cut-off must surface as an
127-
// error, not as a silently truncated name list.
126+
// A valid JSON array longer than maxBody (#251): the entries before the
127+
// cut-off are kept, and the cut-off is reported as ErrTruncated rather
128+
// than losing every name or passing silently.
128129
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
129-
_, _ = io.WriteString(w, `[{"name_value": "a.example.com"},{"name_value": "`)
130+
_, _ = io.WriteString(w, `[{"name_value": "a.example.com"},{"name_value": "b.example.com"},{"name_value": "`)
130131
chunk := strings.Repeat("x", 1<<20)
131132
for written := 0; written <= maxBody; written += len(chunk) {
132133
if _, err := io.WriteString(w, chunk); err != nil {
@@ -138,11 +139,22 @@ func TestNamesOversizedBody(t *testing.T) {
138139
defer srv.Close()
139140

140141
got, err := names(context.Background(), srv, 0)
141-
if err == nil {
142-
t.Fatalf("an oversized answer was accepted: %d names", len(got))
142+
if !errors.Is(err, ErrTruncated) {
143+
t.Errorf("error = %v, want one wrapping ErrTruncated", err)
144+
}
145+
if want := []string{"a", "b"}; !slices.Equal(got, want) {
146+
t.Errorf("Names = %v, want the names before the cut-off %v", got, want)
143147
}
144-
if !errors.Is(err, io.ErrUnexpectedEOF) {
145-
t.Errorf("error = %v, want one wrapping io.ErrUnexpectedEOF", err)
148+
}
149+
150+
func TestNamesNotAnArray(t *testing.T) {
151+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
152+
_, _ = w.Write([]byte(`{"name_value": "a.example.com"}`))
153+
}))
154+
defer srv.Close()
155+
156+
if got, err := names(context.Background(), srv, 0); err == nil || errors.Is(err, ErrTruncated) {
157+
t.Errorf("a JSON object: names %v, err %v; want a non-truncation error", got, err)
146158
}
147159
}
148160

‎internal/passive/passive.go‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ package passive
66

77
import (
88
"context"
9+
"errors"
910
"fmt"
1011
"slices"
1112
"strings"
@@ -76,9 +77,22 @@ func ParseList(s string) ([]string, error) {
7677
func New(name string, opts Options) (Source, error) {
7778
switch name {
7879
case "ct":
79-
return ct.Client{BaseURL: opts.BaseURLs["ct"], UserAgent: opts.UserAgent}, nil
80+
return ctSource{ct.Client{BaseURL: opts.BaseURLs["ct"], UserAgent: opts.UserAgent}}, nil
8081
case "wayback":
8182
return Wayback{BaseURL: opts.BaseURLs["wayback"], UserAgent: opts.UserAgent}, nil
8283
}
8384
return nil, fmt.Errorf("unknown source %q", name)
8485
}
86+
87+
// ctSource reports a crt.sh answer cut off at its read limit as Truncated,
88+
// with the names read before the cut (#251).
89+
type ctSource struct{ ct.Client }
90+
91+
func (s ctSource) Names(ctx context.Context, domain string) ([]string, error) {
92+
names, err := s.Client.Names(ctx, domain)
93+
if errors.Is(err, ct.ErrTruncated) {
94+
_, why, _ := strings.Cut(err.Error(), ": ")
95+
return names, &Truncated{Source: "ct", Why: why}
96+
}
97+
return names, err
98+
}

‎internal/passive/passive_test.go‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,30 @@ func TestWaybackTruncated(t *testing.T) {
106106
}
107107
}
108108

109+
// TestCTTruncated covers #251: a crt.sh answer cut off at the read limit
110+
// gives the names before the cut with a Truncated error.
111+
func TestCTTruncated(t *testing.T) {
112+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
113+
_, _ = w.Write([]byte(`[{"name_value": "a.example.com"},{"name_value": "`))
114+
chunk := strings.Repeat("x", 1<<20)
115+
for range 65 { // past crt.sh's 64 MiB read limit
116+
if _, err := w.Write([]byte(chunk)); err != nil {
117+
return // the client stopped reading
118+
}
119+
}
120+
}))
121+
defer srv.Close()
122+
src, err := New("ct", Options{BaseURLs: map[string]string{"ct": srv.URL + "/"}})
123+
if err != nil {
124+
t.Fatal(err)
125+
}
126+
names, err := src.Names(context.Background(), "example.com")
127+
var tr *Truncated
128+
if !errors.As(err, &tr) || tr.Source != "ct" || !strings.Contains(err.Error(), "read limit") || !slices.Equal(names, []string{"a"}) {
129+
t.Errorf("names %q, err %v; want [a] and a ct Truncated error naming the read limit", names, err)
130+
}
131+
}
132+
109133
func TestWaybackError(t *testing.T) {
110134
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
111135
w.WriteHeader(http.StatusServiceUnavailable)

‎pkg/subenum/sources.go‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ import (
66
"fmt"
77
"sync"
88

9-
"github.com/TMHSDigital/subenum/internal/ct"
109
"github.com/TMHSDigital/subenum/internal/passive"
1110
"github.com/TMHSDigital/subenum/internal/wordlist"
1211
)
@@ -26,7 +25,11 @@ const sourceUserAgent = "subenum-library (+https://github.com/TMHSDigital/subenu
2625
// Certificate Transparency logs, through crt.sh. One HTTPS request per scan,
2726
// no API key.
2827
func CertificateTransparency() Source {
29-
return ct.Client{UserAgent: sourceUserAgent}
28+
src, err := passive.New("ct", passive.Options{UserAgent: sourceUserAgent})
29+
if err != nil { // "ct" is a built-in source
30+
panic(err)
31+
}
32+
return src
3033
}
3134

3235
// WaybackMachine returns the source behind the CLI's -sources wayback:

0 commit comments

Comments
 (0)