You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bb98c4c
Browse filesBrowse the repository at this point in the historyBrowse files
Stream crt.sh answers and keep names before the read limit (#251)
The crt.sh response was decoded as one JSON array, so memory scaled
with the whole body and an answer that hit the 64 MiB cap failed with
unexpected EOF, losing every CT name. Decode the array one entry at a
time into the name set instead. When the limit cuts the body short,
return the names read so far with ct.ErrTruncated; the passive registry
turns that into a Truncated error, so the CLI, report and library treat
it like a truncated Wayback answer. pkg's CertificateTransparency now
builds its source through passive.New to get the same handling.
Fixes#251
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
35
35
- A name's record types are looked up at the same time instead of one after another, so with the default `A,AAAA` (or more types) a name costs its slowest answer rather than the sum of them, and a timed-out A no longer delays AAAA by the whole `-timeout`. `-rate` still caps the queries per second; results keep the order of `-type` (#250).
36
36
- Passive sources (`-sources`, `-ct`, `Config.Sources`) retry a busy answer (429, 502, 503, 504) or a dropped connection twice, after 2 and 5 seconds or the server's `Retry-After` (at most 30 seconds), within each source's 60-second limit; crt.sh often answers 502 or 503 under load, which used to cost the scan all of its names. A target's sources are now asked at the same time instead of one after another (#271).
37
37
- When the Wayback Machine's answer reaches the 200,000-URL or 64 MiB limit, a note says its results were truncated and some hosts may be missing, the report's `sources` entry (and the MCP report's) is marked `truncated`, `explain_stats` mentions it, and the library sends a `NoticeSource` event; the names it did return are still scanned. It used to stop silently (#272).
38
+
- crt.sh's answer is read one entry at a time instead of decoded whole, so memory follows the names rather than the body. An answer that reaches the 64 MiB limit keeps the names read before the cut and is reported as truncated, like Wayback's, instead of failing and losing every CT name (#251).
38
39
- The TUI runs on Bubble Tea v2: `charm.land/bubbletea/v2` v2.1.1, `bubbles/v2` v2.2.1 and `lipgloss/v2` v2.0.6 replace the v1 modules, dropping v1-only helpers that had not had a release in years. Keys, screens and the form work as before; the `go 1.26.0` minimum is unchanged (#188).
39
40
- With text output (the default) and no `-show-records` or `-diff`, the CLI keeps only the names of the current target's results for `-permute` and the `-resume` state, plus the few results with a takeover hint, an internal address or a permutation, instead of every result with its records. A simulated 1M-name scan at 100% hits peaks at about 240 MB instead of 750 MB, against 233 MB at 1% hits. A resumed run re-emits the same text. Structured formats still keep the records a resumed run writes back (#183).
40
41
- The scan dispatcher hands jobs to workers in small chunks (up to 8, fewer when there are few names per worker) over buffered channels, and each worker reports a chunk's jobs at once, instead of two unbuffered hand-offs per job. A 1M-name simulated scan at 50 workers (`make bench`) takes about 0.4 s instead of 1.6 s. A resumed scan may repeat up to one chunk per worker instead of one lookup per worker (#184).
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -455,7 +455,7 @@ The run-quality report counts them per target (`ct_names`). If crt.sh is down or
455
455
subenum -sources ct,wayback example.com
456
456
```
457
457
458
-
The sources of a target are asked at the same time, and a busy answer (429, 502, 503, 504) or a dropped connection is retried twice with a short backoff, honouring `Retry-After`, within the source's 60-second limit. The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. The Wayback Machine is asked for at most 200,000 URLs and 64 MiB; when it reaches either limit its names are still used, a note says some hosts may be missing, and its entry is marked `truncated`. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
458
+
The sources of a target are asked at the same time, and a busy answer (429, 502, 503, 504) or a dropped connection is retried twice with a short backoff, honouring `Retry-After`, within the source's 60-second limit. The report has one `sources` entry per source, with the names it added or the `error` it hit; a failed source is a warning, and the scan goes on with the others. The Wayback Machine is asked for at most 200,000 URLs and 64 MiB, and crt.sh's answer is read up to 64 MiB; when a source reaches a limit its names are still used, a note says some hosts may be missing, and its entry is marked `truncated`. Simulation ignores `-sources`, lab mode refuses it, and a resumed scan reuses what each source returned.
0 commit comments