You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1fcd5bc
Browse filesBrowse the repository at this point in the historyBrowse files
perf(scan): hand jobs out in chunks over buffered channels (#184)
Every job crossed two unbuffered channels (jobs and completed), so 50
workers spent most of a cheap scan in channel lock contention: a
1M-name simulated scan took about 1.6 s, 65% of it in semasleep and
lock2.
The dispatcher now admits up to maxChunk (8) jobs at a time, fewer
when there are few names left per worker so a short wordlist still
reaches every worker, and sends them as one slice over a buffered
jobs channel. A worker reports a chunk's completions as one slice
over a buffered completed channel, and still offers it after an
interrupt when there is room; on cancellation the dispatcher counts
the reports already buffered before computing the resume point.
Deadline skips, -max-queries and EventDone ordering are unchanged.
BenchmarkRun1M (simulate, 50 workers): about 0.4 s instead of 1.6 s.
A resumed scan may now repeat up to one chunk per worker.
Fixes#184
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,6 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
26
26
-`-diff-records` makes `-diff` compare the records of names found again, so a CNAME re-pointed at an unclaimed service or a moved address is reported as `changed` (`~ name` in text; `"change": "changed"` with `previous_records` in JSON and JSONL), counted in the report's `diff.changed`, and exits 4. When either side has a CNAME only the alias targets are compared, so CDN address rotation is not a change. The previous file must be JSON, JSONL or CSV. The GitHub Action gains a `changed` output (#189).
27
27
28
28
### Changed
29
+
- The scan dispatcher hands jobs to workers in small chunks (up to 8, fewer when there are few names per worker) over buffered channels, and each worker reports a chunk's jobs at once, instead of two unbuffered hand-offs per job. A 1M-name simulated scan at 50 workers (`make bench`) takes about 0.4 s instead of 1.6 s. A resumed scan may repeat up to one chunk per worker instead of one lookup per worker (#184).
29
30
- Live UDP scans reuse sockets instead of opening and closing one per query, which capped throughput near 10k queries per second with nearly all CPU in socket teardown. A 100k-name scan against a loopback server now takes about 2 s instead of 12 s at `-t 200`. A late answer to an earlier query on a reused socket is ignored, as before, and cannot make a lookup count as refused (#180).
30
31
-`-format json` streams its array instead of holding every result and then encoding the whole document, so its peak memory matches `-format jsonl` (a 300k-result run: about 200 MB instead of 420 MB). The output is byte for byte the same (#181).
31
32
- Tests no longer flake under load: lab end-to-end runs use a 5 s timeout with 3 attempts and fail with a clear message if the lab 2 wildcard probe times out, the resolver-pool spread test counts lookups per member instead of timing the scan (the timing view is now `BenchmarkPoolSpreadsLoad`), and CI runs tests in shuffled order (#167).
0 commit comments