Repository navigation
Replace implementation with async/http #11
Description
This should make link checking a lot faster: https://github.com/socketry/benchmark-http/blob/main/lib/benchmark/http/spider.rb
This should make link checking a lot faster: https://github.com/socketry/benchmark-http/blob/main/lib/benchmark/http/spider.rb
Activity
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
Add LinkChecker::Async::HTTP, a link checker based on async-http
dblock commentedon Aug 18, 2026
@ioquatix I implemented this in #19, Anecdotal results show
LinkChecker::Async::HTTPusing significantly less CPU time thanLinkChecker::Typhoeus::Hydra, while wall-clock time is comparable between the two and depends on timeouts and the responsiveness of the URLs being checked.ioquatix commentedon Aug 18, 2026
Did you use HTTP/2? It will unlock more concurrency.
dblock commentedon Aug 18, 2026
Added in #22.
http1 0.185039 0.091212 0.276251 ( 10.08s)
http2 0.138420 0.028928 0.167348 ( 10.08s)
Similar success/error/failure counts (network-dependent). HTTP/2 uses noticeably less CPU (system time ~3x lower) with comparable wall-clock time here — consistent with HTTP/2's connection multiplexing reducing overhead, though wall-clock is mostly gated by timeouts/latency of the URLs themselves rather than protocol choice.