log-crawl-analyzer

See how search & AI bots actually crawl you — and how much crawl budget they waste.

---

Search Console's crawl stats are a sampled summary. Your access log is the ground truth — every hit, every bot, every wasted request. Log-file analysis is the most underused technique in technical SEO mostly because the tooling is enterprise-priced. This is the free version.

Parses standard access logs (common/combined format, .gz supported), identifies 35 search / AI / SEO-tool crawlers — Googlebot family, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and friends — and reports:

• hits per bot and per family (search vs AI vs SEO tools)
• status breakdown per bot: how much budget dies on 404s, redirect hops, 5xx
• parameter-URL crawling (classic budget leak)
• most-crawled paths — is Googlebot spending its budget where your money pages are?

Crawl Budget Report  1,086 log lines · 585 bot hits
========================================================================
  Search engines: 505 · AI crawlers: 80

  Googlebot     425 hits   2xx 325 · 3xx 40 · 404 60 · params 25   waste 24%
  Bingbot        80 hits   2xx 80  · clean                          waste 0%
  GPTBot         50 hits   …
  ClaudeBot      30 hits   …

  ≈100 bot hits went to 404s/redirects/errors — crawl budget you can reclaim.

Usage

python3 log_crawl_analyzer.py access.log
python3 log_crawl_analyzer.py access.log.2.gz access.log.1.gz access.log   # rotations
python3 log_crawl_analyzer.py access.log --bot gptbot     # one bot family
python3 log_crawl_analyzer.py access.log --json           # dashboards

Try it instantly with the bundled fixture:

python3 log_crawl_analyzer.py fixtures/sample-access.log

Note: identification is by user-agent string, which can be spoofed. For forensic verification cross-check IPs against published ranges (Google & Bing publish theirs). For budget analysis, UA-level is what you want anyway.

License

MIT © SEO Pro Check · built by @seoprocheck.