anti-bot-doctor

Your scraper is getting blocked. This tool tells you why — and what to do about it.

anti-bot-doctor is a diagnostic CLI and library for scrapers that have stopped working. Point it at a URL and it fetches the page with a realistic browser header set, inspects the response, and returns a ranked list of findings. Each finding is a diagnosis: a symptom (what it saw), a likely cause, a recommended fix, and a link to a guide that walks through the fix in depth.

Think of it as a doctor for a sick scrape: symptom → diagnosis → prescription.

What it diagnoses

Diagnosis	Triggered by	Prescription
JavaScript rendering required	near-empty body, empty SPA mount point, __NEXT_DATA__/__NUXT__ with no content	render with Playwright/Selenium, or parse the embedded JSON
Cloudflare / Akamai / DataDome / HUMAN (PerimeterX) / Imperva / Kasada / AWS WAF	status + signature headers, cookies, and body markers	TLS impersonation, stealth headless, residential proxies
TLS / JA3 fingerprint block	a clean browser-like request still gets 403, no named product	impersonate a real browser's TLS with curl_cffi
Header / User-Agent block	(with --compare) a naive request is blocked but a browser-like one passes	send realistic headers, rotate User-Agents
Rate limiting	429, Retry-After, RateLimit-* headers	back off with retries, slow down, add proxies
Geo / IP block	451 or region-block body markers	proxies in the target country
Login / session gate	401 or a redirect to a login page	authenticate and reuse session cookies
robots.txt disallow	the path is disallowed	throttle, scrape only what you need, or ask permission
Honeypot / bot-trap	hidden links or trap form fields in the HTML	skip CSS-hidden elements, follow only visible links

If nothing fires, you get a clean bill of health.

How it differs from scrape-check

scrape-check is a planning tool: run it before you write code to profile what you'll be up against (robots.txt, anti-bot stack, rendering mode, a recommended strategy).

anti-bot-doctor is a diagnostic tool: run it after things go wrong, when your scraper is being blocked, to pin down the specific cause and get a concrete fix. The two are complementary — profile first, diagnose when a scrape breaks.

Neither tool executes JavaScript, solves challenges, or bypasses anything. They observe and explain.

Install

anti-bot-doctor is not published to PyPI — install it from source, straight from GitHub:

pip install git+https://github.com/python-web-scraping-com/anti-bot-doctor.git

Since it's a command-line tool, pipx is a good way to install it in an isolated environment:

pipx install git+https://github.com/python-web-scraping-com/anti-bot-doctor.git

Requires Python 3.10+.

Usage

anti-bot-doctor example.com
anti-bot-doctor https://www.example.com --compare      # add a naive-vs-browser diff
anti-bot-doctor example.com --json                     # machine-readable output
anti-bot-doctor example.com --proxy http://user:pass@host:port
anti-bot-doctor example.com --user-agent "MyBot/1.0" --timeout 20

The exit code reflects the worst finding, which is handy in CI or scripts:

Exit code	Meaning
0	looks scrapable
1	scrapable, but it needs work
2	hard block — plain HTTP won't get you in

Example diagnosis

╭─ anti-bot-doctor ───────────────────────────────────────────╮
│ https://shop.example.com                                    │
╰─────────────────────────────────────────────────────────────╯

                          Requests
                    browser        naive
 status             403            403
 server             cloudflare     cloudflare
 body               2,014 bytes    2,014 bytes

╭─ Finding 1 ─────────────────────────────────────────────────╮
│ [CRITICAL] Cloudflare detected                              │
│                                                             │
│ Symptom    HTTP 403 returned an interstitial challenge page │
│            carrying Cloudflare fingerprints.                │
│ Cause      Cloudflare served a bot challenge instead of the │
│            page.                                            │
│ Fix        Use curl_cffi to impersonate a real browser's    │
│            TLS, or drive a stealth headless browser, and    │
│            route through residential proxies.               │
│ Evidence   __cf_bm cookie set; cf-ray: 8a…; server:         │
│            cloudflare; body marker: 'challenges.cloudflare' │
│ Guide      https://python-web-scraping.com/advanced-…       │
╰─────────────────────────────────────────────────────────────╯
╭─ Diagnosis ─────────────────────────────────────────────────╮
│ Hard block — plain HTTP will not get you in                 │
│ exit code: 2                                                │
╰─────────────────────────────────────────────────────────────╯

Use as a library

from anti_bot_doctor import diagnose

report = diagnose("https://example.com", compare=True)

print(report.headline)        # "Looks scrapable"
print(report.exit_code)       # 0

for finding in report.findings:
    print(finding.severity.label, finding.title)
    print("  cause:", finding.cause)
    print("  fix:  ", finding.fix)
    print("  guide:", finding.guide_url)

report.to_dict()              # JSON-serialisable dict of the whole diagnosis

diagnose() returns a Diagnosis dataclass. Its .findings are ranked worst-first, and every Finding carries a .guide_url.

How it works

1. Fetches the target with a full, realistic Chrome header set (and, with --compare, a second bare "naive" request for the diff).
2. Fetches /robots.txt and checks whether your path is allowed.
3. Runs each diagnostic rule over the response(s) — headers, cookies, status, and body markers — with earlier rules (product/TLS detection) informing later ones.
4. Ranks the findings by severity and maps the worst one to an exit code.

Only a couple of real HTTP requests happen, and only at runtime — the test suite is fully offline (respx/mocked transports), so CI never touches a live site.

What it can't tell you

• It only loads the initial HTML. Content gated behind scroll, click, or a later XHR can look fine here and still block a real crawl.
• Products are matched from signatures. A missing detection doesn't prove a site is unprotected — the defense may be dormant for a single request.
• The TLS/JA3 diagnosis is an inference from an unattributed 403, not a real handshake capture. Treat it as a strong hint, not proof.

Going deeper

Once anti-bot-doctor has named the problem, these guides on python-web-scraping.com cover the fix:

• Advanced Scraping & Anti-bot Evasion — the hub for everything below.
• Bypassing Cloudflare & Akamai protections — for commercial bot walls.
• TLS & JA3 fingerprint evasion and using curl_cffi to impersonate browsers — when a clean request still gets a 403.
• Using Playwright for modern web automation and Mastering Selenium for dynamic websites — for JavaScript-rendered pages.
• Browser fingerprint & stealth configuration — for PerimeterX/HUMAN and behavioural defenses.
• Rotating proxies & managing IP blocks and best free & paid proxy providers — for geo/IP blocks.
• How to rotate user agents in Python — for header-based blocks.
• Retrying failed requests with tenacity — for rate limiting and backoff.
• Managing cookies & sessions — for login/session gates.
• How to scrape a static website without getting blocked — polite defaults and honeypot avoidance.

New here? Start with the Complete Guide to Python Web Scraping.

Development

git clone https://github.com/python-web-scraping-com/anti-bot-doctor.git
cd anti-bot-doctor
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
pytest

The whole suite runs offline against mocked HTTP — no network required.

License

MIT — see LICENSE. Copyright (c) 2026 python-web-scraping.com.