Move the anti-bot rules into antibot_rules.json, and stop counting Imperva's loader as a challenge

CHANGELOG.md

@@ -38,6 +38,15 @@ the ladder falling back to `compatibility` on the second handshake.
   JavaScript challenge, which no HTTP client passes, so the URL should go to a
   real browser rather than be retried
 - **`Response.attempts`** lists what the ladder tried, in order
+- Imperva's `_Incapsula_Resource` loader script no longer counts as a
+  challenge. Imperva appends it to pages it let through, so ordinary pages and
+  the origin's own error pages were judged `needs_browser`. Imperva's incident
+  page still counts
+- **The anti-bot rules are data.** What `protection` recognises now lives in
+  `antibot_rules.json` instead of code, and `blasthttp.set_antibot_rules(path)`
+  (`antibot::set_rules` in Rust) replaces them at runtime, so a rule can change
+  without a rebuild. `Vendor` is a name rather than a fixed list, so a rules
+  file can add a product
 - **TLS failures now say why.** The reason used to be destroyed in transit:
   hyper-util's error displays as the literal "client error (Connect)" for every
   connect-time failure, so a cipher mismatch, a rejected certificate, a DNS

README.md

@@ -26,7 +26,7 @@ cargo add blasthttp
 - **Rust performance** — async I/O, zero-copy where possible, and native concurrency give significant speed improvements over pure Python HTTP clients
 - **SSL cert info on every request** — extracts CN, SANs, emails, issuer, validity dates, and fingerprint during the TLS handshake that's already happening, eliminating the need for a separate connection to gather this information
 - **Connection profiles, and automatic shifting between them**: `modern` by default (11 cipher suites, TLS 1.2+), `compatibility` for servers that speak only RC4, 3DES, SEED, Camellia, anonymous DH or SSLv3, and `chrome` to imitate a browser's TLS, HTTP/2 and headers together. A handshake the peer cannot negotiate widens the offer; a refusal from a recognised protection product changes what the client claims to be. What works is remembered per host
-- **Says why it failed**: a blocked request reports which product blocked it, and a JavaScript challenge reports `needs_browser`, because no HTTP client passes that tier and retrying will not change it
+- **Says why it failed**: a blocked request reports which product blocked it, and a JavaScript challenge reports `needs_browser`, because no HTTP client passes that tier and retrying will not change it. The products and how to recognise them are rules in [`antibot_rules.json`](antibot_rules.json), and `blasthttp.set_antibot_rules(path)` swaps in your own at runtime
 - **No cert validation by default** — offensive-first: connects to self-signed, expired, and misconfigured TLS without extra config
 - **HTTP/2 support** — automatic via ALPN negotiation, falls back to HTTP/1.1
 - **Low-level primitives** — `RawConnection` for byte-level TCP/TLS I/O (bypasses HTTP framing entirely) and `blasthttp.h2` for manual H2 frame construction (includes a permissive HPACK encoder that emits bytes a strict encoder refuses, the building block for H2 smuggling / CRLF-injection tooling)

antibot_rules.json

@@ -0,0 +1,126 @@
+{
+  "refusal_statuses": [401, 403, 405, 406, 429, 503],
+  "rules": [
+    {
+      "vendor": "cloudflare",
+      "outcome": "challenge",
+      "when": {"header_contains": {"name": "cf-mitigated", "value": "challenge"}},
+      "why": "The unambiguous one, and the only header Cloudflare sets specifically to say it intervened."
+    },
+    {
+      "vendor": "cloudflare",
+      "outcome": "challenge",
+      "when": {"any": [
+        {"body_contains": "/cdn-cgi/challenge-platform/"},
+        {"body_contains": "window._cf_chl_opt"}
+      ]},
+      "why": "The challenge page's own markers, for when cf-mitigated is missing."
+    },
+    {
+      "vendor": "akamai",
+      "outcome": "blocked",
+      "when": {"any": [
+        {"body_contains": "edgesuite"},
+        {"all": [
+          {"any": [
+            {"header_contains": {"name": "server", "value": "akamaighost"}},
+            {"header_present": "akamai-grn"},
+            {"header_present": "x-akam-sw-version"},
+            {"header_present": "x-akamai-transformed"},
+            {"header_contains": {"name": "server-timing", "value": "ak_p"}}
+          ]},
+          {"body_contains": "Access Denied"}
+        ]}
+      ]},
+      "why": "The edge error page. It HTML-entity-encodes its own punctuation, so the body carries errors.edgesuite.net and a search for errors.edgesuite.net never fires; the bare label edgesuite survives that. server: AkamaiGHost is not reliably present either: www.akamai.com's own 403 sends no server header and identifies itself through akamai-grn, x-akam-sw-version and an ak_p server-timing entry instead, so any one of the edge headers will do. Reference # on its own stays insufficient, because plenty of unrelated error pages carry one."
+    },
+    {
+      "vendor": "akamai",
+      "outcome": "challenge",
+      "when": {"all": [
+        {"status": [428]},
+        {"any": [{"cookie_set": "sec_cpt"}, {"body_contains": "sec-cpt"}]}
+      ]},
+      "why": "sec-cpt is an adaptive challenge that fires after _abck already passed, and it answers 428 rather than 403. Its own outcome, not a block."
+    },
+    {
+      "vendor": "datadome",
+      "outcome": "challenge",
+      "when": {"body_contains": "geo.captcha-delivery.com"}
+    },
+    {
+      "vendor": "perimeterx",
+      "outcome": "challenge",
+      "when": {"any": [{"body_contains": "/px/captcha"}, {"body_contains": "px-captcha"}]}
+    },
+    {
+      "vendor": "imperva",
+      "outcome": "challenge",
+      "when": {"any": [
+        {"body_contains": "_Incapsula_Resource?CWUDNSAI="},
+        {"all": [
+          {"body_contains": "_Incapsula_Resource"},
+          {"not": {"body_contains": "&ns=1&cb="}}
+        ]}
+      ]},
+      "why": "The &ns=1&cb= script is the loader Imperva adds to pages it let through, the origin's own error pages included, so on its own it's not an interception. Imperva's incident page frames itself as CWUDNSAI."
+    },
+    {
+      "vendor": "awswaf",
+      "outcome": "challenge",
+      "when": {"header_present": "x-amzn-waf-action"},
+      "why": "The header carries the action, and there is more than one: challenge for the silent JavaScript kind, captcha for the kind wanting a person. Matching only challenge missed wizzair.com, which answers 405 with captcha, so the ladder read a genuine interception as an ordinary refusal and declined to try anything else. Match the header's presence and let the value be detail."
+    },
+    {
+      "vendor": "kasada",
+      "outcome": "blocked",
+      "when": {"all": [
+        {"any": [{"header_present": "x-kpsdk-ct"}, {"body_contains": "KPSDK"}]},
+        {"status": [429]}
+      ]},
+      "why": "Kasada answers 429 rather than 403 for a refusal."
+    },
+    {
+      "vendor": "kasada",
+      "outcome": "present",
+      "when": {"any": [{"header_present": "x-kpsdk-ct"}, {"body_contains": "KPSDK"}]}
+    },
+    {
+      "vendor": "cloudflare",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"any": [{"cookie_set": "__cf_bm"}, {"cookie_set": "cf_clearance"}]},
+      "why": "From here on, sightings of a product on a response nothing above matched. One that set its cookie while refusing us is still a block."
+    },
+    {
+      "vendor": "akamai",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"any": [{"cookie_set": "_abck"}, {"cookie_set": "ak_bmsc"}, {"cookie_set": "bm_sz"}]}
+    },
+    {
+      "vendor": "datadome",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"cookie_set": "datadome"}
+    },
+    {
+      "vendor": "perimeterx",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"cookie_prefix": "_px"}
+    },
+    {
+      "vendor": "imperva",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"any": [{"cookie_prefix": "visid_incap_"}, {"cookie_prefix": "incap_ses_"}]}
+    },
+    {
+      "vendor": "f5",
+      "outcome": "present",
+      "on_refusal": "blocked",
+      "when": {"header_contains": {"name": "server", "value": "volt-adc"}}
+    }
+  ]
+}