Repository navigation
feat(renderer): fresh impersonated session per hop attempt - #613
Conversation
|
Design note from follow-up research (claudedocs/research_impersonation_landscape_2026-10-11.md in the fork, not linked): the fresh-session-per-hop choice is correct for this tier's population, and it would be wrong for bulk. Recording the boundary here so it is not silently widened later. Anti-bot literature in 2026 treats identity churn from one IP as a bot signal in itself. Incogniton documents that a never-seen fingerprint is suspicious. Pre-HTML detection evaluates browser consistency above the handshake. Translated: rotating the impersonated identity per-request at high volume from one address tells the wall you are not a person. pywaze went fresh-per-request because Waze's wall punishes continuity, but that is the minority case. The split this PR implements is the safe subset: the hop is a cold retry that fires once per walled page, so fresh-per-attempt costs nothing and cannot create churn. The pin arm keeps the pooled client because deliberate bulk use is where identity consistency wins. If the pin ever grows per-host bulk crawling, the correct next step is a sticky per-host identity (client + cookie jar + slow TTL rotation), not widening fresh-per-attempt to the pin. Also pushed as a second commit: a deterministic sentinel for the preset's ClientHello shape (bd15a26), asserting the post-quantum key share and HTTP/2 fingerprint from a fingerprint-echo endpoint. This catches preset aging (a Chrome preset that stops carrying X25519MLKEM768 is detectable by post-quantum-aware walls) without depending on a wall firing. |
|
Hey @paoloantinori thanks for this beautiful pr, is it ready? or still working on it, asking cuz its still draft. so if its ready i can start to review and merge! |
|
Ready, thanks. The check failure is the rustup network flake on ubuntu-latest, not from these changes. Everything else is green. |
|
@paoloantinori thank you! will review shortly |
The hop is a cold retry path, so it now builds a use-once ImpersonatedFetcher per attempt: new TLS session, no pooled identity. A persistent impersonated session is exactly what lets a wall vendor build per-fingerprint reputation and quietly stop serving it, the failure pywaze paid to learn (eifinger/pywaze#110: a pinned impersonated session was re-blocked in six days; fastcrw#111 fixed it with a fresh session per fallback request). The pin arm keeps the warm pooled client, where pooling pays. REQUEST_PROXY stays honored fail-closed through the same with_proxy constructor. The builder is a factory field (type ImpersonatedHopFactory) so tests can inject a counter: hop_builds_a_fresh_session_per_attempt asserts exactly one construction per hop attempt, clean_site_builds_no_hop_session asserts none on clean sites. Gates: /simplify, code-review
Asserts the pinned Chrome149 preset carries the post-quantum key share (X25519MLKEM768) in both supported_groups and the actual key_share, plus an HTTP/2 fingerprint hash, by probing a fingerprint-echo endpoint (tls.peet.ws/api/all). A preset that claims Chrome 149 without the PQ share is detectable by post-quantum-aware walls before any HTTP traffic; when this test goes red the answer is a preset bump, not a code fix. Deterministic: reads the handshake shape, not a wall that may not fire. Gates: /simplify, code-review
- sentinel asserts X25519MLKEM768 inside the supported_groups and key_share entries separately, so an offered but unsent PQ share fails - cargo fmt on the sentinel (fmt-check was red) - freshness assert folded into fetch_auto_impersonated_still_blocked_continues_to_ladder via inject_impersonated's build counter; duplicate hop tests dropped - hop factory and client cache docs match behavior: same preset per hop, cache serves the pin arm only
6764e06 to
1b30a66
Compare
|
merged, thanks a lot @paoloantinori! the pywaze write-up and the hop vs pin split were really well reasoned. i pushed one top-up commit on your branch before landing:
it's heading to production now. thanks again for the careful work! |
2 participants
What
Each impersonated hop attempt now runs on a fresh, use-once client: new TLS session, no pooled identity. The pin arm keeps the warm pooled client.
Why
A pattern arrived from outside the project. Waze blocked the unofficial endpoint behind Home Assistant's Waze Travel Time on 2026-09-30, and pywaze's recovery documents a lesson this tier should absorb:
chrome136fingerprint. The wall had built reputation for that fingerprint.crw's hop currently fetches through one shared
wreq::Clientbuilt at startup (plus warm per-proxy clients). If a wall vendor tracks per-fingerprint reputation, a busy crw instance retrying the same walled host through that shared client accumulates reputation, and the tier stops winning for that host without any error.Honest caveat: this is insurance, not an observed bug here. The walls measured for this tier (the Amazon.it class) behave per-request intermittent and IP/rate-based; no fingerprint-reputation wall is in the dataset. Two supporting observations from the live runs while preparing this change: the wall has been seen to hit the impersonated client on isolated single requests (the direct live test failed once on a wall-shaped body and passed on retry, in the same run where the chain e2e passed), and preset aging is already handled by graceful fall-through plus the sentinel live tests. Still, the hardening costs nothing where it applies: the hop is by construction a cold retry path, so a per-attempt client costs one TLS handshake and no pooling benefit is lost.
The split matches the two documented roles of the module. The hop is a reactive retry, so freshness wins; the pin is deliberate bulk use, so pooling wins.
Shape
ImpersonatedHopFactorytype alias andimpersonated_hop_factoryfield onFallbackRenderer: production builds a freshImpersonatedFetcherper call, honoring REQUEST_PROXY fail-closed through the samewith_proxyconstructor the warm path uses.try_impersonated_hoptakes its fetcher from the factory instead of the shared lookup; the pin arm still usesimpersonated_fetcher_for_request(doc updated to say pin-only).hop_builds_a_fresh_session_per_attemptasserts exactly one construction per hop attempt (wall miss falls through to the ladder),clean_site_builds_no_hop_sessionasserts zero on clean sites. The eleven existing hop tests inject through a smallinject_impersonatedhelper that sets both the presence field and the factory.Verification
cargo test -p crw-renderer --features impersonated(796 passed, 4 ignored live), leancargo check -p crw-renderer(no wreq in the tree),cargo fmt --check, andcargo clippy -p crw-renderer --features impersonated,camoufox,cloak,cdp -- -D warningsall run locally. The four live#[ignore]tests run against the real Amazon wall (one single-attempt flake of the direct tier test on a walled request, passing on retry; the auto-chain e2e green in the same run).Marked draft because the design question deserves maintainer eyes: is per-fingerprint reputation a threat class worth hardening against now (this change), or should the tier stay on the pooled client until a wall is observed doing it (in which case closing this is also fine)?