diff --git a/CHANGELOG.md b/CHANGELOG.md index 256a939..3c8ae4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,7 +41,7 @@ layout survives. `.env` loading with documented precedence (`env_config.py`), captcha detection and solving (`captcha_solver.py`), 2Captcha fingerprints (`fingerprint_client.py`) and a two-run differ (`diff_runs.py`). -- **520 offline checks** (`smoke_test.py`, wrapped for `pytest`), with every +- **525 offline checks** (`smoke_test.py`, wrapped for `pytest`), with every fixture cut from a real capture by `make_fixtures.py` and verified to parse identically to the untrimmed original before being committed. - **CI**: the offline suite on the oldest and newest supported Python, a @@ -87,6 +87,15 @@ UAE residential exit. row the DOM fallback produced. - **No scroll loop**, measured: captures taken with no scrolling at all carried the same counts as captures taken with four scroll rounds. +- **The captcha is Google reCAPTCHA, and it guards the login flow, not the + listings.** Found by reading the site's own bundles rather than by guessing + from a vendor list: its key is `RECAPTCHA_KEY` in the argument object + handed to `showAuthPopup(...)`, called with `intent: "login"` or + `"phoneverify"`. Nothing this scraper fetches carries the loader, so the + v2-vs-v3 question is deliberately left unanswered rather than guessed. The + key is a marker — it appears in 0 of 17 captures, so its presence means the + site rendered its own challenge — while the empty + `` mount point on every page deliberately is not. - **`dubizzle.com.bh`, `.om` and `.eg` are refused with the reason** — they carry the same brand and run the OLX platform, which publishes no `__NEXT_DATA__` and a different DOM. `dubizzle.com.lb` redirects to diff --git a/README.md b/README.md index 1fd5f05..9a4e704 100644 --- a/README.md +++ b/README.md @@ -355,15 +355,33 @@ actually buy here: * **Proxies** (`--proxy`, `--proxy-file`) — the same exit, your own browser. Use `region-ae`. A bigger pool of non-UAE addresses buys nothing. * **Captcha solving** (`--solve-captcha`, default `when-blocked`) — nothing - to solve here today, and that is a measurement rather than an assumption. - Across 15 captures this site rendered no reCAPTCHA, hCaptcha, Turnstile or - DataDome challenge to an anonymous visitor. But it **does** ship its own - captcha mount point — ``, empty, on - every page including the good ones — so the right question is not "did we - meet one" but "is one wired up, and would we recognise it". The detectors - cover the shapes the site's own widget would take, and the bare tag is - deliberately **not** a marker: a marker that matches every good page is - worse than no marker. + to solve here, and that is measured rather than assumed. Across 17 captures + and 5 live runs this site rendered no challenge at all to an anonymous + visitor, and Imperva does not challenge either: it refuses outright, with + no widget and no form on either refusal page. + + It **does** have one, though, and reading the site's own bundles says + exactly which. **Google reCAPTCHA**, site key + `6LeubLYqAAAAAK7-X6nc1fW2ggot_vTvQAv0RxdU`, which appears in dubizzle's + JavaScript as `RECAPTCHA_KEY` in the argument object handed to + `showAuthPopup(...)` — beside `GOOGLE_APP_ID` and `FACEBOOK_APP_ID` — and + as `recaptchaSiteKey` in the app config. `showAuthPopup` is called with + `intent: "login"` or `"phoneverify"`, so the captcha guards **signing in + and verifying a phone number**, not reading listings. + + Two consequences worth stating. The version is **not** claimed here: none + of the site's 104 listing chunks carries the `recaptcha/api.js` loader or a + single `grecaptcha.*` call, so the `render=` parameter that settles v2 + against v3 is not observable from anything this scraper fetches — the + widget belongs to a separate auth application the listing pages never + load. And the key itself IS a marker, because it is safe to be one: it + appears in 0 of 17 captures, so seeing it in a page means the site has + rendered its own challenge into it. + + The empty mount point it would render into — + `` — is on every page including the good + ones, so the bare tag is deliberately **not** a marker: one that matches + every good page is worse than no marker. * **Fingerprints** (`--fingerprint`) — a consistent device identity. Pass **one** OS-family tag to `--fp-tags` (`Windows`, `Microsoft Windows` or `Android`); the API rejects a list, and `Chrome`, `Desktop` and `Mobile` @@ -412,7 +430,7 @@ All 2026-09-14, through a UAE residential exit unless stated. | Asset-host references, served page vs refusal | 126–2,571 vs **0** | | Scroll rounds needed | none — captures taken with 0 scrolls matched those taken with 4 | | Block-page shapes seen | 1,160 B (HTTP 403) and 6,183 B (HTTP **200**) | -| Offline checks | 520 | +| Offline checks | 525 | Everything except the DOM cross-check comes out of `__NEXT_DATA__`, which is in the first response and needs no JavaScript, so a readiness wait that times @@ -464,7 +482,7 @@ to compare, not the positions. ## Testing ```bash -python3 smoke_test.py # 520 offline checks, no engine library needed +python3 smoke_test.py # 525 offline checks, no engine library needed pytest # the same checks, wrapped as one test python3 env_config.py # what config was picked up, without secrets python3 .github/ci_checks.py --all # what CI runs diff --git a/product_parser.py b/product_parser.py index a83088a..bcc3517 100644 --- a/product_parser.py +++ b/product_parser.py @@ -749,10 +749,38 @@ def price_in(text: str) -> Optional[float]: "challenges.cloudflare.com/turnstile", "_Incapsula_Resource?SWCGHOEL", "Incapsula incident ID", + # dubizzle's OWN reCAPTCHA key (see RECAPTCHA_SITE_KEY below). Absent + # from 17 of 17 captures, so its appearance means the site rendered its + # own challenge into the page rather than that we recognised a vendor. + "6LeubLYqAAAAAK7-X6nc1fW2ggot_vTvQAv0RxdU", ) -# The site ships an EMPTY mount point on every page it serves — -# ``, 1 occurrence on all 15 captures, +# WHICH CAPTCHA THIS SITE ACTUALLY USES, found by reading its own bundles +# rather than by guessing from a vendor list (§18). +# +# Google reCAPTCHA. The key below is dubizzle's, and it is in the site's own +# JavaScript twice — as `RECAPTCHA_KEY` in the argument object handed to +# `showAuthPopup(...)`, beside `GOOGLE_APP_ID` and `FACEBOOK_APP_ID`, and as +# `recaptchaSiteKey` in the app config. `showAuthPopup` is called with +# `intent: "login"` or `"phoneverify"`, which is the whole story: the captcha +# guards SIGNING IN and VERIFYING A PHONE NUMBER, not reading listings. +# +# That is why an anonymous listing scrape never meets it — 15 captures and 5 +# live runs, zero rendered challenges — and it is also why the version (v2 vs +# v3) is NOT claimed here: none of the site's 104 listing chunks contains the +# `recaptcha/api.js` loader or a single `grecaptcha.*` call. The widget is +# rendered by a separate auth application that the listing pages never load, +# so the loader's `render=` parameter, which is what settles v2 against v3 +# (§8), is not observable from anything this scraper fetches. +# +# Kept as a MARKER because it is safe to be one: the key appears in 0 of 17 +# captures, so its presence in a page's HTML means the site has rendered its +# own challenge into the page we were given. Checked before adding, which is +# the rule that keeps a marker from matching every good page. +RECAPTCHA_SITE_KEY = "6LeubLYqAAAAAK7-X6nc1fW2ggot_vTvQAv0RxdU" + +# The site also ships an EMPTY mount point on every page it serves — +# ``, 1 occurrence on all 17 captures, # good pages included. So the bare tag is a fact about the site and not a # marker (§18), and it is deliberately absent from the list above. What a # rendered challenge would look like is the same element with something diff --git a/smoke_test.py b/smoke_test.py index 44d8c56..4fda80d 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -2034,6 +2034,36 @@ def test_captcha(): not any(product_parser.captcha_mount_is_populated(fx(n)[0]) for n in ("motors_p1", "property_rent", "classified", "jobs", "community"))) + # WHICH captcha this site uses, found by reading its own bundles rather + # than by guessing from a vendor list (§18). Google reCAPTCHA, on the + # login / phone-verification flow, never on a listing. + # Against the raw tuple, not the lowercased set above: a reCAPTCHA key + # is case-sensitive and the page emits it verbatim, so the marker has to + # keep its case. + ok &= check("the site's own reCAPTCHA key is known and is a marker", + product_parser.RECAPTCHA_SITE_KEY.startswith("6L") + and product_parser.RECAPTCHA_SITE_KEY + in product_parser.BOT_CHALLENGE_MARKERS) + ok &= check("...and it is safe to be one: absent from every fixture, so " + "seeing it means the site rendered its own challenge", + not any(product_parser.RECAPTCHA_SITE_KEY in FIX[n]["html"] + for n in FIX)) + ok &= check("a page carrying it IS a challenge", + product_parser.detect_bot_challenge( + '
' + % product_parser.RECAPTCHA_SITE_KEY) is not None) + # The version is deliberately NOT claimed anywhere: the loader that would + # settle v2 against v3 lives in a separate auth application that a + # listing page never loads, so nothing this scraper fetches can observe + # it. Pinned so that nobody later writes a guess into the docs. + for name in ("README.md", "product_parser.py"): + text = open(os.path.join(REPO_ROOT, name), encoding="utf-8").read() + lowered = text.lower() + ok &= check("%s does not claim a reCAPTCHA version it cannot see" + % name, + "recaptcha v2" not in lowered + and "recaptcha v3" not in lowered) + # The extension's own hunters reference turnstile, arkoselabs and # recaptcha on EVERY page fetched over --cdp-endpoint: 3, 2 and 2 # occurrences on a good motors page, and 0 of each after the extension