Scraper API: send waitFor as an object, read the target status from http_code - #40
Merged
Merged
Conversation
…ttp_code
Measured 2026-09-23 against scraper.2captcha.com/tasks/sync: waitFor sent
as a JSON-encoded string is refused with HTTP 422 ("params.waitFor must be
an object") and still billed; as an object it answers 200. The response's
status field is the API's verdict ("success"); the target's HTTP code is
http_code. Adds a no-network regression check, controlled red against the
unfixed client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in
scraper_api_client.py, both measured 2026-09-23 againsthttps://scraper.2captcha.com/tasks/sync.1.
waitForwas sent as a JSON-encoded string. The live API now answers that with HTTP 422params.waitFor must be an object— and still bills the task ($0.0005). So every run with--wait-text/--wait-element/--wait-stateended in exit 5. Sent as an object it answers 200._build_wait_fornow returns a dict (Optional[dict]); it is logged withjson.dumps. The module and function docstrings that said it had to be a double-encoded string now say what was measured.2. The target status was read from
status. The response is{"status": "success", "http_code": <int>, "headers": …, "body": …}:statusis the API's own verdict string,http_codeis the target site's HTTP code. The client logged "Upstream page status success" and a target 403/503 was never visible. New_target_status()readshttp_code(int), falling back tostatusonly if that is an int.Regression check —
check_scraper_api_waitfor_object_and_http_codeinsmoke_test.py: drives the realfetch_htmlwithrequests.postmonkeypatched (no network), asserts the payload'swaitForis a dict for--wait-text, and that the logged upstream status is the int 403 from a fake{"status":"success","http_code":403}response.Control: with
scraper_api_client.pyrestored toorigin/mainin a scratch copy (edit asserted to change the file), the suite went RED with both of this check's messages and nothing else; with the fix it is green.Live: one call through the fixed client,
--wait-text '$',https://www.farfetch.com/shopping/kids/girls-clothing-4/items.aspx: API HTTP 200, upstream status200(int), 1,016,665 bytes, cost $0.0005. The returned HTML carried no challenge marker and parsed to 96 products.Not changed: the module docstring and
--helpstill say this engine does not work against farfetch.com's Akamai-protected category pages. The single live call above returned the full catalogue, but one call is an anecdote, not a re-measurement, so that claim is left for a separate change with more runs behind it. No version bump; CHANGELOG entry under[Unreleased].🤖 Generated with Claude Code