Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 25 additions & 28 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,31 +12,28 @@ body:

A few things answer most reports faster than we can:

- **Everything exit 3?** Check whether the run was **headless** before
you check anything else. Akamai refuses a headless browser on this
site whatever the exit address — measured HTTP 403 from four
residential exits and one datacentre address, against HTTP 200 and
the full catalogue from those very same addresses with a real
window. `--headful` is the default here; a proxy does not fix it,
and `--cdp-endpoint` is the other path measured to work.
- **`price` null on some rows?** That is almost certainly correct:
every blank price is a lot whose reserve has not been met, and the
row carries `reserve_price_set: true` to say so. What is NOT
correct is a null price with no reserve flag — that one is worth
this issue.
- **Zero rows?** You may be on a page that has none: one page past the
end of a listing, or a search that matched nothing (that one answers
200 with "No results" **and 24 suggested lots**, which this scraper
deliberately does not parse). Both are exit 4 and correct answers.
See [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).
- **`rating` and `review_count` empty on every row?** Expected: this
site publishes no per-lot rating. The seller's feedback is in
`seller_score` and `seller_feedback_count` on a `--mode lot` row.
- **The run stopped at page 100?** That is the site's own cap, not a
bug. Past it the site returns page 100's own lots under HTTP 200.
- **Wrong language?** The locale is a path segment (`/en/`, `/nl/`,
`/de/`), not something the exit IP decides. Prices are in euro on
all 18 locales.
- **Everything exit 3?** That is Expedia's "Bot or Not?" handler
(HTTP 429) refusing your exit address, and the scoring is noisy:
the same address serves some requests and challenges others
minutes apart. Slow down (`--delay 10`), move to another exit
(`--proxy` / `--proxy-file`), or wait. The log names the vendor
the handler picked, e.g. `blocked (datadome-challenge)`; only a
reCAPTCHA pick is ever offered to the solver.
- **Exit 6 after page 1?** Look at `stop_reason` in the sidecar.
`next_page_throttled` means the page turn (`POST /graphql`) was
rate limited while the HTML kept answering 200; `cards_missing`
means a page delivered fewer cards than its own counter states.
- **Every price changed overnight?** Check `stay_dates`. A search
with no dates in the URL prices each property on its own cheapest
night; pin `startDate`/`endDate` for comparable prices.
- **`image_url` or `amenities` empty on most rows?** Expected: the
gallery is not mounted below the fold, and the amenity strip is an
A/B variant of the card.
- **Zero rows?** See
[TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) first.
- **Wrong currency or language?** The storefront is the hostname
(`vrbo.com`, `fewo-direkt.de`, `abritel.fr`, `bookabach.co.nz`,
`stayz.com.au`), not something the exit IP decides.

- type: dropdown
id: engine
Expand Down Expand Up @@ -84,8 +81,8 @@ body:
render: shell
placeholder: |
python3 playwright_scraper.py \
--url "https://www.vrbo.com/en/c/333-watches" \
--pages 3 --out grills
--url "https://www.vrbo.com/search?destination=Orlando,%20Florida,%20United%20States%20of%20America" \
--pages 2 --out orlando
validations:
required: true

Expand All @@ -105,7 +102,7 @@ body:
attributes:
label: What you expected instead
placeholder: >-
96 products, as the README says a category page yields.
50 properties per page, as the README says a search page holds.
validations:
required: true

Expand Down
62 changes: 26 additions & 36 deletions .github/ISSUE_TEMPLATE/site_changed.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,36 +9,28 @@ body:
its own report because the fix is different from a code bug:
something on vrbo.com moved.

Useful to know before filing, because it narrows the fix a lot — this
parser reads **two different sources for two halves of a row**:

- **The rows** come from the site's own SSR payload,
`<script id="__NEXT_DATA__">`, which is in the first response and
needs no JavaScript. Titles, ids, urls, images, auction ids and the
reserve flag all come from there.
- **The money** comes from the rendered card
(`.c-lot-card__price` and its siblings), because the payload does
not carry it: bids arrive at hydration.

So "no rows at all" and "rows with no prices" are two different
breakages with two different fixes, and `price_source` on the rows
tells you which happened: `next_data+dom` is healthy, `next_data`
means nothing hydrated.
Useful to know before filing, because it narrows the fix a lot:
there is **no structured data on a listing page**, so every column
comes from the rendered card, anchored on Vrbo's own `data-stid`
attributes. The card is `[data-stid="lodging-card-responsive"]`,
and the price container has two spellings —
`data-stid="product-price-summary"` on vrbo.com and
`data-test-id="price-summary"` on the local storefronts — so say
which host you ran. `price_source` on the rows says which node was
read: `card` / `card-a11y` on a listing, `detail` / `detail-a11y`
on a property page.

- type: dropdown
id: what_broke
attributes:
label: What stopped matching
options:
- No rows at all on a page that visibly has lots (the payload moved)
- Rows come out with no prices (the card, or its price class, moved)
- bid_kind is null on rows that have a price (the status label moved)
- A null price with no reserve_price_set flag (the invariant broke)
- favorite_count empty on every row (the card's counter moved)
- Pagination stops early, repeats a page, or runs past the 100 cap
- A lot page returns nothing (--mode lot)
- The auctions index returns nothing (--mode auctions)
- A no-results search returns 24 suggested lots as if they were real
- No rows at all on a page that visibly has properties (the card moved)
- Rows come out with no prices (the price container moved)
- Prices missing on one storefront only (a new spelling of the container)
- Fewer rows than the page's own counter says (cards_missing)
- Pagination stops early or repeats a page (the next button moved)
- A property page returns nothing (--mode property)
- Something else
validations:
required: true
Expand All @@ -48,10 +40,9 @@ body:
attributes:
label: URL
description: >-
The exact URL, its locale prefix, and how the browser was reached.
Please say whether the run was headful or over --cdp-endpoint: a
headless run is refused by the site and is not a parser problem.
placeholder: "https://www.vrbo.com/en/c/333-watches (locale: /en/, headful local Chrome, exit: NL residential)"
The exact URL, which storefront, how the browser was reached
(local headful, headless, or --cdp-endpoint) and from which exit.
placeholder: "https://www.vrbo.com/search?destination=... (storefront: vrbo.com, headful local Chrome, exit: BE residential)"
validations:
required: true

Expand All @@ -63,12 +54,11 @@ body:
Whichever of these you can get. A capture beats a description, and
`--dump-html` writes one on success as well as on failure.

- The payload as the page actually serves it:
`python3 -c "import json,re;h=open('dump.html').read();d=json.loads(re.search(r'id=.__NEXT_DATA__.[^>]*>(.*?)</script>',h,1).group(1));print(sorted(d['props']['pageProps']))"`
- For a price problem: one whole `article.c-lot-card__container` from
devtools, and the amount the page shows for that lot.
- For a `bid_kind` problem: the exact label the card prints
("Current bid", "Aktuelles Gebot", ...) and the locale.
- For a price problem: one whole
`[data-stid="lodging-card-responsive"]` card from devtools, and
the amount the page shows for that property.
- For a missing-rows problem: the counter text
(`1 - 50 of 300+`) and `cards_missing` from the sidecar.
render: text
validations:
required: true
Expand All @@ -78,5 +68,5 @@ body:
attributes:
label: Anything else
description: >-
Whether it reproduces on a second engine, whether it reproduces in a
second locale, and when it last worked.
Whether it reproduces on a second engine, whether it reproduces on a
second storefront, and when it last worked.
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
# every CI job died with FileNotFoundError while the whole suite was green
# locally. Exactly the class of thing only a clean checkout finds.
!fixtures_generated.json
catawiki_products.*
catawiki_products_*.*
vrbo_products.*
vrbo_products_*.*

# --- the live-test harness's working directory ---------------------------
live_results/
Expand Down
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,37 @@ that is a fix, and the release notes lead with it.
logged "Upstream page status success". `status` is now a fallback only
when it is an integer.

### Docs — leftovers from the repos this one was copied from

Text describing another site as if it were Vrbo, removed or rewritten from
this repo's own README, TROUBLESHOOTING and parser:

- **`captcha_solver.py`** said no challenge had ever been observed here and
that a block was a headless browser refused by Akamai with a 394-byte
"Access Denied". Both were another site's. It now says what the README
measured: Expedia's "Bot or Not?" handler (HTTP 429) picked DataDome on
every recorded refusal, and there is deliberately no DataDome path. The
"No DataDome solver" section the docstring pointed at now exists.
- **`CONTRIBUTING.md`**: the site-change anchors (a different site's
`data-testid` grid and tile markers), the list of pinned properties
(auction lots, reserve prices, bid counts, a 100-page cap) and the
`akamai` marker note (which contradicted this repo's own measurement)
are rewritten for Vrbo. Also corrected: the canary note that the browser
build is the gate, and a "datacentre gets no response at all" claim.
- **Issue templates**: the bug-report hints and the site-change template
described another site's payload, lots and locales; both now describe
Vrbo's cards, storefronts and `stop_reason`s. Example commands use a Vrbo
search URL.
- **`diff_runs.py`**: example filenames, a `sold`/`sold_is_floor` note for a
column this repo does not have, and a docstring listing
`original_price`/`discount_pct`/`in_stock` and JSON-LD provenance. The
docstring now lists the tracked fields and the `stay_changed` bucket.
- **`.gitignore`** ignored another repo's default output prefix; it now
ignores `vrbo_products.*`, the engines' real `--out` default.
- **`SECURITY.md`** named only Akamai as the protection, said the project
has no releases (v0.1.0 exists) and pointed at a README section that does
not exist.

## [0.1.0] — 2026-09-14

First release on this repo family's architecture. The previous contents of
Expand Down
Loading
Loading