Skip to content

Oracle: Chrome reference corpus with network record/replay and diff tool #2

Open

Description

@x-ror

Every later stage is measured against this corpus. Without it we cannot tell which domains Larva can take over.

Corpus

  • Take 100–200 URLs from the default JavaScript-rendering path of the crawling service: requests served by a plain headless Chrome.
  • Exclude domains already routed to special paths (headful Chrome, challenge solvers, real-user browsers, residential renderers).
  • Weight the selection by real volume (top domains), plus a long tail.
  • For each URL, store the request parameters the caller actually sends (waits, scroll, click selectors, per-domain defaults).

Capture: record once, replay forever

  • Run real Chrome through the existing production Chrome tooling, so the flags, stealth script and waits match production.
  • Record:
    • final outerHTML;
    • document status, final URL and response headers;
    • Network.getAllCookies;
    • the list of XHR/fetch requests with their bodies;
    • timings.
  • Also record a full network archive for every request (HAR-like: request, response headers and body bytes).
  • Larva stages 3–4 then run against the replayed network. That separates DOM/JS compatibility from network-fingerprint and ban noise.
  • Keep the corpus outside git, in object storage or a local dir; commit only the manifest.

Comparison (not byte equality)

  • Visible-text similarity of the visible DOM text.
  • Presence and values of the selectors the scrapers actually read (the per-domain extractors).
  • Element counts per tag; link and image sets.
  • Status code and cookie names.
  • The caller's response-health and bot-page checks must pass.

Deliverables

  • tools/oracle/: capture script, replay server, diff tool, and a per-domain report (pass / partial / fail with a reason).
  • This is the gate for stage 5: a domain moves to browser=larva only once it passes.

Activity

  1. x-ror commented on Nov 18, 2026

    @x-ror
    OwnerAuthor

    From #10: once the corpus exists, compare the two parser options on it — lexbor's tree as the DOM (what pkg/html does now, #68) against lexbor's tokenizer feeding our own tree builder — on parse time, RSS per page, and the effort the second would take. The pause and document.write are already solved without forking lexbor, so the comparison decides only who owns the tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions