Skip to content

feat: schema-validated router labels via standard-schema#3748

Open
B4nan wants to merge 130 commits into
v4from
feat/typed-router-schema-validation
Open

feat: schema-validated router labels via standard-schema#3748
B4nan wants to merge 130 commits into
v4from
feat/typed-router-schema-validation

Conversation

@B4nan

@B4nan B4nan commented Jun 16, 2026

Copy link
Copy Markdown
Member

Makes the router's request label drive the type of request.userData end-to-end — from declaring routes, through the request handler, all the way to the methods that enqueue new requests. Targets v4.

The compile-time-only subset (typed route map for the router) was split out and already merged to master in #3747. This PR is the full v4 version: it adds runtime schema validation and propagates the route map to the crawler/context request methods.

1. Typed route map (compile-time)

Declare a label → userData map and pass it as the router's second type argument. Handlers get request.userData typed per label, and unknown labels are a compile error. Fully backwards compatible — the default route map is open (Record<string, …>), so existing untyped usage and the legacy flat-userData generic keep working.

interface Routes {
    PRODUCT: { sku: string; price: number };
    CATEGORY: { categoryId: string };
}

const router = createCheerioRouter<CheerioCrawlingContext, Routes>();

router.addHandler('PRODUCT', async ({ request }) => {
    request.userData.sku; // string
    request.userData.price; // number
});

router.addHandler('TYPO', async () => {}); // ❌ not a known label

The default handler (addDefaultHandler) is a fallback for any request, so its userData stays loosely typed.

2. Schema-validated labels (types + runtime validation)

Passing a per-label Standard Schema (Zod, Valibot, ArkType, …) both infers the userData types and validates them at runtime before the handler runs, replacing request.userData with the parsed value. Invalid userData throws a new non-retryable RequestValidationError (the data won't change between attempts) carrying the schema issues.

import { z } from 'zod';

const router = createCheerioRouter({
    PRODUCT: z.object({ sku: z.string(), price: z.coerce.number() }),
    CATEGORY: z.object({ categoryId: z.string() }),
});

router.addHandler('PRODUCT', async ({ request }) => {
    request.userData.price; // number — inferred from the schema, coerced/validated at runtime
});

3. Propagation to crawler & context request methods

When a typed router is used, the route map now also types the request inputs: providing a declared label requires the matching userData shape (and rejects unknown labels); unlabeled requests keep loose userData (they hit the default handler).

  • Handler contextctx.addRequests / ctx.enqueueLinks are typed from the route map. Driven by the router itself, so it works for every crawler type.
  • Crawler instanceRoutes is inferred from the requestHandler option and types crawler.addRequests / crawler.run. Threaded through all crawler classes (Basic/Http/Cheerio/JSDOM/LinkeDOM and the browser family: Browser/Playwright/Puppeteer/Stagehand/AdaptivePlaywright).
const crawler = new PlaywrightCrawler({ requestHandler: router });

await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 's', price: 1 } }]); // ✅
await crawler.addRequests([{ url, label: 'PRODUCT', userData: { sku: 1 } }]);             // ❌ wrong userData
await crawler.addRequests([{ url, label: 'NOPE' }]);                                      // ❌ unknown label

How

  • New RouteSchemas / RoutesFromSchemas (via StandardSchemaV1.InferOutput); Router.create and the createXRouter factories gain overloads for the route map, the legacy flat-userData form, and the schema map (disambiguated by value type). Adds the tiny, types-only @standard-schema/spec dep to @crawlee/core (zod is already a core dep on v4 and implements the interface).
  • LabeledSource / TypedRequestsLike build a label-discriminated request-input type; a signature-preserving transform retypes enqueueLinks's label/userData without changing argument optionality or return type (which differ per crawler).
  • A Routes generic is inferred from requestHandler and threaded through the crawler-class hierarchy.

All of it is backwards compatible via the open route-map default.

Relates to #3082. The remaining parts of that issue (typed pushData / key-value store) are a separate axis and are not in scope here.

🤖 Generated with Claude Code

B4nan and others added 30 commits March 5, 2026 15:46
BREAKING CHANGE:

The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`.

Also all the dependencies are updated to the latest versions, including cheerio v1.
BREAKING CHANGE:

The crawler following options are removed:

- `handleRequestFunction` -> `requestHandler`
- `handlePageFunction` -> `requestHandler`
- `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs`
- `handleFailedRequestFunction` -> `failedRequestHandler`
BREAKING CHANGE:

The crawling context no longer includes the `Error` object for failed requests. Use the second parameter of the `errorHandler` or `failedRequestHandler` callbacks to access the error.

Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
….retireOnBlockedStatusCodes`

BREAKING CHANGE:

`additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
….retireOnBlockedStatusCodes`

BREAKING CHANGE:

`additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
also tries to bump better-sqlite3 to latest version to have prebuilds for node 22
- closes #2479
- closes #3106
- closes #3107
- closes #3078

In my opinion, it makes a lot of sense to do the remaining changes in a
separate PR.

- [x] Introduce a `ContextPipeline` abstraction
- [x] Update crawlers to use it
- [x] Make sure that existing tests pass
- [ ] Refine the `ContextPipeline.compose` signature and the semantics
of `BasicCrawlerOptions.contextPipelineEnhancer` to maximize DX
- [x] Write tests for the `contextPipelineEnhancer`
- [x] Resolve added TODO comments (fix immediately or make issues)
- [ ] Update documentation

The `context-pipeline` branch introduces a fundamental architectural
change to how Crawlee crawlers build and enhance the crawling context
passed to request handlers. The core motivation is to fix the
composition and extensibility nightmare in the current crawler
hierarchy.

1. **Rigid inheritance hierarchy**: Crawlers were stuck in a brittle
inheritance chain where each layer manipulated the context object while
assuming that it already satisfied its final type. Multiple overrides of
`BasicCrawler` lifecycle methods made the execution flow even harder to
follow.

2. **Context enhancement via monkey-patching**: Manual property
assignment (`crawlingContext.page = page`, `crawlingContext.$ = $`)
scattered everywhere. It was a mess to follow and impossible to reason
about.

3. **Cleanup coordination**: Resource cleanup was handled by separate
`_cleanupContext` methods that were not co-located with the
initialization.

4. **Extension mechanism was broken**: The `CrawlerExtension.use()` API
tried to let you extend crawlers (the ones based on `HttpCrawler`) by
overwriting properties - completely type-unsafe and fragile as hell.

Introduces `ContextPipeline` - a **middleware-based composition
pattern** where:

- Each crawler layer defines how it enhances the context through
explicit `action` functions
- Cleanup logic is co-located with initialization via optional `cleanup`
functions
- Type safety is maintained through TypeScript generics that track
context transformations
- The pipeline executes middleware sequentially with proper error
handling and guaranteed cleanup

Declarative middleware composition with co-located cleanup:

```typescript
contextPipeline.compose({
  action: async (context) => ({ page, $ }),
  cleanup: async (context) => { await page.close(); }
})
```

The `ContextPipeline<TBase, TFinal>` tracks type transformations through
the chain:

```typescript
ContextPipeline<CrawlingContext, CrawlingContext>
  .compose<{ page: Page }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page }>
  .compose<{ $: CheerioAPI }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page, $: CheerioAPI }>
```

The `CrawlerExtension.use()` is gone. New approach via
`contextPipelineEnhancer`:

```typescript
new BasicCrawler({
  contextPipelineEnhancer: (pipeline) =>
    pipeline.compose({
      action: async (context) => ({ myCustomProp: ... })
    })
})
```

The current way to express a context pipeline middleware has some
shortcomings (`ContextPipeline.compose`,
`BasicCrawlerOptions.contextPipelineEnhancer`). I suggest resolving this
in another PR.

For most legitimate use cases, this should be non-breaking. Those who
extend the Crawler classes in non-trivial ways may need to adjust their
code though - the non-public interface of `BasicCrawler` and
`HttpCrawler` changed quite a bit.

The pipeline uses `Object.defineProperties` for each middleware. Is this
a serious performance consideration?

---------

Co-authored-by: Martin Adámek <banan23@gmail.com>
Extracts `ProxyConfiguration` to `BasicCrawler` (related to discussion under #2917).

Pass the `ProxyConfiguration` instance to the `SessionPool` for new `Session` object creation.

Store and read the `ProxyInfo` from the `Session` instance instead of calling the `ProxyConfiguration` methods in the crawlers.

closes #3198
Phasing out `got-scraping`-specific interfaces in favour of native
`fetch` API.

Related to #3071
Fixes build toolchain errors caused by the recent rebase onto the
current `master` ([more details
here](https://apify.slack.com/archives/C02JQSN79V4/p1764373034961859)).

The largest thing is probably updating the dependency versions in
`package.json` - if `turborepo` doesn't find the matching version in the
local workspace, it will build against the package pulled from `npm`
(which doesn't match the v4 API at this point).
…client (#3286)

Removes incorrect implementation `KVS.getPublicUrl()` implementation from `@crawlee/core` and proxies the call to the storage client.

Closes #3272 
Closes #3076
…rfaces (#3295)

Works towards removing `got-scraping` as a direct Crawlee dependency.

Related to #3275
Related to #3071
Related to #3275

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
janbuchar and others added 10 commits June 17, 2026 16:20
- default handler keeps `request.userData` loosely typed (it is a fallback
  for any request, including labels not in the route map)
- split factory/`Router.create` into explicit overloads (route map vs legacy
  flat userData) for backwards compatibility, keeping the schema overload
- drop the exported `RouteMap` alias (referenced as prose in docs instead)
When a typed router is used, the route map now also types the request inputs:

- handler context: `ctx.addRequests` and `ctx.enqueueLinks` require the
  `userData` shape matching the request's `label` (and reject unknown labels);
  this is driven by the router, so it works for every crawler type.
- crawler instance: `Routes` is inferred from the `requestHandler` option and
  used to type `crawler.addRequests`/`crawler.run` for the HTTP-based crawlers
  (Basic/Http/Cheerio/JSDOM/LinkeDOM).

Unlabeled requests keep loose `userData` (they hit the default handler). All
typing is backwards compatible via the open-map default.

Note: crawler-instance `addRequests` typing is not yet wired for the browser
crawlers (Playwright/Puppeteer/Stagehand) — their `requestHandler` is redefined
in BrowserCrawlerOptions which breaks generic inference through the hierarchy;
their handler-context methods are still fully typed via the router.
Threads the `Routes` generic through the browser crawler hierarchy
(`BrowserCrawler` + Playwright/Puppeteer/Stagehand/AdaptivePlaywright) so
`crawler.addRequests`/`run` are route-typed for them as well, completing the
propagation started for the HTTP-based crawlers.

The missing piece was that `PlaywrightCrawlerOptions`/`StagehandCrawlerOptions`
redefined `requestHandler` without the route-aware `RouterHandler` overload,
shadowing the one on `BrowserCrawlerOptions` and preventing `Routes` inference.
`Routes` is also placed before the internal `__Browser*` generics on
`BrowserCrawler(Options)` so subclasses can forward it positionally.
The route-aware requestHandler union paired `RouterHandler<ExtendedContext>`
with `StagehandRequestHandler` (which wraps the context in `LoadedContext`).
The two union members had different call-signature parameter types, so TS
could not contextually type an inline `requestHandler({ page, request, ... })`
and reported the params as implicit `any` (breaking the Stagehand guide
examples). Use `RequestHandler<ExtendedContext>` for the plain member so both
signatures are identical; `StagehandCrawlingContext` already carries a
`LoadedRequest`, so nothing is lost.
Makes `@crawlee/impit-client` the default HTTP client for
`@crawlee/basic`'s `BasicCrawler`.
Adds recent changes to the `SessionPool` and related classes to the
existing guides.

Closes #796
Drops the `SDK_` prefix (replaces w/ `CRAWLEE_`) on multiple spots in
the codebase.

Closes #1729
Utilizes the byte-stream prescan
(https://html.spec.whatwg.org/multipage/parsing.html#prescan-a-byte-stream-to-determine-its-encoding)
to determine the HTML document encoding, if not specified in HTTP
headers (or explicitly forced by user).

Closes #2317
@B4nan
B4nan requested a review from barjin June 30, 2026 12:14
barjin and others added 3 commits July 1, 2026 10:35
Due to the nature of the concurrency model in JS, we cannot reliably
abort the `requestHandler` execution on timeouts.

This can lead to "race conditions" with, e.g., the `requestHandler`
modifying the storages while the `errorHandler` / `failedRequestHandler`
is already running.

This PR adds a `tryCancel()` check throwing on elapsed timeouts before
each storage method call.

Closes #2889

@barjin barjin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you @B4nan !

I suppose this is intended specifically for routers and not (arrow function) requestHandler nor pre/postNavigationHandler, right? Although a cool nice-to-have, I cannot imagine how it would work anyway, so approving this now. 👍

barjin and others added 12 commits July 1, 2026 15:43
Avoid double-counting requests in `RequestManagerTandem`. 

Closes #3787
…a-validation

# Conflicts:
#	packages/core/package.json
#	pnpm-lock.yaml
The merge commit staged the lockfile before regenerating it, so the
`@crawlee/core` importer was missing the `@standard-schema/spec` specifier,
breaking the frozen-lockfile install in CI.
# Conflicts:
#	docs/public-api/crawlee-core.api.md
B4nan added a commit that referenced this pull request Jul 16, 2026
…3851)

## What

Backports the schema-based router to the stable v3 line as a fully
**opt-in, non-breaking** feature, complementing the compile-time
route-map typing from #3747. A router can now take a [Standard
Schema](https://standardschema.dev) (Zod, Valibot, ArkType, …) per
label, which drives **both** the inferred `request.userData` type
**and** its runtime validation.

```ts
import { createHttpRouter, HttpCrawler } from 'crawlee';
import { z } from 'zod';

const router = createHttpRouter({
    LIST: z.object({ offset: z.number() }),
    POKEMON: z.object({ id: z.string() }),
});

router.addHandler('POKEMON', async ({ request }) => {
    request.userData.id; // string — inferred from the schema, validated before the handler runs
});
```

## How

- **Schema router (`@crawlee/core`)** — a third `Router.create` /
`createXRouter` overload accepts a `{ label: schema }` map. The route
map (`Routes`) is inferred from the schemas via `RoutesFromSchemas`. At
dispatch, `request.userData` is validated against the label's schema and
replaced with the parsed (coerced) value before the handler runs.
- **Validation on add** — the crawler validates `userData` against the
label's schema on the add paths it owns: `crawler.addRequests` (and thus
`crawler.run` / `context.addRequests`) and `context.enqueueLinks`. A bad
shape fails fast at insertion instead of deep inside a handler. This
lives at the crawler level — the storage layer (`RequestProvider`) stays
decoupled from the router — so a direct `requestQueue.addRequest` that
bypasses the crawler is validated at dispatch rather than on add.
- A mismatch throws the new non-retryable `RequestValidationError`
(re-running the request would fail identically).

Everything is opt-in and backwards compatible: a router created without
a schema map behaves exactly as before, with no runtime cost, and labels
with no registered schema are left untouched. The only difference vs.
the v4 variant (#3748) is that `crawler.run([...])` inputs are not
type-checked against the route map here — that needs a crawler-level
generic, which is a breaking change kept for v4.

Because this turns the v3 router work into a runtime feature rather than
a type-only patch, it ships as a **minor** (3.18) with its own docs
snapshot — so this PR also reverts the `version-3.17` snapshot edits
that #3747 made (a frozen snapshot shouldn't have been touched) and
documents the schema feature in the live TypeScript guide instead.

Relates to #3082
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants