feat: schema-validated router labels via standard-schema#3748
Open
B4nan wants to merge 130 commits into
Open
Conversation
BREAKING CHANGE: The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`. Also all the dependencies are updated to the latest versions, including cheerio v1.
BREAKING CHANGE: The crawler following options are removed: - `handleRequestFunction` -> `requestHandler` - `handlePageFunction` -> `requestHandler` - `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs` - `handleFailedRequestFunction` -> `failedRequestHandler`
BREAKING CHANGE: The crawling context no longer includes the `Error` object for failed requests. Use the second parameter of the `errorHandler` or `failedRequestHandler` callbacks to access the error. Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
also tries to bump better-sqlite3 to latest version to have prebuilds for node 22
- closes #2479 - closes #3106 - closes #3107 - closes #3078 In my opinion, it makes a lot of sense to do the remaining changes in a separate PR. - [x] Introduce a `ContextPipeline` abstraction - [x] Update crawlers to use it - [x] Make sure that existing tests pass - [ ] Refine the `ContextPipeline.compose` signature and the semantics of `BasicCrawlerOptions.contextPipelineEnhancer` to maximize DX - [x] Write tests for the `contextPipelineEnhancer` - [x] Resolve added TODO comments (fix immediately or make issues) - [ ] Update documentation The `context-pipeline` branch introduces a fundamental architectural change to how Crawlee crawlers build and enhance the crawling context passed to request handlers. The core motivation is to fix the composition and extensibility nightmare in the current crawler hierarchy. 1. **Rigid inheritance hierarchy**: Crawlers were stuck in a brittle inheritance chain where each layer manipulated the context object while assuming that it already satisfied its final type. Multiple overrides of `BasicCrawler` lifecycle methods made the execution flow even harder to follow. 2. **Context enhancement via monkey-patching**: Manual property assignment (`crawlingContext.page = page`, `crawlingContext.$ = $`) scattered everywhere. It was a mess to follow and impossible to reason about. 3. **Cleanup coordination**: Resource cleanup was handled by separate `_cleanupContext` methods that were not co-located with the initialization. 4. **Extension mechanism was broken**: The `CrawlerExtension.use()` API tried to let you extend crawlers (the ones based on `HttpCrawler`) by overwriting properties - completely type-unsafe and fragile as hell. Introduces `ContextPipeline` - a **middleware-based composition pattern** where: - Each crawler layer defines how it enhances the context through explicit `action` functions - Cleanup logic is co-located with initialization via optional `cleanup` functions - Type safety is maintained through TypeScript generics that track context transformations - The pipeline executes middleware sequentially with proper error handling and guaranteed cleanup Declarative middleware composition with co-located cleanup: ```typescript contextPipeline.compose({ action: async (context) => ({ page, $ }), cleanup: async (context) => { await page.close(); } }) ``` The `ContextPipeline<TBase, TFinal>` tracks type transformations through the chain: ```typescript ContextPipeline<CrawlingContext, CrawlingContext> .compose<{ page: Page }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page }> .compose<{ $: CheerioAPI }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page, $: CheerioAPI }> ``` The `CrawlerExtension.use()` is gone. New approach via `contextPipelineEnhancer`: ```typescript new BasicCrawler({ contextPipelineEnhancer: (pipeline) => pipeline.compose({ action: async (context) => ({ myCustomProp: ... }) }) }) ``` The current way to express a context pipeline middleware has some shortcomings (`ContextPipeline.compose`, `BasicCrawlerOptions.contextPipelineEnhancer`). I suggest resolving this in another PR. For most legitimate use cases, this should be non-breaking. Those who extend the Crawler classes in non-trivial ways may need to adjust their code though - the non-public interface of `BasicCrawler` and `HttpCrawler` changed quite a bit. The pipeline uses `Object.defineProperties` for each middleware. Is this a serious performance consideration? --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
Extracts `ProxyConfiguration` to `BasicCrawler` (related to discussion under #2917). Pass the `ProxyConfiguration` instance to the `SessionPool` for new `Session` object creation. Store and read the `ProxyInfo` from the `Session` instance instead of calling the `ProxyConfiguration` methods in the crawlers. closes #3198
Phasing out `got-scraping`-specific interfaces in favour of native `fetch` API. Related to #3071
Fixes build toolchain errors caused by the recent rebase onto the current `master` ([more details here](https://apify.slack.com/archives/C02JQSN79V4/p1764373034961859)). The largest thing is probably updating the dependency versions in `package.json` - if `turborepo` doesn't find the matching version in the local workspace, it will build against the package pulled from `npm` (which doesn't match the v4 API at this point).
Related to #3275 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- default handler keeps `request.userData` loosely typed (it is a fallback for any request, including labels not in the route map) - split factory/`Router.create` into explicit overloads (route map vs legacy flat userData) for backwards compatibility, keeping the schema overload - drop the exported `RouteMap` alias (referenced as prose in docs instead)
When a typed router is used, the route map now also types the request inputs: - handler context: `ctx.addRequests` and `ctx.enqueueLinks` require the `userData` shape matching the request's `label` (and reject unknown labels); this is driven by the router, so it works for every crawler type. - crawler instance: `Routes` is inferred from the `requestHandler` option and used to type `crawler.addRequests`/`crawler.run` for the HTTP-based crawlers (Basic/Http/Cheerio/JSDOM/LinkeDOM). Unlabeled requests keep loose `userData` (they hit the default handler). All typing is backwards compatible via the open-map default. Note: crawler-instance `addRequests` typing is not yet wired for the browser crawlers (Playwright/Puppeteer/Stagehand) — their `requestHandler` is redefined in BrowserCrawlerOptions which breaks generic inference through the hierarchy; their handler-context methods are still fully typed via the router.
Threads the `Routes` generic through the browser crawler hierarchy (`BrowserCrawler` + Playwright/Puppeteer/Stagehand/AdaptivePlaywright) so `crawler.addRequests`/`run` are route-typed for them as well, completing the propagation started for the HTTP-based crawlers. The missing piece was that `PlaywrightCrawlerOptions`/`StagehandCrawlerOptions` redefined `requestHandler` without the route-aware `RouterHandler` overload, shadowing the one on `BrowserCrawlerOptions` and preventing `Routes` inference. `Routes` is also placed before the internal `__Browser*` generics on `BrowserCrawler(Options)` so subclasses can forward it positionally.
The route-aware requestHandler union paired `RouterHandler<ExtendedContext>`
with `StagehandRequestHandler` (which wraps the context in `LoadedContext`).
The two union members had different call-signature parameter types, so TS
could not contextually type an inline `requestHandler({ page, request, ... })`
and reported the params as implicit `any` (breaking the Stagehand guide
examples). Use `RequestHandler<ExtendedContext>` for the plain member so both
signatures are identical; `StagehandCrawlingContext` already carries a
`LoadedRequest`, so nothing is lost.
Makes `@crawlee/impit-client` the default HTTP client for `@crawlee/basic`'s `BasicCrawler`.
Adds recent changes to the `SessionPool` and related classes to the existing guides. Closes #796
Utilizes the byte-stream prescan (https://html.spec.whatwg.org/multipage/parsing.html#prescan-a-byte-stream-to-determine-its-encoding) to determine the HTML document encoding, if not specified in HTTP headers (or explicitly forced by user). Closes #2317
Due to the nature of the concurrency model in JS, we cannot reliably abort the `requestHandler` execution on timeouts. This can lead to "race conditions" with, e.g., the `requestHandler` modifying the storages while the `errorHandler` / `failedRequestHandler` is already running. This PR adds a `tryCancel()` check throwing on elapsed timeouts before each storage method call. Closes #2889
barjin
approved these changes
Jul 1, 2026
barjin
left a comment
Member
There was a problem hiding this comment.
lgtm, thank you @B4nan !
I suppose this is intended specifically for routers and not (arrow function) requestHandler nor pre/postNavigationHandler, right? Although a cool nice-to-have, I cannot imagine how it would work anyway, so approving this now. 👍
Avoid double-counting requests in `RequestManagerTandem`. Closes #3787
…a-validation # Conflicts: # packages/core/package.json # pnpm-lock.yaml
The merge commit staged the lockfile before regenerating it, so the `@crawlee/core` importer was missing the `@standard-schema/spec` specifier, breaking the frozen-lockfile install in CI.
# Conflicts: # docs/public-api/crawlee-core.api.md
B4nan
added a commit
that referenced
this pull request
Jul 16, 2026
…3851) ## What Backports the schema-based router to the stable v3 line as a fully **opt-in, non-breaking** feature, complementing the compile-time route-map typing from #3747. A router can now take a [Standard Schema](https://standardschema.dev) (Zod, Valibot, ArkType, …) per label, which drives **both** the inferred `request.userData` type **and** its runtime validation. ```ts import { createHttpRouter, HttpCrawler } from 'crawlee'; import { z } from 'zod'; const router = createHttpRouter({ LIST: z.object({ offset: z.number() }), POKEMON: z.object({ id: z.string() }), }); router.addHandler('POKEMON', async ({ request }) => { request.userData.id; // string — inferred from the schema, validated before the handler runs }); ``` ## How - **Schema router (`@crawlee/core`)** — a third `Router.create` / `createXRouter` overload accepts a `{ label: schema }` map. The route map (`Routes`) is inferred from the schemas via `RoutesFromSchemas`. At dispatch, `request.userData` is validated against the label's schema and replaced with the parsed (coerced) value before the handler runs. - **Validation on add** — the crawler validates `userData` against the label's schema on the add paths it owns: `crawler.addRequests` (and thus `crawler.run` / `context.addRequests`) and `context.enqueueLinks`. A bad shape fails fast at insertion instead of deep inside a handler. This lives at the crawler level — the storage layer (`RequestProvider`) stays decoupled from the router — so a direct `requestQueue.addRequest` that bypasses the crawler is validated at dispatch rather than on add. - A mismatch throws the new non-retryable `RequestValidationError` (re-running the request would fail identically). Everything is opt-in and backwards compatible: a router created without a schema map behaves exactly as before, with no runtime cost, and labels with no registered schema are left untouched. The only difference vs. the v4 variant (#3748) is that `crawler.run([...])` inputs are not type-checked against the route map here — that needs a crawler-level generic, which is a breaking change kept for v4. Because this turns the v3 router work into a runtime feature rather than a type-only patch, it ships as a **minor** (3.18) with its own docs snapshot — so this PR also reverts the `version-3.17` snapshot edits that #3747 made (a frozen snapshot shouldn't have been touched) and documents the schema feature in the live TypeScript guide instead. Relates to #3082
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the router's request
labeldrive the type ofrequest.userDataend-to-end — from declaring routes, through the request handler, all the way to the methods that enqueue new requests. Targetsv4.1. Typed route map (compile-time)
Declare a
label → userDatamap and pass it as the router's second type argument. Handlers getrequest.userDatatyped per label, and unknown labels are a compile error. Fully backwards compatible — the default route map is open (Record<string, …>), so existing untyped usage and the legacy flat-userDatageneric keep working.The default handler (
addDefaultHandler) is a fallback for any request, so itsuserDatastays loosely typed.2. Schema-validated labels (types + runtime validation)
Passing a per-label Standard Schema (Zod, Valibot, ArkType, …) both infers the
userDatatypes and validates them at runtime before the handler runs, replacingrequest.userDatawith the parsed value. InvaliduserDatathrows a new non-retryableRequestValidationError(the data won't change between attempts) carrying the schema issues.3. Propagation to crawler & context request methods
When a typed router is used, the route map now also types the request inputs: providing a declared
labelrequires the matchinguserDatashape (and rejects unknown labels); unlabeled requests keep looseuserData(they hit the default handler).ctx.addRequests/ctx.enqueueLinksare typed from the route map. Driven by the router itself, so it works for every crawler type.Routesis inferred from therequestHandleroption and typescrawler.addRequests/crawler.run. Threaded through all crawler classes (Basic/Http/Cheerio/JSDOM/LinkeDOM and the browser family: Browser/Playwright/Puppeteer/Stagehand/AdaptivePlaywright).How
RouteSchemas/RoutesFromSchemas(viaStandardSchemaV1.InferOutput);Router.createand thecreateXRouterfactories gain overloads for the route map, the legacy flat-userDataform, and the schema map (disambiguated by value type). Adds the tiny, types-only@standard-schema/specdep to@crawlee/core(zodis already a core dep onv4and implements the interface).LabeledSource/TypedRequestsLikebuild a label-discriminated request-input type; a signature-preserving transform retypesenqueueLinks'slabel/userDatawithout changing argument optionality or return type (which differ per crawler).Routesgeneric is inferred fromrequestHandlerand threaded through the crawler-class hierarchy.All of it is backwards compatible via the open route-map default.
Relates to #3082. The remaining parts of that issue (typed
pushData/ key-value store) are a separate axis and are not in scope here.🤖 Generated with Claude Code