Skip to content

feat: backwards-compatibility shim for globs/regexps/pseudoUrls (proposal)#3871

Draft
l2ysho wants to merge 1 commit into
3409-align-enqueuelinksoptions-with-crawlee-pythonfrom
claude/enqueuelinks-bc-shim
Draft

feat: backwards-compatibility shim for globs/regexps/pseudoUrls (proposal)#3871
l2ysho wants to merge 1 commit into
3409-align-enqueuelinksoptions-with-crawlee-pythonfrom
claude/enqueuelinks-bc-shim

Conversation

@l2ysho

@l2ysho l2ysho commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Draft / proposal — not for merge as-is. Stacked on top of #3533 (targets its branch) to explore the backwards-compatibility question raised in #3533 (comment).

Context

In #3533, @B4nan suggested reconsidering whether we should drop globs/regexps/pseudoUrls without a backwards-compatibility layer, since the Apify Console input components emit these fields and a v3 → v4 migration window will last months (the same input needs to keep working on both).

This draft is a concrete prototype of what a lightweight BC layer could look like, so the approach can be evaluated against a hard removal.

What this does

  • Re-introduces globs, regexps, and pseudoUrls on EnqueueLinksOptions as deprecated aliases, mapped onto include:
    • globsinclude (1:1)
    • regexpsinclude (1:1)
    • pseudoUrlsinclude via purlToRegExp
  • Emits a log.deprecated(...) warning whenever an alias is used.
  • Per-pattern request options (method, payload, label, userData, headers) that the old pattern objects accepted are not restored — they are ignored with a warning, since transformRequestFunction is now the only supported way to customize per-request options.
  • The aliases feed the same include pipeline, so the point-1 (strategy AND-ing) and point-5 (empty-array rejection) behavior from feat: simplify enqueueLinks interface #3533 is unchanged.

Deliberate scope boundaries

  • Covers only the plain string/object pattern arrays the Console generates — intentionally not the removed per-pattern request options.
  • Applied to core enqueueLinks only; not replicated in enqueueLinksByClickingElements (Playwright/Puppeteer) or SitemapRequestList yet.
  • ⚠️ Re-adds the @apify/pseudo_url dependency, which reverses one of feat: simplify enqueueLinks interface #3533's stated goals. That trade-off is the main thing to decide — a BC layer for pseudoUrls isn't possible without it (or without inlining the PURL → RegExp conversion).

Tests

Adds coverage for alias mapping (globs/regexps/pseudoUrls), per-pattern options being ignored, and aliases combined with include.

Open questions for the team

  1. Do we want a BC layer at all, or is a clean break acceptable for a major (with the Console team branching the UI by version)?
  2. If yes — is re-adding @apify/pseudo_url acceptable, or should the PURL→RegExp conversion be inlined to avoid the dependency?
  3. Should the shim also cover enqueueLinksByClickingElements and SitemapRequestList?

🤖 Generated with Claude Code

https://claude.ai/code/session_01QEq1cCENtNPmFUjQev67wk


Generated by Claude Code

Prototype (not part of PR #3533's approved scope) exploring a BC layer for
the deprecated `globs`, `regexps`, and `pseudoUrls` enqueueLinks options, so
input generated for v3 (e.g. by the Apify Console input components) keeps
working on v4 during the migration window.

- Map the deprecated options onto `include` via
  `normalizeDeprecatedPatternOptions`, emitting deprecation warnings.
- Per-pattern request options (method/payload/label/userData/headers) are no
  longer supported; they are ignored with a warning.
- Re-adds the `@apify/pseudo_url` dependency for `pseudoUrls` -> regexp
  conversion (reverses one of the PR's goals; flagged for discussion).
- Add tests covering alias mapping, ignored per-pattern options, and
  combining aliases with `include`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QEq1cCENtNPmFUjQev67wk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants