url-reader-worker

An URL reader like jina-ai/reader for AI Agents which can be deployed very easily as a Cloudflare Worker.

Usage

IMPORTANT: You need Worker Paid plan to use this worker due to the limitation of Browser ability.

Deploy

npm run deploy:all

This deploys the private url-reader-parser Worker first, then the public reader Worker. Wrangler also provisions the KV namespace and Durable Object binding declared in wrangler.jsonc.

Params

[YOUR_WORKER_URL]?target={targetUrl}&mode={mode}&retain_media={retainMedia}&auth_key={authKey}

• targetUrl: URL of the target page, read image / audio / video directly is not supported.

• mode (optional): markdown returns readable Markdown, text returns plain text, get_response streams the origin response without browser rendering, and any other value returns cleaned rendered HTML.

• retainMedia (optional): Controls how <video>, <audio>, and supported video embeds appear in Markdown. Supported values are link (default), text, image, html, and none. Media and image URLs are resolved against the page URL, including lazy-loaded srcset and data-src values.

• authKey (optional): If AUTH_KEY set, you need to pass the key in the query. We recommend to use Cloudflare's abilities to protect your worker.

Env

• CACHE_CONTROL: Response cache control which follows Cloudflare's rules.
• BROWSER_TIMEOUT: General timeout for puppeteer.
• BROWSER_KEEP_ALIVE: Browser session inactivity timeout in milliseconds, up to 10 minutes.
• FETCH_CACHE_TTL: General ttl for caching HEAD and GET requests to original content.
• BROWSER_USER_AGENT: User-Agent string for puppeteer.
• SALVAGE_USER_AGENT: User-Agent string for salvage fetching.
• AUTH_KEY: An optional access key. Set production values with wrangler secret put AUTH_KEY; do not commit them to wrangler.jsonc.

Development

npm test
npm run typecheck
npm run check

Security

Please using WAF / Rules on Cloudflare to secure your worker because it can access any url from any source by default, THIS WORKER CAN BE ABUSED POTENTIALLY.

If you want to add some extra logic to block some evil requests, please fork it and modify the code.

Inspired by

jina-ai/reader

License

Apache-2.0