This file is 184 lines long; read all of them.
From a plain-English prompt to a working Scrapy spider.
Using a specific coding agent? See Zyte Coding Agent Add-Ons for alternatives.
This is an agent plugin: a portable, vendor-neutral package of skills that any agent implementing the standard can load. Install it however your agent installs plugins, or use the skills CLI, which sets it up for every supported agent on your system at once:
npx skills add zytedata/skillsSee skills.sh for the list of agents it supports and how to enable skills in each.
Any agent that reads agent plugins or Agent Skills can use this plugin, including editors such as Cursor. For Claude Code, Codex CLI and GitHub Copilot CLI we publish dedicated packages tuned to each of them; prefer those if you use one.
The plugin declares the Zyte MCP server in mcp.json, for agents that read MCP servers from plugins; sign in to it once, in the browser. Elsewhere, add it the way your agent documents MCP servers, as an HTTP server named zyte at https://mcp.zyte.com/v1/mcp.
This is Zyte's official agent plugin that generates production-ready Scrapy spiders with web-poet page objects from a plain-English prompt. Give it a URL and describe what you want to extract. It handles site exploration, schema discovery, code generation, and smoke testing: no boilerplate, no manual selector hunting.
The plugin explores the target site, discovers available fields, and presents a schema for your approval before generating a single line of code. After you confirm the schema, it creates a Scrapy project with all dependencies configured, generates web-poet page objects and test fixtures, wires up the spider, and runs a smoke test to verify that extraction is working before handing the project back to you.
Optionally, use /zyte to deploy directly to Scrapy Cloud for scheduled runs, job history, and monitoring, which the Zyte MCP handles once the spider is deployed. A free tier is available.
The /scrape skill works on any website with repeating structured content: detail pages linked from a listing or category page. Examples from the skill:
- Product catalogs
- Job listings
- Recipes
The /scrape skill orchestrates two stages automatically:
1. Plan and validate the scrape → /scrape-plan
2. Build the project and spider → /scrapy-extra
Each stage feeds directly into the next. When the pipeline completes, you have a runnable spider and a passing test suite:
uv run scrapy crawl <spider_name>
uv run pytest fixtures/| Skill | Description |
|---|---|
scrape |
End-to-end web scraping workflow — from URL to working spider with web-poet page objects |
| Skill | Description |
|---|---|
scrape-plan |
Plan the scrape and author a validated extraction spec: discover fields, download diverse pages, compare HTML variants, optional browser review |
scrape-analyze-page |
Extract all available fields with values from a detail page |
scrapy-extra |
Hands-on Scrapy coding: write/debug spiders, web-poet page objects, and projects; configure scrapy-poet and scrapy-zyte-api |
| Skill | Description |
|---|---|
zyte |
Zyte work the Zyte MCP does not cover: set up your Zyte account and credentials; deploy projects to Scrapy Cloud and export all items, logs or requests of a job; look up Zyte API plan pricing; and answer how-to and documentation questions about Zyte from the official docs |
- An AI coding agent that supports agent plugins or Agent Skills
uv— used to create and manage the Scrapy project- A Zyte account, signed in once through the Zyte MCP (see Install) — used for Scrapy Cloud jobs, Zyte API usage stats and per-website prices
Project dependencies (scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, pytest) are installed automatically by the skills.
Any scraping prompt triggers the skill automatically. For example:
Scrape books.toscrape.com
The plugin walks you through schema approval interactively, then generates a complete, tested Scrapy project.
To update:
npx skills update zytedata/skillsWe automatically evaluate skills and track both wall time and cost. We measure and aim to improve these metrics over time.
If you find any issue — such as prompts that did not work as expected, or that caused excessive wall time or cost — please open a GitHub issue.
Provide as much detail as possible to help us reproduce the issue. You are welcome to anonymize target websites or other data.
No. The generated spider is a standard Scrapy project that runs locally with uv. A Zyte account is required only if you want to deploy to Scrapy Cloud or use Zyte API to access sites that block standard scrapers. If you want to use Zyte API, you'll need an account to generate an API key.
The generated project includes scrapy-zyte-api as a dependency. Enabling headless browser rendering requires a Zyte API key. The /zyte skill guides you through setting up your credentials.
The project template includes scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, and pytest. All dependencies are installed automatically via uv sync.
Yes. The plugin generates a standard Scrapy project. Run it directly with:
uv run scrapy crawl <spider_name>You can extend, modify, and deploy it independently of your AI coding agent.
See LICENSE.md for the Zyte End User License Agreement.