Walmart Scraper

Extract Walmart product and search data as structured JSON, using the Scrapeless cloud Scraping Browser — no local Chrome, no proxy fleet.

Four interchangeable surfaces (Python, Node.js, CLI, MCP) emit the same JSON shape, documented in DATA_MODEL.md.

Use case

Walmart renders its catalogue from a __NEXT_DATA__ JSON blob rather than from server-side HTML, so a scraper needs a real browser session to get the page at all. That makes this the data layer for:

• Price monitoring and repricing — track priceInfo on a set of item IDs over time.
• Assortment and catalogue research — page through a search query and collect what actually ranks.
• MAP enforcement — detect sellers below an agreed price.
• Competitive intelligence — compare Walmart listings against your own or Amazon's.

Because extraction reads the data blob and not CSS selectors, a Walmart front-end redesign does not break it — the shape only changes when Walmart changes its own data layer.

Requirements

• A Scrapeless API key — create a free account
• Python 3.10+ or Node.js 18+ (each surface stands alone)

Setup

git clone https://github.com/<owner>/walmart-scraper.git
cd walmart-scraper
cp .env.example .env          # then fill in SCRAPELESS_API_KEY
export SCRAPELESS_API_KEY=your_key_here

Python

cd browser/python
pip install scrapeless playwright parsel loguru python-dotenv
python run.py                             # prints JSON to stdout
SAVE_TEST_RESULTS=true python run.py      # writes results/*.json instead

Node.js

cd browser/nodejs
pnpm install
node run.mjs

Override the samples with environment variables instead of editing code:

WALMART_SAMPLE_PRODUCT_URLS="https://www.walmart.com/ip/1736740710,https://www.walmart.com/ip/715596133" \
WALMART_SAMPLE_QUERY="laptop" python run.py

Surfaces

Surface	Path	Built on
Python	browser/python	official scrapeless SDK + Playwright over CDP
Node.js	browser/nodejs	official @scrapeless-ai/sdk + puppeteer-core over CDP
CLI	browser/cli	scrapeless-scraping-browser CLI with in-page eval
MCP	browser/mcp	Scrapeless MCP server — conversational, no code

Python	Node.js
scrape_products(urls)	scrapeProducts(urls)
scrape_search(query, pages)	scrapeSearch(query, pages)

Expected output

A real run on 2026-08-12 (Python surface): 2 product pages and 3 search pages for "laptop" → 116 search records, of which 111 are products. Committed samples in browser/python/results/ come from that run.

products.json — one wrapper per URL, product filtered to a stable key set, reviews passed through:

[
  {
    "product": {
      "id": "5H2BSTM7O0FR",
      "name": "HP 15.6\" Laptop, Intel Core i3-1215U, 8GB RAM, 256GB SSD",
      "brand": "HP",
      "availabilityStatus": "IN_STOCK",
      "averageRating": 4.2,
      "priceInfo": { "currentPrice": { "price": 279, "priceString": "$279.00" } }
    },
    "reviews": { "...": "raw Walmart reviews blob" }
  }
]

search.json — Walmart's own item objects, verbatim.

The gotcha: search results are not all products

Walmart's itemStacks mixes real listings with sponsored slots and layout placeholders. In the run above, 116 records broke down as:

__typename	Count
Product	111
AdPlaceholder	3
TileTakeOverProductPlaceholder	2

Filter on __typename — it is the field that actually distinguishes them:

products = [item for item in search if item.get("__typename") == "Product"]

Placeholders also lack an id, so item.get("id") works as a fallback, and DATA_MODEL.md marks id as not-required for exactly this reason. What you must not do is treat len(search) as a product count — it inflates every downstream number by whatever ad load Walmart served that minute, and the ratio changes between runs.

The scraper's own log line has this same off-by-ads problem. It reported scraped 116 product listings from search pages for a run whose real product count was 111. Trust the filtered length, not the log.

Records are large and mostly empty: Walmart's data layer ships a wide schema per item, so this run's untrimmed search.json was 1.70 MB for 116 records. The committed copies are trimmed to 3 records (see the .TRIMMED.md note beside each). Select the keys you need rather than storing the blob:

slim = [
    {
        "id": item["id"],
        "name": item.get("name"),
        "price": (item.get("priceInfo") or {}).get("currentPrice", {}).get("price"),
        "rating": item.get("averageRating"),
        "reviews": item.get("numberOfReviews"),
        "url": item.get("canonicalUrl"),
    }
    for item in products
]

Tests

Each language surface ships a live validator suite that asserts the documented shape:

cd browser/python && poetry install --with dev && poetry run pytest    # needs pytest-asyncio + pytest-rerunfailures
cd browser/nodejs && node --test test.mjs

The Python suite skips itself when SCRAPELESS_API_KEY is unset, so it is safe in CI without a key. Both suites are declared in pyproject.toml / package.json.

Troubleshooting

Symptom	Cause and fix
Hangs, then a net:: or ERR_TUNNEL_CONNECTION_FAILED error	Transient CDN teardown. The code already retries these; raise the retry count for long runs.
search.json has more records than products	Expected — filter on id, see above.
Empty priceInfo	Walmart omits price on some marketplace and out-of-stock items. Treat it as nullable.
SCRAPELESS_API_KEY errors	The key must be exported in the shell that runs the script; .env alone is not read by every surface.
Fewer search pages than requested	Walmart caps deep pagination for some queries; the scraper stops when a page returns no items.

Verification status

• Verified live on 2026-08-12 — the Python surface completed a full run in 173 s: 2 product pages plus 3 search pages, 116 search records (111 products), no block and no CAPTCHA. The committed browser/python/results/ fixtures come from that run; search.json is trimmed to 3 representative records.
• Not re-run in this pass: the Node.js, CLI and MCP surfaces. They were carried over unchanged from the upstream monorepo and share the data model, but their committed fixtures are older than the Python ones.

Project structure

walmart-scraper/
├── DATA_MODEL.md                 # the JSON contract every surface emits
├── browser/
│   ├── python/                   # SDK + Playwright over CDP, with pytest validators
│   ├── nodejs/                   # SDK + puppeteer-core over CDP
│   ├── cli/                      # scrapeless-scraping-browser CLI
│   └── mcp/                      # Scrapeless MCP server config
├── .env.example
└── LICENSE

Related

• Product: Scraping Browser · Scraping API
• Guides: Walmart scraper · Scrape Walmart product data · Walmart scraping proxies

License

MIT — see LICENSE.