The Wall Street Journal is a subscription publication, so the first question for any wsj api project is not how to fetch a page. It is which pages you are entitled to read at all. This repo draws that line first and then covers the surface that sits outside it, which turns out to be the more useful half anyway: the Market Data Center, where the index quotes live behind stable HTML ids.
Built on ScrapingBee's web scraping API. Verified live on 2026-09-15.
In scope. The Market Data Center and its quote pages, index levels and daily changes, headline text, deks, bylines and section tags as published on free listing pages, and article metadata exposed in tags for social preview.
Out of scope. Full article bodies behind the subscription, anything reachable only after signing in, and any technique for getting at either. Scraping under login credentials is prohibited by ScrapingBee's terms of service, and a paywall is the publisher's access control whether or not a given request happens to slip past it.
That is not a limitation this project works around. It is the shape of the project.
The free surface is genuinely large. The Market Data Center carries index levels, quote pages per symbol, money rates, bond data and the market news river, none of which sit behind the subscription.
The hub and the quote pages behave differently, and the gap matters at volume:
| Page | mode=auto cost |
Bytes | What it carries |
|---|---|---|---|
wsj.com/market-data |
25, then 75 on a later run | 1,383,465 | Navigation, headline river, section links |
wsj.com/market-data/quotes/index/DJIA |
75 | 557,066 | The actual index level and change |
Quote pages need stealth every time. The hub is less predictable: it settled on the premium rung at 25 credits on one run and on stealth at 75 on another, same URL, same parameters. That is auto mode doing its job, escalating only as far as the site forces it on that request, but it means you should budget the hub at 75 and treat 25 as a good day. Read spb-cost per response rather than assuming a fixed figure.
The quote pages use element ids rather than classes, which is the most durable selector type on any site. No build hashes, no CSS module suffixes, nothing that rotates on deploy:
rules = {
"level": "#quote_val",
"change": "#quote_change",
"change_pct": "#quote_changePer",
"as_of": "#quote_dateTime",
}Live values from wsj.com/market-data/quotes/index/DJIA:
{
"level": "52421.20",
"change": "-152.09",
"change_pct": "-0.29%",
"as_of": "4:34 PM EDT 09/14/26"
}Two details worth building on. #quote_dateTime gives you the exchange timestamp with its timezone, so you can tell a live quote from a stale one rather than assuming the fetch time is the quote time. And direction is also encoded structurally: the surrounding #quote_deltaBar carries a deltaBar-neg or deltaBar-pos class, so you can read the sign without parsing the minus out of the number.
Swap the symbol in the path for other indices. SPX is the S&P 500, COMP the Nasdaq Composite. The same id set works across them because it is one template.
Contrast that with the hub, where the classes look like style--link-item--2QGPfq6e and WSJTheme-- appears 1,901 times. Those are generated names with hashed suffixes and they will move. Select on ids where ids exist.
rules = {
"title": "title",
"canonical": {"selector": 'link[rel="canonical"]', "output": "@href"},
"summary": {"selector": 'meta[name="description"]', "output": "@content"},
"headlines": {"selector": "h3 a, h2 a", "type": "list"},
}Live result, billed 25 on one run and 75 on another:
{
"title": "Market Data",
"canonical": "https://www.wsj.com/market-data",
"summary": "Market Data Center",
"headlines": [
"European Chip Stocks Mixed After Monday Rout",
"Euro Could Fall if Fed Lifts Rates"
]
}Headline text and the link are free. The article those links point to is not, and this project does not follow them into the subscription.
For brand monitoring, which is the job the WSJ scraper page leads with, headlines plus deks are usually enough: you want to know that a company was written about and where, and for the body you follow the link in a browser like any other reader.
Measured from spb-cost response headers:
| Call | Credits |
|---|---|
Market Data hub, mode=auto |
25 or 75, varies per request |
Index quote page, mode=auto |
75 |
| Rejected request | 0 |
mode=auto walks the ladder cheapest first and bills only the rung that worked, and nothing if every rung fails. It cannot be combined with render_js, premium_proxy or stealth_proxy, and sending both returns HTTP 400 while billing nothing.
Practical shape: polling three indices every fifteen minutes during a trading session is 3 quotes times 26 polls times 75, which is 5,850 credits a day. That eats an entry plan quickly, so poll on the cadence your decision actually needs rather than on the cadence the market moves. ScrapingBee does not cache, so every repeat is billed in full.
Plan tiers are on the pricing page.
Public Market Data Center pages and free listing pages only. Article bodies behind the subscription are out of scope, and so is anything requiring a signed in session.
Headlines and quotes are the Wall Street Journal's copyrighted output. Using them as a monitoring signal inside your own systems is a different thing from republishing them, so attribute and link back if anything reaches an audience. Dow Jones publishes commercial data and content licensing for cases that need redistribution rights, which is the correct route when that is what you are doing. Dow Jones terms of use govern the material.
Reference: extraction rules, data extraction feature, markdown output when you are feeding headlines to a model.
Adjacent news and finance endpoints: news results API, Bloomberg API, economic news API, financial news feed API, Investopedia scraper API, Google news scraper API, Nasdaq API, Yahoo finance scraper API.
Can I scrape full WSJ articles? Not through this project. Article bodies sit behind a subscription, and scraping under login credentials is prohibited by ScrapingBee's terms of service. Headlines, deks and market data are the free surface.
Why does the same page cost 25 one minute and 75 the next?
Auto mode escalates only as far as the site forces it on that request, and WSJ is not consistent about how hard it pushes back. Quote pages needed stealth every time. Read spb-cost per response and budget for the higher number.
Which selectors should I use?
The ids on quote pages: #quote_val, #quote_change, #quote_changePer, #quote_dateTime. Avoid the hub's generated class names, which carry hashed suffixes.
How do I know a quote is current?
Read #quote_dateTime. It carries the exchange timestamp and timezone, for example 4:34 PM EDT 09/14/26, which is not the same as when you fetched it.
Is there an official WSJ API? Dow Jones licenses content and data commercially. That is the right route for redistribution. This project reads the public pages for monitoring purposes.
MIT. See LICENSE.