A collection of web scraping and data analysis utilities for Chinese social media platforms and macro-economic data sources.
- Overview
- Quick Start
- Available Tasks
- Browser Profiles
- Cache Behavior
- Proxy Configuration
- Troubleshooting
This repository contains standalone scraping and analysis tools for:
- Social Media: Twitter/X lists, Weibo groups and user profiles
- Financial Data: Eastmoney futures market data
- Macro Economics: Chinese macro-economic indicators from official sources
- Content Analysis: LLM-powered analysis of collected data
All scrapers use browser automation (Patchright - Playwright fork) with persistent browser profiles to maintain authentication sessions.
# Install dependencies
npm install
# Create browser profile for authenticated scraping
npm run browser 1
# Run your first scraper
npm run scrape:twitter- Node.js >= 14.0.0
- Chrome browser installed
Scrapes tweets (text + images) from X lists with support for retweets and quote tweets.
Script: other-task/twitter-task/scrape-twitter-list.js
# Basic usage (default: 5 tweets, profile 1)
npm run scrape:twitter
# With custom arguments
npm run scrape:twitter -- --count 10 --since today
npm run scrape:twitter -- --profile 2 --since 24h
npm run scrape:twitter -- --no-cache # Force regeneratenode other-task/twitter-task/scrape-twitter-list.js [options]| Parameter | Type | Default | Description |
|---|---|---|---|
--profile <id> |
number | 1 |
Browser profile ID to use |
--count <n> |
number | 5 |
Number of tweets to scrape |
--since <when> |
string | - | Time filter (see below) |
--url <url> |
string | Built-in | Custom X list URL |
--output-dir <dir> |
string | twitter-YYYY-MM-DD/ |
Output directory |
--headless |
flag | false | Run browser in headless mode |
--scroll-delay <ms> |
number | 2000 |
Delay between scrolls (ms) |
--no-cache |
flag | - | Force regenerate (ignore cache) |
--help |
flag | - | Show help message |
today- Start of today (midnight)yesterday- Start of yesterday24h,12h,2h,1h- Relative hours ago2d,7d- Relative days ago30m,90m- Relative minutes agoYYYY-MM-DD- Specific date (midnight)
Note: Using --since removes the default 5-tweet limit. Add --count N to cap results.
# Latest 5 tweets (default)
npm run scrape:twitter
# All tweets from today
npm run scrape:twitter -- --since today
# Last 24 hours, max 20 tweets
npm run scrape:twitter -- --since 24h --count 20
# Since specific date
npm run scrape:twitter -- --since 2026-02-09
# Custom list URL with profile 2
npm run scrape:twitter -- --url https://x.com/i/lists/YOUR_LIST_ID --profile 2
# Show browser window (non-headless)
npm run scrape:twitter -- --no-headless
# Force regenerate cached data
npm run scrape:twitter -- --no-cacheFor a single X profile, use:
# Default: https://x.com/aimikoda, down to 2026-02-07 inclusive
npm run scrape:twitter:user
# Resume is enabled by default through crawl-state.json
npm run scrape:twitter:user -- --profile 2
# Start over
npm run scrape:twitter:user -- --fresh
# Custom profile / cutoff
npm run scrape:twitter:user -- --url https://x.com/aimikoda --handle aimikoda --until 2026-02-07
# Optional: also download images
npm run scrape:twitter:user -- --download-imagesOutput defaults to other-task/twitter-task/twitter-user-aimikoda-2026-02-07/.
Each post is saved as blog-0001-<tweet-id>/content.md; progress is recorded in crawl-state.json so interrupted runs skip previously saved post URLs.
twitter-2026-03-03/
blog-1/
content.md # Tweet text with metadata
image-1.jpg # High-res images
image-2.jpg
blog-2/
content.md
...
✅ Cached by date - If twitter-YYYY-MM-DD/ exists, scraping is skipped.
Use --no-cache to force regeneration.
Scrapes posts (text + images) from Weibo group feeds with support for reposts.
Script: other-task/weibo-task/scrape-weibo-list.js
# Basic usage (default: 5 posts, profile 1)
npm run scrape:weibo
# With custom arguments
npm run scrape:weibo -- --count 10 --since today
npm run scrape:weibo -- --profile 2 --since 2d
npm run scrape:weibo -- --no-cachenode other-task/weibo-task/scrape-weibo-list.js [options]| Parameter | Type | Default | Description |
|---|---|---|---|
--profile <id> |
number | 1 |
Browser profile ID to use |
--count <n> |
number | 5 |
Number of posts to scrape |
--since <when> |
string | - | Time filter (same as Twitter) |
--url <url> |
string | Built-in | Custom Weibo group URL |
--output-dir <dir> |
string | weibo-YYYY-MM-DD/ |
Output directory |
--headless |
flag | false | Run browser in headless mode |
--scroll-delay <ms> |
number | 2000 |
Delay between scrolls (ms) |
--no-cache |
flag | - | Force regenerate (ignore cache) |
--help |
flag | - | Show help message |
# Latest 5 posts (default)
npm run scrape:weibo
# All posts from today
npm run scrape:weibo -- --since today
# Last 2 days, max 20 posts
npm run scrape:weibo -- --since 2d --count 20
# Custom group URL with profile 3
npm run scrape:weibo -- --url https://weibo.com/mygroups/YOUR_GROUP --profile 3
# Force regenerate
npm run scrape:weibo -- --no-cacheweibo-2026-03-03/
blog-1/
content.md # Post text with metadata
image-1.jpg # Best quality images
image-2.jpg
blog-2/
content.md
...
✅ Cached by date - If weibo-YYYY-MM-DD/ exists, scraping is skipped.
Use --no-cache to force regeneration.
Scrapes posts from a specific Weibo user's profile page.
Script: other-task/weibo-task/scrape-weibo-user.js
# Must provide user URL
npm run scrape:weibo-user -- --url https://weibo.com/u/USER_ID
npm run scrape:weibo-user -- --url https://weibo.com/u/USER_ID --count 20node other-task/weibo-task/scrape-weibo-user.js <url> [options]
# or
node other-task/weibo-task/scrape-weibo-user.js --url <url> [options]| Parameter | Type | Default | Description |
|---|---|---|---|
--url <url> |
string | Required | Weibo user profile URL |
<url> |
string | Required | Weibo user URL (positional) |
--profile <id> |
number | 1 |
Browser profile ID to use |
--count <n> |
number | 50 |
Max number of posts to scrape |
--start <n> |
number | 0 |
Skip first N posts, start from N+1 |
--since <when> |
string | - | Time filter (same as Twitter) |
--output-dir <dir> |
string | weibo-user-{uid}-YYYY-MM-DD/ |
Output directory |
--headless |
flag | false | Run browser in headless mode |
--scroll-delay <ms> |
number | 2000 |
Delay between scrolls (ms) |
--help |
flag | - | Show help message |
# Scrape user's latest 50 posts
npm run scrape:weibo-user -- https://weibo.com/u/1234567890
# Latest 20 posts
npm run scrape:weibo-user -- --url https://weibo.com/u/1234567890 --count 20
# All posts from today
npm run scrape:weibo-user -- https://weibo.com/u/1234567890 --since today
# Skip first 10 posts, scrape next 20
npm run scrape:weibo-user -- https://weibo.com/u/1234567890 --start 10 --count 20
# Use profile 2
npm run scrape:weibo-user -- https://weibo.com/u/1234567890 --profile 2weibo-user-1234567890-2026-03-03/
blog-1/
content.md
image-1.jpg
blog-2/
content.md
...
❌ No cache - Always fetches fresh data (user posts are frequently updated).
Scrapes futures market data (news, stock info, forum posts) from Eastmoney.
Script: other-task/eastmoney-task/scrape-eastmoney-futures.js
# Scrape default symbols (ma, pp, eg, sc)
npm run scrape:eastmoney
# Scrape specific symbols
npm run scrape:eastmoney -- ma pp eg
npm run scrape:eastmoney -- au ag cu # Gold, silver, copper
npm run scrape:eastmoney -- --no-cache # Force regeneratenode other-task/eastmoney-task/scrape-eastmoney-futures.js <symbol>... [options]| Parameter | Type | Default | Description |
|---|---|---|---|
<symbol>... |
string[] | ma pp eg sc |
Futures symbols to scrape |
--output-dir <dir> |
string | YYYY-MM-DD_eastmoney_review/ |
Output directory |
--no-headless |
flag | - | Show browser window |
--no-cache |
flag | - | Force regenerate all symbols |
--help |
flag | - | Show help message |
ma- Methanol (甲醇)pp- Polypropylene (聚丙烯)eg- Ethylene Glycol (乙二醇)sc- Crude Oil (原油)ta- PTAau- Gold (黄金)ag- Silver (白银)cu- Copper (铜)
# Scrape default symbols
npm run scrape:eastmoney
# Scrape specific symbols
npm run scrape:eastmoney -- ma pp
# Scrape metals
npm run scrape:eastmoney -- au ag cu
# Show browser window
npm run scrape:eastmoney -- ma pp --no-headless
# Force regenerate (ignore cache)
npm run scrape:eastmoney -- ma pp eg --no-cache2026-03-03_eastmoney_review/
ma/
news/
1.md # Full news articles
2.md
stock.md # Price & market data
blog.md # Forum posts (title, author, date, stats)
pp/
news/
stock.md
blog.md
...
✅ Cached by date + symbol - Each symbol is cached independently.
Running npm run scrape:eastmoney -- ma pp eg twice will:
- First run: Scrape all three (MA, PP, EG)
- Second run: Skip all (cached)
- Add new symbol:
ma pp eg au→ Only scrape AU
Use --no-cache to force regenerate all symbols.
Fetches and analyzes macro-economic and industry data from official Chinese sources with optional LLM analysis.
Script: other-task/macro-news-task/fetch-macro-news.js
# Fetch all sources with LLM analysis
npm run fetch:macro-news
# Fetch macro data only
npm run fetch:macro-news:macro
# Fetch industry data only
npm run fetch:macro-news:industry
# With custom arguments
npm run fetch:macro-news -- --no-llm --concurrency 3
npm run fetch:macro-news -- --source M2_M1_M0 --source CPI_PPI_PMI
npm run fetch:macro-news -- --no-cache # Force regeneratenode other-task/macro-news-task/fetch-macro-news.js [options]| Parameter | Type | Default | Description |
|---|---|---|---|
--category <type> |
string | all | Filter: macro or industry |
--source <name> |
string | - | Specific data source (repeatable) |
--output-dir <path> |
string | ./output |
Output directory |
--headless |
flag | true | Run browser in headless mode |
--no-headless |
flag | - | Show browser window |
--no-llm |
flag | - | Skip LLM analysis (scrape only) |
--no-cache |
flag | - | Force regenerate report |
--concurrency <n> |
number | 2 |
Concurrent browsers |
--timeout <ms> |
number | 30000 |
Page load timeout (ms) |
--help |
flag | - | Show help message |
Macro Sources:
M2_M1_M0- Money supply (PBC)CPI_PPI_PMI- Inflation & manufacturing (Stats Bureau)利率_LPR- Interest rates (PBC)外汇储备- Foreign reserves (SAFE)进出口数据- Trade data (Customs)财政收支- Fiscal data (MOF)能源产量- Energy production- More...
Industry Sources:
中汽协销量- Auto sales (CAAM)港交所交易量- HKEX trading volume巨潮资讯网公告- CNINFO announcements
# Fetch all sources with LLM
npm run fetch:macro-news
# Macro data only
npm run fetch:macro-news:macro
# Industry data only
npm run fetch:macro-news:industry
# Specific sources
npm run fetch:macro-news -- --source M2_M1_M0 --source CPI_PPI_PMI
# No LLM analysis (faster)
npm run fetch:macro-news -- --no-llm
# Higher concurrency & timeout
npm run fetch:macro-news -- --concurrency 4 --timeout 45000
# Show browser window
npm run fetch:macro-news -- --no-headless
# Force regenerate cached report
npm run fetch:macro-news -- --no-cacheFile: output/macro-news-report-YYYY-MM-DD.md
Contents:
- Table of contents by category (macro/industry)
- Data source URLs and access status
- Raw extracted content (collapsed details)
- AI analysis for each source (if LLM enabled)
- Statistics summary
Create other-task/macro-news-task/.env:
VOLCENGINE_API_KEY=your-volcengine-api-key
VOLCENGINE_MODEL=doubao-pro-32k
LLM_ENDPOINT=https://ark.cn-beijing.volces.com/api/v3
LLM_MAX_TOKENS=2000
LLM_TEMPERATURE=0.7Both VOLCENGINE_API_KEY and VOLCENGINE_MODEL are required.
✅ Cached by date - If macro-news-report-YYYY-MM-DD.md exists, fetching is skipped.
Use --no-cache to force regeneration.
Runs the macro news fetching as a structured workflow with step-level progress tracking and resume capability.
Script: other-task/macro-news-task/run-workflow.js
# Run full workflow
npm run workflow:macro-news
# With custom arguments
npm run workflow:macro-news -- --concurrency 3 --no-llm
npm run workflow:macro-news -- --resume # Resume from saved state
npm run workflow:macro-news -- --no-cache # Clear state, fresh startnode other-task/macro-news-task/run-workflow.js [options]| Parameter | Type | Default | Description |
|---|---|---|---|
--category <type> |
string | all | Filter: macro or industry |
--source <name> |
string | - | Specific data source (repeatable) |
--output-dir <path> |
string | ./output |
Output directory |
--headless |
flag | true | Run browser in headless mode |
--no-headless |
flag | - | Show browser window |
--no-llm |
flag | - | Skip all LLM analysis steps |
--concurrency <n> |
number | 2 |
Concurrent steps |
--timeout <ms> |
number | 30000 |
Page load timeout (ms) |
--retries <n> |
number | 2 |
Retry count for failed steps |
--resume |
flag | - | Resume from saved state |
--no-cache |
flag | - | Clear state file, start fresh |
--dry-run |
flag | - | Show workflow structure only |
--help |
flag | - | Show help message |
# Run full workflow
npm run workflow:macro-news
# Resume interrupted workflow
npm run workflow:macro-news -- --resume
# Clear state and restart
npm run workflow:macro-news -- --no-cache
# Show workflow structure without running
npm run workflow:macro-news -- --dry-run
# Custom concurrency and retries
npm run workflow:macro-news -- --concurrency 3 --retries 3
# Skip LLM analysis
npm run workflow:macro-news -- --no-llmEach data source creates a pipeline:
fetch-page- Scrape source URLanalyze- LLM analysis of contentgenerate-report- Compile final report
State file: output/workflow-state.json
- Automatically saves progress after each step
--resumecontinues from last completed step--no-cacheclears state for fresh start- Each step tracks: status, output, error, retry count
- State:
output/workflow-state.json(progress tracking) - Report:
output/macro-news-report-YYYY-MM-DD.md(final report)
🔄 State-based resume (not traditional cache):
- Uses
workflow-state.jsonto track step completion - Auto-resumes if state exists
--resumeexplicitly continues from state--no-cacheclears state for fresh start
All scrapers (except Eastmoney) use persistent browser profiles to maintain login sessions.
# Launch browser with profile ID
npm run browser 1
# Or directly
node launch-browser.js 1This opens Chrome with:
- Persistent storage at
browser-profiles/browser-1/ - Interactive welcome page with login instructions
- No auto-close (stays open for manual login)
Steps:
- Run
npm run browser 1 - Log in to required platforms:
- Twitter/X: Visit x.com and log in
- Weibo: Visit weibo.com and log in
- Close browser when done
- Your session is saved!
# Create multiple profiles
npm run browser 1 # Profile 1
npm run browser 2 # Profile 2
npm run browser 3 # Profile 3
# Use in scrapers
npm run scrape:twitter -- --profile 1
npm run scrape:twitter -- --profile 2
npm run scrape:weibo -- --profile 3Use cases:
- Multiple accounts
- Different login sessions
- Proxy rotation
- Parallel scraping
Each task has optimized cache logic:
| Task | Cache Strategy | Auto-invalidate | Override |
|---|---|---|---|
| Date-based | Daily | --no-cache |
|
| Weibo List | Date-based | Daily | --no-cache |
| Weibo User | ❌ No cache | Always fresh | N/A |
| Eastmoney | Date + Symbol | Daily | --no-cache |
| Macro News | Date-based | Daily | --no-cache |
| Macro Workflow | State resume | Manual | --no-cache |
Date-based cache:
- Output directory/file includes date:
twitter-2026-03-03/ - Scraper checks if it exists before running
- If exists: Fast exit with cache hit message
- If not exists: Run scraper normally
- Next day: New date → automatic invalidation
Symbol-based cache (Eastmoney):
- Each symbol cached independently
ma ppcached → Addeg→ Only scrapeseg- All cached → Fast exit
- Use
--no-cacheto regenerate all
No cache (Weibo User):
- Always fetches fresh data
- User posts are frequently updated
- Ensures accuracy
State resume (Workflow):
- Not a traditional cache
- Tracks step completion in
workflow-state.json --resumecontinues from last step--no-cacheclears state
# First run: fresh scrape
$ npm run scrape:twitter
🚀 Launching browser...
✅ Scraped 10 tweets
# Second run: cache hit
$ npm run scrape:twitter
📦 Cache hit: twitter-2026-03-03/ already exists
Contains 10 blog(s)
Use --no-cache to regenerate
# Force regenerate
$ npm run scrape:twitter -- --no-cache
🗑️ Clearing existing output
🚀 Launching browser...(Optional) Configure proxies for browser profiles.
Create proxy-config.json in project root:
[
{
"server": "proxy1.example.com:8080",
"username": "user1",
"password": "pass1"
},
{
"server": "proxy2.example.com:8080",
"username": "user2",
"password": "pass2"
},
{
"server": "proxy3.example.com:8080",
"username": "user3",
"password": "pass3"
}
]Profile index maps to proxy array index:
- Profile 1 → Proxy 1 (array[0])
- Profile 2 → Proxy 2 (array[1])
- Profile 3 → Proxy 3 (array[2])
# Create profile 1 with proxy 1
npm run browser 1
# Scrape with profile 2 (uses proxy 2)
npm run scrape:twitter -- --profile 2Proxies work with:
- ✅ Twitter scraper
- ✅ Weibo scrapers (list + user)
- ✅ Browser launcher
- ❌ Eastmoney (no profile, no proxy)
- ❌ Macro news (no profile, no proxy)
Error: "❌ Not logged in to X/Weibo"
Solution:
- Create/refresh profile:
npm run browser <profile-id> - Log in to the platform
- Close browser
- Run scraper again
Error: "❌ Browser profile not found"
Solution:
# Create the missing profile
npm run browser 1 # or 2, 3, etc.Symptoms: Scraper hangs or times out
Solutions:
- Check internet connection
- Increase scroll delay:
--scroll-delay 5000 - Increase timeout (macro only):
--timeout 60000 - Show browser to debug:
--no-headless - Check if platform changed page structure
Symptoms: Some images missing in output
Causes: Network issues, invalid URLs, rate limiting
Solutions:
- Scraper auto-retries with fallback URLs
- Check console output for specific errors
- Reduce concurrency if rate-limited
- Increase
--scroll-delayto give images time to load
Error: "LLM call failed" or missing analysis
Solutions:
-
Verify
.envconfiguration:cd other-task/macro-news-task cat .env # Should show LLM_API_KEY
-
Check API quota and rate limits
-
Skip LLM to test scraping only:
npm run fetch:macro-news -- --no-llm
-
Test with single source:
npm run fetch:macro-news -- --source M2_M1_M0
Problem: Stale cached data
Solution:
# Force regenerate with --no-cache
npm run scrape:twitter -- --no-cache
npm run fetch:macro-news -- --no-cache
npm run scrape:eastmoney -- ma pp --no-cacheProblem: Want to skip cache permanently
Solution: Delete cached output:
# Twitter
rm -rf other-task/twitter-task/twitter-2026-03-03/
# Weibo
rm -rf other-task/weibo-task/weibo-2026-03-03/
# Eastmoney
rm -rf other-task/eastmoney-task/2026-03-03_eastmoney_review/
# Macro news
rm other-task/macro-news-task/output/macro-news-report-2026-03-03.mdProblem: Workflow can't resume or shows errors
Solution: Clear state and restart
npm run workflow:macro-news -- --no-cacheOr manually:
rm other-task/macro-news-task/output/workflow-state.json
npm run workflow:macro-news- Show browser window: Add
--no-headlessto see what's happening - Check terminal logs: All scrapers print detailed progress
- Test with small counts: Use
--count 1for quick tests - Verify platform access: Open URLs manually in browser
- Check rate limits: Reduce
--concurrencyif hitting limits
# Browser Profiles
npm run browser # Launch browser profile 1
npm run browser 2 # Launch browser profile 2
# Twitter
npm run scrape:twitter # Latest 5 tweets, cache enabled
npm run scrape:twitter -- --count 10 # Latest 10 tweets
npm run scrape:twitter -- --since today # All tweets from today
npm run scrape:twitter -- --no-cache # Force regenerate
# Weibo List
npm run scrape:weibo # Latest 5 posts, cache enabled
npm run scrape:weibo -- --count 10 # Latest 10 posts
npm run scrape:weibo -- --since today # All posts from today
npm run scrape:weibo -- --no-cache # Force regenerate
# Weibo User
npm run scrape:weibo-user -- <url> # User URL required (no cache)
npm run scrape:weibo-user -- https://weibo.com/u/1234567890
# Eastmoney
npm run scrape:eastmoney # Default symbols, per-symbol cache
npm run scrape:eastmoney -- ma pp eg # Specific symbols
npm run scrape:eastmoney -- --no-cache # Force regenerate all
# Macro News
npm run fetch:macro-news # All sources, cache enabled
npm run fetch:macro-news:macro # Macro only
npm run fetch:macro-news:industry # Industry only
npm run fetch:macro-news -- --no-llm # Skip LLM analysis
npm run fetch:macro-news -- --no-cache # Force regenerate
# Macro Workflow
npm run workflow:macro-news # Full workflow with state
npm run workflow:macro-news -- --resume # Resume from state
npm run workflow:macro-news -- --no-cache # Clear state, restart
npm run workflow:macro-news -- --dry-run # Show structure onlyarticle-generator/
├── browser-profiles/ # Persistent browser sessions
│ ├── browser-1/ # Profile 1 data
│ ├── browser-2/ # Profile 2 data
│ └── ...
├── other-task/
│ ├── twitter-task/
│ │ ├── scrape-twitter-list.js
│ │ └── twitter-YYYY-MM-DD/ # Output
│ ├── weibo-task/
│ │ ├── scrape-weibo-list.js
│ │ ├── scrape-weibo-user.js
│ │ ├── weibo-YYYY-MM-DD/ # List output
│ │ └── weibo-user-*-YYYY-MM-DD/ # User output
│ ├── eastmoney-task/
│ │ ├── scrape-eastmoney-futures.js
│ │ └── YYYY-MM-DD_eastmoney_review/ # Output
│ └── macro-news-task/
│ ├── fetch-macro-news.js
│ ├── run-workflow.js
│ ├── data-sources.js
│ ├── llm-client.js
│ ├── workflow-engine.js
│ ├── .env # LLM config
│ └── output/
│ ├── macro-news-report-YYYY-MM-DD.md
│ └── workflow-state.json
├── launch-browser.js # Browser profile launcher
├── proxy-config.json # Proxy configuration (optional)
├── package.json
└── README.md
MIT
yanan.wu
Contributions welcome! Please:
- Test changes thoroughly
- Update documentation
- Follow existing code style
- Add examples for new features
For issues or questions:
- Check Troubleshooting section
- Review error messages carefully
- Test with
--no-headlessto see browser behavior - Verify platform website hasn't changed structure