Full-stack application that scrapes the 10 major Spanish newspapers, groups headlines covering the same event via semantic clustering, and quantifies media bias using three academically validated NLP techniques. Results are displayed through an interactive dark-themed visualizer.
Zero local dependencies. Everything runs inside Docker.
# 1. Clone and configure
cp .env.example .env # fill in your Azure OpenAI credentials
# 2. Start the stack (PostgreSQL + backend + frontend)
docker compose up --build -d
# 3. Populate the database (first time — takes ~5 min)
docker compose exec backend python -m app.cli populate
# 4. Open the app
# Frontend → http://localhost
# API → http://localhost:8000/api/healthRSS feeds (24 feeds, 10 newspapers)
│
▼
┌─────────┐ ┌────────────────────────┐ ┌───────────────┐
│ Scraper │────▶│ sentence-transformers │────▶│ HDBSCAN │
│ (httpx + │ │ paraphrase-multilingual │ │ clustering │
│ feedpar) │ │ -mpnet-base-v2 │ │ + coherence │
└─────────┘ └────────────────────────┘ └───────┬───────┘
│
┌─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────┐
│ Bias analysis (per cluster) │
│ │
│ 1. Target-dependent Sentiment (pysentimiento + spaCy) │
│ 2. Word Choice & Labeling (charged lexicon + Jaccard) │
│ 3. Coverage Gap (which papers ignored the story) │
│ │
│ 4. Political beneficiary → Azure OpenAI GPT-4o │
└──────────────────────────────────────────────────────────┘
│
▼
PostgreSQL (persistent, all history preserved)
│
▼
FastAPI ──▶ React frontend
24 RSS feeds across 10 newspapers, scraped concurrently with httpx + feedparser. Headlines are deduplicated by URL both in-memory and in PostgreSQL (ON CONFLICT DO NOTHING).
| Newspaper | Feeds | Typical volume |
|---|---|---|
| El País | 1 (portada) | ~140 |
| El Mundo | 4 (portada, españa, economía, internacional) | ~160 |
| ABC | 5 (portada, última hora, economía, internacional, opinión) | ~80 |
| La Vanguardia | 1 (home) | ~140 |
| El Confidencial | 4 (españa, economía, mundo, sociedad) | ~57 |
| La Razón | 5 (españa, economía, internacional, sociedad, opinión) | ~75 |
| elDiario.es | 1 | ~100 |
| InfoLibre | 1 | ~70 |
| OK Diario | 1 | ~50 |
| 20 Minutos | 1 | ~210 |
Each headline is embedded using paraphrase-multilingual-mpnet-base-v2 (768-dim, L2-normalized). Embeddings are persisted in the headlines table as JSONB — only new headlines without an existing embedding are processed.
HDBSCAN groups headlines from the last 48 hours into clusters representing the same news event. Three filters are applied:
min_cluster_size=3,min_samples=2(conservative to avoid loose groupings)- Clusters with a single newspaper are discarded (nothing to compare)
- Intra-cluster coherence (average pairwise cosine similarity) must exceed
0.45
Three techniques applied per cluster:
Target-dependent Sentiment Analysis — spaCy (es_core_news_lg) detects political entities (NER + curated list of 25 Spanish political actors). pysentimiento (Spanish-native model) measures sentiment in [-1, +1] toward each entity. Result: {newspaper: {party: score}}.
Word Choice & Labeling (WCL) — Compares vocabulary across newspapers for the same event. A lexicon of ~60 emotionally charged Spanish words detects loaded language. Framing diversity is quantified via average pairwise Jaccard distance between headline token sets.
Coverage Gap — Identifies which of the 10 newspapers did NOT cover the event. Systematic omission patterns are tracked across clusters to detect bias by omission.
Azure OpenAI GPT-4o receives the headlines and sentiment scores for each cluster and determines:
- Which political actor benefits most from the aggregate coverage
- Which newspaper shows the strongest favorable bias
- Step-by-step reasoning (always exposed in the UI for transparency)
| Layer | Technologies |
|---|---|
| Frontend | React 19, Vite 6, TailwindCSS 4, Framer Motion, Recharts, React Query |
| Backend | Python 3.11, FastAPI, Pydantic v2 |
| Database | PostgreSQL 16, SQLAlchemy 2.0 (async), asyncpg |
| NLP | spaCy es_core_news_lg, sentence-transformers, pysentimiento, HDBSCAN |
| LLM | Azure OpenAI GPT-4o |
| Infra | Docker Compose (local), Railway (production) |
All commands run inside the backend container:
docker compose exec backend python -m app.cli <command>| Command | Description |
|---|---|
populate |
Full pipeline: scrape → embed → cluster → analyze → store |
recluster |
Re-cluster and re-analyze existing headlines (no scraping) |
recluster --hours 72 |
Same, but use headlines from a custom time window |
create-tables |
Create database tables (idempotent) |
backup -o file.sql |
pg_dump the entire database to a file |
backup |
Dump to stdout (pipe wherever you want) |
Headlines and embeddings persist across runs. To test different clustering settings:
# 1. Edit clustering.py (min_cluster_size, coherence_threshold, etc.)
# 2. Rebuild and re-cluster — no re-scraping, no re-embedding
docker compose up backend --build -d
docker compose exec backend python -m app.cli reclusterEach execution creates a new pipeline_run. The API always serves the latest completed run; previous runs remain in the DB for direct SQL analysis.
| Method | Route | Description |
|---|---|---|
GET |
/api/clusters |
All clusters from the latest pipeline run |
GET |
/api/clusters/{id} |
Single cluster with full bias analysis |
GET |
/api/newspapers |
Aggregated stats per newspaper |
GET |
/api/bias-map |
Scatter plot data (ideology x bias intensity) |
GET |
/api/party-media-matrix |
Sentiment heatmap: parties x newspapers |
POST |
/api/refresh |
Trigger pipeline (requires X-API-Key header) |
GET |
/api/health |
Service status, cluster count, last update timestamp |
pipeline_runs
id SERIAL PRIMARY KEY
started_at TIMESTAMPTZ
finished_at TIMESTAMPTZ
headlines_scraped INT
headlines_new INT
clusters_created INT
status VARCHAR(20) -- running | completed | failed | empty
headlines
id VARCHAR(12) PRIMARY KEY
newspaper VARCHAR(50) INDEXED
title TEXT
url TEXT UNIQUE -- deduplication key
published_at TIMESTAMPTZ INDEXED
embedding JSONB -- 768-dim float vector
scraped_at TIMESTAMPTZ
clusters
id VARCHAR(12) PRIMARY KEY
topic_summary TEXT -- GPT-4o generated
pipeline_run_id INT → pipeline_runs(id)
created_at TIMESTAMPTZ
cluster_headlines -- M2M join table
cluster_id → clusters(id)
headline_id → headlines(id)
bias_analyses
id SERIAL PRIMARY KEY
cluster_id → clusters(id) UNIQUE
entity_sentiment JSONB -- {newspaper: {party: score}}
beneficiary_entity VARCHAR(100)
beneficiary_confidence FLOAT
beneficiary_reasoning TEXT -- LLM chain-of-thought
beneficiary_sentiment_scores JSONB
coverage_gap JSONB -- [newspaper names]
framing_diversity FLOAT -- 0..1
wcl_highlights JSONB -- {newspaper: [charged_words]}
The pipeline runs as a cron job, not as an in-process scheduler.
- Deploy the backend as the
webservice - Create a second service from the same repo as a Cron Job
- Configure:
- Root directory:
backend - Start command:
python -m app.cli populate - Schedule:
0 3 * * *(3:00 AM UTC daily)
- Root directory:
- Share the same environment variables (Railway injects
DATABASE_URLfrom the Postgres addon automatically via service references)
docker compose exec backend python -m app.cli populateRailway PostgreSQL does not provide automatic backups on free/starter plans.
# Create a backup
docker compose exec backend python -m app.cli backup -o /app/backup.sql
docker compose cp backend:/app/backup.sql ./backup.sql
# Restore a backup
docker compose exec -T db psql -U mediabias mediabias < backup.sqlWhat survives without a backup: A fresh populate captures whatever is currently in the RSS feeds (~last 24h). What doesn't: all historical headlines, embeddings, clusters, and analyses older than the RSS window. Backups are critical.
| # | Service | Type | Root dir | Start command |
|---|---|---|---|---|
| 1 | PostgreSQL | Addon | — | — |
| 2 | Backend (API) | Web | backend |
uvicorn app.main:app --host 0.0.0.0 --port $PORT |
| 3 | Pipeline (nightly) | Cron | backend |
python -m app.cli populate |
| 4 | Frontend | Web | frontend |
Nginx (via Dockerfile) |
DATABASE_URL ← auto-injected by Railway Postgres addon
AZURE_OPENAI_ENDPOINT ← base URL only: https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY ← from Azure Portal → Keys and Endpoint
AZURE_OPENAI_DEPLOYMENT ← gpt-4o
AZURE_OPENAI_API_VERSION ← 2025-01-01-preview
REFRESH_API_KEY ← your own secret for POST /api/refresh
.
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI application + endpoints
│ │ ├── cli.py # CLI: populate, recluster, backup, create-tables
│ │ ├── pipeline.py # Orchestration: scrape → embed → cluster → analyze
│ │ ├── scraper.py # RSS feed scraping (httpx + feedparser)
│ │ ├── embeddings.py # sentence-transformers embedding generation
│ │ ├── clustering.py # HDBSCAN clustering + coherence filtering
│ │ ├── bias_analyzer.py # 3 bias techniques (sentiment, WCL, coverage gap)
│ │ ├── llm_client.py # Azure OpenAI GPT-4o (summarize + beneficiary)
│ │ ├── models.py # Pydantic schemas, RSS config, political entities
│ │ ├── database.py # SQLAlchemy async engine + session factory
│ │ ├── db_models.py # ORM table definitions
│ │ └── store.py # Data access layer (all DB reads/writes)
│ ├── Dockerfile
│ ├── Procfile # Railway web process
│ ├── railway.toml # Railway build/deploy config
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── pages/
│ │ │ ├── Home.jsx # Cluster feed (main view)
│ │ │ ├── Cluster.jsx # Cluster detail with bias breakdown
│ │ │ └── Newspapers.jsx # Newspaper stats + bias map
│ │ ├── components/
│ │ │ ├── ClusterCard.jsx # Event card for the feed
│ │ │ ├── ClusterDetail.jsx # Headlines with WCL highlights
│ │ │ ├── BiasMap.jsx # Scatter plot (ideology × bias intensity)
│ │ │ ├── SentimentBar.jsx # Horizontal bar chart (sentiment × entity)
│ │ │ ├── FramingScore.jsx # Gauge visualization (0–100)
│ │ │ ├── PartyMediaMatrix.jsx # Heatmap (parties × newspapers)
│ │ │ └── MethodologyNote.jsx # Expandable methodology explainers
│ │ ├── lib/
│ │ │ ├── api.js # API client (React Query)
│ │ │ └── constants.js # Colors, newspaper palette
│ │ ├── App.jsx # Router + layout
│ │ └── main.jsx # Entry point
│ ├── Dockerfile # Multi-stage: build + Nginx
│ ├── nginx.conf
│ └── package.json
├── docker-compose.yml # PostgreSQL + backend + frontend
├── .env.example
└── README.md
- Ideological position is computed, never assigned. Each newspaper's position on the left-right spectrum is derived from its aggregate sentiment toward political parties, weighted by party ideology scores. No manual labeling.
- Headlines accumulate forever. The
headlinestable is append-only (upsert by URL). Clustering operates on a sliding 48h window, but the full corpus remains available for analysis. - Pipeline runs are immutable. Each
populateorreclustercreates a newpipeline_run. The API serves the latest; old runs stay in the DB for comparison via SQL. - Spanish-native NLP.
pysentimientois trained on Spanish text — no translation step. spaCyes_core_news_lgprovides entity recognition tuned for Spanish political discourse. - LLM transparency. GPT-4o's reasoning is stored verbatim and always shown in the UI. The model never assigns scores — it interprets the NLP-generated scores.
- HDBSCAN over K-means. No need to predefine K. Handles variable-size clusters and naturally produces a noise label for headlines that don't belong to any event.