A public macro-risk dashboard that doubles as a privacy-minimized behavioral bot-detection canary.
Live at macrocanary.pages.dev — deployed on Cloudflare Pages + Workers + D1. The dashboard is a genuinely useful India-aware macro indicator board. The security layer uses that real traffic as a research canary — collecting privacy-minimized telemetry, recording honeypot hits, and running offline analysis to study bot behavior and session-level fleet patterns.
Not financial advice. This dashboard summarizes selected public macro indicators with simple scoring rules. Source dates and freshness are shown because stale data should look stale, not authoritative.
- What it is
- Why it exists
- Architecture
- Dashboard and indicators
- Privacy model
- Bot canary design
- Detection pipeline
- Synthetic harness
- Running locally
- Cloudflare deployment
- Testing
- Limitations
- Roadmap
- Task tracking
MacroCanary has two layers:
Layer 1 — Public macro dashboard. A traffic-light view of 7 selected macroeconomic indicators (6 automated via FRED, 1 manual India CPI). Every indicator shows its source, observation date, and freshness state. Nothing is hidden, smoothed, or overconfidently labelled. The overall signal is Green / Amber / Red.
Layer 2 — Bot-detection canary. Because the page is public, it attracts ordinary readers, crawlers, scrapers, and bots. The security layer collects privacy-minimized session telemetry, records honeypot-path hits, captures Cloudflare edge metadata, and runs a weekly offline Python pipeline to classify sessions and detect coordinated bot fleets.
The two layers reinforce each other: the dashboard's usefulness ensures real human traffic, making bot behavior meaningful to study alongside it.
Most bot-detection portfolios are built on fake honeypots or closed data. MacroCanary is a real public site attracting real traffic — making the bot signals authentic and the methodology defensible.
The project addresses a specific gap in standard bot detection:
Per-session WAF/CDN scoring and real-time blocking miss coordinated campaigns. Coordinated bots deliberately keep individual sessions clean. The coordination signal only exists in aggregate — across sessions, over time. Retrospective fleet analysis is the only layer that sees it.
| Layer | Scope | What it catches | What it misses |
|---|---|---|---|
| 1 — Per-session scoring | Single session | Obvious bots, zero-engagement scrapers | Coordinated bots with clean individual profiles |
| 2 — Fleet clustering | Cross-session similarity | Groups of similar sessions | Fleets that add noise to look individually normal |
| 3 — Campaign intent | Aggregate path topology | Mapping, extraction, and wave campaigns | Nothing that erases all aggregate coordination signals |
Layer 3 is MacroCanary's differentiating contribution: classifying what the fleet is collectively trying to do, not just that similar sessions exist. This signal is invisible at the session level and only emerges in retrospective analysis.
The project also demonstrates privacy-aware telemetry design: no raw IPs, bucketed timing, short retention, notice-first collection, and aggregate-only reporting — producing useful threat intelligence within GDPR/DPDP constraints.
┌──────────────────────────────┐
│ GitHub Actions (weekly cron) │
│ FRED fetch → merge → score │
│ Mon 04:17 UTC │
└────────────┬─────────────────┘
│ writes snapshot rows
v
┌──────────────┐
│ Cloudflare D1 │
│ (SQLite/edge) │
└──────┬───────┘
│ reads latest snapshot
v
Visitor ───────► Cloudflare Pages (dashboard)
│ index.html / app.js / styles.css
│ methodology.html / privacy.html
│
├── telemetry.js (notice-first, honors DNT/GPC)
│ │ POST /collect
│ v
│ Cloudflare Worker ──────────────────► D1 sessions/events
│
└── canary/honeypot routes ─────────────► D1 canary_hits
Offline detection (weekly, GitHub Actions):
D1 sessions+events+canary_hits
└── Python pipeline (Mon 04:47 UTC, after macro run)
├── extract_all() → feature vectors
├── score() → Isolation Forest anomaly flags
├── detect() → DBSCAN fleet clusters
├── fuse() → labeled verdicts
└── generate() → Markdown report
└── write_live_results() → D1 detection_runs + session_verdicts
See docs/architecture.md for component details and data flow.
| # | Indicator | Region | Source | Tier |
|---|---|---|---|---|
| 1 | Sahm Rule (3mo avg U − 12mo low) | US | FRED SAHMREALTIME |
A — Automated |
| 2 | Initial jobless claims (4wk / 26wk avg) | US | FRED ICSA |
A — Automated |
| 3 | High-yield OAS (ICE/BofA) | US | FRED BAMLH0A0HYM2 |
A — Automated |
| 4 | UST 10y–3m yield curve slope | US | FRED T10Y3M |
A — Automated |
| 5 | Brent crude (USD/bbl) | Global | FRED DCOILBRENTEU |
A — Automated |
| 6 | Copper vs 200-period MA (%) | Global | FRED PCOPPUSDM |
A — Automated |
| 7 | India CPI inflation y/y (%) | India | MoSPI (manual) | B — Manual verified |
Indicator thresholds, sources, and freshness windows are configured in macros/indicators.yaml. Adding or deferring an indicator requires only a config change — no code changes.
| State | Score | Overall (7 indicators, max 14) |
|---|---|---|
| Green | 0 | 0–3 — low visible macro stress |
| Amber | 1 | 4–7 — watchful; multiple caution signs |
| Red | 2 | 8+ — elevated stress across several indicators |
Freshness rules:
- Daily/weekly market data: stale after 10 calendar days
- Monthly economic data: stale after 45 calendar days
- Manual release data: stale after expected release + 15 days
- If >30% of indicators are stale, a freshness warning banner appears
Stateful yield-curve rule (§8.3): if the 10y–3m slope fell below −1.00% and later turned positive within 9 months, the yield-curve indicator is bumped one level worse for the post-inversion risk window. This requires comparing the current snapshot against historical snapshots — explained in detail on the methodology page.
| File | Purpose |
|---|---|
| dashboard/index.html | Main traffic-light dashboard |
| dashboard/methodology.html | Scoring rules and stateful yield-curve rule explained |
| dashboard/privacy.html | Privacy notice (required before telemetry collects) |
| dashboard/app.js | Snapshot renderer; reads snapshot.json by default, overridable via ?snapshot= param |
| dashboard/styles.css | Local stylesheet; no external dependencies |
The design is pseudonymous, not anonymous. A random session ID plus network metadata (ASN, country, JA4 bucket) plus behavioral sequence can be linkable in principle. The risk is mitigated, not eliminated, through defense-in-depth:
| Category | Detail | How it's stored |
|---|---|---|
| Session ID | Random, 24-char prefix, 30-min TTL | sessions.session_id |
| Navigation | Path sequence, referrer category (not full URL), entry type | events.path |
| Timing | Dwell time, inter-page intervals | Bucketed: none/lt_5s/5_15s/15_60s/1_3m/3m_plus |
| Engagement | Scroll depth, interaction flag, visibility state | Bucketed: none/0_25/25_50/50_75/75_100 |
| Canary hits | Disallowed paths, unlinked tokens, deep links | canary_hits table |
| Edge metadata | Country, ASN, JA4 bucket (if available), verified-bot category | Coarsened; JA4 stored as 6-char prefix only |
Raw IP address · full user-agent string · names · emails · phone numbers · precise location · keystrokes · form contents · cross-site tracking IDs · third-party ad IDs · full browser fingerprints.
| Data | Retention |
|---|---|
| Raw session events | 14 days (configurable via RETENTION_DAYS) |
| Canary hits | 30 days, then aggregate |
| Detection verdicts | Kept in session_verdicts (aggregate run-level data only in reports) |
| Published reports | Aggregate only — no per-session data |
- Notice-first: telemetry does not collect until the privacy notice has been shown and stored in localStorage (
mc_notice_seen) - DNT / Sec-GPC: if either header is set, nothing is collected
/no-telemetry: visiting this path disables collection- Sampling: configurable
TELEMETRY_SAMPLE_RATEenv var on the Worker limits collection during traffic spikes
See docs/privacy-model.md for DPDP posture, data minimization decisions, and retention enforcement.
| Type | Path example | Label strength | How it works |
|---|---|---|---|
robots_disallowed_path |
/private-for-bots-only/ |
High | Listed in robots.txt as disallowed; only crawlers ignoring robots.txt visit |
unlinked_token_path |
/canary/<token>/ |
High | Never linked publicly; requires prior knowledge or enumeration |
entryless_deep_link |
/indicators/deep-canary-detail |
Medium | Deep indicator page never surfaced in navigation |
Each canary hit writes a row to canary_hits with canary_type, path, label_strength, and reason. Canary hits label sessions retrospectively — no visitor is ever blocked or punished.
The detection pipeline uses exactly 7 labels:
| Label | Meaning |
|---|---|
likely_human |
Behavior consistent with normal reading; not proof of human |
known_good_bot |
Verified crawler or expected automation (e.g., search engine) |
benign_automation |
Non-human but expected and low-risk (e.g., uptime checks) |
honeypot_bot |
Hit a high-confidence canary/honeypot path |
likely_automation |
Session-level anomaly or scraper-like behavior |
suspected_fleet |
Cluster-level uniformity across multiple sessions |
unknown |
Insufficient evidence — ambiguity is better than fake certainty |
| Report group | Labels included |
|---|---|
| Known-good crawlers | known_good_bot |
| Benign automation | benign_automation |
| Suspicious automation | likely_automation |
| Canary-confirmed automation | honeypot_bot |
| Suspected coordinated fleet | suspected_fleet (only if fleet gates pass) |
| Unknown | unknown |
Detection runs fully offline — nothing is blocked in real time. The Python pipeline runs weekly via GitHub Actions.
sessions + events + canary_hits (from D1)
│
▼
topology analysis detection/topology.py
Build site navigation graph from all observed events
Classify topological orphan pages (accessed but never linked)
│
▼
extract_all() detection/features.py
Feature groups: navigation, timing, engagement, network/edge, fleet
New: terminal_path, orphan_access_ratio (topology-aware)
│
▼
score() detection/anomaly.py
Isolation Forest on 14 features including orphan_access_ratio
Guard: skips high-interaction sessions (interaction_ratio ≥ 0.10)
Zero-engagement rule: catches headless bots with no scroll/dwell/interaction
│
▼
detect() detection/fleet.py
Layer 2: DBSCAN clustering on path-set Jaccard + ASN distance
Layer 3: Coverage efficiency, terminal concentration, wave coordination
Campaign intent classification per cluster
│
▼
fuse() detection/fuse.py
Priority fusion: canary → verified bot → rules → fleet → anomaly → human
Every verdict carries a reasons list
│
▼
generate() detection/report.py
Campaign-typed fleet sections with intent descriptions
Evidence claim ladder enforced mechanically
Aggregate only — no per-session data in any published output
The core methodological contribution. Four signals computed per fleet cluster:
| Signal | Formula | High value means |
|---|---|---|
| Coverage efficiency | union(all paths) / (n_sessions × mean_path_count) | Systematic site mapping |
| Terminal concentration | top terminal path count / cluster size | Targeted extraction |
| Wave coordination score | CoV of inter-session entry intervals | Rate-limit evasion |
| Orphan access ratio | orphan pages hit / total pages hit | URL enumeration |
These signals are invisible at the session level. They only emerge in aggregate.
Campaign type taxonomy:
| Campaign type | Signal signature | Plain-English meaning |
|---|---|---|
site_mapping |
Coverage efficiency > 0.65, terminal concentration < 0.5 | Fleet systematically indexed the site's pages |
targeted_extraction |
Terminal concentration > 0.65 | Fleet converged on specific content from diverse routes |
wave_campaign |
Wave CoV < 0.40 | Fleet entered at regular intervals to evade rate limits |
coordinated_automation |
Fleet gate passed, no specific signature | Coordinated but unclassified intent |
| Gate | Minimum |
|---|---|
| Sessions in analysis window | 100 (MIN_SESSIONS) |
| Candidate cluster size | 5 (MIN_CLUSTER_SIZE) |
| Shared route steps | 3 (MIN_SHARED_ROUTE_STEPS) |
If gates fail: "Repeated pattern observed; insufficient volume for fleet classification." — never "Coordinated fleet detected."
| Evidence available | Maximum allowed claim |
|---|---|
| Synthetic harness only | "Pipeline validated against controlled scenarios." |
| Synthetic + canary hit | "Live canary confirmed at least one automation interaction." |
| Synthetic + ≥100 live sessions + gates pass | "Retrospective analysis found suspected coordinated automation patterns." |
| Edge fields present + offline disagreement | "Available edge-time signals and retrospective analysis disagreed." |
| Repeated across windows | "The pattern repeated across observation windows." |
Forbidden phrases: "Cloudflare failed" · "beats enterprise bot management" · "proves humans vs bots" · "predicts recession" · "coordinated fleet detected" (below gate threshold).
If live session count is below MIN_SESSIONS (100), the pipeline falls back to synthetic data. Synthetic verdicts are never written to D1.
See docs/detection-pipeline.md for full detail on all signals, feature extraction, and model parameters.
The synthetic harness validates the pipeline before live traffic exists and provides a floor that guarantees a publishable result regardless of real-world traffic luck.
13 session archetypes defined in synthetic/scenarios.yaml:
| # | Archetype | Expected label | Validates |
|---|---|---|---|
| 1 | Normal reader | likely_human |
Baseline human engagement |
| 2 | Mobile skimmer | likely_human |
Low-dwell human variant |
| 3 | Returning human | likely_human |
Backtracking, high engagement |
| 4 | Search crawler | known_good_bot |
Verified bot edge signal |
| 5 | Fast scraper | likely_automation |
Sub-second dwell, no engagement |
| 6 | Headless browser | likely_automation |
Zero engagement, regular timing |
| 7 | Robots-trap visitor | honeypot_bot |
Canary path hit |
| 8 | Fleet — identical | suspected_fleet |
Layer 2 fleet clustering |
| 9 | Fleet — noisy | suspected_fleet |
Clustering with jitter |
| 10 | Fleet — adversarial | suspected_fleet / likely_automation |
Honest evasion documentation |
| 11 | Mapping fleet | suspected_fleet |
Layer 3: coverage efficiency, orphan access |
| 12 | Extraction fleet | suspected_fleet |
Layer 3: terminal concentration |
| 13 | Wave fleet | suspected_fleet |
Layer 3: wave coordination score |
synthetic/generate.py emits sessions.json, events.json, canary_hits.json, and expected-labels.jsonl (310 sessions with ground-truth labels). Archetypes 11–13 specifically validate the Layer 3 campaign intent signals. The detection test suite validates expected vs detected labels for all 13 archetypes.
- Python 3.11+
pip install -r requirements.txt
# Fetch FRED data (requires FRED_API_KEY env var)
python macros/fetch_fred.py
# Or run the full pipeline with the scoring engine
python macros/score_snapshot.pyOutput lands in macros/runs/<run_id>/snapshots/.
Open dashboard/index.html in a browser. By default it reads dashboard/snapshot.json (the committed dev fixture). To load a different snapshot:
dashboard/index.html?snapshot=../macros/runs/<run_id>/snapshots/snapshot-<timestamp>.json
The dashboard works fully with telemetry absent — no network requests required.
python -m synthetic.generate synthetic/scenarios.yaml /tmp/synthetic-out/Outputs sessions.json, events.json, canary_hits.json to the target directory.
# Against synthetic data (no D1 needed)
python detection/run_detection.py --output-dir reports
# Against live D1 (requires Cloudflare secrets in environment)
CLOUDFLARE_API_TOKEN=... \
CLOUDFLARE_ACCOUNT_ID=... \
CLOUDFLARE_D1_DATABASE_ID=... \
python detection/run_detection.py --output-dir reports --window-days 7If fewer than 100 live sessions are found, the script falls back to synthetic data and says so in the report filename (report-<timestamp>-synthetic.md vs report-<timestamp>-live.md).
python reports/build_report.py \
--verdicts-jsonl path/to/verdicts.jsonl \
--canary-hits-json path/to/canary_hits.json \
--fleet-result-json path/to/fleet_result.json \
--output reports/my-report.mdDeployment is live. The full stack is running on Cloudflare.
| Component | URL |
|---|---|
| Dashboard (Cloudflare Pages) | macrocanary.pages.dev |
| Snapshot API (Pages Function) | /api/snapshot |
| Telemetry collector (Worker) | macrocanary-worker.nitinkoshy.workers.dev/collect |
| Database | Cloudflare D1 — macrocanary |
| Secret | Used by |
|---|---|
FRED_API_KEY |
Macro pipeline GitHub Action |
CLOUDFLARE_API_TOKEN |
Pages/Workers deploy + detection runner |
CLOUDFLARE_ACCOUNT_ID |
All Cloudflare operations |
CLOUDFLARE_D1_DATABASE_ID |
Detection runner D1 queries |
For local Worker development, copy wrangler.example.toml to wrangler.toml and fill in
your D1 database ID. The real wrangler.toml is intentionally ignored so public clones do
not inherit production infrastructure identifiers.
| Action | Schedule | Purpose |
|---|---|---|
| Macro pipeline | Mon 04:17 UTC (09:47 IST) | Fetch FRED, score indicators, write snapshot to D1 |
| Detection runner | Mon 04:47 UTC (10:17 IST) | Query D1, run pipeline, write verdicts + report |
The 30-minute gap ensures the macro snapshot is in D1 before detection runs. Schedules run only from the default branch. If a scheduled run is missed, check that the workflow is enabled in GitHub Actions and that the repo has not hit GitHub's public-repo inactivity disablement for scheduled workflows.
All tests run with standard pytest. No network calls are made in any test — fixtures and injectable callables replace external dependencies.
# Run all tests
python -m pytest
# Run specific suites
python -m pytest macros/test_scoring.py # Scoring engine (12 tests)
python -m pytest detection/test_detection.py # Detection pipeline (29 tests)
python -m pytest dashboard/test_dashboard.py # Dashboard contract (4 tests)
python -m pytest worker/test_worker_static.py # Worker static analysis (4 tests)
python -m pytest detection/test_run_detection.py # Detection runner (3 tests)| File | Tests | What it covers |
|---|---|---|
| macros/test_scoring.py | 12 | Indicator thresholds, staleness rules, stateful yield-curve bump, freshness warning logic |
| detection/test_detection.py | 47 | Topology graph + orphan detection, Layer 3 intent signals (coverage efficiency, terminal concentration, wave coordination, campaign classification), all 13 synthetic archetypes, report evidence ladder, forbidden phrase checks |
| dashboard/test_dashboard.py | 4 | Snapshot fixture shape, privacy/disclaimer language, no external trackers, local-only assets |
| worker/test_worker_static.py | 4 | D1 schema column names, canary contract, privacy invariants (grep for raw-IP patterns), DNT/GPC/sampling/retention presence |
| detection/test_run_detection.py | 3 | Synthetic fallback (no D1 writes), live-data path (writes detection_runs + session_verdicts), edge-disagreement detection |
worker/test_worker_static.py statically verifies the Worker source never writes raw IP-like fields. The detection test suite verifies no per-session identifiers appear in published reports. These are not optional.
MacroCanary/
├── macros/ # Macro data pipeline (Python)
│ ├── indicators.yaml # Indicator config — thresholds, sources, freshness
│ ├── manual-values.yaml # India CPI manual entries
│ ├── score_snapshot.py # Scoring engine and snapshot builder
│ ├── merge_source_history.py
│ ├── fetch_fred.py
│ ├── fetchers/ # FRED, World Bank, IMF, OECD fetchers
│ └── test_scoring.py
├── dashboard/ # Public front-end (HTML/CSS/JS)
│ ├── index.html
│ ├── app.js
│ ├── styles.css
│ ├── methodology.html
│ ├── privacy.html
│ ├── telemetry.js # Notice-first privacy-minimized telemetry
│ ├── robots.txt # Canary trap paths disallowed
│ ├── snapshot.json # Dev fixture (committed intentionally)
│ └── test_dashboard.py
├── worker/ # Cloudflare Worker (JavaScript)
│ ├── collect.js # /collect endpoint, sampling, retention cleanup
│ ├── canary-routes.js # Honeypot route handlers
│ └── test_worker_static.py
├── synthetic/ # Synthetic session harness
│ ├── scenarios.yaml # 13 session archetypes
│ ├── generate.py # Emits sessions/events/canary_hits JSON
│ └── expected-labels.jsonl # Ground-truth labels for 310 sessions
├── detection/ # Offline detection pipeline (Python)
│ ├── topology.py # Site graph + topological orphan page detection
│ ├── features.py # Feature extraction (incl. terminal_path, orphan_access_ratio)
│ ├── anomaly.py # Isolation Forest per-session scoring
│ ├── fleet.py # DBSCAN clustering + campaign intent classification
│ ├── fuse.py # Verdict fusion
│ ├── report.py # Campaign-typed report with evidence ladder
│ ├── run_detection.py # D1 → pipeline → D1 + report (production entry point)
│ ├── test_detection.py # 47 tests
│ └── test_run_detection.py # 3 tests
├── reports/ # Report CLI and generated outputs
│ ├── build_report.py # CLI: loads local JSON → calls report.generate → writes file
│ └── sample-report.md # Generated example (synthetic input)
├── docs/ # Documentation
│ ├── macrocanary-plan-v0_4.md # Full design plan
│ ├── contracts.md # D1 table contracts and verdict schema
│ ├── article-draft.md # Article draft — complete; live observations section pending real traffic
│ ├── architecture.md # Component architecture details
│ └── detection-pipeline.md # Detection pipeline deep-dive
├── functions/
│ └── api/
│ └── snapshot.js # Cloudflare Pages Function — serves /api/snapshot from D1
├── .github/workflows/
│ ├── macro.yml # Weekly macro pipeline (Mon 04:17 UTC)
│ └── detect.yml # Weekly detection action (Mon 04:47 UTC)
├── macrocanary-codex-tasks.md # Build task tracker
├── requirements.txt
└── .gitignore
These are stated plainly — the project is honest about its scope.
-
Thin early traffic. A new public site may receive too little human traffic for strong statistical conclusions. The synthetic harness is the floor that guarantees a publishable result even if live traffic is sparse.
-
Synthetic data is not reality. Synthetic sessions validate the pipeline's mechanics, not the internet's actual behavior.
-
Free-tier edge signals are limited. Enterprise bot-management fields (bot scores, full TLS fingerprints) may not be available on Cloudflare's free plan. The pipeline is designed to work with missing edge fields — all edge metadata is treated as empirically present, never assumed.
-
Behavioral detection is probabilistic. Privacy tools, VPNs, and human-like bots can confuse the classifier. Labels include confidence scores and reasons precisely because certainty would be dishonest.
-
Fleet detection can be evaded. Adversaries can inject timing noise. The project demonstrates a useful analytical seam, not a permanent detection advantage.
-
Economic scoring is simplified. Seven indicators with transparent thresholds. Not a macroeconomic model; not a recession forecast.
-
Manual India CPI can go stale. The monthly MoSPI release requires a manual update. The freshness mechanic surfaces this immediately when it lapses.
v1 (current) — fully deployed and live:
- Macro pipeline (FRED fetch, scoring engine, stateful yield-curve rule)
- Synthetic harness (13 archetypes, 310 sessions, ground-truth labels)
- Detection pipeline (features, Isolation Forest, DBSCAN fleet, fusion, report)
- Public dashboard (HTML/CSS/JS, methodology, privacy notice)
- Cloudflare Worker collector (telemetry.js, /collect, canary routes)
- Report CLI + article scaffold
- Detection runner with D1 integration and synthetic fallback
- Cloudflare deploy + D1 wiring (Pages + Worker + D1 live at macrocanary.pages.dev)
v1.1 — fast-follow after live data exists:
canaryctlGo CLI wrapping the Python pipeline into a portable binary- Automated India CPI source (removes the one manual update requirement)
- Repeated-across-windows detection (second weekly run confirms a pattern)
v2 — optional:
CanaryInspectorC# desktop tool for local session timeline inspection- Additional India indicators graduating from the deferred list
See macrocanary-codex-tasks.md for the full build task log, Codex session instructions for each task, and the shared contract that keeps independently-built components compatible.
Status as of 2026-06-13:
| Task | Status | Tests |
|---|---|---|
| 1 — Macro pipeline + scoring engine | Complete | 12/12 |
| 2 — Synthetic harness + detection pipeline | Complete | 26/26 |
| 3 — Public dashboard front-end | Complete | 4/4 |
| 4 — Cloudflare Worker collector + telemetry | Complete | 4/4 |
| 5 — Cloudflare deploy + wiring | Complete — live | — |
| 6 — Report CLI + article scaffold | Complete | 29/29 |
| 7 — Detection runner + scheduled action | Complete | 32/32 |
| Methodology sharpening — Layer 3 intent signals | Complete | 72/72 total |
This project is licensed under the MIT License.
The dashboard content is educational, and the detection pipeline is a research tool. See docs/macrocanary-plan-v0_4.md §5 for explicit non-goals.