Skip to content

Clusters: majors refresh + Epoch satellite cross-validation + freshness stamping - #26

Draft
aurph wants to merge 4 commits into
mainfrom
feat/clusters-freshness
Draft

aurph wants to merge 4 commits into
mainfrom
feat/clusters-freshness

Conversation

@aurph

@aurph aurph commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Two builds, one theme: the clusters dataset stops rotting in public, and the last freshness blind spot closes. Independent of PR #25 (branched off main; merge in either order — no conflicts).

Commit 1: clusters majors refresh (was 62 days stale)

clusters.json last moved June 26. This is a majors-focused, verified-facts pass — every change sourced, mostly primaries — not a re-verification of all 238 entries (that wants a fresh research session; this one's search budget was spent on the deals work).

Corrections:

  • Santa Teresa was tracked twicestargate-dona-ana and the STACK "Project Jupiter" entry are the same campus, silently double-counting 1 GW planned. Merged (STACK's precise site coordinates kept, sources unioned).
  • Stargate Abilene brought to live reality per Epoch's tracker (re-fetched today): 4 of 8 buildings live, rated 421 MW IT, ~202k GPUs live vs the 450k announced target; full buildout ~Nov 2026; the 600 MW expansion was scrapped in March.
  • Stargate Lordstown 1,500 → 300 MW (estimated) per Epoch's site assessment — the announced ambition survives in the notes, the assessed reality is what's booked. Milam 1,500 → 1,200 per OpenAI/SB Energy primaries.
  • Hyperion's note claimed "10 Entergy gas plants" — only three are approved (~2.26 GW, LPSC Aug 2025); seven more pend a ~Nov 2026 vote. Corrected, plus the July $50B+ expansion and full-build ~2032.

Updates: Colossus 1 TVA grid power ~300 MW + Clean Air Act litigation noted; Colossus 2 unpermitted-turbine record + NAACP suit noted; Ellendale/Polaris Forge leases now 400 MW with building 2 delivered July 1; Helios approved capacity past 1.6 GW; IREN Horizon 1 accepted by Microsoft Aug 13; Stargate Michigan's DTE supply-agreement details.

Added (4): OpenAI Project Camellia (3.2 GW, Effingham GA, Georgia Power); Microsoft's Kilby campus at Pecos (2 GW, Chevron); AWS/STACK Caddo-Bossier ($18B, MW undisclosed so zero booked, Amazon); Savannah River Site (selection stage, DOE/NNSA).

Net: 238 clusters, planned 129.1 GW, and the GPU total honestly deflates 1.58M → 1.33M (Abilene's target number replaced by its live count). New deal-row links (Kilby, Camellia) are deliberately not wired via linkedDeal yet — those ids live on PR #25; link after both merge.

Commit 2: datacenters freshness stamping (audit gap)

datacenters.json reported "unknown" freshness forever — it's a bare array with no date field, so the monitor could never alarm on it. The ingester now maintains a sidecar (datacenters-freshness.json) with the news-scanner semantics: lastChecked moves on every completed scan, lastRefreshed only when an approval changes the approved dataset. The registry reads the sidecar; the data file's shape is untouched.

Seeded 2026-05-19 — the last committed change of datacenters.json per git — so the monitor tells the truth immediately: that dataset is stale and its in-process 6h refresher (long suspected unreliable on autoscale) is now actually being watched.

Commit 3 (Sept 1): Epoch AI cross-validation — dedup, calibrate, recover 3.7 GW of live power

Cross-validated the full registry against Epoch AI's satellite-verified dataset (86 sites, CC-BY; direct CSV at epoch.ai/data/data_centers/data_centers.csv). 56 Epoch rows matched 58 of ours; marquee sites agree (Abilene byte-identical at 421 MW IT). What it exposed, now fixed:

  • Five internal duplicate pairs merged — Fairwater Atlanta, Goodnight, Susquehanna, Project Jade, and the Kilby pair this branch introduced (caught by the cross-check).
  • ~3.1 GW of satellite-visible live IT power was invisible: phased campuses sat at construction/rated-0 while first phases run (Prometheus 560 MW, Pryor Creek 370, five AWS/Meta campuses, Helios 130...). Flipped to operational-with-build-continuing per the IREN convention.
  • Stale figures raised: Colossus 2 350 → 950 MW IT (Epoch: 946, 1.11M H100e, tracking to ~1,530 by early 2027), Fairwater Atlanta → 636, Google New Albany → 453. Overstated figures cut: Eagle Mountain 520 → 150 (full-campus capacity was stored as rated), three more Meta campuses ~3x over, Ellendale down to 68 MW live (delivered ≠ energized), Fort Wayne back to construction (no live load). Denton's 16k GPU count withdrawn as inconsistent with its 262 MW. Dona Ana planned → 1,750 MW IT.
  • Four verified missed Google sites added (Columbus 303 MW, Bristow 238, Omaha 237, Cedar Rapids u/c). Every calibrated field is flagged estimated and carries the Epoch source; CC-BY attribution added to the dataset methodology.

Net: 237 clusters; operational 65 → 80 at 11.2 → 14.9 GW live. Follow-up for a later pass: 15 further Epoch-verified US sites (~1.5 GW, incl. the QTS Richmond trio whose overlap with meta-henrico-va needs manual resolution) and the 11 non-US sites if the registry ever goes global.

Verification

Commits 1-2: 486/486 tests (2 new), tsc clean, build green. Commit 3: full gate re-run green (exit 0). Changed: clusters.json, datacenter-ingester.ts, freshness-registry.ts, the new sidecar, and the ingester tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FY4ziPhfu8VBPFVy5cDMNy

aurph added 2 commits August 27, 2026 16:32
Verified-facts pass over the major clusters, sourced per entry:
Abilene to live reality (4/8 buildings, 421MW IT, ~202k GPUs vs the
450k target; 600MW expansion scrapped); Lordstown 1500->300 per
Epoch's site assessment; Milam 1500->1200 per the primary releases;
Hyperion notes corrected (3 Entergy CCGTs approved, 7 pending the
LPSC vote - the '10 plants' claim was premature) + $50B/full ~2032;
Colossus 1 TVA grid now ~300MW + CAA litigation noted; Ellendale
leases 400MW w/ building 2 delivered; Helios approved past 1.6GW;
IREN Horizon 1 accepted by Microsoft 8/13. Merged the duplicate
Santa Teresa entries (stargate-dona-ana + STACK Project Jupiter were
the same campus, double-counting 1GW). Added 4 verified sites:
OpenAI Camellia (3.2GW, Effingham GA), Microsoft Kilby campus
(Pecos), AWS/STACK Caddo-Bossier ($18B, MW undisclosed, 0 booked),
Savannah River (Amentum, selection stage). 238 clusters, planned
129.1GW; GPU total honestly deflates 1.58M->1.33M. 484 tests green.
Majors-focused pass, not all 238 re-verified; full sweep needs a
fresh search-budget session.
…s age

datacenters.json is a bare array whose consumers expect that shape, so
the ingester now maintains a sidecar envelope
(datacenters-freshness.json) with the news-scanner semantics:
lastChecked moves on every completed scan, lastRefreshed only when an
approval actually changes the approved dataset. The freshness registry
reads the sidecar instead of kind:none, closing the audit gap where
this dataset reported 'unknown' forever and could never alarm.

Sidecar seeded 2026-05-19 = the last committed change of
datacenters.json (git-verified), so the monitor tells the truth from
day one: that dataset is stale and its in-process 6h refresher needs
watching. stampPath is optional so path-injecting tests keep working.

486 tests (2 new), tsc, build green.
… live

Cross-validated all 238 entries against Epoch AI's satellite-verified
dataset (86 sites, CC-BY, epoch.ai/data/ai-data-centers): 56 Epoch rows
matched 58 of ours, and the marquee sites agree (Abilene byte-identical
at 421MW IT).

Fixed what it exposed: FIVE internal duplicate pairs merged (Fairwater
Atlanta, Goodnight, Susquehanna, Project Jade, and the Kilby pair this
branch itself introduced - mea culpa); ~3.1GW of satellite-visible live
IT power was invisible because phased campuses sat at construction/
rated-0 (Prometheus 560MW, Pryor Creek 370, five AWS/Meta campuses,
Helios 130...) - flipped to operational-with-build-continuing per the
IREN precedent; stale figures raised (Colossus 2 350->950MW IT,
Fairwater Atlanta ->636, Google New Albany ->453); full-campus figures
stored as rated cut to measured live (Eagle Mountain 520->150, three
more Meta campuses ~3x over); Ellendale corrected down to 68MW live
(delivered != energized); Denton GPU count withdrawn as implausible;
Dona Ana planned ->1,750MW IT. Four verified missed Google sites added
(Columbus 303MW, Bristow 238, Omaha 237, Cedar Rapids u/c). Every
calibrated field flagged estimated + sourced to Epoch (CC-BY credit in
entry sources and dataset methodology).

Net: 237 clusters, operational 65->80 @ 11.2->14.9GW live. Follow-up
(PR body): 15 further missed US sites incl. the QTS Richmond trio need
per-site placement. Full gate green (test/tsc/build, exit 0).
@aurph aurph changed the title Clusters majors refresh + datacenters freshness stamping Clusters: majors refresh + Epoch satellite cross-validation + freshness stamping Sep 1, 2026
Two advisories published upstream since the last merge (browserslist
unbounded memory growth, HIGH; postcss-selector-parser AST recursion,
low) trip CI's npm-audit gate on every new push, blocking the whole
open PR train. npm audit fix; 0 vulnerabilities after; tests/tsc/build
green. Lockfile-only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant