diff --git a/KNOWN_RISKS.md b/KNOWN_RISKS.md index 6ce9fd2..33d9f7d 100644 --- a/KNOWN_RISKS.md +++ b/KNOWN_RISKS.md @@ -195,6 +195,24 @@ --- +## 13. Context Bloat — Token Cost Scales with Canon Size + +**Risk:** The `ContextAssembler` injects ALL canon entities into every generation prompt. This is already consuming ~1500–2000 tokens per call at 36 entities. At 100+ entities this grows linearly, exhausting daily token limits on budget provider tiers. + +**Trigger:** Every generation call. The current implementation has no cap on context entities. + +**Impact:** Token costs escalate. Groq daily TPD limit (100K tokens) gets exhausted after ~50 generations. Provider rate limits fire sooner. Unnecessary tokens inflate every prompt with entities irrelevant to the current generation type. + +**Mitigation:** +- Implement `getRelevantContext(entityType, hints)` that caps at 15 entities +- Always include: top 10 by relationship count, world-core.json, campaign context +- Fill remaining slots with type-relevant neighbors +- This reduces token overhead ~60% at current canon size + +**Status: PENDING — requires implementation before next generation run** + +--- + ## Summary of Urgency | Risk | Urgency | Needs Phase | diff --git a/NEXT_STEPS.md b/NEXT_STEPS.md index c7f7c07..c3ed71c 100644 --- a/NEXT_STEPS.md +++ b/NEXT_STEPS.md @@ -1,42 +1,38 @@ -# Next Steps — Phase 4 (Post-Deployment) +# Next Steps — IMMEDIATE NEXT SESSION -Campaign: "Echoes of the Northern Fracture" — Region: Northern Velkaris -Status: Phase 3 complete. Phase 4 not started. +## 1 task only: Fix context-assembler.ts token bloat -## Options +**Why:** `ContextAssembler` currently injects ALL 36 canon entities into every generation prompt. This uses ~1500–2000 tokens per call. At 100+ entities this will exhaust daily token limits on all providers. -### A) Push to GitHub + configure secrets + deploy -- Create GitHub repo (if not existing) -- Push `main` branch -- Add 4 secrets: CF_API_TOKEN, CF_ACCOUNT_ID, GEMINI_API_KEY, GROQ_API_KEY -- Create Cloudflare Pages project `aetherion-archive` -- Trigger first deploy via workflow_dispatch +**Required changes in `src/engine/context-assembler.ts`:** -### B) SEO improvements -- Add sitemap.xml (`pages/sitemap.xml.ts`) -- Add robots.txt -- Add schema.org structured data (WebPage, CreativeWork) to entity pages -- Add Open Graph / Twitter Card meta tags +Add function: +```typescript +getRelevantContext(entityType: string, hints?: string[]): Entity[] +``` -### C) Generate more entities (target 50+) -- Run `generate-batch` locally or via workflow_dispatch -- Target thin entity types (religions, races, regions) -- Add missing metadata to close warnings (school, power level, threat level, lifespan) - -### D) Content quality pass -- Rewrite entities with thin descriptions or content -- Fill in attributes that are empty strings -- Standardize lore tone across all 11 types +- Max **15 entities** injected into prompt (not all canon) +- Top 10 by relationship count always included +- `world-core.json` always included +- Campaign / generation context always included +- Remaining slots filled with type-relevant neighbors --- ### Canon status - 36 unique entities across 11 types -- 174 relationships, 0 orphans, 0 bidirectional gaps +- 182 relationships, 0 orphans, 0 bidirectional gaps - 1 error (forbidden-moon-rituals placeholder) - 18 warnings (non-blocking metadata) -### Astro site status -- 48 pages, static output, ~3.9s build time +### Site +- LIVE at aetherion-archive.pages.dev +- 49 pages, static output, ~5s build time - Tailwind v4, dark fantasy theme -- GitHub Actions CI/CD ready (pending secrets) +- Cloudflare Pages auto-deploys on push to master + +### Generation +- 3x/day via GitHub Actions (2am/9am/3pm UTC) +- Rotating provider strategy (balanced/cheap/quality) +- 429 → backoff → fall through +- Manual trigger with provider + count options diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index a3163c3..d778ee9 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -1,11 +1,11 @@ # Project State -## Phase: 3 — CI/CD & Deployment (COMPLETE) ## Phase: 2 — Astro Frontend (COMPLETE) +## Phase: 3 — CI/CD & Deployment (COMPLETE) ## Phase 1C — Sandbox Canon Generation (COMPLETE) ## Pre-Phase 2 Cleanup (COMPLETE) ## Version: 0.1.x -## Status: 36 unique entities, 174 relationships, 0 orphans, 0 bidirectional gaps, 1 remaining error (pre-existing). Generation system: 3 runs/day, rotating provider strategy (balanced/cheap/quality). Phase 3 complete — GitHub Actions workflows ready, SEO layer complete. Deployed on Cloudflare Pages. +## Status: 36 unique entities, 182 relationships, 0 orphans, 0 bidirectional gaps, 1 remaining error (pre-existing). Site LIVE at aetherion-archive.pages.dev. Generation: 3x/day rotating provider strategy (balanced/cheap/quality). Known issue: context-assembler injects all 36 entities per generation → token cost too high. Fix PENDING. ### What exists - Full generation pipeline (context → AI → parse → validate → retry → register → JSONL append → markdown → memory snapshot) @@ -34,7 +34,7 @@ - **Regions (1)**: northern-velkaris ### Relationships -- 174 total, 0 orphans, 0 bidirectional gaps +- 182 total, 0 orphans, 0 bidirectional gaps - 1 remaining error (forbidden-moon-rituals placeholder — pre-existing, separate entity) - 18 warnings (all non-blocking — missing school/power level/threat level/lifespan metadata) @@ -77,9 +77,9 @@ ### Phase 3 — CI/CD & Deployment (COMPLETE) - deploy.yml: install → typecheck → validate → build-and-deploy (Cloudflare Pages) +- Site LIVE at aetherion-archive.pages.dev - generate.yml: 3 daily runs (2am/9am/3pm UTC), rotating strategy (balanced/cheap/quality), generate-batch → commit → triggers deploy - DEPLOYMENT_SETUP.md: created, includes search engine submission guide -- Status: READY FOR DEPLOYMENT (pending: GitHub Secrets + CF project creation) ### SEO Layer (COMPLETE) - sitemap-index.xml (auto-generated via @astrojs/sitemap) @@ -92,7 +92,7 @@ ### What does NOT exist yet - No search, no embeddings, no graph visualization -- No deployment (production URL, domain, hosting) — pending GitHub Secrets + CF project +- No custom domain (using default pages.dev) - No analytics ### Graph health @@ -109,5 +109,8 @@ - 3 runs/day (2am, 9am, 3pm UTC) - Rotating provider: balanced→cheap→quality - BATCH_COUNT=1 per run (token efficient) -- 429 → immediate skip to next provider +- 429 → exponential backoff + skip to next provider - Manual trigger: workflow_dispatch with provider + count options + +### Known issues +- Context-assembler injects all 36 entities per generation → ~1500–2000 tokens per call. Fix PENDING: cap at 15 entities (top 10 by relationship count + world-core + campaign).