diff --git a/CHANGELOG.md b/CHANGELOG.md index c12496a..5fedd0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,18 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Changed + +- **Substrate now auto-runs in Claude Code.** The `UserPromptSubmit` hook injects the full substrate advisory (assumption gate + model routing + blast radius + memory + verify) when it matters, silent otherwise. Load-only — never builds/writes `.forge/` from a hook, fail-safe, never blocks. +- **Cross-tool auto-use.** Added a `substrate` section to `source/rules.json` so `forge init` emits the "run substrate before risky work" rule into every tool's config (AGENTS.md, .cursor/rules, …). +- **Docs rewritten** — `docs/cognitive-substrate/README.md` is a professional, example-rich guide (real command output, auto-use setup, extension points); README/SKILL aligned. + +### Fixed + +- **Security (research prototype): removed pickle-based cache** in `impact_oracle/world_model.py` — it was an insecure-deserialization (RCE) vector on a caller-supplied `cache_dir`. Now JSON node-link only, with `cache_dir` containment inside `root`. +- **Smaller npm package** — stopped publishing `docs/cognitive-substrate` (a ~2 MB whitepaper + zips); tarball dropped to ~280 KB. Removed the redundant `*_src.zip` binaries (source lives unzipped in `research/`). +- **Perf** — `substrateCheck` no longer recomputes the assumption assessment. + ### Added - **Forge Cognitive Substrate** — one pre-action command (`forge substrate`) and MCP surface (`substrate_check`, `predict_impact`, `assumption_gate`) that combines assumption gating, transparent model routing, impact prediction, scope decomposition, Cortex lessons, minimality warnings, and verification planning. diff --git a/README.md b/README.md index 367226f..c04d41e 100644 --- a/README.md +++ b/README.md @@ -17,26 +17,41 @@ enforces the non-negotiables as deterministic guards, and adds a code-graph, cross-session memory, and a cost governor. Works with **Claude Code, Codex, Cursor, Gemini CLI, Aider, Copilot, Windsurf/Devin, and Zed**. +**Install** — pick one: + +```bash +# A) Claude Code / Codex — as a plugin (recommended) +/plugin marketplace add CodeWithJuber/forgekit +/plugin install forgekit + +# B) Clone and run the installer (puts `forge` on your PATH) +git clone https://github.com/CodeWithJuber/forgekit.git +cd forgekit && bash install.sh +``` + +**Use** — in any repository: + ```bash -# clone, then: -bash install.sh # symlinks into ~/.forge + ~/.claude, puts `forge` on PATH -forge init # in any repo: emit every tool's config from one source +forge init # emit every tool's config from one source forge substrate "task" # assumption gate + route + impact + scope + verify forge doctor # verify everything is wired ``` -Prefer a plugin? `/plugin marketplace add ` then `/plugin install forgekit`. -Prefer npm? Forge publishes to **GitHub Packages** as `@codewithjuber/forgekit`. Point the -scope at the registry and authenticate once (GitHub Packages requires a token even for public -installs), then run it: +
+CI / devcontainer install via npm (GitHub Packages) + +Forge publishes to GitHub Packages as `@codewithjuber/forgekit`. GitHub Packages requires a +token even for public installs, so authenticate the scope once, then run it: ```bash echo "@codewithjuber:registry=https://npm.pkg.github.com" >> ~/.npmrc -echo "//npm.pkg.github.com/:_authToken=YOUR_GITHUB_TOKEN" >> ~/.npmrc # token needs read:packages +echo "//npm.pkg.github.com/:_authToken=YOUR_GITHUB_TOKEN" >> ~/.npmrc # needs read:packages npx @codewithjuber/forgekit init ``` -All three channels drive the **same** `global/` tree. +
+ +All channels drive the **same** `global/` tree. ## Why @@ -92,44 +107,44 @@ On **Claude Code** it's fully ambient (hooks). Other tools read the lessons from and a zero-dependency MCP server (`forge cortex-mcp`). Everything lives in `.forge/lessons/` — git-committable and auditable. Try it: `node examples/cortex-demo.mjs`. -## Forge Cognitive Substrate — one pre-action gate +## Forge Cognitive Substrate — the check that runs before every edit -Forge now wraps the agent loop with the paper's cognitive-substrate controls. Run one command before ambiguous, expensive, or mutating work: +A frozen model can't remember, can't foresee, and can't see what an edit will break. The +**cognitive substrate** supplies those faculties from the outside: a fast, mostly-deterministic +check (no extra LLM call) that runs _before_ the agent touches code. One command does it all: ```bash -forge substrate "Fix the checkout bug in `src/payments.ts` and add tests" -forge substrate "Refactor auth" --json -forge impact computeTax +forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests" ``` -`substrate` returns the assumption gate, model route, impact radius, scope clusters, relevant Cortex lessons, minimality warnings, and a verification checklist. MCP-capable extensions get the same flow through `substrate_check`, `predict_impact`, and `assumption_gate`. Deterministic checks are asserted; memory relevance, routing fit, and minimality remain advisory where the research is not a hard guarantee. - -Full paper bundle and original artifacts live in -[`docs/cognitive-substrate/`](docs/cognitive-substrate/): PDF, HTML, evidence map, -ecosystem map, and the original router-gate / impact-oracle prototype packages. - -## Forge Preflight — size the work before spending tokens - -An LLM is a fixed-capacity stochastic predictor. Most of the cost/quality bleed comes from -feeding it the wrong-sized task with the wrong-sized context and then over-trusting the output. -**Preflight** is the cheap, deterministic layer that runs _before_ the tokens — no LLM, no guessing: - -- **`forge preflight ""`** — the assumption detector. Scans the task for symbols/files the - repo doesn't define — the things the model would otherwise _assume_ — and surfaces them so it - asks instead of confabulating. Also fires on `UserPromptSubmit`. (The research whitespace: no - shipping tool pre-scans the repo before acting.) -- **`forge route ""`** — recommends the cheapest _capable_ model (Haiku → Sonnet → Opus → - Fable) from code-task complexity (files, fan-out, churn, past-mistake density, ambiguity). A - prime-finder gets Haiku, not Fable. `forge route gateway` emits a LiteLLM config for real - auto-routing. -- **`forge scope `** — a zero-dep import graph → independent clusters (“run these as - separate sessions”) + the coupled files you didn't mention. -- **`forge uicheck `** — exact WCAG contrast math. The design rules (anti-slop, empty - states, specific errors, AI-UX patterns) emit to every tool; the frontend-verifier _asserts_ - only the deterministic and keeps taste _advisory_ — so AI UI-audits stop hallucinating. - -Everything is advisory and never blocks. Cross-tool via the `preflight_check` / `route_task` / -`scope_files` MCP tools. +It returns, in one contract: the **assumption gate** (is the task clear enough to start?), +the cheapest **capable model**, the predicted **blast radius** (which files an edit breaks), +**scope** clusters (what to split into separate sessions), relevant **Cortex lessons**, +**minimality** warnings, and a **verification** checklist. + +**It runs itself.** In Claude Code a `UserPromptSubmit` hook fires the substrate on every +prompt and adds a short advisory only when something needs attention — never blocking, never +nagging on a clean task. Other tools (Codex, Cursor, Gemini, Aider…) get a rule in their +config telling the agent to run it, plus the MCP tools `substrate_check`, `assumption_gate`, +`predict_impact`, `route_task`, and `scope_files`. + +Each check is also its own command: + +| Command | Answers | +| ----------------------------- | ------------------------------------------------------------------ | +| `forge preflight ""` | Is this clear enough to start? (flags unknown names + vague words) | +| `forge route ""` | Cheapest capable model — Haiku → Sonnet → Opus → Fable | +| `forge impact ` | What will this edit break? (reverse-dependency blast radius) | +| `forge scope ` | Independent vs. coupled files → separate sessions | +| `forge uicheck ` | Exact WCAG contrast math for UI work | + +Deterministic checks (repo grounding, graph traversal, routing arithmetic) are **asserted**; +model fit, minimality, and memory relevance stay **advisory**. Everything is advisory overall +and never blocks — tests and human corrections always win. + +**→ Full guide with worked examples, auto-use setup, and how to extend it: +[`docs/cognitive-substrate/`](docs/cognitive-substrate/)** (also holds the white paper, +evidence map, and ecosystem map). ## Commands diff --git a/biome.json b/biome.json index abe63b0..19e1794 100644 --- a/biome.json +++ b/biome.json @@ -1,5 +1,5 @@ { - "$schema": "https://biomejs.dev/schemas/2.0.0/schema.json", + "$schema": "https://biomejs.dev/schemas/2.5.2/schema.json", "vcs": { "enabled": true, "clientKind": "git", "useIgnoreFile": true }, "files": { "ignoreUnknown": true, diff --git a/docs/cognitive-substrate/README.md b/docs/cognitive-substrate/README.md index 5e85ea3..c47d7aa 100644 --- a/docs/cognitive-substrate/README.md +++ b/docs/cognitive-substrate/README.md @@ -1,74 +1,243 @@ -# Cognitive Substrate Paper Bundle +# The Forge Cognitive Substrate -This directory is the research-backed product map for Forge's cognitive-substrate system. -It keeps the full paper, evidence maps, and original prototype packages next to the -production Node implementation so users can install once and use the system from any -Forge-supported agent. +**Coding agents forget what they learned, assume what they don't know, and break code they +can't see.** The substrate is a fast, mostly-deterministic check that runs *before* an agent +edits your code: it flags an unclear task, picks the cheapest capable model, and shows what +an edit will break — all from the repo you already have, with no extra LLM call. -## Fastest Path +In Claude Code it runs **automatically**. In other tools you (or the agent) run one command. + +> Why this exists, in one line: a frozen model is a stateless function `y = f(x)` — no memory, +> no foresight, a fixed window. Those faculties can't be prompted in; they have to be supplied +> from the outside. This is that outside layer. Full argument: the white paper +> ([PDF](./cognitive_substrate_whitepaper.pdf) · [HTML](./cognitive_substrate_whitepaper.html)). + +--- + +## Install (about 2 minutes) + +```bash +# Claude Code / Codex — install as a plugin +/plugin marketplace add CodeWithJuber/forgekit +/plugin install forgekit + +# …or clone and run the installer (puts `forge` on your PATH) +git clone https://github.com/CodeWithJuber/forgekit.git +cd forgekit && bash install.sh +``` + +Then, inside any project: + +```bash +forge init # writes each AI tool's native config from one source +forge atlas # builds the code graph (needed for blast-radius checks) +``` + +That's it. `forge init` configures Claude Code, Codex, Cursor, Gemini, Aider, Copilot, Zed, +Continue, and Roo where supported. + +--- + +## It runs itself (the main benefit) + +You don't have to remember to use it. + +**In Claude Code** — a hook runs the substrate on **every prompt** and adds a short note *only +when something needs attention* (unclear task, big blast radius, pricey model). It never blocks +and never nags on a clean, simple task. Real example — you type *"refactor computeTax in +math.js"* and the agent silently receives: + +```text +Forge substrate — pre-action advisory (advisory, never blocks): +- Under-specified (high risk). Ask before editing: + • What constraints must be respected: performance, dependencies, style, or compatibility? +- Suggested model: Haiku 4.5 (simple); escalate only on a verifier failure. +- Predicted blast radius (2): invoice.js, math.js. Review these before editing. +- Verify with: review impacted files before editing · run the narrowest affected test first +``` + +**In other AI tools** (Codex, Cursor, Gemini, Aider…) — `forge init` writes a rule into their +config file telling the agent to run the check itself, and exposes it as MCP tools it can call. +See [In other AI tools](#in-other-ai-tools). + +--- + +## The one command ```bash -# one time, from this repo -bash install.sh +forge substrate "" +``` + +It runs the whole check and gives a plain verdict. Two real examples: -# one time, inside any project -forge init +**Vague task → it tells you to ask first:** -# every day, before ambiguous or mutating work -forge substrate "Fix the checkout bug and add tests" -forge substrate "Refactor auth safely" --json -forge impact computeTax +```console +$ forge substrate "make the auth better" + + proceed: ASK FIRST + assumption: high risk · completeness 0.23 + clarify: + - What exactly should this produce, and how will we know it is correct? + impact: 0 file(s) predicted ``` -For MCP-capable tools, call the same flow through: +**Clear task → it clears you and shows the blast radius:** + +```console +$ forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests" + + proceed: yes + assumption: medium risk · completeness 0.63 + route: Haiku 4.5 (simple) + impact: 3 file(s) predicted + - src/auth.js + - src/login.js (imports verifyToken — you didn't mention it) + - src/session.js (imports verifyToken — you didn't mention it) + verify: + - run the narrowest affected test first, then the broader suite +``` -- `substrate_check` — full pre-action gate. -- `assumption_gate` — ask/proceed decision and clarifying questions. -- `predict_impact` — blast-radius prediction for a symbol or file. +The second run found the two files that import `verifyToken` but you never named — the +"forgot the coupled file" bug, caught *before* the edit. Add `--json` for machine-readable +output (see [Use it in a script](#use-it-in-a-script)). -## Install Choices +--- -| Path | Best For | Steps | +## The five checks (each is also its own command) + +`forge substrate` bundles these. Run any one on its own when that's all you need. + +| Command | Answers | One-line example | | --- | --- | --- | -| Claude/Codex plugin | Extension-style install | Install the plugin, then run `forge init` in each repo. | -| Git clone | Lowest friction local setup | `bash install.sh`, then `forge init`. | -| npm package | CI/devcontainer usage | `npx @codewithjuber/forgekit init`. | +| `forge preflight ""` | Is this task clear enough to start? | flags names not in the repo + vague words | +| `forge route ""` | Which model is cheapest-but-capable? | trivial → Haiku · hard → Opus/Fable | +| `forge impact ` | What will this edit break? | reverse-dependency blast radius | +| `forge scope ` | Can this be split into sessions? | independent vs. coupled files | +| `forge verify` | Did it actually work? | runs the real tests/build, not the model's word | + +Real output for the two most-used: -After `forge init`, Forge emits native config for Claude Code, Codex, Cursor, Gemini, -Aider, Copilot/VS Code, Windsurf/Devin, Zed, Continue, and Roo where supported. Tools -without hooks receive advisory context plus MCP config; Forge does not pretend it can -force hooks into hosts that do not expose them. +```console +$ forge preflight "fix the thing in authManager so it works properly" -## Included Artifacts +- `authManager` — not found in the code. Different name, or should it be created? +- Ambiguous: "properly" — state concrete acceptance criteria. +- Which specific file, module, component, or symbol should this change touch? +``` + +```console +$ forge impact verifyToken + target: verifyToken ✓ found + impacted files: 3 + - src/auth.js + - src/login.js + - src/session.js +``` -- [`cognitive_substrate_whitepaper.pdf`](./cognitive_substrate_whitepaper.pdf) — full paper. -- [`cognitive_substrate_whitepaper.html`](./cognitive_substrate_whitepaper.html) — browser-readable paper. -- [`deliverable-package.md`](./deliverable-package.md) — package overview and headline results. -- [`evidence_map.md`](./evidence_map.md) — source/status map for load-bearing evidence. -- [`ecosystem_map.md`](./ecosystem_map.md) — capability-vs-tooling map. -- [`impact_oracle_src.zip`](./impact_oracle_src.zip) — original Prototype I package. -- [`router_gate_src.zip`](./router_gate_src.zip) — original Prototype II package. -- [`../../research/python-prototypes/`](../../research/python-prototypes/) — unzipped prototype source preserved for auditability. +--- -## Paper-To-Forge Map +## Read the result (what to do) -| Paper capability | Production Forge surface | Status | +| Field | What it means | Do this | | --- | --- | --- | -| Memory | `forge recall`, `forge cortex` | File-backed and auditable; relevance is advisory. | -| Learning | `forge cortex` | Outcome-confirmed lessons; model weights do not change. | -| Imagination | `forge impact`, `forge substrate` | Static graph blast-radius simulation. | -| Self-correction | `forge verify`, doom-loop guard | External checks beat model claims. | -| Impact-awareness | `forge atlas`, `forge impact` | Known symbols/files and likely dependents surfaced. | -| M1 routing | `forge route`, `forge substrate` | Transparent cheapest-capable tier recommendation. | -| M2 assumption gate | `forge preflight`, `forge substrate` | Under-specified tasks return questions. | -| M3 decomposition | `forge scope`, `forge substrate` | Independent/coupled file clusters. | -| M4 goal anchoring | `forge substrate` | One pre-action objective/risk/verification summary. | -| M5 anti-over-engineering | `forge substrate`, `lean-guard` | Broad work gets minimality warnings. | -| M6 inline verification | `forge verify` | Checklist and external verification discipline. | - -## Honest Boundary - -Deterministic parts are asserted: repo symbol/file grounding, graph traversal, emitted -config, protected-file guards, and test/build commands. Research-edge judgments remain -advisory: memory relevance, model fit, scope minimality, and whether a verification -checklist is sufficient for a particular production environment. +| `okToProceed: false` | task is under-specified | ask the `assumption.questions`, don't guess | +| `route.tier` | cheapest capable model | start there; only escalate if a verifier fails | +| `impact.impactedFiles` | predicted blast radius | read these before editing | +| `scope.clusters` | independent vs. coupled work | split independent groups into separate sessions | +| `memory.advisory` | past lessons for this area | context, not law — tests override it | +| `verification.checklist` | how to prove it works | run it, show the output, then say "done" | + +--- + +## In other AI tools + +Tools without a hook surface get the substrate two ways, both written by `forge init`: + +1. **A rule in their config** (`AGENTS.md`, `.cursor/rules`, `GEMINI.md`, …): *"Before + ambiguous, expensive, multi-file, or mutating work, run `forge substrate "" --json` + (or the MCP tool `substrate_check`). If `okToProceed` is false, ask the questions first."* + +2. **MCP tools** any MCP-capable agent can call directly: + + | MCP tool | Does | + | --- | --- | + | `substrate_check` | full pre-action check | + | `assumption_gate` | ask/proceed + questions | + | `predict_impact` | blast radius | + | `route_task` | model recommendation | + | `scope_files` | independent vs. coupled | + +Forge never pretends it can force a hook into a tool that has none — it's ambient on Claude +Code, and agent-invoked everywhere else. + +--- + +## Use it in a script + +```bash +forge substrate "update verifyToken in src/auth.js" --json +``` + +```jsonc +{ + "okToProceed": false, + "assumption": { "risk": "high", "shouldAsk": true, "questions": ["…"] }, + "route": { "tier": "simple", "model": { "name": "Haiku 4.5" } }, + "impact": { "impactedFiles": ["src/auth.js", "src/login.js"] }, + "verification": { "checklist": ["npm test", "npm run typecheck"] } +} +``` + +Gate your agent's next step on `okToProceed`; feed `route.tier` to your model picker; read +`impact.impactedFiles` before editing. + +--- + +## Extend it + +Small, pure functions — change the one piece you need, then run `npm test`. + +| To change… | Edit | +| --- | --- | +| how often it asks | `source/substrate.json` → `defaults.askThreshold` (0.6) | +| blast-radius sensitivity | `source/substrate.json` → `defaults.impactThreshold` (0.1) | +| a routing signal | `src/route.js` → `rubricComplexity()` | +| an assumption question | `src/preflight.js` → `DIMENSIONS[]` | +| the verify checklist | `src/substrate.js` → `verificationChecklist()` | +| when the ambient hook speaks | `src/substrate.js` → `substrateContext()` | +| the cross-tool wording | `source/rules.json` → `substrate` section (then `forge init`) | + +--- + +## Honest limits + +- **Heuristic, not benchmarked.** Rubrics were tuned on small hand-labeled sets. Judge after + real use. +- **The graph is regex-approximate.** Dynamic dispatch / DI / generated code can be missed — + impact is *conservative* (catches the obvious dependents), not a sound call graph. +- **Assumption detection is lexical.** It catches unclear *names and wording*, not wrong + *intent*. +- **Auto-run needs a hook surface** — fully ambient on Claude Code; agent-invoked elsewhere. + +What's **asserted** (safe to gate on): repo symbol/file grounding, graph traversal, scope +decomposition, the routing arithmetic, and the test/build commands. What's **advisory** +(flagged, never asserted): whether the model is capable enough, whether a change is +over-engineered, and whether a past lesson is relevant. Tests and human corrections always win. + +--- + +## Learn more + +- **White paper** — the full argument: [PDF](./cognitive_substrate_whitepaper.pdf) · + [HTML](./cognitive_substrate_whitepaper.html) +- **[Package overview](./deliverable-package.md)** — headline results and prototypes +- **[Evidence map](./evidence_map.md)** — every load-bearing statistic re-graded against + primary sources (5 confirmed, 5 vendor-reported, 2 dropped) +- **[Ecosystem map](./ecosystem_map.md)** — each capability vs. the real 2026 tool stack +- **[Prototype source](../../research/python-prototypes/)** — the auditable Python originals + +**How the paper maps to what ships:** memory → `recall`/`cortex` · learning → `cortex` · +imagination → `impact` · self-correction → `verify` · impact-awareness → `atlas`/`impact` · +M1 routing → `route` · M2 assumption gate → `preflight` · M3 decomposition → `scope` · +M4 goal-anchoring, M5 minimality, M6 verification → `substrate`. diff --git a/docs/cognitive-substrate/deliverable-package.md b/docs/cognitive-substrate/deliverable-package.md index dbc8f98..89f9578 100644 --- a/docs/cognitive-substrate/deliverable-package.md +++ b/docs/cognitive-substrate/deliverable-package.md @@ -1,41 +1,48 @@ # A Cognitive Substrate for Coding Agents — Deliverable Package + ### Theory → Evidence → Build-Map edition (v2) -**One-line thesis:** The faculties a coding agent lacks — memory, learning, imagination, self-correction, impact-awareness — are not gaps in the model's *knowledge* but structural consequences of what a frozen transformer *is* (a stateless map `y = f_θ(x)`, fixed weights, bounded window). They cannot be prompted or tooled away; they can only be supplied by **re-wrapping the input→process→output loop** into a closed, stateful cycle around the frozen model. +**One-line thesis:** The faculties a coding agent lacks — memory, learning, imagination, self-correction, impact-awareness — are not gaps in the model's _knowledge_ but structural consequences of what a frozen transformer _is_ (a stateless map `y = f_θ(x)`, fixed weights, bounded window). They cannot be prompted or tooled away; they can only be supplied by **re-wrapping the input→process→output loop** into a closed, stateful cycle around the frozen model. -**What v2 adds.** The first edition argued the five faculties from first principles and prototyped the one that is buildable today. This edition (1) **grounds the argument in the field's own evidence** — twelve load-bearing pain-point statistics independently re-grounded from primary sources and graded *confirmed / vendor-reported / unverifiable*; (2) adds **six metacognitive mechanisms** the frozen loop also lacks (routing, assumption gate, decomposition, goal-anchoring, anti-over-engineering, inline verification); (3) **maps all eleven capabilities against the real 2026 Claude-Code stack**, marking each solved / partial / residual-gap so we say clearly *what not to build*; and (4) ships a **second runnable prototype** — a complexity-aware router + assumption gate, evaluated live on real models. +**What v2 adds.** The first edition argued the five faculties from first principles and prototyped the one that is buildable today. This edition (1) **grounds the argument in the field's own evidence** — twelve load-bearing pain-point statistics independently re-grounded from primary sources and graded _confirmed / vendor-reported / unverifiable_; (2) adds **six metacognitive mechanisms** the frozen loop also lacks (routing, assumption gate, decomposition, goal-anchoring, anti-over-engineering, inline verification); (3) **maps all eleven capabilities against the real 2026 Claude-Code stack**, marking each solved / partial / residual-gap so we say clearly _what not to build_; and (4) ships a **second runnable prototype** — a complexity-aware router + assumption gate, evaluated live on real models. -> **Governing discipline (the user's, adopted throughout):** *AI output is mathematically-calculated probability — non-deterministic, and never blindly trusted.* Every claim in this package is graded by how well it is sourced; every prototype decision is a transparent, attributable rule rather than another opaque model call; and trust is always earned by an **external** check, never asserted by the model. +> **Governing discipline (the user's, adopted throughout):** _AI output is mathematically-calculated probability — non-deterministic, and never blindly trusted._ Every claim in this package is graded by how well it is sourced; every prototype decision is a transparent, attributable rule rather than another opaque model call; and trust is always earned by an **external** check, never asserted by the model. --- ## What's in this package ### 1. The white paper (core deliverable) — 48 pp + - **`cognitive_substrate_whitepaper.pdf`** / **`cognitive_substrate_whitepaper.html`** — the full study, 13 sections + 3 appendices, 7 figures. - - **§1–3** the root cause and the five faculties (from v1): *why* each faculty is structurally absent (P1 statelessness, P2 frozen weights, P3 bounded context), each grounded in the real literature. - - **§4 Evidence** *(new)* — the twelve statistics, re-grounded. 5 confirmed, 5 vendor-reported, 2 unverifiable. + - **§1–3** the root cause and the five faculties (from v1): _why_ each faculty is structurally absent (P1 statelessness, P2 frozen weights, P3 bounded context), each grounded in the real literature. + - **§4 Evidence** _(new)_ — the twelve statistics, re-grounded. 5 confirmed, 5 vendor-reported, 2 unverifiable. - **§5** the Qur'anic epistemic lens — design framing/ethics, never technical authority. - - **§6 Six mechanisms** *(new)* — M1 routing, M2 assumption gate, M3 decomposition, M4 goal-anchoring, M5 anti-over-engineering, M6 inline verification — each formalized, with ecosystem status and a Qur'anic anchor. + - **§6 Six mechanisms** _(new)_ — M1 routing, M2 assumption gate, M3 decomposition, M4 goal-anchoring, M5 anti-over-engineering, M6 inline verification — each formalized, with ecosystem status and a Qur'anic anchor. - **§7** the cognitive substrate, now with the six-mechanism metacognitive control layer (Figure 3). - - **§8 Prototype I** the impact oracle (from v1). **§9 Prototype II** *(new)* the router + gate. **§10 Build-map** *(new)* the ranked opportunity list. + - **§8 Prototype I** the impact oracle (from v1). **§9 Prototype II** _(new)_ the router + gate. **§10 Build-map** _(new)_ the ranked opportunity list. - **§11** new-vs-reinvented. **§12** limitations. **§13** conclusion. ### 2. Prototype I — Codebase World-Model + Impact Oracle -- **`impact_oracle_src.zip`** — parses a codebase (AST) into a **persistent dependency graph**, predicts the **blast radius** of a proposed edit via reverse-dependency traversal with confidence decay. `python demo.py` runs end-to-end; `pytest` → **36 tests pass** with zero setup. Builds opportunity #3. -### 3. Prototype II — Complexity-aware router + Assumption gate *(new)* -- **`router_gate_src.zip`** — the two mechanisms at the top of the build-map, composed as `gate → route → execute → verify → escalate`. Both are **transparent additive rubrics**, not opaque LLM calls; escalation is driven by an external check. `python demo.py`, `pytest` → **19 tests pass**, `python evaluate.py --live` reproduces the live numbers. +- [`research/python-prototypes/impact_oracle/`](../../research/python-prototypes/impact_oracle/) — parses a codebase (AST) into a **persistent dependency graph**, predicts the **blast radius** of a proposed edit via reverse-dependency traversal with confidence decay. `python demo.py` runs end-to-end; `pytest` → **36 tests pass** with zero setup. Builds opportunity #3. (Shipped in production as `forge impact` / `forge atlas`.) + +### 3. Prototype II — Complexity-aware router + Assumption gate _(new)_ + +- [`research/python-prototypes/router_gate/`](../../research/python-prototypes/router_gate/) — the two mechanisms at the top of the build-map, composed as `gate → route → execute → verify → escalate`. Both are **transparent additive rubrics**, not opaque LLM calls; escalation is driven by an external check. `python demo.py`, `pytest` → **19 tests pass**, `python evaluate.py --live` reproduces the live numbers. (Shipped in production as `forge route` / `forge preflight`.) - **`eval_results.json`** — the live evaluation record (real measured tokens). -### 4. Evidence & ecosystem maps *(new)* +### 4. Evidence & ecosystem maps _(new)_ + - **`evidence_map.json`** / **`evidence_map.md`** — every load-bearing statistic, its primary source, and its status. - **`ecosystem_map.json`** / **`ecosystem_map.md`** — every faculty & mechanism vs. the real stack, with residual gap and proposed contribution. ### 5. Figures & schematics -- **`schematic_loop.png`** (Fig 1) · **`schematic_system.png`** (Fig 2) · **`schematic_extended.png`** (Fig 3, *new* — the six-mechanism control layer) · **`impact_graph.png`** (Fig 4) · **`eval_precision_recall.png`** (Fig 5) · **`schematic_router_loop.png`** (Fig 6, *new*) · **`router_eval.png`** (Fig 7, *new*). + +- **`schematic_loop.png`** (Fig 1) · **`schematic_system.png`** (Fig 2) · **`schematic_extended.png`** (Fig 3, _new_ — the six-mechanism control layer) · **`impact_graph.png`** (Fig 4) · **`eval_precision_recall.png`** (Fig 5) · **`schematic_router_loop.png`** (Fig 6, _new_) · **`router_eval.png`** (Fig 7, _new_). ### 6. Supporting artifacts (from v1) + - **`gap_map.json`/`.md`**, **`references.json`** (32 sources), **`quran_lens.json`/`.md`** (14-row concept→faculty→design-principle mapping). --- @@ -43,43 +50,46 @@ ## The honest headline results ### Prototype I — Impact Oracle (against mutation-derived ground truth, 5 real edits) -| Method | Precision | Recall | F1 | -|---|---|---|---| -| **Graph Oracle** (ours) | 0.63 | **1.00** | 0.75 | -| Grep baseline (what agents do today) | 0.73 | 0.94 | **0.79** | -| Edited-file-only | 1.00 | 0.53 | 0.65 | + +| Method | Precision | Recall | F1 | +| ------------------------------------ | --------- | -------- | -------- | +| **Graph Oracle** (ours) | 0.63 | **1.00** | 0.75 | +| Grep baseline (what agents do today) | 0.73 | 0.94 | **0.79** | +| Edited-file-only | 1.00 | 0.53 | 0.65 | The oracle does **not** dominate F1 — grep edges it at the default threshold, and we say so. What the oracle uniquely provides is **guaranteed recall**: for "show me everything my edit could break," a silent miss costs far more than an extra file to check, and only the structural oracle drives false negatives to zero (precision tunable, best F1 = 0.79 at threshold 0.4). ### Prototype II — Router + Gate (live, on real models: haiku / sonnet / opus) -| Metric | Result | -|---|---| -| Gate accuracy (should-ask) | 30/30 · precision 1.00 · recall 1.00 | -| Routing accuracy (well-specified tasks) | 21/21 exact tier | -| **Real cost saved vs always-premium** | **62.1%** (same measured tokens) | -| Execution-verified sub-experiment | 3/3 routed-down outputs passed real test cases | -**Honest caveat (both prototypes):** these are **demonstrations, not benchmarks**. The router's 30-task set is hand-labeled and the rubric thresholds were tuned against it, so perfect separation shows the rubric *can* distinguish these cases — not field accuracy. The oracle's evaluation is 5 mutations + 2 stdlib scale checks. We apply the "retired SWE-bench Verified" caution (§4, confirmed) to our own numbers. +| Metric | Result | +| --------------------------------------- | ---------------------------------------------- | +| Gate accuracy (should-ask) | 30/30 · precision 1.00 · recall 1.00 | +| Routing accuracy (well-specified tasks) | 21/21 exact tier | +| **Real cost saved vs always-premium** | **62.1%** (same measured tokens) | +| Execution-verified sub-experiment | 3/3 routed-down outputs passed real test cases | + +**Honest caveat (both prototypes):** these are **demonstrations, not benchmarks**. The router's 30-task set is hand-labeled and the rubric thresholds were tuned against it, so perfect separation shows the rubric _can_ distinguish these cases — not field accuracy. The oracle's evaluation is 5 mutations + 2 stdlib scale checks. We apply the "retired SWE-bench Verified" caution (§4, confirmed) to our own numbers. ## What the evidence re-grounding caught -The independent re-grounding **changed our claims** — three widely-repeated numbers did not survive and are *not* used as fact in this paper: +The independent re-grounding **changed our claims** — three widely-repeated numbers did not survive and are _not_ used as fact in this paper: + - **"2.74× more vulnerabilities"** is not traceable to Veracode's own report (only their 45% OWASP figure is); likely conflated with a separate study. - **"17% lower comprehension / 400K sessions"** merges two different studies — the session study contains no comprehension finding. - **GitClear 4× vs 8×** internal inconsistency and **JetBrains 77%** could not be located in primary form. -That a re-grounding pass corrected the paper is the point, not an embarrassment: it is the same discipline the architecture makes structural — *a stored fact is provisional until an external check confirms it.* +That a re-grounding pass corrected the paper is the point, not an embarrassment: it is the same discipline the architecture makes structural — _a stored fact is provisional until an external check confirms it._ ## The build-opportunity map (what to build, what to skip) -**Already solved — do not rebuild:** M1 routing (model tiering + gateways like LiteLLM/OpenRouter) and M3 decomposition (subagents, Agent-Teams). The router prototype's honest contribution is only the *transparency layer*, and we say so. +**Already solved — do not rebuild:** M1 routing (model tiering + gateways like LiteLLM/OpenRouter) and M3 decomposition (subagents, Agent-Teams). The router prototype's honest contribution is only the _transparency layer_, and we say so. **The genuine whitespace, ranked:** (1) **assumption/uncertainty gate** — the project's named root failure and the field's named gap; nothing supplies calibrated known-unknowns. (2) **validity-anchored memory** — backends store notes, none tracks invalidation-by-correction. (3) **mandatory pre-action impact gate** — indexers retrieve, none is a deterministic blast-radius check. (4) outcome-validated learning. (5) doom-loop / root-cause correction. (6) scope-minimality. This paper prototypes #1 and #3 — the two where a single session can produce checkable ground truth. ## What is genuinely new vs. reinvented -Most components are borrowed (external memory, fast/slow learning, code graphs, model tiering — all exist). The contribution is **the composition and the framing**: the closed-loop shape; **validity-anchored memory** (prune by whether a past prediction was confirmed by an *external* oracle, not by the model's own judgment); wiring exact impact analysis into a **mandatory pre-action gate**; a **transparent** router/gate that explains every decision; and deriving *which* safeguards are non-negotiable from a coherent epistemology. That turns scattered literatures and named-but-unsolved gaps into one buildable architecture aimed squarely at coding agents. +Most components are borrowed (external memory, fast/slow learning, code graphs, model tiering — all exist). The contribution is **the composition and the framing**: the closed-loop shape; **validity-anchored memory** (prune by whether a past prediction was confirmed by an _external_ oracle, not by the model's own judgment); wiring exact impact analysis into a **mandatory pre-action gate**; a **transparent** router/gate that explains every decision; and deriving _which_ safeguards are non-negotiable from a coherent epistemology. That turns scattered literatures and named-but-unsolved gaps into one buildable architecture aimed squarely at coding agents. ## Scope & limitations (stated honestly) -Two faculties/mechanisms are prototyped, not eleven. The impact oracle's static analysis is single-language (Python) and conservative on dynamic dispatch. The router/gate rubrics are keyword heuristics tuned on a small hand-labeled set. Memory validity, outcome learning, and doom-loop diagnosis remain *specified but unbuilt* — the harder research gaps, marked as such rather than gestured at with a demo. The lens is framing: reject it and you lose the organizing vocabulary but none of the technical content. +Two faculties/mechanisms are prototyped, not eleven. The impact oracle's static analysis is single-language (Python) and conservative on dynamic dispatch. The router/gate rubrics are keyword heuristics tuned on a small hand-labeled set. Memory validity, outcome learning, and doom-loop diagnosis remain _specified but unbuilt_ — the harder research gaps, marked as such rather than gestured at with a demo. The lens is framing: reject it and you lose the organizing vocabulary but none of the technical content. diff --git a/docs/cognitive-substrate/impact_oracle_src.zip b/docs/cognitive-substrate/impact_oracle_src.zip deleted file mode 100644 index 3354953..0000000 Binary files a/docs/cognitive-substrate/impact_oracle_src.zip and /dev/null differ diff --git a/docs/cognitive-substrate/router_gate_src.zip b/docs/cognitive-substrate/router_gate_src.zip deleted file mode 100644 index b64ca05..0000000 Binary files a/docs/cognitive-substrate/router_gate_src.zip and /dev/null differ diff --git a/global/tools/cognitive-substrate/SKILL.md b/global/tools/cognitive-substrate/SKILL.md index 0fcfb96..5575f99 100644 --- a/global/tools/cognitive-substrate/SKILL.md +++ b/global/tools/cognitive-substrate/SKILL.md @@ -11,21 +11,61 @@ description: >- # Cognitive Substrate -Before acting on non-trivial or mutating code tasks, run: +Wrap the frozen model in a pre-action check: gate assumptions, route model effort, predict +blast radius, decompose scope, surface past lessons, and plan verification — **before** editing. + +## When to run it + +Trigger on any ambiguous, expensive, multi-file, or mutating task: edit/refactor/fix/design +production code, integrate a feature, choose model effort, inspect blast radius, or when a +request is vague. Skip it for one-line, well-specified fixes. + +> In Claude Code this fires automatically on every prompt (UserPromptSubmit hook). In other +> agents, run it yourself with the CLI, or call the MCP tool `substrate_check`. + +## Run ```bash -forge substrate "" --json +forge substrate "" --json # full contract; use the fields below +``` + +## Act on the result + +1. **`okToProceed: false`** → ask the returned `assumption.questions` before editing. Do not guess. +2. **`route.tier`** → start at that model tier (cheapest capable); escalate only after an external verifier fails. +3. **`impact.impactedFiles`** → read these before editing a named symbol/file (the blast radius). +4. **`scope.clusters`** → split independent groups into separate sessions; note coupled files you didn't name. +5. **`memory.advisory`** → context, not law; tests and human corrections override it. +6. **`verification.checklist`** → run it and show output before claiming done. + +## Worked example + +```console +$ forge substrate "make the auth better" + proceed: ASK FIRST + assumption: high risk · completeness 0.23 + clarify: + - What exactly should this produce, and how will we know it is correct? +``` + +→ Under-specified. Ask the clarify question instead of editing. + +```console +$ forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests" + proceed: yes + impact: 3 file(s) predicted + - src/auth.js + - src/login.js (importer you did not name) + - src/session.js (importer you did not name) ``` -Use the result this way: +→ Cleared to proceed, but review the two coupled importers first. -1. If `okToProceed` is false, ask the returned `assumption.questions` before editing. -2. Use `route.model` as the cheapest capable tier recommendation; do not escalate without a verifier failure. -3. Read `impact.impactedFiles` before editing a named symbol/file. -4. Use `scope.clusters` to split independent work into separate sessions. -5. Treat `memory.advisory` as context, not law; tests and human corrections override it. -6. Run the `verification.checklist` before claiming completion. +## Single-mechanism commands -For details, read `references/capability-map.md` only when you need to explain how the paper's faculties map to Forge commands. +`forge preflight ""` (assumptions) · `forge route ""` (model tier) · +`forge impact ` (blast radius) · `forge scope ` (decomposition). +MCP equivalents: `assumption_gate`, `route_task`, `predict_impact`, `scope_files`. -The full paper bundle is in `docs/cognitive-substrate/` when the repository is available: PDF, HTML, evidence map, ecosystem map, and original prototype packages. +For the full guide (how it works, extending it, the honesty boundary) and the white paper, +see `docs/cognitive-substrate/README.md`. `references/capability-map.md` maps faculties to commands. diff --git a/package-lock.json b/package-lock.json index df412c4..b4d6202 100644 --- a/package-lock.json +++ b/package-lock.json @@ -12,7 +12,7 @@ "forge": "src/cli.js" }, "devDependencies": { - "@biomejs/biome": "^2.0.0", + "@biomejs/biome": "2.5.2", "@types/node": "^26", "typescript": "^6" }, diff --git a/package.json b/package.json index 28134a8..5405bd3 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "@codewithjuber/forgekit", "version": "0.3.1", - "description": "One config, every AI coding tool — a cross-tool configuration layer for agentic coding assistants (Claude Code, Codex, Cursor, Gemini, Aider, and more).", + "description": "One config, every AI coding tool \u2014 a cross-tool configuration layer for agentic coding assistants (Claude Code, Codex, Cursor, Gemini, Aider, and more).", "type": "module", "bin": { "forge": "src/cli.js" @@ -30,7 +30,6 @@ "source", "global", "templates", - "docs/cognitive-substrate", "plugin", "hooks", ".claude-plugin", @@ -69,7 +68,7 @@ "zero-dependencies" ], "devDependencies": { - "@biomejs/biome": "^2.0.0", + "@biomejs/biome": "2.5.2", "@types/node": "^26", "typescript": "^6" } diff --git a/research/python-prototypes/impact_oracle/impact_oracle/world_model.py b/research/python-prototypes/impact_oracle/impact_oracle/world_model.py index 936d37e..cfd7bdc 100644 --- a/research/python-prototypes/impact_oracle/impact_oracle/world_model.py +++ b/research/python-prototypes/impact_oracle/impact_oracle/world_model.py @@ -2,24 +2,22 @@ Builds and maintains a directed graph whose nodes are symbols and whose edges are structural dependencies. The graph is persisted to disk as -JSON (node-link format) plus a NetworkX pickle. On subsequent runs only -files whose content hash has changed are re-parsed (incremental update). +JSON (node-link format) only — never pickle, which would execute arbitrary +code on load. On subsequent runs only files whose content hash has changed +are re-parsed (incremental update). """ from __future__ import annotations import json import os -import pickle from pathlib import Path from typing import Any import networkx as nx from impact_oracle.parser import ( - DependencyEdge, ParseResult, - SymbolNode, content_hash, discover_python_files, parse_file, @@ -43,7 +41,17 @@ def __init__(self, root: str, cache_dir: str | None = None): Directory to persist the graph. Defaults to ``/.impact_oracle_cache``. """ self.root = os.path.abspath(root) - self.cache_dir = cache_dir or os.path.join(self.root, ".impact_oracle_cache") + # Defense-in-depth: a caller-supplied cache_dir must resolve inside root, so a + # hostile MCP argument cannot point persistence I/O at an arbitrary location. + if cache_dir is None: + self.cache_dir = os.path.join(self.root, ".impact_oracle_cache") + else: + resolved = os.path.abspath(cache_dir) + if os.path.commonpath([resolved, self.root]) != self.root: + raise ValueError( + f"cache_dir {cache_dir!r} must be inside root {self.root!r}" + ) + self.cache_dir = resolved self.graph: nx.DiGraph = nx.DiGraph() # content-hash cache: filepath (relative) -> hash self._file_hashes: dict[str, str] = {} @@ -174,29 +182,26 @@ def summary(self) -> dict[str, Any]: def _save_cache(self): os.makedirs(self.cache_dir, exist_ok=True) - # 1. JSON node-link (portable) + # JSON node-link is the ONLY persistence format. Pickle was removed: loading a + # pickle executes arbitrary code, and cache_dir can be caller-supplied, which made + # graph.pkl an insecure-deserialization (RCE) vector. JSON is data-only and safe. data = nx.node_link_data(self.graph) with open(os.path.join(self.cache_dir, "graph.json"), "w") as f: json.dump(data, f, indent=1) - # 2. Pickle (fast reload) - with open(os.path.join(self.cache_dir, "graph.pkl"), "wb") as f: - pickle.dump(self.graph, f, protocol=pickle.HIGHEST_PROTOCOL) - # 3. File hashes with open(os.path.join(self.cache_dir, "file_hashes.json"), "w") as f: json.dump(self._file_hashes, f) - # 4. File-node mapping with open(os.path.join(self.cache_dir, "file_nodes.json"), "w") as f: json.dump(self._file_nodes, f) def _load_cache(self): - pkl_path = os.path.join(self.cache_dir, "graph.pkl") + json_path = os.path.join(self.cache_dir, "graph.json") hash_path = os.path.join(self.cache_dir, "file_hashes.json") fn_path = os.path.join(self.cache_dir, "file_nodes.json") - if os.path.exists(pkl_path): + if os.path.exists(json_path): try: - with open(pkl_path, "rb") as f: - self.graph = pickle.load(f) + with open(json_path) as f: + self.graph = nx.node_link_graph(json.load(f)) except Exception: self.graph = nx.DiGraph() @@ -221,8 +226,11 @@ def _apply_parse_result(self, result: ParseResult): for edge in result.edges: self.graph.add_edge( - edge.source, edge.target, - kind=edge.kind, confidence=edge.confidence, lineno=edge.lineno, + edge.source, + edge.target, + kind=edge.kind, + confidence=edge.confidence, + lineno=edge.lineno, ) self._file_nodes[result.file] = node_names @@ -251,15 +259,30 @@ def _resolve_cross_module_edges(self): continue # Try parent.v as a resolution # e.g. target 'utils.helper_func' might live as 'demo_package.utils.helper_func' - candidates = [n for n in existing if n.endswith(f".{v}") or n.endswith(f".{v.split('.')[-1]}")] + candidates = [ + n + for n in existing + if n.endswith(f".{v}") or n.endswith(f".{v.split('.')[-1]}") + ] if len(candidates) == 1: edges_to_remove.append((u, v)) - edges_to_add.append((u, candidates[0], {**data, "confidence": data.get("confidence", 1.0) * 0.9})) + edges_to_add.append( + ( + u, + candidates[0], + {**data, "confidence": data.get("confidence", 1.0) * 0.9}, + ) + ) elif len(candidates) > 1: # ambiguous: pick the best match (longest common suffix) - best = max(candidates, key=lambda c: len(os.path.commonprefix([c[::-1], v[::-1]]))) + best = max( + candidates, + key=lambda c: len(os.path.commonprefix([c[::-1], v[::-1]])), + ) edges_to_remove.append((u, v)) - edges_to_add.append((u, best, {**data, "confidence": data.get("confidence", 1.0) * 0.7})) + edges_to_add.append( + (u, best, {**data, "confidence": data.get("confidence", 1.0) * 0.7}) + ) for u, v in edges_to_remove: if self.graph.has_edge(u, v): diff --git a/skills/cognitive-substrate/SKILL.md b/skills/cognitive-substrate/SKILL.md index 0fcfb96..5575f99 100644 --- a/skills/cognitive-substrate/SKILL.md +++ b/skills/cognitive-substrate/SKILL.md @@ -11,21 +11,61 @@ description: >- # Cognitive Substrate -Before acting on non-trivial or mutating code tasks, run: +Wrap the frozen model in a pre-action check: gate assumptions, route model effort, predict +blast radius, decompose scope, surface past lessons, and plan verification — **before** editing. + +## When to run it + +Trigger on any ambiguous, expensive, multi-file, or mutating task: edit/refactor/fix/design +production code, integrate a feature, choose model effort, inspect blast radius, or when a +request is vague. Skip it for one-line, well-specified fixes. + +> In Claude Code this fires automatically on every prompt (UserPromptSubmit hook). In other +> agents, run it yourself with the CLI, or call the MCP tool `substrate_check`. + +## Run ```bash -forge substrate "" --json +forge substrate "" --json # full contract; use the fields below +``` + +## Act on the result + +1. **`okToProceed: false`** → ask the returned `assumption.questions` before editing. Do not guess. +2. **`route.tier`** → start at that model tier (cheapest capable); escalate only after an external verifier fails. +3. **`impact.impactedFiles`** → read these before editing a named symbol/file (the blast radius). +4. **`scope.clusters`** → split independent groups into separate sessions; note coupled files you didn't name. +5. **`memory.advisory`** → context, not law; tests and human corrections override it. +6. **`verification.checklist`** → run it and show output before claiming done. + +## Worked example + +```console +$ forge substrate "make the auth better" + proceed: ASK FIRST + assumption: high risk · completeness 0.23 + clarify: + - What exactly should this produce, and how will we know it is correct? +``` + +→ Under-specified. Ask the clarify question instead of editing. + +```console +$ forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests" + proceed: yes + impact: 3 file(s) predicted + - src/auth.js + - src/login.js (importer you did not name) + - src/session.js (importer you did not name) ``` -Use the result this way: +→ Cleared to proceed, but review the two coupled importers first. -1. If `okToProceed` is false, ask the returned `assumption.questions` before editing. -2. Use `route.model` as the cheapest capable tier recommendation; do not escalate without a verifier failure. -3. Read `impact.impactedFiles` before editing a named symbol/file. -4. Use `scope.clusters` to split independent work into separate sessions. -5. Treat `memory.advisory` as context, not law; tests and human corrections override it. -6. Run the `verification.checklist` before claiming completion. +## Single-mechanism commands -For details, read `references/capability-map.md` only when you need to explain how the paper's faculties map to Forge commands. +`forge preflight ""` (assumptions) · `forge route ""` (model tier) · +`forge impact ` (blast radius) · `forge scope ` (decomposition). +MCP equivalents: `assumption_gate`, `route_task`, `predict_impact`, `scope_files`. -The full paper bundle is in `docs/cognitive-substrate/` when the repository is available: PDF, HTML, evidence map, ecosystem map, and original prototype packages. +For the full guide (how it works, extending it, the honesty boundary) and the white paper, +see `docs/cognitive-substrate/README.md`. `references/capability-map.md` maps faculties to commands. diff --git a/source/rules.json b/source/rules.json index 35a9e65..49ea743 100644 --- a/source/rules.json +++ b/source/rules.json @@ -11,6 +11,18 @@ "Fix root causes, not symptoms. Never suppress an error to make a check pass." ] }, + { + "id": "substrate", + "title": "Cognitive substrate (run before acting)", + "rules": [ + "Before ambiguous, expensive, multi-file, or mutating work, run `forge substrate \"\" --json` (or call the MCP tool `substrate_check`). In Claude Code this fires automatically on each prompt; in other tools invoke it yourself.", + "If it returns `okToProceed:false`, ask the returned `assumption.questions` before editing — do not guess an under-specified task.", + "Start at the recommended `route.tier` (cheapest capable model); escalate only after an external verifier fails, never preemptively.", + "Read `impact.impactedFiles` (predicted blast radius) before editing a named symbol/file, and split `scope.clusters` of independent work into separate sessions.", + "Treat `memory.advisory` and minimality warnings as advisory; tests, graph membership, and human corrections always override the model.", + "Run `verification.checklist` and show the output before claiming done." + ] + }, { "id": "reuse", "title": "Reuse & dependencies", diff --git a/src/cortex_hook_main.js b/src/cortex_hook_main.js index da08ae8..721c3ef 100644 --- a/src/cortex_hook_main.js +++ b/src/cortex_hook_main.js @@ -5,6 +5,8 @@ // // modes: capture (PostToolUse Edit|Write|Bash) — log a signal event // prompt (UserPromptSubmit) — log a user-utterance event +// preflight (UserPromptSubmit) — inject the substrate pre-action advisory +// pre-edit (PreToolUse Edit|Write) — advise on lessons/risk before an edit // stop (Stop) — distill the session into lessons // session-start (SessionStart) — inject learned lessons as context import { applyDistillation, lessonsForContext, startupBlock } from "./cortex.js"; @@ -16,7 +18,7 @@ import { readSession, } from "./cortex_hook.js"; import { load } from "./lessons_store.js"; -import { clarifyBlock, preflightRepo } from "./preflight.js"; +import { substrateCheck, substrateContext } from "./substrate.js"; // Opt-in: distill newly-created lessons into real prose via a cheap model call. Off by // default (deterministic template is used); fail-safe (any error → keep the template). @@ -70,10 +72,12 @@ async function main() { const advice = await preEditAdvisory(root, hook.tool_input?.file_path, today); if (advice) emit("PreToolUse", advice); } else if (mode === "preflight") { - // Assumption detector: does the task name things the repo doesn't define? + // Ambient cognitive substrate: assumption gate + (when an atlas is already cached) + // model routing, blast-radius, memory, and minimality — surfaced before the agent acts. + // allowBuild:false keeps it cheap and never writes .forge/ from a hook; advisory only. if (typeof hook.prompt === "string" && hook.prompt.trim()) { - const block = clarifyBlock(preflightRepo(root, hook.prompt, { allowBuild: false })); - if (block) emit("UserPromptSubmit", block); + const advisory = substrateContext(substrateCheck(root, hook.prompt, { allowBuild: false })); + if (advisory) emit("UserPromptSubmit", advisory); } } } diff --git a/src/substrate.js b/src/substrate.js index 75c6481..cafc5b3 100644 --- a/src/substrate.js +++ b/src/substrate.js @@ -8,7 +8,7 @@ import { fileURLToPath } from "node:url"; import { build as buildAtlas, impact as impactGraph, load as loadAtlas } from "./atlas.js"; import { matchingLessons } from "./cortex.js"; import { load as loadLessons } from "./lessons_store.js"; -import { assessTask, clarifyBlock, preflightRepo, referencedEntities } from "./preflight.js"; +import { clarifyBlock, preflightRepo, referencedEntities } from "./preflight.js"; import { routeTask } from "./route.js"; import { decompose } from "./scope.js"; @@ -65,15 +65,22 @@ export function predictImpact(root, target, { threshold = 0.1 } = {}) { return impactGraph(atlas, target, { threshold }); } -export function substrateCheck(root, task, { threshold = 0.1, askThreshold = 0.6 } = {}) { +export function substrateCheck( + root, + task, + { threshold = 0.1, askThreshold = 0.6, allowBuild = true } = {}, +) { const text = String(task || ""); const entities = referencedEntities(text); - const preflight = preflightRepo(root, text, { askThreshold }); - const assumption = assessTask(text, { askThreshold }); + const preflight = preflightRepo(root, text, { askThreshold, allowBuild }); const route = routeTask(root, text); - const atlas = loadAtlas(root) || buildAtlas({ root }); + // allowBuild:false (ambient hooks) uses the atlas only if one is already cached — never + // builds or writes .forge/atlas.json from a hook. Impact is then best-effort. + const atlas = loadAtlas(root) || (allowBuild ? buildAtlas({ root }) : null); const impactTargets = [...new Set([...entities.symbols, ...entities.files])].slice(0, 8); - const impacts = impactTargets.map((target) => impactGraph(atlas, target, { threshold })); + const impacts = atlas + ? impactTargets.map((target) => impactGraph(atlas, target, { threshold })) + : []; const impactedFiles = [...new Set(impacts.flatMap((r) => r.impactedFiles || []))].sort(); const scopedFiles = [...new Set([...entities.files, ...impactedFiles])]; const scope = scopedFiles.length @@ -84,9 +91,9 @@ export function substrateCheck(root, task, { threshold = 0.1, askThreshold = 0.6 symbols: entities.symbols, }); const result = { - okToProceed: !preflight.assumption.shouldAsk && !assumption.shouldAsk, + okToProceed: !preflight.assumption.shouldAsk, task: text, - assumption: preflight.assumption.shouldAsk ? preflight.assumption : assumption, + assumption: preflight.assumption, clarify: clarifyBlock(preflight), route, entities, @@ -94,9 +101,11 @@ export function substrateCheck(root, task, { threshold = 0.1, askThreshold = 0.6 scope, memory: { matchingLessons: lessons.length, - advisory: lessons - .slice(0, 5) - .map((lesson) => ({ id: lesson.id, status: lesson.status, scope: lesson.scope })), + advisory: lessons.slice(0, 5).map((lesson) => ({ + id: lesson.id, + status: lesson.status, + scope: lesson.scope, + })), }, minimality: { warnings: minimalityWarnings(text, route, preflight) }, verification: { checklist: verificationChecklist(root) }, @@ -147,3 +156,37 @@ export function renderSubstrate(result) { for (const c of result.verification.checklist) lines.push(` - ${c}`); return lines.join("\n"); } + +// Compact advisory for AMBIENT injection (Claude Code UserPromptSubmit additionalContext). +// Returns "" unless there is something worth surfacing — never nags on a well-specified, +// low-impact task. Gated on: must-ask assumptions, a premium model recommendation, +// predicted blast radius, or a minimality warning. +export function substrateContext(result) { + const worthSaying = + result.assumption.shouldAsk || + result.impact.impactedFiles.length > 0 || + result.minimality.warnings.length > 0 || + ["opus", "fable"].includes(result.route.key); + if (!worthSaying) return ""; + const lines = ["Forge substrate — pre-action advisory (advisory, never blocks):"]; + if (result.assumption.shouldAsk) { + lines.push( + `- Under-specified (${result.assumption.risk} risk). Ask before editing:`, + ...result.assumption.questions.map((q) => ` • ${q}`), + ); + } + lines.push( + `- Suggested model: ${result.route.model.name} (${result.route.tier}); escalate only on a verifier failure.`, + ); + if (result.impact.impactedFiles.length) { + const files = result.impact.impactedFiles; + lines.push( + `- Predicted blast radius (${files.length}): ${files.slice(0, 8).join(", ")}${files.length > 8 ? " …" : ""}. Review these before editing.`, + ); + } + for (const w of result.minimality.warnings) lines.push(`- Minimality: ${w}`); + if (result.memory.matchingLessons) + lines.push(`- ${result.memory.matchingLessons} past lesson(s) match this area (advisory).`); + lines.push(`- Verify with: ${result.verification.checklist.join(" · ")}`); + return lines.join("\n"); +}