refactor(knowledge): modularize KB and RAG internals

AGENTS.md

@@ -2,22 +2,31 @@
 
 Use this package when an agent needs persistent, source-grounded knowledge that improves over time.
 
-## Repo layering — this package depends on agent-eval, never the reverse
+## Package layering
 
 ```
-agent-knowledge (this repo) ─┐
-                             ├──► agent-eval (substrate — the bottom)
-agent-runtime ───────────────┘
+agent-runtime
+      |
+agent-knowledge
+    /       \
+agent-eval  agent-interface
 ```
 
-**Rule: agent-knowledge depends on agent-eval. agent-eval MUST NOT import from agent-knowledge.** No upward imports, no peerDependency declaration in agent-eval pointing at agent-knowledge. Substrate primitives that agent-knowledge needs (`AnalystFinding`, `RunRecord`, optimizer types, release-confidence types) live in agent-eval; this repo consumes them.
+Imports flow downward in this diagram.
+`agent-knowledge` may import `agent-eval` and `agent-interface`, but it must not import `agent-runtime`.
+Agent-powered work enters this package through callbacks.
+`agent-runtime` owns live agent execution and composes this package when a complete agent workflow is needed.
+`agent-eval` must not import `agent-knowledge`.
+Shared run and experiment types that knowledge needs live in `agent-eval`.
 
 Types that stay in THIS repo because they're knowledge-domain-shaped:
 - `KbStore`, `KnowledgeFragment`, `KnowledgeChange`
 - `KnowledgeDiscoveryDispatcher`, source adapters (`createCornellLiiSource`, `createIrsPublicationsSource`)
 - Freshness store + change-detection primitives
 
-**The test for "where does a type live?"** — does this concept make sense WITHOUT persistent knowledge or sourced fragments? If yes, it's substrate (agent-eval). If no, it's a knowledge-domain concept (stays here).
+**The test for "where does a type live?"**
+If the concept makes sense without persistent knowledge or sourced fragments, it belongs in `agent-eval` or `agent-interface`.
+Otherwise, it stays in this package.
 
 ## Rules
 
@@ -72,36 +81,42 @@ Use `knowledgeReleaseReport()` before promotion. It folds the candidate and base
 
 ## Integration Boundaries
 
-- Use `KbStore` for storage. Implement D1 in the consuming app when needed.
-- Use `KnowledgeDiscoveryDispatcher` for research workers. Production apps should wire this to their own swarm/fleet runtime.
+- Use `KbStore` for storage. Applications may provide any durable backend that implements it.
+- Use `KnowledgeDiscoveryDispatcher` for research workers. Applications should connect it to their own runtime.
 - Do not bypass `lint` or `validate` before using generated knowledge in an agent.
 
 ## Pluggable Sources + Freshness + Changes
 
 Agents that need to stay current against external authorities should compose:
 
-- `createCornellLiiSource({ selectors })` — US Code + Wex from law.cornell.edu.
-- `createIrsPublicationsSource({ publications, revenueProcedures })` — IRS index + named pubs.
-- `createStateSosSource({ state, baseUrl, entities })` — generic state SOS adapter.
+- `createCornellLiiSource({ selectors })`: US Code and Wex from law.cornell.edu.
+- `createIrsPublicationsSource({ publications, revenueProcedures })`: IRS index and named publications.
+- `createStateSosSource({ state, baseUrl, entities })`: generic state SOS adapter.
 
-Every fetch returns `KnowledgeFragment[]` with `provenance.verifiable` indicating whether the authority was successfully authenticated. Refuse to cite fragments with `verifiable: false`.
+Every fetch returns `KnowledgeFragment[]` with `provenance.verifiable` indicating whether the configured URL returned an acceptable response and the expected content was extracted.
+This flag does not authenticate the publisher or cryptographically prove the content.
+Refuse to cite fragments with `verifiable: false`.
 
 Track per-tenant freshness with `createFileSystemFreshnessStore({ root })` and re-fetch only when `stale({ workspaceId, sourceId, ttlMs })` returns true.
 
-Diff snapshots with `detectChanges(prev, next)`. Each `KnowledgeChange` carries `affectedDimensions` — pass those to your eval scheduler to re-run only the relevant campaigns.
+Diff snapshots with `detectChanges(prev, next)`.
+Each `KnowledgeChange` carries `affectedDimensions`; pass those to your eval scheduler to run only the relevant campaigns again.
 
 ## Authorship
 
-Do not add `Co-Authored-By:` trailers (or any other AI-attribution lines) to commits, PR descriptions, or other artifacts in this repo. Author = the human running the session. Applies to every contributor, including AI agents and subagents — do not include the default Claude Code template trailer.
+Do not add `Co-Authored-By:` trailers or other AI-attribution lines to commits, PR descriptions, or repository artifacts.
+The author is the human running the session.
 
 ## Comment & doc discipline (no historical narrative)
 
-Comments describe **what the code does and why** — never what it used to do, what it replaced, which audit found a bug, or what the prior version looked like. History belongs in commit messages and PR descriptions, not the source tree.
+Comments describe **what the code does and why**.
+They must not describe what code used to do, what it replaced, which audit found a bug, or what a prior version looked like.
+History belongs in commit messages and PR descriptions.
 
 - Bad: `// replaces the inline retry loop`, `// fix for the silent-zero bug`, `// the 2yr rewrite added this`, `// audit fix`
-- Good: `// value: null when retries exhaust — callers must inspect succeeded`
+- Good: `// value is null when retries exhaust; callers must inspect succeeded`
 
-Applies to docstrings, README sections, SKILL.md, AGENTS.md, CLAUDE.md — anywhere the source tree carries prose.
+This applies anywhere the repository carries prose.
 
 ## No fallbacks. Fail loud.
 
@@ -110,4 +125,3 @@ Sloppy fallbacks corrupt every signal downstream. No silent zeros, no `?? defaul
 External-boundary calls (LLM, network, FS, subprocess) return *typed outcomes* (`{ succeeded, value, error }`). Callers MUST inspect `succeeded` before using `value`. Named, opted-in fallback rotations (`policy.fallbackModels: [...]`) are fine; deep `?? "kimi"` helpers are not.
 
 Full doctrine: `~/dotfiles/claude/AGENTS.md` → "No fallbacks. Fail loud."
-

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 4.1.0
+
+### Added
+
+- Added optional scenario input to `knowledgeReleaseReport()` so a required holdout can prove both scenario and run coverage.
+
+### Changed
+
+- Split knowledge-base improvement and RAG evaluation internals into focused modules while preserving public exports and implementation behavior.
+- Split the knowledge improvement tests into candidate, promotion, activation, and integrity suites with shared setup in one support module.
+- Removed unused development dependencies and internal-only exports.
+- Updated the test runner to Vitest 4 and Node type definitions to 26; TypeScript remains on 5.9 because tsup's declaration bundler does not yet support TypeScript 7.
+- Declared Node types explicitly in TypeScript configuration instead of relying on ambient type discovery.
+- Corrected the package dependency guide and RAG roadmap to reflect runtime-owned agent execution through `runKnowledgeImprovementJob()`.
+- Removed historical commentary and em dashes from repository documentation.
+
+### Fixed
+
+- Replaced the stale versioned HTTP user agent with a stable package identity and added request-header coverage.
+
 ## 4.0.1
 
 ### Changed

README.md

@@ -14,7 +14,7 @@ This package turns raw sources and generated markdown knowledge into a versionab
 - [Benchmark runner](#benchmark-runner): BEIR/MTEB/qrels, RAG answer, hallucination, KB-improvement cases
 - [Agent-Eval integration](#agent-eval-integration): retrieval eval, readiness bundles, and release reports
 - [Memory adapters](#memory-adapters): Mem0, Graphiti, Neo4j, branching, and automatic improvement
-- [Research loop](#research-loop): `runKnowledgeResearchLoop` plus the control-loop adapter
+- [Research loops](#research-loop): direct and worker/reviewer knowledge growth
 - [Runtime integration](#runtime-integration): how agent runners plug into the pure KB loop
 - [Pluggable knowledge sources](#pluggable-knowledge-sources): live authorities → eval re-runs
 
@@ -31,7 +31,7 @@ Two ways in, depending on what you're doing:
 - **Author / inspect a KB by hand** → the [CLI](#cli) (`init` → `source-add` → `index` → `search` → `lint`). Fastest way to see the shape on disk.
 - **Drive it from an agent** → pick the primitive by intent:
   - *"Does the agent have enough context to run?"* → [`buildEvalKnowledgeBundle`](#agent-eval-integration) (block / ask / acquire before execution).
-  - *"Grow the KB as a researcher"* → [`runKnowledgeResearchLoop`](#research-loop) (deterministic mechanics; your agent owns judgment) or [`runTwoAgentResearchLoop`](#two-agent-research-loop) (researcher proposes, verifier checks + fills gaps).
+  - *"Grow the KB as a researcher"* → [`runKnowledgeResearchLoop`](#research-loop) (deterministic mechanics; your agent owns judgment) or [`runVerifiedResearchLoop`](#verified-research-loop) (researcher proposes, reviewer checks and fills gaps).
   - *"Spawn live agents to improve a KB"* → pass an `updateKnowledge` callback to `improveKnowledgeBase`; runtime-backed supervisors live in `@tangle-network/agent-runtime`.
   - *"Tune retrieval for a knowledge base"* → `runRetrievalImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section.
   - *"Run an operator-grade KB improvement cycle"* → `improveKnowledgeBase` in the [Agent-Eval integration](#agent-eval-integration) section.
@@ -419,6 +419,17 @@ To answer whether a candidate knowledge base improves agent task success, run an
 
 Use `knowledgeReleaseReport()` before promotion: pass the candidate and baseline `RunRecord[]` (plus optional `ReleaseTraceEvidence` and the gate decision) and it folds them into a `ReleaseConfidenceScorecard` and a `KnowledgeRelease` using `agent-eval`'s release gates and `RunRecord` validation.
 
+When holdout evidence is required, pass both the holdout-tagged run records and the scenario split:
+
+```ts
+const release = knowledgeReleaseReport({
+  candidateId,
+  candidateRuns,
+  scenarios,
+  hasHoldout: true,
+})
+```
+
 Use `buildEvalKnowledgeBundle()` before execution when the question is whether
 the agent has enough task-world context to run:
 
@@ -805,15 +816,13 @@ await runAgentControlLoop({
 })
 ```
 
-## Two-agent research loop
+## Verified research loop
 
-`runTwoAgentResearchLoop()` is the offline sibling of `runKnowledgeResearchLoop`
-with a differentiated worker/driver split over ONE knowledge base: the `worker`
-does primary research (discovers sources, proposes pages for the open gaps); the
-`driver` verifies each candidate source before it commits, optionally gap-fills
-with its own pass (`driverResearches: true`), and gates on the readiness check.
-Both are yours (no creds). The loop owns the deterministic mechanics (indexing,
-applying write blocks, scoring readiness) and stops once no blocking gap remains.
+`runVerifiedResearchLoop()` coordinates a worker and reviewer over one knowledge base.
+The worker discovers sources and proposes pages for open gaps.
+The reviewer checks each source before admission and can run its own gap-filling pass with `driverResearches: true`.
+Callers provide both roles and any credentials.
+The loop owns indexing, write-block application, readiness scoring, and stopping when no blocking gap remains.
 
 An equal-compute comparison across 9 ML topics using `glm-5.2` is documented in
 [docs/two-agent-research-ab.md](docs/two-agent-research-ab.md).
@@ -824,10 +833,10 @@ The document includes limitations and reproduction steps.
 ```ts
 import {
   defineReadinessSpec,
-  runTwoAgentResearchLoop,
+  runVerifiedResearchLoop,
 } from '@tangle-network/agent-knowledge'
 
-await runTwoAgentResearchLoop({
+await runVerifiedResearchLoop({
   root: './kb',
   goal: 'Build a grounded onboarding wiki for billing support',
   readinessSpecs: [defineReadinessSpec({
@@ -836,14 +845,14 @@ await runTwoAgentResearchLoop({
     query: 'refund policy customer request',
     requiredFor: ['support-agent'],
   })],
-  // WORKER: primary research targeting `ctx.gaps`. Returns a ResearchContribution.
+  // The worker researches the current gaps.
   async worker({ gaps, index }) {
     return {
       sources: [/* AddSourceTextInput for the open gaps */],
       proposalText: '/* ---FILE: knowledge/…--- write-protocol blocks */',
     }
   },
-  // DRIVER: verifies each candidate source before it commits, then gates.
+  // The reviewer checks each candidate source before admission.
   driver: {
     verifySource(source, { gaps }) {
       return source.uri ? { accept: true } : { accept: false, reason: 'no uri' }
@@ -915,11 +924,14 @@ three primitives that bridge "live authority" → "eval re-runs":
 
 - `KnowledgeSource`: pluggable contract (`fetch(opts) → KnowledgeFragment[]`).
   Every fragment carries `provenance` (URL, source-attested timestamp,
-  jurisdiction, `verifiable` flag) and `dimensionHints` (which eval
-  dimensions a change in this fragment should re-score).
+  jurisdiction, and fetch/extraction status) and `dimensionHints` (which eval
+  dimensions a change in this fragment should re-score). The `verifiable`
+  flag rejects failed, blocked, or malformed responses; it does not
+  cryptographically authenticate the publisher.
 - `KnowledgeFreshnessStore`: per-`(workspaceId, sourceId)` last-refresh
-  tracker. Filesystem adapter ships in-package; D1 / Postgres adapter
-  scaffold is shipped as `createD1FreshnessStoreStub(adapter)`.
+  tracker. The package ships a filesystem implementation and
+  `createD1FreshnessStoreStub(adapter)`, a complete bridge for an
+  application-owned D1, PostgreSQL, SQLite, or equivalent adapter.
 - `detectChanges(prev, next)`: diffs two fragment snapshots, emits
   `KnowledgeChange[]` tagged with the affected eval dimensions so a cron
   scheduler knows exactly which campaigns to re-run.
@@ -998,4 +1010,4 @@ async function tick({ workspaceId, prevSnapshots }: {
 Polite-by-default: every HTTP fetch carries the package User-Agent, is
 throttled to 1 req/sec/origin, caches successful responses to disk, and
 marks `verifiable: false` on block pages / 4xx rather than promoting
-un-grounded content. See `src/sources/http.ts` for the invariants.
+unusable content. See `src/sources/http.ts` for the checks.