Bounded search foundations and source-navigation recall

.github/workflows/ci.yml

@@ -19,4 +19,7 @@ jobs:
       - run: npm run build
       - run: npm test
       - run: npm run benchmark:tokens:check
+      - run: npm run benchmark:tokens:parity
+      - run: node --test benchmarks/scale/sqlite.test.mjs benchmarks/scale/postings.test.mjs
+      - run: node benchmarks/scale/postings-bench.mjs --records 10000
       - run: npm run test:packages

GOALS.md

@@ -0,0 +1,83 @@
+# Z1P Core release goals
+
+Updated: 21 September 2026.
+
+We want coding agents to spend less time and inference rediscovering a codebase, without making their answers or changes less reliable.  The open-source core comes first.  It must be useful, dependable and independently installable before private services become a dependency.
+
+This is an acceptance plan, not a claim that the gates have passed.  Existing code and tests are a starting point.  Record fresh evidence against the release commit before closing a gate.
+
+Current results and open blockers are recorded in [RELEASE_EVIDENCE.md](RELEASE_EVIDENCE.md).
+
+## Order of work
+
+1. Establish the release baseline and remaining gaps.
+2. Prove trust boundaries, extraction and retrieval behaviour.
+3. Prove installation and real consumer workflows.
+4. Measure complete tasks, then publish a verified release.
+5. Only then build private operational features justified by real use.
+
+The free/private boundary remains [OPEN_CORE.md](OPEN_CORE.md).  Local extraction, verification, bounded retrieval, CLI, MCP and portable formats stay MIT licensed.  Provider routing and worker execution belong in consumers.  A generally useful core fix belongs here even if a private pilot discovers it.
+
+## G0: A reproducible baseline
+
+- [ ] Record the candidate commit, Node version, platform, commands and complete results in a release evidence document.
+- [ ] Run `npm run check`, `npm run benchmark:tokens:check` and `npm run benchmark:tokens:parity` from a clean checkout using `npm ci --ignore-scripts`.
+- [ ] Map the existing tests to G1-G3 below.  Record missing cases explicitly rather than treating a green suite as complete coverage.
+- [ ] Classify release blockers by correctness, security, compatibility and usability; give each an owner and a regression test where applicable.
+- [ ] Verify CI for the candidate commit.  If infrastructure prevents it running, record the blocker and local evidence separately; do not call CI passed.
+
+Exit: another developer can reproduce the baseline, and every known release blocker has a concrete acceptance condition.
+
+## G1: Trust boundaries we can rely on
+
+- [ ] Exercise valid and tampered signatures, wrong keys, malformed input, unsupported versions and invalid grants through public APIs.
+- [ ] Prove collection and project isolation, scoped access and revocation on subsequent reads.  Document that revocation cannot erase plaintext already obtained by an authorised reader.
+- [ ] Prove graph edges, repository text and retrieved instructions cannot grant access, execute code or become trusted instructions merely by being retrieved.
+- [ ] Test repository path escapes, symlinks, ignored/secret files, oversized inputs and traversal limits.  Document what is excluded and what is not guaranteed to be detected.
+- [ ] Check CLI, MCP, errors and logs for unintended disclosure of keys, credentials and plaintext.  No network transfer without an explicit configured operation.
+- [ ] Document the threat model, key custody, supported security properties and a private vulnerability-reporting route.
+
+Exit: negative tests protect each stated boundary, with no unresolved release-blocking security or data-isolation defect.
+
+## G2: Evidence that is correct enough to use
+
+- [ ] Prove indexed queries, refresh and resource budgets through the [enterprise scale plan](docs/ENTERPRISE-SCALE-REVIEW.md) before making whole-codebase claims or raising the 128-record limit.  ADR-001 remains provisional until its query and trust-boundary gaps are resolved.
+- [ ] Verify deterministic output and stable source identity on repeat scans, with fixtures for edits, deletion, renames, duplicate names and ambiguous links.
+- [ ] State snapshot freshness and how to replace or invalidate stale evidence.  A stale record must not silently masquerade as the current source.
+- [ ] Keep TypeScript/JavaScript compiler-derived evidence separate from lexical and inferred evidence in output and documentation.
+- [ ] Cover multi-repository manifests, explicit cross-repository links, missing repositories and ambiguous package names without widening access.
+- [ ] Exercise retrieval budgets, truncation, empty results, corrections and path limits.  Return enough provenance to inspect the source and recognise incomplete answers.
+- [ ] Retain full predeclared required-source recall in both existing benchmark gates.  Report raw answer evidence and navigation compression separately.
+
+Exit: documented behaviour matches executable fixtures, including the cases where extraction cannot answer reliably.
+
+## G3: A release someone else can actually use
+
+- [ ] Test real package tarballs outside the workspace: imports, exports, browser isolation, CLI, MCP, notices and licences.
+- [ ] Walk through the documented install, scan, persist, retrieve, export and import path on a clean machine or isolated environment.
+- [ ] Exercise an actual MCP client session, including initialisation, errors, cancellation and bounded output.  Name the tested client and version.
+- [ ] Verify compatibility fixtures for existing package names, APIs and signed wire formats.  Document any required migration before release.
+- [ ] Test intended consumer integration in isolated branches or fixtures.  Do not silently repin or migrate live consumers.
+- [ ] Reconcile README commands, package versions, changelog and extraction provenance.  Distinguish a prepared tarball from an available registry release.
+- [ ] Publish through the authorised release process, then install the exact published versions from the registry outside the workspace and repeat the smoke workflow.
+
+Exit: a newcomer can follow the published instructions successfully without local workspace links or unpublished dependencies.
+
+## G4: Useful savings, with quality held constant
+
+- [ ] Define representative tasks and acceptance tests before comparing baseline and Z1P-assisted runs.  Include repository orientation, bug investigation, change impact and an accepted code change.
+- [ ] Hold repository revisions, task instructions, model settings and acceptance standards constant.  Separate context improvements from any later model-routing experiment.
+- [ ] Count the whole task: retrieved source, input/output tokens, cached tokens where reported, retries, tool calls, failures and review time.  Include scan and refresh overhead.
+- [ ] Record missing provider usage as unknown, never zero.  Keep observed billing, estimated token cost and fixed subscription costs distinct.
+- [ ] Report accepted tasks, regressions and unsuccessful runs alongside cost.  Navigation compression alone is not an inference-bill saving.
+- [ ] Publish a reproducible, non-sensitive evaluation and its limitations.  Keep private code, prompts, invoices and customer data out of this repository.
+
+Exit: the tested workflow reduces cost per accepted outcome without lowering the agreed quality bar.  If it does not, record the result and fix the cause before making a savings claim.
+
+## Definition of done
+
+G0-G4 have linked evidence and reviewer sign-off for a named release.  No known release-blocking defect remains.  The free local route works without a Z1P account or service, and the published packages have passed an independent install check.
+
+For each gate, record: status (`not assessed`, `in progress`, `blocked`, `passed`), owner, commit, environment, command or workflow, result, evidence link, reviewer and remaining limitations.  Unchecked items are unverified, not necessarily unimplemented.
+
+Start with G0.  Do not broaden language support, build a graph editor or add hosted dependencies to avoid fixing the release path.

README.md

@@ -41,6 +41,11 @@ The free core is intended to be useful on its own, not a time-limited trial.
 See [OPEN_CORE.md](OPEN_CORE.md) for the durable product boundary and
 compatibility policy.
 
+See [GOALS.md](GOALS.md) for the core-first release gates, required evidence
+and whole-task inference-cost evaluation.
+The current candidate results and blockers are recorded in
+[RELEASE_EVIDENCE.md](RELEASE_EVIDENCE.md).
+
 The formats and APIs are project-agnostic. A collection can describe one
 repository or an explicitly assembled ecosystem; graph operations never make
 another collection visible or turn an extracted relationship into authority.

RELEASE_EVIDENCE.md

@@ -0,0 +1,94 @@
+# Z1P Core release candidate evidence ledger
+
+## Source shipment verification, 21 September 2026
+
+The scale-foundations shipment passes 33 core tests, 28 tools tests, independent packed-package smoke, both token benchmark gates, 22 synthetic scale tests and the 10k posting-index runner. The memory blob fixture now copies Node Buffer inputs independently and rejects non-string digests before coercion. CI includes navigation recall, both scale suites and the 10k probe. These are source-shipment checks, not closure of G0-G4 or an npm publication. PR and main CI provide commit-specific remote evidence; earlier entries below remain historical snapshots.
+
+Updated: 21 September 2026.
+
+## Candidate metadata
+
+| Field | Value |
+| --- | --- |
+| Evidence date | 21 September 2026 |
+| Tested commit | `5540c4d` |
+| Working tree | Uncommitted `README.md`, `GOALS.md` and this ledger are documentation changes only |
+| Runtime | Node v24.21.0 |
+| Platform | macOS arm64 |
+| Registry | `@forgesworn/context-tools@0.3.0` is not published |
+
+Overall status: **G0 in progress, not passed.**
+
+The tested commit builds and its current automated tests pass.  CI has not been rerun for the eventual release commit, known release gaps remain, and no readiness or security conclusion follows from this evidence.
+
+## Commands and observed outcomes
+
+```sh
+npm run check && npm run benchmark:tokens:check && npm run benchmark:tokens:parity
+```
+
+| Check | Outcome | Evidence |
+| --- | --- | --- |
+| TypeScript builds | Passed | Both workspace packages compiled |
+| `@forgesworn/context` tests | Passed | 4 files, 25 tests |
+| `@forgesworn/context-tools` tests | Passed | 4 files, 24 tests |
+| Packed-package smoke | Passed | Independent Node import, declarations and browser bundle; independent CLI create, append and restart recovery |
+| Raw evidence benchmark | Passed | 26 files, 67,087 baseline tokens, 6 queries, aggregate 28.30x reduction multiplier, 96.47% reduction, minimum declared-source recall 1.0; 10x regression floor passed |
+| Navigation benchmark | Passed with material limitation | 25 files, 60,541 baseline tokens, 8 queries, average 140.71x, minimum 138.54x and median 140.47x, minimum declared-source recall 1.0; 71.5x reference target passed |
+
+The navigation scan retained 128 graph records and dropped 111.  This makes the present collection ceiling a release and product blocker for whole-codebase claims.  The navigation result measures compact pointers to required sources.  It is not sufficient answer evidence and is not a measured inference-bill saving.
+
+## Gate status
+
+| Gate | Status | Evidence and remaining gaps |
+| --- | --- | --- |
+| G0 reproducible baseline | In progress | Local build, 49 tests, packed smoke and both benchmarks passed. The release commit and a current CI run remain outstanding. Test-to-gate coverage is recorded below. |
+| G1 trust boundaries | Partially evidenced | Tests cover grant and scope rechecks, proof refusal, uploader signing boundaries, bounded traversal and scanner path/symlink handling. The full negative boundary matrix and published threat model remain outstanding. |
+| G2 usable evidence | Partially evidenced | Tests cover deterministic scans, corrections, provenance, ambiguity and response bounds. Repository refresh across deletion and rename is not implemented or proven. The 128-record collection ceiling prevents whole-codebase claims. |
+| G3 independent use | Partially evidenced | Real tarballs work outside the workspace. A published-registry install, documented end-to-end walkthrough and named real MCP client robustness check remain outstanding. |
+| G4 cost per accepted task | Not started | No paired pilot trial has run. Current token benchmarks do not measure accepted task quality, retries, review time or billed cost. |
+
+## Existing test evidence
+
+- `retrieval.test.ts`: bounded query hits and one-hop links without external reads; current-head corrections; exact UTF-8 caps and oversized omission; source revisions are not conflated.
+- `graph.test.ts`: atomic links and cycles; bounded query and path operations without crossing collections; provenance; invalid, absent and corrected endpoints; byte, depth and node caps; corrected stale topology; deterministic shortest paths.
+- `index.test.ts`: grant and scope rechecks; identity sharing and restore; malformed provenance; unknown proof and personal-sharing refusal; host proof adapter; uploader signing boundaries.
+- Scanner tests: deterministic JS/TS and conservative broad-language output; hidden, generated and symlinked trees ignored or rejected; UTF-8 and resource bounds; append-safe relations; ambiguous packages and links handled.
+- `blossom.test.ts`: published envelope fixture compatibility.
+- The package smoke builds real tarballs and checks the independent consumers described above.
+
+These statements describe the named tests.  They do not extend to untested environments, attacks or integrations.
+
+## Known blockers
+
+1. Design and prove a route beyond 128 records while keeping each read bounded.
+2. Define and test repository refresh for changed, deleted and renamed files, including explicit stale-snapshot behaviour.
+3. Exercise initialisation, errors, cancellation and bounded results through a named real MCP client.
+4. Publish the matching packages through the authorised process, then repeat the independent smoke against the registry versions.
+5. Run CI on the eventual release commit and record its result separately from this local pass.
+6. Publish the threat model and complete the negative trust-boundary test matrix.
+7. Run the paired cost-per-accepted-task trial.  Include retries, failures, review effort, scan/refresh cost and the actual charging basis.
+
+## Claim boundaries
+
+- Passing these checks does not establish that Z1P Core is secure, release-ready or production-ready.
+- The broad-language scanner is conservative lexical navigation, not compiler-level semantic analysis.
+- The benchmark corpora and declared-source checks do not prove task answer quality.
+- No cost saving is claimed until the paired trial measures complete accepted tasks.
+- Registry availability, current CI, a real client workflow and normal-work use remain separate gates.
+
+## Next action
+
+Follow the [enterprise scale review](docs/ENTERPRISE-SCALE-REVIEW.md): prove indexed queries and resource budgets before implementing a new cryptographic paging format.
+
+## Subsequent working-tree checks
+
+Adding the experimental paging memory stores increased the test count to 55; both builds, all tests and independent package smoke passed.  Raw-evidence reduction was 28.90x with minimum declared-source recall 1.0.
+
+The expanded navigation corpus failed: 27 files, 128 retained records, 122 dropped records, and `retrieveView` source recall 0.  The earlier passing baseline above remains historical evidence only.  The current working tree does not pass the full gate.  No scanner repair has been applied, and no benchmark threshold has been relaxed.
+
+### Worker implementation follow-up
+
+The scanner regression above has now been repaired by adding bounded exported-name anchors to file records and allocating symbol detail round-robin across files.  Existing IDs, source fields and the 128-record limit are preserved.  Full workspace verification passed: 31 core tests, 28 tools tests, independent package smoke and both unchanged token benchmark gates.  Navigation minimum declared-source recall is again 1.0, with aggregate reduction 144.73x on this working tree.
+
+The separate [synthetic scale probe](benchmarks/scale/README.md) adds eight passing tests and successful 10k/50k and 100k/500k record/edge runs.  This prototype is not integrated into Core and does not lift the current collection limit or qualify private-data storage.