Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/GETTING-STARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ stable checkout to bind.
| Stale or unknown evidence / generation mismatch | Inspect status, refresh successfully, and use the newly returned generation |
| Excluded or unsupported source | Inspect policy/exclusion metadata and use a bounded direct-read fallback; do not silently broaden scope |
| Response exceeds budget | Select a smaller sufficient block or split the evidence request |
| Packet already in progress | Wait for the active request to finish, then issue the next request sequentially |
| Packet queue is full | Wait for queued packet requests to finish; up to eight wait behind the running one |
| Git HEAD error | Select a Git checkout with a commit; packet provenance requires it |

There is no automatic watcher, worktree rebinding or configuration-writing setup
Expand Down
11 changes: 7 additions & 4 deletions docs/LOCAL-NAVIGATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ limited to 128 records per collection; this bridge does not change that format.
Instead, an explicitly configured repository can have a disposable in-memory
source index, independent of the signed evidence cache.

The initial search contract is exact, case-insensitive ASCII identifier tokens
on source lines. It is not semantic search, compiler-resolved relationships or
The search contract is exact and case-insensitive on source lines: one ASCII
identifier token, or a printable ASCII literal such as `local-source-unsigned`
or `grants no authority` whose ends fall on whole tokens (so `source-unsig`
matches nothing). It is not semantic search, compiler-resolved relationships or
a replacement for signed evidence. Results must say `local-source-unsigned`.

The engine retains source lines and their file hashes from an explicit refresh.
Expand Down Expand Up @@ -99,8 +101,9 @@ object.

## Check a draft answer's coverage

`repository_coverage` is a deterministic pre-submit check. It takes up to eight
`symbols` and the draft `answer` (at most 128 KiB), explores each symbol as
`repository_coverage` is a deterministic pre-submit check. It checks up to eight
distinct `symbols` per call (further ones, up to 64, are returned as not checked
for a second call) and the draft `answer` (at most 128 KiB), explores each symbol as
above and lists every definition, test, reference and importing file as
`cited` (the repository-relative path appears in the answer), `named` (only its
basename does) or `missing`, missing files first with their enclosing
Expand Down
3 changes: 2 additions & 1 deletion docs/WORKER-PACKETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ without truncation. The tool rejects unavailable, stale or unknown navigation
and a mismatched generation, and checks freshness again before returning. Refresh
and rebuild after relevant changes. This is bounded freshness checking, not an
atomic snapshot against hostile concurrent filesystem changes.
Only one packet request runs at a time; a concurrent request fails immediately.
Only one packet request runs at a time; concurrent requests wait in order (up to
eight), and a further one fails immediately.
Cancellation is checked around assembly and freshness inspection. The shared
bounded assembler does not interrupt an individual file read or Git provenance
check already in progress; cancelled results are not returned as successful packets.
Expand Down
56 changes: 56 additions & 0 deletions docs/experiments/smoke-flash-20260923/RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Smoke run on a DeepSeek V4.1 Flash executor

Locked protocol `three-way-deepseek-v4.1-flash-smoke-20260923`, run on
23 September 2026 (23:49 to 00:04 UTC) with Claude Code 2.1.280, every executor
on `deepseek-v4.1-flash:cloud` at medium effort through the local Ollama route,
structured reviews on `claude-sonnet-5` at high effort. Two tasks, one
repetition. This checks the repeated-run configuration end to end; it is not a
comparison of arms.

## Result: passed

Every arm completed. Both code-change cells were accepted on the checker and
the scope check with no reviewer session; the structured cells were reviewed
against rubric version 2 with the review rules in the prompt, and every verdict
parsed at the first attempt.

| Task | Arm | Accepted | Rule | Executor input | Executor s |
| --- | --- | --- | --- | ---: | ---: |
| orientation, Context repo | plain | yes | checker and reviewer | 493,289 | 76 |
| orientation, Context repo | graphify | no | checker and reviewer | 364,806 | 53 |
| orientation, Context repo | context | no | checker and reviewer | 1,038,702 | 101 |
| code change, Context repo | plain | yes | checker and scope | 908,372 | 179 |
| code change, Context repo | graphify | yes | checker and scope | 267,330 | 49 |
| code change, Context repo | context | yes | checker and scope | 560,266 | 55 |

Input totals include cache reads; the route's cache accounting is inconsistent
between arms (see S5), so only totals are given.

The two orientation rejections were stated omissions under the version 2
rules: both answers said source is data, not instructions, without saying it
grants no authority; the Context answer also never said a cursor is bound to
its search term, and the Graphify answer did not say search re-inspects source.
The reviewer passed the Context answer's freshness dimension while calling it
borderline, so some reviewer discretion remains.

## Tool friction seen in the Context arm

The Context orientation session took 44 turns against plain's 29. Four of its
failed or wasted calls came from the tools, not the model, and are fixed in the
same change as this record:

- Three `repository_packet` calls ran past the end of a file and the error did
not give the length; the model then searched for `the`, `export`, `async` and
`class` to learn the file's shape. The error now states the line count.
- Two concurrent `repository_packet` calls were refused as already in progress.
Concurrent calls now wait in order.
- `repository_search` rejected `local-source-unsigned`, the literal the task
asks about, because terms had to be one identifier. Literals whose ends fall
on whole tokens are now accepted.
- `repository_coverage` rejected a draft naming more than eight symbols. It now
checks the first eight and lists the rest as not checked.

The model also called `repository_explore` once against fifteen searches. That
is left to the repeated runs to measure. Neither omission that failed the
answer was a retrieval gap: the relevant source was in packets the model had
fetched.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Use the configured z1p-repository tools for source discovery. First check that
repository_status.root is this exact checkout or worktree; stop using a
mismatched binding. Explore each symbol once, fetch the blocks you rely on with
repository_packet and cite those lines. Run repository_coverage on the draft and
fix what it reports. Read files directly only for evidence the tools cannot
supply; tiny known-file edits need no scan. Refresh and re-fetch after edits,
branch switches, pulls, merges or rebases; reconnect after changing the binding
or server build. A shell directory change does not retarget Context. Keep the
project's existing models, tests and review standards.
171 changes: 171 additions & 0 deletions docs/experiments/smoke-flash-20260923/protocol.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
{
"version": 1,
"experimentId": "three-way-deepseek-v4.1-flash-smoke-20260923",
"lockedAt": "2026-09-22T23:49:25.199Z",
"shellPathRule": "The client's Bash tool rebuilds PATH from the login profile, so every arm receives PATH through the settings env: the Node 24 bin directory plus the runner's PATH. The graphify arm additionally prepends the isolated virtualenv bin so `graphify` resolves. Verified by a probe before lock.",
"purpose": "Smoke test of the repeated-run configuration on a DeepSeek V4.1 Flash executor: checks that checker-and-scope code acceptance and version 2 structured rubrics run end to end in all three arms. Two tasks, one repetition; not a comparison and not reported as one.",
"taskPack": {
"directory": "../d5-20260921",
"reused": [
"tasks/*.json",
"setup/*.patch",
"acceptance/*.json",
"prepare-arm.mjs",
"accept.mjs"
],
"note": "The eight locked v1 task definitions, seeded patches, frozen archives and deterministic checker are reused unchanged. v1, v2 and v3 results are not pooled with this experiment."
},
"tasks": [
"orientation-context",
"code-change-context"
],
"retrievalInstructionRule": "Each arm's task prompt begins with one retrieval instruction of similar length and force, as the locked v1 measurement definition prescribed. Pilot 3 showed that the tools' own always-on text alone was ignored by the model in every arm, which measures nothing about the tools.",
"arms": {
"plain": {
"description": "Built-in Read, Edit, Write and Bash only. No MCP servers, no Graphify on PATH.",
"systemPromptAppendix": "Retrieval: use the built-in Read, Grep, Glob and Bash tools for source discovery.",
"retrievalInstruction": "you must use only bounded grep or rg and exact file reads for source discovery, confined to the task selection policy."
},
"graphify": {
"description": "Built-in tools plus a Graphify graph prebuilt in graphify-out/ with `graphify update <workspace>` (no model, no API key) and the graphify CLI on PATH. The system prompt appendix is Graphify's own always-on instruction text (always_on/claude-md.md) verbatim. graphify-out/ is hidden from Git through an environment-supplied core.excludesFile so no hashed file changes.",
"retrievalInstruction": "a Graphify knowledge graph is prebuilt in graphify-out/ and the graphify CLI is on PATH. Before any grep, find, cat or file read you must run `graphify query \"<question>\"` and `graphify explain \"<symbol>\"` (and `graphify path \"<A>\" \"<B>\"` for relationships) for the concepts in this task, then read only the file locations they return, confined to the task selection policy.",
"package": "graphifyy",
"packageVersion": "0.9.65",
"build": "graphify update <workspace>",
"instructionSource": "graphify/always_on/claude-md.md",
"instructionSha256": "5a599e9f148ebbf0a8786c4508c3954daf95d0d74b88223d890dbf37aad23ec6",
"graphOutExcludedFromGit": true
},
"context": {
"description": "Built-in tools plus the z1p-repository stdio MCP server (repository_status, repository_refresh, repository_explore, repository_coverage, repository_search, repository_packet) bound to the arm workspace, with the task selection policy in .z1p-navigation.json. Search and packet responses render as compact text; repository_explore answers one symbol per call. The system prompt appendix is the project instruction text from docs/GETTING-STARTED.md step 3 verbatim (context-instructions.txt). repository_coverage is a deterministic pre-submit check that lists explored files a draft answer leaves uncited.",
"package": "@forgesworn/context-tools",
"packageVersion": "source build at the recorded commit (repository_explore, repository_coverage with exact-quote check, compact rendering, pathPrefix); clean working tree, identified by dist digests below",
"server": "encrypted-context.mjs navigate <workspace>",
"instructionSource": "docs/GETTING-STARTED.md step 3 (context-instructions.txt in this directory)",
"retrievalInstruction": "the z1p-repository MCP tools are bound to this workspace. Before any grep, find, cat or file read you must call repository_status (and repository_refresh if it is unavailable, stale or unknown), then call repository_explore for each identifier in this task, use repository_search with pathPrefix only for literals, and obtain the source and tests you cite with repository_packet using the current expectedGeneration, reading files directly only for evidence those tools cannot supply, confined to the task selection policy. Before writing answer.json, call repository_coverage with your draft answer and the task's identifiers, then address each missing file or state why it does not bear on the task.",
"instructionSha256": "cb7f696ea84c42f5923cfc0470a5416b1b6053f4df29bb162e80bb8795b9f934",
"build": {
"headCommit": "7d436fc1c17fca78ac39bf74e74b9a87f33ed1a1",
"workingTreeClean": true,
"dist": {
"repository-coverage.js": "3a112ab4f27478836a9a5c785a4875d40922d4e4b6ce1f7e1bc499470a72df75",
"repository-explore.js": "54269956eee2297c8318c5fd430c2bdd0417ac16298027b52ce8840fbb9e3249",
"repository-navigation.js": "79459fa72f754522fdd4b2858c82fde73298d1480bb6af1bec37b341b69daafb",
"repository-navigation-mcp.js": "ad57575b2b1cac2c75a94491f0b16d40467564dd22c5a7ebf1ea8b2ffbec442c"
}
}
}
},
"armOrders": [
[
"plain",
"graphify",
"context"
],
[
"graphify",
"context",
"plain"
],
[
"context",
"plain",
"graphify"
]
],
"armOrderRule": "Task i uses armOrders[i mod 3]. Arms of one task run sequentially in that order; each arm has its own prepared workspace, fresh headless session and, for the context arm, its own server process. No answer, diff, index or transcript is shared between arms.",
"executor": {
"client": "claude-code",
"clientVersion": "2.1.280",
"model": "deepseek-v4.1-flash:cloud",
"effort": "medium",
"maxTurns": 80,
"maxBudgetUsd": null,
"settingSources": "project",
"settingsOverride": {
"enabledPlugins": {
"agents-md@builtin": false
}
},
"settingsOverrideReason": "A probe showed the built-in agents-md plugin injects the frozen tree's AGENTS.md into every arm; the Context revision's AGENTS.md instructs use of z1p-repository tools, so it is disabled for all arms. A second probe confirmed no project instruction text remains in context.",
"disallowedTools": [
"WebFetch",
"WebSearch",
"Agent",
"ToolSearch",
"Workflow",
"EnterWorktree",
"ExitWorktree",
"RemoteTrigger",
"CronCreate",
"CronDelete",
"CronList",
"ScheduleWakeup",
"SendMessage",
"PushNotification",
"Monitor",
"DesignSync",
"ListAgents",
"TaskStop",
"NotebookEdit"
],
"builtInToolsObserved": "This Claude Code build exposes Bash, Read, Edit and Write for source work; Grep and Glob are not separate tools, so plain-arm search runs through Bash.",
"permissions": "dangerously-skip-permissions inside a disposable frozen workspace",
"skills": "disabled",
"mcp": "strict; only the arm's --mcp-config",
"sessionPersistence": false,
"promptDelivery": "identical task prompt on stdin for every arm; the only difference is the arm's system prompt appendix, PATH and MCP config",
"modelRoute": "Claude Code 2.1.280 with ANTHROPIC_BASE_URL pointing at the local Ollama 0.34.2 daemon (Anthropic-compatible /v1/messages), which proxies the cloud tag to ollama.com; verified by a smoke test with Bash and MCP tool calls",
"env": {
"ANTHROPIC_BASE_URL": "http://127.0.0.1:11435",
"ANTHROPIC_API_KEY": "ollama",
"ANTHROPIC_AUTH_TOKEN": "ollama"
},
"costNote": "The client reports a list-price estimate for an unrecognised model; it is meaningless here and no budget cap is applied. Provider usage counters (input, cache read, output) are recorded; Ollama Cloud credit consumption is read from the account, not from receipts.",
"effortNote": "--effort medium is accepted by the client; whether the route honours it is unknown. The smoke test reported zero thinking tokens."
},
"reviewer": {
"client": "claude-code",
"model": "claude-sonnet-5",
"effort": "high",
"maxTurns": 3,
"disallowedTools": [
"Bash",
"Read",
"Edit",
"Write",
"Grep",
"Glob",
"TodoWrite"
],
"retry": "one further attempt if the structured verdict does not parse; both attempts are recorded and their usage and time are summed",
"tools": "none (Bash, Read, Edit, Write, Grep and Glob disallowed as well); receives the answer or diff, the private rubric, the deterministic checker result and bounded excerpts of the cited frozen source",
"blind": "does not see the arm name or any other arm's output"
},
"measures": {
"accepted": "code tasks: deterministic checker and scope check passed (no model reviewer); structured tasks: checker passed and the blind reviewer accepted against rubric version 2 with no material issue",
"executorToolCalls": "count of tool_use blocks by tool name from the stream",
"executorInputTokens": "input_tokens + cache_creation_input_tokens + cache_read_input_tokens from the result usage; uncached = input_tokens + cache_creation_input_tokens",
"executorOutputTokens": "output_tokens from the result usage",
"costEstimateUsd": "Claude Code total_cost_usd, a list-price estimate, not subscription billing",
"executorSeconds": "wall time of the headless session",
"armSeconds": "graphify build (arm-specific setup) + executor + checker + reviewer wall time",
"reviewerTokens": "reviewer session usage",
"setupSeconds": "dependency installation, shared by all arms and excluded"
},
"decisionRule": "None: a smoke test. It passes if every arm completes, every code cell records a scope result and no reviewer, and every structured cell has a parsed verdict whose prompt contains the version 2 review rules.",
"knownAsymmetries": [
"Graphify indexes the whole frozen workspace; Context indexes only the task selection policy. Graphify is not disadvantaged by this.",
"Graphify's arm-specific build time is counted in armSeconds; Context's refresh happens inside the session and is counted as a tool call.",
"Arm instructions are delivered by system prompt appendix rather than a CLAUDE.md file so the frozen tree stays identical across arms."
],
"derivedFrom": "three-way-deepseek-v4-pro-20260922-s5",
"codeAcceptance": "checker-and-scope",
"rubricDir": "../rubric-v2-20260923",
"changesFromS5": [
"Executor deepseek-v4.1-flash:cloud (same route and effort).",
"Two tasks, one repetition.",
"Code acceptance by checker plus scope check; structured rubrics version 2.",
"Context arm at the current build with repository_coverage; instruction text is docs/GETTING-STARTED.md step 3 as of the recorded commit; retrieval instruction as v3."
]
}
3 changes: 2 additions & 1 deletion packages/context-tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ over an unsigned in-memory index of that root: `repository_status`,
references with enclosing declarations, importing files and tests),
`repository_coverage` (which explored files a draft answer leaves uncited,
and whether each cited token is quoted exactly),
`repository_search` (exact identifier lines grouped by file, with `pathPrefix`)
`repository_search` (exact identifier or literal lines grouped by file, with
`pathPrefix`)
and `repository_packet` (verbatim ranges or complete TypeScript/JavaScript
blocks). Search, explore and packet responses are compact text by default and
JSON with `format: "json"`. See the
Expand Down
Loading
Loading