Summary
Add a reproducible, offline benchmark for code-context retrieval that reports both retrieval cost and context delivered to the agent. The benchmark should make Perseus's efficiency claims auditable without conflating local retrieval work with provider-billed prompt usage.
Motivation
The GrapeRoot thread made several distinct claims that are easy to blur together: graph lookup can avoid model exploration, context can be pre-injected, and users can estimate money/token savings from reduced exploration. Those are useful hypotheses, but the measurements need separate arms and explicit accounting.
Perseus already has the right honesty precedent in its token A/B harness and context decision records. This issue would extend that discipline to code-graph retrieval.
Proposed benchmark arms
At minimum, on the same fixture repositories, prompts, model/tool contract, and cache policy:
- Baseline agentic exploration: the agent receives the same starting instructions and uses ordinary file listing/search/read operations.
- Lexical/structured Perseus retrieval: existing context routing without a code graph.
- Code-graph retrieval: graph-assisted candidate selection/injection.
- Optional graph plus targeted follow-up reads: tests whether pre-injection and agent exploration compose.
Metrics to report separately
- index build and incremental update latency
- candidate retrieval latency
- render/injection latency
- prompt/context tokens delivered to the model
- tool-schema and tool-call tokens, if applicable
- output tokens and total provider-reported usage when a live provider run is enabled
- task success/answer coverage and evidence attribution
- cold vs warm cache
- corpus size and language mix
- model/provider/version and exact configuration
Do not present local retrieval work as "zero tokens" without clarifying that this means no model-context tokens, not zero CPU, memory, disk, or latency cost.
Acceptance criteria
- Offline deterministic smoke benchmark runs without network access or credentials.
- Live-provider mode is opt-in and records authoritative provider usage separately from estimates.
- Both baseline and product arms count the same request components; no hidden fixed penalty or omitted user/tool-schema tokens.
- Results emit a versioned JSON artifact with configuration, corpus fingerprint, method labels, and SHA-256 content hash.
- A quality gate prevents a lower-token result from being called a win when task coverage/evidence attribution falls below the declared tolerance.
- Public claims can link to the committed artifact and methodology.
Related work
benchmark/tokenab/ and its measured A/B report.docs/context-decision-record.md explicitly distinguishes rendered accounting from provider-billed savings.- Existing LongMemEval and context-position/provenance benchmark work.
Summary
Add a reproducible, offline benchmark for code-context retrieval that reports both retrieval cost and context delivered to the agent. The benchmark should make Perseus's efficiency claims auditable without conflating local retrieval work with provider-billed prompt usage.
Motivation
The GrapeRoot thread made several distinct claims that are easy to blur together: graph lookup can avoid model exploration, context can be pre-injected, and users can estimate money/token savings from reduced exploration. Those are useful hypotheses, but the measurements need separate arms and explicit accounting.
Perseus already has the right honesty precedent in its token A/B harness and context decision records. This issue would extend that discipline to code-graph retrieval.
Proposed benchmark arms
At minimum, on the same fixture repositories, prompts, model/tool contract, and cache policy:
Metrics to report separately
Do not present local retrieval work as "zero tokens" without clarifying that this means no model-context tokens, not zero CPU, memory, disk, or latency cost.
Acceptance criteria
Related work
benchmark/tokenab/and its measured A/B report.docs/context-decision-record.mdexplicitly distinguishes rendered accounting from provider-billed savings.