Test sources are labelled in the answer and rerank prompts - #186
Merged
Merged
Conversation
Fusion demotes a test hit and the walk orders it last, but nothing after that says what it is: the answer prompt and the reranker header print repo, path, line and symbol, so the model reads a test fake as the client and cites the test where the mechanism sits in the file beside it. Both headers now carry (test) on retrieve.IsTestPath, and each prompt gets one rule: a test shows what the code is expected to do, a claim about the mechanism cites the code, a claim standing only on a test says so, and a question about testing is answered from tests. Tests stay indexed and citable; the exclusion was considered and rejected on a thread whose question asked how profiles are tested.
Rubric 27/30 twice on master, 28/30 twice with the label; test citations 5 to 3, the remaining ones named as tests in the sentence. TestFlowGathered 28/30 on both sides. AGENTS.md records that tests are labelled, never excluded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A production thread cited registry_test.go for a rejection that sits in registry.go beside it. Fusion demotes test hits and the walk orders them last, but nothing after that says what a source is, so the answer model read the test as the mechanism and the reranker can promote a test whose excerpt describes the code better than the code does.
Both prompt headers now carry (test) on retrieve.IsTestPath, with one rule each: a test shows what the code is expected to do; a claim about the mechanism cites the code, a test beside it at most as a second marker; a claim standing only on a test says so; a question about testing is answered from tests.
Tests stay indexed and citable. Excluding them was considered and rejected: the same thread's third turn asked how profiles are tested, and its answer had no source but the tests.
Flow corpus, TestEvalMeasureAnswers twice per arm against same-day master: rubric 27/30 and 27/30 on master, 28/30 and 28/30 with the label; test citations 5 to 3, the remaining ones named as tests in the sentence. TestFlowGathered 28/30 on both sides. Tables in docs/measurements/2026-09-17-test-label.md.