Measurement instruments: store probes, judge-only re-grade, a render gate, and the guards that came from getting it wrong - #209
Merged
Conversation
… the gate that proves it ran
The Bitemporal ON ablation turned write-side supersession on and moved nothing in aggregate
(46/60 against 47/60), but its misses shifted from "gold value ABSENT" (11 to 9) toward "HAD the
value and chose the wrong side" (2 to 5). The reading that followed: currency now exists in the
store and is not reaching the model legibly.
The renderer for exactly that was already built. SupersessionProjectionFeature emits
"(since D; previously Y)" and its docstring names the loss verbatim -- "a superseded fact is not
'shown as old', it is absent" -- but it is gated on MemoryProjectionOptions.ResolveSupersessions,
which defaults false, and no harness had ever set it. So the renderer was dark in every run this
project has scored, including the ablation built to measure supersession.
That is the SECOND reachable-but-never-fed lever inside one intervention, six days after the first.
The doctrine sentence ("a path being reachable is not the same as its inputs existing") was right
and still did not catch this, because it was applied to the option under investigation rather than
to the config surface as a whole.
What lands here:
- --resolve-supersessions, threaded to MemoryOptions.Projection. Assigned CONDITIONALLY: the
shared MemoryProjectionOptions.Default instance is reference-compared to tell "unset" from "set
to the defaults", so an unconditional `with`-copy would change identity for every sealed
measurement while leaving all property values equal. A test asserts the off state keeps the
shared instance, and it is the only one of the three that fails when the conditional is removed --
which is the point, since property assertions cannot see an identity change.
- The arm names it. Token "render", so write-only ("supersede"), render-only ("render") and
both-on ("supersede-render") are distinguishable on disk. Appended LAST in the positional record
rather than beside its sibling, because inserting mid-record silently re-slots every positional
call site -- the mistake that once turned a factwt arm into a supersede arm. One positional test
call site converted to named arguments while here.
- A render-state probe, because a config echo would have detected neither of the two dark levers:
in both cases the flag that was set was not the flag that mattered. It reads the OUTPUT instead --
what projection annotated, and whether that text is present in the prompt string the model was
handed. Two counts, deliberately: annotated-but-absent is a render-surface bug that a single
counter would report as success. renderConfirmed is the only affirmative form and requires a
retained sample, so the gate is eyeballable rather than arithmetic. A null block means the probe
never ran, which is a failure and never a pass.
The probe is red-first tested against the counter-check "would this gate still pass if the thing it
guards were deleted?", and the drift guard caught the half-wired flag a fifth time before this
compiled clean. Solution builds 0-warn; LongMemEval suite 719/719.
Note for whoever builds next: global.json pins SDK 10.0.102 and that SDK has been uninstalled on
this machine, leaving an empty husk directory, so nothing builds until it is reinstalled or the pin
is revisited. Verified here under a strictly local, reverted override; the pin is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…y machine here global.json pinned SDK 10.0.102 exactly, with no rollForward. That SDK has been uninstalled from the development machine -- its directory survives with a single leftover Roslyn folder where 10.0.400 has 192 entries -- so `dotnet` correctly refused to resolve it and nothing built at all. Pinned to 10.0.400 with rollForward: latestFeature, so a future SDK update cannot break the build the same way again. CI follows automatically: every workflow resolves its SDK through actions/setup-dotnet with global-json-file: global.json. The target frameworks are untouched -- the libraries still multi-target net10.0;net9.0;net8.0. This changes only which SDK compiles them. Solution builds 0-warn and the LongMemEval suite is 719/719 under the new pin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
A 2-question pre-flight with both levers on reported "projection annotated NOTHING", which is the gate refusing to confirm -- correctly, because that message has two very different causes: the renderer never ran, or the corpus had nothing to supersede in those two questions. The whole point of the gate is that these are indistinguishable from a counter, so the ambiguity was resolved rather than assumed away. Two links were untested, and both are deterministic, so neither needed to be bought: - SupersessionProjectionFeature takes IFactRepository? and reports itself DISABLED when it is absent, by design, so a container without repositories stays resolvable. The consequence is that the flag can be true, the option can reach MemoryOptions, and the feature can still never run -- reporting exactly what an unsuperseded corpus reports. Now asserted against the harness's own DI graph. It resolves. This test needs `await using`, unlike every sibling: resolving the repository constructs Neo4jDriverFactory, which implements IAsyncDisposable ONLY, and a synchronous Dispose on a provider holding one throws. The first version of this test failed for that reason and not for the reason under test -- worth the note, because a test that fails at teardown looks exactly like a test that failed at its assertion. - The engine renders through ProjectionRenderer, but this harness builds its own answer prompt, so "projection produced a note" and "the model saw a note" are separate claims that fail separately. BuildAnswerPrompt is now asserted to carry a SupersessionNote through to the prompt text, with the probe agreeing on that real prompt rather than on a fixture string. What remains unverified is corpus-side only -- whether :SUPERSEDED_BY edges exist for a given question -- and that is what the run reports, on a corpus where the ON ablation already proved supersession fires (facts per question 8.25 -> 7.92, tme-bit-037 flipped). So the pre-flight zero is now explained rather than excused: two questions, nothing superseded in them. LongMemEval suite 721/721. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…AgentEval's own runner AgentEval disclosed that the Bitemporal vertical shipped with NO judge body -- it fell through to StandardBody, and the shared preamble's "premature" definition made the judge grade the gold answer's justification rather than its value, returning Premature where the label says Wrong. Measured bitemporal judge agreement 0.750/0.792/0.792. Every bitemporal number on record was produced by that judge, including both baselines a pre-registered arm is anchored to. The answers were NOT affected, and that is verified rather than assumed: the answer-side adapter never receives a judge client, and no path feeds judge output back into generation. So the repair is to re-grade the stored answers, not to re-run ~5h of pipeline per arm. Design decisions worth keeping: - REPLAY THROUGH THE REAL RUNNER, do not call a judge directly. A hand-rolled grading loop would be a second implementation of scoring, free to drift from the one that produces every number we cite -- and the drift would look like a finding. TypedMemEvalReplayAdapter satisfies IEvaluableAgent by handing back the stored answer, so question selection, judge body, typed-outcome derivation and attribution are all AgentEval's code, unchanged, and whatever the fixed package changes is inherited rather than reimplemented. - SELF-VALIDATING, which is why it can be trusted before the fixed judge exists. Re-grading an artifact under the SAME judge that produced it must reproduce its verdicts, and the tool prints that agreement rate. An instrument that cannot reproduce a known result has no business re-anchoring a baseline. (Not yet exercised: it shares an Azure deployment with a run that is 2h30m in, and a rate-limit collision there would cost far more than waiting.) - SAMPLING COMES FROM THE ARTIFACT'S OWN SIDECAR, never from the command line, and a missing sidecar is refused rather than defaulted. A re-grade that re-specified the seed by hand could select a different question set and the difference would present as a judge effect -- the one conclusion this tool exists to support. - AN UNMATCHED QUESTION ABORTS. It would be replayed as empty text, and an empty answer grades as WRONG rather than as MISSING: indistinguishable in the final score from a genuine model failure, and it would quietly LOWER the baseline being established. Counted, surfaced, and fatal. - Agreement is compared by QuestionId, not by position, so a change in selection cannot silently pair different questions. Re-gradability confirmed for all three bitemporal arms (60/60 answers with gold and question ids). Pre-unblinding record: the prereg was amended at 2026-08-28T14:56:55Z, while the render arm was still executing and before any score from it existed. That file lives in gitignored strategy/, so its sha256 is pinned here instead: b2be4a2aca07247dc5b6ed3f382714d1a1021ced8deae3800a2c4f9c1c82c6ce LongMemEval suite 725/725. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
… in this corpus
The self-validation refused to run and was right to. Replaying the stored ON artifact reported 48
distinct answers for 60 rows: twelve bitemporal questions share their text with another question --
and the twin has a DIFFERENT gold answer.
"According to the record, which department was Colm Whitaker at in February?"
tme-bit-007 belief-at-instant gold: Lowick
tme-bit-037 correction-depth gold: Marchmont
That is a property of the corpus, not a defect in it. Every question is asked against its own
injected history, so identical words over different memory legitimately have different right
answers. But it means the text is not an identity, and the replay had keyed on it.
The consequence had this shipped: 12 of 60 questions would have been handed an answer written for a
different question, and GRADED. Not a blank row -- a wrong one, yielding a complete and plausible
number that would then have become the baseline a pre-registered arm is adjudicated against. The
duplicate guard was written on the general principle that a re-grade must not be able to understate
a baseline; it turned out to be load-bearing on its first real input.
Now keyed by POSITION, since InvokeAsync receives only the prompt and ordering is the only identity
available. The assumption is asserted rather than trusted: the text at each position is checked
against the stored row, so a change in the runner's selection or ordering aborts instead of silently
pairing the wrong answers. Two failure modes are now distinct -- OrderingMismatches (paired wrong)
and UnmatchedQuestions (ran past the end of the artifact) -- and both are fatal to the caller.
INSTRUMENT VALIDATED. Re-grading the ON artifact under the same judge that produced it reproduces
it exactly: 60/60 verdict agreement, 46/60 correct, OverallAccuracy 76.6667 in both. The judge is
deterministic on this corpus, which is a stronger result than the "high agreement, not bit-identity"
this was designed to tolerate. AgentEval 0.28.0-beta+345d5b6.
An instrument that cannot reproduce a known result has no business re-anchoring a baseline. This
one can, so it may.
Also proven here: the detached launch. The rendering arm died at 3h05m because it ran as a harness
background task and the task was stopped, taking the run and its container with it and leaving no
artifact. Five previous kills in this project were harmless because those tasks were WAITERS around
runs launched elsewhere; this one WAS the run. Both re-grade passes were launched via Start-Process
instead, which survives independently -- and that is how the next paid run gets launched.
LongMemEval suite 727/727.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…poral body The Bitemporal vertical shipped with no judge body and fell through to StandardBody, where the shared preamble's "premature" definition made the judge grade the gold answer's justification instead of its value. Measured agreement 0.750/0.792/0.792. 0.29.0-beta is the fix. Authorized after the coordinator's byte-probe cleared the package (tag v0.29.0-beta = 623cb3ba; published nupkg embedded resources hash-identical to the tag blobs) and after an independent check here: the embedded bitemporal corpus is BYTE-IDENTICAL across the two versions (f5b384d7..., 1,534,988 bytes), so re-grading the stored arms isolates the judge and nothing else. That corpus check walked into a trap first, which is worth recording because it is the same shape as the strings-not-installed false negative: loading both assemblies in ONE process made the second LoadFrom throw "assembly with same name is already loaded", so the 0.29 pass silently re-read the 0.28 assembly and reported the hashes identical. The right answer, from a measurement incapable of producing any other. Redone one process per assembly, each printing the version it actually loaded before hashing. API-compatible: solution builds 0-warn and the LongMemEval suite is 727/727 under the new pin, with no source changes. Verified before requesting the gate rather than after. One thing to carry forward, raised with AgentEval as SEND-11 Q-1 and NOT resolved here: their own oracle-answerability probe now reports tme-bit-028 as not answerable from gold (v1 True -> False; roll-ups 60->59, pair-flip 30->29, belief-at-instant 36->35). That question is scored in all three bitemporal arms. If it is genuinely unanswerable the honest denominator is 59, and the re-graded aggregates stay UNREAD until they certify it -- changing a denominator after seeing the numbers looks like tuning, whatever the intent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
… null verdict as unscored The ON re-grade crashed with an unhandled exception and lost its artifact. Two defects, and the ordering one is the serious half. 1. A NULL VERDICT IS NOT A DISAGREEMENT. The new judge returned no verdict for one question (tme-bit-009, JudgeStatus 3, SafeFailureCode invalid_finish_reason -- a truncated or filtered completion, i.e. a failed call rather than an opinion). Agreement() called GetBoolean() on it and threw. Folding a null into "agreed" or "disagreed" would have moved the agreement rate for a reason unrelated to the judge's view, so it is counted on its own as unscoredQuestions: a re-grade carrying unscored questions is a VALIDITY question before it is a score. 2. THE DIAGNOSTIC RAN BEFORE THE PERSIST. Agreement() executed before the artifact reached disk, so a statistic destroyed a full judge pass that had already been paid for. Persist now runs first and the agreement figures are amended into the sidecar afterwards. A cheap derived metric must never stand between an expensive result and the disk -- that is the general rule, and this is the second avoidable loss of the day after the harness-background-task kill. Both arms are now re-graded under 0.29.0-beta with 60/60 questions scored and zero unscored. The transient cleared on the single re-run permitted by a rule written down BEFORE the re-run started (prereg 7f, 08:43:16Z): re-run once; a second failure on the same question is not transient and gets escalated rather than re-run again; whatever the second pass returns is the record, including if it fails elsewhere. Re-running a grader until no call fails is selection, and repeated it selects on the score as surely as on the failure. Aggregates remain UNREAD pending AgentEval's certification of the tme-bit-028 denominator (SEND-11 Q-1). tme-bit-009 is unrelated to that question and the two are not to be conflated: one is corpus answerability, the other a failed HTTP call. LongMemEval suite 728/728. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
… this corpus at all The 30.9d render arm returned 0 supersession notes in 60 of 60 prompts. An absolute zero is a disconnected wire, not a weak effect, so the chain was walked link by link on live Neo4j. EVERY DOWNSTREAM LINK WORKS (SupersessionRenderingPathIntegrationTests, staged so that whichever stage fails names the break): 1. Supersede writes (loser)-[:SUPERSEDED_BY]->(winner) and stamps invalidated_at. 2. GetSupersessionPredecessorsAsync finds the predecessor. 3. LIVE recall renders "(since D; previously Marchmont)". 4. AS-OF recall renders it too. So the as-of hypothesis is dead and the memory layer is not broken. Both recall paths render correctly, and neither had any test coverage for this before today. THE LINK THAT WALK BYPASSED is the one extraction actually goes through. It calls SupersedeAsync directly; extraction first asks WriteTimeFactResolution.CanSupersede, which is MemoryRelationCardinality.IsSingleValued(predicate) -- decided by a vocabulary of EXACTLY SIX relations with the `single` cardinality (belongs to, costs, expires, lives in, weighs, works at), with no aliases declared and the documented behaviour "false for anything unrecognised". The bitemporal corpus is about departments, sites, branches, regions and offices, and the eval profile leaves usePredicateVocabulary OFF, so the extractor writes free-form predicates. The guard test added here confirms `was at`, `assigned to`, `department`, `branch`, `region`, `office`, `site` and `located in` all return false. A predicate outside those six makes SupersedeReplacedFacts=true indistinguishable from false: no edge is ever written, so there is nothing to render. CONSEQUENCE, AND IT REACHES BACKWARDS: ledger row 51 claimed the ON ablation was "a real null, not a silent no-write" on three behavioural proofs. The facts-per-question drop it leaned on (8.25 -> 7.92) is now bracketed by the render arm's 7.65 under the same lever -- three runs, monotonic, no lever change between the last two, which is extraction variance. The two question flips are single rows in a pipeline whose own churn analysis measured 6 gained / 7 lost between identical arms. So SupersedeReplacedFacts has never been measured on this corpus, and the compositional result that survived re-grading (absent 3->10, chose-wrong 9->5) is the difference between two off-state arms. The mechanism statement built on it is withdrawn. That is the same error this arc already corrected once, one layer deeper: an off-state measurement wearing an on-state label. The G2 gate is the only reason this surfaced as a void instead of a published finding -- 45/60 would have been read as "rendering does not help", from two arms that were both off. Unit suite +14, integration suite +3 (both previously uncovered paths). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
The store-state gate used to argue from facts-placed-per-question falling (8.25 -> 7.92) that
supersession had fired. Three arms later the same lever produced 7.65 with no lever change in
between, so that signal was extraction variance -- and the gate had been certifying an off-state arm
as "verified ON". That is precisely the failure the gate existed to prevent, occurring inside the
gate itself.
Replaced with a COUNT, read before container teardown and written into every run's sidecar:
:SUPERSEDED_BY edges, invalidated facts, and the extracted-predicate histogram. A count cannot be
mistaken for noise: zero means the mechanism did not run, and the histogram says why in the same
breath, since write-time supersession is refused for any predicate outside the six the relation
vocabulary declares single-valued.
Written as three separate statements rather than one joined query, deliberately: a single MATCH over
facts AND edges returns NO ROW when the graph holds zero edges, and zero is the single most important
thing this can report. It must arrive as a number, never as an absent row.
FIRST RUN, and it settles the question:
supersession store — 0 :SUPERSEDED_BY edge(s), 0/229 facts invalidated.
Zero of 229. And the histogram carries a second, unanticipated finding: the dominant predicates are
SPEECH ACTS -- `came up with` x46, `said` x43, `was consistent between` x15, `said as much` x10,
`stated` x8. Extraction on this corpus records who said what, not what is true about whom. The single
locational predicate present, `is at` x7, still misses the vocabulary, which declares `works at`.
So supersession is refused twice over, and the consequence reaches backwards: ledger row 51's ON
ablation never superseded either, so 0.783 / 0.767 and their re-graded successors 48/60 and 45/60 are
all off-state numbers, and the compositional movement between them is the difference between two arms
that were both off.
A store whose triples are mostly `X said Y` is also a weak substrate for any fact-grain retrieval
claim on this corpus -- which bears on the unit-of-retrieval statement already relayed to AgentEval.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…etimes unattributable
C1 showed extraction writing speech-act predicates across three unrelated corpora. The histogram
could not distinguish an amount stored as the object of `said as much about` from an amount never
captured -- a grain problem and a capture problem need different fixes, so the reading was registered
before this probe was written, not merely before it ran.
Result on arithmetic (232 facts): 15 amount-bearing objects, 9 other-numeric, 208 non-numeric.
Amounts are PRESENT, so the severer capture finding is ruled out and the extraction track opens on
grain.
The samples carry more than the counts:
payment | has_amount | $40.64
Padstow Green plumbing payment | has_amount | $424.38
Yarrow Quennxby | said | Padstow Green plumbing payment was $424.38 ...
EXTRACTION DOES ASSERT FACTS OF ENTITIES. `has_amount` is exactly the right grain and it exists, so
the C1 headline narrows: the extractor is not incapable of assertional triples, it emits them AND a
large volume of speech-act triples restating the same content. Dilution and duplication, not
inability -- a materially different problem from the one the histogram alone suggested.
Two things outrank the original hypothesis. First, amount-bearing objects are 15 of 232 (6.5%), which
is a sharper mechanism for the fact-weighted-budget null than "utterance triples" was: raising facts
per question 9.0 -> 16.4 adds mostly non-amount facts because only about one fact in fifteen carries
an amount. Second -- flagged, not concluded -- three amount triples carry the bare subject `payment`
while others carry `Padstow Green plumbing payment`. If a generic subject cannot be attributed to a
specific payment, a retrieved amount is unusable for a delta question even when retrieval succeeds,
which would sit downstream of every selection fix attempted so far. That needs its own probe.
Samples print verbatim per bin so the classification is eyeballable rather than trusted -- the same
standard G2 places on the render gate, and the reason the render gate caught what it caught.
Supersession: 0 edges again, now on a fourth store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
`CreateAboutRelationshipAsync` is public API on IFactRepository and IPreferenceRepository, appears in the phase-0 API snapshot, and is covered by four test files -- and no ingestion path in this library calls it. Verified against a prepared store: 26,887 of 26,887 facts carry no :ABOUT edge. It is not, however, unread. NodeDistanceQueries walks [:RELATED_TO|ABOUT*..4], and its own docstring names ABOUT as one of the two edges that carry meaning between an entity and a fact. So the code documents as load-bearing a relationship the pipeline never creates, and any consumer who enables NodeDistanceReranking today gets a re-ranker traversing RELATED_TO alone -- weaker than its own query describes, with nothing reporting the difference. That affects users whether or not anyone ever calls the public verb. Documented rather than changed, deliberately. Removing ABOUT from the traversal is the cheapest fix and it is not free: the writer is PUBLIC, so a consumer who created links manually has them honoured today, and dropping the edge would change their results silently. Wiring a pipeline writer is the larger option and should follow a measured need, not precede one. So this ships the non-breaking step -- the verbs, the option, and the traversal each now say what is actually true -- and leaves both real options open for triage. Also: the three store-shape probes added this week became REQUIRED interface members and broke six unrelated test fakes that have no graph to report on. That is a signal about the contract, not about the fakes: a store-shape reading is something a probe MAY offer. They are now default-implemented to empty, which is the honest answer when a probe has no store, and no fake needs to know they exist. Solution builds 0-warn; 5187 + 728 unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…zero" The three store-shape probes default to empty so a fake without a graph need not implement them. That default is safe only while readers keep two things apart, so the contract now says so: EMPTY MEANS "NO STORE TO READ", NOT "MEASURED ZERO". A default that silently satisfies every caller is how a constant column is born -- a value that could not have come out any other way, read as though it were evidence. That has already cost this project once this week: "12 of 12 pairs with zero entity links" was reported as a finding when every fact in every store had zero, because nothing writes those links. The number was real and carried no information. So the remark points a caller at the render gate's rule as the pattern to copy: a null block is a failure, never a pass, and "the mechanism wrote nothing" is a different claim from "nobody looked". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…subject collision The Cell B/C gate needs our extractor run over corpora that are not embedded resources, and needs a metric that is functional-only without a hand-classified predicate list -- the C-0 lesson, where a general ambiguity rate turned out to be 100% legitimate multi-valued predicates. So the measure is narrow by construction: among subject/predicate groups whose OBJECTS ARE AMOUNTS, the share holding more than one distinct amount. 'This payment's amount' is functional by construction, so restricting to amount-valued groups makes the metric functional-only without anyone deciding which predicates count. Extraction and store reads only -- no judge, no questions, no scoring -- so it is materially cheaper than an arm and cannot emit an accuracy number that might later be read as a result it was never designed to support. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…r print 0% over no denominator The Cell-B sanity gate returned "0 of 0 amount-bearing groups = 0.0%" against AgentEval's calibrated 100.0%. The pipeline was not at fault and neither was the corpus: payment sentences in this corpus start at ENTRY 14, and the registered sampling rule was "the first 12 entries". The window held zero of the thing being measured, so the metric divided by zero and rendered the result as 0.0%. Cost of that: ~2,224 LLM calls and two hours, for a number that could not have come out any other way. It is the constant column again -- the third time this project has been bitten by one, and the first time by an instrument written two commits after documenting the hazard in a probe contract. Two guards, both cheap, both at the point where the failure is detectable: - A PRECONDITION on the window, checked before a single LLM call: count amount-bearing sentences in the selected entries and ABORT if there are none. A sampling rule that cannot contain the phenomenon is a defect in the rule, and it costs nothing to detect. Verified: the old window now aborts in under a second instead of spending an hour. - A ZERO-DENOMINATOR GUARD on the output: an empty denominator prints "NOT MEASURED ... This is not 0%", never a percentage. A share over nothing is not a small share. Also adds --skip-entries so a window can be placed where the phenomenon actually lives, which is what the re-registered sampling rule needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
…e pattern before spending The first cell run spent ~2,224 LLM calls and two hours measuring a window that contained none of the phenomenon. One free read of the corpus would have shown it. So the probe can now do everything the real run does except the paid part: confirm the corpus loads, the window contains amount sentences, the paired cell aligns entry-for-entry on question ids, and the amount pattern actually matches this corpus's own sentences -- printing a real matched sentence rather than asserting the regex works. Verified across three cases: the bad window aborts, and both cells at the ratified window 14-19 pass with 114 amount sentences each and matching question ids. The two sample sentences also show the controlled variable directly -- same job, same amount, same item, different subject. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
… run A 1-entry wiring run showed 1 of 3 amount groups holding 8 of 10 amounts. The group-level rate is therefore unstable at small denominators and UNDERSTATES the collapse it exists to measure -- and it is not the denominator AgentEval calibrated (48/48 per payment). So both are now reported: GROUPS answers how many subjects are overloaded, AMOUNTS answers how many values sit under an overloaded subject. Registering both before the paired run closes off selecting between them afterwards, which is the failure mode a metric change discovered mid-experiment invites. The reason is structural -- visible from the SHAPE of the result, not its value -- but it was noticed after a run, so both ship rather than one being swapped in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
There was a problem hiding this comment.
🟡 Changes recommended
There are a few concrete correctness issues in the new CLI/diagnostic paths (option validation omissions, doc-comment mix-up, and a re-grade agreement null-verdict crash case) that should be fixed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds new measurement and validity instrumentation around TypedMemEval runs (store probes, render-state gate) and introduces a judge-only re-grade path that replays stored answers through AgentEval’s runner, plus a lightweight external-corpus “cell probe” for extraction/store diagnostics. This strengthens the repo’s evaluation tooling by making “feature on/off” verifiable via counted artifacts and by preventing expensive runs from being invalidated by cheap post-hoc diagnostics.
Changes:
- Add
--regrade(judge-only replay of stored artifacts) and--cell-probe(external corpus ingestion/measurement) CLI flows. - Wire
--resolve-supersessionsthrough the LongMemEval harness and persist/print render + store probes in provenance sidecars. - Add unit/integration guards to pin supersession gating behavior and to assert the full supersession rendering path (live + as-of).
File summaries
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/TypedMemEvalReplayAdapter.cs | New replay adapter for returning stored answers to the runner for re-grading |
| tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs | New --regrade program to load a stored report + provenance and re-run grading only |
| tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs | Adds --resolve-supersessions wiring plus render/store probe capture and sidecar output |
| tools/AgentMemory.LongMemEval/TypedMemEvalArm.cs | Extends arm identity/tokening to include ResolveSupersessions |
| tools/AgentMemory.LongMemEval/Program.cs | Dispatches new sub-verbs and updates option validation list |
| tools/AgentMemory.LongMemEval/LongMemEvalSupersessionRenderProbe.cs | New probe + summary for verifying supersession notes reached the actual prompt |
| tools/AgentMemory.LongMemEval/LongMemEvalMemoryProfile.cs | Wires resolveSupersessions into MemoryOptions without changing off-state identity |
| tools/AgentMemory.LongMemEval/LongMemEvalGraphProbe.cs | Adds graph/store probe queries and new diagnostic record types |
| tools/AgentMemory.LongMemEval/CellProbeProgram.cs | New external-corpus ingestion + measurement “cell probe” with dry-run preconditions |
| tools/AgentMemory.LongMemEval/AgentMemoryLongMemEvalAdapter.cs | Observes prompt-level supersession render evidence via the new probe |
| tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj | Bumps AgentEval package version |
| tests/AgentMemory.Tests.Unit/Services/SupersessionPredicateGateTests.cs | Pins supersession gate reach (single-valued predicate vocabulary) |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalReplayAdapterTests.cs | Ensures replay refuses mismatches/unmatched questions to avoid baseline understatement |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalCommandLineTests.cs | Ensures --resolve-supersessions is carried onto the options record |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalArmProvenanceTests.cs | Ensures supersession write/render levers are distinguishable in tokens/sidecars |
| tests/AgentMemory.Tests.Unit.LongMemEval/SupersessionRenderProbeTests.cs | Validates the render-state gate fails closed and checks harness prompt builder |
| tests/AgentMemory.Tests.Unit.LongMemEval/GraphRagWiringTests.cs | Asserts resolve-supersessions wiring, off-state identity, and optional repo registration |
| tests/AgentMemory.Tests.Integration/Services/SupersessionRenderingPathIntegrationTests.cs | End-to-end Neo4j test for supersession note rendering (live and as-of recall) |
| src/AgentMemory.Neo4j/Queries/NodeDistanceQueries.cs | Documents ABOUT traversal reality vs pipeline behavior in node-distance reranking query |
| src/AgentMemory.Abstractions/Repositories/IPreferenceRepository.cs | Documents that ABOUT links are public API but not written by ingestion pipeline |
| src/AgentMemory.Abstractions/Repositories/IFactRepository.cs | Documents that ABOUT links are public API but not written by ingestion pipeline |
| src/AgentMemory.Abstractions/Options/MemoryOptions.cs | Documents that NodeDistanceReranking’s ABOUT traversal won’t activate on pipeline-built stores |
| global.json | Updates pinned SDK version and roll-forward behavior |
Review details
Suppressed comments (2)
tools/AgentMemory.LongMemEval/Program.cs:503
- The top-level argument validator in Program.Parse() uses this KnownOptions list;
--dry-runand--pair-withare supported by CellProbeProgram but missing here, soLongMemEvalArgumentValidator.Validatewill reject them as unknown options.
private static readonly string[] KnownOptions =
[
"--reference-arm", "--surface-probe", "--predicate-distribution", "--prepared-pair",
"--procedural-benefit", "--typedmemeval", "--attempts", "--regrade", "--cell-probe", "--max-entries", "--skip-entries",
"--oracle-decomposition", "--max-sub-questions", "--question-ids", "--no-content",
"--oracle-precision", "--distractor-sessions", "--gold-fraction", "--oracle-representation",
"--capture-headroom", "--artifacts",
"--probe-answer-determinism", "--repeats", "--probe-questions", "--include-text",
"--answer-seed", "--typed-report", "--reports", "--arm",
"--procedure-retrieval", "--min-scores", "--task", "--query-formulation", "--time-grounded-oracle", "--upstream-oracle",
"--list-prepared-corpora",
"--extraction-compare", "--help",
"--chronological-context", "--dataset", "--evidence-detail",
"--exclude-synthetic-messages", "--judge-retries", "--max-items-per-session",
"--max-relevant", "--memory-mode", "--oracle", "--output", "--questions", "--seed",
"--units", "--turns", "--repeat", "--extraction-seed", "--memory-types",
tools/AgentMemory.LongMemEval/TypedMemEvalReplayAdapter.cs:81
- This returns the mutable backing List instance; callers can downcast and mutate it, which can corrupt the adapter's accounting. Return a read-only wrapper or a snapshot instead.
public IReadOnlyList<string> OrderingMismatches => _orderingMismatches;
- Files reviewed: 23/23 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
All three are real; the second is the one that mattered.
1. AN ORPHANED DOC COMMENT. Inserting PrintSupersessionStore left PrintRenderState's <summary> and
<remarks> stranded above it, so one method carried two summaries and the other carried none.
Each method now documents itself.
2. Agreement() GUARDED THE RE-GRADED SIDE AND NOT THE STORED SIDE. `row.GetProperty("Correct")
.GetBoolean()` throws on a stored artifact whose verdict is null -- and we have already seen that
happen (a judge call returning invalid_finish_reason). It is the SAME defect fixed one commit
earlier on the other half of the same method, and it would have thrown AFTER the judge pass was
paid for: precisely the cheap-thing-destroys-expensive-result failure that the
persist-before-diagnostics ordering exists to prevent. A stored row with no verdict is simply not
comparable, and is now skipped like its re-graded counterpart.
Worth stating plainly: I wrote the guard, understood why it was needed, and applied it to one of
two symmetric branches. A review caught the half I missed.
3. THE ACCOUNTING LISTS WERE HANDED OUT MUTABLE. UnmatchedQuestions and OrderingMismatches returned
their backing List, so a caller could downcast and clear the counts the re-grade gate aborts on --
a gate whose only job is to stop a baseline being silently understated. Both now return snapshots,
with a regression test that mutates through the property and asserts the count survives.
Release build 0-warn; LongMemEval suite 729/729.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
Owner
Author
|
@copilot review |
Co-authored-by: joslat <9831011+joslat@users.noreply.github.com>
Contributor
Owner
Author
|
Second review round addressed. Copilot's review summary named "option validation omissions" without attaching an inline comment, so it would have been easy to miss. It was real, and it was the most consequential finding of the four:
Both verbs now validate first, and the cell probe presents the failure as a refusal rather than an unhandled exception. Verified both directions: the typo is refused with a suggestion, the correct flag still completes free. Fixed in the commit above. Release 0-warn, 5187 + 729 + 54 unit tests green. |
…g hazard
Copilot's review summary named "option validation omissions" without an inline comment. It was real,
and it was the most consequential of the four things that review found.
--dry-run and --pair-with were missing from the central KnownOptions list, and neither new verb
called the argument validator at all. The consequence is specific rather than cosmetic: a mistyped
`--dry-runn` was SILENTLY IGNORED, the dry-run branch never fired, and an operator who asked for the
free check got a paid extraction instead. Unknown-option tolerance is a spending hazard on a verb
whose entire purpose is to let you look before you spend -- and this project has already burned
~2,224 calls on a run that should not have been paid for.
Both verbs now validate first. The validator's message already names the typo and suggests the
correction ("Unknown option '--dry-runn'. Did you mean --dry-run?"), so the cell probe now catches
it and prints it as a refusal rather than letting it surface as an unhandled exception -- a stack
trace on top of a good message only obscures that the run was stopped deliberately.
Verified both ways: the typo is refused, and the correct flag still completes the dry run free.
Release build 0-warn; 5187 + 729 + 54 unit tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
Owner
Author
|
Rebased onto var mismatch = entries.Length != paired.Length ? -1 : ... // -1 means "no mismatch"If the two windows had different lengths, the pairing check reported Honest scope note: with both cells at 50 entries and the same window applied to both, the mismatch branch is currently unreachable through the CLI, so this never produced a wrong answer in practice. That is luck, not design — the guard was inverted, and it would have lied the first time the inputs differed. The fix is correct and I have kept it as-is. Also in this push: argument validation on both new verbs, from the "option validation omissions" line in the review summary that had no inline comment attached. Release 0-warn; LongMemEval 730/730 after the rebase. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The branch is named for a hypothesis that is now closed as a clean negative. What it actually carries
is the instruments built while testing it — and two code facts they uncovered.
The code facts, both verifiable in this repo
Write-time supersession cannot fire on free-form predicates.
PersistenceStagegates onWriteTimeFactResolution.CanSupersede→MemoryRelationCardinality.IsSingleValued, which requiresmembership in the relation vocabulary's
singlecardinality set — exactly six relations(
belongs to, costs, expires, lives in, weighs, works at), no aliases, documented "false foranything unrecognised". A corpus whose extracted predicates fall outside those six cannot produce a
single
:SUPERSEDED_BYedge, soSupersedeReplacedFacts = trueis indistinguishable fromfalseonit. Measured across four separate stores: 0 edges, 0 invalidated facts. A guard test now pins the
gate's reach.
CreateAboutRelationshipAsyncis public API that no ingestion path calls. Verified against aprepared store: 26,887 of 26,887 facts carry no
:ABOUTedge. It is not unread, though —NodeDistanceQueriestraverses[:RELATED_TO|ABOUT*..4]and its docstring namesABOUTasload-bearing, so anyone enabling
NodeDistanceRerankingtoday gets a re-ranker walking a graphmissing every
ABOUTedge it names, weaker than its own query describes, with nothing reporting thedifference.
Documented rather than changed: the writer is public under SemVer, so a consumer who created links
manually has them honoured today and removing the edge from the traversal would change their results
silently. All three surfaces — both verbs, the option, and the traversal — now say what is true.
Wiring a writer or dropping the edge stay open as real options.
Instruments
:SUPERSEDED_BYcount, invalidated-fact count, extracted-predicate histogram, fact-object shape, amount-subject collision. A mechanism's firing is verified
by counting its artifacts, never by a downstream statistic moving — the previous check inferred
supersession from facts-per-question falling, and that band turned out to be extraction variance.
--regrade: judge-only re-grade of a stored artifact, replayed through the evaluator's ownrunner rather than a second grading loop, so scoring cannot drift from the path that produces every
other number. Self-validating — re-grading under the judge that produced an artifact reproduces it
(60/60 verdicts, identical accuracy). Keyed by position, after the duplicate guard caught that
12 of 60 questions in one corpus share their text with a question having a different gold.
block is a failure, never a pass.
--cell-probefor extracting external corpus files, with--dry-run.--resolve-supersessionswiring, plus a staged live-Neo4j test proving supersession renders onboth the live and as-of recall paths — neither had any coverage before.
The guards, and why they exist
Each one is a mistake made in this branch, converted into something that cannot recur:
phenomenon (the sampling rule said the first 12 entries; the phenomenon starts at entry 14). The
probe now counts occurrences and aborts before the first paid call.
0 of 0printsNOT MEASURED … this is not 0%. A share over nothingis not a small share.
already been paid for. Nothing cheap may stand between an expensive result and durability.
that empty means "no store to read", not "measured zero", because a default that silently
satisfies every caller is how a constant column is born.
Verification
Release build 0 warnings. Unit 5,187 + 728 + 54. Integration 486 passed / 29 skipped / 0
failed against live Neo4j; the skips are the NAMS live tests behind their opt-in flag.
Also bumps the SDK pin to 10.0.400 (10.0.102 was uninstalled and unavailable) and the evaluator
package to the release whose judge has a body for the vertical it was grading.