Wire the composition levers into the benchmark harness, and measure what they move

[joslat]

What this is

The benchmark harness could not switch on the engine's own composition machinery, so every
TypedMemEval number this repository holds was measured with it off. This wires it, guards that it
arrives, and adds a value-grained instrument for reading the results.

The defect

AgentMemoryLongMemEvalAdapter builds recall with two levers whose own comments name the failure
they exist for:

• ExpandFactsByPredicate — "a relation returned whole, for the aggregation questions top-K
  structurally cannot answer"
• ResolveQueryRelations — "the multi-relation case top-K structurally cannot nominate"

Both were set only by LongMemEvalPreparedPairProgram. TypedMemEvalProgram set neither and
exposed no flag. RecallFanOutOptions.Enabled was set by nothing under tools/ at all.

Those are precisely the question shapes that scored zero across five verticals — alias-then-count
0/15, arithmetic delta 0/10 and sum 0/14, procedural step-order 0/20 — and 58–99% of every
failure in that family was a failure where the required evidence was incomplete in the answer
context.

What changed

• --expand-facts, --resolve-query-relations, --recall-fan-out, threaded through to
  RecallOptions and MemoryOptions.FanOut.Enabled.
• Each lever stamps itself into the arm token (expand / qrel / fanout). An ON artifact must
  never be mistakable for a default one on disk — a run that silently took the default path would
  reproduce the baseline and read as "the levers do nothing", which is the most dangerous outcome
  available here.
• LongMemEvalGoldValueCoverageProbe — record-only, reads what recall already returned. It reports
  how many of the gold-bearing turns' values reached the recalled facts, separating "retrieval never
  had it" from "retrieval had it and the answer still missed".

Off stays off. An unflagged run is byte-identical to every sealed measurement before it, and a
test asserts it.

Measured effect

Same corpus, same seed, same judge, gates passed on both:

vertical	default	expand-qrel
arithmetic	7/50 = 14%	15/50 = 30%
procedural	9/80 = 11.2%	21/80 = 26.2%

Arithmetic's duration shape — the one that never needed aggregation — did not move (5/12 both
arms). That is what separates a mechanism from global inflation.

Reported against it: gross +11/−3 and +18/−6. One arithmetic question hit MaxExpandedFacts
exactly (10 → 60 facts) and regressed; expansion injects distractors as well as evidence.

Tests

767 in the harness suite, 5,192 in the main unit suite. The new ones follow each lever from the
command line to the object the engine actually reads — a flag that parses is not a flag that
arrives, which is what all five reachable-but-not-fed instances in this repo had in common.

They also caught two real defects in the instrument before it touched any data: an integer rule
matching the digit groups inside decimals (nine values reported for three), and an amount pattern
that could not match a bare 1113.71, matching the tail 3.71 and reporting a miss for a value that
was present.

One thing shipped deliberately imperfect

The instrument's first version sourced required values from the gold answer, which for an
aggregation question states a total that is never stored. Validation against real corpus text showed
it measured the wrong thing (arithmetic: mean 1.00 value per gold — the result, not the inputs) and
was entirely silent on procedural (0 of 80 golds carry a value). The commit history keeps that
finding rather than hiding it, and the follow-up commit re-sources from gold-bearing turns.

It remains numeric-only by design: procedural's values are step names, and a name matcher tuned
on invented multi-word corpus names would be tuned on padding. The line prints NOT MEASURABLE
rather than a confident 0.00 — unmeasurable is not zero.

[Copilot]

🟡 Changes recommended

The new gold-value coverage reporting contains a misleading “NOT MEASURABLE” message and an XML doc comment issue that should be corrected to avoid incorrect diagnostics and documentation warnings.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the AgentMemory.LongMemEval benchmark harness so TypedMemEval can explicitly toggle the engine’s composition-related retrieval levers (expand facts, resolve query relations, recall fan-out), and adds a record-only probe to measure whether numeric “gold-bearing” values actually reached retrieved facts (separating retrieval misses from answering misses).

Changes:

• Adds --expand-facts, --resolve-query-relations, and --recall-fan-out flags to TypedMemEval and threads them through to the adapter/memory options and arm token stamping.
• Introduces LongMemEvalGoldValueCoverageProbe plus reporting to measure value-grained retrieval coverage without affecting retrieval/prompting.
• Adds focused unit tests to ensure flags are recognized, carried end-to-end, and that the new coverage metric behaves correctly (normalization, null vs zero, partial coverage).

File summaries

File	Description
tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs	Wires new CLI levers into the TypedMemEval run path and prints gold-value coverage summary.
tools/AgentMemory.LongMemEval/TypedMemEvalArm.cs	Extends arm identity to include the new levers and stamps them into file tokens/descriptions.
tools/AgentMemory.LongMemEval/LongMemEvalMemoryProfile.cs	Threads recall fan-out into MemoryOptions so the engine can actually see it.
tools/AgentMemory.LongMemEval/LongMemEvalGoldValueCoverage.cs	Adds the numeric-only extraction/normalization + probe for value-grained retrieval coverage.
tools/AgentMemory.LongMemEval/AgentMemoryLongMemEvalAdapter.cs	Records gold-turn values vs recalled fact values via the optional probe.
tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj	Bumps AgentEval dependency version.
tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalCommandLineTests.cs	Verifies new flags are advertised and carried onto the options record.
tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalArmProvenanceTests.cs	Updates arm provenance assertions for the new levers.
tests/AgentMemory.Tests.Unit.LongMemEval/GraphRagWiringTests.cs	Threads recall fan-out through the service wiring tests.
tests/AgentMemory.Tests.Unit.LongMemEval/GoldValueCoverageTests.cs	Adds unit coverage for extraction/measurement/probe correctness and failure boundaries.
tests/AgentMemory.Tests.Unit.LongMemEval/FacadeCallerFeedsTests.cs	Updates options construction to include new lever fields.
tests/AgentMemory.Tests.Unit.LongMemEval/CompositionLeverWiringTests.cs	New end-to-end tests proving each lever reaches the actual consumer object(s).

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[joslat]

W1c added to this branch — expansion inside point-in-time recall

Two composition arms could not run at all: prospective and temporal declare timestamped grounding, so recall anchors at RecallAsOfAsync, which had no expansion parameters at all. An existing guard refused rather than running flags that would silently do nothing — it saved two 5-hour runs for about a minute of cost.

That is a user-facing gap, not only a harness one: any application asking "what did I know as of March" got top-K similarity and no aggregation support, on the one path whose whole purpose is point-in-time questions.

The invariant is the feature

Expansion widens a query from "the K most similar facts" to "this relation, whole", and a relation returned whole from an as-of query must be the relation as it stood then. A dropped clock here does not lose a row — it adds one, and the extra fact looks exactly like legitimate completeness.

So the invariant is structural rather than only tested:

• SearchByCanonicalPredicatesAsOfAsync is a separate method with non-optional clocks — passing the wrong one is not reachable by argument.
• Its default returns empty and never delegates to the unbounded overload. A non-overriding implementor degrades to unexpanded recall, which is merely incomplete. Delegating would make every one of them silently leak facts from outside the window. Incomplete is recoverable; wrong-and-confident is not.
• The Cypher carries the same four clock predicates as SearchFactsAsOf, in the same order, so the similarity half and the expansion half of one recall cannot disagree about what "as of" means.

Written red-first, and verified against a live store

AsOfExpansionTests did not compile before the overload existed, then failed 2 of 3 once it did — with the off-state test passing, proving the harness was sound and the feature was the gap.

Unit tests assert at the service seam over substitutes, so they prove the clocks are passed, not that the Cypher uses them. AsOfExpansionIntegrationTests runs it against live Neo4j: three facts under one predicate, not-yet-true excluded by the valid clock, no-longer-believed by the transaction clock. Its control test earned itself immediately — the first run came back empty from both queries, identifying the fault as my seeding rather than the new query.

The guard was narrowed by making its claim true, never relaxed

It now refuses GraphRAG alone, which genuinely remains unimplemented as-of.

Three pre-existing guard tests failed and every one was right to: AsOfRecallDivergenceTests required the three options be struck from the documented divergence list deliberately. I added the converse case too — expansion options are now accepted — because either test alone permits a half-wired capability: reachable but refused, or accepted but not implemented.

5,195 + 767 green.