Gate the re-grade on corpus identity: ids and text cannot detect a redraw

[joslat]

From the SHA-only keying audit.

AgentEval redraws corpora keeping the question_id set 100% identical with zero byte-identical
items, and neither corpus_id nor revision moves — corpus_sha256 is the only distinguishing
field. For bitemporal, 27 of 60 items keep the same question text with a different gold.

That defeats the protection the re-grade already had. The replay is position-keyed and verifies
question text per position: it catches a redraw that reworded anything, and catches nothing
when the text is stable and only the gold moved. Re-grading across such a redraw replays the right
answers against the wrong keys and reports an agreement rate that means nothing — after paying for
the judge pass.

The gate

Corpus sha compared before any judge call. Stored value from the artifact's own
TypedOutcomes.CorpusSha256; current value hashed from the embedded resource this build carries,
rather than read from a manifest — the bytes the run would use, not a claim about them.

Three outcomes, deliberately distinct:

• match → proceed
• mismatch → ABORT, naming both shas
• no sha recorded → warn, don't fail — artifacts predating provenance capture are still
  re-gradable, but it is the one case where nothing can be verified, so it says so rather than
  passing quietly

Verified both ways: a matching artifact proceeds; an artifact whose sha is moved to the current
bitemporal lineage (abf2f3f4) aborts with both shas named.

Also documents the adapter's blindness as a test rather than pretending it away — identical text
passes the positional check whatever the gold now says, which is exactly why identity belongs one
layer up.

Release 0-warn; 734/734.

[Copilot]

🟡 Changes recommended

The new gate can proceed silently when the current corpus SHA can’t be resolved and can throw on malformed/short SHA strings, undermining the intended “verify-before-spend” behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds a corpus-identity gate to the TypedMemEval re-grade tool to prevent re-grading artifacts against a different (redrawn) corpus with identical question ids/text, and adds a unit test documenting the replay adapter’s positional-blindness to “gold moved behind unchanged text”.

Changes:

• Abort re-grade early when the artifact’s stored TypedOutcomes.CorpusSha256 doesn’t match the corpus bytes embedded in the current build.
• Warn (non-fatal) when an artifact has no recorded corpus SHA (pre-provenance artifacts).
• Add a unit test that documents the replay adapter’s inability to detect moved gold when question text is unchanged.

File summaries

File	Description
tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs	Adds the corpus SHA gate and computes current corpus SHA from embedded resources.
tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalReplayAdapterTests.cs	Adds a unit test documenting a known limitation of position-keyed replay.

Review details

Suppressed comments (1)

tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs:140

• If CurrentCorpusSha returns null (resource not found or ambiguous), the code currently proceeds silently even though the doc comment says this should be treated as "cannot verify". Also, slicing storedCorpusSha[..16] can throw if the stored value is shorter than 16 chars. Add an explicit warning branch for the null case and make the abort message length-safe.

            else if (currentCorpusSha is not null &&
                     !string.Equals(storedCorpusSha, currentCorpusSha, StringComparison.OrdinalIgnoreCase))
            {
                Console.Error.WriteLine(
                    $"regrade: ABORT — corpus mismatch. The artifact was produced against " +
                    $"{storedCorpusSha[..16]}… and this build carries {currentCorpusSha[..16]}…. " +
                    "Question ids and text are NOT sufficient to detect this: corpora are redrawn " +
                    "keeping ids identical, and gold can move behind unchanged text. Re-grade with " +
                    "the package the artifact was produced against.");
                return 5;

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.