Say something when SupersedeReplacedFacts is on and inert - #210
Merged
Conversation
A caller sets SupersedeReplacedFacts = true. Their extractor writes free-form predicates -- "was at", "assigned to", "department". CanSupersede refuses every one of them, because it requires membership in the six relations the vocabulary declares single-valued and is documented false for anything unrecognised. No :SUPERSEDED_BY edge is written, no invalidated_at is stamped, and until now nothing anywhere said so. The feature is on and doing nothing, silently. That is not a hypothetical failure mode. A benchmark ran FOUR scored arms against exactly this state and reported them as measurements of the feature; the cause was found only because the graph could be queried directly and a probe was written to count the edges. A consumer has neither recourse. They would conclude bitemporal memory does not work. Three signals, at the level each belongs: - DEBUG, per refusal: names the predicate that was refused and lists the qualifying relations. This is what makes the batch warning actionable once you are already looking. - WARNING, once per batch, and only when the option was actually enabled AND nothing qualified: "on and inert". Per-batch rather than per-fact so a large ingestion cannot bury it; conditioned on the option so a feature nobody asked for never warns -- a warning that fires when things are working is how a signal stops being read. - The option's own documentation, where someone enabling it will actually see the constraint. Tested in both directions, because a guard that cannot stay quiet is as bad as one that cannot speak: an unrecognised predicate warns and names both the predicate and the qualifying set; a qualifying predicate does NOT warn; and the feature switched off is silent. Release build 0-warn; 5191 + 730 + 54 unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
There was a problem hiding this comment.
🟡 Changes recommended
The new refusal debug log currently duplicates a structured placeholder name and allocates string.Join(...) even when Debug is disabled, and the updated test file has an unused using directive.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Improves observability for ExtractionOptions.SupersedeReplacedFacts when enabled against non-single-valued predicates by emitting per-refusal debug logs and a once-per-batch warning when the feature is effectively inert, plus documents the constraint on the option.
Changes:
- Add per-refusal debug logging and a batch-level warning when supersession is enabled but no facts qualify.
- Extend unit coverage to assert the “on and inert” warning behavior (and silence when qualifying/off).
- Update option documentation to clarify the single-valued predicate constraint and the new signals.
File summaries
| File | Description |
|---|---|
| tests/AgentMemory.Tests.Unit/Extraction/WriteTimeSupersessionTests.cs | Adds logger-capture-based assertions to verify the new warning/debug signals and silence cases. |
| src/AgentMemory.Core/Extraction/PersistenceStage.cs | Implements refusal counters, debug logging per refusal, and a batch-level warning when supersession is enabled but inapplicable. |
| src/AgentMemory.Abstractions/Options/ExtractionOptions.cs | Documents that supersession only applies to vocabulary-declared single-valued predicates and explains the new signals. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Both from Copilot's review of #210. The debug refusal ran `string.Join` over the relation list unconditionally. Arguments are evaluated whether or not Debug is enabled, and this fires once per REFUSED fact -- so on the exact ingestion this feature exists to diagnose, where nothing qualifies, that is once per fact in the batch. Real cost for output nobody is reading. Now behind IsEnabled, with a test that asserts the message is not built when Debug is off. The batch warning stays unguarded on purpose: it fires at most once and only when something is wrong. Also fixes the duplicated `{P}` structured placeholder (named parameters now, one each) and removes the NullLogger using the capturing logger replaced. Note on the test: it asserts against the REFUSAL message rather than "no Debug entries at all". The first version did the latter and failed -- this stage also logs an unrelated per-fact "Persisted fact" line, and a capturing fake records what it is handed rather than re-checking IsEnabled the way a real provider does. Asserting on the absence of a whole level was testing the fake, not the guard. Release 0-warn; 5192 unit tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
joslat
added a commit
that referenced
this pull request
Aug 31, 2026
Copilot's review of #211 found consecutive <summary> tags above DryRun. Same cause as the one it found on #210: inserting a method between an existing doc comment and its body leaves the comment attached to the newcomer, so one method carries two summaries and another carries none. Twice in one session is a pattern rather than a slip, so this fixes both sites and then SWEEPS the repository for the shape instead of waiting to be told about the next one. The sweep found a third, pre-existing and unrelated to this branch: LeadingToken's summary had been stranded above LeadingTokenAfterLabel in LongMemEvalPostRunDiagnostics, leaving the parser it describes undocumented. Reattached. Repo-wide sites remaining: none. Release build 0-warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
joslat
added a commit
that referenced
this pull request
Aug 31, 2026
* Give the typedmemeval verb a real stage-1 dry run The three-stage run protocol is a maintainer directive covering every paid run, and the verb that runs most of them had no stage 1. This adds one: it loads the corpus through TypedMemEvalCorpus.Load using the same option mapping the real run uses -- so it proves the corpus the run will actually read, not a simulation of it -- and reports the vertical, the arm token, the seed, the selected question count and the shape distribution. It also carries the containment precondition: an empty selection ABORTS, because a run that selects nothing measures nothing, and a window holding none of the phenomenon has already cost this project a paid run. The first version of this sat inside RunVerticalAsync, which meant it booted a Neo4j container and read Azure credentials before printing "nothing spent". A dry run that spends anything -- money or two minutes of container start -- is not the thing the directive asked for. It now runs before any client, container or credential read, and completes in under three seconds. Verified on the row 53 configuration: arithmetic, seed 20260821, arm default, 50 questions selected, shapes count 14 / sum 14 / duration 12 / delta 10, no container created. The drift guard now names --dry-run, so an advertised option that stops being carried fails the suite. Release 0-warn; LongMemEval 730/730. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo * Reattach two stranded doc comments, and sweep the repo for the rest Copilot's review of #211 found consecutive <summary> tags above DryRun. Same cause as the one it found on #210: inserting a method between an existing doc comment and its body leaves the comment attached to the newcomer, so one method carries two summaries and another carries none. Twice in one session is a pattern rather than a slip, so this fixes both sites and then SWEEPS the repository for the shape instead of waiting to be told about the next one. The sweep found a third, pre-existing and unrelated to this branch: LeadingToken's summary had been stranded above LeadingTokenAfterLabel in LongMemEvalPostRunDiagnostics, leaving the parser it describes undocumented. Reattached. Repo-wide sites remaining: none. Release build 0-warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SupersedeReplacedFacts = truedoes nothing at all unless the extracted predicate is one of the sixrelations the vocabulary declares single-valued (
belongs to, costs, expires, lives in, weighs, works at).CanSupersedeis documented "false for anything unrecognised", so free-form predicates —was at,assigned to,department— are refused. No:SUPERSEDED_BYedge, noinvalidated_at,and no signal of any kind.
This is not hypothetical. Four scored benchmark arms ran against precisely that state and were
reported as measurements of the feature. The cause was found only because the graph could be queried
directly and a probe was written to count edges. A consumer has neither recourse — they would
conclude bitemporal memory does not work.
Three signals, each at the level it belongs
actionable once you are looking.
inert". Per-batch so a large ingestion cannot bury it; conditioned on the option so a feature
nobody asked for never warns.
Tested both directions
A guard that cannot stay quiet is as bad as one that cannot speak, so: an unrecognised predicate
warns and names both the predicate and the qualifying relations; a qualifying predicate does
not warn; the feature off is silent.
Release 0-warn; 5191 + 730 + 54 unit tests green. No behaviour change — nothing supersedes that did
not supersede before.