Skip to content

Say something when SupersedeReplacedFacts is on and inert - #210

Merged
joslat merged 2 commits into
mainfrom
feat/supersession-gate-observability
Aug 31, 2026
Merged

Say something when SupersedeReplacedFacts is on and inert#210
joslat merged 2 commits into
mainfrom
feat/supersession-gate-observability

Conversation

@joslat

@joslat joslat commented Aug 31, 2026

Copy link
Copy Markdown
Owner

SupersedeReplacedFacts = true does nothing at all unless the extracted predicate is one of the six
relations the vocabulary declares single-valued (belongs to, costs, expires, lives in, weighs, works at). CanSupersede is documented "false for anything unrecognised", so free-form predicates —
was at, assigned to, department — are refused. No :SUPERSEDED_BY edge, no invalidated_at,
and no signal of any kind.

This is not hypothetical. Four scored benchmark arms ran against precisely that state and were
reported as measurements of the feature. The cause was found only because the graph could be queried
directly and a probe was written to count edges. A consumer has neither recourse — they would
conclude bitemporal memory does not work.

Three signals, each at the level it belongs

  • Debug, per refusal — names the refused predicate and the qualifying set. Makes the warning
    actionable once you are looking.
  • Warning, once per batch — only when the option is enabled and nothing qualified: "on and
    inert". Per-batch so a large ingestion cannot bury it; conditioned on the option so a feature
    nobody asked for never warns.
  • The option's own docs — where someone enabling it will read the constraint.

Tested both directions

A guard that cannot stay quiet is as bad as one that cannot speak, so: an unrecognised predicate
warns and names both the predicate and the qualifying relations; a qualifying predicate does
not warn; the feature off is silent.

Release 0-warn; 5191 + 730 + 54 unit tests green. No behaviour change — nothing supersedes that did
not supersede before.

A caller sets SupersedeReplacedFacts = true. Their extractor writes free-form predicates -- "was at",
"assigned to", "department". CanSupersede refuses every one of them, because it requires membership
in the six relations the vocabulary declares single-valued and is documented false for anything
unrecognised. No :SUPERSEDED_BY edge is written, no invalidated_at is stamped, and until now nothing
anywhere said so. The feature is on and doing nothing, silently.

That is not a hypothetical failure mode. A benchmark ran FOUR scored arms against exactly this state
and reported them as measurements of the feature; the cause was found only because the graph could be
queried directly and a probe was written to count the edges. A consumer has neither recourse. They
would conclude bitemporal memory does not work.

Three signals, at the level each belongs:

- DEBUG, per refusal: names the predicate that was refused and lists the qualifying relations. This
  is what makes the batch warning actionable once you are already looking.
- WARNING, once per batch, and only when the option was actually enabled AND nothing qualified: "on
  and inert". Per-batch rather than per-fact so a large ingestion cannot bury it; conditioned on the
  option so a feature nobody asked for never warns -- a warning that fires when things are working is
  how a signal stops being read.
- The option's own documentation, where someone enabling it will actually see the constraint.

Tested in both directions, because a guard that cannot stay quiet is as bad as one that cannot
speak: an unrecognised predicate warns and names both the predicate and the qualifying set; a
qualifying predicate does NOT warn; and the feature switched off is silent.

Release build 0-warn; 5191 + 730 + 54 unit tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
Copilot AI lite review requested due to automatic review settings August 31, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new refusal debug log currently duplicates a structured placeholder name and allocates string.Join(...) even when Debug is disabled, and the updated test file has an unused using directive.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Improves observability for ExtractionOptions.SupersedeReplacedFacts when enabled against non-single-valued predicates by emitting per-refusal debug logs and a once-per-batch warning when the feature is effectively inert, plus documents the constraint on the option.

Changes:

  • Add per-refusal debug logging and a batch-level warning when supersession is enabled but no facts qualify.
  • Extend unit coverage to assert the “on and inert” warning behavior (and silence when qualifying/off).
  • Update option documentation to clarify the single-valued predicate constraint and the new signals.
File summaries
File Description
tests/AgentMemory.Tests.Unit/Extraction/WriteTimeSupersessionTests.cs Adds logger-capture-based assertions to verify the new warning/debug signals and silence cases.
src/AgentMemory.Core/Extraction/PersistenceStage.cs Implements refusal counters, debug logging per refusal, and a batch-level warning when supersession is enabled but inapplicable.
src/AgentMemory.Abstractions/Options/ExtractionOptions.cs Documents that supersession only applies to vocabulary-declared single-valued predicates and explains the new signals.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/AgentMemory.Core/Extraction/PersistenceStage.cs
Both from Copilot's review of #210.

The debug refusal ran `string.Join` over the relation list unconditionally. Arguments are evaluated
whether or not Debug is enabled, and this fires once per REFUSED fact -- so on the exact ingestion
this feature exists to diagnose, where nothing qualifies, that is once per fact in the batch. Real
cost for output nobody is reading. Now behind IsEnabled, with a test that asserts the message is not
built when Debug is off.

The batch warning stays unguarded on purpose: it fires at most once and only when something is
wrong.

Also fixes the duplicated `{P}` structured placeholder (named parameters now, one each) and removes
the NullLogger using the capturing logger replaced.

Note on the test: it asserts against the REFUSAL message rather than "no Debug entries at all". The
first version did the latter and failed -- this stage also logs an unrelated per-fact "Persisted
fact" line, and a capturing fake records what it is handed rather than re-checking IsEnabled the way
a real provider does. Asserting on the absence of a whole level was testing the fake, not the guard.

Release 0-warn; 5192 unit tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
@joslat
joslat merged commit aabddd9 into main Aug 31, 2026
7 checks passed
@joslat
joslat deleted the feat/supersession-gate-observability branch August 31, 2026 09:13
joslat added a commit that referenced this pull request Aug 31, 2026
Copilot's review of #211 found consecutive <summary> tags above DryRun. Same cause as the one it
found on #210: inserting a method between an existing doc comment and its body leaves the comment
attached to the newcomer, so one method carries two summaries and another carries none.

Twice in one session is a pattern rather than a slip, so this fixes both sites and then SWEEPS the
repository for the shape instead of waiting to be told about the next one. The sweep found a third,
pre-existing and unrelated to this branch: LeadingToken's summary had been stranded above
LeadingTokenAfterLabel in LongMemEvalPostRunDiagnostics, leaving the parser it describes
undocumented. Reattached.

Repo-wide sites remaining: none. Release build 0-warn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
joslat added a commit that referenced this pull request Aug 31, 2026
* Give the typedmemeval verb a real stage-1 dry run

The three-stage run protocol is a maintainer directive covering every paid run, and the verb that
runs most of them had no stage 1. This adds one: it loads the corpus through TypedMemEvalCorpus.Load
using the same option mapping the real run uses -- so it proves the corpus the run will actually
read, not a simulation of it -- and reports the vertical, the arm token, the seed, the selected
question count and the shape distribution.

It also carries the containment precondition: an empty selection ABORTS, because a run that selects
nothing measures nothing, and a window holding none of the phenomenon has already cost this project
a paid run.

The first version of this sat inside RunVerticalAsync, which meant it booted a Neo4j container and
read Azure credentials before printing "nothing spent". A dry run that spends anything -- money or
two minutes of container start -- is not the thing the directive asked for. It now runs before any
client, container or credential read, and completes in under three seconds.

Verified on the row 53 configuration: arithmetic, seed 20260821, arm default, 50 questions selected,
shapes count 14 / sum 14 / duration 12 / delta 10, no container created.

The drift guard now names --dry-run, so an advertised option that stops being carried fails the
suite. Release 0-warn; LongMemEval 730/730.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo

* Reattach two stranded doc comments, and sweep the repo for the rest

Copilot's review of #211 found consecutive <summary> tags above DryRun. Same cause as the one it
found on #210: inserting a method between an existing doc comment and its body leaves the comment
attached to the newcomer, so one method carries two summaries and another carries none.

Twice in one session is a pattern rather than a slip, so this fixes both sites and then SWEEPS the
repository for the shape instead of waiting to be told about the next one. The sweep found a third,
pre-existing and unrelated to this branch: LeadingToken's summary had been stranded above
LeadingTokenAfterLabel in LongMemEvalPostRunDiagnostics, leaving the parser it describes
undocumented. Reattached.

Repo-wide sites remaining: none. Release build 0-warn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants