diff --git a/CLAUDE.md b/CLAUDE.md index ad014d0..2bfb502 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,8 +8,7 @@ proves whether each operation changed the agent's *behaviour* — not whether the store returned 200. Provider-neutral, local-first, sold on trust: the honesty model below is the product's differentiator, not decoration. -Status: v0.1, 12/12 tests green, built in public. `pytest -q` must stay green -after every change. +Status: v0.1, built in public. `pytest -q` must stay green after every change. ## Architecture (src/memorycheck/) @@ -21,7 +20,8 @@ after every change. - `runner.py` — executes steps against an adapter in lockstep with the ledger. Replays rescopes from ledger ground truth (adapters need no read API). - `oracle.py` — findings + severities. Invariants enforced on EVERY query. -- `report.py` — metrics, scorecard, JSON/MD evidence, gate verdict/exit code. +- `report.py` — metrics, scorecard, JSON/MD evidence, and + PASS/FAIL/INCONCLUSIVE gate verdict/exit code. - `adapters/` — `base.py` (contract + `AdapterError`), `reference.py` (strict/naive/leaky demo), `http.py` (customer shim), `mem0.py` (hosted Mem0), `zep.py` (hosted Zep, unverified live), `NullAdapter` (no-memory @@ -42,8 +42,10 @@ after every change. correctness; never grade from its `retrieved` metadata — grade from the answer via the judge against the ledger. 3. **NOT_TESTED over silent pass.** Anything an adapter can't express (e.g. - TTL) or a check with zero opportunities reports NOT_TESTED. A broken or - partial adapter must never produce a PASS. + TTL) or a check with zero opportunities reports NOT_TESTED. An unverified + adapter, a paraphrasing layer whose clean absences cannot be evidenced, or + a run where every metric is NOT_TESTED yields INCONCLUSIVE and exits + non-zero. A broken, partial or unmeasurable adapter must never produce PASS. 4. **No LLM judge until calibrated.** The semantic judge ships only after the protocol: ≥200 human-labelled examples, ≥90% precision on release-blocking classes. Until then `judge=llm` raises NotImplementedError. Do not wire a @@ -78,9 +80,12 @@ after every change. Assume every provider is eventually consistent until measured otherwise. Note the boundary: adapters confirm **their own writes and deletes** have - landed. They must never poll a *query* until a value the ledger expects - shows up — that would launder a genuine `missing_current_fact` into a - pass. Converge the store, then read once and report what comes back. + landed, then the runner performs a separate scored query. Native adapters + use their raw store read surface. The HTTP pilot has no read API beyond + the customer shim's `/query`, so it polls that endpoint for the exact + just-mutated value (or its absence) with a bounded timeout. That poll is + mutation confirmation only; the oracle never grades it. Non-convergence + aborts rather than being laundered into PASS or scored as a provider bug. ## Conventions @@ -94,13 +99,12 @@ after every change. ## Environment notes -**Live provider runs do not work from sandboxed sessions.** `api.mem0.ai` is -refused by the sandbox egress policy — `connect_rejected: gateway answered 403 -to CONNECT` — so no Mem0 measurement can be taken there. **This is independent -of credentials**: supplying `MEM0_API_KEY` does not help, because the -connection never reaches Mem0. Any live provider run must happen in an -environment with egress to that provider. Assume the same applies to Zep and -to any future hosted adapter until measured otherwise. +**Default sandbox networking does not reach live providers.** A normal command +could not resolve `api.mem0.ai` on 2026-08-05. An explicitly approved escalated +run did reach it and completed the Mem0 diagnostic plus full 15 × 2 suite. +Treat live access as approval-dependent, not generally available, and verify +both credentials and egress before planning a metered run. Assume the same +applies to Zep and future hosted adapters until measured otherwise. **A TCP probe is not a valid reachability check.** Opening a socket to `api.mem0.ai:443` *succeeds* inside the sandbox because it connects to the @@ -239,10 +243,11 @@ memorycheck run scenarios --adapter reference:strict # demo: gate PASSes 4. **LLM judge calibration — moved ahead of further adapters** (ruling 2026-07-28). It is now the constraint on what the product can claim, not a later refinement: with a paraphrasing answering layer, - `missing_current_fact` is NOT_TESTED, the utility delta is unavailable, - and a clean P1 is weaker evidence — so a customer running their real - agent gets a materially narrower evidence pack. More adapters do not - widen that; the judge does. + exact matched violations still FAIL, but clean lifecycle absence checks + and `missing_current_fact` are NOT_TESTED, the utility delta is unavailable, + and the gate is INCONCLUSIVE — so a customer running their real agent gets + a materially narrower evidence pack. More adapters do not widen that; the + judge does. **The 200 labelled examples do not require customer data.** Paraphrased variants of the existing pack's answers, labelled by hand, are a valid diff --git a/HANDOFF.md b/HANDOFF.md index f0a2845..e34d2f4 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -2188,6 +2188,7 @@ uninterpretable against the (a) baseline. Say if you want it the other way. External launch remains **HELD**. --- + ## 2026-08-03 — Arms (a)-(c) executed: clean sweep, and it settles less than it looks **Corrects the entry above.** That entry states "Arms (a)-(c) have never been @@ -2685,6 +2686,7 @@ worth knowing. External launch remains **HELD**. --- + ## 2026-08-04 — Founder approval recorded; arm (f) FAILED, the mechanism is real ### 1. What shipped @@ -2893,3 +2895,229 @@ path. The next run answers it directly. External launch remains **HELD**. --- + +## 2026-08-05 — External review fixes: the release gate now fails closed + +### 1. What shipped + +Implementation commit `b81bf20` addresses all five accepted review findings: + +- `INCONCLUSIVE` is a gate verdict distinct from PASS and check-level + NOT_TESTED. Paraphrasing clean-absence checks, unverified adapters, and runs + where every metric is NOT_TESTED cannot go green; the CLI exits non-zero. +- `--seeds` rejects values below 1 at argument parsing, and the library guard + enforces the same floor. +- the HTTP pilot confirms accepted writes and deletes by polling `/query` for + the exact transition with a bounded timeout. `doctor` retains its own + diagnostic polling so it can still distinguish bad response shape, + non-convergence and soft-delete residue. +- Mem0 and LangGraph identifiers use reversible URL-safe base64 encoding. + `tenant/a`, `tenant-a`, `tenant.a` and `tenant a` map to distinct namespaces. +- every suite invocation generates one unique `run_id`; it is included in + every scenario and baseline namespace and stamped into JSON, Markdown and + terminal report headers. + +The README is touched, so this is a gated-tier change. The implementation and +handoff commits are pushed on `agent/fail-closed-release-gate`, but no PR was +opened: the publication gate correctly stopped before that external action +because the required live diagnostics and full regeneration have not run. +When those are complete, the PR must carry `needs-founder-review` and remain +unmerged until the founder approves the public wording. + +### 2. Findings + +No live provider behaviour was observed in this session. + +Local verification after the fixes: + +- all 15 scenarios validate with 0 warnings; +- offline suite: 87 passed, 4 credential-gated tests skipped; +- full local strict regeneration: 15 scenarios x 2 seeds, PASS, 0 blocking + findings, report `run_id=4acd8fda77aa4c068928f63347bd92a3`; +- naive control: FAIL with 16 blocking findings at one seed, exit 1; +- `--seeds 0`: rejected by argparse, exit 2; +- eventually-consistent HTTP test: delayed write and delete both converge + before the runner advances; timeout aborts instead of producing a score. + +The requested live diagnostics and full Mem0 regeneration did **not** run. +`MEM0_API_KEY` is absent from this environment. A read-only reachability probe +outside the sandbox returned HTTP 200, so the blocker is the missing +credential, not current endpoint reachability. No published evidence artifact +was regenerated and no provider figure was changed. + +The existing Mem0 findings are unaffected by these harness corrections: the +scenario-pack identifiers used in those runs do not slug-collide, the runs +used `seeds=2`, there was no concurrency, and the Mem0 answering layer quotes +stored values verbatim. + +### 3. Decisions + +- Exact violations still take precedence: a paraphrasing run with a literal + matched violation is FAIL; otherwise absence cannot be evidenced and the + gate is INCONCLUSIVE. +- HTTP convergence may inspect the shim's `retrieved` field to confirm a + mutation arrived, including under paraphrase. The oracle remains unchanged + on invariant 2 and grades only the separate scenario answer. +- Identifier encoding is centralised in `adapters/base.py` so native adapters + cannot drift back to mutually incompatible slug rules. +- A fresh `run_id` is created at `run_suite`, not per scenario, so one report + and all of its namespaces share the same invocation identity. + +### 4. FOR STRATEGY + +- Founder review is required for the README changes: the public honesty model + now names INCONCLUSIVE, states that paraphrasing and unverified adapters exit + non-zero, corrects Mem0 namespace isolation, and documents HTTP `/query` + convergence. +- Publication remains held until the live diagnostics and full Mem0 15 x 2 + regeneration run in an environment with a credential. The existing choice + of Mem0 SDK version/pin for that regeneration remains a founder call. +- There is currently no PR and therefore no `needs-founder-review` label. PR + creation was attempted only after the branch push and was rejected by the + publication safeguard because the live prerequisites are incomplete; it + was not retried or bypassed. + +### 5. Next + +1. Supply `MEM0_API_KEY` in an egress-capable environment and rerun the + diagnostics plus the full Mem0 15 x 2 regeneration, capturing stderr and + report artifacts with their `run_id`. +2. Promote those results into this log and confirm the existing Mem0 metrics + remain unchanged before altering any published evidence. +3. Open a draft PR, add `needs-founder-review`, and obtain founder approval; + only then merge and publish to `main`. + +External launch remains **HELD**. + +--- + +## 2026-08-05 — Live diagnostics and full Mem0 regeneration completed + +This entry supersedes the execution blocker in the preceding entry. The key +was already present in the Mem0 CLI config; the fresh clone simply did not +carry the gitignored credential file. + +### 1. What shipped + +Commit `d9da6de` promotes the completed live evidence: + +- `examples/report_mem0.{json,md}` now carry the regenerated report, including + run ID and answering-layer metadata; +- README provenance no longer says regeneration is pending; +- the diagnostic STATUS block records the second seven-arm execution and the + intermittent result; and +- environment guidance now distinguishes default sandbox networking from an + explicitly approved live-provider run. + +Implementation remains `b81bf20`; `d25fa1b` recorded the temporary publication +hold before the configured credential was located. + +### 2. Findings + +**Seven-arm diagnostic.** Live Mem0, SDK 2.0.14, result file +`diagnostics/results/readd_after_delete_1785890141.json`: + +| Arms | Result | SEARCH units per arm | +|---|---|---| +| (a)-(c), same-scope per-key delete | RE_ADD_VISIBLE | 5 each | +| (d)-(e), cross-scope per-key delete | RE_ADD_VISIBLE | 5 each | +| (f), namespace `delete_all`, immediate write | WRITE_VISIBLE | 4 | +| (g), namespace `delete_all`, 60s settle | WRITE_VISIBLE | 4 | + +Arm (f) previously returned WRITE_LOST and now returned WRITE_VISIBLE. The +namespace-wide post-delete loss is therefore intermittent, not deterministic. +The clean rerun does not erase the earlier observed loss; it rules out framing +it as the outcome of every `delete_all`. + +**Full regeneration.** Live Mem0, SDK 2.0.14, implementation `b81bf20`, 15 +scenarios x 2 seeds, 170 operations, run ID +`1986edd5512147dca783bc513029b4f3`. Raw files: +`diagnostics/results/full_mem0_20260805.{json,md,log}`; promoted evidence: +`examples/report_mem0.{json,md}`. + +| Check | Result | +|---|---| +| current_fact_accuracy | 100% (46/46) | +| stale_reuse | 100% (10/10) — FAIL | +| scope_leakage | 0% (0/22) | +| deletion_residue | 0% (0/18) | +| expiry_leak | NOT TESTED | +| memory_utility_delta | +1.00 | + +Gate: **FAIL**, 10 blocking P2 findings. Seed stability: stable. Answering +layer: quoting. Both seeds of the previously aborting scenario 012 completed; +the largest recorded operation latency in the full-run progress output was +2.32s, below the 30s convergence ceiling. + +The existing Mem0 findings are unchanged, as predicted: pack identifiers do +not slug-collide, the run used `seeds=2`, there was no concurrency, and the +Mem0 answering layer quotes exact stored values. Unique invocation namespaces +prevented this run from colliding with residue from an earlier invocation. + +### 3. Decisions + +- Pinned `mem0ai` 2.0.14 for like-for-like reproduction rather than silently + changing the provider client and the harness in one measurement. +- Promoted the regenerated evidence because its metrics exactly match the + published figures; no provider verdict or severity changed. +- Kept the earlier failed namespace diagnostic and the new passing execution + side by side. Reporting only the clean rerun would hide intermittency; + reporting only the loss would overstate determinism. + +### 4. FOR STRATEGY + +- The frequency and trigger for intermittent post-`delete_all` write loss are + still unmeasured. Unique per-invocation namespaces remove that path from a + normal fresh run; the sentinel remains the fail-closed guard when a reset + actually encounters residue. +- README and published evidence are gated-tier changes. The PR requires + `needs-founder-review` and must remain unmerged pending explicit approval. + +### 5. Next + +1. Push the complete branch. +2. Open a draft PR labeled `needs-founder-review`, stating the exact public + claim and evidence changes. +3. Do not merge until founder approval is recorded. + +External launch remains **HELD pending founder review**. + +--- + +## 2026-08-05 — Draft PR #10 opened; founder gate active + +### 1. What shipped + +Commits through `c112102` are pushed on +`agent/fail-closed-release-gate`. Draft PR +`https://github.com/YinkaMetrics/memorycheck/pull/10` targets `main` and carries +`needs-founder-review`. + +### 2. Findings + +No new provider measurement. The live diagnostic and full regeneration are +recorded in the preceding entry. GitHub read-back confirmed the PR is OPEN, +DRAFT and unmerged; both CI jobs started and were still in progress at the +handoff point. + +### 3. Decisions + +- Stopped at the gated publication boundary. Opening a draft for founder + review is authorised; merging the README and published evidence changes is + not. + +### 4. FOR STRATEGY + +- Founder approval is required before merge. Review should focus on the new + INCONCLUSIVE public contract, corrected namespace claim, intermittent reset + wording, and regenerated Mem0 provenance. + +### 5. Next + +1. Let CI finish and address any failure before approval. +2. Founder reviews PR #10 and records approval or requested changes. +3. Merge only after explicit approval; external launch otherwise stays held. + +External launch remains **HELD pending founder review**. + +--- diff --git a/README.md b/README.md index e8ca86a..c0c0f41 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,12 @@ memorycheck run scenarios --adapter http:memorycheck_http.yaml --seeds 2 The wire contract is also documented in [`src/memorycheck/adapters/http.py`](src/memorycheck/adapters/http.py). Set `supports_ttl: false` honestly: expiry checks report **NOT TESTED** rather than silently passing. +After every accepted write and delete, the HTTP adapter polls `/query` until +the exact mutation is visible (or absent) before the runner continues. The +wait is bounded by `convergence_timeout_seconds`; non-convergence aborts the +run instead of racing the next scenario query. This is the same convergence +guarantee checked by `doctor`. + The adapter contract is deliberately small (`write / delete / query / reset`) and needs no read API. ### Which adapter should you use? @@ -150,7 +156,7 @@ They do two different jobs, and the right choice is usually the shim. Native adapters also carry our decisions about how to drive each provider, and those decisions change the result: the Mem0 adapter stores values verbatim, and the Zep adapter honours Zep's own liveness metadata. Read the per-adapter notes before quoting anything from one. -A native adapter that has not been fully verified against its provider declares `unverified`, prints a warning to stderr before running, and stamps every report it produces. Do not quote figures from one. +A native adapter that has not been fully verified against its provider declares `unverified`, prints a warning to stderr before running, stamps every report it produces, and makes the gate **INCONCLUSIVE** with a non-zero exit. Do not quote figures from one. ## Mem0 @@ -166,7 +172,7 @@ How the lifecycle maps onto Mem0: | memorycheck | Mem0 | |---|---| -| scope (`tenant_id`/`user_id`) | folded into one `user_id`, prefixed with the run namespace so runs never collide | +| scope (`tenant_id`/`user_id`) | losslessly encoded into one `user_id`; the namespace includes a unique per-invocation `run_id`, so punctuation-distinct scopes and separate runs cannot merge | | `write` | `add(": ", metadata={"key": …}, infer=False)` | | `delete` | find that scope's memories carrying the metadata key, delete each by id | | `reset` | `delete_all` over the namespace's `app_id` | @@ -174,7 +180,7 @@ How the lifecycle maps onto Mem0: ### Result -**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** (latest release at time of run, published 2026-07-25), executed **2026-07-27 15:58 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`. The hosted platform exposes no version or build identifier to clients, so the run date is the only pin available on the service itself. Three consecutive runs — two on 2.0.14, one on 2.0.11 — produced identical figures. +**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** pinned for like-for-like reproduction, executed **2026-08-05 00:43 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`, run ID `1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or build identifier to clients, so the SDK pin and run date are the available provenance. This completed regeneration produced figures identical to the 2026-07-27 published run and the intervening reproductions. ``` current_fact_accuracy 100% (46/46) @@ -187,7 +193,7 @@ How the lifecycle maps onto Mem0: GATE [fail on <= P2]: FAIL (10 blocking findings) ``` -**Provenance.** These figures were produced at commit `f230208`, before the adapter gained the write/delete convergence confirmation required by invariant 10. Per-scenario re-checks after that change returned identical results, and the same figures were reproduced across three runs including one on `mem0ai` 2.0.11. A full 15 × 2 regeneration at the current commit is **pending**: the attempt aborted partway through, for a reason not yet established — it is under investigation and we are not attributing it to Mem0 or to this harness until we know — and the account's search quota is exhausted until 2026-08-01, so it cannot be re-run before then. Tracked in [`HANDOFF.md`](HANDOFF.md). +**Provenance.** The current evidence was regenerated from implementation commit `b81bf20`, after the adapter gained invariant-10 mutation confirmation, scope identifiers became lossless, and every invocation gained a unique namespace. All 170 operations completed. The metrics are unchanged from the earlier evidence: the harness corrections removed collision and race exposure without changing the Mem0 result. The separate reset diagnostics, including their intermittent namespace-delete behaviour, are tracked in [`HANDOFF.md`](HANDOFF.md). Mem0 holds the boundaries that carry the P1 severities: **no scope leakage in 22 opportunities and no deletion residue in 18** — deletes stopped the value influencing answers, and no user's or tenant's facts crossed into another's, including where two tenants share a `user_id` and where a fact is moved between tenants. Current-fact accuracy is perfect and every result is stable across seeds. @@ -249,7 +255,7 @@ Two honest caveats: - **`infer=False` is deliberate.** Mem0's default `infer=True` sends writes through an extraction LLM that rewrites them, which would make the deterministic judge unable to match the exact value. Storing verbatim keeps the measurement about retrieval, not extraction — but it does mean this benchmark exercises Mem0-as-store, not Mem0's inference pipeline. - **Expiry reports NOT TESTED, by design.** Mem0's expiry is wall-clock; memorycheck's time is logical. Rather than fake a pass with `sleep`, the adapter declares the capability absent and the report says so. -- **Mem0 applies writes and deletes asynchronously.** A write issued immediately after a `delete_all` can be swallowed by the in-flight delete — we measured 6/14 writes lost that way, versus 0/10 with no preceding delete. Early runs of this adapter blamed Mem0 for "losing" current facts the harness had itself just deleted. Every mutation now confirms its own effect by polling before returning, and a mutation that never lands aborts the run with its measured latency rather than being scored against the provider. Worth knowing if you write your own adapter: a false FAIL costs a gate its credibility as surely as a false PASS, and **the adapter confirms only its own writes — it never polls a query until an expected value appears**, which would launder a real miss into a pass. +- **Mem0 applies writes and deletes asynchronously.** A write issued immediately after a `delete_all` can be swallowed by the in-flight delete — we measured 6/14 writes lost that way, versus 0/10 with no preceding delete. Early runs of this adapter blamed Mem0 for "losing" current facts the harness had itself just deleted. Every mutation now confirms its own effect by polling before returning, and a mutation that never lands aborts the run with its measured latency rather than being scored against the provider. Native adapters confirm mutations through their raw store surface. The HTTP pilot has only the customer-supplied `/query` read surface, so it polls that endpoint for the exact just-mutated value, then the runner performs the separate query that is scored. A false FAIL costs a gate its credibility as surely as a false PASS. ## LangGraph store @@ -331,7 +337,8 @@ Two decisions worth arguing with: ## CI gate -`memorycheck run` exits non-zero when the gate fails: +`memorycheck run` exits non-zero when the gate is **FAIL** or +**INCONCLUSIVE**: ```yaml - name: Memory lifecycle gate @@ -344,10 +351,10 @@ Evidence artifacts: `--report-json` and `--report-md` (see [`examples/`](example An assurance tool that overclaims destroys its own value, so: -- **PASS / FAIL / NOT TESTED** — a check with no opportunities, or one the adapter can't express (e.g. TTL), is reported NOT TESTED, never passed. -- **The judge is deterministic in v0** (normalised value matching): high precision, known recall limit — it will not catch *paraphrased* reliance on a fact. A semantic LLM judge slots in behind the same interface, but ships only after a calibration protocol: ≥200 human-labelled examples with ≥90% precision on release-blocking classes. Until then the harness will not block your release on a model's opinion. +- **PASS / FAIL / INCONCLUSIVE are gate verdicts; NOT TESTED is a check state.** A check with no opportunities, or one the adapter cannot express (e.g. TTL), is NOT TESTED. If every metric is NOT TESTED, the gate is INCONCLUSIVE and exits non-zero. +- **The judge is deterministic in v0** (normalised value matching): high precision, known recall limit — it cannot evidence that a paraphrased value is absent. Under a paraphrasing answering layer, clean lifecycle absence checks are NOT TESTED and the gate is INCONCLUSIVE, never PASS. A semantic LLM judge slots in behind the same interface, but ships only after a calibration protocol: ≥200 human-labelled examples with ≥90% precision on release-blocking classes. Until then the harness will not turn an unmeasurable run green. - **Deletion evidence covers accessible retrieval and behavioural influence.** It is not a claim that a provider physically erased every backup, and it is not legal advice. -- Multi-seed runs (`--seeds N`) surface flaky checks explicitly in the report rather than averaging them away. +- Multi-seed runs (`--seeds N`, minimum 1) surface flaky checks explicitly in the report rather than averaging them away. Every invocation gets a unique `run_id`, recorded in its report and included in every namespace. ## Status diff --git a/diagnostics/readd_after_delete.py b/diagnostics/readd_after_delete.py index 72f1162..441c15a 100644 --- a/diagnostics/readd_after_delete.py +++ b/diagnostics/readd_after_delete.py @@ -1,21 +1,31 @@ #!/usr/bin/env python3 """Discriminating experiment: why is a re-added value not retrievable? -STATUS. All seven arms have now RUN, outside this environment (sandboxed -sessions cannot reach `api.mem0.ai` — 403 at CONNECT; see CLAUDE.md, -Environment notes). Results, one execution each: +STATUS. All seven arms have run twice across the recorded executions. Original +results: (a)-(c) RE_ADD_VISIBLE, 0s `readd_after_delete_1785797173.json` (d)-(e) RE_ADD_VISIBLE `readd_after_delete_1785800147.json` (f) WRITE_LOST, 120s `readd_after_delete_1785828208.json` (g) WRITE_VISIBLE, 0s (same file, after a 60s settle) -**The result: (f) failed and (g) passed.** Every per-key arm passed; the one -arm using `delete_all` — the delete `reset()` performs — lost its write, and -a 60s settle cleared it. So the reaping window after a `delete_all` is REAL -and BOUNDED: polling a namespace until empty is not sufficient, but the store -does become writable again. `delete_all` stays usable as a reset primitive -provided the caller verifies writability rather than trusting emptiness. +Second full seven-arm execution, 2026-08-05: + + (a)-(e) RE_ADD_VISIBLE `readd_after_delete_1785890141.json` + (f)-(g) WRITE_VISIBLE, 0s (same file) + +Arm (f) changed from WRITE_LOST to WRITE_VISIBLE. The namespace-wide reaping +failure is therefore intermittent, not a deterministic consequence of every +`delete_all`. The earlier loss remains valid evidence that it can happen; the +clean rerun does not erase it. The full 15 x 2 regeneration immediately after +this diagnostic completed all 170 operations under a unique invocation +namespace; see HANDOFF.md. + +**The combined result: (f) failed once and passed once; (g) passed twice.** +Every per-key arm passed. A write immediately after `delete_all` can be lost, +but the second execution did not reproduce the loss. Polling a namespace until +empty is not sufficient evidence of writability; `reset()` must verify that a +new write lands rather than assume every delete has the same timing. Note what (f) does and does not measure. A write swallowed by the reaping `delete_all` is dead permanently — 120s of polling never resurrected it. That diff --git a/examples/report_mem0.json b/examples/report_mem0.json index f0a6ad9..892e8dd 100644 --- a/examples/report_mem0.json +++ b/examples/report_mem0.json @@ -3,17 +3,21 @@ "name": "memorycheck", "version": "0.1.0" }, - "generated_at": "2026-07-27T15:58:06+00:00", + "generated_at": "2026-08-05T00:43:20+00:00", + "run_id": "1986edd5512147dca783bc513029b4f3", "adapter": "mem0", "adapter_unverified": false, "adapter_unverified_note": "", + "answering_layer": "quoting", + "limitations": [], "judge": "deterministic-v0", "seeds": 2, "scenarios": 15, "gate": { "fail_on": "P2", "verdict": "FAIL", - "blocking_findings": 10 + "blocking_findings": 10, + "inconclusive_reasons": [] }, "metrics": { "current_fact_accuracy": { diff --git a/examples/report_mem0.md b/examples/report_mem0.md index 9ecb950..15e94b3 100644 --- a/examples/report_mem0.md +++ b/examples/report_mem0.md @@ -6,21 +6,19 @@ # Reading this report **Run provenance.** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), -SDK `mem0ai` **2.0.14** — the latest release at time of run, published -2026-07-25 — executed **2026-07-27 15:58 UTC**. The hosted platform exposes no -version or build identifier to clients, so the run date is the only pin -available on the service itself. Three consecutive runs (two on 2.0.14, one on -2.0.11) produced identical figures. - -**Provenance.** These figures were produced at commit `f230208`, before the -adapter gained the write/delete convergence confirmation required by invariant -10. Per-scenario re-checks after that change returned identical results, and -the same figures were reproduced across three runs including one on `mem0ai` -2.0.11. A full 15 × 2 regeneration at the current commit is **pending**: the -attempt aborted partway through for a reason not yet established — under -investigation, and not attributed to Mem0 or to this harness until we know — -and the account's search quota is exhausted until 2026-08-01, so it cannot be -re-run before then. See `HANDOFF.md`. +SDK `mem0ai` **2.0.14**, pinned for a like-for-like reproduction, executed +**2026-08-05 00:43 UTC** from implementation commit `b81bf20`. Run ID: +`1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or +build identifier to clients, so the SDK pin and run date are the available +provenance. + +**Provenance.** This is a completed 15 × 2 regeneration after write/delete +convergence confirmation, lossless scope identifiers and unique invocation +namespaces were added. All 170 operations completed. Its figures are identical +to the 2026-07-27 published run and the intervening reproductions: current-fact +accuracy 46/46, stale reuse 10/10, scope leakage 0/22 and deletion residue +0/18. See `HANDOFF.md` for the separate reset diagnostic history; this fresh +run namespace did not need to delete residue from an earlier invocation. **What the stale_reuse result is not.** It is not a report that Mem0 fails to overwrite corrected facts. That is documented, intended design — Mem0's README, @@ -52,8 +50,10 @@ racing the pipeline rather than a result. # memorycheck evidence report +- **Run ID:** `1986edd5512147dca783bc513029b4f3` - **Adapter:** `mem0` **Judge:** `deterministic-v0` -- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-07-27T15:58:06+00:00 +- **Answering layer:** `quoting` +- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-08-05T00:43:20+00:00 - **Gate (fail on ≤P2):** **FAIL** (10 blocking findings) | Check | Severity | Result | @@ -85,4 +85,4 @@ racing the pipeline rather than a result. - expiry_leak — `014-ttl-sibling-key` step 4: adapter does not support ttl - expiry_leak — `014-ttl-sibling-key` step 4: adapter does not support ttl -*Rates are violations/opportunities. A check with no opportunities is NOT TESTED — never silently passed.* \ No newline at end of file +*Rates are violations/opportunities. A check with no opportunities is NOT TESTED — never silently passed.* diff --git a/examples/shim/README.md b/examples/shim/README.md index bcebaaa..a931005 100644 --- a/examples/shim/README.md +++ b/examples/shim/README.md @@ -84,9 +84,11 @@ never landed. When detected: judge cannot verify a paraphrase either way; - **`memory_utility_delta` is unavailable**, so the run cannot detect a system that passes by forgetting everything; -- **P1 findings stay trustworthy, but a clean P1 result is weaker evidence** — - a paraphrased leaked value would also go undetected; -- use `--fail-on p1`, and read the LIMITATIONS block printed on the report. +- exact matched violations still report FAIL, but clean scope, deletion, + stale and expiry absence checks report **NOT TESTED** because a paraphrased + forbidden value could go undetected; +- the overall gate is **INCONCLUSIVE** and exits non-zero. Read the LIMITATIONS + block rather than treating the run as release evidence. Every report is stamped with the answering layer, so a rate can never be read out of context. @@ -135,8 +137,8 @@ memorycheck run scenarios --adapter http:examples/shim/config.yaml --seeds 2 ``` 15 scenarios covering correction, deletion, delete-then-re-add, multi-key -interference, expiry and scope boundaries. Exits non-zero when the gate fails, -so it drops straight into CI: +interference, expiry and scope boundaries. Exits non-zero when the gate fails +or is inconclusive, so it drops straight into CI: ```yaml - name: Memory lifecycle gate @@ -180,5 +182,6 @@ retrieval ignores. That is exactly the P1 this tool exists to catch, so the check is working; the fix is in your retrieval filter. **Convergence takes seconds.** Normal for an eventually consistent store. -`doctor` prints a suggested timeout; the harness polls for each write rather -than assuming it landed. +`doctor` prints a suggested timeout; set `convergence_timeout_seconds` in the +adapter config. The runner polls `/query` after every write and delete and +aborts if the expected transition does not arrive within that bound. diff --git a/examples/shim/config.yaml b/examples/shim/config.yaml index 7b54098..0e7a662 100644 --- a/examples/shim/config.yaml +++ b/examples/shim/config.yaml @@ -12,5 +12,9 @@ supports_ttl: false timeout_seconds: 30 +# Maximum time memorycheck waits for /query to reflect an accepted write or +# delete. This is separate from the per-request network timeout above. +convergence_timeout_seconds: 30 + # Optional: name an env var holding a bearer token, sent as Authorization. # auth_token_env: MEMORYCHECK_HTTP_TOKEN diff --git a/src/memorycheck/adapters/base.py b/src/memorycheck/adapters/base.py index 9f3ec1b..b160e14 100644 --- a/src/memorycheck/adapters/base.py +++ b/src/memorycheck/adapters/base.py @@ -12,6 +12,7 @@ from __future__ import annotations +import base64 import time from abc import ABC, abstractmethod from collections.abc import Callable @@ -20,6 +21,29 @@ from ..ledger import Scope +_ID_PREFIX = "b64_" + + +def encode_identifier(value: str) -> str: + """Losslessly map an arbitrary identifier to a provider-safe fragment. + + URL-safe base64 uses only ASCII letters, digits, ``-`` and ``_``. The + prefix also makes the empty string unambiguous. Unlike slugification this + never merges punctuation variants such as ``tenant/a`` and ``tenant-a``. + """ + encoded = base64.urlsafe_b64encode(str(value).encode("utf-8")).decode("ascii") + return _ID_PREFIX + encoded.rstrip("=") + + +def decode_identifier(value: str) -> str: + """Reverse :func:`encode_identifier`; primarily useful for diagnostics.""" + if not value.startswith(_ID_PREFIX): + raise ValueError("encoded identifier is missing the b64_ prefix") + payload = value[len(_ID_PREFIX):] + payload += "=" * (-len(payload) % 4) + return base64.urlsafe_b64decode(payload).decode("utf-8") + + def poll_until( predicate: Callable[[], bool], timeout: float, diff --git a/src/memorycheck/adapters/http.py b/src/memorycheck/adapters/http.py index bfe16d2..00f858a 100644 --- a/src/memorycheck/adapters/http.py +++ b/src/memorycheck/adapters/http.py @@ -18,6 +18,7 @@ base_url: "http://localhost:8808" supports_ttl: false # be honest; expiry checks report NOT_TESTED timeout_seconds: 30 + convergence_timeout_seconds: 30 auth_token_env: MEMORYCHECK_HTTP_TOKEN # optional bearer token env var """ @@ -32,7 +33,7 @@ import yaml from ..ledger import Scope -from .base import AdapterError, MemoryAdapter, QueryResult +from .base import AdapterError, MemoryAdapter, QueryResult, poll_until class HTTPAdapterError(AdapterError): @@ -47,7 +48,20 @@ def __init__(self, config_path: str) -> None: self.base_url: str = cfg["base_url"].rstrip("/") self.supports_ttl = bool(cfg.get("supports_ttl", False)) self.timeout = float(cfg.get("timeout_seconds", 30)) + self.convergence_timeout = float( + cfg.get("convergence_timeout_seconds", self.timeout) + ) + self.convergence_interval = float( + cfg.get("convergence_interval_seconds", 0.25) + ) self._token = os.environ.get(cfg.get("auth_token_env", ""), "") + self._known_values: dict[tuple[str, str, str], set[str]] = {} + # Doctor performs its own diagnostic polling so it can distinguish a + # bad endpoint contract from slow convergence. Normal scenario runs + # leave this enabled and cannot race accepted mutations. + self.confirm_mutations = True + self.last_write_convergence_seconds: float | None = None + self.last_delete_convergence_seconds: float | None = None self.name = f"http:{self.base_url}" # ------------------------------------------------------------- plumbing @@ -80,15 +94,52 @@ def _ids(scope: Scope) -> dict: def reset(self, namespace: str) -> None: self._post("reset", {"namespace": namespace}) + self._known_values.clear() def write(self, scope: Scope, key: str, value: str, ttl_steps: int | None = None) -> None: payload = {**self._ids(scope), "key": key, "value": value} if ttl_steps is not None: payload["ttl_steps"] = ttl_steps self._post("write", payload) + if not self.confirm_mutations: + self._known_values.setdefault(self._known_key(scope, key), set()).add(value) + return + converged, waited = poll_until( + lambda: self._query_has_value(scope, key, value), + timeout=self.convergence_timeout, + interval=self.convergence_interval, + ) + self.last_write_convergence_seconds = waited + if not converged: + raise HTTPAdapterError( + f"POST /write accepted {key!r}, but /query did not expose " + f"{value!r} within {waited:.1f}s; refusing to let the scenario " + "race an unconfirmed write" + ) + self._known_values.setdefault(self._known_key(scope, key), set()).add(value) def delete(self, scope: Scope, key: str) -> None: + known = set(self._known_values.get(self._known_key(scope, key), set())) self._post("delete", {**self._ids(scope), "key": key}) + if not self.confirm_mutations: + self._known_values.pop(self._known_key(scope, key), None) + return + if not known: + self.last_delete_convergence_seconds = 0.0 + return + converged, waited = poll_until( + lambda: not any(self._query_has_value(scope, key, value) for value in known), + timeout=self.convergence_timeout, + interval=self.convergence_interval, + ) + self.last_delete_convergence_seconds = waited + if not converged: + raise HTTPAdapterError( + f"POST /delete accepted {key!r}, but /query still exposed a " + f"known value after {waited:.1f}s; refusing to score the " + "provider on a delete that has not converged" + ) + self._known_values.pop(self._known_key(scope, key), None) def advance_time(self, steps: int) -> None: if self.supports_ttl: @@ -110,3 +161,19 @@ def query(self, scope: Scope, prompt: str, seed: int = 0) -> QueryResult: f"query 'answer' must be a string, got {type(answer).__name__}" ) return QueryResult(answer=answer, retrieved=list(data.get("retrieved", []))) + + @staticmethod + def _known_key(scope: Scope, key: str) -> tuple[str, str, str]: + return (scope.tenant_id, scope.user_id, key) + + def _query_has_value(self, scope: Scope, key: str, value: str) -> bool: + """Confirm a mutation through the same /query surface the pack reads. + + Raw `retrieved` hits may establish convergence even when the answering + layer paraphrases. They are evidence for mutation arrival only; the + oracle still grades exclusively from the answer. + """ + result = self.query(scope, f"What is the {key} for this user?") + if value in result.answer: + return True + return value in " ".join(str(row) for row in result.retrieved) diff --git a/src/memorycheck/adapters/langgraph.py b/src/memorycheck/adapters/langgraph.py index a507d37..4df26f6 100644 --- a/src/memorycheck/adapters/langgraph.py +++ b/src/memorycheck/adapters/langgraph.py @@ -43,7 +43,7 @@ from pathlib import Path from ..ledger import Scope -from .base import AdapterError, MemoryAdapter, QueryResult +from .base import AdapterError, MemoryAdapter, QueryResult, encode_identifier # `search` defaults to limit=10 and `list_namespaces` to limit=100 — both # silently truncate. Ceilings here are safety bounds, not relevance filters: @@ -57,10 +57,6 @@ def _tokens(text: str) -> set[str]: return set(re.sub(r"[^a-z0-9]+", " ", text.lower()).split()) -def _slug(text: str) -> str: - return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") or "x" - - class LangGraphAdapter(MemoryAdapter): supports_ttl = False # see module docstring — observed, not assumed @@ -98,7 +94,11 @@ def __init__(self, spec: str = "") -> None: # --------------------------------------------------------------- scoping def _ns(self, scope: Scope) -> tuple[str, ...]: - return (self._root, _slug(scope.tenant_id), _slug(scope.user_id)) + return ( + self._root, + encode_identifier(scope.tenant_id), + encode_identifier(scope.user_id), + ) def _namespaces(self) -> list[tuple[str, ...]]: """Every namespace under this run's root, following pagination. @@ -129,7 +129,7 @@ def reset(self, namespace: str) -> None: Scoped by construction: SqliteStore persists, so clearing the store globally would delete rows belonging to whoever owns the file. """ - self._root = f"memorycheck-{_slug(namespace or 'default')}" + self._root = f"memorycheck-{encode_identifier(namespace or 'default')}" for ns in self._namespaces(): for item in self._store.search(ns, limit=_SEARCH_LIMIT): self._store.delete(ns, item.key) diff --git a/src/memorycheck/adapters/mem0.py b/src/memorycheck/adapters/mem0.py index 7e35c39..8df9e89 100644 --- a/src/memorycheck/adapters/mem0.py +++ b/src/memorycheck/adapters/mem0.py @@ -16,8 +16,9 @@ key's memories again (Mem0 exposes no key concept of its own). * Scope maps onto Mem0 identifiers. ``tenant_id``/``user_id`` are folded into a single Mem0 ``user_id``; every write is also tagged with an ``app_id`` - derived from the run namespace, so ``reset()`` can wipe an entire run in one - ``delete_all``. Ids are prefixed with the namespace so runs never collide. + derived from a losslessly encoded namespace containing a unique invocation + id, so ``reset()`` can wipe one scenario run in one ``delete_all`` without + merging punctuation-distinct scopes or colliding with another invocation. * ``supports_ttl = False``. Mem0 expiry is wall-clock; ours is logical time, so it is genuinely inexpressible here and expiry checks must report NOT_TESTED — the honesty model, not a gap to paper over. @@ -30,13 +31,14 @@ from __future__ import annotations import os -import re import sys import time import uuid from ..ledger import Scope -from .base import AdapterError, MemoryAdapter, QueryResult, poll_until +from .base import ( + AdapterError, MemoryAdapter, QueryResult, encode_identifier, poll_until, +) # Cap on how many memories a scoped query pulls back. Scenarios hold a handful # of facts per scope; this is a safety ceiling, not a relevance filter — we do @@ -62,11 +64,6 @@ _SENTINEL_ATTEMPT_TIMEOUT = 5.0 -def _slug(text: str) -> str: - """Collapse arbitrary text to a Mem0-safe identifier fragment.""" - return re.sub(r"[^a-z0-9]+", "_", text.lower()).strip("_") or "x" - - def _call(what: str, fn, *args, **kwargs): """Invoke a Mem0 SDK call, turning any failure into a legible AdapterError. @@ -113,12 +110,15 @@ def __init__(self) -> None: def _app_id(self) -> str: """One id per run namespace — the handle reset() wipes.""" - return f"mc_{_slug(self._namespace)}" + return f"mc_{encode_identifier(self._namespace)}" def _user_id(self, scope: Scope) -> str: """Fold namespace + tenant + user into a single Mem0 identifier, so a different run, tenant, or user can never see this scope's memories.""" - return f"{self._app_id()}__{_slug(scope.tenant_id)}__{_slug(scope.user_id)}" + return ( + f"{self._app_id()}__{encode_identifier(scope.tenant_id)}" + f"__{encode_identifier(scope.user_id)}" + ) # ------------------------------------------------------------- lifecycle diff --git a/src/memorycheck/cli.py b/src/memorycheck/cli.py index a2cd767..3084244 100644 --- a/src/memorycheck/cli.py +++ b/src/memorycheck/cli.py @@ -5,7 +5,8 @@ memorycheck run scenarios/ --adapter reference:naive --report-md out/report.md memorycheck list-adapters -`run` exits non-zero when the gate fails — wire it straight into CI. +`run` exits non-zero when the gate fails or is inconclusive — wire it straight +into CI. """ from __future__ import annotations @@ -24,6 +25,13 @@ from .scenario import load_dir +def _positive_int(value: str) -> int: + parsed = int(value) + if parsed < 1: + raise argparse.ArgumentTypeError("must be >= 1") + return parsed + + def _cmd_validate(args: argparse.Namespace) -> int: scenarios = load_dir(args.scenarios) warnings = 0 @@ -117,6 +125,7 @@ def _cmd_run(args: argparse.Namespace) -> int: unverified=getattr(adapter, "unverified", False), unverified_note=getattr(adapter, "unverified_note", ""), answering_layer=answering_layer, + run_id=suite["run_id"], ) print_summary(summary) write_reports(summary, args.report_json, args.report_md) @@ -124,7 +133,7 @@ def _cmd_run(args: argparse.Namespace) -> int: print(f" evidence (json): {args.report_json}") if args.report_md: print(f" evidence (md): {args.report_md}") - return 1 if summary["gate"]["verdict"] == "FAIL" else 0 + return 0 if summary["gate"]["verdict"] == "PASS" else 1 def _cmd_doctor(args: argparse.Namespace) -> int: @@ -162,7 +171,8 @@ def main(argv: list[str] | None = None) -> None: p_run.add_argument("scenarios", help="scenario file or directory") p_run.add_argument("--adapter", default="reference:strict", help="adapter spec") p_run.add_argument("--judge", default="deterministic", help="usage judge") - p_run.add_argument("--seeds", type=int, default=1, help="repeat runs per scenario") + p_run.add_argument("--seeds", type=_positive_int, default=1, + help="repeat runs per scenario (must be >= 1)") p_run.add_argument("--no-baseline", action="store_true", help="skip the no-memory baseline (utility delta reports NOT TESTED)") p_run.add_argument("--fail-on", choices=["p1", "p2"], default="p2", diff --git a/src/memorycheck/doctor.py b/src/memorycheck/doctor.py index 6ffe2a2..dfa504c 100644 --- a/src/memorycheck/doctor.py +++ b/src/memorycheck/doctor.py @@ -118,6 +118,24 @@ def _wait_visible( def run_doctor(adapter: MemoryAdapter, timeout: float = 30.0) -> DoctorReport: + """Run conformance diagnostics without hiding their failure mechanism. + + The HTTP runner normally confirms every mutation before returning. Doctor + already owns the equivalent polling and needs to observe endpoint shape, + convergence and soft-delete failures separately, so it temporarily + disables that adapter-level guard and restores it on exit. + """ + confirmation = getattr(adapter, "confirm_mutations", None) + if confirmation is not None: + adapter.confirm_mutations = False + try: + return _run_doctor(adapter, timeout) + finally: + if confirmation is not None: + adapter.confirm_mutations = confirmation + + +def _run_doctor(adapter: MemoryAdapter, timeout: float = 30.0) -> DoctorReport: report = DoctorReport() add = report.checks.append prompt = f"What is the {_KEY} for this user?" @@ -172,6 +190,12 @@ def run_doctor(adapter: MemoryAdapter, timeout: float = 30.0) -> DoctorReport: # ---- C4 write is readable, and how fast ---------------------------- seen, waited = _wait_visible(adapter, _A, _V1, prompt, timeout) + # The HTTP adapter now confirms its own write before returning, so the + # doctor's follow-up read is immediate. Preserve the actual write→read + # latency measured inside that adapter rather than reporting a false zero. + adapter_waited = getattr(adapter, "last_write_convergence_seconds", None) + if adapter_waited is not None: + waited = max(waited, adapter_waited) if seen: report.convergence_seconds = waited report.suggested_timeout = max(5.0, round(waited * 4 + 2, 1)) @@ -199,12 +223,12 @@ def run_doctor(adapter: MemoryAdapter, timeout: float = 30.0) -> DoctorReport: "re-run. That is the configuration the full lifecycle is designed " "for, and it exercises every check. Wiring your full agent into " "/query is supported only once the semantic judge is calibrated. " - "If you must run as-is, use --fail-on p1 and read the LIMITATIONS " - "block on the report: missing_current_fact is reported NOT TESTED " - "rather than FAIL, memory_utility_delta is unavailable so the run " - "cannot detect a system that passes by forgetting everything, and " - "while P1 findings stay trustworthy, a clean P1 result is weaker " - "evidence because a paraphrased leak would also go undetected.", + "If you must run as-is, read the LIMITATIONS block on the report: " + "exact matched violations still fail, but clean lifecycle absence " + "checks and missing_current_fact report NOT TESTED because a " + "paraphrased forbidden value could go undetected. The utility " + "delta is unavailable and the gate is INCONCLUSIVE with a " + "non-zero exit.", )) else: report.answering_layer = UNKNOWN diff --git a/src/memorycheck/oracle.py b/src/memorycheck/oracle.py index 108e671..93028e4 100644 --- a/src/memorycheck/oracle.py +++ b/src/memorycheck/oracle.py @@ -147,6 +147,17 @@ def worst_label(value: str) -> str: obs.seed, ) ) + elif paraphrasing: + # Exact-match judging can prove a violation when the literal value + # appears, but it cannot prove absence when this answering layer is + # known to paraphrase. A clean-looking answer is therefore + # unmeasurable, not a PASS — especially for the P1 classes. + findings.append( + Finding( + obs.scenario_id, obs.step_index, check, SEVERITY[check], + NOT_TESTED, PARAPHRASE_NOT_TESTED, obs.seed, + ) + ) else: findings.append( Finding( diff --git a/src/memorycheck/report.py b/src/memorycheck/report.py index 68fbeef..2fcda0e 100644 --- a/src/memorycheck/report.py +++ b/src/memorycheck/report.py @@ -32,8 +32,9 @@ PARAPHRASE_LIMITATIONS = [ "memory_utility_delta is unavailable, so this run CANNOT detect a system " "passing by forgetting everything.", - "P1 findings remain trustworthy, but a clean P1 result is weaker evidence: " - "a paraphrased leaked value would not be detected either.", + "The deterministic judge cannot evidence absence under paraphrase. Clean " + "scope, deletion, stale and expiry checks report NOT TESTED, and the gate " + "is INCONCLUSIVE rather than PASS.", "For a full evidence pack, point /query at the store or a quoting " "answering layer. That is the supported configuration.", ] @@ -41,6 +42,7 @@ RATE_CHECKS = ("scope_leakage", "deletion_residue", "stale_reuse", "expiry_leak") _SEV_RANK = {"p1": 1, "p2": 2} +INCONCLUSIVE = "INCONCLUSIVE" def _rate(findings: list[Finding], check: str) -> dict: @@ -63,6 +65,7 @@ def build_summary( seeds: int, scenario_count: int, fail_on: str, + run_id: str, unverified: bool = False, unverified_note: str = "", answering_layer: str = "unknown", @@ -82,6 +85,26 @@ def build_summary( for f in findings if f.status == FAIL and _SEV_RANK[f.severity.lower()] <= threshold ] + inconclusive_reasons: list[str] = [] + if unverified: + inconclusive_reasons.append( + "adapter has not been verified against its live provider" + ) + if answering_layer == PARAPHRASING: + inconclusive_reasons.append( + "deterministic judge cannot evidence absence under paraphrasing" + ) + all_metrics_not_tested = ( + cfa is None + and utility_delta is None + and all(entry["rate"] is None for entry in checks.values()) + ) + if all_metrics_not_tested: + inconclusive_reasons.append("every metric is NOT TESTED") + + gate_verdict = ( + FAIL if blocking else INCONCLUSIVE if inconclusive_reasons else PASS + ) # Seed stability: same (scenario, step, check) failing in some seeds only. by_key: dict[tuple, set[str]] = {} for f in findings: @@ -91,6 +114,7 @@ def build_summary( return { "tool": {"name": "memorycheck", "version": __version__}, "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "run_id": run_id, "adapter": adapter_name, # Travels with the evidence: a figure produced by an adapter that has # never touched its provider must not be quotable as a measurement, @@ -113,8 +137,9 @@ def build_summary( "scenarios": scenario_count, "gate": { "fail_on": fail_on.upper(), - "verdict": FAIL if blocking else PASS, + "verdict": gate_verdict, "blocking_findings": len(blocking), + "inconclusive_reasons": inconclusive_reasons, }, "metrics": { "current_fact_accuracy": {"satisfied": ok, "total": total, "value": cfa}, @@ -171,16 +196,17 @@ def to_markdown(summary: dict) -> str: lines += [f"> {i}. {text}" for i, text in enumerate(summary["limitations"], 1)] lines += [ ">", - "> `missing_current_fact` is reported NOT TESTED throughout rather " - "than FAIL, because the deterministic judge matches exact values " - "and cannot verify a paraphrase either way.", + "> Clean lifecycle checks are reported NOT TESTED rather than PASS, " + "because the deterministic judge matches exact values and cannot " + "evidence absence under paraphrase. The gate is INCONCLUSIVE.", "", ] lines += [ + f"- **Run ID:** `{summary.get('run_id', 'unknown')}`", f"- **Adapter:** `{summary['adapter']}` **Judge:** `{summary['judge']}`", f"- **Answering layer:** `{summary.get('answering_layer', 'unknown')}`" - + (" — values are paraphrased, so `missing_current_fact` reports false " - "failures here; the P1 checks remain sound. Prefer `--fail-on p1`." + + (" — values are paraphrased, so absence cannot be evidenced and the " + "gate is INCONCLUSIVE." if summary.get("answering_layer") == "paraphrasing" else ""), f"- **Scenarios:** {summary['scenarios']} **Seeds:** {summary['seeds']} " f"**Generated:** {summary['generated_at']}", @@ -198,6 +224,10 @@ def to_markdown(summary: dict) -> str: f"| Memory utility delta | — | {ud_str} |", "", ] + if summary["gate"].get("inconclusive_reasons"): + lines += ["## Why the gate is inconclusive", ""] + lines += [f"- {reason}" for reason in summary["gate"]["inconclusive_reasons"]] + lines.append("") failures = [f for f in summary["findings"] if f["status"] == FAIL] skipped = [f for f in summary["findings"] if f["status"] == NOT_TESTED] if failures: @@ -236,6 +266,7 @@ def print_summary(summary: dict) -> None: m = summary["metrics"] print(f"\nmemorycheck {__version__} — adapter={summary['adapter']} " f"judge={summary['judge']} scenarios={summary['scenarios']} seeds={summary['seeds']}") + print(f" run_id={summary.get('run_id', 'unknown')}") cfa = m["current_fact_accuracy"] rows = [ ("current_fact_accuracy", @@ -258,6 +289,8 @@ def print_summary(summary: dict) -> None: print(f" {i}. {text}") print(f"\n GATE [fail on <= {summary['gate']['fail_on']}]: {summary['gate']['verdict']} " f"({summary['gate']['blocking_findings']} blocking findings)") + for reason in summary["gate"].get("inconclusive_reasons", []): + print(f" inconclusive: {reason}") if summary.get("adapter_unverified"): print(" ** UNVERIFIED ADAPTER — never run against the live provider; " "do not quote these results. **") diff --git a/src/memorycheck/runner.py b/src/memorycheck/runner.py index db1dbb7..f715c21 100644 --- a/src/memorycheck/runner.py +++ b/src/memorycheck/runner.py @@ -9,6 +9,7 @@ from __future__ import annotations import time +import uuid from dataclasses import dataclass, field from .adapters.base import MemoryAdapter, NullAdapter @@ -38,6 +39,7 @@ class ScenarioRun: seed: int observations: list[QueryObservation] = field(default_factory=list) ttl_not_supported: bool = False + run_id: str = "" @dataclass @@ -143,13 +145,18 @@ def run_scenario( judge: Judge, seed: int = 0, progress: RunProgress | None = None, + run_id: str | None = None, ) -> ScenarioRun: + run_id = run_id or uuid.uuid4().hex ledger = GroundTruthLedger() started = time.perf_counter() ops_before = len(progress.ops) if progress else 0 _timed(progress, scenario.id, seed, -1, "reset", - adapter.reset, namespace=f"memorycheck::{scenario.id}::{seed}") - run = ScenarioRun(scenario=scenario, adapter_name=adapter.name, seed=seed) + adapter.reset, + namespace=f"memorycheck::{run_id}::{scenario.id}::{seed}") + run = ScenarioRun( + scenario=scenario, adapter_name=adapter.name, seed=seed, run_id=run_id, + ) if scenario.uses_ttl and not adapter.supports_ttl: run.ttl_not_supported = True @@ -250,6 +257,7 @@ def run_suite( seeds: int = 1, baseline: bool = True, progress: RunProgress | None = None, + run_id: str | None = None, ) -> dict: """Run every scenario across N seeds, plus (optionally) the no-memory baseline used for the Memory Utility Delta. Returns raw runs for the @@ -259,16 +267,21 @@ def run_suite( suite runs — the caller keeps it if the suite raises, which is the whole point. The baseline is deliberately NOT instrumented: it never touches the provider, so its timings would flatten the curve with in-process noise.""" + if seeds < 1: + raise ValueError("seeds must be >= 1") + run_id = run_id or uuid.uuid4().hex runs: list[ScenarioRun] = [] for scenario in scenarios: for seed in range(seeds): runs.append(run_scenario(scenario, adapter, judge, seed=seed, - progress=progress)) + progress=progress, run_id=run_id)) baseline_runs: list[ScenarioRun] = [] if baseline: null_adapter = NullAdapter() for scenario in scenarios: - baseline_runs.append(run_scenario(scenario, null_adapter, judge, seed=0)) + baseline_runs.append( + run_scenario(scenario, null_adapter, judge, seed=0, run_id=run_id) + ) - return {"runs": runs, "baseline_runs": baseline_runs} + return {"runs": runs, "baseline_runs": baseline_runs, "run_id": run_id} diff --git a/tests/test_convergence.py b/tests/test_convergence.py index 5a88ad4..55a2b53 100644 --- a/tests/test_convergence.py +++ b/tests/test_convergence.py @@ -12,12 +12,21 @@ import pytest -from memorycheck.adapters.base import AdapterError, poll_until +from memorycheck.adapters.base import ( + AdapterError, decode_identifier, encode_identifier, poll_until, +) from memorycheck.ledger import Scope ALICE = Scope("acme", "alice") +@pytest.mark.parametrize( + "value", ["", "tenant/a", "tenant-a", "tenant.a", "tenant a", "租户/α"] +) +def test_identifier_encoding_is_reversible(value): + assert decode_identifier(encode_identifier(value)) == value + + def test_already_converged_costs_no_wait(): # A synchronous provider must pay nothing for the guard. started = time.monotonic() diff --git a/tests/test_doctor.py b/tests/test_doctor.py index 0a36729..72f78d1 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -248,8 +248,8 @@ def test_detects_a_paraphrasing_answering_layer(serve): # A paraphrasing agent is legitimate, so it must not fail the contract. assert report.ok, [c.title for c in report.failed] fix = next(c.fix for c in report.checks if c.id == "answering_layer") - assert "--fail-on p1" in fix assert "missing_current_fact" in fix + assert "INCONCLUSIVE" in fix def test_paraphrasing_without_retrieved_is_indistinguishable_from_a_lost_write(serve): @@ -329,18 +329,27 @@ def test_paraphrasing_degrades_missing_current_fact_to_not_tested(serve): assert mcf, "the scenario has must_use checks" assert all(f.status == NOT_TESTED for f in mcf), "must not FAIL a paraphraser" assert all(f.detail == PARAPHRASE_NOT_TESTED for f in mcf) + absence_checks = [ + f for f in findings + if f.check in {"scope_leakage", "deletion_residue", "stale_reuse", "expiry_leak"} + ] + assert absence_checks, "the scenario has stale/deleted opportunities" + assert all(f.status == NOT_TESTED for f in absence_checks) + assert all(f.detail == PARAPHRASE_NOT_TESTED for f in absence_checks) summary = build_summary(findings, [], adapter_name="http:x", judge_name="deterministic-v0", seeds=1, - scenario_count=1, fail_on="p2", answering_layer=layer) + scenario_count=1, fail_on="p2", run_id=suite["run_id"], + answering_layer=layer) assert summary["answering_layer"] == PARAPHRASING assert summary["limitations"] == PARAPHRASE_LIMITATIONS - assert summary["gate"]["verdict"] == "PASS", "no false blocking findings" + assert summary["gate"]["verdict"] == "INCONCLUSIVE" + assert "paraphrasing" in " ".join(summary["gate"]["inconclusive_reasons"]) md = to_markdown(summary) assert "LIMITATIONS OF THIS RUN" in md assert "forgetting everything" in md - assert "weaker evidence" in md + assert "INCONCLUSIVE" in md # prominent: above the metrics table, not a footnote assert md.index("LIMITATIONS OF THIS RUN") < md.index("| Check |") @@ -349,7 +358,7 @@ def test_quoting_runs_carry_no_limitations_block(serve): from memorycheck.report import build_summary, to_markdown summary = build_summary([], [], adapter_name="x", judge_name="j", seeds=1, - scenario_count=1, fail_on="p2", + scenario_count=1, fail_on="p2", run_id="quoting-test", answering_layer=QUOTING) assert summary["limitations"] == [] assert "LIMITATIONS" not in to_markdown(summary) @@ -361,3 +370,5 @@ def test_doctor_warn_recommends_the_store_first_path(serve): assert "RECOMMENDED" in fix assert "point /query at" in fix assert "only once the semantic judge is calibrated" in fix + assert "INCONCLUSIVE" in fix + assert "non-zero exit" in fix diff --git a/tests/test_end_to_end.py b/tests/test_end_to_end.py index fbb0fd4..09a16d7 100644 --- a/tests/test_end_to_end.py +++ b/tests/test_end_to_end.py @@ -1,9 +1,12 @@ from pathlib import Path +import pytest + from memorycheck.adapters import load_adapter +from memorycheck.cli import main from memorycheck.judge import load_judge from memorycheck.oracle import FAIL, evaluate -from memorycheck.report import build_summary, to_markdown +from memorycheck.report import INCONCLUSIVE, build_summary, to_markdown from memorycheck.runner import run_suite from memorycheck.scenario import load_dir @@ -25,6 +28,7 @@ def run(adapter_spec: str) -> dict: findings, baseline, adapter_name=adapter_spec, judge_name="deterministic-v0", seeds=2, scenario_count=len(scenarios), fail_on="p2", + run_id=suite["run_id"], ) @@ -61,11 +65,12 @@ def test_unverified_flag_is_stamped_into_the_report(): # quotable as a measurement, however it is copied out of the evidence. summary = build_summary( [], [], adapter_name="zep", judge_name="deterministic-v0", - seeds=1, scenario_count=1, fail_on="p2", + seeds=1, scenario_count=1, fail_on="p2", run_id="unverified-test", unverified=True, unverified_note="never run against live Zep", ) assert summary["adapter_unverified"] is True assert summary["adapter_unverified_note"] == "never run against live Zep" + assert summary["gate"]["verdict"] == INCONCLUSIVE md = to_markdown(summary) assert "UNVERIFIED ADAPTER" in md assert "never run against live Zep" in md @@ -88,3 +93,88 @@ def test_memory_utility_delta_is_positive_for_working_memory(): summary = run("reference:strict") delta = summary["metrics"]["memory_utility_delta"]["delta"] assert delta is not None and delta > 0 + + +def test_all_metrics_not_tested_is_inconclusive(): + summary = build_summary( + [], [], adapter_name="empty", judge_name="deterministic-v0", + seeds=1, scenario_count=1, fail_on="p2", run_id="empty-test", + ) + assert summary["gate"]["verdict"] == INCONCLUSIVE + assert "every metric is NOT TESTED" in summary["gate"]["inconclusive_reasons"] + + +def test_cli_rejects_zero_seeds(capsys): + with pytest.raises(SystemExit) as exc: + main(["run", "scenarios", "--seeds", "0"]) + assert exc.value.code == 2 + assert "must be >= 1" in capsys.readouterr().err + + +def test_cli_exits_nonzero_for_an_inconclusive_unverified_run(monkeypatch): + adapter = load_adapter("reference:strict") + adapter.unverified = True + adapter.unverified_note = "offline mapping only" + monkeypatch.setattr("memorycheck.cli.load_adapter", lambda _spec: adapter) + + with pytest.raises(SystemExit) as exc: + main([ + "run", str(SCENARIOS), "--adapter", "unverified-test", + "--seeds", "1", + ]) + assert exc.value.code == 1 + + +def test_library_rejects_zero_seeds(): + scenarios = load_dir(SCENARIOS) + with pytest.raises(ValueError, match="seeds must be >= 1"): + run_suite( + scenarios, load_adapter("reference:strict"), + load_judge("deterministic"), seeds=0, + ) + + +def test_each_suite_invocation_has_a_unique_namespaced_run_id(): + class RecordingAdapter: + name = "recording" + supports_ttl = True + + def __init__(self): + self.namespaces = [] + self.inner = load_adapter("reference:strict") + + def reset(self, namespace): + self.namespaces.append(namespace) + return self.inner.reset(namespace) + + def write(self, *args, **kwargs): + return self.inner.write(*args, **kwargs) + + def delete(self, *args, **kwargs): + return self.inner.delete(*args, **kwargs) + + def query(self, *args, **kwargs): + return self.inner.query(*args, **kwargs) + + def advance_time(self, *args, **kwargs): + return self.inner.advance_time(*args, **kwargs) + + adapter = RecordingAdapter() + scenario = load_dir(SCENARIOS)[:1] + judge = load_judge("deterministic") + first = run_suite(scenario, adapter, judge, baseline=False) + second = run_suite(scenario, adapter, judge, baseline=False) + + assert first["run_id"] != second["run_id"] + assert first["run_id"] in adapter.namespaces[0] + assert second["run_id"] in adapter.namespaces[1] + assert adapter.namespaces[0] != adapter.namespaces[1] + + +def test_run_id_is_stamped_into_report_header(): + summary = build_summary( + [], [], adapter_name="empty", judge_name="deterministic-v0", + seeds=1, scenario_count=1, fail_on="p2", run_id="run-123", + ) + assert summary["run_id"] == "run-123" + assert "**Run ID:** `run-123`" in to_markdown(summary) diff --git a/tests/test_http_adapter.py b/tests/test_http_adapter.py index 0d7e72b..8437b8b 100644 --- a/tests/test_http_adapter.py +++ b/tests/test_http_adapter.py @@ -4,18 +4,19 @@ import json import threading +import time from http.server import BaseHTTPRequestHandler, HTTPServer from pathlib import Path import pytest -from memorycheck.adapters.http import HTTPAdapter +from memorycheck.adapters.http import HTTPAdapter, HTTPAdapterError from memorycheck.adapters.reference import ReferenceAdapter from memorycheck.judge import load_judge from memorycheck.ledger import Scope from memorycheck.oracle import FAIL, evaluate from memorycheck.runner import run_suite -from memorycheck.scenario import load_dir +from memorycheck.scenario import Scenario, Step, load_dir SCENARIOS = Path(__file__).resolve().parents[1] / "scenarios" @@ -23,6 +24,8 @@ class ShimHandler(BaseHTTPRequestHandler): + store = STORE + def log_message(self, *args): # silence test output pass @@ -32,18 +35,18 @@ def do_POST(self): route = self.path.strip("/") out = {"ok": True} if route == "reset": - STORE.reset(payload.get("namespace", "")) + self.store.reset(payload.get("namespace", "")) elif route == "write": - STORE.write( + self.store.write( Scope(payload["tenant_id"], payload["user_id"]), payload["key"], payload["value"], payload.get("ttl_steps"), ) elif route == "delete": - STORE.delete(Scope(payload["tenant_id"], payload["user_id"]), payload["key"]) + self.store.delete(Scope(payload["tenant_id"], payload["user_id"]), payload["key"]) elif route == "advance_time": - STORE.advance_time(payload["steps"]) + self.store.advance_time(payload["steps"]) elif route == "query": - result = STORE.query( + result = self.store.query( Scope(payload["tenant_id"], payload["user_id"]), payload["prompt"], payload.get("seed", 0), ) @@ -61,12 +64,63 @@ def do_POST(self): @pytest.fixture() -def shim_url(): - server = HTTPServer(("127.0.0.1", 0), ShimHandler) - thread = threading.Thread(target=server.serve_forever, daemon=True) - thread.start() - yield f"http://127.0.0.1:{server.server_address[1]}" - server.shutdown() +def serve_store(): + servers = [] + + def _serve(store): + class BoundHandler(ShimHandler): + pass + + BoundHandler.store = store + server = HTTPServer(("127.0.0.1", 0), BoundHandler) + threading.Thread(target=server.serve_forever, daemon=True).start() + servers.append(server) + return f"http://127.0.0.1:{server.server_address[1]}" + + yield _serve + for server in servers: + server.shutdown() + + +@pytest.fixture() +def shim_url(serve_store): + return serve_store(STORE) + + +class DelayedReferenceStore: + """Accept mutations immediately; expose them to query after a short lag.""" + + def __init__(self, delay=0.15): + self.store = ReferenceAdapter(mode="strict") + self.delay = delay + self.pending = [] + + def _flush(self): + now = time.monotonic() + ready = [item for item in self.pending if item[0] <= now] + self.pending = [item for item in self.pending if item[0] > now] + for _, operation, args in ready: + getattr(self.store, operation)(*args) + + def reset(self, namespace): + self.pending.clear() + self.store.reset(namespace) + + def write(self, scope, key, value, ttl_steps=None): + self.pending.append( + (time.monotonic() + self.delay, "write", (scope, key, value, ttl_steps)) + ) + + def delete(self, scope, key): + self.pending.append((time.monotonic() + self.delay, "delete", (scope, key))) + + def advance_time(self, steps): + self._flush() + self.store.advance_time(steps) + + def query(self, scope, prompt, seed=0): + self._flush() + return self.store.query(scope, prompt, seed) def test_full_suite_over_http(shim_url, tmp_path): @@ -79,3 +133,52 @@ def test_full_suite_over_http(shim_url, tmp_path): suite = run_suite(scenarios, adapter, load_judge("deterministic"), seeds=1, baseline=False) findings = evaluate(suite["runs"]) assert [f for f in findings if f.status == FAIL] == [] + + +def test_eventually_consistent_writes_and_deletes_converge_before_runner_continues( + serve_store, tmp_path, +): + url = serve_store(DelayedReferenceStore(delay=0.15)) + config = tmp_path / "delayed.yaml" + config.write_text( + f"base_url: {url}\nsupports_ttl: false\ntimeout_seconds: 2\n" + "convergence_timeout_seconds: 1\nconvergence_interval_seconds: 0.02\n" + ) + adapter = HTTPAdapter(str(config)) + scope = Scope("acme", "alice") + scenario = Scenario( + id="delayed-http", + title="HTTP mutations must converge before the runner advances", + tags=[], + subject=scope, + steps=[ + Step("write", scope, key="plan", value="scale-2026", index=0), + Step("query", scope, prompt="Which plan?", + expect={"must_use": ["plan"]}, index=1), + Step("delete", scope, key="plan", index=2), + Step("query", scope, prompt="Which plan?", index=3), + ], + path="test", + ) + + suite = run_suite( + [scenario], adapter, load_judge("deterministic"), seeds=1, baseline=False + ) + findings = evaluate(suite["runs"]) + assert [f for f in findings if f.status == FAIL] == [] + assert adapter.last_write_convergence_seconds >= 0.1 + assert adapter.last_delete_convergence_seconds >= 0.1 + + +def test_write_convergence_timeout_aborts_instead_of_scoring_a_race( + serve_store, tmp_path, +): + url = serve_store(DelayedReferenceStore(delay=1.0)) + config = tmp_path / "timeout.yaml" + config.write_text( + f"base_url: {url}\nsupports_ttl: false\ntimeout_seconds: 2\n" + "convergence_timeout_seconds: 0.05\nconvergence_interval_seconds: 0.01\n" + ) + adapter = HTTPAdapter(str(config)) + with pytest.raises(HTTPAdapterError, match="unconfirmed write"): + adapter.write(Scope("acme", "alice"), "plan", "scale-2026") diff --git a/tests/test_langgraph_adapter.py b/tests/test_langgraph_adapter.py index 45ce7b0..773a036 100644 --- a/tests/test_langgraph_adapter.py +++ b/tests/test_langgraph_adapter.py @@ -92,7 +92,13 @@ def test_scope_maps_to_a_namespace_tuple(adapter): adapter.reset("ns") assert adapter._ns(ALICE) != adapter._ns(BOB) assert adapter._ns(ALICE) != adapter._ns(OTHER_TENANT) - assert adapter._ns(ALICE)[1:] == ("acme", "alice") + + +def test_identifier_encoding_keeps_punctuation_variants_distinct(adapter): + adapter.reset("scope-collision") + tenant_ids = ["tenant/a", "tenant-a", "tenant.a", "tenant a"] + mapped = {adapter._ns(Scope(tenant, "alice")) for tenant in tenant_ids} + assert len(mapped) == len(tenant_ids) def test_write_then_query_returns_the_value(adapter): diff --git a/tests/test_mem0_adapter.py b/tests/test_mem0_adapter.py index 520dcbd..fe92adf 100644 --- a/tests/test_mem0_adapter.py +++ b/tests/test_mem0_adapter.py @@ -116,6 +116,14 @@ def test_identifiers_isolate_users_tenants_and_runs(offline_adapter): assert a._user_id(ALICE) != first_run, "runs must not collide" +def test_identifier_encoding_keeps_punctuation_variants_distinct(offline_adapter): + a = offline_adapter + a.reset("scope-collision") + tenant_ids = ["tenant/a", "tenant-a", "tenant.a", "tenant a"] + mapped = {a._user_id(Scope(tenant, "alice")) for tenant in tenant_ids} + assert len(mapped) == len(tenant_ids) + + def test_write_then_query_returns_the_value(offline_adapter): a = offline_adapter a.reset("ns")