From e2701914289fb547625e2f4e3e6cf5beb259aea0 Mon Sep 17 00:00:00 2001 From: Yinka Metrics <115735904+YinkaMetrics@users.noreply.github.com> Date: Wed, 5 Aug 2026 02:25:48 +0100 Subject: [PATCH 1/4] docs: gate live runs on complete provenance --- ADAPTER_PREFLIGHT.md | 24 ++++++- CLAUDE.md | 15 ++++- HANDOFF.md | 98 +++++++++++++++++++++++++++++ README.md | 15 ++--- diagnostics/readd_after_delete.py | 93 ++++++++++++++++++++++----- examples/report_mem0.json | 10 +-- examples/report_mem0.md | 31 +++++---- tests/test_diagnostic_provenance.py | 43 +++++++++++++ 8 files changed, 280 insertions(+), 49 deletions(-) create mode 100644 tests/test_diagnostic_provenance.py diff --git a/ADAPTER_PREFLIGHT.md b/ADAPTER_PREFLIGHT.md index 3c9fbf7..e81f32c 100644 --- a/ADAPTER_PREFLIGHT.md +++ b/ADAPTER_PREFLIGHT.md @@ -12,6 +12,22 @@ never by reading documentation. Record answers in `HANDOFF.md` before the first scenario. An adapter keeps `unverified = True` until every item has an observed answer. +For every live execution, create this provenance block before interpreting +the result: + +```text +status: verified | reported, unverified +executed_by: +environment: +search_quota_before: +search_quota_after: +results_file: +``` + +All four facts plus the filename are mandatory. If any is missing, the run is +`reported, unverified`: log it internally, but do not change README, a +published report, adapter verification status, or another gated claim from it. + --- ## 1. Quota and rate model — from live response headers @@ -26,8 +42,9 @@ incomplete every time so far. - [ ] What is the request rate limit, and does confirmation polling threaten it? *Why:* Mem0 meters `SEARCH` (1,000/period) separately from `ADD` -(10,000/period). A full run costs ~106 SEARCH — about a tenth of the period — -and confirmation reads are most of it. Zep meters ingestion credits instead, so +(10,000/period). The current full 15 × 2 path has a structural minimum of +~182 SEARCH — 91 per seed — and extra convergence polls can raise it. Zep +meters ingestion credits instead, so the same polling is free but a request-rate ceiling applies. Two providers, opposite constraints; neither was guessable. @@ -102,3 +119,6 @@ residue as leakage. project so far has been ours, not the provider's. - **Record what you could not test.** An untested assumption is not a passed one; say so in `HANDOFF.md` and keep `unverified = True`. +- **Bracket every live run with quota observations.** Estimates are planning + inputs. Only the provider-reported before/after pair closes the spend and + qualifies the run for gated publication. diff --git a/CLAUDE.md b/CLAUDE.md index 2bfb502..799a86e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -86,6 +86,14 @@ Status: v0.1, built in public. `pytest -q` must stay green after every change. just-mutated value (or its absence) with a bounded timeout. That poll is mutation confirmation only; the oracle never grades it. Non-convergence aborts rather than being laundered into PASS or scored as a provider bug. +11. **Live-run provenance is a publication gate.** Every live-provider run + records four facts contemporaneously: who executed it, the execution + environment, the provider's SEARCH quota immediately before and after, + and the results filename. Missing any field makes the run **reported, + unverified**. It may be logged internally, but it must not update README, + `examples/report_*`, clear an adapter's `unverified` flag, or support any + other gated claim. Estimates and reconstructed quota arithmetic are useful + diagnostics, never substitutes for the before/after pair. ## Conventions @@ -156,8 +164,11 @@ stops short has not finished its task and must say which of the four steps it stopped at. Blocked work still gets an entry: the blocker, what was established anyway, and what remains unverified. -**Any diagnostic run executed outside this environment must be promoted into -`HANDOFF.md` by hand, citing its results filename.** `diagnostics/results/` is +**Every live-provider run, here or elsewhere, must be promoted into +`HANDOFF.md` with its complete invariant-11 provenance block:** executor, +environment, SEARCH quota before, SEARCH quota after, and results filename. +If one is absent, label it **reported, unverified** and do not change a gated +file from it. `diagnostics/results/` is gitignored by design — those are open investigations whose output is not publishable until ruled on. The cost of that choice is that a run done on a laptop leaves no commit, no artefact and no trace here, so it is invisible to diff --git a/HANDOFF.md b/HANDOFF.md index e34d2f4..5e89454 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -3084,6 +3084,104 @@ External launch remains **HELD pending founder review**. --- +## 2026-08-05 — Live-run quota reconciliation and provenance gate + +### 1. What changed + +- Added invariant 11: every live-provider execution must contemporaneously + record `executed_by`, `environment`, SEARCH quota before, SEARCH quota after, + and the exact results filename. +- A run missing any field is **reported, unverified**. It may be retained in + this internal handoff, but it cannot update README, `examples/report_*`, an + adapter verification flag, or another gated claim. +- `diagnostics/readd_after_delete.py` now refuses a live execution without the + executor and environment, writes the initial provenance record before any + mutation, and records the quota pair and result filename in its JSON output. +- Withdrew the 2026-08-05 evidence promotion from README and + `examples/report_mem0.*`. The published artifact remains the 2026-07-27 run; + the later execution remains available internally as reported, unverified. +- Corrected the documented current Mem0 cost from the historical ~106 estimate + to a structural minimum of 91 SEARCH per seed, or 182 for 15 × 2, before + extra convergence polls. + +### 2. Quota accounting + +The founder-reported account movement is **885 → 599 = 286 SEARCH units**. +The surviving artifacts support this reconciliation: + +| Bucket | SEARCH units | Evidence and limit | +|---|---:|---| +| Seven-arm diagnostic | 41 | 33 in per-arm header deltas, plus 7 before-arm probes and the initial quota probe | +| Full 15 × 2 execution | at least 182 | structural minimum: 30 reset pre-reads, 60 write confirmations, 32 delete reads/confirmations, and 60 scenario queries | +| Unallocated before/around the first recorded checkpoint | at most 63 | arithmetic remainder; no complete run-specific quota pair exists, so this cannot be assigned more precisely | +| **Total** | **286** | 41 + 182 + up to 63 | + +The diagnostic's first recorded provider checkpoint was **822 remaining**. +Its result file reports arm deltas `5, 5, 5, 5, 5, 4, 4`; those 33 units +include the seven after-arm probes, while the seven before-arm probes and the +initial probe bring the diagnostic to 41. + +No surviving artifact evidences a repeat full 15 × 2 run in this work. The +full-run log contains no residue-reset/sentinel sequence because the unique +run namespace started empty; its 30 reset pre-reads are already included in +the 182 minimum. No separate post-checkpoint development run is evidenced. +Any extra convergence poll, standalone quota probe, or earlier development +call must therefore remain inside the unallocated remainder rather than being +invented as a specific cause. + +### 3. Provenance status of today's live executions + +**Seven-arm diagnostic** + +```text +status: reported, unverified +executed_by: Codex (OpenAI), on the founder's instruction +environment: Codex desktop on macOS; local writable clone; explicitly approved live network access +search_quota_before: no run-start value recorded; first provider checkpoint was 822 after the initial probe +search_quota_after: not contemporaneously recorded as a run-level value +results_file: diagnostics/results/readd_after_delete_1785890141.json +``` + +**Full 15 × 2 execution** + +```text +status: reported, unverified +executed_by: Codex (OpenAI), on the founder's instruction +environment: Codex desktop on macOS; local writable clone; explicitly approved live network access +search_quota_before: not contemporaneously recorded for this run +search_quota_after: not contemporaneously recorded for this run; 599 is only the founder-reported final account value across the combined work +results_file: diagnostics/results/full_mem0_20260805.json (with matching .md and .log) +``` + +Both executions are retained for investigation, but neither qualifies to +change a gated file under invariant 11. + +### 4. FOR STRATEGY + +- The 63-unit maximum remainder is closed as **unallocated, unverified**, not + silently attributed to repeat, sentinel, or development runs that the + artifacts do not prove. Do not spend more quota merely to reconstruct it. +- The next live run must begin only after the five-field provenance record is + prepared and must finish with the provider-reported quota pair in the same + results artifact. That single artifact should settle both evidentiary status + and spend without a follow-up reconstruction. +- Existing Mem0 findings are unaffected: the scenario-pack identifiers do not + slug-collide, the execution used `seeds=2`, there was no concurrency, and + the Mem0 answering layer is quoting. This is an evidence-provenance hold, + not a change to the previously published metric values. +- README/report edits remain gated-tier and PR #10 must retain + `needs-founder-review`; do not merge without explicit founder approval. + +### 5. Next + +1. Run offline validation only; do not spend additional Mem0 quota. +2. Push the provenance rule and accounting to draft PR #10. +3. Confirm the PR stays draft, labeled `needs-founder-review`, and unmerged. + +External launch remains **HELD pending founder review**. + +--- + ## 2026-08-05 — Draft PR #10 opened; founder gate active ### 1. What shipped diff --git a/README.md b/README.md index c0c0f41..0ec3bef 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ How the lifecycle maps onto Mem0: ### Result -**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** pinned for like-for-like reproduction, executed **2026-08-05 00:43 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`, run ID `1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or build identifier to clients, so the SDK pin and run date are the available provenance. This completed regeneration produced figures identical to the 2026-07-27 published run and the intervening reproductions. +**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** (latest release at time of run, published 2026-07-25), executed **2026-07-27 15:58 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`. The hosted platform exposes no version or build identifier to clients, so the run date is the only pin available on the service itself. Three consecutive runs — two on 2.0.14, one on 2.0.11 — produced identical figures. ``` current_fact_accuracy 100% (46/46) @@ -193,7 +193,7 @@ How the lifecycle maps onto Mem0: GATE [fail on <= P2]: FAIL (10 blocking findings) ``` -**Provenance.** The current evidence was regenerated from implementation commit `b81bf20`, after the adapter gained invariant-10 mutation confirmation, scope identifiers became lossless, and every invocation gained a unique namespace. All 170 operations completed. The metrics are unchanged from the earlier evidence: the harness corrections removed collision and race exposure without changing the Mem0 result. The separate reset diagnostics, including their intermittent namespace-delete behaviour, are tracked in [`HANDOFF.md`](HANDOFF.md). +**Provenance.** These figures were produced at commit `f230208`, before the adapter gained the write/delete convergence confirmation required by invariant 10. Per-scenario re-checks after that change returned identical results, and the same figures were reproduced across three runs including one on `mem0ai` 2.0.11. A later 15 × 2 execution on 2026-08-05 is recorded internally as **reported, unverified** because it lacks its own contemporaneous SEARCH-quota before/after pair. Under invariant 11, that execution is not a source for this published report. Tracked in [`HANDOFF.md`](HANDOFF.md). Mem0 holds the boundaries that carry the P1 severities: **no scope leakage in 22 opportunities and no deletion residue in 18** — deletes stopped the value influencing answers, and no user's or tenant's facts crossed into another's, including where two tenants share a `user_id` and where a fact is moved between tenants. Current-fact accuracy is perfect and every result is stable across seeds. @@ -240,13 +240,14 @@ Mem0 meters two independent counters, and this adapter spends the scarcer one. | `SEARCH` | 1,000 per billing period | every query, plus each read used to confirm a write or delete landed | | `ADD` | 10,000 per billing period | one per `write` | -Measured on the bundled 15-scenario pack: **~53 SEARCH units per seed**, so a -default 2-seed run costs **~106** — roughly a tenth of a 1,000-unit period. -Confirmation reads are the bulk of it, and they are not optional: without them -the harness races the provider and manufactures false failures (invariant 10). +The current adapter has a structural minimum of **91 SEARCH units per seed**: +15 reset pre-reads, 30 write confirmations, 16 delete reads/confirmations and +30 scenario queries. A default 2-seed run therefore costs **~182**, before any +extra convergence polls. Confirmation reads are not optional: without them the +harness races the provider and manufactures false failures (invariant 10). Budget accordingly if you wire this into CI. Running the full pack on every -commit will exhaust a 1,000-unit period in ~9 runs. The intended shape is a +commit will exhaust a 1,000-unit period in ~5 runs. The intended shape is a small smoke subset per commit and the full pack nightly or per release. Watch the `x-quota-remaining` response header; when the counter is exhausted the API returns `429` and the run aborts rather than reporting degraded numbers. diff --git a/diagnostics/readd_after_delete.py b/diagnostics/readd_after_delete.py index 441c15a..e2ccab1 100644 --- a/diagnostics/readd_after_delete.py +++ b/diagnostics/readd_after_delete.py @@ -19,7 +19,9 @@ `delete_all`. The earlier loss remains valid evidence that it can happen; the clean rerun does not erase it. The full 15 x 2 regeneration immediately after this diagnostic completed all 170 operations under a unique invocation -namespace; see HANDOFF.md. +namespace. That run did not record its own quota before/after pair, so under +invariant 11 it is reported, unverified and cannot update gated evidence; see +HANDOFF.md. **The combined result: (f) failed once and passed once; (g) passed twice.** Every per-key arm passed. A write immediately after `delete_all` can be lost, @@ -111,9 +113,11 @@ **not** license "012 refuted" — it licenses only "not reproduced same-scope, with a per-key delete". - python diagnostics/readd_after_delete.py # prints cost, then asks - python diagnostics/readd_after_delete.py --yes # no prompt python diagnostics/readd_after_delete.py --dry-run # cost only, spends nothing + python diagnostics/readd_after_delete.py \ + --executed-by "" --environment "" + python diagnostics/readd_after_delete.py --yes \ + --executed-by "" --environment "" The key is read from MEM0_API_KEY, falling back to ~/.mem0/config.json, so no export is needed. Results are written to diagnostics/results/ as well as @@ -376,10 +380,18 @@ def main() -> int: ap.add_argument("--yes", action="store_true", help="skip the cost prompt") ap.add_argument("--dry-run", action="store_true", help="print the cost estimate and exit without spending") + ap.add_argument( + "--executed-by", default=os.environ.get("MEMORYCHECK_EXECUTED_BY"), + help="person or agent identity executing the live run", + ) + ap.add_argument( + "--environment", default=os.environ.get("MEMORYCHECK_RUN_ENVIRONMENT"), + help="machine/runtime and network mode used for the live run", + ) args = ap.parse_args() - # Four quota probes (one up front, one either side of each arm) are - # themselves SEARCH calls; count them so the estimate is not optimistic. + # Quota probes (one up front, one either side of each arm) are themselves + # SEARCH calls; count them so the estimate is not optimistic. probes = 1 + 2 * len(COST_ESTIMATE) total = sum(COST_ESTIMATE.values()) + probes print("Estimated SEARCH cost (worst case, the scarce counter):") @@ -392,6 +404,14 @@ def main() -> int: print("dry run: nothing spent") return 0 + if not args.executed_by or not args.environment: + print( + "Live-run provenance required: pass --executed-by and --environment " + "(or set MEMORYCHECK_EXECUTED_BY and MEMORYCHECK_RUN_ENVIRONMENT).", + file=sys.stderr, + ) + return 2 + api_key = resolve_api_key() if not api_key: print("No Mem0 key: set MEM0_API_KEY or populate ~/.mem0/config.json.", @@ -403,16 +423,44 @@ def main() -> int: print('Mem0 SDK missing. pip install "mem0ai>=2.0.14"', file=sys.stderr) return 2 - remaining = quota_remaining(api_key) - if remaining is not None and remaining >= 0: - print(f"SEARCH quota remaining now: {remaining}") - if remaining < total: - print(f"REFUSING TO RUN: need ~{total}, have {remaining}. A run that " + stamp = int(time.time()) + out_dir = Path(__file__).resolve().parent / "results" + out_dir.mkdir(exist_ok=True) + out = out_dir / f"readd_after_delete_{stamp}.json" + + quota_before = quota_remaining(api_key) + initial_report = { + "run_at_epoch": stamp, + "provenance": { + "status": "reported, unverified", + "executed_by": args.executed_by, + "environment": args.environment, + "search_quota_before": quota_before, + "search_quota_after": None, + "results_file": str(out), + }, + "state": "started", + "results": [], + } + # Persist the provenance before mutations. If the process is interrupted, + # the partial record remains explicitly unverified instead of disappearing. + out.write_text(json.dumps(initial_report, indent=2) + "\n") + + if quota_before is not None and quota_before >= 0: + print(f"SEARCH quota remaining now: {quota_before}") + if quota_before < total: + initial_report["state"] = "refused: insufficient SEARCH quota" + out.write_text(json.dumps(initial_report, indent=2) + "\n") + print(f"REFUSING TO RUN: need ~{total}, have {quota_before}. A run that " "dies partway proves nothing and spends the rest.", file=sys.stderr) + print(f"provenance written to {out}") return 3 if not args.yes: if input(f"Spend ~{total} SEARCH units? [y/N] ").strip().lower() != "y": print("aborted; nothing spent beyond the quota probe") + initial_report["state"] = "aborted by operator after quota probe" + out.write_text(json.dumps(initial_report, indent=2) + "\n") + print(f"provenance written to {out}") return 0 # Pass the resolved key explicitly. MemoryClient() reads MEM0_API_KEY from @@ -421,7 +469,6 @@ def main() -> int: # probe and then die here with "Mem0 API Key not provided" — after the # operator had already confirmed the spend. client = MemoryClient(api_key=api_key) - stamp = int(time.time()) # (name, first value, second value, extra settle, cross-scope) plan = [ ("a_identical", f"wintergreen-{stamp}", f"wintergreen-{stamp}", 0.0, False), @@ -440,6 +487,7 @@ def main() -> int: ] results = [] + quota_after = None for name, first, second, settle, cross in plan: print(f"\n--- arm {name}{' (cross-scope)' if cross else ''} ---") print(f" expected SEARCH cost, worst case: ~{COST_ESTIMATE[name]} units") @@ -448,6 +496,7 @@ def main() -> int: target = f"mc_diag__{name}_{stamp}__scope_b" if cross else None result = Arm(client, name, source, target).run(first, second, settle) after = quota_remaining(api_key) + quota_after = after if before is not None and after is not None and before >= 0 and after >= 0: result["search_units_spent"] = before - after results.append(result) @@ -464,6 +513,7 @@ def main() -> int: [f"residue-{i}-{stamp}" for i in range(_RESIDUE_N)], f"probe-{name}-{stamp}", settle) after = quota_remaining(api_key) + quota_after = after if before is not None and after is not None and before >= 0 and after >= 0: result["search_units_spent"] = before - after results.append(result) @@ -557,18 +607,31 @@ def main() -> int: print("\nNo conclusion may be published from a single execution of this " "script. Re-run to establish stability first.") - out_dir = Path(__file__).resolve().parent / "results" - out_dir.mkdir(exist_ok=True) - out = out_dir / f"readd_after_delete_{stamp}.json" + provenance_verified = all( + isinstance(value, int) and value >= 0 + for value in (quota_before, quota_after) + ) + provenance = { + "status": "verified" if provenance_verified else "reported, unverified", + "executed_by": args.executed_by, + "environment": args.environment, + "search_quota_before": quota_before, + "search_quota_after": quota_after, + "results_file": str(out), + } out.write_text(json.dumps({ "run_at_epoch": stamp, + "provenance": provenance, + "state": "completed", "poll_interval_s": POLL_INTERVAL, "confirm_timeout_s": CONFIRM_TIMEOUT, "extra_settle_s": EXTRA_SETTLE, "estimated_search_units": total, "results": results, "caveat": "single execution; not a finding until reproduced", - }, indent=2)) + }, indent=2) + "\n") + print(f"provenance status: {provenance['status']}") + print(f"SEARCH quota before/after: {quota_before} -> {quota_after}") print(f"results written to {out}") return 0 diff --git a/examples/report_mem0.json b/examples/report_mem0.json index 892e8dd..ba341e5 100644 --- a/examples/report_mem0.json +++ b/examples/report_mem0.json @@ -3,21 +3,17 @@ "name": "memorycheck", "version": "0.1.0" }, - "generated_at": "2026-08-05T00:43:20+00:00", - "run_id": "1986edd5512147dca783bc513029b4f3", + "generated_at": "2026-07-27T15:58:06+00:00", "adapter": "mem0", "adapter_unverified": false, "adapter_unverified_note": "", - "answering_layer": "quoting", - "limitations": [], "judge": "deterministic-v0", "seeds": 2, "scenarios": 15, "gate": { "fail_on": "P2", "verdict": "FAIL", - "blocking_findings": 10, - "inconclusive_reasons": [] + "blocking_findings": 10 }, "metrics": { "current_fact_accuracy": { @@ -187,4 +183,4 @@ "seed": 1 } ] -} \ No newline at end of file +} diff --git a/examples/report_mem0.md b/examples/report_mem0.md index 15e94b3..91ee244 100644 --- a/examples/report_mem0.md +++ b/examples/report_mem0.md @@ -6,19 +6,20 @@ # Reading this report **Run provenance.** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), -SDK `mem0ai` **2.0.14**, pinned for a like-for-like reproduction, executed -**2026-08-05 00:43 UTC** from implementation commit `b81bf20`. Run ID: -`1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or -build identifier to clients, so the SDK pin and run date are the available -provenance. - -**Provenance.** This is a completed 15 × 2 regeneration after write/delete -convergence confirmation, lossless scope identifiers and unique invocation -namespaces were added. All 170 operations completed. Its figures are identical -to the 2026-07-27 published run and the intervening reproductions: current-fact -accuracy 46/46, stale reuse 10/10, scope leakage 0/22 and deletion residue -0/18. See `HANDOFF.md` for the separate reset diagnostic history; this fresh -run namespace did not need to delete residue from an earlier invocation. +SDK `mem0ai` **2.0.14** — the latest release at time of run, published +2026-07-25 — executed **2026-07-27 15:58 UTC**. The hosted platform exposes no +version or build identifier to clients, so the run date is the only pin +available on the service itself. Three consecutive runs (two on 2.0.14, one on +2.0.11) produced identical figures. + +**Provenance.** These figures were produced at commit `f230208`, before the +adapter gained the write/delete convergence confirmation required by invariant +10. Per-scenario re-checks after that change returned identical results, and +the same figures were reproduced across three runs including one on `mem0ai` +2.0.11. A later 15 × 2 execution on 2026-08-05 is recorded internally as +**reported, unverified** because it lacks its own contemporaneous SEARCH-quota +before/after pair. Under invariant 11, that execution is not a source for this +published report. See `HANDOFF.md`. **What the stale_reuse result is not.** It is not a report that Mem0 fails to overwrite corrected facts. That is documented, intended design — Mem0's README, @@ -50,10 +51,8 @@ racing the pipeline rather than a result. # memorycheck evidence report -- **Run ID:** `1986edd5512147dca783bc513029b4f3` - **Adapter:** `mem0` **Judge:** `deterministic-v0` -- **Answering layer:** `quoting` -- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-08-05T00:43:20+00:00 +- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-07-27T15:58:06+00:00 - **Gate (fail on ≤P2):** **FAIL** (10 blocking findings) | Check | Severity | Result | diff --git a/tests/test_diagnostic_provenance.py b/tests/test_diagnostic_provenance.py new file mode 100644 index 0000000..7b87936 --- /dev/null +++ b/tests/test_diagnostic_provenance.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +SCRIPT = Path(__file__).resolve().parents[1] / "diagnostics" / "readd_after_delete.py" + + +def _environment_without_provenance() -> dict[str, str]: + env = os.environ.copy() + env.pop("MEMORYCHECK_EXECUTED_BY", None) + env.pop("MEMORYCHECK_RUN_ENVIRONMENT", None) + return env + + +def test_live_diagnostic_refuses_missing_provenance_before_api_access(): + completed = subprocess.run( + [sys.executable, str(SCRIPT), "--yes"], + capture_output=True, + check=False, + env=_environment_without_provenance(), + text=True, + ) + + assert completed.returncode == 2 + assert "Live-run provenance required" in completed.stderr + assert "No Mem0 key" not in completed.stderr + + +def test_dry_run_does_not_require_live_provenance(): + completed = subprocess.run( + [sys.executable, str(SCRIPT), "--dry-run"], + capture_output=True, + check=False, + env=_environment_without_provenance(), + text=True, + ) + + assert completed.returncode == 0 + assert "dry run: nothing spent" in completed.stdout From 56a83b1d7bf8ab484f9add641ef46e74833d78a1 Mon Sep 17 00:00:00 2001 From: Yinka Metrics <115735904+YinkaMetrics@users.noreply.github.com> Date: Wed, 5 Aug 2026 02:28:08 +0100 Subject: [PATCH 2/4] docs: record post-merge publication hold --- HANDOFF.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/HANDOFF.md b/HANDOFF.md index 5e89454..81c77e9 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -3084,6 +3084,48 @@ External launch remains **HELD pending founder review**. --- +## 2026-08-05 — PR #10 merged before provenance correction landed + +### 1. What changed + +GitHub read-back at this handoff found PR #10 already merged at 01:09 UTC at +head `a5aad4e`. The quota reconciliation and invariant-11 correction are in +later commit `e270191`, pushed to the same branch after that merge, so they are +not part of `main`. + +### 2. Findings + +The merge was an external state change, not an action taken in this task. Its +published 2026-08-05 evidence is exactly the promotion now classified as +reported, unverified. Commit `e270191` withdraws that promotion and preserves +the older verified published artifact; no new provider call was made. + +### 3. Decisions + +- Do not rewrite, revert, or otherwise mutate the completed merge without + explicit founder authority. +- Open the isolated post-merge correction as a new draft PR from + `agent/fail-closed-release-gate`, labeled `needs-founder-review`. +- Do not merge the follow-up PR automatically. + +### 4. FOR STRATEGY + +Until the follow-up is reviewed and merged, `main` contains an evidence +promotion that fails the newly adopted provenance rule. Treat that published +2026-08-05 run as reported, unverified regardless of its current location in +`main`. + +### 5. Next + +1. Open and label the follow-up draft PR. +2. Confirm it contains only the provenance, accounting, cost correction and + evidence rollback. +3. Await explicit founder review; do not merge. + +External launch remains **HELD pending founder review**. + +--- + ## 2026-08-05 — Live-run quota reconciliation and provenance gate ### 1. What changed @@ -3219,3 +3261,35 @@ handoff point. External launch remains **HELD pending founder review**. --- + +## 2026-08-05 — Current publication state after PR #10 merge + +### 1. What shipped + +PR #10 merged at head `a5aad4e`. The later provenance correction is pushed at +`e270191` and is not in `main`. + +### 2. Findings + +No new provider measurement and no additional quota spend. The detailed +286-unit reconciliation and both reported-unverified provenance blocks are in +the entries above. + +### 3. Decisions + +Preserve the completed merge and route `e270191` through a separate draft PR +with `needs-founder-review`; do not merge it automatically. + +### 4. FOR STRATEGY + +Until the follow-up correction is approved, treat the 2026-08-05 live evidence +currently present on `main` as reported, unverified. + +### 5. Next + +Open the isolated follow-up draft, confirm its scope and gate, then await +explicit founder review. + +External launch remains **HELD pending founder review**. + +--- From 8d0a456c0746a3e8c89c0101802e52425e86d21c Mon Sep 17 00:00:00 2001 From: Yinka Metrics <115735904+YinkaMetrics@users.noreply.github.com> Date: Wed, 5 Aug 2026 02:32:37 +0100 Subject: [PATCH 3/4] fix: keep provenance checks dependency-free --- HANDOFF.md | 34 +++++++++++++++++++++++++++++++ diagnostics/readd_after_delete.py | 6 ++++-- 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 81c77e9..c67de3d 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -3293,3 +3293,37 @@ explicit founder review. External launch remains **HELD pending founder review**. --- + +## 2026-08-05 — Follow-up PR #11 CI repair + +### 1. What shipped + +Draft PR #11 opened with `needs-founder-review`. Its first CI run failed only +because the new subprocess tests exercised the diagnostic in the base install, +while `httpx` was imported at module load despite being an optional Mem0-path +dependency. + +### 2. Findings + +The provenance checks themselves behaved correctly locally. CI failed before +reaching them with `ModuleNotFoundError: httpx`; no provider call was possible +and no quota was spent. + +### 3. Decisions + +Move the optional `httpx` import into the live quota-probe function. Dry-run +and fail-closed provenance validation must work in the base install, while an +actual Mem0 run continues to use the dependency supplied by the Mem0 extra. + +### 4. FOR STRATEGY + +This is a packaging boundary correction only. It does not alter the quota +reconciliation, provenance ruling, published metrics, or founder gate. + +### 5. Next + +Push the repair, confirm PR #11 CI passes, and leave the PR draft and unmerged. + +External launch remains **HELD pending founder review**. + +--- diff --git a/diagnostics/readd_after_delete.py b/diagnostics/readd_after_delete.py index e2ccab1..b914892 100644 --- a/diagnostics/readd_after_delete.py +++ b/diagnostics/readd_after_delete.py @@ -140,8 +140,6 @@ import time from pathlib import Path -import httpx - POLL_INTERVAL = 5.0 # seconds between reads; each read costs one SEARCH unit CONFIRM_TIMEOUT = 120.0 # ceiling on waiting for a value to become retrievable EXTRA_SETTLE = 60.0 # arm (c): wait this long AFTER the delete reads empty @@ -193,6 +191,10 @@ def resolve_api_key() -> str | None: def quota_remaining(api_key: str) -> int | None: """Read the SEARCH counter. Costs one SEARCH unit itself.""" try: + # Keep the optional live-provider dependency off the dry-run and + # missing-provenance paths. Core CI installs only the base package. + import httpx + r = httpx.post( "https://api.mem0.ai/v3/memories/", headers={"Authorization": f"Token {api_key}", "Content-Type": "application/json"}, From c37f419470b25aacd656e630b477b56757ae0825 Mon Sep 17 00:00:00 2001 From: Yinka Metrics <115735904+YinkaMetrics@users.noreply.github.com> Date: Wed, 5 Aug 2026 19:59:38 +0100 Subject: [PATCH 4/4] docs: verify Mem0 evidence by quota reconciliation --- ADAPTER_PREFLIGHT.md | 16 ++-- CLAUDE.md | 12 ++- HANDOFF.md | 136 +++++++++++++++++++++--------- README.md | 11 +-- diagnostics/readd_after_delete.py | 7 +- examples/report_mem0.json | 8 +- examples/report_mem0.md | 33 ++++---- 7 files changed, 148 insertions(+), 75 deletions(-) diff --git a/ADAPTER_PREFLIGHT.md b/ADAPTER_PREFLIGHT.md index e81f32c..63a8bc8 100644 --- a/ADAPTER_PREFLIGHT.md +++ b/ADAPTER_PREFLIGHT.md @@ -12,8 +12,8 @@ never by reading documentation. Record answers in `HANDOFF.md` before the first scenario. An adapter keeps `unverified = True` until every item has an observed answer. -For every live execution, create this provenance block before interpreting -the result: +For every live execution after adoption of invariant 11, create this +provenance block before interpreting the result: ```text status: verified | reported, unverified @@ -24,9 +24,15 @@ search_quota_after: results_file: ``` -All four facts plus the filename are mandatory. If any is missing, the run is +All four facts plus the filename are mandatory prospectively. If any is +missing, the run is `reported, unverified`: log it internally, but do not change README, a published report, adapter verification status, or another gated claim from it. +The 2026-08-05 Mem0 evidence is the recorded legacy exception: the founder +independently read the live SEARCH counter at 599 against approximately 885 +before the work, verifying the run retrospectively by quota reconciliation. +Its provenance was not self-recorded at execution time — the reason this rule +now exists. --- @@ -42,8 +48,8 @@ incomplete every time so far. - [ ] What is the request rate limit, and does confirmation polling threaten it? *Why:* Mem0 meters `SEARCH` (1,000/period) separately from `ADD` -(10,000/period). The current full 15 × 2 path has a structural minimum of -~182 SEARCH — 91 per seed — and extra convergence polls can raise it. Zep +(10,000/period). The current full 15 × 2 path costs at least 182 SEARCH — a +91-per-seed structural minimum — and extra convergence polls can raise it. Zep meters ingestion credits instead, so the same polling is free but a request-rate ceiling applies. Two providers, opposite constraints; neither was guessable. diff --git a/CLAUDE.md b/CLAUDE.md index 799a86e..8624e5f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -86,14 +86,19 @@ Status: v0.1, built in public. `pytest -q` must stay green after every change. just-mutated value (or its absence) with a bounded timeout. That poll is mutation confirmation only; the oracle never grades it. Non-convergence aborts rather than being laundered into PASS or scored as a provider bug. -11. **Live-run provenance is a publication gate.** Every live-provider run +11. **Live-run provenance is a publication gate.** From adoption of this rule, + every live-provider run records four facts contemporaneously: who executed it, the execution environment, the provider's SEARCH quota immediately before and after, and the results filename. Missing any field makes the run **reported, unverified**. It may be logged internally, but it must not update README, `examples/report_*`, clear an adapter's `unverified` flag, or support any other gated claim. Estimates and reconstructed quota arithmetic are useful - diagnostics, never substitutes for the before/after pair. + diagnostics, never substitutes for the before/after pair. The 2026-08-05 + Mem0 evidence is a documented legacy exception: it was verified + retrospectively by the founder's independent live counter reading of 599 + against approximately 885 before the work. Its provenance was not + self-recorded at execution time — the reason this prospective rule exists. ## Conventions @@ -164,7 +169,8 @@ stops short has not finished its task and must say which of the four steps it stopped at. Blocked work still gets an entry: the blocker, what was established anyway, and what remains unverified. -**Every live-provider run, here or elsewhere, must be promoted into +**Every live-provider run after adoption of invariant 11, here or elsewhere, +must be promoted into `HANDOFF.md` with its complete invariant-11 provenance block:** executor, environment, SEARCH quota before, SEARCH quota after, and results filename. If one is absent, label it **reported, unverified** and do not change a gated diff --git a/HANDOFF.md b/HANDOFF.md index c67de3d..5b90b7f 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -3084,42 +3084,43 @@ External launch remains **HELD pending founder review**. --- -## 2026-08-05 — PR #10 merged before provenance correction landed +## 2026-08-05 — PR #10 merge and retrospective provenance review ### 1. What changed GitHub read-back at this handoff found PR #10 already merged at 01:09 UTC at -head `a5aad4e`. The quota reconciliation and invariant-11 correction are in -later commit `e270191`, pushed to the same branch after that merge, so they are -not part of `main`. +head `a5aad4e`. The quota reconciliation and invariant-11 process rule are in +later commits on the follow-up branch. ### 2. Findings The merge was an external state change, not an action taken in this task. Its -published 2026-08-05 evidence is exactly the promotion now classified as -reported, unverified. Commit `e270191` withdraws that promotion and preserves -the older verified published artifact; no new provider call was made. +published 2026-08-05 evidence is **verified retrospectively by quota +reconciliation; provenance was not self-recorded at execution time — the +reason the rule now exists.** The founder independently read Mem0's live +SEARCH counter at 599 on 2026-08-05, against approximately 885 before the +work. Mem0 owns that counter outside the repository; repository code cannot +write it. No new provider call was made during this documentation review. ### 3. Decisions - Do not rewrite, revert, or otherwise mutate the completed merge without explicit founder authority. -- Open the isolated post-merge correction as a new draft PR from - `agent/fail-closed-release-gate`, labeled `needs-founder-review`. +- Open the isolated post-merge documentation/process correction as a new draft + PR from `agent/fail-closed-release-gate`, labeled `needs-founder-review`. - Do not merge the follow-up PR automatically. ### 4. FOR STRATEGY -Until the follow-up is reviewed and merged, `main` contains an evidence -promotion that fails the newly adopted provenance rule. Treat that published -2026-08-05 run as reported, unverified regardless of its current location in -`main`. +The 2026-08-05 evidence on `main` remains verified. The process gap is that +the run did not self-record its provenance contemporaneously; apply invariant +11 prospectively so the next run does not require retrospective reconciliation. ### 5. Next 1. Open and label the follow-up draft PR. -2. Confirm it contains only the provenance, accounting, cost correction and - evidence rollback. +2. Confirm it retains the verified 2026 evidence and contains only provenance, + accounting, cost and process-rule corrections. 3. Await explicit founder review; do not merge. External launch remains **HELD pending founder review**. @@ -3135,21 +3136,25 @@ External launch remains **HELD pending founder review**. and the exact results filename. - A run missing any field is **reported, unverified**. It may be retained in this internal handoff, but it cannot update README, `examples/report_*`, an - adapter verification flag, or another gated claim. + adapter verification flag, or another gated claim. This applies to runs + executed after the rule's adoption. - `diagnostics/readd_after_delete.py` now refuses a live execution without the executor and environment, writes the initial provenance record before any mutation, and records the quota pair and result filename in its JSON output. -- Withdrew the 2026-08-05 evidence promotion from README and - `examples/report_mem0.*`. The published artifact remains the 2026-07-27 run; - the later execution remains available internally as reported, unverified. +- Retained the 2026-08-05 evidence in README and `examples/report_mem0.*`. + It is verified retrospectively by the founder's independent live SEARCH + counter reading; its provenance was not self-recorded at execution time. - Corrected the documented current Mem0 cost from the historical ~106 estimate - to a structural minimum of 91 SEARCH per seed, or 182 for 15 × 2, before - extra convergence polls. + to a structural minimum of 91 SEARCH per seed, or at least 182 for 15 × 2, + before extra convergence polls. ### 2. Quota accounting -The founder-reported account movement is **885 → 599 = 286 SEARCH units**. -The surviving artifacts support this reconciliation: +The founder independently took a live Mem0 SEARCH counter reading of **599** +on 2026-08-05, against approximately **885 before the work**: a movement of +approximately 286 SEARCH units. Mem0's counter is external to this repository +and cannot be written by repository code. The surviving artifacts support this +reconciliation: | Bucket | SEARCH units | Evidence and limit | |---|---:|---| @@ -3176,31 +3181,33 @@ invented as a specific cause. **Seven-arm diagnostic** ```text -status: reported, unverified +status: verified retrospectively by quota reconciliation; provenance not self-recorded at execution time — the reason the rule now exists executed_by: Codex (OpenAI), on the founder's instruction environment: Codex desktop on macOS; local writable clone; explicitly approved live network access -search_quota_before: no run-start value recorded; first provider checkpoint was 822 after the initial probe -search_quota_after: not contemporaneously recorded as a run-level value +search_quota_before: approximately 885 before the combined work, recorded by the founder +search_quota_after: 599, independently read live from Mem0 by the founder on 2026-08-05 results_file: diagnostics/results/readd_after_delete_1785890141.json ``` **Full 15 × 2 execution** ```text -status: reported, unverified +status: verified retrospectively by quota reconciliation; provenance not self-recorded at execution time — the reason the rule now exists executed_by: Codex (OpenAI), on the founder's instruction environment: Codex desktop on macOS; local writable clone; explicitly approved live network access -search_quota_before: not contemporaneously recorded for this run -search_quota_after: not contemporaneously recorded for this run; 599 is only the founder-reported final account value across the combined work +search_quota_before: approximately 885 before the combined work, recorded by the founder +search_quota_after: 599, independently read live from Mem0 by the founder on 2026-08-05 results_file: diagnostics/results/full_mem0_20260805.json (with matching .md and .log) ``` -Both executions are retained for investigation, but neither qualifies to -change a gated file under invariant 11. +The combined live work is verified retrospectively by the independent provider +counter reconciliation. It qualifies as evidence; the missing self-recorded +run-level pair is a process deficiency and the reason invariant 11 now applies +prospectively. ### 4. FOR STRATEGY -- The 63-unit maximum remainder is closed as **unallocated, unverified**, not +- The 63-unit maximum remainder is closed as **unallocated**, not silently attributed to repeat, sentinel, or development runs that the artifacts do not prove. Do not spend more quota merely to reconstruct it. - The next live run must begin only after the five-field provenance record is @@ -3209,15 +3216,15 @@ change a gated file under invariant 11. and spend without a follow-up reconstruction. - Existing Mem0 findings are unaffected: the scenario-pack identifiers do not slug-collide, the execution used `seeds=2`, there was no concurrency, and - the Mem0 answering layer is quoting. This is an evidence-provenance hold, - not a change to the previously published metric values. -- README/report edits remain gated-tier and PR #10 must retain + the Mem0 answering layer is quoting. The evidence remains verified and the + metric values remain unchanged. +- README/report edits remain gated-tier and PR #11 must retain `needs-founder-review`; do not merge without explicit founder approval. ### 5. Next 1. Run offline validation only; do not spend additional Mem0 quota. -2. Push the provenance rule and accounting to draft PR #10. +2. Push the corrected provenance rule and accounting to draft PR #11. 3. Confirm the PR stays draft, labeled `needs-founder-review`, and unmerged. External launch remains **HELD pending founder review**. @@ -3272,18 +3279,20 @@ PR #10 merged at head `a5aad4e`. The later provenance correction is pushed at ### 2. Findings No new provider measurement and no additional quota spend. The detailed -286-unit reconciliation and both reported-unverified provenance blocks are in -the entries above. +286-unit reconciliation and retrospective-verification provenance blocks are +in the entries above. ### 3. Decisions -Preserve the completed merge and route `e270191` through a separate draft PR -with `needs-founder-review`; do not merge it automatically. +Preserve the completed merge and route the prospective provenance rule and +customer-facing cost correction through a separate draft PR with +`needs-founder-review`; do not merge it automatically. ### 4. FOR STRATEGY -Until the follow-up correction is approved, treat the 2026-08-05 live evidence -currently present on `main` as reported, unverified. +Treat the 2026-08-05 live evidence on `main` as **verified retrospectively by +quota reconciliation; provenance not self-recorded at execution time — the +reason the rule now exists.** ### 5. Next @@ -3327,3 +3336,46 @@ Push the repair, confirm PR #11 CI passes, and leave the PR draft and unmerged. External launch remains **HELD pending founder review**. --- + +## 2026-08-05 — Founder confirms retrospective verification + +### 1. What changed + +Corrected HANDOFF, README, the published Mem0 report framing and PR #11 to +retain the 2026-08-05 evidence as verified. Removed the proposed evidence +rollback. Invariant 11 remains as a prospective self-recording requirement. + +### 2. Findings + +The evidence is **verified retrospectively by quota reconciliation; provenance +not self-recorded at execution time — the reason the rule now exists.** The +founder independently read Mem0's live SEARCH quota at **599** on 2026-08-05, +against approximately **885 before the work**. That approximately 286-unit +movement is consistent with 41 for the seven-arm diagnostic, at least 182 for +the full 15 × 2, and at most 63 unallocated. Mem0's counter is external to the +repository and cannot be written by repository code. + +### 3. Decisions + +- Keep the regenerated 2026-08-05 report and run ID + `1986edd5512147dca783bc513029b4f3` as the current evidence. +- State the customer-facing full-run cost as **at least 182 SEARCH** under + invariant 10, not the historical ~106 estimate. +- Require future live runs to self-record executor, environment, quota pair + and exact results filename; retrospective verification is not the default + operating procedure. + +### 4. FOR STRATEGY + +This correction changes provenance framing, not provider findings. Existing +Mem0 results remain unaffected: identifiers do not slug-collide, seeds=2, no +concurrency, and a quoting answering layer. + +### 5. Next + +Update draft PR #11, rerun offline validation and CI, retain +`needs-founder-review`, and do not merge without explicit approval. + +External launch remains **HELD pending founder review**. + +--- diff --git a/README.md b/README.md index 0ec3bef..69244fa 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ How the lifecycle maps onto Mem0: ### Result -**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** (latest release at time of run, published 2026-07-25), executed **2026-07-27 15:58 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`. The hosted platform exposes no version or build identifier to clients, so the run date is the only pin available on the service itself. Three consecutive runs — two on 2.0.14, one on 2.0.11 — produced identical figures. +**Run:** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), SDK `mem0ai` **2.0.14** pinned for like-for-like reproduction, executed **2026-08-05 00:43 UTC**, 15 scenarios × 2 seeds, judge `deterministic-v0`, run ID `1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or build identifier to clients, so the SDK pin and run date are the available provenance. This completed regeneration produced figures identical to the 2026-07-27 published run and the intervening reproductions. ``` current_fact_accuracy 100% (46/46) @@ -193,7 +193,7 @@ How the lifecycle maps onto Mem0: GATE [fail on <= P2]: FAIL (10 blocking findings) ``` -**Provenance.** These figures were produced at commit `f230208`, before the adapter gained the write/delete convergence confirmation required by invariant 10. Per-scenario re-checks after that change returned identical results, and the same figures were reproduced across three runs including one on `mem0ai` 2.0.11. A later 15 × 2 execution on 2026-08-05 is recorded internally as **reported, unverified** because it lacks its own contemporaneous SEARCH-quota before/after pair. Under invariant 11, that execution is not a source for this published report. Tracked in [`HANDOFF.md`](HANDOFF.md). +**Provenance.** The current evidence was regenerated from implementation commit `b81bf20`, after the adapter gained invariant-10 mutation confirmation, scope identifiers became lossless, and every invocation gained a unique namespace. All 170 operations completed. The run is **verified retrospectively by quota reconciliation; provenance was not self-recorded at execution time — the reason invariant 11 now exists.** The founder independently read Mem0's live SEARCH counter at **599** on 2026-08-05, against approximately **885** before the work. That external counter movement is consistent with 41 SEARCH for the seven-arm diagnostic, at least 182 for the full 15 × 2, and at most 63 unallocated. Mem0's counter is outside this repository and cannot be written by it. The metrics are unchanged from the earlier evidence. Full accounting is in [`HANDOFF.md`](HANDOFF.md). Mem0 holds the boundaries that carry the P1 severities: **no scope leakage in 22 opportunities and no deletion residue in 18** — deletes stopped the value influencing answers, and no user's or tenant's facts crossed into another's, including where two tenants share a `user_id` and where a fact is moved between tenants. Current-fact accuracy is perfect and every result is stable across seeds. @@ -242,9 +242,10 @@ Mem0 meters two independent counters, and this adapter spends the scarcer one. The current adapter has a structural minimum of **91 SEARCH units per seed**: 15 reset pre-reads, 30 write confirmations, 16 delete reads/confirmations and -30 scenario queries. A default 2-seed run therefore costs **~182**, before any -extra convergence polls. Confirmation reads are not optional: without them the -harness races the provider and manufactures false failures (invariant 10). +30 scenario queries. A default 2-seed run therefore costs **at least 182 +SEARCH**, with extra convergence polls increasing that total. Confirmation +reads are not optional: without them the harness races the provider and +manufactures false failures (invariant 10). Budget accordingly if you wire this into CI. Running the full pack on every commit will exhaust a 1,000-unit period in ~5 runs. The intended shape is a diff --git a/diagnostics/readd_after_delete.py b/diagnostics/readd_after_delete.py index b914892..0440b39 100644 --- a/diagnostics/readd_after_delete.py +++ b/diagnostics/readd_after_delete.py @@ -19,9 +19,10 @@ `delete_all`. The earlier loss remains valid evidence that it can happen; the clean rerun does not erase it. The full 15 x 2 regeneration immediately after this diagnostic completed all 170 operations under a unique invocation -namespace. That run did not record its own quota before/after pair, so under -invariant 11 it is reported, unverified and cannot update gated evidence; see -HANDOFF.md. +namespace. It is verified retrospectively by quota reconciliation: the founder +independently read Mem0's live SEARCH counter at 599 on 2026-08-05, against +approximately 885 before the work. Provenance was not self-recorded at +execution time — the reason invariant 11 now exists. See HANDOFF.md. **The combined result: (f) failed once and passed once; (g) passed twice.** Every per-key arm passed. A write immediately after `delete_all` can be lost, diff --git a/examples/report_mem0.json b/examples/report_mem0.json index ba341e5..53f4697 100644 --- a/examples/report_mem0.json +++ b/examples/report_mem0.json @@ -3,17 +3,21 @@ "name": "memorycheck", "version": "0.1.0" }, - "generated_at": "2026-07-27T15:58:06+00:00", + "generated_at": "2026-08-05T00:43:20+00:00", + "run_id": "1986edd5512147dca783bc513029b4f3", "adapter": "mem0", "adapter_unverified": false, "adapter_unverified_note": "", + "answering_layer": "quoting", + "limitations": [], "judge": "deterministic-v0", "seeds": 2, "scenarios": 15, "gate": { "fail_on": "P2", "verdict": "FAIL", - "blocking_findings": 10 + "blocking_findings": 10, + "inconclusive_reasons": [] }, "metrics": { "current_fact_accuracy": { diff --git a/examples/report_mem0.md b/examples/report_mem0.md index 91ee244..681eb1b 100644 --- a/examples/report_mem0.md +++ b/examples/report_mem0.md @@ -6,20 +6,21 @@ # Reading this report **Run provenance.** Mem0 hosted platform via `api.mem0.ai` (`/v3` endpoints), -SDK `mem0ai` **2.0.14** — the latest release at time of run, published -2026-07-25 — executed **2026-07-27 15:58 UTC**. The hosted platform exposes no -version or build identifier to clients, so the run date is the only pin -available on the service itself. Three consecutive runs (two on 2.0.14, one on -2.0.11) produced identical figures. - -**Provenance.** These figures were produced at commit `f230208`, before the -adapter gained the write/delete convergence confirmation required by invariant -10. Per-scenario re-checks after that change returned identical results, and -the same figures were reproduced across three runs including one on `mem0ai` -2.0.11. A later 15 × 2 execution on 2026-08-05 is recorded internally as -**reported, unverified** because it lacks its own contemporaneous SEARCH-quota -before/after pair. Under invariant 11, that execution is not a source for this -published report. See `HANDOFF.md`. +SDK `mem0ai` **2.0.14**, pinned for a like-for-like reproduction, executed +**2026-08-05 00:43 UTC** from implementation commit `b81bf20`. Run ID: +`1986edd5512147dca783bc513029b4f3`. The hosted platform exposes no version or +build identifier to clients, so the SDK pin and run date are the available +provenance. + +**Provenance.** This completed 15 × 2 regeneration followed write/delete +convergence confirmation, lossless scope identifiers and unique invocation +namespaces. All 170 operations completed. It is **verified retrospectively by +quota reconciliation; provenance was not self-recorded at execution time — the +reason invariant 11 now exists.** The founder independently read Mem0's live +SEARCH counter at **599** on 2026-08-05, against approximately **885** before +the work. The 286-unit movement is consistent with 41 for the diagnostic, at +least 182 for this full run, and at most 63 unallocated. Mem0's counter is +outside this repository and cannot be written by it. See `HANDOFF.md`. **What the stale_reuse result is not.** It is not a report that Mem0 fails to overwrite corrected facts. That is documented, intended design — Mem0's README, @@ -51,8 +52,10 @@ racing the pipeline rather than a result. # memorycheck evidence report +- **Run ID:** `1986edd5512147dca783bc513029b4f3` - **Adapter:** `mem0` **Judge:** `deterministic-v0` -- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-07-27T15:58:06+00:00 +- **Answering layer:** `quoting` +- **Scenarios:** 15 **Seeds:** 2 **Generated:** 2026-08-05T00:43:20+00:00 - **Gate (fail on ≤P2):** **FAIL** (10 blocking findings) | Check | Severity | Result |