diff --git a/HANDOFF.md b/HANDOFF.md index 8ff1d2b..6157998 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -2085,9 +2085,10 @@ whatever comes back. ### 2. Findings -**Both runs are blocked. Egress is still refused.** A Mem0 credential is now -present in the session (`~/.mem0/config.json`), but `api.mem0.ai` remains -blocked: +**Both runs are blocked. Egress is still refused.** ~~A Mem0 credential is now +present in the session (`~/.mem0/config.json`)~~ — **CORRECTED 2026-08-03: this +was wrong, there is no credential here; see the correction entry below** — and +`api.mem0.ai` remains blocked: ``` curl https://api.mem0.ai/v1/ping/ -> CONNECT tunnel failed, response 403 @@ -2166,7 +2167,9 @@ uninterpretable against the (a) baseline. Say if you want it the other way. ### 4. FOR STRATEGY - **An environment with egress is now the single blocker for all Mem0 work.** - The credential is in place; the network is not. Everything else is staged + ~~The credential is in place; the network is not.~~ **CORRECTED + 2026-08-03: neither is in place — see the correction entry below.** + Everything else is staged and push-button. - Unchanged: whether to pin `mem0ai` 2.0.14 for the regen, whether the extras should be exact pins, whether an open and contested #6017 changes the @@ -2430,3 +2433,133 @@ establish. External launch remains **HELD**. --- +## 2026-08-03 — Falsified abort message replaced; arms (f)/(g); credential claim corrected + +### 1. What shipped + +Three things, all internal tier. + +| Change | File | +|---|---| +| Abort message: falsified trigger replaced with the evidenced one | `adapters/mem0.py` | +| Arms (f)/(g): the `delete_all` namespace condition | `diagnostics/readd_after_delete.py` | +| Credential claim corrected in two prior entries | `HANDOFF.md` | + +### 2. Findings + +**The abort message was asserting a falsified mechanism.** `write()` said: + +> Known trigger: re-adding text identical to a value deleted moments earlier +> (see HANDOFF, scenario 012) + +Three things now contradict that. The **2026-08-03 abort fired at +`003-scope-boundaries`, op #26** — reported, not observed here — with no +re-add of deleted text anywhere near it. `003` does not delete at all. And +arms (a)-(e) were built to test exactly that mechanism. Replaced with the +evidenced trigger: **a write issued after a `delete_all`, even when the +namespace was polled until it read empty**, citing `reset()`'s own measurement +of **6/14 writes lost following a delete_all versus 0/10 with no preceding +delete**. That measurement was in the codebase the whole time, one method +above the message that ignored it. + +This mattered beyond tidiness: the message is what an operator reads at the +moment a run dies, and it was pointing them at a mechanism five arms had +already excluded. + +**Arms (f)/(g) — and why (a)-(e) were the wrong instrument.** Every arm so far +uses a **per-key** delete (fetch scope, filter on metadata key, delete by id). +`reset()` does something categorically different: one +`delete_all(app_id=...)` wiping the whole namespace across every scope, then a +poll until empty, then return — after which the runner immediately writes. + +That per-key substitution was **my call**, made when (d)/(e) were added: the +brief said `delete_all` and I used per-key to keep the delete mechanism fixed +across arms for comparability. Comparability was preserved and the mechanism +under investigation was not tested. It is a plausible reason all five passed. + +- **(f)** seeds `_RESIDUE_N`=5 values so the namespace holds real residue, + calls the same `delete_all(app_id=...)`, polls until the namespace reads + empty, then writes and polls for retrievability. +- **(g)** identical, plus a 60s settle after the namespace reads empty. + +Together they separate *"reads empty means propagated"* from *"reads empty but +still reaping"*. + +**A design flaw in (f)/(g), caught by simulation before any spend.** The first +draft opened with a cleanup `delete_all` before seeding. Under the very +hypothesis being tested, that opening call would swallow the seed writes and +the arm would die in setup having measured nothing — the simulation returned +`SETUP_FAILED` for both reaping behaviours. Fixed by giving each namespace arm +a **fresh `app_id`**, so the namespace is empty by construction and the arm +contains exactly one `delete_all`: the one under test. + +Validated offline against three known `delete_all` behaviours, spending +nothing: + +| delete_all behaviour | (f) no settle | (g) 60s settle | +|---|---|---| +| clean | WRITE_VISIBLE | WRITE_VISIBLE | +| reaps, transient | **WRITE_LOST** | WRITE_VISIBLE | +| reaps, persistent | **WRITE_LOST** | **WRITE_LOST** | + +In every simulated case the namespace **read empty before the write**, which +is the whole point: emptiness is what `reset()` currently trusts. + +**Reading stated in advance.** If (f) fails and (g) passes, polling a +namespace until empty is not sufficient — and since that is precisely what +`reset()` does before returning, **every run is exposed**, with the first +writes after any reset liable to be reaped. That would be a harness-side +defect as much as a provider behaviour, and the fix is a settle sized from the +measured empty-to-safe gap, not a fixed sleep. + +**Cost.** All seven arms now ~228 units worst case; (f)+(g) are ~66 of that, +~10-12 if they pass. Worst case remains the failing case. + +**Credential claim corrected.** Two prior entries state a Mem0 credential is +present in this environment. **It is not.** `~/.mem0/config.json` is 57 bytes +containing a single `user_id` field — the Mem0 CLI's local identity file, not +an API key. There is no `platform.api_key`, so the script's `resolve_api_key()` +returns `None` from it, and `MEM0_API_KEY` is unset. I inferred the credential +from the file existing and never opened it. Both entries are struck through in +place and point here. + +So **both** blockers are live, not one: no key, and no egress +(`connect_rejected api.mem0.ai:443`, re-verified). Either alone is sufficient +to stop every Mem0 run from this environment. + +### 3. Decisions + +- **Per-key delete is not substituted in (f)/(g).** Explicit instruction, and + correct: the substitution is what made the earlier arms miss. +- **Fresh `app_id` per namespace arm**, so the arm holds exactly one + `delete_all`. Adopted after the simulation showed the alternative aborts in + setup under the hypothesis being tested. +- **The struck-through claims were left visible** rather than deleted. The + entries are merged and public; silently rewriting them would hide that the + log was wrong for several hours. + +### 4. FOR STRATEGY + +- **The `003` abort is a stronger data point than anything the arms have + produced, and it is second-hand here.** Worth promoting properly: the + results filename, the full latency curve from the new instrumentation, and + whether it was the regen or a shorter run. Per the promotion rule, an + unlogged run is invisible to the next session — right now this one is a + single sentence. +- If (f)/(g) implicate `reset()`, the fix lands in `adapters/mem0.py` and + **could flip provider findings**, so it is gated tier and needs a ruling + before merge. +- Unchanged: pinning `mem0ai` 2.0.14 for the regen, exact pins for extras, + whether #6017 changes the publication plan, and the `get_all` pagination + limitation. + +### 5. Next + +1. Run (f)/(g) on the Mac — the leading hypothesis, ~66 units worst case. +2. Promote the `003` abort properly, with its curve and results filename. +3. Full 15 × 2 regen, capturing stderr for the latency curve. +4. Founder ruling before any of it touches published text. + +External launch remains **HELD**. + +--- diff --git a/diagnostics/readd_after_delete.py b/diagnostics/readd_after_delete.py index d05e2bb..d68b033 100644 --- a/diagnostics/readd_after_delete.py +++ b/diagnostics/readd_after_delete.py @@ -61,10 +61,34 @@ (e) as (d), but WAIT an additional 60s before writing to B Separates propagation timing from the cross-scope factor itself. -Read the arms together and across the pair boundary. (a) alone says nothing; -the informative comparisons are (a) vs (b), (a) vs (c), (a) vs (d), and -(d) vs (e). In particular a clean sweep of (a)-(c) does **not** license -"012 refuted" — it licenses only "not reproduced same-scope". +NAMESPACE ARMS (f)-(g). **The leading hypothesis, and the only arms that use +the delete the harness actually performs.** + +Arms (a)-(e) all use a PER-KEY delete: fetch the scope, filter on the metadata +key, delete by id. `reset()` does something different — a single +`delete_all(app_id=...)` that wipes the whole namespace across every scope. +The 2026-08-03 abort at `003-scope-boundaries` (op #26) followed a `reset()`, +with no re-add of deleted text anywhere near it, which per-key arms cannot +model. Substituting per-key for delete_all is very likely why (a)-(e) passed. + + (f) write N values so the namespace holds real residue, call the SAME + `delete_all(app_id=...)` that `reset()` calls, poll until the namespace + reads empty, then write and poll for retrievability + The evidenced condition. `reset()`'s own docstring records 6/14 writes + lost after a delete_all versus 0/10 without one. + + (g) as (f), but WAIT an additional 60s after the namespace reads empty + Separates "reads empty means propagated" from "reads empty but the + delete is still reaping". If (f) fails and (g) passes, then polling a + namespace until empty — which is exactly what `reset()` does before + returning — is NOT sufficient, and every run is exposed. + +Read the arms together and across the pair boundaries. (a) alone says nothing; +the informative comparisons are (a) vs (b), (a) vs (c), (a) vs (d), (d) vs (e), +and — now the important one — (a)-(e) vs (f)/(g), which is per-key delete +versus namespace delete_all. In particular a clean sweep of (a)-(c) does +**not** license "012 refuted" — it licenses only "not reproduced same-scope, +with a per-key delete". python diagnostics/readd_after_delete.py # prints cost, then asks python diagnostics/readd_after_delete.py --yes # no prompt @@ -100,6 +124,11 @@ APP_ID = "mc_diag_readd" KEY = "handover-note" +# How many values (f)/(g) seed before wiping, so the namespace holds real +# residue rather than a single row. A delete_all over one memory is not the +# condition a run hits — `reset()` fires when a namespace has accumulated. +_RESIDUE_N = 5 + # Worst-case SEARCH reads per arm: 1 pre-read + write-confirm polls # + 1 delete lookup + delete-confirm polls + re-add-confirm polls. # @@ -115,6 +144,9 @@ "c_settle_then_identical": 2 + 2 + 1 + _MAX_POLLS, "d_cross_scope_identical": 2 + 2 + 1 + _MAX_POLLS, "e_cross_scope_settle": 2 + 2 + 1 + _MAX_POLLS, + # (f)/(g) seed N values, so add N confirm reads to the usual shape. + "f_namespace_delete_all": 2 + _RESIDUE_N + 2 + _MAX_POLLS, + "g_namespace_settle": 2 + _RESIDUE_N + 2 + _MAX_POLLS, } @@ -149,8 +181,14 @@ def quota_remaining(api_key: str) -> int | None: class Arm: def __init__(self, client, name: str, user_id: str, - target_user_id: str | None = None): + target_user_id: str | None = None, app_id: str = APP_ID): self.c, self.name, self.uid = client, name, user_id + # Per-arm namespace for (f)/(g): a fresh `app_id` starts empty by + # construction, so the arm contains exactly ONE delete_all — the one + # under test. An opening cleanup delete_all would, under the very + # hypothesis being tested, swallow the seed writes and abort the arm + # in setup before it measured anything. + self.app_id = app_id # Where the re-add goes. Same scope for (a)-(c); a different `user_id` # under the same `app_id` for the cross-scope arms (d)-(e), which is # what the 012 rescope actually does. @@ -181,7 +219,7 @@ def _wait_until(self, predicate, timeout: float) -> tuple[bool, float]: time.sleep(POLL_INTERVAL) def _add(self, value: str, uid: str | None = None) -> None: - self.c.add(f"{KEY}: {value}", user_id=uid or self.uid, app_id=APP_ID, + self.c.add(f"{KEY}: {value}", user_id=uid or self.uid, app_id=self.app_id, metadata={"key": KEY}, infer=False) def _delete_key(self) -> int: @@ -243,12 +281,74 @@ def run(self, first: str, second: str, extra_settle: float) -> dict: if landed else f"re-added value NOT retrievable {where} within " f"{CONFIRM_TIMEOUT:.0f}s").replace(" ", " "), + "delete_mechanism": "per-key delete by id", "cross_scope": self.cross_scope, "delete_observed_empty_after_s": round(empty_after), "extra_settle_s": extra_settle, "reads": self.reads, } + def run_namespace(self, values: list[str], probe: str, + extra_settle: float) -> dict: + """Arms (f)/(g): the `reset()` condition, reproduced exactly. + + Deliberately NOT the per-key delete used by (a)-(e). `reset()` issues + `delete_all(app_id=...)` — one call wiping the namespace across every + scope — then polls the namespace until it reads empty and returns, + after which the runner immediately writes. That sequence is what this + reproduces, including the emptiness poll, because the open question is + whether reading empty actually means the delete has propagated. + """ + # No opening cleanup: this arm uses a fresh `app_id`, so the namespace + # is empty already and the delete_all below is the only one in the arm. + # Seed real residue across a couple of scopes, as a live namespace has. + for i, value in enumerate(values): + self._add(value, f"{self.uid}__s{i % 2}") + seeded, _ = self._wait_until( + lambda: len(self._namespace_rows()) >= len(values), CONFIRM_TIMEOUT) + if not seeded: + return {"arm": self.name, "outcome": "SETUP_FAILED", + "detail": f"only {len(self._namespace_rows())} of " + f"{len(values)} seed writes became retrievable", + "reads": self.reads} + + # The call reset() makes, verbatim. + self.c.delete_all(app_id=self.app_id) + emptied, empty_after = self._wait_until( + lambda: not self._namespace_rows(), CONFIRM_TIMEOUT) + if not emptied: + return {"arm": self.name, "outcome": "NAMESPACE_NEVER_EMPTIED", + "detail": f"namespace still non-empty {CONFIRM_TIMEOUT}s " + "after delete_all", "reads": self.reads} + + if extra_settle: + time.sleep(extra_settle) + + target = f"{self.uid}__s0" + self._add(probe, target) + landed, took = self._wait_until( + lambda: self._visible(probe, target), CONFIRM_TIMEOUT) + self.c.delete_all(app_id=self.app_id) + + return { + "arm": self.name, + "outcome": "WRITE_VISIBLE" if landed else "WRITE_LOST", + "detail": (f"post-reset write retrievable after {took:.0f}s" + if landed else + f"post-reset write NOT retrievable within " + f"{CONFIRM_TIMEOUT:.0f}s"), + "delete_mechanism": "delete_all(app_id) — as reset()", + "residue_seeded": len(values), + "namespace_read_empty_after_s": round(empty_after), + "extra_settle_s": extra_settle, + "reads": self.reads, + } + + def _namespace_rows(self) -> list[dict]: + self.reads += 1 + resp = self.c.get_all(filters={"app_id": self.app_id}) + return resp.get("results", []) if isinstance(resp, dict) else (resp or []) + def main() -> int: ap = argparse.ArgumentParser(description=__doc__) @@ -312,6 +412,11 @@ def main() -> int: ("e_cross_scope_settle", f"saltmarsh-{stamp}", f"saltmarsh-{stamp}", EXTRA_SETTLE, True), ] + # (name, extra settle) — namespace arms, run via run_namespace(). + namespace_plan = [ + ("f_namespace_delete_all", 0.0), + ("g_namespace_settle", EXTRA_SETTLE), + ] results = [] for name, first, second, settle, cross in plan: @@ -328,6 +433,22 @@ def main() -> int: for k, v in result.items(): print(f" {k}: {v}") + for name, settle in namespace_plan: + print(f"\n--- arm {name} (namespace delete_all — the reset() condition) ---") + print(f" expected SEARCH cost, worst case: ~{COST_ESTIMATE[name]} units") + before = quota_remaining(api_key) + arm = Arm(client, name, f"mc_diag__{name}_{stamp}", + app_id=f"{APP_ID}_{name}_{stamp}") + result = arm.run_namespace( + [f"residue-{i}-{stamp}" for i in range(_RESIDUE_N)], + f"probe-{name}-{stamp}", settle) + after = quota_remaining(api_key) + if before is not None and after is not None and before >= 0 and after >= 0: + result["search_units_spent"] = before - after + results.append(result) + for k, v in result.items(): + print(f" {k}: {v}") + print("\n" + "=" * 68) for r in results: print(f" {r['arm']:26s} {r['outcome']}") @@ -383,6 +504,35 @@ def main() -> int: else: print(" Pattern not anticipated. Record the outcomes and reason from them.") + f, g = by.get("f_namespace_delete_all"), by.get("g_namespace_settle") + W_LOST, W_OK = "WRITE_LOST", "WRITE_VISIBLE" + print("\n NAMESPACE (f)-(g) — the reset() condition, and the leading " + "hypothesis:") + if f == W_LOST and g == W_OK: + print(" (f) failed, (g) succeeded -> POLLING A NAMESPACE UNTIL EMPTY IS\n" + " NOT SUFFICIENT. reset() does exactly that and then returns, so\n" + " every run is exposed: the first writes after any reset can be\n" + " reaped by a delete_all that has stopped being visible. This is\n" + " a harness-side defect as much as a provider behaviour — the fix\n" + " is a settle after reset, sized from the measured empty-to-safe\n" + " gap, not a fixed sleep. Founder ruling before publication.") + elif f == W_LOST and g == W_LOST: + print(" (f) and (g) both failed -> waiting does not clear it either. A\n" + " delete_all suppresses subsequent writes to the namespace for\n" + " longer than 60s, or until something other than time changes.\n" + " The most serious form, and it would make delete_all unusable as\n" + " a reset primitive.") + elif f == W_OK and all(x == "RE_ADD_VISIBLE" for x in (a, b, c, d, e) if x): + print(" (f) succeeded and every other arm passed -> no isolated arm\n" + " reproduces the abort. Combined with the standing caveat, that\n" + " points hard at load/backlog dependence, which only a full run\n" + " can exhibit. Do NOT read it as 'nothing is wrong'.") + elif f == W_OK: + print(" (f) succeeded -> the reset() condition does not reproduce in\n" + " isolation. Read it against the per-key arms before concluding.") + else: + print(" Pattern not anticipated. Record the outcomes and reason from them.") + print("\nNo conclusion may be published from a single execution of this " "script. Re-run to establish stability first.") diff --git a/src/memorycheck/adapters/mem0.py b/src/memorycheck/adapters/mem0.py index 4e41eb3..56f8995 100644 --- a/src/memorycheck/adapters/mem0.py +++ b/src/memorycheck/adapters/mem0.py @@ -165,9 +165,12 @@ def write( f"mem0 write {key!r}={value!r} for scope " f"{scope.tenant_id}/{scope.user_id} was not retrievable after " f"{waited:.1f}s — the store did not accept the write, so no " - "result from this run would be meaningful. Known trigger: " - "re-adding text identical to a value deleted moments earlier " - "(see HANDOFF, scenario 012)" + "result from this run would be meaningful. Evidenced trigger: " + "a write issued after a delete_all, even when the namespace " + "was polled until it read empty first — see reset() above, " + "where 6/14 writes were lost following a delete_all versus " + "0/10 with no preceding delete. Reading empty is not proof " + "the delete finished propagating." ) def delete(self, scope: Scope, key: str) -> None: