Repository navigation
Nominate independently operated memory systems for Phase 3 experiment #6
Description
Phase 3 system nominations
LLM Errata is seeking nominations for three independently operated AI-memory
systems for a synthetic-data interoperability experiment.
Repository: https://github.com/thomaswillner/llm-errata
Current verdict: NOT_PROD_READY. The experiment has not started.
Required experiment shape
One synthetic memory root must receive a correction, supersession, and erasure
across three independently operated systems. The set must include:
- one intentionally nonconforming importer;
- one system whose coverage is genuinely incomplete or opaque; and
- at least one system capable of recording derivation across mixed artifacts.
Measurements include observation-to-quarantine time, known-descendant coverage,
stale-behavior rate, replacement activation, collateral retention, stale-reimport
resistance, opaque coverage, operator effort, and user-visible friction.
Nomination requirements
Please name:
- system, operator, public documentation, and version;
- memory stores and derivation surfaces exposed;
- API, account, cost, and data-residency requirements;
- deletion, correction, export, and audit capabilities;
- known opaque stores or unsupported coverage; and
- whether the operator can authorize a synthetic experiment and publish results.
Nomination is not approval. Every system, account, integration, cost, and
interaction requires explicit owner authorization before the experiment begins.
Activity
thomaswillner commentedon Aug 12, 2026
Phase 3 nomination contract is now included in draft PR #8:
#8
Exact current implementation target:
35052868cde8b6f16fc6c20b4136ae0277f9a00e.No experiment has started. Three independently operated systems, two independent adapters, a separate validator, owner authorization for synthetic data/account/cost, and external review remain required before execution. Please nominate systems using the fields in
PHASE3_SYSTEMS.md, including opaque stores and authorization/publication constraints.Status remains
NOT_PROD_READY; a nomination is not approval or interoperability evidence.thomaswillner commentedon Aug 12, 2026
@kartik-mem0 — inviting Mem0 as a candidate independently operated system for a later synthetic-data LLM Errata interoperability experiment.
Why this fit is specific: Mem0 is a large, actively maintained memory layer with multiple SDKs/integrations and vector/graph-backed behavior. The experiment needs real systems where correction, descendant coverage, stale recall, and opaque provider-controlled surfaces can be measured rather than assumed.
Nomination contract: https://github.com/thomaswillner/llm-errata/blob/agent/g2-publication/PHASE3_SYSTEMS.md
Draft program: #8
This is only a nomination request:
NOT_PROD_READY.If this is of interest, a brief nomination naming OSS or hosted scope and an operator contact is enough for the next planning gate.
thomaswillner commentedon Aug 12, 2026
@Vasilije1990 — inviting Cognee as a candidate independently operated system for the later three-system LLM Errata experiment.
Why this fit is specific: Cognee spans knowledge graph, vector, relational/session, and multiple storage backends, including explicitly demo-vs-production boundaries. That makes it useful for measuring known-descendant coverage and honestly reporting stores that cannot be fully inspected or repaired.
Nomination contract: https://github.com/thomaswillner/llm-errata/blob/agent/g2-publication/PHASE3_SYSTEMS.md
Draft program: #8
This is only a nomination request:
NOT_PROD_READY.A preliminary “possible/impossible and why” response would already improve experiment design.
thomaswillner commentedon Aug 12, 2026
Final Phase 2 target available for three-system nominations
Internal Phase 2 is complete and frozen at:
50e895fbfec544b16c94caa07bf2d1f4049a42e29547aec8328b601489dda067c6e62f287229b2b24a413dac2c9e7be98e429804Three independently operated real memory systems are still needed for a synthetic-data experiment. A nomination should identify system and exact version, operator, stores and lineage visibility, correction/deletion/export/audit capabilities, opaque or provider-controlled surfaces, account and cost requirements, data constraints, and whether raw artifacts and results may be published.
No experiment has started. Nomination does not authorize account creation, credentials, cost, deployment, data transfer, or publication. Execution additionally requires two independent adapters, a separate validator, qualifying external review, and explicit operator authorization for every system.
G5 remains
BLOCKED; repository remainsNOT_PROD_READY. Zero systems have supplied qualifying authorization or experiment evidence so far.thomaswillner commentedon Aug 13, 2026
Corrected Phase 3 target and operator nomination request
The earlier frozen target in this thread is historical. The current experiment contract is:
08b95263c9ed700c43aea0b285696956cc23e87803abc492319b875a7d528e0e8de05714bc5a7219b42031fc7c3f42cff1f0bf14@kartik-mem0 — Mem0 remains relevant because its hosted and open-source surfaces expose real vector/graph-backed memory behavior and provider-controlled boundaries. Would you kindly nominate a specific Mem0 version and OSS or hosted scope that an independent operator could use for the synthetic-data experiment, or state why the experiment is impractical?
@Vasilije1990 — Cognee remains relevant because its graph, vector, relational/session, and multiple-backend surfaces can test descendant coverage and honest reporting of opaque stores. Would you kindly nominate a specific Cognee version/backend and operator scope, or state why the experiment is impractical?
A nomination is not authorization. No account creation, credential use, cost, deployment, data transfer, or experiment is requested by this comment. A later run requires explicit operator authorization, synthetic data, independent adapters, a separate validator, and publishable raw evidence. Invitations do not satisfy G5. Current verdict remains
NOT_PROD_READY; G5 remainsBLOCKED.