AI Memory Systems Benchmark

Comparative evaluation framework for AI memory architectures using standardized benchmarks (LoCoMo, LongMemEval).

Purpose

Side-by-side comparison of memory systems for learning/research:

• Graphiti (baseline) - Your existing FalkorDB + LanceDB deployment (port 8000)
• OpenMemory ✅ DEPLOYED - CaviraOSS HSG v3 system (port 8080)
• Zep (~75.1% LoCoMo) - Session-aware, production-ready
• Mem0 (66.9% LoCoMo) - Pre-1.0, for reference only

Structure

memory-benchmarks/
├── benchmarks/
│   ├── locomo_eval.py      # LoCoMo benchmark implementation
│   ├── longmemeval.py      # LongMemEval implementation
│   └── latency_test.py     # Query latency comparison
├── systems/
│   ├── graphiti/           # Config for existing deployment
│   ├── openmemory/         # ✅ CaviraOSS OpenMemory (deployed)
│   └── zep/                # Docker deployment (planned)
├── results/                # Benchmark results
├── docs/                   # Research and documentation
└── docker-compose.yml      # Multi-system orchestration

Quick Start

# Run LoCoMo benchmark against Graphiti
python benchmarks/locomo_eval.py --system graphiti

# Compare all systems
python benchmarks/compare_all.py

Benchmark Standards

LoCoMo (Long-term Conversational Memory)

• 1,540 questions across multi-session conversations
• Tests: single-hop recall, multi-hop reasoning, temporal ordering, commonsense
• Evaluation: Exact match + LLM-as-a-judge scoring
• Reference: https://arxiv.org/abs/2402.01440

LongMemEval

• Tests long-context memory retention
• Complementary to LoCoMo

Current Benchmark Leaderboard

System	LoCoMo	Status	Notes
Graphiti/Zep	State of the Art	✅ Port 8000	Paper: arXiv:2501.13956 (Jan 2025)
OpenMemory	80.1% (claimed)	✅ Port 8080	HSG v3 + FAISS + BM25
Mem0	66.9%	⚠️ Pre-1.0	SSRF vulnerability

Key Discoveries

2025-12-11: Zep = Graphiti's parent company! Your deployment already has state-of-the-art technology.

2025-12-12: OpenMemory deployed and benchmarked:

• 100% accuracy on simple queries (both systems tied)
• OpenMemory: 2.8ms warm query latency, 5 cognitive memory sectors
• Graphiti: 3-6ms query latency, knowledge graph with temporal validity
• See results/2025-12-11_openmemory_vs_graphiti.md for full comparison

Prerequisites

• Python 3.11+
• Docker + Docker Compose
• Existing Graphiti deployment (port 8000)
• LiteLLM proxy (port 4000) for LLM-as-a-judge

Related Research

• docs/work/research/reports/2025-12-11_mem0-ai-memory-layer-comprehensive-analysis.md
• docs/work/research/reports/2025-12-11_openmemory-ai-memory-benchmarking.md