Shared Memory Failures in Multi-Agent LLM Systems

Research code that systematically studies how shared memory in multi-agent LLM systems can cause systematic recall failures — and whether authority- and recency-aware retrieval mitigates them.

The framework spins up small teams of role-played LLM agents (e.g. a senior partner, associate, and client in a legal domain) that converse over a scripted, multi-turn scenario sharing a common vector memory store. At certain turns, an agent is asked to recall a fact from earlier in the conversation. The experiment measures how often that recall is correct, and breaks failures down into three named failure modes.

Failure Modes Studied

• Belief Asymmetry — an agent's own prior reasoning persists in its private context, causing it to contradict updated facts in shared memory (stale reasoning dominates fresh evidence).
• Authority Collapse — a low-authority agent's (possibly wrong) statement overrides a higher-authority agent's correct one in retrieval results.
• Memory Anchoring — early information (right or wrong) anchors later recall even after it has been corrected, reinforced by multiple agents repeating the error.

Experiment Conditions

Condition	Shared Memory	Scoring	Invalidation
Baseline	Off	—	—
Naive	On	similarity + recency only (no authority weighting)	No
Lightweight	On	similarity + authority + recency	Optional removal of low-authority conflicting entries

Memory retrieval uses a composite score:

score = alpha * similarity + beta * authority + gamma * recency

where:

recency = 1 / (1 + (current_turn - entry_turn))

The weights (alpha, beta, gamma) are defined per experiment condition in config.py.

For example:

• In the Naive condition, beta = 0 (authority is ignored).
• In the Lightweight condition, beta = 0.4 (authority is weighted).

Domains

Three role-played domains ship out of the box, each with 5 roles and authority weights (0–1):

Legal

• senior_lawyer (1.0)
• associate (0.7)
• paralegal (0.4)
• external_counsel (0.9)
• client_rep (0.2)

Medical

• attending_physician (1.0)
• resident (0.6)
• nurse (0.4)
• pharmacist (0.8)
• patient (0.0)

Finance

• portfolio_manager (1.0)
• financial_analyst (0.75)
• risk_officer (0.85)
• trader (0.6)
• compliance_officer (0.8)

Pre-generated scripts for Legal (10 scripts) and Medical (5 scripts) are included in the scripts/ folder.

Additional scripts (including Finance) can be generated using the --generate-scripts flag if you provide a generate_scripts.py module.

---

Architecture

File	Purpose
main.py	CLI entry point — parses args and wires everything together
config.py	Domain roles, system prompts, authority weights, experiment conditions
shared_memory.py	SharedMemory class — ChromaDB-backed vector store with composite scoring and authority-based invalidation
agents.py	LLM call wrapper (with retries) and the agent node that reads memory, calls the LLM, and writes back
graph.py	LangGraph StateGraph that loops the agent node over every turn in a script
experiment_runner.py	Orchestrates scripts × conditions × repetitions, preloads initial facts, and collects results
evaluation.py	Recall scoring (exact substring match or llm-as-judge) and metric aggregation
utils.py	Script loading/validation, seeding, and logging setup

---

Installation

git clone https://github.com/ebaadraheem/shared-memory-failures.git
cd shared-memory-failures
pip install -r requirements.txt

Set your OpenAI API key (used for both role-played agents and the optional LLM-judge evaluator):

echo "OPENAI_API_KEY=sk-..." > .env

If no key is set, agents fall back to mock responses, which is useful for dry-running the pipeline.

---

Scripts

Scenarios live as JSON files under:

scripts/<domain>/*.json

The repository includes pre-generated Legal and Medical scripts.

You can also author your own scripts or provide a generate_scripts.py module exposing:

generate_all_scripts(output_dir, scripts_per_domain, seed)

This function is referenced by --generate-scripts but is not shipped with the repository.

Expected Script Schema

{
  "script_id": "legal_01",
  "domain": "legal",
  "initial_facts": [
    {
      "id": "F1",
      "text": "...",
      "tier": "structural",
      "authority_level": 1
    }
  ],
  "turns": [
    {
      "turn": 1,
      "role": "user",
      "content": "Shared matter file initialized...",
      "is_recall": false,
      "target_fact_id": null,
      "ground_truth": [],
      "recall_query": "...",
      "failure_mode": "",
      "write_to_memory": false
    },
    {
      "turn": 6,
      "role": "paralegal",
      "content": "Shared memory update: deadline corrected...",
      "is_recall": false,
      "target_fact_id": null,
      "ground_truth": [],
      "recall_query": "...",
      "failure_mode": "",
      "write_to_memory": true
    },
    {
      "turn": 9,
      "role": "senior_lawyer",
      "content": "We remain anchored to the old date...",
      "is_recall": true,
      "target_fact_id": "F5",
      "ground_truth": ["July 22, 2026"],
      "recall_query": "What is the current deadline?",
      "failure_mode": "belief_asymmetry",
      "write_to_memory": false
    }
  ]
}

Field Notes

• write_to_memory should be true only on turns where an agent explicitly updates shared memory (for example, a paralegal correcting a deadline).

• For all other turns, set it to false to avoid memory pollution.

• For every is_recall: true turn, you must provide:

  • target_fact_id
  • recall_query
  • ground_truth
  • failure_mode

---

Usage

Run all conditions with 10 repetitions per script (default):

python main.py

Run a single condition:

python main.py --conditions Baseline

Run multiple conditions with custom repetitions:

python main.py --conditions Baseline Lightweight --repetitions 3

Use an LLM judge instead of exact substring matching:

python main.py --eval-method llm

Generate synthetic scripts before running:

python main.py --generate-scripts --scripts-per-domain 10

Full CLI Options

Flag	Choices / Type	Default	Description
--conditions	Baseline, Naive, Lightweight, all	all	Which experiment conditions to run
--repetitions	int	10	Repetitions per script per condition
--scripts-dir	path	scripts	Path to scripts directory
--results-dir	path	results	Output directory for CSV files
--eval-method	exact or llm	exact	Recall evaluation method
--seed	int	42	Base random seed
--generate-scripts	flag	False	Generate synthetic scripts before running
--scripts-per-domain	int	10	Scripts to generate per domain
--log-level	DEBUG, INFO, WARNING, ERROR	INFO	Logging verbosity
--no-save-intermediate	flag	False	Skip per-condition intermediate CSVs

---

Output

Results are written to --results-dir (default: results/):

• results_<condition>.csv — raw per-turn results for each condition (unless --no-save-intermediate)
• results_all.csv — full combined results with evaluation columns (correct, partial_score, matched)
• metrics_summary.csv — accuracy grouped by domain × failure mode × condition (main paper table)
• condition_summary.csv — accuracy grouped by condition only
• experiment.log — run log

A summary table is also printed to stdout at the end of each run.

---

Requirements

Python 3.10+ with:

langgraph
langchain-core
chromadb
sentence-transformers
openai
pandas
numpy
tqdm
python-dotenv

(See requirements.txt for exact versions.)

OpenAI API key:

• Optional
• Falls back to mock responses if unavailable

---

Reproducibility

All experiments use a fixed random seed (--seed, default 42).

The SharedMemory class resets the vector database before each run.

Combined with the fixed script structure, this ensures experiments are fully reproducible.

---

License

MIT License .