Repository navigation
"Prompt as Memory" (Memory Surface) #25
Description
🎯 Feature Design: Adaptive Context Snapshotting & Multi-Dimensional Change Memory
1. Context & Goal
Following extensive review, we are hardening our Meta-Memory Database. Instead of treating memory as a flat text log, we treat it as an active training surface for RAG optimization.
By recording multi-dimensional scores of context execution, and storing compressed snapshots of failed contexts (negative examples), we build a predictive engine that learns exactly what context configuration is required for any given FQN before the agent even writes code.
2. Hardened Data Contract
# Pseudo-schema inside src/cgis/prompting/memory.py
from pydantic import BaseModel, Field
from typing import Dict, Any, Optional
class ContextSnapshot(BaseModel):
# Identifiers
context_hash: str = Field(..., description="MD5 hash of the compiled XML context")
focal_fqn: str = Field(..., description="Target FQN being modified")
commit_sha: str = Field(..., description="Git commit hash containing the change")
# Context Metrics
context_depth: int
retrieved_nodes_count: int
token_count: int
# Multi-dimensional Effectiveness Scores
correctness: bool = Field(default=True, description="Did the change pass unit tests / CI?")
confidence: float = Field(default=1.0, ge=0.0, le=1.0, description="Agent's self-assessed confidence")
blast_radius: int = Field(default=1, description="Number of files modified in this commit (proxy for regression)")
efficiency: float = Field(default=0.0, description="Tokens used per line of modified code (lower is better)")
# Portable Topic Indexing for AgentRelay
topic_tag: str = Field(default="generic", description="Semantic category (e.g., 'db.session', 'ts.imports')")
# Failed Context Payload (Stored ONLY on failure to prevent DB bloat)
failed_context_payload: Optional[str] = Field(default=None, description="Raw XML context (only set if correctness=False)")-- Upgraded Database Schema for SQLite
CREATE TABLE IF NOT EXISTS context_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
context_hash TEXT NOT NULL,
focal_fqn TEXT NOT NULL,
commit_sha TEXT NOT NULL,
-- Metrics
context_depth INTEGER NOT NULL,
retrieved_nodes_count INTEGER NOT NULL,
token_count INTEGER NOT NULL,
-- Multi-dimensional Evaluation
correctness INTEGER DEFAULT 1, -- Boolean represented as 0 or 1
confidence REAL DEFAULT 1.0,
blast_radius INTEGER DEFAULT 1,
efficiency REAL DEFAULT 0.0,
topic_tag TEXT DEFAULT 'generic',
-- Memory Footprint Optimization (BLOB for compressed XML context on failure)
failed_context_blob BLOB DEFAULT NULL,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
-- Compound Index for finding pattern failures
CREATE INDEX IF NOT EXISTS idx_context_eval ON context_history (correctness, focal_fqn, topic_tag);3. Algorithmic Blueprint
Part A: Zero-Bloat Git Reference Integration
To prevent SQLite bloating from raw diffs, we abstract the diff retrieval. The database stores only the commit_sha. When TruLens or Ragas requests evaluation, we query Git dynamically on-demand.
# Pseudo-code for retrieving diffs on-demand
import subprocess
def get_git_diff_for_commit(commit_sha: str, file_path: str) -> str:
"""Queries git dynamically for the diff of a specific file in a commit."""
try:
result = subprocess.run(
["git", "show", commit_sha, "--", file_path],
capture_output=True,
text=True,
check=True
)
return result.stdout
except subprocess.CalledProcessError:
return ""Part B: Negative Example Compression
We only save the XML payload of failed attempts to conserve space. We compress it using zlib.
# Pseudo-code for compression
import zlib
def compress_failed_context(xml_payload: str) -> bytes:
return zlib.compress(xml_payload.encode("utf-8"))
def decompress_failed_context(blob: bytes) -> str:
return zlib.decompress(blob).decode("utf-8")4. TDD Acceptance Criteria
- Test 1 (Iterative Commit Safety): Inserting multiple rows with the same
context_hashbut differentcommit_sha(iterative attempts) must succeed without throwing unique constraint violations. - Test 2 (Git Diff Abstraction): Verify that the test runner can fetch a real git diff from a temporary git repository using only the
commit_shastored in the row. - Test 3 (Asymmetric Storage): Verify that a successful snapshot (
correctness=True) leavesfailed_context_blobasNULL, whereas a failed snapshot (correctness=False) correctly compresses and stores the XML payload.
5. Design constraints (added 2026-10-02 from review comments)
From hegu-1's comment:
- Snapshot ≠ memory. A snapshot is evidence about one run, not a reusable rule. Lifecycle: immutable execution snapshot → attached evaluation → candidate lesson / retrieval rule → reviewed promotion to durable memory → generated index used for context assembly. A low score alone never promotes anything. (Prior art in this repo:
FinderRecordinginsrc/cgis/guardian/recording.pyalready freezes a run together with its diff.) - Provenance and expiry. Every snapshot and candidate carries repository + commit, focal FQN, model and tool versions, prompt compiler version and evaluator version, plus a supersession/expiry path, so an old negative example stops steering retrieval once the code or task contract changes.
- No MD5 for
context_hash. Use sha256 over a canonical serialization of the compiled context. (The MD5 inpipeline.pyis file change detection only and is out of scope.)
Not adopted: mapping to the external "Universal Memory Protocol" spec. No concrete need for cross-framework interop yet.
Open question on the metrics: blast_radius (files per commit) is a noisy proxy for missing context. The sufficiency / allocation-efficiency metrics proposed in #543 are likely better signals.
6. Sequencing
- Prerequisite: bench: agent A/B benchmark (with vs without cgis MCP) as the headline metric #543 (agent A/B benchmark). It produces the first real "context → outcome" data and tells us which metrics are worth persisting here.
- Depends on this: Technical Strategy: Bridging the Resolution Gap #66 phase 2 (self-healing resolution loop), absorbed here on 2026-09-16.
Activity
zaebee commentedon Jun 7, 2026
Hi @kehansama,
This is a great expansion of the schema.
We have upgraded our proposed architecture draft (Issue #25) to incorporate these guidelines. Here is how we are structuring the implementation:
1. Multi-Dimensional Effectiveness Scoring
We replaced the flat
is_successfulboolean with your suggested four-dimensional matrix:correctness(Boolean): Test suite outcome.efficiency(Float): Calculated astoken_count / lines_modifiedin the resulting git diff. Lower ratio means the agent achieved high-density changes with minimal context.confidence(Float): Self-assessed by the agent prior to commit.blast_radius(Integer): Total number of files modified in the commit. If the blast radius is high for a supposedly local change, it mathematically indicates that the context provided to the agent was incomplete (missing transitive upstream/downstream boundaries).2. Asymmetric Failure Retention (The Power of Negative Examples)
To solve our "Git Bloat" database constraint while capturing high-value negative examples, we designed an asymmetric storage strategy:
correctness=True): We store only the FQN, thecommit_sha, and metadata. We do not store the compiled XML context. If we ever need the code diff, we query Git dynamically on-demand (git show <commit_sha> -- <file>), keeping our SQLite database footprint minimal.correctness=False): We strictly serialize, compress viazlib, and store the entire compiled XML context payload in afailed_context_blobcolumn. This provides us with a high-density, portable corpus of exactly what went wrong, allowing us to evaluate what information was missing from the agent's prompt during the failure.3. Topic-Indexed Memory Surfaces for AgentRelay
To make these profiles portable across projects (cross-agent generalization), we introduced a
topic_tagmetadata field (e.g.,lang.ts.import_resolution,framework.fastapi.route_binding).This directly aligns with your AgentRelay concept of "Memory Surfaces". Once indexed, we can evaluate which context depths and centrality-pruning thresholds yield the highest
efficiencyandcorrectnessfor specific topic tags across multiple distinct codebases.4. RAG Evaluation Collaboration
We are extremely excited about the TruLens integration. This dataset acts as the ground truth of "context vs. outcome." We would love to collaborate on standardizing this
context_historyschema so that any project running CGIS can output standardized benchmarks compatible with TruLens/Ragas evaluations.Let's sync up on the integration design as soon as our cross-file symbol resolution is locked down!
edihasaj commentedon Jun 29, 2026
Nice design and thoughtful tradeoffs around storing only failed payloads and using commit SHAs for diffs. If you want to make that schema interoperable across agent tools and vector stores, you might map these fields to a portable memory contract instead of a project-local table; that makes it easier to share snapshots, provenance, and failure payloads between frameworks. I maintain an open spec and reference implementation for a Universal Memory Protocol that defines identifiers, metadata (timestamps, provenance, consent), and sync semantics you could adapt for ContextSnapshot -- see the repo at https://github.com/edihasaj/universal-memory-protocol.
hegu-1 commentedon Jul 15, 2026
The snapshot model is useful, but I would keep an execution snapshot separate from durable memory.
A failed-context record is evidence about one run under a specific code state, prompt compiler, model, tool set, and task. It should not become a reusable rule merely because its score was low. A safer lifecycle is:
Each candidate should retain repository/commit, FQN, model/tool versions, evaluator version, and a supersession/expiry path. Otherwise an old negative example can keep steering retrieval after the code or task contract changes.
I would also avoid MD5 for a content-addressed authority boundary; use a collision-resistant digest and canonical serialization.
This Continuity/Snapshot/Trace split is close to the distinction I mean: snapshot captures a phase/run, trace explains the path, and continuity carries only what remains active: https://github.com/hegu-1/personal-ai-os/blob/main/docs/continuity-snapshot-trace.md
zaebee commentedon Sep 16, 2026
Absorbing Phase 2 of #66 (self-healing resolution loop), which depends on this memory surface.
zaebee commentedon Oct 2, 2026
Review of the comments so far (2026-10-02):
Generated by Claude Code