Repository navigation
Hybrid Vector-Graph Retrieval Layer (Qdrant / SQLite-VSS Adapter) #60
Description
🎯 Feature Design: Hybrid Vector-Graph Retrieval Layer (Qdrant / SQLite-VSS Adapter)
1. Context & Problem Statement
Vector search (Naive RAG) is excellent at fuzzy semantic matching (e.g., matching the query "where is the database connection initialized?" to the code block containing SQLiteStore.connect). However, it is completely blind to relationships. Conversely, our Graph Engine is excellent at deterministic relationships, but cannot perform semantic searches.
We must combine them into a Hybrid Retrieval Layer.
A user or AI agent executes a fuzzy semantic query. The Vector Store identifies the top
2. Ports & Adapters Architecture (Hexagonal Design)
To support both zero-dependency local runs and enterprise SaaS scaling, we isolate the vector database behind an interface.
Ingestion / Query Pipeline (Core)
↓ (Calls Port)
[ BaseVectorStore ] (Port)
┌──────────┴──────────┐
▼ ▼
[ SQLiteVSSStore ] [ QdrantStore ] (SaaS Adapter)
(Local Adapter)
The Interface:
# Pseudo-code for src/cgis/storage/vector_base.py
from abc import ABC, abstractmethod
from typing import List, Tuple
class BaseVectorStore(ABC):
@abstractmethod
def connect(self) -> None:
pass
@abstractmethod
def upsert_embeddings(self, records: List[Tuple[str, List[float]]]) -> None:
"""Upserts a batch of (node_fqn, embedding_vector) records."""
pass
@abstractmethod
def delete_embeddings(self, node_fqns: List[str]) -> None:
"""Deletes embeddings for specific FQNs (used during incremental updates)."""
pass
@abstractmethod
def search_similarity(self, query_vector: List[float], limit: int = 5) -> List[Tuple[str, float]]:
"""Returns List of (node_fqn, cosine_similarity_score)."""
pass3. Algorithmic Blueprint
Part A: Incremental Embedding Ingestion (pipeline.py)
We must only generate and upsert embeddings for nodes that belong to changed files during the incremental ingest phase. We index the Node.metadata["docstring"] or the Node.name if no docstring is present.
# Pseudo-code for src/cgis/pipeline.py
def _persist_incremental(self, store: SQLiteStore, vector_store: BaseVectorStore, changed_files: Dict[str, str], ...):
# 1. Save SQLite graph atomically as usual
store.save_incremental_batch(...)
# 2. Extract and generate embeddings ONLY for modified nodes
embedding_records = []
for node in all_nodes:
if node.file_path in changed_files:
# Construct semantic text payload (FQN + Docstring or Name)
docstring = node.metadata.get("docstring", "")
payload = f"Node FQN: {node.id}\nType: {node.type.value}\nDescription: {docstring or node.name}"
# Generate embedding vector (using local fast-embed or external API)
vector = self._embedding_service.embed_text(payload)
embedding_records.append((node.id, vector))
# 3. Upsert to Vector Database
if embedding_records:
# First, delete old stale embeddings for these FQNs
stale_fqns = [record[0] for record in embedding_records]
vector_store.delete_embeddings(stale_fqns)
vector_store.upsert_embeddings(embedding_records)Part B: The Hybrid Query Orchestrator (query/hybrid.py)
This coordinator merges vector retrieval with graph traversal.
# Pseudo-code for src/cgis/query/hybrid.py
class HybridRetriever:
def __init__(self, store: SQLiteStore, vector_store: BaseVectorStore, embedding_service):
self.store = store
self.vector_store = vector_store
self.embedding_service = embedding_service
self.query_engine = QueryEngine(store)
def retrieve_context(self, query_text: str, limit: int = 3) -> dict:
"""
Executes a hybrid semantic-structural query.
"""
# 1. Generate query embedding vector
query_vector = self.embedding_service.embed_text(query_text)
# 2. Query Vector DB for top semantic matches
matches = self.vector_store.search_similarity(query_vector, limit=limit)
results = []
# 3. For each matched FQN, pull its structural and behavioral call graphs
for fqn, score in matches:
node = self.store.get_node(fqn)
if node:
# Retrieve execution flow graph (depth 1 to save tokens)
nodes, edges = self.query_engine.get_flow_graph(fqn, max_depth=1)
results.append({
"focal_node": node,
"similarity_score": score,
"subgraph": {"nodes": nodes, "edges": edges}
})
return {"query": query_text, "results": results}4. TDD Acceptance Criteria
- Test 1 (Fuzzy Semantic Match): Given a node
sqlite_store.save_graphwith a docstring "Saves nodes and edges to SQLite". A semantic query "how do we persist data?" must returncgis.storage.sqlite_store.SQLiteStore.save_graphas the top vector match. - Test 2 (Incremental Deletion): If a file containing mapped nodes is deleted, verify that running incremental ingest successfully deletes both the nodes from SQLite and their corresponding embedding vectors from the Vector Database.
- Test 3 (Stateless Portability): Verify that we can instantiate the
HybridRetrieverusing either a mockSQLiteVSSStoreor a mockQdrantStoreadapter without changing the query orchestrator logic.
5. Architectural High-Thinking (Watch-outs)
- Join Key Invariant (The FQN Anchor): We must never store raw file paths or database auto-increment IDs as the vector reference. The FQN string is our absolute, platform-independent anchor. This allows the vector DB (which might run in the cloud on Qdrant) and the graph DB (which runs locally on SQLite) to be completely decentralized while maintaining perfect referential integrity.
- Local Zero-Dependency Fallback: For local CLI usage, the system must not require a running Docker container. We can implement
SQLiteVSSStoreusingsqlite-vss(or simple in-memory numpy cosine similarity for small repositories), making the entire vector layer zero-setup out of the box.
Activity
feat: cgis_find_symbol — partial name → ranked FQNs (#173) (#206)
zaebee commentedon Oct 2, 2026
Folding into #220: the lexical retrieval rung shipped as
cgis_find_symbol(#206); vector/semantic search is the remaining rung on top ofcgis context.Generated by Claude Code