Conversational Knowledge Graph System

A dynamic knowledge graph system that learns from conversations in real-time. As users chat with the AI, their conversations are automatically indexed into both a vector database and a personalized knowledge graph, enabling context-aware responses without sending entire chat histories.

Features

• Session-Based Knowledge Graphs: Each conversation/session has its own isolated knowledge graph
• Smart Context Management: Maintains only 3 most recent messages + retrieval-based context
• Async Graph Building: Background processing with queue management to prevent race conditions
• Hybrid Retrieval: Combines sliding window + vector search + graph traversal
• Real-time Learning: Captures topics, entities, and relationships from each conversation

Architecture

Components

1. Agent System (agent/)

   • Conversational AI agent with memory tools
   • User-specific context management
   • Hybrid retrieval strategy

2. Storage Layer (storage/)

   • PostgreSQL for vector storage and conversation history
   • Neo4j for personalized knowledge graphs
   • Async queue for graph updates

3. API Layer (api/)

   • FastAPI with streaming responses
   • Session management
   • Background task handling

How It Works

User Message → Agent Processing → Response
                    ↓
              [Async Queue]
                    ↓
        Vector DB + Knowledge Graph
                    ↓
        [Background Worker]

Message Flow:

1. User sends a message
2. System retrieves context using:
   • Last 3 messages (sliding window)
   • Relevant past conversations (vector search)
   • User preferences & topics (knowledge graph)
3. Agent generates response
4. Message pair queued for async processing
5. Background worker updates vector DB and knowledge graph
6. Queue ensures no race conditions

Knowledge Graph Structure:

• Topic Nodes: Discussion subjects and themes
• Entity Nodes: People, places, concepts mentioned
• Relationship Edges: How topics and entities relate
• Episode Nodes: Conversation turns with timestamps
• Each session maintains its own isolated graph

Prerequisites

• Python 3.11+
• PostgreSQL with pgvector extension
• Neo4j database
• LLM API key (OpenAI, Ollama, etc.)

Installation

1. Set up virtual environment

python -m venv venv
.\venv\Scripts\activate  # Windows
source venv/bin/activate  # Linux/macOS

2. Install dependencies

pip install -r requirements.txt

3. Set up databases

Execute the SQL schema:

psql -d $DATABASE_URL -f sql/schema.sql

4. Configure environment

Create .env file:

# Database
DATABASE_URL=postgresql://user:password@host:port/dbname

# Neo4j
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password

# LLM Provider
LLM_PROVIDER=openai
LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=sk-your-key
LLM_CHOICE=gpt-4.1-mini

# Embeddings
EMBEDDING_PROVIDER=openai
EMBEDDING_BASE_URL=https://api.openai.com/v1
EMBEDDING_API_KEY=sk-your-key
EMBEDDING_MODEL=text-embedding-3-small

# Application
APP_PORT=8059
LOG_LEVEL=INFO

Usage

Start the API Server

python -m api.main

Use the CLI

python cli.py

Visualize the Knowledge Graph

To visualize the knowledge graph for a specific session or user:

# Visualize a specific conversation/session
python visualize_graph.py --conversation conv_user123_abc123

# Visualize all conversations for a user
python visualize_graph.py --user user123

# Visualize entire graph (all sessions)
python visualize_graph.py --limit 200

# Show detailed statistics
python visualize_graph.py --stats

# Advanced options
python visualize_graph.py --conversation conv_xyz --type interactive --output session_graph

This creates two files:

• knowledge_graph.png - Static image of the graph
• knowledge_graph.html - Interactive visualization (open in browser!)

The interactive HTML version allows you to:

• 🖱️ Click and drag nodes to explore
• 🔍 Hover over nodes to see details
• 🎯 Zoom in/out and pan around
• 📊 See all entities and relationships

You can also access graph data via the API:

# Get graph data for a specific conversation
curl "http://localhost:8059/graph/data?conversation_id=conv_xyz&limit=100"

# Get graph statistics for a conversation
curl "http://localhost:8059/graph/stats?conversation_id=conv_xyz"

# Get all graphs for a user
curl "http://localhost:8059/graph/data?user_id=user123&limit=200"

Example Conversation

You: I'm interested in learning about Python programming

🤖 Assistant: Great! Python is an excellent programming language to learn.
What specifically about Python would you like to focus on? Web development,
data science, automation, or general programming concepts?

You: I think data science sounds interesting

🤖 Assistant: Perfect! Since you mentioned data science, I can help you get started. 
The main libraries you'll want to learn are NumPy, Pandas, and Matplotlib...

[Type "new" to start a new session with a fresh knowledge graph]

You: new
✓ Started new conversation

You: What programming topics should I explore?

🤖 Assistant: There are many exciting areas in programming! What are your 
interests or goals? Are you looking to build web applications, work with data, 
create games, or something else?

Notice: Each session maintains its own isolated knowledge graph. Starting a new conversation creates a fresh context!

System Architecture

Dynamic Learning Flow

┌─────────────┐
│ User Message│
└──────┬──────┘
       │
       ├──────────────────┐
       │                  │
       ▼                  ▼
┌──────────────┐   ┌──────────────┐
│   Retrieve   │   │  Store in    │
│   Context    │   │  Vector DB   │
└──────┬───────┘   └──────────────┘
       │
       │ ┌────────────────┐
       ├─┤ Sliding Window │
       │ └────────────────┘
       │ ┌────────────────┐
       ├─┤ Vector Search  │
       │ └────────────────┘
       │ ┌────────────────┐
       └─┤ Graph Context  │
         └────────┬───────┘
                  │
                  ▼
         ┌────────────────┐
         │ Agent Response │
         └────────┬───────┘
                  │
                  ▼
         ┌────────────────┐
         │ Async Queue    │
         │ (No blocking!) │
         └────────┬───────┘
                  │
                  ▼
         ┌────────────────┐
         │ Background     │
         │ Worker Updates │
         │ Knowledge Graph│
         └────────────────┘

Race Condition Prevention

The system uses an async queue to prevent race conditions:

1. User sends message → Agent responds immediately
2. Response queued → Added to background processing queue
3. Sequential processing → Queue worker processes updates one at a time
4. Even if user sends next message → It won't interfere with graph building
5. Queue ensures order → Messages processed in the order they were sent

Context Retrieval Strategy

For each user message, the system retrieves:

1. Sliding Window (3 messages): Immediate context from current conversation
2. Vector Search (Top 5): Similar past conversations across all sessions
3. Knowledge Graph: User preferences, topics, and relationships

This hybrid approach provides both immediate and historical context without sending entire chat histories to the LLM.

Key Features Explained

Session-Based Knowledge Graphs

Each conversation has its own isolated knowledge graph:

• Complete Isolation: Knowledge from one session doesn't leak into another
• Clean Context: Each conversation maintains its own coherent knowledge base
• Privacy: Different topics/projects can have completely separate contexts
• Easy Management: Close or archive individual session graphs when conversations end

See SESSION_ARCHITECTURE.md for detailed architecture documentation.

Smart Context Management

Instead of sending entire chat histories:

• Only 3 most recent messages sent directly
• Historical context retrieved via similarity search
• Knowledge graph provides learned facts and preferences
• Reduces token usage while maintaining context

Async Graph Building

The graph building happens in the background:

• API responds immediately (no waiting for graph updates)
• Messages queued for processing
• Background worker processes sequentially
• No race conditions between rapid messages
• User can continue chatting while graph updates

What Gets Stored in the Graph

From each conversation turn, Graphiti extracts:

• Entities: Topics discussed, concepts mentioned
• Relationships: How topics relate to each other
• User Preferences: User likes, interests, goals
• Conversation Flow: How discussions evolve
• Temporal Facts: Information with timestamps

Quick Start Guide

1. Set up databases

PostgreSQL with pgvector:

psql -d your_database < sql/schema.sql

Neo4j:

• Install Neo4j Desktop or use cloud instance
• Start database and note credentials

2. Configure environment

Copy .env.example to .env and fill in:

cp .env.example .env
# Edit .env with your credentials

3. Install dependencies

pip install -r requirements.txt

4. Start the server

python -m api.main

5. Use the CLI

In a separate terminal:

python cli.py --user your_username

API Endpoints

POST /chat

Non-streaming chat endpoint.

Request:

{
  "message": "Tell me about machine learning",
  "user_id": "user123",
  "conversation_id": "optional_conversation_id",
  "metadata": {}
}

Response:

{
  "message": "Machine learning is...",
  "conversation_id": "conv_user123_abc123",
  "user_id": "user123",
  "sources_used": ["sliding_window", "vector_search", "knowledge_graph"],
  "metadata": {"queue_size": 2}
}

POST /chat/stream

Streaming chat endpoint with Server-Sent Events.

Events:

• conversation: Conversation ID
• text: Response text chunks
• end: End of response with queue size
• error: Error message

GET /health

Health check endpoint.

Response:

{
  "status": "healthy",
  "database": true,
  "graph_database": true,
  "llm_connection": true,
  "queue_size": 3,
  "version": "1.0.0",
  "timestamp": "2025-11-30T..."
}

Configuration Options

Environment variables for customization:

# Context Management
SLIDING_WINDOW_SIZE=3          # Number of recent messages to include
MAX_RETRIEVAL_RESULTS=5        # Max vector search results
ENABLE_GRAPH_CONTEXT=true      # Enable knowledge graph context

# Background Processing
GRAPH_UPDATE_DELAY=0.5         # Delay between graph updates (seconds)
MAX_QUEUE_SIZE=100             # Maximum queue size

# LLM Configuration
LLM_CHOICE=gpt-4.1-mini       # Fast model for responses
EMBEDDING_MODEL=text-embedding-3-small  # Efficient embeddings

Troubleshooting

Queue backing up

If the queue size keeps growing:

• Check Neo4j connection
• Verify LLM API is responding
• Consider using a faster/smaller model
• Increase GRAPH_UPDATE_DELAY

Context not working

If the agent doesn't remember previous conversations:

• Check that embeddings are being generated
• Verify vector search is returning results
• Check knowledge graph is being updated (see queue)

Slow responses

If responses are taking too long:

• Reduce MAX_RETRIEVAL_RESULTS
• Use a faster LLM model
• Check database connection latency

Development

Project Structure

conversational-knowledge-graph/
├── agent/              # AI agent and prompts
│   ├── agent.py       # Context retrieval and agent execution
│   └── prompts.py     # System prompts
├── api/               # FastAPI application
│   └── main.py        # API endpoints and streaming
├── storage/           # Data layer
│   ├── db_utils.py    # PostgreSQL operations
│   ├── graph_utils.py # Neo4j/Graphiti operations
│   └── queue_manager.py # Async queue for graph updates
├── sql/               # Database schema
├── models.py          # Pydantic models
├── providers.py       # LLM provider configuration
├── cli.py             # Command-line interface
└── README.md          # This file

Testing

# Test database connection
python -c "from storage.db_utils import test_connection; import asyncio; print(asyncio.run(test_connection()))"

# Test API health
curl http://localhost:8059/health

# Test chat endpoint
curl -X POST http://localhost:8059/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello", "user_id": "test_user"}'

Roadmap

• Add user preference extraction
• Implement topic clustering
• Add conversation summaries
• Multi-modal support (images, files)
• Export/import user knowledge graphs
• Analytics dashboard
• Multi-language support

License

MIT License - Feel free to use and modify!

---

Built with ❤️ using Pydantic AI, PostgreSQL, Neo4j, and Graphiti.