This document outlines the core architecture design decisions for the RAG Agent system.
A basic RAG system usually uses embeddings and vector similarity search. This works well for semantic similarity, but enterprise documents often contain exact identifiers, numbers, company names, contract terms, ticket IDs, stock symbols, API names, and table fields. Vector search can miss these exact-match signals.
Use vector search as one retrieval channel, but not the only one.
Vector search is good at finding semantically similar chunks. However, it is less reliable for exact strings such as:
- contract IDs
- company names
- token symbols
- order numbers
- API endpoints
- exact dollar amounts
- spreadsheet field names
Adding non-vector retrieval increases system complexity, but improves recall and robustness.
- Vector-only search
- BM25-only search
- SQL full-text search
- Elasticsearch / OpenSearch
- Qdrant hybrid search
- pgvector + Postgres full-text search
Vector-only can be enough for a small, clean, semantic-only corpus where users ask broad natural language questions.
I do not treat embedding search as a silver bullet. It is strong for semantic recall, but enterprise RAG also needs exact-match retrieval for IDs, numbers, names, and domain-specific terms. 中文讲法：纯向量检索不是银弹。它适合语义相似，但对数字、ID、公司名、合约地址、字段名这些精确匹配不够稳。所以我不会只依赖 vector search。
Different queries require different retrieval signals. Some are semantic, others are lexical. For example:
- "What is the project price?" is semantic.
- "HIP-3 500K HYPE collateral" is lexical and exact-match heavy.
Use BM25 and vector search together. Retrieve candidates from both channels, merge, deduplicate, and rerank.
BM25 improves exact keyword recall.
Vector search improves semantic recall.
Together they produce a more complete candidate set.
Hybrid search adds complexity:
- two retrieval channels
- score normalization or rank fusion
- deduplication
- more candidates to rerank It can also increase latency.
- Vector-only
- BM25-only
- Reciprocal Rank Fusion
- Elasticsearch hybrid
- Qdrant hybrid
- LlamaIndex / LangChain retrievers
If the corpus is tiny or queries are purely semantic, hybrid search may be unnecessary.
I use hybrid search because retrieval recall determines the upper bound of the whole RAG system. If the correct evidence does not enter the candidate set, rerank and generation cannot fix it. 关键句：Retriever 决定上限，Reranker 优化排序，LLM 决定生成质量。
Hybrid retrieval returns candidates that are potentially relevant, but not always directly useful for answering the question. Some chunks may share keywords but not answer the question. Others may be semantically related but too vague.
Use a reranker to score query-chunk pairs and select the final top-k evidence.
A reranker can directly compare the query and the candidate chunk. It is better at identifying whether a chunk actually helps answer the question.
Rerank improves precision but adds latency and compute cost. Candidate size must be controlled.
- No rerank
- LLM-based rerank
- Embedding-only similarity
- RRF only
- Vendor rerank API
- Local cross-encoder reranker
If latency is extremely sensitive and retrieval precision is already high, rerank may be skipped.
Retrieval should be wide enough to avoid missing evidence; rerank narrows it down. I usually recall top-20 or top-50 candidates, then rerank to top-5 evidence.
Retrievers always return top-k chunks, even when the corpus does not contain the answer. If the LLM is forced to answer, it may hallucinate.
Before generating the final answer, check whether retrieved evidence is sufficient to answer the question.
Relevant evidence is not the same as sufficient evidence. A chunk may mention "pricing" but not contain the actual price.
Adding answerability check may require an extra LLM call and can introduce false refusals.
- Rerank score threshold
- LLM judge
- Extract-then-answer
- Rule-based refusal
- Confidence calibration
For low-risk brainstorming or creative generation, answerability check is less important.
In enterprise RAG, saying "I don't know based on the current evidence" is often much safer than producing a confident but unsupported answer. 中文讲法：相关不等于可回答。检索系统一定会返回 top-k，但这些 evidence 未必足够支持答案。所以我加 answerability，证据不足就拒答。
Free-form LLM output is hard to parse and unreliable for downstream systems. The frontend, API, eval harness, and workflow engine need stable fields such as answer, citations, answerable, confidence, and missing_information.
Require the model to output JSON and validate it with Pydantic.
Structured output turns the LLM from a free-form text generator into a typed system component.
Structured output can make prompting stricter and may require retry logic when validation fails.
- Free-form text
- JSON mode only
- Function calling
- PydanticAI
- Guardrails
- Instructor
For casual chat or creative writing, strict schemas may be unnecessary.
I use structured output because Agent systems need machine-consumable outputs, not just human-readable text. Validation also gives me a clean failure point.
User questions are often vague, short, conversational, or contain references such as "this", "that", "it", or "the previous one". Raw queries are often poor retrieval queries.
Add a query rewrite node before retrieval.
Query rewrite improves retrieval recall by turning natural user questions into standalone, retrieval-optimized queries.
It adds one LLM call and can accidentally change user intent if poorly prompted.
- Use raw question directly
- Rule-based expansion
- HyDE
- Query expansion
- Multi-query generation
For clear keyword queries or latency-critical paths, raw search may be enough.
Many RAG failures are not embedding failures but query understanding failures. Query rewrite lets me fix the input before blaming retrieval.
Complex user questions often contain multiple information needs. A single retrieval pass may over-focus on one aspect and miss others. Example: "What are the pricing, risks, and delivery timeline?"
Decompose complex questions into sub-questions, retrieve evidence for each, then synthesize the final answer.
This improves recall, reduces missing sections, and makes evaluation more granular.
More LLM calls, more retrieval calls, more complexity.
- Single broad retrieval
- LLM planner
- Sub-question query engine
- Agentic retrieval
Simple fact lookup should not be over-decomposed.
Query decomposition is the bridge from RAG to Agent workflow. It turns a complex question into a plan-execute-synthesize process.
A simple chain is not enough for production Agent behavior. Real agents need branching, retries, memory, checkpoint, streaming, and human-in-the-loop.
Use LangGraph to represent the Agent as a state machine.
LangGraph makes state, nodes, edges, conditional routing, checkpoint, streaming, and interrupt explicit.
It has more upfront complexity than a simple chain.
- Plain Python orchestration
- LangChain AgentExecutor
- LlamaIndex workflows
- CrewAI
- AutoGen / Microsoft Agent Framework
- Temporal / Celery workflow
For a simple one-shot RAG endpoint, plain Python or LCEL may be enough.
I use LangGraph not because it is trendy, but because Agent engineering is state management. LangGraph gives me explicit state transitions, conditional edges, checkpoint, streaming, and human-in-the-loop. 强表达：我把 Agent 理解成状态机：state_n + node/tool execution → state_{n+1}。这和区块链 state transition 很像。
In a simple chain, intermediate values are hidden in function calls. This makes debugging, retrying, checkpointing, and branching difficult.
Use an explicit AgentState that stores question, rewritten_query, evidence, answer, tool_calls, steps, errors, retry_count, and status.
Explicit state makes the system observable, recoverable, and debuggable.
State design requires discipline. If too much is stored, state becomes bloated.
- Local variables in a function
- Conversation history only
- External DB only
- Full event sourcing
For very simple synchronous workflows, explicit graph state may be overkill.
State is the Agent's runtime working memory. I keep raw input, transformed input, retrieval candidates, final evidence, answer, errors, and tool calls separate so I can locate failures precisely.
Agent runs can be long-running, interrupted, or dependent on user approval. Without checkpointing, failures or pauses lose state.
Use checkpointing keyed by thread_id.
Checkpoint enables:
- resume after interrupt
- human-in-the-loop
- conversation memory
- time-travel debugging
- fault recovery
Checkpoint storage adds persistence complexity and state size must be controlled.
- No checkpoint
- Store only final result
- Store full event log
- External workflow engine
For stateless, short-lived requests that do not need recovery.
Checkpoint is what turns an Agent run from a single request into a recoverable workflow. It is essential for HITL and long-running agents.
Large language models are stateless. To support multi-turn conversation, they need access to previous turns. Storing everything in the prompt can exceed context limits and degrade prompt performance.
Store recent chat history in the conversation state and checkpointers, and pass it dynamically to the LLM context.
It keeps conversation context across turns, allowing the agent to resolve references like "it" or "the previous document" and maintain thread continuity.
Increases input token count and cost; requires truncation or summarization strategies.
- Window-based memory
- Summary memory
- No conversational history
Single-turn search tasks where context is not carried over.
Short-term conversational memory is managed via thread states and LangGraph checkpointers, making it persistent across sessions but bounded to control context usage.
Users have persistent preferences or facts that should span across different conversations (threads). Checking database tables every time is inefficient.
Store long-term memories in a structured/semantic memory store (e.g. memories table) and retrieve relevant memories based on user ID and semantic similarity.
Allows personalization and persistence of facts (e.g. user preferences, corporate rules) across threads.
Adds DB lookups, vector matching, and background memory extraction/consolidation tasks.
- Prompt-based defaults
- No personalization
Strictly stateless, anonymized tools where personalization is not needed.
Long-term memory is updated asynchronously or on-demand, storing key insights in a dedicated memories table to guide agent behavior globally.
Agents may propose actions with side effects, such as saving memory, sending email, modifying data, deleting files, or executing transactions. Letting the LLM execute these directly is unsafe.
Use interrupt / resume for approval workflows.
The LLM can propose an action, but the system and human must authorize execution.
Human approval adds friction and slows execution.
- Fully automatic execution
- Rule-based approval
- Role-based policy only
- Manual-only workflow
Low-risk read-only tools usually do not need approval.
For irreversible or state-changing actions, LLM should never have final authority. It can draft or propose, but execution requires policy checks and often human approval. 区块链类比：HITL = 多签 / 签名确认。
Agent runs can take several seconds or longer. If the UI only shows a spinner, users do not know what the system is doing.
Use SSE to stream workflow events from LangGraph to the frontend.
Streaming makes the Agent transparent:
- query rewrite
- retrieval progress
- evidence found
- rerank
- answer generation
- interrupt waiting
SSE adds connection management complexity.
- Polling
- WebSocket
- Long polling
- Final-only response
Very short tasks may not need streaming.
Streaming is not just UX decoration. It exposes the Agent's execution trace to the user and builds trust.
A wrong answer can come from many layers: query rewrite, retrieval, rerank, answerability, generation, citation, or tool selection. Final-answer-only eval cannot tell where the failure happened.
Use layered evals. Evaluate:
- standalone question
- retrieval hit
- rerank top-k
- answerability
- citation validity
- structured output
- final answer
- tool trajectory
Layered eval makes failures actionable.
More eval cases and more implementation work.
- Manual review
- LLM judge only
- Final answer string match
- User feedback only
For prototypes, final-answer eval may be enough temporarily.
I treat Agent eval like transaction trace debugging. I need to know which layer failed, not just whether the final output looked right.
An Agent can produce a correct-looking answer while using the wrong tool path. For example, table aggregation should use a structured table query, not free-form RAG and LLM arithmetic.
Record and evaluate tool calls, arguments, ordering, forbidden tool usage, and approval compliance.
Tool trajectory eval ensures the Agent uses the right process, not just the right words.
Tool eval requires structured tool traces and more detailed eval cases.
- Final answer eval only
- Manual trace review
- LLM judge trajectory scoring
If the system has no tools or only one fixed tool.
For tool-using Agents, correctness includes tool selection and execution path. I evaluate expected_tools, forbidden_tools, required_order, and tool argument quality.
A conversation or workflow thread can contain multiple user requests. Each request should be tracked independently.
Use thread_id for conversation / workflow context and run_id for a single execution.
This enables:
- multi-turn memory
- run history
- event streaming per run
- replay and audit
- interrupt resume
- eval and debugging
More API and storage complexity.
- Single chat_id
- Request-only stateless API
- Session-only tracking
For a stateless one-shot API.
thread_id is the stateful conversation boundary; run_id is the execution instance. This separation is important for streaming, checkpointing, resume, and observability.
LLMs are not security boundaries. Prompt instructions can fail, and RAG evidence may contain prompt injection.
Apply authorization in retrieval, memory access, and tool execution layers.
The model should never see unauthorized content in the first place.
Requires ACL metadata, user context, permission filters, and tool policy.
- Prompt-only safety
- Post-generation filtering
- Manual review
Never rely only on prompt for sensitive authorization.
Permissions must be enforced before data reaches the LLM. The model is not the access control layer. 关键金句：The model is not the access control layer.
Frameworks hide complexity. This is good for speed, but bad for learning and debugging if the underlying workflow is not understood.
Implement V1 from first principles, then use frameworks in V2.
This builds deep understanding of:
- chunking, metadata, embeddings, retrieval, rerank, answerability, structured output, state, memory, eval, and observability.
V1 takes more effort than using a framework directly.
- Start with LangChain
- Start with LlamaIndex
- Start with Dify / Coze
- Start with CrewAI
If the goal is a quick customer PoC, using a framework or low-code platform first may be better.
I intentionally built V1 from the bottom up so I could understand and debug each layer. In V2, I can map each layer to LangChain, LlamaIndex, LangGraph, CrewAI, or Dify based on the use case.
V1 implements the system from first principles.
V2 will add framework adapters:
| Framework | Role |
|---|---|
| LangChain | Model, prompt, tool, retriever, output parser components |
| LangGraph | Stateful workflow runtime |
| LlamaIndex | RAG indexing, query engine, document workflows |
| CrewAI | Role-based multi-agent workflows |
| Microsoft Agent Framework | Enterprise multi-agent orchestration |
| Dify / Coze | Low-code customer-facing workflow PoC |
| LangSmith / RAGAS / DeepEval | Observability and eval |
The goal of V2 is not to replace the architecture, but to compare how different frameworks express the same underlying Agent engineering concepts (用主流框架表达同一套底层 Agent 工程逻辑).