[Feature]: ask about project

[TentenMarchhhh]

Problem statement

Question 1: Knowhere positions itself as a structured document memory layer that reconstructs full document hierarchies using a proprietary tree-like algorithm instead of flattening them into linear sequences. From a structural engineering standpoint, how is this navigation tree represented in the data schema, and how do retrieval agents computationally "walk" this tree alongside standard vector-based semantic search to drill down into deep document regions?

Question 2: The repository emphasizes the construction of a lightweight "Memory Graph" that links chunks, navigation trees, summaries, and graph links into an agent-ready context. How does Knowhere dynamically determine and create edges/relationships between disparate document chunks or cross-document nodes during the graph construction phase without triggering excessive, expensive LLM calls?

Question 3: Knowhere supports high-fidelity extraction from complex multi-modal assets (PDFs, Office files, images) and uses Vision-Language Models (VLMs) to generate summaries and extract features from images/tables. How does the indexing pipeline map and sync these visual asset coordinates or VLM-generated features back into the parent text node within the unified tree structure, ensuring that retrieval agents can cite specific tables or images at inference time?

Question 4: According to the release notes, the platform now handles ultra-long PDFs (300+ to 500+ pages) and routes technical atlases or drawing collections through a dedicated layout-aware parser. How does the ingestion engine optimize memory foot-prints and scale horizontally when parsing long-form PDFs? Does it chunk the file processing at the binary/page level before layout analysis, and how does it stitch the broken hierarchical trees back together seamlessly?

Question 5: Knowhere includes an Agentic Retrieval engine that combines a first-pass traditional/semantic search via Reciprocal Rank Fusion (RRF) with autonomous agent navigation. What specific policy or decision-making loop does the retrieval agent follow to decide whether to stop searching based on raw text snippets and instead initiate a tree-navigation or graph-walking routine?

Question 6: The workspace utilizes uv for dependency synchronization and supports custom model names, provider URLs, and concurrency limits via configuration overrides. How decoupled is the underlying execution layer from proprietary LLM APIs? If we deploy Knowhere in a completely air-gapped environment using local open-source models (e.g., Llama 3 or Mistral) for parsing, summary generation, and graph indexing, what interfaces or abstract base classes must be implemented to ensure feature parity?

Affected area

Other

Proposed solution

m

Alternatives considered

No response

Compatibility and migration notes

No response

Additional context

No response

[EricNGOntos]

HI, thanks for asking, I am trying to asnwer these:

1. the data is mainly represented in json, and as it is a tree (if there are more than one docs, it will be a linked tree, i.e., a network with connected nodes and a node often refers to a bottom-level section). The agent starts from the root, and for each step, it can see the overall section hierarchy at its current node (for instance, if it is at the root, it will see all level 1/2 titles), then, it can determine to expand a node, collect a node, jump to a deepper node, or back to a coarse node. These process is called navigation, and once the agent believes it collects all needed information, it will Finish the process and assemble all information based on their original hierarchy.

2. well, to be honest, the function is still testing and not published, but the cross-doc link can be determined by LLM which judeges if two bottom-level sections share pre-defined relationships (e.g., if they are talking about the same topic, if they contradict to each other, if they share same entities which can be also pre-defined). We planned to set a few default relations (e.g., similar, contradict, causal) and entities (e.g., person, organization, location).
   In this way, the process does not heavy bottom-up and recursive summary and entity extraction.

3. all assets, e.g, tables and images are connected to the bottom level sections where they exist in the raw document, as we can detect the hierarchy, we can correctly place the assets. We use placeholders once we detect an image or table in raw content, and use the key "connectto" in the json data schema to establish explicit linkage.

4. this is a very good question that we are still exploring, currently, we segment long PDFs first based on H1/H2 headings in the table of contents (the agent will try to extract TOC first). If there is no TOC, we segment PDF based on blank or low-content pages. And we stitch the pages according to the hierarchy as well (in fact, as long as we can detect the TOC accurately, the stiching process is simple and straightforward, and in most cases, TOC can be accurately detected as it usually appears in consecutive pages and there are often no other confusing content)

5. the two methods are used in parallel. we think that the navigation needs to explore from top to bottom, which is not efficient in some cases and can ignore nodes that hidden in deep levels. The traditional searching especially detect these deep nodes to supplement the navigation process, once the agent believes that the current navigation status and (or) the bottom discovered nodes can satisfy the query, it will stop.

6. Knowhere's LLM layer is already fully decoupled. All model calls go through a single OpenAICompatibleClientSync that uses the standard OpenAI /v1/chat/completions protocol. For air-gapped deployment, no code changes or custom interfaces are needed. Just set env vars:

DS_URL=http://localhost:8000/v1 # your local vLLM / Ollama endpoint
DS_KEY=local # any non-empty string
NORMOL_MODEL=your-model
IMAGE_MODEL=your-local-vlm # must support vision (e.g. InternVL, LLaVA)

Key requirements for feature parity:
Your local server must be OpenAI-compatible (vLLM, Ollama, llama.cpp all work)
A vision-language model is needed — PDF classification, image OCR, and image description all use VLM calls
JSON mode support (response_format: {"type": "json_object"}) is expected by several prompts
MinerU (PDF parsing) has a self-hosted mode; PPTX conversion already has a built-in LibreOffice fallback
No ABCs or interfaces to implement, namely, it's entirely config-driven.