Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.references/**
**~
*.DS_Store

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down
46 changes: 46 additions & 0 deletions .plans/implemented/implementation_step_3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Step 3: Scoped retrieval and context assembly

## Goal and dependency

Find relevant DocLang nodes within authorized URI and XPath scopes, then assemble bounded context with exact citations. Depends on Steps 1–2. The system must answer lexical queries with no embedding model installed.

## Index records and interfaces

- A retrieval unit is a source document or semantic node with `tenant_id`, `uri`, `document_id`, `revision_id`, `xpath`, `parent_xpath`, `tier`, page, bounding box, text for indexing, and an internal stable `uint64` vector ID. SQLite maps the vector ID to the full address and stores lifecycle state.
- Use SQLite tables for tree/path filters and FTS5 for lexical search. Keep a `VectorIndex` interface (`upsert`, `remove`, `search(query, allowed_ids, k)`, `checkpoint`, `rebuild`) so the retrieval planner is independent of index implementation.
- Store the embedding model ID, dimensions, normalization rule, input hash, and index generation with each record. Never mix vectors from different model/dimension generations; rebuild a new generation and switch it atomically.
- Index L0 collection/document summaries, L1 outlines/section descriptions, and selected L2 semantic units. Preserve DocLang structure: split at headings and safe semantic boundaries; tables, formulas, and captions remain addressable units. Keep source XML untouched.

## Local vector store choice

- Evaluate [turbovec `IdMapIndex`](https://github.com/ryancodrai/turbovec) as the initial local implementation because it exposes stable external IDs, `remove`, `search(..., allowlist=...)`, and `write`/`load`/`sync` persistence. Use `IdMapIndex`, not positional IDs that can move after deletion. Its Python API expects contiguous `float32` arrays; document and validate dimensional requirements.
- Resolve tenant, URI-prefix, tier, and XPath-prefix constraints in SQLite first. Supply the resulting vector IDs as turbovec's allowlist, then combine vector results with FTS results. Recheck every returned ID against current revision and principal before opening a package.
- Treat turbovec's index file as a rebuildable projection, not the canonical document store. Record an index generation and committed outbox offset in SQLite. On restart, load a compatible snapshot and replay later events; if missing or inconsistent, rebuild from canonical DCLX packages and metadata.
- Benchmark filtered recall, search latency, index size, restart time, update/delete behavior, and embedding dimensions on representative local corpora. If a platform or dimension is unsupported, retain the FTS path and switch the vector adapter rather than changing public APIs. Repository benchmark claims are hypotheses for this workload, not acceptance results.

## Retrieval algorithm

1. Authenticate and normalize tenant, URI subtree, optional document, and optional XPath scope.
2. Read permitted candidates from SQLite, then run FTS and optional allowlist vector search only within those IDs.
3. Merge and rerank by relevance, tier, freshness, and structural proximity; collapse overlapping parent/child hits.
4. Fetch cited nodes from the exact DCLX revision. Return L0/L1 first when they satisfy the requested budget; load L2 only for selected nodes. Respect per-result and total token limits.
5. Return score components, URI, revision, XPath, page/bbox when present, tier, and truncation status. No citation may point to a stale or missing revision.

## Work sequence

1. Build deterministic node extraction and SQLite FTS records.
2. Implement scoped lexical search and cited XPath reads.
3. Add embedding provider interface and turbovec adapter with checkpoint/rebuild logic.
4. Add hybrid ranking and context assembly.
5. Measure quality and latency against a flat-search baseline with a fixed corpus and query set.

## Acceptance checks

- Scoped results never include another tenant, sibling subtree, or excluded XPath branch, including when an index contains stale IDs.
- Delete and replace a document, restart, and verify no old vector result is returned. Corrupt or remove the vector snapshot and verify a correct rebuild.
- Return the same cited DocLang content before and after index rebuild; enforce requested budgets and deterministic truncation.
- Report measured recall and latency for lexical-only, vector-only, and hybrid modes; set performance targets from the baseline rather than promising a fixed subsecond result in advance.

## Exit artifact

A local retrieval engine with exact DocLang citations, optional embeddings, scoped hybrid search, and a replaceable vector backend.
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

A local context tree that stores documents as immutable DocLang `.dclx` packages.

See the [capability guide](docs/index.md), [CLI reference](docs/cli.md), and
See the [capability guide](docs/index.md), [retrieval guide](docs/retrieval.md),
[CLI reference](docs/cli.md), and
[single PDF and folder examples](examples/adding_resources/README.md).

Ingest native `.dclg`, `.dclg.xml`, and `.dclx` sources with `Ingestor.add_resource`.
Expand Down Expand Up @@ -40,11 +41,19 @@ uv run dc --store ./context-data add-resource ./paper.pdf --collection papers --
uv run dc --store ./context-data add-resource ./documents -r --from docx --from image
uv run dc --store ./context-data worker --once
uv run dc --store ./context-data tree docling://resources/papers -L 2
uv run dc --store ./context-data index status
uv run dc --store ./context-data search "invoice" --uri docling://resources/papers
uv run --extra vectors dc --store ./context-data index vector rebuild
uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid
```

Use `uv run dc` or activate the project environment before running `dc` by
itself; some systems also provide a calculator named `dc`. See the
[full CLI guide](docs/cli.md) and [resource examples](examples/adding_resources/README.md).
Ingestion builds the lexical index automatically. Vector indexing is optional;
the `vectors` extra provides local embeddings and `dc index vector rebuild`.
`dc index lex rebuild` reconstructs search records from stored DCLX packages.
Search results cite exact source nodes or `@summary:` and `@toc:` sidecars.

## Local usage

Expand Down
51 changes: 51 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,17 @@ uv run --extra conversion dc --store ./context-data add-resource ./paper.pdf \
uv run dc --store ./context-data task status JOB_ID
uv run dc --store ./context-data worker --once
uv run dc --store ./context-data status
uv run dc --store ./context-data index status
uv run dc --store ./context-data index lex status
uv run dc --store ./context-data index lex rebuild
uv run --extra vectors dc --store ./context-data index vector status
uv run --extra vectors dc --store ./context-data index vector rebuild
uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid
uv run dc --store ./context-data ls docling://resources/
uv run dc --store ./context-data tree docling://resources/papers -L 2
uv run dc --store ./context-data find "invoice" --uri docling://resources/papers
uv run dc --store ./context-data grep "total" --uri docling://resources/papers
uv run dc --store ./context-data search "invoice" --uri docling://resources/papers
```

`add-resource` accepts one document or a folder of documents. With the
Expand All @@ -41,8 +48,52 @@ job; `add-resource` prints the revision and its `task_id`. `task status` reports
that job. Run `worker --once` for one due job, or `worker` continuously. A
collection summary may remain stale until its latest job completes.

Every `status` command prints a table by default. Pass `--json` for structured
output, for example `dc status --json`, `dc task status JOB_ID --json`, or
`dc index vector status --json`.

`find` and `grep` are case-insensitive substring scans of current documents.
They scan at most 1,000 documents, 2,000 nodes per document, and 50,000 nodes
total, returning at most 100 hits. They accept
an optional URI subtree scope and enforce tenant and user access. They do not
yet provide embedding search or relevance ranking.

`search` uses the SQLite retrieval index and returns exact DCLX citations.
New or changed documents are indexed during ingestion. `index status` reports
both lexical and vector coverage. `index lex status` and `index vector status`
show each index separately. `index lex rebuild` regenerates the SQLite index
from stored DCLX packages without rerunning conversion or OCR. It invalidates
an existing vector snapshot; a configured embedding provider rebuilds it on its
next use. Status and lexical rebuild honor the selected tenant and user scope.

The CLI defaults to lexical search until a vector model has been built, then
uses hybrid search. `--mode lexical|vector|hybrid` selects a mode explicitly.
Install the `vectors` extra and run `dc index vector rebuild` to generate
embeddings with the local default `BAAI/bge-small-en-v1.5` model.
`--model MODEL` selects another supported FastEmbed text model;
`Qdrant/clip-ViT-B-32` selects paired
text and image encoders. The first build downloads model weights unless
`--offline` is passed. On Apple Silicon, `--backend auto` selects MLX for
supported BGE models and uses ONNX for other models. Use `--backend mlx` or
`--backend onnx` to choose explicitly. MLX uses separate model weights and
cannot be combined with `--offline`; `--backend auto --offline` uses ONNX.
The selected backend is saved with the vector generation, so subsequent
searches use the same encoder. Both lexical and vector rebuilds display a
progress bar on interactive terminals; progress goes to stderr.
After a vector build, document writes and deletes replay from the outbox on
the next vector search. Missing or corrupt snapshots trigger a full rebuild.
`dc index vector status` reports embedding coverage and the
stored model. `search --image PATH --mode vector` accepts an image query with
CLIP; text and image may be supplied together. A text-only model skips the
image input. Vector builds currently require all documents in the store to be
accessible to the selected principal because the snapshot is store-wide.
With an image-capable model, referenced picture assets are indexed alongside
text nodes. Search results include the picture XPath and asset path; image-only
results show an image marker in the table. After upgrading an existing store,
run `dc index lex rebuild` and `dc index vector rebuild` to add picture assets.

Use `--uri`, `--document`, and `--xpath` to restrict scope; repeat `--tier`
to select context tiers. `-k`, `--max-tokens`, and `--per-result-tokens`
control output size. It prints a compact table by default; pass `--json` for
full citation records. See [scoped retrieval](retrieval.md) for the Python
vector and hybrid API.
8 changes: 5 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@ them through local Python and `dc` command-line APIs.
| Conversion | Typed PDF and image settings for OCR, tables, charts, enrichment, images, and limits |
| Remote conversion | Opt-in Docling Serve adapter; explicit endpoint and bounded response |
| Inspection | `dc status`, `ls`, `tree`, and task status |
| Search | `dc find` and `grep` provide bounded lexical scans of current documents |
| Search | `dc search` provides scoped lexical, vector, and hybrid retrieval, including picture assets with an image-capable model |

Search currently scans stored documents and has no semantic ranking or vector
index. The indexed retrieval work is planned separately.
`find` and `grep` retain their bounded substring scans. Indexed retrieval
verifies citations against the current DCLX revision before returning text.

- [Conversion settings](conversion.md)
- [Scoped retrieval](retrieval.md)
- [Retrieval benchmark](../examples/retrieval/benchmark_corpus.py)
- [CLI commands](cli.md)
- [Storage and jobs](storage.md)
- [Single PDF and folder examples](../examples/adding_resources/README.md)
Loading
Loading