Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
- name: Install native build dependencies
run: sudo apt-get update && sudo apt-get install -y libxml2-dev ninja-build
- name: Install uv and Python
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
uses: astral-sh/setup-uv@v7
with:
python-version: ${{ matrix.python-version }}
enable-cache: true
Expand Down Expand Up @@ -52,7 +52,7 @@ jobs:
steps:
- uses: actions/checkout@v7
- name: Install uv and Python
uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0
uses: astral-sh/setup-uv@v7
with:
python-version: "3.12"
enable-cache: true
Expand Down
46 changes: 46 additions & 0 deletions .plans/implemented/implementation_step_2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Step 2: Ingestion and hierarchical summaries

## Goal and dependency

Turn source files and existing DocLang artifacts into revisioned DCLX packages, with L0 and L1 context in DocLang sidecars and the full L2 document intact. Depends on the package and outbox contracts from Step 1.

## Input and package contract

- Support `.dclg`/`.dclg.xml` and `.dclx` first. Add local Docling conversion for selected formats behind a `Converter` interface; add a `docling-serve` adapter only after local conversion works. Record converter version, options, source hash, and origin URI in provenance.
- The canonical result is one DCLX package per logical document revision. If input is DCLG, wrap its validated XML in DCLX. If input is DCLX, preserve its recognized sidecars and opaque assets. Never flatten a source document to Markdown.
- L0 is a short DocLang summary using `set_document_summary`; L1 uses the native DocLang TOC sidecar via `set_toc`. The full `document.xml` is L2. Concepts may use `set_concepts`. For future sidecar types, use a context-specific package manifest and DocLang or typed structured parts, with explicit schema version and source revision.
- Collection-level summaries are separate DocLang documents stored as DCLX packages under reserved collection URIs. Their provenance lists child revision IDs. Rebuild them when children change. This avoids pretending a directory is a sidecar of one child package.

## Pipeline

1. Fingerprint source bytes and conversion settings. Repeated ingestion with an identical fingerprint is idempotent unless `force` is requested.
2. Convert or load, validate DocLang, preserve source geometry/assets, and capture conversion warnings.
3. Walk heading and semantic nodes to build an outline. Use deterministic extraction first; optional local LLM/VLM providers enrich summaries. Model output must be validated as DocLang and tied to the exact revision.
4. Create L0/L1 sidecars. On model failure, generate bounded extractive summaries and structural outlines, mark their generation method, and continue ingestion.
5. Package, validate, commit the new revision, and emit `DocumentCommitted`. Schedule collection-summary and indexing jobs through the local outbox.

## Job and update semantics

- Jobs have durable IDs, input revision, status (`queued`, `running`, `complete`, `failed`), attempt count, and retry time in SQLite. Workers claim leases so a crash can be retried safely.
- Derived sidecars are immutable per revision. Enrichment creates a new package revision and emits an update; consumers never combine a sidecar from one revision with XML from another.
- If a source document is replaced or deleted, mark dependent collection summaries stale, then rebuild them. Expose stale status to retrieval until the new derived revision is committed.
- Limit converter resources and archive sizes. Keep source path/URL access behind explicit allow rules; no automatic network fetching is required for fully local operation.

## Work sequence

1. Implement the DCLG/DCLX import path and package round trips.
2. Implement heading-tree extraction, TOC generation, and deterministic L0 fallback.
3. Add local Docling conversion and optional model enrichment.
4. Add durable jobs, retries, and collection-summary invalidation.
5. Add remote conversion adapter with the same output contract.

## Acceptance checks

- Ingest one native DCLX, one DCLG XML file, and one converted PDF. Reopen every result with `DoclangXDocument`, validate L0/L1 as DocLang, and resolve sample XPaths and geometry.
- Repeat ingestion and verify no duplicate revision or jobs; change source bytes and verify a new revision.
- Kill a worker after conversion and resume without duplicate visible state.
- Fail the model provider and verify local ingestion and structural retrieval still work.

## Exit artifact

An `add-resource` library operation and background worker that yield portable DCLX packages with traceable, revision-consistent context tiers.
50 changes: 49 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,20 @@
# docling-context

A local context tree that stores documents as immutable DocLang `.dclx` packages.

See the [capability guide](docs/index.md), [CLI reference](docs/cli.md), and
[single PDF and folder examples](examples/adding_resources/README.md).

Ingest native `.dclg`, `.dclg.xml`, and `.dclx` sources with `Ingestor.add_resource`.
Pass bytes with a filename, or a local `Path` under an explicit `allowed_roots` entry.
Each revision stores its full document, a short DocLang summary, and a DocLang TOC
in one DCLX package. Collection summaries are rebuilt by `CollectionWorker`;
`CollectionWorker.status` reports whether a summary is stale.

Document conversion is local and optional: install the `conversion` extra to
enable `LocalDoclingConverter` for Docling's supported formats, including PDF,
Office documents, and images. `DoclingServeConverter` requires an explicit HTTPS
endpoint or loopback HTTP endpoint.
SQLite tracks logical URIs, revisions, tenant ownership, and an append-only change
outbox. Package bytes live under SHA-256 paths. The API is designed so another
storage backend can implement the same contracts later.
Expand All @@ -15,9 +29,23 @@ published `doclang` package, which may not yet have the required native API.

```bash
uv sync
uv run pytest -q tests/test_step1.py
uv run pytest -q tests
```

## CLI quick start

```bash
uv sync --extra conversion
uv run dc --store ./context-data add-resource ./paper.pdf --collection papers --no-ocr
uv run dc --store ./context-data add-resource ./documents -r --from docx --from image
uv run dc --store ./context-data worker --once
uv run dc --store ./context-data tree docling://resources/papers -L 2
```

Use `uv run dc` or activate the project environment before running `dc` by
itself; some systems also provide a calculator named `dc`. See the
[full CLI guide](docs/cli.md) and [resource examples](examples/adding_resources/README.md).

## Local usage

```python
Expand All @@ -35,6 +63,26 @@ with LocalContextStore("./context-data") as store:
print(store.read_node(principal, citation).text)
```

For ingestion, allow the source directory explicitly and then process queued
collection summaries. A worker can call `run_forever(stop_event)` in its own
process with its own store connection.

```python
from pathlib import Path
from docling_context import CollectionWorker, Ingestor, LocalContextStore, Principal

principal = Principal("acme", "alice")
source_dir = Path("./documents").resolve()
with LocalContextStore("./context-data") as store:
record = Ingestor(store, allowed_roots=(source_dir,)).add_resource(
principal, "docling://resources/manuals/guide", source_dir / "guide.dclx"
)
worker = CollectionWorker(store)
while worker.run_once() is not None:
pass
print(worker.status(principal, "docling://resources/manuals"))
```

Document URIs use `docling://resources/{collection}/{path}` or
`docling://users/{tenant}/{user}/{memories|sessions|skills}/{path}`. Namespace
names are lowercase; path segments keep their case and normalize to Unicode NFC.
Expand Down
48 changes: 48 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# `dc` CLI

Install the package to expose `dc`. For PDF input, install the `conversion`
extra. The command uses a local store at `~/.local/share/docling-context` by
default; pass `--store PATH` to choose another directory. `--tenant` and
`--user` select the principal; their defaults are `default` and `local`.
Use `uv run dc` or activate the project environment; a system calculator also
uses the name `dc` on some machines.

```bash
uv run --extra conversion dc --store ./context-data add-resource ./paper.pdf \
--collection papers --no-ocr --tables --no-charts
uv run dc --store ./context-data task status JOB_ID
uv run dc --store ./context-data worker --once
uv run dc --store ./context-data status
uv run dc --store ./context-data ls docling://resources/
uv run dc --store ./context-data tree docling://resources/papers -L 2
uv run dc --store ./context-data find "invoice" --uri docling://resources/papers
uv run dc --store ./context-data grep "total" --uri docling://resources/papers
```

`add-resource` accepts one document or a folder of documents. With the
`conversion` extra installed, it accepts Docling's supported formats, including
PDF, DOCX, PPTX, XLSX, images, HTML, and plain text. Native DCLG and DCLX files
also work without that extra. Folder imports scan the top level by default.
Add `-r` or `--recursive` to include subfolders; those paths are retained in
the URI. Use repeatable `--from FORMAT` options to select Docling format names,
such as `--from pdf --from docx --from image` (default: all). Filtering uses
filename extensions; Docling then detects the actual input format. Files with
the same stem and different extensions receive distinct URIs. A single file
can use `--uri`; otherwise `--collection` defaults to `documents`. Use `--config`
for a [JSON conversion config](conversion.md), with command flags overriding it.
Use `--force` to create a new revision for unchanged input.
Table reconstruction always uses accurate mode; there is no CLI mode switch.
Page and picture images are on by default, and there is no default page limit.
`--no-page-images`, `--no-picture-images`, and `--max-pages N` override these
defaults. OCR uses RapidOCR with English PP-OCRv6 `tiny` models when enabled.

Ingestion is synchronous. Each new resource revision queues a collection-summary
job; `add-resource` prints the revision and its `task_id`. `task status` reports
that job. Run `worker --once` for one due job, or `worker` continuously. A
collection summary may remain stale until its latest job completes.

`find` and `grep` are case-insensitive substring scans of current documents.
They scan at most 1,000 documents, 2,000 nodes per document, and 50,000 nodes
total, returning at most 100 hits. They accept
an optional URI subtree scope and enforce tenant and user access. They do not
yet provide embedding search or relevance ranking.
51 changes: 51 additions & 0 deletions docs/conversion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Document conversion settings

Install local document conversion with `uv sync --extra conversion`. Pass a
`PdfConversionConfig` to `LocalDoclingConverter`, or supply a JSON file to
`dc add-resource --config`. Docling runs locally by default; this adapter keeps
external services and external plugins disabled. The conversion extra includes
RapidOCR and its ONNX Runtime engine for the default OCR configuration. The
first OCR run may download model weights; later runs use the local copy.

```python
from docling_context import LocalDoclingConverter, PdfConversionConfig

config = PdfConversionConfig(
do_ocr=False,
do_table_structure=True,
do_chart_extraction=False,
table_mode="accurate",
max_pages=None,
)
converter = LocalDoclingConverter(config)
```

| Setting | Default | Effect |
| --- | --- | --- |
| `do_ocr` | `true` | Read scanned text with RapidOCR, English PP-OCRv6 `tiny` |
| `do_table_structure` | `true` | Reconstruct tables and cells |
| `do_chart_extraction` | `false` | Extract structured chart data |
| `do_code_enrichment` | `false` | Enrich code blocks |
| `do_formula_enrichment` | `false` | Enrich formulas |
| `force_backend_text` | `false` | Prefer embedded PDF text |
| `generate_page_images` | `true` | Include rendered page images when available |
| `generate_picture_images` | `true` | Include extracted picture images when available |
| `table_mode` | `accurate` | Fixed to `accurate`; `fast` is rejected |
| `max_pages` | `null` | Unlimited pages by default; set a positive number to limit conversion |
| `max_file_bytes` | `50000000` | Maximum PDF source size |
| `document_timeout` | `null` | Optional Docling processing timeout in seconds |
| `artifacts_path` | `null` | Optional local model-artifact directory |

The PDF and image pipelines use the OCR and image settings; other formats use
Docling's format-specific defaults. The entire config is recorded in each
document's provenance and contributes to its ingestion fingerprint. Changing a
setting creates a new revision on the next ingestion. `dc` flags such as
`--no-ocr`, `--no-tables`, and `--charts` override the corresponding JSON fields.
See the [example JSON](../examples/adding_resources/conversion_config.json).

The opt-in `DoclingServeConverter(endpoint, config=config)` forwards supported
OCR, table, chart, code, formula, and text options to the service. Local limits
such as `max_pages`, model artifact paths, and image generation settings are
not enforced by the service adapter; configure those on the service separately.
Only explicit HTTPS or loopback HTTP endpoints are accepted, and redirects are
rejected. Source URLs are never fetched during ingestion.
24 changes: 24 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Capabilities

`docling-context` stores document revisions as DocLang DCLX packages and exposes
them through local Python and `dc` command-line APIs.

| Capability | Current behavior |
| --- | --- |
| Storage | SQLite metadata and outbox; immutable, SHA-256-addressed DCLX packages |
| Namespaces | Tenant-scoped resources and user-scoped memories, sessions, and skills |
| Ingestion | Native DCLG and DCLX, plus Docling formats through optional local conversion |
| Context tiers | L0 document summary, L1 heading TOC, L2 source document in one package |
| Collection summaries | Durable jobs with retries, leases, and visible stale status |
| Conversion | Typed PDF and image settings for OCR, tables, charts, enrichment, images, and limits |
| Remote conversion | Opt-in Docling Serve adapter; explicit endpoint and bounded response |
| Inspection | `dc status`, `ls`, `tree`, and task status |
| Search | `dc find` and `grep` provide bounded lexical scans of current documents |

Search currently scans stored documents and has no semantic ranking or vector
index. The indexed retrieval work is planned separately.

- [Conversion settings](conversion.md)
- [CLI commands](cli.md)
- [Storage and jobs](storage.md)
- [Single PDF and folder examples](../examples/adding_resources/README.md)
28 changes: 28 additions & 0 deletions docs/storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Storage, revisions, and jobs

`LocalContextStore` stores metadata in SQLite and DCLX blobs under SHA-256
paths. A logical URI points to the latest revision; old revisions remain
addressable by document ID, revision ID, and XPath. Every write also inserts
an outbox event in the same SQLite transaction. Blob reads verify their hash.

Resource URIs have the form `docling://resources/{collection}/{path}`. User
URIs have the form
`docling://users/{tenant}/{user}/{memories|sessions|skills}/{path}`. Tenant
ownership is checked on reads and writes. Local file ingestion requires an
explicit `allowed_roots` entry; bytes can be ingested without filesystem access.

Each ingested document is a DCLX package containing `document.xml` (L2), a
DocLang summary (L0), and a DocLang TOC (L1). Source assets and sidecars other
than the generated summary and TOC are kept from DCLX input. `context/manifest.json` records the
summary method and exact source revision. Source hashes, origin, converter
version/options, and warnings are in the revision's provenance.

Collection summaries are separate DCLX documents at
`docling://resources/{collection}/_context/summary`. Writes and deletes mark
the collection stale and enqueue a durable job. `CollectionWorker` claims jobs
with leases, retries failures, and can resume after a crash without adding a
duplicate summary revision. Its `status` method exposes staleness and the
summary revision ID.

The local and in-memory stores implement the same basic `ContextStore`
contract, leaving room for another storage backend later.
8 changes: 8 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ classifiers = [
]
dependencies = ["doclang"]

[project.optional-dependencies]
conversion = ["docling[rapidocr]>=2.134.0,<3"]

[project.scripts]
dc = "docling_context.cli:main"

[project.urls]
Repository = "https://github.com/docling-project/docling-context"

Expand All @@ -42,6 +48,8 @@ default-groups = ["dev"]
include = [
"/src/docling_context",
"/tests",
"/examples",
"/docs",
"/README.md",
"/LICENSE",
"/pyproject.toml",
Expand Down
19 changes: 19 additions & 0 deletions src/docling_context/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
"""Local-first context storage backed by DocLang packages."""

from .contracts import ContextStore
from .conversion_config import PdfConversionConfig
from .converters import (
ConversionResult,
Converter,
DoclingServeConverter,
LocalDoclingConverter,
)
from .ingestion import Ingestor, SummaryProvider
from .jobs import CollectionStatus, CollectionWorker, Job
from .local import LocalContextStore, RecordMissing, RevisionConflict
from .memory import MemoryContextStore
from .models import (
Expand All @@ -19,21 +28,31 @@
__all__ = [
"AccessDenied",
"ChangeEvent",
"CollectionStatus",
"CollectionWorker",
"ContextStore",
"ContextURI",
"ConversionResult",
"Converter",
"DoclingServeConverter",
"DocumentRecord",
"FilePackageStore",
"Ingestor",
"InvalidURI",
"Job",
"LocalContextStore",
"LocalDoclingConverter",
"MemoryContextStore",
"NodeAddress",
"NodeContent",
"PackageError",
"PackageMissing",
"PackageRef",
"PdfConversionConfig",
"Principal",
"RecordMissing",
"RevisionConflict",
"SummaryProvider",
"TenantScope",
"TreeEntry",
"authorize_uri",
Expand Down
Loading
Loading