Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .plans/implemented/implementation_step_5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Step 5: Agent harness installation, CLI, SDK, and MCP

## Goal and dependency

Make the context service easy to install and use from agent harnesses, locally first and with the same tools against a cloud service. Depends on stable storage, ingestion, retrieval, and memory operations from Steps 1–4.

## Installable product

- Publish a Python package with a stable CLI and `docling-context mcp` stdio entry point. Provide a remote HTTP MCP endpoint for deployments.
- Provide `docling-context integrate {codex,claude,hermes,pi}` with `--scope user|project` and `--mode local|remote`, plus `doctor` and `remove` operations. These are planned project commands, to be implemented in this step.
- Generate or update only this project's config entry. Preserve unrelated configuration, detect name conflicts, and make repeated installation idempotent. For project scope, create a reviewable project configuration file where the harness supports one; do not place credentials in it.
- Installation verifies the executable, MCP handshake, tool discovery, backend access, and one cited retrieval. `doctor` reports actionable failures; removal deletes only entries created by this installer. Test upgrade and rollback of generated configuration.
- Ship short harness-specific usage guidance so agents know when to call `search`, `outline`, and `show`, and how to retain URI/XPath citations. Keep memory capture opt-in and expose write tools explicitly.

## Public service contract

- Define a Python service layer with typed request/response objects for `list`, `tree`, `outline`, `show`, `search`, `ingest`, `job_status`, `session_append`, `memory_list`, `memory_review`, and `memory_recall`.
- `show` accepts a document URI, optional immutable revision, and XPath, then returns bounded DocLang XML or a structured representation with exact citation metadata. `outline` returns the TOC sidecar and node links. `search` returns matches and optional assembled context without silently changing tiers.
- Standardize errors: invalid URI/XPath, unauthorized scope, missing revision, stale citation, package failure, conversion failure, unsupported provider, and budget exceeded. Include stable machine-readable codes.
- Version the public request/response schema before adding network transport. Keep backend choice in configuration; callers use the same operations locally and with a deployed service.

## Surfaces

1. **Python SDK:** synchronous core operations, with asynchronous wrappers only where jobs or remote calls benefit. Return typed records rather than raw database rows. Keep package readers available for callers needing DocLang node operations.
2. **CLI:** commands such as `add`, `ls`, `tree`, `outline`, `show`, `find`, `jobs`, `sessions`, and `memories`. Provide human output and structured JSON output; bound tree depth and result count. `show` can emit DocLang XML, with no Markdown storage/export implied.
3. **MCP server:** expose read/search tools first. Define tool input schemas with URI, scope, XPath, tier, and limits. Add writes only with explicit authenticated principal and operation-specific authorization. Return citations and truncation markers in every content-bearing response.

## Harness targets

| Harness | Initial installation route |
| --- | --- |
| Codex | Register the MCP server through its CLI or configuration; evaluate a packaged plugin after direct installation works. |
| Claude Code | Register through `claude mcp` at user or project scope. |
| Hermes Agent | Manage a named `mcp_servers` entry in its configuration. |
| Pi | Register through built-in `pi mcp`; create a Pi package only if native Pi behavior adds value. |

Each adapter supports a local stdio server. Where the harness supports it, also support the remote HTTP endpoint with appropriate authentication. Harness adapters remain thin: every tool calls the same service contract. Maintain a compatibility matrix with tested harness versions, install scope, transport, verification, and removal behavior.

## Operational behavior

- Do not accept a caller-provided tenant or user ID as authorization by itself. Derive the principal from local configuration or authenticated transport, then check scope in the service layer.
- Enforce timeouts and budgets around conversion, model calls, retrieval, and XML output. Expose job IDs for slow ingestion rather than holding an interactive request indefinitely.
- Log operation ID, URI scope, latency, index generation, and result count; omit document content and credentials from default logs.
- Package optional agent framework adapters separately after the SDK is stable; harness installation is part of this step and calls the public service layer without adding independent storage semantics.

## Work sequence

1. Freeze the Python service schema and shared error types; package the local service, CLI, and stdio MCP entry point.
2. Implement MCP read tools with the same contract and bounds, then authorized write tools.
3. Implement harness configuration adapters, `doctor`, and removal. Test a fresh local install in each harness with a small DCLX fixture.
4. Add remote HTTP setup and authentication checks.
5. Publish installation guides and a compatibility matrix. Add framework adapters only where a concrete integration test exists.

## Acceptance checks

- For the same fixture and principal, SDK, CLI JSON, and MCP return equivalent URIs, revisions, XPaths, and content bounds.
- Bad scopes, oversized results, invalid XPath, and missing packages produce the defined errors on every surface.
- A local installation runs ingest, outline, search, and cited `show` without credentials, network, or a cloud account.
- A long conversion returns a job ID and can be polled after process restart.
- A new user can install the package, run one integration command, restart each supported harness, and retrieve a cited DocLang node.
- User and project installs coexist without overwriting unrelated settings. Reinstall, upgrade, and removal behave predictably for all four harnesses.
- All four harnesses expose equivalent read tools and citation fields. Local setup requires no cloud account; remote setup does not put secrets in project configuration.

## Exit artifact

An installable local product and verified integrations for Codex, Claude Code, Hermes, and Pi whose public operations can be served by the cloud backend without a contract change.

## Harness documentation to recheck at implementation time

- [Codex MCP setup](https://developers.openai.com/learn/docs-mcp)
- [Claude Code MCP setup](https://code.claude.com/docs/en/mcp)
- [Hermes Agent MCP setup](https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp/)
- [Pi MCP setup](https://pi.dev/docs/latest/mcp)
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ A local context tree that stores documents as immutable DocLang `.dclx` packages

See the [capability guide](docs/index.md), [retrieval guide](docs/retrieval.md),
[session guide](docs/sessions.md), [memory guide](docs/memory.md),
[agent harness guide](docs/agents.md),
[CLI reference](docs/cli.md), and
[single PDF and folder examples](examples/adding_resources/README.md).

Expand Down Expand Up @@ -49,6 +50,8 @@ uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid
uv run dc --store ./context-data session start --id chat-1
uv run dc --store ./context-data session list
uv run dc --store ./context-data memory list --status proposed
uv run docling-context integrate codex --scope project --store ./context-data
uv run docling-context doctor codex --scope project --store ./context-data
```

Use `uv run dc` or activate the project environment before running `dc` by
Expand Down
127 changes: 127 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Agent harness installation

Install `docling-context` into a Python environment visible to the harness. The
package exposes `dc` for document work and `docling-context` for MCP serving and
harness integration. Local serving uses the same SQLite store and DCLX packages
as `dc`; no cloud account is needed.

```bash
uv sync --extra conversion
uv run dc add-resource ./paper.pdf --collection papers
uv run docling-context integrate codex --scope project
uv run docling-context doctor codex --scope project
```

`integrate` launches a temporary MCP server and checks handshake, tool discovery,
backend listing, and one immutable citation when indexed content exists. Then it
adds a named `docling-context` entry to the harness configuration. Restart the
harness to load it. `doctor` repeats the checks; `remove` deletes only an entry
still owned by this installer. Existing entries with the same name are reported
as conflicts. Repeating `integrate` is safe, and changing options upgrades only
the owned entry. The installer writes a small ownership marker next to the
config; keep it alongside the config for future upgrades and removal.

## Harnesses

| Harness | Local version observed | User config | Project config | Local stdio | Remote HTTP | Verification |
| --- | --- | --- | --- | --- | --- | --- |
| Codex | 0.160.0 | `~/.codex/config.toml` | `.codex/config.toml` | Yes | Yes | Config test, MCP handshake, CLI lists entry |
| Claude Code | 2.1.278 | `~/.claude.json` | `.mcp.json` | Yes | Yes | Config test, MCP handshake, CLI sees entry pending project approval |
| Hermes Agent | 0.21.1 | `~/.hermes/config.yaml` | Unsupported by Hermes | Yes | Yes | Config test, MCP handshake, CLI lists entry |
| Pi | Not installed locally | `~/.pi/agent/mcp.json` | `.pi/mcp.json` | Yes | Yes | Config test and MCP SDK handshake; native CLI unverified |

The config tests check preservation, repeat installs, upgrades, and removal. The
MCP handshake is tested with the Python MCP client; harness binaries are not
required for the test suite. Project configuration is reviewable in Git.
Codex and Pi may require project trust before loading project MCP entries.
Configuration paths and transports follow the current
[Codex](https://developers.openai.com/codex/mcp),
[Claude Code](https://code.claude.com/docs/en/mcp),
[Hermes](https://hermes-agent.nousresearch.com/docs/reference/mcp-config-reference/),
and [Pi](https://pi.dev/docs/latest/mcp) MCP guides.

```bash
uv run docling-context integrate claude --scope user
uv run docling-context integrate hermes --scope user
uv run docling-context integrate pi --scope project
uv run docling-context remove pi --scope project
```

The local entry uses the absolute Python executable of the current environment.
Keep that environment installed and accessible after restarting the harness.
Use `--store PATH`, `--tenant ID`, and `--user ID` on `integrate` and `doctor`
to select a local store and principal. Those IDs select local scope; they are
not remote authentication credentials. If the chosen store has no indexed
document, `doctor` reports that cited retrieval could not be exercised.

## Using the tools

Ask the agent to call `search` with a narrow URI scope, inspect relevant
documents with `outline`, and call `show` using the returned document URI,
immutable `revision_id`, and XPath. Keep those three values with quoted text
for a reproducible citation. `show` accepts `format="xml"` for bounded DocLang
XML. `search(include_context=True)` also assembles the bounded cited hits into
one context string. `@toc:` and `@summary:` search paths refer to package
sidecars; `show` accepts those paths, and `outline` exposes the TOC links.
With a configured image-capable vector model, `search` accepts an optional
base64 image query (up to 2.5 MB) alongside query text. A text-only model
silently skips the image contribution.

Read tools are on by default. Local write tools require `--allow-writes` on
`integrate` or `docling-context mcp`; write tools include `ingest`,
`session_append`, and `memory_review`. Memory capture remains opt-in. Each
operation checks the principal's tenant and user scope. The current local
principal comes from the server command, not from tool arguments.

MCP `ingest` accepts base64 source bytes up to 2.5 MB and returns a durable job
ID. Run `docling-context ingest-worker` alongside the agent, or
`docling-context ingest-worker --once` for one job. `job_status` reports queued,
running, completed, or failed state and the resulting revision. Jobs survive
server and worker restarts. Larger documents can be imported with `dc
add-resource`, which converts synchronously and indexes the result. Use
`dc jobs status JOB_ID` to inspect either job type; add `--json` for a record.
Queued Docling conversions use a five-minute document timeout.

## Remote service

Run the HTTP service behind a TLS terminator and provide a bearer token in an
environment variable. The server binds that token to the configured tenant and
user; callers cannot substitute a principal in requests. The server refuses to
start without the token.

```bash
export DOCLING_CONTEXT_TOKEN='your-deployment-token'
uv run docling-context mcp --transport http --host 127.0.0.1 --port 8765 \
--store ./context-data --tenant team --user agent
uv run docling-context integrate codex --mode remote \
--url https://context.example.com/mcp
```

`integrate` and `doctor` need the token in their own environment to verify the
remote MCP endpoint. Generated config contains only the variable name or
`${VARIABLE}` reference, never the token value. Remote write tools are enabled
only on the server with `--allow-writes`. Restrict the deployment token and use
separate server instances or authentication mappings for distinct principals.

The same version 1 operations are available to Python through
`RemoteContextService` at `/v1/{operation}` and locally through
`ContextService`. For example:

```python
from docling_context import ContextService, LocalContextStore, Principal, SearchRequest

with LocalContextStore("./context-data") as store:
service = ContextService(store, Principal("team", "agent"))
hits = service.search(SearchRequest("Koonap", uri="docling://resources/papers"))
for hit in hits.data["hits"]:
print(hit["uri"], hit["revision_id"], hit["xpath"])
```

For a deployment, use the same `SearchRequest` with
`RemoteContextService("https://context.example.com/mcp", token)`. The token
stays in the client process and is sent only as a bearer header.

The HTTP API requires `schema_version: 1`, returns an operation ID, and uses
stable error codes such as `invalid_uri`, `unauthorized_scope`,
`missing_revision`, `stale_citation`, `package_failure`, and
`budget_exceeded`. Content-bearing responses mark truncation.
15 changes: 15 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ uses the name `dc` on some machines.
uv run --extra conversion dc --store ./context-data add-resource ./paper.pdf \
--collection papers --no-ocr --tables --no-charts
uv run dc --store ./context-data task status JOB_ID
uv run dc --store ./context-data jobs status JOB_ID --json
uv run dc --store ./context-data worker --once
uv run dc --store ./context-data status
uv run dc --store ./context-data index status
Expand All @@ -21,12 +22,17 @@ uv run --extra vectors dc --store ./context-data index vector rebuild
uv run --extra vectors dc --store ./context-data search "invoice" --mode hybrid
uv run dc --store ./context-data ls docling://resources/
uv run dc --store ./context-data tree docling://resources/papers -L 2
uv run dc --store ./context-data outline docling://resources/papers/paper
uv run dc --store ./context-data show docling://resources/papers/paper \
'/doclang[1]/text[1]' --format xml --json
uv run dc --store ./context-data find "invoice" --uri docling://resources/papers
uv run dc --store ./context-data grep "total" --uri docling://resources/papers
uv run dc --store ./context-data search "invoice" --uri docling://resources/papers
uv run dc --store ./context-data session start --id chat-1
uv run dc --store ./context-data session list --status open
uv run dc --store ./context-data memory list --status proposed
uv run docling-context integrate codex --scope project --store ./context-data
uv run docling-context doctor codex --scope project --store ./context-data
```

`add-resource` accepts one document or a folder of documents. With the
Expand All @@ -51,6 +57,15 @@ job; `add-resource` prints the revision and its `task_id`. `task status` reports
that job. Run `worker --once` for one due job, or `worker` continuously. A
collection summary may remain stale until its latest job completes.

`outline URI` reads the TOC sidecar and `show URI XPATH` reads one cited node.
Use `--revision ID` for an immutable revision, `--format xml` for DocLang XML,
and `--max-chars N` to bound output. Both commands support `--json`; JSON
includes the URI, document ID, revision ID, XPath, and truncation marker.

`docling-context` exposes the existing `dc` commands and adds `mcp`,
`integrate`, `doctor`, `remove`, and `ingest-worker`. See the
[agent harness guide](agents.md) for scopes, remote tokens, and verification.

Every `status` command prints a table by default. Pass `--json` for structured
output, for example `dc status --json`, `dc task status JOB_ID --json`, or
`dc index vector status --json`.
Expand Down
5 changes: 4 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Capabilities

`docling-context` stores document revisions as DocLang DCLX packages and exposes
them through local Python and `dc` command-line APIs.
them through Python, `dc`, MCP, and authenticated HTTP APIs.

| Capability | Current behavior |
| --- | --- |
Expand All @@ -16,6 +16,8 @@ them through local Python and `dc` command-line APIs.
| Search | `dc search` provides scoped lexical, vector, and hybrid retrieval, including picture assets with an image-capable model |
| Sessions | `dc session` records, lists, replays, closes, and purges revisioned event packages |
| Durable memory | `dc memory` lists and reviews DCLX claims, runs compilation jobs, and supports scoped recall |
| Agent harnesses | One-command MCP integration for Codex, Claude Code, Hermes, and Pi |
| Public service | Versioned Python and authenticated HTTP operations for inspection, search, ingestion, jobs, sessions, and memory |

`find` and `grep` retain their bounded substring scans. Indexed retrieval
verifies citations against the current DCLX revision before returning text.
Expand All @@ -27,4 +29,5 @@ verifies citations against the current DCLX revision before returning text.
- [Storage and jobs](storage.md)
- [Agent sessions](sessions.md)
- [Durable memory](memory.md)
- [Agent harness installation and MCP](agents.md)
- [Single PDF and folder examples](../examples/adding_resources/README.md)
11 changes: 10 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,15 @@ classifiers = [
"Topic :: Database",
"Topic :: Text Processing :: Markup :: XML",
]
dependencies = ["doclang", "tabulate>=0.10,<1", "tqdm>=4.66,<5"]
dependencies = [
"doclang",
"httpx>=0.27,<1",
"mcp>=1.20,<2",
"ruamel.yaml>=0.18,<1",
"tabulate>=0.10,<1",
"tomlkit>=0.13,<1",
"tqdm>=4.66,<5",
]

[project.optional-dependencies]
conversion = ["docling[rapidocr]>=2.134.0,<3"]
Expand All @@ -33,6 +41,7 @@ vectors = [

[project.scripts]
dc = "docling_context.cli:main"
docling-context = "docling_context.app:main"

[project.urls]
Repository = "https://github.com/docling-project/docling-context"
Expand Down
Loading
Loading