Skip to content

feat: add concepts of projects - #5

Open
PeterStaar-IBM wants to merge 5 commits into
mainfrom
feat/add-concepts-of-projects
Open

PeterStaar-IBM wants to merge 5 commits into
mainfrom
feat/add-concepts-of-projects

Conversation

@PeterStaar-IBM

@PeterStaar-IBM PeterStaar-IBM commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Summary

This branch reorganizes docling-context around projects and a shared resource library. Before, documents
lived in per-collection folders. Now every source is converted to DCLX once, stored in a flat shared library, and
linked into as many projects as need it. A project represents a topic of work.

On top of that model, the branch adds:

  • Typed resources: concepts, skills, memories and structured knowledge, each a first-class, citable DCLX
    resource.
  • Concept-grounded knowledge: datasets of cited facts, validated against a pinned concept schema.
  • Project-scoped agent sessions: a coding agent (Codex, Claude Code, Hermes, Pi) runs inside a project with one
    command.
  • A simpler, canonical CLI, with one command per operation and project-aware search.
  • A single SQLite catalog: no database server and no PostgreSQL setup.

Scale: 46 files changed, about 5,900 lines added and 1,000 removed, plus about 1,300 new lines of tests.


Why this matters

Before After
Documents were tied to one collection. Reusing a paper in two efforts meant importing it twice. One library
document can be linked into any number of projects without copying.
Re-importing the same file created a new document. Source bytes are SHA-256 hashed before conversion. A
repeat import, from a local file or a URL, reuses the existing library URI.
Knowledge was unstructured text. Knowledge is typed facts. They are validated against a concept schema, with
units, and each one cites an exact source (URI, revision, XPath).
Agents could reach the store through MCP, but sessions had no project or agent identity. `dc session start
--project X --agent codex` launches the agent, records the session under the project, and closes it on exit.
The CLI had overlapping routes (project session-*, top-level search, status, collection commands). One
canonical command tree: init, overview, project, library, skill, concept, knowledge, index, session, memory, agent.

|
| Search was scoped by URI prefix only. | library search --project ID returns only resources linked to that
project. |


What's new

1. Shared resources and projects

A new namespace layout:

docling://
├── resources/
│   ├── library/{doc_id}         original sources converted to DCLX
│   ├── memories/{memory_id}     session outcomes and conclusions
│   ├── skills/{skill_id}        reusable methods
│   ├── concepts/{concept_id}    entity and relationship definitions
│   └── knowledge/{knowledge_id} structured knowledge descriptors
├── projects/{project_id}/
│   ├── .description.dclx        optional short description
│   ├── .abstract.dclx           title, description, linked counts
│   ├── .overview.dclx           linked resource inventory
│   └── sessions/{session_id}
└── user/{user_id}/
  • catalog.py (new, about 780 lines) is the SQLite catalog. Its tables cover projects, resources, revisions,
    project-resource links, typed resource-to-resource links, project sessions, project documents, knowledge datasets and
    facts, and project-context jobs.
  • Tenant boundaries: foreign keys and unique constraints are tenant-scoped.
  • Project membership: project link / project unlink change membership only. Unlinking never deletes the
    resource.
  • Default skill: every project gets the shared doclang-authoring skill linked automatically. Projects that
    already exist when a store is opened are back-filled.
  • dc overview lists every project, including empty ones. For each it counts linked documents, memories, skills,
    concepts, knowledge datasets and sessions. It also merges in the old store-status and pending-work views.

2. Project context documents

  • project_context.py (new): a durable worker that regenerates each project's .abstract.dclx and
    .overview.dclx deterministically.
  • Triggers: project creation, membership changes and description changes queue a refresh job.
  • Commands:
    • dc project context worker processes jobs.
    • dc project context jobs ID inspects them.
    • dc project context refresh ID queues a refresh manually.
    • dc project context set attaches project-owned DCLX documents.

3. Concepts and concept-bound knowledge

  • Concepts are DCLX definitions with a mandatory machine-readable schema in context/concept.json:
    • non-empty entity_types and properties;
    • typed properties (text, integer, number, boolean, json) with optional units, such as {"type": "number", "unit": "%"};
    • optional relationship_types with source and target entity types.
  • Creating a dataset: dc knowledge create ID --concept CONCEPT_ID --project PROJECT creates a dataset
    descriptor. It pins the concept's URI and immutable revision, and copies the validated schema into the catalog
    entry.
  • Adding facts: dc knowledge add-fact validates entity type, property name, value type and unit before storing
    a fact with an exact source citation.
  • Concept edits never silently change existing datasets. Old datasets that aren't bound to a concept must be
    recreated explicitly before they accept new facts.
  • Reuse: concepts can be shared across many datasets and projects. Dataset-to-concept and resource-to-resource
    relationships can be queried globally (resource_links, resource_projects, concept projects ID, skill projects ID).

4. Skills and memories as typed resources

  • Commands: dc skill create|list|projects and dc concept create|list|projects. These keep methods and
    definitions out of the source-document library.
  • Memories are reviewed DCLX conclusions, syntheses and session outcomes, with citations. They support
    project-scoped create, list and search, and authoring in DCLG or DCLX. The existing review, correction, recall and
    compilation commands are kept.
  • Session compilation: closing a project session queues compilation into shared memory proposals that link back
    to the project
    .

5. DocLang authoring guidance (doclang_authoring.py, new)

  • Built-in skill: doclang-authoring teaches agents to write schema-valid DCLG XML and DCLX packages: validate
    against the bundled XSD and Schematron, and avoid invented elements such as <toc>.
  • Validation: helpers validate generated summaries and outlines as real DocLang documents. Outlines are now
    stored as native annotations/toc.dclg and annotations/summary.dclg companion documents.

6. Importing from URLs (source_urls.py, new)

  • URL sources: dc library add https://arxiv.org/pdf/2608.23083 --project X works, including PDF URLs with no
    .pdf suffix. The file type comes from the response Content-Type.
  • Bounded downloads: downloads are size-limited and use the same source-hash deduplication as local files.

7. Project-aware agent sessions (agent_bridge.py, new)

There are two entry points into one session model.

Launch an agent inside a project:

dc session start --project materials --agent codex
  1. Checks that the agent executable exists.
  2. Installs the agent's MCP connection and the dc skill.
  3. Starts a DCLX session at docling://projects/{project}/sessions/{id}, with the agent's identity recorded.
  4. Runs the agent in the current terminal, passing DOCLING_CONTEXT_PROJECT, DOCLING_CONTEXT_SESSION and
    store/tenant/user details as environment variables. MCP searches default to that project.
  5. When the agent exits, closes the session, queues memory compilation and returns the agent's exit code.

--no-launch creates the session without starting an agent.

Use an agent you start yourself:

dc agent install codex --scope project   # or: status / remove
  • What it installs: the MCP entry plus a dc skill. Codex invokes it as $dc; Claude Code, Hermes and Pi use
    /dc.
  • Safe and idempotent: it refuses to overwrite files it doesn't own and removes only its own files.
  • Supported agents: Codex, Claude Code, Hermes, Pi.
  • Session listing: dc session list --project ID shows each session's agent and state.

8. Service, SDK and MCP surface

service.py, remote.py and mcp_server.py expose the new model through Python, HTTP and MCP:

Access Operations
Read overview, project_show, project_resources, resource_links, resource_projects,
knowledge_facts
Write project_create, project_link, project_unlink, resource_link, knowledge_add_fact,
project_session_start, project_session_close

Existing MCP tool names stay the same.

9. Retrieval and indexing hardening

  • Project scoping: SearchScope.project_id limits lexical, vector and hybrid search to the project's linked
    resources. The allowed set of IDs is computed without loading text or embeddings.
  • Vector checks: verified_vector_state checks the snapshot hash before loading a possibly expensive embedding
    model. Missing, corrupt or model-mismatched vector storage now fails with a clear VectorStorageUnavailable error
    and a rebuild hint. A warning appears when the vector projection lags behind the current searchable units.
  • Citations: results still carry the canonical URI, immutable revision ID and XPath.

10. CLI simplification

  • Command tree: init, overview, project, library, skill, concept, knowledge, index, session, memory, agent.
  • library owns import, listing, tree, show, outline, find/grep, indexed search and ingest jobs.
  • Removed duplicates: project session-* and collection-specific commands are gone.
  • Bare dc shows the overview plus starter examples.
  • Session IDs returned by the CLI work in later commands. Ambiguous IDs are rejected rather than resolved
    silently.
  • Output: tabular by default; --json for scripts and agents.

11. Documentation and dependencies

  • Docs rewritten around SQLite and projects: README.md, docs/index.md (with a "5 minutes to fun" quick
    start), and the cli, storage, agents, sessions, retrieval, memory and conversion guides.
  • Dependency updates: docling[rapidocr]>=2.138.0 plus docling-parse>=7.22.3. DocLang is pinned to 24b7ca6.
  • Implementation records in .plans/planned/:
    • refactor-to-have-projects.md
    • simplify-cli-and-concept-knowledge.md
    • project-agent-sessions.md

Quick start

uv sync --extra conversion
uv run dc init
uv run dc project create materials --title "Materials research"
uv run dc library add ./paper.pdf --project materials
uv run dc overview
uv run dc library search "material properties" --project materials
uv run dc session start --project materials --agent claude

Testing

New and expanded tests:

Test file What it covers
test_projects_catalog.py (new) Deduplication (including concurrent imports), multi-project links, unlink
preservation, overview counts, tenant isolation, concept schemas, pinned revisions, fact rejection
test_agent_sessions.py (new) Project and agent requirements, launch and return, environment and argument
construction, exit and failure handling, idempotent installer ownership
test_source_urls.py (new) Bounded URL downloads and content-type detection
test_retrieval.py, test_cli.py, test_service_mcp.py, test_session_memory_cli.py Updated for project
scoping and the canonical CLI routes

The implementation notes report Ruff, mypy and the full pytest suite passing (119 tests). The installed codex,
claude, hermes and pi executables were checked with their help and version commands. Interactive agent runs
were tested with a subprocess stand-in, not live model sessions.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant