Repository navigation
feat: add concepts of projects - #5
Open
PeterStaar-IBM wants to merge 5 commits into
Open
PeterStaar-IBM wants to merge 5 commits into
PeterStaar-IBM wants to merge 5 commits into
Conversation
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch reorganizes
docling-contextaround projects and a shared resource library. Before, documentslived in per-collection folders. Now every source is converted to DCLX once, stored in a flat shared library, and
linked into as many projects as need it. A project represents a topic of work.
On top of that model, the branch adds:
resource.
command.
Scale: 46 files changed, about 5,900 lines added and 1,000 removed, plus about 1,300 new lines of tests.
Why this matters
collection. Reusing a paper in two efforts meant importing it twice.project session-*, top-levelsearch,status, collection commands).init, overview, project, library, skill, concept, knowledge, index, session, memory, agent.|
| Search was scoped by URI prefix only. |
library search --project IDreturns only resources linked to thatproject. |
What's new
1. Shared resources and projects
A new namespace layout:
catalog.py(new, about 780 lines) is the SQLite catalog. Its tables cover projects, resources, revisions,project-resource links, typed resource-to-resource links, project sessions, project documents, knowledge datasets and
facts, and project-context jobs.
project link/project unlinkchange membership only. Unlinking never deletes theresource.
doclang-authoringskill linked automatically. Projects thatalready exist when a store is opened are back-filled.
dc overviewlists every project, including empty ones. For each it counts linked documents, memories, skills,concepts, knowledge datasets and sessions. It also merges in the old store-status and pending-work views.
2. Project context documents
project_context.py(new): a durable worker that regenerates each project's.abstract.dclxand.overview.dclxdeterministically.dc project context workerprocesses jobs.dc project context jobs IDinspects them.dc project context refresh IDqueues a refresh manually.dc project context setattaches project-owned DCLX documents.3. Concepts and concept-bound knowledge
context/concept.json:entity_typesandproperties;text,integer,number,boolean,json) with optional units, such as{"type": "number", "unit": "%"};relationship_typeswith source and target entity types.dc knowledge create ID --concept CONCEPT_ID --project PROJECTcreates a datasetdescriptor. It pins the concept's URI and immutable revision, and copies the validated schema into the catalog
entry.
dc knowledge add-factvalidates entity type, property name, value type and unit before storinga fact with an exact source citation.
recreated explicitly before they accept new facts.
relationships can be queried globally (
resource_links,resource_projects,concept projects ID,skill projects ID).4. Skills and memories as typed resources
dc skill create|list|projectsanddc concept create|list|projects. These keep methods anddefinitions out of the source-document library.
project-scoped create, list and search, and authoring in DCLG or DCLX. The existing review, correction, recall and
compilation commands are kept.
to the project.
5. DocLang authoring guidance (
doclang_authoring.py, new)doclang-authoringteaches agents to write schema-valid DCLG XML and DCLX packages: validateagainst the bundled XSD and Schematron, and avoid invented elements such as
<toc>.stored as native
annotations/toc.dclgandannotations/summary.dclgcompanion documents.6. Importing from URLs (
source_urls.py, new)dc library add https://arxiv.org/pdf/2608.23083 --project Xworks, including PDF URLs with no.pdfsuffix. The file type comes from the responseContent-Type.7. Project-aware agent sessions (
agent_bridge.py, new)There are two entry points into one session model.
Launch an agent inside a project:
dcskill.docling://projects/{project}/sessions/{id}, with the agent's identity recorded.DOCLING_CONTEXT_PROJECT,DOCLING_CONTEXT_SESSIONandstore/tenant/user details as environment variables. MCP searches default to that project.
--no-launchcreates the session without starting an agent.Use an agent you start yourself:
dc agent install codex --scope project # or: status / removedcskill. Codex invokes it as$dc; Claude Code, Hermes and Pi use/dc.dc session list --project IDshows each session's agent and state.8. Service, SDK and MCP surface
service.py,remote.pyandmcp_server.pyexpose the new model through Python, HTTP and MCP:overview,project_show,project_resources,resource_links,resource_projects,knowledge_factsproject_create,project_link,project_unlink,resource_link,knowledge_add_fact,project_session_start,project_session_closeExisting MCP tool names stay the same.
9. Retrieval and indexing hardening
SearchScope.project_idlimits lexical, vector and hybrid search to the project's linkedresources. The allowed set of IDs is computed without loading text or embeddings.
verified_vector_statechecks the snapshot hash before loading a possibly expensive embeddingmodel. Missing, corrupt or model-mismatched vector storage now fails with a clear
VectorStorageUnavailableerrorand a rebuild hint. A warning appears when the vector projection lags behind the current searchable units.
10. CLI simplification
init, overview, project, library, skill, concept, knowledge, index, session, memory, agent.libraryowns import, listing, tree,show,outline,find/grep, indexed search and ingest jobs.project session-*and collection-specific commands are gone.dcshows the overview plus starter examples.silently.
--jsonfor scripts and agents.11. Documentation and dependencies
README.md,docs/index.md(with a "5 minutes to fun" quickstart), and the
cli,storage,agents,sessions,retrieval,memoryandconversionguides.docling[rapidocr]>=2.138.0plusdocling-parse>=7.22.3. DocLang is pinned to24b7ca6..plans/planned/:refactor-to-have-projects.mdsimplify-cli-and-concept-knowledge.mdproject-agent-sessions.mdQuick start
Testing
New and expanded tests:
test_projects_catalog.py(new)test_agent_sessions.py(new)test_source_urls.py(new)test_retrieval.py,test_cli.py,test_service_mcp.py,test_session_memory_cli.pyThe implementation notes report Ruff, mypy and the full pytest suite passing (119 tests). The installed
codex,claude,hermesandpiexecutables were checked with their help and version commands. Interactive agent runswere tested with a subprocess stand-in, not live model sessions.