Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .plans/planned/project-agent-sessions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Project agent sessions and agent command installation

Status: implemented in the working tree. The plan remains here as the requested implementation record.

## Goal

Support two entry points into one project-scoped session model:

1. `dc session start --project materials --agent codex` starts a durable project session, prepares the selected agent's Docling Context MCP connection and `dc` skill, runs the agent in the current terminal, and returns to the caller when it exits.
2. `dc agent install codex` installs the same MCP connection and reusable `dc` skill for an agent started independently. The skill guides project selection, session recording, scoped search, citations, and closure.

Support Codex, Claude Code, Hermes, and Pi. Use the existing local SQLite store and DCLX session packages. Keep the original caller's working directory and terminal.

## Current state

- The CLI has separate user-scoped `session` commands and `project session-*` commands. Project sessions already have DCLX URIs and a catalog link, but no agent identity.
- `docling-context integrate` installs an MCP entry only. The MCP tool set already includes project-aware search, project session start/close, and session append.
- Project refresh jobs generate deterministic abstract/overview inventory documents. They are unrelated to launching agents or session recording.
- Graphify installs agent skills under per-agent user/project paths. Codex's current official skill syntax is `$dc`; Claude, Hermes, and Pi expose skills through their own command menus.

## Proposed implementation

### Session model and CLI

- Store `agent` in `dc_project_sessions`. Keep a local additive schema update for existing SQLite stores.
- Make `dc session start` require `--project` and `--agent`. Generate or accept `--id`. Add `--no-launch` for an agent already running (the skill path). Return the session URI, project, and agent.
- On normal launch, verify the agent executable and install its MCP and skill integration, then start the DCLX session. Pass the full session URI, project ID, agent, store, tenant, and user to the child as environment variables and a short initial prompt. MCP searches default to that project. Wait for the agent process; on exit, close the session, which queues memory compilation. Preserve exit code and terminal control. Do not automatically run a memory worker.
- Show project sessions, including agent, with `dc session list --project PROJECT_ID`; project sessions also appear in `dc project session-list`.
- Keep explicit project session commands and the internal user session API for compatibility; the main human start path is project first.

### Agent installation

- Add `dc agent install|status|remove AGENT --scope user|project`. Installation reuses the owned MCP entry in `integration.py` and adds a small `dc` skill in the agent's skill directory. Installation is idempotent; refuse to overwrite unmanaged files and remove only owned files.
- Launcher installs user-scoped integration so it works from any directory. Explicit project scope writes into the current working directory for Codex, Claude, and Pi. Hermes uses user scope for its MCP integration. Skill paths follow the installed agents and the Graphify reference.
- The skill uses the Docling Context MCP tools for project-scoped retrieval and DCLX session events. When independently invoked, it starts a session for an explicit project, or asks for a project if several are available. It does not launch another agent.
- The launcher reuses the installer. No shell interpolation for agent arguments; use an argument list and inherited terminal streams.

### Documentation and verification

- Update `docs/index.md`, `docs/agents.md`, `docs/sessions.md`, and `docs/cli.md` with the two workflows, project scoping, agent-specific invocation names, and job distinctions.
- Unit tests cover project/agent requirement, DCLX and catalog association, subprocess argument/environment construction, exit and failure handling, and idempotent installer ownership in temporary homes/project roots.
- Run CLI help and discovery checks against installed `codex`, `claude`, `hermes`, and `pi` binaries. Do not start live model conversations during tests.

## Acceptance

- Running `dc session start --project materials --agent codex` enters Codex and returns to the original shell when Codex exits.
- The session is visible under the project with its agent and closed state.
- `dc agent install` makes the agent's Docling Context tools and skill available without starting through `dc`.
- Existing unrelated agent configuration and skills are preserved.
- Docs identify that Codex invokes the skill as `$dc`, while other harnesses may display `/dc`.

## Verification

- Full test suite: 111 passed.
- Ruff and mypy passed for the source tree.
- The installed Codex, Claude Code, Hermes, and Pi executables answered their version and help commands. Interactive model sessions were exercised with a subprocess test double rather than live prompts.
72 changes: 72 additions & 0 deletions .plans/planned/refactor-to-have-projects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Shared resources and projects

## Agreed scope

Use a fresh local store. SQLite is the single catalog for resource identity,
revisions, project membership, typed relationships, jobs, sessions, search
indexes, and cited structured knowledge. DCLX packages are content-addressed
files beside the SQLite database. No PostgreSQL server or migration is needed.

```text
docling://
├── resources/
│ ├── library/{doc_id}
│ ├── memories/{memory_id}
│ ├── skills/{skill_id}
│ ├── concepts/{concept_id}
│ └── knowledge/{knowledge_id}
├── projects/{project_id}/
│ ├── .description.dclx
│ ├── .abstract.dclx
│ ├── .overview.dclx
│ └── sessions/{session_id}
└── user/{user_id}/
```

A project is a topic of work. Its membership links to subsets of flat shared
resources without copying packages. Source bytes are hashed before conversion;
a repeat import reuses the existing library document. `add-resource --project`
links the imported document. `dc overview` lists every project with counts of
linked documents, memories, skills, concepts, knowledge, and sessions.

## Implementation areas

1. Use `LocalContextStore` and one SQLite catalog for project and resource
tables. Keep tenant-scoped foreign keys and unique constraints. Store raw
source hashes separately from DCLX package hashes and immutable revisions.
2. Route `add-resource` to the shared library; replace `--collection` with
`--project`. Preserve exact URI, revision, and XPath citations. Support
project-filtered lexical and vector retrieval.
3. Store reviewed memories in the flat memory list, compile project sessions
into linked memory proposals, and validate DCLX skill, concept, and
knowledge descriptors. Store typed knowledge facts with source citations.
4. Refresh project abstract and overview DCLX documents after project changes.
Expose project and resource links, overview, and knowledge facts in CLI,
local SDK, HTTP, and MCP.
5. Retire old collection-specific public commands. Keep private user sessions
available until a project is
selected; new project sessions live under the project URI.
6. Rewrite `README.md` and `docs/` around SQLite, a five-minute quick start,
and tested installer recipes for Codex, Claude Code, Hermes, and Pi.

## Acceptance checks

- Duplicate imports reuse a document ID, including concurrent imports.
- A resource can link to multiple projects; unlinking preserves it.
- Overview counts are correct for empty and populated projects.
- Project search returns only linked resources and citations remain readable.
- Tenant boundaries and memory owner review checks hold.
- Cited knowledge facts validate field types and source revisions.
- Project sessions compile into shared, project-linked memory proposals.
- CLI, SDK, MCP, docs, and lockfile agree on SQLite; no PostgreSQL setup is
required.
- Run tests, lint, type checking, and smoke-test the quick start.

## Current implementation

The SQLite catalog, shared library deduplication, project membership and
search, flat memories, typed descriptors, knowledge facts, project context jobs,
CLI overview, and service/MCP project operations are implemented. Old
collection summary commands are removed from the CLI; internal compatibility
code and private user sessions remain. The documentation describes the SQLite
store and project workflow.
76 changes: 76 additions & 0 deletions .plans/planned/simplify-cli-and-concept-knowledge.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Simplify the CLI and ground knowledge in concepts

## Agreed behavior

- Use one canonical command for each operation; remove duplicate CLI routes.
- The top level is `init`, `overview`, `project`, `library`, `session`, `memory`,
`knowledge`, `concept`, `skill`, `index`, and `agent`. The two extra typed groups
keep concepts and skills out of the source-document library.
- `library` owns document import, listing, tree navigation, exact node reading,
outlines, lexical lookup (`find` and `grep`), indexed search, and ingest jobs.
- Projects manage titles, shared resource links, and project context. Group context
refresh, show, set, jobs, and worker under `project context`.
- Remove `project session-*`. `session start --project ID --agent NAME` launches
the agent in the calling terminal, closes the session on exit, and returns to
the caller. `--no-launch` explicitly creates a session without launching.
- Session IDs returned by the CLI work in subsequent commands; reject ambiguous
IDs instead of choosing a project silently.
- Merge store status and pending-work counts into overview. Bare `dc` shows the
overview and examples for creating a project, importing documents, and launching
an agent. Keep the default SQLite store and global `--store` override.
- Memories are reviewed DCLX conclusions, syntheses, and session outcomes, with
citations. Support project-scoped create/list/search and DCLX authoring, retain
review, correction, recall, and compilation commands.
- Concepts are DCLX definitions with a mandatory machine-readable schema for
entity types, typed properties, optional units, and relationship types.
- `knowledge create ID --concept CONCEPT_ID --project PROJECT` creates a DCLX
dataset descriptor and stores cited facts in SQLite. It never takes an
independent `--schema`. Pin the concept URI and immutable revision and copy the
validated schema into the dataset catalog entry. Validate entity types, property
names, value types, and units when adding facts. Later concept edits do not
mutate existing datasets.
- Concepts can be reused by multiple datasets and projects. Dataset-to-concept,
resource-to-resource, and project-to-resource relationships remain globally
queryable. Source citations retain URI, revision, and XPath.
- No CLI compatibility aliases or automatic data migration. Existing documents
and projects remain readable; unbound old datasets require explicit recreation
before accepting new facts.

## Implementation steps

1. Refactor argparse command groups and dispatch while reusing existing import,
search, session launcher, job, and inspection implementations.
2. Add typed concept/skill creation and listing, ID-to-URI resolution, concept
schema validation, and concept-bound knowledge creation/show/list/filtering.
3. Resolve project session IDs and add project-scoped memory workflows without
bypassing review or authorization rules.
4. Update CLI/index/agent/storage/conversion/retrieval/session documentation,
README examples, generated `/dc` skill, and command references in source.
Keep MCP tool names stable: recording a session through MCP is separate from
launching an external interactive agent through the CLI.
5. Update existing CLI tests to the canonical routes. Test help, bare invocation,
agent launch/return, resource scoping, memory project links, mandatory concept
schemas, pinned revisions, fact rejection, and exact citations. Run the full
suite and required lint/type checks.

## Schema contract

Concept metadata lives in `context/concept.json` inside DCLX. Its `schema` has
nonempty `entity_types` and `properties`; properties map names to definitions
such as `{"type":"number","unit":"%"}`. Supported value types are `text`,
`integer`, `number`, `boolean`, and `json`. Optional `relationship_types` map names
to source/target entity types. Definitions and documentation live in the DCLG
document body. DCLG input requires descriptor metadata with the mandatory schema;
DCLX input may carry the descriptor directly.

## Verification and completion

- Implemented the canonical CLI groups and removed duplicate top-level and
project-session routes. Bare `dc` shows the overview and starting examples.
- Added document, skill, and concept relationship commands; project-scoped
memory authoring and lookup; and authored DCLG/DCLX memory preservation.
- Knowledge datasets now require a concept, pin its URI and revision, and
validate typed facts and units against the copied schema.
- Updated the CLI, agent, storage, session, retrieval, conversion, and index
documentation, plus the generated agent skill.
- Verified with Ruff, mypy, and the full pytest suite (119 tests passed).
Loading
Loading