diff --git a/.agents/skills/docs-governance/SKILL.md b/.agents/skills/docs-governance/SKILL.md index 1727fa5..c664180 100644 --- a/.agents/skills/docs-governance/SKILL.md +++ b/.agents/skills/docs-governance/SKILL.md @@ -15,6 +15,7 @@ Each doc has exactly one job, keep content in its lane: | `packages/*/CHANGELOG.md` | Each package's version history, written by changesets | Marketing prose, roadmap | | `apps/docs/content/reference/pi/comparison.mdx` | Audited Pi peer comparison; the page and the source are one file | Feature marketing, unverified cells | | `apps/docs/content/reference/opencode/comparison.mdx` | Audited OpenCode v2 peer comparison; the page and the source are one file | Feature marketing, unverified cells | +| `apps/docs/content/reference/deepseek/comparison.mdx` | Audited DeepSeek peer comparison; the page and the source are one file | Feature marketing, unverified cells | | `apps/docs/content/_snippets/**` | Shared Markdown or MDX fragments, spliced into the per-harness wrapper pages with `` | Frontmatter, a second full copy of a page, or the other harness's content | | `packages/core/README.md` | The engine's API: the entry, the two ports, the boundary rule | Pi-specific behaviour, the split's rationale | | `docs/roadmap.md` | Forward-looking work, evidence gates, rejected items | Shipped-feature history | @@ -53,8 +54,8 @@ Each doc has exactly one job, keep content in its lane: "Last verified: …" before treating a cell as settled. Credit a peer's win in the cell: a comparison that shows only our wins is marketing. Peer package names appear in - `apps/docs/content/reference/{pi,opencode}/comparison.mdx`, and nowhere else - in this repo or on the site (owner policy). Those two pages are both the + `apps/docs/content/reference/{pi,opencode,deepseek}/comparison.mdx`, and nowhere else + in this repo or on the site (owner policy). Those three pages are both the audited source and the rendered page: edit a cell there, with its verification note and date, never in a second copy. There are no inspiration credits: the design cites the LCM paper (algorithmic @@ -71,7 +72,7 @@ Each doc has exactly one job, keep content in its lane: the landing npm shows, `packages//CHANGELOG.md` is the version history changesets writes in place, and `packages//LICENSE` is a copy of the root licence. The comparisons live in - `apps/docs/content/reference/{pi,opencode}/comparison.mdx`, so the tarball + `apps/docs/content/reference/{pi,opencode,deepseek}/comparison.mdx`, so the tarball no longer ships them. The holder named there is the org brand, `stainless-code`, as in every other stainless-code package, never a personal name. npm diff --git a/.agents/skills/harden-pr/LEDGER.md b/.agents/skills/harden-pr/LEDGER.md new file mode 100644 index 0000000..f6c4387 --- /dev/null +++ b/.agents/skills/harden-pr/LEDGER.md @@ -0,0 +1,86 @@ +# Harden ledger + +Local working file. A fresh clone does not carry it; reads become no-ops there. + +## Rejections + +By-design drops from review passes. Findings matching an entry are dropped +without re-litigation. + +- Exported-symbol JSDoc stays. Public-API docs on exported functions, types, + and fields ship in dist typings and hovers; narration flags against them + are rejected (2026-10-25 full-branch sicko pass). +- Host line-number pins and drift-rule comments stay. Each marks what mirrors + the seam so a pin move has a checklist; the pin itself is the mechanism + note (same pass). +- Plan-sketch code fences are documentation, not corpses. Fenced sketches in + docs/deepseek-plan.md describe future shapes; flags against their `//` + lines are rejected (same pass). +- MUST KILL reshapes are out of bounds. dshRef, UsageTracker, + commitDagTransaction, DagStabilityError, resolveSummarizationTarget, + MessageIndex, resolveEntryId, SpanAlreadyMemoryError, withSummaryRefs, + cross-session caps, selectDagRange, lcmCommandDefinition, pressureZone, + writeDumpTarget, whitespace sentinel, mechanicalToolNameOf split, + SidecarStores opened handles, scopedGrepAll pushdown, toolScopeMessageIds + scope state, liveStems, valueOf arg parsing, healInFlight guard, + backfillSession accounting and visited budget: all need new shapes + (redesigns), so their comments stay and the flags report open (same pass). +- UI routing comments stay. Sidebar and harness-nav comments record the + reference-only constraint and cross-file sync points a reader cannot + re-derive from the loop (same pass). +- Test-rationale comments stay when they state fixture, divergence, or + expectation rationale; pure assertion echoes go (same pass). +- Negative metric limit reads all rows by design. `readMetricRows` treats a + negative limit as unbounded and zero as empty, pinned by + `util-metrics-sink.test.ts` (2026-10-25 gauntlet pass 1). +- Root README names the host ports and bundling once. Day-1 architecture + needs the two-port contract and the inline-into-dist note; deeper internals + stay out (same pass). +- Changesets describe behavior, not just the headline. This repo ships + detailed per-package entries; the deepseek entry stays behavior-complete + rather than trimmed to a headline (same pass). +- backfillSession Budget-type and visitedBudget-function MUST KILLs are + rejected. The visited/consumed accounting is five explicit lines with the + rule in one comment plus three regression tests; a Budget type would add + indirection without removing a branch (2026-10-26 gauntlet). + +## Deferred + +- Recall scope closer (`toolScopeMessageIds` quoted-marker scan beyond + toolResult opening lines), bare `tool ` matching, cross-session + `offset`/`summary`/`include_removed` + forwarding, ghost-session read-only open for minimal deps, dispose-then-close + for live/healing stores: each + needs a design decision (sharded past-search semantics, scope state machine, + lifecycle close scheduling) beyond a polish pass. Recorded open, not fixed. + Fixed from this row: `/lcm compact` error-code mapping tests (landed with + the failure-code test), and the toolResult-row half of the scope scan + (opening-line rule now matches `toolNameOfText`, pinned by regression test). +- Warm-scan probe cost: a warm remount re-probes every stored row (one indexed + point lookup each) plus marker derivation, so per-mount work scales with + history length while only writes and new evaluations are capped. Bounding the + cheap probes too needs a persisted frontier (resume) design. Progress is + guaranteed; the cost is accepted until a frontier design lands. +- Mount fan-out: one backfill per live session runs under one Promise.all. + Live sessions per composition are inherently few, and serializing would slow + the common single-session mount. A concurrency cap waits for a composition + with enough live sessions to matter. +- Engine cold rescan: the first span miss re-derives the live log to rebuild + the index. That is the documented cold-start path (fail-closed); changing it + reshapes documented behavior. +- dag-commit point mapping: one indexed `messageByEntryId` read per shadowed + seq. A range rewrite would change the per-seq fail-closed mapping into + filter semantics; indexed point reads stand until a measured need appears. +- hydrateCheckpoint full load: cold ladder runs read all stored summaries to + rebuild one checkpoint. Loading latest-plus-children instead reshapes the + resume path; the one-time cold cost stands. +- Lossless preset forks: each `deepseek-lossless-web` preset is a full copy of + the pinned harness preset with only the backend row swapped (2026-10-26 + harden pass). Generating them from a base-plus-diff or upstreaming a + narrower preset-override hook would delete ~400 duplicated lines, but both + need a design beyond polish (harness-owned composition shape). +- Host core registry: the module-level shared core with owner counting could + move onto Cordis scoping once a host-provided service is the established + pattern for cross-realm sharing. The current shape is explicit and tested; + reshaping it would change mount/dispose semantics, so it waits for a design + pass with a live web session behind it. diff --git a/.changeset/core-deepseek-support.md b/.changeset/core-deepseek-support.md new file mode 100644 index 0000000..0dc02c0 --- /dev/null +++ b/.changeset/core-deepseek-support.md @@ -0,0 +1,5 @@ +--- +"lossless-core": patch +--- + +Support the DeepSeek adapter from core. `readMetricRows` takes `session` and `limit` filters, the metrics sink adds usage-tally helpers, `SidecarStores` evicts closed handles, `token-window` selects the compactable range, and `toolNameOfText` is exported for scope matching. diff --git a/.changeset/deepseek-bundle-manifest.md b/.changeset/deepseek-bundle-manifest.md new file mode 100644 index 0000000..b892949 --- /dev/null +++ b/.changeset/deepseek-bundle-manifest.md @@ -0,0 +1,5 @@ +--- +"deepseek-lossless": patch +--- + +Ship the profile bundle. The package declares `dsh.bundle`, ships its patch layer, icon, locale metadata, and detail doc, and exposes the Cordis plugin shape (`name`, `inject`, `Config`, `apply`) so `dsh plugin --profile add deepseek-lossless` installs as a profile layer instead of a plain dependency. Install stays per profile; the guide covers `web`, `headless`, and custom names plus the enable step that replaces the base compaction backend. diff --git a/.changeset/deepseek-json-stdout-e2e.md b/.changeset/deepseek-json-stdout-e2e.md new file mode 100644 index 0000000..8099000 --- /dev/null +++ b/.changeset/deepseek-json-stdout-e2e.md @@ -0,0 +1,5 @@ +--- +"deepseek-lossless": patch +--- + +Route the sidecar dir notice through the harness logger so headless `--json` stdout stays clean NDJSON, and scope `run-e2e.sh` sidecars to the scratch `DSH_HOME=/tmp/e2e-dsh` from the README. diff --git a/.changeset/deepseek-lossless.md b/.changeset/deepseek-lossless.md new file mode 100644 index 0000000..1a26d3a --- /dev/null +++ b/.changeset/deepseek-lossless.md @@ -0,0 +1,5 @@ +--- +"deepseek-lossless": patch +--- + +Add the DeepSeek adapter. `applyLossless` composes lossless memory into one harness composition, opt-in through `enabled`: every session mirrors into its own sidecar, and the returned disposer stops the mirror, the tools, and the command. It registers `lcm_grep`, `lcm_describe`, `lcm_expand`, and `lcm_expand_query`, plus the `/lcm` verbs `status`, `report`, `dump`, `backup`, `export`, `import`, `gc`, `migrate`, `heal`, `diagnose`, and `compact`. The summarizer engine mounts as the host's compaction backend: automatic pressure compaction stays off until `auto` opts in, manual `/lcm compact` commits through the DAG bracket, and `bracketAuto` routes the automatic path through the bracket too. `mechanicalTools` names noisy tools whose spans compact into deterministic `mechanical` leaves with zero summarizer spend. `lcm_grep` takes `query` with `scope` into past sessions (pattern stays session-scoped) of the same global sidecar directory, where `sessions` and `all_sessions` are equivalent, and `lcm_expand` reads a past session's sidecar read-only through `session`. `lcm_expand_query` runs a bounded in-process reader over `dshReader` and returns findings only. Recall covers grep filters (role, date range, tool name) and anchored expand listings. `/lcm` also grows `dump` with save and removed-row flags, gc session and age selectors, effective-threshold status and diagnose output, a configured heal model fallback, per-sidecar restorable exports, and stores that close on dispose. Model calls stamp the calling session for per-conversation routing, and span-mapping failures record a failed pass row. Sessions already live at mount backfill through the same mirror path, paged and capped, with already-mirrored rows not counting toward the visit ceiling so remounts still reach new events. Every seam writes diagnostic rows to the sidecar directory log, and `/lcm report` rolls up the session's rows. `/lcm diagnose` builds a redacted bundle, `backup`, `export`, and `diagnose --save` write owner-only (0600) files, and `/lcm gc` is age-based and dry-run by default. The composition value-imports its `@deepseek-ai/*` host packages, which stay peer dependencies. diff --git a/.changeset/deepseek-preset-close.md b/.changeset/deepseek-preset-close.md new file mode 100644 index 0000000..4f70f5a --- /dev/null +++ b/.changeset/deepseek-preset-close.md @@ -0,0 +1,6 @@ +--- +"deepseek-lossless": patch +"deepseek-lossless-web": patch +--- + +Fix preset teardown leaking open SQLite handles: whichever disposer releases the last core ownership (host or preset) now runs the shared guarded close, so idle stores always close on final release instead of only when the host happens to stop last. Also corrects the uninstall answer to lead with `dsh plugin remove` plus restart, since the shipped bundle patch owns the enable/disable rows and fresh installs create no user rows to delete. diff --git a/.changeset/deepseek-web-surface.md b/.changeset/deepseek-web-surface.md new file mode 100644 index 0000000..cd790d8 --- /dev/null +++ b/.changeset/deepseek-web-surface.md @@ -0,0 +1,6 @@ +--- +"deepseek-lossless": patch +"deepseek-lossless-web": patch +--- + +Support every DSH surface. The host composition now runs on a shared core with identity-tied, once-only release, so a stale disposer after a remount cannot evict the replacement, and the last owner alone closes shared handles. A preset-realm backend entrypoint (`deepseek-lossless/preset`) mounts the compaction engine inside agent presets without mirroring twice or closing shared handles; it fails fast naming the host row when the core is absent. The new `deepseek-lossless-web` companion bundle ships Lossless variants of all four shipped presets with only the backend row swapped, pinned to the same harness version as the peers (plus a punctuation normalization the repo requires). Installing the host bundle switches the host-plane backend with no further edits; web and desktop sessions select a Lossless preset. diff --git a/.changeset/import-hash-check.md b/.changeset/import-hash-check.md index 62a1648..cbfe3eb 100644 --- a/.changeset/import-hash-check.md +++ b/.changeset/import-hash-check.md @@ -1,4 +1,5 @@ --- +"deepseek-lossless": patch "lossless-core": patch "pi-lossless": patch "opencode-lossless": patch diff --git a/.changeset/mechanical-noisy-tier.md b/.changeset/mechanical-noisy-tier.md index 54e1469..3b3412f 100644 --- a/.changeset/mechanical-noisy-tier.md +++ b/.changeset/mechanical-noisy-tier.md @@ -2,6 +2,7 @@ "lossless-core": patch "pi-lossless": patch "opencode-lossless": patch +"deepseek-lossless": patch --- -Add a mechanical tier for noisy tools. Name tools in `mechanicalTools` in summarizer-config, `/lcm` config, or the OpenCode plugin options. Their spans compact into deterministic `mechanical` leaves with zero summarizer spend at compaction time. Every message stays stored verbatim and covered, so the integrity coverage check passes, and heal re-derives a skipped leaf from provenance on demand. There is no ingest filtering. +Add a mechanical tier for noisy tools. Name tools in `mechanicalTools` in summarizer-config, `/lcm` config, the OpenCode plugin options, or the DeepSeek composition config (`mechanicalTools`). Their spans compact into deterministic `mechanical` leaves with zero summarizer spend at compaction time. Every message stays stored verbatim and covered, so the integrity coverage check passes, and heal re-derives a skipped leaf from provenance on demand. There is no ingest filtering. diff --git a/.changeset/orphan-sidecar-report.md b/.changeset/orphan-sidecar-report.md index 83bdf4d..f3b1956 100644 --- a/.changeset/orphan-sidecar-report.md +++ b/.changeset/orphan-sidecar-report.md @@ -1,7 +1,8 @@ --- +"deepseek-lossless": patch "lossless-core": patch "pi-lossless": patch "opencode-lossless": patch --- -Report orphan sidecars in doctor. `integritySnapshot` now lists sidecar files and stale `wal` and `shm` files with no live session (`.sqlite` by default, `.db` for Pi). `checkIntegrity` reports them as a read-only `orphan-sidecar` finding through `lcm doctor` and the diagnose bundle. Report only. Deletion stays manual. +Report orphan sidecars in doctor. `integritySnapshot` now lists sidecar files and stale `wal` and `shm` files with no live session (`.sqlite` by default, `.db` for Pi). `checkIntegrity` reports them as a read-only `orphan-sidecar` finding through `lcm doctor` and the diagnose bundle. Report only. Deletion stays manual. DeepSeek carries the finding through the diagnose bundle only, since it has no doctor verb. diff --git a/.changeset/recall-search-upgrades.md b/.changeset/recall-search-upgrades.md index 89af75f..e26da7b 100644 --- a/.changeset/recall-search-upgrades.md +++ b/.changeset/recall-search-upgrades.md @@ -2,6 +2,7 @@ "lossless-core": patch "pi-lossless": patch "opencode-lossless": patch +"deepseek-lossless": patch --- -Upgrade recall search. `lcm_grep` takes `role`, `date_start`/`date_end`, and `tool_name` filters. Role and time are SQL predicates on the stored columns with a covering index. Tool scope is an FTS rewrite against `[tool:Name]` markers with no new table or index. `lcm_expand` takes an `anchor` entry id that centers the listing window with edge clamping. The recall manifest and the retrieval brief order by stop-word stripped query-word overlap over summary headers. A read-only files view lists each linked file once with an incomplete-coverage flag and a `large-inline` mark. No migration and no new writer. +Upgrade recall search. `lcm_grep` takes `role`, `date_start`/`date_end`, and `tool_name` filters. Role and time are SQL predicates on the stored columns with a covering index. Tool scope is an FTS rewrite against `[tool:Name]` markers with no new table or index; a call embedded in prose still anchors, and removed rows stay out of the visible pass unless requested. `lcm_expand` takes an `anchor` entry id that centers the listing window with edge clamping. The recall manifest and the retrieval brief order by stop-word stripped query-word overlap over summary headers. A read-only files view lists each linked file once with an incomplete-coverage flag and a `large-inline` mark. No migration and no new writer. diff --git a/.changeset/static-sidecar-dump.md b/.changeset/static-sidecar-dump.md index cca4ac1..adc1437 100644 --- a/.changeset/static-sidecar-dump.md +++ b/.changeset/static-sidecar-dump.md @@ -1,6 +1,7 @@ --- "pi-lossless": patch "opencode-lossless": patch +"deepseek-lossless": patch --- -Add a read-only static sidecar dump. `/lcm dump` and `/lossless-dump` open the session sidecar `queryOnly` and reuse the recall views and metric rows. The dump redacts displayed seams, labels removed rows, and reports old generations as named skips. Output is capped at 16 KB. +Add a read-only static sidecar dump. `/lcm dump` (plus `/lossless-dump` on OpenCode) opens the session sidecar `queryOnly` and reuses the recall views and metric rows. The dump redacts displayed seams, labels removed rows, and reports old generations as named skips. Output is capped at 16 KB. diff --git a/README.md b/README.md index 219278f..8b04ce0 100644 --- a/README.md +++ b/README.md @@ -6,18 +6,24 @@ stays searchable. Nothing is lost, and the model sees less. This repository holds the engine and the adapters that consume it: -| Package | Directory | What it is | -| -------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------ | -| [`pi-lossless`](packages/pi/README.md) | `packages/pi` | The Pi extension (`pi install npm:pi-lossless`): recall tools, cockpit, commands. | -| [`opencode-lossless`](packages/opencode/README.md) | `packages/opencode` | The OpenCode v2 plugin: recall tools, lifecycle commands, DAG projection, per-session sidecar. | -| [`lossless-core`](packages/core/README.md) | `packages/core` | The engine: ingest, compaction, and retrieval behind a two-port host contract, with no agent SDK dependency. | - -`lossless-core` is a workspace `devDependency` of the Pi and OpenCode -adapters, and the pack inlines it into `dist/index.mjs`, so each published -package is one self-contained artifact with no runtime dependencies. The Pi +| Package | Directory | What it is | +| ---------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------ | +| [`pi-lossless`](packages/pi/README.md) | `packages/pi` | The Pi extension (`pi install npm:pi-lossless`): recall tools, cockpit, commands. | +| [`opencode-lossless`](packages/opencode/README.md) | `packages/opencode` | The OpenCode v2 plugin: recall tools, lifecycle commands, DAG projection, per-session sidecar. | +| [`deepseek-lossless`](packages/deepseek/README.md) | `packages/deepseek` | The DeepSeek composition: recall tools, `/lcm` verbs, DAG bracket, per-session sidecars. | +| [`deepseek-lossless-web`](packages/deepseek-web/README.md) | `packages/deepseek-web` | Lossless presets for the DeepSeek web surface: the shipped presets with the lossless backend. | +| [`lossless-core`](packages/core/README.md) | `packages/core` | The engine: ingest, compaction, and retrieval behind a two-port host contract, with no agent SDK dependency. | + +`lossless-core` is a workspace `devDependency` of the three adapters. The packs +inline it into `dist/index.mjs`, so those packages ship with no runtime +dependencies except the host SDKs each adapter peer-depends on: the DeepSeek +bundle keeps `@deepseek-ai/*` in `peerDependencies`, resolved from the +consumer's install. The Pi adapter implements both host ports. The OpenCode adapter implements `ModelHost` and runs retrieval in a host child session, because its plugin API has no -in-process tool-calling completion. +in-process tool-calling completion. The DeepSeek adapter implements `ModelHost` +through `dshModelHost`, reads through the in-process `dshReader`, and mounts +its summarizer engine as the host's compaction backend. ## Where to work @@ -29,6 +35,9 @@ in-process tool-calling completion. | `packages/pi/test/**` | adapter tests, the Pi-shaped doubles, and the end-to-end spawn | | `packages/opencode/src/**` | the OpenCode v2 adapter, the only OpenCode-aware code in the tree | | `packages/opencode/test/**` | adapter tests, the host-shaped doubles, and the host-compat guard | +| `packages/deepseek/src/**` | the DeepSeek adapter, the only DeepSeek-aware code in the tree | +| `packages/deepseek/test/**` | adapter tests and the host-shaped doubles | +| `packages/deepseek-web/**` | the Lossless web presets bundle (patch, locale, docs) plus its surface test | | `apps/docs/**` | the documentation site (Blume), its generated pages, and the per-harness page source | | `apps/docs/content/_snippets/**` | shared guide and reference fragments; the per-harness wrappers include them | | `scripts/**` | repo tools: the long-context evaluation, the metrics report, the peer census, and the OpenCode live loop | @@ -62,7 +71,7 @@ bun run eval # the procedural long-context ladder, on demand under `packages/*/src` run it and restart Pi. A running OpenCode server reloads the plugin when its configured directory changes; its loop is `bun run opencode:check`, `bun run opencode:smoke`, and `bun run opencode:compact` -(see [packages/opencode/README.md](packages/opencode/README.md#live-loop)). +(see [packages/opencode/README.md](packages/opencode/README.md)). The docs site has its own gates: ```bash @@ -71,17 +80,19 @@ cd apps/docs && bun run build && bun run validate --strict && bun run audit ## Docs -| | | -| ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -| [The docs site](https://stainless-code.com/lossless/) | How LCM behaves in a session, the config keys, the limits | -| [pi-lossless](packages/pi/README.md) | Install, the commands, and the three recall tools | -| [opencode-lossless](packages/opencode/README.md) | Install, the commands, the four recall tools, and the live loop | -| [lossless-core](packages/core/README.md) | The engine's API and the ports a host supplies | -| [pi/comparison.mdx](apps/docs/content/reference/pi/comparison.mdx) | Audited Pi peer comparison, page and source in one file | -| [opencode/comparison.mdx](apps/docs/content/reference/opencode/comparison.mdx) | Audited OpenCode v2 peer comparison, page and source in one file | -| [docs/roadmap.md](docs/roadmap.md) | Forward-looking work, evidence gates, and rejected items | -| [pi CHANGELOG.md](packages/pi/CHANGELOG.md), [core CHANGELOG.md](packages/core/CHANGELOG.md), [opencode CHANGELOG.md](packages/opencode/CHANGELOG.md) | Per-package version history, generated from changesets | -| [AGENTS.md](AGENTS.md) | Agent rules, with the skills and lessons under `.agents/` | +| | | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| [The docs site](https://stainless-code.com/lossless/) | How LCM behaves in a session, the config keys, the limits | +| [pi-lossless](packages/pi/README.md) | Install, the commands, and the three recall tools | +| [opencode-lossless](packages/opencode/README.md) | Install, the commands, the four recall tools, and the live loop | +| [lossless-core](packages/core/README.md) | The engine's API and the ports a host supplies | +| [pi/comparison.mdx](apps/docs/content/reference/pi/comparison.mdx) | Audited Pi peer comparison, page and source in one file | +| [opencode/comparison.mdx](apps/docs/content/reference/opencode/comparison.mdx) | Audited OpenCode v2 peer comparison, page and source in one file | +| [deepseek/comparison.mdx](apps/docs/content/reference/deepseek/comparison.mdx) | Audited DeepSeek peer comparison, page and source in one file | +| [deepseek-lossless-web](packages/deepseek-web/README.md) | Install the Lossless presets for the DeepSeek web surface | +| [docs/roadmap.md](docs/roadmap.md) | Forward-looking work, evidence gates, and rejected items | +| [pi CHANGELOG.md](packages/pi/CHANGELOG.md), [core CHANGELOG.md](packages/core/CHANGELOG.md), [opencode CHANGELOG.md](packages/opencode/CHANGELOG.md), [deepseek CHANGELOG.md](packages/deepseek/CHANGELOG.md), [deepseek-web CHANGELOG.md](packages/deepseek-web/CHANGELOG.md) | Per-package version history, generated from changesets | +| [AGENTS.md](AGENTS.md) | Agent rules, with the skills and lessons under `.agents/` | ## License diff --git a/apps/docs/components/blume/Sidebar.astro b/apps/docs/components/blume/Sidebar.astro index 3a58042..6ca4746 100644 --- a/apps/docs/components/blume/Sidebar.astro +++ b/apps/docs/components/blume/Sidebar.astro @@ -17,12 +17,6 @@ interface Props { const { items, currentRoute, fragmentBase, ids, strings } = Astro.props; -const nextHost = { - label: "DeepSeek", - description: "Coming next", - icon: withBase("/logos/deepseek.svg"), -}; - const active: HostId = hostFromRoute(currentRoute) ?? "pi"; const harnessOf = (node: NavNode): HostId | undefined => @@ -54,6 +48,21 @@ const filtered = !showSwitcher const current = HOSTS.find((host) => host.id === active) ?? HOSTS[0]; +// Every route the built tree serves. The host switcher swaps the section +// segment only against this set, so a host without a page in the current +// section falls through to its own entry instead of linking a 404. +const servedRoutes = new Set(); +const collectRoutes = (nodes: NavNode[]): void => { + for (const node of nodes) { + if (node.kind === "page") servedRoutes.add(node.route); + else { + if (node.route) servedRoutes.add(node.route); + collectRoutes(node.children); + } + } +}; +collectRoutes(items); + const firstRoute = (nodes: NavNode[]): string | undefined => { for (const node of nodes) { if (node.kind === "page") return node.route; @@ -69,8 +78,13 @@ const firstRoute = (nodes: NavNode[]): string | undefined => { }; const hrefFor = (host: HostId): string => { - const match = /^\/(guides|reference)\/(pi|opencode)(\/.*)?$/.exec(currentRoute); - if (match) return `/${match[1]}/${host}${match[3] ?? ""}`; + // DeepSeek ships reference only: a same-page swap inside guides/ has no + // target, so the swap applies only when the built tree serves the route. + const match = /^\/(guides|reference)\/(pi|opencode|deepseek)(\/.*)?$/.exec(currentRoute); + if (match) { + const candidate = `/${match[1]}/${host}${match[3] ?? ""}`; + if (servedRoutes.has(candidate)) return candidate; + } // A neutral page keeps the current section: switching from `/reference/...` // lands on the other host's reference entry, not its guides entry. @@ -78,6 +92,9 @@ const hrefFor = (host: HostId): string => { const first = group?.kind === "group" ? firstRoute(group.children) : undefined; if (first) return first; + const entry = HOSTS.find((candidate) => candidate.id === host)?.entry; + if (entry) return entry; + return `/guides/${host}/getting-started`; }; @@ -123,21 +140,7 @@ const currentHref = (host: HostId): string => )) } - -
- - - {nextHost.label} - {nextHost.description} - - - NEXT -
- ) diff --git a/apps/docs/components/mdx/HarnessCards.astro b/apps/docs/components/mdx/HarnessCards.astro index 8f23db5..449fe7f 100644 --- a/apps/docs/components/mdx/HarnessCards.astro +++ b/apps/docs/components/mdx/HarnessCards.astro @@ -2,7 +2,7 @@ import Card from "blume/components/content/Card.astro"; import CardGroup from "blume/components/content/CardGroup.astro"; -import { OPENCODE_ICON, PI_ICON } from "../../harness-icons.ts"; +import { DEEPSEEK_ICON, OPENCODE_ICON, PI_ICON } from "../../harness-icons.ts"; interface Props { variant: "guides" | "comparison"; @@ -43,6 +43,13 @@ const cards = cta: "View comparison", body: "Context engines and memory services in two matrices, because they solve different problems.", }, + { + title: "DeepSeek", + href: "/reference/deepseek/comparison", + icon: DEEPSEEK_ICON, + cta: "View comparison", + body: "The basic compaction backend on one matrix, with where it leads and the gaps this package keeps.", + }, ]; --- diff --git a/apps/docs/components/seo/FaqSection.astro b/apps/docs/components/seo/FaqSection.astro index 4a6f368..08dadff 100644 --- a/apps/docs/components/seo/FaqSection.astro +++ b/apps/docs/components/seo/FaqSection.astro @@ -2,16 +2,24 @@ import Accordion from "blume/components/content/Accordion.astro"; import AccordionItem from "blume/components/content/AccordionItem.astro"; import { contentHref } from "blume/components/content/base-href.ts"; +import { DEEPSEEK_FAQS } from "./deepseek-faqs.ts"; import { LCM_FAQS } from "./lcm-faqs.ts"; import { OPENCODE_FAQS } from "./opencode-faqs.ts"; import { PI_FAQS } from "./pi-faqs.ts"; interface Props { - host?: "pi" | "opencode"; + host?: "pi" | "opencode" | "deepseek"; } const { host } = Astro.props; -const faqs = host === "pi" ? PI_FAQS : host === "opencode" ? OPENCODE_FAQS : LCM_FAQS; +const faqs = + host === "pi" + ? PI_FAQS + : host === "opencode" + ? OPENCODE_FAQS + : host === "deepseek" + ? DEEPSEEK_FAQS + : LCM_FAQS; const toAnswerHtml = (text: string): string => text diff --git a/apps/docs/components/seo/deepseek-faqs.ts b/apps/docs/components/seo/deepseek-faqs.ts new file mode 100644 index 0000000..332bb92 --- /dev/null +++ b/apps/docs/components/seo/deepseek-faqs.ts @@ -0,0 +1,49 @@ +import { SHARED_FAQS } from "./lcm-faqs.ts"; +import type { LcmFaq } from "./lcm-faqs.ts"; + +/** Questions that are specific to the DeepSeek bundle. */ +export const DEEPSEEK_FAQS: readonly LcmFaq[] = [ + { + question: "Does it modify my DeepSeek session record?", + answer: + "No. The harness session record stays the source of truth; the adapter mirrors into a per-session SQLite sidecar under the configured `sidecarDir` (default `$DSH_HOME/lossless`, else `~/.dsh/lossless`, else `./.lossless`). Deleting a sidecar does not edit the session, but it does forget that session's recall.", + }, + SHARED_FAQS.lossy, + { + question: "Do I have to configure anything?", + answer: + "No. Installing the bundle switches the backend over with no further edits, and the defaults (pressure at 80% of the priced window, keep 16% verbatim, summary and leaf budgets of 1500 and 3000 tokens) work with no further options. Configuration is the `deepseek-lossless` row in the profile patch; there is no separate config file. The [configuration guide](/guides/deepseek/configuration) lists every key.", + }, + { + question: "Where does the data live?", + answer: + "One SQLite sidecar per session under `sidecarDir` (default `$DSH_HOME/lossless`, else `~/.dsh/lossless`, else `./.lossless`). The directory is 0700; the database, WAL, and shared-memory files are 0600. Metrics sit beside the stores. Nothing leaves the machine except the provider calls the summarizer and reader make.", + }, + { + question: "Does it replace DeepSeek compaction?", + answer: + "It mounts as the profile compaction backend once the base `compaction-basic` row is disabled and the bundle row is enabled. Aged turns become summary pointers the model can navigate, and the engine returns the same frontier as the host compaction checkpoint.", + }, + { + question: "Can the model read a past DeepSeek session?", + answer: + 'Yes. `lcm_grep` scans the global sidecar directory, so `"sessions"` and `"all_sessions"` are equivalent today. Hits are pointers; nothing from a past session is injected automatically.', + }, + { + question: "How do I use it on the web?", + answer: + "Install both packages into the web profile, restart, and select a Lossless preset for the session. The host switch alone does not reach preset sessions, which keep the shipped backend. To switch back, select a shipped preset again.", + }, + SHARED_FAQS.caching, + { + question: "What commands and tools does it add?", + answer: + "The `/lcm` verbs `status`, `report`, `dump`, `backup`, `export`, `import`, `gc`, `migrate`, `heal`, `diagnose`, and `compact`. Tools: `lcm_grep`, `lcm_describe`, `lcm_expand`, and `lcm_expand_query`. The [command reference](/reference/deepseek/command) and [tool reference](/reference/deepseek/tools) document each one.", + }, + SHARED_FAQS.secrets, + { + question: "How do I remove it?", + answer: + "Run `dsh plugin --profile remove deepseek-lossless` and restart to restore the base backend, then delete the configured `sidecarDir`. To switch back without uninstalling, re-enable `compaction-basic` and disable `deepseek-lossless` from the Plugins page or the profile patch. On preset-bearing profiles, also remove `deepseek-lossless-web` or select a shipped preset again. `/lcm backup` copies a sidecar, `/lcm export` writes a JSONL transcript, and `/lcm gc --apply` exports and deletes retired sidecars.", + }, +]; diff --git a/apps/docs/content/_snippets/reference/command/deepseek.mdx b/apps/docs/content/_snippets/reference/command/deepseek.mdx new file mode 100644 index 0000000..e12cb36 --- /dev/null +++ b/apps/docs/content/_snippets/reference/command/deepseek.mdx @@ -0,0 +1,120 @@ +```text +/lcm status +/lcm report +/lcm dump [query] [--include-removed] [--save ] +/lcm backup [file] +/lcm export [file] +/lcm import +/lcm gc [--apply] [--session ] [--older-than ] +/lcm migrate [--apply] +/lcm heal [--apply] [--max-nodes N] +/lcm diagnose [--save ] +/lcm compact +``` + +Eleven verbs on one `/lcm` command. + +## Settings + +There is no settings panel. Edit the capability `config` in the composition and restart it. [Config](/reference/deepseek/config) lists every key. + +## status + +```text +/lcm status +``` + +Message and compaction-summary counts for the live sidecars, plus the meter window with thresholds and the last pass outcome for this session. + +## report + +```text +/lcm report +``` + +Pass, checkpoint, backfill, recall, and retrieval rows for this session. With no rows, it reports that the session has none. + +## dump + +```text +/lcm dump [query] +/lcm dump [query] --include-removed +/lcm dump --save # or --save= +``` + +Read-only sidecar dump: store facts, history, summaries, message samples, and this session's metrics, redacted. Free-text args sample one FTS5 query; `--include-removed` keeps removed rows; `--save` writes the dump at mode 0600 instead of printing it. + +## backup + +```text +/lcm backup [file] +``` + +Copies this session's sidecar to `.backup-`, or to a named path, at mode 0600. + +## export + +```text +/lcm export [file] +``` + +Writes the verbatim transcript and summaries as JSONL at mode 0600. The default lands beside the sidecars. + +## import + +```text +/lcm import +``` + +Restores rows from a file written by `/lcm export`. Re-importing the same file inserts nothing and reports the skips. Refused without a file argument, or into a store that already holds foreign messages. + +## gc + +```text +/lcm gc # dry-run list +/lcm gc --apply # export each transcript, then delete +/lcm gc --session # narrow to one sidecar +``` + +Collects sidecars older than 30 days that no live session holds (`--older-than` sets the window). It never touches a live session's sidecar and exports each transcript before deleting it. A no-op apply writes no export file. + +## migrate + +```text +/lcm migrate # dry-run list of older-schema sidecars +/lcm migrate --apply # upgrade them in place +``` + +Generation-gated: sidecars older than the current schema version are listed, then upgraded in place. Sidecars of live sessions are excluded. + +## heal + +```text +/lcm heal # dry-run: count the tainted nodes and their ancestors +/lcm heal --apply # re-summarize them from their stored source +/lcm heal --apply --max-nodes 8 +``` + +A non-clean summary is a candidate. Heal reads the exact source rows that node points at, re-summarizes the tainted node, then recomputes every ancestor whose child changed. `--max-nodes` bounds one run; `summarizerTokensPerHeal` bounds its spend. The dry run makes no provider call. Refused while a compaction pass is in flight, or while a repair already runs for the session. + +## diagnose + +```text +/lcm diagnose +/lcm diagnose --save # or --save= +``` + +One redacted, pasteable block: versions, store facts, the effective options, and this session's metrics. `--save` writes the exact bytes at mode 0600 instead of printing them. + +## compact + +```text +/lcm compact +``` + +Runs one manual compaction immediately. Returns null with "No compactable history yet" when no safe useful range exists. Automatic pressure stays on the harness seam unless `bracketAuto` is set. + +## Related + +- [Config](/reference/deepseek/config): the keys and thresholds behind these verbs. +- [Files](/reference/deepseek/files): where backup, export, and gc write. diff --git a/apps/docs/content/_snippets/reference/config/deepseek.mdx b/apps/docs/content/_snippets/reference/config/deepseek.mdx new file mode 100644 index 0000000..1913377 --- /dev/null +++ b/apps/docs/content/_snippets/reference/config/deepseek.mdx @@ -0,0 +1,41 @@ +The config is the capability's `config` object in the composition; there is no separate config file. `enabled` is opt-in: anything but exactly `true` registers nothing. + +## Keys + +| Key | Type | Default | Rides | Effect | +| -------------------------- | ------------------ | --------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `enabled` | boolean | unset | mount | Master flag. Anything but exactly `true` returns a no-op. | +| `auto` | boolean | `false` | seam | Automatic step-boundary pressure and overflow-recovery listeners. | +| `bracketAuto` | boolean | `false` | adapter deps | Route automatic pressure compaction through the DAG bracket instead of the seam. | +| `sidecarDir` | string | `$DSH_HOME/lossless`, else `~/.dsh/lossless`, else `./.lossless` | adapter | One SQLite file per session; created when missing. An absolute `DSH_HOME` wins (source `default:DSH_HOME`), else the home dir (source `default:home`), else `./.lossless` (source `fallback`). An explicit value overrides the default (source `explicit`); a blank value falls back to the default. | +| `summaryTokens` | number | `1500` | adapter deps | Tokens one summary aims for. | +| `leafChunkTokens` | number | `3000` | adapter deps | Tokens one leaf chunk may hold before it splits. | +| `summarizationProvider` | string | unset | seam | Summary provider; unset follows the routed model. | +| `summarizationModel` | string | unset | seam | Summary model; unset follows the routed model. | +| `thresholdRatio` | number | `0.8` | seam | Window fraction for pressure; capped at the window minus reserved output and headroom. | +| `retainTokens` | number | unset | seam | Recent context retained as an absolute token budget. Mutually exclusive with `retainRatio`. | +| `retainRatio` | number | `0.16` | seam | Recent context retained as a fraction of the priced window. Mutually exclusive with `retainTokens`. | +| `headroomTokens` | number | `65536` | seam | Pressure headroom beyond the routed output reservation. | +| `modelPolicies` | object[] | unset | seam | Exact provider/model overrides, merged over the defaults per routed target. Duplicate targets fail load. | +| `summarizerTokensPerCall` | number | `60000` | adapter-only | Ceiling on one summarizer request, refused before the provider is called. | +| `summarizerTokensPerPass` | number | `250000` | adapter-only | Ceiling on one pass across its calls. | +| `summarizerTokensPerHeal` | number | `120000` | adapter-only | Ceiling on one `/lcm heal --apply` repair run. | +| `healSourceTokens` | number | derived | adapter-only | Source tokens one heal map window may read. Unset derives `8 x summaryTokens`, clamped to `3000` through `24000`. | +| `mechanicalTools` | string[] | `[]` | adapter-only | Tool names whose spans compact without a model call. Empty keeps the LLM path. | + +The scope column names where each key takes effect. Seam keys reach the harness config, where unknown keys fail load. The rest take effect locally; mount gates the whole composition. The manual `/lcm compact` path ignores retain settings. + +The summarizer follows the routed provider and model unless both summarizer fields are set. The reader calls the calling agent's model: the latest routed request first, agent options second. `reasoning` stays false on every adapter model call, so a reasoning model never silently spends the output cap. + +## Parse rules + +- `enabled` mounts the mirror, the tools, the engine, and the command only when it is exactly `true`. +- `auto` and `bracketAuto` stay off unless exactly `true`. +- `retainTokens` and `retainRatio` are mutually exclusive; the seam's default shape is the ratio form. +- `modelPolicies` entries match an exact provider plus model route; duplicates fail plugin load. +- Unknown keys on the seam config are rejected by the seam resolver; adapter-only keys never reach it. + +## Related + +- [Compaction zones](/concepts/compaction-zones): the resolved lines in context. +- [Commands](/reference/deepseek/command): the verbs that read and repair these values. diff --git a/apps/docs/content/_snippets/reference/files/deepseek.mdx b/apps/docs/content/_snippets/reference/files/deepseek.mdx new file mode 100644 index 0000000..953c6c5 --- /dev/null +++ b/apps/docs/content/_snippets/reference/files/deepseek.mdx @@ -0,0 +1,73 @@ +The adapter writes under `sidecarDir`. Named export and backup paths are the exception. + +## Store + + + +- sidecarDir/ + - ses_0123456789abcdef.sqlite + - ses_0123456789abcdef.sqlite-wal + - ses_0123456789abcdef.sqlite-shm + - ses_0123456789abcdef.sqlite.backup-2026-10-25T12-00-00 + - metrics.jsonl + - metrics.jsonl.1730000000000 + - lcm-export-2026-10-25T12-00-00.jsonl + - lcm-export-gc-ses_0123456789abcdef-2026-10-25T12-00-00.jsonl + + + +One sidecar per session holds the verbatim mirror, the DAG, and the checkpoint marker. The `.sqlite-wal` and `.sqlite-shm` files belong to it. Disk stems are sanitized session ids. The directory is created 0700; the database, WAL, and shared-memory files are 0600. + +The live session log remains the source of truth for the conversation. Deleting a sidecar forgets that session's recall until remount: a session still live at mount backfills from the session history source when the query backend is available, within the paging and cap below. `/lcm export` keeps a portable copy; `/lcm backup` keeps a byte copy. + +Sessions already live when the plugin mounts backfill through the same mirror path as the live feed, paged at 50 with a 500 inserted-row per-session cap; re-mirroring inserts nothing. Each backfill writes one `backfill` metrics row with its ingested count and span. + +## Metrics + +`metrics.jsonl` holds one row per event, tagged with the session. Groups: + +| Group | Kinds | +| ----------- | ------------------------------------------ | +| Pass | `pass` | +| Checkpoint | `compaction-checkpoint` | +| Backfill | `backfill` | +| Recall | `recall` | +| Reader | `retrieval` | + +One `pass` row lands per ladder run with its outcome, leaf and condensed counts, and token usage; one `compaction-checkpoint` row lands per DAG bracket commit with condensed plus leaf seqs. `/lcm report` is a view over these rows. The log rotates at 5 MB into a millisecond-timestamped sibling, keeps three generations, and is created 0600. + +## Exports, backups, and gc + +| Path | Written by | Contents | +| --------------------------------------------------- | --------------------- | ----------------------------------------------------- | +| `.backup-` beside the sidecar | `/lcm backup` | A byte copy of the sidecar. Never deleted by LCM. | +| `lcm-export-.jsonl` | `/lcm export` | Every message and summary, verbatim, mode 0600. | +| `lcm-export-gc--.jsonl` | `/lcm gc --apply` | The transcripts of every sidecar it is about to delete. | + +`/lcm import ` restores missing rows and skips present ones. It runs on its named file, so a fresh session can restore history it never mirrored. + +## Diagnostics + +```text +/lcm diagnose # rendered, not written +/lcm diagnose --save # exact bytes, mode 0600 +``` + +The block carries versions, store facts, the effective options, and this session's metrics. [Commands](/reference/deepseek/command#diagnose) lists the flags. + +## Footprint + +One sidecar per session, so the directory grows with use. `/lcm gc` removes the ones older than 30 days that no live session holds; a deleted sidecar stays forgotten unless its session is still live at remount with history available, in which case backfill restores transcript rows within the cap. + +## Config + +The capability options live in the composition config, not under `sidecarDir`. [Config](/reference/deepseek/config) lists every key. + + + Stores, backups, and exports hold the session verbatim. Treat them like the session file. + + +## Related + +- [Security](/concepts/security): modes, contents, and what each surface exposes. +- [Commands](/reference/deepseek/command): the subcommands that read and write these files. diff --git a/apps/docs/content/_snippets/reference/tools/deepseek.mdx b/apps/docs/content/_snippets/reference/tools/deepseek.mdx new file mode 100644 index 0000000..55fa228 --- /dev/null +++ b/apps/docs/content/_snippets/reference/tools/deepseek.mdx @@ -0,0 +1,115 @@ +Four tools reach stored history. `lcm_expand_query` runs in process over one streaming call per step; there is no child session on this harness. + +| Question | Tool | +| --------------------------------------------------------- | ------------------ | +| "Where did we talk about X?" | `lcm_grep` | +| "What is in summary #12, and what would reading it cost?" | `lcm_describe` | +| "Read the originals." | `lcm_expand` | +| "Read the originals and answer from them." | `lcm_expand_query` | + +## lcm_grep + +Searches stored history, including content already compacted out of the visible context. + +| Parameter | Type | Default | Meaning | +| ----------------- | ------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------- | +| `query` | string | required* | FTS5 search: words, quoted phrases, `AND`/`OR`, prefix `*`. Matches the first 100,000 characters of a message. | +| `pattern` | string | none | JavaScript regular expression over the full text of every stored message. Give one of `query` or `pattern`, not both. | +| `case_sensitive` | boolean | `false` | Match a `pattern` with its own case. | +| `offset` | integer | `0` | Hits to skip, for the next page. | +| `after` | string | none | Continue a pattern scan from this entry id, which is the `after` value a deadline stop names. | +| `limit` | integer | `20` | Hits one page holds. | +| `summary_id` | integer | none | Restrict hits to the messages this summary covers. | +| `include_removed` | boolean | `false` | Include matches from a branch that left the active path. | +| `scope` | `"session" \| "sessions" \| "all_sessions"` | `"session"` | Which sidecar to read. The last two are equivalent because the sidecar directory is global. | + +\* One of `query` or `pattern` is required in session scope; past-session scopes take `query` only. A pattern scan stays session-scoped. + +Bounds, each named in the output that hits it: + +| Bound | Value | +| ----------------------------------- | ------------------------------------ | +| Indexed characters per message | 100,000 | +| One hit line | 400 characters, whitespace collapsed | +| Cross-session sidecars scanned | 20 most recent, newest first | +| Cross-session hits per store | 10 | +| Cross-session search deadline | 1.5 seconds | +| Cross-session hits | 40 | + +Every hit names the entry id that `lcm_expand` reads in full, and the summary (`#N`) that covers it. Past-session hits carry an `[lcm:session ]` label; read one with `lcm_expand({session, entry_id})`. + +## lcm_describe + +```text +lcm_describe({ id: 12 }) +``` + +`id` is the `N` in an `[lcm:summary #N ...]` header. Returns a node without expanding it: + +- kind, depth, and token count +- the covered span (`e104 .. e238`) +- costs: the source text behind it, in tokens and messages, and the whole subtree +- tiers, when the node keeps a thorough text beside its terse one +- the files its children cover, each with the first line of its exploration summary +- children and parents, with a per-child cost line + +The thorough tier is what a describe returns when both exist, and both sizes are named. This reads the calling session only. An id the store does not hold is refused. + +## lcm_expand + +Reads one stored message, or lists a summary's messages, without a model call. + +| Parameter | Type | Default | Meaning | +| ------------ | --------------------- | ------- | ----------------------------------------------------------------------- | +| `id` | integer | none | List this summary's messages, or check that an entry is covered. | +| `entry_id` | string | none | Read this one message. | +| `session` | string | none | Read an entry from this past session's sidecar. | +| `offset` | integer `>= 0` | `0` | Message index to start listing from. | +| `limit` | integer `1..50` | `25` | Messages to list. | +| `char_offset`| integer `>= 0` | `0` | Character offset inside `entry_id`. | +| `max_chars` | integer `200..20000` | `20000` | Characters to read from `entry_id`. | + +Pass `entry_id` to read one message, or `id` to list a summary. `session` names a past session id from an `[lcm:session ]` hit; that sidecar opens read-only and the text is labelled as a past session's history. A reader-loop call with a past `session` is refused: the reader is session-scoped. + +Bounds: + +| Bound | Value | +| -------------- | -------------------- | +| One read | 20,000 characters | +| Messages listed| 50 per call | + +## lcm_expand_query + +Reads stored originals in process and answers from them. + +| Parameter | Type | Default | Meaning | +| ------------ | --------------------- | -------- | ---------------------------------------------------------------------------------------------------- | +| `query` | string | required | What to search for in the stored history. | +| `prompt` | string | none | What to find out, and the form the answer takes. Ask for a verbatim quote when the exact words matter. | +| `summary_id` | integer | none | Read this node's originals, from an `[lcm:summary #N ...]` header. | +| `entry_id` | string | none | Read one stored message, from a grep hit. | +| `session` | string | none | Read another session kept in this process instead of the calling one; names a session id, never a file. | +| `max_tokens` | integer `500..40000` | `10000` | Retrieved content budget, in estimated tokens. | + +Pass `summary_id` or `entry_id`, not both. With neither, the reader searches. Findings come back wrapped in ``; the stored text does not enter this conversation. A past session's answer is labelled with its session id. + +The reader is one streaming call per step, at most 6 steps, over `lcm_grep`, `lcm_describe`, and `lcm_expand`; it cannot recurse. Without a routed provider and model, the run refuses before reading anything. + +## Failure modes + +| Result | Cause | +| ------------------------------------------------------------------- | ------------------------------------------------------------------ | +| `No stored history for this session yet` | Nothing has been ingested, or the call arrived outside any agent. | +| `No summary with id N` | The address is not in this store. | +| `Message eNNN is not covered by summary #N` | The entry and summary were mismatched. | +| `Nothing was read: the reader is session-scoped` | The reader tried a past-session scope. | +| `Nothing was read: no stored session with id ...` | The named past session has no sidecar. | +| `LCM: past sessions are not reachable in this process` | No sidecar directory is wired. | +| `LCM: a pattern scan searches this session` | A `pattern` was passed with a past-session scope. | +| `lossless retrieval: no provider/model available for the reader` | No route resolves for the reader's model call. | + +## Related + +- [Recall and provenance](/concepts/recall-and-provenance): what the headers and addresses mean. +- [Limits](/concepts/limits): the caps in one place. +- [Commands](/reference/deepseek/command): `/lcm compact` and the verbs that read these rows. diff --git a/apps/docs/content/guides/deepseek/configuration.mdx b/apps/docs/content/guides/deepseek/configuration.mdx new file mode 100644 index 0000000..83189e1 --- /dev/null +++ b/apps/docs/content/guides/deepseek/configuration.mdx @@ -0,0 +1,58 @@ +--- +title: "Configuration for DeepSeek" +description: "DeepSeek: set options in the profile patch: the enable flag, the compaction thresholds, the summarizer budgets, and what stays adapter-only." +search: + tags: ["guide", "config"] + boost: 3 + keywords: [config, cordis.patch.yml, thresholds, options] +related: + - /concepts/compaction-zones + - /reference/deepseek/config + - /guides/deepseek/troubleshooting +sidebar: + label: "Configuration" +--- + +There is no separate config file. Options live in the profile patch row for `deepseek-lossless` at `~/.dsh/profiles//cordis.patch.yml`. A patch replaces the whole row config, so restating only the keys you change is enough: the rest fall back to the bundled schema defaults, which the Plugins page validates. The shipped profile row mounts enabled, so a fresh install needs no edits; the schema default stays off as the code-level safe default for direct composition callers. + +```yaml +- id: deepseek-lossless + config: + enabled: true + auto: false + sidecarDir: ./.lossless + summaryTokens: 1500 +``` + +Preset sessions read the same keys from the preset's own backend row, which +ships enabled. Keep `sidecarDir` identical in both rows: the realm backend +borrows the host core and fails fast when the directories differ. The rows +are configured independently, so keep shared policy knobs (thresholds, +budgets, summarizer chain) aligned on both. Enable `auto` in at most one of +the two rows so pressure compaction has a single driver per session. + +If you customized a shipped preset, your edits do not transfer to the +Lossless twin, which starts from stock content. Either redo them on the +Lossless preset through the editor, or bring lossless to your preset +instead: open your customized preset, replace its compaction backend row +with this block, and keep the host row enabled with a matching +`sidecarDir`: + +```yaml +- id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true +``` + +A fully custom preset of your own keeps the base backend otherwise, since +no later layer can address rows inside it. +Lossless presets start from the stock shipped presets: customizations you +made to a shipped preset do not transfer, so redo them on the Lossless +preset through the editor, which addresses our rows the same way. A fully +custom preset of your own keeps the base backend, since no later layer can +address rows inside it. + +The [config reference](/reference/deepseek/config#keys) lists every key with its type, default, and effect. `enabled` is the master flag and only exactly `true` mounts anything. `retainTokens` and `retainRatio` are mutually exclusive. `modelPolicies` entries match an exact provider plus model route and duplicate targets fail load. Seam keys ride the harness config where unknown keys fail load; adapter-only keys (`summarizerTokensPerCall`, `summarizerTokensPerPass`, `summarizerTokensPerHeal`, `healSourceTokens`, `mechanicalTools`) never reach the seam. + +The summarizer follows the routed provider and model unless both summarizer fields are set. `/lcm status` reports the effective values, so a bad value is visible before it costs anything. diff --git a/apps/docs/content/guides/deepseek/cross-session-recall.mdx b/apps/docs/content/guides/deepseek/cross-session-recall.mdx new file mode 100644 index 0000000..be50867 --- /dev/null +++ b/apps/docs/content/guides/deepseek/cross-session-recall.mdx @@ -0,0 +1,18 @@ +--- +title: "Cross-session recall for DeepSeek" +description: "DeepSeek: how `lcm_grep` reaches past sidecars, which scopes match what, and why a hit stays a pointer until the model expands it." +search: + tags: ["guide", "recall"] +related: + - /concepts/recall-and-provenance + - /concepts/security + - /reference/deepseek/tools +sidebar: + label: "Cross-session recall" +--- + +Recall reads the local sidecars, never the live session. `lcm_grep` searches past sessions, up to 10 hits per past store within 40 hits across 20 stores and a 1.5 s deadline. Each hit is a pointer: session id, entry id, and an excerpt. The model then calls `lcm_describe` for the lineage around a hit or `lcm_expand` to read the full verbatim event. `lcm_expand_query` runs a bounded in-process reader over the same store for follow-up questions. + +Scopes stay session aware. A bare query stays in the calling session sidecar. Cross-session scopes search the same global sidecar directory the composition was mounted with, so `sessions` and `all_sessions` match the same set today. Past sessions open read-only; the mount never writes to them. + +The [tools reference](/reference/deepseek/tools) lists every tool with its parameters and output shape. Only observed events flow through the mirror path with dedup, and unmatched queries fail closed and resolve to misses. diff --git a/apps/docs/content/guides/deepseek/faq.mdx b/apps/docs/content/guides/deepseek/faq.mdx new file mode 100644 index 0000000..96ec74f --- /dev/null +++ b/apps/docs/content/guides/deepseek/faq.mdx @@ -0,0 +1,17 @@ +--- +title: DeepSeek FAQ +description: "Short answers about the DeepSeek bundle: the sidecar, the store, secrets, cost, cross-session recall, and uninstalling." +sidebar: + label: "FAQ" +search: + tags: ["guide", "faq", "deepseek"] +--- + + + +## Related + +- [Security](/concepts/security): the store, redaction, and the surfaces that carry stored bytes. +- [Limits](/concepts/limits): what LCM does not do. +- [Configuration](/guides/deepseek/configuration): the keys the answers above refer to. +- [Getting started](/guides/deepseek/getting-started): install and enable per profile. diff --git a/apps/docs/content/guides/deepseek/getting-started.mdx b/apps/docs/content/guides/deepseek/getting-started.mdx new file mode 100644 index 0000000..6ab6efc --- /dev/null +++ b/apps/docs/content/guides/deepseek/getting-started.mdx @@ -0,0 +1,54 @@ +--- +title: "Getting started for DeepSeek" +description: "DeepSeek: install `deepseek-lossless` into each profile you boot, switch the compaction backend over, and verify with `/lcm status`." +search: + tags: ["guide", "install"] + boost: 5 + keywords: [install, setup, quickstart, deepseek, dsh, profile] +sidebar: + label: "Getting started" +--- + +Install into each profile you boot, starting with the one you run. Host-plane +profiles (`headless`, `acp`, `sdk`, and custom profiles built from the base +bundle) need only the host package: + +```sh +dsh plugin --profile headless add deepseek-lossless +``` + +Preset-bearing profiles (`web`, `desktop`, or a custom profile built on +`web-app`) need the preset companion too. It ships Lossless variants of all +four shipped presets whose compaction backend runs the shared core: + +```sh +dsh plugin --profile web add deepseek-lossless +dsh plugin --profile web add deepseek-lossless-web +``` + +For a custom profile, replace the name. Without a shipped template, the +profile starts from the base bundle. The install resolves through the npm +registry, so the version you publish is the version they get. Each install +appends its package to `dsh.profile.bundles` automatically because both +declare `dsh.bundle`; without that manifest a package would warn and install +as a plain dependency with no profile layer. + +The install switches the backend over with no further edits: the shipped +patch quiets the base row and mounts our row enabled, matching the manager's +rule that installation enables a new bundle. Restart the profile, then +verify with `/lcm status` in a booted session. It reports per-session +message and summary counts, the meter window, the resolved swap and re-cut +thresholds, and the last pass outcome. Then run `/lcm compact` once on an +idle session to prove the manual path: aged turns become summary pointers +and every verbatim message stays in the local SQLite sidecar. + +On a preset-bearing profile, install both packages, restart, then select a +Lossless preset for the session instead of a shipped one. The preset +backend borrows the shared core and fails fast naming the host row when it +is absent, instead of running a split-brain copy. To switch back, select a +shipped preset again; the base backend returns for those sessions. To switch +a host-plane profile back, re-enable `compaction-basic` and disable +`deepseek-lossless` from the Plugins page or the profile patch; removing +the bundle likewise restores the base row. + +Reference holds the full surface: [command](/reference/deepseek/command), [tools](/reference/deepseek/tools), [config](/reference/deepseek/config), and [files](/reference/deepseek/files). Common keys live under the plugin config: `auto` for pressure compaction inside the turn, `sidecarDir` for one SQLite file per session, `summaryTokens` and `leafChunkTokens` for summary budgets, and `mechanicalTools` for spans that compact without a model call. The Plugins page validates them against the bundled schema. diff --git a/apps/docs/content/guides/deepseek/meta.ts b/apps/docs/content/guides/deepseek/meta.ts new file mode 100644 index 0000000..2c4b4c8 --- /dev/null +++ b/apps/docs/content/guides/deepseek/meta.ts @@ -0,0 +1,9 @@ +import { defineMeta } from "blume"; + +import { DEEPSEEK_ICON } from "../../../harness-icons.ts"; + +export default defineMeta({ + title: "DeepSeek", + icon: DEEPSEEK_ICON, + pages: ["getting-started", "configuration", "cross-session-recall", "troubleshooting", "faq"], +}); diff --git a/apps/docs/content/guides/deepseek/troubleshooting.mdx b/apps/docs/content/guides/deepseek/troubleshooting.mdx new file mode 100644 index 0000000..84c52c8 --- /dev/null +++ b/apps/docs/content/guides/deepseek/troubleshooting.mdx @@ -0,0 +1,18 @@ +--- +title: "Troubleshooting for DeepSeek" +description: "DeepSeek: what to run when compactions do not happen, when the backend stays inert, and how to file a useful report." +search: + tags: ["guide", "troubleshooting"] +related: + - /reference/deepseek/files + - /concepts/limits + - /reference/deepseek/command +sidebar: + label: "Troubleshooting" +--- + +Start with `/lcm status`. It reports per-session message and summary counts, the meter window, the resolved swap and re-cut thresholds with a `(clamped)` marker when the clamp pulled a line back, and the last pass outcome. When compactions do not happen, check three things in order: the bundle is installed and its row is enabled (it ships enabled; the Plugins page shows otherwise), the base `compaction-basic` row is disabled in the same composition, and the profile was restarted after any hand edit. On a preset-bearing profile, also confirm the session uses a Lossless preset: the host switch alone does not reach preset sessions, which keep the shipped backend. With the row disabled the bundle mounts no `/lcm` surface at all, so a missing `/lcm` command means the bundle is still inert; the harness `/compact` path follows whichever compaction service the profile mounts. If you run a third-party compaction backend of your own, ours conflicts with it at boot: disable the `deepseek-lossless` row and keep yours. + +When the window gauge disagrees with the harness, compare the thresholds section of `/lcm status` against the routed model window. `thresholdRatio`, `retainRatio`, and `headroomTokens` ride the seam, and the clamp pulls an unusable line back with a `(clamped)` marker. + +To file a useful report, run `/lcm diagnose --save `. It builds a redacted bundle without secrets. Include the profile name, the thresholds section from `/lcm status`, and the files list from the [files reference](/reference/deepseek/files). Limits live under [limits](/concepts/limits): what LCM does not do. diff --git a/apps/docs/content/guides/meta.ts b/apps/docs/content/guides/meta.ts index 741bbc6..9774a58 100644 --- a/apps/docs/content/guides/meta.ts +++ b/apps/docs/content/guides/meta.ts @@ -3,5 +3,5 @@ import { defineMeta } from "blume"; export default defineMeta({ title: "Guides", icon: "book-open", - pages: ["pi", "opencode"], + pages: ["pi", "opencode", "deepseek"], }); diff --git a/apps/docs/content/reference/comparison.mdx b/apps/docs/content/reference/comparison.mdx index 43c3ce1..caf02a0 100644 --- a/apps/docs/content/reference/comparison.mdx +++ b/apps/docs/content/reference/comparison.mdx @@ -1,6 +1,6 @@ --- title: Comparison -description: How the Pi and OpenCode v2 adapters compare with the engines and memory services their users choose between, and why the comparisons are separate. +description: How the Pi, OpenCode v2, and DeepSeek adapters compare with the engines and memory services their users choose between, and why the comparisons are separate. sidebar: hidden: true mode: "center" @@ -9,12 +9,12 @@ pagination: false search: tags: ["reference", "comparison"] boost: 3 - keywords: [comparison, peers, pi, opencode] + keywords: [comparison, peers, pi, opencode, deepseek] --- The engine is shared. The peer sets are not. -Pi sessions pick among compaction engines. OpenCode v2 sessions pick among context engines, memory services, or both, so each adapter has its own page. +Pi sessions pick among compaction engines. OpenCode v2 sessions pick among context engines, memory services, or both, so each adapter has its own page. DeepSeek sessions pick between the harness basic backend and this package on one matrix. @@ -23,10 +23,11 @@ Pi sessions pick among compaction engines. OpenCode v2 sessions pick among conte - [Pi](/reference/pi/comparison): four Pi compaction engines on one matrix, with where each one leads and the gaps this package keeps. - [OpenCode v2](/reference/opencode/comparison): context engines and memory services in two matrices, because they solve different problems. +- [DeepSeek](/reference/deepseek/comparison): the basic compaction backend on one matrix, with where it leads and the gaps this package keeps. ## What a cell means Each cell comes from the peer's source at the commit named in that page's verification note, not from a README. The date on the page covers the peer cells and this package's own. Each page names where a peer leads and which gaps this package keeps. -The audited sources are [comparison/pi.mdx](https://github.com/stainless-code/lossless/blob/main/apps/docs/content/reference/pi/comparison.mdx) and [comparison/opencode.mdx](https://github.com/stainless-code/lossless/blob/main/apps/docs/content/reference/opencode/comparison.mdx). The page and the source are the same file. +The audited sources are [comparison/pi.mdx](https://github.com/stainless-code/lossless/blob/main/apps/docs/content/reference/pi/comparison.mdx), [comparison/opencode.mdx](https://github.com/stainless-code/lossless/blob/main/apps/docs/content/reference/opencode/comparison.mdx), and [comparison/deepseek.mdx](https://github.com/stainless-code/lossless/blob/main/apps/docs/content/reference/deepseek/comparison.mdx). The page and the source are the same file. diff --git a/apps/docs/content/reference/deepseek/command.mdx b/apps/docs/content/reference/deepseek/command.mdx new file mode 100644 index 0000000..56d28cb --- /dev/null +++ b/apps/docs/content/reference/deepseek/command.mdx @@ -0,0 +1,13 @@ +--- +title: "Commands for DeepSeek" +description: "DeepSeek: every `/lcm` verb, with its flags, what it prints, and the limits of the host command API." +search: + tags: ["reference", "command"] + keywords: [commands, lcm, lcm-compact] +sidebar: + label: "Commands" +--- + +/_snippets/reference/command/intro.mdx + +/_snippets/reference/command/deepseek.mdx diff --git a/apps/docs/content/reference/deepseek/comparison.mdx b/apps/docs/content/reference/deepseek/comparison.mdx new file mode 100644 index 0000000..cc77855 --- /dev/null +++ b/apps/docs/content/reference/deepseek/comparison.mdx @@ -0,0 +1,119 @@ +--- +title: DeepSeek comparison +description: "deepseek-lossless against the harness native compaction backend: one matrix, where the harness leads, and what is not comparable." +sidebar: + label: "Comparison" +search: + tags: ["reference", "comparison"] +--- + +deepseek-lossless is the DeepSeek harness adapter for a lossless context engine. It +mirrors every session event into a per-session SQLite sidecar and summarizes the aged +span into a DAG. The original bytes stay addressable, so what compaction removed +can be read back. + +**Scope.** This page is DeepSeek only. The peer below is the harness's own replay-aware +basic compaction backend, with the pruner service and the token meter it drives. A +session runs one backend, so the cells compare what each one does with the same window. + +Cells are read from the pinned packages, never from a project's README. This package's +own cells are read from this repository. + +Last verified: 2026-10-25. + +That date covers the peer cells and the audits named in the verification note, +and this package's own cells, measured from this repository (`packages/core` +plus `packages/deepseek`). + +## One matrix on the same axes + +Legend: ✅ full · 🟡 partial · ❌ none. Inventory rows name what ships, not how well it works, and carry no rating. + +| Dimension | deepseek-lossless | harness basic backend | +| ---------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | +| Owns the compaction boundary | 🟡 manual `/lcm compact` commits through the DAG bracket; auto pressure stays on the seam unless `bracketAuto` is set | ✅ `compactIfNeeded`, `compactNow`, and `compactRegion` on one engine per composition, one checkpoint per transaction | +| Model-visible history after a pass | condensed replacement naming `[lcm:summary #N]` refs; leaves stay sidecar only, so the log holds one summary per pass | checkpoint preamble plus a tagged summary block; a prior checkpoint merges instead of re-summarizing | +| Own store of the thread | ✅ every derived text block verbatim in a per-session SQLite sidecar plus FTS5 | ❌ none; reads the live session log | +| Verbatim original reachable | ✅ any stored entry by id, any summary by id, character-windowed reads, cross-session reads | ❌ originals persist as shadowed seqs in the harness append-only log, readable through opt-in session-query tools; no built-in recall path | +| Recall mechanism | ✅ FTS5 `lcm_grep`, `lcm_describe`, `lcm_expand`, and in-process `lcm_expand_query` over stored history | ❌ none; the checkpoint carries goals and decisions forward as prose | +| Recall tools | ✅ `lcm_grep`, `lcm_describe`, `lcm_expand`, `lcm_expand_query` | ❌ none | +| Cross-session scope | ✅ one global sidecar directory; a 20-store, 40-hit, 1.5 s scan with 10 hits per store; reads by session id | ❌ none; continuity is the prior-checkpoint merge within one session | +| Child sessions | ❌ none; the reader is one streaming call per step, at most 6 steps, inside the calling process | n/a; the backend makes no reader calls | +| Prefix-cache behavior | ❌ single-message calls cannot express prefix reuse | ✅ replays the conversation prefix with the instruction last, so the provider's warm prefix cache is reused | +| Model-free reduction | 🟡 opt-in `mechanicalTools` compacts named spans into deterministic `mechanical` leaves with zero model calls; unset keeps the model-backed ladder | ✅ a pruner service trims tool results before pressure and overflow measurement, with no model call | +| Checkpoint structure | `[lcm:summary #N]` refs that resolve in the sidecar; resume rebuilds the last checkpoint from stored rows | ✅ preamble framing, sectioned prompt with a verbatim-quoted goals section, snapshot pricing of the replayed range | +| Pressure measurement | seam meter drives pressure; status shows total against window plus thresholds and the last pass | ✅ per-request uncached, cache-read, and cache-write buckets; pressure includes cache traffic; budgets scale to routed capacity | +| Summary quality control | ✅ leaf plus condensed ladder with escalation to a deterministic floor; shrink check on the condensed replace | ✅ one retry after the first compaction when pressure remains; overflow recovery; shrink check on the framed checkpoint | +| Safety controls | ✅ redaction at every seam that hands stored text to a model; the store stays verbatim | labelled established-background framing on the checkpoint; a hostile id cannot escape the sidecar directory on reads | +| Token budgeting and thresholds | seam knobs (`thresholdRatio`, retain, headroom, per-model policies) plus adapter caps per call, pass, and heal | ✅ threshold ratio with capacity cap, retain ratio or tokens, headroom, exact-target policy table, generation caps | +| Runtime dependencies | `node:sqlite`, Node `>=22.19`; the engine inlines `typebox`; the harness packages stay external | the harness itself; no sidecar, no store | +| Operator surface | `/lcm` with status, report, dump, backup, export, import, gc, migrate, heal, diagnose, and compact | the harness compaction surface the backend mounts behind | +| Resume behavior | ✅ pre-mount backfill pages history at 50 with a 500 inserted-row cap, then rebuilds the last checkpoint from the sidecar | prior checkpoint text in the log carries continuity; no backfill of older history | + +## Where each one leads + +**deepseek-lossless.** Verbatim recall with provenance, a DAG behind one checkpoint +per transaction, cross-session search and reads, and a resume path that rebuilds +continuity from the sidecar. + +**The harness backend.** The closest peer and the default. It reuses the provider +prefix cache by construction, trims tool results without spending a model call, +frames checkpoints with merge rules and a goals section, prices pressure with +cache-bucket resolution, and recovers from overflow. None of that needs a store. + +## By design, not missing + +Gaps against this backend, each with the reason it stays that way: + +- **No prefix-cache riding on summarizer calls.** The ladder sends one message per + call, which cannot express prefix reuse. Deterministic input ordering ships as + hygiene; cache-share claims wait on a replay prototype plus measured cache-read + share plus a live key. +- **No pruner call on the adapter paths.** The seam runs the pruner before its own + pressure and overflow measurement. The bracket paths summarize the selected span + as stored; pruning live history would rewrite what the verbatim promise covers. + The opt-in `mechanicalTools` tier is different: it compacts stored spans without + a model call instead of trimming live history. +- **No curated goals section.** The checkpoint prompt captures evolving goals as + prose. This package keeps the full transcript addressable, so goals are + recoverable by address rather than carried as a section. +- **No child-session reader.** The harness has no hosted child session to borrow, + so the reader runs in process with a step and token budget. Findings stay wrapped + so raw transcript never enters the caller's context. + +## Live proof + +Run 2026-10-25 on the pinned harness with a live session: auto pressure stored 6 +leaves plus 1 condensed node with a checkpoint replacement on the surface and one +`pass` metrics row. The summarizer wrote prose leaves plus a condensed parent with +billed usage; a depth-2 DAG of 61 summaries formed over 4 swaps. A fresh session +answered from aged history verbatim with cross-session entry citations. A defaults +quiet run mirrored 7 rows with 0 summaries and left the log untouched. Two boot +defects were fixed with tests (uninjectable session query degrades to +live-mirror-only; the cold message map re-derives from the live log). Suite +green across `vp check`, `tsc`, `bun run test`, and `bun run pack`. + +Caveat, kept as a tuning follow-up rather than a loop defect: one model route was +pathologically verbose on rung prompts and hit the output cap where another route +wrote clean prose on the same prompts, so those nodes fell back mechanical. + +## When to pick which + +| You want | Reach for | +| ----------------------------------------------------------------------------------------- | ------------------------ | +| A lossless transcript, a navigable DAG, recall by address, and cross-session search | **deepseek-lossless** | +| The harness default: cache-reusing summaries, model-free pruning, and overflow recovery | the basic backend | + +## Verification note + +Method and coverage: + +- The peer cells were read from the installed tarballs at `dsh-compaction-basic` + `0.2.1-rc.3` with `cordis` `4.0.5`, against harness checkout `22ddee3792`: + the summarizer envelope and prefix-cache comment, the checkpoint preamble and + prior-merge rule, the goals section, the pruner call sites, the meter buckets + and pressure arithmetic, and the retry and overflow configuration vocabulary + all came from package source. +- This package's cells came from this repository (`packages/core` + plus `packages/deepseek`). +- This field moves fast. Re-verify a cell before relying on it. diff --git a/apps/docs/content/reference/deepseek/config.mdx b/apps/docs/content/reference/deepseek/config.mdx new file mode 100644 index 0000000..6d5794d --- /dev/null +++ b/apps/docs/content/reference/deepseek/config.mdx @@ -0,0 +1,13 @@ +--- +title: "Config for DeepSeek" +description: "DeepSeek: every capability option, its default, which ride the harness seam, and what stays adapter-only." +search: + tags: ["reference", "config"] + keywords: [config, options, thresholds, bracketAuto] +sidebar: + label: "Config" +--- + +/_snippets/reference/config/intro.mdx + +/_snippets/reference/config/deepseek.mdx diff --git a/apps/docs/content/reference/deepseek/files.mdx b/apps/docs/content/reference/deepseek/files.mdx new file mode 100644 index 0000000..db25f25 --- /dev/null +++ b/apps/docs/content/reference/deepseek/files.mdx @@ -0,0 +1,13 @@ +--- +title: "Files for DeepSeek" +description: "DeepSeek: where the adapter writes sidecars, metrics, exports, backups, and diagnostics, with the modes each file carries." +search: + tags: ["reference", "files"] + keywords: [sidecar, metrics, backup, export, deepseek] +sidebar: + label: "Files" +--- + +/_snippets/reference/files/intro.mdx + +/_snippets/reference/files/deepseek.mdx diff --git a/apps/docs/content/reference/deepseek/meta.ts b/apps/docs/content/reference/deepseek/meta.ts new file mode 100644 index 0000000..0c01979 --- /dev/null +++ b/apps/docs/content/reference/deepseek/meta.ts @@ -0,0 +1,9 @@ +import { defineMeta } from "blume"; + +import { DEEPSEEK_ICON } from "../../../harness-icons.ts"; + +export default defineMeta({ + title: "DeepSeek", + icon: DEEPSEEK_ICON, + pages: ["tools", "command", "config", "files", "comparison"], +}); diff --git a/apps/docs/content/reference/deepseek/tools.mdx b/apps/docs/content/reference/deepseek/tools.mdx new file mode 100644 index 0000000..3fcc600 --- /dev/null +++ b/apps/docs/content/reference/deepseek/tools.mdx @@ -0,0 +1,14 @@ +--- +title: "Recall tools for DeepSeek" +description: "DeepSeek: every parameter and bound for `lcm_grep`, `lcm_describe`, `lcm_expand`, and `lcm_expand_query`, and which one a question needs." +search: + tags: ["reference", "tools"] + boost: 3 + keywords: [recall, search, lcm_grep, lcm_describe, lcm_expand, lcm_expand_query] +sidebar: + label: "Recall tools" +--- + +/_snippets/reference/tools/intro.mdx + +/_snippets/reference/tools/deepseek.mdx diff --git a/apps/docs/content/reference/meta.ts b/apps/docs/content/reference/meta.ts index 30e26c0..e76358c 100644 --- a/apps/docs/content/reference/meta.ts +++ b/apps/docs/content/reference/meta.ts @@ -3,5 +3,5 @@ import { defineMeta } from "blume"; export default defineMeta({ title: "Reference", icon: "code", - pages: ["pi", "opencode"], + pages: ["pi", "opencode", "deepseek"], }); diff --git a/apps/docs/harness-icons.ts b/apps/docs/harness-icons.ts index 3e8220e..72018ab 100644 --- a/apps/docs/harness-icons.ts +++ b/apps/docs/harness-icons.ts @@ -6,3 +6,4 @@ const icon = (name: string, light: string, dark: string): string => export const PI_ICON = icon("cmp-pi", "pi-light.svg", "pi-dark.svg"); export const OPENCODE_ICON = icon("cmp-opencode", "opencode-light.svg", "opencode-dark.svg"); +export const DEEPSEEK_ICON = icon("cmp-deepseek", "deepseek.svg", "deepseek.svg"); diff --git a/apps/docs/harness-nav.ts b/apps/docs/harness-nav.ts index 6260e69..82fb2f3 100644 --- a/apps/docs/harness-nav.ts +++ b/apps/docs/harness-nav.ts @@ -1,14 +1,21 @@ import { withBase } from "blume/components/islands/base-path.ts"; -import { OPENCODE_ICON } from "./harness-icons.ts"; +import { DEEPSEEK_ICON, OPENCODE_ICON } from "./harness-icons.ts"; -export type HostId = "pi" | "opencode"; +export type HostId = "pi" | "opencode" | "deepseek"; export interface Host { id: HostId; label: string; description: string; icon: string; + /** + * Static entry route used when the current section holds no group for this + * host. Guides and reference both ship for DeepSeek now; the entry mirrors + * the first reference page for sections without a DeepSeek group. + * Mirrors the first entry of `content/reference/deepseek/meta.ts`. + */ + entry?: string; } export const HOSTS: readonly Host[] = [ @@ -24,6 +31,13 @@ export const HOSTS: readonly Host[] = [ description: "The OpenCode plugin", icon: OPENCODE_ICON, }, + { + id: "deepseek", + label: "DeepSeek", + description: "The DeepSeek bundle", + icon: DEEPSEEK_ICON, + entry: "/reference/deepseek/tools", + }, ]; export const under = (route: string, prefix: string): boolean => @@ -34,5 +48,6 @@ export const hostFromRoute = (route: string): HostId | undefined => { if (under(route, "/guides/opencode") || under(route, "/reference/opencode")) { return "opencode"; } + if (under(route, "/guides/deepseek") || under(route, "/reference/deepseek")) return "deepseek"; return undefined; }; diff --git a/apps/docs/pages/_home/CopyCommand.astro b/apps/docs/pages/_home/CopyCommand.astro index 0b9c434..2bcda39 100644 --- a/apps/docs/pages/_home/CopyCommand.astro +++ b/apps/docs/pages/_home/CopyCommand.astro @@ -35,7 +35,7 @@ const buttonClass = compact ---
diff --git a/apps/docs/pages/_home/FinalCta.astro b/apps/docs/pages/_home/FinalCta.astro index b866926..7342de1 100644 --- a/apps/docs/pages/_home/FinalCta.astro +++ b/apps/docs/pages/_home/FinalCta.astro @@ -52,7 +52,7 @@ import CopyCommand from "./CopyCommand.astro";

Every new session is logged, searchable, and readable in full.

-
+
{host.label} - {host.next && ( - NEXT - )} )) }
-
+
diff --git a/apps/docs/pages/_home/Packages.astro b/apps/docs/pages/_home/Packages.astro index 17efe5f..948720c 100644 --- a/apps/docs/pages/_home/Packages.astro +++ b/apps/docs/pages/_home/Packages.astro @@ -20,6 +20,13 @@ const shells = [ body: "The same engine as a plugin: recall tools, the swap point, and a per-session sidecar beside the session file.", command: "opencode plugin add opencode-lossless", }, + { + name: "deepseek-lossless", + icon: "/logos/deepseek.svg", + title: "The DeepSeek bundle", + body: "The same engine as a profile bundle: recall tools, the compaction backend, and a per-session sidecar beside the session file.", + command: "dsh plugin --profile headless add deepseek-lossless", + }, ]; --- @@ -47,7 +54,7 @@ const shells = [

Ingest, compaction and retrieval behind a two-port host contract. One - engine runs in Pi, OpenCode v2, or the next host. Each install is one + engine runs in Pi, OpenCode v2, or DeepSeek. Each install is one package.

diff --git a/bun.lock b/bun.lock index cf85716..e9f0a9d 100644 --- a/bun.lock +++ b/bun.lock @@ -23,14 +23,53 @@ }, "packages/core": { "name": "lossless-core", - "version": "0.1.1", + "version": "0.1.2", "dependencies": { "typebox": ">=1.4.3 <2", }, }, + "packages/deepseek": { + "name": "deepseek-lossless", + "version": "0.1.0", + "dependencies": { + "@deepseek-ai/schemastery": "^3.18.5", + }, + "devDependencies": { + "@deepseek-ai/cordis": "4.0.5", + "@deepseek-ai/dsh-agent": "0.2.1-rc.3", + "@deepseek-ai/dsh-commands": "0.2.1-rc.3", + "@deepseek-ai/dsh-compaction": "0.2.1-rc.3", + "@deepseek-ai/dsh-compaction-basic": "0.2.1-rc.3", + "@deepseek-ai/dsh-llm": "0.2.1-rc.3", + "@deepseek-ai/dsh-session": "0.2.1-rc.3", + "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", + "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3", + "@deepseek-ai/dsh-token-meter": "0.2.1-rc.3", + "@deepseek-ai/dsh-tools": "0.2.1-rc.3", + "lossless-core": "workspace:*", + }, + "peerDependencies": { + "@deepseek-ai/cordis": "*", + "@deepseek-ai/dsh-agent": "^0.2.1-rc.3", + "@deepseek-ai/dsh-commands": "^0.2.1-rc.3", + "@deepseek-ai/dsh-compaction": "^0.2.1-rc.3", + "@deepseek-ai/dsh-compaction-basic": "^0.2.1-rc.3", + "@deepseek-ai/dsh-llm": "^0.2.1-rc.3", + "@deepseek-ai/dsh-session": "^0.2.1-rc.3", + "@deepseek-ai/dsh-token-meter": "^0.2.1-rc.3", + "@deepseek-ai/dsh-tools": "^0.2.1-rc.3", + }, + }, + "packages/deepseek-web": { + "name": "deepseek-lossless-web", + "version": "0.1.0", + "peerDependencies": { + "deepseek-lossless": "^0.1.0", + }, + }, "packages/opencode": { "name": "opencode-lossless", - "version": "0.1.1", + "version": "0.1.2", "devDependencies": { "@opencode/ai": "2.1.4", "@opencode/plugin": "2.1.4", @@ -39,7 +78,7 @@ }, "packages/pi": { "name": "pi-lossless", - "version": "0.1.1", + "version": "0.1.2", "devDependencies": { "@earendil-works/pi-agent-core": "1.4.0", "@earendil-works/pi-ai": "1.4.0", @@ -57,6 +96,24 @@ }, }, "overrides": { + "@deepseek-ai/dsh-commands": { + "zod": "^4.6.6", + }, + "@deepseek-ai/dsh-llm": { + "zod": "^4.6.6", + }, + "@deepseek-ai/dsh-llm-retry": { + "zod": "^4.6.6", + }, + "@deepseek-ai/dsh-sandbox-policy": { + "zod": "^4.6.6", + }, + "@deepseek-ai/dsh-session-projection": { + "zod": "^4.6.6", + }, + "@deepseek-ai/dsh-token-meter": { + "zod": "^4.6.6", + }, "@earendil-works/pi-coding-agent": { "undici": "^8.12.2", }, @@ -267,6 +324,62 @@ "@clack/prompts": ["@clack/prompts@1.8.0", "", { "dependencies": { "@clack/core": "1.5.0", "fast-string-width": "^3.0.2", "fast-wrap-ansi": "^0.2.0", "sisteransi": "^1.0.5" } }, "sha512-PXzLZ8N34rxmuo4dJg3xtOXhcBse94qGjDqsteoEYrFrrZ5FSjIGwMAuOcv64ln8rHVBBD06XeVGr+/JX+plcA=="], + "@deepseek-ai/cordis": ["@deepseek-ai/cordis@4.0.5", "", { "dependencies": { "@deepseek-ai/cosmokit": "~1.8.6", "@standard-schema/spec": "^1.1.0" }, "peerDependencies": { "@deepseek-ai/cordis-plugin-include": "~1.0.10", "@deepseek-ai/cordis-plugin-loader": "~1.0.6" }, "optionalPeers": ["@deepseek-ai/cordis-plugin-include", "@deepseek-ai/cordis-plugin-loader"], "bin": { "cordis": "bin.js" } }, "sha512-Sktq5axy0owk9PdnivBRNOJjHEVEibCt0+TDDZcv0+UH2KS1M910fX4kn5y45YFsrpZIi1mJf/1s5VzywMUnyg=="], + + "@deepseek-ai/cosmokit": ["@deepseek-ai/cosmokit@1.8.6", "", {}, "sha512-ZSD/fN52L498LBIOYwjIOlaLv5veY6ZEwI2yzvZUKQtctn0QAPqu8tKdJ+OF5+Tv6NBq1U+wJWH+QI6OGy5EvQ=="], + + "@deepseek-ai/dsh-agent": ["@deepseek-ai/dsh-agent@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-scope": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3", "@deepseek-ai/dsh-typert-protocol": "0.2.1-rc.3", "@deepseek-ai/dsh-util-values": "0.2.1-rc.3", "@deepseek-ai/dsh-workspace": "0.2.1-rc.3" }, "optionalPeers": ["@deepseek-ai/dsh-workspace"] }, "sha512-S6x/deneAsd+pOgecRLM1PC1No9O472kHjS9pS57nuTXi2Vtdo9l7qF+q7PfiHRezL5dxv5gUBQMORPmIYDNvQ=="], + + "@deepseek-ai/dsh-attachment": ["@deepseek-ai/dsh-attachment@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-brand": "0.2.1-rc.3" } }, "sha512-JH7sWHhRdyPJri4H0rFcrv1Bv+XYfpTfYV9VwcokTZ7zxieiR7hHwFr4T2EBcuF6Jpg4vaJ2s2Raw+7/d26acg=="], + + "@deepseek-ai/dsh-brand": ["@deepseek-ai/dsh-brand@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-bPpKuu05gZj7gdW15z6PL2dz4+A4MDEafgWOrhexFO7f08g1NeI4fhE4YNOxd8N2ua+tZJjsKCMAIobFMAaGOw=="], + + "@deepseek-ai/dsh-commands": ["@deepseek-ai/dsh-commands@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-util-crypto": "0.2.1-rc.3", "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-attachment": "0.2.1-rc.3", "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-scope": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-typert-protocol": "0.2.1-rc.3" } }, "sha512-Kuk7WW8aSEAgGaib45BtVQkUrSf1JaERJtvtZ2/YIt9e32SjAyMpIOHa0tcdpyjnNZ9dA2euw6bwxCAy0EGbEA=="], + + "@deepseek-ai/dsh-compaction": ["@deepseek-ai/dsh-compaction@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-commands": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3" } }, "sha512-ibeUgQhjoyNUme+LIcx5V3NYFXv5oMJY3t6UouHilLBnJyi7Rt+VY8zCu2lsWGKv7r4X+42d1o8Q//tRjVd14Q=="], + + "@deepseek-ai/dsh-compaction-basic": ["@deepseek-ai/dsh-compaction-basic@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-util-values": "0.2.1-rc.3", "@deepseek-ai/schemastery": "~3.18.5" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-commands": "0.2.1-rc.3", "@deepseek-ai/dsh-compaction": "0.2.1-rc.3", "@deepseek-ai/dsh-compaction-tool-result-pruner": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-token-meter": "0.2.1-rc.3" }, "optionalPeers": ["@deepseek-ai/dsh-compaction-tool-result-pruner"] }, "sha512-28cB6jgAILPk4w/q/J2bo29/vq7sGAaxpCoEGhf4e8G+743BtzK4niAPtLx85T7ZPTmqUzCtIzIIu/mUM0BXog=="], + + "@deepseek-ai/dsh-compaction-image-offload": ["@deepseek-ai/dsh-compaction-image-offload@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-util-values": "0.2.1-rc.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-compaction": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3" } }, "sha512-QX769h8IXaaDwzNpdJV8LQPRn/5BnOEp7d8K45LPHAusFW1x50Ov3lWhvbWh/wzhkyI6Gc2E2SjUfdP9KxHCkg=="], + + "@deepseek-ai/dsh-fs": ["@deepseek-ai/dsh-fs@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-sandbox": "0.2.1-rc.3" } }, "sha512-tENLWe471WlAXMc13J4cns8xU6lFyu1smwpkSKR85vhGr6FyWZhTlpVohZYm66LbwKTmSWiDhd9Urk7VxN9B9w=="], + + "@deepseek-ai/dsh-llm": ["@deepseek-ai/dsh-llm@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-timeout": "0.2.1-rc.3", "@deepseek-ai/dsh-typert-protocol": "0.2.1-rc.3", "@deepseek-ai/dsh-util-crypto": "0.2.1-rc.3", "@deepseek-ai/dsh-util-values": "0.2.1-rc.3", "@deepseek-ai/schemastery": "~3.18.5", "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-7Fq7zS3yQb1J1PsaNmahQ1gHDdsb338UhTkmZCexlNecmc5I44c/PbkHcT5pJ5SO4hnLplJOdDhiIMFpvvyAaw=="], + + "@deepseek-ai/dsh-llm-retry": ["@deepseek-ai/dsh-llm-retry@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/schemastery": "~3.18.5", "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", "@deepseek-ai/dsh-timeout": "0.2.1-rc.3" } }, "sha512-JoNZ0Cw8xYxko1rQ1H2oMtsvQ+Qti10UAcPUUaHv/SR22gPQOT+g8BiynLKyS7wMJam4jdAlVqjiA6gGfBf/xA=="], + + "@deepseek-ai/dsh-ptc-runtime": ["@deepseek-ai/dsh-ptc-runtime@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-sandbox": "0.2.1-rc.3" } }, "sha512-nFUzb1abLvCQfUsrp/fMKQMCYU+O+ttiwqUFMa+uO4rFE6Nd+xNjus3V5aaqyy78jdciyBzzNysblZeryJAcXQ=="], + + "@deepseek-ai/dsh-sandbox": ["@deepseek-ai/dsh-sandbox@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-util-values": "0.2.1-rc.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3" } }, "sha512-7JXzEw2FBNAD3hNhiMzpxCjm8Zedc6DpFfsJfvh1Uh4HuIzX2BFEtzjvEUSwXGSWA+pCZOEvaYD9v0sZoMx3fQ=="], + + "@deepseek-ai/dsh-sandbox-policy": ["@deepseek-ai/dsh-sandbox-policy@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/schemastery": "~3.18.5", "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-sandbox": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3" } }, "sha512-/PlQEx/F6pwjjC/eOEBCHWG9InCu+aSEIC1k9V32noS43B/KS/FRSgP+SZ0YkbjMfk6iyS27sgglJDIY7RG1zQ=="], + + "@deepseek-ai/dsh-scope": ["@deepseek-ai/dsh-scope@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-3oCu1zZhbJTzV9TtYXjVUyxc9yOaQOYzxSGQvMuZ/q/O7/1Y1ofIS2j2GETfc79mCoiQIoQAjr2gu/ncXSgALw=="], + + "@deepseek-ai/dsh-session": ["@deepseek-ai/dsh-session@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-util-values": "0.2.1-rc.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-scope": "0.2.1-rc.3" } }, "sha512-z1Dfn6yB2YMm9iNThPHaIYCx0Y/mKjm+fMrWFtnPZmX0fJXVmTIr4RAfLF7ZFJk6qd52SUl8d2uX+uNbrPJESg=="], + + "@deepseek-ai/dsh-session-projection": ["@deepseek-ai/dsh-session-projection@0.2.1-rc.3", "", { "dependencies": { "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-session": "0.2.1-rc.3" } }, "sha512-kyDxXb0avo0m4ZmAb8P+X6T/963NOZCvH/MgrWl3AUs5fHkXA6NfeVnalj488o5RpkRtZdqI7KtCEFNkyKAEsA=="], + + "@deepseek-ai/dsh-system-prompt": ["@deepseek-ai/dsh-system-prompt@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/schemastery": "~3.18.5" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-scope": "0.2.1-rc.3" } }, "sha512-2CXnigUF0N2Ps0AG/lOvvY9qeEhvo5uUCGCqvoGrH9IWbhoh/tfvzo9KTa5svyZIBW6mLMAyLyHOdi7f1T9tfg=="], + + "@deepseek-ai/dsh-timeout": ["@deepseek-ai/dsh-timeout@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-DOuGjmY5dMlXqXqhuXlHNgEVajTLTYuj/EabS5b8IIBC5VN36YDOtyPgaT8aq1+LI7XoReDMUYQRUHCI3SOXqw=="], + + "@deepseek-ai/dsh-token-meter": ["@deepseek-ai/dsh-token-meter@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-util-values": "0.2.1-rc.3", "@deepseek-ai/schemastery": "~3.18.5", "zod": "4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-compaction": "0.2.1-rc.3", "@deepseek-ai/dsh-compaction-image-offload": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-llm-retry": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3" } }, "sha512-K2lxx7axpJb9mW3jzvRxiTf19DbTAyXjWkeKgq58DmZyVm64RuWjzR6fB+6NMr3ZVlMYn5/GvFN1Fm/n8Tspuw=="], + + "@deepseek-ai/dsh-tools": ["@deepseek-ai/dsh-tools@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-util-values": "0.2.1-rc.3", "@deepseek-ai/schemastery": "~3.18.5" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-ptc-runtime": "0.2.1-rc.3", "@deepseek-ai/dsh-sandbox": "0.2.1-rc.3", "@deepseek-ai/dsh-sandbox-policy": "0.2.1-rc.3", "@deepseek-ai/dsh-scope": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3", "@deepseek-ai/dsh-user-approval": "0.2.1-rc.3", "@deepseek-ai/dsh-working-directory": "0.2.1-rc.3" } }, "sha512-fHmhIFaztPt9+uvOlk3p4wXMU62F+TRe0ozVPasV3seIsFcxpca74QGSK5dFHICDhKIY1T9MUh0F3itMTdMi3A=="], + + "@deepseek-ai/dsh-typert-protocol": ["@deepseek-ai/dsh-typert-protocol@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "zod": "^4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-ycKq2jF0h/iE/gwQFXn7Fx/pdKsVnPe6rkaA8F0SIZ8QfbAK8KcYmpG1Kn9zjwZAWPyb6pLMhD2GI/gXlM7f0Q=="], + + "@deepseek-ai/dsh-user-approval": ["@deepseek-ai/dsh-user-approval@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/schemastery": "~3.18.5" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-brand": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-scope": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3" } }, "sha512-n2AG2Oh3/32UNStxdkoTOYwheOjSjrJWPj9pPvgISw6vBURPUYrquoVD0AmvmV8TeWpxaCKIGsFNO08yrP9ODw=="], + + "@deepseek-ai/dsh-util-crypto": ["@deepseek-ai/dsh-util-crypto@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-AKznKjQmc2JHDQO+DV2Z68zwGrr3+0qAkxNqZE3SnuBTpO7TeTaS7v2cOzQi81EhZUlYFArmKEbmjcsCrsQvvw=="], + + "@deepseek-ai/dsh-util-values": ["@deepseek-ai/dsh-util-values@0.2.1-rc.3", "", { "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5" } }, "sha512-YyHu30xQBdp7sIr4ZNUliNxm2S9yRNOwdUPiBvfQZ+KaRtekI2j+T8+TFy7qgyBiGM7YYiG0xzJVibS894WIrg=="], + + "@deepseek-ai/dsh-working-directory": ["@deepseek-ai/dsh-working-directory@0.2.1-rc.3", "", { "dependencies": { "@deepseek-ai/schemastery": "~3.18.5", "zod": "^4.4.3" }, "peerDependencies": { "@deepseek-ai/cordis": "~4.0.5", "@deepseek-ai/dsh-agent": "0.2.1-rc.3", "@deepseek-ai/dsh-fs": "0.2.1-rc.3", "@deepseek-ai/dsh-llm": "0.2.1-rc.3", "@deepseek-ai/dsh-session": "0.2.1-rc.3", "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3" } }, "sha512-Ec/MG+j4jWu/4+ym9UieB2zHt22mwSFfS2s2xZfnoCQIrxkIH5ayVcY1SNttrmHFYDwsyZEG4YY2dzYRtcuIfw=="], + + "@deepseek-ai/schemastery": ["@deepseek-ai/schemastery@3.18.5", "", { "dependencies": { "@deepseek-ai/cosmokit": "~1.8.6", "@standard-schema/spec": "^1.1.0" } }, "sha512-Mnobvqf9rHsZYv4g5edj5ddxS1oF1kIdSlz0DunUgnY3LWG5F26MqvH3YOOsSaodCgyzJ16OMN9O99/s5Crtdw=="], + "@earendil-works/pi-agent-core": ["@earendil-works/pi-agent-core@1.4.0", "", { "dependencies": { "@earendil-works/pi-ai": "1.4.0", "typebox": "1.3.27" } }, "sha512-PoKY84SKUwnt7S5QBZaGUqfNO1kjP3KHpHUA9QpFTj4cOjaVWMYIcBPJ6FDRCdE8Zen9OSECQUd0uUfVgtSmgA=="], "@earendil-works/pi-ai": ["@earendil-works/pi-ai@1.4.0", "", { "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", "@earendil-works/pi-telemetry": "1.4.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", "https-proxy-agent": "9.1.0", "openai": "7.19.0", "partial-json": "0.1.7", "typebox": "1.3.27" }, "bin": { "pi-ai": "dist/cli.js" } }, "sha512-hQM/C4+1w8rHYACzDDf+0vQMnEnpzIop1tptIERVgR3ZeT/hHSzBNG8PaWPgxWQjqSeKCyBZSyp8uTT6eKK1pQ=="], @@ -1387,6 +1500,10 @@ "decode-named-character-reference": ["decode-named-character-reference@1.3.0", "", { "dependencies": { "character-entities": "^2.0.0" } }, "sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q=="], + "deepseek-lossless": ["deepseek-lossless@workspace:packages/deepseek"], + + "deepseek-lossless-web": ["deepseek-lossless-web@workspace:packages/deepseek-web"], + "default-browser": ["default-browser@5.5.1", "", { "dependencies": { "bundle-name": "^4.1.0", "default-browser-id": "^5.0.0" } }, "sha512-m1pAzaJgZ/gssEqlOhJkPJp8Xly7QyW6xcrkUa2KKcDeDSEMP7X8xipU3snUcfisTQx0w1AGae+9UtJSfVnXGw=="], "default-browser-id": ["default-browser-id@5.0.1", "", {}, "sha512-x1VCxdX4t+8wVfd1so/9w+vQ4vx7lKd2Qp5tDRutErwmR85OgmfX7RlLRMWafRMY7hbEiXIbudNrjOAPa/hL8Q=="], diff --git a/docs/deepseek-plan.md b/docs/deepseek-plan.md new file mode 100644 index 0000000..1ea96dc --- /dev/null +++ b/docs/deepseek-plan.md @@ -0,0 +1,545 @@ +# DeepSeek plan + +Status: active. The code lives on `feat/deepseek`. OpenCode v2 already shipped alongside Pi; this plan +tracks the DeepSeek adapter. Roadmap item 10 describes the live-proof gate. + +Goal: answer whether `lossless-core` can serve the DeepSeek harness as one +more adapter, without weakening the engine boundary or the Pi adapter. + +## Constraints the design must honor + +These come from the tree, not from preference. + +- The engine boundary holds. `packages/core/src/host.ts` is the only host + surface (`ModelHost` plus `Reader`), and the boundary check fails any module + under `packages/core/src/` that imports a host SDK. Another adapter proves + the boundary instead of widening it. +- The target harness owns surface mutation. Its session is an append-only log, + `deriveMessages()` renders the model history, and loop-built requests arrive + frozen. There is no equivalent of the Pi `context` event that rewrites the + outgoing request in flight. Model-visible change must be a logged event, normally + a `user/message` with a `replace` operation inside the `compaction/start` to + `compaction/end` bracket. +- The target's compaction seam is `compactIfNeeded`, `compactNow`, and + `compactRegion` on one `CompactionEngine` per composition, with tool-pairing + balance checks on both edges and a shrink check that rejects a summary no + smaller than its span. +- Lossless behavior worth keeping: verbatim ingest, the leaf plus condensed DAG + with escalation to a deterministic floor, addressable summary ids, a budgeted + reader loop over `lcm_grep`, `lcm_describe`, and `lcm_expand`, per-model + calibration, and the metrics rows that make a pass auditable. +- Names stay unscoped until a collision forces the one-commit migration that + `docs/roadmap.md` already describes. The working name below is a placeholder + the owner replaces. +- Comparison claims ship on audited per-harness pages, including + `apps/docs/content/reference/deepseek/comparison.mdx`, per roadmap item 10. + +## Candidates considered + +Three shapes, structurally distinct. + +### A. Recall-only sidecar + +Keep the shipped basic compactor. Add a plugin that mirrors session history +into a sidecar store and registers the three recall tools plus one command. + +```yaml +- name: basic-compaction-backend +- name: lossless-recall +``` + +Smallest diff. It proves ingest mapping, the `ModelHost` mapping over +streaming calls, and tool registration. It concedes compaction policy entirely, +so the DAG, the swap timing, and the pointer projection buy nothing yet. + +### B. Summarizer subclass + +Subclass the shipped `BasicCompactionEngine` and override only `summarize()`. +Route that hook through lossless escalation and the sidecar store. Keep the +target's transaction, meter, retention, and lock behavior untouched. + +```ts +class LosslessSummarizerEngine extends BasicCompactionEngine { + protected summarize( + input: SummarizationInput, + agent: Agent, + signal?: AbortSignal, + ): Promise { + // not implemented: build the lossless prompt from input.messages, + // call through the host port, frame the checkpoint + } +} +``` + +Medium diff. It changes summary quality without owning range selection or +surface writes. It still concedes the DAG shape, because the seam expects one +checkpoint per transaction, not a frontier of nodes. + +### C. Full engine backend + +Implement `CompactionEngine` directly. Map lossless retention onto region +selection, lossless `runCompaction` onto the bracket transaction, and the +frontier onto the replacement checkpoint. + +Largest diff and largest risk. It must reimplement lock handling, stability +checks for whole-surface versus selected-span calls, and the busy taxonomy +correctly. It is the only shape that carries the full lossless policy across. + +## Synthesis decision + +Order the work A, then B, then C only on evidence. Each phase is usable on its +own, and each is the prerequisite of the next. This follows Foundational +thinking (scaffold before features) and Sequence verifiable units (one checkable +state per phase). + +## Candidate C plan (DAG parity slices) + +Verified against the published tarball at the pinned release: the package +root exposes config types and `BasicCompactionEngine` only. No region +internals, no range selection, and not even the `summarize()` contract cross +the published boundary. So any C built on published code stays inside the +subclass seam, and the risky machinery stays the target's without any +reimplementation: the lock is a durable unmatched `compaction/start` marker +read from the log, and stability checks, the busy taxonomy, the shrink check, +and the commit framing all run behind `super.compactRegion`. The earlier +warning that C must reimplement them applied only to a C that bypasses the +engine; that C is not planned. + +The DAG already runs in the sidecar under B: `runCompaction` stores leaf and +condensed nodes with coverage, proven by the engine tests. The remaining gap +to voltropy parity is three slices, each independently testable: + +- C1, checkpoint pointers. The ladder outcome already returns leaf and + condensed summary ids. Include `[lcm:summary #N]` refs in the checkpoint + text so the model and the recall tools can cite and resolve DAG nodes. + Evidence: a real compaction whose derived checkpoint names ids that + resolve in the sidecar. +- C2, previous summary. Feed the prior checkpoint as `previousSummary` so + the ladder merges instead of re-summarizing. The plugin tracks the last + checkpoint text per session (projection state), which avoids parsing + checkpoint tags. Evidence: a second compaction over checkpointed history + whose ladder input carries the previous summary. +- C3, already-memory mapping. Map a ladder `nothing` outcome (span rows all + covered) to a graceful `null` instead of a failed compaction. The planned + coverage-aware narrowing was dropped after analysis: replace semantics keep + the surface head always uncovered, so a covered prefix cannot occur, and + covered-but-live rows (range expansion catching retained rows older than a + checkpoint) stay truly covered under shadowing, so compacting over them is + already correct. Evidence: forcibly compacting an already-covered live span + rejects with `SpanAlreadyMemoryError` instead of the seam's generic + could-not-summarize failure. Null is not the answer there: the forcible + seam always compacts or throws by contract, and the auto path never selects + a fully covered span because replace semantics keep the head uncovered. + +Seam constraint, accepted: one checkpoint per transaction. The DAG lives in +the sidecar and the checkpoint is its projection, which is sufficient for +recall because the reader resolves pointers. Optional later upstream ask: a +small PR exporting the region functions and the summarize contract from the +package root, which removes the structural-copy drift rule. Not blocking. + +## Custom transaction flow (DAG-wins assumption, design sketch) + +The C slices are done, so this section designs what the confirmed +assumption commits us to: the DAG itself living in the DeepSeek log +instead of projected from the sidecar into one checkpoint per +transaction. If the item 1 numbers, once runnable, show the DAG does not +beat structured checkpoints, this flow was unnecessary and candidate B +stays the ceiling. + +### Problem: what a transaction is, and what constrains a custom one + +Traced from the pinned `dsh-compaction-basic` lib. Config types and +`BasicCompactionEngine` are the whole published boundary. + +- One transaction is `compaction/start`, then `compaction/summary` (a log + record with shadowed range, seqs, token count, provider, model, usage) + plus `user/message` with a `replace` surface op, then + `compaction/end`. Failure still appends `compaction/end` carrying the + error, which is what releases the durable lock (an unmatched + `compaction/start` read from the log). +- The shrink check rejects a framed checkpoint at or above the shadowed + route token count. Near-verbatim leaf nodes cannot pass it as replaces. +- Stability is span-scoped or whole-surface; the busy taxonomy separates + manual-idle (`owner: null`, no open turn) from auto (open turn + required). +- Checkpoint framing (preamble, `` tags, + `compactCheckpointSource`) is reader and tooling surface: custom commits + must reuse it byte for byte. +- No live-model verification exists on this branch: every slice proves + itself through scripted adapters only. The live-proof Evidence bullet below + covers the adapter loop on a live session; the transaction-flow numbers + (item 1) still wait on a DeepSeek key. + +### Candidate A: orchestrated multi-compact through the published seam + +Run the ladder once, then commit one DAG node per `super.compactRegion` +call, staging each node text for the per-transaction `summarize()` hook. +No reimplementation, no upstream PR, every bracket independently valid to +their tooling. + +It fails on the shrink check: leaf text is near-verbatim, so leaf replaces +are rejected, and the seam always pairs a summary record with a replace. +Through the published seam only condensed nodes can be committed, which is +a chain the C2 carryover already yields across compactions, not a DAG in +the log. A is status quo with orchestration, not parity. + +### Candidate B: one bracket, custom commit body + +Reimplement the bracket in the adapter: open, run the ladder once outside +the seam, commit one condensed replace plus one log-only +`compaction/summary` record per leaf (no surface op, linked by +`sourceEventSeqs`), close. The DAG lives in the log while the surface +pays only for the condensed node. + +Caller usage first: + +```ts +// Implements this shape, with one amendment recorded below +const result = await commitDagTransaction(deps, agent, selection, { + owner: null, // or a non-null marker for the auto path + stability: "whole-surface", // or "selected-span" + signal, +}); +// result: { compactionId, startSeq, summarySeqs, leafSeqs, endSeq } +``` + +Types derived from it: `DagCommitResult` as above (no `DagNodeCommit`: +the ladder runs inside the bracket, see the amendment). Module map: a new +`packages/deepseek/src/dag-commit.ts` holds the bracket; `engine.ts` gives +up a `runLadder(ctx, lossless, target, sessionId, span, signal)` helper +that `summarize()` keeps using for the B path, so the ladder runs +identically inside and outside the seam. Framing, lock check, stability +predicate, the shrink check on the condensed replace, and the manual/auto +error taxonomy are reimplemented against the pinned lib line for line, +each marked with the source line it mirrors so a pin move has a checklist. + +Amendment (B1): the sketch passed precomputed `nodes` in. The bracket +owns the ladder instead, mirroring how `compactSurfaceRegion` owns its +summarizer. Passing nodes in would split open/commit across the caller +boundary and leave ladder errors outside the failure taxonomy, which is +the half the spike exists to prove. Two deliberate divergences from the +lib, both marked B1 in code: single ladder attempt (no recover retry +loop; arrives with production wiring) and one log-only summary record +per stored DAG node, appended before the condensed record so the seam's +shadow-price adjacency contract still holds. + +### Synthesis decision + +B is the only shape that puts the DAG in the log; A cannot carry leaves +through the seam. Build B in slices: B0 this sketch, B1 bracket spike +against scripted models (open, commit, close, plus the failure path +releasing the lock), B2 production wiring through the plugin paths, with +convergence onto exported region primitives as a later option if the +upstream PR ever lands. Each slice re-verifies the B path has not +regressed, because the custom flow and the subclass seam now share the +ladder helper. + +B1 landed: `packages/deepseek/src/dag-commit.ts` (`commitDagTransaction`), +`packages/deepseek/test/dag-commit.test.ts` (3 tests: condensed replace +plus leaf records in one bracket, ladder failure closes with error and +releases the lock, manual taxonomy on idle), and the `runLadder` / +`resolveSummarizationTarget` / `withSummaryRefs` extraction in +`engine.ts` that `summarize()` keeps using. Gates green: `vp check`, 584 +tests, typecheck, pack. + +B2 landed (manual production path through the bracket): + +- `packages/deepseek/src/ladder.ts` holds the shared ladder + (`LosslessEngineDeps`, `runLadder`, `resolveSummarizationTarget`, + `withSummaryRefs`, `SpanAlreadyMemoryError`, `checkpointMemory`); + `engine.ts` re-exports the pre-split names so no consumer changed. + The split keeps the module graph acyclic (engine references the + bracket, the bracket references the ladder). +- `DagCommitResult` is a superset of the seam's `CompactionResult` + (branded ids plus `summarySeqs`/`leafSeqs`), so the engine override + and `/lcm compact` read it unchanged. +- `selectDagRange` mirrors `selectCompactableRange`; the engine + `compactNow` override mirrors the seam's idle admission, selection, + and flush, with the bracket as the transaction. +- Auto pressure (`compactRegion`) stays on the B seam, so the B-path + regression suite keeps pinning it. Routing auto through the bracket + waits until the manual path has proven itself live. + +B3 landed (flag-gated auto routing): `bracketAuto` on `LosslessEngineDeps` +and `LosslessPluginConfig` routes `compactRegion` through the bracket +(owner current-turn, whole-surface stability) while the default stays the +seam. The flag rides our own deps because the seam's `resolveConfig` +rejects unknown keys, so it cannot travel on `BasicCompactionConfig`. +Tests pin the bracket commit on an open turn and the raw (unwrapped) +ladder failure on the auto path. Gates green: `vp check`, 594 tests, +typecheck, pack. + +- Tests: `selectDagRange` (null, range shape, drift fails closed), + `compactNow` (bracket commit, null, busy on open turn, busy on + refused maintenance, cancelled), and the `/lcm compact` production + test now pins the DAG shape (condensed plus leaf records under one + id). Gates green: `vp check`, 592 tests, typecheck, pack. + +## Comparison docs direction (staged, waits on live proof) + +- The shipped docs keep per-harness reference pages (`pi`, `opencode`, + and `deepseek` when it ships): host-specific facts only. A root page + with shared guarantees stays a staged idea until the adapter proves + itself live; nothing is ever copied between pages. +- The docs-governance skill gains a lane for the root-plus-pages structure + only if the root page happens. +- Package READMEs stay as they are. Only the docs site changes. + +## Quality bar for this workstream + +Every slice ships production grade: clean, efficient, performant, composable, +and extensible, with keep-it-stupid-simple winning wherever it does not cost +one of the others. Concretely: no new machinery where the seam already owns +it, no speculative abstraction, no dead config, per-session state instead of +globals, fail-closed errors the engine's recovery path already handles, and +green gates (`vp check`, `tsc`, `bun run test`, `bun run pack`) before every +commit. + +## Module map + +One new package holds every host-aware line. Core stays host-free. + +```text +packages/deepseek/ + src/ + index.ts plugin entry: engine registration, tools, command + model-host.ts ModelHost over the streaming call API + reader.ts Reader over the recall tools + ingest.ts log observation mapped to ingestible entries + tools.ts recall tool definitions + engine.ts summarizer subclass over the lossless ladder + ladder.ts shared ladder the seam and bracket paths both run + dag-commit.ts bracket transaction around one condensed replace + command.ts operator command and config verbs + lifecycle.ts read-only and dry-run verbs behind the command + metrics.ts JSONL rows beside the sidecar + plugin.ts composition wiring behind the enabled flag +``` + +The caller configures it the way the target composes everything else, by +selecting the capability in configuration: + +```yaml +- name: lossless-recall +- name: lossless-compaction + config: + keepRecentTokens: 20000 + summaryTokens: 1500 +``` + +## Phases and evidence gates + +Run `vp check` after every phase. Add `bun run test` and `bun run typecheck` +for any phase that touches `packages/*/src/**`. + +1. Port proof in a scratch composition. Implement the host port over one + streaming call and run one summary plus one retrieval against the + mechanical path. Evidence: the test log showing text assembled from chunks + and mapped to the `ModelAnswer` union, including the capped case. +2. Ingest mirror. Observe the live log and the persisted copy, write both to + a throwaway store, and compare entry counts against the derived message + count. Evidence: counts match across a session that includes tool calls, + a steering injection, and a resume. +3. Recall tools. Register the three tools in agent scope, ask a question whose + answer lives only in aged history, and require the answer to cite an entry + id or a summary id. Evidence: cited answer plus the retrieval budget rows. +4. Engine hook behind a flag. Ship candidate A or B with the engine disabled + by default, then enable it on one composition and replay a compacted + session deterministically. Evidence: identical derived messages on replay + and the target's own compaction tests still green. +5. Policy, command, and config. Wire thresholds, per-model overrides, the + operator command, and the metrics rows. Evidence: a pressure run that swaps + before the window fills and a quiet run that leaves the log untouched. + +Stop after any phase whose evidence fails. A failed phase is a finding for +roadmap item 10, not a reason to start the next phase. + +## Owner decisions (recorded) + +- Package name: `deepseek-lossless`, in `packages/deepseek/` to mirror the + `packages/pi` holding `pi-lossless` convention (internal only; the shipped + name is what matters). +- Comparison docs: restructure toward a root page with per-package sub-pages + (see below). Implementation waits on live proof, per roadmap item 10. +- Candidate C: proceeded as DAG-parity work in slices (done: C1 refs, C2 + carryover, C3 already-memory). The item 1 measurement is blocked (no + DeepSeek account, so no live-model DeepSeek arm); the work continues on + the confirmed DAG-wins assumption, with the numbers as a stop/confirm + gate when a key exists. Roadmap item 10 tracks this work until then. +- Parity program (slices 1-6 done on `feat/deepseek`, scripted models only): + answered retrieval, metrics rows, full `/lcm` verb set, recall param + parity with cross-session reads, seam knob and budget forwarding, async + backfill. Suite stands at 79 package tests green with gates clean. +- `enabled` default stays opt-in (`enabled !== true` returns noop). + OpenCode defaults true; flipping here changes mount behavior for every + existing composition, so it waits on an explicit owner call. Recorded as + a known divergence, not an oversight. +- `bracketAuto` default stays off until the bracket manual path proves + itself live. The B-path regression suite pins the seam default. +- Blocked pending harness seams (verified in harness source, not assumed): + settings cockpit overlay (no custom-render seam on the composition ctx), + Pi-style projection policy (auto pressure belongs to the seam engine), + synchronous pre-mount scan (prohibited; async observation is the + permitted path and is now the backfill), provider window auto-correction + (no named-window error seam; `tokenMeter.measure` covers steady state). + +## Evidence (spike baseline) + +Target checkout `38286a6cf9` (release `dsh-0.1.7-alpha.2`, vendor `4.0.4`). +Pins match exactly: `dsh-*@0.1.7-alpha.2`, `cordis@4.0.4`. +Current pins: `dsh-*@0.2.1-rc.3`, `cordis@4.0.5`, against harness HEAD `22ddee3792`. + +- Target baseline: `pnpm vitest run packages/compaction` in the checkout, + 13 files and 263 tests green. +- Lossless suite with the new package: 578 tests green (`vp check`, `tsc`, + `bun run test`, `bun run pack` all pass, with the harness left external). +- Phase 1: scripted adapter through the real escalation ladder and one + `runRetrieval` in two steps against a real store. +- Phase 2: every derived text block lands in the sidecar, re-mirroring + inserts nothing, and a replace checkpoint shrinks derived history while + shadowed entries stay stored. +- Phase 3: three tools registered on a real tools composition; answers read + from a real store. +- Phase 4: one real `compactRegion` through the ladder with leaves in the + sidecar and the envelope on the summary event; fail-closed on unmapped + history; the engine shrink check fires on an irreducible span. +- Phase 5: disabled-by-default plugin; live-feed mirror with `/lcm status`; + one full manual compaction through the `/lcm compact` handler; one quiet + pressure run returning null with the log untouched. +- C1: the derived checkpoint names `[lcm:summary #N]` refs that all resolve + in the sidecar. +- C2: a second compaction over checkpointed history produces a checkpoint + starting with the first checkpoint's exact text, through the live feed + across two manual compactions. +- C3: spans order by sidecar rowid (core reads a span as a storage range); + forcibly compacting an already-covered live span rejects with + `SpanAlreadyMemoryError` while the surface stays untouched. +- Suite with the C slices: 581 tests green. +- B1: `commitDagTransaction` commits one condensed replace plus log-only + leaf records in one bracket; ladder failure closes with error and + releases the lock; manual taxonomy on idle. `runLadder` is shared with + the seam `summarize()` so the summarizer cannot drift. +- B2: the production manual path runs the bracket (`compactNow` override + mirrors idle admission, selection, flush); `DagCommitResult` stays a + superset of the seam result so `/lcm compact` reads it unchanged; the + `/lcm compact` test pins the production DAG shape. +- B3: flag-gated auto routing (`bracketAuto`, default off) through the + bracket; the flag rides our own deps because the seam's `resolveConfig` + rejects unknown keys. +- Suite with the B slices: 594 tests green (`vp check` clean, `tsc`, + `bun run test`, `bun run pack`). +- Memory hydration: `checkpointMemory` takes a sidecar lookup and rebuilds + the last checkpoint from the newest stored summary plus its children + when the process has no noted text, so a resumed session compacts with + continuity. Exact for leaf-plus-condensed runs; a single-leaf run + replaces its row's text on condensation, so the rebuild carries the + stored form. Suite: 596 tests green. +- Finding for a future pin move: the package root omits the `summarize()` + contract members, so the override repeats their shape (drift rule in + `packages/deepseek/src/engine.ts`). +- Parity slice 1: `lcm_expand_query` answers from aged history through the + in-process reader and cites the entry or summary id, with one + retrieval-kind row per run and no cross-session search yet. Suite: 41 + tests green (`vp check`, `tsc`, `bun run test`). +- Parity slice 2: the DeepSeek adapter writes metrics rows beside the sidecar + (new `packages/deepseek/src/metrics.ts` with the OpenCode rotation policy + and 0600 modes): one `pass`-kind row per ladder run with the outcome plus + leaf and condensed counts and tracker usage, one `compaction-checkpoint`-kind + row per DAG bracket commit with condensed plus leaf seqs, and `recall` plus + `retrieval` rows through the wired recorder. Suite: 44 tests green + (`vp check`, `tsc`, `bun run test`, `bun run pack`). +- Parity slice 3a: `/lcm` grows `status|report|backup|export|diagnose` beside + `compact` (new `packages/deepseek/src/lifecycle.ts`, mirroring opencode; + single registration with manual subcommand dispatch, since the registry + rejects a duplicate `lcm` name). Status keeps the counts lines and adds the + meter window with thresholds plus the last pass outcome; report rolls up the + slice-2 rows in the OpenCode vocabulary; backup and export write 0600 files; + diagnose builds the redacted core bundle with `--save` support. Suite: 50 + tests green (`vp check`, `tsc`, `bun run test packages/deepseek`, + `bun run pack`). +- Parity slice 3b: `/lcm` grows `import|gc|migrate|heal`; gc/migrate/heal + dry-run by default with `--apply` the only destructive path, and import + runs on its named file (new verbs in + `packages/deepseek/src/lifecycle.ts`, dispatched in `command.ts`, `storeFor` + wired in `plugin.ts`). Import dedups through `importTranscript`; gc collects + age-based sidecar candidates and never touches live stems; migrate gates on + `storeGeneration` against `SCHEMA_VERSION`; heal runs core `runHeal` with + core default budgets until slice 5 calibrates this adapter. Suite: 54 + tests green (`vp check`, `tsc`, `bun run test packages/deepseek`, + `bun run pack`). +- Parity slice 4: `lcm_grep` reaches Pi param parity (`pattern`, + `case_sensitive`, `offset`, `after`, `summary_id`, `include_removed`, + `scope`) over core `grepView`, `lcm_expand` gains `session`, and + cross-session search plus reads run through the sidecar listing with Pi + caps (20 stores, 40 hits, 1.5 s) and `sessionsScanned` recorded; the + default scope stays the current session. Suite: 63 tests green + (`vp check`, `tsc`, `bun run test packages/deepseek`). +- Parity slice 5: seam policy knobs (`thresholdRatio`, `retainTokens` / + `retainRatio`, `headroomTokens`, `modelPolicies`) ride the plugin config + into the resolved seam config with adapter-only keys kept off it; + summarizer caps thread into the ladder `makeLlm` config and the heal + `makeLlm` plus `runHeal` budgets, with an unset heal source deriving + 8 x target clamped 3000..24000 inside core `runHeal`; `reasoning` stays + false per the harness sources. Suite: 72 tests green (`vp check`, `tsc`, + `bun run test packages/deepseek`, `bun run pack`). +- Parity slice 6: pre-mount history backfills through the async observation + seam (the same `observeSession` call the host history controller pages + from in `packages/api/session-controller/src/history.ts`), paged at 50 + events with a 500-event per-session cap and mirrored through the live + `mirrorEvent` path, so re-mirroring inserts nothing; each backfill writes + one `backfill` metrics row with its ingested count and span. Suite: 79 + tests green (`vp check`, `tsc`, `bun run test packages/deepseek`, + `bun run pack`). +- Live proof 2026-10-25 (installed dsh 0.2.1-rc.3, throwaway `lossless-proof` + profile, wrapper outside the repos, agent on OpenCode Go + `deepseek-v4.1-flash`): auto pressure fired on a live session; the seam + path stored 6 leaves plus 1 condensed node with a checkpoint replace on + the surface and one `pass` metrics row. Summarizer routed to OpenRouter + `deepseek/deepseek-v4.1-flash-1017` wrote real prose leaves plus a + condensed parent (usage billed); the Go route 400s direct summarizer calls + (`MissingSessionID`: no `x-opencode-session` header), which escalation + absorbed into mechanical nodes. A fresh session called `lcm_grep` plus + `lcm_expand` and answered verbatim with cross-session entry citations. + A defaults quiet run mirrored 7 rows with 0 summaries. Two boot bugs fixed + with TDD on the branch (uninjectable `sessionQuery` degrades to + live-mirror-only; cold message map re-derives from the live log). Suite at that run: + 873 tests green. Merge and item 10 stay owner decisions. +- Live proof, continued: the Go `MissingSessionID` 400s are fixed on our end + by stamping the calling session id into `GenerateOptions` (the port field + pi-ai maps to the header; agent calls always had it, ours never sent it). + Depth-2 DAG formed live (61 summaries: 47 leaves, 14 condensed, max depth + 2, over 4 swaps with billed usage). Caveat as tuning follow-up, not a loop + defect: Flash via Go is pathologically verbose on rung prompts (60k + output-cap hits on 2-3k-token sources), so those nodes fell back + mechanical, while OpenRouter Flash wrote clean prose on the same prompts. + +## Open questions for the owner + +- Whether the single-prompt `complete()` shape is acceptable when the target + reuses provider prefix cache only for verbatim replays of system prompt, + tools, and messages. +- Whether the sidecar store stays a file beside the session or moves onto the + target storage backend later. +- The package name, the config key spellings, and the compat pins against a + harness that warns of breaking changes. +- Whether item 10 ever advances past tracking, and whether the extracted + branch merges to main. The item 1 evaluation stays unrunnable until a + DeepSeek key exists; its numbers are a stop/confirm gate on the + transaction flow, not a precondition. + +## What this plan does not change + +The branch carries the adapter source, config, version, `.changeset/deepseek-lossless.md`, +plus docs site and package README edits. The B1-B3 bracket spike was proven on +scripted models; roadmap item 10 stays a tracking item, so no spike log +attaches to it beyond this plan. + +Spike verdict: the custom transaction flow is proven on scripted models +across all three production paths (seam auto, bracket manual, bracket auto +behind a flag), with the B path's regression suite pinning the default. +What scripted models cannot prove is live-harness behavior (real pressure +timing, real flush backends, real prefix-cache effects), so merging the +spike to main and reopening item 10 stay owner decisions. The recommended +next work is roadmap item 1, the long-context evaluation: it is the only +source of measured evidence for the rest of the list, and its numbers are +the stop/confirm gate on this transaction flow. diff --git a/docs/roadmap.md b/docs/roadmap.md index 31b1ae6..0fb1cb4 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,8 +4,9 @@ Forward-looking work only. What the current release does is in the [README](../README.md) and on the [docs site](https://stainless-code.com/lossless/); behavior lives in the code, and the peer comparisons with their verified cells are in -[`pi/comparison.mdx`](../apps/docs/content/reference/pi/comparison.mdx) for Pi and -[`opencode/comparison.mdx`](../apps/docs/content/reference/opencode/comparison.mdx) for OpenCode v2. Each item below names the evidence +[`pi/comparison.mdx`](../apps/docs/content/reference/pi/comparison.mdx) for Pi, +[`opencode/comparison.mdx`](../apps/docs/content/reference/opencode/comparison.mdx) for OpenCode v2, and +[`deepseek/comparison.mdx`](../apps/docs/content/reference/deepseek/comparison.mdx) for DeepSeek. Each item below names the evidence that would close it and what it costs. Nothing here is built before it has a measurement or a defect behind it. @@ -127,12 +128,13 @@ Where this implementation departs from the paper, and what it does not claim: 10. **A third consumer: DeepSeek.** OpenCode v2 shipped on its own adapter (`packages/opencode`); the live gate and the two host limits (a child-session reader, estimated summarizer usage) are recorded in that package's README. - The next consumer is the DeepSeek harness. It stays on the spike branch - until it is proven live and extracted on its own branch; the open questions - are its plugin surface, its reader substitute, and whether its summarizer - reports usage. The comparison has an audited page for each shipping adapter, - [Pi](../apps/docs/content/reference/pi/comparison.mdx) and - [OpenCode v2](../apps/docs/content/reference/opencode/comparison.mdx). + The DeepSeek harness adapter lives on `feat/deepseek` with live proof + recorded in `docs/deepseek-plan.md`; the remaining step is the merge + decision. The open questions are its plugin surface, its reader substitute, + and whether its summarizer reports usage. The comparison has an audited page + for each adapter, [Pi](../apps/docs/content/reference/pi/comparison.mdx), + [OpenCode v2](../apps/docs/content/reference/opencode/comparison.mdx), and + [DeepSeek](../apps/docs/content/reference/deepseek/comparison.mdx). 11. **The gallery asset.** A `video` (mp4, hover-autoplay) or `image` field per the Pi packages documentation: a long session, context filling, the swap notification, then `lcm_expand_query` recovering verbatim @@ -254,18 +256,47 @@ Each of these was verified in peer source and is worth doing only with a reason. today: every writer of provider text uses `error`, which is on the list. The item becomes live the first time a metrics writer puts foreign text under a new key. +- **Four grafts from the DeepSeek harness, none touching the DAG.** Cache-aligned + ladder inputs (ordered span replay, instruction trailing, so calls ride the + provider prefix cache); structured rung prompts (sectioned shape with verbatim + quoting rules inside the existing thorough/terse levels); untrusted framing + for past-history answers (labelled, immutable, never instructions); provider + usage reuse in estimation (believe reported usage over chars-per-token math, + worked under item 5). Each passes the three-property test: verbatim originals + stay reachable, child pointers never break, the model sees less. The merge + half of structured prompts carries one hard constraint: merging prose must + never drop child pointers. Deliberately not grafted: surface pruning (rewrites + live history this package promises verbatim; externalization already covers + the need) and goals/objectives (a second product, not a better memory). +- **Arena verdict on the grafts (three-lens viability pass, order 4, 3, 2, 1).** + Usage reuse first: it denominates every later spend claim in believed tokens, + through calibration gates only, never synthesized, stale-report guard stays. + Untrusted framing second: past-session answers only (same-session use would + train the model to ignore it), one or two sentences mirroring proven wording, + cosmetic until a live misread. Structured prompts third: section wording under + fidelity and oversize gates, merge as text concatenation with id-set equality + green, deterministic prepend stays the default. Cache inputs last, with a + correction to the entry above: deterministic ordering ships as hygiene, but + cache-riding claims need a message-replay prototype plus measured cacheRead + share plus a live key, because the single-message `complete()` envelope cannot + express prefix reuse. Kill gates: cacheRead share under 10 pct, oversize rate + up over 5 points, any orphan-child or fidelity regression, calibration null + rate above baseline. ## Names -**All three packages are unscoped.** `pi-lossless` is the Pi adapter, -`opencode-lossless` is the OpenCode v2 adapter, and `lossless-core` +**All five workspace packages are unscoped.** `pi-lossless` is the Pi adapter, +`opencode-lossless` is the OpenCode v2 adapter, `deepseek-lossless` is the +unshipped DeepSeek adapter with its `deepseek-lossless-web` preset companion, +and `lossless-core` is the engine. A `@stainless-code/` scope arrives only on a real collision, and -then it arrives once, for every package at the same time. `pi-lossless` 0.1.0 and `lossless-core` 0.1.0 are -published; `opencode-lossless` is pending its first release. The unscoped `lossless` on npm is an unrelated published +then it arrives once, for every package at the same time. `pi-lossless` 0.1.2, `lossless-core` 0.1.2, and +`opencode-lossless` 0.1.2 are published; `deepseek-lossless` 0.1.0 and +`deepseek-lossless-web` 0.1.0 are unshipped. The unscoped `lossless` on npm is an unrelated published package, which is what the harness prefix is for; and the docs app is already scoped and private (`@stainless-code/lossless-docs`), so scoping stays a decision available rather than one made by accident. What would force the scope -is a squat on any of the four names, or a cost that +is a squat on any of the five names, or a cost that the engine's adjacency to the unrelated `lossless` package inflicts on a consumer of the published engine. diff --git a/package.json b/package.json index 35fe3d3..fab50cd 100644 --- a/package.json +++ b/package.json @@ -20,7 +20,7 @@ "opencode:check": "bun scripts/opencode-live.ts", "opencode:smoke": "bun scripts/opencode-smoke.ts", "opencode:compact": "bun scripts/opencode-compact.ts", - "pack": "vp run --filter ./packages/core prepack && vp run --filter ./packages/pi prepack && vp run --filter ./packages/opencode prepack", + "pack": "vp run --filter ./packages/core prepack && vp run --filter ./packages/pi prepack && vp run --filter ./packages/deepseek prepack && vp run --filter ./packages/opencode prepack", "changeset": "changeset", "version": "changeset version && rm -f bun.lock && bun install && bun scripts/sync-skill-versions.ts && vp fmt packages/*/CHANGELOG.md", "release": "changeset publish", @@ -38,6 +38,24 @@ "vitest": "catalog:" }, "overrides": { + "@deepseek-ai/dsh-commands": { + "zod": "^4.6.6" + }, + "@deepseek-ai/dsh-llm": { + "zod": "^4.6.6" + }, + "@deepseek-ai/dsh-llm-retry": { + "zod": "^4.6.6" + }, + "@deepseek-ai/dsh-sandbox-policy": { + "zod": "^4.6.6" + }, + "@deepseek-ai/dsh-session-projection": { + "zod": "^4.6.6" + }, + "@deepseek-ai/dsh-token-meter": { + "zod": "^4.6.6" + }, "@earendil-works/pi-coding-agent": { "undici": "^8.12.2" }, diff --git a/packages/core/src/recall-view.ts b/packages/core/src/recall-view.ts index 5083173..cca48e5 100644 --- a/packages/core/src/recall-view.ts +++ b/packages/core/src/recall-view.ts @@ -9,7 +9,7 @@ import { type PatternScanOutcome, } from "./pattern-scan.ts"; import type { LcmStore, StoredMessage, SummaryNode } from "./store.ts"; -import { mechanicalToolNameOf } from "./util/summarizer-config.ts"; +import { mechanicalToolNameOf, toolNameOfText } from "./util/summarizer-config.ts"; /** * Every view masks stored text with the injected `redact` before it is cut down @@ -80,41 +80,48 @@ export function withToolScope(query: string, toolName: string): string { * only in the result. A trailing run of toolResult rows after a marker row * belongs to that call batch; any other role ends the run, as does a * toolResult row that names a different tool. Bracket text is checked - * directly so one row naming two tools anchors both, and - * mechanicalToolNameOf covers the `tool ` and payload forms. User and + * directly so one row naming two tools anchors both, + * mechanicalToolNameOf covers the `tool ` and payload forms, and + * toolNameOfText re-reads the marker lines without the mechanical tool-only + * gate so a call embedded in prose still anchors. User and * custom rows never open a scope, so quoting a marker in chat pulls in no * results; a mixed assistant block still opens through its literal marker. */ -export function toolScopeMessageIds( - store: LcmStore, - toolName: string, - opts?: { includeRemoved?: boolean; withinMessageIds?: ReadonlySet }, -): Set { - const want = toolName.trim().toLowerCase(); +/** The order-dependent scope state machine over already filtered rows. A + * REMOVED row never reaches the visible pass, so it cannot close the scope; + * the unfiltered pass sees it and closes, which preserves the pre-refactor + * visible-only behavior exactly. */ +function scopeIdsFromRows(rows: readonly StoredMessage[], want: string): Set { const out = new Set(); - if (want.length === 0) return out; - const scope = opts?.withinMessageIds; - const includeRemoved = opts?.includeRemoved === true; - const visible = store - .allMessages() - .filter( - (m) => - (includeRemoved || m.removedAt === undefined) && (scope === undefined || scope.has(m.id)), - ); let open = false; - for (const row of visible) { + const escaped = want.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const bracketTest = new RegExp(`\\[tool:${escaped}\\]`, "i"); + for (const row of rows) { + // A result body may quote another tool's marker further down, so only the + // opening line can open or close a scope for the tool that produced it. + const scopeText = + row.role === "toolResult" + ? (row.text.split("\n").find((line) => line.trim().length > 0) ?? "") + : row.text; + // toolNameOfText reads the same marker lines without the mechanical + // tool-only gate, so a call embedded in prose still anchors a scope. + const scoped = row.role !== "user" && row.role !== "custom"; + const textName = scoped ? toolNameOfText(row.text, row.role) : undefined; const namesTool = - row.role !== "user" && - row.role !== "custom" && - (row.text.toLowerCase().includes(`[tool:${want}]`) || - mechanicalToolNameOf({ text: row.text, payload: row.payload, role: row.role }) === want); + scoped && + (bracketTest.test(scopeText) || + mechanicalToolNameOf({ text: row.text, payload: row.payload, role: row.role }) === want || + textName === want); if (namesTool) { out.add(row.id); open = true; continue; } if (row.role === "toolResult" && open) { - const other = mechanicalToolNameOf({ text: row.text, payload: row.payload }) ?? ""; - if ((other.length > 0 && other !== want) || /\[tool:/i.test(row.text)) { + const other = + mechanicalToolNameOf({ text: row.text, payload: row.payload, role: row.role }) ?? + textName ?? + ""; + if ((other.length > 0 && other !== want) || /\[tool:/i.test(scopeText)) { open = false; continue; } @@ -126,6 +133,24 @@ export function toolScopeMessageIds( return out; } +export function toolScopeMessageIds( + store: LcmStore, + toolName: string, + opts?: { includeRemoved?: boolean; withinMessageIds?: ReadonlySet }, +): Set { + const want = toolName.trim().toLowerCase(); + if (want.length === 0) return new Set(); + const scope = opts?.withinMessageIds; + const includeRemoved = opts?.includeRemoved === true; + const visible = store + .allMessages() + .filter( + (m) => + (includeRemoved || m.removedAt === undefined) && (scope === undefined || scope.has(m.id)), + ); + return scopeIdsFromRows(visible, want); +} + export function grepView( store: LcmStore, redact: (text: string) => string, @@ -355,10 +380,17 @@ function toolScopedView( ...(params.dateStart === undefined ? {} : { dateStart: params.dateStart }), ...(params.dateEnd === undefined ? {} : { dateEnd: params.dateEnd }), }; - const visibleIds = toolScopeMessageIds(store, toolName, { - includeRemoved, - withinMessageIds: params.withinMessageIds, - }); + const want = toolName.trim().toLowerCase(); + // One table load; the order-dependent scope machine runs twice over + // in-memory arrays so a REMOVED row cannot close the visible scope. + const allRows = store.allMessages(); + const baseRows = + params.withinMessageIds === undefined + ? allRows + : allRows.filter((m) => params.withinMessageIds!.has(m.id)); + const visibleRows = includeRemoved ? baseRows : baseRows.filter((m) => m.removedAt === undefined); + const visibleIds = want.length === 0 ? new Set() : scopeIdsFromRows(visibleRows, want); + const allIds = want.length === 0 ? new Set() : scopeIdsFromRows(baseRows, want); let pattern = query; const runAll = (text: string, ids: ReadonlySet, removed: boolean) => scopedGrepAll(store, text, { includeRemoved: removed, ...roleOpts, scoped: ids }); @@ -386,10 +418,6 @@ function toolScopedView( const hits = all.slice(offset, offset + limit); if (hits.length === 0) { if (!includeRemoved) { - const allIds = toolScopeMessageIds(store, toolName, { - includeRemoved: true, - withinMessageIds: params.withinMessageIds, - }); let hidden: ReturnType = []; try { hidden = runAll(pattern, allIds, true); diff --git a/packages/core/src/util/metrics-sink.ts b/packages/core/src/util/metrics-sink.ts index 54ee404..1f5c8af 100644 --- a/packages/core/src/util/metrics-sink.ts +++ b/packages/core/src/util/metrics-sink.ts @@ -69,8 +69,58 @@ export function appendMetricRow(dir: string, row: Record): void } catch {} } +/** Filter for {@link readMetricRows}: narrows to one session and keeps only the newest rows. */ +export interface ReadMetricRowsOptions { + readonly session?: string; + readonly limit?: number; +} + /** Read rotated generations oldest-first plus the live log, skipping missing files and malformed lines. */ -export function readMetricRows(dir: string): unknown[] { +export function readMetricRows(dir: string, opts?: ReadMetricRowsOptions): unknown[] { + const session = + typeof opts?.session === "string" && opts.session.length > 0 ? opts.session : undefined; + let limit = + typeof opts?.limit === "number" && Number.isFinite(opts.limit) ? opts.limit : undefined; + if (limit !== undefined && limit < 0) limit = undefined; + if (session === undefined && limit === undefined) { + return readAllMetricRows(dir); + } + if (limit !== undefined && limit <= 0) return []; + const capped = limit === undefined ? undefined : Math.floor(limit); + const paths = metricPaths(dir); + const kept: unknown[] = []; + for (let i = paths.length - 1; i >= 0; i -= 1) { + let text: string; + try { + text = readFileSync(paths[i] as string, "utf8"); + } catch { + continue; + } + const lines = text.split("\n"); + for (let j = lines.length - 1; j >= 0; j -= 1) { + const trimmed = (lines[j] as string).trim(); + if (trimmed.length === 0) continue; + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch { + continue; + } + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) continue; + if (session !== undefined && (parsed as Record).session !== session) + continue; + kept.push(parsed); + if (capped !== undefined && kept.length >= capped) { + kept.reverse(); + return kept; + } + } + } + kept.reverse(); + return kept; +} + +function metricPaths(dir: string): string[] { const paths: string[] = []; const base = metricsFilePath(dir); try { @@ -84,6 +134,11 @@ export function readMetricRows(dir: string): unknown[] { (a, b) => Number(a.slice(a.lastIndexOf(".") + 1)) - Number(b.slice(b.lastIndexOf(".") + 1)), ); paths.push(base); + return paths; +} + +function readAllMetricRows(dir: string): unknown[] { + const paths = metricPaths(dir); const rows: unknown[] = []; for (const path of paths) { @@ -109,3 +164,84 @@ export function readMetricRows(dir: string): unknown[] { } return rows; } + +/** Running provider counts one ladder sums across calls. Host-free so every + * adapter tallies the same four legs without importing a provider SDK. */ +export interface UsageTally { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; +} + +/** Empty tally: no call has been added yet. */ +export function createUsageTally(): UsageTally { + return { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }; +} + +/** Add one call's counts. Legs are required numbers per the core Usage type. */ +export function addUsageTally( + tally: UsageTally, + delta: { input: number; output: number; cacheRead: number; cacheWrite: number }, +): void { + tally.input += delta.input; + tally.output += delta.output; + tally.cacheRead += delta.cacheRead; + tally.cacheWrite += delta.cacheWrite; +} + +/** Fold a tally into provider token names. Undefined when nothing was billed, + * so the caller keeps the absent usage signal instead of a zero row. */ +export function tokenUsageOfTally(tally: UsageTally): + | { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; + } + | undefined { + if (tally.input === 0 && tally.output === 0) return undefined; + return { + inputTokens: tally.input, + outputTokens: tally.output, + totalTokens: tally.input + tally.output, + cacheReadTokens: tally.cacheRead, + cacheWriteTokens: tally.cacheWrite, + }; +} + +/** Build the usage record with zeroed cost legs: nothing offline knows what a + * token costs here, so every cost leg is zero and the totals stay exact. */ +export function zeroCostUsageOfTokens( + usage: + | { + inputTokens: number; + outputTokens: number; + totalTokens?: number; + cacheReadTokens?: number; + cacheWriteTokens?: number; + } + | undefined, +): + | { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; + totalTokens: number; + cost: { input: number; output: number; cacheRead: number; cacheWrite: number; total: number }; + } + | undefined { + if (!usage) return undefined; + const input = usage.inputTokens; + const output = usage.outputTokens; + return { + input, + output, + cacheRead: usage.cacheReadTokens ?? 0, + cacheWrite: usage.cacheWriteTokens ?? 0, + totalTokens: usage.totalTokens ?? input + output, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }; +} diff --git a/packages/core/src/util/sidecar-store.ts b/packages/core/src/util/sidecar-store.ts index 862ab47..5e0dac7 100644 --- a/packages/core/src/util/sidecar-store.ts +++ b/packages/core/src/util/sidecar-store.ts @@ -142,4 +142,14 @@ export class SidecarStores { for (const store of this.stores.values()) store.close(); this.stores.clear(); } + + /** Forget one cached store without closing or touching disk; the caller closes first. */ + evict(sessionId: string): void { + this.stores.delete(sessionId); + } + + /** Forget every held store without closing or touching disk; the caller closes first so a live handle can stay open. */ + clear(): void { + this.stores.clear(); + } } diff --git a/packages/core/src/util/summarizer-config.ts b/packages/core/src/util/summarizer-config.ts index e9dadac..5cb47b3 100644 --- a/packages/core/src/util/summarizer-config.ts +++ b/packages/core/src/util/summarizer-config.ts @@ -71,10 +71,14 @@ export function mechanicalToolNameOf(message: { return toolNameOfPayload(message.payload); } -function toolNameOfText(text: string, role?: string): string | undefined { +/** The literal marker line a row carries, if any. Same line scan the + * mechanical decision reads, without the tool-only gate: a scope matcher + * calls this directly so a call embedded in prose still anchors. */ +export function toolNameOfText(text: string, role?: string): string | undefined { const head = text.match(/^\s*tool\s+([A-Za-z0-9_:\-/.]+)/i); if (head?.[1]) return head[1].toLowerCase(); - const lineMarker = /^\s*(?:\[tool:([^\]]+)\]|tool[_-]?call:\s*([A-Za-z0-9_:\-/.]+))/i; + const lineMarker = + /^\s*(?:\[tool:([^\]]+)\]|tool[_-]?call:\s*([A-Za-z0-9_:\-/.]+)|tool\s+([A-Za-z0-9_:\-/.]+))/i; if (role === "toolResult") { // A result body may quote another tool's marker further down, so only the // opening line can name the tool that produced it. @@ -85,7 +89,7 @@ function toolNameOfText(text: string, role?: string): string | undefined { } for (const line of text.split("\n")) { const hit = line.match(lineMarker); - const name = (hit?.[1] ?? hit?.[2])?.trim().toLowerCase(); + const name = (hit?.[1] ?? hit?.[2] ?? hit?.[3])?.trim().toLowerCase(); if (name && name.length > 0) return name; } return undefined; diff --git a/packages/core/src/util/token-window.ts b/packages/core/src/util/token-window.ts index d86bd78..30c8d21 100644 --- a/packages/core/src/util/token-window.ts +++ b/packages/core/src/util/token-window.ts @@ -1,9 +1,10 @@ /** Shared token and window math every harness adapter uses. The estimator * counts four characters per token, the overflow parser reads the window a * provider named, and the lookup and key helpers name a model the same way - * in config zones and metric rows. Pi already reads the estimator and the - * parser from the core entry; OpenCode reads the lookup and the key from - * here through its thin wrappers, so the two adapters cannot drift. */ + * in config zones and metric rows. Pi reads the estimator and the parser + * from the core entry; OpenCode and DeepSeek read the lookup, the key, and + * the range selector through their thin wrappers, so the adapters cannot + * drift from the contract stated here. */ /** Token math shared with adapters: `tokensFromChars` counts chars at four per token, `scaledBudget` calibrates a budget. */ export { scaledBudget, tokensFromChars } from "../estimate-tokens.ts"; @@ -114,6 +115,51 @@ export function createWindowLookup( }; } +/** One priced surface position the range selector reads. */ +export interface CompactableNode { + readonly seq: S; + readonly tokens: number; +} + +/** Inclusive seq range to compact. */ +export interface CompactableRange { + readonly start: S; + readonly end: S; +} + +/** The next range starting at `firstIdx` while retaining a priced recent tail + * and never splitting a tool-call/result pair. Adapters keep host-side + * checks beside their session; the accumulation, the admission, and the + * boundary walk live here. Null means no safe useful range exists: no nodes, + * the retained tail already reaches the head, or no balanced boundary is + * left. */ +export function selectCompactableRange( + nodes: readonly CompactableNode[], + firstIdx: number, + retainTokens: number, + isBalanced: (seq: S) => boolean, +): CompactableRange | null { + if (nodes.length === 0) return null; + let accumulated = 0; + let keepFromIdx = nodes.length; + for (let index = nodes.length - 1; index >= 0; index -= 1) { + accumulated += nodes[index]?.tokens ?? 0; + keepFromIdx = index; + if (accumulated >= retainTokens) break; + } + if (keepFromIdx <= firstIdx) return null; + while (keepFromIdx > firstIdx) { + const seq = nodes[keepFromIdx]?.seq; + if (seq !== undefined && isBalanced(seq)) break; + keepFromIdx -= 1; + } + if (keepFromIdx <= firstIdx) return null; + const start = nodes[firstIdx]?.seq; + const end = nodes[keepFromIdx - 1]?.seq; + if (start === undefined || end === undefined) return null; + return { start, end }; +} + function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null; } diff --git a/packages/core/test/recall-tool-scope-removed.test.ts b/packages/core/test/recall-tool-scope-removed.test.ts new file mode 100644 index 0000000..1fd88a2 --- /dev/null +++ b/packages/core/test/recall-tool-scope-removed.test.ts @@ -0,0 +1,58 @@ +import assert from "node:assert/strict"; + +import { test } from "vite-plus/test"; + +import { ingestEntries } from "../src/ingest.ts"; +import { grepView, toolScopeMessageIds } from "../src/recall-view.ts"; +import { redactSecrets } from "../src/redact.ts"; +import { LcmStore } from "../src/store.ts"; + +function hitIds(view: ReturnType): string[] { + assert.equal(view.ok, true); + return view.text + .split("\n") + .slice(1) + .filter((l) => l.startsWith("[")) + .map((l) => l.slice(1, l.indexOf("]"))); +} + +test("toolScope: a removed result naming another tool does not end the visible scope", () => { + const store = new LcmStore(":memory:"); + try { + ingestEntries(store, [ + { + entryId: "a00", + role: "assistant", + text: 'tool-call: read {"path": "/a"}', + timestamp: 1, + }, + { entryId: "r00", role: "toolResult", text: "firstword body here", timestamp: 2 }, + { + entryId: "r01", + role: "toolResult", + text: 'tool-call: bash {"cmd": "ls"}\nsecondword', + timestamp: 3, + }, + { entryId: "r02", role: "toolResult", text: "thirdword body here", timestamp: 4 }, + ]); + store.setRemoved(["r01"], 999); + const byEntry = new Map(store.allMessages().map((m) => [m.entryId, m.id])); + const visible = toolScopeMessageIds(store, "read"); + assert.ok(visible.has(byEntry.get("a00")!), "call row opens scope"); + assert.ok(visible.has(byEntry.get("r00")!), "plain result stays in"); + assert.ok(!visible.has(byEntry.get("r01")!), "removed row stays out"); + assert.ok( + visible.has(byEntry.get("r02")!), + "visible continuation after a removed other-tool row stays in", + ); + const unfiltered = toolScopeMessageIds(store, "read", { includeRemoved: true }); + assert.ok(unfiltered.has(byEntry.get("a00")!), "unfiltered call row opens scope"); + assert.ok(unfiltered.has(byEntry.get("r00")!), "unfiltered plain result stays in"); + assert.ok(!unfiltered.has(byEntry.get("r01")!), "other-tool row ends scope"); + assert.ok(!unfiltered.has(byEntry.get("r02")!), "rows after the other tool stay out"); + const kept = hitIds(grepView(store, redactSecrets, { query: "thirdword", toolName: "read" })); + assert.deepEqual(kept, ["r02"]); + } finally { + store.close(); + } +}); diff --git a/packages/core/test/recall-upgrades.test.ts b/packages/core/test/recall-upgrades.test.ts index 5b2dbd6..8510c10 100644 --- a/packages/core/test/recall-upgrades.test.ts +++ b/packages/core/test/recall-upgrades.test.ts @@ -21,6 +21,7 @@ import { import { redactSecrets } from "../src/redact.ts"; import { searchSessions } from "../src/sessions.ts"; import { LcmStore } from "../src/store.ts"; +import { mechanicalToolNameOf } from "../src/util/summarizer-config.ts"; function seedRoleTime(): LcmStore { const store = new LcmStore(":memory:"); @@ -704,6 +705,129 @@ test("toolScope: a user-quoted marker opens no scope but mixed assistant still m } }); +test("toolScope: a result quoting another marker below its opening line stays in scope", () => { + const store = new LcmStore(":memory:"); + try { + ingestEntries(store, [ + { + entryId: "a00", + role: "assistant", + text: 'tool-call: read {"path": "/a"}', + timestamp: 1, + }, + { + entryId: "r00", + role: "toolResult", + text: "quoteword body here\n[tool:bash] quoted further down", + timestamp: 2, + }, + { + entryId: "r01", + role: "toolResult", + text: "quotecallword body here\ntool-call: bash quoted further down", + timestamp: 3, + }, + ]); + const byEntry = new Map(store.allMessages().map((m) => [m.entryId, m.id])); + const scoped = toolScopeMessageIds(store, "read"); + assert.ok(scoped.has(byEntry.get("a00")!), "call row opens scope"); + assert.ok( + scoped.has(byEntry.get("r00")!), + "quoted bracket marker below the opening line opens no bash scope", + ); + assert.ok( + scoped.has(byEntry.get("r01")!), + "quoted call below the opening line closes no scope", + ); + const bash = toolScopeMessageIds(store, "bash"); + assert.equal(bash.size, 0, "no row names bash on its opening line"); + const kept = hitIds(grepView(store, redactSecrets, { query: "quoteword", toolName: "read" })); + assert.deepEqual(kept, ["r00"]); + } finally { + store.close(); + } +}); + +test("toolScope: an assistant prose-plus-tool-call row opens scope for its tool", () => { + const store = new LcmStore(":memory:"); + try { + ingestEntries(store, [ + { + entryId: "a00", + role: "assistant", + text: 'Here is why the change is safe.\ntool-call: read {"path": "/a"}', + timestamp: 1, + }, + { entryId: "r00", role: "toolResult", text: "prosecallword body here", timestamp: 2 }, + { + entryId: "a01", + role: "assistant", + text: "note: the log said tool-call: read mid-line", + timestamp: 3, + }, + { entryId: "r01", role: "toolResult", text: "midlineword body here", timestamp: 4 }, + ]); + const byEntry = new Map(store.allMessages().map((m) => [m.entryId, m.id])); + const scoped = toolScopeMessageIds(store, "read"); + assert.ok(scoped.has(byEntry.get("a00")!), "prose-plus-call row opens scope"); + assert.ok(scoped.has(byEntry.get("r00")!), "result after prose-plus-call stays in"); + assert.ok(!scoped.has(byEntry.get("a01")!), "mid-line quote opens no scope"); + assert.ok(!scoped.has(byEntry.get("r01")!), "result after a mid-line quote stays out"); + const kept = hitIds( + grepView(store, redactSecrets, { query: "prosecallword", toolName: "read" }), + ); + assert.deepEqual(kept, ["r00"]); + const missed = grepView(store, redactSecrets, { query: "midlineword", toolName: "read" }); + assert.equal(missed.ok, true); + assert.match(missed.text, /No matches/); + // The tier decision is untouched: the mixed row stays on the summarizer path. + assert.equal( + mechanicalToolNameOf({ + text: 'Here is why the change is safe.\ntool-call: read {"path": "/a"}', + role: "assistant", + }), + undefined, + ); + } finally { + store.close(); + } +}); + +test("toolScope: a result naming another tool-call ends the scope", () => { + const store = new LcmStore(":memory:"); + try { + ingestEntries(store, [ + { + entryId: "a00", + role: "assistant", + text: 'Here is why the change is safe.\ntool-call: read {"path": "/a"}', + timestamp: 1, + }, + { entryId: "r00", role: "toolResult", text: "firstword body here", timestamp: 2 }, + { + entryId: "r01", + role: "toolResult", + text: 'tool-call: bash {"cmd": "ls"}\nsecondword', + timestamp: 3, + }, + { entryId: "r02", role: "toolResult", text: "thirdword body here", timestamp: 4 }, + ]); + const byEntry = new Map(store.allMessages().map((m) => [m.entryId, m.id])); + const scoped = toolScopeMessageIds(store, "read"); + assert.ok(scoped.has(byEntry.get("a00")!), "prose-plus-call row opens scope"); + assert.ok(scoped.has(byEntry.get("r00")!), "plain result after the call stays in"); + assert.ok(!scoped.has(byEntry.get("r01")!), "result naming another tool ends scope"); + assert.ok(!scoped.has(byEntry.get("r02")!), "rows after the other tool stay out"); + const kept = hitIds(grepView(store, redactSecrets, { query: "firstword", toolName: "read" })); + assert.deepEqual(kept, ["r00"]); + const missed = grepView(store, redactSecrets, { query: "secondword", toolName: "read" }); + assert.equal(missed.ok, true); + assert.match(missed.text, /No matches/); + } finally { + store.close(); + } +}); + test("grepView: an unbalanced-quote query falls back to a phrase search", () => { const store = new LcmStore(":memory:"); try { diff --git a/packages/core/test/util-metrics-sink.test.ts b/packages/core/test/util-metrics-sink.test.ts new file mode 100644 index 0000000..20bf2c0 --- /dev/null +++ b/packages/core/test/util-metrics-sink.test.ts @@ -0,0 +1,65 @@ +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { test } from "vite-plus/test"; + +import { appendMetricRow, readMetricRows } from "../src/util/metrics-sink.ts"; +import { + addUsageTally, + createUsageTally, + tokenUsageOfTally, + zeroCostUsageOfTokens, +} from "../src/util/metrics-sink.ts"; + +function seedDir(): string { + const dir = mkdtempSync(join(tmpdir(), "metrics-sink-")); + appendMetricRow(dir, { session: "a", kind: "x", n: 1 }); + appendMetricRow(dir, { session: "b", kind: "x", n: 2 }); + appendMetricRow(dir, { session: "a", kind: "x", n: 3 }); + return dir; +} + +test("metrics-sink: no-session negative limit reads all rows", () => { + const dir = seedDir(); + const rows = readMetricRows(dir, { limit: -1 }); + assert.equal(rows.length, 3); +}); + +test("metrics-sink: session negative limit reads all matching rows", () => { + const dir = seedDir(); + const rows = readMetricRows(dir, { session: "a", limit: -5 }); + assert.equal(rows.length, 2); + assert.ok( + rows.every((row) => (row as Record).session === "a"), + JSON.stringify(rows), + ); +}); + +test("metrics-sink: limit zero still returns no rows", () => { + const dir = seedDir(); + assert.deepEqual(readMetricRows(dir, { limit: 0 }), []); + assert.deepEqual(readMetricRows(dir, { session: "a", limit: 0 }), []); +}); + +test("metrics-sink: empty tally folds to undefined usage", () => { + const tally = createUsageTally(); + assert.equal(tokenUsageOfTally(tally), undefined); +}); + +test("metrics-sink: tally adds one call and folds to provider names", () => { + const tally = createUsageTally(); + addUsageTally(tally, { input: 10, output: 5, cacheRead: 2, cacheWrite: 1 }); + const usage = tokenUsageOfTally(tally); + assert.deepEqual(usage, { + inputTokens: 10, + outputTokens: 5, + totalTokens: 15, + cacheReadTokens: 2, + cacheWriteTokens: 1, + }); + const zeroed = zeroCostUsageOfTokens(usage); + assert.equal(zeroed?.input, 10); + assert.equal(zeroed?.output, 5); +}); diff --git a/packages/core/test/util-token-window.test.ts b/packages/core/test/util-token-window.test.ts index b304d08..4a9d6f1 100644 --- a/packages/core/test/util-token-window.test.ts +++ b/packages/core/test/util-token-window.test.ts @@ -2,7 +2,7 @@ import assert from "node:assert/strict"; import { test } from "vite-plus/test"; -import { createWindowLookup, modelKey } from "../src/util/token-window.ts"; +import { createWindowLookup, modelKey, selectCompactableRange } from "../src/util/token-window.ts"; test("token window: modelKey spells the OpenCode providerID spelling", () => { assert.equal(modelKey({ providerID: "test", id: "m1" }), "test/m1"); @@ -121,3 +121,46 @@ test("token window: lookup filters invalid catalogue entries", async () => { assert.equal(await windowOf({ providerID: "test", id: "bad-zero" }), 200_000); assert.equal(await windowOf({ providerID: "test", id: "bad-missing-limit" }), 200_000); }); + +test("token window: empty nodes select no range", () => { + assert.equal( + selectCompactableRange([], 0, 100, () => true), + null, + ); +}); + +test("token window: retain reaching the head selects no range", () => { + const nodes = [ + { seq: "a", tokens: 10 }, + { seq: "b", tokens: 10 }, + ]; + assert.equal( + selectCompactableRange(nodes, 0, 100, () => true), + null, + ); +}); + +test("token window: unbalanced boundary selects no range", () => { + const nodes = [ + { seq: "a", tokens: 50 }, + { seq: "b", tokens: 50 }, + ]; + assert.equal( + selectCompactableRange(nodes, 0, 10, () => false), + null, + ); +}); + +test("token window: balanced boundary selects the head range", () => { + const nodes = [ + { seq: "a", tokens: 50 }, + { seq: "b", tokens: 50 }, + ]; + assert.deepEqual( + selectCompactableRange(nodes, 0, 10, (seq) => seq === "b"), + { + start: "a", + end: "a", + }, + ); +}); diff --git a/packages/deepseek-web/CHANGELOG.md b/packages/deepseek-web/CHANGELOG.md new file mode 100644 index 0000000..455e75c --- /dev/null +++ b/packages/deepseek-web/CHANGELOG.md @@ -0,0 +1 @@ +# deepseek-lossless-web diff --git a/packages/deepseek-web/LICENSE b/packages/deepseek-web/LICENSE new file mode 100644 index 0000000..0e8c8a8 --- /dev/null +++ b/packages/deepseek-web/LICENSE @@ -0,0 +1,13 @@ +MIT License + +This software is an original implementation of the Lossless Context Management +(LCM) architecture, specified in the paper "Lossless Context Management" by +Clint Ehrlich and Theodore Blackman, Voltropy PBC +(https://papers.voltropy.com/LCM). The paper is credited as the algorithmic +specification; all code in this repository was written from that spec. + +The "smart zone" framing (quality degradation at a token budget, not the +window limit) is credited to Matt Pocock, AI Coding Dictionary +(https://www.aihero.dev/ai-coding-dictionary/smart-zone). + +Copyright (c) 2026 stainless-code diff --git a/packages/deepseek-web/README.md b/packages/deepseek-web/README.md new file mode 100644 index 0000000..8e5985c --- /dev/null +++ b/packages/deepseek-web/README.md @@ -0,0 +1,40 @@ +# deepseek-lossless-web + +Lossless presets for the DeepSeek web surface: Lossless variants of all four +shipped agent presets (`standard`, `ptc`, `minimal`, `cordis`) whose +compaction backend runs the shared lossless core. Preset sessions keep their +catalog; the backend is swapped, with normalized punctuation and a preset +description added. + +## Install + +Install alongside `deepseek-lossless` on a preset-bearing profile (`web`, +`desktop`, or a custom profile built on `web-app`): + +```sh +dsh plugin --profile web add deepseek-lossless +dsh plugin --profile web add deepseek-lossless-web +``` + +Host-plane profiles (`headless`, `acp`, `sdk`, custom base profiles) do not +need this package: the host switch from `deepseek-lossless` alone covers +them. + +## Use + +Restart the profile, then select a Lossless preset for the session. The +preset backend borrows the shared host core and fails fast naming the host +row when it is absent. To switch back, select a shipped preset again. + +## Pins + +The four presets are byte-for-byte copies of the harness presets at +dsh v0.2.1-rc.3 (the version our peers admit) except the backend row, a +punctuation normalization the repo requires, and a config.description per +preset that the UI renders. Re-sync them when the peer floor moves: diff +each file against the new harness preset and carry over everything except +the backend row. + +## License + +MIT, see [LICENSE](./LICENSE). diff --git a/packages/deepseek-web/cordis.presets.patch.yml b/packages/deepseek-web/cordis.presets.patch.yml new file mode 100644 index 0000000..3779e7f --- /dev/null +++ b/packages/deepseek-web/cordis.presets.patch.yml @@ -0,0 +1,496 @@ +# Lossless presets: one `@deepseek-ai/dsh-agent-preset` declaration per +# shipped web preset, inserted after the web patch. Each entry is generated +# by scripts/sync-lossless-presets.ts from the pinned harness presets +# (dsh v0.2.1-rc.3, the runtime our peers admit) with exactly three +# deliberate differences: the compaction group backend becomes +# `deepseek-lossless/preset`, em dashes in copied plan-mode prose read +# as commas (this repo ships no em dashes), and each preset declares a +# config.description (the UI renders declaration config.description for +# non built-in ids, and the lossless ids have no locale key). Everything +# else is byte for byte the pinned harness preset, so preset sessions keep +# their catalog while compaction runs through the shared lossless core. +# +# Selecting a Lossless preset is the opt-in; the backend row ships enabled +# because the selection itself is the opt-in. The host row stays disabled +# by default, so enable it too: the realm backend borrows the shared core +# and fails fast naming the host row when it is absent. +# +- insert: + - id: preset-lossless-standard + name: "@deepseek-ai/dsh-agent-preset" + config: + id: lossless-standard + description: "Full coding agent with planning, subagents, and web tools. Compaction runs through the shared lossless core." + order: 11 + plugins: + - id: persona + name: "@deepseek-ai/dsh-persona" + config: + prefix: You are a coding agent powered by the {{model}} model. + - id: agent-instructions + name: "@deepseek-ai/dsh-agent-instructions" + config: + maxBytes: 65536 + - id: time-context + name: "@deepseek-ai/dsh-time-context" + - id: tool-bash + name: "@deepseek-ai/dsh-tool-bash" + disabled: !!js process.platform === 'win32' + - id: tool-pwsh + name: "@deepseek-ai/dsh-tool-pwsh" + disabled: !!js process.platform !== 'win32' + - id: tool-fs + name: "@deepseek-ai/dsh-tool-fs" + - id: tool-fs-search + name: "@deepseek-ai/dsh-tool-fs-search" + config: + sampleOverCapGlobResults: false + - id: tool-jobs + name: "@deepseek-ai/dsh-tool-jobs" + - id: tool-schedule + name: "@deepseek-ai/dsh-tool-schedule" + - id: skill-filesystem + name: "@deepseek-ai/dsh-skill-filesystem" + - id: tool-skill + name: "@deepseek-ai/dsh-tool-skill" + - id: command-goal + name: "@deepseek-ai/dsh-command-goal" + - id: tool-goal + name: "@deepseek-ai/dsh-tool-goal" + - id: planning + name: cordis:group + group: true + isolate: + planMode: true + config: + - id: plan-mode + name: "@deepseek-ai/dsh-plan-mode" + config: + section: | + You are in plan mode. Stay in plan mode until the user approves your plan through exit_plan_mode or switches the session mode. Imperative language to implement changes means plan the implementation, not execute it. A user's conversational agreement, including an answer confirming something you asked, approves nothing and does not end plan mode; fold the confirmed decision into the plan and submit it through exit_plan_mode. + + Explore first. Use non-mutating reads, searches, static analysis, and checks to ground the plan in the actual repository. Do not edit or write files, change configuration, run formatters or code generation that rewrites tracked files, commit, or otherwise carry out the plan. Prefer existing functions and patterns over new machinery. + + The tool catalog stays the same across modes for request-cache stability. These plan-mode rules override any later tool description or guidance that suggests using mutation tools; those tools remain listed to keep the tool catalog unchanged. Do not use todo_write to track this planning phase: it tracks implementation after an approved plan, while the plan itself belongs in exit_plan_mode. + + Resolve discoverable facts by inspection. Use ask_user_question only for user-owned choices or material ambiguity that inspection cannot answer. Do not ask the user where code lives or how current behavior works when you can find out. + + Make the plan decision-complete: state the goal and success criteria; group implementation changes by subsystem; identify public API, schema, and data-flow changes; cover edge cases, failure modes, tests, acceptance criteria, and explicit assumptions. Keep it concise enough to review but detailed enough that another engineer can implement it without making design decisions. + + When ready, call exit_plan_mode with the complete plan markdown, starting with a # title. Make exit_plan_mode the only and final tool call in that assistant response: it presents the plan for approval, and implementation begins only in a later step after approval. Do not paste the final plan as a plain reply or ask "should I proceed?" through prose or ask_user_question. If review rejects it, incorporate the feedback and present again. If the review channel is unavailable or aborted, stay in plan mode and ask the user to switch modes manually; do not proceed with implementation. + - id: compaction + name: cordis:group + group: true + isolate: + compaction: true + toolResultPruner: true + config: + - id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true + - id: command-compact + name: "@deepseek-ai/dsh-command-compact" + - id: tool-result-pruner + name: "@deepseek-ai/dsh-compaction-tool-result-pruner" + config: + thresholdChars: 8192 + headChars: 4096 + tailChars: 1024 + - id: delegation + name: cordis:group + group: true + isolate: + workflowEngine: true + config: + - id: tool-subagent-control + name: "@deepseek-ai/dsh-tool-subagent-control" + - id: tool-subagent-list-agents + name: "@deepseek-ai/dsh-tool-subagent-control/list-agents" + - id: tool-subagent + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: spawn + toolName: subagent + modelSelectionSettings: true + - id: tool-subagent-fork + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: fork + toolName: subagent_fork + - id: workflow-ptc + name: "@deepseek-ai/dsh-workflow-ptc" + config: + provider: spawn + - id: tool-workflow + name: "@deepseek-ai/dsh-tool-workflow" + - id: tool-ask-user + name: "@deepseek-ai/dsh-tool-ask-user" + - id: tool-todo + name: "@deepseek-ai/dsh-tool-todo" + config: + allowParallelInProgress: true + - id: tool-web + name: "@deepseek-ai/dsh-tool-web" + config: + fetch: true + searchTimeoutMs: 60000 + - id: present + name: "@deepseek-ai/dsh-tool-present" + - id: tool-plugin-manager + name: "@deepseek-ai/dsh-plugin-manager/tools" + disabled: true + - id: tool-plugin-market + name: "@deepseek-ai/dsh-plugin-market/tools" + disabled: true + +- insert: + - id: preset-lossless-ptc + name: "@deepseek-ai/dsh-agent-preset" + config: + id: lossless-ptc + description: "Full coding agent with PTC presentation mode. Compaction runs through the shared lossless core." + order: 12 + plugins: + - id: persona + name: "@deepseek-ai/dsh-persona" + config: + prefix: You are a coding agent powered by the {{model}} model. + - id: agent-instructions + name: "@deepseek-ai/dsh-agent-instructions" + config: + maxBytes: 65536 + - id: time-context + name: "@deepseek-ai/dsh-time-context" + - id: tool-bash + name: "@deepseek-ai/dsh-tool-bash" + disabled: !!js process.platform === 'win32' + - id: tool-pwsh + name: "@deepseek-ai/dsh-tool-pwsh" + disabled: !!js process.platform !== 'win32' + - id: tool-fs + name: "@deepseek-ai/dsh-tool-fs" + - id: tool-fs-search + name: "@deepseek-ai/dsh-tool-fs-search" + config: + sampleOverCapGlobResults: false + - id: tool-jobs + name: "@deepseek-ai/dsh-tool-jobs" + - id: tool-schedule + name: "@deepseek-ai/dsh-tool-schedule" + - id: skill-filesystem + name: "@deepseek-ai/dsh-skill-filesystem" + - id: tool-skill + name: "@deepseek-ai/dsh-tool-skill" + - id: command-goal + name: "@deepseek-ai/dsh-command-goal" + - id: tool-goal + name: "@deepseek-ai/dsh-tool-goal" + - id: planning + name: cordis:group + group: true + isolate: + planMode: true + config: + - id: plan-mode + name: "@deepseek-ai/dsh-plan-mode" + config: + section: | + You are in plan mode. Stay in plan mode until the user approves your plan through exit_plan_mode or switches the session mode. Imperative language to implement changes means plan the implementation, not execute it. A user's conversational agreement, including an answer confirming something you asked, approves nothing and does not end plan mode; fold the confirmed decision into the plan and submit it through exit_plan_mode. + + Explore first. Use non-mutating reads, searches, static analysis, and checks to ground the plan in the actual repository. Do not edit or write files, change configuration, run formatters or code generation that rewrites tracked files, commit, or otherwise carry out the plan. Prefer existing functions and patterns over new machinery. + + The tool catalog stays the same across modes for request-cache stability. These plan-mode rules override any later tool description or guidance that suggests using mutation tools; those tools remain listed to keep the tool catalog unchanged. Do not use todo_write to track this planning phase: it tracks implementation after an approved plan, while the plan itself belongs in exit_plan_mode. + + Resolve discoverable facts by inspection. Use ask_user_question only for user-owned choices or material ambiguity that inspection cannot answer. Do not ask the user where code lives or how current behavior works when you can find out. + + Make the plan decision-complete: state the goal and success criteria; group implementation changes by subsystem; identify public API, schema, and data-flow changes; cover edge cases, failure modes, tests, acceptance criteria, and explicit assumptions. Keep it concise enough to review but detailed enough that another engineer can implement it without making design decisions. + + When ready, call exit_plan_mode with the complete plan markdown, starting with a # title. Make exit_plan_mode the only and final tool call in that assistant response: it presents the plan for approval, and implementation begins only in a later step after approval. Do not paste the final plan as a plain reply or ask "should I proceed?" through prose or ask_user_question. If review rejects it, incorporate the feedback and present again. If the review channel is unavailable or aborted, stay in plan mode and ask the user to switch modes manually; do not proceed with implementation. + - id: compaction + name: cordis:group + group: true + isolate: + compaction: true + toolResultPruner: true + config: + - id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true + - id: command-compact + name: "@deepseek-ai/dsh-command-compact" + - id: tool-result-pruner + name: "@deepseek-ai/dsh-compaction-tool-result-pruner" + config: + thresholdChars: 8192 + headChars: 4096 + tailChars: 1024 + - id: delegation + name: cordis:group + group: true + isolate: + workflowEngine: true + config: + - id: tool-subagent-control + name: "@deepseek-ai/dsh-tool-subagent-control" + - id: tool-subagent-list-agents + name: "@deepseek-ai/dsh-tool-subagent-control/list-agents" + - id: tool-subagent + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: spawn + toolName: subagent + modelSelectionSettings: true + - id: tool-subagent-fork + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: fork + toolName: subagent_fork + - id: workflow-ptc + name: "@deepseek-ai/dsh-workflow-ptc" + disabled: true + config: + provider: spawn + - id: tool-workflow + name: "@deepseek-ai/dsh-tool-workflow" + disabled: true + - id: tool-ask-user + name: "@deepseek-ai/dsh-tool-ask-user" + - id: tool-todo + name: "@deepseek-ai/dsh-tool-todo" + config: + allowParallelInProgress: true + - id: tool-web + name: "@deepseek-ai/dsh-tool-web" + config: + fetch: true + searchTimeoutMs: 60000 + - id: tool-presentation + name: "@deepseek-ai/dsh-agent-tool-presentation" + config: + mode: ptc + - id: present + name: "@deepseek-ai/dsh-tool-present" + - id: tool-plugin-manager + name: "@deepseek-ai/dsh-plugin-manager/tools" + disabled: true + - id: tool-plugin-market + name: "@deepseek-ai/dsh-plugin-market/tools" + disabled: true + +- insert: + - id: preset-lossless-minimal + name: "@deepseek-ai/dsh-agent-preset" + config: + id: lossless-minimal + description: "Lightweight shell preset with a fixed persona and image reads. Compaction runs through the shared lossless core." + order: 13 + plugins: + - id: persona + name: "@deepseek-ai/dsh-persona" + config: + prefix: You are a helpful software engineer assistant. + complete: true + includeRuntimeContext: false + - id: tool-fs + name: "@deepseek-ai/dsh-tool-fs" + config: + read: false + readImage: true + write: false + edit: false + - id: compaction + name: cordis:group + group: true + isolate: + compaction: true + config: + - id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true + - id: command-compact + name: "@deepseek-ai/dsh-command-compact" + - id: persistent-shell + name: cordis:group + group: true + isolate: + terminals: true + config: + - id: pty + name: "@deepseek-ai/dsh-terminal" + - id: terminal-bash + name: "@deepseek-ai/dsh-terminal-bash" + disabled: !!js process.platform === 'win32' + config: + timeoutMs: 300000 + - id: persistent-bash + name: "@deepseek-ai/dsh-tool-bash-persistent" + disabled: !!js process.platform === 'win32' + config: + timeoutMs: 300000 + description: |- + Run commands in a bash shell + * When invoking this tool, the contents of the "command" parameter does NOT need to be XML-escaped. + * Network access depends on the task environment. Prefer configured mirrors/proxies when they are available. + * State is persistent across command calls and discussions with the user. + * To inspect a particular line range of a file, e.g. lines 10-25, try 'sed -n 10,25p /path/to/the/file'. + * Please avoid commands that may produce a very large amount of output. + * Please run long lived commands in the background, e.g. 'sleep 10 &' or start a server in the background. + - id: terminal-pwsh + name: "@deepseek-ai/dsh-terminal-bash" + disabled: !!js process.platform !== 'win32' + config: + shellDialect: pwsh + timeoutMs: 300000 + - id: persistent-pwsh + name: "@deepseek-ai/dsh-tool-pwsh-persistent" + disabled: !!js process.platform !== 'win32' + config: + timeoutMs: 300000 + description: |- + Run commands in a PowerShell shell + * When invoking this tool, the contents of the "command" parameter does NOT need to be XML-escaped. + * You don't have access to the internet via this tool. + * State is persistent across command calls and discussions with the user. + * Use native Windows paths (C:\...) and $env:NAME variables; this is PowerShell, not bash. + * Please avoid commands that may produce a very large amount of output. + * Please run long lived commands in the background, e.g. 'Start-Job' or start a server with Start-Process. + +- insert: + - id: preset-lossless-cordis + name: "@deepseek-ai/dsh-agent-preset" + config: + id: lossless-cordis + description: "Full coding agent with Cordis preset authoring tools. Compaction runs through the shared lossless core." + order: 14 + plugins: + # Same persona as `standard`: tool descriptions and the skill catalog carry + # the operating detail, and the skills own the preset-authoring rules. + - id: persona + name: "@deepseek-ai/dsh-persona" + config: + prefix: >- + You are a coding agent powered by the {{model}} model. + - id: agent-instructions + name: "@deepseek-ai/dsh-agent-instructions" + config: + maxBytes: 65536 + - id: time-context + name: "@deepseek-ai/dsh-time-context" + - id: tool-bash + name: "@deepseek-ai/dsh-tool-bash" + disabled: !!js process.platform === 'win32' + - id: tool-pwsh + name: "@deepseek-ai/dsh-tool-pwsh" + disabled: !!js process.platform !== 'win32' + - id: tool-fs + name: "@deepseek-ai/dsh-tool-fs" + - id: tool-fs-search + name: "@deepseek-ai/dsh-tool-fs-search" + config: + sampleOverCapGlobResults: false + - id: tool-jobs + name: "@deepseek-ai/dsh-tool-jobs" + - id: tool-schedule + name: "@deepseek-ai/dsh-tool-schedule" + - id: command-goal + name: "@deepseek-ai/dsh-command-goal" + - id: tool-goal + name: "@deepseek-ai/dsh-tool-goal" + - id: planning + name: cordis:group + group: true + isolate: + planMode: true + config: + - id: plan-mode + name: "@deepseek-ai/dsh-plan-mode" + config: + section: | + You are in plan mode. Stay in plan mode until the user approves your plan through exit_plan_mode or switches the session mode. Imperative language to implement changes means plan the implementation, not execute it. A user's conversational agreement, including an answer confirming something you asked, approves nothing and does not end plan mode; fold the confirmed decision into the plan and submit it through exit_plan_mode. + + Explore first. Use non-mutating reads, searches, static analysis, and checks to ground the plan in the actual repository. Do not edit or write files, change configuration, run formatters or code generation that rewrites tracked files, commit, or otherwise carry out the plan. Prefer existing functions and patterns over new machinery. + + The tool catalog stays the same across modes for request-cache stability. These plan-mode rules override any later tool description or guidance that suggests using mutation tools; those tools remain listed to keep the tool catalog unchanged. Do not use todo_write to track this planning phase: it tracks implementation after an approved plan, while the plan itself belongs in exit_plan_mode. + + Resolve discoverable facts by inspection. Use ask_user_question only for user-owned choices or material ambiguity that inspection cannot answer. Do not ask the user where code lives or how current behavior works when you can find out. + + Make the plan decision-complete: state the goal and success criteria; group implementation changes by subsystem; identify public API, schema, and data-flow changes; cover edge cases, failure modes, tests, acceptance criteria, and explicit assumptions. Keep it concise enough to review but detailed enough that another engineer can implement it without making design decisions. + + When ready, call exit_plan_mode with the complete plan markdown, starting with a # title. Make exit_plan_mode the only and final tool call in that assistant response: it presents the plan for approval, and implementation begins only in a later step after approval. Do not paste the final plan as a plain reply or ask "should I proceed?" through prose or ask_user_question. If review rejects it, incorporate the feedback and present again. If the review channel is unavailable or aborted, stay in plan mode and ask the user to switch modes manually; do not proceed with implementation. + - id: compaction + name: cordis:group + group: true + isolate: + compaction: true + toolResultPruner: true + config: + - id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true + - id: command-compact + name: "@deepseek-ai/dsh-command-compact" + - id: tool-result-pruner + name: "@deepseek-ai/dsh-compaction-tool-result-pruner" + config: + thresholdChars: 8192 + headChars: 4096 + tailChars: 1024 + - id: delegation + name: cordis:group + group: true + isolate: + workflowEngine: true + config: + - id: tool-subagent-control + name: "@deepseek-ai/dsh-tool-subagent-control" + - id: tool-subagent-list-agents + name: "@deepseek-ai/dsh-tool-subagent-control/list-agents" + - id: tool-subagent + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: spawn + toolName: subagent + modelSelectionSettings: true + - id: tool-subagent-fork + name: "@deepseek-ai/dsh-tool-subagent" + config: + provider: fork + toolName: subagent_fork + - id: workflow-ptc + name: "@deepseek-ai/dsh-workflow-ptc" + config: + provider: spawn + - id: tool-workflow + name: "@deepseek-ai/dsh-tool-workflow" + - id: tool-ask-user + name: "@deepseek-ai/dsh-tool-ask-user" + - id: tool-todo + name: "@deepseek-ai/dsh-tool-todo" + config: + allowParallelInProgress: true + - id: tool-web + name: "@deepseek-ai/dsh-tool-web" + config: + fetch: true + searchTimeoutMs: 60000 + - id: tool-cordis + name: "@deepseek-ai/dsh-tool-cordis" + - id: skill-filesystem + name: "@deepseek-ai/dsh-skill-filesystem" + config: + customSkillDirs: + - !!js process.getBuiltinModule('node:path').join(process.getBuiltinModule('node:path').dirname(process.getBuiltinModule('node:module').createRequire(baseUrl).resolve('@deepseek-ai/dsh-agent-preset/package.json')), 'skills') + - id: tool-skill + name: "@deepseek-ai/dsh-tool-skill" + - id: present + name: "@deepseek-ai/dsh-tool-present" + - id: tool-plugin-manager + name: "@deepseek-ai/dsh-plugin-manager/tools" + disabled: !!js "!ctx.get('profileContext')" + - id: tool-plugin-market + name: "@deepseek-ai/dsh-plugin-market/tools" + disabled: !!js "!ctx.get('profileContext')" diff --git a/packages/deepseek-web/docs/DETAIL.md b/packages/deepseek-web/docs/DETAIL.md new file mode 100644 index 0000000..93942d3 --- /dev/null +++ b/packages/deepseek-web/docs/DETAIL.md @@ -0,0 +1,29 @@ +# Lossless presets for DeepSeek web + +Four agent presets that mirror the shipped web presets. The compaction +backend is the shared lossless core instead of the base backend, with +punctuation normalized and a preset description added. +Selecting one is the opt-in; the backend row ships enabled because the +selection itself is the opt-in. + +## Install + +Install alongside the host package on a preset-bearing profile (`web`, +`desktop`, or a custom profile built on `web-app`): + +```sh +dsh plugin --profile web add deepseek-lossless +dsh plugin --profile web add deepseek-lossless-web +``` + +Host-plane profiles (`headless`, `acp`, `sdk`, custom base profiles) do not +need this package: the host switch from `deepseek-lossless` alone covers +them. + +## Enable + +No edits needed: the host bundle mounts its row enabled on install, so the +shared core is up once the profile restarts. Select a Lossless preset for +the session. A missing or disabled host row fails fast naming it instead of +running split-brain copies. To switch back, select a shipped preset again; +the base backend returns for those sessions. diff --git a/packages/deepseek-web/icon.svg b/packages/deepseek-web/icon.svg new file mode 100644 index 0000000..51892fe --- /dev/null +++ b/packages/deepseek-web/icon.svg @@ -0,0 +1 @@ + diff --git a/packages/deepseek-web/locale/en.json b/packages/deepseek-web/locale/en.json new file mode 100644 index 0000000..fc3f89c --- /dev/null +++ b/packages/deepseek-web/locale/en.json @@ -0,0 +1,7 @@ +{ + "meta": { + "title": "Lossless presets", + "description": "Lossless variants of the shipped agent presets whose compaction backend runs the shared lossless core.", + "detail": "docs/DETAIL.md" + } +} diff --git a/packages/deepseek-web/package.json b/packages/deepseek-web/package.json new file mode 100644 index 0000000..cf20c06 --- /dev/null +++ b/packages/deepseek-web/package.json @@ -0,0 +1,52 @@ +{ + "name": "deepseek-lossless-web", + "version": "0.1.0", + "description": "Lossless presets for the DeepSeek web surface: Lossless variants of every shipped agent preset whose compaction backend runs the shared lossless core.", + "keywords": [ + "coding-agent", + "context", + "deepseek", + "dsh-plugin", + "lcm", + "memory" + ], + "homepage": "https://github.com/stainless-code/lossless#readme", + "bugs": { + "url": "https://github.com/stainless-code/lossless/issues" + }, + "license": "MIT", + "author": "stainless-code", + "repository": { + "type": "git", + "url": "git+https://github.com/stainless-code/lossless.git", + "directory": "packages/deepseek-web" + }, + "files": [ + "cordis.presets.patch.yml", + "CHANGELOG.md", + "icon.svg", + "locale/*.json", + "docs/DETAIL.md" + ], + "type": "module", + "exports": { + "./package.json": "./package.json", + "./cordis.presets.patch.yml": "./cordis.presets.patch.yml", + "./locale/*.json": "./locale/*.json" + }, + "scripts": { + "prepublishOnly": "vp check" + }, + "peerDependencies": { + "deepseek-lossless": "^0.1.0" + }, + "icon": "./icon.svg", + "engines": { + "node": ">=22.19.0" + }, + "dsh": { + "bundle": { + "patch": "./cordis.presets.patch.yml" + } + } +} diff --git a/packages/deepseek-web/test/publish-surface.test.ts b/packages/deepseek-web/test/publish-surface.test.ts new file mode 100644 index 0000000..0cb902b --- /dev/null +++ b/packages/deepseek-web/test/publish-surface.test.ts @@ -0,0 +1,90 @@ +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { test } from "vite-plus/test"; + +const PACKAGE_DIR = fileURLToPath(new URL("..", import.meta.url)); +const REPO_ROOT = fileURLToPath(new URL("../../..", import.meta.url)); + +interface Manifest { + name?: string; + files?: string[]; + exports?: Record; + repository?: { directory?: string }; + icon?: string; + dsh?: { bundle?: { patch?: string } }; + peerDependencies?: Record; +} + +const manifest = JSON.parse(readFileSync(join(PACKAGE_DIR, "package.json"), "utf8")) as Manifest; + +test("publish surface: the web companion ships only the preset layer", () => { + assert.equal(manifest.name, "deepseek-lossless-web"); + assert.deepEqual((manifest.files ?? []).toSorted(), [ + "CHANGELOG.md", + "cordis.presets.patch.yml", + "docs/DETAIL.md", + "icon.svg", + "locale/*.json", + ]); + assert.equal(manifest.dsh?.bundle?.patch, "./cordis.presets.patch.yml"); + assert.equal(manifest.icon, "./icon.svg"); + assert.equal(manifest.repository?.directory, "packages/deepseek-web"); + for (const subpath of ["./package.json", "./cordis.presets.patch.yml", "./locale/*.json"]) { + assert.ok(manifest.exports?.[subpath] !== undefined, `${subpath} is exported`); + } + for (const name of [ + "cordis.presets.patch.yml", + "icon.svg", + "locale/en.json", + "docs/DETAIL.md", + "README.md", + "LICENSE", + "CHANGELOG.md", + ]) { + assert.ok(existsSync(join(PACKAGE_DIR, name)), `${name} ships in the tarball`); + } + assert.deepEqual( + readFileSync(join(PACKAGE_DIR, "LICENSE")), + readFileSync(join(REPO_ROOT, "LICENSE")), + ); +}); + +test("publish surface: the lossless presets mirror the pinned harness presets", () => { + const patch = readFileSync(join(PACKAGE_DIR, "cordis.presets.patch.yml"), "utf8"); + for (const preset of ["standard", "ptc", "minimal", "cordis"]) { + assert.ok( + patch.includes(`id: preset-lossless-${preset}`), + `lossless ${preset} preset is declared`, + ); + assert.ok(patch.includes(`id: lossless-${preset}`), `lossless ${preset} identity is set`); + } + assert.ok(!patch.includes("- id: compaction-basic"), "no preset keeps the base backend row"); + assert.ok(!patch.includes("\u2014"), "no em dashes in shipped prose"); + assert.ok( + patch.includes("deepseek-lossless/preset"), + "the realm backend resolves the sibling entrypoint", + ); +}); + +test("publish surface: every lossless preset declares a description", () => { + const patch = readFileSync(join(PACKAGE_DIR, "cordis.presets.patch.yml"), "utf8"); + for (const preset of ["standard", "ptc", "minimal", "cordis"]) { + const head = patch.match( + new RegExp(`id: lossless-${preset}\\n([\\s\\S]*?)\\n\\s*plugins:`), + )?.[1]; + assert.ok(head, `lossless ${preset} config head is present`); + const description = head.match(/description:\s*(.+)/)?.[1]?.trim(); + assert.ok( + description !== undefined && description.length > 0, + `lossless ${preset} declares a non-empty description`, + ); + assert.ok(!description.includes("\u2014"), `lossless ${preset} description has no em dash`); + } +}); + +test("publish surface: the web companion peers on the host package", () => { + assert.equal(manifest.peerDependencies?.["deepseek-lossless"], "^0.1.0"); +}); diff --git a/packages/deepseek/CHANGELOG.md b/packages/deepseek/CHANGELOG.md new file mode 100644 index 0000000..b123572 --- /dev/null +++ b/packages/deepseek/CHANGELOG.md @@ -0,0 +1 @@ +# deepseek-lossless diff --git a/packages/deepseek/LICENSE b/packages/deepseek/LICENSE new file mode 100644 index 0000000..0e8c8a8 --- /dev/null +++ b/packages/deepseek/LICENSE @@ -0,0 +1,13 @@ +MIT License + +This software is an original implementation of the Lossless Context Management +(LCM) architecture, specified in the paper "Lossless Context Management" by +Clint Ehrlich and Theodore Blackman, Voltropy PBC +(https://papers.voltropy.com/LCM). The paper is credited as the algorithmic +specification; all code in this repository was written from that spec. + +The "smart zone" framing (quality degradation at a token budget, not the +window limit) is credited to Matt Pocock, AI Coding Dictionary +(https://www.aihero.dev/ai-coding-dictionary/smart-zone). + +Copyright (c) 2026 stainless-code diff --git a/packages/deepseek/README.md b/packages/deepseek/README.md new file mode 100644 index 0000000..cdabdb8 --- /dev/null +++ b/packages/deepseek/README.md @@ -0,0 +1,133 @@ +# deepseek-lossless + +**Lossless Context Management (LCM) for the DeepSeek harness. Long sessions keep working past the point where the model would forget, and every verbatim message stays searchable and readable.** + +Mount backfill mirrors at most 500 pre-mount events per session by default, while live events keep mirroring past the cap. + +Apply it, and the adapter mirrors every session into its own SQLite sidecar. +The model reads that history back through `lcm_grep`, `lcm_describe`, +`lcm_expand`, and `lcm_expand_query`. The summarizer engine mounts as the +compaction backend: automatic pressure compaction stays off until `auto` opts +in, and `/lcm compact` runs one manual compaction through the DAG bracket. + +This package is the DeepSeek adapter over +[`lossless-core`](https://npmx.dev/package/lossless-core), the engine that +ingests the history and builds the summaries. + +## Pins + +The adapter compiles against these exact published versions: + +- `@deepseek-ai/dsh-agent@0.2.1-rc.3` +- `@deepseek-ai/dsh-commands@0.2.1-rc.3` +- `@deepseek-ai/dsh-compaction@0.2.1-rc.3` +- `@deepseek-ai/dsh-compaction-basic@0.2.1-rc.3` +- `@deepseek-ai/dsh-llm@0.2.1-rc.3` +- `@deepseek-ai/dsh-session@0.2.1-rc.3` +- `@deepseek-ai/dsh-session-projection@0.2.1-rc.3` (tests only) +- `@deepseek-ai/dsh-system-prompt@0.2.1-rc.3` (tests only) +- `@deepseek-ai/dsh-token-meter@0.2.1-rc.3` (tests only) +- `@deepseek-ai/dsh-tools@0.2.1-rc.3` +- `@deepseek-ai/cordis@4.0.5` + +## Install + +Opt-in per composition; nothing registers, mirrors, or mounts until `enabled` +is true: + +```ts +import { applyLossless } from "deepseek-lossless"; + +const stop = applyLossless(ctx, { enabled: true }); +``` + +## Config + +| Key | Default | +| ------------------------- | ---------------------------------------------------------------- | +| `enabled` | unset (stays off) | +| `auto` | `false` | +| `bracketAuto` | `false` | +| `sidecarDir` | `$DSH_HOME/lossless`, else `~/.dsh/lossless`, else `./.lossless` | +| `summaryTokens` | `1500` | +| `leafChunkTokens` | `3000` | +| `summarizationProvider` | routed model | +| `summarizationModel` | routed model | +| `thresholdRatio` | `0.8` | +| `retainTokens` | unset | +| `retainRatio` | `0.16` | +| `headroomTokens` | `65536` | +| `modelPolicies` | unset | +| `summarizerTokensPerCall` | `60000` | +| `summarizerTokensPerPass` | `250000` | +| `summarizerTokensPerHeal` | `120000` | +| `healSourceTokens` | `8` x `summaryTokens`, clamped `3000`..`24000` | +| `mechanicalTools` | `[]` (empty keeps the LLM path) | + +## `/lcm` + +- `status`: session counts plus the meter window and last pass outcome. +- `report`: rolls up the metrics rows. +- `dump [query] [--include-removed] [--save ]`: read-only sidecar dump; free-text query samples one FTS5 query, `--include-removed` keeps removed rows, `--save` writes the file at mode 0600. +- `backup [file]`: writes a `0600` store copy. +- `export [file]`: writes a `0600` transcript copy. +- `import `: dedups the named file. +- `gc [--apply] [--session ] [--older-than ]`: dry-runs age-based sidecar collection; `--apply` deletes. `--session` narrows to one sidecar. +- `migrate [--apply]`: lists older-schema sidecars; `--apply` migrates. +- `heal [--apply] [--max-nodes N]`: lists truncated summaries; `--apply` repairs. +- `diagnose [--save ]`: builds the redacted diagnose bundle. +- `compact`: one manual compaction. + +## Tools + +The model gets four recall tools: + +- `lcm_grep`: full-text search over this session. `query` is an FTS5 search + (words, quoted phrases, `AND`/`OR`, prefixes); `pattern` is a JavaScript + regular expression over the full text of every message. `scope` reaches past + sessions in the same sidecar directory (`sessions` and `all_sessions` are + equivalent because the directory is global). +- `lcm_describe`: inspect one summary (kind, depth, costs, children, parents, + full text) without expanding it. +- `lcm_expand`: read the verbatim originals a summary covers, or one message by + `entry_id`. `session` reads a past session's sidecar read-only, labelled as a + past session's history. +- `lcm_expand_query`: a bounded reader expands the stored originals and returns + findings only, so the raw transcript never enters the context. + +Full command, config, and file reference: [DeepSeek reference](https://stainless-code.com/lossless/reference/deepseek/tools). + +## Verify + +From a checkout of this repo: + +```sh +bun -e "await import('./packages/deepseek/dist/index.mjs')" # plugin loads from its build +vp test packages/deepseek/test/plugin.test.ts packages/deepseek/test/tools.test.ts # mount, /lcm status, recall tools; fakes only, no model calls +``` + +In a live session with the plugin mounted, `/lcm status` is the read-only +check and `lcm_grep` searches the mirrored history. + +## Documentation + +| | | +| ------------------------------------------------------------------------------- | --------------------------------------------------------- | +| [The docs site](https://stainless-code.com/lossless/) | How LCM behaves in a session, the config keys, the limits | +| [Command](https://stainless-code.com/lossless/reference/deepseek/command) | What each `/lcm` verb does | +| [Comparison](https://stainless-code.com/lossless/reference/deepseek/comparison) | The competing engines on one set of axes, verified cells | +| [CHANGELOG.md](./CHANGELOG.md) | Version history (changesets) | + +## Credits + +- **Lossless Context Management (LCM)**, the algorithmic specification this + package implements: "Lossless Context Management", Clint Ehrlich and + Theodore Blackman, Voltropy PBC, . +- **"Smart zone"**, the framing that model quality degrades at a token + budget, not at the context-window limit, Matt Pocock, AI Coding + Dictionary, . +- All code is an original implementation written from the paper's spec. + +## License + +MIT, see [LICENSE](./LICENSE). diff --git a/packages/deepseek/cordis.patch.yml b/packages/deepseek/cordis.patch.yml new file mode 100644 index 0000000..d03a2e1 --- /dev/null +++ b/packages/deepseek/cordis.patch.yml @@ -0,0 +1,15 @@ +# Installing this bundle switches the host-plane compaction backend over with +# no further edits: the base row goes quiet and our row mounts enabled. This +# matches the manager's own rule that installation enables a new bundle by +# default. The code-level default stays disabled (see the schema), so direct +# composition callers still opt in explicitly; only the shipped profile row +# opts in. To switch back, re-enable `compaction-basic` and disable +# `deepseek-lossless` from the Plugins page or the profile patch. +- id: compaction-basic + disabled: true + +- insert: + - id: deepseek-lossless + name: deepseek-lossless + config: + enabled: true diff --git a/packages/deepseek/docs/DETAIL.md b/packages/deepseek/docs/DETAIL.md new file mode 100644 index 0000000..94b1e20 --- /dev/null +++ b/packages/deepseek/docs/DETAIL.md @@ -0,0 +1,50 @@ +# Lossless memory for DeepSeek + +Long sessions keep working past the point where the model would forget. Aged turns become summary pointers the model can navigate, and every verbatim message stays in a local SQLite sidecar, searchable and readable in full. + +## Install + +Install is per profile. The host package covers host-plane profiles, +verified end to end: `headless`, `acp`, `sdk`, and custom profiles built +from the base bundle. + +```sh +dsh plugin --profile headless add deepseek-lossless +``` + +For a custom profile, replace `headless` with its name. Without a shipped +template, the profile starts from the base bundle. Preset-bearing profiles +(`web`, `desktop`, or a custom profile built on `web-app`) need the preset +companion as well: + +```sh +dsh plugin --profile web add deepseek-lossless +dsh plugin --profile web add deepseek-lossless-web +``` + +## Enable + +No edits needed: installing the bundle switches the host-plane backend +over, since the shipped patch quiets the base row and mounts our row +enabled. The code-level default stays disabled, so direct composition +callers still opt in explicitly. + +On a preset-bearing profile, install the companion, restart, then select a +Lossless preset for the session. The preset backend borrows the shared core +and fails fast naming the host row when it is absent. + +Restart the profile, then verify with `/lcm status`. To switch back, +re-enable `compaction-basic` and disable `deepseek-lossless` from the +Plugins page or the profile patch; removing the bundle restores the base +row the same way. + +## Configure + +All options ride the plugin config. The Plugins page validates them against the bundled schema. Common keys: + +- `auto`: automatic pressure compaction inside the turn. +- `sidecarDir`: one SQLite file per session. Defaults to `$DSH_HOME/lossless` when `DSH_HOME` names an absolute path, else `~/.dsh/lossless`, else `./.lossless`. +- `summaryTokens`, `leafChunkTokens`: summary budgets. +- `mechanicalTools`: tool names whose spans compact without a model call. + +Reference holds the full surface: command, tools, config, and files. diff --git a/packages/deepseek/icon.svg b/packages/deepseek/icon.svg new file mode 100644 index 0000000..51892fe --- /dev/null +++ b/packages/deepseek/icon.svg @@ -0,0 +1 @@ + diff --git a/packages/deepseek/locale/en.json b/packages/deepseek/locale/en.json new file mode 100644 index 0000000..bdee543 --- /dev/null +++ b/packages/deepseek/locale/en.json @@ -0,0 +1,7 @@ +{ + "meta": { + "title": "Lossless memory", + "description": "Long sessions keep working past the point where the model would forget, and every verbatim message stays searchable and readable.", + "detail": "docs/DETAIL.md" + } +} diff --git a/packages/deepseek/package.json b/packages/deepseek/package.json new file mode 100644 index 0000000..5060d02 --- /dev/null +++ b/packages/deepseek/package.json @@ -0,0 +1,91 @@ +{ + "name": "deepseek-lossless", + "version": "0.1.0", + "description": "Lossless Context Management for the DeepSeek harness: long sessions keep working past the point where the model would forget, and every verbatim message stays searchable and readable.", + "keywords": [ + "coding-agent", + "context", + "deepseek", + "dsh-plugin", + "lcm", + "memory", + "tanstack-intent" + ], + "homepage": "https://github.com/stainless-code/lossless#readme", + "bugs": { + "url": "https://github.com/stainless-code/lossless/issues" + }, + "license": "MIT", + "author": "stainless-code", + "repository": { + "type": "git", + "url": "git+https://github.com/stainless-code/lossless.git", + "directory": "packages/deepseek" + }, + "files": [ + "dist", + "CHANGELOG.md", + "skills", + "cordis.patch.yml", + "icon.svg", + "locale/*.json", + "docs/DETAIL.md" + ], + "type": "module", + "main": "./dist/index.mjs", + "exports": { + ".": { + "types": "./dist/index.d.mts", + "default": "./dist/index.mjs" + }, + "./preset": { + "types": "./dist/preset.d.mts", + "default": "./dist/preset.mjs" + }, + "./package.json": "./package.json", + "./cordis.patch.yml": "./cordis.patch.yml", + "./locale/*.json": "./locale/*.json" + }, + "scripts": { + "pack:watch": "vp pack --watch", + "prepack": "vp pack", + "prepublishOnly": "vp check && vp test && intent validate" + }, + "dependencies": { + "@deepseek-ai/schemastery": "^3.18.5" + }, + "devDependencies": { + "@deepseek-ai/cordis": "4.0.5", + "@deepseek-ai/dsh-agent": "0.2.1-rc.3", + "@deepseek-ai/dsh-commands": "0.2.1-rc.3", + "@deepseek-ai/dsh-compaction": "0.2.1-rc.3", + "@deepseek-ai/dsh-compaction-basic": "0.2.1-rc.3", + "@deepseek-ai/dsh-llm": "0.2.1-rc.3", + "@deepseek-ai/dsh-session": "0.2.1-rc.3", + "@deepseek-ai/dsh-session-projection": "0.2.1-rc.3", + "@deepseek-ai/dsh-system-prompt": "0.2.1-rc.3", + "@deepseek-ai/dsh-token-meter": "0.2.1-rc.3", + "@deepseek-ai/dsh-tools": "0.2.1-rc.3", + "lossless-core": "workspace:*" + }, + "peerDependencies": { + "@deepseek-ai/cordis": "*", + "@deepseek-ai/dsh-agent": "^0.2.1-rc.3", + "@deepseek-ai/dsh-commands": "^0.2.1-rc.3", + "@deepseek-ai/dsh-compaction": "^0.2.1-rc.3", + "@deepseek-ai/dsh-compaction-basic": "^0.2.1-rc.3", + "@deepseek-ai/dsh-llm": "^0.2.1-rc.3", + "@deepseek-ai/dsh-session": "^0.2.1-rc.3", + "@deepseek-ai/dsh-token-meter": "^0.2.1-rc.3", + "@deepseek-ai/dsh-tools": "^0.2.1-rc.3" + }, + "icon": "./icon.svg", + "engines": { + "node": ">=22.19.0" + }, + "dsh": { + "bundle": { + "patch": "./cordis.patch.yml" + } + } +} diff --git a/packages/deepseek/skills/deepseek-lossless/SKILL.md b/packages/deepseek/skills/deepseek-lossless/SKILL.md new file mode 100644 index 0000000..4ffb282 --- /dev/null +++ b/packages/deepseek/skills/deepseek-lossless/SKILL.md @@ -0,0 +1,76 @@ +--- +name: deepseek-lossless +description: "Use when a DeepSeek session has deepseek-lossless applied: search compacted history with lcm_grep, lcm_describe, lcm_expand, or lcm_expand_query; run /lcm status, report, dump, backup, export, import, gc, migrate, heal, diagnose, or compact; or tune the compaction thresholds." +license: MIT +metadata: + type: composition + library: "deepseek-lossless" + library_version: "0.1.0" +requires: [] +sources: + - stainless-code/lossless:packages/deepseek/src +--- + +# deepseek-lossless + +`applyLossless` composes lossless memory into a harness context: every session +mirrors into its own SQLite sidecar, the summarizer engine mounts as the +compaction backend, and the composition registers four recall tools plus `/lcm`. +Nothing registers, mirrors, or mounts until `enabled` is true. Past the +pressure point the engine compacts automatically when `auto` is on; `/lcm +compact` compacts on demand. + +## Install + +```ts +import { applyLossless } from "deepseek-lossless"; + +const stop = applyLossless(ctx, { enabled: true }); +``` + +Enable on one composition at a time; the returned disposer stops the mirror, +the tools, and the command. + +## Commands + +- `/lcm status`: session counts plus the meter window and last pass outcome. +- `/lcm report`: this session's metrics rows. +- `/lcm dump [query] [--include-removed] [--save ]`: read-only sidecar dump; free-text query samples one FTS5 query, `--include-removed` keeps removed rows, `--save` writes the file at mode 0600. +- `/lcm backup [file]`: copy this session's sidecar (mode 0600). +- `/lcm export [file]`: write the verbatim transcript and summaries as JSONL (mode 0600). +- `/lcm import `: restore rows from an export. +- `/lcm gc [--apply] [--session ] [--older-than ]`: dry-run age-based sidecar collection; `--apply` deletes. `--session` narrows to one sidecar. +- `/lcm migrate [--apply]`: dry-run the older-schema list; `--apply` migrates. +- `/lcm heal [--apply] [--max-nodes N]`: dry-run the truncated list; `--apply` spends model calls. +- `/lcm diagnose [--save ]`: redacted, pasteable diagnostics bundle. +- `/lcm compact`: one manual compaction on demand. + +## Recall tools + +- `lcm_grep`: full-text search over this session. `query` is FTS5 (words, quoted phrases, `AND`/`OR`, prefixes); `pattern` is a JavaScript regex over the full text of every message, case-insensitive unless `case_sensitive` is set. `scope: "sessions"` or `"all_sessions"` searches past sessions in the same sidecar directory; the two are equivalent because the directory is global. Caps: 20 stores, 40 hits, 10 per store, 1.5s deadline. +- `lcm_describe`: inspect one summary (kind, depth, costs, children, parents, full text) without expanding it. +- `lcm_expand`: read the verbatim originals a summary covers, or one entry by `entry_id`. `session` reads a past session's sidecar read-only; the text is labelled as a past session's history. +- `lcm_expand_query`: a bounded reader expands the stored originals and returns findings only, so the raw transcript never enters the context. Ask for verbatim quotes in `prompt` when the exact words matter. + +## Config essentials + +```json +{ "enabled": true, "sidecarDir": "./.lossless", "auto": false } +``` + +- `sidecarDir` holds one SQLite file per session; deleting one removes LCM recall only. WAL and shared-memory companions, exports, backups, and the live session log keep their text. Stop the session first, then remove the sidecar plus its companions and any exports or backups. +- `auto: false` (default) runs no automatic pressure compaction; `/lcm compact` works regardless. `bracketAuto: true` routes automatic compaction through the DAG bracket. +- `summarizationProvider` / `summarizationModel` unset follow the routed model. +- `thresholdRatio` (default `0.8`) sets the window fraction where pressure compacts; `retainRatio` (default `0.16`) keeps that fraction of recent context verbatim. `retainTokens` replaces the fraction with an absolute budget; the two are mutually exclusive. +- `summaryTokens` (default `1500`) targets one summary; `leafChunkTokens` (default `3000`) caps one leaf chunk before it splits. +- `headroomTokens` (default `65536`) reserves pressure headroom past the routed output reservation; `modelPolicies` unset merges exact provider and model overrides per routed target. +- Budgets: `summarizerTokensPerCall`, `summarizerTokensPerPass`, `summarizerTokensPerHeal`; `healSourceTokens` unset derives 8x `summaryTokens`, clamped to 3000 through 24000. +- `mechanicalTools` (default `[]`) compacts named tool spans without a model call; ingest still stores every message verbatim. + +## Common mistakes + +- **Expecting anything with `enabled` unset.** The master flag defaults off, so no mirror, no tools, and no command exist until one composition sets it true. +- **Expecting automatic compaction by default.** `auto` defaults false; past the pressure point nothing compacts until `auto` is on or `/lcm compact` runs. +- **Forgetting the `scope` equivalence.** `scope: "sessions"` and `"all_sessions"` read the same global sidecar directory; neither reaches outside it. +- **Expecting the raw transcript from the reader.** `lcm_expand_query` returns answers from the stored originals, never the transcript; ask for verbatim quotes when the exact words matter. +- **Looking for a delete command.** There is no delete tool. Removing that session sidecar file removes LCM recall only; WAL and shared-memory companions, exports, backups, and the live session log keep their text. Stop the session first, then remove the sidecar plus its companions and any exports or backups. diff --git a/packages/deepseek/src/command.ts b/packages/deepseek/src/command.ts new file mode 100644 index 0000000..f9f85ae --- /dev/null +++ b/packages/deepseek/src/command.ts @@ -0,0 +1,119 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { CommandInvocation, CommandResult } from "@deepseek-ai/dsh-commands"; +import { CommandDefinitionId } from "@deepseek-ai/dsh-commands/brand"; +import { ManualCompactionError } from "@deepseek-ai/dsh-compaction"; + +import { + handleBackup, + handleDiagnose, + handleDump, + handleExport, + handleGc, + handleHeal, + handleImport, + handleMigrate, + handleReport, + handleStatus, + type LcmCommandDeps, +} from "./lifecycle.ts"; + +export type { LcmCommandDeps } from "./lifecycle.ts"; + +const USAGE = + "Usage: /lcm [status|report|dump [--save ] [--include-removed] [query]|backup [file]|export [file]|import |gc [--apply] [--session ] [--older-than ]|migrate [--apply]|heal [--apply] [--max-nodes N]|diagnose [--save ]|compact]"; + +function assertNever(value: never): never { + throw new Error(`unknown manual compaction error code: ${String(value)}`); +} + +function expectedFailure(error: ManualCompactionError): CommandResult { + switch (error.code) { + case "busy": + return { + kind: "error", + text: "Compaction is unavailable because this process has an active compaction, or the agent is not idle.", + }; + case "cancelled": + return { kind: "error", text: "Compaction cancelled." }; + case "changed": + return { + kind: "error", + text: "The history selected for compaction changed before it could be replaced. The attempt is recorded in the session log.", + }; + case "summary": + return { + kind: "error", + text: "Compaction could not produce a useful summary. The attempt is recorded in the session log.", + }; + case "commit": + return { + kind: "error", + text: "Compaction did not finish cleanly; some session history may have changed. Inspect the current session state before retrying.", + }; + case "persistence": + return { + kind: "error", + text: "Compaction finished, but the session could not be saved.", + }; + default: + return assertNever(error.code); + } +} + +async function handleCompact(ctx: Context, invocation: CommandInvocation): Promise { + try { + const result = await ctx.compaction.compactNow( + invocation.agent, + invocation.signal, + invocation.commandId, + ); + if (result === null) return { kind: "success", text: "No compactable history yet." }; + return { + kind: "success", + text: `Compacted ${result.shadowedSeqs.length} history items (~${result.shadowedTokenCount} tokens).`, + sourceEventSeq: result.summarySeq, + }; + } catch (error: unknown) { + if (invocation.signal.aborted) return { kind: "error", text: "Compaction cancelled." }; + if (error instanceof ManualCompactionError) return expectedFailure(error); + throw error; + } +} + +/** The `/lcm` command: read-only operator verbs, mutating verbs with dry-run + * defaults, plus one manual compaction. Subcommand routing is manual dispatch + * inside this one handler: the command registry rejects a second registration + * under the same name, so `/lcm report` and friends arrive here as raw input. */ +export function lcmCommandDefinition(ctx: Context, deps: LcmCommandDeps) { + return { + definitionId: CommandDefinitionId("deepseek-lossless"), + name: "lcm", + description: + "Lossless session memory: status, report, dump, backup, export, import, gc, migrate, heal, diagnose, compact", + input: { + hint: "[status|report|dump|backup|export|import|gc|migrate|heal|diagnose|compact] ...", + }, + handler: (invocation: CommandInvocation): Promise => { + const tokens = invocation.rawInput.trim().split(/\s+/).filter(Boolean); + const verb = tokens[0] ?? ""; + const args = tokens.slice(1); + if (verb === "" || verb === "status") return handleStatus(ctx, deps, invocation); + if (verb === "report") return Promise.resolve(handleReport(deps, invocation)); + if (verb === "dump") return Promise.resolve(handleDump(deps, invocation, args)); + if (verb === "backup") return Promise.resolve(handleBackup(deps, invocation, args)); + if (verb === "export") return Promise.resolve(handleExport(deps, invocation, args)); + if (verb === "import") return Promise.resolve(handleImport(deps, invocation, args)); + if (verb === "gc") return Promise.resolve(handleGc(deps, invocation, args)); + if (verb === "migrate") return Promise.resolve(handleMigrate(deps, invocation, args)); + if (verb === "heal") return handleHeal(ctx, deps, invocation, args); + if (verb === "diagnose") return handleDiagnose(ctx, deps, invocation, args); + if (verb === "compact") return handleCompact(ctx, invocation); + return Promise.resolve({ kind: "error", text: USAGE }); + }, + }; +} + +/** Register `/lcm` on a composition with a commands service. Returns its disposer. */ +export function registerLcmCommand(ctx: Context, deps: LcmCommandDeps): () => void { + return ctx.commands.register(lcmCommandDefinition(ctx, deps)); +} diff --git a/packages/deepseek/src/core.ts b/packages/deepseek/src/core.ts new file mode 100644 index 0000000..d2f5695 --- /dev/null +++ b/packages/deepseek/src/core.ts @@ -0,0 +1,143 @@ +import { resolve } from "node:path"; + +import { SidecarStores, redactSecrets, type LcmStore } from "lossless-core"; + +import { checkpointMemory, type CheckpointMemory } from "./engine.ts"; +import { MessageIndex } from "./ingest.ts"; +import { createRecord } from "./metrics.ts"; + +/** Directory default shared with the loader schema. */ +export const DEFAULT_SIDECAR_DIR = "./.lossless"; + +/** Per-session sidecars: stores delegate to the shared core seam while each + * session keeps its own live message index beside its store. */ +export class SessionSidecars extends SidecarStores { + private readonly indexes = new Map(); + + constructor(dir: string) { + super({ dir }); + } + + entryFor(sessionId: string): { store: LcmStore; index: MessageIndex } { + const store = this.for(sessionId); + let index = this.indexes.get(sessionId); + if (!index) { + index = new MessageIndex(); + this.indexes.set(sessionId, index); + } + return { store, index }; + } + + peekIndex(sessionId: string): MessageIndex | undefined { + return this.indexes.get(sessionId); + } + + storesMap(): Map { + const map = new Map(); + for (const id of this.ids()) { + const store = this.peek(id); + if (store) map.set(id, store); + } + return map; + } + + override clear(): void { + super.clear(); + this.indexes.clear(); + } + + override evict(sessionId: string): void { + super.evict(sessionId); + this.indexes.delete(sessionId); + } +} + +/** Shared composition state: one per sidecar directory per process. The host + * row owns it (creates on mount, closes on dispose). Preset-realm backends + * borrow it and never close it. */ +export interface LosslessCore { + /** Resolved sidecar directory this core writes. */ + readonly dir: string; + readonly sidecars: SessionSidecars; + readonly record: (row: Record) => void; + readonly memory: CheckpointMemory; + /** Redaction for recall surfaces. */ + readonly redact: typeof redactSecrets; +} + +const cores = new Map(); + +/** Resolve the registry key so host and realm rows naming the same directory + * share one core even when one side passes a relative path. */ +export function resolveCoreDir(dir: string): string { + return resolve(dir); +} + +/** Whether this exact core currently occupies the registry. A repeat dispose + * uses it to tell a drained-and-deleted core (safe to sweep) from a live + * shared or replaced one (hands off). */ +export function isCoreHeld(core: LosslessCore): boolean { + const held = cores.get(core.dir); + return held !== undefined && held.core === core; +} + +/** Tool names that skip the model. One name or a list; blanks and repeats + * drop, and an empty result keeps the LLM path. */ +export function readMechanicalTools(raw: unknown): string[] | undefined { + const list = typeof raw === "string" ? [raw] : Array.isArray(raw) ? raw : undefined; + if (list === undefined) return undefined; + const seen = new Set(); + const out: string[] = []; + for (const entry of list) { + if (typeof entry !== "string") continue; + const name = entry.trim(); + if (name.length === 0) continue; + const key = name.toLowerCase(); + if (seen.has(key)) continue; + seen.add(key); + out.push(name); + } + return out.length === 0 ? undefined : out; +} + +/** Borrow or create the core for a directory. The first mount creates it; + * every mount, host or realm, holds one ownership count. Realm rows fail fast + * when no core exists instead of running a split-brain copy. */ +export function acquireCore(dir: string, owner: boolean): LosslessCore { + const key = resolveCoreDir(dir); + const held = cores.get(key); + if (held !== undefined) { + held.owners += 1; + return held.core; + } + if (!owner) { + throw new Error( + `deepseek-lossless: no shared core for "${dir}". Enable the deepseek-lossless host row (enabled: true) in the same profile so the mirror, tools, and sidecars mount first.`, + ); + } + const sidecars = new SessionSidecars(key); + const core: LosslessCore = { + dir: key, + sidecars, + record: createRecord(key), + memory: checkpointMemory((sessionId) => sidecars.peek(sessionId)), + redact: redactSecrets, + }; + cores.set(key, { core, owners: 1 }); + return core; +} + +/** Release one ownership of a core. Tied to the acquired core object, not the + * directory: a stale disposer from a previous mount cannot release the + * replacement. Each disposer must also call this at most once (guarded at the + * call site), so a double dispose after a remount cannot spend the + * replacement's ownership. Returns true when this release dropped the last + * count; the caller then runs the shared guarded close. */ +export function releaseCore(core: LosslessCore): boolean { + const held = cores.get(core.dir); + if (held === undefined || held.core !== core || held.owners <= 0) return false; + held.owners -= 1; + if (held.owners > 0) return false; + cores.delete(core.dir); + return true; +} diff --git a/packages/deepseek/src/dag-commit.ts b/packages/deepseek/src/dag-commit.ts new file mode 100644 index 0000000..f673156 --- /dev/null +++ b/packages/deepseek/src/dag-commit.ts @@ -0,0 +1,627 @@ +import { randomUUID } from "node:crypto"; +import { isDeepStrictEqual } from "node:util"; + +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import type { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import { + CompactionId, + ManualCompactionError, + compactCheckpointSource, + toolPairingBalancedAfter, + toolPairingBalancedBefore, +} from "@deepseek-ai/dsh-compaction"; +import { createUserMessage, errorChain } from "@deepseek-ai/dsh-llm"; +import type { ContentBlock } from "@deepseek-ai/dsh-llm"; +import { SessionSeq } from "@deepseek-ai/dsh-session"; +import type { Session, SessionEvent } from "@deepseek-ai/dsh-session"; +import type { TokenMeasurement, TokenMeter, TokenSurfaceNode } from "@deepseek-ai/dsh-token-meter"; +import { selectCompactableRange, type CompactionSpanEntry } from "lossless-core"; + +import { backfillSession, historySourceFrom } from "./ingest.ts"; +import type { LadderOutcome, LosslessEngineDeps } from "./ladder.ts"; +import { + SpanAlreadyMemoryError, + resolveSummarizationTarget, + runLadder, + seqOfEntry, + withSummaryRefs, +} from "./ladder.ts"; +import type { DshRoute } from "./model-host.ts"; + +/** Deps for the DAG bracket: the ladder deps, the composition the ladder + * runs inside, and the summarization fallback the target resolves from. The + * token meter is read off the composition, which the host harness provides; + * the import is type-only, so the runtime boundary stays honest. */ +export interface DagCommitDeps extends LosslessEngineDeps { + readonly ctx: Context; + /** Empty means unset: resolution falls through to the routed request, then + * the agent options, exactly like the seam. */ + readonly summarizationProvider: string; + readonly summarizationModel: string; +} + +/** Inclusive surface-node span to condense, mirroring the seam's start/end. */ +export interface DagCommitSelection { + readonly start: SessionSeq; + readonly end: SessionSeq; +} + +export interface DagCommitOptions { + /** Null compacts an idle session (manual taxonomy); any other value + * requires an open turn and reports that turn as the lifecycle owner. */ + readonly owner: string | null; + readonly stability: "whole-surface" | "selected-span"; + readonly sourceCommandId?: CommandId; + readonly flush?: () => Promise; +} + +/** Bracket result: the seam's CompactionResult fields plus the DAG node + * records. The singular summary fields name the condensed record (the event + * directly before the checkpoint). The session log holds exactly one + * compaction/summary per bracket; leaves stay sidecar only. summarySeqs + * holds the single condensed seq, leafSeqs remains empty for the seam shape, + * and the sidecar leaf and condensed ids stay on the result for metrics. The + * seam shape is kept so the engine override and the /lcm command read this + * result unchanged. */ +export interface DagCommitResult { + readonly compactionId: CompactionId; + readonly sourceCommandId?: CommandId; + readonly startSeq: SessionSeq; + readonly summarySeq: SessionSeq; + readonly endSeq: SessionSeq; + readonly summary: ContentBlock[]; + readonly shadowedRange: { readonly start: SessionSeq; readonly end: SessionSeq }; + readonly shadowedSeqs: SessionSeq[]; + readonly shadowedTokenCount: number; + /** Single condensed seq; the only compaction/summary in the session log. */ + readonly summarySeqs: readonly SessionSeq[]; + /** Always empty: leaves stay sidecar only, never session log records. */ + readonly leafSeqs: readonly SessionSeq[]; + /** Sidecar ids stored by the ladder run, kept for metrics. */ + readonly leafSummaryIds: readonly number[]; + /** Sidecar ids stored by the ladder run, kept for metrics. */ + readonly condensedSummaryIds: readonly number[]; +} + +/** Stability failure inside the bracket: the span moved under the ladder. + * Plays the seam's internal SurfaceChangedError role in the taxonomy. */ +export class DagStabilityError extends Error { + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = "DagStabilityError"; + } +} + +/** Checkpoint framing copied from the seam so a DAG checkpoint reads like + * one: dsh-compaction-basic@0.1.7-alpha.2 lib/index.js:234-235, 280. */ +const DAG_SUMMARY_OPEN_TAG = ""; +const DAG_SUMMARY_CLOSE_TAG = ""; +const DAG_CHECKPOINT_PREAMBLE = + "This is an automatically generated checkpoint condensing an earlier span of the conversation to free up context. Treat the captured context as established background and build on it without restating it. Continue the task directly from the messages that follow, without acknowledging this checkpoint."; + +type StartEvent = Extract; + +interface DagValidated { + readonly start: SessionSeq; + readonly end: SessionSeq; + readonly startIdx: number; + readonly endIdx: number; + readonly shadowedSeqs: SessionSeq[]; +} + +interface DagPrepared extends DagValidated { + readonly sessionId: string; + readonly target: DshRoute; + readonly measurement: TokenMeasurement; + readonly selectedNodes: TokenSurfaceNode[]; + readonly shadowedTokenCount: number; + readonly shadowedRouteTokenCount: number; + readonly span: CompactionSpanEntry[]; +} + +type DagStage = "summary" | "commit"; + +/** One custom compaction transaction: the seam's bracket (lock check, open, + * stability, shrink check, commit, close, failure taxonomy) with the + * lossless ladder as the summarizer and a single condensed summary record. + * Structure mirrors compactSurfaceRegion line for line + * (dsh-compaction-basic@0.1.7-alpha.2 lib/index.js:449-521); the two + * deliberate divergences are marked B1 below. */ +export async function commitDagTransaction( + deps: DagCommitDeps, + agent: Agent, + selection: DagCommitSelection, + options: DagCommitOptions, + signal?: AbortSignal, +): Promise { + const session = agent.session; + if (options.owner === null) signal?.throwIfAborted(); + const validated = validateDagRegion(session, selection.start, selection.end); + const entryState = inspectDagEntryState(session); + assertDagInactive(entryState.unmatchedStart, entryState.latestEndSeedSeq, "compaction"); + let owner: number | null; + if (options.owner === null) { + if (entryState.openTurn !== null) { + throw new ManualCompactionError( + "busy", + "manual compaction: the session already has an open turn", + ); + } + owner = null; + } else { + if (entryState.openTurn === null) { + throw new Error( + "compactRegion: no open turn: automatic compaction events must be enclosed in a turn", + ); + } + owner = entryState.openTurn; + } + const compactionId = CompactionId(randomUUID()); + const lifecycle = { + compactionId, + ...(options.sourceCommandId === undefined ? {} : { sourceCommandId: options.sourceCommandId }), + turn: owner, + }; + const startEvent = session.append("compaction/start", lifecycle); + const assertStable = + options.stability === "whole-surface" ? assertWholeSurfaceUnchanged : assertSelectedSpanStable; + let failure: { error: unknown; stage: DagStage } | undefined; + let flushFailure: unknown; + let result: DagCommitResult | undefined; + let closed = false; + let closing = false; + let stage: DagStage = "summary"; + try { + const prepared = await prepareDagCommit(deps, agent, session, validated); + signal?.throwIfAborted(); + const ladder = await runLadder( + deps.ctx, + deps, + prepared.target, + prepared.sessionId, + prepared.span, + signal, + ); + const condensed = frameCondensed(deps, prepared, ladder, compactionId, options.sourceCommandId); + if (options.owner === null) signal?.throwIfAborted(); + assertStable(deps.ctx.tokenMeter, session, prepared); + stage = "commit"; + const pending = commitDagBody(session, startEvent, prepared, ladder, condensed); + closing = true; + const endEvent = session.append("compaction/end", lifecycle); + closed = true; + result = { ...pending, endSeq: endEvent.seq }; + // Note only after the end marker lands: earlier notes survive failed + // shrink, stability, and commit paths as phantom checkpoints. + deps.memory?.note(prepared.sessionId, condensed.condensedText); + } catch (error) { + failure = { error, stage: closing ? "commit" : stage }; + if (!closing) { + closing = true; + try { + session.append("compaction/end", { ...lifecycle, error: errorChain(error) }); + closed = true; + } catch (closeError) { + failure = { error: closeError, stage: "commit" }; + } + } + } + if (closed && options.flush !== undefined) { + try { + await options.flush(); + } catch (error) { + flushFailure = error; + } + } + if (options.owner === null) signal?.throwIfAborted(); + if (failure !== undefined) { + if (options.owner === null) throwDagFailure(failure); + throw failure.error; + } + if (flushFailure !== undefined) { + throw new ManualCompactionError( + "persistence", + "manual compaction durability checkpoint failed", + { + cause: flushFailure, + }, + ); + } + if (result === undefined) throw new Error("compaction committed without a result"); + // One row per bracket commit: the condensed seq plus the sidecar leaf and + // condensed ids, so the checkpoint stays countable in the metrics log + // without leaf records in the session log. + deps.record?.({ + ts: Date.now(), + event: "lcm", + kind: "compaction-checkpoint", + session: String(agent.session.id), + summarySeqs: [...result.summarySeqs], + leafSeqs: [...result.leafSeqs], + leaf: result.leafSummaryIds.length, + condensed: result.condensedSummaryIds.length, + leafSummaryIds: [...result.leafSummaryIds], + condensedSummaryIds: [...result.condensedSummaryIds], + }); + return result; +} + +/** Classify one closed manual attempt without weakening cancellation + * precedence. Mirrors throwManualFailure (lib/index.js:523-528). */ +function throwDagFailure(failure: { error: unknown; stage: DagStage }): never { + if (failure.error instanceof SpanAlreadyMemoryError) throw failure.error; + if (failure.stage === "commit") { + throw new ManualCompactionError("commit", "manual compaction did not commit cleanly", { + cause: failure.error, + }); + } + if (failure.error instanceof DagStabilityError) { + throw new ManualCompactionError( + "changed", + "the compacted history changed during manual compaction", + { + cause: failure.error, + }, + ); + } + throw new ManualCompactionError( + "summary", + "manual compaction could not produce a smaller summary", + { + cause: failure.error, + }, + ); +} + +/** Reject a durable unmatched compaction marker unless a later + * constructor-seed boundary proves its owner belongs to an earlier session + * lifecycle. Mirrors assertCompactionInactive (lib/index.js:535-542). */ +function assertDagInactive( + unmatchedStart: StartEvent | undefined, + latestEndSeedSeq: SessionSeq | undefined, + stage: string, +): void { + if (unmatchedStart === undefined) return; + if (latestEndSeedSeq !== undefined && latestEndSeedSeq > unmatchedStart.seq) return; + throw new ManualCompactionError( + "busy", + `${stage}: compaction already in progress; the session compaction lock is already active`, + ); +} + +/** Resolve the next range starting at the first non-system surface node + * while retaining a priced recent tail and never splitting an assistant + * tool-call/result pair. The index math lives in the core seam's + * selectCompactableRange (which mirrors the host seam at + * lib/index.js:409-448); this wrapper keeps the fail-closed surface check + * and the system-head offset. */ +export function selectDagRange( + session: Session, + measurement: TokenMeasurement, + retainTokens: number, +): DagCommitSelection | null { + const pricedNodes = measurement.nodes; + const surfaceNodes = session.surface.nodes; + if ( + surfaceNodes.length !== pricedNodes.length || + surfaceNodes.some((seq, index) => seq !== pricedNodes[index]?.seq) + ) { + throw new Error( + "lossless dag commit: token-meter surface does not match the current session surface", + ); + } + const head = surfaceNodes[0]; + if (head === undefined) return null; + // A system/message at surface node 0 is the retained head, never inside + // the range; without one the range starts at node 0. + const firstIdx = dagSystemHead(session, head) === undefined ? 0 : 1; + return selectCompactableRange( + surfaceNodes.map((seq, index) => ({ seq, tokens: pricedNodes[index]?.tokens ?? 0 })), + firstIdx, + retainTokens, + (seq) => toolPairingBalancedBefore(session, seq), + ); +} + +function dagSystemHead(session: Session, headSeq: SessionSeq) { + const head = session.eventAt(headSeq); + return head?.type === "system/message" ? head : undefined; +} + +/** Validate one requested surface-position span before the bracket opens. + * Mirrors validateSurfaceRegion (lib/index.js:549-565). */ +function validateDagRegion(session: Session, start: SessionSeq, end: SessionSeq): DagValidated { + const nodes = session.surface.nodes; + const startIdx = nodes.indexOf(start); + const endIdx = nodes.indexOf(end); + const startNode = nodes[startIdx]; + const endNode = nodes[endIdx]; + if (startNode === undefined) + throw new Error(`dag commit: start seq ${start} not found in surface`); + if (endNode === undefined) throw new Error(`dag commit: end seq ${end} not found in surface`); + if (startIdx > endIdx) { + throw new Error( + `dag commit: start seq ${start} (position ${startIdx}) is after end seq ${end} (position ${endIdx}) on the surface`, + ); + } + if (!toolPairingBalancedBefore(session, startNode)) { + throw new Error( + `dag commit: start seq ${start} is not a balanced boundary (would split a step's tool-call/result pair)`, + ); + } + if (!toolPairingBalancedAfter(session, endNode)) { + throw new Error( + `dag commit: end seq ${end} is not a balanced boundary (would split a step, or the step is still open)`, + ); + } + return { start, end, startIdx, endIdx, shadowedSeqs: nodes.slice(startIdx, endIdx + 1) }; +} + +/** Snapshot pricing, resolve the summarization target, and map the span + * through the sidecar entry ids. Mirrors prepareCompaction + * (lib/index.js:567-579); the span mapping follows the seam's spanOf through + * entry ids the mirror records, never through text matching. */ +async function prepareDagCommit( + deps: DagCommitDeps, + agent: Agent, + session: Session, + validated: DagValidated, +): Promise { + const meter = deps.ctx.tokenMeter; + const measurement = meter.measure(session); + const selectedNodes = measurement.nodes.slice(validated.startIdx, validated.endIdx + 1); + if ( + selectedNodes.length !== validated.shadowedSeqs.length || + selectedNodes.some((node, index) => node.seq !== validated.shadowedSeqs[index]) + ) { + throw new DagStabilityError( + "lossless dag commit: selected surface changed before summarization began", + ); + } + const sessionId = String(session.id); + const target = resolveSummarizationTarget( + { + summarizationProvider: deps.summarizationProvider, + summarizationModel: deps.summarizationModel, + }, + agent, + ); + if (!target) { + throw new Error( + "lossless compaction: no provider/model available for summarization: set both engine summarization fields, route one request, or set both agent options fields", + ); + } + const store = deps.store(sessionId); + if (!store) throw new Error(`lossless compaction: no sidecar store for session ${sessionId}`); + const buildSpan = (): CompactionSpanEntry[] => { + // Core reads a span as a storage range, so frontierFrom must name the + // seq minimum. Catch-up backfill lands older seqs on newer rows, so + // rowid order no longer matches narrative order. + const items: Array<{ seq: number; entry: CompactionSpanEntry }> = []; + for (const seq of validated.shadowedSeqs) { + // Entry ids mirror the ingest projection, so the seq maps to its row without touching the live message index. + const entryId = `${session.id}:${seq}`; + const row = store.messageByEntryId(entryId); + if (!row) throw new Error(`lossless dag commit: sidecar holds no row for ${entryId}`); + items.push({ + seq: seqOfEntry(entryId), + entry: { + entryId, + role: row.role, + text: row.text, + ...(row.payload === undefined ? {} : { payload: row.payload }), + }, + }); + } + items.sort((a, b) => a.seq - b.seq); + return items.map((item) => item.entry); + }; + const base = { + ...validated, + sessionId, + target, + measurement, + selectedNodes: [...selectedNodes], + shadowedTokenCount: selectedNodes.reduce((total, node) => total + node.heuristicTokens, 0), + shadowedRouteTokenCount: selectedNodes.reduce((total, node) => total + node.tokens, 0), + }; + try { + return { ...base, span: buildSpan() }; + } catch (error) { + if (!isUnmappedSpanError(error)) throw error; + const source = historySourceFrom(deps.ctx); + if (source === undefined) throw error; + await backfillSession( + session, + store, + source, + deps.index(sessionId), + deps.record === undefined ? {} : { record: deps.record }, + ); + return { ...base, span: buildSpan() }; + } +} + +function isUnmappedSpanError(error: unknown): boolean { + return ( + error instanceof Error && + (error.message.includes("has no sidecar row; the span is not fully mirrored") || + error.message.includes("sidecar holds no row for ")) + ); +} + +interface CondensedCommit { + readonly checkpointMessage: ReturnType; + readonly condensedText: string; + readonly condensedBlocks: ContentBlock[]; +} + +/** Frame the condensed checkpoint and run the shrink check. Mirrors the + * framing half of summarizeCompaction (lib/index.js:581-606); leaves stay + * sidecar only, so no leaf record is read or written here. */ +function frameCondensed( + deps: DagCommitDeps, + prepared: DagPrepared, + ladder: LadderOutcome, + compactionId: ReturnType, + sourceCommandId: CommandId | undefined, +): CondensedCommit { + const condensedText = withSummaryRefs(ladder.summaryText, [ + ...ladder.leafSummaryIds, + ...ladder.condensedSummaryIds, + ]); + const condensedBlocks: ContentBlock[] = [{ type: "text", text: condensedText }]; + const checkpointMessage = createUserMessage({ + content: [ + { type: "text", text: `${DAG_CHECKPOINT_PREAMBLE}\n\n${DAG_SUMMARY_OPEN_TAG}` }, + { type: "text", text: condensedText }, + { type: "text", text: DAG_SUMMARY_CLOSE_TAG }, + ], + source: compactCheckpointSource(compactionId, sourceCommandId), + }); + const framedSummaryTokenCount = deps.ctx.tokenMeter.estimateMessage(checkpointMessage); + if (framedSummaryTokenCount >= prepared.shadowedRouteTokenCount) { + throw new Error( + `summary is not smaller than the shadowed content (${framedSummaryTokenCount} estimated framed tokens >= ${prepared.shadowedRouteTokenCount})`, + ); + } + return { checkpointMessage, condensedText, condensedBlocks }; +} + +/** Append the single condensed summary record and the replacement body + * without yielding, preserving the seam's shadow-price adjacency contract + * (lib/index.js:627-675; compaction/summary doc at types/types.d.ts). + * Leaves stay sidecar only: the condensed text cites their ids, so a + * harness reload sees exactly one summary per bracket. */ +function commitDagBody( + session: Session, + startEvent: StartEvent, + prepared: DagPrepared, + ladder: LadderOutcome, + condensed: CondensedCommit, +): Omit { + const compactionId = startEvent.data.compactionId; + const sourceCommandId = + startEvent.data.sourceCommandId === undefined + ? {} + : { sourceCommandId: startEvent.data.sourceCommandId }; + const summaryEvent = session.append("compaction/summary", { + compactionId, + ...sourceCommandId, + summary: condensed.condensedBlocks, + rawOutput: condensed.condensedBlocks, + shadowedRange: { start: prepared.start, end: prepared.end }, + shadowedSeqs: [...prepared.shadowedSeqs], + shadowedTokenCount: prepared.shadowedTokenCount, + provider: ladder.provider, + model: ladder.model, + ...(ladder.usage === undefined ? {} : { usage: ladder.usage }), + }); + session.append("user/message", condensed.checkpointMessage, { + surfaceOp: { op: "replace", startSeq: prepared.start, endSeq: prepared.end }, + sourceEventSeqs: [startEvent.seq, summaryEvent.seq, ...prepared.shadowedSeqs], + }); + return { + compactionId, + ...sourceCommandId, + startSeq: startEvent.seq, + summarySeq: summaryEvent.seq, + summary: condensed.condensedBlocks, + shadowedRange: { start: prepared.start, end: prepared.end }, + shadowedSeqs: [...prepared.shadowedSeqs], + shadowedTokenCount: prepared.shadowedTokenCount, + summarySeqs: [summaryEvent.seq], + leafSeqs: [], + leafSummaryIds: [...ladder.leafSummaryIds], + condensedSummaryIds: [...ladder.condensedSummaryIds], + }; +} + +/** Reject a summary prepared against any earlier surface generation. + * Mirrors assertWholeSurfaceUnchanged (lib/index.js:608-610). */ +function assertWholeSurfaceUnchanged( + meter: TokenMeter, + session: Session, + prepared: DagPrepared, +): void { + if (!isDeepStrictEqual(meter.measure(session).nodes, prepared.measurement.nodes)) { + throw new DagStabilityError( + "lossless dag commit: session surface changed during summarization", + ); + } +} + +/** Require only that the selected span remain the same present, contiguous, + * equally priced, balanced replacement target. Mirrors + * assertSelectedSpanStable (lib/index.js:616-626). */ +function assertSelectedSpanStable( + meter: TokenMeter, + session: Session, + prepared: DagPrepared, +): void { + let current: DagValidated; + try { + current = validateDagRegion(session, prepared.start, prepared.end); + } catch (error) { + throw new DagStabilityError( + "lossless dag commit: the selected span is no longer a valid replacement target", + { cause: error }, + ); + } + if (!isDeepStrictEqual([...current.shadowedSeqs], [...prepared.shadowedSeqs])) { + throw new DagStabilityError( + "lossless dag commit: the selected span changed during summarization", + ); + } + if ( + !isDeepStrictEqual( + meter.measure(session).nodes.slice(current.startIdx, current.endIdx + 1), + prepared.selectedNodes, + ) + ) { + throw new DagStabilityError( + "lossless dag commit: the selected span was rewritten during summarization", + ); + } +} + +/** Latest bracket and turn state walking the log tailward. Mirrors + * inspectCompactionEntryState (lib/index.js:705-741). */ +function inspectDagEntryState(session: Session): { + readonly openTurn: number | null; + readonly unmatchedStart: StartEvent | undefined; + readonly latestEndSeedSeq: SessionSeq | undefined; +} { + let openTurn: number | null = null; + let openTurnStateKnown = false; + let unmatchedStart: StartEvent | undefined; + let startKnown = false; + let latestEndSeedSeq: SessionSeq | undefined; + for (let seq = session.seq - 1; seq >= 0; seq -= 1) { + const event = session.eventAt(SessionSeq(seq)); + if (event === undefined) + throw new Error(`lossless dag commit: session log has no event at seq ${seq}`); + if (latestEndSeedSeq === undefined && event.type === "session/end-seed") { + latestEndSeedSeq = event.seq; + } + if (!startKnown) { + if (event.type === "compaction/start") { + unmatchedStart = event; + startKnown = true; + } else if (event.type === "compaction/end") { + startKnown = true; + } + } + if (!openTurnStateKnown) { + if (event.type === "turn/start") { + openTurn = event.data.turn; + openTurnStateKnown = true; + } else if (event.type === "turn/end") { + openTurnStateKnown = true; + } + } + if (openTurnStateKnown && startKnown && latestEndSeedSeq !== undefined) break; + } + return { openTurn, unmatchedStart, latestEndSeedSeq }; +} diff --git a/packages/deepseek/src/default-sidecar-dir.ts b/packages/deepseek/src/default-sidecar-dir.ts new file mode 100644 index 0000000..421056e --- /dev/null +++ b/packages/deepseek/src/default-sidecar-dir.ts @@ -0,0 +1,57 @@ +import { homedir as osHomedir } from "node:os"; +import { isAbsolute, join, posix, win32 } from "node:path"; + +export type DefaultSidecarSource = "default:DSH_HOME" | "default:home" | "fallback"; + +export type DefaultSidecarEnv = Readonly>; + +export interface ResolveDefaultSidecarDirInput { + readonly env: DefaultSidecarEnv; + readonly homedir: string | undefined; +} + +export interface ResolvedDefaultSidecarDir { + readonly dir: string; + readonly source: DefaultSidecarSource; +} + +/** This path stays relative. It resolves from the process working directory. */ +export const FALLBACK_SIDECAR_DIR = "./.lossless"; + +function nonBlank(value: string | undefined): string | undefined { + if (value === undefined) return undefined; + const trimmed = value.trim(); + return trimmed.length === 0 ? undefined : trimmed; +} + +function isAbsolutePath(path: string): boolean { + return isAbsolute(path) || posix.isAbsolute(path) || win32.isAbsolute(path); +} + +/** Pure resolver for the default sidecar directory. Only a non-blank absolute + * DSH_HOME wins; anything else falls through to the home dir, then fallback. */ +export function resolveDefaultSidecarDir( + input: ResolveDefaultSidecarDirInput, +): ResolvedDefaultSidecarDir { + const dshHome = nonBlank(input.env["DSH_HOME"]); + if (dshHome !== undefined && isAbsolutePath(dshHome)) { + return { dir: join(dshHome, "lossless"), source: "default:DSH_HOME" }; + } + const home = nonBlank(input.homedir); + if (home !== undefined) { + return { dir: join(home, ".dsh", "lossless"), source: "default:home" }; + } + return { dir: FALLBACK_SIDECAR_DIR, source: "fallback" }; +} + +/** Resolve the default from the live process. Never throws: anything odd falls back to {@link FALLBACK_SIDECAR_DIR}. */ +export function defaultSidecarDir(): ResolvedDefaultSidecarDir { + try { + return resolveDefaultSidecarDir({ + env: process.env, + homedir: osHomedir(), + }); + } catch { + return { dir: FALLBACK_SIDECAR_DIR, source: "fallback" }; + } +} diff --git a/packages/deepseek/src/engine.ts b/packages/deepseek/src/engine.ts new file mode 100644 index 0000000..2ecf077 --- /dev/null +++ b/packages/deepseek/src/engine.ts @@ -0,0 +1,326 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import type { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import { ManualCompactionError } from "@deepseek-ai/dsh-compaction"; +import BasicCompactionEngine from "@deepseek-ai/dsh-compaction-basic"; +import type { BasicCompactionConfig } from "@deepseek-ai/dsh-compaction-basic"; +import type { ContentBlock, Message, TokenUsage, ToolSchema } from "@deepseek-ai/dsh-llm"; +import type { Session } from "@deepseek-ai/dsh-session"; +import type { CompactionSpanEntry, LcmStore } from "lossless-core"; + +import { commitDagTransaction, selectDagRange } from "./dag-commit.ts"; +import type { DagCommitResult } from "./dag-commit.ts"; +import { backfillSession, historySourceFrom, MessageIndex } from "./ingest.ts"; +import type { CheckpointMemory, LadderOutcome, LosslessEngineDeps } from "./ladder.ts"; +import { + SpanAlreadyMemoryError, + checkpointMemory, + resolveSummarizationTarget, + runLadder, + seqOfEntry, + withSummaryRefs, +} from "./ladder.ts"; + +/** Import surface this module keeps stable for the package root. The ladder + * lives in ladder.ts; these re-exports hold the pre-split import paths so + * the move changes no consumer. */ +export { SpanAlreadyMemoryError, checkpointMemory }; +export type { CheckpointMemory, LosslessEngineDeps }; + +/** The pinned `summarize()` contract, repeated here because the package root + * does not export these members at this pin. The override check below fails + * the build on renames and removals; additions need a look when the pin + * moves. */ +interface SpanInput { + readonly tools?: readonly ToolSchema[]; + readonly messages: readonly Message[]; +} + +interface SpanSummary { + summary: ContentBlock[]; + provider: string; + model: string; + maxTokens?: number; + usage?: TokenUsage; + rawOutput?: ContentBlock[]; +} + +/** The shipped backend keeps range selection, the transaction, + * the meter, and the shrink check. Only the summarizer runs through the + * lossless ladder and the sidecar, and the checkpoint text it returns is + * what the engine frames and commits. Manual compaction (`compactNow`) + * commits through the DAG bracket instead; automatic pressure compaction + * (`compactRegion`) stays on this seam unless `bracketAuto` routes it + * through the bracket too, so the regression suite keeps pinning it. */ +export class LosslessSummarizerEngine extends BasicCompactionEngine { + constructor( + ctx: Context, + config: BasicCompactionConfig = {}, + private readonly lossless: LosslessEngineDeps, + ) { + super(ctx, config); + } + + override async compactRegion( + ...args: Parameters + ): Promise>> { + if (this.lossless.bracketAuto === true) { + const [start, end, agent, signal] = args; + return commitDagTransaction( + { + ctx: this.ctx, + summarizationProvider: this.config.summarizationProvider, + summarizationModel: this.config.summarizationModel, + ...this.lossless, + }, + agent, + { start, end }, + { owner: "current-turn", stability: "whole-surface" }, + signal, + ); + } + try { + return await super.compactRegion(...args); + } catch (error) { + // The seam wraps summarize failures in a generic could-not-summarize + // error; unwrap ours so the caller hears already-memory instead. + // Matched structurally (name plus code) so a second copy of the root + // package cannot break the recognition, and without importing it. + const cause = isManualSummaryError(error) ? error.cause : undefined; + if (cause instanceof SpanAlreadyMemoryError) throw cause; + throw error; + } + } + + /** Manual compaction through the DAG bracket: idle-agent admission, + * range selection, and the durability flush mirror the seam's compactNow + * (dsh-compaction-basic@0.1.7-alpha.2 lib/index.js:974-1009); the + * transaction itself is commitDagTransaction. Returns null when no safe + * useful range exists. */ + override compactNow( + agent: Agent, + signal: AbortSignal, + sourceCommandId?: CommandId, + ): Promise { + signal.throwIfAborted(); + try { + // Bare return, not await: only a synchronous admission refusal maps + // to busy here. A task failure rejects the returned promise past this + // catch, preserving the seam's cancellation precedence. + return agent.runMaintenance(async (agentSignal) => { + const operationSignal = AbortSignal.any([agentSignal, signal]); + try { + operationSignal.throwIfAborted(); + // Retain 0, exactly like the seam's compactNow: the manual path + // forces one useful reduction, so retainTokens prices only the + // pressure path the seam owns. + const range = selectDagRange( + agent.session, + this.ctx.tokenMeter.measure(agent.session), + 0, + ); + if (range === null) return null; + return await commitDagTransaction( + { + ctx: this.ctx, + summarizationProvider: this.config.summarizationProvider, + summarizationModel: this.config.summarizationModel, + ...this.lossless, + }, + agent, + range, + { + owner: null, + stability: "selected-span", + ...(sourceCommandId === undefined ? {} : { sourceCommandId }), + flush: async () => { + await this.ctx.sessions.flush(agent.session); + }, + }, + operationSignal, + ); + } catch (error) { + if (agentSignal.aborted && operationSignal.reason === agentSignal.reason) { + throw new ManualCompactionError("cancelled", "manual compaction was cancelled", { + cause: error, + }); + } + operationSignal.throwIfAborted(); + throw error; + } + }); + } catch (error) { + throw new ManualCompactionError( + "busy", + "manual compaction requires an idle agent with no waking queued work", + { cause: error }, + ); + } + } + + protected override async summarize( + input: SpanInput, + agent: Agent, + signal?: AbortSignal, + ): Promise { + const sessionId = String(agent.session.id); + const store = this.lossless.store(sessionId); + if (!store) throw new Error(`lossless compaction: no sidecar store for session ${sessionId}`); + const index = this.lossless.index(sessionId); + if (!index) { + throw new Error(`lossless compaction: session ${sessionId} has no message map`); + } + const target = resolveSummarizationTarget(this.config, agent); + if (!target) { + throw new Error( + "lossless compaction: no provider/model available for summarization: set both engine summarization fields, route one request, or set both agent options fields", + ); + } + let span: CompactionSpanEntry[]; + const started = Date.now(); + try { + try { + span = this.spanOf(input, store, index, agent.session); + } catch (error) { + // A dead mirror heals on retry; anything else must stay fail-closed. + if (!isUnmappedSpanError(error)) throw error; + const source = historySourceFrom(this.ctx); + if (source === undefined) throw error; + await backfillSession( + agent.session, + store, + source, + index, + this.lossless.record === undefined ? {} : { record: this.lossless.record }, + ); + span = this.spanOf(input, store, index, agent.session); + } + } catch (error) { + // Span mapping never reaches the ladder, so this is the only pass row. + this.lossless.record?.({ + ts: Date.now(), + event: "lcm", + kind: "pass", + session: sessionId, + stage: "blocking", + outcome: "failed", + error: error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300), + durationMs: Date.now() - started, + }); + throw error; + } + const ladder = await runLadder(this.ctx, this.lossless, target, sessionId, span, signal); + const checkpoint = withSummaryRefs(ladder.summaryText, [ + ...ladder.leafSummaryIds, + ...ladder.condensedSummaryIds, + ]); + // Outcome unobservable here; next success overwrites, so staleness self-heals. + this.lossless.memory?.note(sessionId, checkpoint); + return toSummaryResult(checkpoint, ladder, this.config.maxTokens); + } + + private spanOf( + input: SpanInput, + store: LcmStore, + index: MessageIndex, + session: Session, + ): CompactionSpanEntry[] { + // A leading system message is the retained head, not shadowed content. + const startAt = input.messages[0]?.role === "system" ? 1 : 0; + const items: Array<{ seq: number; entry: CompactionSpanEntry }> = []; + let warmed = false; + for (const message of input.messages.slice(startAt)) { + let entryId = index.entryId(message.id); + if (entryId === undefined) { + if (warmed) { + entryId = undefined; + } else { + warmed = true; + entryId = this.resolveEntryId(session, index, message.id); + } + } + if (entryId === undefined) { + throw new Error( + `lossless compaction: message ${message.id} has no sidecar row; the span is not fully mirrored`, + ); + } + const row = store.messageByEntryId(entryId); + if (!row) { + throw new Error(`lossless compaction: sidecar holds no row for ${entryId}`); + } + items.push({ + seq: seqOfEntry(entryId), + entry: { + entryId, + role: row.role, + text: row.text, + ...(row.payload === undefined ? {} : { payload: row.payload }), + }, + }); + } + if (items.length === 0) + throw new Error("lossless compaction: no mirrorable messages in the span"); + // Core reads a span as a storage range, so frontierFrom must name the + // seq minimum. Catch-up backfill lands older seqs on newer rows, so + // rowid order no longer matches narrative order. + items.sort((a, b) => a.seq - b.seq); + return items.map((item) => item.entry); + } + + /** Re-derive message ids to sidecar entries from the live log. The map dies + * with the process; one scan repopulates every derived id it finds, so a + * cold map heals on first miss and later spans stay on the fast path. A + * message with no live event stays unmapped and the caller fails closed. */ + private resolveEntryId( + session: Session, + index: MessageIndex, + messageId: string, + ): string | undefined { + let found: string | undefined; + for (const event of session.snapshotEvents()) { + const message = session.deriveEventMessage(event); + if (message === null) continue; + const entryId = `${session.id}:${event.seq}`; + index.record(message.id, entryId); + if (message.id === messageId) found = entryId; + } + return found; + } +} + +function toSummaryResult( + checkpoint: string, + ladder: Pick, + maxTokens: number, +): SpanSummary { + if (checkpoint.trim().length === 0) { + throw new Error("lossless compaction: ladder produced no text"); + } + return { + summary: [{ type: "text", text: checkpoint }], + rawOutput: [{ type: "text", text: checkpoint }], + provider: ladder.provider, + model: ladder.model, + maxTokens, + ...(ladder.usage === undefined ? {} : { usage: ladder.usage }), + }; +} + +/** A gapped mirror is worth one live reread; every other failure keeps its path. */ +function isUnmappedSpanError(error: unknown): boolean { + return ( + error instanceof Error && + (error.message.includes("has no sidecar row; the span is not fully mirrored") || + error.message.includes("sidecar holds no row for ")) + ); +} + +/** True for the seam's generic summary failure, which wraps whatever + * `summarize` threw as its cause. */ +function isManualSummaryError(error: unknown): error is Error & { cause?: unknown } { + return ( + error instanceof Error && + error.name === "ManualCompactionError" && + (error as { code?: unknown }).code === "summary" + ); +} diff --git a/packages/deepseek/src/index.ts b/packages/deepseek/src/index.ts new file mode 100644 index 0000000..2c41e94 --- /dev/null +++ b/packages/deepseek/src/index.ts @@ -0,0 +1,13 @@ +export { dshModelHost, dshRef, usageOf } from "./model-host.ts"; +export type { DshHostOptions, DshRoute } from "./model-host.ts"; +export { dshReader } from "./reader.ts"; +export type { DshReaderOptions } from "./reader.ts"; +export { entryFor, mirrorEvent, observeSession, MessageIndex } from "./ingest.ts"; +export { LosslessSummarizerEngine, SpanAlreadyMemoryError, checkpointMemory } from "./engine.ts"; +export type { CheckpointMemory, LosslessEngineDeps } from "./engine.ts"; +export { lcmCommandDefinition, registerLcmCommand } from "./command.ts"; +export type { LcmCommandDeps } from "./command.ts"; +export { apply, applyLossless, Config, inject, name } from "./plugin.ts"; +export type { LosslessPluginConfig } from "./plugin.ts"; +export { registerRecallTools } from "./tools.ts"; +export type { RecallDeps } from "./tools.ts"; diff --git a/packages/deepseek/src/ingest.ts b/packages/deepseek/src/ingest.ts new file mode 100644 index 0000000..0a8160d --- /dev/null +++ b/packages/deepseek/src/ingest.ts @@ -0,0 +1,414 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { ContentBlock, Message } from "@deepseek-ai/dsh-llm"; +import type { Session, SessionEvent, SessionId } from "@deepseek-ai/dsh-session"; +import { ingestEntries } from "lossless-core"; +import type { IngestStats, IngestibleEntry, LcmRole, LcmStore } from "lossless-core"; + +/** Live projection state: derived message id to sidecar entry id, plus the + * tool name per call id. The engine hook resolves its span through this map + * instead of matching text; a cold map re-derives entries from the live log + * on first miss, and only a message with no live event fails closed rather + * than guessing rows. The tool map carries the name from a tool call to its + * later result without a history scan, so a result keeps its identity even + * when the assistant row that requested it mixes prose with the call. */ +export class MessageIndex { + private readonly ids = new Map(); + private readonly toolNames = new Map(); + + record(messageId: string, entryId: string): void { + this.ids.set(messageId, entryId); + } + + entryId(messageId: string): string | undefined { + return this.ids.get(messageId); + } + + recordToolCall(callId: string, name: string): void { + const id = callId.trim(); + const tool = name.trim(); + if (id.length === 0 || tool.length === 0) return; + if (!this.toolNames.has(id)) this.toolNames.set(id, tool); + } + + toolName(callId: string): string | undefined { + return this.toolNames.get(callId.trim()); + } +} + +/** One sidecar row per message-producing event, keyed by session and sequence. + * A compaction checkpoint ingests like any other model-visible message, and + * the entries it shadows stay stored: the log moves on, the sidecar keeps + * every address the reader can name. */ +export function entryFor(session: Session, event: SessionEvent): IngestibleEntry | undefined { + return projectedEntry(session, event)?.entry; +} + +/** Mirror one committed event into the sidecar. Ingest dedups on the entry + * id, so a re-delivered event inserts nothing. */ +export function mirrorEvent( + session: Session, + event: SessionEvent, + store: LcmStore, + index?: MessageIndex, +): IngestStats { + const projected = projectedEntry(session, event, index); + if (projected === undefined) { + return { inserted: 0, skipped: 1, piCompactions: [], files: [], largeInline: [] }; + } + return mirrorProjected(store, projected, index); +} + +/** Mirror an already-derived projection. Backfill probes the projection to + * account visited budget, then mirrors without deriving twice. */ +function mirrorProjected( + store: LcmStore, + projected: { entry: IngestibleEntry; messageId: string }, + index?: MessageIndex, +): IngestStats { + const stats = ingestEntries(store, [projected.entry]); + if (stats.inserted > 0) index?.record(projected.messageId, projected.entry.entryId); + return stats; +} + +/** Follow one session's live feed into its sidecar. The feed starts at + * subscription time: synchronous history scans are prohibited for new + * production code in this harness, so there is no sync backfill here. + * Pre-mount history arrives through backfillSession over the async + * observation seam instead, which keeps the sync readers test-only. A + * reopened sidecar dedups against its stored rows. */ +export function observeSession( + ctx: Context, + session: Session, + store: LcmStore, + index?: MessageIndex, +): () => void { + return ctx.on("session/event", (live, event) => { + if (live.id !== session.id) return; + mirrorEvent(live, event, store, index); + }); +} + +/** Mirror slice size for pre-mount backfill. Matches the host history page + * default (DEFAULT_MAX_MESSAGES in the harness session-controller history + * module), so one seam means one paging vocabulary. */ +export const BACKFILL_PAGE_SIZE = 50; + +/** Per-session ceiling on backfilled rows. Counts INSERTED rows only, so + * dedup and non-message skips never burn budget; live events keep mirroring + * past the cap. */ +export const BACKFILL_MAX_EVENTS = 500; + +/** Ceiling on total visited events per backfill. Bounds the scan when skips + * dominate; defaults to four times the effective inserted cap. */ +export const BACKFILL_MAX_VISITED = BACKFILL_MAX_EVENTS * 4; + +/** One frozen pre-mount cut for a session, read through the async observation + * seam. The harness materializes it behind its own migration waiver, so this + * package still makes no synchronous history call. */ +export interface HistoryObservation { + readonly events: readonly SessionEvent[]; +} + +/** Async history seam for pre-mount backfill. Production reads come from the + * composed query backend; tests fake this interface, never the harness. */ +export interface SessionHistorySource { + observe(sessionId: SessionId): Promise; +} + +/** Structural shape of the composed `ctx.sessionQuery` backend. Declared + * locally so this package takes no dependency on the query backend and adds + * no required service: when the backend is absent from the composition the + * mount skips backfill and the live mirror still covers new events. */ +interface SessionQueryLike { + observeSession( + sessionId: SessionId, + options?: { signal?: AbortSignal; projectionMode?: "all" | "none" }, + ): Promise; +} + +/** Well-known dispose without the esnext.disposable lib: the lease drops a + * cache pin on prepared cuts and is a no-op flag on live cuts. */ +const DISPOSE = (Symbol as unknown as { dispose: symbol }).dispose; + +function disposeObservation(observation: { [key: symbol]: unknown }): void { + const dispose = observation[DISPOSE]; + if (typeof dispose === "function") dispose(); +} + +/** Optional async history source for one composition. Returns undefined when + * the query backend is not mounted; the caller then skips backfill. This is + * the same `observeSession` call the host history controller pages from, + * read optionally so the ctx surface does not widen. */ +export function historySourceFrom(ctx: Context): SessionHistorySource | undefined { + let query: SessionQueryLike | undefined; + try { + // Live scopes reject undeclared service reads; treat that exactly like + // an unmounted backend so the mount degrades to live-mirror-only. + query = (ctx as unknown as { sessionQuery?: SessionQueryLike }).sessionQuery; + } catch { + return undefined; + } + if (query === undefined) return undefined; + return { + observe: async (sessionId) => { + const observation = await query.observeSession(sessionId, { projectionMode: "none" }); + try { + return { events: observation.events }; + } finally { + disposeObservation(observation); + } + }, + }; +} + +/** Options for one session backfill. */ +export interface BackfillOptions { + /** Mirror slice size. Default BACKFILL_PAGE_SIZE. */ + pageSize?: number; + /** Per-session inserted-row ceiling. Default BACKFILL_MAX_EVENTS. */ + maxEvents?: number; + /** Per-session visited-event ceiling. Only unmirrored messages count: + * stored rows cost one indexed probe and non-message events short-circuit + * before any store call. Default 4x the effective maxEvents. */ + maxVisited?: number; + /** Metrics recorder; receives one `backfill` row per session. */ + record?: (row: Record) => void; +} + +/** Outcome of one session backfill. */ +export interface BackfillResult { + ingested: number; + skipped: number; + /** Seq of the first visited event, -1 when nothing visited. */ + fromSeq: number; + /** Seq of the last visited event, -1 when nothing visited. */ + throughSeq: number; + /** True when the observation held more events than the cap allowed. */ + capped: boolean; +} + +/** Mirror one session's pre-mount history through the same mirrorEvent path + * the live feed uses, so backfill plus live mirror cannot double-insert: + * ingest dedups on the entry id. Pages the frozen observation in bounded + * slices with an event-loop yield between pages, and stops at the cap. */ +export async function backfillSession( + session: Session, + store: LcmStore, + source: SessionHistorySource, + index?: MessageIndex, + options: BackfillOptions = {}, +): Promise { + const pageSize = Math.max(1, Math.floor(options.pageSize ?? BACKFILL_PAGE_SIZE)); + const maxEvents = Math.max(0, Math.floor(options.maxEvents ?? BACKFILL_MAX_EVENTS)); + const maxVisited = Math.max(0, Math.floor(options.maxVisited ?? maxEvents * 4)); + const { events } = await source.observe(session.id); + let ingested = 0; + let skipped = 0; + let visited = 0; + let consumed = 0; + let fromSeq = -1; + let throughSeq = -1; + for ( + let start = 0; + start < events.length && ingested < maxEvents && visited < maxVisited; + start += pageSize + ) { + for (const event of events.slice(start, start + pageSize)) { + if (ingested >= maxEvents || visited >= maxVisited) break; + consumed += 1; + // Only unmirrored messages cost visited budget: stored rows need one + // indexed probe and markers never touch the store, so a warm history + // still reaches new events instead of stalling on its own prefix. + const projected = projectedEntry(session, event, index); + if ( + projected !== undefined && + store.messageByEntryId(projected.entry.entryId) === undefined + ) { + visited += 1; + } + const seq = Number(event.seq); + if (fromSeq === -1) fromSeq = seq; + throughSeq = seq; + const stats = + projected === undefined + ? { inserted: 0, skipped: 1, piCompactions: [], files: [], largeInline: [] } + : mirrorProjected(store, projected, index); + ingested += stats.inserted; + skipped += stats.skipped; + } + if (start + pageSize < events.length && ingested < maxEvents && visited < maxVisited) { + await new Promise((resolve) => setImmediate(resolve)); + } + } + const capped = (ingested >= maxEvents || visited >= maxVisited) && consumed < events.length; + options.record?.({ + ts: Date.now(), + event: "lcm", + kind: "backfill", + session: String(session.id), + ingested, + skipped, + fromSeq, + throughSeq, + capped, + }); + return { ingested, skipped, fromSeq, throughSeq, capped }; +} + +function projectedEntry( + session: Session, + event: SessionEvent, + index?: MessageIndex, +): { entry: IngestibleEntry; messageId: string } | undefined { + if (event.type === "tool/call") { + const data = event.data as unknown as { callId?: unknown; name?: unknown }; + if (typeof data.callId === "string" && typeof data.name === "string") { + index?.recordToolCall(data.callId, data.name); + } + return undefined; + } + const message = session.deriveEventMessage(event); + if (message === null) return undefined; + if (message.role === "assistant") { + for (const block of message.content) { + if (block.type === "tool-call") { + index?.recordToolCall(String(block.id), block.name); + } + } + } + const role = dshRole(message.role); + if (role === undefined) return undefined; + const toolName = message.role === "tool" ? toolNameForResult(message, index) : undefined; + const text = renderText(message, toolName); + if (text === undefined) return undefined; + return { + entry: { + entryId: `${session.id}:${event.seq}`, + role, + text, + timestamp: event.time, + }, + messageId: message.id, + }; +} + +/** The tool a result answers, from the live call map. The call event and the + * assistant tool-call block both record the name ahead of the result, so the + * stored row keeps its identity without a history scan. */ +function toolNameForResult(message: Message, index: MessageIndex | undefined): string | undefined { + if (message.role !== "tool" || index === undefined) return undefined; + const candidates: unknown[] = [ + (message as { toolCallId?: unknown }).toolCallId, + (message as { source?: { callId?: unknown } }).source?.callId, + ]; + for (const candidate of candidates) { + if (typeof candidate !== "string" || candidate.trim().length === 0) continue; + const name = index.toolName(candidate); + if (name !== undefined && name.trim().length > 0) return name.trim(); + } + return undefined; +} + +function dshRole(role: Message["role"]): LcmRole | undefined { + switch (role) { + case "user": + return "user"; + case "assistant": + return "assistant"; + case "tool": + return "toolResult"; + case "system": + case "developer": + return "custom"; + default: + return undefined; + } +} + +/** Verbatim text plus the calls the transcript needs to stay replayable. A + * message with blocks but no text keeps a descriptor row instead, and blank + * text keeps a whitespace marker, so every span message holds a sidecar + * address for compaction to map. A tool result keeps its tool name on the + * opening line, the marker core reads, so a listed tool stays mechanical + * even when the assistant row that requested it mixes prose with the call. */ +function renderText(message: Message, toolName?: string): string | undefined { + if (message.content.length === 0) return undefined; + const parts: string[] = []; + for (const block of message.content) { + if (block.type === "text") parts.push(block.text); + else if (block.type === "tool-call") parts.push(`tool-call: ${block.name} ${block.arguments}`); + } + const text = parts.join("\n"); + if (text.trim().length > 0) return withToolName(message, text, toolName); + if (message.content.some((block) => block.type !== "text")) { + return withToolName(message, describeNonText(message), toolName); + } + // Blank text still needs an address or spans wedge; the marker keeps it honest. + return withToolName(message, `[lcm:whitespace ${text.length} chars]`, toolName); +} + +/** Prefix a tool result with its tool marker line. Other roles keep their + * text untouched, and a result that already names the same tool keeps its + * opening line instead of doubling it. */ +function withToolName(message: Message, text: string, toolName?: string): string { + if (message.role !== "tool") return text; + const name = toolName?.trim(); + if (!name) return text; + const first = text.split("\n").find((line) => line.trim().length > 0) ?? ""; + const marker = new RegExp(`^\\s*tool\\s+${escapeRegExp(name)}\\b`, "i"); + if (marker.test(first)) return text; + const callMarker = new RegExp( + `^\\s*(?:\\[tool:${escapeRegExp(name)}\\]|tool[_-]?call:\\s*${escapeRegExp(name)}\\b)`, + "i", + ); + if (callMarker.test(first)) return text; + return `tool ${name}\n${text}`; +} + +function escapeRegExp(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** Bracketed stand-in mirroring the core file-handle shape: block kinds plus + * stable identifiers, never bytes, marked so no reader takes it for seen text. */ +function describeNonText(message: Message): string { + const kinds = message.content.map(describeBlock).join(", "); + return `[lcm:non-text ${kinds}; content not seen, descriptor only]`; +} + +function describeBlock(block: ContentBlock): string { + switch (block.type) { + case "image": { + if ((block as { attachment?: unknown }).attachment === undefined) { + return "image (attachment missing)"; + } + const ref = block.attachment; + const details = [ + ref.mediaType, + `${ref.bytes} bytes`, + `${ref.width}x${ref.height}px`, + String(ref.attachmentId), + ]; + if (block.offloaded === true) details.push("offloaded"); + const name = ref.name === undefined ? "" : ` ${JSON.stringify(ref.name)}`; + return `image${name} (${details.join(", ")})`; + } + case "file": { + if ((block as { attachment?: unknown }).attachment === undefined) { + return "file (attachment missing)"; + } + const ref = block.attachment; + return `file ${JSON.stringify(ref.name)} (${ref.bytes} bytes, ${String(ref.attachmentId)})`; + } + case "tool-addition": + case "tool-removal": + return `${block.type} ${JSON.stringify(block.toolName)}`; + case "reasoning": + case "text": + return block.type; + case "tool-call": + return `tool-call ${block.name}`; + default: + return (block as { type: string }).type; + } +} diff --git a/packages/deepseek/src/ladder.ts b/packages/deepseek/src/ladder.ts new file mode 100644 index 0000000..b0b2cc1 --- /dev/null +++ b/packages/deepseek/src/ladder.ts @@ -0,0 +1,375 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import type { TokenUsage } from "@deepseek-ai/dsh-llm"; +import { + addUsageTally, + createUsageTally, + makeLlm, + runCompaction, + SUMMARIZER_CALL_TOKENS, + SUMMARIZER_HEAL_TOKENS, + tokenUsageOfTally, + type CompactionSpanEntry, + type CompletionRequest, + type LcmStore, + type MetricsSink, + type ModelAnswer, + type ModelHost, + type ModelRef, + type SummarizerConfig, + type UsageTally, +} from "lossless-core"; + +import type { MessageIndex } from "./ingest.ts"; +import { dshModelHost } from "./model-host.ts"; +import { agentRoute } from "./model-host.ts"; +import type { DshRoute } from "./model-host.ts"; + +/** Engine dependencies for one composition: store, index, redact, and budgets. */ +export interface LosslessEngineDeps { + /** Store for one session id. Undefined when the session has no sidecar yet. */ + readonly store: (sessionId: string) => LcmStore | undefined; + /** Live message-id to entry-id map, fed by the mirror. Spans resolve through it, never through text matching. */ + readonly index: (sessionId: string) => MessageIndex | undefined; + /** Last checkpoint text per session, fed back as the next ladder's previous + * summary. Absent until the engine stores its first checkpoint here. */ + readonly memory?: CheckpointMemory; + /** Redacts stored text before it reaches a model call. */ + readonly redact: (text: string) => string; + /** Diagnostic sink: one run row per ladder run plus every engine row the + * run causes. Optional so scripted tests can observe without a metrics + * file; production passes the per-composition JSONL writer. */ + readonly record?: (row: Record) => void; + /** Tokens one summary aims for. Unset keeps the core default. */ + readonly summaryTokens?: number; + /** Tokens one leaf chunk may hold before it splits. Unset keeps the core default. */ + readonly leafChunkTokens?: number; + /** Tokens one summarizer request may cost before it is refused rather than + * paid for. Unset keeps the core default (60,000). */ + readonly summarizerTokensPerCall?: number; + /** Tokens one pass may spend across all of its calls. Unset keeps the core + * default (250,000). */ + readonly summarizerTokensPerPass?: number; + /** Tokens one heal run may spend across all of its calls. Unset keeps the + * core heal default (120,000). */ + readonly summarizerTokensPerHeal?: number; + /** Estimated tokens one heal node may read as source. Unset derives the + * core default inside runHeal: 8 x the node target, clamped to 3,000 + * through 24,000 and to the request budget. */ + readonly healSourceTokens?: number; + /** Route automatic pressure compaction through the DAG bracket instead of + * the seam. Default false: the seam stays the observable auto behavior + * while the bracket proves itself on the manual path. */ + readonly bracketAuto?: boolean; + /** Tool names whose spans compact without a model call. Empty keeps the + * LLM path; ingest still stores every message verbatim. */ + readonly mechanicalTools?: readonly string[]; +} + +/** Per-session checkpoint memory: the plugin owns one, the engine reads the + * previous summary from it and notes every checkpoint it stores. Absent + * until the session stores its first checkpoint, in memory or in the + * sidecar. */ +export interface CheckpointMemory { + previous(sessionId: string): string | undefined; + note(sessionId: string, text: string): void; +} + +const DEFAULT_SUMMARY_TOKENS = 1500; +const DEFAULT_LEAF_CHUNK_TOKENS = 3000; + +/** makeLlm config for one ladder pass. Set keys ride the plugin config; + * absent keys stay absent so makeLlm keeps the core defaults. */ +export function passLlmConfig( + deps: Pick, +): SummarizerConfig { + return { + ...(deps.summarizerTokensPerCall === undefined + ? {} + : { summarizerTokensPerCall: deps.summarizerTokensPerCall }), + ...(deps.summarizerTokensPerPass === undefined + ? {} + : { summarizerTokensPerPass: deps.summarizerTokensPerPass }), + }; +} + +/** Run budgets for one heal, mirroring the pi and opencode adapters: the run + * cap is the per-heal budget, the node target is the summary target, the + * per-request cap is the per-call budget, and the source window rides only + * when configured. An unset source derives inside core runHeal as 8 x the + * target, clamped to 3,000 through 24,000 and to the request budget. */ +export function healRunBudgets( + deps: Pick< + LosslessEngineDeps, + "summaryTokens" | "summarizerTokensPerCall" | "summarizerTokensPerHeal" | "healSourceTokens" + >, +): { + readonly maxTokens: number; + readonly targetTokens: number; + readonly callBudgetTokens: number; + readonly sourceTokens?: number; +} { + return { + maxTokens: deps.summarizerTokensPerHeal ?? SUMMARIZER_HEAL_TOKENS, + targetTokens: deps.summaryTokens ?? DEFAULT_SUMMARY_TOKENS, + callBudgetTokens: deps.summarizerTokensPerCall ?? SUMMARIZER_CALL_TOKENS, + ...(deps.healSourceTokens === undefined ? {} : { sourceTokens: deps.healSourceTokens }), + }; +} + +/** makeLlm config for one heal run: the per-request cap plus the heal run + * budget standing in for the per-pass budget, like pi and opencode. */ +export function healLlmConfig( + deps: Pick, +): SummarizerConfig { + return { + ...(deps.summarizerTokensPerCall === undefined + ? {} + : { summarizerTokensPerCall: deps.summarizerTokensPerCall }), + summarizerTokensPerPass: deps.summarizerTokensPerHeal ?? SUMMARIZER_HEAL_TOKENS, + }; +} + +/** Per-session checkpoint texts for the ladder. `previous` returns the noted text, else hydrates the latest checkpoint from the sidecar and caches it; stays undefined with no store or no stored summary. Called with no lookup, it serves noted texts only. */ +export function checkpointMemory( + stores?: (sessionId: string) => LcmStore | undefined, +): CheckpointMemory { + const texts = new Map(); + return { + previous: (sessionId) => { + const cached = texts.get(sessionId); + if (cached !== undefined) return cached; + const hydrated = hydrateCheckpoint(stores?.(sessionId)); + if (hydrated !== undefined) texts.set(sessionId, hydrated); + return hydrated; + }, + note: (sessionId, text) => { + texts.set(sessionId, text); + }, + }; +} + +/** Rebuild the last checkpoint from the sidecar: the newest stored summary + * plus its stored children, cited the same way the ladder cites them. The + * newest row is the last successful run's final node. Exact when the run + * stored leaves plus a fresh condensed row; a single-leaf run replaces its + * row's text on condensation, so the rebuild carries the stored form; with + * deeper condensation every cited id still resolves. Either way a resumed + * session compacts with continuity instead of from scratch. */ +function hydrateCheckpoint(store: LcmStore | undefined): string | undefined { + if (store === undefined) return undefined; + const summaries = store.allSummaries(); + const latest = summaries[summaries.length - 1]; + if (latest === undefined) return undefined; + return withSummaryRefs(latest.text, [...store.childSummaryIds(latest.id), latest.id]); +} + +/** The ladder stored nothing because every row in the span is already covered: + * the span is memory, not history. Thrown instead of the seam's generic + * summary failure so the caller hears the true state. */ +export class SpanAlreadyMemoryError extends Error { + constructor(span: string) { + super(`lossless compaction: span ${span} is already memory; every row is covered`); + this.name = "SpanAlreadyMemoryError"; + } +} + +/** Shared provider/model fallback chain (explicit config, latest routed + * request, agent options) for the seam path and the DAG bracket alike. Reads + * only the host session header and agent options beside the host SDK. + * Per-model policy overrides (modelPolicies) ride the seam config untouched: + * the seam merges them per routed target on its pressure path, while this + * chain stays the one summarization target both paths share so the + * summarizer cannot drift between them. */ +export function resolveSummarizationTarget( + config: { summarizationProvider: string; summarizationModel: string }, + agent: Agent, +): DshRoute | undefined { + const configured = + config.summarizationProvider.length > 0 && config.summarizationModel.length > 0 + ? { provider: config.summarizationProvider, model: config.summarizationModel } + : undefined; + return configured ?? agentRoute(agent); +} + +/** Seq suffix of a sidecar entry id (`:`), split on the last + * colon. Throws exactly like a missing row: an unparseable id never guesses + * span order. Shared by both span builders so the rule cannot drift. */ +export function seqOfEntry(entryId: string): number { + const tail = entryId.slice(entryId.lastIndexOf(":") + 1); + if (!/^\d+$/.test(tail)) { + throw new Error( + `lossless compaction: entry ${entryId} has no parseable seq; refusing to guess span order`, + ); + } + return Number(tail); +} + +/** What one ladder run stored: the condensed text plus the sidecar ids and + * the model call facts for the checkpoint record. */ +export interface LadderOutcome { + readonly summaryText: string; + readonly leafSummaryIds: readonly number[]; + readonly condensedSummaryIds: readonly number[]; + readonly provider: string; + readonly model: string; + readonly usage?: TokenUsage; +} + +/** One lossless ladder run against a resolved span: leaf chunking, then + * condensation, sharing one usage tracker. The seam `summarize()` and the + * DAG bracket commit through here, so the summarizer cannot drift between + * the two paths. Throws SpanAlreadyMemoryError when every row in the span + * is already covered. */ +export async function runLadder( + ctx: Context, + lossless: LosslessEngineDeps, + target: DshRoute, + sessionId: string, + span: CompactionSpanEntry[], + signal?: AbortSignal, +): Promise { + const first = span[0]; + const last = span[span.length - 1]; + if (!first || !last) throw new Error("lossless compaction: span mapping came back empty"); + const host = dshModelHost(ctx, { + defaultRoute: target, + sessionId, + ...(signal === undefined ? {} : { signal }), + }); + const tracker = new UsageTracker(); + const record = lossless.record; + // Every engine and completer row belongs to this run, so they go through + // the same record callback as the run row. Without one this stays unset + // and the rows fall back to the global log, never a half-wired file. + const sink: MetricsSink | undefined = + record === undefined ? undefined : (row) => record({ ...row }); + const llm = makeLlm(tracker.wrap(host), passLlmConfig(lossless), { + session: sessionId, + stage: "blocking", + span: { firstEntryId: first.entryId, lastEntryId: last.entryId }, + ...(sink === undefined ? {} : { sink }), + }); + if (!llm) throw new Error("lossless compaction: no summarizer model resolves"); + const store = lossless.store(sessionId); + if (!store) throw new Error(`lossless compaction: no sidecar store for session ${sessionId}`); + const { token } = store.openRun({ session: sessionId }); + const started = Date.now(); + const spanPair: [string, string] = [first.entryId, last.entryId]; + try { + const outcome = await runCompaction( + store.forRun(token), + { + span, + targetTokens: lossless.summaryTokens ?? DEFAULT_SUMMARY_TOKENS, + frontierFrom: first.entryId, + previousSummary: lossless.memory?.previous(sessionId), + }, + llm, + { + leafChunkTokens: lossless.leafChunkTokens ?? DEFAULT_LEAF_CHUNK_TOKENS, + redact: lossless.redact, + session: sessionId, + stage: "blocking", + // The seam summarize and the DAG bracket both commit through here, + // so one forward covers both paths. + ...(lossless.mechanicalTools === undefined + ? {} + : { mechanicalTools: lossless.mechanicalTools }), + }, + ); + store.commitRun(token); + if (outcome.kind === "nothing") { + record?.({ + ts: Date.now(), + event: "lcm", + kind: "pass", + session: sessionId, + stage: "blocking", + outcome: "nothing", + reason: outcome.reason, + ingested: outcome.ingested, + span: spanPair, + durationMs: Date.now() - started, + }); + throw new SpanAlreadyMemoryError(`${first.entryId}..${last.entryId} (${outcome.reason})`); + } + const route = tracker.route ?? target; + const usage = tracker.tokenUsage(); + record?.({ + ts: Date.now(), + event: "lcm", + kind: "pass", + session: sessionId, + stage: "blocking", + outcome: "stored", + leaf: outcome.leafSummaryIds.length, + condensed: outcome.condensedSummaryIds.length, + ingested: outcome.ingested, + ...(usage === undefined ? {} : { usage }), + span: spanPair, + durationMs: Date.now() - started, + }); + return { + summaryText: outcome.summaryText, + leafSummaryIds: outcome.leafSummaryIds, + condensedSummaryIds: outcome.condensedSummaryIds, + provider: route.provider, + model: route.model, + ...(usage === undefined ? {} : { usage }), + }; + } catch (error) { + if (error instanceof SpanAlreadyMemoryError) throw error; + record?.({ + ts: Date.now(), + event: "lcm", + kind: "pass", + session: sessionId, + stage: "blocking", + outcome: "failed", + error: error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300), + span: spanPair, + durationMs: Date.now() - started, + }); + throw error; + } finally { + store.abandonRun(token); + } +} + +/** Name the stored DAG nodes in the checkpoint using the same refs the + * recall tools resolve, so a future reader pulls full nodes from a short + * citation instead of re-deriving them. Shared with the DAG bracket so its + * checkpoint cites the same nodes. */ +export function withSummaryRefs(text: string, ids: readonly number[]): string { + if (ids.length === 0) return text; + return `${text}\n\nStored summaries: ${ids.map((id) => `[lcm:summary #${id}]`).join(" ")}`; +} + +/** Sums provider counts across every ladder call into one envelope. The route reported is the one that produced the stored text. The accumulation lives in the core usage tally seam; this wrapper keeps only the route beside the host. */ +class UsageTracker { + private tally: UsageTally = createUsageTally(); + route: DshRoute | undefined = undefined; + + wrap(host: ModelHost): ModelHost { + return { + ...host, + complete: async ( + model: ModelRef, + request: CompletionRequest, + ): Promise => { + const answer = await host.complete(model, request); + this.route = model.model; + const usage = answer.usage; + if (usage) { + addUsageTally(this.tally, usage); + } + return answer; + }, + }; + } + + tokenUsage(): TokenUsage | undefined { + return tokenUsageOfTally(this.tally); + } +} diff --git a/packages/deepseek/src/lifecycle.ts b/packages/deepseek/src/lifecycle.ts new file mode 100644 index 0000000..f51fc2f --- /dev/null +++ b/packages/deepseek/src/lifecycle.ts @@ -0,0 +1,998 @@ +import { chmodSync, readFileSync, renameSync, statSync, unlinkSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { basename, dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import type { CommandInvocation, CommandResult } from "@deepseek-ai/dsh-commands"; +import type { ModelCompactPolicyConfig } from "@deepseek-ai/dsh-compaction-basic"; +import { + appendMetric, + buildDiagnoseBundle, + checkIntegrity, + depthStats, + dropSidecarFiles, + effectiveThresholds, + exportTranscript, + fmtTokens, + importTranscript, + listSidecarRefs, + makeLlm, + mergeZone, + modelKey, + packageVersionNear, + parseExport, + readMetricRows, + redactSecrets, + runHeal, + sanitizeSidecarId, + SCHEMA_VERSION, + storeGeneration, + type HealOutcome, + LcmStore, + type LlmComplete, + type ThresholdConfig, +} from "lossless-core"; + +import { healLlmConfig, healRunBudgets } from "./ladder.ts"; +import { agentRoute, dshModelHost, type DshRoute } from "./model-host.ts"; +import { dumpSidecar } from "./sidecar-dump.ts"; + +/** Command dependencies for one composition: stores, paths, and budgets. */ +export interface LcmCommandDeps { + /** Open stores by session id for status and report views. */ + readonly stores: () => ReadonlyMap; + /** Open-or-create the calling session sidecar. Import uses it so a fresh + * session can restore history it has never mirrored. */ + readonly storeFor?: (sessionId: string) => LcmStore; + /** Sidecar directory: the metrics log lives here and default export paths land here. */ + readonly sidecarDir: string; + /** File path of one session's sidecar, for backup targets. */ + readonly storePathFor: (sessionId: string) => string; + /** Tokens one repaired node targets. Unset keeps the ladder default. */ + readonly summaryTokens?: number; + /** Tokens one summarizer request may cost. Unset keeps the core default. */ + readonly summarizerTokensPerCall?: number; + /** Tokens one heal run may spend across all of its calls. Unset keeps the + * core heal default. */ + readonly summarizerTokensPerHeal?: number; + /** Estimated tokens one heal node may read as source. Unset derives the + * core default (8 x target, clamped) inside runHeal. */ + readonly healSourceTokens?: number; + /** Configured summary route. A set pair names the heal model; unset + * follows the calling agent's route, like the ladder. */ + readonly summarizationProvider?: string; + readonly summarizationModel?: string; + /** Seam pressure keys status and diagnose report as effective values. */ + readonly thresholdRatio?: number; + readonly retainTokens?: number; + readonly retainRatio?: number; + readonly headroomTokens?: number; + /** Exact provider/model pressure overrides, merged per routed target. */ + readonly modelPolicies?: ModelCompactPolicyConfig[]; +} + +const STAMP = (): string => new Date().toISOString().replace(/[:.]/g, "-"); + +async function resolveWindowTokens(ctx: Context, agent: Agent): Promise { + const route = agentRoute(agent); + if (!route) return undefined; + try { + const info = await ctx.llm?.resolveModelInfo(route.provider, route.model); + const window = info?.context?.contextWindow; + return typeof window === "number" && window > 0 ? window : undefined; + } catch { + return undefined; + } +} + +function measureTotalTokens(ctx: Context, agent: Agent): number | undefined { + try { + const measurement = ctx.tokenMeter?.measure(agent.session); + return measurement === undefined ? undefined : Math.max(0, measurement.totalTokens); + } catch { + return undefined; + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +const METRIC_ROWS_LIMIT = 500; + +function metricRows(deps: LcmCommandDeps, sessionId?: string): Record[] { + const opts = + sessionId === undefined + ? { limit: METRIC_ROWS_LIMIT } + : { session: sessionId, limit: METRIC_ROWS_LIMIT }; + return readMetricRows(deps.sidecarDir, opts).filter((row): row is Record => + isRecord(row), + ); +} + +function sessionRows( + rows: readonly Record[], + sessionId: string, +): Record[] { + return rows.filter((row) => row.session === sessionId); +} + +function numberOf(value: unknown): number { + return typeof value === "number" && Number.isFinite(value) ? value : 0; +} + +/** Heal model: the configured pair first, the calling route after. */ +function healRoute(deps: LcmCommandDeps, agent: Agent | undefined): DshRoute | undefined { + const provider = deps.summarizationProvider; + const model = deps.summarizationModel; + if (provider !== undefined && provider.length > 0 && model !== undefined && model.length > 0) { + return { provider, model }; + } + return agentRoute(agent); +} + +/** Pressure override for one routed model: exact match, like the seam. */ +function policyFor( + deps: LcmCommandDeps, + route: DshRoute | undefined, +): ModelCompactPolicyConfig | undefined { + if (route === undefined) return undefined; + return deps.modelPolicies?.find( + (policy) => policy.provider === route.provider && policy.model === route.model, + ); +} + +function pressureZone(deps: LcmCommandDeps, route: DshRoute | undefined): ThresholdConfig { + const override = policyFor(deps, route)?.thresholdRatio; + return mergeZone( + deps.thresholdRatio === undefined ? {} : { swapAtRatio: deps.thresholdRatio }, + override === undefined ? undefined : { swapAtRatio: override }, + ); +} + +function lastPassLine(passRows: readonly Record[]): string { + const last = passRows[passRows.length - 1]; + if (!last) return "none recorded"; + const outcome = typeof last.outcome === "string" ? last.outcome : "unknown"; + if (outcome === "stored") { + return `stored (${numberOf(last.leaf)} leaf, ${numberOf(last.condensed)} condensed)`; + } + if (outcome === "nothing") { + const reason = typeof last.reason === "string" ? last.reason : "already memory"; + return `nothing (${reason})`; + } + if (outcome === "failed") { + const error = typeof last.error === "string" ? last.error : "unknown error"; + return `failed (${error.slice(0, 120)})`; + } + return outcome; +} + +/** Enriched status: the mirrored counts plus the meter window, thresholds, + * and the last pass outcome. Count lines stay additive to the old shape. */ +export async function handleStatus( + ctx: Context, + deps: LcmCommandDeps, + invocation: CommandInvocation, +): Promise { + const stores = deps.stores(); + if (stores.size === 0) return { kind: "success", text: "LCM has mirrored no sessions yet." }; + const sessionId = String(invocation.agent.session.id); + const lines = [...stores].map(([id, store]) => { + const messages = store.messageCount(); + // Summary payloads stay small beside message bodies, so listing them costs little. + const summaries = store.allSummaries().length; + return `${id}: ${messages} messages, ${summaries} summaries`; + }); + lines.unshift(`LCM sidecar status: ${stores.size} session(s)`); + const window = await resolveWindowTokens(ctx, invocation.agent); + const total = measureTotalTokens(ctx, invocation.agent); + if (window !== undefined && total !== undefined) { + const occupancy = (total / window) * 100; + lines.push( + `window (${sessionId}): ${total}/${fmtTokens(window)} tokens (${occupancy.toFixed(0)}%)`, + ); + const route = agentRoute(invocation.agent); + const resolved = effectiveThresholds(pressureZone(deps, route), window); + lines.push( + `thresholds: swap ${(resolved.swap * 100).toFixed(0)}% / recut ${(resolved.recut * 100).toFixed(0)}%${resolved.clamped ? " (clamped)" : ""}`, + ); + } else if (window !== undefined) { + lines.push(`window: ${fmtTokens(window)} (pressure unread: token meter unavailable)`); + lines.push("thresholds: window known, pressure unknown"); + } else { + lines.push("window: unknown (no model window resolved)"); + lines.push("thresholds: window unknown"); + } + const passes = sessionRows(metricRows(deps, sessionId), sessionId).filter( + (row) => row.kind === "pass", + ); + lines.push(`last pass (${sessionId}): ${lastPassLine(passes)}`); + return { kind: "success", text: lines.join("\n") }; +} + +/** Session-scoped metrics rollup over the metrics rows, in the OpenCode vocabulary. */ +export function formatSessionReport( + rows: readonly Record[], + sessionId: string, +): string { + const mine = sessionRows(rows, sessionId); + if (mine.length === 0) return "LCM metrics: no rows recorded for this session yet."; + const lines = [ + `LCM metrics: ${mine.length} row(s) for session ${sessionId} (${rows.length} in the log).`, + ]; + const passes = mine.filter((row) => row.kind === "pass"); + const stored = passes.filter((row) => row.outcome === "stored").length; + const nothing = passes.filter((row) => row.outcome === "nothing").length; + const failed = passes.filter((row) => row.outcome === "failed").length; + const leaf = passes.reduce((sum, row) => sum + numberOf(row.leaf), 0); + const condensed = passes.reduce((sum, row) => sum + numberOf(row.condensed), 0); + lines.push( + `passes: ${passes.length} (stored ${stored}, nothing ${nothing}, failed ${failed}); ${leaf} leaf, ${condensed} condensed`, + ); + const checkpoints = mine.filter((row) => row.kind === "compaction-checkpoint").length; + lines.push(`checkpoints: ${checkpoints} commit(s)`); + const backfills = mine.filter((row) => row.kind === "backfill"); + const backfilled = backfills.reduce((sum, row) => sum + numberOf(row.ingested), 0); + lines.push(`backfill: ${backfills.length} session(s), ${backfilled} ingested`); + const recall = mine.filter((row) => row.kind === "recall"); + const recallOutcomes = ["hit", "miss", "refused", "error"].flatMap((outcome) => { + const count = recall.filter((row) => row.outcome === outcome).length; + return count === 0 ? [] : [`${outcome} ${count}`]; + }); + lines.push( + `recall: ${recall.length}${recallOutcomes.length === 0 ? "" : ` (${recallOutcomes.join(", ")})`}`, + ); + const retrieval = mine.filter((row) => row.kind === "retrieval"); + const stops = ["answered", "refused", "failed", "stopped"].flatMap((stop) => { + const count = retrieval.filter((row) => row.stop === stop).length; + return count === 0 ? [] : [`${stop} ${count}`]; + }); + lines.push(`retrieval: ${retrieval.length}${stops.length === 0 ? "" : ` (${stops.join(", ")})`}`); + lines.push(`last pass: ${lastPassLine(passes)}`); + const text = lines.join("\n"); + return text.length <= 2048 ? text : `${text.slice(0, 2040)}...`; +} + +export function handleReport(deps: LcmCommandDeps, invocation: CommandInvocation): CommandResult { + const sessionId = String(invocation.agent.session.id); + return { kind: "success", text: formatSessionReport(metricRows(deps, sessionId), sessionId) }; +} + +/** Timestamped copy of the calling session's sidecar, mode 0600. */ +export function handleBackup( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const store = deps.stores().get(sessionId); + if (!store) return { kind: "error", text: "LCM backup: no stored history for this session yet." }; + const target = + args[0] === undefined ? `${deps.storePathFor(sessionId)}.backup-${STAMP()}` : resolve(args[0]); + try { + store.backup(target); + try { + chmodSync(target, 0o600); + } catch {} + } catch (error) { + return { + kind: "error", + text: `LCM backup failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } + return { kind: "success", text: `LCM backup written: ${target} (mode 0600).` }; +} + +/** JSONL export of the calling session's sidecar, mode 0600. */ +export function handleExport( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const store = deps.stores().get(sessionId); + if (!store) return { kind: "error", text: "LCM export: no stored history for this session yet." }; + const target = + args[0] === undefined ? join(deps.sidecarDir, `lcm-export-${STAMP()}.jsonl`) : resolve(args[0]); + try { + const out = exportTranscript(store); + writeFileSync(target, `${out.toJSONL()}\n`, { encoding: "utf8", mode: 0o600 }); + try { + chmodSync(target, 0o600); + } catch {} + return { + kind: "success", + text: `LCM export: ${out.messages.length} messages + ${out.summaries.length} summaries -> ${target} (mode 0600).`, + }; + } catch (error) { + return { + kind: "error", + text: `LCM export failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } +} + +type SaveFlag = { kind: "path"; path: string } | { kind: "absent" } | { kind: "missing-path" }; + +function parseSaveFlag(flags: readonly string[]): SaveFlag { + for (let i = 0; i < flags.length; i++) { + const flag = flags[i]; + if (flag === "--save") { + const next = flags[i + 1]; + // `--save --repair` is a missing value, not a file named `--repair`. + return next === undefined || next.startsWith("--") + ? { kind: "missing-path" } + : { kind: "path", path: next }; + } + if (flag?.startsWith("--save=")) { + const path = flag.slice("--save=".length); + return path === "" ? { kind: "missing-path" } : { kind: "path", path }; + } + } + return { kind: "absent" }; +} + +/** Value after `--name value` or inside `--name=value`. A missing value or a + * follow-on flag reads as absent, so `--save --repair` never names a file. */ +function valueOf(args: readonly string[], name: string): string | undefined { + const inline = args.find((arg) => arg.startsWith(`${name}=`)); + if (inline !== undefined) return inline.slice(name.length + 1); + const index = args.indexOf(name); + const value = index >= 0 ? args[index + 1] : undefined; + return value !== undefined && !value.startsWith("--") ? value : undefined; +} + +function writeDumpTarget(target: string, text: string): void { + // Unique temp with exclusive creation, then an atomic rename: a planted + // predictable path is never touched and the destination swaps at once. + const dir = dirname(target); + const base = basename(target); + let tmp = ""; + for (let attempt = 0; attempt < 5; attempt++) { + const candidate = join( + dir, + `.${base}.${process.pid}.${Date.now()}.${Math.floor(Math.random() * 1e9)}.tmp`, + ); + try { + writeFileSync(candidate, text, { encoding: "utf8", mode: 0o600, flag: "wx" }); + tmp = candidate; + break; + } catch (error) { + if ((error as { code?: string })?.code === "EEXIST") continue; + // A write can create the private file and then fail (disk full), so + // remove the candidate before surfacing the error. EEXIST keeps its + // continue above and never unlinks a pre-existing collision. + try { + unlinkSync(candidate); + } catch {} + throw error; + } + } + if (tmp === "") throw new Error(`LCM dump: cannot create temp file beside ${target}.`); + try { + chmodSync(tmp, 0o600); + renameSync(tmp, target); + } catch (error) { + try { + unlinkSync(tmp); + } catch {} + throw error; + } +} + +function pluginLabel(): string { + const version = packageVersionNear(fileURLToPath(import.meta.url), "deepseek-lossless"); + return `deepseek-lossless ${version ?? "unknown"}`; +} + +/** Redacted bundle via the core helper, inline or `--save`d to a 0600 file. */ +export async function handleDiagnose( + ctx: Context, + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): Promise { + const sessionId = String(invocation.agent.session.id); + const store = deps.stores().get(sessionId); + if (!store) { + return { kind: "error", text: "LCM diagnose: no stored history for this session yet." }; + } + const save = parseSaveFlag(args); + if (save.kind === "missing-path") { + return { + kind: "error", + text: "LCM diagnose --save: name the file to write, e.g. `/lcm diagnose --save lcm-diagnose.txt`.", + }; + } + const stats = store.stats(); + const integrity = checkIntegrity(store); + const route = agentRoute(invocation.agent); + const policy = policyFor(deps, route); + const zone = pressureZone(deps, route); + const model = + route === undefined + ? "(session model)" + : modelKey({ provider: route.provider, id: route.model }); + const window = await resolveWindowTokens(ctx, invocation.agent); + const bundle = buildDiagnoseBundle({ + versions: { plugin: pluginLabel, host: () => "deepseek harness" }, + runtime: { + node: process.version, + platform: process.platform, + arch: process.arch, + mode: "plugin", + }, + session: { + store: basename(deps.storePathFor(sessionId)), + file: "(deepseek session)", + format: "deepseek session log", + }, + facts: { + stats, + integrity: { checks: integrity.checks, findings: integrity.findings, repairs: [] }, + depthRows: depthStats(store.allSummaries()), + pin: null, + model, + window: + window === undefined + ? null + : { tokens: window, effective: effectiveThresholds(zone, window) }, + chain: { level: "info", state: "summarizer: deepseek ladder (leaf plus condensed DAG)" }, + zoneConfigured: policy !== undefined, + config: { + swapAtRatio: zone.swapAtRatio ?? 0.7, + recutAtRatio: zone.recutAtRatio ?? 0.85, + assemblyEnabled: false, + }, + keepRecentTokens: deps.retainTokens ?? 0, + configPath: "plugin options", + }, + config: { + effective: JSON.stringify({ + swapAtRatio: zone.swapAtRatio ?? 0.7, + recutAtRatio: zone.recutAtRatio ?? 0.85, + ...(deps.retainTokens === undefined ? {} : { retainTokens: deps.retainTokens }), + ...(deps.retainRatio === undefined ? {} : { retainRatio: deps.retainRatio }), + ...(deps.headroomTokens === undefined ? {} : { headroomTokens: deps.headroomTokens }), + ...(deps.modelPolicies === undefined ? {} : { modelPolicies: deps.modelPolicies }), + }), + path: "plugin options", + }, + metrics: sessionRows(metricRows(deps, sessionId), sessionId), + redact: redactSecrets, + home: homedir(), + }); + if (save.kind === "path") { + const target = resolve(save.path); + try { + writeFileSync(target, `${bundle}\n`, { encoding: "utf8", mode: 0o600 }); + chmodSync(target, 0o600); + } catch (error) { + return { + kind: "error", + text: `LCM diagnose: cannot write ${target} (${error instanceof Error ? error.message : String(error)}).`, + }; + } + return { + kind: "success", + text: `LCM diagnose: wrote ${Buffer.byteLength(bundle, "utf8")} bytes to ${target} (mode 0600); the block is in the file, not in this message.`, + }; + } + return { kind: "success", text: bundle }; +} + +/** Restore rows from a file written by export. Dedup is the importer's own: + * re-importing the same file inserts nothing and reports the skips. */ +export function handleImport( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const arg = args[0]; + if (arg === undefined || arg.length === 0) { + return { + kind: "error", + text: "LCM import: name a file written by /lcm export, e.g. `/lcm import lcm-export.jsonl`.", + }; + } + let text: string; + try { + text = readFileSync(resolve(arg), "utf8"); + } catch (error) { + return { + kind: "error", + text: `LCM import failed: cannot read ${arg} (${error instanceof Error ? error.message : String(error)}).`, + }; + } + const parsed = parseExport(text); + if ("error" in parsed) { + return { kind: "error", text: `LCM import refused: ${parsed.error}.` }; + } + const store = deps.storeFor?.(sessionId) ?? deps.stores().get(sessionId); + if (!store) { + return { kind: "error", text: "LCM import: no stored history for this session yet." }; + } + let result: ReturnType; + try { + result = importTranscript(store, parsed.rows); + } catch (error) { + return { + kind: "error", + text: `LCM import failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } + if (result.refused !== undefined) { + return { kind: "error", text: `LCM import refused: ${result.refused}.` }; + } + const restored = result.messagesInserted + result.summariesInserted; + const skipped = result.messagesSkipped + result.summariesSkipped; + return { + kind: "success", + text: `LCM import: ${restored} row(s) restored (${result.messagesInserted} messages, ${result.summariesInserted} summaries); ${skipped} row(s) already present.`, + }; +} + +/** Read-only dump of the calling session's sidecar, opened with `queryOnly` + * so a dump never mutates the store. Free-text args sample one FTS5 query; + * `--save` writes the dump through the atomic writer instead of printing it. */ +export function handleDump( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const includeRemoved = args.includes("--include-removed"); + const saveValue = valueOf(args, "--save") || undefined; + const hasSave = args.includes("--save") || args.some((arg) => arg.startsWith("--save=")); + if (hasSave && saveValue === undefined) { + return { + kind: "error", + text: "LCM dump --save: name the file to write, e.g. `/lcm dump --save lcm-dump.txt`.", + }; + } + const save = saveValue; + const saveIndex = args.findIndex((arg) => arg === "--save" || arg.startsWith("--save=")); + const saveArgs = saveIndex === -1 ? new Set() : new Set([saveIndex]); + if (saveIndex !== -1 && args[saveIndex] === "--save" && save !== undefined) { + const valueIndex = args.findIndex((arg, i) => i > saveIndex && arg === save); + if (valueIndex !== -1) saveArgs.add(valueIndex); + } + const query = args.filter((arg, i) => !arg.startsWith("--") && !saveArgs.has(i)).join(" "); + const result = dumpSidecar({ + path: deps.storePathFor(sessionId), + sessionId, + redact: redactSecrets, + ...(includeRemoved ? { includeRemoved: true as const } : {}), + metricRows: metricRows(deps, sessionId), + ...(query.length > 0 ? { sampleQuery: query } : {}), + }); + if (result.status === "skipped") { + return { + kind: "error", + text: `LCM dump skipped: ${result.reason} (${result.detail}).`, + }; + } + if (save !== undefined) { + const target = resolve(save); + try { + writeDumpTarget(target, `${result.text}\n`); + } catch (error) { + return { + kind: "error", + text: `LCM dump failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } + return { + kind: "success", + text: `LCM dump: wrote ${Buffer.byteLength(result.text, "utf8")} bytes to ${target} (mode 0600).`, + }; + } + return { kind: "success", text: result.text }; +} + +/** Live stems guard gc: disk stems are sanitized session ids, so the check + * compares those, not raw ids, and fails closed by also covering the calling + * session even when its store is not in the map yet. */ +function liveStems(deps: LcmCommandDeps, sessionId: string): Set { + const stems = new Set(); + for (const id of deps.stores().keys()) stems.add(sanitizeSidecarId(id)); + stems.add(sanitizeSidecarId(sessionId)); + return stems; +} + +const GC_DEFAULT_DAYS = 30; + +function gcCandidates( + deps: LcmCommandDeps, + sessionId: string, + days: number, + onlySession: string | undefined, +): { + candidates: Array<{ id: string; path: string; sizeBytes: number; ageDays: number }>; + /** Sidecars skipped because a -shm file marks them open in another process. + * A crashed owner can leave a stale -shm behind; deleting it collects the sidecar. */ + heldOpen: number; +} { + const live = liveStems(deps, sessionId); + const refs = listSidecarRefs(deps.sidecarDir).filter((ref) => !live.has(ref.id)); + // A named session bypasses the age window: the filter itself is the intent. + const selected = + onlySession === undefined + ? refs + : refs.filter((ref) => ref.id === onlySession || ref.id === sanitizeSidecarId(onlySession)); + const cutoff = Date.now() - days * 86_400_000; + const candidates: Array<{ id: string; path: string; sizeBytes: number; ageDays: number }> = []; + let heldOpen = 0; + for (const ref of selected) { + try { + const stat = statSync(ref.path); + if (!stat.isFile()) continue; + let mtimeMs = stat.mtimeMs; + for (const suffix of ["-wal", "-shm"]) { + try { + mtimeMs = Math.max(mtimeMs, statSync(`${ref.path}${suffix}`).mtimeMs); + } catch {} + } + // A -shm file exists only while a process holds the database open. + // liveStems covers this process; the shm probe covers every other one + // sharing the directory, so gc never deletes an open database. + try { + statSync(`${ref.path}-shm`); + heldOpen += 1; + continue; + } catch {} + if (onlySession === undefined && mtimeMs > cutoff) continue; + candidates.push({ + id: ref.id, + path: ref.path, + sizeBytes: stat.size, + ageDays: Math.max(0, (Date.now() - mtimeMs) / 86_400_000), + }); + } catch { + continue; + } + } + return { candidates, heldOpen }; +} + +/** Age-based sidecar collection. Dry-run default; only --apply deletes, and + * it never touches a live sidecar. A no-op apply writes no export file. */ +export function handleGc( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const apply = args.includes("--apply"); + const sessionValue = valueOf(args, "--session") || undefined; + const hasSession = args.includes("--session") || args.some((arg) => arg.startsWith("--session=")); + if (hasSession && sessionValue === undefined) { + return { kind: "error", text: "LCM gc: --session takes a session id." }; + } + const hasOlderThan = + args.includes("--older-than") || args.some((arg) => arg.startsWith("--older-than=")); + if (hasOlderThan && valueOf(args, "--older-than") === undefined) { + return { kind: "error", text: "LCM gc: --older-than takes a number of days." }; + } + const onlySession = sessionValue; + if (onlySession !== undefined && liveStems(deps, sessionId).has(sanitizeSidecarId(onlySession))) { + return { + kind: "error", + text: `LCM gc: session ${onlySession} is open in this process; gc never deletes an open sidecar.`, + }; + } + if (onlySession !== undefined) { + // Explicit target: fail closed when another process holds it open, the + // same guarantee as the live guard above but across processes. + const target = listSidecarRefs(deps.sidecarDir).find( + (ref) => ref.id === onlySession || ref.id === sanitizeSidecarId(onlySession), + ); + if (target !== undefined) { + try { + statSync(`${target.path}-shm`); + return { + kind: "error", + text: `LCM gc: session ${onlySession} is open in another process; gc never deletes an open sidecar.`, + }; + } catch {} + } + } + const olderThanRaw = valueOf(args, "--older-than"); + const rawDays = olderThanRaw ?? String(GC_DEFAULT_DAYS); + // Strict shape stops suffixes like 2weeks from parsing as 2. + const daysMatch = /^(\d+(?:\.\d+)?)d?$/.exec(rawDays.trim()); + const days = daysMatch === null ? Number.NaN : Number(daysMatch[1]); + if (!Number.isFinite(days) || days < 0) { + return { kind: "error", text: "LCM gc: --older-than takes a number of days." }; + } + let found: { + candidates: Array<{ id: string; path: string; sizeBytes: number; ageDays: number }>; + heldOpen: number; + }; + try { + found = gcCandidates(deps, sessionId, days, onlySession); + } catch (error) { + return { + kind: "error", + text: `LCM gc failed: cannot list ${deps.sidecarDir} (${error instanceof Error ? error.message : String(error)}).`, + }; + } + const candidates = found.candidates; + const heldNote = + found.heldOpen === 0 + ? "" + : ` ${found.heldOpen} sidecar(s) held open by another process were skipped (a crashed owner leaves a stale -shm behind; deleting it collects the sidecar).`; + if (onlySession !== undefined && candidates.length === 0) { + return { + kind: "success", + text: `LCM gc: no sidecar for session ${onlySession} (the current session is never a candidate).`, + }; + } + if (candidates.length === 0) { + return { + kind: "success", + text: `LCM gc: nothing eligible (no sidecar older than ${days} day(s); live sidecars are never candidates).${heldNote}`, + }; + } + const lines = candidates.map( + (c) => ` ${c.id}: ${(c.sizeBytes / 1024).toFixed(0)} KB, ${c.ageDays.toFixed(1)} day(s) old`, + ); + if (!apply) { + // A named session bypasses the age window, so the hint omits an --older-than that would have no effect. + const selectors = + (onlySession !== undefined ? ` --session ${onlySession}` : "") + + (onlySession === undefined && olderThanRaw !== undefined + ? ` --older-than ${olderThanRaw}` + : ""); + return { + kind: "success", + text: `LCM gc dry-run: ${candidates.length} sidecar(s) removable:\n${lines.join("\n")}\nRun \`/lcm gc --apply${selectors}\` to export then delete them. Backups are never touched.${heldNote}`, + }; + } + const stamp = STAMP(); + let removed = 0; + let exported = 0; + let failed = 0; + let skippedOpen = found.heldOpen; + const written: string[] = []; + for (const candidate of candidates) { + try { + // Re-check right before deleting: another process may have opened the + // sidecar since the sweep probed it. + statSync(`${candidate.path}-shm`); + skippedOpen += 1; + continue; + } catch {} + try { + const probe = new LcmStore(candidate.path, { queryOnly: true }); + let rows = ""; + try { + rows = exportTranscript(probe).toJSONL(); + } finally { + probe.close(); + } + if (rows.length > 0) { + // One file per sidecar: ordinals restart at 0 per store. + const target = join(deps.sidecarDir, `lcm-export-gc-${candidate.id}-${stamp}.jsonl`); + writeFileSync(target, `${rows}\n`, { encoding: "utf8", mode: 0o600 }); + try { + chmodSync(target, 0o600); + } catch {} + written.push(target); + exported++; + } + dropSidecarFiles(candidate.path); + removed++; + } catch { + failed++; + } + } + const heldTail = skippedOpen === 0 ? "" : `; ${skippedOpen} skipped (open in another process)`; + if (exported > 0) { + return { + kind: "success", + text: `LCM gc: removed ${removed} of ${candidates.length} sidecar(s); ${exported} transcript(s) exported to ${written.join(", ")}${failed === 0 ? "" : `; ${failed} skipped (migrate or inspect first)`}${heldTail}.`, + }; + } + return { + kind: "success", + text: `LCM gc: removed ${removed} of ${candidates.length} sidecar(s); transcripts were empty so no export file was written${failed === 0 ? "" : `; ${failed} skipped (migrate or inspect first)`}${heldTail}.`, + }; +} + +/** Generation-gated migration. A store this process holds open is already + * current, so live stems are excluded: opening a second writer would only + * race the live session. Dry-run default; only --apply opens writers. */ +export function handleMigrate( + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): CommandResult { + const sessionId = String(invocation.agent.session.id); + const apply = args.includes("--apply"); + const live = liveStems(deps, sessionId); + const refs = listSidecarRefs(deps.sidecarDir).filter((ref) => !live.has(ref.id)); + const stale: Array<{ id: string; path: string; generation: number }> = []; + let current = deps.stores().size; + let broken = 0; + for (const ref of refs) { + const generation = storeGeneration(ref.path); + if (generation === undefined) broken++; + else if (generation < SCHEMA_VERSION) stale.push({ ...ref, generation }); + else current++; + } + if (stale.length === 0) { + return { + kind: "success", + text: `LCM migrate: nothing to do (${current} sidecar(s) at generation ${SCHEMA_VERSION}${broken === 0 ? "" : `, ${broken} unreadable`}).`, + }; + } + const summary = stale + .map((s) => ` ${s.id}: generation ${s.generation} -> ${SCHEMA_VERSION}`) + .join("\n"); + if (!apply) { + return { + kind: "success", + text: `LCM migrate dry-run: ${stale.length} sidecar(s) older than generation ${SCHEMA_VERSION} (${current} current${broken === 0 ? "" : `, ${broken} unreadable`}):\n${summary}\nRun \`/lcm migrate --apply\` to upgrade them in place.`, + }; + } + let migrated = 0; + let failed = 0; + for (const s of stale) { + try { + const candidate = new LcmStore(s.path); + candidate.close(); + migrated++; + } catch { + failed++; + } + } + return { + kind: "success", + text: `LCM migrate: upgraded ${migrated} of ${stale.length} sidecar(s) to generation ${SCHEMA_VERSION}${failed === 0 ? "" : `, ${failed} failed`}.`, + }; +} + +type MaxNodesFlag = { kind: "absent" } | { kind: "value"; value: number } | { kind: "invalid" }; + +function parseMaxNodesFlag(flags: readonly string[]): MaxNodesFlag { + for (let i = 0; i < flags.length; i++) { + const flag = flags[i]; + if (flag === "--max-nodes") { + const next = flags[i + 1]; + if (next === undefined || next.startsWith("--")) return { kind: "invalid" }; + return maxNodesValue(next); + } + if (flag?.startsWith("--max-nodes=")) { + return maxNodesValue(flag.slice("--max-nodes=".length)); + } + } + return { kind: "absent" }; +} + +function maxNodesValue(raw: string): MaxNodesFlag { + const value = Number(raw); + return Number.isInteger(value) && value > 0 ? { kind: "value", value } : { kind: "invalid" }; +} + +/** A dry run reads the plan and makes no provider call. */ +const healDryRunLlm: LlmComplete = Object.assign( + async (): Promise => { + throw new Error("lcm: a heal dry run never calls the model"); + }, + { metrics: appendMetric }, +); + +function healReport(outcome: HealOutcome): string { + if (outcome.dryRun) { + return outcome.counts.candidates === 0 + ? "LCM heal dry-run: no truncated summaries or stale ancestors; nothing to repair." + : `LCM heal dry-run: ${outcome.counts.candidates} candidate(s), ${outcome.counts.tainted} tainted, ${outcome.sourceTokens} source tokens; run \`/lcm heal --apply\` to repair.`; + } + const stop = outcome.stop === "done" ? "" : ` (stopped: ${outcome.stop})`; + const changed = outcome.counts.changed === 0 ? "" : `, changed ${outcome.counts.changed}`; + const blockedSource = + outcome.counts.blockedSource === 0 ? "" : ` (${outcome.counts.blockedSource} source)`; + return `LCM heal: repaired ${outcome.counts.repaired} clean, improved ${outcome.counts.improved}, failed ${outcome.counts.failed}, blocked ${outcome.counts.blocked}${blockedSource}${changed}, remaining ${outcome.counts.notAttempted}; ${outcome.calls} model call(s)${stop}.`; +} + +const healInFlight = new Set(); + +/** Repair mechanically truncated nodes from their stored sources. Dry-run + * default; only --apply calls the model. Budgets ride the plugin config + * through these deps; unset keys keep the core heal defaults. */ +export async function handleHeal( + ctx: Context, + deps: LcmCommandDeps, + invocation: CommandInvocation, + args: readonly string[], +): Promise { + const sessionId = String(invocation.agent.session.id); + const apply = args.includes("--apply"); + const maxNodes = parseMaxNodesFlag(args); + if (maxNodes.kind === "invalid") { + return { + kind: "error", + text: "LCM heal: --max-nodes takes a positive whole number, e.g. `/lcm heal --apply --max-nodes 8`.", + }; + } + const store = deps.stores().get(sessionId); + if (!store) { + return { kind: "error", text: "LCM heal: no stored history for this session yet." }; + } + const span = store.entrySpan(); + if (!span) { + return { kind: "error", text: "LCM heal: this session has no stored entries yet." }; + } + if (store.hasLiveRun()) { + return { + kind: "error", + text: "LCM heal: a compaction pass is in flight; retry when it finishes.", + }; + } + if (healInFlight.has(sessionId)) { + return { kind: "error", text: "LCM heal: a repair is already running for this session." }; + } + healInFlight.add(sessionId); + try { + const maxNodeInput = maxNodes.kind === "value" ? { maxNodes: maxNodes.value } : {}; + if (!apply) { + const outcome = await runHeal( + store, + { + session: sessionId, + redact: redactSecrets, + dryRun: true, + ...healRunBudgets(deps), + ...maxNodeInput, + }, + healDryRunLlm, + ); + return { kind: "success", text: healReport(outcome) }; + } + const route = healRoute(deps, invocation.agent); + const host = dshModelHost(ctx, { + ...(route === undefined ? {} : { defaultRoute: route }), + signal: invocation.signal, + sessionId, + }); + const llm = makeLlm(host, healLlmConfig(deps), { + session: sessionId, + stage: "heal", + span: { firstEntryId: span.first, lastEntryId: span.last }, + }); + if (!llm) { + return { + kind: "error", + text: "LCM heal: no summarizer model resolves for this session; nothing was repaired.", + }; + } + const outcome = await runHeal( + store, + { session: sessionId, redact: redactSecrets, ...healRunBudgets(deps), ...maxNodeInput }, + llm, + ); + return { kind: "success", text: healReport(outcome) }; + } catch (error) { + return { + kind: "error", + text: `LCM heal failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } finally { + healInFlight.delete(sessionId); + } +} + +/** Session-less lookup for tests: the gc dry-run default window in days. */ +export function gcDryRunDefaultDays(): number { + return GC_DEFAULT_DAYS; +} + +/** Session-less lookup for tests, consulted by dispose: the ids of sessions with a heal in flight. */ +export function healInFlightSessions(): readonly string[] { + return [...healInFlight]; +} diff --git a/packages/deepseek/src/metrics.ts b/packages/deepseek/src/metrics.ts new file mode 100644 index 0000000..bd858f5 --- /dev/null +++ b/packages/deepseek/src/metrics.ts @@ -0,0 +1,24 @@ +/** Per-composition JSONL sink beside the sidecar directory. The rows live in + * the core so adapters cannot drift; this module keeps the import path its + * callers already use and binds one recorder per composition. */ + +export { + METRICS_FILE, + METRICS_KEEP_GENERATIONS, + METRICS_MAX_BYTES, + appendMetricRow, + readMetricRows, +} from "lossless-core"; +export { + metricsFilePath as metricsPath, + rotateMetricsIfNeeded as rotateIfNeeded, +} from "lossless-core"; + +import { appendMetricRow } from "lossless-core"; + +/** One JSONL recorder bound to a sidecar directory. Appends through the + * shared core seam, so placement (beside the sidecars), 0600 file modes, + * and the rotation policy stay identical to the OpenCode precedent. */ +export function createRecord(dir: string): (row: Record) => void { + return (row) => appendMetricRow(dir, row); +} diff --git a/packages/deepseek/src/model-host.ts b/packages/deepseek/src/model-host.ts new file mode 100644 index 0000000..6ec4ad2 --- /dev/null +++ b/packages/deepseek/src/model-host.ts @@ -0,0 +1,184 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import { BlockAssembler } from "@deepseek-ai/dsh-llm"; +import type { FinishReason, GenerateOptions, TokenUsage } from "@deepseek-ai/dsh-llm"; +import type { + CompletionRequest, + ModelAnswer, + ModelHost, + ModelRef, + Notices, + Usage, +} from "lossless-core"; +import { zeroCostUsageOfTokens } from "lossless-core"; + +/** One model route on the composition: the provider key that selects the + * adapter instance, and the model id that adapter serves. */ +export interface DshRoute { + readonly provider: string; + readonly model: string; +} + +/** Host options for one composition: default route, abort, and session stamp. */ +export interface DshHostOptions { + /** The route `session()` answers with. Absent until the plugin wires config. */ + readonly defaultRoute?: DshRoute; + /** Abort for every call this host makes. The reader owns its own signal. */ + readonly signal?: AbortSignal; + /** Calling session stamped into GenerateOptions so adapters can attach + * per-conversation transport (e.g. a session routing header). Absent sends + * nothing. */ + readonly sessionId?: string; +} + +/** One ref for one route. `reasoning` stays false: core adds + * REASONING_HEADROOM to the request cap when it is true (lossless-core + * llm.ts requestOf), and no harness source hands this host a synchronous + * boolean. resolveModelInfo is async and the DeepSeek adapter always reports + * an efforts table, off-only when thinking is disabled, never a per-model + * boolean (harness llm-deepseek model-info.ts modelInfo); the sibling + * opencode adapter hardwires the same false (opencode model-host.ts). A + * reasoning model therefore never silently spends the output cap here. */ +export function dshRef(route: DshRoute): ModelRef { + return { key: `${route.provider}/${route.model}`, reasoning: false, model: route }; +} + +/** The calling session's route: header first, agent options second. */ +export function agentRoute(agent: Agent | undefined): DshRoute | undefined { + // requestHeader crosses the host boundary and can throw. + try { + const header = agent?.session.requestHeader?.()?.config; + if ( + header !== undefined && + typeof header.provider === "string" && + header.provider.length > 0 && + typeof header.model === "string" && + header.model.length > 0 + ) { + return { provider: header.provider, model: header.model }; + } + } catch {} + const options = agent?.options as { provider?: unknown; model?: unknown } | undefined; + if ( + typeof options?.provider === "string" && + options.provider.length > 0 && + typeof options?.model === "string" && + options.model.length > 0 + ) { + return { provider: options.provider, model: options.model }; + } + return undefined; +} + +/** The core port over one streaming call. `session()` answers the configured + * default route, `find()` builds a ref for any well-formed request and lets + * the stream fail `NO_ADAPTER` for routes nothing registered, and `complete()` + * sends the pre-joined prompt as one user message with `purpose: compaction`. */ +export function dshModelHost(ctx: Context, options?: DshHostOptions): ModelHost { + const signal = options?.signal; + const fallback = options?.defaultRoute; + const sessionId = options?.sessionId; + return { + notices: { kind: "none" } satisfies Notices, + session: () => (fallback === undefined ? undefined : dshRef(fallback)), + find: (requested) => + requested.provider.length > 0 && requested.id.length > 0 + ? dshRef({ provider: requested.provider, model: requested.id }) + : undefined, + complete: (model, request) => complete(ctx, { signal, ref: model, request, sessionId }), + }; +} + +interface CompleteArgs { + readonly signal: AbortSignal | undefined; + readonly sessionId: string | undefined; + readonly ref: ModelRef; + readonly request: CompletionRequest; +} + +async function complete(ctx: Context, args: CompleteArgs): Promise { + const assembler = new BlockAssembler(); + const options: GenerateOptions = { + provider: args.ref.model.provider, + model: args.ref.model.model, + messages: [{ role: "user", content: [{ type: "text", text: args.request.prompt }] }], + maxTokens: args.request.maxTokens, + purpose: "compaction", + ...(args.signal === undefined ? {} : { signal: args.signal }), + // Brand is harness-side identity; the string carries it. Omitted when the + // caller has no session, so direct calls stay routable without one. + ...(args.sessionId === undefined + ? {} + : { sessionId: args.sessionId as GenerateOptions["sessionId"] }), + }; + try { + for await (const chunk of ctx.llm.stream(options)) assembler.push(chunk); + } catch (error) { + return { + outcome: "failed", + reason: message(error), + capped: false, + stop: "threw", + usage: usageOf(assembler.usage), + }; + } + return answerOf(assembler); +} + +/** The terminal chunk decides, with the same words Pi uses: `stop` answers, + * `max-tokens` fails capped so the ladder stores nothing partial, and anything + * else fails open with the provider's own code in the reason. */ +function answerOf(assembler: BlockAssembler): ModelAnswer { + const usage = usageOf(assembler.usage); + const finish: FinishReason = assembler.finish; + switch (finish.kind) { + case "stop": + return { outcome: "answer", text: joinedText(assembler), stop: finish.kind, usage }; + case "max-tokens": + return { + outcome: "failed", + reason: "generation hit the token cap and the summary is incomplete", + capped: true, + stop: finish.kind, + usage, + }; + case "tool-calls": + return { + outcome: "failed", + reason: "the model returned tool calls for a request that sent no tools", + capped: false, + stop: finish.kind, + usage, + }; + case "aborted": + case "error": + return { + outcome: "failed", + reason: `${finish.failure.code}: ${finish.failure.message}`, + capped: false, + stop: finish.kind, + usage, + }; + default: { + const _exhaustive: never = finish; + throw new Error(`unhandled finish kind: ${JSON.stringify(_exhaustive)}`); + } + } +} + +function joinedText(assembler: BlockAssembler): string { + return assembler + .blocks() + .flatMap((block) => (block.type === "text" ? [block.text] : [])) + .join("\n"); +} + +/** The provider's counts with no prices attached. The zeroing lives in the + * core seam; this wrapper keeps only the TokenUsage to Usage shape. */ +export function usageOf(usage: TokenUsage | undefined): Usage | undefined { + return zeroCostUsageOfTokens(usage); +} + +function message(error: unknown): string { + return error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300); +} diff --git a/packages/deepseek/src/plugin.ts b/packages/deepseek/src/plugin.ts new file mode 100644 index 0000000..29cf528 --- /dev/null +++ b/packages/deepseek/src/plugin.ts @@ -0,0 +1,328 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { ModelCompactPolicyConfig } from "@deepseek-ai/dsh-compaction-basic"; +import z from "@deepseek-ai/schemastery"; +import { redactSecrets } from "lossless-core"; + +import { registerLcmCommand } from "./command.ts"; +import { defaultSidecarDir } from "./default-sidecar-dir.ts"; +import { LosslessSummarizerEngine } from "./engine.ts"; +import { backfillSession, historySourceFrom, mirrorEvent } from "./ingest.ts"; +import { healInFlightSessions } from "./lifecycle.ts"; +import { registerExpandQueryTool, registerRecallTools } from "./tools.ts"; + +/** Composition options for `applyLossless`. Each field carries its own default. */ +export interface LosslessPluginConfig { + /** Master flag. Nothing registers, mirrors, or mounts until true. + * @default false */ + readonly enabled?: boolean; + /** Automatic pressure compaction inside the turn. + * @default false */ + readonly auto?: boolean; + /** Route automatic pressure compaction through the DAG bracket instead of + * the seam. + * @default false */ + readonly bracketAuto?: boolean; + /** Directory for one sidecar file per session. Created when missing. + * @default global sidecar dir, "./.lossless" only as fallback */ + readonly sidecarDir?: string; + /** Tokens one summary aims for. Rides the engine deps, not the seam. + * @default 1500 */ + readonly summaryTokens?: number; + /** Tokens one leaf chunk may hold before it splits. Rides the engine + * deps, not the seam. + * @default 3000 */ + readonly leafChunkTokens?: number; + /** Provider the summarizer calls. Unset follows the routed model. + * @default unset */ + readonly summarizationProvider?: string; + /** Model the summarizer calls. Unset follows the routed model. + * @default unset */ + readonly summarizationModel?: string; + /** Pressure fraction of the priced window that trips automatic compaction. + * Rides the seam config. + * @default 0.8 */ + readonly thresholdRatio?: number; + /** Recent context retained as an absolute token budget. Mutually exclusive + * with retainRatio; the seam default shape is the ratio form (0.16). + * Rides the seam config; the manual path keeps the seam's 0 retain. */ + readonly retainTokens?: number; + /** Recent context retained as a fraction of the priced window. Mutually + * exclusive with retainTokens. Rides the seam. + * @default 0.16 */ + readonly retainRatio?: number; + /** Pressure headroom beyond the routed output reservation. The seam also + * defaults its summary cap to it. Rides the seam. + * @default 65536 */ + readonly headroomTokens?: number; + /** Exact provider/model policy overrides, merged over the defaults by the + * seam per routed target. Duplicate targets fail load. Rides the seam. */ + readonly modelPolicies?: ModelCompactPolicyConfig[]; + /** Tokens one summarizer request may cost before it is refused. + * Adapter-only: never rides the seam config. + * @default 60000 */ + readonly summarizerTokensPerCall?: number; + /** Tokens one pass may spend across all of its calls. Adapter-only. + * @default 250000 */ + readonly summarizerTokensPerPass?: number; + /** Tokens one heal run may spend across all of its calls. Adapter-only. + * @default 120000 */ + readonly summarizerTokensPerHeal?: number; + /** Estimated tokens one heal node may read as source. Unset derives 8 x + * summaryTokens, clamped to 3000 through 24000 and the request budget. + * Adapter-only. + * @default 8 x summaryTokens, clamped 3000..24000 */ + readonly healSourceTokens?: number; + /** Tool names whose spans compact without a model call. Empty keeps the + * LLM path. Adapter-only: never rides the seam config. + * @default [] */ + readonly mechanicalTools?: readonly string[]; +} + +import { + acquireCore, + isCoreHeld, + readMechanicalTools, + releaseCore, + type LosslessCore, +} from "./core.ts"; + +/** Close and evict every idle sidecar store in a core. Stores with a live run + * or an in-flight heal stay open so the commit or repair can land; failures + * are reported through `warn` instead of thrown so one bad store never blocks + * the rest. Runs on whichever dispose releases the last ownership count, host + * or realm. */ +export function closeIdleStores(core: LosslessCore, warn: (message: string) => void): void { + // Dispose-then-use is an error: a live run keeps its store open so the commit can land, and a session under heal repair keeps its store for the same reason. + const healing = new Set(healInFlightSessions()); + for (const [id, store] of core.sidecars.storesMap()) { + try { + if (store.hasLiveRun() || healing.has(id)) continue; + store.close(); + core.sidecars.evict(id); + } catch (error: unknown) { + warn( + `lossless dispose skipped closing session "${id}": ${error instanceof Error ? error.message : String(error)}`, + ); + } + } +} + +/** Cordis plugin name used by loader diagnostics. */ +export const name = "deepseek-lossless"; + +/** Capability services the composition reads. Session query stays out: + * history backfill degrades gracefully without it, so it remains optional. + * Compaction stays out: the composition provides it, so requiring it would + * deadlock startup. */ +export const inject = ["sessions", "tools", "commands", "llm", "tokenMeter"]; + +/** Loader schema for the composition options. Mirrors + * `LosslessPluginConfig` with loader defaults and generated docs. */ +export const Config = z.object({ + enabled: z.boolean().default(false), + auto: z.boolean().default(false), + bracketAuto: z.boolean().default(false), + sidecarDir: z.string(), + summaryTokens: z.number().step(1).min(1).default(1500), + leafChunkTokens: z.number().step(1).min(1).default(3000), + summarizationProvider: z.string(), + summarizationModel: z.string(), + thresholdRatio: z.number().min(0).max(1).default(0.8), + retainTokens: z.number().step(1).min(0), + // No default: the seam owns the ratio default and rejects the pair when + // both ride the config. A schema default here would also populate the ratio + // when only the absolute budget is set. + retainRatio: z.number().min(0).max(1), + headroomTokens: z.number().step(1).min(0).default(65536), + modelPolicies: z.array(z.any()), + summarizerTokensPerCall: z.number().step(1).min(1).default(60000), + summarizerTokensPerPass: z.number().step(1).min(1).default(250000), + summarizerTokensPerHeal: z.number().step(1).min(1).default(120000), + healSourceTokens: z.number().step(1).min(1), + mechanicalTools: z.array(z.string()).default([]), +}); + +/** Cordis plugin entrypoint. Delegates to `applyLossless` so the loader + * and direct composition share one path. */ +export function apply(ctx: Context, config: LosslessPluginConfig = {}): () => void { + return applyLossless(ctx, config); +} + +/** Split the engine arguments so the host composition and the preset-realm + * backend share one construction path. The seam object carries only seam + * policy knobs (the seam resolver rejects unknown keys); every adapter-only + * key stays on the engine deps. */ +export function buildEngineArgs( + config: LosslessPluginConfig, + core: LosslessCore, +): [ + seam: Exclude[1], undefined>, + deps: ConstructorParameters[2], +] { + const mechanicalTools = readMechanicalTools(config.mechanicalTools); + return [ + { + auto: config.auto ?? false, + ...(config.summarizationProvider === undefined + ? {} + : { summarizationProvider: config.summarizationProvider }), + ...(config.summarizationModel === undefined + ? {} + : { summarizationModel: config.summarizationModel }), + ...(config.thresholdRatio === undefined ? {} : { thresholdRatio: config.thresholdRatio }), + ...(config.retainTokens === undefined ? {} : { retainTokens: config.retainTokens }), + ...(config.retainRatio === undefined ? {} : { retainRatio: config.retainRatio }), + ...(config.headroomTokens === undefined ? {} : { headroomTokens: config.headroomTokens }), + ...(config.modelPolicies === undefined ? {} : { modelPolicies: config.modelPolicies }), + }, + { + store: (sessionId) => core.sidecars.peek(sessionId), + index: (sessionId) => core.sidecars.peekIndex(sessionId), + memory: core.memory, + redact: redactSecrets, + record: core.record, + ...(config.summaryTokens === undefined ? {} : { summaryTokens: config.summaryTokens }), + ...(config.leafChunkTokens === undefined ? {} : { leafChunkTokens: config.leafChunkTokens }), + ...(config.bracketAuto === undefined ? {} : { bracketAuto: config.bracketAuto }), + ...(config.summarizerTokensPerCall === undefined + ? {} + : { summarizerTokensPerCall: config.summarizerTokensPerCall }), + ...(config.summarizerTokensPerPass === undefined + ? {} + : { summarizerTokensPerPass: config.summarizerTokensPerPass }), + ...(config.summarizerTokensPerHeal === undefined + ? {} + : { summarizerTokensPerHeal: config.summarizerTokensPerHeal }), + ...(config.healSourceTokens === undefined + ? {} + : { healSourceTokens: config.healSourceTokens }), + ...(mechanicalTools === undefined ? {} : { mechanicalTools }), + }, + ]; +} + +/** Compose lossless memory into a harness context: mirror every session + * into its own sidecar, register the recall tools against the calling + * session, mount the summarizer engine as the compaction backend, and + * register `/lcm`. Disabled by default; enable on one composition at a + * time. The returned disposer stops the mirror, the tools, and the command, + * and closes every open sidecar. The engine follows the composition lifetime. + * @example + * ```ts + * const stop = applyLossless(ctx, { enabled: true }); + * ``` */ +export function applyLossless(ctx: Context, config: LosslessPluginConfig = {}): () => void { + if (config.enabled !== true) return () => undefined; + const sidecar = readSidecarDir(config.sidecarDir); + const dir = sidecar.dir; + logSidecarDir(sidecar.dir, sidecar.source, (message) => ctx.logger.info(message)); + const core = acquireCore(dir, true); + const { sidecars, record } = core; + const stopMirror = ctx.on("session/event", (live, event) => { + const entry = sidecars.entryFor(String(live.id)); + mirrorEvent(live, event, entry.store, entry.index); + }); + // Pre-mount backfill: sessions already live when the plugin mounts mirror + // their earlier history through the same mirrorEvent path as the live feed + // above, paged and capped so a huge log cannot stall mount. The query + // backend is optional: without it only new events are mirrored. Contained + // per session and never rejecting, so mount cannot fail over history. + const history = historySourceFrom(ctx); + if (history !== undefined) { + const pending = ctx.sessions.list().map((session) => { + const entry = sidecars.entryFor(String(session.id)); + return backfillSession(session, entry.store, history, entry.index, { record }).catch( + (error: unknown) => { + ctx.logger.warn( + `lossless backfill skipped session "${String(session.id)}": ${error instanceof Error ? error.message : String(error)}`, + ); + }, + ); + }); + void Promise.all(pending); + } + const stopTools = registerRecallTools(ctx, { + store: (sessionId) => + sessionId === undefined ? undefined : (sidecars.peek(sessionId) ?? undefined), + redact: redactSecrets, + record, + pastSessions: { list: (exclude) => sidecars.list(exclude) }, + }); + const stopExpandQuery = registerExpandQueryTool(ctx, { + store: (sessionId) => + sessionId === undefined ? undefined : (sidecars.peek(sessionId) ?? undefined), + redact: redactSecrets, + record, + pastSessions: { list: (exclude) => sidecars.list(exclude) }, + }); + // Hydration fallback: a resumed session rebuilds its last checkpoint + // from its sidecar until it compacts again. Reads only; a lookup miss + // creates no store. + const [seamConfig, engineDeps] = buildEngineArgs(config, core); + void new LosslessSummarizerEngine(ctx, seamConfig, engineDeps); + const stopCommand = registerLcmCommand(ctx, { + stores: () => sidecars.storesMap(), + storeFor: (sessionId) => sidecars.for(sessionId), + sidecarDir: dir, + storePathFor: (sessionId) => sidecars.pathFor(sessionId), + // The heal verbs budget from these; unset keys keep core defaults. + ...(config.summaryTokens === undefined ? {} : { summaryTokens: config.summaryTokens }), + ...(config.summarizerTokensPerCall === undefined + ? {} + : { summarizerTokensPerCall: config.summarizerTokensPerCall }), + ...(config.summarizerTokensPerHeal === undefined + ? {} + : { summarizerTokensPerHeal: config.summarizerTokensPerHeal }), + ...(config.healSourceTokens === undefined ? {} : { healSourceTokens: config.healSourceTokens }), + // The heal model rides these; unset keeps the agent route fallback. + ...(config.summarizationProvider === undefined + ? {} + : { summarizationProvider: config.summarizationProvider }), + ...(config.summarizationModel === undefined + ? {} + : { summarizationModel: config.summarizationModel }), + // Status and diagnose report effective values from these seam keys. + ...(config.thresholdRatio === undefined ? {} : { thresholdRatio: config.thresholdRatio }), + ...(config.retainTokens === undefined ? {} : { retainTokens: config.retainTokens }), + ...(config.retainRatio === undefined ? {} : { retainRatio: config.retainRatio }), + ...(config.headroomTokens === undefined ? {} : { headroomTokens: config.headroomTokens }), + ...(config.modelPolicies === undefined ? {} : { modelPolicies: config.modelPolicies }), + }); + // The guarded close below runs only for the last owner, so one mount never + // closes stores out from under a still-mounted sibling, and a repeat call + // only ever sweeps this mount's own released core. + let released = false; + const closeIdle = (): void => { + closeIdleStores(core, (message) => ctx.logger.warn(message)); + }; + return () => { + stopMirror(); + stopTools(); + stopExpandQuery(); + stopCommand(); + if (!released) { + released = true; + if (releaseCore(core)) closeIdle(); + return; + } + // Repeat dispose (e.g. after a heal drains): sweep my own released core, + // but only when no replacement has taken the directory since. + if (!isCoreHeld(core)) closeIdle(); + }; +} + +/** Resolve the sidecar dir an explicit option wins, else the global default. + * Exported so the preset row shares the host fallback without the log line. */ +export function readSidecarDir(raw: unknown): { dir: string; source: string } { + if (typeof raw === "string" && raw.trim().length > 0) { + return { dir: raw, source: "explicit" }; + } + const resolved = defaultSidecarDir(); + return { dir: resolved.dir, source: resolved.source }; +} + +function logSidecarDir(dir: string, source: string, info: (message: string) => void): void { + try { + info(`LCM sidecar dir: ${dir} (${source})`); + } catch {} +} diff --git a/packages/deepseek/src/preset.ts b/packages/deepseek/src/preset.ts new file mode 100644 index 0000000..4fc5834 --- /dev/null +++ b/packages/deepseek/src/preset.ts @@ -0,0 +1,60 @@ +import type { Context } from "@deepseek-ai/cordis"; + +import { acquireCore, releaseCore } from "./core.ts"; +import { LosslessSummarizerEngine } from "./engine.ts"; +import { + buildEngineArgs, + closeIdleStores, + Config, + readSidecarDir, + type LosslessPluginConfig, +} from "./plugin.ts"; + +/** Cordis plugin name used by loader diagnostics. Distinct from the host row + * so host and preset fibers are told apart in activation warnings. */ +export const name = "deepseek-lossless-preset"; + +/** Capability services the realm backend reads. Everything here resolves + * from the outer scope: preset groups isolate only the compaction service + * itself, so sessions, the model route, and the meter stay shared. Tools and + * commands stay out on purpose: the host row registers recall and `/lcm` + * once against the shared registries. */ +export const inject = ["sessions", "llm", "tokenMeter"]; + +/** The loader schema, identical to the host row's: the preset row takes the + * same composition options, including `enabled` and `sidecarDir`. */ +export { Config }; +export type { LosslessPluginConfig }; + +/** Realm-only compaction backend for agent presets. Pairs with the host + * row's `apply`/`applyLossless`, which must mount first: this entry mounts + * the summarizer engine against the shared host core (same sidecars, same + * record sink, same checkpoint memory) and registers nothing else. It + * mirrors nothing, and it closes only idle shared handles when its own + * release is the last ownership count left. + * + * Fails fast when the host row has not mounted, with the fix in the message, + * instead of running a split-brain copy. + * @example + * ```ts + * const stop = apply(ctx, { enabled: true }); + * ``` */ +export function apply(ctx: Context, config: LosslessPluginConfig = {}): () => void { + if (config.enabled !== true) return () => undefined; + const dir = readSidecarDir(config.sidecarDir).dir; + const core = acquireCore(dir, false); + const [seamConfig, engineDeps] = buildEngineArgs(config, core); + void new LosslessSummarizerEngine(ctx, seamConfig, engineDeps); + // Borrowers release like owners: when this release drops the last + // ownership count, sweep the shared idle stores so repeated mounts cannot + // accumulate open handles. Live and healing stores stay open. Once-only, + // like the host. + let released = false; + return () => { + if (released) return; + released = true; + if (releaseCore(core)) { + closeIdleStores(core, (message) => ctx.logger.warn(message)); + } + }; +} diff --git a/packages/deepseek/src/reader.ts b/packages/deepseek/src/reader.ts new file mode 100644 index 0000000..e718259 --- /dev/null +++ b/packages/deepseek/src/reader.ts @@ -0,0 +1,175 @@ +import type { Context } from "@deepseek-ai/cordis"; +import { + BlockAssembler, + ToolCallId, + createAssistantMessage, + createToolResultMessage, +} from "@deepseek-ai/dsh-llm"; +import type { + ContentBlock, + FinishReason, + GenerateOptions, + RequestMessage, + ToolSchema, +} from "@deepseek-ai/dsh-llm"; +import type { + ModelRef, + Reader, + ReaderCall, + ReaderReply, + ReaderRequest, + ReaderResult, + Reading, +} from "lossless-core"; + +import type { DshRoute } from "./model-host.ts"; +import { usageOf } from "./model-host.ts"; + +/** Reader options for one composition: route ref, abort, and session stamp. */ +export interface DshReaderOptions { + /** Model the reader calls, as provider plus model on one route. */ + readonly ref: ModelRef; + /** Cancellation for the streaming call; an abort fails the step. */ + readonly signal?: AbortSignal; + /** Calling session, stamped into each step so adapters can attach + * per-conversation transport. Absent sends nothing. */ + readonly sessionId?: string; +} + +/** The core reader port over one streaming call per step. The loop owns the + * transcript the way Pi does: each step sends the brief plus every answered + * call so far, and a step that asks for reads appends its assistant message + * before the results land. Tool schemas pass through untouched. */ +export function dshReader(ctx: Context, options: DshReaderOptions): Reader { + return { begin: (request) => beginReading(ctx, options, request) }; +} + +function beginReading(ctx: Context, options: DshReaderOptions, request: ReaderRequest): Reading { + const messages: RequestMessage[] = [ + { role: "user", content: [{ type: "text", text: request.brief }] }, + ]; + let opened = false; + return { + async next(results: readonly ReaderResult[]): Promise { + if (opened) for (const result of results) messages.push(toolResultMessage(result)); + opened = true; + const assembler = new BlockAssembler(); + const stream: GenerateOptions = { + provider: options.ref.model.provider, + model: options.ref.model.model, + system: request.systemPrompt, + messages, + tools: request.tools.map(toolSchema), + ...(options.signal === undefined ? {} : { signal: options.signal }), + ...(options.sessionId === undefined + ? {} + : { sessionId: options.sessionId as GenerateOptions["sessionId"] }), + }; + try { + for await (const chunk of ctx.llm.stream(stream)) assembler.push(chunk); + } catch (error) { + return { outcome: "failed", reason: message(error), usage: usageOf(assembler.usage) }; + } + const usage = usageOf(assembler.usage); + let blocks: ContentBlock[]; + try { + blocks = assembler.blocks(); + } catch (error) { + return { outcome: "failed", reason: message(error), usage }; + } + const reply = replyOf(assembler.finish, blocks, usage); + if (reply.outcome === "reads") { + messages.push(assistantMessage(options.ref, blocks)); + } + return reply; + }, + }; +} + +/** What one reader step means, with the same words Pi uses: error and aborted + * fail, calls ask for execution, and anything else answers, including a + * max-tokens stop whose partial text stays usable. */ +function replyOf( + finish: FinishReason, + blocks: ContentBlock[], + usage: ReaderReply["usage"], +): ReaderReply { + switch (finish.kind) { + case "error": + case "aborted": + return { + outcome: "failed", + reason: `${finish.failure.code}: ${finish.failure.message}`, + usage, + }; + case "stop": + case "max-tokens": + case "tool-calls": + break; + default: { + const _exhaustive: never = finish; + throw new Error(`unhandled finish kind: ${JSON.stringify(_exhaustive)}`); + } + } + const calls: ReaderCall[] = []; + for (const block of blocks) { + if (block.type !== "tool-call") continue; + const parsed: unknown = safeParse(block.arguments); + if (!isRecord(parsed)) { + return { + outcome: "failed", + reason: `tool call ${block.id} carries arguments that are not a JSON object`, + usage, + }; + } + calls.push({ id: block.id, name: block.name, arguments: parsed }); + } + if (calls.length > 0) return { outcome: "reads", calls, text: joinedText(blocks), usage }; + if (finish.kind === "tool-calls") { + return { outcome: "failed", reason: "the model finished with tool-calls but sent none", usage }; + } + return { outcome: "answered", text: joinedText(blocks), usage }; +} + +function toolSchema(tool: { name: string; description: string; parameters: unknown }): ToolSchema { + if (!isRecord(tool.parameters)) throw new Error(`tool ${tool.name} has no parameters object`); + return { name: tool.name, description: tool.description, parameters: tool.parameters }; +} + +function assistantMessage(ref: ModelRef, blocks: ContentBlock[]) { + return createAssistantMessage({ + content: blocks, + source: { provider: ref.model.provider, model: ref.model.model }, + }); +} + +function toolResultMessage(result: ReaderResult) { + return createToolResultMessage({ + callId: ToolCallId(result.call.id), + content: [{ type: "text", text: result.text }], + isError: false, + }); +} + +function joinedText(blocks: ContentBlock[]): string { + return blocks + .flatMap((block) => (block.type === "text" ? [block.text] : [])) + .join("\n") + .trim(); +} + +function safeParse(json: string): unknown { + try { + return JSON.parse(json) as unknown; + } catch { + return undefined; + } +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function message(error: unknown): string { + return error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300); +} diff --git a/packages/deepseek/src/sidecar-dump.ts b/packages/deepseek/src/sidecar-dump.ts new file mode 100644 index 0000000..5d5d9ac --- /dev/null +++ b/packages/deepseek/src/sidecar-dump.ts @@ -0,0 +1,279 @@ +import { existsSync } from "node:fs"; +import { basename } from "node:path"; + +import { + describeView, + expandView, + grepView, + historyManifest, + LcmStore, + StaleStoreError, +} from "lossless-core"; + +/** Max bytes one dump prints; callers copy the file when they need more. */ +const DUMP_MAX_BYTES = 16_384; +/** Summaries described in full; the header still counts the rest. */ +const DUMP_MAX_SUMMARIES = 10; +/** Metric rows kept from the tail of the session's rows. */ +const DUMP_MAX_METRIC_ROWS = 20; +/** Top-level summaries the manifest names. */ +const DUMP_MAX_HISTORY = 20; +/** Messages read as windows from the first summary. */ +const DUMP_MAX_SAMPLE_MESSAGES = 3; +/** Chars one sampled message window holds. */ +const DUMP_SAMPLE_CHARS = 400; +/** Bytes kept from each described summary; the tail names the cut. */ +const DUMP_MAX_SUMMARY_BYTES = 1_200; + +/** Why a sidecar could not be dumped. Each reason is a count the caller + * reports, never an error it raises. */ +export type DumpSkipReason = "missing" | "unreadable" | "older-generation"; + +export type SidecarDumpResult = + | { status: "dumped"; text: string } + | { status: "skipped"; reason: DumpSkipReason; detail: string }; + +export interface SidecarDumpOptions { + /** Sidecar file to open read-only. Never created when absent. */ + readonly path: string; + /** Session id naming the dump and filtering metric rows. */ + readonly sessionId: string; + /** Masks secret-shaped text on every displayed seam. */ + readonly redact: (text: string) => string; + /** Include matches from a branch that left the active path. */ + readonly includeRemoved?: boolean; + /** Metric rows for this sidecar directory, oldest first. */ + readonly metricRows?: readonly Record[]; + /** Optional FTS5 query sampled through grepView. */ + readonly sampleQuery?: string; +} + +/** Read-only dump of one session sidecar plus its metrics. Opens the file + * with `queryOnly`, so SQLite refuses writes, and closes it before returning. + * An older generation reports a named skip instead of migrating. */ +export function dumpSidecar(options: SidecarDumpOptions): SidecarDumpResult { + if (!existsSync(options.path)) { + return { + status: "skipped", + reason: "missing", + detail: `no sidecar at ${basename(options.path)}`, + }; + } + let store: LcmStore | undefined; + try { + store = new LcmStore(options.path, { queryOnly: true }); + } catch (error) { + if (error instanceof StaleStoreError) { + return { + status: "skipped", + reason: "older-generation", + detail: + `generation ${error.generation} is older than this build. ` + + `Run migrate first. No rows were changed.`, + }; + } + return { + status: "skipped", + reason: "unreadable", + detail: error instanceof Error ? error.message.slice(0, 200) : String(error), + }; + } + try { + return { status: "dumped", text: buildDump(store, options) }; + } catch (error) { + return { + status: "skipped", + reason: "unreadable", + detail: error instanceof Error ? error.message.slice(0, 200) : String(error), + }; + } finally { + try { + store.close(); + } catch {} + } +} + +function buildDump(store: LcmStore, options: SidecarDumpOptions): string { + const includeRemoved = options.includeRemoved === true; + const stats = store.stats(); + const removed = stats.removedMessages > 0 ? ` (${stats.removedMessages} removed, retained)` : ""; + const head = [ + `LCM sidecar dump: ${options.sessionId}`, + `store: ${basename(options.path)}, generation ${store.generation}, ` + + `${stats.messages} messages${removed}, ${stats.summaries} summaries, ` + + `${(stats.dbBytes / 1024).toFixed(0)} KB`, + `history:`, + historyManifest(store, DUMP_MAX_HISTORY), + ]; + const allSummaries = store.allSummaries(); + const summaries = allSummaries.slice(0, DUMP_MAX_SUMMARIES); + const totalSummaries = allSummaries.length; + const sections: string[] = [...head]; + if (summaries.length > 0) { + sections.push(`summaries: ${summaries.length} shown of ${totalSummaries}`); + for (const summary of summaries) { + const view = describeView(store, options.redact, summary.id); + sections.push(view.ok ? capSummaryText(view.text) : `Summary #${summary.id}: unreadable.`); + } + const first = summaries[0]!; + const coveredIds = store.coveredMessageIds(first.id); + const coveredMsgs = store.messagesByIds(coveredIds); + const coveredIdSet = new Set(coveredMsgs.map((m) => m.entryId)); + const removedIds = new Set( + coveredMsgs.filter((m) => m.removedAt !== undefined).map((m) => m.entryId), + ); + const visibleMsgs = includeRemoved + ? coveredMsgs + : coveredMsgs.filter((m) => m.removedAt === undefined); + const list = expandView(store, options.redact, { + mode: "list", + id: first.id, + offset: 0, + limit: DUMP_MAX_SAMPLE_MESSAGES, + maxChars: DUMP_SAMPLE_CHARS, + }); + sections.push( + `messages (summary #${first.id}):`, + list.ok + ? shapeSampleList( + list.text, + first.id, + coveredIdSet, + removedIds, + includeRemoved, + visibleMsgs.length, + ) + : "No messages.", + ); + for (const message of visibleMsgs.slice(0, DUMP_MAX_SAMPLE_MESSAGES)) { + const window = expandView(store, options.redact, { + mode: "message", + entryId: message.entryId, + charOffset: 0, + maxChars: DUMP_SAMPLE_CHARS, + }); + sections.push(window.ok ? window.text : `No stored message ${message.entryId}.`); + } + } else { + sections.push("summaries: none stored yet."); + } + if (options.sampleQuery !== undefined && options.sampleQuery.length > 0) { + const found = grepView(store, options.redact, { + query: options.sampleQuery, + limit: DUMP_MAX_SAMPLE_MESSAGES, + includeRemoved, + }); + sections.push( + `search ${JSON.stringify(options.sampleQuery)}${includeRemoved ? " (including removed)" : ""}:`, + found.text, + ); + } + const rows = (options.metricRows ?? []).filter((row) => row.session === options.sessionId); + const shown = rows.slice(Math.max(0, rows.length - DUMP_MAX_METRIC_ROWS)); + sections.push(`metrics: ${rows.length} row(s) for this session, ${shown.length} shown`); + for (const row of shown) sections.push(` ${JSON.stringify(sanitizeRow(row, options.redact))}`); + sections.push( + `omitted: summary text past ${DUMP_MAX_SUMMARY_BYTES} bytes each, message bodies past ${DUMP_SAMPLE_CHARS} chars per sample, ` + + `summaries past ${DUMP_MAX_SUMMARIES}, metric rows past ${DUMP_MAX_METRIC_ROWS}. ` + + `Removed rows carry [removed]. Read them with include_removed.`, + ); + const text = options.redact(sections.join("\n")); + if (Buffer.byteLength(text, "utf8") <= DUMP_MAX_BYTES) return text; + return clipDump(text); +} + +/** Reshape the sampled list for the dump: removed rows stay out by default + * and carry a [removed] label when --include-removed keeps them. Untouched + * when no covered row left the active path. */ +function shapeSampleList( + text: string, + summaryId: number, + coveredIds: ReadonlySet, + removedIds: ReadonlySet, + includeRemoved: boolean, + visibleTotal: number, +): string { + if (removedIds.size === 0) return text; + const groups: { id: string | undefined; chunks: string[] }[] = []; + for (const chunk of text.split("\n\n").slice(1)) { + const at = chunk.indexOf("\n"); + const first = at === -1 ? chunk : chunk.slice(0, at); + const id = /^\[([^\]]+)\] \((?:user|assistant|toolResult|custom)\)$/.exec(first)?.[1]; + if (id !== undefined && coveredIds.has(id)) groups.push({ id, chunks: [chunk] }); + else if (groups.length > 0) groups[groups.length - 1]!.chunks.push(chunk); + else groups.push({ id: undefined, chunks: [chunk] }); + } + const kept: string[] = []; + let shown = 0; + for (const group of groups) { + const removed = group.id !== undefined && removedIds.has(group.id); + if (removed && !includeRemoved) continue; + if (group.id !== undefined) shown++; + kept.push( + group.chunks + .map((chunk, index) => (index === 0 && removed ? labelRemoved(chunk) : chunk)) + .join("\n\n"), + ); + } + const head = + `Summary #${summaryId} → ${visibleTotal} original messages ` + + `(showing ${shown} from 0)${visibleTotal > shown ? ", use offset to page" : ""}`; + return [head, ...kept].join("\n\n"); +} + +function labelRemoved(chunk: string): string { + const at = chunk.indexOf("\n"); + return at === -1 ? `${chunk} [removed]` : `${chunk.slice(0, at)} [removed]${chunk.slice(at)}`; +} + +/** Keep one described summary bounded so large summaries cannot clip the tail. */ +function capSummaryText(text: string): string { + if (Buffer.byteLength(text, "utf8") <= DUMP_MAX_SUMMARY_BYTES) return text; + let used = 0; + let end = 0; + for (const chunk of text) { + const cost = Buffer.byteLength(chunk, "utf8"); + if (used + cost > DUMP_MAX_SUMMARY_BYTES) break; + used += cost; + end += chunk.length; + } + const more = Buffer.byteLength(text, "utf8") - used; + return `${text.slice(0, end)}\n[lcm:more ${more} bytes]`; +} + +/** Redact string seams in one metric row; keys and stub pointers stay intact. */ +function sanitizeRow( + row: Record, + redact: (text: string) => string, +): Record { + const out: Record = {}; + for (const [key, value] of Object.entries(row)) out[key] = sanitizeValue(value, redact); + return out; +} + +function sanitizeValue(value: unknown, redact: (text: string) => string): unknown { + if (typeof value === "string") return redact(value); + if (Array.isArray(value)) return value.map((item) => sanitizeValue(item, redact)); + if (typeof value === "object" && value !== null) { + const out: Record = {}; + for (const [key, item] of Object.entries(value as Record)) { + out[key] = sanitizeValue(item, redact); + } + return out; + } + return value; +} + +function clipDump(text: string): string { + const marker = `... truncated at the ${DUMP_MAX_BYTES} byte limit`; + const budget = DUMP_MAX_BYTES - Buffer.byteLength(marker, "utf8") - 1; + let kept = ""; + let used = 0; + for (const line of text.split("\n")) { + const cost = Buffer.byteLength(line, "utf8") + 1; + if (used + cost > budget) break; + kept += (kept.length === 0 ? "" : "\n") + line; + used += cost; + } + return `${kept}\n${marker}`; +} diff --git a/packages/deepseek/src/tools.ts b/packages/deepseek/src/tools.ts new file mode 100644 index 0000000..c340ead --- /dev/null +++ b/packages/deepseek/src/tools.ts @@ -0,0 +1,933 @@ +import type { Context } from "@deepseek-ai/cordis"; +import type { Agent } from "@deepseek-ai/dsh-agent"; +import { defineTool } from "@deepseek-ai/dsh-tools"; +import type { ToolRunContext } from "@deepseek-ai/dsh-tools"; +import { + DEFAULT_RETRIEVAL_TOKENS, + NO_STORE, + RECALL_SCOPES, + RETRIEVAL_STEPS, + billedCounters, + describeAnswer as coreDescribeAnswer, + expandAnswer as coreExpandAnswer, + grepView, + runRetrieval, + LcmStore, + type Reader, + type RecallToolDeps as CoreRecallDeps, + type RetrievalScope, + type SidecarRef, +} from "lossless-core"; + +import type { DshRoute } from "./model-host.ts"; +import { agentRoute, dshRef } from "./model-host.ts"; +import { dshReader } from "./reader.ts"; + +export interface PastSessions { + /** Sidecars past sessions are read from, newest first. The listing + * excludes the calling session, so no scope can reach the store a + * normal grep already searched. Backed by `SidecarStores.list`, which + * reads the directory rather than the open cache, so sessions from + * earlier processes are reachable. */ + readonly list: (excludeSessionId?: string) => readonly SidecarRef[]; +} + +/** Recall dependencies for one composition: store, redact, record, and past scope. */ +export interface RecallDeps { + /** Sidecar by calling-agent session. Undefined answers the no-history + * text, so a call outside any agent fails explained rather than empty. */ + readonly store: (sessionId: string | undefined) => LcmStore | undefined; + /** Redacts stored text before it reaches a model call. */ + readonly redact: (text: string) => string; + /** Diagnostic sink for retrieval rows. Optional, so a test can observe + * retrieval without a metrics file; production passes the per-composition + * JSONL writer. */ + readonly record?: (row: Record) => void; + /** The sidecars a cross-session scope scans. Absent answers explained. */ + readonly pastSessions?: PastSessions; +} + +/** Caps for one cross-session scan, matching the Pi precedent + * (`SESSION_SCAN_LIMIT`, `SESSION_HITS_LIMIT`, + * `SESSION_SEARCH_DEADLINE_MS`): an unbounded directory read inside a + * tool call is a latency bug, so whichever cap stops the scan is named in + * the answer. OpenCode carries the same numbers. */ +export const CROSS_SESSION_STORE_LIMIT = 20; +export const CROSS_SESSION_HITS_LIMIT = 40; +export const CROSS_SESSION_DEADLINE_MS = 1_500; +/** Hits one past session may contribute. A single chatty store must not + * fill the whole page while newer stores go unread. */ +export const CROSS_SESSION_PER_STORE_HITS = 10; + +/** Register the three recall tools on a composition with a tools service. + * Returns one disposer for all three. The descriptions match the loop tools + * core hands the reader, so both seams present the same surface. */ +export function registerRecallTools(ctx: Context, deps: RecallDeps): () => void { + const stop = [ + ctx.tools.register(lcmGrepTool(deps)), + ctx.tools.register(lcmDescribeTool(deps)), + ctx.tools.register(lcmExpandTool(deps)), + ]; + return () => { + for (const dispose of stop) dispose(); + }; +} + +/** Register the fourth recall tool, the answered reader over `runRetrieval`. + * Separate from the three direct tools because it needs the composition + * context to run the reader. Returns its disposer. */ +export function registerExpandQueryTool(ctx: Context, deps: RecallDeps): () => void { + const dispose = ctx.tools.register(lcmExpandQueryTool(ctx, deps)); + return () => dispose(); +} + +function toCoreDeps(deps: RecallDeps): CoreRecallDeps { + if (!deps.pastSessions) { + return { + storeFor: (sessionId) => deps.store(sessionId === "" ? undefined : sessionId), + redact: deps.redact, + ...(deps.record === undefined ? {} : { record: deps.record }), + }; + } + const past = deps.pastSessions; + const redact = deps.redact; + return { + storeFor: (sessionId) => deps.store(sessionId === "" ? undefined : sessionId), + redact, + ...(deps.record === undefined ? {} : { record: deps.record }), + searchPast: (session, query, scope, limit, filters) => { + const result = searchPastSessions({ past, redact }, session, query, { + ...(typeof limit === "number" ? { limit } : {}), + ...(filters?.role === undefined ? {} : { role: filters.role }), + ...(filters?.dateStart === undefined ? {} : { dateStart: filters.dateStart }), + ...(filters?.dateEnd === undefined ? {} : { dateEnd: filters.dateEnd }), + ...(filters?.toolName === undefined ? {} : { toolName: filters.toolName }), + }); + return { + text: result.text, + hits: result.hits, + scanned: result.scanned, + skipped: result.skipped, + }; + }, + openPast: (session, pastSession) => { + const opened = openPastSession(past, session, pastSession); + if (opened.kind !== "opened") return opened; + return { kind: "opened" as const, sessionId: opened.ref.id, store: opened.store }; + }, + }; +} + +/** Past-search outcome for the recorder: redacted text plus hit, scanned, + * and skipped session counts. */ +export interface PastSearchResult { + readonly text: string; + readonly hits: number; + readonly scanned: number; + readonly skipped: number; +} + +export type PastSearchStop = "stores" | "hits" | "deadline"; + +export interface PastSearchDeps { + readonly past: PastSessions; + readonly redact: (text: string) => string; + readonly now?: () => number; +} + +/** Search every past sidecar for `query` and format the labelled hits. A + * store that cannot be opened is counted and skipped: an older generation + * or a file in use is not an error the caller can act on mid-turn. The + * past store opens with `queryOnly` so a scan can never rewrite a session + * it is only reading. */ +export function searchPastSessions( + deps: PastSearchDeps, + callingSessionId: string | undefined, + query: string, + opts: { + limit?: number; + now?: () => number; + role?: "user" | "assistant" | "toolResult" | "custom"; + dateStart?: number; + dateEnd?: number; + toolName?: string; + } = {}, +): PastSearchResult & { stoppedBy?: PastSearchStop; elapsedMs: number } { + const now = opts.now ?? deps.now ?? Date.now; + const started = now(); + const hitLimit = clampLimit(opts.limit, CROSS_SESSION_HITS_LIMIT); + const deadline = started + CROSS_SESSION_DEADLINE_MS; + const all = deps.past.list(callingSessionId); + const refs = all.slice(0, CROSS_SESSION_STORE_LIMIT); + + const lines: string[] = []; + const sessions = new Set(); + let hits = 0; + let scanned = 0; + let skipped = 0; + let stoppedBy: PastSearchStop | undefined = all.length > refs.length ? "stores" : undefined; + + for (const ref of refs) { + if (hits >= hitLimit) { + stoppedBy = "hits"; + break; + } + if (now() > deadline) { + stoppedBy = "deadline"; + break; + } + let store: LcmStore | undefined; + try { + store = new LcmStore(ref.path, { queryOnly: true }); + } catch { + skipped++; + continue; + } + try { + const view = grepView(store, deps.redact, { + query, + limit: Math.min(CROSS_SESSION_PER_STORE_HITS, hitLimit - hits), + ...(opts.role === undefined ? {} : { role: opts.role }), + ...(opts.dateStart === undefined ? {} : { dateStart: opts.dateStart }), + ...(opts.dateEnd === undefined ? {} : { dateEnd: opts.dateEnd }), + ...(opts.toolName === undefined ? {} : { toolName: opts.toolName }), + expandTool: "lcm_expand", + }); + if (!view.ok) { + // A query this store rejects was rejected by every store; report the + // view's own message rather than a fake "no matches". + skipped++; + return finishPastSearch(view.text, { + hits: 0, + scanned, + skipped, + sessions: 0, + started, + now, + stoppedBy, + }); + } + scanned++; + for (const line of view.text.split("\n")) { + if (!line.startsWith("[")) continue; + if (hits >= hitLimit) break; + lines.push(`[lcm:session ${ref.id}] ${line}`); + hits++; + sessions.add(ref.id); + } + } catch { + skipped++; + } finally { + store.close(); + } + } + + if (hits >= hitLimit && stoppedBy === undefined) stoppedBy = "hits"; + const skippedNote = skipped === 0 ? "" : `, skipped ${skipped} unreadable`; + const body = + hits === 0 + ? `No matches in ${scanned} past session(s)${skippedNote}.` + : [ + `Found ${hits} match(es) across ${sessions.size} past session(s) in ${now() - started} ms (scanned ${scanned}${skippedNote}).`, + ...lines, + "Read a hit with lcm_expand({session, entry_id}); the text it returns is that past session's history, never this one's.", + ].join("\n"); + const stopped = stoppedBy === undefined ? undefined : stopNote(stoppedBy, hitLimit); + return finishPastSearch(stopped === undefined ? body : `${body}\n${stopped}`, { + hits, + scanned, + skipped, + sessions: sessions.size, + started, + now, + ...(stoppedBy === undefined ? {} : { stoppedBy }), + }); +} + +/** The cap that ended the scan, named so a short answer is not read as a + * complete one. */ +export function stopNote( + stoppedBy: PastSearchStop | undefined, + hitLimit: number = CROSS_SESSION_HITS_LIMIT, +): string | undefined { + switch (stoppedBy) { + case "stores": + return `Stopped at the ${CROSS_SESSION_STORE_LIMIT}-store cap; older sessions were not searched.`; + case "hits": + return `Stopped at the ${hitLimit}-hit cap; narrow the query or read one session.`; + case "deadline": + return `Stopped at the ${CROSS_SESSION_DEADLINE_MS} ms deadline; narrow the query.`; + default: + return undefined; + } +} + +/** The past sidecar a caller named, or why it cannot be read. The id is + * looked up in the listing rather than joined into a path, so a hostile id + * cannot escape the sidecar directory. */ +export function openPastSession( + past: PastSessions, + callingSessionId: string | undefined, + sessionId: string, +): { kind: "opened"; ref: SidecarRef; store: LcmStore } | { kind: "missing" | "unreadable" } { + const ref = past.list(callingSessionId).find((candidate) => candidate.id === sessionId); + if (ref === undefined) return { kind: "missing" }; + try { + return { kind: "opened", ref, store: new LcmStore(ref.path, { queryOnly: true }) }; + } catch { + return { kind: "unreadable" }; + } +} + +function finishPastSearch( + text: string, + state: { + hits: number; + scanned: number; + skipped: number; + sessions: number; + started: number; + now: () => number; + stoppedBy?: PastSearchStop; + }, +): PastSearchResult & { stoppedBy?: PastSearchStop; elapsedMs: number } { + return { + text, + hits: state.hits, + scanned: state.scanned, + skipped: state.skipped, + elapsedMs: state.now() - state.started, + ...(state.stoppedBy === undefined ? {} : { stoppedBy: state.stoppedBy }), + }; +} + +function clampLimit(raw: number | undefined, max: number): number { + const value = raw !== undefined && Number.isFinite(raw) && raw > 0 ? Math.trunc(raw) : max; + return Math.min(max, Math.max(1, value)); +} + +// SQLite returns no rows for LIMIT 0 and every row for a negative LIMIT. +function clampSessionLimit(raw: number): number | undefined { + return Number.isFinite(raw) ? Math.max(1, Math.trunc(raw)) : undefined; +} + +export interface GrepArgs { + readonly query?: string; + readonly pattern?: string; + readonly case_sensitive?: boolean; + readonly offset?: number; + readonly after?: string; + readonly limit?: number; + readonly summary_id?: number; + readonly include_removed?: boolean; + readonly role?: "user" | "assistant" | "toolResult" | "custom"; + readonly date_start?: number; + readonly date_end?: number; + readonly tool_name?: string; + readonly scope?: string; +} + +export interface GrepTestOptions { + readonly pastSessions?: PastSessions; + readonly sessionId?: string; + readonly record?: (row: Record) => void; +} + +/** Session-less answer for tests: binds one store through the single + * converter above with the empty session id. Query plus limit behave + * exactly as before; the extra params map one to one onto `grepView`, + * which already honors every one of them (verified against + * `packages/core/src/recall-view.ts`: query, pattern, caseSensitive, + * offset, after, limit, summaryId, includeRemoved). `scope` defaults to + * the calling session, so existing callers see no behavior change. */ +export function grepAnswer( + store: LcmStore | undefined, + redact: (text: string) => string, + args: GrepArgs, + opts: GrepTestOptions = {}, +): string { + const deps: RecallDeps = { + store: () => store, + redact, + ...(opts.record === undefined ? {} : { record: opts.record }), + ...(opts.pastSessions === undefined ? {} : { pastSessions: opts.pastSessions }), + }; + return answerGrep(deps, opts.sessionId ?? "", args); +} + +/** Session grep plus past-session search behind `lcm_grep`, mirroring Pi + * params and caps: this session reads `grepView` directly with every + * named param, while `sessions` and `all_sessions` scan the sidecar + * directory (equivalent here because it is global, as in OpenCode) with a + * 20-store, 40-hit, 1.5 s cap and record sessionsScanned/Skipped. A + * pattern scan stays session-scoped, as in Pi. */ +export function answerGrep(deps: RecallDeps, callerSessionId: string, args: GrepArgs): string { + const started = Date.now(); + const session = callerSessionId; + const rawScope = args.scope; + const scope = rawScope === undefined ? "session" : asRecallScope(rawScope); + const query = args.query; + const queryChars = query?.length ?? 0; + if (scope === undefined) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "refused", + queryChars, + durationMs: Date.now() - started, + ...(typeof rawScope === "string" ? { scope: rawScope } : {}), + }); + return "lcm_grep scope must be session, sessions, or all_sessions."; + } + if (scope !== "session") { + if (args.pattern !== undefined) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "refused", + queryChars, + scope, + durationMs: Date.now() - started, + }); + return "LCM: a pattern scan searches this session. Pass query for past sessions."; + } + if (query === undefined) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "error", + queryChars, + scope, + durationMs: Date.now() - started, + }); + return "LCM: pass a query to search past sessions."; + } + if (!deps.pastSessions) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "refused", + queryChars, + scope, + durationMs: Date.now() - started, + }); + return "LCM: past sessions are not reachable in this process (no sidecar directory is wired)."; + } + const result = searchPastSessions( + { past: deps.pastSessions, redact: deps.redact }, + session, + query, + { + ...(typeof args.limit === "number" ? { limit: args.limit } : {}), + ...(args.role === undefined ? {} : { role: args.role }), + ...(args.date_start === undefined ? {} : { dateStart: args.date_start }), + ...(args.date_end === undefined ? {} : { dateEnd: args.date_end }), + ...(args.tool_name === undefined ? {} : { toolName: args.tool_name }), + }, + ); + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: result.hits === 0 ? "miss" : "hit", + hits: result.hits, + queryChars, + scope, + sessionsScanned: result.scanned, + sessionsSkipped: result.skipped, + durationMs: Date.now() - started, + }); + return result.text; + } + if (query === undefined && args.pattern === undefined) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "error", + queryChars, + durationMs: Date.now() - started, + }); + return "lcm_grep needs a query string."; + } + const store = deps.store(callerSessionId === "" ? undefined : callerSessionId); + if (!store) { + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: "miss", + hits: 0, + queryChars, + durationMs: Date.now() - started, + }); + return NO_STORE; + } + const sessionLimit = typeof args.limit === "number" ? clampSessionLimit(args.limit) : undefined; + const view = grepView(store, deps.redact, { + ...(query === undefined ? {} : { query }), + ...(args.pattern === undefined ? {} : { pattern: args.pattern }), + ...(args.case_sensitive === undefined ? {} : { caseSensitive: args.case_sensitive }), + ...(typeof args.offset === "number" ? { offset: args.offset } : {}), + ...(args.after === undefined ? {} : { after: args.after }), + ...(sessionLimit === undefined ? {} : { limit: sessionLimit }), + ...(typeof args.summary_id === "number" ? { summaryId: args.summary_id } : {}), + ...(args.include_removed === undefined ? {} : { includeRemoved: args.include_removed }), + ...(args.role === undefined ? {} : { role: args.role }), + ...(args.date_start === undefined ? {} : { dateStart: args.date_start }), + ...(args.date_end === undefined ? {} : { dateEnd: args.date_end }), + ...(args.tool_name === undefined ? {} : { toolName: args.tool_name }), + expandTool: "lcm_expand", + }); + const hits = view.ok && typeof view.details.hits === "number" ? view.details.hits : 0; + recordGrep(deps, session, { + tool: "lcm_grep", + outcome: !view.ok ? "error" : hits === 0 ? "miss" : "hit", + hits, + queryChars, + durationMs: Date.now() - started, + }); + return view.text; +} + +function asRecallScope(raw: unknown): (typeof RECALL_SCOPES)[number] | undefined { + return typeof raw === "string" && (RECALL_SCOPES as readonly string[]).includes(raw) + ? (raw as (typeof RECALL_SCOPES)[number]) + : undefined; +} + +function recordGrep( + deps: Pick, + session: string, + row: Record, +): void { + deps.record?.({ ts: Date.now(), session, kind: "recall", ...row }); +} + +/** Session-less answer for tests: see grepAnswer. */ +export function describeAnswer( + store: LcmStore | undefined, + redact: (text: string) => string, + args: { id: number }, +): string { + return coreDescribeAnswer(toCoreDeps({ store: () => store, redact }), "", args); +} + +export interface ExpandArgs { + readonly id?: number; + readonly entry_id?: string; + readonly anchor?: string; + readonly session?: string; + readonly offset?: number; + readonly limit?: number; + readonly char_offset?: number; + readonly max_chars?: number; +} + +/** Session-less answer for tests: see grepAnswer. The `session` param + * resolves through the same sidecar lookup the tool uses; without wired + * past sessions a foreign session falls back to the bound store so + * single-store tests keep working. */ +export function expandAnswer( + store: LcmStore | undefined, + redact: (text: string) => string, + args: ExpandArgs, + opts: GrepTestOptions = {}, +): string { + const deps: RecallDeps = { + store: () => store, + redact, + ...(opts.record === undefined ? {} : { record: opts.record }), + ...(opts.pastSessions === undefined ? {} : { pastSessions: opts.pastSessions }), + }; + return coreExpandAnswer(toCoreDeps(deps), opts.sessionId ?? "", args); +} + +export interface ExpandQueryArgs { + readonly query: string; + readonly prompt?: string; + readonly summary_id?: number; + readonly entry_id?: string; + readonly session?: string; + readonly max_tokens?: number; +} + +/** Session-less answered retrieval for tests: binds one store and one reader + * through the same converter the tool uses, with the same record seam. */ +export function expandQueryAnswer( + store: LcmStore | undefined, + redact: (text: string) => string, + reader: Reader, + sessionId: string | undefined, + args: ExpandQueryArgs, + record?: (row: Record) => void, + pastSessions?: PastSessions, +): Promise { + const deps: RecallDeps = { + store: () => store, + redact, + ...(record === undefined ? {} : { record }), + ...(pastSessions === undefined ? {} : { pastSessions }), + }; + return answerExpandQuery(toCoreDeps(deps), reader, sessionId, args); +} + +const DEFAULT_EXPAND_QUERY_PROMPT = + "Answer the question from the stored history, citing the entry or summary ids relied on."; + +/** Answered retrieval behind `lcm_expand_query`: the `session` param names + * another session kept in this process, resolved read-only through the + * sidecar listing when it names a past session (closed after the run, so + * a ghost id creates no file), and every run that reaches past validation + * emits one retrieval-kind row when a recorder is present. A past + * session's answer is labelled as such, mirroring Pi, and findings stay + * wrapped so the raw transcript never enters the caller's context. */ +async function answerExpandQuery( + deps: CoreRecallDeps, + reader: Reader, + sessionId: string | undefined, + args: ExpandQueryArgs, +): Promise { + const budgetTokens = args.max_tokens ?? DEFAULT_RETRIEVAL_TOKENS; + const scope = expandScope(args); + if (typeof scope === "string") { + recordRetrieval(deps, sessionId, { + queryChars: args.query.length, + budgetTokens, + steps: 0, + retrievedTokens: 0, + stop: "refused", + reason: scope, + }); + return `Nothing was read: ${scope}.`; + } + if (args.session !== undefined && args.session.trim() === "") { + recordRetrieval(deps, sessionId, { + queryChars: args.query.length, + budgetTokens, + steps: 0, + retrievedTokens: 0, + stop: "refused", + reason: "session must be a non-empty session id", + }); + return "Nothing was read: session must be a non-empty session id."; + } + const caller = sessionId ?? ""; + const pastId = args.session === undefined || args.session === caller ? undefined : args.session; + let target: LcmStore | undefined; + let pastSession: string | undefined; + let closePast = false; + if (pastId === undefined) { + target = deps.storeFor(sessionId ?? ""); + } else if (deps.openPast !== undefined && sessionId !== undefined) { + const opened = deps.openPast(sessionId, pastId); + if (opened.kind !== "opened") { + recordRetrieval(deps, sessionId, { + queryChars: args.query.length, + budgetTokens, + steps: 0, + retrievedTokens: 0, + stop: "refused", + reason: + opened.kind === "missing" + ? `no stored session with id ${pastId}` + : `session ${pastId} could not be opened (an older generation, or in use)`, + }); + return opened.kind === "missing" + ? `Nothing was read: no stored session with id ${pastId}.` + : `Nothing was read: session ${pastId} could not be opened (an older generation, or in use).`; + } + target = opened.store; + pastSession = opened.sessionId; + closePast = true; + } else { + target = deps.storeFor(pastId); + } + if (!target) { + recordRetrieval(deps, sessionId, { + queryChars: args.query.length, + budgetTokens, + steps: 0, + retrievedTokens: 0, + stop: "refused", + reason: "no stored history for this session yet", + }); + return NO_STORE; + } + let run: Awaited>; + try { + run = await runRetrieval( + target, + { + query: args.query, + prompt: + args.prompt !== undefined && args.prompt.length > 0 + ? args.prompt + : DEFAULT_EXPAND_QUERY_PROMPT, + budgetTokens, + maxSteps: RETRIEVAL_STEPS, + redact: deps.redact, + ...(scope === undefined ? {} : { scope }), + }, + reader, + ); + } finally { + if (closePast) target.close(); + } + recordRetrieval(deps, sessionId, { + queryChars: args.query.length, + budgetTokens, + steps: run.steps, + retrievedTokens: run.retrievedTokens, + stop: run.kind, + ...(run.kind === "answered" ? {} : { reason: run.reason }), + ...(scope === undefined ? {} : { scope: scope.kind }), + ...(scope?.kind === "summary" ? { summaryId: scope.summaryId } : {}), + ...(scope?.kind === "entry" ? { entryId: scope.entryId } : {}), + ...billedCounters(run.usage), + }); + if (run.kind === "refused") { + return `Nothing was read: ${run.reason}.`; + } + if (run.kind === "failed") { + return `Retrieval failed: ${run.reason}`; + } + if (run.kind === "stopped" && run.text.length === 0) { + return ( + `Retrieval stopped (${run.reason}) after ${run.steps} step(s) without an answer; ` + + `narrow the query or raise max_tokens.` + ); + } + const stopped = run.kind === "stopped" ? `, stopped: ${run.reason}` : ""; + const where = + pastSession === undefined + ? "stored history" + : `a past session's history (session ${pastSession})`; + const header = + `Read ${where} in ${run.steps} step(s) ` + + `(${run.retrievedTokens} of ${budgetTokens} tokens)${stopped}.`; + const tag = + pastSession === undefined + ? "recovered_findings" + : `recovered_findings session="${pastSession}"`; + return `${header}\n<${tag}>\n${run.text}\n`; +} + +function expandScope(args: { + summary_id?: number; + entry_id?: string; +}): RetrievalScope | string | undefined { + if (args.summary_id !== undefined && args.entry_id !== undefined) { + return "pass summary_id or entry_id, not both"; + } + if (args.summary_id !== undefined) return { kind: "summary", summaryId: args.summary_id }; + if (args.entry_id === undefined) return undefined; + if (args.entry_id.length === 0) return "entry_id is empty"; + return { kind: "entry", entryId: args.entry_id }; +} + +function recordRetrieval( + deps: CoreRecallDeps, + sessionId: string | undefined, + row: Record, +): void { + deps.record?.({ + event: "lcm", + kind: "retrieval", + ...(sessionId === undefined ? {} : { session: sessionId }), + ...row, + }); +} + +/** The reader calls the calling agent's model: the latest routed request + * first, agent options second, mirroring the ladder's fallback order. */ +function readerRoute(agent: Agent | undefined): DshRoute | undefined { + // One route contract; the guards live in model-host. + return agentRoute(agent); +} + +function sessionOf(exec: ToolRunContext): string | undefined { + const id = exec.agent?.session.id; + return id === undefined ? undefined : String(id); +} + +function lcmGrepTool(deps: RecallDeps) { + return defineTool({ + name: "lcm_grep", + description: + "Full-text search over the stored session history. Each hit names the summary (#N) that covers it, which lcm_expand reads. Pass `query` for a full-text search, or `pattern` for a JavaScript regular expression over the full text of every message. `scope` widens the search to past sessions in this sidecar directory; sessions and all_sessions are equivalent here because the directory is global.", + parameters: { + query: { + type: "string", + description: "FTS5 query: quoted phrases, AND/OR, prefix*", + }, + pattern: { + type: "string", + description: + "JavaScript regular expression over the full text of every stored message. Case-insensitive unless case_sensitive is set. Give one of query or pattern, not both.", + }, + case_sensitive: { + type: "boolean", + description: "Match a pattern with its own case (default false)", + }, + offset: { type: "integer", description: "Hits to skip, for the next page" }, + after: { + type: "string", + description: + "Continue a pattern scan from this entry id, which is the `after` value a deadline stop names", + }, + limit: { type: "integer", description: "Hits to return" }, + summary_id: { + type: "integer", + description: "Restrict hits to the messages this summary covers", + }, + include_removed: { + type: "boolean", + description: + "Include matches from a branch that left the active path (excluded by default)", + }, + role: { + type: "string", + enum: ["user", "assistant", "toolResult", "custom"], + description: "Keep only rows stored with this role", + }, + date_start: { + type: "number", + description: "Keep only rows at or after this timestamp", + }, + date_end: { + type: "number", + description: "Keep only rows at or before this timestamp", + }, + tool_name: { + type: "string", + description: + "Keep only tool-result rows for this tool. It matches [tool:Name] markers. It applies to query searches only.", + }, + scope: { + type: "string", + enum: [...RECALL_SCOPES], + description: + "session (default) searches this session. sessions and all_sessions search past sessions in this sidecar directory and are equivalent here.", + }, + }, + output: { + schema: { type: "string" }, + render: (_args, value) => [{ type: "text", text: value }], + }, + async execute(args, exec) { + return answerGrep(deps, sessionOf(exec) ?? "", args); + }, + }); +} + +function lcmDescribeTool(deps: RecallDeps) { + return defineTool({ + name: "lcm_describe", + description: + "Inspect a summary without expanding it: kind, depth, own and source token costs, children with their costs, parents, and full text.", + parameters: { + id: { + type: "integer", + required: true, + description: "Summary id, the N in [lcm:summary #N ...]", + }, + }, + output: { + schema: { type: "string" }, + render: (_args, value) => [{ type: "text", text: value }], + }, + async execute(args, exec) { + return coreDescribeAnswer(toCoreDeps(deps), sessionOf(exec) ?? "", args); + }, + }); +} + +function lcmExpandTool(deps: RecallDeps) { + return defineTool({ + name: "lcm_expand", + description: + "Read the original messages a summary covers, or one named entry. Without entry_id, list `limit` messages from message index `offset`, or centered on `anchor` when it names an entry id. With entry_id, read that one message from `char_offset` for up to `max_chars` characters; continue from the char_offset the result reports. `id` is the summary to list, or to check that the entry is covered; omit it to read an entry that no summary covers yet. Pass `session` with the id from an [lcm:session ] hit to read that past session's sidecar; the text is labelled as a past session's history and the read is read-only.", + parameters: { + id: { type: "integer", description: "Summary id, the N in [lcm:summary #N ...]" }, + entry_id: { type: "string", description: "Read this one message by entry id" }, + anchor: { + type: "string", + description: "Center the listing window on this entry id (with id, without entry_id)", + }, + session: { + type: "string", + description: + "The past session id from an [lcm:session ] hit. Its sidecar is read read-only and the text is labelled as a past session's history.", + }, + offset: { type: "integer", description: "Message index to start listing from" }, + limit: { type: "integer", description: "Messages to list" }, + char_offset: { type: "integer", description: "Character offset inside entry_id" }, + max_chars: { type: "integer", description: "Characters to read from entry_id" }, + }, + output: { + schema: { type: "string" }, + render: (_args, value) => [{ type: "text", text: value }], + }, + async execute(args, exec) { + return coreExpandAnswer(toCoreDeps(deps), sessionOf(exec) ?? "", args); + }, + }); +} + +function lcmExpandQueryTool(ctx: Context, deps: RecallDeps) { + return defineTool({ + name: "lcm_expand_query", + description: + "Recover content from compacted history and answer a question about it. Use it whenever lcm_grep or a summary in context points at what you need but you do not have its text. Pass summary_id to read one summary's originals, or entry_id to read one stored message; without either it searches. A bounded reader expands the stored originals and returns findings only, so the raw transcript never enters this context. Ask for verbatim quotes in `prompt` when the exact words matter.", + parameters: { + query: { + type: "string", + required: true, + description: "What to search for in the stored history", + }, + prompt: { + type: "string", + description: + "What to find out, and the form the answer should take (ask for a verbatim quote when the exact words matter)", + }, + summary_id: { + type: "integer", + description: "Read this summary's originals: the N from an [lcm:summary #N ...] header", + }, + entry_id: { + type: "string", + description: "Read this one stored message: the entry id a grep hit names", + }, + session: { + type: "string", + description: + "Read another session kept in this process instead of the calling one; names a session id, never a file.", + }, + max_tokens: { + type: "integer", + description: `Retrieved content budget in estimated tokens (default ${DEFAULT_RETRIEVAL_TOKENS})`, + }, + }, + output: { + schema: { type: "string" }, + render: (_args, value) => [{ type: "text", text: value }], + }, + async execute(args, exec) { + const sessionId = sessionOf(exec); + const coreDeps = toCoreDeps(deps); + const route = readerRoute(exec.agent); + if (route === undefined) { + recordRetrieval(coreDeps, sessionId, { + queryChars: args.query.length, + budgetTokens: args.max_tokens ?? DEFAULT_RETRIEVAL_TOKENS, + steps: 0, + retrievedTokens: 0, + stop: "failed", + reason: "no provider/model available for the reader", + }); + return "lossless retrieval: no provider/model available for the reader: route one request or set both agent options fields."; + } + return answerExpandQuery( + coreDeps, + dshReader(ctx, { + ref: dshRef(route), + signal: exec.signal, + ...(sessionId === undefined ? {} : { sessionId }), + }), + sessionId, + args, + ); + }, + }); +} diff --git a/packages/deepseek/test/backfill.test.ts b/packages/deepseek/test/backfill.test.ts new file mode 100644 index 0000000..70881c2 --- /dev/null +++ b/packages/deepseek/test/backfill.test.ts @@ -0,0 +1,455 @@ +import { existsSync, mkdtempSync, readdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import { + ToolCallId, + createMessage, + createSystemMessage, + createToolResultMessage, + createUserMessage, +} from "@deepseek-ai/dsh-llm"; +import SessionStore, { Session, SessionId, type SessionEvent } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { LcmStore, readMetricRows } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { + BACKFILL_MAX_EVENTS, + BACKFILL_MAX_VISITED, + backfillSession, + historySourceFrom, + mirrorEvent, + type SessionHistorySource, +} from "../src/ingest.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { applyLossless } from "../src/plugin.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function buildSession() { + const session = Session.create(SessionId("backfill-1")); + session.append("turn/start", { turn: 1 }); + session.append( + "system/message", + { turn: 1, step: 1, message: createSystemMessage("You are a test.") }, + { surfaceOp: "append" }, + ); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "start the backfill work" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 1, step: 1 }); + const callId = ToolCallId("call-1"); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [ + { type: "text", text: "reading now" }, + { type: "tool-call", id: callId, name: "read", arguments: '{"path":"backfill.conf"}' }, + ], + source: { kind: "model", provider: ROUTE.provider, model: ROUTE.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("tool/call", { + turn: 1, + step: 1, + callId, + name: "read", + arguments: '{"path":"backfill.conf"}', + }); + session.append( + "tool/result", + { + turn: 1, + step: 1, + message: createToolResultMessage({ + callId, + content: [{ type: "text", text: "config contents here" }], + isError: false, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 1, step: 1 }); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + return session; +} + +function buildLongSession(turns: number) { + const session = Session.create(SessionId("backfill-long")); + for (let turn = 1; turn <= turns; turn++) { + session.append("turn/start", { turn }); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `request ${turn}` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append( + "assistant/message", + { + stream: [], + turn, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: `response ${turn}` }], + source: { kind: "model", provider: ROUTE.provider, model: ROUTE.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn, reason: { kind: "completed" } }); + } + return session; +} + +function fakeSource( + events: readonly SessionEvent[], + seen?: { calls: number }, +): SessionHistorySource { + return { + observe: async () => { + if (seen) seen.calls += 1; + return { events }; + }, + }; +} + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-backfill-")); +} + +async function mountBase(ctx: Context): Promise { + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +describe("pre-mount backfill", () => { + it("ingests pre-mount messages including tool calls through one async read", async () => { + const session = buildSession(); + const store = new LcmStore(":memory:"); + const rows: Record[] = []; + const seen = { calls: 0 }; + // The snapshot walk is test-only: production reads the frozen cut through + // the async observation seam the fake stands in for. + const result = await backfillSession( + session, + store, + fakeSource(session.snapshotEvents(), seen), + undefined, + { + record: (row) => rows.push(row), + }, + ); + const derived = session.deriveMessages(); + expect(seen.calls).toBe(1); + expect(result.ingested).toBe(derived.length); + expect(result.capped).toBe(false); + expect(store.allMessages()).toHaveLength(derived.length); + expect(store.allMessages().some((row) => row.text.includes("tool-call: read"))).toBe(true); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + event: "lcm", + kind: "backfill", + session: "backfill-1", + ingested: derived.length, + capped: false, + }); + }); + + it("re-mirroring the same span inserts nothing", async () => { + const session = buildSession(); + const store = new LcmStore(":memory:"); + const events = session.snapshotEvents(); + const first = await backfillSession(session, store, fakeSource(events)); + expect(first.ingested).toBe(session.deriveMessages().length); + const second = await backfillSession(session, store, fakeSource(events)); + expect(second.ingested).toBe(0); + expect(second.skipped).toBe(events.length); + let liveInserted = 0; + for (const event of events) liveInserted += mirrorEvent(session, event, store).inserted; + expect(liveInserted).toBe(0); + expect(store.allMessages()).toHaveLength(session.deriveMessages().length); + }); + + it("live events after mount append normally", async () => { + const session = buildSession(); + const store = new LcmStore(":memory:"); + await backfillSession(session, store, fakeSource(session.snapshotEvents())); + const before = store.allMessages().length; + const followup = session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "live followup after backfill" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + const stats = mirrorEvent(session, followup, store); + expect(stats.inserted).toBe(1); + expect(store.allMessages()).toHaveLength(before + 1); + expect( + store.allMessages().some((row) => row.text.includes("live followup after backfill")), + ).toBe(true); + }); + + it("respects the cap on an oversized history", async () => { + const session = buildLongSession(30); + const events = session.snapshotEvents(); + expect(events.length).toBeGreaterThan(25); + const store = new LcmStore(":memory:"); + const rows: Record[] = []; + const seen = { calls: 0 }; + const result = await backfillSession(session, store, fakeSource(events, seen), undefined, { + pageSize: 10, + maxEvents: 25, + record: (row) => rows.push(row), + }); + expect(seen.calls).toBe(1); + expect(result.capped).toBe(true); + expect(result.ingested).toBe(25); + expect(result.skipped).toBe(25); + expect(result.ingested + result.skipped).toBe(50); + expect(result.fromSeq).toBe(Number(events[0]?.seq)); + expect(result.throughSeq).toBe(Number(events[49]?.seq)); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + kind: "backfill", + capped: true, + throughSeq: result.throughSeq, + }); + }); + + it("dedup skips do not burn the inserted budget", async () => { + const session = buildLongSession(30); + const events = session.snapshotEvents(); + const store = new LcmStore(":memory:"); + // Seed the first five turns so their replays dedup. + for (const event of events.slice(0, 20)) mirrorEvent(session, event, store); + expect(store.allMessages()).toHaveLength(10); + const result = await backfillSession(session, store, fakeSource(events), undefined, { + pageSize: 10, + maxEvents: 10, + }); + expect(result.capped).toBe(true); + expect(result.ingested).toBe(10); + expect(result.ingested + result.skipped).toBe(39); + expect(result.throughSeq).toBe(Number(events[38]?.seq)); + expect(store.allMessages()).toHaveLength(20); + }); + + it("ceiling counts unmirrored messages, not markers or stored rows", async () => { + const session = buildLongSession(30); + const events = session.snapshotEvents(); + const store = new LcmStore(":memory:"); + const result = await backfillSession(session, store, fakeSource(events), undefined, { + pageSize: 10, + maxEvents: 1000, + maxVisited: 20, + }); + expect(result.capped).toBe(true); + expect(result.ingested).toBe(20); + expect(result.skipped).toBe(19); + expect(result.ingested + result.skipped).toBe(39); + expect(result.throughSeq).toBe(Number(events[38]?.seq)); + }); + + it("honors the default cap on a log larger than BACKFILL_MAX_EVENTS", async () => { + const session = buildLongSession(350); + const events = session.snapshotEvents(); + expect(events.length).toBeGreaterThan(BACKFILL_MAX_EVENTS); + expect(session.deriveMessages().length).toBeGreaterThan(BACKFILL_MAX_EVENTS); + const store = new LcmStore(":memory:"); + const result = await backfillSession(session, store, fakeSource(events)); + expect(result.capped).toBe(true); + expect(result.ingested).toBe(BACKFILL_MAX_EVENTS); + expect(result.ingested + result.skipped).toBe(999); + expect(result.ingested + result.skipped).toBeLessThan(BACKFILL_MAX_VISITED); + expect(store.allMessages()).toHaveLength(BACKFILL_MAX_EVENTS); + }); + + it("mirrors a 600-event session across one call and inserts nothing on repeat", async () => { + const session = buildLongSession(150); + const events = session.snapshotEvents(); + expect(events).toHaveLength(600); + const store = new LcmStore(":memory:"); + const first = await backfillSession(session, store, fakeSource(events)); + expect(first.capped).toBe(false); + expect(first.ingested).toBe(session.deriveMessages().length); + expect(first.ingested + first.skipped).toBe(600); + expect(store.allMessages()).toHaveLength(session.deriveMessages().length); + const second = await backfillSession(session, store, fakeSource(events)); + expect(second.ingested).toBe(0); + expect(second.skipped).toBe(600); + expect(store.allMessages()).toHaveLength(session.deriveMessages().length); + }); + + it("reaches newer events past a stored prefix larger than the visited ceiling", async () => { + const session = buildLongSession(600); + const events = session.snapshotEvents(); + expect(events.length).toBeGreaterThan(BACKFILL_MAX_VISITED); + const store = new LcmStore(":memory:"); + for (const event of events.slice(0, 2100)) mirrorEvent(session, event, store); + const before = store.allMessages().length; + expect(before).toBeGreaterThan(0); + const result = await backfillSession(session, store, fakeSource(events)); + expect(store.allMessages()).toHaveLength(session.deriveMessages().length); + expect(result.capped).toBe(false); + }); + + it("reports uncapped when inserts hit the cap exactly on the final event", async () => { + const session = buildLongSession(600); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "trailing request" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + const events = session.snapshotEvents(); + const store = new LcmStore(":memory:"); + for (const event of events.slice(0, 1901)) mirrorEvent(session, event, store); + const stored = store.allMessages().length; + const remaining = session.deriveMessages().length - stored; + expect(remaining).toBeGreaterThan(0); + const result = await backfillSession(session, store, fakeSource(events), undefined, { + maxEvents: remaining, + }); + expect(store.allMessages()).toHaveLength(session.deriveMessages().length); + expect(result.capped).toBe(false); + }); + + it("still trips the visited cap on new rows past a warm prefix", async () => { + const session = buildLongSession(60); + const events = session.snapshotEvents(); + const store = new LcmStore(":memory:"); + for (const event of events.slice(0, 40)) mirrorEvent(session, event, store); + const result = await backfillSession(session, store, fakeSource(events), undefined, { + maxEvents: 1000, + maxVisited: 10, + }); + expect(result.capped).toBe(true); + expect(result.ingested).toBe(10); + expect(store.allMessages()).toHaveLength(30); + }); + + it("mirrors an attachment-less image block as a marker row", async () => { + const session = Session.create(SessionId("backfill-missing-attachment")); + session.append("turn/start", { turn: 1 }); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "image" } as unknown as { type: "text"; text: string }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + const result = await backfillSession(session, store, fakeSource(session.snapshotEvents())); + expect(result.capped).toBe(false); + expect(result.ingested).toBe(1); + const rows = store.allMessages(); + expect(rows).toHaveLength(1); + expect(rows[0]?.text).toContain("image (attachment missing)"); + }); +}); + +describe("history source seam", () => { + it("treats an uninjectable query backend as absent instead of throwing", () => { + // Live cordis scopes reject undeclared service reads with + // `cannot get property "sessionQuery" without inject`; the mount must + // degrade to live-mirror-only, exactly like a missing backend. + const throwing = { + get sessionQuery(): never { + throw new Error('cannot get property "sessionQuery" without inject'); + }, + }; + expect(historySourceFrom(throwing as unknown as Context)).toBeUndefined(); + }); + + it("is absent without the query backend and reads through it when mounted", async () => { + const ctx = new Context(); + expect(historySourceFrom(ctx)).toBeUndefined(); + const session = buildSession(); + const events = session.snapshotEvents(); + let disposed = false; + const disposeKey = (Symbol as unknown as { dispose: symbol }).dispose; + (ctx as unknown as Record).sessionQuery = { + observeSession: async () => ({ + events, + cursor: events.at(-1)?.seq ?? -1, + [disposeKey]: () => { + disposed = true; + }, + }), + }; + const source = historySourceFrom(ctx); + expect(source).toBeDefined(); + const observation = await source?.observe(SessionId("backfill-1")); + expect(observation?.events).toHaveLength(events.length); + expect(disposed).toBe(true); + }); + + it("backfills pre-mount sessions at plugin mount and records the row", async () => { + const { ctx } = setupContext(ROUTE, [textChunks("summary")], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const session = ctx.sessions.create(SessionId("premout-1"), { meta: { cwd: tmpdir() } }); + appendConversation(session, ROUTE, "premount notes"); + const derived = session.deriveMessages().length; + const events = session.snapshotEvents(); + (ctx as unknown as Record).sessionQuery = { + observeSession: async () => ({ events }), + }; + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const path = join(dir, "premout-1.sqlite"); + let stored = 0; + for (let i = 0; i < 100 && stored !== derived; i++) { + await new Promise((resolve) => setImmediate(resolve)); + if (existsSync(path)) stored = new LcmStore(path).allMessages().length; + } + expect(stored).toBe(derived); + const backfills = readMetricRows(dir) + .map((row) => row as Record) + .filter((row) => row.kind === "backfill"); + expect(backfills).toHaveLength(1); + expect(backfills[0]).toMatchObject({ + session: "premout-1", + ingested: derived, + capped: false, + }); + void readdirSync(dir); + } finally { + dispose(); + } + }); +}); diff --git a/packages/deepseek/test/calibration.test.ts b/packages/deepseek/test/calibration.test.ts new file mode 100644 index 0000000..347ad49 --- /dev/null +++ b/packages/deepseek/test/calibration.test.ts @@ -0,0 +1,505 @@ +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import type { BasicCompactionConfig } from "@deepseek-ai/dsh-compaction-basic"; +import { ToolCallId, createMessage, createToolResultMessage } from "@deepseek-ai/dsh-llm"; +import SessionStore, { Session, SessionId } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { LcmStore, TRUNCATED_SUMMARY_PREFIX, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { lcmCommandDefinition, type LcmCommandDeps } from "../src/command.ts"; +import { LosslessSummarizerEngine } from "../src/engine.ts"; +import type { LosslessEngineDeps } from "../src/engine.ts"; +import { MessageIndex, mirrorEvent } from "../src/ingest.ts"; +import { healLlmConfig, healRunBudgets, passLlmConfig, runLadder } from "../src/ladder.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { applyLossless } from "../src/plugin.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-lcm-cal-")); +} + +async function mountBase(ctx: Context): Promise { + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function mountMeter(ctx: Context): void { + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function engineOf(ctx: Context): LosslessSummarizerEngine { + const engine = ctx.compaction as unknown as LosslessSummarizerEngine; + expect(engine).toBeInstanceOf(LosslessSummarizerEngine); + return engine; +} + +const identity = (text: string): string => text; + +describe("seam policy knobs", () => { + it("forwards threshold, retain, headroom, and model policies into the resolved seam config", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { + enabled: true, + sidecarDir: tempDir(), + thresholdRatio: 0.5, + retainTokens: 100, + headroomTokens: 4096, + modelPolicies: [{ provider: ROUTE.provider, model: ROUTE.model, thresholdRatio: 0.9 }], + }); + try { + const config = engineOf(ctx).config; + expect(config.thresholdRatio).toBe(0.5); + expect(config.retainTokens).toBe(100); + expect(config.headroomTokens).toBe(4096); + expect(config.maxTokens).toBe(4096); + expect(config.modelPolicies).toHaveLength(1); + expect(config.modelPolicies[0]).toMatchObject({ + provider: ROUTE.provider, + model: ROUTE.model, + thresholdRatio: 0.9, + }); + } finally { + dispose(); + } + }); + + it("forwards the retain ratio form when set", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { + enabled: true, + sidecarDir: tempDir(), + retainRatio: 0.1, + }); + try { + const config = engineOf(ctx).config; + expect(config.retainRatio).toBe(0.1); + expect(config.retainTokens).toBeUndefined(); + } finally { + dispose(); + } + }); + + it("keeps seam defaults when the plugin config omits the knobs", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: tempDir() }); + try { + const config = engineOf(ctx).config; + expect(config.thresholdRatio).toBe(0.8); + expect(config.retainRatio).toBe(0.16); + expect(config.retainTokens).toBeUndefined(); + expect(config.headroomTokens).toBe(65_536); + expect(config.maxTokens).toBe(65_536); + expect(config.modelPolicies).toEqual([]); + expect(config.auto).toBe(false); + } finally { + dispose(); + } + }); + + it("rejects adapter-only keys on the seam config", async () => { + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + expect( + () => + new LosslessSummarizerEngine( + ctx, + { auto: false, bracketAuto: true } as unknown as BasicCompactionConfig, + { store: () => store, index: () => index, redact: identity }, + ), + ).toThrow('unknown key "bracketAuto"'); + }); +}); + +describe("summarizer caps", () => { + it("leaves the pass llm config empty when caps are unset", () => { + expect(passLlmConfig({})).toEqual({}); + expect(healLlmConfig({})).toEqual({ summarizerTokensPerPass: 120_000 }); + expect(healRunBudgets({})).toEqual({ + maxTokens: 120_000, + targetTokens: 1500, + callBudgetTokens: 60_000, + }); + }); + + it("records a capped run through a tiny per-call cap", async () => { + const session = Session.create(SessionId("cap-1")); + appendConversation(session, ROUTE, "working notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [textChunks("never used")]); + mountMeter(ctx); + const rows: Record[] = []; + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { + store: () => store, + index: () => index, + redact: identity, + summarizerTokensPerCall: 10, + record: (row) => rows.push(row), + }, + ); + const nodes = session.surface.nodes; + // A refused pass stores nothing billable, so the run may succeed + // mechanical or fail the shrink check; the capped row is the assertion. + await engine + .compactRegion(nodes[1]!, nodes[nodes.length - 1]!, fakeAgent(session, ROUTE)) + .catch(() => undefined); + const capped = rows.filter((row) => row.kind === "summarizer-capped"); + expect(capped).toHaveLength(1); + expect(capped[0]).toMatchObject({ reason: "call", budgetTokens: 10 }); + }); +}); + +describe("summarizer budgets", () => { + function losslessOf(ctx: Context): LosslessEngineDeps { + const engine = engineOf(ctx); + return (engine as unknown as { lossless: LosslessEngineDeps }).lossless; + } + + it("forwards non-empty budgets through plugin wiring", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { + enabled: true, + sidecarDir: tempDir(), + summaryTokens: 40, + leafChunkTokens: 500, + summarizerTokensPerPass: 1_000, + summarizerTokensPerHeal: 2_000, + }); + try { + const lossless = losslessOf(ctx); + expect(lossless.summaryTokens).toBe(40); + expect(lossless.leafChunkTokens).toBe(500); + expect(lossless.summarizerTokensPerPass).toBe(1_000); + expect(lossless.summarizerTokensPerHeal).toBe(2_000); + expect(passLlmConfig(lossless)).toEqual({ summarizerTokensPerPass: 1_000 }); + expect(healLlmConfig(lossless)).toEqual({ summarizerTokensPerPass: 2_000 }); + expect(healRunBudgets(lossless)).toMatchObject({ maxTokens: 2_000, targetTokens: 40 }); + } finally { + dispose(); + } + }); + + it("keeps bracketAuto off unless configured", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: tempDir() }); + try { + expect(losslessOf(ctx).bracketAuto ?? false).toBe(false); + } finally { + dispose(); + } + }); +}); + +describe("mechanical tools", () => { + const TOOL_TEXT = `tool noisy-read {"path":"/a"}\n${"x".repeat(2000)}`; + + function toolSpan( + sessionId: string, + ): Array<{ entryId: string; role: "toolResult"; text: string }> { + return [ + { entryId: `${sessionId}:1`, role: "toolResult", text: TOOL_TEXT }, + { entryId: `${sessionId}:2`, role: "toolResult", text: TOOL_TEXT }, + ]; + } + + it("compacts a listed tool-only span with zero model calls", async () => { + const { ctx, adapter } = setupContext(ROUTE, []); + const store = new LcmStore(":memory:"); + try { + const outcome = await runLadder( + ctx, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + mechanicalTools: ["noisy-read"], + }, + ROUTE, + "mech-listed", + toolSpan("mech-listed"), + ); + expect(adapter.requests).toHaveLength(0); + expect(outcome.leafSummaryIds.length).toBeGreaterThan(0); + for (const id of outcome.leafSummaryIds) { + expect(store.getSummary(id)?.quality).toBe("mechanical"); + } + } finally { + store.close(); + } + }); + + it("keeps a listed result mechanical when ingest mirrors prose plus the call", async () => { + const session = Session.create(SessionId("mech-mixed-ingest")); + session.append("turn/start", { turn: 1 }); + const callId = ToolCallId("call-mixed-mech"); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [ + { type: "text", text: "reading now" }, + { type: "tool-call", id: callId, name: "noisy-read", arguments: '{"path":"/a"}' }, + ], + source: { kind: "model", provider: ROUTE.provider, model: ROUTE.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("tool/call", { + turn: 1, + step: 1, + callId, + name: "noisy-read", + arguments: '{"path":"/a"}', + }); + const resultEvent = session.append( + "tool/result", + { + turn: 1, + step: 1, + message: createToolResultMessage({ + callId, + content: [{ type: "text", text: `config body ${"x".repeat(2000)}` }], + isError: false, + }), + }, + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + try { + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const row = store.messageByEntryId(`${session.id}:${resultEvent.seq}`); + expect(row?.role).toBe("toolResult"); + expect(row?.text.startsWith("tool noisy-read\n")).toBe(true); + const { ctx, adapter } = setupContext(ROUTE, []); + const outcome = await runLadder( + ctx, + { + store: () => store, + index: () => index, + redact: identity, + mechanicalTools: ["noisy-read"], + }, + ROUTE, + "mech-mixed-ingest", + [{ entryId: row!.entryId, role: row!.role, text: row!.text }], + ); + expect(adapter.requests).toHaveLength(0); + expect(outcome.leafSummaryIds.length).toBeGreaterThan(0); + for (const id of outcome.leafSummaryIds) { + expect(store.getSummary(id)?.quality).toBe("mechanical"); + } + } finally { + store.close(); + } + }); + + it("still calls the model for unlisted tools", async () => { + const { ctx, adapter } = setupContext(ROUTE, [ + textChunks("prose summary"), + textChunks("prose summary"), + textChunks("prose summary"), + textChunks("prose summary"), + ]); + const store = new LcmStore(":memory:"); + try { + await runLadder( + ctx, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + mechanicalTools: ["other-tool"], + }, + ROUTE, + "mech-unlisted", + toolSpan("mech-unlisted"), + ); + expect(adapter.requests.length).toBeGreaterThan(0); + } finally { + store.close(); + } + }); + + it("keeps the LLM path when the list is empty or unset", async () => { + for (const mechanicalTools of [undefined, []] as const) { + const { ctx, adapter } = setupContext(ROUTE, [ + textChunks("prose summary"), + textChunks("prose summary"), + textChunks("prose summary"), + textChunks("prose summary"), + ]); + const store = new LcmStore(":memory:"); + try { + await runLadder( + ctx, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + ...(mechanicalTools === undefined ? {} : { mechanicalTools }), + }, + ROUTE, + "mech-default", + toolSpan("mech-default"), + ); + expect(adapter.requests.length).toBeGreaterThan(0); + } finally { + store.close(); + } + } + }); + + it("forwards the plugin list and drops an empty one", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dispose = applyLossless(ctx, { + enabled: true, + sidecarDir: tempDir(), + mechanicalTools: ["noisy-read", " noisy-read ", ""], + }); + try { + expect(losslessDepsOf(ctx).mechanicalTools).toEqual(["noisy-read"]); + } finally { + dispose(); + } + const again = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(again.ctx); + const disposeEmpty = applyLossless(again.ctx, { + enabled: true, + sidecarDir: tempDir(), + mechanicalTools: [], + }); + try { + expect(losslessDepsOf(again.ctx).mechanicalTools).toBeUndefined(); + } finally { + disposeEmpty(); + } + }); + + function losslessDepsOf(ctx: Context): LosslessEngineDeps { + const engine = engineOf(ctx); + return (engine as unknown as { lossless: LosslessEngineDeps }).lossless; + } +}); + +describe("heal source default", () => { + function depsFor(dir: string, stores: Map, extra?: Partial) { + return { + stores: () => stores, + storeFor: (sessionId: string) => { + let store = stores.get(sessionId); + if (!store) { + store = new LcmStore(join(dir, `${sessionId}.sqlite`)); + stores.set(sessionId, store); + } + return store; + }, + sidecarDir: dir, + storePathFor: (sessionId: string) => join(dir, `${sessionId}.sqlite`), + ...extra, + }; + } + + function seedHealTarget(store: LcmStore): void { + ingestEntries( + store, + [ + `heal source alpha ${"detail ".repeat(60)}`, + `heal source beta ${"context ".repeat(60)}`, + ].map((text, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text, + timestamp: i, + })), + ); + const messageIds = store.messagesInSpan("e00", "e01").map((message) => message.id); + store.insertSummary({ + kind: "leaf", + text: `${TRUNCATED_SUMMARY_PREFIX}\nmechanical`, + tokens: 5, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds, + quality: "mechanical", + }); + } + + async function dryHeal( + sessionId: string, + extra?: Partial, + ): Promise<{ kind: string; text?: string }> { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId(sessionId), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores, extra); + seedHealTarget(deps.storeFor(sessionId)); + const definition = lcmCommandDefinition(ctx, deps); + return definition.handler({ + agent: fakeAgent(ctx.sessions.get(SessionId(sessionId))!, ROUTE), + signal: new AbortController().signal, + commandId: CommandId("cmd-heal"), + rawInput: "heal", + attachments: [], + }) as Promise<{ kind: string; text?: string }>; + } + + it("derives 8x the summary target when the source window is unset", async () => { + const dry = await dryHeal("heal-default"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("12000 source tokens"); + }); + + it("clamps a small summary target to the 3000 floor", async () => { + const dry = await dryHeal("heal-floor", { summaryTokens: 40 }); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("3000 source tokens"); + }); + + it("honors an explicit heal source window", async () => { + const dry = await dryHeal("heal-explicit", { healSourceTokens: 5000 }); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("5000 source tokens"); + }); +}); diff --git a/packages/deepseek/test/cross-session.test.ts b/packages/deepseek/test/cross-session.test.ts new file mode 100644 index 0000000..7db6165 --- /dev/null +++ b/packages/deepseek/test/cross-session.test.ts @@ -0,0 +1,416 @@ +import { existsSync, mkdtempSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { LcmStore, SidecarStores, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import type { DshRoute } from "../src/model-host.ts"; +import { dshRef } from "../src/model-host.ts"; +import { dshReader } from "../src/reader.ts"; +import { + CROSS_SESSION_DEADLINE_MS, + CROSS_SESSION_HITS_LIMIT, + CROSS_SESSION_PER_STORE_HITS, + CROSS_SESSION_STORE_LIMIT, + answerGrep, + expandAnswer, + expandQueryAnswer, + searchPastSessions, + type RecallDeps, +} from "../src/tools.ts"; +import { setupContext, textChunks, toolCallChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +const CURRENT = "ses_current"; +const PAST = "ses_past"; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-lossless-cross-")); +} + +function seed(dir: string, sessionId: string, texts: readonly string[]): void { + const store = new LcmStore(join(dir, `${sessionId}.sqlite`)); + try { + ingestEntries( + store, + texts.map((text, index) => ({ + entryId: `${sessionId}:prompt:msg_${index + 1}`, + role: "user" as const, + text, + timestamp: index + 1, + })), + ); + } finally { + store.close(); + } +} + +const identity = (text: string): string => text; + +function deps( + dir: string, + current: LcmStore | undefined, +): { sidecars: SidecarStores; recall: RecallDeps } { + const sidecars = new SidecarStores({ dir }); + return { + sidecars, + recall: { + store: () => current, + redact: identity, + pastSessions: { list: (exclude?: string) => sidecars.list(exclude) }, + }, + }; +} + +describe("cross-session recall", () => { + it("labels a hit from another sidecar and never reads the calling session", () => { + const dir = tempDir(); + seed(dir, CURRENT, ["the passphrase is MARIGOLD-4421"]); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const { sidecars, recall } = deps(dir, undefined); + try { + const found = answerGrep(recall, CURRENT, { query: "ORCHID", scope: "sessions" }); + expect(found).toContain("ORCHID-7319"); + expect(found).toContain(`[lcm:session ${PAST}]`); + expect(found).toContain("lcm_expand({session, entry_id})"); + + const own = answerGrep(recall, CURRENT, { query: "MARIGOLD", scope: "sessions" }); + expect(own).not.toContain("MARIGOLD-4421"); + expect(own).toContain("No matches"); + + const all = answerGrep(recall, CURRENT, { query: "ORCHID", scope: "all_sessions" }); + expect(all).toContain(`[lcm:session ${PAST}]`); + } finally { + sidecars.close(); + } + }); + + it("records sessionsScanned on a cross-session hit", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const { sidecars, recall } = deps(dir, undefined); + const rows: Record[] = []; + try { + const found = answerGrep({ ...recall, record: (row) => rows.push(row) }, CURRENT, { + query: "ORCHID", + scope: "sessions", + }); + expect(found).toContain("ORCHID-7319"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ tool: "lcm_grep", scope: "sessions" }); + expect(typeof rows[0]?.["sessionsScanned"]).toBe("number"); + expect(rows[0]?.["sessionsScanned"] as number).toBeGreaterThanOrEqual(1); + } finally { + sidecars.close(); + } + }); + + it("keeps the default scope on the current session", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const current = new LcmStore(":memory:"); + try { + ingestEntries(current, [ + { entryId: "cur-1", role: "user", text: "the passphrase is MARIGOLD-4421", timestamp: 1 }, + ]); + const { sidecars, recall } = deps(dir, current); + try { + const found = answerGrep(recall, CURRENT, { query: "MARIGOLD" }); + expect(found).toContain("MARIGOLD-4421"); + expect(found).not.toContain(`[lcm:session ${PAST}]`); + const scoped = answerGrep(recall, CURRENT, { query: "MARIGOLD", scope: "session" }); + expect(scoped).toBe(found); + } finally { + sidecars.close(); + } + } finally { + current.close(); + } + }); + + it("reads a past entry through lcm_expand and refuses a missing session or entry", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const { sidecars, recall } = deps(dir, undefined); + try { + const entryId = `${PAST}:prompt:msg_1`; + const past = { pastSessions: recall.pastSessions }; + const read = expandAnswer(undefined, identity, { session: PAST, entry_id: entryId }, past); + expect(read).toContain(`[lcm:session ${PAST}]`); + expect(read).toContain("past session's history"); + expect(read).toContain("ORCHID-7319"); + + const ghost = expandAnswer( + undefined, + identity, + { session: "ses_ghost", entry_id: "ses_ghost:prompt:msg_1" }, + past, + ); + expect(ghost).toContain("no stored session with id ses_ghost"); + expect(existsSync(join(dir, "ses_ghost.sqlite"))).toBe(false); + + const missing = expandAnswer( + undefined, + identity, + { session: PAST, entry_id: `${PAST}:prompt:msg_404` }, + past, + ); + expect(missing).toContain("No stored message with entry id"); + expect(missing).not.toContain("ORCHID-7319"); + + const empty = expandAnswer(undefined, identity, { session: " ", entry_id: entryId }, past); + expect(empty).toContain("session must be a non-empty session id"); + } finally { + sidecars.close(); + } + }); + + it("answers expand_query from a past sidecar through the listing", async () => { + const dir = tempDir(); + seed(dir, PAST, ["deciding the recall pipeline rollout for next week"]); + const sidecars = new SidecarStores({ dir }); + try { + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "recall pipeline rollout" }), + textChunks( + "The recall pipeline rollout was approved for next week [ses_past:prompt:msg_1].", + ), + ]); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + undefined, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + CURRENT, + { query: "recall pipeline rollout", session: PAST }, + (row) => rows.push(row), + { list: (exclude?: string) => sidecars.list(exclude) }, + ); + expect(answer).toContain("a past session's history (session ses_past)"); + expect(answer).toContain(`session="ses_past"`); + expect(answer).toContain("recall pipeline rollout"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "answered" }); + expect(existsSync(join(dir, `${CURRENT}.sqlite`))).toBe(false); + } finally { + sidecars.close(); + } + }); + + it("does not modify a past store", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const path = join(dir, `${PAST}.sqlite`); + const before = { + bytes: readFileSync(path), + mtimeMs: statSync(path).mtimeMs, + }; + const { sidecars, recall } = deps(dir, undefined); + try { + expect(answerGrep(recall, CURRENT, { query: "ORCHID", scope: "sessions" })).toContain( + "ORCHID-7319", + ); + expect( + expandAnswer( + undefined, + identity, + { session: PAST, entry_id: `${PAST}:prompt:msg_1` }, + { pastSessions: recall.pastSessions }, + ), + ).toContain("ORCHID-7319"); + } finally { + sidecars.close(); + } + expect(readFileSync(path).equals(before.bytes)).toBe(true); + expect(statSync(path).mtimeMs).toBe(before.mtimeMs); + expect(existsSync(join(dir, `${CURRENT}.sqlite`))).toBe(false); + }); + + it("forwards role, date, and tool filters to past sessions", () => { + const dir = tempDir(); + const past = new LcmStore(join(dir, `${PAST}.sqlite`)); + try { + ingestEntries(past, [ + { + entryId: `${PAST}:prompt:msg_1`, + role: "user", + text: "zephyr routing decision from a user", + timestamp: 100, + }, + { + entryId: `${PAST}:prompt:msg_2`, + role: "assistant", + text: "zephyr routing decision from the assistant", + timestamp: 200, + }, + { + entryId: `${PAST}:prompt:msg_3`, + role: "assistant", + text: "[tool:Read] zephyr routing decision file", + timestamp: 300, + }, + { + entryId: `${PAST}:prompt:msg_4`, + role: "toolResult", + text: "[tool:Read] zephyr routing decision contents", + timestamp: 400, + }, + ]); + } finally { + past.close(); + } + const { sidecars, recall } = deps(dir, undefined); + try { + const role = answerGrep(recall, CURRENT, { + query: "zephyr", + scope: "sessions", + role: "assistant", + }); + expect(role).toContain("msg_2"); + expect(role).toContain("msg_3"); + expect(role).not.toContain("msg_1"); + expect(role).not.toContain("msg_4"); + const window = answerGrep(recall, CURRENT, { + query: "zephyr", + scope: "sessions", + date_start: 150, + date_end: 250, + }); + expect(window).toContain("msg_2"); + expect(window).not.toContain("msg_1"); + expect(window).not.toContain("msg_3"); + const scoped = answerGrep(recall, CURRENT, { + query: "zephyr", + scope: "sessions", + tool_name: "Read", + }); + expect(scoped).toContain("msg_3"); + expect(scoped).toContain("msg_4"); + expect(scoped).not.toContain("msg_1"); + expect(scoped).not.toContain("msg_2"); + } finally { + sidecars.close(); + } + }); +}); + +describe("cross-session caps", () => { + it("stops at the store cap and names it", () => { + const dir = tempDir(); + for (let i = 0; i < CROSS_SESSION_STORE_LIMIT + 5; i += 1) { + seed(dir, `ses_cap_${String(i).padStart(2, "0")}`, ["the passphrase is ORCHID-7319"]); + } + const { sidecars, recall } = deps(dir, undefined); + try { + const result = searchPastSessions( + { past: recall.pastSessions!, redact: identity }, + CURRENT, + "ORCHID", + ); + expect(result.stoppedBy).toBe("stores"); + expect(result.scanned).toBe(CROSS_SESSION_STORE_LIMIT); + expect(result.text).toContain(`${CROSS_SESSION_STORE_LIMIT}-store cap`); + } finally { + sidecars.close(); + } + }); + + it("stops at the hit cap with at most ten hits per store", () => { + const dir = tempDir(); + for (let s = 0; s < 5; s += 1) { + seed( + dir, + `ses_hits_${s}`, + Array.from({ length: 12 }, (_, i) => `ORCHID marker s${s} m${i}`), + ); + } + const { sidecars, recall } = deps(dir, undefined); + try { + const result = searchPastSessions( + { past: recall.pastSessions!, redact: identity }, + CURRENT, + "ORCHID", + ); + expect(result.stoppedBy).toBe("hits"); + expect(result.hits).toBe(CROSS_SESSION_HITS_LIMIT); + expect(result.text).toContain(`${CROSS_SESSION_HITS_LIMIT}-hit cap`); + } finally { + sidecars.close(); + } + }); + + it("caps one chatty store at ten hits", () => { + const dir = tempDir(); + seed( + dir, + PAST, + Array.from({ length: 15 }, (_, i) => `ORCHID marker m${i}`), + ); + const { sidecars, recall } = deps(dir, undefined); + try { + const result = searchPastSessions( + { past: recall.pastSessions!, redact: identity }, + CURRENT, + "ORCHID", + ); + expect(result.stoppedBy).toBeUndefined(); + expect(result.hits).toBe(CROSS_SESSION_PER_STORE_HITS); + expect(result.scanned).toBe(1); + } finally { + sidecars.close(); + } + }); + + it("stops at the deadline when the clock runs past it", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const { sidecars, recall } = deps(dir, undefined); + try { + let calls = 0; + const now = (): number => { + calls += 1; + return calls <= 1 ? 1_000 : 1_000 + CROSS_SESSION_DEADLINE_MS + 1; + }; + const result = searchPastSessions( + { past: recall.pastSessions!, redact: identity }, + CURRENT, + "ORCHID", + { now }, + ); + expect(result.stoppedBy).toBe("deadline"); + expect(result.scanned).toBe(0); + expect(result.text).toContain(`${CROSS_SESSION_DEADLINE_MS} ms deadline`); + } finally { + sidecars.close(); + } + }); + + it("counts an unreadable sidecar as skipped", () => { + const dir = tempDir(); + seed(dir, PAST, ["the passphrase is ORCHID-7319"]); + const { sidecars, recall } = deps(dir, undefined); + try { + const good = recall.pastSessions!.list(CURRENT).find((ref) => ref.id === PAST); + expect(good).toBeDefined(); + const corruptPath = join(dir, "ses_corrupt.sqlite"); + writeFileSync(corruptPath, "not a database"); + const result = searchPastSessions( + { + past: { list: () => [good!, { id: "ses_corrupt", path: corruptPath }] }, + redact: identity, + }, + CURRENT, + "ORCHID", + ); + expect(result.skipped).toBe(1); + expect(result.scanned).toBe(1); + expect(result.hits).toBe(1); + expect(result.stoppedBy).toBeUndefined(); + expect(result.text).toContain("skipped 1 unreadable"); + } finally { + sidecars.close(); + } + }); +}); diff --git a/packages/deepseek/test/dag-commit.test.ts b/packages/deepseek/test/dag-commit.test.ts new file mode 100644 index 0000000..6898d04 --- /dev/null +++ b/packages/deepseek/test/dag-commit.test.ts @@ -0,0 +1,553 @@ +import { Context } from "@deepseek-ai/cordis"; +import { isCompactCheckpointSource, ManualCompactionError } from "@deepseek-ai/dsh-compaction"; +import { createMessage, createSystemMessage, createUserMessage } from "@deepseek-ai/dsh-llm"; +import { Session, SessionId } from "@deepseek-ai/dsh-session"; +import type { SessionEvent } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import { LcmStore } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { commitDagTransaction, selectDagRange } from "../src/dag-commit.ts"; +import type { DagCommitDeps } from "../src/dag-commit.ts"; +import { LosslessSummarizerEngine } from "../src/engine.ts"; +import { MessageIndex, mirrorEvent } from "../src/ingest.ts"; +import { resolveSummarizationTarget, SpanAlreadyMemoryError } from "../src/ladder.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { errorChunks, setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; +const SUMMARY = "Checkpoint: engine work started and the config was read."; + +function buildSession(text: string): Session { + const session = Session.create(SessionId("dag-commit-1")); + appendConversation(session, ROUTE, text); + session.append("turn/start", { turn: 3 }); + return session; +} + +function mountMeter(ctx: Context): void { + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function ofType( + events: readonly SessionEvent[], + type: T, +): Extract[] { + return events.filter((event): event is Extract => event.type === type); +} + +function deps(ctx: Context, store: LcmStore, index: MessageIndex): DagCommitDeps { + return { + ctx, + summarizationProvider: "", + summarizationModel: "", + store: () => store, + index: () => index, + redact: (text) => text, + }; +} + +describe("selectDagRange", () => { + it("returns null when only the retained system head exists", () => { + const session = Session.create(SessionId("dag-range-1")); + session.append( + "system/message", + { turn: 1, step: 1, message: createSystemMessage("You are a test.") }, + { surfaceOp: "append" }, + ); + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + expect(selectDagRange(session, ctx.tokenMeter.measure(session), 0)).toBeNull(); + }); + + it("starts after the system head and leaves the tail edge", () => { + const session = buildSession("ranged notes ".repeat(20)); + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + const nodes = session.surface.nodes; + const range = selectDagRange(session, ctx.tokenMeter.measure(session), 0); + expect(range).not.toBeNull(); + expect(range?.start).toBe(nodes[1]); + expect(range?.end).toBe(nodes[nodes.length - 2]); + }); + + it("fails closed when the surface moved under the measurement", () => { + const session = buildSession("drift notes ".repeat(20)); + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + const measurement = ctx.tokenMeter.measure(session); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "drift intruder" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + expect(() => selectDagRange(session, measurement, 0)).toThrow(/does not match/); + }); +}); + +describe("commitDagTransaction", () => { + it("commits one condensed summary in a single bracket with leaves sidecar only", async () => { + const session = buildSession("working notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + const before = session.deriveMessages().length; + const result = await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + expect(result.compactionId.length).toBeGreaterThan(0); + expect(result.summarySeqs).toHaveLength(1); + expect(result.summarySeqs[0]).toBe(result.summarySeq); + expect(result.leafSeqs).toHaveLength(0); + expect(result.leafSummaryIds.length).toBeGreaterThan(0); + expect(result.leafSummaryIds.length + result.condensedSummaryIds.length).toBeGreaterThan(0); + expect(result.startSeq).toBeLessThan(result.summarySeqs[0]!); + expect(result.summarySeqs[result.summarySeqs.length - 1]!).toBeLessThan(result.endSeq); + const events = session.snapshotEvents(); + const starts = ofType(events, "compaction/start"); + const ends = ofType(events, "compaction/end"); + expect(starts).toHaveLength(1); + expect(ends).toHaveLength(1); + expect(starts[0]?.data.compactionId).toBe(result.compactionId); + expect(ends[0]?.data.compactionId).toBe(result.compactionId); + expect(ends[0]?.data.error).toBeUndefined(); + const summaries = ofType(events, "compaction/summary"); + expect(summaries).toHaveLength(1); + expect(summaries).toHaveLength(result.summarySeqs.length); + for (const summary of summaries) { + expect(summary.data.compactionId).toBe(result.compactionId); + } + const checkpoints = ofType(events, "user/message").filter((event) => + isCompactCheckpointSource(event.data.source), + ); + expect(checkpoints).toHaveLength(1); + expect(session.deriveMessages().length).toBeLessThan(before); + let depth = 0; + for (const event of events) { + if (event.type === "compaction/start") depth += 1; + if (event.type === "compaction/end") depth -= 1; + } + expect(depth).toBe(0); + }); + + it("writes exactly one session log summary per bracket across two compactions", async () => { + const session = buildSession("repeat notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const firstNodes = session.surface.nodes; + const first = await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: firstNodes[1]!, end: firstNodes[2]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const pre = [...session.surface.nodes]; + const second = await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: pre[1]!, end: pre[pre.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + const events = session.snapshotEvents(); + const summaries = ofType(events, "compaction/summary"); + expect(summaries).toHaveLength(2); + expect(new Set(summaries.map((summary) => summary.data.compactionId)).size).toBe(2); + expect(summaries.map((summary) => summary.seq)).toEqual([first.summarySeq, second.summarySeq]); + expect(first.summarySeqs).toHaveLength(1); + expect(second.summarySeqs).toHaveLength(1); + expect(first.leafSeqs).toHaveLength(0); + expect(second.leafSeqs).toHaveLength(0); + }); + + it("emits one compaction-checkpoint row carrying the committed seq plus sidecar ids", async () => { + const session = buildSession("checkpoint notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + const rows: Record[] = []; + const result = await commitDagTransaction( + { ...deps(ctx, store, index), record: (row) => rows.push(row) }, + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + const checkpoints = rows.filter((row) => row.kind === "compaction-checkpoint"); + expect(checkpoints).toHaveLength(1); + expect(checkpoints[0]).toMatchObject({ + event: "lcm", + session: "dag-commit-1", + summarySeqs: [...result.summarySeqs], + leafSeqs: [...result.leafSeqs], + leaf: result.leafSummaryIds.length, + condensed: result.condensedSummaryIds.length, + leafSummaryIds: [...result.leafSummaryIds], + condensedSummaryIds: [...result.condensedSummaryIds], + }); + expect(result.summarySeqs).toHaveLength(1); + expect(result.leafSeqs).toHaveLength(0); + expect(result.leafSummaryIds.length).toBeGreaterThan(0); + expect(typeof checkpoints[0]?.ts).toBe("number"); + }); + + it("orders the prepared span by seq when rows landed out of order", async () => { + const session = buildSession("backfill order ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + const events = session.snapshotEvents(); + // Two mirror batches shaped like a real catch-up: the first batch holds + // a prefix plus newer live rows, the second backfills the gap in order, + // so gap rows land on newer rowids than the newer rows they precede. + const gap = events.slice(4, 7); + const rest = events.filter((event) => !gap.includes(event)); + for (const event of rest) mirrorEvent(session, event, store, index); + for (const event of gap) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const pre = [...session.surface.nodes]; + await commitDagTransaction( + { ...deps(ctx, store, index), record: (row) => rows.push(row) }, + agent, + { start: pre[1]!, end: pre[pre.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + const seqs = pre.slice(1).map(Number); + const sid = String(session.id); + const pass = rows.find((row) => row.kind === "pass"); + expect(pass?.span).toEqual([`${sid}:${Math.min(...seqs)}`, `${sid}:${Math.max(...seqs)}`]); + }); + + it("keeps the checkpoint at its seq position after a replace", async () => { + const session = buildSession("checkpoint order ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const firstNodes = session.surface.nodes; + await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: firstNodes[1]!, end: firstNodes[2]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const rows: Record[] = []; + const pre = [...session.surface.nodes]; + await commitDagTransaction( + { ...deps(ctx, store, index), record: (row) => rows.push(row) }, + agent, + { start: pre[1]!, end: pre[pre.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + const seqs = pre.slice(1).map(Number); + const sid = String(session.id); + const pass = rows.find((row) => row.kind === "pass"); + expect(pass?.span).toEqual([`${sid}:${Math.min(...seqs)}`, `${sid}:${Math.max(...seqs)}`]); + }); + + it("closes the bracket with the error and releases the lock when the ladder fails", async () => { + const session = buildSession("failing notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [errorChunks()]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + await expect( + commitDagTransaction( + deps(ctx, store, index), + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ), + ).rejects.toThrow(); + const events = session.snapshotEvents(); + const ends = ofType(events, "compaction/end"); + expect(ends).toHaveLength(1); + expect(typeof ends[0]?.data.error).toBe("string"); + await expect( + commitDagTransaction( + deps(ctx, store, index), + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ), + ).rejects.toThrow(/^(?!.*already in progress).*$/s); + }); + + it("maps a ladder failure to the manual summary taxonomy on an idle session", async () => { + const session = buildSession("manual notes ".repeat(20)); + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [errorChunks()]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + const failure = await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: null, stability: "selected-span" }, + ).catch((error: unknown) => error); + expect(failure).toBeInstanceOf(ManualCompactionError); + expect((failure as ManualCompactionError).code).toBe("summary"); + const ends = ofType(session.snapshotEvents(), "compaction/end"); + expect(ends).toHaveLength(1); + expect(ends[0]?.data.turn).toBeNull(); + }); + + it("leaves memory untouched when the shrink check fails", async () => { + const session = buildSession("ok "); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const bloated = "bloated summary ".repeat(400); + const { ctx } = setupContext(ROUTE, [ + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + textChunks(bloated), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + let notes = 0; + const memory = { + previous: (_sessionId: string) => undefined, + note: (_sessionId: string, _text: string) => { + notes += 1; + }, + }; + await expect( + commitDagTransaction( + { ...deps(ctx, store, index), memory }, + agent, + { start: nodes[1]!, end: nodes[2]! }, + { owner: "current-turn", stability: "whole-surface" }, + ), + ).rejects.toThrow(/not smaller/); + expect(notes).toBe(0); + }); + + it("notes memory exactly once when the bracket commits", async () => { + const session = buildSession("working notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + let notes = 0; + let noted: string | undefined; + const memory = { + previous: (_sessionId: string) => undefined, + note: (_sessionId: string, text: string) => { + notes += 1; + noted = text; + }, + }; + await commitDagTransaction( + { ...deps(ctx, store, index), memory }, + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + expect(notes).toBe(1); + expect(noted).toContain(SUMMARY); + }); + + it("heals a gapped mirror with one backfill catch-up instead of throwing", async () => { + const session = buildSession("catchup notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + const events = session.snapshotEvents(); + for (const event of events.slice(0, 3)) mirrorEvent(session, event, store, index); + expect(store.allMessages().length).toBeGreaterThan(0); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const liveEvents = session.snapshotEvents(); + (ctx as unknown as Record).sessionQuery = { + observeSession: async () => ({ events: liveEvents }), + }; + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const nodes = session.surface.nodes; + const result = await commitDagTransaction( + { ...deps(ctx, store, index), record: (row) => rows.push(row) }, + agent, + { start: nodes[1]!, end: nodes[nodes.length - 1]! }, + { owner: "current-turn", stability: "whole-surface" }, + ); + expect(result.summarySeqs).toHaveLength(1); + expect(rows.some((row) => row.kind === "backfill")).toBe(true); + }); + + it("rethrows already-memory on the manual path instead of mapping it to summary", async () => { + const session = buildSession("covered live ".repeat(30)); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `${"covered live ".repeat(30)} third request` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 3, step: 1 }); + session.append("request/header", { + header: { config: { provider: ROUTE.provider, model: ROUTE.model } }, + reason: "initial", + }); + session.append( + "assistant/message", + { + stream: [], + turn: 3, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: `${"covered live ".repeat(30)} third response` }], + source: { kind: "model", provider: ROUTE.provider, model: ROUTE.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 3, step: 1 }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks("First span."), + textChunks("Second span."), + textChunks("Third span."), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: (text) => text }, + ); + const nodes = session.surface.nodes; + await engine.compactRegion(nodes[1]!, nodes[2]!, agent); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const afterFirst = session.surface.nodes; + await engine.compactRegion(afterFirst[1]!, afterFirst[3]!, agent); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + const covered = session.surface.nodes; + const failure = await commitDagTransaction( + deps(ctx, store, index), + agent, + { start: covered[covered.length - 2]!, end: covered[covered.length - 1]! }, + { owner: null, stability: "selected-span" }, + ).catch((error: unknown) => error); + expect(failure).toBeInstanceOf(SpanAlreadyMemoryError); + }); +}); + +describe("resolveSummarizationTarget", () => { + it("falls back to the agent route when only one field is set", () => { + const session = Session.create(SessionId("summ-target-1")); + const agent = fakeAgent(session, ROUTE); + expect( + resolveSummarizationTarget( + { summarizationProvider: "cfg-provider", summarizationModel: "cfg-model" }, + agent, + ), + ).toEqual({ provider: "cfg-provider", model: "cfg-model" }); + expect( + resolveSummarizationTarget( + { summarizationProvider: "cfg-provider", summarizationModel: "" }, + agent, + ), + ).toEqual(ROUTE); + expect( + resolveSummarizationTarget( + { summarizationProvider: "", summarizationModel: "cfg-model" }, + agent, + ), + ).toEqual(ROUTE); + expect( + resolveSummarizationTarget({ summarizationProvider: "", summarizationModel: "" }, agent), + ).toEqual(ROUTE); + }); +}); diff --git a/packages/deepseek/test/default-sidecar-dir.test.ts b/packages/deepseek/test/default-sidecar-dir.test.ts new file mode 100644 index 0000000..a643b25 --- /dev/null +++ b/packages/deepseek/test/default-sidecar-dir.test.ts @@ -0,0 +1,111 @@ +import { join } from "node:path"; + +import { describe, expect, it } from "vite-plus/test"; + +import { + defaultSidecarDir, + resolveDefaultSidecarDir, + type DefaultSidecarEnv, +} from "../src/default-sidecar-dir.ts"; +import { readSidecarDir } from "../src/plugin.ts"; + +function resolve( + env: DefaultSidecarEnv, + homedir: string | undefined, +): { dir: string; source: string } { + return resolveDefaultSidecarDir({ env, homedir }); +} + +describe("resolveDefaultSidecarDir", () => { + it("prefers a non-blank absolute DSH_HOME", () => { + expect(resolve({ DSH_HOME: "/data" }, "/home/u")).toEqual({ + dir: join("/data", "lossless"), + source: "default:DSH_HOME", + }); + }); + + it("treats an empty DSH_HOME as unset", () => { + expect(resolve({ DSH_HOME: "" }, "/home/u")).toEqual({ + dir: join("/home/u", ".dsh", "lossless"), + source: "default:home", + }); + }); + + it("treats a whitespace-only DSH_HOME as unset", () => { + expect(resolve({ DSH_HOME: " " }, "/home/u")).toEqual({ + dir: join("/home/u", ".dsh", "lossless"), + source: "default:home", + }); + }); + + it("ignores a relative DSH_HOME", () => { + expect(resolve({ DSH_HOME: "rel/path" }, "/home/u")).toEqual({ + dir: join("/home/u", ".dsh", "lossless"), + source: "default:home", + }); + }); + + it("falls back to the home dir when DSH_HOME is unset", () => { + expect(resolve({}, "/Users/u")).toEqual({ + dir: join("/Users/u", ".dsh", "lossless"), + source: "default:home", + }); + }); + + it("treats a blank homedir as unset and falls back", () => { + expect(resolve({}, " ")).toEqual({ + dir: "./.lossless", + source: "fallback", + }); + }); + + it("falls back to ./.lossless with no home", () => { + expect(resolve({}, undefined)).toEqual({ + dir: "./.lossless", + source: "fallback", + }); + }); + + it("falls back to ./.lossless with no home even when DSH_HOME is relative", () => { + expect(resolve({ DSH_HOME: "rel/path" }, undefined)).toEqual({ + dir: "./.lossless", + source: "fallback", + }); + }); +}); + +describe("defaultSidecarDir", () => { + it("resolves without throwing", () => { + const resolved = defaultSidecarDir(); + expect(typeof resolved.dir).toBe("string"); + expect(["default:DSH_HOME", "default:home", "fallback"]).toContain(resolved.source); + }); +}); + +describe("readSidecarDir", () => { + it("passes an explicit relative dir through byte identical", () => { + const options = readSidecarDir("./.lossless"); + expect(options.dir).toBe("./.lossless"); + expect(options.source).toBe("explicit"); + }); + + it("passes an explicit absolute dir through byte identical", () => { + const options = readSidecarDir("/tmp/lcm-abs"); + expect(options.dir).toBe("/tmp/lcm-abs"); + expect(options.source).toBe("explicit"); + }); + + it("falls back to the default for a blank dir", () => { + const expected = defaultSidecarDir(); + const options = readSidecarDir(" "); + expect(options.dir).toBe(expected.dir); + expect(options.source).toBe(expected.source); + }); + + it("falls back to the default when unset", () => { + const expected = defaultSidecarDir(); + const options = readSidecarDir(undefined); + expect(options.dir).toBe(expected.dir); + expect(options.source).toBe(expected.source); + }); +}); diff --git a/packages/deepseek/test/e2e/README.md b/packages/deepseek/test/e2e/README.md new file mode 100644 index 0000000..8bdaa43 --- /dev/null +++ b/packages/deepseek/test/e2e/README.md @@ -0,0 +1,142 @@ +# Headless e2e probes + +Live harness probes for `deepseek-lossless`. These are manual recipes plus a +runner script, not unit tests: each probe boots a real `dsh` profile, calls a +real model over OpenRouter, and writes a real sidecar. The sidecar directory +defaults to the global dsh home path (`$DSH_HOME/lossless`, else +`~/.dsh/lossless`), so point `DSH_HOME` at a scratch dir to keep sidecars out +of the repo: `export DSH_HOME=/tmp/e2e-dsh`. + +## Scratch profiles + +Two disposable profiles under `~/.dsh/profiles/`. Real profiles +(`web`, `desktop`, `headless`) stay untouched. + +- `lossless-e2e`: base bundle plus the packed `deepseek-lossless` tarball, + with the OpenRouter route and default model pinned to the free + `cohere/north-mini-code:free` model in its `cordis.patch.yml`. Purpose: prove + the shipped bundle answers headless tasks and mirrors messages to the + sidecar. +- `lossless-plain`: base bundle only, same route and default model. Purpose: + control profile that proves a hang or failure is environmental rather than + caused by our bundle. + +## Tarball rebuild + +After changing the plugin, repack and reinstall into the scratch profile: + +```sh +cd packages/deepseek && npm pack +cd ../deepseek-web && npm pack +dsh plugin --profile lossless-e2e add file:../deepseek/deepseek-lossless-0.1.0.tgz +``` + +## Probe 1: shipped headless profile plus model overlay + +The shipped `headless` profile (base plus `@deepseek-ai/dsh-headless` +bundles) has the OpenRouter route but needs the default model pinned, so it +runs with the model overlay: + +```sh +cd /tmp +DSH_PERMISSION_MODE=danger-full-access timeout 60 dsh --profile headless --patch ./overlays/model-overlay.yml --json "reply with exactly: pong" > /tmp/e2e-probe1.json +``` + +(Replace `./overlays/model-overlay.yml` with the path to +`packages/deepseek/test/e2e/overlays/model-overlay.yml` in this repo.) + +## Probe 2: plain base plus headless overlay + +The `lossless-plain` profile has no headless rows, so they are injected with +the headless overlay: + +```sh +cd /tmp +DSH_PERMISSION_MODE=danger-full-access timeout 60 dsh --profile lossless-plain --patch ./overlays/headless-overlay.yml headless --json "reply with exactly: pong" > /tmp/e2e-probe2.json +``` + +## Probe 3: lossless bundle pong plus fresh sidecar + +No overlay needed: `lossless-e2e` already carries the route, the default +model, and the bundle: + +```sh +cd /tmp +DSH_PERMISSION_MODE=danger-full-access timeout 60 dsh --profile lossless-e2e headless --json "reply with exactly: pong" > /tmp/e2e-lossless.json +ls /tmp/e2e-dsh/lossless/*.sqlite +``` + +The scratch dir starts empty and the sidecar must postdate the probe, so a +pass proves this run wrote it rather than a previous one. + +## Probe 4: second pong to warm backfill metrics + +A pong plus sidecar rows alone need not record anything: live mirroring +writes database rows without metrics. The second mount sees probe 3's +sidecar, so mount backfill records its row even when everything dedups: + +```sh +cd /tmp +DSH_PERMISSION_MODE=danger-full-access timeout 60 dsh --profile lossless-e2e headless --json "reply with exactly: pong" > /tmp/e2e-lossless2.json +test -s /tmp/e2e-dsh/lossless/metrics.jsonl +``` + +## Expected pong output shape + +With `--json`, stdout is newline-delimited run events. A passing run ends with +the final event on the last line: + +```json +{ "type": "final", "text": "pong" } +``` + +Without `--json`, stdout is just `pong` and diagnostics go to stderr. Any +timeout (exit 124), empty output, or missing final event is a failure. Known +history: on 2026-10-27 these same commands hung with zero output and no +sidecar writes even without our bundle, which pointed at the environment +(free-tier queue, approval gate, or session startup) rather than the plugin, +because raw `curl` to the same OpenRouter model answered in about one second. + +## Sidecar check paths + +After a passing probe 3, the sqlite files exist; after probe 4, the metrics +log joins them (mount backfill records on the second mount): + +- `/tmp/e2e-dsh/lossless/*.sqlite` (one per headless session) +- `/tmp/e2e-dsh/lossless/metrics.jsonl` + +## Recall note + +The `/lcm` slash commands exist only in TUI and Web sessions. Typing `/lcm +status` into a headless shell just fails with `/lcm: No such file or +directory` (see the `e2e-lcm-status` probe pattern). In headless runs, recall +goes through the `lcm_grep` family of tools instead: resume the earlier +session with `--session-id ` and ask the model to search with `lcm_grep`, +which returns matches like `Found 4 match(es)` followed by the answer. + +## Free model fallback order + +If the pinned model stalls or rate-limits, try the next model in this order +(copied from `~/.dsh/profiles/desktop/cordis.patch.yml`, probed working +2026-10-25): + +1. `cohere/north-mini-code:free` +2. `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free` +3. `nvidia/nemotron-3-ultra-550b-a55b:free` +4. `poolside/laguna-s-2.1:free` +5. `xingchen-agi/xing4.0-29b-a4b:free` + +Swap the model id in `overlays/model-overlay.yml` (or the scratch profile +patch) and rerun the probe. + +## Runner script + +`run-e2e.sh` executes probes 1 to 3 in order with `timeout 60` guards, +asserts the pong final event in each output file, and lists the sidecar +files. It exports `DSH_HOME=/tmp/e2e-dsh` so sidecars land in +`/tmp/e2e-dsh/lossless`, matching the scratch dir above. It takes no +arguments and stops nonzero on the first failure: + +```sh +./packages/deepseek/test/e2e/run-e2e.sh +``` diff --git a/packages/deepseek/test/e2e/overlays/headless-overlay.yml b/packages/deepseek/test/e2e/overlays/headless-overlay.yml new file mode 100644 index 0000000..5e8583d --- /dev/null +++ b/packages/deepseek/test/e2e/overlays/headless-overlay.yml @@ -0,0 +1,13 @@ +# Probe 2 overlay: inject headless rows over lossless-plain base. +- insert: + - id: headless-startup + name: "@deepseek-ai/dsh-headless/startup" + + # Reads its task and run options from the ordinary headlessStartup provider. + - id: headless-runner + name: "@deepseek-ai/dsh-headless" + inject: [headlessStartup] + config: + task: !!js ctx.headlessStartup.task + sessionId: !!js ctx.headlessStartup.sessionId + json: !!js ctx.headlessStartup.json diff --git a/packages/deepseek/test/e2e/overlays/model-overlay.yml b/packages/deepseek/test/e2e/overlays/model-overlay.yml new file mode 100644 index 0000000..df6a89b --- /dev/null +++ b/packages/deepseek/test/e2e/overlays/model-overlay.yml @@ -0,0 +1,4 @@ +- id: agent-default-model + config: + provider: openrouter + model: cohere/north-mini-code:free diff --git a/packages/deepseek/test/e2e/run-e2e.sh b/packages/deepseek/test/e2e/run-e2e.sh new file mode 100755 index 0000000..f401293 --- /dev/null +++ b/packages/deepseek/test/e2e/run-e2e.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# +# Headless e2e probes for deepseek-lossless. +# +# Runs probe 1 (shipped headless profile + model overlay), probe 2 (plain +# base + headless overlay), probe 3 (lossless bundle pong + fresh sidecar), +# and probe 4 (second pong to warm backfill metrics). Each dsh call is +# guarded with `timeout 60`. Takes no arguments, exits nonzero on the first +# failure, and prints EXIT codes plus sidecar paths. Probes export +# DSH_HOME=/tmp/e2e-dsh so sidecars land in /tmp/e2e-dsh/lossless. +# +# Usage: ./packages/deepseek/test/e2e/run-e2e.sh +set -uo pipefail + +E2E_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OVERLAYS_DIR="$E2E_DIR/overlays" +OUT_DIR="/tmp" +export DSH_HOME="/tmp/e2e-dsh" +SIDECAR_DIR="$DSH_HOME/lossless" +TASK="reply with exactly: pong" + +export DSH_PERMISSION_MODE=danger-full-access + +if ! command -v dsh >/dev/null 2>&1; then + echo "FAIL: dsh not found on PATH" >&2 + exit 1 +fi + +mkdir -p "$SIDECAR_DIR" || exit 1 +# Fresh scratch every run: stale sidecars from a previous pass must not mask +# a regression that stops writing them. +rm -rf "$SIDECAR_DIR" || exit 1 +mkdir -p "$SIDECAR_DIR" || exit 1 +cd "$OUT_DIR" || exit 1 + +check_pong() { + local label="$1" + local file="$2" + if grep -qF '{"type":"final","text":"pong"}' "$file" 2>/dev/null; then + echo "$label: pong OK ($file)" + else + echo "$label: FAIL: no pong final event in $file" >&2 + exit 1 + fi +} + +echo "--- probe 1: shipped headless profile + model overlay ---" +timeout 60 dsh --profile headless --patch "$OVERLAYS_DIR/model-overlay.yml" --json "$TASK" > "$OUT_DIR/e2e-probe1.json" +rc=$? +echo "probe1 EXIT=$rc" +[ "$rc" -eq 0 ] || { echo "probe1: FAIL: dsh exited $rc" >&2; exit 1; } +check_pong "probe1" "$OUT_DIR/e2e-probe1.json" + +echo "--- probe 2: plain base + headless overlay ---" +timeout 60 dsh --profile lossless-plain --patch "$OVERLAYS_DIR/headless-overlay.yml" headless --json "$TASK" > "$OUT_DIR/e2e-probe2.json" +rc=$? +echo "probe2 EXIT=$rc" +[ "$rc" -eq 0 ] || { echo "probe2: FAIL: dsh exited $rc" >&2; exit 1; } +check_pong "probe2" "$OUT_DIR/e2e-probe2.json" + +echo "--- probe 3: lossless bundle pong ---" +touch "$OUT_DIR/e2e-probe3-start" +timeout 60 dsh --profile lossless-e2e headless --json "$TASK" > "$OUT_DIR/e2e-lossless.json" +rc=$? +echo "probe3 EXIT=$rc" +[ "$rc" -eq 0 ] || { echo "probe3: FAIL: dsh exited $rc" >&2; exit 1; } +check_pong "probe3" "$OUT_DIR/e2e-lossless.json" + +echo "--- sidecars in $SIDECAR_DIR ---" +shopt -s nullglob +sidecars=("$SIDECAR_DIR"/*.sqlite) +shopt -u nullglob +if [ "${#sidecars[@]}" -eq 0 ]; then + echo "probe3: FAIL: no sidecar sqlite files in $SIDECAR_DIR" >&2 + exit 1 +fi +printf '%s\n' "${sidecars[@]}" +# Freshness: the sidecar must postdate probe 3, not a previous run. +if ! find "$SIDECAR_DIR" -name '*.sqlite' -newer "$OUT_DIR/e2e-probe3-start" | grep -q .; then + echo "probe3: FAIL: no sidecar written during this run in $SIDECAR_DIR" >&2 + exit 1 +fi +echo "probe3: fresh sidecar OK" + +echo "--- probe 4: lossless bundle pong again (warms backfill metrics) ---" +# A pong plus sidecar rows alone need not record anything: live mirroring +# writes database rows without metrics. The second mount sees probe 3's +# sidecar, so mount backfill records its row even when everything dedups. +timeout 60 dsh --profile lossless-e2e headless --json "$TASK" > "$OUT_DIR/e2e-lossless2.json" +rc=$? +echo "probe4 EXIT=$rc" +[ "$rc" -eq 0 ] || { echo "probe4: FAIL: dsh exited $rc" >&2; exit 1; } +check_pong "probe4" "$OUT_DIR/e2e-lossless2.json" + +echo "--- metrics in $SIDECAR_DIR ---" +if [ ! -s "$SIDECAR_DIR/metrics.jsonl" ]; then + echo "probe4: FAIL: no metrics in $SIDECAR_DIR/metrics.jsonl" >&2 + exit 1 +fi +echo "probe4: metrics OK ($SIDECAR_DIR/metrics.jsonl)" + +echo "ALL PROBES PASSED" diff --git a/packages/deepseek/test/e2e/run-e2e.test.ts b/packages/deepseek/test/e2e/run-e2e.test.ts new file mode 100644 index 0000000..2f9d3ff --- /dev/null +++ b/packages/deepseek/test/e2e/run-e2e.test.ts @@ -0,0 +1,21 @@ +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { describe, expect, it } from "vite-plus/test"; + +const dir = dirname(fileURLToPath(import.meta.url)); +const script = readFileSync(join(dir, "run-e2e.sh"), "utf8"); + +describe("run-e2e.sh", () => { + it("scopes sidecars to the scratch DSH_HOME", () => { + expect(script).toContain('export DSH_HOME="/tmp/e2e-dsh"'); + expect(script).toContain('SIDECAR_DIR="$DSH_HOME/lossless"'); + expect(script).not.toContain("/tmp/.lossless"); + }); + + it("asserts the probed sidecars in the scratch dir", () => { + expect(script).toContain('sidecars=("$SIDECAR_DIR"/*.sqlite)'); + expect(script).toContain("no sidecar sqlite files in $SIDECAR_DIR"); + }); +}); diff --git a/packages/deepseek/test/engine.test.ts b/packages/deepseek/test/engine.test.ts new file mode 100644 index 0000000..8e262d9 --- /dev/null +++ b/packages/deepseek/test/engine.test.ts @@ -0,0 +1,714 @@ +import { Context } from "@deepseek-ai/cordis"; +import { ManualCompactionError } from "@deepseek-ai/dsh-compaction"; +import { createMessage, createSystemMessage, createUserMessage } from "@deepseek-ai/dsh-llm"; +import type { Message } from "@deepseek-ai/dsh-llm"; +import SessionStore, { Session, SessionId } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import { LcmStore } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { + LosslessSummarizerEngine, + SpanAlreadyMemoryError, + checkpointMemory, +} from "../src/engine.ts"; +import type { LosslessEngineDeps } from "../src/engine.ts"; +import { MessageIndex, mirrorEvent } from "../src/ingest.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; +const SUMMARY = "Checkpoint: engine work started and the config was read."; + +function buildSession(text: string): Session { + const session = Session.create(SessionId("engine-1")); + appendConversation(session, ROUTE, text); + // The engine writes automatic compaction events into an open turn. + session.append("turn/start", { turn: 3 }); + return session; +} + +function mountMeter(ctx: Context): void { + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +const identity = (text: string): string => text; + +describe("LosslessSummarizerEngine", () => { + function engineFor( + ctx: Context, + store: LcmStore, + index: MessageIndex, + extra?: Pick, + ): LosslessSummarizerEngine { + return new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity, ...extra }, + ); + } + + it("compacts a region through the ladder and stores leaves in the sidecar", async () => { + const session = buildSession("working notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity }, + ); + const nodes = session.surface.nodes; + const before = session.deriveMessages().length; + const result = await engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent); + expect(result.summary).toHaveLength(1); + expect(result.summary[0]?.type).toBe("text"); + const summaryEvent = session + .snapshotEvents() + .find((event) => event.type === "compaction/summary"); + expect(summaryEvent?.data.provider).toBe(ROUTE.provider); + expect(summaryEvent?.data.model).toBe(ROUTE.model); + expect(summaryEvent?.data.usage?.outputTokens).toBeGreaterThan(0); + expect(result.shadowedSeqs.length).toBeGreaterThan(0); + const derived = session.deriveMessages(); + expect(derived.length).toBeLessThan(before); + expect( + derived.some((message) => + message.content.some((block) => block.type === "text" && block.text.includes(SUMMARY)), + ), + ).toBe(true); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + expect(store.allSummaries().length).toBeGreaterThan(0); + expect(store.allMessages().some((row) => row.text.includes(SUMMARY))).toBe(true); + const checkpoint = derived.find((message) => + message.content.some((block) => block.type === "text" && block.text.includes(SUMMARY)), + ); + const refs = checkpoint + ? checkpoint.content.flatMap((block) => + block.type === "text" ? [...block.text.matchAll(/\[lcm:summary #(\d+)\]/g)] : [], + ) + : []; + expect(refs.length).toBeGreaterThan(0); + for (const ref of refs) { + expect(store.getSummary(Number(ref[1]))).toBeDefined(); + } + }); + + it("resolves span messages without the live index after a restart", async () => { + // Live every headless round is a new process: the in-memory map is gone + // but the sidecar and the live log remain. The span must re-derive from + // the live session instead of failing closed. + const session = buildSession("resume notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const liveIndex = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, liveIndex); + const coldIndex = new MessageIndex(); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => coldIndex, redact: identity }, + ); + const nodes = session.surface.nodes; + const result = await engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent); + expect(result.summary).toHaveLength(1); + expect(result.shadowedSeqs.length).toBeGreaterThan(0); + const mapped = session.deriveMessages().filter((message) => coldIndex.entryId(message.id)); + expect(mapped.length).toBeGreaterThan(0); + expect(coldIndex.entryId("never-derived")).toBeUndefined(); + }); + + it("carries the previous checkpoint into the next ladder", async () => { + const session = buildSession("carryover notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const FIRST = "First checkpoint alpha."; + const SECOND = "Second checkpoint beta."; + const { ctx } = setupContext(ROUTE, [ + textChunks(FIRST), + textChunks(SECOND), + textChunks(SECOND), + ]); + mountMeter(ctx); + const memory = checkpointMemory(); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, memory, redact: identity, summaryTokens: 40 }, + ); + const nodes = session.surface.nodes; + const first = await engine.compactRegion(nodes[1]!, nodes[2]!, agent); + const firstBlock = first.summary[0]; + if (!firstBlock || firstBlock.type !== "text") throw new Error("first summary is not text"); + expect(firstBlock.text).toContain(FIRST); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const afterFirst = session.surface.nodes; + const second = await engine.compactRegion( + afterFirst[1]!, + afterFirst[afterFirst.length - 1]!, + agent, + ); + const secondBlock = second.summary[0]; + if (!secondBlock || secondBlock.type !== "text") throw new Error("second summary is not text"); + expect(secondBlock.text).toContain(SECOND); + expect(secondBlock.text.startsWith(firstBlock.text)).toBe(true); + }); + + it("rejects an already-covered span as memory instead of failing the summary", async () => { + const session = buildSession("covered live ".repeat(30)); + // A third turn inside the open turn 3: rows the later checkpoint lands + // after, which is what makes them covered-but-live intruders. + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `${"covered live ".repeat(30)} third request` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 3, step: 1 }); + session.append("request/header", { + header: { config: { provider: ROUTE.provider, model: ROUTE.model } }, + reason: "initial", + }); + session.append( + "assistant/message", + { + stream: [], + turn: 3, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: `${"covered live ".repeat(30)} third response` }], + source: { kind: "model", provider: ROUTE.provider, model: ROUTE.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 3, step: 1 }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks("First span."), + textChunks("Second span."), + textChunks("Third span."), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { + store: () => store, + index: () => index, + memory: checkpointMemory(), + redact: identity, + summaryTokens: 40, + }, + ); + const nodes = session.surface.nodes; + await engine.compactRegion(nodes[1]!, nodes[2]!, agent); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const afterFirst = session.surface.nodes; + await engine.compactRegion(afterFirst[1]!, afterFirst[3]!, agent); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const covered = session.surface.nodes; + const before = session.deriveMessages().length; + await expect( + engine.compactRegion(covered[covered.length - 2]!, covered[covered.length - 1]!, agent), + ).rejects.toThrow(SpanAlreadyMemoryError); + expect(session.deriveMessages().length).toBe(before); + }); + + it("fails closed when the span is not mirrored", async () => { + const session = buildSession("unmirrored notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => new MessageIndex(), redact: identity }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "sidecar holds no row", + ); + }); + + it("records a failed pass when span mapping throws", async () => { + const session = buildSession("unmirrored notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + record: (row) => rows.push(row), + }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "sidecar holds no row", + ); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + event: "lcm", + kind: "pass", + session: String(session.id), + stage: "blocking", + outcome: "failed", + error: expect.stringContaining("sidecar holds no row"), + }); + expect(typeof rows[0]?.durationMs).toBe("number"); + }); + + it("throws cleanly without a recorder when span mapping throws", async () => { + const session = buildSession("unmirrored notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => new MessageIndex(), redact: identity }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "sidecar holds no row", + ); + }); + + it("heals a partial mirror through one backfill catch-up", async () => { + const session = buildSession("catchup notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + const events = session.snapshotEvents(); + for (const event of events.slice(0, 3)) mirrorEvent(session, event, store, index); + expect(store.allMessages().length).toBeGreaterThan(0); + expect(store.allMessages().length).toBeLessThan(session.deriveMessages().length); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const liveEvents = session.snapshotEvents(); + (ctx as unknown as Record).sessionQuery = { + observeSession: async () => ({ events: liveEvents }), + }; + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity, record: (row) => rows.push(row) }, + ); + const nodes = session.surface.nodes; + const expected = session.deriveMessages().length; + const result = await engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent); + expect(result.summary).toHaveLength(1); + expect(result.shadowedSeqs.length).toBeGreaterThan(0); + expect(rows.some((row) => row.kind === "backfill")).toBe(true); + expect(store.allMessages().length).toBe(expected); + }); + + it("orders a backfilled span by seq when rows landed out of order", async () => { + const session = buildSession("backfill order ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + const events = session.snapshotEvents(); + // Two mirror batches shaped like a real catch-up: the first batch holds + // a prefix plus newer live rows, the second backfills the gap in order, + // so gap rows land on newer rowids than the newer rows they precede. + const gap = events.slice(4, 7); + const rest = events.filter((event) => !gap.includes(event)); + for (const event of rest) mirrorEvent(session, event, store, index); + for (const event of gap) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity, record: (row) => rows.push(row) }, + ); + const pre = [...session.surface.nodes]; + await engine.compactRegion(pre[1]!, pre[pre.length - 1]!, agent); + const seqs = pre.slice(1).map(Number); + const sid = String(session.id); + const pass = rows.find((row) => row.kind === "pass"); + expect(pass?.span).toEqual([`${sid}:${Math.min(...seqs)}`, `${sid}:${Math.max(...seqs)}`]); + }); + + it("scans the live log once for consecutive index misses", () => { + const session = buildSession("scan once ".repeat(20)); + const store = new LcmStore(":memory:"); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store); + const derived = session.deriveMessages(); + const real = derived.find((message) => message.role !== "system"); + if (!real) throw new Error("conversation is missing a non-system message"); + const ghost = { id: "ghost-1", role: "user" } as unknown as Message; + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const engine = engineFor(ctx, store, new MessageIndex()); + const spanOf = ( + engine as unknown as { + spanOf( + input: { messages: readonly Message[] }, + store: LcmStore, + index: MessageIndex, + session: Session, + ): unknown[]; + } + ).spanOf.bind(engine); + let scans = 0; + const snapshotEvents = session.snapshotEvents.bind(session); + session.snapshotEvents = (() => { + scans += 1; + return snapshotEvents(); + }) as typeof session.snapshotEvents; + const index = new MessageIndex(); + let failure: unknown; + try { + spanOf({ messages: [real, ghost] }, store, index, session); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(Error); + expect((failure as Error).message).toBe( + "lossless compaction: message ghost-1 has no sidecar row; the span is not fully mirrored", + ); + expect(scans).toBe(1); + }); + + it("rethrows the original span error when no history source exists", async () => { + const session = buildSession("gap notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + const events = session.snapshotEvents(); + for (const event of events.slice(0, 3)) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity, record: (row) => rows.push(row) }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "sidecar holds no row", + ); + expect(rows.some((row) => row.kind === "backfill")).toBe(false); + }); + + it("leaves non-span failures alone without a backfill call", async () => { + const session = buildSession("hi"); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + let backfillCalls = 0; + const liveEvents = session.snapshotEvents(); + (ctx as unknown as Record).sessionQuery = { + observeSession: async () => { + backfillCalls += 1; + return { events: liveEvents }; + }, + }; + const rows: Record[] = []; + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity, record: (row) => rows.push(row) }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "not smaller than the shadowed content", + ); + expect(backfillCalls).toBe(0); + expect(rows.some((row) => row.kind === "backfill")).toBe(false); + }); + + describe("compactNow", () => { + // The durability flush needs a live session, so these build on + // ctx.sessions instead of the detached buildSession helper. + function idleSession(ctx: Context, id: string, text: string): Session { + const session = ctx.sessions.create(SessionId(id)); + appendConversation(session, ROUTE, text); + session.append("turn/start", { turn: 3 }); + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + return session; + } + + it("commits the manual path through the DAG bracket", async () => { + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + void new SessionStore(ctx); + const session = idleSession(ctx, "compact-now-1", "manual notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const before = session.deriveMessages().length; + const result = await engineFor(ctx, store, index).compactNow( + fakeAgent(session, ROUTE), + new AbortController().signal, + ); + expect(result).not.toBeNull(); + expect(result?.shadowedSeqs.length).toBeGreaterThan(0); + expect(result?.summarySeqs).toHaveLength(1); + expect(result?.leafSeqs).toHaveLength(0); + expect(result?.leafSummaryIds.length ?? 0).toBeGreaterThan(0); + expect(result?.summarySeq).toBe(result?.summarySeqs[0]); + expect(session.deriveMessages().length).toBeLessThan(before); + const events = session.snapshotEvents(); + expect(events.filter((event) => event.type === "compaction/start")).toHaveLength(1); + const ends = events.filter((event) => event.type === "compaction/end"); + expect(ends).toHaveLength(1); + expect(ends[0]?.type).toBe("compaction/end"); + }); + + it("returns null when no safe range exists", async () => { + const session = Session.create(SessionId("compact-now-empty")); + session.append( + "system/message", + { turn: 1, step: 1, message: createSystemMessage("You are a test.") }, + { surfaceOp: "append" }, + ); + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + void new SessionStore(ctx); + const result = await engineFor(ctx, new LcmStore(":memory:"), new MessageIndex()).compactNow( + fakeAgent(session, ROUTE), + new AbortController().signal, + ); + expect(result).toBeNull(); + }); + + it("reports busy when the session still has an open turn", async () => { + const session = buildSession("open turn ".repeat(20)); + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + void new SessionStore(ctx); + const failure = await engineFor(ctx, new LcmStore(":memory:"), new MessageIndex()) + .compactNow(fakeAgent(session, ROUTE), new AbortController().signal) + .catch((error: unknown) => error); + expect(failure).toBeInstanceOf(ManualCompactionError); + expect((failure as ManualCompactionError).code).toBe("busy"); + }); + + it("reports busy when the agent refuses maintenance", async () => { + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + void new SessionStore(ctx); + const session = idleSession(ctx, "compact-now-4", "refused notes ".repeat(20)); + const agent = { + ...fakeAgent(session, ROUTE), + runMaintenance(): Promise { + throw new Error("waking work queued"); + }, + }; + // A synchronous admission refusal throws before any promise exists, + // so .catch cannot see it; take it with try/catch instead. + let failure: unknown; + try { + await engineFor(ctx, new LcmStore(":memory:"), new MessageIndex()).compactNow( + agent, + new AbortController().signal, + ); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(ManualCompactionError); + expect((failure as ManualCompactionError).code).toBe("busy"); + }); + + describe("checkpointMemory", () => { + it("hydrates the last checkpoint from the sidecar on a fresh instance", async () => { + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + void new SessionStore(ctx); + const sessionId = "compact-memory-1"; + const session = idleSession(ctx, sessionId, "hydrated notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const first = checkpointMemory(() => store); + const engine = engineFor(ctx, store, index, { memory: first }); + const result = await engine.compactNow( + fakeAgent(session, ROUTE), + new AbortController().signal, + ); + expect(result).not.toBeNull(); + const noted = first.previous(sessionId); + expect(noted).toBeDefined(); + // A fresh instance over the same sidecar rebuilds continuity: the + // narrative plus refs that all resolve. (A single-leaf run replaces + // its row's text on condensation, so the rebuilt text is the stored + // form, not byte-identical to the noted form.) + const resumed = checkpointMemory(() => store); + const rebuilt = resumed.previous(sessionId); + expect(rebuilt).toBeDefined(); + expect(rebuilt).toContain("Checkpoint: engine work started"); + const ids = [...rebuilt!.matchAll(/\[lcm:summary #(\d+)\]/g)].map((match) => + Number(match[1]), + ); + expect(ids.length).toBeGreaterThan(0); + for (const id of ids) expect(store.getSummary(id)).toBeDefined(); + resumed.note(sessionId, "fresh checkpoint"); + expect(resumed.previous(sessionId)).toBe("fresh checkpoint"); + }); + + it("stays absent without a store or without a condensed summary", () => { + expect(checkpointMemory().previous("ghost")).toBeUndefined(); + expect(checkpointMemory(() => new LcmStore(":memory:")).previous("ghost")).toBeUndefined(); + }); + }); + + it("reports cancelled when the maintenance signal aborts", async () => { + // No scripts: the first ladder call fails past the aborted signal. + const { ctx } = setupContext(ROUTE, []); + mountMeter(ctx); + void new SessionStore(ctx); + const session = idleSession(ctx, "compact-now-5", "cancelled notes ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const controller = new AbortController(); + controller.abort(new Error("waking")); + const agent = { + ...fakeAgent(session, ROUTE), + runMaintenance(task: (signal: AbortSignal) => Promise): Promise { + return task(controller.signal); + }, + }; + const failure = await engineFor(ctx, store, index) + .compactNow(agent, new AbortController().signal) + .catch((error: unknown) => error); + expect(failure).toBeInstanceOf(ManualCompactionError); + expect((failure as ManualCompactionError).code).toBe("cancelled"); + }); + }); + + describe("compactRegion with bracketAuto", () => { + it("commits the auto path through the bracket on an open turn", async () => { + const session = buildSession("auto bracket ".repeat(20)); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = engineFor(ctx, store, index, { bracketAuto: true }); + const nodes = session.surface.nodes; + const before = session.deriveMessages().length; + const result = await engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent); + expect(result.shadowedSeqs.length).toBeGreaterThan(0); + expect(result.summarySeq).toBeDefined(); + expect(session.deriveMessages().length).toBeLessThan(before); + const summaries = session + .snapshotEvents() + .filter((event) => event.type === "compaction/summary"); + expect(summaries.length).toBe(1); + }); + + it("lets a ladder failure through unwrapped on the auto path", async () => { + const session = buildSession("unmirrored auto ".repeat(20)); + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = engineFor(ctx, new LcmStore(":memory:"), new MessageIndex(), { + bracketAuto: true, + }); + const nodes = session.surface.nodes; + const failure = await engine + .compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent) + .catch((error: unknown) => error); + expect(failure).toBeInstanceOf(Error); + expect(failure).not.toBeInstanceOf(ManualCompactionError); + expect(String((failure as Error).message)).toMatch(/sidecar holds no row/); + }); + }); + + it("lets the engine shrink check fire on a span the ladder cannot reduce", async () => { + const session = buildSession("hi"); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + mountMeter(ctx); + const agent = fakeAgent(session, ROUTE); + const engine = new LosslessSummarizerEngine( + ctx, + { auto: false }, + { store: () => store, index: () => index, redact: identity }, + ); + const nodes = session.surface.nodes; + await expect(engine.compactRegion(nodes[1]!, nodes[nodes.length - 1]!, agent)).rejects.toThrow( + "not smaller than the shadowed content", + ); + }); +}); diff --git a/packages/deepseek/test/expand-query.test.ts b/packages/deepseek/test/expand-query.test.ts new file mode 100644 index 0000000..be09511 --- /dev/null +++ b/packages/deepseek/test/expand-query.test.ts @@ -0,0 +1,385 @@ +import { Context } from "@deepseek-ai/cordis"; +import { ToolCallId, type StreamChunk } from "@deepseek-ai/dsh-llm"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import ToolRuntime, { type ToolRunContext } from "@deepseek-ai/dsh-tools"; +import { LcmStore, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import type { DshRoute } from "../src/model-host.ts"; +import { dshRef } from "../src/model-host.ts"; +import { dshReader } from "../src/reader.ts"; +import { expandQueryAnswer, registerExpandQueryTool } from "../src/tools.ts"; +import { errorChunks, setupContext, textChunks, toolCallChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function seedStore(): LcmStore { + const store = new LcmStore(":memory:"); + ingestEntries( + store, + Array.from({ length: 8 }, (_, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text: `message ${i}: deciding the ${i % 2 === 0 ? "compaction engine" : "recall pipeline"} rollout ${"detail ".repeat(20)}`, + timestamp: i, + })), + ); + return store; +} + +function seedStoreWithSummary(): { store: LcmStore; summaryId: number } { + const store = seedStore(); + const messageIds = store.messagesInSpan("e00", "e01").map((message) => message.id); + const summaryId = store.insertSummary({ + kind: "leaf", + text: "leaf over the rollout decision", + tokens: 4, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds, + }).id; + return { store, summaryId }; +} + +const identity = (text: string): string => text; + +/** One step carrying both findings text and a tool call, so the run can + * exhaust its steps with a partial answer in hand. */ +function textAndCallChunks( + text: string, + callId: string, + name: string, + args: unknown, +): StreamChunk[] { + const id = ToolCallId(callId); + const json = JSON.stringify(args); + return [ + { type: "block-start", index: 0, blockType: "text" }, + { type: "text-delta", index: 0, text }, + { type: "block-end", index: 0, block: { type: "text", text } }, + { type: "block-start", index: 1, blockType: "tool-call" }, + { type: "tool-call-delta", index: 1, id, name, argumentsDelta: json }, + { type: "block-end", index: 1, block: { type: "tool-call", id, name, arguments: json } }, + { type: "usage", usage: { inputTokens: 10, outputTokens: 5 } }, + { type: "finish", reason: { kind: "tool-calls" } }, + ]; +} + +describe("lcm_expand_query", () => { + it("registers the fourth tool whose schema names every param", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const store = seedStore(); + const dispose = registerExpandQueryTool(ctx, { store: () => store, redact: identity }); + const schemas = ctx.tools.schemas(); + expect(schemas.map((schema) => schema.name)).toContain("lcm_expand_query"); + const params = schemas.find((schema) => schema.name === "lcm_expand_query"); + expect(Object.keys(params?.parameters.properties ?? {})).toEqual( + expect.arrayContaining([ + "query", + "prompt", + "summary_id", + "entry_id", + "session", + "max_tokens", + ]), + ); + expect(params?.parameters.required).toContain("query"); + dispose(); + expect(ctx.tools.schemas().map((schema) => schema.name)).not.toContain("lcm_expand_query"); + }); + + it("answers from aged history and cites the entry id", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "recall pipeline rollout" }), + textChunks("The recall pipeline rollout was approved for next week [e01]."), + ]); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { + query: "recall pipeline rollout", + prompt: "What did we decide about the recall pipeline rollout?", + }, + (row) => rows.push(row), + ); + expect(answer).toContain(""); + expect(answer).toContain("[e01]"); + expect(answer).toContain("Read stored history in 2 step(s)"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ event: "lcm", kind: "retrieval", stop: "answered" }); + expect(rows[0]?.["steps"]).toBe(2); + }); + + it("reads a named summary and cites the summary id", async () => { + const { store, summaryId } = seedStoreWithSummary(); + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_expand", { id: summaryId }), + textChunks(`Summary #${summaryId} covers the compaction engine decision [e00].`), + ]); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { + query: "compaction engine decision", + prompt: "What did summary cover?", + summary_id: summaryId, + }, + (row) => rows.push(row), + ); + expect(answer).toContain(`#${summaryId}`); + expect(answer).toContain(""); + expect(rows[0]).toMatchObject({ + kind: "retrieval", + stop: "answered", + scope: "summary", + summaryId, + }); + }); + + it("reads a named session through the same store lookup", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "recall pipeline rollout" }), + textChunks("The recall pipeline rollout was approved for next week [e01]."), + ]); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", session: "other-session" }, + (row) => rows.push(row), + ); + expect(answer).toContain("[e01]"); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "answered" }); + }); + + it("refuses a call that names both a summary and an entry", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", summary_id: 1, entry_id: "e01" }, + (row) => rows.push(row), + ); + expect(answer).toBe("Nothing was read: pass summary_id or entry_id, not both."); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "refused" }); + }); + + it("refuses an entry id the store does not hold", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", entry_id: "missing" }, + (row) => rows.push(row), + ); + expect(answer).toBe("Nothing was read: no stored message with entry id missing."); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "refused" }); + }); + + it("answers the no-history text when no store resolves", async () => { + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + undefined, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + undefined, + { query: "recall pipeline rollout" }, + (row) => rows.push(row), + ); + expect(answer).toContain("no stored history"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ event: "lcm", kind: "retrieval", stop: "refused" }); + }); + + it("reports a reader failure with its reason and billed counters", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "recall pipeline rollout" }), + errorChunks(), + ]); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout" }, + (row) => rows.push(row), + ); + expect(answer).toBe("Retrieval failed: AUTH: bad key"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + kind: "retrieval", + stop: "failed", + reason: "AUTH: bad key", + steps: 2, + input: 10, + output: 5, + }); + }); + + it("stops with partial findings when the steps run out", async () => { + const store = seedStore(); + const scripts = Array.from({ length: 6 }, (_, i) => + textAndCallChunks(`partial note ${i} [e01]`, `call-${i}`, "lcm_grep", { + query: "recall pipeline rollout", + }), + ); + const { ctx } = setupContext(ROUTE, scripts); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout" }, + (row) => rows.push(row), + ); + expect(answer).toContain("stopped: steps"); + expect(answer).toContain(""); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "stopped", steps: 6 }); + }); + + it("stops empty when no step left an answer", async () => { + const store = seedStore(); + const scripts = Array.from({ length: 6 }, (_, i) => + toolCallChunks(`call-${i}`, "lcm_grep", { query: "recall pipeline rollout" }), + ); + const { ctx } = setupContext(ROUTE, scripts); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout" }, + (row) => rows.push(row), + ); + expect(answer).toContain("Retrieval stopped (steps) after 6 step(s) without an answer"); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "stopped", reason: "steps" }); + }); + + it("refuses an empty entry id", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", entry_id: "" }, + (row) => rows.push(row), + ); + expect(answer).toBe("Nothing was read: entry_id is empty."); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "refused" }); + }); + + it("refuses a summary id the store does not hold", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", summary_id: 4242 }, + (row) => rows.push(row), + ); + expect(answer).toBe("Nothing was read: no summary with id 4242."); + expect(rows[0]).toMatchObject({ + kind: "retrieval", + stop: "refused", + reason: "no summary with id 4242", + }); + }); + + it("refuses an empty session string", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", session: "" }, + (row) => rows.push(row), + ); + expect(answer).toBe("Nothing was read: session must be a non-empty session id."); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "refused" }); + }); + + it("refuses a past session no sidecar holds", async () => { + const store = seedStore(); + const { ctx } = setupContext(ROUTE, []); + const rows: Record[] = []; + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { query: "recall pipeline rollout", session: "ghost" }, + (row) => rows.push(row), + { list: () => [] }, + ); + expect(answer).toBe("Nothing was read: no stored session with id ghost."); + expect(rows[0]).toMatchObject({ kind: "retrieval", stop: "refused" }); + }); + + it("fails the tool run when no provider backs the reader", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const store = seedStore(); + const rows: Record[] = []; + const dispose = registerExpandQueryTool(ctx, { + store: () => store, + redact: identity, + record: (row) => rows.push(row), + }); + try { + const tool = ctx.tools.get("lcm_expand_query"); + expect(tool).toBeDefined(); + const exec = { + agent: undefined, + signal: new AbortController().signal, + } as unknown as ToolRunContext; + const answer = await tool?.execute({ query: "recall pipeline rollout" }, exec); + expect(answer).toBe( + "lossless retrieval: no provider/model available for the reader: route one request or set both agent options fields.", + ); + expect(rows[0]).toMatchObject({ + kind: "retrieval", + stop: "failed", + reason: "no provider/model available for the reader", + steps: 0, + }); + } finally { + dispose(); + } + }); +}); diff --git a/packages/deepseek/test/ingest.test.ts b/packages/deepseek/test/ingest.test.ts new file mode 100644 index 0000000..b1bd192 --- /dev/null +++ b/packages/deepseek/test/ingest.test.ts @@ -0,0 +1,470 @@ +import { + ToolCallId, + createMessage, + createSystemMessage, + createToolResultMessage, + createUserMessage, +} from "@deepseek-ai/dsh-llm"; +import type { FileBlock, ImageBlock } from "@deepseek-ai/dsh-llm"; +import { Session, SessionId } from "@deepseek-ai/dsh-session"; +import { LcmStore } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { MessageIndex, entryFor, mirrorEvent } from "../src/ingest.ts"; + +const MODEL = "test-model"; + +function buildSession(): Session { + const session = Session.create(SessionId("mirror-1")); + session.append("turn/start", { turn: 1 }); + session.append( + "system/message", + { turn: 1, step: 1, message: createSystemMessage("You are a test.") }, + { surfaceOp: "append" }, + ); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "start the compaction engine work" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 1, step: 1 }); + session.append("request/header", { + header: { config: { provider: MODEL, model: MODEL } }, + reason: "initial", + }); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: "engine work started" }], + source: { kind: "model", provider: MODEL, model: MODEL }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 1, step: 1 }); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + session.append("turn/start", { turn: 2 }); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "read the engine config" }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 2, step: 1 }); + const callId = ToolCallId("call-1"); + session.append( + "assistant/message", + { + stream: [], + turn: 2, + step: 1, + message: createMessage({ + role: "assistant", + content: [ + { type: "text", text: "reading now" }, + { type: "tool-call", id: callId, name: "read", arguments: '{"path":"engine.conf"}' }, + ], + source: { kind: "model", provider: MODEL, model: MODEL }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("tool/call", { + turn: 2, + step: 1, + callId, + name: "read", + arguments: '{"path":"engine.conf"}', + }); + session.append( + "tool/result", + { + turn: 2, + step: 1, + message: createToolResultMessage({ + callId, + content: [{ type: "text", text: "config contents here" }], + isError: false, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 2, step: 1 }); + session.append("turn/end", { turn: 2, reason: { kind: "completed" } }); + return session; +} + +describe("ingest mirror", () => { + it("mirrors every derived message and nothing else, idempotently", () => { + const session = buildSession(); + const store = new LcmStore(":memory:"); + // The snapshot walk is test-only: production code mirrors the live feed. + const events = session.snapshotEvents(); + let inserted = 0; + for (const event of events) inserted += mirrorEvent(session, event, store).inserted; + const derived = session.deriveMessages(); + expect(inserted).toBe(derived.length); + expect(inserted).toBe(6); + const stored = new Map(store.allMessages().map((row) => [row.entryId, row])); + expect(stored.size).toBe(6); + for (const message of derived) { + for (const block of message.content) { + if (block.type !== "text" || block.text.trim().length === 0) continue; + const kept = [...stored.values()].some((row) => row.text.includes(block.text)); + expect(kept).toBe(true); + } + } + const turnStart = events.find((event) => event.type === "turn/start"); + expect(turnStart).toBeDefined(); + if (turnStart) expect(entryFor(session, turnStart)).toBeUndefined(); + let second = 0; + let skipped = 0; + for (const event of events) { + const stats = mirrorEvent(session, event, store); + second += stats.inserted; + skipped += stats.skipped; + } + expect(second).toBe(0); + expect(skipped).toBe(events.length); + }); + + it("keeps shadowed entries when a checkpoint replaces surface nodes", () => { + const session = buildSession(); + const store = new LcmStore(":memory:"); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store); + const before = session.deriveMessages().length; + const firstUser = session.snapshotEvents().find((event) => event.type === "user/message"); + const firstAssistant = session + .snapshotEvents() + .find((event) => event.type === "assistant/message"); + expect(firstUser).toBeDefined(); + expect(firstAssistant).toBeDefined(); + if (!firstUser || !firstAssistant) throw new Error("conversation is missing its first turn"); + const checkpoint = session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: "checkpoint: engine work started, config read next" }], + source: { kind: "user" }, + }), + { + surfaceOp: { op: "replace", startSeq: firstUser.seq, endSeq: firstAssistant.seq }, + sourceEventSeqs: [firstUser.seq, firstAssistant.seq], + }, + ); + const stats = mirrorEvent(session, checkpoint, store); + expect(stats.inserted).toBe(1); + expect(session.deriveMessages().length).toBe(before - 1); + expect(store.allMessages()).toHaveLength(before + 1); + const texts = store.allMessages().map((row) => row.text); + expect(texts.some((text) => text.includes("start the compaction engine work"))).toBe(true); + expect(texts.some((text) => text.includes("engine work started"))).toBe(true); + expect(texts.some((text) => text.includes("checkpoint: engine work started"))).toBe(true); + }); + + it("mirrors an image-only message as one descriptor row", () => { + const session = Session.create(SessionId("mirror-image")); + session.append("turn/start", { turn: 1 }); + const imageEvent = session.append( + "user/message", + createUserMessage({ + content: [ + { + type: "image", + attachment: { + attachmentId: + "sha256:test-image-1" as unknown as ImageBlock["attachment"]["attachmentId"], + mediaType: "image/png", + bytes: 1234, + width: 800, + height: 600, + name: "photo.png", + }, + }, + ], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + let inserted = 0; + for (const event of session.snapshotEvents()) + inserted += mirrorEvent(session, event, store).inserted; + expect(inserted).toBe(1); + const rows = store.allMessages(); + expect(rows).toHaveLength(1); + const row = rows[0]; + expect(row).toBeDefined(); + if (!row) throw new Error("expected one descriptor row"); + expect(row.entryId).toBe(`${session.id}:${imageEvent.seq}`); + expect(row.role).toBe("user"); + expect(row.timestamp).toBe(imageEvent.time); + expect(row.text).toContain("image"); + expect(row.text).toContain("photo.png"); + expect(row.text).toContain("descriptor only"); + }); + + it("mirrors a file-only message as one descriptor row", () => { + const session = Session.create(SessionId("mirror-file")); + session.append( + "user/message", + createUserMessage({ + content: [ + { + type: "file", + attachment: { + attachmentId: + "sha256:test-file-1" as unknown as FileBlock["attachment"]["attachmentId"], + name: "notes.pdf", + bytes: 5678, + }, + }, + ], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + const store = new LcmStore(":memory:"); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store); + const rows = store.allMessages(); + expect(rows).toHaveLength(1); + const row = rows[0]; + expect(row).toBeDefined(); + if (!row) throw new Error("expected one descriptor row"); + expect(row.text).toContain("file"); + expect(row.text).toContain("notes.pdf"); + }); + + it("maps a span containing a non-text message instead of throwing", () => { + const session = buildSession(); + session.append("turn/start", { turn: 3 }); + session.append( + "user/message", + createUserMessage({ + content: [ + { + type: "image", + attachment: { + attachmentId: + "sha256:test-image-1" as unknown as ImageBlock["attachment"]["attachmentId"], + mediaType: "image/png", + bytes: 1234, + width: 800, + height: 600, + name: "photo.png", + }, + }, + ], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const derived = session.deriveMessages(); + expect(derived.length).toBe(7); + for (const message of derived) { + const entryId = index.entryId(message.id); + expect(entryId).toBeDefined(); + if (entryId === undefined) throw new Error(`no sidecar entry for ${message.id}`); + expect(store.messageByEntryId(entryId)).toBeDefined(); + } + }); + + it("keeps no row for a message with no content at all", () => { + const session = Session.create(SessionId("mirror-empty")); + const event = session.append( + "user/message", + createUserMessage({ content: [], source: { kind: "user" } }), + { surfaceOp: "append" }, + ); + expect(entryFor(session, event)).toBeUndefined(); + const store = new LcmStore(":memory:"); + expect(mirrorEvent(session, event, store).inserted).toBe(0); + expect(store.allMessages()).toHaveLength(0); + }); + + it("keeps leading indentation, trailing spaces, and outer blank lines verbatim", () => { + const session = Session.create(SessionId("mirror-ws")); + const original = "\n\n indented line\nsecond line \n\n"; + const event = session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: original }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + const store = new LcmStore(":memory:"); + expect(mirrorEvent(session, event, store).inserted).toBe(1); + const rows = store.allMessages(); + expect(rows).toHaveLength(1); + expect(rows[0]?.text).toBe(original); + }); + + it("maps whitespace-only text through a descriptor row", () => { + const session = Session.create(SessionId("mirror-ws-keep")); + const original = " \n "; + const event = session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: original }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + expect(entryFor(session, event)?.text).toBe("[lcm:whitespace 6 chars]"); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + expect(mirrorEvent(session, event, store, index).inserted).toBe(1); + const rows = store.allMessages(); + expect(rows).toHaveLength(1); + expect(rows[0]?.text).toBe("[lcm:whitespace 6 chars]"); + for (const message of session.deriveMessages()) { + const entryId = index.entryId(message.id); + expect(entryId).toBeDefined(); + if (entryId === undefined) throw new Error(`no sidecar entry for ${message.id}`); + expect(store.messageByEntryId(entryId)).toBeDefined(); + } + }); + + it("mirrors a blank tool output as one descriptor row and maps its span", () => { + const session = Session.create(SessionId("mirror-blank-tool")); + session.append("turn/start", { turn: 1 }); + const callId = ToolCallId("call-blank"); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [ + { type: "tool-call", id: callId, name: "read", arguments: '{"path":"blank.txt"}' }, + ], + source: { kind: "model", provider: MODEL, model: MODEL }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("tool/call", { + turn: 1, + step: 1, + callId, + name: "read", + arguments: '{"path":"blank.txt"}', + }); + const blank = " \n "; + const resultEvent = session.append( + "tool/result", + { + turn: 1, + step: 1, + message: createToolResultMessage({ + callId, + content: [{ type: "text", text: blank }], + isError: false, + }), + }, + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const row = store.messageByEntryId(`${session.id}:${resultEvent.seq}`); + expect(row).toBeDefined(); + expect(row?.text).toBe("tool read\n[lcm:whitespace 6 chars]"); + for (const message of session.deriveMessages()) { + const entryId = index.entryId(message.id); + expect(entryId).toBeDefined(); + if (entryId === undefined) throw new Error(`no sidecar entry for ${message.id}`); + expect(store.messageByEntryId(entryId)).toBeDefined(); + } + }); + + it("keeps the tool name on a result even when the assistant mixes prose with the call", () => { + const session = Session.create(SessionId("mirror-mixed-tool")); + session.append("turn/start", { turn: 1 }); + const callId = ToolCallId("call-mixed"); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [ + { type: "text", text: "reading now" }, + { type: "tool-call", id: callId, name: "read", arguments: '{"path":"a.txt"}' }, + ], + source: { kind: "model", provider: MODEL, model: MODEL }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("tool/call", { + turn: 1, + step: 1, + callId, + name: "read", + arguments: '{"path":"a.txt"}', + }); + const resultEvent = session.append( + "tool/result", + { + turn: 1, + step: 1, + message: createToolResultMessage({ + callId, + content: [{ type: "text", text: "file body here" }], + isError: false, + }), + }, + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + const store = new LcmStore(":memory:"); + const index = new MessageIndex(); + try { + for (const event of session.snapshotEvents()) mirrorEvent(session, event, store, index); + const assistantRow = store.allMessages().find((row) => row.role === "assistant"); + expect(assistantRow?.text).toContain("reading now"); + expect(assistantRow?.text).toContain("tool-call: read"); + const resultRow = store.messageByEntryId(`${session.id}:${resultEvent.seq}`); + expect(resultRow?.role).toBe("toolResult"); + expect(resultRow?.text).toBe("tool read\nfile body here"); + } finally { + store.close(); + } + }); + + it("skips a message with an unknown harness role instead of throwing", () => { + const session = { + id: "mirror-unknown-role", + deriveEventMessage: () => ({ + id: "m-unknown", + role: "harness-new-role", + content: [{ type: "text", text: "future role text" }], + }), + } as unknown as Session; + const event = { seq: 1 } as unknown as Parameters[1]; + expect(entryFor(session, event)).toBeUndefined(); + const store = new LcmStore(":memory:"); + expect(mirrorEvent(session, event, store).inserted).toBe(0); + }); +}); diff --git a/packages/deepseek/test/lifecycle-mutating.test.ts b/packages/deepseek/test/lifecycle-mutating.test.ts new file mode 100644 index 0000000..e980bb4 --- /dev/null +++ b/packages/deepseek/test/lifecycle-mutating.test.ts @@ -0,0 +1,831 @@ +import { mkdtempSync, existsSync, readdirSync, utimesSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { DatabaseSync } from "node:sqlite"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import SessionStore, { SessionId } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { + LcmStore, + TRUNCATED_SUMMARY_PREFIX, + ingestEntries, + sanitizeSidecarId, + storeGeneration, + SCHEMA_VERSION, +} from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { lcmCommandDefinition, type LcmCommandDeps } from "../src/command.ts"; +import { healInFlightSessions } from "../src/lifecycle.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { applyLossless } from "../src/plugin.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-lcm-mut-")); +} + +async function mountBase(ctx: Context): Promise { + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function depsFor(dir: string, stores: Map): LcmCommandDeps { + return { + stores: () => stores, + storeFor: (sessionId) => { + let store = stores.get(sessionId); + if (!store) { + store = new LcmStore(join(dir, `${sanitizeSidecarId(sessionId)}.sqlite`)); + stores.set(sessionId, store); + } + return store; + }, + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sanitizeSidecarId(sessionId)}.sqlite`), + }; +} + +function invoke( + ctx: Context, + deps: LcmCommandDeps, + sessionId: string, + rawInput: string, +): Promise<{ kind: string; text?: string }> { + const session = ctx.sessions.get(SessionId(sessionId)); + if (!session) throw new Error(`no session ${sessionId}`); + const definition = lcmCommandDefinition(ctx, deps); + return definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId(`cmd-${rawInput.split(" ")[0] || "status"}`), + rawInput, + attachments: [], + }) as Promise<{ kind: string; text?: string }>; +} + +function seedMessages(store: LcmStore, texts: string[]): void { + ingestEntries( + store, + texts.map((text, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text, + timestamp: i, + })), + ); +} + +function ageFile(path: string, daysAgo: number): void { + const old = new Date(Date.now() - daysAgo * 86_400_000); + utimesSync(path, old, old); +} + +describe("lcm mutating verbs", () => { + it("export then import into a fresh store yields identical message counts", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("round-a"), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("round-b"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const source = deps.storeFor!("round-a"); + seedMessages(source, ["round trip one", "round trip two"]); + const target = join(dir, "round.jsonl"); + const exported = await invoke(ctx, deps, "round-a", `export ${target}`); + expect(exported.kind).toBe("success"); + expect(exported.text).toContain("2 messages"); + const first = await invoke(ctx, deps, "round-b", `import ${target}`); + expect(first.kind).toBe("success"); + expect(first.text).toContain("2 messages"); + expect(deps.storeFor!("round-b").allMessages().length).toBe(2); + const second = await invoke(ctx, deps, "round-b", `import ${target}`); + expect(second.kind).toBe("success"); + expect(second.text).toContain("already present"); + const missing = await invoke(ctx, deps, "round-b", "import"); + expect(missing.kind).toBe("error"); + }); + + it("import against a closed store returns an error instead of throwing", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("closed-src"), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("closed-dst"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + seedMessages(deps.storeFor!("closed-src"), ["closed store one", "closed store two"]); + const target = join(dir, "closed.jsonl"); + const exported = await invoke(ctx, deps, "closed-src", `export ${target}`); + expect(exported.kind).toBe("success"); + deps.storeFor!("closed-dst").close(); + let result: { kind: string; text?: string } | undefined; + let threw: unknown = null; + try { + result = await invoke(ctx, deps, "closed-dst", `import ${target}`); + } catch (error) { + threw = error; + } + expect(threw).toBeNull(); + expect(result?.kind).toBe("error"); + expect(result?.text).toContain("LCM import failed:"); + }); + + it("gc dry-run lists without deleting and apply deletes only stale sidecars", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "live.session:1"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const liveStem = sanitizeSidecarId(liveId); + const live = deps.storeFor!(liveId); + seedMessages(live, ["live history stays"]); + const stalePath = join(dir, "stale-old.sqlite"); + const staleDetached = new LcmStore(stalePath); + try { + seedMessages(staleDetached, ["stale history goes"]); + } finally { + staleDetached.close(); + } + ageFile(stalePath, 31); + ageFile(join(dir, `${liveStem}.sqlite`), 31); + const dry = await invoke(ctx, deps, liveId, "gc"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("dry-run"); + expect(dry.text).toContain("stale-old"); + expect(dry.text).not.toContain(liveStem); + expect(existsSync(stalePath)).toBe(true); + const applied = await invoke(ctx, deps, liveId, "gc --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("removed 1 of 1"); + expect(existsSync(stalePath)).toBe(false); + expect(existsSync(join(dir, `${liveStem}.sqlite`))).toBe(true); + expect(live.allMessages().length).toBe(1); + const again = await invoke(ctx, deps, liveId, "gc --apply"); + expect(again.kind).toBe("success"); + expect(again.text).toContain("nothing eligible"); + const gcExports = readdirSync(dir).filter((name) => name.startsWith("lcm-export-gc-")); + expect(gcExports.length).toBe(1); + }); + + it("gc skips a sidecar held open by another process and refuses it by name", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "live.session:1"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + // Open but never registered: simulates another profile holding the file. + const foreignPath = join(dir, "foreign.sqlite"); + const foreign = new LcmStore(foreignPath); + seedMessages(foreign, ["foreign history stays"]); + const stalePath = join(dir, "stale-old.sqlite"); + const staleDetached = new LcmStore(stalePath); + try { + seedMessages(staleDetached, ["stale history goes"]); + } finally { + staleDetached.close(); + } + ageFile(stalePath, 31); + ageFile(foreignPath, 31); + const dry = await invoke(ctx, deps, liveId, "gc"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("stale-old"); + expect(dry.text).not.toContain("foreign"); + expect(dry.text).toContain("held open"); + const named = await invoke(ctx, deps, liveId, "gc --session foreign --apply"); + expect(named.kind).toBe("error"); + expect(named.text).toMatch(/another process/); + expect(existsSync(foreignPath)).toBe(true); + const applied = await invoke(ctx, deps, liveId, "gc --apply"); + expect(applied.kind).toBe("success"); + expect(existsSync(stalePath)).toBe(false); + expect(existsSync(foreignPath)).toBe(true); + foreign.close(); + // Closing the last WAL handle checkpoints into the sqlite file and refreshes + // its mtime, so re-age before the final collect. + ageFile(foreignPath, 31); + const afterClose = await invoke(ctx, deps, liveId, "gc --apply"); + expect(afterClose.kind).toBe("success"); + expect(existsSync(foreignPath)).toBe(false); + }); + + it("gc apply writes one export per sidecar so each restores", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "live-gc-split"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-split"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const seedStale = (stem: string, texts: string[]): string => { + const path = join(dir, `${stem}.sqlite`); + const stale = new LcmStore(path); + try { + seedMessages(stale, texts); + const ids = stale.messagesInSpan("e00", "e01").map((message) => message.id); + stale.insertSummary({ + kind: "leaf", + text: `leaf for ${stem}`, + tokens: 4, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds: ids, + }); + } finally { + stale.close(); + } + ageFile(path, 31); + return path; + }; + const staleA = seedStale("stale-a", ["stale a one", "stale a two"]); + const staleB = seedStale("stale-b", ["stale b one", "stale b two"]); + ageFile(join(dir, `${sanitizeSidecarId(liveId)}.sqlite`), 31); + const applied = await invoke(ctx, deps, liveId, "gc --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("removed 2 of 2"); + const gcExports = readdirSync(dir) + .filter((name) => name.startsWith("lcm-export-gc-")) + .sort(); + expect(gcExports.length).toBe(2); + expect(applied.text).toContain(gcExports[0]!); + expect(applied.text).toContain(gcExports[1]!); + expect(existsSync(staleA)).toBe(false); + expect(existsSync(staleB)).toBe(false); + ctx.sessions.create(SessionId("restore-a"), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("restore-b"), { meta: { cwd: dir } }); + const fileA = gcExports.find((name) => name.includes("stale-a"))!; + const fileB = gcExports.find((name) => name.includes("stale-b"))!; + expect(fileA).toBeDefined(); + expect(fileB).toBeDefined(); + const impA = await invoke(ctx, deps, "restore-a", `import ${join(dir, fileA)}`); + expect(impA.kind).toBe("success"); + expect(deps.storeFor!("restore-a").allMessages().length).toBe(2); + expect(deps.storeFor!("restore-a").allSummaries().length).toBe(1); + const impB = await invoke(ctx, deps, "restore-b", `import ${join(dir, fileB)}`); + expect(impB.kind).toBe("success"); + expect(deps.storeFor!("restore-b").allMessages().length).toBe(2); + expect(deps.storeFor!("restore-b").allSummaries().length).toBe(1); + const againA = await invoke(ctx, deps, "restore-a", `import ${join(dir, fileA)}`); + expect(againA.kind).toBe("success"); + expect(againA.text).toContain("already present"); + expect(deps.storeFor!("restore-a").allMessages().length).toBe(2); + }); + + it("gc apply on an empty stale sidecar removes it without naming an export file", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "live-gc-empty"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-empty"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const live = deps.storeFor!(liveId); + seedMessages(live, ["live stays"]); + const emptyPath = join(dir, "empty-old.sqlite"); + const empty = new LcmStore(emptyPath); + empty.close(); + ageFile(emptyPath, 31); + ageFile(join(dir, `${sanitizeSidecarId(liveId)}.sqlite`), 31); + const applied = await invoke(ctx, deps, liveId, "gc --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("removed 1 of 1"); + expect(applied.text).toContain("transcripts were empty so no export file was written"); + expect(applied.text).not.toContain("lcm-export-gc-"); + expect(existsSync(emptyPath)).toBe(false); + const gcExports = readdirSync(dir).filter((name) => name.startsWith("lcm-export-gc-")); + expect(gcExports.length).toBe(0); + }); + + it("migrate dry-run describes a stale sidecar and apply upgrades it", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("mig-live"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!("mig-live"); + appendConversation(ctx.sessions.get(SessionId("mig-live"))!, ROUTE, "live notes"); + const stalePath = join(dir, "old-session.sqlite"); + const old = new DatabaseSync(stalePath); + try { + old.exec(` + CREATE TABLE messages ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + entry_id TEXT NOT NULL UNIQUE, + role TEXT NOT NULL, + text TEXT NOT NULL, + tokens INTEGER NOT NULL, + timestamp INTEGER NOT NULL, + payload TEXT, + removed_at INTEGER + ); + CREATE TABLE summaries ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + kind TEXT NOT NULL CHECK (kind IN ('leaf','condensed')), + text TEXT NOT NULL, + tokens INTEGER NOT NULL, + depth INTEGER NOT NULL, + first_entry_id TEXT NOT NULL, + last_entry_id TEXT NOT NULL, + created_at INTEGER NOT NULL, + thorough_text TEXT, + thorough_tokens INTEGER, + status TEXT NOT NULL DEFAULT 'committed' CHECK (status IN ('pending','committed')), + run TEXT + ); + CREATE TABLE provenance ( + summary_id INTEGER NOT NULL, + child_type TEXT NOT NULL CHECK (child_type IN ('message','summary')), + child_id INTEGER NOT NULL, + PRIMARY KEY (summary_id, child_type, child_id) + ); + PRAGMA user_version = 1; + `); + } finally { + old.close(); + } + expect(storeGeneration(stalePath)).toBe(1); + const dry = await invoke(ctx, deps, "mig-live", "migrate"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("dry-run"); + expect(dry.text).toContain("old-session"); + expect(dry.text).toContain("generation 1"); + const applied = await invoke(ctx, deps, "mig-live", "migrate --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("upgraded 1 of 1"); + expect(storeGeneration(stalePath)).toBe(SCHEMA_VERSION); + const quiet = await invoke(ctx, deps, "mig-live", "migrate"); + expect(quiet.kind).toBe("success"); + expect(quiet.text).toContain("nothing to do"); + }); + + it("heal --apply uses the configured summary model over the agent route", async () => { + const scripts = [[...textChunks("a repaired leaf summary")]]; + const { ctx, adapter } = setupContext(ROUTE, scripts); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("heal-cfg"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps: LcmCommandDeps = { + ...depsFor(dir, stores), + summarizationProvider: ROUTE.provider, + summarizationModel: "configured-heal-model", + }; + const store = deps.storeFor!("heal-cfg"); + seedMessages(store, [ + `heal source alpha ${"detail ".repeat(60)}`, + `heal source beta ${"context ".repeat(60)}`, + ]); + const messageIds = store.messagesInSpan("e00", "e01").map((message) => message.id); + store.insertSummary({ + kind: "leaf", + text: `${TRUNCATED_SUMMARY_PREFIX}\nmechanical`, + tokens: 5, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds, + quality: "mechanical", + }); + const applied = await invoke(ctx, deps, "heal-cfg", "heal --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("repaired 1 clean"); + expect(adapter.requests.length).toBeGreaterThan(0); + expect(adapter.requests[0]?.model).toBe("configured-heal-model"); + }); + + it("heal --apply falls back to the agent route when no summary model is configured", async () => { + const scripts = [[...textChunks("a repaired leaf summary")]]; + const { ctx, adapter } = setupContext(ROUTE, scripts); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("heal-fb"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const store = deps.storeFor!("heal-fb"); + seedMessages(store, [ + `heal source alpha ${"detail ".repeat(60)}`, + `heal source beta ${"context ".repeat(60)}`, + ]); + const messageIds = store.messagesInSpan("e00", "e01").map((message) => message.id); + store.insertSummary({ + kind: "leaf", + text: `${TRUNCATED_SUMMARY_PREFIX}\nmechanical`, + tokens: 5, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds, + quality: "mechanical", + }); + const applied = await invoke(ctx, deps, "heal-fb", "heal --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("repaired 1 clean"); + expect(adapter.requests.length).toBeGreaterThan(0); + expect(adapter.requests[0]?.model).toBe(ROUTE.model); + }); + + it("dispose leaves the healing session's store open", async () => { + const scripts = [[...textChunks("a repaired leaf summary")]]; + const { ctx } = setupContext(ROUTE, scripts); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + const originalClose = Reflect.get(LcmStore.prototype, "close") as (this: LcmStore) => void; + let closes = 0; + LcmStore.prototype.close = function (this: LcmStore): void { + closes += 1; + Reflect.apply(originalClose, this, []); + }; + const stores = new Map(); + try { + ctx.sessions.create(SessionId("heal-live"), { meta: { cwd: dir } }); + const session = ctx.sessions.get(SessionId("heal-live"))!; + appendConversation(session, ROUTE, "heal live history ".repeat(20)); + const deps = depsFor(dir, stores); + const store = deps.storeFor!("heal-live"); + const rows = store.allMessages(); + expect(rows.length).toBeGreaterThan(0); + store.insertSummary({ + kind: "leaf", + text: `${TRUNCATED_SUMMARY_PREFIX}\nmechanical`, + tokens: 5, + depth: 0, + firstEntryId: rows[0]!.entryId, + lastEntryId: rows[rows.length - 1]!.entryId, + messageIds: rows.map((row) => row.id), + quality: "mechanical", + }); + const pending = invoke(ctx, deps, "heal-live", "heal --apply"); + expect(healInFlightSessions()).toContain("heal-live"); + dispose(); + expect(closes).toBe(0); + const applied = await pending; + expect(applied.kind).toBe("success"); + expect(healInFlightSessions()).toHaveLength(0); + dispose(); + expect(closes).toBeGreaterThan(0); + } finally { + LcmStore.prototype.close = originalClose; + for (const store of stores.values()) { + try { + store.close(); + } catch {} + } + } + }); + + it("gc --session collects only that session", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-sel"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-sel"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const live = deps.storeFor!(liveId); + seedMessages(live, ["live history stays"]); + for (const stem of ["stale-sel-a", "stale-sel-b"]) { + const stale = new LcmStore(join(dir, `${stem}.sqlite`)); + try { + seedMessages(stale, [`history for ${stem}`]); + } finally { + stale.close(); + } + ageFile(join(dir, `${stem}.sqlite`), 31); + } + const dry = await invoke(ctx, deps, liveId, "gc --session stale-sel-a"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("stale-sel-a"); + expect(dry.text).not.toContain("stale-sel-b"); + const applied = await invoke(ctx, deps, liveId, "gc --session stale-sel-a --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("removed 1 of 1"); + expect(existsSync(join(dir, "stale-sel-a.sqlite"))).toBe(false); + expect(existsSync(join(dir, "stale-sel-b.sqlite"))).toBe(true); + }); + + it("gc --older-than collects only sidecars older than the window", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-age"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-age"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const oldPath = join(dir, "stale-old-age.sqlite"); + const old = new LcmStore(oldPath); + try { + seedMessages(old, ["two days old"]); + } finally { + old.close(); + } + ageFile(oldPath, 2); + const freshPath = join(dir, "stale-fresh-age.sqlite"); + const fresh = new LcmStore(freshPath); + try { + seedMessages(fresh, ["fresh today"]); + } finally { + fresh.close(); + } + const recent = await invoke(ctx, deps, liveId, "gc --older-than 1d"); + expect(recent.kind).toBe("success"); + expect(recent.text).toContain("stale-old-age"); + expect(recent.text).not.toContain("stale-fresh-age"); + const dflt = await invoke(ctx, deps, liveId, "gc"); + expect(dflt.kind).toBe("success"); + expect(dflt.text).toContain("nothing eligible"); + }); + + it("gc --session collects a fresh sidecar despite the age window", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-fresh"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const stale = new LcmStore(join(dir, "stale-fresh-a.sqlite")); + try { + seedMessages(stale, ["history for stale-fresh-a"]); + } finally { + stale.close(); + } + ageFile(join(dir, "stale-fresh-a.sqlite"), 31); + const fresh = new LcmStore(join(dir, "fresh-named.sqlite")); + try { + seedMessages(fresh, ["history for fresh-named"]); + } finally { + fresh.close(); + } + const dry = await invoke(ctx, deps, liveId, "gc --session fresh-named --older-than 30d"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("fresh-named"); + expect(dry.text).toContain("/lcm gc --apply --session fresh-named"); + expect(dry.text).not.toContain("--older-than"); + }); + + it("gc dry-run hint echoes session and older-than selectors", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-hint"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-hint"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + for (const stem of ["stale-hint-a", "stale-hint-b"]) { + const stale = new LcmStore(join(dir, `${stem}.sqlite`)); + try { + seedMessages(stale, [`history for ${stem}`]); + } finally { + stale.close(); + } + ageFile(join(dir, `${stem}.sqlite`), 31); + } + const sessionDry = await invoke(ctx, deps, liveId, "gc --session stale-hint-a"); + expect(sessionDry.kind).toBe("success"); + expect(sessionDry.text).toContain("/lcm gc --apply --session stale-hint-a"); + const ageDry = await invoke(ctx, deps, liveId, "gc --older-than 1d"); + expect(ageDry.kind).toBe("success"); + expect(ageDry.text).toContain("/lcm gc --apply --older-than 1d"); + const bothDry = await invoke(ctx, deps, liveId, "gc --session stale-hint-a --older-than 1d"); + expect(bothDry.kind).toBe("success"); + expect(bothDry.text).toContain("--session stale-hint-a"); + expect(bothDry.text).not.toContain("--older-than"); + }); + + it("gc apply after a session hint deletes only the previewed session", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-hint-apply"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-hint-apply"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + for (const stem of ["stale-apply-a", "stale-apply-b"]) { + const stale = new LcmStore(join(dir, `${stem}.sqlite`)); + try { + seedMessages(stale, [`history for ${stem}`]); + } finally { + stale.close(); + } + ageFile(join(dir, `${stem}.sqlite`), 31); + } + const dry = await invoke(ctx, deps, liveId, "gc --session stale-apply-a"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("stale-apply-a"); + expect(dry.text).not.toContain("stale-apply-b"); + expect(dry.text).toContain("/lcm gc --apply --session stale-apply-a"); + const applied = await invoke(ctx, deps, liveId, "gc --session stale-apply-a --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("removed 1 of 1"); + expect(existsSync(join(dir, "stale-apply-a.sqlite"))).toBe(false); + expect(existsSync(join(dir, "stale-apply-b.sqlite"))).toBe(true); + }); + + it("gc older-than accepts only a number with an optional d suffix", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-parse"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-parse"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + for (const valid of ["2", "2d", "2.5d"]) { + const result = await invoke(ctx, deps, liveId, `gc --older-than ${valid}`); + expect(result.kind).toBe("success"); + } + for (const invalid of ["2weeks", "d", "-1"]) { + const result = await invoke(ctx, deps, liveId, `gc --older-than ${invalid}`); + expect(result.kind).toBe("error"); + expect(result.text).toContain("--older-than takes a number of days"); + } + const empty = await invoke(ctx, deps, liveId, "gc --older-than="); + expect(empty.kind).toBe("error"); + expect(empty.text).toContain("--older-than takes a number of days"); + }); + + it("gc refuses a missing or invalid selector value", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-bad"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-bad"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const missingSession = await invoke(ctx, deps, liveId, "gc --session"); + expect(missingSession.kind).toBe("error"); + expect(missingSession.text).toContain("--session takes a session id"); + const missingDays = await invoke(ctx, deps, liveId, "gc --older-than"); + expect(missingDays.kind).toBe("error"); + expect(missingDays.text).toContain("--older-than takes a number of days"); + const invalidDays = await invoke(ctx, deps, liveId, "gc --older-than soon"); + expect(invalidDays.kind).toBe("error"); + expect(invalidDays.text).toContain("--older-than takes a number of days"); + const liveSelected = await invoke(ctx, deps, liveId, `gc --session ${liveId}`); + expect(liveSelected.kind).toBe("error"); + expect(liveSelected.text).toContain("never deletes an open sidecar"); + const unknown = await invoke(ctx, deps, liveId, "gc --session no-such-sidecar"); + expect(unknown.kind).toBe("success"); + expect(unknown.text).toContain("no sidecar for session no-such-sidecar"); + }); + + it("dump --save= with an empty inline value names the file error", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-empty-save"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + seedMessages(deps.storeFor!("dump-empty-save"), ["empty save history"]); + const result = await invoke(ctx, deps, "dump-empty-save", "dump --save="); + expect(result.kind).toBe("error"); + expect(result.text).toContain("name the file to write"); + }); + + it("gc --session= with an empty inline value asks for a session id", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-empty-session"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + ctx.sessions.create(SessionId("other-empty-session"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const result = await invoke(ctx, deps, liveId, "gc --session="); + expect(result.kind).toBe("error"); + expect(result.text).toContain("--session takes a session id"); + }); + + it("gc --older-than= with an inline value filters by age", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + const liveId = "gc-live-inline-older"; + ctx.sessions.create(SessionId(liveId), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + deps.storeFor!(liveId); + const result = await invoke(ctx, deps, liveId, "gc --older-than=1d"); + expect(result.kind).toBe("success"); + }); + + it("dump keeps a query term that matches the save filename", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-query-save"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + seedMessages(deps.storeFor!("dump-query-save"), ["save-me query term save-me"]); + const target = join(tempDir(), "save-me.txt"); + const result = await invoke(ctx, deps, "dump-query-save", `dump --save ${target} ${target}`); + expect(result.kind).toBe("success"); + expect(result.text).toContain("wrote"); + const { readFileSync } = await import("node:fs"); + const written = readFileSync(target, "utf8"); + expect(written).toContain(`search ${JSON.stringify(target)}`); + }); + + it("dump lists each stored summary once from a single read", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-single-read"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const store = deps.storeFor!("dump-single-read"); + seedMessages(store, ["single read one", "single read two"]); + const ids = store.messagesInSpan("e00", "e01").map((message) => message.id); + store.insertSummary({ + kind: "leaf", + text: "leaf for single read", + tokens: 4, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds: ids, + }); + const result = await invoke(ctx, deps, "dump-single-read", "dump"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("LCM sidecar dump: dump-single-read"); + expect(result.text).toContain("1 shown of 1"); + expect(result.text).toContain("single read one"); + }); + + it("heal dry-run lists a truncated node and apply repairs it on a scripted model", async () => { + const scripts = [[...textChunks("a repaired leaf summary")]]; + const { ctx } = setupContext(ROUTE, scripts); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("heal-1"), { meta: { cwd: dir } }); + const stores = new Map(); + const deps = depsFor(dir, stores); + const store = deps.storeFor!("heal-1"); + seedMessages(store, [ + `heal source alpha ${"detail ".repeat(60)}`, + `heal source beta ${"context ".repeat(60)}`, + ]); + const messageIds = store.messagesInSpan("e00", "e01").map((message) => message.id); + store.insertSummary({ + kind: "leaf", + text: `${TRUNCATED_SUMMARY_PREFIX}\nmechanical`, + tokens: 5, + depth: 0, + firstEntryId: "e00", + lastEntryId: "e01", + messageIds, + quality: "mechanical", + }); + const badFlag = await invoke(ctx, deps, "heal-1", "heal --max-nodes nope"); + expect(badFlag.kind).toBe("error"); + expect(badFlag.text).toContain("--max-nodes"); + const dry = await invoke(ctx, deps, "heal-1", "heal"); + expect(dry.kind).toBe("success"); + expect(dry.text).toContain("dry-run"); + expect(dry.text).toContain("1 candidate(s)"); + const applied = await invoke(ctx, deps, "heal-1", "heal --apply"); + expect(applied.kind).toBe("success"); + expect(applied.text).toContain("repaired 1 clean"); + const repaired = store.allSummaries()[0]; + expect(repaired?.quality).toBe("clean"); + expect(existsSync(join(dir, "heal-1.sqlite"))).toBe(true); + }); +}); diff --git a/packages/deepseek/test/lifecycle.test.ts b/packages/deepseek/test/lifecycle.test.ts new file mode 100644 index 0000000..4bc2b0a --- /dev/null +++ b/packages/deepseek/test/lifecycle.test.ts @@ -0,0 +1,389 @@ +import { mkdtempSync, readFileSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import SessionStore, { SessionId } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { LcmStore, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { lcmCommandDefinition, type LcmCommandDeps } from "../src/command.ts"; +import { createRecord } from "../src/metrics.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-lcm-")); +} + +async function mountBase(ctx: Context): Promise { + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function depsFor(dir: string, stores: Map): LcmCommandDeps { + return { + stores: () => stores, + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sessionId}.sqlite`), + }; +} + +function seedStore(dir: string, sessionId: string, texts: string[]): LcmStore { + const store = new LcmStore(join(dir, `${sessionId}.sqlite`)); + ingestEntries( + store, + texts.map((text, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text, + timestamp: i, + })), + ); + return store; +} + +function invoke( + ctx: Context, + deps: LcmCommandDeps, + sessionId: string, + rawInput: string, +): Promise<{ kind: string; text?: string }> { + const session = ctx.sessions.get(SessionId(sessionId)); + if (!session) throw new Error(`no session ${sessionId}`); + const definition = lcmCommandDefinition(ctx, deps); + return definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId(`cmd-${rawInput.split(" ")[0] || "status"}`), + rawInput, + attachments: [], + }) as Promise<{ kind: string; text?: string }>; +} + +describe("lcm read-only verbs", () => { + it("status keeps counts and adds window, thresholds, and the last pass", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const session = ctx.sessions.create(SessionId("status-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "status notes"); + const store = seedStore(dir, "status-1", ["status hello"]); + createRecord(dir)({ + kind: "pass", + session: "status-1", + outcome: "stored", + leaf: 2, + condensed: 1, + }); + const result = await invoke( + ctx, + depsFor(dir, new Map([["status-1", store]])), + "status-1", + "status", + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("status-1: 1 messages, 0 summaries"); + expect(result.text).toContain("window (status-1):"); + expect(result.text).toContain("thresholds: swap 70% / recut 85%"); + expect(result.text).toContain("last pass (status-1): stored (2 leaf, 1 condensed)"); + }); + + it("status reports the configured pressure threshold instead of the default", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const session = ctx.sessions.create(SessionId("status-cfg"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "status notes"); + const store = seedStore(dir, "status-cfg", ["status hello"]); + const deps: LcmCommandDeps = { + ...depsFor(dir, new Map([["status-cfg", store]])), + thresholdRatio: 0.5, + }; + const result = await invoke(ctx, deps, "status-cfg", "status"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("thresholds: swap 50% / recut 85%"); + expect(result.text).not.toContain("swap 70%"); + }); + + it("status applies the routed model policy over the base zone", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const session = ctx.sessions.create(SessionId("status-pol"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "status notes"); + const store = seedStore(dir, "status-pol", ["status hello"]); + const deps: LcmCommandDeps = { + ...depsFor(dir, new Map([["status-pol", store]])), + thresholdRatio: 0.5, + modelPolicies: [{ provider: ROUTE.provider, model: ROUTE.model, thresholdRatio: 0.3 }], + }; + const result = await invoke(ctx, deps, "status-pol", "status"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("thresholds: swap 30% / recut 85%"); + expect(result.text).not.toContain("swap 50%"); + }); + + it("diagnose reports the configured pressure threshold", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("diag-cfg"), { meta: { cwd: dir } }); + const store = seedStore(dir, "diag-cfg", ["diagnose hello"]); + const deps: LcmCommandDeps = { + ...depsFor(dir, new Map([["diag-cfg", store]])), + thresholdRatio: 0.5, + }; + const result = await invoke(ctx, deps, "diag-cfg", "diagnose"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("swap 50%"); + expect(result.text).toContain('"swapAtRatio":0.5'); + expect(result.text).not.toContain("swap 70%"); + }); + + it("report rolls up seeded pass, checkpoint, recall, and retrieval rows", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("rep-1"), { meta: { cwd: dir } }); + const record = createRecord(dir); + record({ kind: "pass", session: "rep-1", outcome: "stored", leaf: 2, condensed: 1 }); + record({ kind: "pass", session: "rep-1", outcome: "nothing", reason: "span covered" }); + record({ kind: "compaction-checkpoint", session: "rep-1", summarySeqs: [1], leafSeqs: [2] }); + record({ kind: "recall", session: "rep-1", tool: "lcm_grep", outcome: "hit" }); + record({ + kind: "retrieval", + session: "rep-1", + stop: "answered", + steps: 2, + retrievedTokens: 10, + }); + record({ kind: "backfill", session: "rep-1", ingested: 3, skipped: 1 }); + record({ kind: "backfill", session: "rep-1", ingested: 4, skipped: 0 }); + const result = await invoke(ctx, depsFor(dir, new Map()), "rep-1", "report"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("passes: 2 (stored 1, nothing 1, failed 0); 2 leaf, 1 condensed"); + expect(result.text).toContain("checkpoints: 1 commit(s)"); + expect(result.text).toContain("backfill: 2 session(s), 7 ingested"); + expect(result.text).toContain("recall: 1 (hit 1)"); + expect(result.text).toContain("retrieval: 1 (answered 1)"); + expect(result.text).toContain("last pass: nothing (span covered)"); + }); + + it("backup writes a timestamped copy of the calling session sidecar", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("bak-1"), { meta: { cwd: dir } }); + const store = seedStore(dir, "bak-1", ["backup hello"]); + const target = join(dir, "manual-backup.sqlite"); + const result = await invoke( + ctx, + depsFor(dir, new Map([["bak-1", store]])), + "bak-1", + `backup ${target}`, + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("LCM backup written:"); + expect(statSync(target).mode & 0o777).toBe(0o600); + }); + + it("export writes JSONL with 0600 mode", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("exp-1"), { meta: { cwd: dir } }); + const store = seedStore(dir, "exp-1", ["export one", "export two"]); + const target = join(dir, "out.jsonl"); + const result = await invoke( + ctx, + depsFor(dir, new Map([["exp-1", store]])), + "exp-1", + `export ${target}`, + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("LCM export: 2 messages + 0 summaries"); + expect(statSync(target).mode & 0o777).toBe(0o600); + const lines = readFileSync(target, "utf8").trim().split("\n"); + expect(lines.length).toBe(2); + for (const line of lines) { + const row = JSON.parse(line) as { rowType?: string }; + expect(row.rowType).toBe("message"); + } + }); + + it("diagnose redacts secret material from the bundle", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("diag-1"), { meta: { cwd: dir } }); + const store = seedStore(dir, "diag-1", ["diagnose hello"]); + const secret = `sk-ant-${"A".repeat(20)}`; + createRecord(dir)({ kind: "pass", session: "diag-1", outcome: "failed", error: secret }); + const result = await invoke( + ctx, + depsFor(dir, new Map([["diag-1", store]])), + "diag-1", + "diagnose", + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("LCM diagnose bundle:"); + expect(result.text).not.toContain(secret); + expect(result.text).toContain("[REDACTED]"); + }); + + it("diagnose names the calling provider and model", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("diag-model"), { meta: { cwd: dir } }); + const store = seedStore(dir, "diag-model", ["diagnose model hello"]); + const result = await invoke( + ctx, + depsFor(dir, new Map([["diag-model", store]])), + "diag-model", + "diagnose", + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("test-provider/test-model"); + }); + + it("report caps the rows it reads", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("cap-1"), { meta: { cwd: dir } }); + const record = createRecord(dir); + for (let i = 0; i < 650; i += 1) { + record({ kind: "pass", session: "cap-1", outcome: "stored", leaf: 1, condensed: 0, seq: i }); + } + const result = await invoke(ctx, depsFor(dir, new Map()), "cap-1", "report"); + expect(result.kind).toBe("success"); + expect(result.text).toContain("500 row(s) for session cap-1"); + expect(result.text).not.toContain("650 row(s)"); + }); + + it("diagnose --save writes the bundle to a file", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("diag-2"), { meta: { cwd: dir } }); + const store = seedStore(dir, "diag-2", ["diagnose save hello"]); + const target = join(dir, "diag.txt"); + const deps = depsFor(dir, new Map([["diag-2", store]])); + const missing = await invoke(ctx, deps, "diag-2", "diagnose --save"); + expect(missing.kind).toBe("error"); + expect(missing.text).toContain("name the file to write"); + const result = await invoke(ctx, deps, "diag-2", `diagnose --save ${target}`); + expect(result.kind).toBe("success"); + expect(result.text).toContain("mode 0600"); + expect(readFileSync(target, "utf8")).toContain("LCM diagnose bundle:"); + }); +}); + +describe("lcm dump verb", () => { + it("dump prints the calling session sidecar and answers a free-text query", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-1"), { meta: { cwd: dir } }); + const store = seedStore(dir, "dump-1", ["dump hello world"]); + const result = await invoke( + ctx, + depsFor(dir, new Map([["dump-1", store]])), + "dump-1", + "dump hello", + ); + expect(result.kind).toBe("success"); + expect(result.text).toContain("LCM sidecar dump: dump-1"); + expect(result.text).toContain("dump hello world"); + }); + + it("dump --save writes the dump to a file with 0600 mode", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-2"), { meta: { cwd: dir } }); + const store = seedStore(dir, "dump-2", ["dump save hello"]); + const target = join(dir, "dump-out.txt"); + const deps = depsFor(dir, new Map([["dump-2", store]])); + const result = await invoke(ctx, deps, "dump-2", `dump --save ${target}`); + expect(result.kind).toBe("success"); + expect(result.text).toContain("mode 0600"); + expect(result.text).toContain(target); + expect(statSync(target).mode & 0o777).toBe(0o600); + expect(readFileSync(target, "utf8")).toContain("LCM sidecar dump: dump-2"); + }); + + it("dump --save without a path errors", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-3"), { meta: { cwd: dir } }); + const store = seedStore(dir, "dump-3", ["dump missing hello"]); + const deps = depsFor(dir, new Map([["dump-3", store]])); + const result = await invoke(ctx, deps, "dump-3", "dump --save"); + expect(result.kind).toBe("error"); + expect(result.text).toContain("name the file to write"); + }); + + it("dump --include-removed surfaces removed rows the default dump hides", async () => { + const { ctx } = setupContext(ROUTE, []); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("dump-4"), { meta: { cwd: dir } }); + const store = seedStore(dir, "dump-4", ["dump keeper alpha", "dump hidden beta"]); + store.setRemoved(["e01"], Date.now()); + const deps = depsFor(dir, new Map([["dump-4", store]])); + const plain = await invoke(ctx, deps, "dump-4", "dump beta"); + expect(plain.kind).toBe("success"); + expect(plain.text).not.toContain("dump hidden beta"); + const kept = await invoke(ctx, deps, "dump-4", "dump beta --include-removed"); + expect(kept.kind).toBe("success"); + expect(kept.text).toContain("dump hidden beta"); + }); +}); + +describe("lcm command shape", () => { + it("claims spaced verbs through an input hint and keeps bare dispatch on status", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + ctx.sessions.create(SessionId("shape-1"), { meta: { cwd: dir } }); + const store = seedStore(dir, "shape-1", ["shape hello"]); + const deps = depsFor(dir, new Map([["shape-1", store]])); + const definition = lcmCommandDefinition(ctx, deps); + expect(definition.name).toBe("lcm"); + expect(definition.input?.hint).toBe( + "[status|report|dump|backup|export|import|gc|migrate|heal|diagnose|compact] ...", + ); + const disposeCommand = ctx.commands.register(definition); + try { + const bare = await invoke(ctx, deps, "shape-1", ""); + expect(bare.kind).toBe("success"); + expect(bare.text).toContain("shape-1"); + const spaced = await invoke(ctx, deps, "shape-1", " status"); + expect(spaced.kind).toBe("success"); + expect(spaced.text).toContain("shape-1"); + const dump = await invoke(ctx, deps, "shape-1", " dump hello"); + expect(dump.kind).toBe("success"); + expect(dump.text).toContain("shape hello"); + } finally { + disposeCommand(); + } + }); +}); diff --git a/packages/deepseek/test/metrics.test.ts b/packages/deepseek/test/metrics.test.ts new file mode 100644 index 0000000..5cbc1cd --- /dev/null +++ b/packages/deepseek/test/metrics.test.ts @@ -0,0 +1,209 @@ +import { mkdtempSync, readFileSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { LcmStore, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { MessageIndex } from "../src/ingest.ts"; +import { SpanAlreadyMemoryError, runLadder } from "../src/ladder.ts"; +import { createRecord, metricsPath, readMetricRows } from "../src/metrics.ts"; +import { dshRef } from "../src/model-host.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { dshReader } from "../src/reader.ts"; +import { expandQueryAnswer } from "../src/tools.ts"; +import { setupContext, textChunks, toolCallChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; +const SUMMARY = "Checkpoint: metrics slice writes its rows beside the sidecar."; +const identity = (text: string): string => text; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-metrics-")); +} + +function seedStore(): LcmStore { + const store = new LcmStore(":memory:"); + ingestEntries( + store, + Array.from({ length: 8 }, (_, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text: `message ${i}: deciding the ${i % 2 === 0 ? "compaction engine" : "recall pipeline"} rollout ${"detail ".repeat(20)}`, + timestamp: i, + })), + ); + return store; +} + +describe("metrics sink", () => { + it("appends valid JSONL lines with 0600 mode", () => { + const dir = tempDir(); + const record = createRecord(dir); + record({ event: "lcm", kind: "pass", session: "s1", outcome: "stored", leaf: 1, condensed: 1 }); + record({ event: "lcm", kind: "retrieval", session: "s1", stop: "answered" }); + + const path = metricsPath(dir); + expect(statSync(path).mode & 0o777).toBe(0o600); + const lines = readFileSync(path, "utf8").trim().split("\n"); + expect(lines).toHaveLength(2); + for (const line of lines) { + expect(() => JSON.parse(line)).not.toThrow(); + } + const rows = readMetricRows(dir).map((row) => row as Record); + expect(rows).toHaveLength(2); + expect(rows[0]).toMatchObject({ kind: "pass", outcome: "stored" }); + expect(typeof rows[0]?.ts).toBe("number"); + expect(rows[1]).toMatchObject({ kind: "retrieval", stop: "answered" }); + }); + + it("emits exactly one run row per ladder run with the stored outcome", async () => { + const store = new LcmStore(":memory:"); + const rows: Record[] = []; + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + const outcome = await runLadder( + ctx, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + record: (row) => rows.push(row), + }, + ROUTE, + "metrics-ladder-1", + [ + { entryId: "ladder:e1", role: "user", text: `ladder notes ${"detail ".repeat(60)}` }, + { entryId: "ladder:e2", role: "assistant", text: `ladder reply ${"detail ".repeat(60)}` }, + ], + ); + expect(outcome.leafSummaryIds.length).toBeGreaterThan(0); + const passRows = rows.filter((row) => row.kind === "pass"); + expect(passRows).toHaveLength(1); + expect(passRows[0]).toMatchObject({ + event: "lcm", + session: "metrics-ladder-1", + outcome: "stored", + leaf: outcome.leafSummaryIds.length, + condensed: outcome.condensedSummaryIds.length, + }); + const usage = passRows[0]?.usage as { inputTokens?: number } | undefined; + expect(usage?.inputTokens).toBeGreaterThan(0); + }); + + it("records one nothing pass when the span is already memory", async () => { + const store = new LcmStore(":memory:"); + const span = Array.from({ length: 4 }, (_, i) => ({ + entryId: `nothing:e${i}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text: `nothing notes ${i} ${"detail ".repeat(60)}`, + })); + const rows: Record[] = []; + const { ctx } = setupContext(ROUTE, [ + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + textChunks(SUMMARY), + ]); + const shared = { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + record: (row: Record) => rows.push(row), + }; + const first = await runLadder(ctx, shared, ROUTE, "metrics-nothing", span); + expect(first.leafSummaryIds.length).toBeGreaterThan(0); + rows.length = 0; + // A sub-span of a stored run is covered on every row, so the pass records nothing. + await expect( + runLadder(ctx, shared, ROUTE, "metrics-nothing", span.slice(1, 3)), + ).rejects.toThrow(SpanAlreadyMemoryError); + const passRows = rows.filter((row) => row.kind === "pass"); + expect(passRows).toHaveLength(1); + expect(passRows[0]).toMatchObject({ + event: "lcm", + session: "metrics-nothing", + stage: "blocking", + outcome: "nothing", + reason: "covered", + span: ["nothing:e1", "nothing:e2"], + }); + expect(typeof passRows[0]?.durationMs).toBe("number"); + }); + + it("records one failed pass when the ladder throws", async () => { + const store = new LcmStore(":memory:"); + const rows: Record[] = []; + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)]); + // A write failure escapes the escalation fallback, so the run records failed. + store.insertSummary = () => { + throw new Error("sidecar write failed"); + }; + await expect( + runLadder( + ctx, + { + store: () => store, + index: () => new MessageIndex(), + redact: identity, + record: (row) => rows.push(row), + }, + ROUTE, + "metrics-failed", + [ + { entryId: "failed:e1", role: "user", text: `failing notes ${"detail ".repeat(60)}` }, + { + entryId: "failed:e2", + role: "assistant", + text: `failing reply ${"detail ".repeat(60)}`, + }, + ], + ), + ).rejects.toThrow("sidecar write failed"); + const passRows = rows.filter((row) => row.kind === "pass"); + expect(passRows).toHaveLength(1); + expect(passRows[0]).toMatchObject({ + event: "lcm", + session: "metrics-failed", + stage: "blocking", + outcome: "failed", + error: "sidecar write failed", + span: ["failed:e1", "failed:e2"], + }); + expect(typeof passRows[0]?.durationMs).toBe("number"); + }); + + it("records a recall call through the wired recorder", async () => { + const dir = tempDir(); + const store = seedStore(); + const { ctx } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "recall pipeline rollout" }), + textChunks("The recall pipeline rollout was approved for next week [e01]."), + ]); + const answer = await expandQueryAnswer( + store, + identity, + dshReader(ctx, { ref: dshRef(ROUTE) }), + "session-1", + { + query: "recall pipeline rollout", + prompt: "What did we decide about the recall pipeline rollout?", + }, + createRecord(dir), + ); + expect(answer).toContain(""); + const rows = readMetricRows(dir).map((row) => row as Record); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + event: "lcm", + kind: "retrieval", + session: "session-1", + stop: "answered", + }); + }); +}); diff --git a/packages/deepseek/test/model-host.test.ts b/packages/deepseek/test/model-host.test.ts new file mode 100644 index 0000000..e38c80f --- /dev/null +++ b/packages/deepseek/test/model-host.test.ts @@ -0,0 +1,154 @@ +import type { Agent } from "@deepseek-ai/dsh-agent"; +import { summarizeWithEscalation } from "lossless-core"; +import type { LlmComplete } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { agentRoute, dshModelHost } from "../src/model-host.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { cappedChunks, errorChunks, setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function stubAgent(header: unknown, options?: unknown): Agent { + return { + session: + header === undefined + ? {} + : { + requestHeader: () => ({ config: header }), + }, + options, + } as unknown as Agent; +} + +describe("agentRoute", () => { + it("prefers the latest routed header over agent options", () => { + const agent = stubAgent( + { provider: "header-provider", model: "header-model" }, + { provider: "options-provider", model: "options-model" }, + ); + expect(agentRoute(agent)).toEqual({ provider: "header-provider", model: "header-model" }); + }); + + it("falls back to agent options without a header", () => { + const agent = stubAgent(undefined, { provider: "options-provider", model: "options-model" }); + expect(agentRoute(agent)).toEqual({ provider: "options-provider", model: "options-model" }); + }); + + it("returns undefined when nothing resolves", () => { + expect(agentRoute(stubAgent(undefined, undefined))).toBeUndefined(); + expect(agentRoute(undefined)).toBeUndefined(); + }); + + it("falls back to options when requestHeader throws", () => { + const agent = { + session: { + requestHeader: () => { + throw new Error("header blew up"); + }, + }, + options: { provider: "options-provider", model: "options-model" }, + } as unknown as Agent; + expect(agentRoute(agent)).toEqual({ provider: "options-provider", model: "options-model" }); + }); + + it("ignores malformed header strings and returns undefined", () => { + const agent = stubAgent({ provider: 42, model: "" }, undefined); + expect(agentRoute(agent)).toBeUndefined(); + }); +}); + +describe("dshModelHost.complete", () => { + it("answers with text assembled from the chunk stream", async () => { + const { ctx, adapter } = setupContext(ROUTE, [textChunks("hello summary")]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE }); + const ref = host.session(); + expect(ref?.key).toBe("test-provider/test-model"); + if (!ref) throw new Error("session route did not resolve"); + const answer = await host.complete(ref, { prompt: "summarize this", maxTokens: 100 }); + expect(answer).toMatchObject({ + outcome: "answer", + text: "hello summary", + stop: "stop", + usage: { + input: 10, + output: 13, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 23, + }, + }); + expect(adapter.requests).toHaveLength(1); + expect(adapter.requests[0]).toMatchObject({ + provider: "test-provider", + model: "test-model", + maxTokens: 100, + purpose: "compaction", + }); + }); + + it("stamps the calling session id for per-conversation routing", async () => { + const { ctx, adapter } = setupContext(ROUTE, [textChunks("hello summary")]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE, sessionId: "session-1" }); + const ref = host.session(); + if (!ref) throw new Error("session route did not resolve"); + await host.complete(ref, { prompt: "summarize this", maxTokens: 100 }); + expect(adapter.requests).toHaveLength(1); + expect(adapter.requests[0]).toMatchObject({ sessionId: "session-1" }); + }); + + it("omits the session stamp when unset", async () => { + const { ctx, adapter } = setupContext(ROUTE, [textChunks("hello summary")]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE }); + const ref = host.session(); + if (!ref) throw new Error("session route did not resolve"); + await host.complete(ref, { prompt: "summarize this", maxTokens: 100 }); + expect(adapter.requests).toHaveLength(1); + expect(adapter.requests[0]).not.toHaveProperty("sessionId"); + }); + + it("fails capped when the provider stops at its token cap", async () => { + const { ctx } = setupContext(ROUTE, [cappedChunks("partial")]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE }); + const ref = host.session(); + if (!ref) throw new Error("session route did not resolve"); + const answer = await host.complete(ref, { prompt: "summarize this", maxTokens: 4 }); + expect(answer).toMatchObject({ outcome: "failed", capped: true, stop: "max-tokens" }); + }); + + it("fails open with the provider code on an error finish", async () => { + const { ctx } = setupContext(ROUTE, [errorChunks()]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE }); + const ref = host.session(); + if (!ref) throw new Error("session route did not resolve"); + const answer = await host.complete(ref, { prompt: "summarize this", maxTokens: 100 }); + expect(answer.outcome).toBe("failed"); + if (answer.outcome !== "failed") throw new Error("expected a failed answer"); + expect(answer.capped).toBe(false); + expect(answer.stop).toBe("error"); + expect(answer.reason).toContain("AUTH"); + }); + + it("summarizes one span through the real escalation ladder", async () => { + const { ctx } = setupContext(ROUTE, [textChunks("Decisions: ship it. File: src/a.ts.")]); + const host = dshModelHost(ctx, { defaultRoute: ROUTE }); + const ref = host.session(); + if (!ref) throw new Error("session route did not resolve"); + const llm: LlmComplete = async (systemPrompt, userText, maxTokens) => { + const answer = await host.complete(ref, { + prompt: `${systemPrompt}\n\n${userText}`, + maxTokens, + }); + if (answer.outcome !== "answer") throw new Error(answer.reason); + return answer.text; + }; + const result = await summarizeWithEscalation( + [{ role: "user", text: "working notes ".repeat(50) }], + 200, + llm, + ); + expect(result.level).toBe(1); + expect(result.retried).toBe(false); + expect(result.text).toContain("ship it"); + }); +}); diff --git a/packages/deepseek/test/plugin.test.ts b/packages/deepseek/test/plugin.test.ts new file mode 100644 index 0000000..5344441 --- /dev/null +++ b/packages/deepseek/test/plugin.test.ts @@ -0,0 +1,678 @@ +import assert from "node:assert/strict"; +import { existsSync, mkdtempSync, readdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import { CommandId } from "@deepseek-ai/dsh-commands/brand"; +import { ManualCompactionError } from "@deepseek-ai/dsh-compaction"; +import { createUserMessage } from "@deepseek-ai/dsh-llm"; +import SessionStore, { SessionId } from "@deepseek-ai/dsh-session"; +import type { SessionEvent } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime, { type ToolRunContext } from "@deepseek-ai/dsh-tools"; +import { LcmStore, SidecarStores } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { lcmCommandDefinition } from "../src/command.ts"; +import { acquireCore, releaseCore } from "../src/core.ts"; +import { LosslessSummarizerEngine } from "../src/engine.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { apply, applyLossless, buildEngineArgs, Config, inject, name } from "../src/plugin.ts"; +import { fakeAgent } from "./support/agent-fake.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; +const SUMMARY = "Checkpoint: engine work started and the config was read."; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-lossless-")); +} + +async function mountBase(ctx: Context): Promise { + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); +} + +function sidecarMap(dir: string): Map { + const map = new Map(); + for (const file of readdirSync(dir)) { + if (!file.endsWith(".sqlite")) continue; + map.set(file.slice(0, -".sqlite".length), new LcmStore(join(dir, file))); + } + return map; +} + +describe("applyLossless", () => { + it("exposes the Cordis plugin shape the loader applies", () => { + expect(name).toBe("deepseek-lossless"); + for (const service of ["sessions", "tools", "commands", "llm", "tokenMeter"]) { + expect(inject).toContain(service); + } + expect(inject).not.toContain("compaction"); + expect(inject).not.toContain("sessionQuery"); + expect(Config).toBeDefined(); + expect(typeof apply).toBe("function"); + }); + + it("keeps the loader Config keys in sync with the composition options", () => { + const dict = (Config as unknown as { dict?: Record }).dict ?? {}; + assert.deepEqual(Object.keys(dict).sort(), [ + "auto", + "bracketAuto", + "enabled", + "headroomTokens", + "healSourceTokens", + "leafChunkTokens", + "mechanicalTools", + "modelPolicies", + "retainRatio", + "retainTokens", + "sidecarDir", + "summarizationModel", + "summarizationProvider", + "summarizerTokensPerCall", + "summarizerTokensPerHeal", + "summarizerTokensPerPass", + "summaryTokens", + "thresholdRatio", + ]); + }); + + it("delegates the loader entrypoint to the composition on one path", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const dir = join(tempDir(), "apply-delegates"); + const dispose = apply(ctx, { sidecarDir: dir }); + expect(ctx.tools.schemas().map((schema) => schema.name)).not.toContain("lcm_grep"); + expect(existsSync(dir)).toBe(false); + dispose(); + }); + + it("mounts the same composition through apply as through applyLossless", async () => { + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = apply(ctx, { enabled: true, sidecarDir: dir }); + try { + expect(ctx.compaction).toBeInstanceOf(LosslessSummarizerEngine); + expect(ctx.tools.schemas().map((schema) => schema.name)).toContain("lcm_grep"); + const session = ctx.sessions.create(SessionId("delegate-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "delegate notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + expect( + ( + ctx.compaction as unknown as { + lossless: { store: (id: string) => unknown }; + } + ).lossless.store("delegate-1"), + ).toBeDefined(); + } finally { + dispose(); + } + }); + + it("routes auto and bracketAuto to the seam and the engine deps", () => { + const dir = tempDir(); + const core = acquireCore(dir, true); + try { + const [seam, deps] = buildEngineArgs( + { auto: true, bracketAuto: true, thresholdRatio: 0.7 }, + core, + ); + assert.equal(seam.auto, true); + assert.equal(deps.bracketAuto, true); + assert.equal(seam.thresholdRatio, 0.7); + } finally { + releaseCore(core); + } + }); + it("shares one core across overlapping mounts and closes only on the last dispose", async () => { + const { ctx: first } = setupContext(ROUTE, [textChunks(SUMMARY)], { + contextWindow: 1_000_000, + }); + await mountBase(first); + const dir = tempDir(); + const stopFirst = applyLossless(first, { enabled: true, sidecarDir: dir }); + const { ctx: second } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(second); + const stopSecond = applyLossless(second, { enabled: true, sidecarDir: dir }); + try { + const session = second.sessions.create(SessionId("overlap-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "overlap notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + stopFirst(); + // The first dispose released only its ownership: the shared store stays + // open and readable through the still-mounted sibling. + const shared = ( + second as unknown as { + compaction: { lossless: { store: (id: string) => LcmStore } }; + } + ).compaction.lossless.store("overlap-1"); + expect(shared.allMessages().length).toBeGreaterThan(0); + } finally { + stopSecond(); + } + // The last dispose closed the shared handle; the data survived on disk. + const reopened = new LcmStore(join(dir, "overlap-1.sqlite")); + try { + expect(reopened.allMessages().length).toBeGreaterThan(0); + } finally { + reopened.close(); + } + }); + + it("registers nothing until enabled", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const dir = join(tempDir(), "never-created"); + const dispose = applyLossless(ctx, { sidecarDir: dir }); + expect(ctx.tools.schemas().map((schema) => schema.name)).not.toContain("lcm_grep"); + expect(existsSync(dir)).toBe(false); + dispose(); + }); + + it("reports the sidecar dir through the logger, never stdout", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const logged: unknown[][] = []; + const originalInfo = ctx.logger.info; + ctx.logger.info = (format: any, ...rest: any[]) => { + logged.push([format, ...rest]); + }; + const logs: unknown[][] = []; + const originalLog = console.log; + console.log = (...args: unknown[]) => { + logs.push(args); + }; + const written: string[] = []; + const originalWrite = process.stdout.write.bind(process.stdout); + process.stdout.write = ((chunk: unknown) => { + written.push(String(chunk)); + return true; + }) as typeof process.stdout.write; + try { + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + expect(logs.length).toBe(0); + expect(written.some((chunk) => chunk.includes("LCM sidecar dir"))).toBe(false); + expect(logged.some((args) => args[0] === `LCM sidecar dir: ${dir} (explicit)`)).toBe(true); + } finally { + dispose(); + } + } finally { + console.log = originalLog; + process.stdout.write = originalWrite; + ctx.logger.info = originalInfo; + } + }); + + it("mounts the engine, the tools, and the live mirror", async () => { + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + expect(ctx.compaction).toBeInstanceOf(LosslessSummarizerEngine); + expect(ctx.tools.schemas().map((schema) => schema.name)).toContain("lcm_grep"); + const session = ctx.sessions.create(SessionId("live-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "live notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const map = sidecarMap(dir); + expect(map.has("live-1")).toBe(true); + expect(map.get("live-1")?.allMessages().length).toBe(session.deriveMessages().length); + const definition = lcmCommandDefinition(ctx, { + stores: () => map, + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sessionId}.sqlite`), + }); + const status = await definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId("cmd-status"), + rawInput: "status", + attachments: [], + }); + expect(status.kind).toBe("success"); + if (status.kind !== "success") throw new Error("status did not succeed"); + expect(status.text).toContain("live-1"); + } finally { + dispose(); + } + }); + + it("recall reads without creating an empty sidecar", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const grep = ctx.tools.get("lcm_grep"); + expect(grep).toBeDefined(); + const ghostGrep = { + agent: { session: { id: "ghost-recall" } }, + signal: new AbortController().signal, + } as unknown as ToolRunContext; + const grepAnswer = await grep?.execute({ query: "anything" }, ghostGrep); + expect(grepAnswer).toContain("no stored history"); + expect(existsSync(join(dir, "ghost-recall.sqlite"))).toBe(false); + const expandQuery = ctx.tools.get("lcm_expand_query"); + expect(expandQuery).toBeDefined(); + const ghostExpand = { + agent: { + session: { id: "ghost-expand" }, + options: { provider: ROUTE.provider, model: ROUTE.model }, + }, + signal: new AbortController().signal, + } as unknown as ToolRunContext; + await expandQuery?.execute({ query: "anything" }, ghostExpand); + expect(existsSync(join(dir, "ghost-expand.sqlite"))).toBe(false); + } finally { + dispose(); + } + }); + + it("dispose closes every open sidecar and stays safe to run twice", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + const originalClose = Reflect.get(LcmStore.prototype, "close") as (this: LcmStore) => void; + let closes = 0; + LcmStore.prototype.close = function (this: LcmStore): void { + closes += 1; + Reflect.apply(originalClose, this, []); + }; + try { + const session = ctx.sessions.create(SessionId("close-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "close me ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const path = join(dir, "close-1.sqlite"); + expect(existsSync(path)).toBe(true); + const mirrored = session.deriveMessages().length; + expect(mirrored).toBeGreaterThan(0); + dispose(); + expect(closes).toBeGreaterThan(0); + dispose(); + } finally { + LcmStore.prototype.close = originalClose; + } + const reopened = new LcmStore(join(dir, "close-1.sqlite")); + try { + const session = ctx.sessions.get(SessionId("close-1")); + expect(reopened.allMessages().length).toBe(session?.deriveMessages().length); + } finally { + reopened.close(); + } + }); + + it("dispose skips a store with a live run and still closes idle stores", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + const originalClose = Reflect.get(LcmStore.prototype, "close") as (this: LcmStore) => void; + const originalHasLiveRun = Reflect.get(LcmStore.prototype, "hasLiveRun") as ( + this: LcmStore, + ) => boolean; + const liveByStore = new Map(); + const closedStores = new Set(); + LcmStore.prototype.hasLiveRun = function (this: LcmStore): boolean { + const live = Reflect.apply(originalHasLiveRun, this, []) as boolean; + liveByStore.set(this, live); + return live; + }; + LcmStore.prototype.close = function (this: LcmStore): void { + closedStores.add(this); + Reflect.apply(originalClose, this, []); + }; + let holder: LcmStore | undefined; + let token = ""; + try { + const live = ctx.sessions.create(SessionId("live-run"), { meta: { cwd: dir } }); + appendConversation(live, ROUTE, "live run ".repeat(20)); + live.append("turn/start", { turn: 3 }); + const idle = ctx.sessions.create(SessionId("idle-run"), { meta: { cwd: dir } }); + appendConversation(idle, ROUTE, "idle notes ".repeat(20)); + idle.append("turn/start", { turn: 3 }); + holder = new LcmStore(join(dir, "live-run.sqlite")); + token = holder.openRun({ session: "live-run" }).token; + dispose(); + expect(liveByStore.size).toBe(2); + expect([...liveByStore.values()].filter(Boolean).length).toBe(1); + expect(closedStores.size).toBe(1); + for (const [store, isLive] of liveByStore) { + expect(closedStores.has(store)).toBe(!isLive); + } + } finally { + LcmStore.prototype.hasLiveRun = originalHasLiveRun; + LcmStore.prototype.close = originalClose; + if (holder !== undefined) { + if (token !== "") holder.abandonRun(token); + holder.close(); + } + dispose(); + } + }); + + it("dispose stays safe to run twice and reopens after a live run drains", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + const holderPath = join(dir, "drain-run.sqlite"); + let holder: LcmStore | undefined; + let token = ""; + try { + const session = ctx.sessions.create(SessionId("drain-run"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "drain me ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const mirrored = session.deriveMessages().length; + expect(mirrored).toBeGreaterThan(0); + holder = new LcmStore(holderPath); + token = holder.openRun({ session: "drain-run" }).token; + dispose(); + holder.abandonRun(token); + token = ""; + expect(() => dispose()).not.toThrow(); + const reopened = new LcmStore(holderPath); + try { + expect(reopened.allMessages().length).toBe(session.deriveMessages().length); + } finally { + reopened.close(); + } + } finally { + if (holder !== undefined) { + if (token !== "") holder.abandonRun(token); + holder.close(); + } + dispose(); + } + }); + + it("forces a manual compaction through /lcm compact", async () => { + const { ctx } = setupContext( + ROUTE, + [textChunks(SUMMARY), textChunks(SUMMARY), textChunks(SUMMARY)], + { contextWindow: 1_000_000 }, + ); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const session = ctx.sessions.create(SessionId("live-2"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "compact me ".repeat(30)); + session.append("turn/start", { turn: 3 }); + // Manual compaction runs on an idle session: close the turn first. + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + const before = session.deriveMessages().length; + const definition = lcmCommandDefinition(ctx, { + stores: () => sidecarMap(dir), + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sessionId}.sqlite`), + }); + const result = await definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId("cmd-compact"), + rawInput: "compact", + attachments: [], + }); + if (result.kind !== "success") throw new Error(`compact failed: ${result.text}`); + expect(result.text).toMatch(/Compacted \d+ history items/); + expect(session.deriveMessages().length).toBeLessThan(before); + const logSummaries = session + .snapshotEvents() + .filter( + (event): event is Extract => + event.type === "compaction/summary", + ); + expect(logSummaries.length).toBe(1); + expect(new Set(logSummaries.map((summary) => summary.data.compactionId)).size).toBe(1); + const summaries = sidecarMap(dir).get("live-2")?.allSummaries() ?? []; + expect(summaries.length).toBeGreaterThan(0); + } finally { + dispose(); + } + }); + + it("carries the checkpoint across two manual compactions", async () => { + const { ctx } = setupContext( + ROUTE, + [textChunks(SUMMARY), textChunks(SUMMARY), textChunks(SUMMARY)], + { contextWindow: 1_000_000 }, + ); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir, summaryTokens: 40 }); + try { + const session = ctx.sessions.create(SessionId("live-4"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "carryover work ".repeat(30)); + session.append("turn/start", { turn: 3 }); + session.append("turn/end", { turn: 3, reason: { kind: "completed" } }); + const definition = lcmCommandDefinition(ctx, { + stores: () => sidecarMap(dir), + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sessionId}.sqlite`), + }); + const compact = (id: string) => + definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId(id), + rawInput: "compact", + attachments: [], + }); + const first = await compact("cmd-first"); + if (first.kind !== "success") throw new Error(`first compact failed: ${first.text}`); + session.append("turn/start", { turn: 4 }); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `more work ${".".repeat(600)}` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("turn/end", { turn: 4, reason: { kind: "completed" } }); + const second = await compact("cmd-second"); + if (second.kind !== "success") throw new Error(`second compact failed: ${second.text}`); + const texts = (sidecarMap(dir).get("live-4")?.allMessages() ?? []) + .map((row) => row.text) + .filter((text) => text.includes(SUMMARY)); + const bodyOf = (text: string): string => { + const open = ""; + const close = ""; + const start = text.indexOf(open); + const end = text.indexOf(close); + if (start === -1 || end <= start) throw new Error("checkpoint frame not found"); + return text.slice(start + open.length, end).trim(); + }; + const bodies = texts.map(bodyOf); + const carryover = bodies.some((outer) => + bodies.some((inner) => outer !== inner && outer.includes(inner)), + ); + expect(carryover).toBe(true); + } finally { + dispose(); + } + }); + + it("maps every manual compaction failure code to its explained error", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const session = ctx.sessions.create(SessionId("live-codes"), { meta: { cwd: dir } }); + const cases = [ + ["busy", "has an active compaction"], + ["cancelled", "Compaction cancelled."], + ["changed", "changed before it could be replaced"], + ["summary", "could not produce a useful summary"], + ["commit", "did not finish cleanly"], + ["persistence", "could not be saved"], + ] as const; + for (const [code, phrase] of cases) { + (ctx as unknown as { compaction: unknown }).compaction = { + compactNow: async () => { + throw new ManualCompactionError(code, `test ${code}`); + }, + }; + const definition = lcmCommandDefinition(ctx, { + stores: () => new Map(), + sidecarDir: dir, + storePathFor: (sessionId) => join(dir, `${sessionId}.sqlite`), + }); + const result = await definition.handler({ + agent: fakeAgent(session, ROUTE), + signal: new AbortController().signal, + commandId: CommandId("cmd-compact-codes"), + rawInput: "compact", + attachments: [], + }); + expect(result.kind).toBe("error"); + if (result.kind === "error") expect(result.text).toContain(phrase); + } + }); + + it("leaves a quiet session untouched on pressure", async () => { + const { ctx } = setupContext(ROUTE, [textChunks(SUMMARY)], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const session = ctx.sessions.create(SessionId("live-3"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "quiet notes"); + session.append("turn/start", { turn: 3 }); + const eventsBefore = session.snapshotEvents().length; + const result = await ctx.compaction.compactIfNeeded( + fakeAgent(session, ROUTE), + "pressure", + new AbortController().signal, + ); + expect(result).toBeNull(); + expect(session.snapshotEvents().length).toBe(eventsBefore); + } finally { + dispose(); + } + }); + + it("post-dispose engine reads absence instead of closed handles", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const session = ctx.sessions.create(SessionId("evict-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "evict me ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const engine = ctx.compaction as LosslessSummarizerEngine; + const deps = ( + engine as unknown as { + lossless: { + store: (id: string) => LcmStore | undefined; + index: (id: string) => unknown; + memory: { previous: (id: string) => string | undefined }; + }; + } + ).lossless; + expect(deps.store("evict-1")).toBeDefined(); + dispose(); + expect(deps.store("evict-1")).toBeUndefined(); + expect(deps.index("evict-1")).toBeUndefined(); + expect(() => deps.memory.previous("evict-1")).not.toThrow(); + expect(deps.memory.previous("evict-1")).toBeUndefined(); + } finally { + dispose(); + } + }); + + it("post-dispose summarize reports no-sidecar-store", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + try { + const session = ctx.sessions.create(SessionId("evict-2"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "evict me ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const engine = ctx.compaction as LosslessSummarizerEngine; + const summarize = ( + engine as unknown as { + summarize: (input: { messages: readonly unknown[] }, agent: unknown) => Promise; + } + ).summarize.bind(engine); + dispose(); + await expect(summarize({ messages: [] }, fakeAgent(session, ROUTE))).rejects.toThrow( + "no sidecar store", + ); + } finally { + dispose(); + } + }); + + it("double dispose stays absent and safe after eviction", async () => { + const { ctx } = setupContext(ROUTE, [], { contextWindow: 1_000_000 }); + await mountBase(ctx); + const dir = tempDir(); + const dispose = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + const session = ctx.sessions.create(SessionId("evict-3"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "evict me ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const engine = ctx.compaction as LosslessSummarizerEngine; + const deps = ( + engine as unknown as { lossless: { store: (id: string) => LcmStore | undefined } } + ).lossless; + dispose(); + expect(deps.store("evict-3")).toBeUndefined(); + expect(() => dispose()).not.toThrow(); + expect(deps.store("evict-3")).toBeUndefined(); + }); + + it("SidecarStores.clear evicts without closing or deleting files", () => { + const dir = tempDir(); + const caches = new SidecarStores({ dir }); + const store = caches.for("unit-1"); + store.insertMessage({ + entryId: "unit-1:1", + role: "user", + text: "hello", + tokens: 1, + timestamp: Date.now(), + }); + expect(caches.peek("unit-1")).toBe(store); + expect(caches.ids()).toContain("unit-1"); + const path = caches.pathFor("unit-1"); + caches.clear(); + expect(caches.peek("unit-1")).toBeUndefined(); + expect(caches.ids()).toEqual([]); + expect(existsSync(path)).toBe(true); + expect(() => store.allMessages()).not.toThrow(); + store.close(); + }); + + it("SidecarStores.evict forgets one entry and keeps the rest open", () => { + const dir = tempDir(); + const caches = new SidecarStores({ dir }); + const kept = caches.for("unit-keep"); + const dropped = caches.for("unit-drop"); + const droppedPath = caches.pathFor("unit-drop"); + dropped.close(); + caches.evict("unit-drop"); + expect(caches.peek("unit-drop")).toBeUndefined(); + expect(caches.peek("unit-keep")).toBe(kept); + expect(caches.ids()).toEqual(["unit-keep"]); + expect(existsSync(droppedPath)).toBe(true); + kept.close(); + }); +}); diff --git a/packages/deepseek/test/preset.test.ts b/packages/deepseek/test/preset.test.ts new file mode 100644 index 0000000..c633576 --- /dev/null +++ b/packages/deepseek/test/preset.test.ts @@ -0,0 +1,270 @@ +import { existsSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import CommandRuntime from "@deepseek-ai/dsh-commands"; +import type { StreamChunk } from "@deepseek-ai/dsh-llm"; +import SessionStore, { SessionId } from "@deepseek-ai/dsh-session"; +import SessionProjectionRegistry from "@deepseek-ai/dsh-session-projection"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import TokenMeter from "@deepseek-ai/dsh-token-meter"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { LcmStore } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { LosslessSummarizerEngine } from "../src/engine.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { applyLossless } from "../src/plugin.ts"; +import { apply as applyPreset, Config, inject, name } from "../src/preset.ts"; +import { appendConversation } from "./support/conversation.ts"; +import { setupContext, textChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; +const SUMMARY = "Checkpoint: engine work started and the config was read."; + +function tempDir(): string { + return mkdtempSync(join(tmpdir(), "deepseek-preset-")); +} + +/** A realm-like context: the services the preset backend reads, without the + * host registries it must not touch. */ +function realmContext(route: DshRoute, scripts: StreamChunk[][] = []) { + const { ctx } = setupContext(route, scripts, { contextWindow: 1_000_000 }); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); + return ctx; +} + +/** A host-like context: the full registries the core composition mounts. */ +async function hostContext( + route: DshRoute, + scripts: StreamChunk[][], + dir: string, +): Promise<{ ctx: Context; stop: () => void }> { + const { ctx } = setupContext(route, scripts, { contextWindow: 1_000_000 }); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + void new CommandRuntime(ctx); + void new SessionStore(ctx); + new SessionProjectionRegistry(ctx); + void new TokenMeter(ctx); + const stop = applyLossless(ctx, { enabled: true, sidecarDir: dir }); + return { ctx, stop }; +} + +// White-box by design: the engine keeps its deps in a private `lossless` +// field, and these tests pin the cross-entry sharing contract through it. +// If the field moves, update this helper, not the assertions. +function engineDepsOf(engine: LosslessSummarizerEngine): { + store: (id: string) => unknown; +} { + return (engine as unknown as { lossless: { store: (id: string) => unknown } }).lossless; +} + +describe("preset backend", () => { + it("exposes the realm entrypoint shape the loader applies", () => { + expect(name).toBe("deepseek-lossless-preset"); + for (const service of ["sessions", "llm", "tokenMeter"]) { + expect(inject).toContain(service); + } + expect(inject).not.toContain("compaction"); + expect(inject).not.toContain("tools"); + expect(inject).not.toContain("commands"); + expect(inject).not.toContain("sessionQuery"); + expect(Config).toBeDefined(); + expect(typeof applyPreset).toBe("function"); + }); + + it("registers nothing until enabled", () => { + const ctx = realmContext(ROUTE); + const dir = join(tempDir(), "never-created"); + const dispose = applyPreset(ctx, { sidecarDir: dir }); + expect((ctx as unknown as { compaction?: unknown }).compaction).toBeUndefined(); + expect(existsSync(dir)).toBe(false); + dispose(); + }); + + it("fails fast without the host core instead of running split-brain", () => { + const ctx = realmContext(ROUTE); + expect(() => applyPreset(ctx, { enabled: true, sidecarDir: tempDir() })).toThrow( + /no shared core.*host row/, + ); + }); + + it("shares the loader schema with the host row", async () => { + const { Config: hostConfig } = await import("../src/plugin.ts"); + expect(Config).toBe(hostConfig); + }); + + it("provides the realm backend against the shared host core", async () => { + const dir = tempDir(); + const { ctx: host, stop: stopHost } = await hostContext(ROUTE, [], dir); + const realm = realmContext(ROUTE); + try { + // Backend-only: the realm never gets the recall registries, so there + // is no tools service to even query here. + expect((realm as unknown as { tools?: unknown }).tools).toBeUndefined(); + expect((realm as unknown as { commands?: unknown }).commands).toBeUndefined(); + const stopPreset = applyPreset(realm, { enabled: true, sidecarDir: dir }); + try { + expect((realm as unknown as { compaction?: unknown }).compaction).toBeInstanceOf( + LosslessSummarizerEngine, + ); + expect((host as unknown as { compaction?: unknown }).compaction).toBeInstanceOf( + LosslessSummarizerEngine, + ); + // Still backend-only after mount: no recall tools, no command. + expect((realm as unknown as { tools?: unknown }).tools).toBeUndefined(); + expect((realm as unknown as { commands?: unknown }).commands).toBeUndefined(); + } finally { + stopPreset(); + } + } finally { + stopHost(); + } + }); + + it("leaves shared handles open while the host still holds the core", async () => { + const dir = tempDir(); + const { ctx: host, stop: stopHost } = await hostContext(ROUTE, [textChunks(SUMMARY)], dir); + try { + const session = host.sessions.create(SessionId("shared-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "shared notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const realm = realmContext(ROUTE); + const stopPreset = applyPreset(realm, { enabled: true, sidecarDir: dir }); + try { + const hostStore = engineDepsOf( + (host as unknown as { compaction: LosslessSummarizerEngine }).compaction, + ).store("shared-1") as LcmStore; + const realmStore = engineDepsOf( + (realm as unknown as { compaction: LosslessSummarizerEngine }).compaction, + ).store("shared-1"); + expect(realmStore).toBeDefined(); + expect(realmStore).toBe(hostStore); + expect(hostStore.allMessages().length).toBeGreaterThan(0); + } finally { + stopPreset(); + } + // The preset disposer leaves shared handles open and readable. + const kept = engineDepsOf( + (host as unknown as { compaction: LosslessSummarizerEngine }).compaction, + ).store("shared-1") as LcmStore; + expect(kept.allMessages().length).toBeGreaterThan(0); + } finally { + stopHost(); + } + }); + + it("holds the core while a realm borrows it, and drops it after", async () => { + const dir = tempDir(); + const { ctx: host, stop: stopHost } = await hostContext(ROUTE, [textChunks(SUMMARY)], dir); + const session = host.sessions.create(SessionId("borrowed-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "borrowed notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const realm = realmContext(ROUTE); + const stopPreset = applyPreset(realm, { enabled: true, sidecarDir: dir }); + try { + stopHost(); + // The host dispose released its ownership but skipped every shared + // close for the live borrower, so the realm still reads. + const kept = engineDepsOf( + (realm as unknown as { compaction: LosslessSummarizerEngine }).compaction, + ).store("borrowed-1") as LcmStore; + expect(kept.allMessages().length).toBeGreaterThan(0); + // The entry stays for the borrower: another realm mount still finds it. + const realm2 = realmContext(ROUTE); + const stopPreset2 = applyPreset(realm2, { enabled: true, sidecarDir: dir }); + stopPreset2(); + } finally { + stopPreset(); + } + }); + + it("ignores a stale disposer after a remount on the same directory", async () => { + const dir = tempDir(); + const first = await hostContext(ROUTE, [], dir); + first.stop(); + const second = await hostContext(ROUTE, [], dir); + first.stop(); + // The replacement host still serves: the stale dispose changed nothing. + const session = second.ctx.sessions.create(SessionId("remount-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "remount notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const realm = realmContext(ROUTE); + try { + expect(() => applyPreset(realm, { enabled: true, sidecarDir: dir })).not.toThrow(); + const store = engineDepsOf( + (second.ctx as unknown as { compaction: LosslessSummarizerEngine }).compaction, + ).store("remount-1") as LcmStore; + expect(store.allMessages().length).toBeGreaterThan(0); + } finally { + second.stop(); + } + }); + + it("closes idle shared stores when the preset release is the last one", async () => { + const dir = tempDir(); + const { ctx: host, stop: stopHost } = await hostContext(ROUTE, [textChunks(SUMMARY)], dir); + const session = host.sessions.create(SessionId("shared-last-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "shared last ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const realm = realmContext(ROUTE); + const stopPreset = applyPreset(realm, { enabled: true, sidecarDir: dir }); + const originalClose = Reflect.get(LcmStore.prototype, "close") as (this: LcmStore) => void; + let closes = 0; + Reflect.set(LcmStore.prototype, "close", function (this: LcmStore): void { + closes += 1; + Reflect.apply(originalClose, this, []); + }); + try { + stopHost(); + expect(closes).toBe(0); + stopPreset(); + expect(closes).toBeGreaterThan(0); + } finally { + Reflect.set(LcmStore.prototype, "close", originalClose); + } + }); + + it("rejects a preset sidecar directory that differs from the host core", async () => { + const { stop: stopHost } = await hostContext(ROUTE, [], tempDir()); + const realm = realmContext(ROUTE); + try { + expect(() => applyPreset(realm, { enabled: true, sidecarDir: tempDir() })).toThrow( + /no shared core/, + ); + } finally { + stopHost(); + } + }); + + it("closes idle stores on the last host dispose", async () => { + const dir = tempDir(); + const { ctx: host, stop: stopHost } = await hostContext(ROUTE, [], dir); + const session = host.sessions.create(SessionId("owned-1"), { meta: { cwd: dir } }); + appendConversation(session, ROUTE, "owned notes ".repeat(20)); + session.append("turn/start", { turn: 3 }); + const originalClose = Reflect.get(LcmStore.prototype, "close") as (this: LcmStore) => void; + let closes = 0; + Reflect.set(LcmStore.prototype, "close", function (this: LcmStore): void { + closes += 1; + Reflect.apply(originalClose, this, []); + }); + try { + stopHost(); + expect(closes).toBeGreaterThan(0); + } finally { + Reflect.set(LcmStore.prototype, "close", originalClose); + } + // Closed cleanly and the data survived for reopen. + const reopened = new LcmStore(join(dir, "owned-1.sqlite")); + try { + expect(reopened.allMessages().length).toBeGreaterThan(0); + } finally { + reopened.close(); + } + }); +}); diff --git a/packages/deepseek/test/publish-surface.test.ts b/packages/deepseek/test/publish-surface.test.ts new file mode 100644 index 0000000..e1006d2 --- /dev/null +++ b/packages/deepseek/test/publish-surface.test.ts @@ -0,0 +1,91 @@ +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { test } from "vite-plus/test"; + +const PACKAGE_DIR = fileURLToPath(new URL("..", import.meta.url)); +const REPO_ROOT = fileURLToPath(new URL("../../..", import.meta.url)); + +interface Manifest { + name?: string; + main?: string; + files?: string[]; + exports?: Record; + repository?: { directory?: string }; + icon?: string; + dsh?: { bundle?: { patch?: string } }; +} + +const manifest = JSON.parse(readFileSync(join(PACKAGE_DIR, "package.json"), "utf8")) as Manifest; + +test("publish surface: the tarball carries the build, the changelog, and the skill", () => { + assert.equal(manifest.name, "deepseek-lossless"); + // The changelog starts at the first published version: nothing ships yet, + // so no changeset has ever cut one. npm always includes README and LICENSE + // from the package directory, so files names only what npm must be told. + assert.deepEqual((manifest.files ?? []).toSorted(), [ + "CHANGELOG.md", + "cordis.patch.yml", + "dist", + "docs/DETAIL.md", + "icon.svg", + "locale/*.json", + "skills", + ]); + const root = manifest.exports?.["."] as { types?: string; default?: string } | undefined; + assert.equal(root?.default, manifest.main); + assert.equal(root?.types, manifest.main?.replace(/\.mjs$/, ".d.mts")); + const preset = manifest.exports?.["./preset"] as { types?: string; default?: string } | undefined; + assert.equal(preset?.default, "./dist/preset.mjs"); + assert.equal(preset?.types, "./dist/preset.d.mts"); + assert.equal(manifest.repository?.directory, "packages/deepseek"); + for (const name of ["README.md", "LICENSE", "CHANGELOG.md"]) { + assert.ok(existsSync(join(PACKAGE_DIR, name)), `${name} is not in the package`); + } + assert.ok(existsSync(join(PACKAGE_DIR, "skills", "deepseek-lossless", "SKILL.md"))); + assert.deepEqual( + readFileSync(join(PACKAGE_DIR, "LICENSE")), + readFileSync(join(REPO_ROOT, "LICENSE")), + ); +}); + +test("publish surface: the bundle manifest resolves through the profile composer", () => { + assert.equal(manifest.icon, "./icon.svg"); + assert.equal(manifest.dsh?.bundle?.patch, "./cordis.patch.yml"); + for (const subpath of ["./package.json", "./cordis.patch.yml", "./locale/*.json", "./preset"]) { + assert.ok(manifest.exports?.[subpath] !== undefined, `${subpath} is exported`); + } + for (const name of ["cordis.patch.yml", "icon.svg", "locale/en.json", "docs/DETAIL.md"]) { + assert.ok(existsSync(join(PACKAGE_DIR, name)), `${name} ships in the tarball`); + } +}); + +test("publish surface: the host patch switches the backend on install", () => { + const patch = readFileSync(join(PACKAGE_DIR, "cordis.patch.yml"), "utf8"); + assert.ok(patch.includes("- id: compaction-basic"), "the base row is addressed"); + assert.ok(patch.includes("disabled: true"), "the base backend goes quiet on install"); + assert.ok(patch.includes("- id: deepseek-lossless"), "our row is inserted"); + assert.ok(patch.includes("enabled: true"), "our row mounts enabled"); +}); + +test("publish surface: preset realms live in the web companion package", () => { + assert.equal(manifest.dsh?.bundle?.patch, "./cordis.patch.yml"); + const patch = readFileSync(join(PACKAGE_DIR, "cordis.patch.yml"), "utf8"); + assert.ok(!patch.includes("preset-lossless"), "no preset rows in the host patch"); +}); + +test("publish surface: peer ranges admit the pinned prerelease hosts", () => { + // npm rejects prereleases under "*", so the dsh peers name the rc floor instead. + // dsh-session-projection and dsh-system-prompt stay dev-only on purpose: + // nothing under src/ imports them (only the test harness mounts them). + const peers = (manifest as { peerDependencies?: Record }).peerDependencies ?? {}; + for (const [name, range] of Object.entries(peers)) { + if (name === "@deepseek-ai/cordis") { + assert.equal(range, "*"); + } else { + assert.equal(range, "^0.2.1-rc.3"); + } + } +}); diff --git a/packages/deepseek/test/reader.test.ts b/packages/deepseek/test/reader.test.ts new file mode 100644 index 0000000..cbfddb1 --- /dev/null +++ b/packages/deepseek/test/reader.test.ts @@ -0,0 +1,81 @@ +import { LcmStore, ingestEntries, runRetrieval } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { dshRef } from "../src/model-host.ts"; +import type { DshRoute } from "../src/model-host.ts"; +import { dshReader } from "../src/reader.ts"; +import { setupContext, textChunks, toolCallChunks } from "./support/script-adapter.ts"; + +const ROUTE: DshRoute = { provider: "test-provider", model: "test-model" }; + +function seedStore(): LcmStore { + const store = new LcmStore(":memory:"); + ingestEntries( + store, + Array.from({ length: 8 }, (_, i) => ({ + entryId: `e${String(i).padStart(2, "0")}`, + role: (i % 2 === 0 ? "user" : "assistant") as "user" | "assistant", + text: `message ${i}: deciding the ${i % 2 === 0 ? "compaction engine" : "recall pipeline"} rollout ${"detail ".repeat(20)}`, + timestamp: i, + })), + ); + return store; +} + +describe("dshReader retrieval", () => { + it("answers one query through a real grep call and a real store", async () => { + const store = seedStore(); + const { ctx, adapter } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "compaction" }), + textChunks("The rollout decision is recorded under the compaction engine entries."), + ]); + const run = await runRetrieval( + store, + { + query: "compaction engine rollout", + prompt: "What did we decide about the compaction engine rollout?", + budgetTokens: 10000, + maxSteps: 6, + redact: (text) => text, + }, + dshReader(ctx, { ref: dshRef(ROUTE) }), + ); + expect(run.kind).toBe("answered"); + if (run.kind !== "answered") throw new Error(`retrieval did not answer: ${run.kind}`); + expect(run.steps).toBe(2); + expect(run.text).toContain("compaction engine entries"); + expect(adapter.requests).toHaveLength(2); + const first = adapter.requests[0]; + const tools = first?.tools?.map((tool) => tool.name) ?? []; + expect(tools).toContain("lcm_grep"); + expect(tools).toContain("lcm_describe"); + expect(tools).toContain("lcm_expand"); + const second = adapter.requests[1]; + const roles = second?.messages.map((message) => message.role) ?? []; + expect(roles).toEqual(["user", "assistant", "tool"]); + }); + + it("stamps the calling session id on every step", async () => { + const store = seedStore(); + const { ctx, adapter } = setupContext(ROUTE, [ + toolCallChunks("call-1", "lcm_grep", { query: "compaction" }), + textChunks("The rollout decision is recorded under the compaction engine entries."), + ]); + const run = await runRetrieval( + store, + { + query: "compaction engine rollout", + prompt: "What did we decide about the compaction engine rollout?", + budgetTokens: 10000, + maxSteps: 6, + redact: (text) => text, + }, + dshReader(ctx, { ref: dshRef(ROUTE), sessionId: "session-9" }), + ); + expect(run.kind).toBe("answered"); + expect(adapter.requests.length).toBeGreaterThan(0); + for (const request of adapter.requests) { + expect(request).toMatchObject({ sessionId: "session-9" }); + } + }); +}); diff --git a/packages/deepseek/test/sidecar-dump.test.ts b/packages/deepseek/test/sidecar-dump.test.ts new file mode 100644 index 0000000..470aeb2 --- /dev/null +++ b/packages/deepseek/test/sidecar-dump.test.ts @@ -0,0 +1,47 @@ +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { LcmStore, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { dumpSidecar } from "../src/sidecar-dump.ts"; + +function seedStore(path: string): void { + const store = new LcmStore(path); + ingestEntries(store, [ + { + entryId: "s1:1", + role: "user", + text: "dump me", + timestamp: Date.now(), + }, + ]); + store.close(); +} + +describe("sidecar dump", () => { + it("reports a named skip for a missing sidecar", () => { + const result = dumpSidecar({ + path: join(mkdtempSync(join(tmpdir(), "dump-")), "missing.sqlite"), + sessionId: "missing", + redact: (text) => text, + }); + assert.equal(result.status, "skipped"); + if (result.status === "skipped") assert.equal(result.reason, "missing"); + }); + + it("dumps a stored session with metrics", () => { + const dir = mkdtempSync(join(tmpdir(), "dump-")); + const path = join(dir, "ses_dump.sqlite"); + seedStore(path); + const result = dumpSidecar({ + path, + sessionId: "ses_dump", + redact: (text) => text, + }); + expect(result.status).toBe("dumped"); + if (result.status === "dumped") expect(result.text).toContain("LCM sidecar dump: ses_dump"); + }); +}); diff --git a/packages/deepseek/test/support/agent-fake.ts b/packages/deepseek/test/support/agent-fake.ts new file mode 100644 index 0000000..4e6c50f --- /dev/null +++ b/packages/deepseek/test/support/agent-fake.ts @@ -0,0 +1,17 @@ +import type { Agent } from "@deepseek-ai/dsh-agent"; +import type { Session } from "@deepseek-ai/dsh-session"; + +import type { DshRoute } from "../../src/model-host.ts"; + +/** compactNow and compactRegion read session, options, and runMaintenance. A + * real Agent needs the full runtime (inbox, dispatcher); the harness compacts + * its own suite the same way, with a minimal fake cast to the runtime type. */ +export function fakeAgent(session: Session, route: DshRoute): Agent { + return { + session, + options: { provider: route.provider, model: route.model }, + runMaintenance(task: (signal: AbortSignal) => Promise): Promise { + return task(new AbortController().signal); + }, + } as unknown as Agent; +} diff --git a/packages/deepseek/test/support/conversation.ts b/packages/deepseek/test/support/conversation.ts new file mode 100644 index 0000000..fce5fdc --- /dev/null +++ b/packages/deepseek/test/support/conversation.ts @@ -0,0 +1,71 @@ +import { createMessage, createSystemMessage, createUserMessage } from "@deepseek-ai/dsh-llm"; +import type { Session } from "@deepseek-ai/dsh-session"; + +import type { DshRoute } from "../../src/model-host.ts"; + +/** Two closed turns with a system head and a routed request header. Works on + * detached and live sessions alike; the caller opens the turn the engine + * writes into. */ +export function appendConversation(session: Session, route: DshRoute, text: string): void { + session.append("turn/start", { turn: 1 }); + session.append( + "system/message", + { turn: 1, step: 1, message: createSystemMessage("You are a test.") }, + { surfaceOp: "append" }, + ); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `${text} first request` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 1, step: 1 }); + session.append("request/header", { + header: { config: { provider: route.provider, model: route.model } }, + reason: "initial", + }); + session.append( + "assistant/message", + { + stream: [], + turn: 1, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: `${text} first response` }], + source: { kind: "model", provider: route.provider, model: route.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 1, step: 1 }); + session.append("turn/end", { turn: 1, reason: { kind: "completed" } }); + session.append("turn/start", { turn: 2 }); + session.append( + "user/message", + createUserMessage({ + content: [{ type: "text", text: `${text} second request` }], + source: { kind: "user" }, + }), + { surfaceOp: "append" }, + ); + session.append("step/start", { turn: 2, step: 1 }); + session.append( + "assistant/message", + { + stream: [], + turn: 2, + step: 1, + message: createMessage({ + role: "assistant", + content: [{ type: "text", text: `${text} second response` }], + source: { kind: "model", provider: route.provider, model: route.model }, + }), + }, + { surfaceOp: "append" }, + ); + session.append("step/end", { turn: 2, step: 1 }); + session.append("turn/end", { turn: 2, reason: { kind: "completed" } }); +} diff --git a/packages/deepseek/test/support/script-adapter.ts b/packages/deepseek/test/support/script-adapter.ts new file mode 100644 index 0000000..b506fb8 --- /dev/null +++ b/packages/deepseek/test/support/script-adapter.ts @@ -0,0 +1,89 @@ +import { Context } from "@deepseek-ai/cordis"; +import LlmRuntime, { LlmAdapter, ToolCallId } from "@deepseek-ai/dsh-llm"; +import type { GenerateOptions, StreamChunk } from "@deepseek-ai/dsh-llm"; + +import type { DshRoute } from "../../src/model-host.ts"; + +export function textChunks(text: string): StreamChunk[] { + return [ + { type: "block-start", index: 0, blockType: "text" }, + { type: "text-delta", index: 0, text }, + { type: "block-end", index: 0, block: { type: "text", text } }, + { type: "usage", usage: { inputTokens: 10, outputTokens: text.length } }, + { type: "finish", reason: { kind: "stop" } }, + ]; +} + +export function cappedChunks(text: string): StreamChunk[] { + return [ + { type: "block-start", index: 0, blockType: "text" }, + { type: "text-delta", index: 0, text }, + { type: "block-end", index: 0, block: { type: "text", text } }, + { type: "usage", usage: { inputTokens: 10, outputTokens: text.length } }, + { type: "finish", reason: { kind: "max-tokens" } }, + ]; +} + +export function errorChunks(): StreamChunk[] { + return [ + { + type: "finish", + reason: { kind: "error", failure: { message: "bad key", code: "AUTH" } }, + }, + ]; +} + +export function toolCallChunks(callId: string, name: string, args: unknown): StreamChunk[] { + const id = ToolCallId(callId); + const json = JSON.stringify(args); + return [ + { type: "block-start", index: 0, blockType: "tool-call" }, + { type: "tool-call-delta", index: 0, id, name, argumentsDelta: json }, + { type: "block-end", index: 0, block: { type: "tool-call", id, name, arguments: json } }, + { type: "usage", usage: { inputTokens: 10, outputTokens: 5 } }, + { type: "finish", reason: { kind: "tool-calls" } }, + ]; +} + +/** One scripted provider route: each stream consumes the next entry and + * records its request, so a test asserts what the host actually sent. */ +export class ScriptAdapter extends LlmAdapter { + readonly requests: GenerateOptions[] = []; + + constructor( + private readonly scripts: StreamChunk[][], + private readonly contextWindow?: number, + ) { + super(); + } + + override resolveModel(provider: string, model: string) { + return Promise.resolve({ + provider, + id: model, + name: model, + ...(this.contextWindow === undefined + ? {} + : { context: { contextWindow: this.contextWindow } }), + }); + } + + override async *stream(options: GenerateOptions): AsyncIterable { + this.requests.push(options); + const chunks = this.scripts.shift(); + if (!chunks) throw new Error("ScriptAdapter: script exhausted"); + yield* chunks; + } +} + +export function setupContext( + route: DshRoute, + scripts: StreamChunk[][], + opts?: { contextWindow?: number }, +): { ctx: Context; adapter: ScriptAdapter } { + const ctx = new Context(); + void new LlmRuntime(ctx); + const adapter = new ScriptAdapter(scripts, opts?.contextWindow); + ctx.llm.registerAdapter([route.provider], adapter); + return { ctx, adapter }; +} diff --git a/packages/deepseek/test/tools.test.ts b/packages/deepseek/test/tools.test.ts new file mode 100644 index 0000000..84579a9 --- /dev/null +++ b/packages/deepseek/test/tools.test.ts @@ -0,0 +1,509 @@ +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Context } from "@deepseek-ai/cordis"; +import SystemPrompt from "@deepseek-ai/dsh-system-prompt"; +import ToolRuntime from "@deepseek-ai/dsh-tools"; +import { LcmStore, SidecarStores, ingestEntries } from "lossless-core"; +import { describe, expect, it } from "vite-plus/test"; + +import { + answerGrep, + describeAnswer, + expandAnswer, + grepAnswer, + registerRecallTools, + searchPastSessions, + stopNote, +} from "../src/tools.ts"; + +function seedStore(): LcmStore { + const store = new LcmStore(":memory:"); + ingestEntries( + store, + Array.from({ length: 4 }, (_, i) => ({ + entryId: `e${i}`, + role: "user" as const, + text: `message ${i}: the ${i % 2 === 0 ? "compaction engine" : "recall pipeline"} decision ${"detail ".repeat(10)}`, + timestamp: i, + })), + ); + return store; +} + +const identity = (text: string): string => text; + +function seedPast(dir: string, sessionId: string, texts: readonly string[]): void { + const store = new LcmStore(join(dir, `${sessionId}.sqlite`)); + try { + ingestEntries( + store, + texts.map((text, index) => ({ + entryId: `${sessionId}:prompt:msg_${index + 1}`, + role: "user" as const, + text, + timestamp: index + 1, + })), + ); + } finally { + store.close(); + } +} + +function hitLines(text: string): string[] { + return text.split("\n").filter((line) => line.startsWith("[")); +} + +function seedSpan(): { store: LcmStore; leaf: number } { + const store = new LcmStore(":memory:"); + ingestEntries( + store, + Array.from({ length: 10 }, (_, i) => ({ + entryId: `a${i}`, + role: "user" as const, + text: `span message ${i} about the harbor relay`, + timestamp: i, + })), + ); + const messageIds = store.messagesInSpan("a0", "a9").map((message) => message.id); + const leaf = store.insertSummary({ + kind: "leaf", + text: "leaf over the whole span", + tokens: 4, + depth: 0, + firstEntryId: "a0", + lastEntryId: "a9", + messageIds, + }).id; + return { store, leaf }; +} + +describe("recall tools", () => { + it("registers three tools whose schemas reach the wire projection", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const store = seedStore(); + const dispose = registerRecallTools(ctx, { store: () => store, redact: identity }); + const names = ctx.tools.schemas().map((schema) => schema.name); + expect(names).toContain("lcm_grep"); + expect(names).toContain("lcm_describe"); + expect(names).toContain("lcm_expand"); + const grep = ctx.tools.schemas().find((schema) => schema.name === "lcm_grep"); + expect(Object.keys(grep?.parameters.properties ?? {})).toEqual( + expect.arrayContaining([ + "query", + "pattern", + "case_sensitive", + "offset", + "after", + "limit", + "summary_id", + "include_removed", + "scope", + ]), + ); + const expand = ctx.tools.schemas().find((schema) => schema.name === "lcm_expand"); + expect(Object.keys(expand?.parameters.properties ?? {})).toContain("session"); + dispose(); + expect(ctx.tools.schemas().map((schema) => schema.name)).not.toContain("lcm_grep"); + }); + + it("answers grep, describe, and expand from a real store", () => { + const store = seedStore(); + const grep = grepAnswer(store, identity, { query: "compaction" }); + expect(grep).toContain("Found 2 match(es):"); + expect(grep).toContain("[e0]"); + expect(grepAnswer(undefined, identity, { query: "compaction" })).toContain("no stored history"); + expect(describeAnswer(store, identity, { id: 999 })).toBe("No summary with id 999."); + const expanded = expandAnswer(store, identity, { entry_id: "e1" }); + expect(expanded).toContain("recall pipeline"); + expect(expandAnswer(store, identity, {})).toContain("needs an id"); + }); + + it("greps with a pattern, case sensitivity, and an offset", () => { + const store = seedStore(); + const insensitive = grepAnswer(store, identity, { pattern: "MESSAGE" }); + expect(insensitive).toContain("[e0 (user)]"); + expect(insensitive).toContain("[e3 (user)]"); + const sensitiveMiss = grepAnswer(store, identity, { + pattern: "MESSAGE", + case_sensitive: true, + }); + expect(sensitiveMiss).toContain("No matches for pattern"); + const paged = grepAnswer(store, identity, { pattern: "MESSAGE", offset: 1 }); + expect(paged).not.toContain("[e0 (user)]"); + expect(paged).toContain("[e1 (user)]"); + }); + + it("scopes grep to one summary id", () => { + const store = seedStore(); + const messageIds = store.messagesInSpan("e0", "e1").map((message) => message.id); + const summaryId = store.insertSummary({ + kind: "leaf", + text: "leaf over the first two messages", + tokens: 4, + depth: 0, + firstEntryId: "e0", + lastEntryId: "e1", + messageIds, + }).id; + const scoped = grepAnswer(store, identity, { query: "compaction", summary_id: summaryId }); + expect(scoped).toContain("[e0]"); + expect(scoped).not.toContain("[e2]"); + }); + + it("hides removed branch hits unless include_removed is set", () => { + const store = seedStore(); + store.setRemoved(["e1"], 100); + const hidden = grepAnswer(store, identity, { query: "recall pipeline" }); + expect(hidden).toContain("[e3]"); + expect(hidden).not.toContain("[e1]"); + const shown = grepAnswer(store, identity, { + query: "recall pipeline", + include_removed: true, + }); + expect(shown).toContain("[e1]"); + expect(shown).toContain("[removed]"); + expect(shown).toContain("[e3]"); + }); + + it("names the caller limit in the hit-cap note", () => { + const dir = mkdtempSync(join(tmpdir(), "deepseek-lossless-hitcap-")); + seedPast( + dir, + "ses_a", + Array.from({ length: 12 }, () => "narwhal sighted near the floe edge"), + ); + seedPast( + dir, + "ses_b", + Array.from({ length: 12 }, () => "narwhal sighted near the floe edge"), + ); + const sidecars = new SidecarStores({ dir }); + try { + const result = searchPastSessions( + { + past: { list: (exclude?: string) => sidecars.list(exclude) }, + redact: identity, + now: () => 0, + }, + "ses_current", + "narwhal", + { limit: 5 }, + ); + expect(result.hits).toBe(5); + expect(result.stoppedBy).toBe("hits"); + expect(result.text).toContain("5-hit cap"); + expect(result.text).not.toContain("40-hit cap"); + expect(stopNote("hits", 5)).toContain("5-hit cap"); + expect(stopNote("hits")).toContain("40-hit cap"); + } finally { + sidecars.close(); + } + }); + + it("clamps a session grep limit to at least one hit", () => { + const store = seedStore(); + try { + for (const limit of [0, -4, 1.7]) { + const rows: Record[] = []; + const text = grepAnswer( + store, + identity, + { query: "message", limit }, + { record: (row) => rows.push(row) }, + ); + expect(hitLines(text)).toHaveLength(1); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ tool: "lcm_grep", outcome: "hit", hits: 1 }); + } + const floored = grepAnswer(store, identity, { query: "message", limit: 2.7 }); + expect(hitLines(floored)).toHaveLength(2); + } finally { + store.close(); + } + }); + + it("treats a quoted query as a phrase miss and still counts unreadable stores", () => { + const dir = mkdtempSync(join(tmpdir(), "deepseek-lossless-rejected-")); + seedPast(dir, "ses_past", ["hello world"]); + const sidecars = new SidecarStores({ dir }); + try { + const past = { list: (exclude?: string) => sidecars.list(exclude) }; + // Embedded quotes phrase-escape instead of failing: a miss, scanned. + const result = searchPastSessions( + { past, redact: identity, now: () => 0 }, + "ses_current", + `foo"bar`, + ); + expect(result.scanned).toBe(1); + expect(result.skipped).toBe(0); + expect(result.text).toContain("No matches"); + const rows: Record[] = []; + const text = answerGrep( + { + store: () => undefined, + redact: identity, + record: (row) => rows.push(row), + pastSessions: past, + }, + "ses_current", + { query: `foo"bar`, scope: "sessions" }, + ); + expect(text).toContain("No matches"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ tool: "lcm_grep", outcome: "miss", sessionsScanned: 1 }); + // A sidecar that cannot be opened still counts skipped, not scanned. + const ghost = searchPastSessions( + { + past: { list: () => [{ id: "ghost", path: join(dir, "no-such-dir", "ghost.sqlite") }] }, + redact: identity, + now: () => 0, + }, + "ses_current", + "hello", + ); + expect(ghost.skipped).toBe(1); + expect(ghost.scanned).toBe(0); + } finally { + sidecars.close(); + } + }); + + it("records recall rows for session grep outcomes", () => { + const store = seedStore(); + try { + const hitRows: Record[] = []; + expect( + grepAnswer( + store, + identity, + { query: "compaction" }, + { record: (row) => hitRows.push(row) }, + ), + ).toContain("Found 2 match(es):"); + expect(hitRows).toHaveLength(1); + expect(hitRows[0]).toMatchObject({ + tool: "lcm_grep", + outcome: "hit", + hits: 2, + queryChars: 10, + }); + + const missRows: Record[] = []; + expect( + grepAnswer( + store, + identity, + { query: "no-such-token-xyz" }, + { record: (row) => missRows.push(row) }, + ), + ).toContain("No matches"); + expect(missRows).toHaveLength(1); + expect(missRows[0]).toMatchObject({ tool: "lcm_grep", outcome: "miss", hits: 0 }); + + const missingRows: Record[] = []; + expect(grepAnswer(store, identity, {}, { record: (row) => missingRows.push(row) })).toContain( + "needs a query string", + ); + expect(missingRows).toHaveLength(1); + expect(missingRows[0]).toMatchObject({ tool: "lcm_grep", outcome: "error", queryChars: 0 }); + + const refusedRows: Record[] = []; + expect( + grepAnswer( + store, + identity, + { query: "compaction", scope: "bogus" }, + { record: (row) => refusedRows.push(row) }, + ), + ).toContain("must be session"); + expect(refusedRows).toHaveLength(1); + expect(refusedRows[0]).toMatchObject({ + tool: "lcm_grep", + outcome: "refused", + scope: "bogus", + }); + + const bothRows: Record[] = []; + expect( + grepAnswer( + store, + identity, + { query: "compaction", pattern: "foo" }, + { record: (row) => bothRows.push(row) }, + ), + ).toContain("not both"); + expect(bothRows).toHaveLength(1); + expect(bothRows[0]).toMatchObject({ tool: "lcm_grep", outcome: "error", hits: 0 }); + + const phraseRows: Record[] = []; + expect( + grepAnswer( + store, + identity, + { query: `foo"bar` }, + { record: (row) => phraseRows.push(row) }, + ), + ).toContain("No matches"); + expect(phraseRows).toHaveLength(1); + expect(phraseRows[0]).toMatchObject({ tool: "lcm_grep", outcome: "miss", hits: 0 }); + } finally { + store.close(); + } + }); + + it("forwards after and limit through session grep", () => { + const store = seedStore(); + try { + const refused = grepAnswer(store, identity, { query: "message", after: "e0" }); + expect(refused).toContain("after resumes a pattern scan"); + + const resumed = grepAnswer(store, identity, { pattern: "message", after: "e1" }); + expect(resumed).toContain("[e2 (user)]"); + expect(resumed).toContain("[e3 (user)]"); + expect(resumed).not.toContain("[e0 (user)]"); + expect(resumed).not.toContain("[e1 (user)]"); + + const paged = grepAnswer(store, identity, { pattern: "message", after: "e0", limit: 1 }); + expect(paged).toContain("[e1 (user)]"); + expect(paged).not.toContain("[e2 (user)]"); + + const first = grepAnswer(store, identity, { query: "message", limit: 1 }); + expect(first).toContain("1..1 of 4"); + expect(hitLines(first)).toHaveLength(1); + const second = grepAnswer(store, identity, { query: "message", limit: 1, offset: 1 }); + expect(second).toContain("2..2 of 4"); + expect(hitLines(second)).toHaveLength(1); + expect(second).not.toBe(first); + } finally { + store.close(); + } + }); + + it("exposes role, date, tool, and anchor params in the tool schemas", async () => { + const ctx = new Context(); + await ctx.plugin(SystemPrompt, {}); + void new ToolRuntime(ctx); + const store = seedStore(); + const dispose = registerRecallTools(ctx, { store: () => store, redact: identity }); + try { + const schemas = ctx.tools.schemas(); + const grep = schemas.find((schema) => schema.name === "lcm_grep"); + expect(Object.keys(grep?.parameters.properties ?? {})).toEqual( + expect.arrayContaining(["role", "date_start", "date_end", "tool_name"]), + ); + const expand = schemas.find((schema) => schema.name === "lcm_expand"); + expect(Object.keys(expand?.parameters.properties ?? {})).toContain("anchor"); + } finally { + dispose(); + store.close(); + } + }); + + it("forwards role, date, and tool filters through session grep", () => { + const store = new LcmStore(":memory:"); + try { + ingestEntries(store, [ + { entryId: "f0", role: "user", text: "zephyr routing decision", timestamp: 100 }, + { + entryId: "f1", + role: "assistant", + text: "zephyr routing decision confirmed", + timestamp: 200, + }, + { + entryId: "f2", + role: "assistant", + text: "[tool:Read] zephyr routing decision file", + timestamp: 300, + }, + { + entryId: "f3", + role: "toolResult", + text: "[tool:Read] zephyr routing decision contents", + timestamp: 400, + }, + ]); + const role = grepAnswer(store, identity, { query: "zephyr", role: "assistant" }); + expect(role).toContain("[f1]"); + expect(role).toContain("[f2]"); + expect(role).not.toContain("[f0]"); + expect(role).not.toContain("[f3]"); + const window = grepAnswer(store, identity, { + query: "zephyr", + date_start: 150, + date_end: 250, + }); + expect(window).toContain("[f1]"); + expect(window).not.toContain("[f0]"); + expect(window).not.toContain("[f2]"); + const scoped = grepAnswer(store, identity, { query: "zephyr", tool_name: "Read" }); + expect(scoped).toContain("[f2]"); + expect(scoped).toContain("[f3]"); + expect(scoped).not.toContain("[f0]"); + expect(scoped).not.toContain("[f1]"); + const both = grepAnswer(store, identity, { + query: "zephyr", + role: "assistant", + tool_name: "Read", + }); + expect(both).toContain("[f2]"); + expect(both).not.toContain("[f3]"); + } finally { + store.close(); + } + }); + + it("refuses a pattern scoped to a tool and a pattern over past sessions", () => { + const store = seedStore(); + try { + expect(grepAnswer(store, identity, { pattern: "message", tool_name: "Read" })).toContain( + "toolName scopes a query search", + ); + expect( + answerGrep( + { + store: () => store, + redact: identity, + pastSessions: { list: () => [] }, + }, + "ses_current", + { pattern: "message", scope: "sessions" }, + ), + ).toContain("a pattern scan searches this session"); + } finally { + store.close(); + } + }); + + it("centers a summary listing on an anchor", () => { + const { store, leaf } = seedSpan(); + try { + const centered = expandAnswer(store, identity, { id: leaf, anchor: "a5", limit: 5 }); + expect(centered).toContain("anchored at a5"); + for (const id of ["a3", "a4", "a5", "a6", "a7"]) { + expect(centered).toContain(`[${id}]`); + } + expect(centered).not.toContain("[a0]"); + } finally { + store.close(); + } + }); + + it("refuses an anchor without an id and an anchor with an entry id", () => { + const { store, leaf } = seedSpan(); + try { + expect(expandAnswer(store, identity, { anchor: "a5" })).toContain( + "anchor centers a summary listing", + ); + expect(expandAnswer(store, identity, { id: leaf, entry_id: "a5", anchor: "a5" })).toContain( + "pass anchor or entry_id", + ); + } finally { + store.close(); + } + }); +}); diff --git a/packages/deepseek/tsconfig.build.json b/packages/deepseek/tsconfig.build.json new file mode 100644 index 0000000..85acb7a --- /dev/null +++ b/packages/deepseek/tsconfig.build.json @@ -0,0 +1,7 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "paths": {} + }, + "include": ["src/**/*.ts"] +} diff --git a/packages/deepseek/tsconfig.json b/packages/deepseek/tsconfig.json new file mode 100644 index 0000000..e8b02ca --- /dev/null +++ b/packages/deepseek/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../../tsconfig.json", + "include": ["src/**/*.ts", "test/**/*.ts"] +} diff --git a/packages/deepseek/vite.config.ts b/packages/deepseek/vite.config.ts new file mode 100644 index 0000000..e6600e9 --- /dev/null +++ b/packages/deepseek/vite.config.ts @@ -0,0 +1,26 @@ +import { fileURLToPath } from "node:url"; + +import { defineConfig } from "vite-plus"; + +/** The engine read as source for the package-local test run; `pack` resolves it + * through node_modules, so nothing here reaches the tarball. */ +const engine = fileURLToPath(new URL("../core/src/index.ts", import.meta.url)); + +export default defineConfig({ + pack: { + deps: { + resolveDepSubpath: true, + }, + entry: ["src/index.ts", "src/preset.ts"], + format: ["esm"], + dts: true, + sourcemap: true, + outDir: "dist", + target: "node22.19.0", + tsconfig: "tsconfig.build.json", + }, + test: { + alias: { "lossless-core": engine }, + setupFiles: ["../core/test/setup.ts"], + }, +}); diff --git a/packages/pi/src/ui/config-io.ts b/packages/pi/src/ui/config-io.ts index 92c9616..a930f02 100644 --- a/packages/pi/src/ui/config-io.ts +++ b/packages/pi/src/ui/config-io.ts @@ -66,6 +66,7 @@ export interface PassConditions { leafChunkTokens: number; maxAsyncChunks: number; redactSecrets: boolean; + mechanicalTools: string; } const PASS_CONDITION_KEYS = [ @@ -74,6 +75,7 @@ const PASS_CONDITION_KEYS = [ "leafChunkTokens", "maxAsyncChunks", "redactSecrets", + "mechanicalTools", ] as const satisfies readonly (keyof PassConditions)[]; export function passConditions(config: LcmConfig): PassConditions { @@ -83,6 +85,7 @@ export function passConditions(config: LcmConfig): PassConditions { leafChunkTokens: config.leafChunkTokens ?? 3000, maxAsyncChunks: config.maxAsyncChunks ?? 24, redactSecrets: config.redactSecrets !== false, + mechanicalTools: JSON.stringify(config.mechanicalTools ?? []), }; } diff --git a/packages/pi/test/config-io.test.ts b/packages/pi/test/config-io.test.ts index 4776158..3dcefcd 100644 --- a/packages/pi/test/config-io.test.ts +++ b/packages/pi/test/config-io.test.ts @@ -147,6 +147,7 @@ test("config-io: pass conditions default to what the pass reads", () => { leafChunkTokens: 3000, maxAsyncChunks: 24, redactSecrets: true, + mechanicalTools: "[]", }); }); @@ -178,6 +179,7 @@ test("config-io: every pass condition is compared, and only those", () => { leafChunkTokens: 2000, maxAsyncChunks: 6, redactSecrets: false, + mechanicalTools: ["noisy-read"], }); assert.deepEqual(changedPassConditions(passConditions({}), moved), [ "summarizer", @@ -185,6 +187,7 @@ test("config-io: every pass condition is compared, and only those", () => { "leafChunkTokens", "maxAsyncChunks", "redactSecrets", + "mechanicalTools", ]); assert.deepEqual( passConditions({ swapAtRatio: 0.95, keepRecentTokens: 1000, assemblyEnabled: false }), @@ -210,6 +213,7 @@ test("config-io: each setting can make a pass stale, and the row can name it", ( ["leafChunkTokens", { leafChunkTokens: 2000 }], ["maxAsyncChunks", { maxAsyncChunks: 6 }], ["redactSecrets", { redactSecrets: false }], + ["mechanicalTools", { mechanicalTools: ["noisy-read"] }], ]; for (const [key, config] of moved) { assert.deepEqual(staleReason(identity("one/model", {}), identity("one/model", config)), { diff --git a/scripts/sync-lossless-presets.ts b/scripts/sync-lossless-presets.ts new file mode 100644 index 0000000..d9353c2 --- /dev/null +++ b/scripts/sync-lossless-presets.ts @@ -0,0 +1,165 @@ +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +/** + * Regenerate packages/deepseek-web/cordis.presets.patch.yml from the pinned + * harness presets plus the lossless backend swap. + * + * Usage: bun scripts/sync-lossless-presets.ts [--check] + * + * The deepseek harness has no narrower preset-override hook: a patch keyed + * by a preset row id replaces the complete child list, and nested rows are + * not addressable from later layers (the composer rejects composite ids). + * So each Lossless preset is a full copy of the shipped preset with exactly + * three deliberate differences: the compaction group backend becomes + * deepseek-lossless/preset, em dashes in copied plan-mode prose read as + * commas (this repo ships no em dashes), and each preset declares a + * config.description (the UI renders declaration config.description for + * non built-in ids, and the lossless ids have no locale key). Everything + * else stays byte for byte, which --check verifies along with regeneration. + * + * Re-run against a new harness checkout when the dsh peer floor moves and + * review the diff: it must show only the backend swap (plus punctuation). + * Point at the pinned release tag (for example + * dsh-v0.2.1-rc.3), not at master HEAD: master keeps the same package + * version string while carrying unreleased preset rows that npm cannot + * resolve, so the content pins below fail fast on such a checkout. + */ + +const [, , harnessRepo, flag] = process.argv; +if (harnessRepo === undefined || !existsSync(harnessRepo)) { + console.error("sync-lossless-presets: pass the deepseek-harness checkout path"); + process.exit(2); +} +const check = flag === "--check"; + +const PRESETS = [ + { name: "standard", order: 1, losslessOrder: 11 }, + { name: "ptc", order: 2, losslessOrder: 12 }, + { name: "minimal", order: 3, losslessOrder: 13 }, + { name: "cordis", order: 4, losslessOrder: 14 }, +] as const; + +const DESCRIPTIONS = { + standard: + "Full coding agent with planning, subagents, and web tools. Compaction runs through the shared lossless core.", + ptc: "Full coding agent with PTC presentation mode. Compaction runs through the shared lossless core.", + minimal: + "Lightweight shell preset with a fixed persona and image reads. Compaction runs through the shared lossless core.", + cordis: + "Full coding agent with Cordis preset authoring tools. Compaction runs through the shared lossless core.", +} as const; + +const BACKEND_OLD = ` - id: compaction-basic + name: '@deepseek-ai/dsh-compaction-basic'`; +const BACKEND_NEW = ` - id: deepseek-lossless + name: deepseek-lossless/preset + config: + enabled: true`; +const EM_DASH_PROSE = + "Imperative language to implement changes means plan the implementation, not execute it. A user's conversational agreement \u2014 including an answer confirming something you asked \u2014 approves nothing and does not end plan mode;"; +const EM_DASH_FIXED = + "Imperative language to implement changes means plan the implementation, not execute it. A user's conversational agreement, including an answer confirming something you asked, approves nothing and does not end plan mode;"; + +// The version pin is exact because master HEAD keeps the same version +// string while carrying unreleased rows; the content pins below fail fast +// on such a checkout, so run against the release tag. +const HARNESS_VERSION = JSON.parse( + readFileSync(join(harnessRepo, "packages", "bundle", "web-app", "package.json"), "utf8"), +).version as string; +if (HARNESS_VERSION !== "0.2.1-rc.3") { + console.error( + `sync-lossless-presets: unexpected harness version ${HARNESS_VERSION}; expected 0.2.1-rc.3 at tag dsh-v0.2.1-rc.3`, + ); + process.exit(2); +} + +const out: string[] = []; +out.push(`# Lossless presets: one \`@deepseek-ai/dsh-agent-preset\` declaration per +# shipped web preset, inserted after the web patch. Each entry is generated +# by scripts/sync-lossless-presets.ts from the pinned harness presets +# (dsh v${HARNESS_VERSION}, the runtime our peers admit) with exactly three +# deliberate differences: the compaction group backend becomes +# \`deepseek-lossless/preset\`, em dashes in copied plan-mode prose read +# as commas (this repo ships no em dashes), and each preset declares a +# config.description (the UI renders declaration config.description for +# non built-in ids, and the lossless ids have no locale key). Everything +# else is byte for byte the pinned harness preset, so preset sessions keep +# their catalog while compaction runs through the shared lossless core. +# +# Selecting a Lossless preset is the opt-in; the backend row ships enabled +# because the selection itself is the opt-in. The host row stays disabled +# by default, so enable it too: the realm backend borrows the shared core +# and fails fast naming the host row when it is absent. +#`); + +for (const { name, order, losslessOrder } of PRESETS) { + const src = readFileSync( + join(harnessRepo, "packages", "bundle", "web-app", "presets", `${name}.patch.yml`), + "utf8", + ); + if (src.includes("dsh-tool-executor-ptc")) + throw new Error( + `sync-lossless-presets: ${name} has the unreleased executor row; run against tag dsh-v0.2.1-rc.3, not master HEAD`, + ); + if (name === "ptc" && !src.includes("dsh-agent-tool-presentation")) + throw new Error( + `sync-lossless-presets: ptc lacks the pinned presentation row; run against tag dsh-v0.2.1-rc.3`, + ); + if (name !== "minimal" && !src.includes("tool-subagent-list-agents")) + throw new Error( + `sync-lossless-presets: ${name} lacks the pinned list-agents row; run against tag dsh-v0.2.1-rc.3`, + ); + let body = src.replace(/^#[^\n]*\n(?:#.*\n)*/, ""); + const idLine = `- id: preset-${name}\n name: '@deepseek-ai/dsh-agent-preset'`; + if (!body.includes(idLine)) + throw new Error(`sync-lossless-presets: ${name} entry head not found`); + body = body.replace( + idLine, + `- id: preset-lossless-${name}\n name: "@deepseek-ai/dsh-agent-preset"`, + ); + body = body.replace(`id: ${name}\n`, `id: lossless-${name}\n`); + body = body.replace( + `id: lossless-${name}\n`, + `id: lossless-${name}\n description: "${DESCRIPTIONS[name]}"\n`, + ); + if (!body.includes(`description: "${DESCRIPTIONS[name]}"`)) + throw new Error(`sync-lossless-presets: ${name} description not injected`); + body = body.replace(`order: ${order}\n`, `order: ${losslessOrder}\n`); + if (body.split(BACKEND_OLD).length !== 2) + throw new Error(`sync-lossless-presets: ${name} backend row not found exactly once`); + body = body.replace(BACKEND_OLD, BACKEND_NEW); + body = body.split(EM_DASH_PROSE).join(EM_DASH_FIXED); + // Match the repo formatter (vp check): single-quoted `name:` scalars read + // as double-quoted. Nothing else is touched: !!js lines are JavaScript + // and literal-block prose keeps the harness punctuation verbatim. + body = body + .split("\n") + .map((line) => + /^(\s*(?:-\s+)?name: )'([^']*)'\s*$/.test(line) + ? line.replace(/^(\s*(?:-\s+)?name: )'([^']*)'\s*$/, '$1"$2"') + : line, + ) + .join("\n"); + if (body.includes("- id: compaction-basic")) + throw new Error(`sync-lossless-presets: ${name} keeps the base backend`); + if (body.includes("\u2014")) + throw new Error(`sync-lossless-presets: ${name} still has an em dash`); + out.push(body.replace(/\n+$/, "")); + out.push(""); +} + +const generated = `${out.join("\n")}`; +const target = join("packages", "deepseek-web", "cordis.presets.patch.yml"); +const current = existsSync(target) ? readFileSync(target, "utf8") : null; +if (current === generated) { + console.log("sync-lossless-presets: in sync"); +} else if (check) { + console.error( + "sync-lossless-presets: committed presets differ from generated output; re-run without --check", + ); + process.exit(1); +} else { + writeFileSync(target, generated); + console.log("sync-lossless-presets: wrote packages/deepseek-web/cordis.presets.patch.yml"); +}