feat(track-a): context-engine as a pure extraction library (ADR-037) - #2
Merged
Merged
Conversation
Adds a Docling-based file extractor (python/docling_extract.py, spawned by
docling-client.service.ts) as the default, with graceful fallback to the
existing Gemini extractor on any failure. Extracted document structure is
serialized to MINT via the new mint-mapper (encodeDocument from
@q1k-oss/mint-format >=1.1.0) and stored in files.mint_content for
token-efficient prompt injection.
- file.types: DocumentStructure gains figures[] + section pageRef
- schema: files.mint_content column (apply via db:push)
- config: extractor option ('docling' | 'gemini')
- adds CLAUDE.md (extractor flow, Docling/uv setup, MINT usage)
- scripts/smoke-docling.ts verifies the structure -> MINT path
Make context-engine consumable in-process by q1k-controlplane's Temporal activities (no HTTP service in the doc-ingestion path). Sprint M Track A. - Docling device fix: python/docling_extract.py reads DOCLING_DEVICE (default cpu) and pins AcceleratorOptions — fixes the Apple-Silicon RT-DETR-v2 float64-on-MPS crash. Verified: real PDF extracts cleanly, device=cpu, no patch. - New pure chunker: chunkDocument(structure, opts) → DocumentChunk[] (§-boundary chunks + overlap, stable chunkIndex = idempotency key, provenance sectionRef/ pageRef). +6 unit tests. - Delete orchestration (A9): processFile/saveFile/integrateIntoGraph + the file-upload route. CE no longer owns ingestion state; controlplane drives it. No existing tests referenced these (60 existing tests still green). - Clean library exports: doclingClientService, structureToMint/toMintDocument, chunkDocument + types from src/index.ts. - Packaging: ship python/ + pyproject.toml in the npm `files` field so the Docling adapter travels with the library (the consuming worker resolves the Python env via DOCLING_CMD). - Add `npm test` (vitest run) — repo had no test script. - CLAUDE.md rewritten to the library role. Verified real-stack: smoke-docling.ts → MINT round-trip + chunking + real Docling on a sample SOP PDF all green; 66 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The main index.ts barrel re-exports the Express routers, so importing it from a pure-function consumer (controlplane's Temporal worker) drags in the whole server stack. Add `@q1k-oss/context-engine/extraction` — a dependency-light entry exporting only doclingClientService / structureToMint / chunkDocument + types. Verified: controlplane imports it and runs chunk+mint with no express/ router load. Bump to 0.3.0 (library shape). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hshekhar-git
marked this pull request as ready for review
June 24, 2026 09:01
avnsh
force-pushed
the
feat/track-a-library
branch
from
June 25, 2026 02:50
c87c4dc to
7ebff50
Compare
1.1.0 is now published; refresh the lockfile so npm ci resolves it. Verified: npm ci + tsc build pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Track A (ADR-037): turn
@q1k-oss/context-engineinto a pure, importable extraction library so the controlplane can consume Docling extraction + MINT + chunking in-process (no CE HTTP service).python/docling_extract.py) with an Apple-Silicon device fix (DOCLING_DEVICE, default cpu) +pyproject.toml/uv.lockso the Python subprocess ships with the package.mint-mapper.ts) + a new purechunkDocument(chunker.ts, §-boundary chunks with overlap + page/section refs) + tests (__tests__/chunker.test.ts)./extractionsubpath (src/extraction.ts) exportingdoclingClientService/structureToMint/chunkDocument+ types — so importers don't load the Express app.file-processor.service.ts,routes/files.routes.ts) — CE is a library now, not a service.package.jsonpins@q1k-oss/mint-formatat a local tarball (file:../mint-format/...1.1.0.tgz). Depends on mint-format feat(track-a): context-engine as a pure extraction library (ADR-037) #2 — publish@q1k-oss/mint-format@1.1.0, re-pin to the real version. (CInpm ciwill fail until then.)@q1k-oss/context-engine@0.3.0so the consumer can depend on it.Pairs with
🤖 Generated with Claude Code