Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/fast-storage-readers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@ontos-ai/knowhere-sdk": patch
---

Read committed parsed-storage result objects concurrently after validating the commit marker.
8 changes: 0 additions & 8 deletions .changeset/parsed-storage-result-layout.md

This file was deleted.

5 changes: 5 additions & 0 deletions .changeset/quiet-pages-reflect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@ontos-ai/knowhere-mcp": patch
---

Improve MCP read guidance for page assets and default durable asset URLs for display reads.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## 2.1.2

### Patch Changes

- [#115](https://github.com/Ontos-AI/knowhere-node-sdk/pull/115) [`8b7ccca`](https://github.com/Ontos-AI/knowhere-node-sdk/commit/8b7cccaadef54396dcfece2c2ba260e3bf0c20ec) Thanks [@suguanYang](https://github.com/suguanYang)! - Replace the SDK parsed-document storage adapter with a result-relative object interface and store committed parsed results as expanded Knowhere result files (`manifest.json`, `chunks.json`, optional sidecars, and assets) instead of SDK-specific paged snapshots.

Remove `assetUrlPolicy` from `knowledge.readChunks` and the MCP `knowhere_read_chunks` schema. SDK remote chunk reads now always request asset URLs from Knowhere, return those remote asset URLs directly on storage misses, and only rewrite asset URLs to storage object URLs when a configured parsed storage already has the corresponding object.

## 2.1.1

### Patch Changes
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@ontos-ai/knowhere-sdk",
"version": "2.1.1",
"version": "2.1.2",
"description": "Official Node.js SDK for Knowhere document parsing API",
"keywords": [
"knowhere",
Expand Down
11 changes: 11 additions & 0 deletions packages/mcp/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# @ontos-ai/knowhere-mcp

## 2.1.2

### Patch Changes

- [#115](https://github.com/Ontos-AI/knowhere-node-sdk/pull/115) [`8b7ccca`](https://github.com/Ontos-AI/knowhere-node-sdk/commit/8b7cccaadef54396dcfece2c2ba260e3bf0c20ec) Thanks [@suguanYang](https://github.com/suguanYang)! - Replace the SDK parsed-document storage adapter with a result-relative object interface and store committed parsed results as expanded Knowhere result files (`manifest.json`, `chunks.json`, optional sidecars, and assets) instead of SDK-specific paged snapshots.

Remove `assetUrlPolicy` from `knowledge.readChunks` and the MCP `knowhere_read_chunks` schema. SDK remote chunk reads now always request asset URLs from Knowhere, return those remote asset URLs directly on storage misses, and only rewrite asset URLs to storage object URLs when a configured parsed storage already has the corresponding object.

- Updated dependencies [[`8b7ccca`](https://github.com/Ontos-AI/knowhere-node-sdk/commit/8b7cccaadef54396dcfece2c2ba260e3bf0c20ec)]:
- @ontos-ai/knowhere-sdk@2.1.2

## 2.1.1

### Patch Changes
Expand Down
11 changes: 7 additions & 4 deletions packages/mcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,12 +182,13 @@ When logged in with Read only permission, the MCP server exposes only
and makes the parsed document available to outline/read/grep/search tools.
- `knowhere_parse_file`: blocking parse for a file path available to the MCP
process; waits for completion and makes the parsed document available to
outline/read/grep/search tools.
outline/read/grep/search tools. The path is resolved on the machine running
this stdio MCP server.
- `knowhere_async_parse_url`: start parsing a remote URL and return the job
immediately. When checking status, use exponential backoff.
- `knowhere_async_parse_file`: start parsing a local file path, upload it if
needed, and return the job immediately. When checking status, use exponential
backoff.
backoff. The path is resolved on the machine running this stdio MCP server.
- `knowhere_async_get_job_status`: check a parse job status. For large PDFs or
OCR-heavy files, parsing can take 10+ minutes; poll with `5s`, `10s`, `20s`,
`40s`, `80s`, then cap at `120s` between follow-up status checks. After
Expand All @@ -203,12 +204,14 @@ When logged in with Read only permission, the MCP server exposes only
- `knowhere_read_chunks`: read exact chunks from one parsed document by passing
`localDocumentId`, published `documentId`, or completed `jobId`. Use
`page`/`pageSize` for display reads; asset URLs are returned when the source
or configured storage provides them.
or configured storage provides them. Page screenshots are returned under chunk
`metadata.pageAssets`, not as image chunks.
- `knowhere_grep_chunks`: run literal or regex grep over one parsed document by
passing `localDocumentId`, published `documentId`, or completed `jobId`.
Broad workspace search belongs to `knowhere_search`.
- `knowhere_search`: search published documents through the Knowhere API
retrieval query.
retrieval query. Use follow-up `knowhere_read_chunks` calls to inspect page
screenshots under chunk `metadata.pageAssets`.

Local-cache tool responses include `document.resultDirectoryPath`; expanded
chunks are stored at `<resultDirectoryPath>/chunks.json` for direct filesystem
Expand Down
2 changes: 1 addition & 1 deletion packages/mcp/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@ontos-ai/knowhere-mcp",
"version": "2.1.1",
"version": "2.1.2",
"description": "MCP wrapper for the Knowhere Node.js SDK local knowledge tools",
"keywords": [
"knowhere",
Expand Down
13 changes: 13 additions & 0 deletions packages/mcp/src/__tests__/mcp.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ describe('knowhere MCP wrapper', () => {
const toolNames = tools.tools.map((tool) => tool.name).sort();
const statusTool = tools.tools.find((tool) => tool.name === 'knowhere_async_get_job_status');
const listTool = tools.tools.find((tool) => tool.name === 'knowhere_list_documents');
const parseFileTool = tools.tools.find((tool) => tool.name === 'knowhere_parse_file');
const asyncParseFileTool = tools.tools.find(
(tool) => tool.name === 'knowhere_async_parse_file',
);
const readTool = tools.tools.find((tool) => tool.name === 'knowhere_read_chunks');
const searchTool = tools.tools.find((tool) => tool.name === 'knowhere_search');

expect(toolNames).toEqual([
'knowhere_async_get_job_status',
Expand All @@ -30,8 +35,16 @@ describe('knowhere MCP wrapper', () => {
expect(statusTool?.description).toContain('5s, 10s, 20s, 40s, 80s');
expect(statusTool?.description).toContain('Large PDFs or OCR-heavy files can take 10+ minutes');
expect(listTool?.description).toContain('remote API');
expect(parseFileTool?.description).toContain(
'resolved on the machine running this stdio MCP server',
);
expect(asyncParseFileTool?.description).toContain(
'resolved on the machine running this stdio MCP server',
);
expect(readTool?.description).toContain('configured parsed storage first');
expect(readTool?.description).toContain('returns asset URLs');
expect(readTool?.description).toContain('metadata.pageAssets');
expect(searchTool?.description).toContain('metadata.pageAssets');
await client.close();
await server.close();
});
Expand Down
8 changes: 4 additions & 4 deletions packages/mcp/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ export async function createKnowhereMcpServer(
'knowhere_parse_file',
{
description:
'Blocking parse: submit a local file path available to this MCP process, wait for completion, then make the parsed document available to outline/read/grep/search tools.',
'Blocking parse: submit a local file path available to this MCP process, wait for completion, then make the parsed document available to outline/read/grep/search tools. The file path is resolved on the machine running this stdio MCP server, not on a remote chat client.',
inputSchema: {
file: z.string().describe('Local file path available to this MCP server process.'),
fileName: z.string().optional(),
Expand Down Expand Up @@ -150,7 +150,7 @@ export async function createKnowhereMcpServer(
'knowhere_async_parse_file',
{
description:
'Start parsing a local file path available to this MCP process, upload it if needed, and return immediately with the parse job. When checking status, call knowhere_async_get_job_status with exponential backoff: 5s, 10s, 20s, 40s, 80s, then cap at 120s. Large PDFs or OCR-heavy files can take 10+ minutes; prefer sparse follow-up status checks over rapid repeated calls. After completion, use the returned localDocumentId, documentId, or jobId with outline/read/grep tools.',
'Start parsing a local file path available to this MCP process, upload it if needed, and return immediately with the parse job. The file path is resolved on the machine running this stdio MCP server, not on a remote chat client. When checking status, call knowhere_async_get_job_status with exponential backoff: 5s, 10s, 20s, 40s, 80s, then cap at 120s. Large PDFs or OCR-heavy files can take 10+ minutes; prefer sparse follow-up status checks over rapid repeated calls. After completion, use the returned localDocumentId, documentId, or jobId with outline/read/grep tools.',
inputSchema: {
file: z.string().describe('Local file path available to this MCP server process.'),
fileName: z.string().optional(),
Expand Down Expand Up @@ -243,7 +243,7 @@ export async function createKnowhereMcpServer(
'knowhere_read_chunks',
{
description:
'Read exact chunks from one parsed document. Pass localDocumentId, published documentId, or completed jobId. page/pageSize are for display reads and cannot be combined with sectionPath/startChunk/endChunk/chunkId. The SDK reads configured parsed storage first, falls back to remote document chunks for documentId reads, and returns asset URLs when the source or storage provides them.',
'Read exact chunks from one parsed document. Pass localDocumentId, published documentId, or completed jobId. page/pageSize are for display reads and cannot be combined with sectionPath/startChunk/endChunk/chunkId. The SDK reads configured parsed storage first, falls back to remote document chunks for documentId reads, and returns asset URLs when the source or storage provides them. Page screenshots are returned under chunk metadata.pageAssets, not as image chunks.',
inputSchema: {
localDocumentId: z.string().optional(),
documentId: z.string().optional(),
Expand Down Expand Up @@ -291,7 +291,7 @@ export async function createKnowhereMcpServer(
'knowhere_search',
{
description:
'Search published Knowhere documents with the Knowhere API retrieval query. localDocumentIds only map returned server document IDs back to local cache IDs when available.',
'Search published Knowhere documents with the Knowhere API retrieval query. localDocumentIds only map returned server document IDs back to local cache IDs when available. Page screenshots are exposed by follow-up read calls under chunk metadata.pageAssets, not as image chunks.',
inputSchema: {
query: z.string(),
namespace: z.string().optional(),
Expand Down
77 changes: 77 additions & 0 deletions src/knowledge/__tests__/knowledge.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,83 @@ describe('Knowledge', () => {
expect(read.totalChunks).toBe(3);
});

it('should read committed parsed result objects concurrently after the commit gate', async () => {
const cacheDirectory = await createTempDirectory();
const parseResult = createParseResult();
const storage = createInMemoryParsedStorage();
const readEvents: string[] = [];
const delayedResultPaths = new Set(['manifest.json', 'chunks.json', 'doc_nav.json']);
const releaseByPath = new Map<string, () => void>();
let shouldDelayResultReads = true;

storage.seedResult({
documentId: 'doc_remote',
revisionKey: 'jres_remote',
result: parseResult,
});

const readObject = storage.readObject.bind(storage);
storage.readObject = async (
params: ParsedDocumentObjectParams,
): Promise<ParsedDocumentObject | null> => {
readEvents.push(`start:${params.path}`);
if (delayedResultPaths.has(params.path) && shouldDelayResultReads) {
await new Promise<void>((resolve) => {
releaseByPath.set(params.path, resolve);
});
}

const object = await readObject(params);
readEvents.push(`finish:${params.path}`);
return object;
};

const releaseDelayedReads = (): void => {
shouldDelayResultReads = false;
for (const release of releaseByPath.values()) {
release();
}
releaseByPath.clear();
};

const { client, documentsListChunks } = createClient(parseResult);
const knowledge = new Knowledge(client, { cacheDirectory }).withParsedStorage({ storage });
const readPromise = knowledge.readChunks({
documentId: 'doc_remote',
revisionKey: 'jres_remote',
page: 1,
pageSize: 2,
});

try {
await new Promise<void>((resolve) => {
setImmediate(resolve);
});

const commitFinishIndex = readEvents.indexOf('finish:.knowhere-sdk/commit.json');
const resultStartIndexes = ['manifest.json', 'chunks.json', 'doc_nav.json'].map(
(objectPath) => readEvents.indexOf(`start:${objectPath}`),
);

expect(commitFinishIndex).toBeGreaterThanOrEqual(0);
expect(resultStartIndexes.every((index) => index > commitFinishIndex)).toBe(true);
expect(readEvents.filter((event) => event.startsWith('start:'))).toEqual([
'start:.knowhere-sdk/commit.json',
'start:manifest.json',
'start:chunks.json',
'start:doc_nav.json',
]);

releaseDelayedReads();
const read = await readPromise;

expect(documentsListChunks).not.toHaveBeenCalled();
expect(read.chunks.map((chunk) => chunk.chunkId)).toEqual(['chunk-intro', 'chunk-table']);
} finally {
releaseDelayedReads();
}
});

it('should hydrate stored page asset URLs from snake_case chunks metadata', async () => {
const cacheDirectory = await createTempDirectory();
const storage = createInMemoryParsedStorage();
Expand Down
32 changes: 17 additions & 15 deletions src/knowledge/knowledge.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1081,16 +1081,23 @@ export class Knowledge {
return null;
}

const manifestObject = await parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'manifest.json',
});
const chunksObject = await parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'chunks.json',
});
const [manifestObject, chunksObject, docNavObject] = await Promise.all([
parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'manifest.json',
}),
parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'chunks.json',
}),
parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'doc_nav.json',
}),
]);
if (!manifestObject || !chunksObject) {
return null;
}
Expand All @@ -1100,11 +1107,6 @@ export class Knowledge {
payload: readJsonObject<unknown>(chunksObject),
sourceFileName: manifest.sourceFileName,
});
const docNavObject = await parsedStorageConfig.storage.readObject({
documentId,
revisionKey,
path: 'doc_nav.json',
});
const docNav = docNavObject
? readJsonObject<{ readonly sections?: DocNavSection[] }>(docNavObject)
: undefined;
Expand Down
Loading