Problem statement
The SDK parsed-storage layout currently diverges from the Knowhere result ZIP layout. The SDK writes an SDK-specific snapshot shape such as manifest/current.json, chunks/page-N.json, and assets/..., while the Knowhere result contract is an expanded result directory with manifest.json, chunks.json, optional sidecar files, and result-relative asset paths.
This creates extra adapter knowledge, makes Notebook page reads pay for SDK chunk-page behavior, and splits the local cache / MCP vocabulary from parsed storage.
Proposed solution
Implement the result-layout storage plan documented in docs/parsed-storage-result-layout-plan.md.
Core decisions:
- SDK parsed storage should unzip the Knowhere result and write files using the original result-relative paths.
- Storage adapters should map result-relative paths to complete physical Blob/disk/S3 paths; they should not change the logical result layout.
manifest.json plus chunks.json is the minimum committed snapshot.
- Commit/progress metadata should stay minimal and be outside the result layout or in an adapter side channel.
- Writes must commit last; reads must ignore incomplete or failed snapshots.
Jobs.load(jobId) / Knowledge.syncParsedDocument({ jobId }) is the canonical result ZIP path.
Knowledge.syncParsedDocument({ documentId }) should resolve the current published job id, then use the job-result path before falling back to documents.listChunks reconstruction.
- Do not add
chunks.index.json in v1. SDK core treats logical chunks.json as the source of truth. Storage adapters may internally split large chunks.json writes if they can transparently read it back as the same logical path.
Breaking change tracking
This changes the public ParsedDocumentStorage adapter interface from page-oriented methods such as readManifest / readChunkPage / writeChunkPage toward object-path methods such as readObject / writeObject.
Track compatibility work explicitly:
- keep legacy paged snapshot reads during migration
- make all new writes use result layout
- update disk storage, Notebook Blob storage, MCP docs/tests, and SDK tests
- add a changeset and migration notes
Acceptance criteria
- Result-layout parsed storage writes preserve result-relative paths from the ZIP.
readChunks, grepChunks, and getDocumentOutline can read from result-layout storage.
syncParsedDocument({ jobId }) uses the result ZIP path.
syncParsedDocument({ documentId }) resolves the current published job id and uses the result ZIP path when available.
- Incomplete result-layout writes are not readable.
- Legacy paged snapshot storage remains readable during migration.
- Documentation and changeset describe the breaking adapter-interface change.
Alternatives considered
- Keep paged
chunks/page-N.json as the SDK storage layout. Rejected because it duplicates the Knowhere result contract and forces SDK-specific chunk-page semantics into adapters.
- Add
chunks.index.json in v1. Rejected for now; keep the core concept simple and let storage adapters hide internal splitting if needed.
Problem statement
The SDK parsed-storage layout currently diverges from the Knowhere result ZIP layout. The SDK writes an SDK-specific snapshot shape such as
manifest/current.json,chunks/page-N.json, andassets/..., while the Knowhere result contract is an expanded result directory withmanifest.json,chunks.json, optional sidecar files, and result-relative asset paths.This creates extra adapter knowledge, makes Notebook page reads pay for SDK chunk-page behavior, and splits the local cache / MCP vocabulary from parsed storage.
Proposed solution
Implement the result-layout storage plan documented in
docs/parsed-storage-result-layout-plan.md.Core decisions:
manifest.jsonpluschunks.jsonis the minimum committed snapshot.Jobs.load(jobId)/Knowledge.syncParsedDocument({ jobId })is the canonical result ZIP path.Knowledge.syncParsedDocument({ documentId })should resolve the current published job id, then use the job-result path before falling back todocuments.listChunksreconstruction.chunks.index.jsonin v1. SDK core treats logicalchunks.jsonas the source of truth. Storage adapters may internally split largechunks.jsonwrites if they can transparently read it back as the same logical path.Breaking change tracking
This changes the public
ParsedDocumentStorageadapter interface from page-oriented methods such asreadManifest/readChunkPage/writeChunkPagetoward object-path methods such asreadObject/writeObject.Track compatibility work explicitly:
Acceptance criteria
readChunks,grepChunks, andgetDocumentOutlinecan read from result-layout storage.syncParsedDocument({ jobId })uses the result ZIP path.syncParsedDocument({ documentId })resolves the current published job id and uses the result ZIP path when available.Alternatives considered
chunks/page-N.jsonas the SDK storage layout. Rejected because it duplicates the Knowhere result contract and forces SDK-specific chunk-page semantics into adapters.chunks.index.jsonin v1. Rejected for now; keep the core concept simple and let storage adapters hide internal splitting if needed.