Skip to content

[Feature]: Align parsed storage with result ZIP layout #114

Description

@suguanYang

Problem statement

The SDK parsed-storage layout currently diverges from the Knowhere result ZIP layout. The SDK writes an SDK-specific snapshot shape such as manifest/current.json, chunks/page-N.json, and assets/..., while the Knowhere result contract is an expanded result directory with manifest.json, chunks.json, optional sidecar files, and result-relative asset paths.

This creates extra adapter knowledge, makes Notebook page reads pay for SDK chunk-page behavior, and splits the local cache / MCP vocabulary from parsed storage.

Proposed solution

Implement the result-layout storage plan documented in docs/parsed-storage-result-layout-plan.md.

Core decisions:

  • SDK parsed storage should unzip the Knowhere result and write files using the original result-relative paths.
  • Storage adapters should map result-relative paths to complete physical Blob/disk/S3 paths; they should not change the logical result layout.
  • manifest.json plus chunks.json is the minimum committed snapshot.
  • Commit/progress metadata should stay minimal and be outside the result layout or in an adapter side channel.
  • Writes must commit last; reads must ignore incomplete or failed snapshots.
  • Jobs.load(jobId) / Knowledge.syncParsedDocument({ jobId }) is the canonical result ZIP path.
  • Knowledge.syncParsedDocument({ documentId }) should resolve the current published job id, then use the job-result path before falling back to documents.listChunks reconstruction.
  • Do not add chunks.index.json in v1. SDK core treats logical chunks.json as the source of truth. Storage adapters may internally split large chunks.json writes if they can transparently read it back as the same logical path.

Breaking change tracking

This changes the public ParsedDocumentStorage adapter interface from page-oriented methods such as readManifest / readChunkPage / writeChunkPage toward object-path methods such as readObject / writeObject.

Track compatibility work explicitly:

  • keep legacy paged snapshot reads during migration
  • make all new writes use result layout
  • update disk storage, Notebook Blob storage, MCP docs/tests, and SDK tests
  • add a changeset and migration notes

Acceptance criteria

  • Result-layout parsed storage writes preserve result-relative paths from the ZIP.
  • readChunks, grepChunks, and getDocumentOutline can read from result-layout storage.
  • syncParsedDocument({ jobId }) uses the result ZIP path.
  • syncParsedDocument({ documentId }) resolves the current published job id and uses the result ZIP path when available.
  • Incomplete result-layout writes are not readable.
  • Legacy paged snapshot storage remains readable during migration.
  • Documentation and changeset describe the breaking adapter-interface change.

Alternatives considered

  • Keep paged chunks/page-N.json as the SDK storage layout. Rejected because it duplicates the Knowhere result contract and forces SDK-specific chunk-page semantics into adapters.
  • Add chunks.index.json in v1. Rejected for now; keep the core concept simple and let storage adapters hide internal splitting if needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions