Add original/normalized file download API for SDK page-memory citations

[suguanYang]

Problem

The Node SDK issue Ontos-AI/knowhere-node-sdk#110 plans SDK-side page-memory citation assets: render source PDF pages into images, store them through pluggable SDK storage, and attach those assets to page chunks by metadata.page_nums.

The SDK cannot reliably do that later from only a document_id unless the Knowhere API exposes the original or normalized source file. Current staging behavior for doc_c5f9927aa1a5 shows:

• documents.listChunks(... include_asset_urls=true) returns page chunks with asset_url: null.
• page chunk metadata has no original/normalized file URL.
• the job result_url points to a parsed result ZIP only.
• that result ZIP does not include the original file or rendered page assets.

So SDK consumers can only render page images when they still have the source file from parse-time or when the app has its own source storage. Server-side clients like Notebook need an authenticated API path to fetch the source/normalized PDF by document.

Goal

Add a v2 API endpoint that lets authorized clients retrieve the original file or normalized PDF for a document, so SDKs can generate page-memory citation assets after parsing.

Proposed API Shape

Prefer a documents resource because page assets are linked to persisted documents/chunks.

Possible endpoints:

GET /v2/documents/{document_id}/files/original
GET /v2/documents/{document_id}/files/normalized-pdf

or a single endpoint with a variant:

GET /v2/documents/{document_id}/file?variant=original|normalized_pdf

Response options:

1. JSON metadata + signed URL:

{
  "document_id": "doc_...",
  "variant": "normalized_pdf",
  "file_name": "...pdf",
  "content_type": "application/pdf",
  "size_bytes": 12345,
  "url": "https://...signed...",
  "expires_at": "2026-07-02T12:00:00Z"
}

2. Direct streaming response with Content-Type and Content-Disposition.

For SDK usage, signed URL JSON is probably simpler and avoids streaming large files through SDK model parsing, but direct streaming can be more secure if we want the API to enforce auth on every byte.

Variant Semantics

• original: the uploaded/source file when it is safe and available.
• normalized_pdf: the PDF representation used for page-memory page numbering and rendering.

For page-memory citations, normalized_pdf is the important variant because page numbers must match metadata.page_nums. For PDF inputs, original and normalized PDF may be the same object.

For non-PDF inputs, if no normalized PDF is persisted today, page-memory ingestion should persist it or add a durable artifact path so this endpoint can return it.

Authorization / Security

• Require the same document ownership/API key checks as GET /v2/documents/{document_id} and listChunks.
• Respect archived/deleted document state.
• Use short-lived signed URLs if returning URL JSON.
• Avoid exposing raw storage keys or permanent bucket URLs.
• Consider rate limits because SDK page rendering may fetch large PDFs.

SDK Consumer Contract

After this exists, Node SDK can expose:

const file = await client.documents.getFile(documentId, {
  variant: "normalized_pdf",
  responseType: "arrayBuffer",
});

const chunksWithAssets = await attachPageAssetsToChunks({
  chunks,
  sourceFile: file,
  storage,
});

or:

const { url } = await client.documents.getFileUrl(documentId, {
  variant: "normalized_pdf",
});

Acceptance Criteria

• API exposes an authenticated way to fetch or sign the original/normalized PDF for a document.
• For page-memory documents, normalized_pdf page numbering matches metadata.page_nums on page chunks.
• PDF source documents return a PDF successfully.
• Non-PDF page-memory documents either return a persisted normalized PDF or a clear unsupported/not_available error.
• Responses include content type, file name, variant, and expiry when returning signed URLs.
• Access is denied for documents outside the caller's namespace/account.
• Tests cover PDF document success, missing artifact, unauthorized access, archived/deleted document behavior, and signed URL expiry/shape.
• Node SDK issue Ontos-AI/knowhere-node-sdk#110 can depend on this endpoint for post-parse server-side clients.

Open Questions

• Should the endpoint default to normalized_pdf for page-memory documents?
• Should we return signed URL JSON, stream bytes directly, or support both?
• Where should normalized PDFs be stored for non-PDF inputs if they are not currently persisted?
• Should result ZIPs also include normalized source file references, or should source retrieval remain a document API concern only?

[suguanYang]

Follow-up from code inspection: the original file is already stored in source/upload object storage.

Relevant current behavior:

• Job.s3_key persists the source object key.
• File jobs use uploads/{job_id}{file_extension} in apps/api/app/services/document_ingestion/creation_service.py.
• URL jobs also use uploads/{job_id}{file_extension}; the worker downloads the source URL and uploads it via JobFileStorage.upload_source_file(temp_file_path, s3_key).
• Worker parse context later uses the same job_context.s3_key and verifies/downloads it from upload storage before parsing.
• Current code only deletes temporary MinerU shard objects through delete_upload_file; I did not find a normal post-parse delete of the original uploads/{job_id}... object.

For the staging page-memory document discussed in Slock, the API reports job job_29168e1aa4a2 and PDF source file document-aFHqgst2LAg68OKAJKoWJnumXSJhxD.pdf; by current key construction the source object should be uploads/job_29168e1aa4a2.pdf in the uploads bucket. The object is private from unauthenticated HTTP (S3 returns 403), and the public document/job API does not expose s3_key today.

So the endpoint can probably sign/stream the existing Job.s3_key upload object for variant=original. The remaining design work is how to resolve document_id -> current_job_result_id -> job_id -> Job.s3_key, and whether variant=normalized_pdf should be the same object for PDF sources or a separate persisted artifact for non-PDF sources.

[suguanYang]

Decision update from planning thread:

1. Renderer choice: implementation should pick the most performant and compatible open-source PDF-to-image renderer for server-side use. The implementation PR should evaluate/justify the selected renderer against compatibility, license, deployment footprint, CPU/memory behavior, and output quality.
2. Notebook chunk view: do not display page image assets directly in parsed chunk cards. Notebook already has jump-to-original-file behavior for chunk inspection.
3. Chat citations: page image assets are needed for citation display. The UI should display a link to the rendered page image for chat citations.
4. Asset access: use signed URLs for rendered page images. Avoid proxying/binding bandwidth through the original application server when object storage can serve the asset directly.
5. API dependency: Knowhere API must expose an original/normalized file retrieval endpoint for page-memory mode so SDK/server-side clients can render page images from the correct source.

This keeps SDK #110 focused on pluggable storage + page-memory citation asset generation, with Notebook consuming links in chat citations rather than adding images to the parsed chunk cards.

[suguanYang]

API response decision from planning thread: the original/normalized file download API can return a presigned S3 URL directly to the client.

Implications:

• Endpoint should authenticate/authorize the document request first.
• Resolve the existing source object key from Job.s3_key for variant=original.
• Generate a short-lived presigned GET URL with content metadata in the JSON response.
• Do not stream the file through the API server by default; avoid API bandwidth for large source PDFs.
• The SDK can call the endpoint, fetch the presigned URL, and then render page images from that source.

Suggested response shape remains:

{
  "document_id": "doc_...",
  "variant": "original",
  "file_name": "...pdf",
  "content_type": "application/pdf",
  "size_bytes": 12345,
  "url": "https://...presigned...",
  "expires_at": "2026-07-02T12:00:00Z"
}

[suguanYang]

Implementation PR opened: #202

Final API shape from the planning thread:

• GET /v2/documents/{document_id}/files/page-citation-source
• v2-only; no variant request parameter
• returns signed URL JSON for the page-memory normalized PDF artifact (source.pdf, variant: normalized_pdf in response metadata)
• returns 404 for non-page-memory documents, missing source.pdf, archived documents, or documents outside the caller account

Local verification is listed in the PR body.