Skip to content

feat(runtime): materialize native PDF inputs for verified PDF-capable providers #3164

Description

@MoonOld

Problem

Maka synchronizes PDF-input metadata and exposes resolveModelPdfSupport, but the runtime never consumes that capability. AiSdkBackend.appendImageParts filters attachments to kind === image; a PDF remains only a text placeholder even when the selected provider/model has a verified native PDF contract.

The storage side is already present: PDF attachments have a canonical kind and MIME type, are persisted as Session artifacts, and can be read through the existing bounded AttachmentByteReader. The missing boundary is provider request materialization.

Provider research

AI SDK encoding support and provider support are separate facts. The current locked SDK produces these request shapes from one generic file part:

  • OpenAI-compatible Chat: type: file with filename and Base64 file_data;
  • Anthropic Messages: type: document with a Base64 application/pdf source;
  • OpenAI Responses: type: input_file with filename and Base64 file_data.

That proves the SDK can lower a generic file part. It does not prove every OpenAI-compatible provider accepts the resulting wire.

Verified first-party contracts

  • Anthropic Messages: all active Claude models support PDF processing. The API accepts URL, Base64, or Files API references. The complete request is limited to 32 MB; PDFs must be standard and unencrypted; the page limit is 600, or 100 when the request context window is below 1M tokens.
  • OpenAI: native Chat Completions and Responses accept PDF file inputs. Responses uses input_file; PDF processing includes extracted text and page images. Each file and the combined files in one request must remain below 50 MB.

Not yet verified

  • Generic OpenAI-compatible providers: the adapter can emit the OpenAI Chat file_data shape, but each provider must opt in only after its own contract is verified. Unknown compatible providers must remain unsupported.
  • Alibaba Token Plan / qwen3.8-max: Alibaba Chat documentation lists text, image, audio, and video content but no file/file_data PDF shape. Alibaba Responses documents input_file, but currently states that this content type is supported only by qwen3.5-ocr and requires file_url. Therefore qwen3.8-max is not an acceptance target for this issue. Its generated pdf modality is insufficient authority for sending local PDF bytes.
  • Alibaba PDF support, if added later, needs a separate provider contract. If the service continues to require file_url, it also needs a safe upload/temporary-URL design rather than silently exposing local files.

Scope status

This is an independent Runtime attachment-capability issue. It is not a follow-up requirement for Qwen3.8 support, #3156, #3157, #3162, or #3163, and it does not block those deliverables. Alibaba/Qwen remains in this issue only as a negative boundary showing why generic OpenAI-compatible PDF support must not be inferred without provider evidence.

Desired outcome

A model on a verified PDF-capable provider/wire receives an attached PDF as a provider-native file part on the current Turn and during durable replay. Unsupported, unknown, or unverified provider/model combinations continue to receive the bounded placeholder and never receive raw PDF bytes.

Proposed delivery

First slice: Anthropic and native OpenAI

  1. Replace the unused boolean-only assumption with an explicit provider/wire PDF-input contract. Do not infer generic-compatible support from a model name or generated modality alone.
  2. Carry the resolved contract into AiSdkBackend from the production Runtime Host composition. The former Headless production path no longer exists on current main and is not part of this slice.
  3. Materialize AttachmentRef.kind === pdf through the existing attachment byte reader as one generic AI SDK file part with exact application/pdf, filename, and bytes. Let the verified AI SDK provider adapter perform the wire-specific lowering.
  4. Cover native Anthropic Messages and native OpenAI Chat/Responses. Keep all generic OpenAI-compatible providers disabled unless separately verified.
  5. Keep current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction replacement, and restart behavior consistent.

Budget and failure policy

  • Add a PDF sub-budget and a combined raw-binary attachment budget. Separate independent image/PDF limits are not sufficient because Anthropic applies its 32 MB cap to the complete request and Base64 adds roughly one-third overhead.
  • Preserve the existing 12 MB image sub-limit. Choose the PDF and combined limits from measured encoded request size, leaving room for text, schemas, JSON, and other attachments.
  • Deduplicate budget decisions by durable occurrence across provider steps, as the image path does today.
  • Missing, unreadable, oversized, unsupported, or locally invalid PDFs must degrade to a safe bounded explanation; PDF bytes must never enter transcript text, logs, RuntimeEvents, or diagnostics.
  • Preflight parsing for encrypted/page-limit PDFs is tracked in feat(runtime): preflight native PDF inputs against provider document constraints #3284; bounded recovery after provider rejection is tracked in discussion(runtime): define bounded recovery for provider-rejected native PDF inputs #3285. Neither is a merge gate for this first slice. Ordinary provider errors remain governed by the existing bounded ModelFailure handling.

Acceptance criteria

  • resolveModelPdfSupport is no longer dead metadata; the runtime has an explicit verified provider/wire gate.
  • Native Anthropic receives a document block with Base64 application/pdf data through the SDK.
  • Native OpenAI Chat receives type: file; native OpenAI Responses receives type: input_file.
  • A generic OpenAI-compatible connection does not receive PDF bytes by default.
  • Current Turn, RuntimeEvent replay, StoredMessage fallback, steering, compaction, and restart all present the same durable PDF occurrence without duplicating the user message.
  • Image and PDF attachments respect both their subtype caps and the combined binary-request budget.
  • Unsupported models and local attachment failures keep the safe placeholder. Provider request failures use the existing bounded ModelFailure path; this slice does not add PDF-specific automatic retry.
  • Focused live verification covers at least one native Anthropic model and one native OpenAI model when protected credentials are available.

Non-goals

References

Disclosure: this issue was drafted and revised with OpenAI Codex assistance and reviewed and approved by the human contributor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions