Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions packages/core/src/attachments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,20 @@ export const READ_IMAGE_TOO_LARGE_MESSAGE = `Image exceeds the ${MAX_READ_IMAGE_
export const MAX_PROVIDER_IMAGE_REQUEST_BYTES = 12 * 1024 * 1024;
export const PROVIDER_IMAGE_BUDGET_EXCEEDED_MESSAGE = `Image was read, but the per-request image budget (${MAX_PROVIDER_IMAGE_REQUEST_BYTES / 1024 / 1024}MB across all images this turn) was exceeded; earlier images were sent and this one was omitted. Read fewer or smaller images.`;

/**
* Native PDF requests are Base64 encoded. Sixteen raw MiB expands to roughly
* 21.4 MiB, leaving headroom under Anthropic's 32 MiB whole-request limit for
* text, tool schemas, JSON framing, and other content.
*/
export const MAX_PROVIDER_PDF_REQUEST_BYTES = 16 * 1024 * 1024;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Bound the work the request telemetry does on these bytes before raising the cap that makes them routine. capturePreparedProviderRequest runs on every provider request — the tracker exists whenever model-call accounting does, so this is not gated on recordProviderRequestCapture — and canonicalize has no typed-array branch, so a Uint8Array is expanded element-wise through Object.keys().sort(). Measured against this repository's built packages/runtime/dist with capture persistence disabled: an 8 MiB attachment costs 3.5 s per step, 12 MiB (today's image cap) 7.5 s, and 16 MiB (this cap) 9.5 s per step with requestBytes reported as 222.4 MiB; across three steps that is 28.5 s during which a 10 ms timer scheduled beforehand never fires, and RSS grows 4.4 to 7.6 GiB. With capture persistence enabled the bytes are also verbatim in serializedRequest — I matched the %PDF- header inside it — which contradicts this PR's "without copying bytes into transcript text or diagnostics" and #3164's "PDF bytes must never enter transcript text, logs, RuntimeEvents, or diagnostics". This is not introduced here; images take the same path on main. But this slice raises the ceiling to 18 MiB combined and makes ten-plus-MiB attachments ordinary, because a 16 MiB PDF is a normal document and a 16 MiB PNG is not. Give request-shape.ts a binary branch that represents file data as { byteLength, sha256 } — substituting that summary drops the same call from 9.5 s to 0.4 ms and requestBytes from 222.4 MiB to 312 B — and add a regression asserting that a file part's bytes never appear in serializedRequest. Landing it as a separate prerequisite is fine; landing native PDF input without it is not.


/**
* Shared raw-byte ceiling across image and PDF inputs. Eighteen raw MiB
* expands to 24 MiB in Base64, preserving 8 MiB of whole-request headroom on
* the strictest verified native PDF route.
*/
export const MAX_PROVIDER_BINARY_REQUEST_BYTES = 18 * 1024 * 1024;

const MIME_BY_EXTENSION: Readonly<Record<string, string>> = {
png: 'image/png',
jpg: 'image/jpeg',
Expand Down Expand Up @@ -88,10 +102,9 @@ export function guessMimeFromName(fileName: string): string {

/**
* Route a MIME type to an {@link AttachmentRef} kind. The runtime
* consumption split is image vs. everything-else (images become provider
* image parts; other kinds are read on demand by the model via Read), so this
* only needs to single out the kinds that change
* consumption or display. Unknown / unmapped MIME falls back to `other`.
* consumption split singles out images and PDFs (authorized routes can send
* them as provider file parts); text-like kinds are read on demand by the
* model via Read. Unknown / unmapped MIME falls back to `other`.
*
* `fileName` is consulted for kinds whose MIME is unreliable across OSes
* (Office documents arrive as `application/octet-stream` or a long
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,106 @@ test('backend creation does not treat aliased provider metadata as inventory', a
);
});

test('Host composition carries verified native PDF input through the provider wire', async () => {
const modelId = 'gpt-4o';
const provider = await startProvider();
let attachmentReads = 0;
let backend: Awaited<ReturnType<typeof createHostAiSdkBackend>> | undefined;
try {
backend = await createHostAiSdkBackend(
backendCreationFixture({
abortSignal: new AbortController().signal,
modelId,
resolveExecutionConnection: async () => ({
kind: 'ready',
connection: {
slug: 'backend-creation-connection',
providerType: 'openai',
baseUrl: provider.baseUrl,
enabledModelIds: [modelId],
models: [
{
id: modelId,
capabilities: { chat: true, functionCalling: true },
modalities: { input: ['text', 'image', 'pdf'], output: ['text'] },
contextWindow: 8_192,
maxOutputTokens: 1_024,
},
],
},
networkProxy: { enabled: false },
secretMaterial: { connection: { secret: API_KEY } },
}),
readPricing: async () => ({ revision: 0, overrides: [] }),
artifacts: {
readDurableAttachmentBinary: async ({
artifactId,
sessionId,
}: {
artifactId: string;
sessionId: string;
}) => {
attachmentReads += 1;
assert.equal(artifactId, 'brief');
assert.equal(sessionId, 'backend-creation-session');
return { ok: true, base64: 'JVBERi0=', mimeType: 'application/pdf' };
},
} as unknown as HostAiSdkBackendInput['artifacts'],
}),
);

const events = [];
for await (const event of backend.send({
invocationId: 'pdf-composition-invocation',
runId: 'pdf-composition-run',
turnId: 'pdf-composition-turn',
text: 'Read the attached PDF.',
attachments: [
{
kind: 'pdf',
name: 'brief.pdf',
mimeType: 'application/pdf',
bytes: 8,
ref: {
kind: 'session_file',
sessionId: 'backend-creation-session',
relativePath: 'brief',
},
},
],
context: [],
runtimeContext: [],
})) {
events.push(event);
}

assert.equal(
events.find((event) => event.type === 'complete')?.stopReason,
'end_turn',
JSON.stringify({ events, providerRequests: provider.requests }),
);
assert.equal(attachmentReads, 1);
assert.equal(provider.requests.length, 1);
const messages = provider.requests[0]?.body.messages;
assert.ok(Array.isArray(messages));
const filePart = messages
.flatMap((message: { content?: unknown }) =>
Array.isArray(message.content) ? message.content : [],
)
.find((part: { type?: unknown }) => part.type === 'file');
assert.deepEqual(filePart, {
type: 'file',
file: {
filename: 'brief.pdf',
file_data: 'data:application/pdf;base64,JVBERi0=',
},
});
} finally {
await backend?.dispose();
await provider.close();
}
});

test('provider dispatch fails closed when the Run Composition commit fails', async () => {
const provider = await startProvider();
let commits = 0;
Expand Down Expand Up @@ -3180,6 +3280,7 @@ function backendCreationFixture(input: {
recordModelCallAttempt?: BackendFactoryContext['recordModelCallAttempt'];
createFetchTransport?: HostAiSdkBackendInput['createFetchTransport'];
createRunComposer?: HostAiSdkBackendInput['createRunComposer'];
artifacts?: HostAiSdkBackendInput['artifacts'];
}): HostAiSdkBackendInput {
const runtimePolicy =
input.runtimePolicy ??
Expand Down Expand Up @@ -3252,7 +3353,7 @@ function backendCreationFixture(input: {
...(input.oauthCredentials ? { oauthCredentials: input.oauthCredentials } : {}),
...(input.claudeDeviceId ? { claudeDeviceId: input.claudeDeviceId } : {}),
createRunComposer,
artifacts: {},
artifacts: input.artifacts ?? {},
executionArtifacts: {
recordToolArtifacts: async () => undefined,
toolResultArchive: createToolResultArchiveCapability({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import { buildLlmHistorySummarizer } from '@maka/runtime/history-compact-summari
import { buildOpenAiCodexHistoryCompactor } from '@maka/runtime/openai-codex-history-compactor';
import { buildPricingLookup, recordToolInvocation } from '@maka/runtime/telemetry';
import { buildProviderOptions, getAIModel } from '@maka/runtime/model-factory';
import { resolveModelPdfInputContract } from '@maka/runtime/model-runtime';
import { createProviderRequestCaptureRecorder } from '@maka/runtime/provider-request-telemetry';
import {
createProxiedFetchTransport,
Expand Down Expand Up @@ -132,6 +133,7 @@ export async function createHostAiSdkBackend(input: HostAiSdkBackendInput): Prom
input.context.header.thinkingLevel,
);
const contextWindow = resolveSelectedModelContextWindow(target.connection, target.model);
const pdfInputContract = resolveModelPdfInputContract(target.connection, target.model);
let modelComposition: HostRunComposer;
try {
modelComposition = await readDuringBackendCreation(
Expand Down Expand Up @@ -362,6 +364,7 @@ export async function createHostAiSdkBackend(input: HostAiSdkBackendInput): Prom
target.model,
relayModelProfile(target.connection, target.model)?.vision,
),
...(pdfInputContract ? { pdfInputContract } : {}),
readAttachmentBytes: createAttachmentByteReader({
artifactStore: input.artifacts,
sessionId: input.context.sessionId,
Expand Down
Loading
Loading