feat(knowledge): read a PDF's text layer before paying for OCR

Every PDF went to OCR, an external per-document call, even though most carry an
embedded text layer that costs nothing to read. Across a real corpus of 2,693
documents, local extraction produced text for every PDF that OCR could also read,
so the great majority of those calls bought nothing.

A PDF's text layer is now read first and used when it is good enough, leaving OCR
for the documents that actually need it. Three ways a layer fails, none of which
catches the others: there is no text at all (a scan), the text is too sparse to
be the document, or there is plenty of text that is not language — a broken
encoding, or the raw character ids a CID-keyed font emits with no ToUnicode map,
which is common in exactly the contract and procurement material that reaches a
knowledge base and which a length check alone reads as healthy.

Beyond the cost, this narrows an availability dependency: an OCR outage no longer
touches every PDF, only the minority that cannot be read locally. The threshold is
env-tunable so the balance can be moved toward cost or fidelity without a deploy.

Known limitation: the judgement is per document, so a file mixing typeset pages
with scanned inserts can average above the threshold and keep its partial text.
Per-page routing would catch it and needs per-page extraction this does not have.

The opaque-input refusal now asserts against the outbound request rather than the
storage read: local parsing is not model input, so bytes are read before the
projection is checked and still never leave the worker when it refuses.

Author: waleedlatif1

apps/sim/lib/knowledge/documents/document-processor-secret-provenance.test.ts

@@ -91,7 +91,17 @@ describe('knowledge document model-input provenance', () => {
     expect(fetchMock).not.toHaveBeenCalled()
   })
 
+  /**
+   * The refusal guards egress to an external model, so it is asserted against the
+   * outbound request rather than the storage read. A PDF is now parsed locally
+   * first and only reaches OCR when it has no usable text layer — local parsing is
+   * not model input, as the case above establishes — so the bytes are read before
+   * the projection is checked, and never leave the worker when it refuses.
+   */
   it('rejects secret-bearing opaque document bytes before external OCR', async () => {
+    const fetchMock = vi.fn()
+    vi.stubGlobal('fetch', fetchMock)
+
     await expect(
       runWithKnowledgeModelInputProvenance(
         undefined,
@@ -109,7 +119,7 @@ describe('knowledge document model-input provenance', () => {
       )
     ).rejects.toThrow('Knowledge model input could not be safely projected')
 
-    expect(mockDownloadFileFromUrl).not.toHaveBeenCalled()
+    expect(fetchMock).not.toHaveBeenCalled()
   })
 
   it('attaches exact-empty provenance to the internal Mistral OCR request', async () => {

apps/sim/lib/knowledge/documents/document-processor.ts

@@ -22,6 +22,7 @@ import {
   resolveParserExtension,
   resolveStoredArtifactExtension,
 } from '@/lib/knowledge/documents/parser-extension'
+import { assessPdfTextLayer } from '@/lib/knowledge/documents/pdf-text-layer'
 import { retryWithExponentialBackoff } from '@/lib/knowledge/documents/utils'
 import {
   assertKnowledgeOpaqueModelInputSafe,
@@ -295,6 +296,63 @@ async function getMistralApiKey(workspaceId?: string | null): Promise<string | n
   return env.MISTRAL_API_KEY || null
 }
 
+/**
+ * Reads a PDF's embedded text layer, returning it only when it is good enough to
+ * index — otherwise `undefined`, leaving the caller to fall through to OCR.
+ *
+ * A failure to parse is not an error here: an encrypted or malformed PDF simply
+ * has no usable layer, which is precisely a case for OCR. The document is fetched
+ * again on that path, a second read from our own storage, which is a cheap price
+ * for keeping the two extraction routes independent.
+ */
+async function readEmbeddedPdfText(
+  fileUrl: string,
+  filename: string,
+  mimeType: string,
+  userId?: string
+): Promise<
+  | {
+      content: string
+      processingMethod: 'file-parser'
+      cloudUrl?: string
+      metadata?: FileParseMetadata
+    }
+  | undefined
+> {
+  try {
+    const buffer = await downloadFileWithTimeout(fileUrl, userId)
+    const [parsed, pageCount] = await Promise.all([
+      parseBuffer(buffer, 'pdf'),
+      getPdfPageCount(buffer),
+    ])
+
+    const verdict = assessPdfTextLayer(parsed.content, pageCount)
+    if (!verdict.usable) {
+      logger.info('PDF text layer not usable, routing to OCR', {
+        filename,
+        pageCount,
+        reason: verdict.reason,
+      })
+      return undefined
+    }
+
+    logger.info('Using embedded PDF text layer', { filename, pageCount })
+    return {
+      content: parsed.content,
+      processingMethod: 'file-parser',
+      cloudUrl: undefined,
+      metadata: parsed.metadata,
+    }
+  } catch (error) {
+    logger.info('Could not read PDF text layer, routing to OCR', {
+      filename,
+      mimeType,
+      error: toError(error).message,
+    })
+    return undefined
+  }
+}
+
 async function parseDocument(
   fileUrl: string,
   filename: string,
@@ -319,14 +377,23 @@ async function parseDocument(
       MISTRAL_API_KEY: mistralApiKey,
     }).providerId
 
-    if (ocrProvider === 'azure-mistral') {
-      assertKnowledgeOpaqueModelInputSafe()
-      logger.info('Using Azure Mistral OCR')
-      return parseWithAzureMistralOCR(fileUrl, filename, mimeType, userId)
-    }
+    if (ocrProvider === 'azure-mistral' || ocrProvider === 'mistral') {
+      /**
+       * Most PDFs carry a usable text layer, and reading it costs nothing. OCR is
+       * a per-document call to an external service, so it is reserved for the
+       * documents that actually need it — which also means everything else stops
+       * depending on that service being reachable.
+       */
+      const embedded = await readEmbeddedPdfText(fileUrl, filename, mimeType, userId)
+      if (embedded) return embedded
 
-    if (ocrProvider === 'mistral') {
       assertKnowledgeOpaqueModelInputSafe()
+
+      if (ocrProvider === 'azure-mistral') {
+        logger.info('Using Azure Mistral OCR')
+        return parseWithAzureMistralOCR(fileUrl, filename, mimeType, userId)
+      }
+
       logger.info('Using Mistral OCR')
       return parseWithMistralOCR(fileUrl, filename, mimeType, userId, workspaceId, mistralApiKey)
     }

apps/sim/lib/knowledge/documents/pdf-ocr-triage.test.ts

@@ -0,0 +1,127 @@
+/**
+ * @vitest-environment node
+ *
+ * Every PDF used to be sent to OCR, an external per-document call, even though the
+ * large majority carry a usable text layer that costs nothing to read. These pin
+ * the routing: the text layer is tried first, and OCR is reached only when it is
+ * missing or unreadable.
+ */
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+
+const { mockParseBuffer, mockDownload, mockGetDocumentProxy, mockToken, mockBaseUrl } = vi.hoisted(
+  () => ({
+    mockParseBuffer: vi.fn(),
+    mockDownload: vi.fn(),
+    mockGetDocumentProxy: vi.fn(),
+    mockToken: vi.fn(),
+    mockBaseUrl: vi.fn(),
+  })
+)
+
+vi.mock('@/lib/auth/internal', () => ({ generateInternalToken: mockToken }))
+vi.mock('@/lib/core/utils/urls', async (importOriginal) => ({
+  ...(await importOriginal<typeof import('@/lib/core/utils/urls')>()),
+  getInternalApiBaseUrl: mockBaseUrl,
+}))
+
+vi.mock('@/lib/file-parsers', () => ({
+  parseBuffer: mockParseBuffer,
+  isSupportedFileType: (extension: string) => ['pdf'].includes(extension),
+}))
+vi.mock('@/lib/uploads/utils/file-utils.server', () => ({ downloadFileFromUrl: mockDownload }))
+vi.mock('unpdf', () => ({ getDocumentProxy: mockGetDocumentProxy }))
+
+import { env } from '@/lib/core/config/env'
+import { processDocument } from '@/lib/knowledge/documents/document-processor'
+import { runWithKnowledgeModelInputProvenance } from '@/lib/knowledge/model-input-provenance'
+
+/** External, so the OCR path uses the URL directly instead of re-uploading it. */
+const PDF_URL = 'https://example.com/Contract.pdf'
+const typeset = 'The Supplier shall provide the Services described herein. '.repeat(60)
+
+function parse() {
+  return runWithKnowledgeModelInputProvenance(
+    undefined,
+    () => processDocument(PDF_URL, 'Contract.pdf', 'application/pdf', 1024, 200, 1, 'user-1'),
+    { opaqueInputSafe: true }
+  )
+}
+
+describe('PDF OCR triage', () => {
+  beforeEach(() => {
+    vi.clearAllMocks()
+    Object.assign(env, { OCR_PROVIDER: 'mistral', MISTRAL_API_KEY: 'key' })
+    mockDownload.mockResolvedValue(Buffer.from('%PDF-1.7'))
+    mockGetDocumentProxy.mockResolvedValue({ numPages: 2 })
+    mockToken.mockResolvedValue('internal-token')
+    mockBaseUrl.mockReturnValue('http://sim.local')
+  })
+
+  afterEach(() => {
+    vi.unstubAllGlobals()
+  })
+
+  it('uses the embedded text layer and never calls OCR', async () => {
+    mockParseBuffer.mockResolvedValue({ content: typeset, metadata: {} })
+    const fetchMock = vi.fn()
+    vi.stubGlobal('fetch', fetchMock)
+
+    const result = await parse()
+
+    expect(result.metadata.processingMethod).toBe('file-parser')
+    expect(fetchMock).not.toHaveBeenCalled()
+  })
+
+  it('falls through to OCR when the PDF is a scan', async () => {
+    mockParseBuffer.mockResolvedValue({ content: '', metadata: {} })
+    const fetchMock = vi
+      .fn()
+      .mockResolvedValue(
+        new Response(JSON.stringify({ pages: [{ markdown: 'Recognised text' }], usage_info: {} }), {
+          status: 200,
+          headers: { 'Content-Type': 'application/json' },
+        })
+      )
+    vi.stubGlobal('fetch', fetchMock)
+
+    const result = await parse()
+
+    expect(result.metadata.processingMethod).toBe('mistral-ocr')
+    expect(fetchMock).toHaveBeenCalled()
+  })
+
+  /**
+   * The case a length check alone cannot see: a CID-keyed font with no Unicode map
+   * yields plenty of characters, none of them words.
+   */
+  it('falls through to OCR when the text layer is raw CID escapes', async () => {
+    mockParseBuffer.mockResolvedValue({ content: '/31 /8 /18 /12 /44 '.repeat(60), metadata: {} })
+    const fetchMock = vi.fn().mockResolvedValue(
+      new Response(JSON.stringify({ pages: [{ markdown: 'Recognised' }], usage_info: {} }), {
+        status: 200,
+        headers: { 'Content-Type': 'application/json' },
+      })
+    )
+    vi.stubGlobal('fetch', fetchMock)
+
+    const result = await parse()
+
+    expect(result.metadata.processingMethod).toBe('mistral-ocr')
+  })
+
+  /** An encrypted or malformed PDF has no readable layer, which is a case for OCR. */
+  it('falls through to OCR when the text layer cannot be parsed at all', async () => {
+    mockParseBuffer.mockRejectedValue(new Error('Invalid PDF structure.'))
+    const fetchMock = vi.fn().mockResolvedValue(
+      new Response(JSON.stringify({ pages: [{ markdown: 'Recognised' }], usage_info: {} }), {
+        status: 200,
+        headers: { 'Content-Type': 'application/json' },
+      })
+    )
+    vi.stubGlobal('fetch', fetchMock)
+
+    const result = await parse()
+
+    expect(result.metadata.processingMethod).toBe('mistral-ocr')
+  })
+})

apps/sim/lib/knowledge/documents/pdf-text-layer.test.ts

@@ -0,0 +1,78 @@
+/**
+ * @vitest-environment node
+ */
+import { describe, expect, it } from 'vitest'
+import { assessPdfTextLayer } from '@/lib/knowledge/documents/pdf-text-layer'
+
+/** Roughly the character volume of a typeset page. */
+const page = (n: number) =>
+  'The Supplier shall provide the Services described in this Statement of Work. '.repeat(n)
+
+describe('assessPdfTextLayer', () => {
+  it('accepts an ordinary typeset document', () => {
+    expect(assessPdfTextLayer(page(60), 2)).toEqual({ usable: true })
+  })
+
+  it('rejects a scan, which carries no text at all', () => {
+    expect(assessPdfTextLayer('', 12)).toEqual({ usable: false, reason: 'no-text' })
+    expect(assessPdfTextLayer('   \n  ', 12)).toEqual({ usable: false, reason: 'no-text' })
+  })
+
+  /** A scan often still yields a header or a stamp — present, but not the content. */
+  it('rejects text too sparse to be the document', () => {
+    expect(assessPdfTextLayer('CONFIDENTIAL', 40)).toEqual({
+      usable: false,
+      reason: 'sparse-text',
+    })
+  })
+
+  /**
+   * A CID-keyed font with no `ToUnicode` map extracts as raw character ids. There
+   * is plenty of it, so a length check passes and the document would be indexed as
+   * gibberish — the failure mode a characters-per-page test alone cannot see.
+   */
+  it('rejects raw CID escapes from a font with no Unicode mapping', () => {
+    const cid = '/31 /8 /18 /12 /44 /9 /27 /15 /3 /62 '.repeat(40)
+
+    expect(assessPdfTextLayer(cid, 1)).toEqual({ usable: false, reason: 'cid-escapes' })
+  })
+
+  it('rejects a text layer that decoded to replacement characters', () => {
+    expect(assessPdfTextLayer('�'.repeat(500), 1)).toEqual({
+      usable: false,
+      reason: 'unreadable-encoding',
+    })
+  })
+
+  /** Real prose contains slashes and digits; only a dominant share is disqualifying. */
+  it('keeps a document that merely mentions figures and dates', () => {
+    const prose = `${page(40)} Payment of /50 net 30, effective 01/04/2026, ref /12 /9.`
+
+    expect(assessPdfTextLayer(prose, 1)).toEqual({ usable: true })
+  })
+
+  it('keeps accented and non-Latin prose, which is ordinary text', () => {
+    expect(assessPdfTextLayer('Zusammenfassung über Verträge. '.repeat(40), 1)).toEqual({
+      usable: true,
+    })
+    expect(assessPdfTextLayer('契約の概要について説明します。'.repeat(40), 1)).toEqual({
+      usable: true,
+    })
+  })
+
+  /** An unparseable page count must still apply a floor rather than divide by zero. */
+  it('treats an unknown page count as a single page', () => {
+    expect(assessPdfTextLayer('short', 0)).toEqual({ usable: false, reason: 'sparse-text' })
+    expect(assessPdfTextLayer(page(40), 0)).toEqual({ usable: true })
+  })
+
+  it('scales the threshold with length, so one good page does not carry a long scan', () => {
+    const onePageOfText = page(30)
+
+    expect(assessPdfTextLayer(onePageOfText, 1)).toEqual({ usable: true })
+    expect(assessPdfTextLayer(onePageOfText, 200)).toEqual({
+      usable: false,
+      reason: 'sparse-text',
+    })
+  })
+})