feat(ai, ai-openai): EvaluateError keeps usage and answers when OpenAI refuses a question

.changeset/evaluate-refusal-error.md

@@ -0,0 +1,30 @@
+---
+'@tanstack/ai': minor
+'@tanstack/ai-openai': minor
+---
+
+Add `EvaluateError`, a typed error an evaluate adapter throws when the provider answered but the call cannot become a result. `@tanstack/ai` and `@tanstack/ai/adapters` export the class, the `isEvaluateError()` type guard, the `EvaluateErrorInit` type and `EVALUATE_REFUSAL_CODE`. The error keeps what the provider sent back:
+
+- `code`: a stable code to branch on. A refused question uses `EVALUATE_REFUSAL_CODE` (`'refusal'`).
+- `usage`: the tokens the call used.
+- `model`: the model that answered.
+- `answers`: the raw wire answers that came back, before `decide()` maps them. A `choice()` answer has `choice` (for example `{ type: 'choice', choice: 'billing', ... }`), and a `boolean()` answer is `{ type: 'noul', noul: 0.9 }`, not `{ value }`. The refused question has no key.
+
+When OpenAI refuses to answer a question, `decide()` now throws an `EvaluateError` with `code: 'refusal'` instead of a plain `Error`. If more than one question is refused, the message names the first one.
+
+Use `isEvaluateError(error)` to narrow a caught error. It checks `name` and `code`, not `instanceof`, so it still works when the app installs more than one copy of `@tanstack/ai`.
+
+When an adapter throws an `EvaluateError` that carries `usage`, `decide()` reports that usage through the `evaluate:usage` event and the middleware `onUsage` hook, then runs `onError` and rethrows the same error. `onFinish` does not run.
+
+```ts
+try {
+  await decide({ adapter, state, questions })
+} catch (error) {
+  if (isEvaluateError(error) && error.code === EVALUATE_REFUSAL_CODE) {
+    console.log(error.usage, error.answers)
+  }
+  throw error
+}
+```
+
+The message text does not change. The error is still an `Error`, but its `name` is now `'EvaluateError'`, so logs and the OpenTelemetry `error.type` show that name instead of `'Error'`.

docs/adapters/openai.md

@@ -282,7 +282,7 @@ console.log(result.meta.usage);
 
 - `gpt-6-luna` is the only Decisions model.
 - An object `state` goes to OpenAI as JSON text.
-- If OpenAI refuses to answer a question, `decide()` throws an error with the name of that question.
+- If OpenAI refuses to answer a question, `decide()` throws an `EvaluateError` with `code: EVALUATE_REFUSAL_CODE`. The message names the question. The error also has the usage and the raw answers that came back, for example `{ type: 'noul', noul: 0.9 }` for a `boolean()` question. Narrow it with `isEvaluateError(error)`. See [Refused questions](../evaluate/evaluate#refused-questions).
 - `boolean()` questions work. The adapter adds the meaning of true and false to the instructions.
 
 See the [Evaluate guide](../evaluate/evaluate) for the question helpers, the result shape, abort, and middleware.

docs/evaluate/evaluate.md

@@ -411,3 +411,68 @@ try {
   }
 }
 ```
+
+### Refused questions
+
+OpenAI can refuse to answer a question.
+Then `decide()` throws an `EvaluateError` with `code: EVALUATE_REFUSAL_CODE`.
+The error keeps what the provider sent back:
+
+- `usage`: the tokens the call used.
+- `model`: the model that answered.
+- `answers`: the raw answers that came back, before `decide()` maps them. The refused question has no key.
+
+The raw answers do not have `value`.
+A `choice()` answer has `choice`, for example `{ type: 'choice', choice: 'billing', ... }`.
+A `boolean()` answer is `{ type: 'noul', noul: 0.9 }`, where `noul` is the chance of true.
+
+Use `isEvaluateError(error)` to narrow a caught error.
+It checks `name` and `code`, so it works even when the app has two copies of `@tanstack/ai`.
+`instanceof EvaluateError` does not.
+
+```typescript
+import {
+  decide,
+  boolean,
+  choice,
+  isEvaluateError,
+  EVALUATE_REFUSAL_CODE,
+} from '@tanstack/ai'
+import { openaiDecider } from '@tanstack/ai-openai'
+
+try {
+  const result = await decide({
+    adapter: openaiDecider('gpt-6-luna'),
+    state: 'Please refund the extra payment.',
+    questions: {
+      queue: choice({
+        instructions: 'Which team should handle this ticket?',
+        options: {
+          billing: 'Payments, invoices, refunds',
+          tech: 'Bugs, outages, integrations',
+        },
+      }),
+      refund: boolean({
+        instructions: 'Is the customer asking for a refund?',
+      }),
+    },
+  })
+  console.log(result.refund.value)
+} catch (error) {
+  if (isEvaluateError(error) && error.code === EVALUATE_REFUSAL_CODE) {
+    console.error(error.message) // OpenAI declined to answer question "refund"
+    console.log(error.usage)
+    console.log(error.answers?.queue) // { type: 'choice', choice: 'billing', ... }
+  }
+  throw error
+}
+```
+
+The provider still bills a refused call.
+So before `decide()` rethrows an `EvaluateError` that has `usage`, it reports that usage:
+
+1. Middleware `onUsage` runs with the usage.
+2. Middleware `onError` runs with the error.
+3. `decide()` rethrows the same error.
+
+`onFinish` does not run. See [Track usage](#track-usage) for middleware.

packages/ai-openai/src/adapters/evaluate.ts

@@ -1,6 +1,10 @@
 import OpenAI from 'openai'
 import { buildBaseUsage } from '@tanstack/ai'
-import { BaseEvaluateAdapter } from '@tanstack/ai/adapters'
+import {
+  BaseEvaluateAdapter,
+  EVALUATE_REFUSAL_CODE,
+  EvaluateError,
+} from '@tanstack/ai/adapters'
 import { toRunErrorPayload } from '@tanstack/ai/adapter-internals'
 import { getOpenAIApiKeyFromEnv } from '../utils/client'
 import type {
@@ -71,7 +75,7 @@ function toDecisionQuestion(
   }
 }
 
-function toWireAnswer(answer: DecisionAnswer, name: string): WireAnswer {
+function toWireAnswer(answer: DecisionAnswer): WireAnswer | undefined {
   switch (answer.type) {
     case 'predicate':
       return { type: 'noul', noul: answer.probability }
@@ -97,7 +101,7 @@ function toWireAnswer(answer: DecisionAnswer, name: string): WireAnswer {
         confidence: answer.confidence,
       }
     case 'refusal':
-      throw new Error(`OpenAI declined to answer question "${name}"`)
+      return undefined
   }
 }
 
@@ -144,22 +148,35 @@ export class OpenAIEvaluateAdapter<
       // Answers come back in question order. `name` is null only for
       // unnamed questions, and we always send names. Object.fromEntries keeps
       // a question named "__proto__" as an own key.
+      const refused: Array<string> = []
       const answers: Record<string, WireAnswer> = Object.fromEntries(
-        decision.answers.map((answer, index) => {
+        decision.answers.flatMap((answer, index) => {
           const name = answer.name ?? questionKeys[index] ?? String(index)
-          return [name, toWireAnswer(answer, name)]
+          const wire = toWireAnswer(answer)
+          if (!wire) {
+            refused.push(name)
+            return []
+          }
+          return [[name, wire]]
         }),
       )
+      const usage = buildBaseUsage({
+        promptTokens: decision.usage.input_tokens,
+        completionTokens: decision.usage.output_tokens,
+        totalTokens: decision.usage.total_tokens,
+      })
 
-      return {
-        model: decision.model,
-        answers,
-        usage: buildBaseUsage({
-          promptTokens: decision.usage.input_tokens,
-          completionTokens: decision.usage.output_tokens,
-          totalTokens: decision.usage.total_tokens,
-        }),
+      if (refused.length > 0) {
+        throw new EvaluateError({
+          message: `OpenAI declined to answer question "${refused[0]}"`,
+          code: EVALUATE_REFUSAL_CODE,
+          usage,
+          model: decision.model,
+          answers,
+        })
       }
+
+      return { model: decision.model, answers, usage }
     } catch (error) {
       // toRunErrorPayload keeps request headers (and the API key) out of logs.
       logger.errors(`${this.name}.evaluate fatal`, {

packages/ai-openai/tests/evaluate-adapter.test.ts

@@ -1,5 +1,6 @@
 import { describe, expect, it, vi } from 'vitest'
 import { resolveDebugOption } from '@tanstack/ai/adapter-internals'
+import { EVALUATE_REFUSAL_CODE, EvaluateError } from '@tanstack/ai/adapters'
 import { OpenAIEvaluateAdapter } from '../src/adapters/evaluate'
 import type { EvaluateOptions, WireQuestion } from '@tanstack/ai/adapters'
 import type { Decision } from 'openai/resources/decisions'
@@ -193,9 +194,44 @@ describe('OpenAI evaluate adapter', () => {
     expect(Object.keys(result.answers)).toEqual(['__proto__'])
   })
 
-  it('throws when a question is refused', async () => {
-    const { adapter } = setup(decision([{ type: 'refusal', name: 'refund' }]))
-    await expect(run(adapter)).rejects.toThrow('"refund"')
+  it('throws an EvaluateError with usage and the other answers when a question is refused', async () => {
+    const { adapter } = setup(
+      decision([
+        ...allAnswers.answers.slice(0, 2),
+        { type: 'refusal', name: 'refund' },
+      ]),
+    )
+    const error = await run(adapter).catch((e: unknown) => e)
+    expect(error).toBeInstanceOf(EvaluateError)
+    if (!(error instanceof EvaluateError)) return
+    expect(error.code).toBe(EVALUATE_REFUSAL_CODE)
+    expect(error.message).toBe('OpenAI declined to answer question "refund"')
+    expect(error.model).toBe('gpt-6-luna')
+    expect(error.usage).toEqual({
+      promptTokens: 40,
+      completionTokens: 0,
+      totalTokens: 40,
+    })
+    expect(Object.keys(error.answers ?? {})).toEqual(['queue', 'urgency'])
+    expect(error.answers?.queue).toMatchObject({
+      type: 'choice',
+      choice: 'billing',
+    })
+  })
+
+  it('names the first refused question and leaves out every refused key', async () => {
+    const { adapter } = setup(
+      decision([
+        { type: 'refusal', name: 'queue' },
+        allAnswers.answers[1]!,
+        { type: 'refusal', name: 'refund' },
+      ]),
+    )
+    const error = await run(adapter).catch((e: unknown) => e)
+    expect(error).toBeInstanceOf(EvaluateError)
+    if (!(error instanceof EvaluateError)) return
+    expect(error.message).toBe('OpenAI declined to answer question "queue"')
+    expect(Object.keys(error.answers ?? {})).toEqual(['urgency'])
   })
 
   it('rethrows SDK errors', async () => {

packages/ai/src/activities/evaluate/adapter.ts

@@ -137,6 +137,82 @@ export interface EvaluateAdapterResult {
   provider?: string
 }
 
+/** `EvaluateError.code` when the provider declined to answer a question. */
+export const EVALUATE_REFUSAL_CODE = 'refusal'
+
+/** Fields for {@link EvaluateError}. */
+export interface EvaluateErrorInit {
+  /** Human-readable error text. */
+  message: string
+  /** Stable code to branch on, for example `EVALUATE_REFUSAL_CODE`. */
+  code: string
+  /** Tokens the provider reported for this call. */
+  usage?: TokenUsage
+  /** Resolved model id from the provider. */
+  model?: string
+  /**
+   * Wire answers the provider did return, before `decide()` maps them. A
+   * `choice()` answer has `choice`, and a `boolean()` answer is
+   * `{ type: 'noul', noul }`. A refused question has no key.
+   */
+  answers?: Record<string, WireAnswer>
+}
+
+/**
+ * Thrown by an evaluate adapter when the provider answered, but the call
+ * cannot become a result. It carries what the provider already returned,
+ * so callers can still read the billed usage and the answers that came back.
+ *
+ * The `name` is always `EvaluateError`.
+ * Another package can read this shape without an import of this class.
+ * Use {@link isEvaluateError} to narrow a caught value. `instanceof` fails
+ * when the app has more than one copy of `@tanstack/ai`.
+ */
+export class EvaluateError extends Error {
+  /** Stable code to branch on, for example `EVALUATE_REFUSAL_CODE`. */
+  readonly code: string
+  /** Tokens the provider reported for this call. */
+  readonly usage?: TokenUsage
+  /** Resolved model id from the provider. */
+  readonly model?: string
+  /**
+   * Wire answers the provider did return, before `decide()` maps them. A
+   * `choice()` answer has `choice`, and a `boolean()` answer is
+   * `{ type: 'noul', noul }`. A refused question has no key.
+   */
+  readonly answers?: Record<string, WireAnswer>
+
+  constructor(init: EvaluateErrorInit) {
+    super(init.message)
+    this.name = 'EvaluateError'
+    this.code = init.code
+    this.usage = init.usage
+    this.model = init.model
+    this.answers = init.answers
+  }
+}
+
+/**
+ * This function returns true when `value` has the evaluate error shape.
+ *
+ * The check reads `name` and `code`.
+ * A plain object with those fields passes.
+ * The check does not use `instanceof`, so it also matches an error from
+ * another copy of `@tanstack/ai`.
+ *
+ * @param value - A thrown value, or a plain object
+ *
+ * @example
+ * if (isEvaluateError(error) && error.code === EVALUATE_REFUSAL_CODE) {
+ *   error.usage
+ * }
+ */
+export function isEvaluateError(value: unknown): value is EvaluateError {
+  if (typeof value !== 'object' || value === null) return false
+  if (!('name' in value) || value.name !== 'EvaluateError') return false
+  return 'code' in value && typeof value.code === 'string'
+}
+
 /**
  * Evaluate adapter interface with pre-resolved generics.
  *

packages/ai/src/activities/evaluate/index.ts

@@ -17,6 +17,7 @@ import {
   runGenerationStart,
   runGenerationUsage,
 } from '../middleware/run'
+import { isEvaluateError } from './adapter'
 import type { InternalLogger } from '../../logger/internal-logger'
 import type { DebugOption } from '../../logger/types'
 import type { TokenUsage } from '../../types'
@@ -454,6 +455,10 @@ export function boolean(options: {
  * The question key `meta` is reserved. Throws if `questions` is empty or uses
  * that key.
  *
+ * When the adapter throws an `EvaluateError` that carries `usage`, `decide()`
+ * reports that usage through `onUsage` and the `evaluate:usage` event, then
+ * runs `onError` and rethrows the same error.
+ *
  * @param options.adapter Evaluate adapter created with a model.
  * @param options.state Shared state every question judges.
  * @param options.questions Questions built with `choice`, `score`, `boolean`.
@@ -602,15 +607,27 @@ export async function decide<
     })
   } catch (error) {
     const duration = Date.now() - startTime
-    if (isAbortError(error, abortSignal)) {
-      await runGenerationAbort(middleware, mwCtx, {
-        reason: error instanceof Error ? error.message : undefined,
-        duration,
-      })
-    } else {
-      await runGenerationError(middleware, mwCtx, { error, duration })
+    try {
+      if (isEvaluateError(error) && error.usage) {
+        aiEventClient.emit('evaluate:usage', {
+          requestId,
+          model: error.model ?? model,
+          usage: error.usage,
+          timestamp: Date.now(),
+        })
+        await runGenerationUsage(middleware, mwCtx, error.usage)
+      }
+    } finally {
+      if (isAbortError(error, abortSignal)) {
+        await runGenerationAbort(middleware, mwCtx, {
+          reason: error instanceof Error ? error.message : undefined,
+          duration,
+        })
+      } else {
+        await runGenerationError(middleware, mwCtx, { error, duration })
+      }
+      logger.errors('evaluate activity failed', { error, source: 'evaluate' })
     }
-    logger.errors('evaluate activity failed', { error, source: 'evaluate' })
     throw error
   }
 }

packages/ai/src/activities/index.ts

@@ -138,6 +138,10 @@ export {
 
 export {
   BaseEvaluateAdapter,
+  EvaluateError,
+  EVALUATE_REFUSAL_CODE,
+  isEvaluateError,
+  type EvaluateErrorInit,
   type EvaluateAdapter,
   type EvaluateAdapterConfig,
   type AnyEvaluateAdapter,