Skip to content

feat(openai): enable prompt caching for GPT-5.6 - #679

Open
piercebrookins wants to merge 6 commits into
mpfaffenberger:mainfrom
piercebrookins:feat/gpt-5-6-prompt-caching
Open

feat(openai): enable prompt caching for GPT-5.6#679
piercebrookins wants to merge 6 commits into
mpfaffenberger:mainfrom
piercebrookins:feat/gpt-5-6-prompt-caching

Conversation

@piercebrookins

@piercebrookins piercebrookins commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • rebase the branch onto the current upstream main (09085be5)
  • enable stable GPT-5.6 prompt routing keys using the provider model ID and logical agent identity, avoiding per-launch UUID churn
  • replace the per-process UUID only in GPT-5.6 cacheable instructions while retaining the unique internal runtime identity
  • add one explicit prompt_cache_breakpoint after the reusable instruction prefix for the official OpenAI provider
  • keep implicit caching for ChatGPT OAuth, Azure, Azure Foundry, and custom OpenAI-compatible endpoints unless a provider explicitly opts into breakpoints
  • support Responses API and Chat Completions across OpenAI-compatible providers
  • preserve cache-read and cache-write usage for streaming and non-streaming responses, including the distinction between zero and missing values
  • preserve existing extra_body values and expose cache usage with model, agent, API-path, and streaming metadata
  • provide a narrow, default-on GPT-5.6 usage adapter for pinned pydantic-ai 1.56; prompt_cache_breakpoint_enabled controls only the explicit marker
  • isolate cache policy, key construction, API-path selection, compatibility adapters, and focused tests in openai_prompt_cache.py
  • add regression coverage for cross-launch key stability, aliases, explicit boundary placement, provider capability gating, message/attachment ordering, and cache-usage mapping
  • retain upstream GPT-5 legacy reasoning-effort normalization during rebases

This implements the OpenAI team's attached guidance without sending deprecated prompt_cache_retention or overriding implicit cache behavior.

Refactor scope

  • reduced the PR from 13 changed files to 9
  • reduced model_factory.py from 129 additions / 16 deletions to 36 additions / 20 deletions
  • removed cache policy and adapter implementation from model_factory.py; it now delegates through three small integration calls
  • removed redundant changes to base_agent.py, subagent_usage_metrics.py, test_model_factory_coverage.py, and the standalone builder cache test file
  • retained the existing patchable stock-model seams for non-GPT-5.6 models

BreedX feedback addressed

  • cache-key scope is now the stable logical agent name, not assembled instructions containing a per-launch UUID
  • the regression test constructs two separate agents with different identity strings and verifies that both use the same logical cache scope
  • aliases resolving to the same provider model and agent scope produce the same routing key
  • the branch is rebased onto the latest upstream main

Live ChatGPT OAuth validation

  • before ec325fe2, codex-gpt-5.6-sol failed with HTTP 400 because the OAuth Responses backend rejects prompt_cache_breakpoint
  • after ec325fe2, two identical fresh-process requests completed successfully with 5,749 input tokens each
  • both responses reported 0 cached-read tokens, so the observed OAuth sample is 0/2 cache hits; this PR does not claim a measured OAuth hit
  • explicit breakpoint support remains enabled by default for the public OpenAI provider and can be opted into by compatible provider configurations

Live public OpenAI API validation

  • before 9fb5cb2b, three fresh-process Responses requests with an explicit breakpoint reported 0/3 hits because the cacheable prefix contained a random per-process agent UUID
  • local cross-process inspection confirmed the UUID was the only changing instruction field; after the fix, three independent launches produced the same instruction length and SHA-256 hash
  • after 9fb5cb2b, three identical gpt-5.6-sol Responses requests each contained 5,759 input tokens: the cold request read 0 cached tokens, and both warm requests read 5,756 cached tokens
  • observed warm-request cache-hit rate: 2/2 (100%)
  • observed cached-token ratio per warm request: 5,756 / 5,759 (99.95%)

Testing

  • ruff format --check and ruff check pass for all changed files
  • 237 focused model-factory, builder, prompt-cache, subagent, OAuth, and Azure Foundry tests pass after the refactor
  • live ChatGPT OAuth smoke test passes after the provider-capability fix
  • the previous GitHub Actions run passed all quality, macOS Python 3.13, Windows encoding, and computer-use checks; the latest push triggers a fresh run

@breedx

breedx commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

We run 5.6 in production on Azure Chat Completions — 92.6% cache-hit across ~6,600 calls, sustained for weeks, sending no prompt_cache_key.

On this branch, _make_prompt_cache_key returns a different key every launch. Scope is instructions, which ends with get_identity_prompt()self.id = str(uuid.uuid4()):

3 agent instances -> 3 keys;  same instance, 3 calls -> 1 key

Stable within a session, random across them — probably worse than sending nothing, since OpenAI's prefix-hash routing is at least deterministic.

The test misses it because it passes the same instructions string in twice, asserting SHA-256 is deterministic rather than that the input is stable. Nothing builds two agents.

Fix:

_make_prompt_cache_key(model_name, getattr(agent, "name", None))

5 instances → 1 key.

Also: both listed test failures are already fixed on main — needs a rebase.

@piercebrookins
piercebrookins force-pushed the feat/gpt-5-6-prompt-caching branch 2 times, most recently from c0b88fb to 41c7178 Compare August 3, 2026 02:09
@mpfaffenberger

Copy link
Copy Markdown
Owner

@piercebrookins look at @breedx's comments please.

@piercebrookins
piercebrookins force-pushed the feat/gpt-5-6-prompt-caching branch from 41c7178 to 24dcf63 Compare August 3, 2026 15:07
@piercebrookins

Copy link
Copy Markdown
Contributor Author

Addressed the latest live-test issue in ec325fe2. The ChatGPT OAuth Responses backend rejects prompt_cache_breakpoint, so explicit markers are now capability-gated by provider while cache usage normalization remains enabled. Two identical fresh-process OAuth requests now complete successfully (5,749 input tokens each); both reported 0 cached-read tokens, so the current measured OAuth sample is 0/2 hits rather than a claimed hit. The PR description includes the validation details, and 48 focused tests plus formatting/lint pass.

@piercebrookins

Copy link
Copy Markdown
Contributor Author

Verified the cache-prefix fix in 9fb5cb2b against the public OpenAI Responses API with gpt-5.6-sol. Three identical fresh-process requests each had 5,759 input tokens: cold read = 0, warm reads = 5,756 and 5,756. That is a 2/2 (100%) warm-request hit rate and 99.95% cached-token ratio per warm request. Before the fix, both explicit and implicit controls reported zero hits; cross-process inspection isolated the random agent UUID as the only changing instruction field. The PR description now includes the complete data.

@mpfaffenberger

Copy link
Copy Markdown
Owner

Hmm the changes in model_factory.py need to be refactored out as much as possible. That file is already way too big.

13 files changed seems excessive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants