GPT-5.6: correct every rate from the model cards, curate the models (PR-3), and fix the tool that missed them - #980
Merged
Conversation
…-wrong blend `--rates-only` could never have produced a usable number. Three defects, all found by actually running it against dev-ai: 1. It filtered usage types on the substring `gpt-5.6`. No usage type contains a model id, so the filter matched nothing and the script reported "Cost Explorer lags ~24h" — a lag message for a search that was never going to match, which is the worst possible failure mode for a tool whose whole job is to answer "have the numbers landed yet?". 2. It multiplied every rate by 1000 to convert from 1K-token units. These models bill through AWS Marketplace in units of **1M tokens**, and Cost Explorer declares the unit in its own `Unit` field. Every derived rate was overstated 1000x. It now reads the declared unit and converts accordingly. 3. It read MONTHLY. Daily rows come back as exact round numbers; a multi-day window silently blends models into an average that looks like a rate. The blend is not hypothetical, and it is why this needed a guard rather than a fix. Marketplace usage types carry the token bucket and the service tier but never the model, so every OpenAI-family model in the account shares the same four rows — verified against USAGE_TYPE grouped by OPERATION and by BILLING_ENTITY; no finer dimension exists. August shows two distinct price cards ($5.50/$27.50 and $2.20/$11.00) and 2026-08-31 is visibly a blend of the two. A rate is therefore only a given model's rate on a day when it was the sole OpenAI-family model to run, so `--table` now reconciles against what we recorded in sessions-metadata and refuses to vouch for a number otherwise. I nearly shipped the mistake this guard prevents: a first read of Aug 20-31 gave a cache-read rate matching gpt-5.4's 0.1x to four decimals, and a reconcile then showed zero GPT calls in that window. The match was coincidence. This also closes off the spec's Option 1. The Price List API has no Marketplace service code at all (all 269 enumerated), and the Marketplace Catalog API is seller-side. These rates are not unpublished-yet; they are unpublishable through any pricing API while they bill this way, so waiting will not produce them. Bearing on the tier/long-context modelling gap PR-3 must resolve: every row ever seen in this account is `_standard` and no `-long-ctx` usage type has appeared, so a flat standard rate is correct for current traffic and a change would show up as a new usage type. That makes the gap monitorable rather than blocking. Controlled window claimed 2026-09-06 for `us.openai.gpt-5.6-sol` (dev had zero recorded calls beforehand); expected token totals are recorded in the spec so the read is a verification rather than a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…odel cards
I concluded yesterday that these rates existed in no source and had to be
derived empirically. That was wrong, and the error was one of scope: the search
ran against pricing *APIs* — Price List, then Marketplace Catalog — and stopped
there. AWS publishes them in prose on each model's card in the Bedrock User
Guide, alongside caching support, context windows, service tiers and endpoint
support. Absence from an API is not absence from the docs.
Every dev GPT-5.6 row was wrong, and every error over-charged by exactly 20%
(corrected in the dev catalog 2026-09-06T15:59Z):
sol output 26.40 -> 22.00
terra in/out/cache read/write 2.64 / 15.84 / 0.264 / 3.30
-> 2.20 / 13.20 / 0.22 / 2.75
luna in/out/cache read/write 0.264 / 1.584 / 0.0264 / 0.33
-> 0.22 / 1.32 / 0.022 / 0.275
The 1.2x is not coincidence: Terra and Luna were sourced wholesale from the
GovCloud Price List rows, which are exactly 1.2x commercial. Sol's output was
the one figure with no source at all — a 6x input ratio inferred from GovCloud,
where the real ratio is 5x. `openai.gpt-5.4` was already correct, empty
cache-write cell included, so yesterday's prod fix is confirmed by the card.
This also resolves the tier/long-context gap PR-3 was blocked on, rather than
merely downgrading it as the previous commit claimed:
- Service tiers do not apply. Every card says Priority and Flex are not
supported for these models, so the 0.5x/2x dimension does not exist here.
- Long context is real, and the spec's "2x twin" was wrong: above the 272K
threshold input is 2x but output is only 1.5x. A flat 2x would have
over-priced long-context output by a third.
- We do not reach it. All rows carry maxInputTokens 272000, pinned at the
short-context boundary, and compaction runs at 100K — so one short-context
rate is correct, and that cap is what keeps it correct.
Noted for the prod rows: `global.openai.gpt-5.6-*` prices 9.1% below the `us.*`
Geo CRIS card across every bucket, and prod already runs Claude on `global.*`.
Prod should not be a copy of the dev rows.
The empirical work in the previous commit is not wasted — it is now the audit
of these published numbers instead of the source of them, and the 2026-09-06
Sol window should reproduce 4.40 / 0.44 / 5.50 / 22.00 rather than discover it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds `CURATED_BEDROCK_RESPONSES_MODELS` behind a new "Bedrock Responses" catalog tab, so the three GPT-5.6 models are one-click-creatable instead of requiring the escape-hatch form. Rates are the published Geo CRIS short-context row from each AWS model card — Geo CRIS is the tier the `us.*` inference profiles resolve to, and these models are inference-profile-only. Two values are pinned by test because they are pricing correctness, not preference: - `supportsCaching: true`. These models cache implicitly server-side with no way to turn it off, so `false` is not a preference but a false statement, and its only effect is to clear the cache-rate fields — pricing cached tokens at $0.00 while AWS bills them in full. On a warm conversation nearly every input token is a cached one. - `maxInputTokens: 272_000`. These have a 1M window but AWS prices them on two cards: above 272K, input costs 2x and output 1.5x. A CuratedModel holds one flat rate per bucket, so this cap is what keeps that single rate honest. Raising it silently opens the second price card. Fixes the curated `openai.gpt-5.4` Mantle entry in the same pass. It inherited `mantleDefaults()`' `supportsCaching: false`, so one-click-creating it produced exactly the mis-priced row that had to be repaired by hand in prod last night. Its card publishes a cache-read rate at 0.1x input and an em dash for cache write, so caching is on with a literal 0 write rate — 0 is the correct value rather than a missing one, because it makes `compute_wasted_usd` see a non-positive premium and return $0 instead of inventing waste. The `mantleDefaults()` comment claiming Bedrock caching is model-bound to Claude+Nova was simply wrong and is corrected. `claudeRates` becomes `ratesWithDerivedCache`: the 1.25x write / 0.1x read multipliers are not Claude-specific. The GPT-5.6 cards publish the same two, and commercial Cost Explorer billing reproduces them to four decimals — two model families, two independent sources, same ratios. `supportedParams` is deliberately absent from the new entries. AWS publishes no parameter table for GPT-5.6 (`model-parameters-openai.html` covers only the open-weight gpt-oss family), and a declared spec flips the #915 guard from permissive to restrictive — so an invented one would silently block parameters the model actually accepts. Better none than a guess. Not browser-verified: the page is admin-gated against the dev backend, so an unmerged frontend change cannot be signed in to. Layout risk is low — the tab strip is `flex-wrap` and the card grid is unchanged — but the visual check is worth doing on dev after merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three things, one thread
1. The rates
The published rates were in AWS's model cards the whole time (Sol, Terra, Luna, GPT-5.4).
The
1.2×is not coincidence: Terra and Luna had been sourced wholesale from the GovCloud Price List rows, which are exactly 1.2× commercial. Sol's output was the one figure with no source at all — a6×input ratio inferred from GovCloud, where the real ratio is5×.My error, plainly: I concluded these rates existed in no source and had to be derived empirically. The search ran against pricing APIs — Price List, then Marketplace Catalog — and stopped there. Absence from an API is not absence from the docs. The spec now carries that as a standing note: check the model card first. It is where AWS documents rates, caching support, context windows, service tiers and endpoint support together.
2. PR-3 — the models are curated
CURATED_BEDROCK_RESPONSES_MODELSbehind a new Bedrock Responses catalog tab. Two values are pinned by test because they are pricing correctness, not preference:supportsCaching: true— these cache implicitly server-side with no way to turn it off, sofalseis a false statement whose only effect is to clear the cache-rate fields, pricing cached tokens at $0.00 while AWS bills them in full.maxInputTokens: 272_000— the models have a 1M window, but AWS prices them on two cards: above 272K, input costs2×and output1.5×. ACuratedModelholds one flat rate per bucket, so this cap is what keeps that single rate honest. Raising it silently opens the second price card.Also fixes a live bug in the curated
openai.gpt-5.4Mantle entry. It inheritedmantleDefaults()'supportsCaching: false, so one-click-creating it produced exactly the mis-priced row that had to be repaired by hand in prod last night. Its card publishes cache read at 0.1× input and an em dash for cache write, so caching is on with a literal0write rate —0is correct rather than missing, because it makescompute_wasted_usdsee a non-positive premium and return $0 instead of inventing waste. ThemantleDefaults()comment claiming Bedrock caching is model-bound to Claude+Nova was simply wrong, and is corrected.claudeRates→ratesWithDerivedCache: the 1.25×/0.1× multipliers aren't Claude-specific. The GPT-5.6 cards publish the same two, and commercial Cost Explorer billing reproduces them to four decimals — two model families, two independent sources.supportedParamsis deliberately omitted. AWS publishes no parameter table for GPT-5.6, and a declared spec flips the #915 guard from permissive to restrictive — so a guessed one would silently block parameters the model accepts. Better none than a guess.This also resolves PR-3's modelling gap
For prod:
global.openai.gpt-5.6-*prices 9.1% below theus.*rates across every bucket, and prod already runs Claude onglobal.*. The prod rows should not be copies of dev's.3. The tooling
Now the audit of the published numbers rather than the source, and only trustworthy because it's fixed.
--rates-onlyhad three defects, all found by running it:gpt-5.6, but no usage type contains a model id. It matched nothing and printed "Cost Explorer lags ~24h, try tomorrow".Unitfield.Marketplace usage types carry the bucket and tier but never the model, so a rate is only attributable on a single-model day;
--tablenow reconciles againstsessions-metadataand refuses to vouch otherwise. That guard exists because I nearly shipped the mistake it prevents: a first read gave a cache-read rate matchinggpt-5.4's 0.1× to four decimals, and a reconcile then showed zero GPT calls in that window.Also closed
google.gemma-4-31b'ssupportsCaching: falseis correct — across 2026-06-01..09-06 and 455K input tokens it has only ever produced input and output usage types, never a cache SKU.Testing
sidenav.spec.tsfailure; an identical-code re-run was fully green, consistent with the repo's known cross-file flake.us.openai.gpt-5.6-sol.Not browser-verified: the catalog is admin-gated against the dev backend, so an unmerged frontend change can't be signed in to. Layout risk is low (the tab strip is
flex-wrap, the card grid unchanged), but the visual check is worth doing on dev after merge.🤖 Generated with Claude Code