Skip to content

GPT-5.6: correct every rate from the model cards, curate the models (PR-3), and fix the tool that missed them - #980

Merged
philmerrell merged 3 commits into
developfrom
fix/gpt56-rate-derivation-method
Sep 6, 2026
Merged

GPT-5.6: correct every rate from the model cards, curate the models (PR-3), and fix the tool that missed them#980
philmerrell merged 3 commits into
developfrom
fix/gpt56-rate-derivation-method

Conversation

@philmerrell

@philmerrell philmerrell commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Three things, one thread

  1. Every dev GPT-5.6 rate was wrong — all over-charging by exactly 20%. Corrected in the dev catalog 2026-09-06T15:59Z.
  2. PR-3 is now shipped — the three models are curated and one-click-creatable.
  3. The tool that missed the rates is fixed — it had a 1000× unit bug and a filter that could never match.

1. The rates

The published rates were in AWS's model cards the whole time (Sol, Terra, Luna, GPT-5.4).

model field was now
sol output 26.40 22.00
terra input / output / cache read / cache write 2.64 / 15.84 / 0.264 / 3.30 2.20 / 13.20 / 0.22 / 2.75
luna input / output / cache read / cache write 0.264 / 1.584 / 0.0264 / 0.33 0.22 / 1.32 / 0.022 / 0.275

The 1.2× is not coincidence: Terra and Luna had been sourced wholesale from the GovCloud Price List rows, which are exactly 1.2× commercial. Sol's output was the one figure with no source at all — a input ratio inferred from GovCloud, where the real ratio is .

My error, plainly: I concluded these rates existed in no source and had to be derived empirically. The search ran against pricing APIs — Price List, then Marketplace Catalog — and stopped there. Absence from an API is not absence from the docs. The spec now carries that as a standing note: check the model card first. It is where AWS documents rates, caching support, context windows, service tiers and endpoint support together.

2. PR-3 — the models are curated

CURATED_BEDROCK_RESPONSES_MODELS behind a new Bedrock Responses catalog tab. Two values are pinned by test because they are pricing correctness, not preference:

  • supportsCaching: true — these cache implicitly server-side with no way to turn it off, so false is a false statement whose only effect is to clear the cache-rate fields, pricing cached tokens at $0.00 while AWS bills them in full.
  • maxInputTokens: 272_000 — the models have a 1M window, but AWS prices them on two cards: above 272K, input costs and output 1.5×. A CuratedModel holds one flat rate per bucket, so this cap is what keeps that single rate honest. Raising it silently opens the second price card.

Also fixes a live bug in the curated openai.gpt-5.4 Mantle entry. It inherited mantleDefaults()' supportsCaching: false, so one-click-creating it produced exactly the mis-priced row that had to be repaired by hand in prod last night. Its card publishes cache read at 0.1× input and an em dash for cache write, so caching is on with a literal 0 write rate — 0 is correct rather than missing, because it makes compute_wasted_usd see a non-positive premium and return $0 instead of inventing waste. The mantleDefaults() comment claiming Bedrock caching is model-bound to Claude+Nova was simply wrong, and is corrected.

claudeRatesratesWithDerivedCache: the 1.25×/0.1× multipliers aren't Claude-specific. The GPT-5.6 cards publish the same two, and commercial Cost Explorer billing reproduces them to four decimals — two model families, two independent sources.

supportedParams is deliberately omitted. AWS publishes no parameter table for GPT-5.6, and a declared spec flips the #915 guard from permissive to restrictive — so a guessed one would silently block parameters the model accepts. Better none than a guess.

This also resolves PR-3's modelling gap

  • Service tiers don't apply. Every card: "Priority and Flex tiers are not supported for this model."
  • Long context is real, and the spec's "2× twin" was wrong — past 272K input is 2× but output only 1.5×. A flat 2× would have over-priced long-context output by a third.
  • We don't reach it — the 272K cap plus compaction at 100K.

For prod: global.openai.gpt-5.6-* prices 9.1% below the us.* rates across every bucket, and prod already runs Claude on global.*. The prod rows should not be copies of dev's.

3. The tooling

Now the audit of the published numbers rather than the source, and only trustworthy because it's fixed. --rates-only had three defects, all found by running it:

  1. A filter that could never match — it searched usage types for gpt-5.6, but no usage type contains a model id. It matched nothing and printed "Cost Explorer lags ~24h, try tomorrow".
  2. A 1000× error — it assumed 1K-token units; Marketplace rows bill in 1M tokens, and Cost Explorer declares the unit in its own Unit field.
  3. Monthly granularity — daily rows are exact round numbers; a multi-day window blends models into an average that looks like a rate.

Marketplace usage types carry the bucket and tier but never the model, so a rate is only attributable on a single-model day; --table now reconciles against sessions-metadata and refuses to vouch otherwise. That guard exists because I nearly shipped the mistake it prevents: a first read gave a cache-read rate matching gpt-5.4's 0.1× to four decimals, and a reconcile then showed zero GPT calls in that window.

Also closed

google.gemma-4-31b's supportsCaching: false is correct — across 2026-06-01..09-06 and 455K input tokens it has only ever produced input and output usage types, never a cache SKU.

Testing

  • Backend: 7,458 passed, 3 skipped.
  • Frontend: 2,477 passed (220 files). A first run showed one unrelated sidenav.spec.ts failure; an identical-code re-run was fully green, consistent with the repo's known cross-file flake.
  • The rewritten Cost Explorer reader was validated against a window whose ground truth was already known, and both probe arms ran live against us.openai.gpt-5.6-sol.
  • Corrected dev catalog rows verified by reading them back.

Not browser-verified: the catalog is admin-gated against the dev backend, so an unmerged frontend change can't be signed in to. Layout risk is low (the tab strip is flex-wrap, the card grid unchanged), but the visual check is worth doing on dev after merge.

🤖 Generated with Claude Code

philmerrell and others added 2 commits September 6, 2026 09:43
…-wrong blend

`--rates-only` could never have produced a usable number. Three defects, all
found by actually running it against dev-ai:

1. It filtered usage types on the substring `gpt-5.6`. No usage type contains a
   model id, so the filter matched nothing and the script reported "Cost
   Explorer lags ~24h" — a lag message for a search that was never going to
   match, which is the worst possible failure mode for a tool whose whole job
   is to answer "have the numbers landed yet?".
2. It multiplied every rate by 1000 to convert from 1K-token units. These
   models bill through AWS Marketplace in units of **1M tokens**, and Cost
   Explorer declares the unit in its own `Unit` field. Every derived rate was
   overstated 1000x. It now reads the declared unit and converts accordingly.
3. It read MONTHLY. Daily rows come back as exact round numbers; a multi-day
   window silently blends models into an average that looks like a rate.

The blend is not hypothetical, and it is why this needed a guard rather than a
fix. Marketplace usage types carry the token bucket and the service tier but
never the model, so every OpenAI-family model in the account shares the same
four rows — verified against USAGE_TYPE grouped by OPERATION and by
BILLING_ENTITY; no finer dimension exists. August shows two distinct price
cards ($5.50/$27.50 and $2.20/$11.00) and 2026-08-31 is visibly a blend of the
two. A rate is therefore only a given model's rate on a day when it was the
sole OpenAI-family model to run, so `--table` now reconciles against what we
recorded in sessions-metadata and refuses to vouch for a number otherwise.

I nearly shipped the mistake this guard prevents: a first read of Aug 20-31 gave
a cache-read rate matching gpt-5.4's 0.1x to four decimals, and a reconcile then
showed zero GPT calls in that window. The match was coincidence.

This also closes off the spec's Option 1. The Price List API has no Marketplace
service code at all (all 269 enumerated), and the Marketplace Catalog API is
seller-side. These rates are not unpublished-yet; they are unpublishable through
any pricing API while they bill this way, so waiting will not produce them.

Bearing on the tier/long-context modelling gap PR-3 must resolve: every row ever
seen in this account is `_standard` and no `-long-ctx` usage type has appeared,
so a flat standard rate is correct for current traffic and a change would show
up as a new usage type. That makes the gap monitorable rather than blocking.

Controlled window claimed 2026-09-06 for `us.openai.gpt-5.6-sol` (dev had zero
recorded calls beforehand); expected token totals are recorded in the spec so
the read is a verification rather than a guess.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…odel cards

I concluded yesterday that these rates existed in no source and had to be
derived empirically. That was wrong, and the error was one of scope: the search
ran against pricing *APIs* — Price List, then Marketplace Catalog — and stopped
there. AWS publishes them in prose on each model's card in the Bedrock User
Guide, alongside caching support, context windows, service tiers and endpoint
support. Absence from an API is not absence from the docs.

Every dev GPT-5.6 row was wrong, and every error over-charged by exactly 20%
(corrected in the dev catalog 2026-09-06T15:59Z):

  sol    output                     26.40  ->  22.00
  terra  in/out/cache read/write    2.64 / 15.84 / 0.264 / 3.30
                                 ->  2.20 / 13.20 / 0.22  / 2.75
  luna   in/out/cache read/write    0.264 / 1.584 / 0.0264 / 0.33
                                 ->  0.22  / 1.32  / 0.022  / 0.275

The 1.2x is not coincidence: Terra and Luna were sourced wholesale from the
GovCloud Price List rows, which are exactly 1.2x commercial. Sol's output was
the one figure with no source at all — a 6x input ratio inferred from GovCloud,
where the real ratio is 5x. `openai.gpt-5.4` was already correct, empty
cache-write cell included, so yesterday's prod fix is confirmed by the card.

This also resolves the tier/long-context gap PR-3 was blocked on, rather than
merely downgrading it as the previous commit claimed:

- Service tiers do not apply. Every card says Priority and Flex are not
  supported for these models, so the 0.5x/2x dimension does not exist here.
- Long context is real, and the spec's "2x twin" was wrong: above the 272K
  threshold input is 2x but output is only 1.5x. A flat 2x would have
  over-priced long-context output by a third.
- We do not reach it. All rows carry maxInputTokens 272000, pinned at the
  short-context boundary, and compaction runs at 100K — so one short-context
  rate is correct, and that cap is what keeps it correct.

Noted for the prod rows: `global.openai.gpt-5.6-*` prices 9.1% below the `us.*`
Geo CRIS card across every bucket, and prod already runs Claude on `global.*`.
Prod should not be a copy of the dev rows.

The empirical work in the previous commit is not wasted — it is now the audit
of these published numbers instead of the source of them, and the 2026-09-06
Sol window should reproduce 4.40 / 0.44 / 5.50 / 22.00 rather than discover it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@philmerrell philmerrell changed the title fix(costs): derive GPT-5.6 rates from a single-model day, not a 1000x-wrong blend fix(costs): correct every GPT-5.6 rate from the model cards, and fix the tool that missed them Sep 6, 2026
Adds `CURATED_BEDROCK_RESPONSES_MODELS` behind a new "Bedrock Responses"
catalog tab, so the three GPT-5.6 models are one-click-creatable instead of
requiring the escape-hatch form. Rates are the published Geo CRIS
short-context row from each AWS model card — Geo CRIS is the tier the `us.*`
inference profiles resolve to, and these models are inference-profile-only.

Two values are pinned by test because they are pricing correctness, not
preference:

- `supportsCaching: true`. These models cache implicitly server-side with no
  way to turn it off, so `false` is not a preference but a false statement,
  and its only effect is to clear the cache-rate fields — pricing cached
  tokens at $0.00 while AWS bills them in full. On a warm conversation nearly
  every input token is a cached one.
- `maxInputTokens: 272_000`. These have a 1M window but AWS prices them on two
  cards: above 272K, input costs 2x and output 1.5x. A CuratedModel holds one
  flat rate per bucket, so this cap is what keeps that single rate honest.
  Raising it silently opens the second price card.

Fixes the curated `openai.gpt-5.4` Mantle entry in the same pass. It inherited
`mantleDefaults()`' `supportsCaching: false`, so one-click-creating it produced
exactly the mis-priced row that had to be repaired by hand in prod last night.
Its card publishes a cache-read rate at 0.1x input and an em dash for cache
write, so caching is on with a literal 0 write rate — 0 is the correct value
rather than a missing one, because it makes `compute_wasted_usd` see a
non-positive premium and return $0 instead of inventing waste. The
`mantleDefaults()` comment claiming Bedrock caching is model-bound to
Claude+Nova was simply wrong and is corrected.

`claudeRates` becomes `ratesWithDerivedCache`: the 1.25x write / 0.1x read
multipliers are not Claude-specific. The GPT-5.6 cards publish the same two,
and commercial Cost Explorer billing reproduces them to four decimals — two
model families, two independent sources, same ratios.

`supportedParams` is deliberately absent from the new entries. AWS publishes no
parameter table for GPT-5.6 (`model-parameters-openai.html` covers only the
open-weight gpt-oss family), and a declared spec flips the #915 guard from
permissive to restrictive — so an invented one would silently block parameters
the model actually accepts. Better none than a guess.

Not browser-verified: the page is admin-gated against the dev backend, so an
unmerged frontend change cannot be signed in to. Layout risk is low — the tab
strip is `flex-wrap` and the card grid is unchanged — but the visual check is
worth doing on dev after merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@philmerrell philmerrell changed the title fix(costs): correct every GPT-5.6 rate from the model cards, and fix the tool that missed them GPT-5.6: correct every rate from the model cards, curate the models (PR-3), and fix the tool that missed them Sep 6, 2026
@philmerrell
philmerrell merged commit 8a2c371 into develop Sep 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant