[MaxGhenis]

Google re-routed two Gemini Flash names that have rows on the board. gemini-3.7-flash now reaches Gemini 3.8 Flash, and gemini-3.5-flash now reaches Gemini 3.6 Flash. Both rows were recorded before the successor models existed, so the published rows are the original models, but neither name can re-run them. policybench/config.py already records the same situation for DeepSeek's deepseek-v4-flash and deepseek-v4-pro. This PR records it for the two Gemini names. It changes documentation and adds one test; no score, reference or data changes.

What Google says

From the Gemini API changelog, entry dated October 8, 2026 (read 2026-10-13 19:15 UTC):

Gemini 3.7 Flash deprecation: gemini-3.7-flash is deprecated and has been replaced by gemini-3.8-flash. It is no longer available to select in Google AI Studio, and API requests to gemini-3.7-flash are automatically routed to gemini-3.8-flash.

Gemini 3.5 Flash deprecation: gemini-3.5-flash is deprecated and has been replaced by gemini-3.6-flash. It is no longer available to select in Google AI Studio, and API requests to gemini-3.5-flash are automatically routed to gemini-3.6-flash.

The changelog lists no preview of either successor before its GA date: Gemini 3.6 Flash on 2026-07-21 and Gemini 3.8 Flash on 2026-09-02. Those match MODEL_RELEASE_DATES.

What a live call returns (2026-10-13)

I sent one prompt ("Reply with the single word OK.") to each id through the first-party Gemini API, v1beta generateContent:

Requested	modelVersion in the response	responseId	Response date
gemini-3.7-flash	gemini-3.8-flash	x4POapazCbe_39IPxILouQ0	19:17:28 GMT
gemini-3.5-flash	gemini-3.6-flash	yIPOauTkPLSGjrEPj5bQ8Qg	19:17:30 GMT
gemini-3.8-flash	gemini-3.8-flash	yoPOapnWGoPh_uMP3ZavuQk	19:17:32 GMT
gemini-3.6-flash	gemini-3.6-flash	zIPOaqPcE5fy-8YP1PWukQ4	19:17:33 GMT

GET models/gemini-3.7-flash and GET models/gemini-3.5-flash still return each retired model's own version string (3.7-flash-08-2026 and 3.5-flash-05-2026). The model listing does not show the re-route.

A synchronous run would not show the re-route

litellm 1.88.1, the locked version, sets the response model to the requested name (model_response.model = model in llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py). The string modelVersion does not occur anywhere in the litellm package. eval_no_tools._extract_provider_fingerprint stores that field as provider_resolved_model.

I checked this on single requests by capturing Google's raw body and PolicyBench's record of the same call (2026-10-13 19:21 UTC):

litellm id	Google's modelVersion	PolicyBench records provider_resolved_model	Same responseId
gemini/gemini-3.7-flash	gemini-3.8-flash	gemini-3.7-flash	v4TOatjdDLLijrEPy57z4Qg
gemini/gemini-3.5-flash	gemini-3.6-flash	gemini-3.5-flash	yYTOarmcGfqOjrEPlsLoqA0

So on the synchronous path, which row repairs also use (row_repair → eval_no_tools.run_single_no_tools), a Gemini row's provider_resolved_model echoes the request and cannot tell which model answered. A synchronous re-run or row repair under either retired name would be labelled with the retired model. The Gemini batch path is different: GeminiBatchAdapter (policybench/batch_eval.py) calls google-genai directly and records response.model_version, so a batch run would show the re-route. Review r1 caught this, and the comments say so. That's why the claim about the existing rows rests on when they were recorded, not on that field.

When the board's rows were recorded

Both rows have 1,984 rows (100 calls) in the frozen predictions.csv.gz:

• gemini-3.7-flash: every row records request_started_at between 2026-08-18 00:58:33 and 01:08:27 UTC. Gemini 3.8 Flash was released 2026-09-02. The canonical row hash (scripts/freeze_snapshot._model_prediction_rows_sha256) is 6a3c0abb…, the frozen_rows_sha256 the serving config already records.
• gemini-3.5-flash: no request times. These rows predate request-time capture, which #82 added on 2026-07-02. Their canonical hash, a3d96578…, is the same in the first committed frozen predictions, e456371 (Refresh to the June 2026 populace run (US-only) #74, merged 2026-06-15 16:56 UTC). Gemini 3.6 Flash was released 2026-07-21.

Changes

• policybench/config.py: comments beside the gemini-3.5-flash and gemini-3.7-flash MODELS entries, in the style of the DeepSeek note. They give the deprecation date, where each name routes now, the live observation, and that the rows are the original models and the names can no longer re-run them. They also say a synchronous run's provider_resolved_model and GET models/<name> don't show the re-route, while the batch path records modelVersion. The mappings themselves are unchanged.
• docs/runbook.md: a paragraph after the DeepSeek one, which is the existing place that lists names that can't be re-run.
• tests/test_snapshot_artifacts.py: test_board_rows_under_retired_gemini_names_stay_as_recorded, one case per name. It pins a hash of each row's answers and checks that every recorded request time precedes the successor's release. Since provider_resolved_model can't flag a synchronous re-run or repair under these names, this test does.
  • The hash covers scenario_id, variable, prediction, explanation, raw_response and provider_response_id. The cells are read as strings, and only predictions are parsed, with Python's float.
  • It doesn't use freeze_snapshot's canonical row hash, because that hash isn't portable. For the same file it gave a3d96578…/6a3c0abb… on macOS arm64 under both Python 3.12 and 3.14, and 9393d47c…/049865e7… on the Linux CI runner, with pandas 3.0.1 and numpy 2.4.2 on both. The cause isn't isolated; that's split out as a separate task.

Not changed:

• docs/benchmark_card.md and the paper (paper/index.qmd, the tbl-model-runs caption) don't list re-runnable aliases, and neither carries the DeepSeek note. They describe the frozen rows, which this doesn't change.
• MODEL_RELEASE_DATES in paper_results.py holds release dates only.
• No existing test checks the config comments for retired aliases. test_deepseek_models_are_public_defaults pins only the id mapping.

Invariants

These hold for every frozen snapshot. The new test checks them on the committed one.

• The frozen answers under a retired Gemini name are the rows recorded before the re-route, as pinned by the answer hash. By design the hash ignores surrounding whitespace and the way a number is written.
• No request time recorded under a retired name is on or after the successor model's release.

Verification

• pytest tests/test_snapshot_artifacts.py -k retired_gemini passes locally. The answer hash is the same under Python 3.12.14 and 3.14.0. Ruff format and check are clean. CI on 4c2cd37 failed only these two cases (4,419 passed), on the old canonical-hash pins; 6d91fb2 replaces them.
• Mutation check, 21 cases, all as intended (scripts/mutate_guard.py in the evidence folder).
  • These fail the test for both names: an edit inside a raw_response, an edited explanation or prediction, a changed provider_response_id, an extra row, a dropped scenario, and a post-release request_started_at.
  • These pass, as intended: an unchanged copy, an edit to another model's rows, trailing whitespace on a raw_response, and a prediction written another way with the same value.
• git merge-tree against Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221's head (90eb3c1f) merges cleanly. Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221 adds gemini-4-argon below gemini-3.5-flash-lite, and these comments sit above gemini-3.5-flash and gemini-3.7-flash, with unchanged lines between the two edits.

Evidence (changelog HTML snapshot, sha256 f5105a03…; raw live responses; probe scripts) is in ~/reviews/policybench-gemini-flash-alias-retirement-2026-10-13/.

Review

• r1, GPT-6.1 Sol at bd41607: CHANGES REQUESTED, one should-fix. It found that the claim the re-route never shows in a run's output overstated things, because the Gemini batch path records modelVersion. Fixed in 4c2cd37, which limits the claim to synchronous runs and row repairs. r1 independently verified every other claim: the changelog, the live responses, the litellm mechanism, row dates and hashes, test quality, scope, and the clean merge with Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221.
• r2 was cancelled before launch, because its head went stale while the host's battery hold froze Subfleet.
• r3, GPT-6.1 Sol at 6d91fb2: AGREE, no findings. It confirmed that 4c2cd37 resolves r1 exactly. It reproduced both answer-hash pins independently, with the standard-library CSV parser under Python 3.12.14 and 3.14.0. It confirmed all 21 mutation cases, and found nothing further in the full diff. Its output is review_r3.md in the evidence folder.

Follow-up (not in this PR)

Synchronous Gemini runs should record Google's modelVersion as provider_resolved_model, as the batch path already does, so a future re-route shows up in the data. That's a harness change, split out as a separate task.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 13, 2026 10:25pm UTC