Audit the Medicare Part B premium in medical expenses; propose excluding scenario_114's Virginia income tax

[MaxGhenis]

Summary

policyengine-us 2.15.17 treats every Medicare-eligible person as enrolled (takes_up_medicare_if_eligible defaults to true) and adds the modeled Part B premium to medical expenses (medical_expense_health_insurance_premiums). For 2026 that premium is the $202.90 monthly standard premium, $2,434.80 a year. The engine's IRMAA lookback reads 2024 MAGI, which no household sets, so no income-related amount applies.

No PolicyBench prompt states Medicare enrollment or a Part B premium. The prompt rules say to treat unlisted numbers as 0 and not to infer expenses or health coverage.

This PR adds reference_audit/2026-10-05-medicare-part-b/. It changes no reference and no published number. The exclusion it proposes changes published scores, so it waits for Max's ruling, alongside #191 (decision d963).

What the sweep finds

All 1,984 outputs were recomputed on the reference system (2.15.17 + latest_final); the baseline reproduces all 1,928 scored references. Each output was then rerun with each person's Part B premium at 0, with enrollment off (two ways), and with IRMAA read on 2026 MAGI.

• Who is charged. 39 people in 33 households are Medicare-eligible, all by age. The premium is charged to 35 people in 30 households; the Medicare Savings Program pays it for the other 4.

• What moves. Exactly two outputs move by more than $1, both scored, both in scenario_114 (VA, age 69, itemizes):

  Output	Reference	No Part B premium
  federal income tax before refundable credits	10,729.61	11,265.27 (already proposed in Audit the state income tax withheld in federal SALT; propose three exclusions #191 for the withholding estimate)
  Virginia income tax before refundable credits	3,514.15	3,654.15 (proposed here)

• What does not move. No SNAP or eligibility output changes by any amount under any reading. The README explains household by household why the other 29 do not move:

  • 7 are below the 7.5%-of-AGI floor either way;
  • 21 take the standard deduction either way;
  • 072 stops itemizing but owes $0;
  • the state channels that fire leave a $0 or unchanged output;
  • SNAP is $0, or pays the $24 monthly minimum under both readings.

• Model answers. No model is within $1 of either frozen value.

  • GPT-6 Astra and GPT-6.1 Sol are within $1 of the Virginia value without the premium.
  • On the federal output, GPT-6.1 Sol and GPT-6 Astra land on the no-premium + liability value ($11,642.16).
  • Kimi K3 and GPT-5.6 Sol land on the literal reading ($11,969.80): no premium and no state income tax paid.

Records

• proposed_exclusions.json holds the Virginia record.
• It also holds a standalone federal record, to install only if Audit the state income tax withheld in federal SALT; propose three exclusions #191's record for that output is not adopted. Audit the state income tax withheld in federal SALT; propose three exclusions #191's record already quotes the no-premium values.

Leaderboard impact

From verification/leaderboard_impact.json, via policybench.cli analyze on scratch copies. The unchanged copy reproduces the published payload.

Case	Exact-rate change	GPT-6 Sol exact	Exact rank changes
this audit alone (2 records)	+0.18 to +0.26 points	95.00 → 95.27%	none
#191 alone (3 records)	+0.42 to +0.61 points	95.00 → 95.61%	#191's
#191 and the Virginia record	+0.47 to +0.69 points	95.00 → 95.69%	#191's

Related findings

• Stale impact weights, with no effect on scores. The published eligibility impact weights (90 rows) predate 2.15.17, because the 2026-09-29 rebuild rewrote values only. Scoring with 2.15.17's weights changes no compared payload value.
• scenario_031's Medicaid annotations. They explain a correct reference through a Part B subtraction the engine never makes; the engine applies a $230 monthly California disregard instead. A separate task covers the fix.
• Audit the state income tax withheld in federal SALT; propose three exclusions #191's README rounding. Audit the state income tax withheld in federal SALT; propose three exclusions #191's README rounds 95.6146 to 95.62% and 70.3946 to 70.40%; both should round down, to 95.61% and 70.39%.

Invariants checked

• The baseline reproduces every scored reference.
• not_enrolled and not_enrolled_direct agree on all 1,984 outputs.
• Scoring the unchanged copy reproduces the published payload exactly.
• The proposal script refuses to run if the baseline mismatches, or if the enrollment readings move a different output set from no_part_b.
• Rerunning the proposal and explain scripts gives byte-identical files.

Review

Independent reviews (a mechanism trace with hand recomputation, a differential re-implementation, a prompt and answer audit, a completeness critic, and an adversarial Subfleet review) are running against this head. Their reports will be added under verification/.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 5, 2026 6:23pm UTC

[MaxGhenis]

scenario_114 annotation check, for the release that adopts these records (from #197, which fixes scenario_031's Medicaid text and leaves 114 to this PR, as agreed)

The README's "Related findings" says the adopting release rewords both 114 case notes. The row annotations need it too, because excluded outputs keep their row text as description.

Recomputed on 2.15.17 + latest_final:

Reading	Federal medical deduction	VA itemized	VA tax	Federal tax
Reference	4,441.86	19,419.94	3,514.15	10,729.61
Part B = 0	2,007.06	16,985.14	3,654.15	11,265.27
OTC counted, Part B kept	6,441.86	21,419.94	3,399.15	10,289.61
OTC counted, Part B = 0	4,007.06	18,985.14	3,539.15	10,825.27

How the reference medical figure is built: itemized_medical_expenses adds only medical_expense_health_insurance_premiums and other_medical_expenses. So the medical base is $10,000 + the imputed $2,434.80 Part B premium = $12,434.80, less 7.5% of AGI ($7,992.94), giving $4,441.86. The $2,000 of OTC is left out. Virginia's $19,419.94 is the non-SALT federal itemized deductions plus the $4,064.29 real estate tax:

• charitable $10,913.78, after the 0.5%-of-AGI floor;

• medical $4,441.86.

• Virginia note, group (4), is wrong on both halves. The reference itself leaves out OTC and applies the charitable floor. Models at $17,518 itemized come to: charitable without the floor ($11,446.65) + medical without Part B ($2,007.06) + real estate tax. Their gap is the Part B premium less the floor, $2,434.80 − $532.86 = $1,901.94.

• Virginia rows repeat it (agent check, numbers spot-verified):

  • Rows saying "$17,518 … leaving out the $2,000 of over-the-counter health expenses": claude-sonnet-5.5, gemini-3.6-flash, gemini-3.7-flash, gemini-3.8-flash, gpt-5.5 and grok-4.5.
  • gpt-6-astra and gpt-6.1-sol, "$16,985.18". The only gap is the premium, and their answers match the no-Part-B value, $3,654.15.
  • claude-fable-5.1, "$18,985". It counted OTC in place of Part B, a gap of $434.80.
  • gemini-3.5-flash. The text puts medical at "$12,000 … above the floor"; the engine has $12,434.80 and $4,441.86.
  • gpt-6-sol. The text's figure is wrong. Its $57,229 matches its own $10,000 AGI understatement plus medical without Part B.
  • gemini-3.1-pro-preview. Its row infers $17,518; the model gave no itemized figure.

• Federal note, group (3), "missing the Part B premium". This is right about the engine but rests on the unlisted premium. Either omission alone (Part B, or state tax in SALT) still leaves itemizing ahead. Only both together reach the standard deduction, $11,969.80, which kimi-k3 and gpt-5.6-sol answered.

• Federal note side point. It gives SALT state income tax as $5,367.59. state_and_local_sales_or_income_tax is $5,367.30, and $4,064.29 + $5,367.30 = $9,431.59. Six federal rows repeat $5,367.59.

• Wording. The federal note's "non-deductible OTC purchases" is a legal label. The engine only shows that it leaves OTC out.

Full agent report (row-by-row, with suggested wording)

scenario_114 (VA, single, age 69): engine check of two case notes and their row annotations

Bottom line: the federal note's medical figures are right, but they rest on a Part B premium the prompt never gives. The Virginia note's group (4) is wrong on both halves. Eleven Virginia row annotations repeat its mistake, and one is ambiguous. The 18 federal rows that mention medical expenses or the premium all match the engine.

System: policyengine-us 2.15.17 plus reference_audit/2026-09-28/fixes/latest_final.py, household from Scenario.to_pe_household. Recomputed baseline: federal 10,729.61, Virginia 3,514.15. The fix files do not touch medical, charitable, Medicare or Virginia itemized variables. Computations: /private/tmp/claude-501/-Users-maxghenis-Library-Application-Support-Claude-scratch-workspaces-ee3763f4-e7ea-4177-bcaf-1362266768c1-4457aa9d-2dc0-4bcf-a128-e50adae6be25-scratch-2026-09-22-c3da5e/2fe8b63d-984e-40c0-a80b-d2491d3d138e/scratchpad/s114.py. The scenario_114 data in the Part B sweep files (README, sweep_part_b_households.json, part_b_households.csv) agrees with these results.

Below, pe/ = /Users/maxghenis/PolicyEngine/policybench/results/local/adds202609/triage/.venv-pe21517/lib/python3.13/site-packages/policyengine_us.

Inputs

• From scenarios.csv: other_medical_expenses 10,000; over_the_counter_health_expenses 2,000; charitable cash 10,766 and non-cash 680.65; real_estate_taxes 4,064.29.
• No health insurance premium, Medicare enrollment or Part B input is set.

How the engine builds these numbers

• What counts as medical: pe/variables/gov/irs/income/taxable_income/deductions/itemizing/itemized_medical_expenses.py:25-28 adds only medical_expense_health_insurance_premiums and other_medical_expenses. Its docstring (lines 20-22) says over-the-counter (OTC) expenses are excluded.
• Where OTC goes: the only formula that reads over_the_counter_health_expenses is pe/variables/household/income/spm_unit/spm_unit_non_premium_medical_out_of_pocket_expenses.py:18. Nothing in the IRS itemizing code or gov/states/va reads it.
• Where the premium comes from:
  • pe/variables/household/expense/health/medical_expense_health_insurance_premiums.py:18-25: because health_insurance_premiums is 0, the premium term is non-Part-B premiums (0) plus medicare_part_b_premium × medicare_enrolled.
  • pe/variables/gov/hhs/medicare/costs/medicare_enrolled.py:11-12 and pe/variables/gov/hhs/medicare/takes_up_medicare_if_eligible.py:9 (default True) make the filer enrolled.
  • pe/variables/gov/hhs/medicare/eligibility/part_b/base_part_b_premium.py:14-16 with pe/parameters/gov/hhs/medicare/part_b/base_premium.yaml:25 ($202.90 a month) gives $2,434.80. Medicare Savings Program coverage and the income-related surcharge are both 0.
• Medical deduction: .../medical_expense_deduction.py:14-17: 12,434.80 − 7.5% × 106,572.56 (7,992.94) = 4,441.86.
• Charitable deduction: .../charitable_deduction.py:51-75 applies the 2026 floor of 0.5% of AGI (532.86): 11,446.65 − 532.86 = 10,913.78.
• Virginia itemized deductions:
  • pe/variables/gov/irs/income/taxable_income/deductions/itemized_deductions_less_salt.py:13-19 sums the non-SALT itemized deductions: 15,355.64.
  • pe/variables/gov/states/va/tax/income/deductions/va_itemized_deductions.py:16-23, 31-33, 38-41: AGI of 106,572.56 is below the 2026 single applicable amount of $340,200 (computed from the parameter), so no limitation applies. The result is non-SALT itemized deductions plus uncapped real estate tax, with state income tax added and then removed: 15,355.64 + 4,064.29 = 19,419.94.
  • va_deductions.py:17-20 uses it because the filer itemizes federally.
  • So Virginia's medical deduction is the federal 4,441.86: it includes the Part B premium, excludes OTC, and sits next to the charitable deduction after the floor.

Counterfactuals (computed)

Case	Federal medical deduction	VA itemized	VA tax	Federal tax	Itemizes federally
Reference	4,441.86	19,419.94	3,514.15	10,729.61	yes
Part B = 0	2,007.06	16,985.14	3,654.15	11,265.27	yes
OTC added, Part B kept	6,441.86	21,419.94	3,399.15	10,289.61	yes
OTC added, Part B = 0	4,007.06	18,985.14	3,539.15	10,825.27	yes
Part B kept, no state income tax in SALT (sweep grid)	4,441.86	n/a	3,514.15	11,758.09	yes
Part B = 0 and no state income tax (sweep grid)	2,007.06	n/a	4,127.67	11,969.80	no

The model itemized totals break down exactly. Charitable is either 10,913.78 (after the floor) or 11,446.65 (no floor), plus real estate tax 4,064.29:

• 16,985.14 = after floor + 2,007.06 medical (no Part B)
• 17,518.00 = no floor + 2,007.06
• 18,985.14 = after floor + 4,007.06 (OTC counted, no Part B)
• 19,518.00 = no floor + 4,007.06
• 15,510.94 = no floor + no medical

(a) Federal note (federal_income_tax_before_refundable_credits)

(a1) Group (3), "missing the Part B premium". Verdict: correct-but-depends-on-unlisted-input.

• The premium is an engine assumption (enrollment defaults to True) that no prompt line states.
• The clause "and so picking the standard deduction" holds only when both omissions happen. Either one alone still leaves the engine itemizing: 11,265.27 without Part B, 11,758.09 without state income tax. Both together give the standard path at 11,969.80, which is what kimi-k3 (11,969.85) and gpt-5.6-sol (11,970) answered.
• gpt-6-astra (11,642.20) and gpt-6.1-sol (11,642.09) equal the engine's reading with no Part B and the state tax liability in SALT (11,642.16).
• Suggested wording: "(3) understating itemized deductions by leaving state income tax out of SALT or by leaving out the $2,434.80 Medicare Part B premium that the engine counts in medical expenses because it treats a person aged 65 or older as enrolled in Medicare (the prompt lists no premium); in the engine either omission alone still leaves itemizing ahead, and both together make the standard-deduction path ($11,969.80) the result;"

(a2) The medical sentence. Verdict: correct-but-depends-on-unlisted-input.

• The arithmetic checks out (12,434.80 − 7,992.94 = 4,441.86).
• "Non-deductible" is a legal label. The engine only shows that it leaves OTC out.
• Suggested wording: "medical $4,441.86 (the engine counts $12,434.80: the $10,000 of other medical expenses plus a $2,434.80 Medicare Part B premium, $202.90 a month, that it imputes because it treats the 69-year-old as enrolled in Medicare; it does not count the $2,000 of over-the-counter health expenses; the deduction is the amount above 7.5% of AGI, $7,992.94)".

Side point: the note says SALT includes "$5,367.59 state income tax". The engine's state_and_local_sales_or_income_tax is 5,367.30, and 4,064.29 + 5,367.30 = 9,431.59. The same $5,367.59 appears in six federal rows: claude-sonnet-5.5, gemini-3.5-flash, gpt-5.6-terra, gpt-6-astra, gpt-6.1-sol, ox-alpha.

(b) Virginia note (state_income_tax_before_refundable_credits)

Group (4), "leaving out the $2,000 over-the-counter expenses or adding the 0.5%-of-AGI charitable floor". Verdict: wrong on both halves.

• The engine's Virginia figure itself leaves out OTC and itself applies the floor. A model that did either one matches the engine on that item.
• What the short models actually miss is the imputed Part B premium:
  • gpt-6-astra (3,654.16) and gpt-6.1-sol (3,654.66) reproduce the engine's no-Part-B result (3,654.15).
  • Models at 17,518 are short by 1,901.94 = 2,434.80 − 532.86.
  • claude-fable-5.1 at 18,985 is short by 434.80 = 2,434.80 − 2,000.
• Suggested wording: "(4) leaving out the $2,434.80 Medicare Part B premium that the engine imputes and counts in the federal medical deduction Virginia carries over, giving Virginia itemized deductions of $16,985 (gpt-6-astra, gpt-6.1-sol) or, with the 0.5%-of-AGI charitable floor also skipped, $17,518; models that counted the $2,000 of over-the-counter expenses in place of the premium reached $18,985 or $19,518."

The sentence "Virginia itemized deductions of $19,419.94 apply (the filer itemizes federally)". Verdict: correct-but-depends-on-unlisted-input (16,985.14 and 3,654.15 without Part B).

• Suggested wording: "From there, the engine applies Virginia itemized deductions of $19,419.94 (the filer itemizes federally): the federal itemized deductions other than SALT ($10,913.78 charitable after the 0.5%-of-AGI floor and a $4,441.86 medical deduction that includes a $2,434.80 Medicare Part B premium the engine imputes and excludes the $2,000 of over-the-counter expenses) plus the $4,064.29 real estate tax, with state income tax removed; federal AGI is below the $340,200 point where Virginia's limitation begins."

Related: the Virginia reference explanation in us_case_reference_explanations.csv says the itemized deductions include "state/local taxes". The engine removes state income tax (va_itemized_deductions.py:31-33) and keeps only the real estate tax.

(c) Row annotations in us_audit_row_annotations.csv

Federal: 18 rows. All match the engine. Verdict for each: correct-but-depends-on-unlisted-input, because each relies on the imputed premium.

1. claude-fable-5: counted $2,000 OTC instead of the $2,434.80 Part B premium; itemized $19,518 (matches 19,518.00). Change "non-deductible OTC purchases" to "OTC purchases, which the engine does not count".
2. claude-fable-5.1: medical $4,007 versus $4,441.86 from counting OTC instead of Part B. Matches.
3. claude-opus-5: medical limited to $10,000 without Part B, so it chose the standard deduction. Matches; the grid flips to standard when both omissions occur.
4. claude-opus-5.5: medical $4,007, OTC instead of Part B. Matches. Same "non-deductible" rewording.
5. claude-sonnet-5.5: components including the Part B premium. Matches, apart from the $5,367.59 noted above.
6. deepseek-v4-flash-0731: "medical without the $2,434.80 Part B premium". Matches.
7. deepseek-v4-pro: medical $2,007.04 by ignoring Part B, correct $4,441.86. Matches.
8. gemini-3.7-flash: medical $4,441.86. Matches.
9. gemini-3.8-flash: $2,007 medical without Part B. Matches.
10. glm-5.3: same claim. Matches.
11. gpt-5.5: "no Part B premium in medical". Matches.
12. gpt-5.6-sol: $4,441.86 including Part B. Matches.
13. gpt-5.6-terra: same. Matches.
14. gpt-6-astra: same. Matches.
15. gpt-6.1-sol: "$10,000 + $2,434.80 Part B, OTC excluded". Matches.
16. grok-4.7: $4,441.86 including the Part B premium. Matches.
17. kimi-k3: $2,007 without Part B, then the standard deduction, $11,969.80. Matches the grid's no-Part-B, no-state-tax reading.
18. ox-alpha: $4,441.86 including Part B. Matches.

Not relevant: claude-haiku-4.5 (it mentions Medicare payroll tax, not the premium).

Incomplete: claude-sonnet-5 puts its $19,518 down to SALT and the skipped floor only. That total also needs $4,007.06 of medical (OTC counted, Part B left out).

Virginia: rows mentioning OTC, $2,000, Part B or a medical figure

1. claude-sonnet-5.5, gemini-3.6-flash, gemini-3.7-flash, gemini-3.8-flash, gpt-5.5 and grok-4.5 (6 rows) all say: "$17,518 … leaving out the $2,000 of over-the-counter health expenses". Verdict: wrong. The engine also leaves OTC out. 17,518.00 = charitable with no floor + $2,007.06 medical (no Part B) + real estate tax. The $1,902 gap is 2,434.80 (Part B) − 532.86 (floor skipped). Suggested wording: "…$17,518, leaving out the $2,434.80 Medicare Part B premium the engine counts in the medical deduction and skipping the 0.5%-of-AGI charitable floor the engine applies; the Virginia itemized deduction is $19,419.94…"
2. gemini-3.1-pro-preview: "$17,518 itemized total (counting only the $10,000 of other medical expenses)". Verdict: misleading. The model's own explanation gives no itemized figure, so the $17,518 is the annotator's inference and only roughly reproduces its $3,726 (about $3 off). If that total is right, the gap is the Part B premium net of the skipped floor. Reword as in item 1.
3. gpt-6-astra and gpt-6.1-sol: "$16,985.18 by applying a 0.5%-of-AGI floor to charitable gifts and counting only the $10,000 of other medical expenses". Verdict: wrong. The engine applies the same floor and also counts only $10,000 of non-premium medical. The only gap is the $2,434.80 premium, and their answers match the no-Part-B result (3,654.15). Suggested wording: "…$16,985.18, which leaves out the $2,434.80 Medicare Part B premium the engine imputes; with it the Virginia itemized deduction is $19,419.94…"
4. claude-fable-5.1: "$18,985 by applying a 0.5%-of-AGI charitable floor, $435 below". Verdict: wrong. Its own explanation shows charitable 10,914 after the floor, which is correct, and medical 12,000 − 7.5% = 4,007. The $434.80 gap is the $2,434.80 premium minus the $2,000 OTC it counted. Suggested wording: "…$18,985, counting the $2,000 of over-the-counter expenses, which the engine leaves out, instead of the $2,434.80 Part B premium the engine imputes, $435 below…"
5. gemini-3.5-flash: "the medical deduction … is the $12,000 of medical costs above the 7.5%-of-AGI floor". Verdict: wrong. The engine's base is $12,434.80 ($10,000 + $2,434.80 Part B, no OTC) and the deduction is $4,441.86. Its $15,511 is charitable with no floor plus real estate tax, so the shortfall against the reference is $3,909.00. Suggested wording: "…leaves out the medical deduction, which in the engine is $4,441.86: $10,000 of other medical expenses plus the imputed $2,434.80 Part B premium, above 7.5% of federal AGI…"
6. gpt-6-sol: "deducting all $12,000 of medical expenses without the 7.5%-of-AGI floor, which limits the medical deduction to about $4,000". Verdict: wrong on the engine figure ($4,441.86 from $12,434.80). My computed reconstruction also contradicts the diagnosis. Its federal explanation uses AGI of $96,572.80, so VA AGI would be $76,744. Charitable after the floor (10,963.78) + $10,000 medical above 7.5% (2,757.04) + real estate tax = 17,785.11 of itemized deductions, leaving taxable income of 57,228.89, which matches its "about 57229". Suggested wording: "Its about $57,229 of taxable income matches its own $10,000 understatement of AGI carried into Virginia AGI ($76,744), with itemized deductions built from the $10,000 of other medical expenses without the $2,434.80 Part B premium the engine imputes."
7. grok-4.7: "$11,447 of charitable gifts, $4,064 of real estate taxes and the medical deduction make the filer a federal itemizer". Verdict: correct-but-depends-on-unlisted-input. Without state income tax in SALT, the engine still itemizes only because the medical deduction includes the premium.

Related Virginia rows that give itemized figures without describing why: claude-fable-5, claude-opus-5.5, deepseek-v4-pro-0813 and grok-4.6 at $19,518, and deepseek-v4.1-flash at $17,518. The figures are numerically correct. For the $19,518 rows, the +$98.06 is +532.86 (floor skipped) − 434.80 (OTC counted in place of Part B).

Neither us_adjudications.json nor us_wording-amendments.json has any scenario_114 entry. No files were edited. Both outputs are proposed for exclusion (#191 and #193), so this report is coordination input; any rewording would go into the release that adopts those records.

[MaxGhenis]

Shipped in release dashboard-data-20261006 (PR #202, squash-merged as 9ce4ade8, now the latest data release and live on policybench.org). This PR's commits were folded into that release as a merge commit, so its content is on main; closing it rather than merging it a second time. Ruling d974: scenario_114's Virginia income tax before refundable credits is excluded (a modeled Medicare Part B premium in medical expenses).