This page explains how we check that the bill parser extracts dollar amounts correctly, and what the current numbers are. It is generated from the validation suite, so the figures here match the tests.
Our tool reads each appropriations bill and pulls out the dollar amount for every account. To confirm it reads them correctly, we compare its output against an independent source we did not build: the committee report that Congress publishes alongside each bill. The bill and the report are written separately, by different people, for different purposes — so when the tool's numbers match the report, that is real, independent evidence the tool got it right.
Across all 12 regular appropriations subcommittees, the tool's amounts match the independent source about 94% of the time. We then hand-checked every remaining case, and none is a tool error. Each is a spot where the report and the bill legitimately state a number differently — for example, the report gives one account total while the bill splits it into separate lines, or the amount is open-ended ("such sums as may be necessary") so there is no fixed figure in the bill to match. We also confirmed the tool is not just "tuned" to the bills we happened to test (see Guarding against overfitting below).
The rest of this page is the detail behind that claim. Terms: the parser is the tool that reads the bill; an account is one funding line (e.g. "FBI — Salaries and Expenses"); a match is called a recall.
The system under test is our bill parser (bill_tree / diff_bill) reading dollar
amounts out of the bill XML (the authoritative text from govinfo). We do not question the
bill XML; we check that our parser reads it correctly, by comparing the parser's output to
independent ground truth.
- Legislative Branch — 414 line items across 7 enrolled bills (113-hr-3547, 113-hr-83, 114-hr-2029, 115-hr-1625, 115-hr-244, 115-hr-5895, 116-hr-1865), chambers: house, senate. Source: an independently maintained appropriations spreadsheet. Validated structurally (
TestLegBranchValidation): each curated account names amatch_path, and the parser must produce a node there with the expected amount. - 13 committee-report jurisdictions — committee-recommended amounts parsed from the
Senate Appropriations committee reports (govinfo
CRPT-…), compared to what the parser extracts from each reported bill. The report is written by committee staff for a different purpose than the bill, so it is genuinely independent. (Committee reports, not CRS reports.) Most are the FY2025 reported bills; two use an earlier year (the FY2024 CJS overfitting guard, and Homeland Security, whose FY2025 bill the Senate never reported) — see the table. Most are read from the report's 3-line summary blocks; tabular jurisdictions (Defense) print accounts only in the wide comparative statement, so those are read from that table instead (committee-recommendation column, converted from thousands to dollars).
An account is counted as recalled when the report's amount appears in the parser's
extraction under the correct agency, or as a sum of the account's components (totals the bill
states only as parts). See tests/validation_check.py::validate_jurisdiction. For the comparative-
statement source, rollup totals/subtotals and negative reduction/offset lines (rescissions,
offsetting collections) are excluded — they are not leaf appropriation accounts.
| Jurisdiction | Bill | Accounts | Recalled | Recall |
|---|---|---|---|---|
| Commerce-Justice-Science | 118-s-4795 |
75 | 73 | 97% |
| Agriculture-Rural Development-FDA | 118-s-4690 |
44 | 41 | 93% |
| Transportation-HUD | 118-s-4796 |
67 | 64 | 96% |
| State-Foreign Operations | 118-s-4797 |
68 | 67 | 99% |
| Interior-Environment | 118-s-4802 |
72 | 64 | 89% |
| Financial Services-General Government | 118-s-4928 |
100 | 98 | 98% |
| Labor-HHS-Education | 118-s-4942 |
123 | 108 | 88% |
| Military Construction-VA | 118-s-4677 |
19 | 18 | 95% |
| Defense | 118-s-4921 |
77 | 77 | 100% |
| Energy-Water | 118-s-4927 |
67 | 61 | 91% |
| Commerce-Justice-Science (FY2024) | 118-s-2321 |
75 | 73 | 97% |
| Homeland Security (FY2024) | 118-s-2625 |
35 | 30 | 86% |
| Legislative Branch | 118-s-4678 |
31 | 29 | 94% |
Overall: 803 / 853 accounts recalled (94.1%).
Every unrecalled account below was traced by hand. None is a parser extraction error; each is an inherent difference between how the report and the bill state a figure:
- Indefinite "such sums" accounts — the bill states no fixed dollar figure (e.g. Contract Support Costs, Payments for Tribal Leases, the Commodity Credit Corporation reimbursement). The report prints an estimate; there is nothing in the bill text to match.
- Account totals the bill itemizes — the report gives an account total, the bill states the components (e.g. HUD Tenant-Based Rental Assistance = renewals + a separate line). The single total never appears as one number in the bill, so there is nothing for the parser to match.
- Report-side typos — the report's own number is wrong (e.g. NASA Safety, Security and Mission Services: the report narrative prints the budget estimate, while the bill and the report's own comparative statement agree on the correct figure, which the parser extracts).
- Shared account names — sub-allocations under a name shared across bureaus ("Salaries and Expenses"), where the report's granularity differs from the bill's.
The full current remainder:
Commerce-Justice-Science (2)
- DEPARTMENT OF JUSTICE / PUBLIC SAFETY OFFICERS BENEFITS — $280,800,000
- SCIENCE / SAFETY, SECURITY, AND MISSION SERVICES — $3,044,440,000 Agriculture-Rural Development-FDA (3)
- FARM PRODUCTION AND CONSERVATION PROGRAMS / DAIRY INDEMNITY PROGRAM — $500,000
- FARM PRODUCTION AND CONSERVATION PROGRAMS / REIMBURSEMENT FOR NET REALIZED LOSSES — $12,650,463,000
- RELATED AGENCY AND FOOD AND DRUG ADMINISTRATION / FARM CREDIT ADMINISTRATION — $100,430,000 Transportation-HUD (3)
- DEPARTMENT OF TRANSPORTATION / THE NATIONAL RAILROAD PASSENGER CORPORATION — $2,638,072,000
- DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT / COMMUNITY DEVELOPMENT FUND — $4,614,317,000
- INDEPENDENT AGENCIES / SALARIES AND EXPENSES — $32,100,000 State-Foreign Operations (1)
- DEPARTMENT OF STATE AND RELATED AGENCY / PAYMENT TO THE FOREIGN SERVICE RETIREMENT AND DISABILITY FUND — $60,000,000 Interior-Environment (8)
- DEPARTMENT OF THE INTERIOR / SERVICE CHARGES, DEPOSITS, AND FORFEITURES — $30,000,000
- DEPARTMENT OF THE INTERIOR / CONTRACT SUPPORT COSTS — $356,000,000
- DEPARTMENT OF THE INTERIOR / PAYMENTS FOR TRIBAL LEASES — $69,000,000
- DEPARTMENT OF THE INTERIOR / ASSISTANCE TO TERRITORIES — $58,775,000
- RELATED AGENCIES / ACQUISITION OF LANDS TO COMPLETE LAND EXCHANGES — $150,000
- RELATED AGENCIES / RANGE BETTERMENT FUND — $1,719,000
- RELATED AGENCIES / CONTRACT SUPPORT COSTS — $2,036,000,000
- RELATED AGENCIES / PAYMENTS FOR TRIBAL LEASES — $400,000,000 Financial Services-General Government (2)
- EXECUTIVE OFFICE OF THE PRESIDENT AND FUNDS APPROPRIATED TO THE / SALARIES AND EXPENSES — $77,681,000
- EXECUTIVE OFFICE OF THE PRESIDENT AND FUNDS APPROPRIATED TO THE / SALARIES AND EXPENSES — $133,290,000 Labor-HHS-Education (15)
- DEPARTMENT OF LABOR / SPECIAL BENEFITS FOR DISABLED COAL MINERS — $31,367,000
- DEPARTMENT OF LABOR / BLACK LUNG DISABILITY TRUST FUND — $485,706,000
- DEPARTMENT OF LABOR / OFFICE OF THE INSPECTOR GENERAL — $97,028,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / HRSA-WIDE ACTIVITIES AND PROGRAM SUPPORT — $1,092,655,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / IMMUNIZATION AND RESPIRATORY DISEASES — $934,291,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / EMERGING AND ZOONOTIC INFECTIOUS DISEASES — $805,272,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / CHRONIC DISEASE PREVENTION AND HEALTH PROMOTION — $1,463,914,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / PUBLIC HEALTH AND SCIENTIFIC SERVICES — $774,497,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / ENVIRONMENTAL HEALTH — $244,850,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / CDC-WIDE ACTIVITIES — $681,570,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / MENTAL HEALTH — $2,902,546,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / REFUGEE AND ENTRANT ASSISTANCE — $6,428,214,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / AGING AND DISABILITY SERVICES PROGRAMS — $2,543,817,000
- DEPARTMENT OF HEALTH AND HUMAN SERVICES / RETIREMENT PAY AND MEDICAL BENEFITS FOR COMMISSIONED OFFICERS — $894,795,000
- RELATED AGENCIES / OFFICE OF INSPECTOR GENERAL — $114,665,000 Military Construction-VA (1)
- DEPARTMENT OF VETERANS AFFAIRS / MEDICAL CARE COLLECTION FUND — $4,632,000,000 Energy-Water (6)
- DEPARTMENT OF DEFENSE--CIVIL / Water Infrastructure Finance and Innovation Program Account — $10,000,000
- DEPARTMENT OF THE INTERIOR / Central Valley Project Restoration Fund — $55,656,000
- DEPARTMENT OF ENERGY / Nuclear Energy — $1,525,000,000
- DEPARTMENT OF ENERGY / Defense function — $150,000,000
- DEPARTMENT OF ENERGY / Colorado River Basin Fund (sec 307) — $2,000,000
- INDEPENDENT AGENCIES / Salaries and expenses — $942,558,000 Commerce-Justice-Science (FY2024) (2)
- DEPARTMENT OF JUSTICE / SALARIES AND EXPENSES — $3,210,719,000
- DEPARTMENT OF JUSTICE / PUBLIC SAFETY OFFICERS BENEFITS — $208,800,000 Homeland Security (FY2024) (5)
- DEPARTMENTAL MANAGEMENT, OPERATIONS, INTELLIGENCE, AND OVERSIGHT / PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS — $73,365,000
- DEPARTMENTAL MANAGEMENT, OPERATIONS, INTELLIGENCE, AND OVERSIGHT / OPERATIONS AND SUPPORT — $341,497,000
- DEPARTMENTAL MANAGEMENT, OPERATIONS, INTELLIGENCE, AND OVERSIGHT / OPERATIONS AND SUPPORT — $237,000,000
- SECURITY, ENFORCEMENT, AND INVESTIGATIONS / HEALTH CARE FUND CONTRIBUTION — $277,000,000
- RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES / OPERATIONS AND SUPPORT — $455,661,000 Legislative Branch (2)
- LEGISLATIVE BRANCH APPROPRIATIONS / CAPITOL POLICE — $832,556,000
- LEGISLATIVE BRANCH APPROPRIATIONS / SALARIES AND EXPENSES — $60,238,000
The concern behind this work (GitHub #8) is that the parser's heuristics might be tuned to the specific bills we happen to have. Three things push against that:
- A different fiscal year. The table above includes Commerce-Justice-Science (FY2024) — a bill and report not otherwise in our corpus. It recalls at the same rate as FY2025 CJS, with the same two structural misses, so the parser is not keyed to one year's formatting.
- Both chambers. The Legislative Branch source spans House and Senate enrolled bills across FY2014–FY2020. (The committee-report jurisdictions are Senate-only because House appropriations reports render their account tables as embedded images, so there is no text for the reader to extract — a limit of the report source, not the bill parser, which parses House bill XML fine.)
- Many independent jurisdictions. Every committee-report jurisdiction in the table above — spanning very different account structures (flat tabular Defense, deeply-nested Energy-Water, narrative summary blocks) — recalls in the same band, which is the opposite of what overfitting to one structure would produce.
- This validates amount extraction, not the version-to-version diff/comparison, which is covered by separate hand-built fixtures.
- The committee-report check is internal-consistency between two documents of the same legislative process. It is strong for catching the parser misreading unfamiliar structure (which is the overfitting risk), but it is not a third-party audit.
- Coverage now spans all 12 regular Senate appropriations subcommittees: 11 via committee reports here (FY2025, except Homeland Security — the Senate never reported an FY2025 DHS bill, so its FY2024 reported bill is used instead) plus Legislative Branch via the spreadsheet source. Other years and the House chamber are not exhaustively covered.
uv run python scripts/build_validation.py --fetch # fetch sources, rebuild fixtures
uv run pytest tests/test_validate_extraction.py -m slow # run the validation
uv run python scripts/generate_validation_report.py # refresh this page