Add FSA portfolio and ACS S1501 attainment source packages

[DTrim99]

Closes #310

Summary

Seven publisher-backed source packages for the EqRC student debt pilot (PolicyEngine/us-student-debt#1). Microcosm consumes them in PolicyEngine/microcosm#1074 (calibration targets) and PolicyEngine/microcosm#297 (attainment). Chronicle stores the published levels. Aging, reconciliation across FSA tables, the treatment of "Not Reported", and target selection stay in Microcosm.

Federal Student Aid portfolio (packages/ed_fsa/)

All six files were downloaded on 2026-10-12 from https://studentaid.gov/sites/default/files/fsawg/datacenter/library/<file>. Every file has Last-Modified: Thu, 08 Oct 2026, so FSA re-posted them after the issue was written (2026-09-30). The latest period in each file is still the one the issue lists, and the issue's headline numbers match ($1,724.5B, 42.3M recipients, $52.2B Not Reported). studentaid.gov stalls non-browser user agents, so the files were fetched with curl and a browser UA (see Notes).

Each package emits every FY2026 quarter in its file. Earlier quarters are not emitted as facts, but they stay in the parsed source cells. A fact's period is the month in which the quarter ends (period_type: month). period_coverage holds the snapshot date and the FSA label, for example Federal Fiscal Year 2026 Q3, as the MSD quarter-end packages do. Dollars are published in $ billions and scaled with value_scale: 1e9. Counts are published in millions (1e6), or in thousands for the location table (1e3).

Package	File	Periods	Counts	Record sets / facts	Quirks kept as published
ed-fsa-portfolio-summary-fy2026-q3	PortfolioSummary.xls	FY2026 Q1–Q3 (2025-12-31, 2026-03-31, 2026-06-30)	recipients; "Unduplicated Recipients" for the Total	9 / 24	The Total is stored as printed and not recomputed (footnote 1). Recipient means the student, so for parent PLUS loans it is the child.
ed-fsa-portfolio-by-location-fy2026-q2	Portfolio-by-Location.xls	as of Mar 31, 2026 (FY2026 Q2), the only date in the file	borrowers (thousands)	1 / 108	54 rows: 50 states, DC, Puerto Rico, plus "Other" and "Not Reported" ($52.2B; 1,953.3K borrowers). The last two are statistical_scope geographies (ed_fsa:location_other, ed_fsa:location_not_reported) and are not allocated to states. The table has no total row.
ed-fsa-portfolio-by-age-fy2026-q2	Portfolio-by-Age.xls	FY2026 Q1–Q2	borrowers	4 / 24	The bands carry age bounds. "24 and Younger" has no lower bound, because the Definitions sheet describes bands ("Younger than 18", "19 to 24") that the table does not publish. "Not Reported" carries age_reporting_status == not_reported.
ed-fsa-portfolio-by-debt-size-fy2026-q2	Portfolio-by-Debt-Size.xls	FY2026 Q1–Q2	borrowers	4 / 36	Nine bands with federal_student_loan_balance bounds taken from the Definitions sheet (≥ lower, < upper). The table writes "80K-100K" and the Definitions sheet writes "80K to 100K". The guards keep the table's spelling.
ed-fsa-portfolio-by-loan-type-fy2026-q3	PortfoliobyLoanType.xls	FY2026 Q1–Q3	recipients (loan-level)	12 / 42	Stafford (Combined) and its "Unique Recipients" count are in separate record sets (subtotal, federal_student_loan_type_group == stafford), so they don't mix with the loan-level counts.
ed-fsa-dl-portfolio-by-repayment-plan-fy2026-q3	DLPortfoliobyRepaymentPlan.xls	FY2026 Q1–Q3 (through Jun 30, 2026)	recipients (loan-level)	36 / 227	Covers all six data sheets: DL by plan, DL IBR/PAYE by partial financial hardship, ED-held FFEL, ED-held FFEL IBR by PFH, federally managed, and federally managed IBR/PAYE by PFH. The status universe is repayment, deferment and forbearance. There is no RAP column, because the latest quarter ends before RAP launched on Jul 1, 2026. "Other" recipients are published as NA*** (DL) and NA** (FFEL, and federally managed Q1); those cells produce no fact, and a guard on the matching dollars row records the marker. The workbook does not define the asterisks. Federally managed "Other" recipients appear as 0.09M from Q2.

Spot values: Q3 total $1,724,500,000,000; Direct Loans $1,564.8B; SAVE (DL, Q3) $354.3B and 6.52M recipients; California $156.7B and 3,919,500 borrowers; age 35–49 (Q2) $685.1B; Parent PLUS (Q3) $115.0B and 3.5M recipients.

Census ACS S1501 (packages/census/acs_s1501_state_2024/)

• Vintage: 2024 ACS 1-year. As of 2026-10-12 the 2025 1-year tables are not released. data.census.gov returns "There is no table with the ID: ACSST1Y2025.S1501", api.census.gov/data/2025/acs/acs1 returns 404, and the Census Bureau's 6 August 2026 update says their release date "is being determined".
• Source: the data.census.gov table API (/api/access/data/table?id=ACSST1Y2024.S1501&g=010XX00US$0400000). It wraps the api.census.gov group(S1501) array in {"response": {"data": [...]}}. api.census.gov now redirects requests without a key to its missing-key page, so the template's endpoint can't be refetched without a key.
• Parser: the new census_acs_s1501_attainment_json_rows accepts both the wrapped and the bare array. Like the S0101/S2201 parsers, it unpivots each state into long rows with age and educational_attainment columns, so row-backed constraints are checked against the source row.
• Universes, from the Total estimate columns only:
  • 18–24 (S1501_C01_001E–005E): a total plus 4 categories. This universe does not separate associate's degrees; they are inside "Some college or associate's degree".
  • 25 and over (006E–013E): a total plus 7 categories (less than 9th grade; 9th–12th, no diploma; high school; some college, no degree; associate's; bachelor's; graduate or professional).
• Shape: 52 geographies (50 states, DC, PR) × 2 universes = 104 record sets and 676 facts, one record set per state per universe, following the acs_s0101_state_2024 template. The detail rows add up to each total in this release.
• Not included: the cumulative "or higher" rows, the 25–34/35–44/45–64/65+ summary rows (which publish only "HS or higher" and "BA or higher"), race and ethnicity rows, poverty, earnings, the percent and sex columns, and margins of error.

Code changes

• chronicle/sources/rows.py and chronicle/source_package.py: the S1501 parser and its dispatch. Seven SOURCE_PACKAGE_ALIASES entries.
• Constraint variables are source-neutral (federal_student_loan_program, ..._type, ..._repayment_plan, ..._portfolio, ..._status_group, ..._partial_financial_hardship, federal_student_loan_balance, age, educational_attainment), because the consumer contract rejects variables in the ed_fsa. namespace. The groupby dimension ids stay namespaced (ed_fsa.loan_type and similar).
• packages/ed_fsa/build_packages.py and packages/census/acs_s1501_state_2024/build_package.py generate the YAML (precedent: packages/statbel/.../build_package.py). Every guard, header and label string is read from the publisher file, so the next FSA quarter can be regenerated rather than hand-edited.
• tests/test_chronicle_bundle.py: each package is keyed to its file's year (2026 for FSA, 2024 for ACS) and has no artifact_year pin, so the default 2023 bundle skips all seven. Only skipped_source_count (10 → 17) and the skipped list move. No fact, key or warning count changes.

Chronicle Governance

• Approved agent role: ledger-source-ingestor. As in Add BEA NIPA Table 7.8 line 17 (B4923C): employer contributions for private group health insurance #331 and Add IRS SOI Historic Table 2 TY2023 state AGI-band facts #291, the change also touches db/data/** and the alias map / parser dispatch in chronicle/source_package.py, which the role file does not list.
• Deterministic checks, run locally on Windows with Python 3.13 and the locked ruff 0.14.10:
  • Source-package validation: chronicle validate-package is valid for all seven, at --year 2026 (FSA) and --year 2024 (ACS). At --year 2023 each reports only No source artifact for year 2023, which is the bundle's skip condition.
  • Source-cell preservation: chronicle build-suite is valid for all seven, with lineage coverage 1.0 and 0 agent-acceptance errors. The only warning is concept_alignment_validation_skipped, because no Axiom root was passed.
    • The FSA builds parse the whole workbook: 720, 216, 670, 944, 1,008 and 4,876 source cells.
    • The ACS build has 676 source rows and 5,416 cells, and the row-backed age and educational_attainment constraints are evidenced.
    • tests/test_chronicle_ed_fsa_portfolio.py re-reads every selected cell with xlrd and checks fact = cell × scale exactly (Decimal) for all 461 facts.
    • tests/test_chronicle_census_acs_s1501.py checks all 676 facts against the raw JSON by (GEO_ID, variable).
  • Consumer-contract validation: consumer_facts.jsonl was written for every package, and validate_consumer_fact_contract passes.
    • Bundles: build-bundle --year 2026 over the six FSA packages gives 461 facts with 0 errors, 0 warnings and 0 aggregate or semantic duplicate keys.
    • build-bundle --year 2024 over S1501 + S0101 state + S0101 national gives 1,630 facts with 0 errors, 0 warnings and 0 duplicates.
  • Raw-facts boundary: test_chronicle_boundaries.py, test_chronicle_facts_only.py and test_chronicle_governance.py pass locally, except one boundaries test that fails on Windows only (a charmap decode error, also on unmodified files). There is no derived, aged or reconciled value; every fact is one publisher cell times a declared unit scale.
• LLM judge verdicts: ledger-source-fidelity is pending and ledger-boundary is pending. CI does not run them; .github/workflows/ci.yml has no judge step. They need an independent reviewer run, as on Add BEA NIPA Table 7.8 line 17 (B4923C): employer contributions for private group health insurance #331. The author has not self-certified them.

Tests

• New: tests/test_chronicle_ed_fsa_portfolio.py (28 tests) and tests/test_chronicle_census_acs_s1501.py (9 tests). Both pass. They cover pinned counts, cell-by-cell value checks, headline values, constraints, NA markers, Other/Not Reported, no artifact for other years, a guard that rejects a changed expectation, and a selector moved to the previous quarter.
• Also pass: test_chronicle_source_rows.py, test_source_package_alias_drift.py::test_repo_alias_map_matches_packages_on_disk, test_chronicle_consumer_contract.py, test_issue_261_dimension_labels.py and test_issue_266_geography_names.py.
• ruff check chronicle policyengine_chronicle db scripts tests passes.
• I did not run the full default bundle (test_chronicle_bundle.py) locally; CI runs it. The new pins come from _source_unavailable_for_year on each package at 2023.
• Some existing tests fail locally on Windows for environment reasons: checksum mismatches from core.autocrlf rewriting pinned CSVs, charmap decoding, drive letters parsed as URL schemes, symlink privileges, and wrangler. None of them involve these packages. For the boundary, restamp, concept and label suites, I confirmed the same failures on the base commit. The new raw files are binary XLS or a single-line JSON, so autocrlf cannot change them, and the staged blobs hash to the manifest SHA-256s.

Notes for the reviewer

1. R2 upload is still needed. The manifests record the canonical content-addressed ledger-raw keys, the same ones chronicle publish-raw writes. The consumer contract refuses facts without raw_r2_uri. I don't have R2 credentials, so nothing has been uploaded. Before merge, someone with access should run chronicle publish-raw --root db/data/ed_fsa and chronicle publish-raw --root db/data/census/acs_s1501_state_2024, then verify with chronicle verify-raw.
2. Fetching. studentaid.gov stalls python-httpx (and therefore chronicle fetch-artifact and LEDGER_SOURCE_ARTIFACT_FETCH=1). The bytes are committed, so builds don't need to fetch them.
3. Decisions to confirm:
   • (a) Emit only the FY2026 quarters, not the full history back to FY2007 or FY2017. This gives every table a common FY2026 Q2 alongside each table's latest quarter. The generator can widen the window.
   • (b) Use a month period with a snapshot date, rather than quarter.
   • (c) Treat "Other" and "Not Reported" as statistical-scope geographies.
   • (d) Keep Stafford (Combined) in separate record sets.
   • (e) Count only S1501 Total estimates, without margins of error.
   • (f) Use the constraint variable names listed above.

🤖 Generated with Claude Code