[MaxGhenis]

WIP. This is part 1 of milestone D4 of the income-and-resources design (#559: the D4 row of the milestone table). It adds pinned captures and parsers for aggregate tables. Part 2 adds the SSA and BLS sources that refuse plain requests, through Internet Archive captures. The tier-count update waits for the coordinator's slot. Eight independent review rounds asked for fixes, which are in. The eighth round's fix is dfb30420, and 7adafa66 merges master at 0f69bbb2; the round-9 delta review is next.

What it adds

• Captures in data/external/snapshots/income_resources_d4_v1/: 54 pinned sources. Each per-source manifest records time, size, SHA-256 and URL. A folder-level .gitattributes (* -text) keeps the 37 captures with CRLF line endings byte-exact.
• Registries: 874 entries, comprising 861 numeric tables and 13 references:
  • SSA Annual Statistical Supplement Tables 4.B1, 4.B2, 4.B10 and 4.B11;
  • 777 POMS SI 01415 annual state-supplement payment charts;
  • POMS RS 01505 and SL Section 218 coverage references;
  • 18 Federal Reserve Z.1 tables (June 2026 codes);
  • 52 DOL Private Pension Plan Bulletin tables (2023 edition);
  • ten IRS SOI IRA tables (tax year 2023).
• Readers (src/populace_dynamics/income_resources/aggregate_*.py):
  • every public reader checks the capture's size and SHA-256 before reading a cell;
  • cells are typed, with table, row, column, unit and year.

Invariants

• Hash and size match before any cell is read.
• Missing, suppressed and unavailable cells stay None, never zero.
• Valid separators and footnotes keep magnitude and sign. Malformed numbers refuse without echoing their content.
• HTML spans keep a rectangle, its hierarchy and its footnotes.
• Duplicate cell identities refuse.
• POMS cells keep state, claimant, living-arrangement code and category.
• A State's code definitions are read from every definition table in its section and merged. A code that two tables define differently refuses. A jointly published code ("C/H", "A or B") names each single code, and a full-width note row defines no code.
• Every block that ends a paragraph, apart from a table or rule, names its State, claimant or date from its own text, read whole and line by line. A line ends at a <br> and where a nested block starts or ends, and drops Unicode format (Cf) characters. Each block's whole text and each line are read three ways: joined, as HTML renders inline text; bare, without footnote markers; and spaced, with a space at every element edge. A State heading and a claimant come from the joined or bare reading, so Utah<sup>1</sup> heads Utah; the date sentence comes from the joined reading; and every State net reads all three, so no inline edge hides a State. So neither a paragraph left unclosed nor a block inside a heading carries one State's definitions into the next State's chart, and a list item, section or blockquote reads the same after a closed heading as inside one left open.
• Every line passes the State guards, including the lines of a block whose whole text is a State date line, which do not head a section. A line that dates another State refuses, and so does an effective-date sentence or a claimant line that names any State but its section's. A State date line is recognized by the names, D.C. aliases and postal suffixes a section heading takes. A block whose lines head two States refuses, as does a line shaped as another State's heading inside a whole-text date line, and any line shaped as a State heading whose postal code is not that State's. Every line of an h1–h6 heading may name only its section's State; paragraph prose may name any State. A block names at most one claimant type, and every chart needs its own claimant line. "Washington,DC" names the District and "Hawaiʻi" names Hawaii; postal codes, in parentheses or upper case standing as a word, and 14 abbreviations name States to the nets but never head a section. Before the first State heading no section is open, and neither effective-date nor claimant lines are guarded.
• The shared HTML tree builds paragraphs, headings, lists, buttons, cells, rows, table sections, captions and colgroups as the WHATWG parsing rules do, end-tag scopes included, so selectors do not depend on omitted or stray end tags. It does not model the adoption agency, foster parenting, a form start tag inside an open form or a self-closing slash on a non-void element; no pinned capture opens a form inside another. The tree is the same under Python 3.13 and 3.14: raw text is pinned to HTML's six raw-text elements (script, style, xmp, iframe, noembed and noframes; noscript stays markup, as with scripting off), and a plaintext start tag or an abruptly closed empty comment refuses. A later html start tag only adds the attributes the html element lacks, a noscript end tag with a special element open inside it is ignored, and a table part's start tag outside a table builds nothing, as in html5lib.
• No reader function calls itself; the tree builder, the document walk, the line and cell readers and the chart skeleton keep their own stacks. A probe parses and iterates a page nested 300,000 deep under a recursion limit of 200, and a test reads a chart cell nested 50,000 deep in its own process.
• Every chart code is defined, or the chart is recorded in poms_open_questions.json. No definition is inferred.
• Payment-validity dates stay distinct from section-revision dates. Conflicting dates stay named options.
• Narrative state coverage text stays cited text. It never becomes an observed coverage share.
• Totals reconcile within rounding, or keep a named published discrepancy without repair.
• The parsers contain no print or logging calls.

Open items

• POMS:
  • Fifteen Iowa (2006–2011) and Pennsylvania (2008–2010 couple) charts publish codes that no definition table in their section defines. They are recorded, not filled in.
  • The District of Columbia's 1999 tables carry two conflicting dates, so the year stays unknown.
  • Two Hawaii couple tables (2022, 2023) each have one malformed amount, which stays missing with named options.
  • Thirty selectors carry 33 published component-versus-total discrepancies. These are flagged, not corrected, and SSI2 adjudicates them before using them as conformance truth.
• HTML limits, documented in _HTMLTree and _cell_token:
  • The tree does not model HTML's adoption agency rules for misnested inline end tags, foster parenting of content misplaced in a table, or a self-closing slash on a non-void element.
  • A <br> or nested block in one cell joins its lines with a space, so two amounts stacked that way would read as one space-grouped number. No pinned cell joins two amounts by a <br> within one paragraph. POMS charts stack amounts in paragraphs, which the reader splits into subrows.
• Census Annual Survey of Public Pensions: the data portal refused a plain request (SOURCE_BLOCKED), so no coverage share is inferred.
• Part 2:
  • SSA Supplement Tables 7.A–7.E and 3.C6;
  • the SSI Annual Statistical Report 2025;
  • SSI Recipients by State and County 2025;
  • BLS NCS benefits.

Review and fixes

The first review found that the POMS reader kept only the last definition table, so California 2004 lost codes A–F. The fix reads every definition table in the State's section. It also surfaced three more reader problems, now fixed:

• The 2009 and 2010 headings "New Jersey (NJ)" were not recognized, so four New Jersey charts were labelled Nevada. Their registry entries are renamed new_jersey.
• Iowa 1999–2005 defines codes jointly ("C/H", "D/I"); each joint row now defines both codes.
• A full-width note row in Massachusetts 1999 was read as a code; it is now skipped.
• A State named only in an "X Effective" date line would have kept the previous State's definitions. That path now refuses; no pinned capture uses it.

New checks:

• definition codes match an independent raw-HTML locator on all 777 charts;
• registry selectors equal the structural reader;
• each chart's State matches the State its date sentence names (775 charts name one);
• a California 2004 regression;
• Hypothesis properties for definitions split across tables and for joint codes.

Each of the four bugs, reintroduced, fails at least two tests. The five new source files are listed as birth-evidence seal exclusions, with a reachability check.

The second review found that a definition table placed between two charts of one State section would have reached only the later chart. The source does not say which earlier chart such a table serves, so the reader now refuses it. No pinned capture has that order, and all 777 charts describe as before. Added:

• an invented regression of that ordering;
• a Hypothesis property over any order of definition tables and charts within a section;
• a stronger raw-HTML differential, which now locates State sections and definition tables itself and checks them against both the registry and the live reader.

The third review found that a State heading in a paragraph left unclosed, such as <p>Nevada followed by the chart's heading and table, was not recognized. The shared HTML tree never closes a paragraph implicitly, so the paragraph's text ran on into the chart, and the previous State's selector and definitions carried into the next State's chart. The reader now reads each paragraph or heading from its own text, up to the first nested element whose start tag closes a paragraph in HTML (the set was checked against html5lib 1.1). No pinned capture nests a block in a paragraph or heading (none of 68,336), and all 777 registered selectors, definitions and parsed cells are unchanged, by hash. Added:

• the reviewer's minimized regression, with and without </p>;
• a Hypothesis property that selectors do not depend on omitted paragraph ends;
• a Hypothesis property that a block with no nested block reads its full text.

With the old reader, the regression and the omitted-ends property fail.

The fourth review found that this fix cut a heading's text at blocks that HTML keeps inside a heading, so <h3><div>Nevada</div></h3> again kept California's definitions. It also found that an omitted </p> in a stacked payment cell merged its amounts, that a page of thousands of unclosed paragraphs exhausted the stack, and that the closer set lacked dialog and search. The fix (401b3bc7):

• the shared HTML tree follows the WHATWG "in body" rules for optional end tags: it ends a paragraph in button scope at every start tag that closes one, ends a heading when another starts while it is the current node, and closes an open cell or row at a new one;
• the POMS reader reads a heading's text through nested blocks, reads a div as a block that can carry a State heading, refuses a claimant line that names a State other than its section's, and walks the document without recursion.

Against html5lib 1.1, the new tree agreed on 20,000 random pages of paragraphs, headings and block containers; the old tree disagreed on 6,639.

The fifth review found one medium and three low problems:

• Medium: reading a heading through nested blocks up to the next table meant a heading that holds a block after its words, such as <h3>Nevada<p>Invented note</p>, no longer matched the State pattern, so the previous State's selector carried on. Claimant headings failed the same way. The tree also ended a heading only at its own end tag, where HTML ends it at any h1–h6 end tag.
• Low: thead, tbody, tfoot, caption and colgroup start tags did not end an open cell or row, so a header cell took in the next section's text. This predates the round-4 fix.
• Low: heading text was still collected recursively.
• Low: the claimant-State guard refused safe lines: "Washington, D.C." in the District's section, and a claimant line before any State heading.

The fix (8613b313):

• the tree ends the open heading in scope at any h1–h6 end tag, clears the stack back to the table at a caption, colgroup or table-section start tag, and ends an open caption at a row or cell start tag. col is left out: all 4,045 col tags in the 28 pinned captures that have one sit in a colgroup;
• the POMS reader checks a heading's whole text, then each line split at its nested blocks, for the State, the claimant and the effective date. A whole-text "State Effective …" line stays a date line, and a heading whose lines name two States refuses;
• "Washington, D.C." reads as the District of Columbia, and the claimant-State guard waits for the first State heading;
• heading text and cell text are read without recursion; a cell nesting 1,500 spans had raised RecursionError too.

Nothing changes on the pinned captures: the tree comparison builds 44 identical trees, and all 874 registered entries hash identically. (Round 4's commit called these "43 HTML captures". The registry holds 41 HTML captures, and the comparison's byte filter also admits two xlsx workbooks and a zip archive.) Against html5lib 1.1, on 5,000 invented pages per mode, the new tree has no mismatch in headings (401b3bc7: 506) or in tables with optional cell, row, section and caption ends (401b3bc7: 1,230). Each of 27 mutants of the tree and the POMS reader fails at least one test, as does each of five mutants of the cell reader.

Added, on invented HTML only:

• the reviewer's State and claimant heading cases: blocks after the heading's words, open headings and mismatched end tags;
• an effective line after heading words, a table inside an open heading, and a heading whose lines name two States;
• deep headings, paragraphs and cells;
• a claimant line before any State heading, and Washington, D.C.;
• table sections, captions and colgroups ending open cells and rows, and heading end tags ending any heading in scope but stopping at scope;
• a property that wrapping or ending heading text keeps the selectors;
• a differential property of the cell reader against the recursive one.

The sixth review (delta, at 697ea340) found one medium and three low problems:

• Medium: in a heading, a line from a nested section, list item or blockquote set the chart's date and claimant without the State guards that the same line passes in a paragraph, and the result changed with where </h3> sat.
• Low: _payment_rows deep-copied the chart, and ElementTree's copy recurses, so a cell nesting 50,000 spans crashed the interpreter. The recursion invariant above was therefore false.
• Low: a State heading with a <br>, a zero-width character or another State's postal code was misread, and so was a "Washington, D.C." heading.
• Low: mutants of the new tree rules survived the tests.

The fix (4aae2820):

• every line of a block passes both State guards, and every block that ends a paragraph, apart from a table or rule, is read from its own text. A heading whose first line names a State other than its section's refuses;
• the chart is read through a skeleton built without recursion, whose cells hold keys to the original cells. A table nested in a cell stays that cell's text, as before;
• lines end at a <br> and drop zero-width characters. A postal code after a State's name must be its own, and "Washington, D.C." and "Washington DC" name the District in headings;
• the tree builds lists, buttons, nested tables, captions, colgroups, cols and implied table sections and rows as HTML does, and closes end tags by HTML's scope rules.

Nothing changes on the pinned captures: every registered reader's output, cell values included, hashes as at 697ea340. On the 40 pinned HTML captures, the tree's tables, block texts and nesting now equal html5lib 1.1's on all 40 (697ea340: 40, 40 and 32). Comments and noscript are removed from both trees for that comparison. On 5,000 generated pages per mode, the tree has no heading or table mismatch with html5lib (697ea340: 1,561 and 3,969). Each of 46 mutants, one per new or changed rule, fails at least one test. 51 of the 157 tests in test_aggregate_poms_invented.py fail against 697ea340's modules.

Added, on invented HTML only:

• the reviewer's nested-line cases, for a section, list item and blockquote, with and without </h3>, and a property that selectors do not depend on heading ends or block kinds;
• the claimant guard on every claimant line, and a <br>-separated line that dates another State;
• headings with <br>, </br>, </p>, zero-width characters, postal codes and Washington, D.C.;
• headings and paragraphs that name another State, refused or admitted by the first-line rule;
• 12 table pages and 13 block pages whose expected trees are html5lib 1.1's;
• a chart cell nested 50,000 deep, read in its own process, and a table nested in a chart cell.

The seventh review (delta, GPT-6.1 Sol, at 4aae2820) found three medium and three low problems and one note:

• Medium: a block whose whole text is a State date line skipped its lines, so a later line could date or name another State.
• Medium: a claimant line that named another State passed whenever it also named the section's own.
• Medium: an effective-date sentence other than "State Effective ...", or one with a D.C. alias or postal suffix, carried another State's date into the section.
• Low: a space at each inline element's edge split words such as "Utah".
• Low: a later heading line shaped as a State heading, with another State's postal code, kept the section's selector.
• Low: three tree mutants survived the tests: list items around an address, dd and dt ending each other, and </html> after an html start tag.
• Note: the tree's WHATWG wording covered nested forms, which it does not model.

The fix (89b40e46):

• every line of a block passes the State guards, and a whole-text State date line's lines still head no section;
• claimant lines and effective-date sentences may name no State but the section's, and State date lines are recognized by the heading reader's names, aliases and postal suffixes;
• inline elements join without a separator within a line;
• any line shaped as a State heading with another State's postal code refuses;
• the three tree cases are pinned to html5lib 1.1's blocks, and the tree's docstring lists nested forms among what it does not model.

The pinned 2011 California section's date sentence ends charts<span class="poms-bold">:</span>, which the old line reader read as "charts :". Its two charts' parameter_effective_date_text in poms_registry.json now read "charts:". Every registered reader's output, cell values included, hashes as at 4aae2820; only that page's selector digest changes. The html5lib comparisons are unchanged: 40 of 40 pinned captures, and no mismatch on 5,000 generated pages per mode. Against 4aae2820's reader, 18 of the 187 invented POMS tests fail, the same 18 in each of three Hypothesis seeds; the new tree cases and the later-claimant-line case pass there, since 4aae2820 already built those trees and refused that line. This round's 14 mutants (G1–G11 for the guards, O1–O3 the review's tree mutants) and round 7's 41, rerun on the final code, all fail tests. P11 (the claimant guard stops at its first claimant line) passed at first, because a whole text naming another State now refuses on its own; a block too long to read as one claimant line, with its own State's claimant line first, now catches it.

Added, on invented HTML only: the review's four repro fragments and their variants, refused or admitted, including long blocks whose whole text is no effective-date or claimant line, one with the section's own claimant line before another State's; controls that read the section's own State, date and claimant; an effective-date line before any State heading; and five tree pages whose expected blocks are html5lib 1.1's.

The eighth review (delta, Opus 5.5, at 93824c2c) found two medium and four low problems and three notes, and CI failed on Python 3.14.7:

• CI: Python 3.14's HTMLParser reads xmp, iframe, noembed and noframes as raw text and turns the rest of the page into text after a plaintext start tag, so an invented paragraph-closer test failed there.
• Medium: joining inline text without a separator hid a State name that touches a word character across an inline tag, such as a footnote sup, from every State net; 4aae2820 had refused those pages.
• Medium: in a block whose whole text is a State date line, a line shaped as another State's heading kept the section's selector.
• Low: three mutants (R1, R3, R5) survived the tests.
• Low: claimant selection was silent and inconsistent: a block whose claimant lines named two types read one of them, and the claimant carried over after a chart.
• Low: two of round 7's own boundary variants still read wrong without refusing, because heading lines after the first were not netted for another State.
• Low: "Washington,DC" without a space did not name the District.
• Notes: Unicode format characters, "Hawaiʻi", a postal code alone in a date sentence and "N.J."; a mid-body html start tag; and the round-8 count of round 7's mutants.

The fix (dfb30420):

• pins the raw-text set to HTML's six elements, refuses plaintext and abruptly closed empty comments, and closes the parser at the end of the page, so the tree is the same on 3.13 and 3.14;
• reads each block and line joined, bare and spaced, and runs every State net over all three;
• refuses a line shaped as another State's heading inside a whole-text date line;
• refuses a block naming two claimant types, and resets the claimant after each chart;
• nets every line of an h1–h6 heading, reversing an earlier deliberate admission of a second heading line that names another State;
• matches "Washington,DC" and "Washington,D.C.", strips Cf characters, reads "Hawaiʻi", and reads postal codes and 14 abbreviations as naming States without heading a section;
• models a mid-body html start tag as html5lib does, and, found by the extended differential, a noscript end tag with a special element open inside it and a table part outside a table.

Every registered reader's output, cell values included, hashes as at 93824c2c on both Python versions, and all 777 pinned charts' selectors equal 93824c2c's. On 3.13, 92 of the 300 invented POMS tests fail against 93824c2c's reader, the same 92 in each of three Hypothesis seeds. mutants_r9.py's 60 mutants fail tests but one, H1, which nets only a heading's whole text and first line; the whole text holds every line's text, so it differs only for an alias split across lines. Of earlier rounds' mutants, all 7 of mutants_r8.py's and all 31 of mutants_r7.py's that still apply are caught; the rest are ported into mutants_r9.py or match a caught mutant. (Round 8's "round 7's 41" meant the 41 of mutants_r7.py's 46 that applied then.)

Open, found by the round-9 build: a prose line that holds a claimant word, such as "A recipient or couple qualifies if ...", matches the claimant pattern as individual and sets the claimant for the next chart. The pinned captures hold 70 line forms of this kind. No pinned selector moved this round; whether any pinned chart takes its claimant from such a line is the round-9 review's to check.

• Merge (7adafa66): origin/master at 0f69bbb2 (OP1, OP1: open-population harness around the sealed loop #566, and its report fix, Fix review r10's report findings for OP1 #589). It has no conflicts; git auto-merged the birth-evidence check and its script, which both branches changed.

Tests

D4's six modules (test_aggregate_coverage, _numbers, _pensions, _poms, _poms_invented and _ssa): 575 passed on Python 3.13.9 and 575 on 3.14.7. The birth-evidence seal tests' reachability test needs modules this sparse clone leaves out, so CI runs it, as it runs tests/track_u2/test_u2_isolation.py. black -l 79 and ruff are clean.

Tier	Added
unit	319
artifact	256

• Unit tests use invented inputs.
• Artifact tests read the pinned public captures.
• Microdata: none is read, and no Trustees Report cost table is captured.
• Tier counts: tests/tier_counts.json is unchanged and waits for the coordinator's slot.

axiom: n/a: pinned public aggregate tables and their readers for microsim validation; no PolicyEngine country-model or RuleSpec encoding changed

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
social-security-model	Ready	Preview	Oct 26, 2026 5:44am UTC