Publish credible bounds as numbers, and count the rest of the prose - #14
Merged
Conversation
data/national.json carries every national figure the site publishes, including the 846 institutions it names under two statutes, and nothing verified it. Every one of its inputs was gitignored, so a reader who cloned this repository could regenerate data/report.json from data/sample.json and could not regenerate the national artifact from anything at all. The same was true of the three-year IPEDS history the 2% systemic threshold is argued from: two of its three collection years had no committed inputs. Commit the six public archives (3.9 MiB, no key, no quota, no terms) and replay the whole chain against the artifacts the repository ships: crosscheck + national -> data/national.json byte for byte crosscheck + snapshot -> data/snapshots/ipeds/*.json byte for byte, all three years Plus the invariants that catch an artifact disagreeing with itself rather than with this code: applicable == reported + missing + implausible, applicable + suppressed + not_applicable == institutions for every field, share_reported is the division it claims to be or is null, and the named-institution lists match the counts beside them exactly. The gap this closes is measurable. Making _is_an_institution also exclude INSTCAT of -1, which silently drops fourteen real colleges out of every denominator in the country, passes all 297 existing tests. It fails four of these. Runs offline: load_institutions reads the committed archive when the cache path exists, so no push reaches NCES. `make replay` and `make ipeds-snapshots` show the diff when they disagree. Closes #4 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects on the methodology page, plus the class of bug the second one belongs to. `5e+05` is what a prospective student was shown as the ceiling on median earnings. Past four significant digits `,.4g` switches to exponent form and the thousands separator stops applying, so four of the six Scorecard ceilings published as `5e+05`, `4e+05`, `1.5e+05` and `2.5e+05`, on the one page whose entire job is to be argued with by people who are not statisticians, directly under a rationale that says "$1,000" in plain words. Fixed-point with the trailing zeros trimmed, not `,.0f`: every bound today is integral and `,.0f` would be right for all of them, and would also silently round a future credible range of 0.5 to "0", which is a rule misstated on the page that states the rules. The formatter moves out of the closure so it can be tested against values no field has met yet. The "three institutions" count and the peers docstring were already corrected in e5e02b4 and 05da61f. What was not fixed is the reason they were wrong: about twenty more counts were typed into prose by hand with nothing checking them, and until the IPEDS archives were committed there was no data in the repository to check the largest group of them against. So the page now prints the rules rather than restating them. Letter bands come from grading.BANDS, the systemic threshold from drift.SYSTEMIC_THRESHOLD, both alongside MIN_PEERS which already worked this way. The published sentences are unchanged; they are now generated by the constants that make the decisions they describe. And tests/test_published_figures.py recounts everything that cannot be derived, from the archives, the snapshots and the committed report: 4,469 / 6,163 / 1,998 / 2,008 / 1,334 / 674 / 2,377 / 101 / 76 / 284 / 14 / 114 / 11 / 29, the eight figures the 2% threshold is argued from, and the peer group that settled the West Valley finding. Each is checked against its source rather than against another copy of itself, because two files agreeing is how both drift onto a wrong number together. Every one of them is correct today. Eleven single-digit mutations of them pass all 297 existing tests and fail these. Closes #3 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ChelseaKR
changed the base branch from
data/verify-national-artifact
to
master
August 15, 2026 00:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3. Stacked on #12 — based on
data/verify-national-artifact, because the prose tests here need the IPEDS archives that PR commits. Review #12 first; the diff shown here is only this PR's own changes.1.
5e+05Still live on the published page as of today:
Four of the six Scorecard ceilings, in a notation a prospective student will not read, on the one page whose entire job is to be argued with by non-specialists — directly under a rationale that says "$1,000" and "aggregate federal borrowing limits" in plain words. Now:
Fixed-point with trailing zeros trimmed rather than the
,.0fthe issue suggests. Every bound today is integral and,.0fwould be correct for all of them; it would also silently round a future credible range of0.5to"0", which is a rule misstated on the page that states the rules. The defect being repaired is a formatter that was right for the values it was written against and wrong for the values it met, so the replacement is tested against values it has not met either. The formatter moves out of the closure to make that possible.2 and 3. The counts
Already fixed, before this PR.
e5e02b4corrected "three institutions" to two, and05da61fcorrected thepeers.pydocstring to 79 / 78 / 77 / $1,108–$1,571. Both verified against the data here; both are right.What was not fixed is the reason they were wrong. About twenty more counts were typed into prose by hand with nothing checking them, and until the archives were committed there was no data in the repo to check the IPEDS ones against.
Numbers that could be derived, now are. Letter bands print from
grading.BANDS, the systemic threshold fromdrift.SYSTEMIC_THRESHOLD— joiningMIN_PEERS, which already worked this way. The published sentences are byte-identical; they are now generated by the constants that make the decisions they describe, so moving a band moves the page.Numbers that cannot be derived are now counted by a test.
tests/test_published_figures.py, following theTestTheFiguresTheProseStatespattern already intest_disclosure.py:HD2023.zip+IC2023.zipsnapshots/ipeds/*.jsonreport.json+sample.jsonEach is checked against its source, never against another copy of itself. Asserting that two files agree is how both drift onto a wrong number together.
Three of these claims live in
#comments rather than docstrings, including the whole justification forSYSTEMIC_THRESHOLD. They are checked too: a comment is not less published when the page paraphrases it verbatim.Every one of these numbers is correct today
That is not a reason to skip this. They were correct on the day the completion-rate figure was typed too, and what changed was the rule underneath, silently. Verified by mutation — eleven single-digit edits, each run against both suites:
Not done, deliberately
Threading the report into
methodology_page()sofield.rationalerecomputes its own counts at render time.rationaleis a frozen dataclass attribute read in three places (the methodology page, every institution page, anddataset.schema.json), and making it dynamic would be a large refactor of the thing most needing to stay stable. The repo had already chosen the test-based answer for this exact figure, with a docstring explaining why; this extends that rather than replacing it.make verifygreen: 344 passed, coverage 98.76%.Not merged, not deployed.
🤖 Generated with Claude Code