diff --git a/AGENTS.md b/AGENTS.md index 0886b2cf..e4b3ba7e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,11 +5,23 @@ representation, and preserve provenance. Selection and measurement contracts liv in consumers such as Microcosm. Every fact value must trace to a publisher. The boundary is who asserted the -value, not level versus projection: a publisher's own projection (CBO baseline, -BFP outlook, SSA trustees, TPC/JCT score) is a fact typed +value, not level versus projection: a government's own projection (CBO +baseline, BFP outlook, SSA trustees) is a fact typed `assertion: source_projection`. PolicyEngine-computed values — aged, uprated, forecast, or reconciled levels — are never Chronicle facts. +Chronicle holds government-produced records only. Before adding a source +folder or package, confirm the publisher is part of a government and add or +update its entry in `docs/source-publishers.yaml`; `tests/test_source_publishers.py` +fails on an unclassified folder or a new non-government one. Do not add +records from trade associations, charities, think tanks, universities, +firms, or aggregators that republish government data: take the series from +the government publisher instead. Comparisons against outside scores belong +in PolicyEngine/policyengine-scorecard, not here. Edge cases (central banks, +regulators, public corporations, intergovernmental bodies, government records +on a third-party host) follow the 2026-10-21 amendment in +`docs/adr-chronicle-facts-only.md`. + Do not put Microcosm work in Chronicle: - no cross-source reconciliation diff --git a/README.md b/README.md index 8b5007f6..9abaead5 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,9 @@ This repository provides: - **Sources**: Source file references, retrieval metadata, manifests, checksums, and provenance. - **Facts**: Source-backed claims represented with typed values, units, - geography, period, source table, and lineage. Publisher projections (CBO - baselines, BFP outlooks, SSA trustees tables, TPC/JCT scores) are facts - typed `assertion: source_projection`; measured outcomes are the default + geography, period, source table, and lineage. Government projections (CBO + baselines, BFP outlooks, SSA trustees tables) are facts typed + `assertion: source_projection`; measured outcomes are the default `assertion: observation`. Every fact also carries a required, closed `provenance_class` measurement basis; survey aggregates additionally name their `survey_instrument`. @@ -57,6 +57,22 @@ fact's `period` is the period its value refers to. Consumers must enforce any contract that aligns it to another period (see [`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)). +Chronicle holds government-produced records only. A fact's publisher must be +part of a government: a legislature or one of its offices, a ministry or +department and its agencies, a statistics office, or a public body set up to +publish official figures. Records from trade associations, charities, think +tanks, universities, firms, and aggregators that republish government data +are not Chronicle facts; when an aggregator republishes a government series, +Chronicle takes it from the government publisher. Comparisons of PolicyEngine +estimates against outside scores live in the separate +[scorecard](https://github.com/PolicyEngine/policyengine-scorecard). +[`docs/source-publishers.yaml`](docs/source-publishers.yaml) names the +publisher of every source folder, and the ADR's 2026-10-21 amendment lists +the proposed treatment of edge cases (central banks, regulators, public +corporations, intergovernmental bodies, third-party hosts) and the three +folders that predate the rule (`isc`, `ici`, `kff`), which stay until their +migration is ruled on. + | Layer | Owns | Examples | |-------|------|----------| | Chronicle Sources | Source artifacts and provenance | URLs, checksums, source files, parsed tables/cells | diff --git a/chronicle/core.py b/chronicle/core.py index 8ab671f2..b9c1326b 100644 --- a/chronicle/core.py +++ b/chronicle/core.py @@ -258,8 +258,8 @@ class AggregateFact: claim the publisher made: ``observation`` for measured or administered outcomes as published, ``source_projection`` for the publisher's own forward-looking estimate (CBO baselines, BFP outlooks, SSA trustees - tables, TPC/JCT scores). PolicyEngine-computed values are never facts; - aged, uprated, or reconciled levels belong in downstream builds. + tables). PolicyEngine-computed values are never facts; aged, uprated, + or reconciled levels belong in downstream builds. ``provenance_class`` records the publisher's measurement basis and has no default. Survey aggregates additionally require ``survey_instrument``; diff --git a/docs/adr-chronicle-facts-only.md b/docs/adr-chronicle-facts-only.md index 8bfa88dd..89f47d36 100644 --- a/docs/adr-chronicle-facts-only.md +++ b/docs/adr-chronicle-facts-only.md @@ -1,7 +1,8 @@ # ADR: Chronicle is a facts-only store Status: accepted 2026-07-02 (supersedes the facts-plus-projections split -originally proposed in issue #71) +originally proposed in issue #71); amended 2026-10-21 to admit +government-produced records only (see the amendment at the end) ## Decision @@ -44,7 +45,7 @@ Instead of projection objects, Chronicle contributes two guarantees: projections would churn on every CBO update and every aging-model version bump, turning an auditable chronicle into a store of volatile derivations. - **The consumer set does not need it.** Thesis does not want - PolicyEngine-computed aging; validation comparators (TPC, JCT scores) are + PolicyEngine-computed aging; a government's own estimates are source-published and therefore facts; the only consumer of PolicyEngine-computed aged values is Microcosm calibration, so that is where the code belongs (PolicyEngine/microcosm#116). @@ -74,3 +75,61 @@ Instead of projection objects, Chronicle contributes two guarantees: superseded; Belgian contracts also live consumer-side. - Geography vintage translation (microcosm#205) follows the same pattern: a declared consumer-side transform over facts, never an edit to them. + +## Amendment 2026-10-21: government-produced records only + +Chronicle admits a record only when a government produced it. The publisher +that asserted the value must be part of a national, subnational, or +supranational government: a legislature or one of its offices (CBO, JCT), a +ministry or department and its agencies (DWP, HMRC, the IRS), a statistics +office (ONS, Statbel, Stats NZ), or a public body that a government set up to +publish official figures (OBR, the Federal Planning Bureau). Government +estimates of a policy change, such as JCT revenue estimates, are government +records like any other. + +- Records from trade associations, charities, think tanks, universities, + firms, and secondary aggregators are not Chronicle facts, however reliable. + When an aggregator republishes a government series, Chronicle takes the + series from the government publisher. +- Comparisons of PolicyEngine estimates against outside scores and analyses + live in the separate scorecard (PolicyEngine/policyengine-scorecard), which + keeps its own provenance. This amendment removed TPC from the source + projection examples in this ADR, the README, AGENTS.md, the governance doc, + and the `AggregateFact` docstring: TPC is not a government publisher, and + Chronicle never held a TPC package. +- The rule turns on who produced a record, not where it was fetched. A + government record that reached Chronicle through a third-party host keeps + the government as its publisher. +- `docs/source-publishers.yaml` names the publisher of every source folder and + classifies it; `tests/test_source_publishers.py` fails when a folder is + unclassified or a new non-government folder appears. + +### Edge cases (proposed treatment, awaiting a ruling) + +Until a ruling, existing packages in these categories stay as they are. + +| Category | Where | Proposed treatment | Basis | +|---|---|---|---| +| Central banks | `nbb`, `federal_reserve`; central-bank rows in the Thesis observation journal (`codex/thesis-ledger-facts`: Bank of England, ECB, Bank of Japan, Reserve Bank of Australia, Bank of Canada, Federal Reserve) | In, for statistics and policy decisions they publish under a public mandate | The Belgian State holds half of the NBB's shares, and the NBB compiles Belgium's national accounts on behalf of the National Accounts Institute; the Federal Reserve Board is a federal agency | +| Regulators | `ofgem`, `orr` | In | GOV.UK lists both as non-ministerial departments | +| Public corporations and other public bodies | `nithc`, `slc` | In, for records of their own statutory functions | NITHC (Translink) is a public corporation set up under the Transport Act (Northern Ireland) 1967 and wholly owned by government; GOV.UK lists SLC as an executive non-departmental public body | +| Intergovernmental bodies | `eurostat`, `jrc` | In | Both are directorates-general of the European Commission | +| Government records on a third-party host | `dwp/pip_daily_living_foi_2025` (a DWP FOI reply on WhatDoTheyKnow); NRS census tables mirrored by the UK Data Service; `welshgov/brma_boundaries_2012` (University of Edinburgh DataShare); `treasury/an24_01_fiscal_totals` (the NZ Treasury Analytics & Insights GitHub organisation); journal rows fetched through ALFRED or the Internet Archive | In, with the government named as publisher; prefer the government-hosted copy when one exists | The producer decides, not the host | +| Government-funded work published by others | None today | Out, unless a government body publishes it under its own name | A university or contractor is then the publisher | + +### Packages that do not meet the rule + +These predate the amendment. Each stays until its migration is ruled on, so +no consumer loses a fact before a replacement exists. + +| Folder | Publisher | Government replacement | +|---|---|---| +| `isc` (`isc-annual-census-2023` to `isc-annual-census-2026`) | Independent Schools Council, the umbrella body for UK independent-school associations | DfE *Schools, pupils and their characteristics*, which counts independent-school pupils in England (all independent schools, not ISC members only) | +| `ici` (`ici-fact-book-table-30`) | Investment Company Institute, the trade association of US regulated funds | IRS SOI Table 1.4 capital gain distributions, already in Chronicle (a narrower concept: distributions reported on Form 1040) | +| `kff` (`kff-marketplace-effectuated-enrollment`) | KFF, a nonprofit; this table republishes CMS data | CMS effectuated enrollment reports; Chronicle already holds the 2022 edition (`cms_aca/effectuated_enrollment_2022`) | + +Two smaller items fall under the same rule: `db/pe_source_inventory.py` lists +a Disability Rights UK news page under `source_id: dwp` and the ISC census +page as source artifacts, and the curated Statbel NIS 2025 crosswalk +(`statbel/nis_2025_commune_crosswalk`) cites publisys.eu next to Statbel's +REFNIS file. diff --git a/docs/chronicle-governance.md b/docs/chronicle-governance.md index f60969fc..59cdbdb9 100644 --- a/docs/chronicle-governance.md +++ b/docs/chronicle-governance.md @@ -7,9 +7,9 @@ Microcosm turns those facts into active, aged, reconciled calibration targets. ## Boundary The store is facts-only, and the line is who asserted the value, not level -versus projection. A publisher's own forward-looking estimate — a CBO -baseline, a BFP outlook, an SSA trustees table, a TPC or JCT score — is a -source-backed claim like any other and enters Chronicle as a fact typed +versus projection. A government's own forward-looking estimate — a CBO +baseline, a BFP outlook, an SSA trustees table — is a source-backed claim +like any other and enters Chronicle as a fact typed `assertion: source_projection`. A value PolicyEngine computed — an aged, uprated, forecast, or reconciled level — is never a fact, whatever object it hides in. This keeps three properties intact: every value in Chronicle traces to @@ -31,6 +31,11 @@ Chronicle may: Chronicle must not: +- hold records that a non-government publisher produced (a trade + association, charity, think tank, university, firm, or an aggregator that + republishes government data); every source folder's publisher is named and + classified in `docs/source-publishers.yaml`, and edge cases follow the + 2026-10-21 amendment to `docs/adr-chronicle-facts-only.md` - reconcile across sources - age facts to a build year - store PolicyEngine-computed values (aged, uprated, forecast, or reconciled diff --git a/docs/source-publishers.yaml b/docs/source-publishers.yaml new file mode 100644 index 00000000..7aa4be55 --- /dev/null +++ b/docs/source-publishers.yaml @@ -0,0 +1,208 @@ +# Publisher of every Chronicle source folder (packages/ and +# db/data/; every package lives in the same folder as its data). +# +# Chronicle holds government-produced records only: see the 2026-10-21 +# amendment to docs/adr-chronicle-facts-only.md. tests/test_source_publishers.py +# fails when a folder is missing here, when an entry names no folder, or when a +# non-government folder appears beyond the three that predate the rule. +# +# status: +# government part of a national, subnational, or supranational government +# edge_case a category whose treatment awaits a ruling (edge_case names it); +# existing packages stay until then +# non_government predates the rule; stays until its migration is ruled on +schema_version: chronicle.source_publishers.v1 +folders: + bea: + publisher: U.S. Bureau of Economic Analysis + status: government + bfp: + publisher: Federal Planning Bureau (Belgium) + status: government + notes: Same publisher as fpb, under its French acronym. + cbo: + publisher: Congressional Budget Office + status: government + census: + publisher: U.S. Census Bureau + status: government + cms_aca: + publisher: Centers for Medicare & Medicaid Services + status: government + cms_medicaid: + publisher: Centers for Medicare & Medicaid Services + status: government + cms_medicare: + publisher: Centers for Medicare & Medicaid Services + status: government + cms_nhe: + publisher: Centers for Medicare & Medicaid Services + status: government + defra: + publisher: Department for Environment, Food & Rural Affairs + status: government + desnz: + publisher: Department for Energy Security and Net Zero + status: government + dfc_ni: + publisher: Department for Communities (Northern Ireland) + status: government + dfe: + publisher: Department for Education + status: government + dfi_ni: + publisher: Department for Infrastructure (Northern Ireland) + status: government + dft: + publisher: Department for Transport + status: government + dwp: + publisher: Department for Work and Pensions + status: government + notes: >- + pip_daily_living_foi_2025 is a DWP freedom-of-information reply fetched + from WhatDoTheyKnow, a third-party host. + eurostat: + publisher: Eurostat (European Commission) + status: edge_case + edge_case: intergovernmental + federal_reserve: + publisher: Board of Governors of the Federal Reserve System + status: edge_case + edge_case: central_bank + fpb: + publisher: Federal Planning Bureau (Belgium) + status: government + hhs_acf: + publisher: Administration for Children and Families (HHS) + status: government + hhs_acf_liheap: + publisher: Administration for Children and Families (HHS) + status: government + hmlr: + publisher: HM Land Registry + status: government + hmrc: + publisher: HM Revenue & Customs + status: government + ici: + publisher: Investment Company Institute + status: non_government + notes: Trade association of US regulated funds. + ird: + publisher: Inland Revenue (New Zealand) + status: government + irs_soi: + publisher: Internal Revenue Service, Statistics of Income + status: government + isc: + publisher: Independent Schools Council + status: non_government + notes: Umbrella body for UK independent-school associations. + jct: + publisher: Joint Committee on Taxation + status: government + jrc: + publisher: Joint Research Centre (European Commission) + status: edge_case + edge_case: intergovernmental + kff: + publisher: KFF + status: non_government + notes: Nonprofit; the packaged table republishes CMS effectuated enrollment. + lps: + publisher: Land & Property Services (Department of Finance, Northern Ireland) + status: government + mbie: + publisher: Ministry of Business, Innovation and Employment (New Zealand) + status: government + notes: Tenancy bond data from Tenancy Services (tenancy.govt.nz). + mhclg: + publisher: Ministry of Housing, Communities and Local Government + status: government + msd: + publisher: Ministry of Social Development (New Zealand) + status: government + nbb: + publisher: National Bank of Belgium + status: edge_case + edge_case: central_bank + nisra: + publisher: Northern Ireland Statistics and Research Agency + status: government + nithc: + publisher: Northern Ireland Transport Holding Company (Translink) + status: edge_case + edge_case: public_corporation + nrs: + publisher: National Records of Scotland + status: government + notes: >- + Some census 2022 tables are fetched from UK Data Service mirrors, a + third-party host. + obr: + publisher: Office for Budget Responsibility + status: government + ofgem: + publisher: Ofgem + status: edge_case + edge_case: regulator + onem_rva: + publisher: National Employment Office (ONEM/RVA, Belgium) + status: government + ons: + publisher: Office for National Statistics + status: government + onss: + publisher: National Social Security Office (ONSS/RSZ, Belgium) + status: government + opgroeien: + publisher: Opgroeien regie (Flemish Government agency) + status: government + orr: + publisher: Office of Rail and Road + status: edge_case + edge_case: regulator + scotgov: + publisher: Scottish Government + status: government + sfpd: + publisher: Federal Pensions Service (SFP/SFPD, Belgium) + status: government + slc: + publisher: Student Loans Company + status: edge_case + edge_case: public_corporation + spf_finances: + publisher: Federal Public Service Finance (Belgium), released by Statbel + status: government + ssa: + publisher: Social Security Administration + status: government + statbel: + publisher: Statbel (Belgian statistical office) + status: government + notes: >- + The curated nis_2025_commune_crosswalk cites publisys.eu next to + Statbel's own REFNIS file. + stats_nz: + publisher: Stats NZ + status: government + treasury: + publisher: The Treasury (New Zealand) + status: government + notes: >- + an24_01_fiscal_totals is fetched from the NZ Treasury Analytics & + Insights GitHub organisation, a third-party host. + usda_snap: + publisher: USDA Food and Nutrition Service + status: government + voa: + publisher: Valuation Office Agency (now part of HMRC) + status: government + welshgov: + publisher: Welsh Government + status: government + notes: >- + brma_boundaries_2012 is fetched from University of Edinburgh DataShare, + a third-party host. diff --git a/tests/test_source_publishers.py b/tests/test_source_publishers.py new file mode 100644 index 00000000..ca187c4e --- /dev/null +++ b/tests/test_source_publishers.py @@ -0,0 +1,72 @@ +"""Every source folder names its publisher, and only government ones are added. + +Chronicle holds government-produced records only (the 2026-10-21 amendment to +docs/adr-chronicle-facts-only.md). docs/source-publishers.yaml classifies the +publisher of each folder under packages/ and db/data/. +""" + +from __future__ import annotations + +from pathlib import Path + +from chronicle.yaml_io import safe_load + + +ROOT = Path(__file__).resolve().parents[1] +REGISTRY_PATH = ROOT / "docs" / "source-publishers.yaml" + +STATUSES = {"government", "edge_case", "non_government"} +EDGE_CASES = {"central_bank", "regulator", "public_corporation", "intergovernmental"} +# Non-government folders that predate the rule and await a migration ruling. +# A new non-government source must not be added here: take the series from the +# government publisher instead. +PREDATING_NON_GOVERNMENT = {"isc", "ici", "kff"} + + +def _registry() -> dict: + return safe_load(REGISTRY_PATH.read_text()) + + +def _source_folders() -> set[str]: + folders = {path.name for path in (ROOT / "packages").iterdir() if path.is_dir()} | { + path.name for path in (ROOT / "db" / "data").iterdir() if path.is_dir() + } + return {name for name in folders if not name.startswith(("_", "."))} + + +def test_registry_names_exactly_the_source_folders(): + registered = set(_registry()["folders"]) + folders = _source_folders() + + assert sorted(folders - registered) == [], ( + "source folders without a publisher entry in docs/source-publishers.yaml" + ) + assert sorted(registered - folders) == [], ( + "docs/source-publishers.yaml entries with no packages/ or db/data/ folder" + ) + + +def test_every_folder_has_a_publisher_and_a_known_status(): + registry = _registry() + assert registry["schema_version"] == "chronicle.source_publishers.v1" + + for folder, entry in registry["folders"].items(): + assert entry["publisher"].strip(), folder + assert entry["status"] in STATUSES, folder + if entry["status"] == "edge_case": + assert entry["edge_case"] in EDGE_CASES, folder + else: + assert "edge_case" not in entry, folder + + +def test_no_new_non_government_source_folders(): + non_government = { + folder + for folder, entry in _registry()["folders"].items() + if entry["status"] == "non_government" + } + + assert non_government <= PREDATING_NON_GOVERNMENT, ( + "Chronicle holds government-produced records only; " + f"new non-government folders: {sorted(non_government - PREDATING_NON_GOVERNMENT)}" + )