Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,23 @@ representation, and preserve provenance. Selection and measurement contracts liv
in consumers such as Microcosm.

Every fact value must trace to a publisher. The boundary is who asserted the
value, not level versus projection: a publisher's own projection (CBO baseline,
BFP outlook, SSA trustees, TPC/JCT score) is a fact typed
value, not level versus projection: a government's own projection (CBO
baseline, BFP outlook, SSA trustees) is a fact typed
`assertion: source_projection`. PolicyEngine-computed values — aged, uprated,
forecast, or reconciled levels — are never Chronicle facts.

Chronicle holds government-produced records only. Before adding a source
folder or package, confirm the publisher is part of a government and add or
update its entry in `docs/source-publishers.yaml`; `tests/test_source_publishers.py`
fails on an unclassified folder or a new non-government one. Do not add
records from trade associations, charities, think tanks, universities,
firms, or aggregators that republish government data: take the series from
the government publisher instead. Comparisons against outside scores belong
in PolicyEngine/policyengine-scorecard, not here. Edge cases (central banks,
regulators, public corporations, intergovernmental bodies, government records
on a third-party host) follow the 2026-10-21 amendment in
`docs/adr-chronicle-facts-only.md`.

Do not put Microcosm work in Chronicle:

- no cross-source reconciliation
Expand Down
22 changes: 19 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ This repository provides:
- **Sources**: Source file references, retrieval metadata, manifests, checksums,
and provenance.
- **Facts**: Source-backed claims represented with typed values, units,
geography, period, source table, and lineage. Publisher projections (CBO
baselines, BFP outlooks, SSA trustees tables, TPC/JCT scores) are facts
typed `assertion: source_projection`; measured outcomes are the default
geography, period, source table, and lineage. Government projections (CBO
baselines, BFP outlooks, SSA trustees tables) are facts typed
`assertion: source_projection`; measured outcomes are the default
`assertion: observation`. Every fact also carries a required, closed
`provenance_class` measurement basis; survey aggregates additionally name
their `survey_instrument`.
Expand Down Expand Up @@ -57,6 +57,22 @@ fact's `period` is the period its value refers to. Consumers must enforce any
contract that aligns it to another period (see
[`docs/adr-chronicle-facts-only.md`](docs/adr-chronicle-facts-only.md)).

Chronicle holds government-produced records only. A fact's publisher must be
part of a government: a legislature or one of its offices, a ministry or
department and its agencies, a statistics office, or a public body set up to
publish official figures. Records from trade associations, charities, think
tanks, universities, firms, and aggregators that republish government data
are not Chronicle facts; when an aggregator republishes a government series,
Chronicle takes it from the government publisher. Comparisons of PolicyEngine
estimates against outside scores live in the separate
[scorecard](https://github.com/PolicyEngine/policyengine-scorecard).
[`docs/source-publishers.yaml`](docs/source-publishers.yaml) names the
publisher of every source folder, and the ADR's 2026-10-21 amendment lists
the proposed treatment of edge cases (central banks, regulators, public
corporations, intergovernmental bodies, third-party hosts) and the three
folders that predate the rule (`isc`, `ici`, `kff`), which stay until their
migration is ruled on.

| Layer | Owns | Examples |
|-------|------|----------|
| Chronicle Sources | Source artifacts and provenance | URLs, checksums, source files, parsed tables/cells |
Expand Down
4 changes: 2 additions & 2 deletions chronicle/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,8 @@ class AggregateFact:
claim the publisher made: ``observation`` for measured or administered
outcomes as published, ``source_projection`` for the publisher's own
forward-looking estimate (CBO baselines, BFP outlooks, SSA trustees
tables, TPC/JCT scores). PolicyEngine-computed values are never facts;
aged, uprated, or reconciled levels belong in downstream builds.
tables). PolicyEngine-computed values are never facts; aged, uprated,
or reconciled levels belong in downstream builds.

``provenance_class`` records the publisher's measurement basis and has no
default. Survey aggregates additionally require ``survey_instrument``;
Expand Down
63 changes: 61 additions & 2 deletions docs/adr-chronicle-facts-only.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# ADR: Chronicle is a facts-only store

Status: accepted 2026-07-02 (supersedes the facts-plus-projections split
originally proposed in issue #71)
originally proposed in issue #71); amended 2026-10-21 to admit
government-produced records only (see the amendment at the end)

## Decision

Expand Down Expand Up @@ -44,7 +45,7 @@ Instead of projection objects, Chronicle contributes two guarantees:
projections would churn on every CBO update and every aging-model version
bump, turning an auditable chronicle into a store of volatile derivations.
- **The consumer set does not need it.** Thesis does not want
PolicyEngine-computed aging; validation comparators (TPC, JCT scores) are
PolicyEngine-computed aging; a government's own estimates are
source-published and therefore facts; the only consumer of
PolicyEngine-computed aged values is Microcosm calibration, so that is
where the code belongs (PolicyEngine/microcosm#116).
Expand Down Expand Up @@ -74,3 +75,61 @@ Instead of projection objects, Chronicle contributes two guarantees:
superseded; Belgian contracts also live consumer-side.
- Geography vintage translation (microcosm#205) follows the same pattern: a
declared consumer-side transform over facts, never an edit to them.

## Amendment 2026-10-21: government-produced records only

Chronicle admits a record only when a government produced it. The publisher
that asserted the value must be part of a national, subnational, or
supranational government: a legislature or one of its offices (CBO, JCT), a
ministry or department and its agencies (DWP, HMRC, the IRS), a statistics
office (ONS, Statbel, Stats NZ), or a public body that a government set up to
publish official figures (OBR, the Federal Planning Bureau). Government
estimates of a policy change, such as JCT revenue estimates, are government
records like any other.

- Records from trade associations, charities, think tanks, universities,
firms, and secondary aggregators are not Chronicle facts, however reliable.
When an aggregator republishes a government series, Chronicle takes the
series from the government publisher.
- Comparisons of PolicyEngine estimates against outside scores and analyses
live in the separate scorecard (PolicyEngine/policyengine-scorecard), which
keeps its own provenance. This amendment removed TPC from the source
projection examples in this ADR, the README, AGENTS.md, the governance doc,
and the `AggregateFact` docstring: TPC is not a government publisher, and
Chronicle never held a TPC package.
- The rule turns on who produced a record, not where it was fetched. A
government record that reached Chronicle through a third-party host keeps
the government as its publisher.
- `docs/source-publishers.yaml` names the publisher of every source folder and
classifies it; `tests/test_source_publishers.py` fails when a folder is
unclassified or a new non-government folder appears.

### Edge cases (proposed treatment, awaiting a ruling)

Until a ruling, existing packages in these categories stay as they are.

| Category | Where | Proposed treatment | Basis |
|---|---|---|---|
| Central banks | `nbb`, `federal_reserve`; central-bank rows in the Thesis observation journal (`codex/thesis-ledger-facts`: Bank of England, ECB, Bank of Japan, Reserve Bank of Australia, Bank of Canada, Federal Reserve) | In, for statistics and policy decisions they publish under a public mandate | The Belgian State holds half of the NBB's shares, and the NBB compiles Belgium's national accounts on behalf of the National Accounts Institute; the Federal Reserve Board is a federal agency |
| Regulators | `ofgem`, `orr` | In | GOV.UK lists both as non-ministerial departments |
| Public corporations and other public bodies | `nithc`, `slc` | In, for records of their own statutory functions | NITHC (Translink) is a public corporation set up under the Transport Act (Northern Ireland) 1967 and wholly owned by government; GOV.UK lists SLC as an executive non-departmental public body |
| Intergovernmental bodies | `eurostat`, `jrc` | In | Both are directorates-general of the European Commission |
| Government records on a third-party host | `dwp/pip_daily_living_foi_2025` (a DWP FOI reply on WhatDoTheyKnow); NRS census tables mirrored by the UK Data Service; `welshgov/brma_boundaries_2012` (University of Edinburgh DataShare); `treasury/an24_01_fiscal_totals` (the NZ Treasury Analytics & Insights GitHub organisation); journal rows fetched through ALFRED or the Internet Archive | In, with the government named as publisher; prefer the government-hosted copy when one exists | The producer decides, not the host |
| Government-funded work published by others | None today | Out, unless a government body publishes it under its own name | A university or contractor is then the publisher |

### Packages that do not meet the rule

These predate the amendment. Each stays until its migration is ruled on, so
no consumer loses a fact before a replacement exists.

| Folder | Publisher | Government replacement |
|---|---|---|
| `isc` (`isc-annual-census-2023` to `isc-annual-census-2026`) | Independent Schools Council, the umbrella body for UK independent-school associations | DfE *Schools, pupils and their characteristics*, which counts independent-school pupils in England (all independent schools, not ISC members only) |
| `ici` (`ici-fact-book-table-30`) | Investment Company Institute, the trade association of US regulated funds | IRS SOI Table 1.4 capital gain distributions, already in Chronicle (a narrower concept: distributions reported on Form 1040) |
| `kff` (`kff-marketplace-effectuated-enrollment`) | KFF, a nonprofit; this table republishes CMS data | CMS effectuated enrollment reports; Chronicle already holds the 2022 edition (`cms_aca/effectuated_enrollment_2022`) |

Two smaller items fall under the same rule: `db/pe_source_inventory.py` lists
a Disability Rights UK news page under `source_id: dwp` and the ISC census
page as source artifacts, and the curated Statbel NIS 2025 crosswalk
(`statbel/nis_2025_commune_crosswalk`) cites publisys.eu next to Statbel's
REFNIS file.
11 changes: 8 additions & 3 deletions docs/chronicle-governance.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ Microcosm turns those facts into active, aged, reconciled calibration targets.
## Boundary

The store is facts-only, and the line is who asserted the value, not level
versus projection. A publisher's own forward-looking estimate — a CBO
baseline, a BFP outlook, an SSA trustees table, a TPC or JCT score — is a
source-backed claim like any other and enters Chronicle as a fact typed
versus projection. A government's own forward-looking estimate — a CBO
baseline, a BFP outlook, an SSA trustees table — is a source-backed claim
like any other and enters Chronicle as a fact typed
`assertion: source_projection`. A value PolicyEngine computed — an aged,
uprated, forecast, or reconciled level — is never a fact, whatever object it
hides in. This keeps three properties intact: every value in Chronicle traces to
Expand All @@ -31,6 +31,11 @@ Chronicle may:

Chronicle must not:

- hold records that a non-government publisher produced (a trade
association, charity, think tank, university, firm, or an aggregator that
republishes government data); every source folder's publisher is named and
classified in `docs/source-publishers.yaml`, and edge cases follow the
2026-10-21 amendment to `docs/adr-chronicle-facts-only.md`
- reconcile across sources
- age facts to a build year
- store PolicyEngine-computed values (aged, uprated, forecast, or reconciled
Expand Down
208 changes: 208 additions & 0 deletions docs/source-publishers.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
# Publisher of every Chronicle source folder (packages/<folder> and
# db/data/<folder>; every package lives in the same folder as its data).
#
# Chronicle holds government-produced records only: see the 2026-10-21
# amendment to docs/adr-chronicle-facts-only.md. tests/test_source_publishers.py
# fails when a folder is missing here, when an entry names no folder, or when a
# non-government folder appears beyond the three that predate the rule.
#
# status:
# government part of a national, subnational, or supranational government
# edge_case a category whose treatment awaits a ruling (edge_case names it);
# existing packages stay until then
# non_government predates the rule; stays until its migration is ruled on
schema_version: chronicle.source_publishers.v1
folders:
bea:
publisher: U.S. Bureau of Economic Analysis
status: government
bfp:
publisher: Federal Planning Bureau (Belgium)
status: government
notes: Same publisher as fpb, under its French acronym.
cbo:
publisher: Congressional Budget Office
status: government
census:
publisher: U.S. Census Bureau
status: government
cms_aca:
publisher: Centers for Medicare & Medicaid Services
status: government
cms_medicaid:
publisher: Centers for Medicare & Medicaid Services
status: government
cms_medicare:
publisher: Centers for Medicare & Medicaid Services
status: government
cms_nhe:
publisher: Centers for Medicare & Medicaid Services
status: government
defra:
publisher: Department for Environment, Food & Rural Affairs
status: government
desnz:
publisher: Department for Energy Security and Net Zero
status: government
dfc_ni:
publisher: Department for Communities (Northern Ireland)
status: government
dfe:
publisher: Department for Education
status: government
dfi_ni:
publisher: Department for Infrastructure (Northern Ireland)
status: government
dft:
publisher: Department for Transport
status: government
dwp:
publisher: Department for Work and Pensions
status: government
notes: >-
pip_daily_living_foi_2025 is a DWP freedom-of-information reply fetched
from WhatDoTheyKnow, a third-party host.
eurostat:
publisher: Eurostat (European Commission)
status: edge_case
edge_case: intergovernmental
federal_reserve:
publisher: Board of Governors of the Federal Reserve System
status: edge_case
edge_case: central_bank
fpb:
publisher: Federal Planning Bureau (Belgium)
status: government
hhs_acf:
publisher: Administration for Children and Families (HHS)
status: government
hhs_acf_liheap:
publisher: Administration for Children and Families (HHS)
status: government
hmlr:
publisher: HM Land Registry
status: government
hmrc:
publisher: HM Revenue & Customs
status: government
ici:
publisher: Investment Company Institute
status: non_government
notes: Trade association of US regulated funds.
ird:
publisher: Inland Revenue (New Zealand)
status: government
irs_soi:
publisher: Internal Revenue Service, Statistics of Income
status: government
isc:
publisher: Independent Schools Council
status: non_government
notes: Umbrella body for UK independent-school associations.
jct:
publisher: Joint Committee on Taxation
status: government
jrc:
publisher: Joint Research Centre (European Commission)
status: edge_case
edge_case: intergovernmental
kff:
publisher: KFF
status: non_government
notes: Nonprofit; the packaged table republishes CMS effectuated enrollment.
lps:
publisher: Land & Property Services (Department of Finance, Northern Ireland)
status: government
mbie:
publisher: Ministry of Business, Innovation and Employment (New Zealand)
status: government
notes: Tenancy bond data from Tenancy Services (tenancy.govt.nz).
mhclg:
publisher: Ministry of Housing, Communities and Local Government
status: government
msd:
publisher: Ministry of Social Development (New Zealand)
status: government
nbb:
publisher: National Bank of Belgium
status: edge_case
edge_case: central_bank
nisra:
publisher: Northern Ireland Statistics and Research Agency
status: government
nithc:
publisher: Northern Ireland Transport Holding Company (Translink)
status: edge_case
edge_case: public_corporation
nrs:
publisher: National Records of Scotland
status: government
notes: >-
Some census 2022 tables are fetched from UK Data Service mirrors, a
third-party host.
obr:
publisher: Office for Budget Responsibility
status: government
ofgem:
publisher: Ofgem
status: edge_case
edge_case: regulator
onem_rva:
publisher: National Employment Office (ONEM/RVA, Belgium)
status: government
ons:
publisher: Office for National Statistics
status: government
onss:
publisher: National Social Security Office (ONSS/RSZ, Belgium)
status: government
opgroeien:
publisher: Opgroeien regie (Flemish Government agency)
status: government
orr:
publisher: Office of Rail and Road
status: edge_case
edge_case: regulator
scotgov:
publisher: Scottish Government
status: government
sfpd:
publisher: Federal Pensions Service (SFP/SFPD, Belgium)
status: government
slc:
publisher: Student Loans Company
status: edge_case
edge_case: public_corporation
spf_finances:
publisher: Federal Public Service Finance (Belgium), released by Statbel
status: government
ssa:
publisher: Social Security Administration
status: government
statbel:
publisher: Statbel (Belgian statistical office)
status: government
notes: >-
The curated nis_2025_commune_crosswalk cites publisys.eu next to
Statbel's own REFNIS file.
stats_nz:
publisher: Stats NZ
status: government
treasury:
publisher: The Treasury (New Zealand)
status: government
notes: >-
an24_01_fiscal_totals is fetched from the NZ Treasury Analytics &
Insights GitHub organisation, a third-party host.
usda_snap:
publisher: USDA Food and Nutrition Service
status: government
voa:
publisher: Valuation Office Agency (now part of HMRC)
status: government
welshgov:
publisher: Welsh Government
status: government
notes: >-
brma_boundaries_2012 is fetched from University of Edinburgh DataShare,
a third-party host.
Loading
Loading