From 8bcb66a88126a237e4fae0fafd732eee18f77fef Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Wed, 21 Oct 2026 23:12:45 -0400 Subject: [PATCH 1/2] Add Stats NZ regional ERP at 30 June 2023 from the pinned 2025 release G8-d4: add 17 direct Table 1 B7:B22/B25 cells to the existing package. Keep CY2023 / 2023-06-30 coverage, the unmarked numeric 2023 header, REGC2025 geography and existing concept/provenance/assertion. Guard source headings and row labels. Contents and workbook footnotes do not explicitly call 2023 final or revised. No new artifact or computed values. Measured single-source build_bundle delta (before/after outputs deleted): - facts 85 -> 102 (+17); source cells 2265 -> 2265 (+0). - geography keys 17 -> 17 (+0); no added/removed by_geography keys. - sources 1 -> 1 (+0); packages 1 -> 1 (+0). - source-table keys 1 -> 1 (+0); parsed source sheets 2 -> 2 (+0). - bundle warnings 0 -> 0 (+0); errors and duplicate-key counts stay 0. - period keys 1 -> 2 (+1), adding calendar_year:2023 with 17 facts; calendar_year:2025 stays 85. Entity keys 1 -> 1 (+0), person 85 -> 102. - source-region count 5 -> 6 (+1); source-record and consumer-fact counts 85 -> 102 (+17). Unique aggregate and semantic keys 85 -> 102 (+17); unique source-series keys 5 -> 6 (+1); other unique key counts unchanged. - observed measure stats_nz:population +17; observed/canonical concept stats_nz.estimated_resident_population +17 each. - by_source stats_nz +17; existing table key +17: stats_nz:Subnational population estimates: At 30 June 2025 (provisional), Tables 1 and 3 - by_geography +1 each (5 -> 6): country:NZ, region:01, region:02, region:03, region:04, region:05, region:06, region:07, region:08, region:09, region:12, region:13, region:14, region:15, region:16, region:17, region:18. Default-bundle pins = local main 140316a182e6082b5b9d203bd0756f666e626aae (pins identical at assigned base aa288c3032348d5cf2688e6fc1102af52fe4c7fc) plus that measured delta: - facts 1579263 -> 1579280 (+17). - geography keys 102613 -> 102613 (+0); source count 56 -> 56 (+0); packages 405 -> 405 (+0); table keys 347 -> 347 (+0). - warnings 122 -> 122 (+0); period keys 523 -> 523 (+0), because calendar_year:2023 already exists; entity keys 12 -> 12 (+0). - calendar_year:2023 111185 -> 111202 (+17); person 521319 -> 521336 (+17). - stats_nz 2015 -> 2032 (+17); existing Tables 1 and 3 key 85 -> 102 (+17). - country:NZ 6131 -> 6132 (+1); each of the 16 region keys listed above 30 -> 31 (+1); no added/removed by_geography keys. Original 85 facts, consumer rows and full source cells are byte-identical. Both B-to-C selector mutants were killed by the exact-cell regression test; package restored afterward. Independent source-fidelity, boundary and pin arithmetic reviews PASS. Targeted package tests: 58 passed (12 dependency deprecation warnings). Ruff checks pass for both changed test files. Full/default bundle test intentionally left to CI. Co-Authored-By: GPT-6.1 Sol --- .../nz-regional-population-2023.added.md | 5 + docs/pe-nz-population-source-notes.md | 51 +++- docs/pe-nz-source-checklist.md | 2 +- .../README.md | 45 ++- .../source_package.yaml | 272 +++++++++++++++++- tests/test_chronicle_bundle.py | 20 +- tests/test_chronicle_nz_population.py | 195 ++++++++++--- 7 files changed, 517 insertions(+), 73 deletions(-) create mode 100644 changelog.d/nz-regional-population-2023.added.md diff --git a/changelog.d/nz-regional-population-2023.added.md b/changelog.d/nz-regional-population-2023.added.md new file mode 100644 index 00000000..3e57bb08 --- /dev/null +++ b/changelog.d/nz-regional-population-2023.added.md @@ -0,0 +1,5 @@ +Add 17 Stats NZ estimated resident population facts at 30 June 2023: the +16 regions and the directly published New Zealand total from Table 1's +unmarked 2023 column in the existing pinned October 2025 workbook. Preserve +the source's reference date, 2023 census base, REGC 2025 geographies and +unmarked status; compute no ratios, growth rates or aligned values. diff --git a/docs/pe-nz-population-source-notes.md b/docs/pe-nz-population-source-notes.md index 679c6c5b..7d3bc2fb 100644 --- a/docs/pe-nz-population-source-notes.md +++ b/docs/pe-nz-population-source-notes.md @@ -3,7 +3,8 @@ This records the bounded source-ingestion work for demographic packages 1 and 2 of [issue #176](https://github.com/PolicyEngine/chronicle/issues/176), checked on 30 August 2026 UTC, with G8-d2 territorial-authority evidence added on -13 October 2026 UTC. It does not mark the demographic target surface complete. +13 October 2026 UTC and G8-d4's 2023-reference-date cells verified locally on +22 October 2026 UTC. It does not mark the demographic target surface complete. ## Pinned publisher evidence @@ -32,8 +33,9 @@ The NZ hub published the TA 2025 classification to its declared R2 key on 13 October 2026 and verified it by streaming the remote object back: its SHA-256 matched the manifest. The workbook and its existing manifest are unchanged. -The two population references share a 30 June 2025 reference date, not a -publication date. The national landing page is evidence of the missing download +The two population release titles refer to 30 June 2025, not their +publication dates; the subnational workbook also publishes historical 2023 +and 2024 estimates. The national landing page is evidence of the missing download surface, **not** a single-year-age data artifact or a national source package. Both classifications are geography evidence only and emit no population facts. @@ -69,11 +71,44 @@ authority areas; published rounding also qualifies comparisons with the territorial-authority figures. No total, Auckland local-board sum, residual, share, aging, imputation, or reconciliation is computed or emitted. +## G8-d4: regional population at 30 June 2023 + +The existing [regional package](../packages/stats_nz/subnational_population_estimates_2025/README.md) +also reads Table 1!B7:B22 and B25: 16 regional council areas and the separately +published New Zealand total. These 17 all-age ERP facts use `calendar_year: 2023` +and equal coverage dates `2023-06-30`. Each is one publisher cell at scale 1, +with the same ERP concept, REGC codes/names, `regc_2025` regional vintage, +`provenance_class: census`, and `assertion: observation` as the 2025 facts. +The national geography remains `country:NZ`, vintage `current`. + +Verified with openpyxl `data_only`: A3 is `Regional council areas`; A4 is +`At 30 June 2023–2025`; A5 is `Regional council area(2)`; B5 is `At 30 June`; +B6 is numeric `2023`, with no `P`. C6 and D6 are `2024 P` and `2025 P`. +Samples are Northland 198,500 (B7), Auckland 1,755,200 (B8), Wellington +538,000 (B15), Canterbury 676,600 (B20), Southland 102,700 (B22), and New +Zealand 5,200,000 (B25). All 17 cells are compared directly in the tests. + +The Contents sheet and workbook footnotes do not explicitly call 2023 final or +revised. Table 1!A34 defines `P provisional`; the unmarked B6 header is retained +without assigning provisional, final or revised status. A26:A28 define ERP +using the 2023 census usually resident population count, adjusted for net +census under/overcount, residents temporarily overseas, and births, deaths +and net migration up to the estimate date. These 2023 estimates are the +release's 2023-base series, not the 7 March 2023 census-night count. + +Guards cover A3, A4, A5, B5, numeric B6, every selected row header, the +1 January 2025 boundary footnote A29, and the P-symbol definition A34. +Rows 23–24 (island subtotals), the 2024 P column and population-change columns +remain preserved source cells without facts. No new source artifact is added; +the workbook, manifest, 2,265 parsed cells, and the existing 85 facts are +unchanged. The package constructs no ratios, growth rates or aligned values. + ## S7: partial workbook package The [source package](../packages/stats_nz/subnational_population_estimates_2025/README.md) -admits 85 cells from Tables 1 and 3: 17 geographic rows × all ages/four broad -age bands. Its precise scope, semantics, and guards are documented there. +admits 102 cells from Tables 1 and 3: 85 at 30 June 2025 (17 geographic rows × +all ages/four broad age bands), plus the 17 G8-d4 all-age cells at 30 June 2023. +Its precise scope, semantics, and guards are documented there. The workbook does not contain the requested five-year-age × sex × region cut. Stats NZ's public metadata catalog lists `POPES_SUB_004`: @@ -97,7 +132,7 @@ revision vintage before it can extend this package. In particular, `POPES_SUB_003` advertises **2026 boundaries** and is not a substitute. The regional five-year-age × sex table is a **follow-up**, not part of this -package. This package lands only the 85 partial counts above. The follow-up +package. This package lands only the 102 partial counts above. The follow-up extends it once an authorized export passes those checks; until then the package emits no five-year-age or sex-specific fact. @@ -129,7 +164,7 @@ registered, and the national facts count added by S8 is **zero**. The partial S7 package passes source-package validation, exact source-cell comparisons, geography mapping checks, source-semantic drift tests, consumer contract validation, and an explicit one-package build. The bounded source -build records 85/85 lineaged facts and 2,265 parsed cells. +build records 102/102 lineaged facts and 2,265 parsed cells. The source build warns that Axiom CLI canonical-concept validation did not run. All mappings are source-label relations; this work claims neither Axiom @@ -137,6 +172,6 @@ canonical certification nor calibration quality. Microcosm retains responsibility for choosing constraints, reconciling source universes, aging, imputing, constructing solver targets, and activating a -population. The 85 counts are available source facts, not proof that these +population. The 102 counts are available source facts, not proof that these downstream steps have run. Independent source-fidelity and boundary review is still required before landing. diff --git a/docs/pe-nz-source-checklist.md b/docs/pe-nz-source-checklist.md index 4b9f13fe..2bed5172 100644 --- a/docs/pe-nz-source-checklist.md +++ b/docs/pe-nz-source-checklist.md @@ -53,7 +53,7 @@ tests, and record a verified `raw/nz/...` R2 URI. | Package from #176 | Artifact pinned | Package valid | `raw/nz` verified | Notes | |---|---:|---:|---:|---| -| `stats_nz/subnational_population_estimates_2025` | [x] | [ ] | [x] | **Partial:** 85 validated workbook counts (16 regions + published NZ total; all ages/four broad age bands). Five-year-age × sex coverage is missing; its ADE structure request returned HTTP 401. Package-completion box stays unchecked. [Evidence and access notes](pe-nz-population-source-notes.md). | +| `stats_nz/subnational_population_estimates_2025` | [x] | [ ] | [x] | **Partial:** 102 workbook counts: 85 at 30 June 2025 (16 regions + published NZ total; all ages/four broad age bands), plus G8-d4's 17 all-age counts at 30 June 2023 from Table 1!B7:B22 and B25. The 2023 header has no P; Contents and workbook footnotes do not explicitly call 2023 final or revised. Same pinned workbook and REGC 2025 geographies; no new artifact, ratios or alignment. Five-year-age × sex coverage is missing; its ADE structure request returned HTTP 401. Package-completion box stays unchecked. [Evidence and access notes](pe-nz-population-source-notes.md). | | `stats_nz/subnational_population_ta_2025` | [x] | [x] | [x] | G8-d2: 67 provisional all-age estimated resident populations at 30 June 2025, one Table 2!E cell per territorial authority; `local_authority`, `nz-ta-NNN`, `ta_2025`. Same unchanged workbook; codes and names come from the pinned 68-feature Stats NZ TA 2025 classification. The hub published the classification to its `raw/nz/stats_nz/...` key and verified it by SHA-256 on 2026-10-13. Local boards, previous years, change/percent columns and the duplicate NZ row are preserved but not emitted; no sums or reconciliation. [Scope](../packages/stats_nz/subnational_population_ta_2025/README.md) and [evidence](pe-nz-population-source-notes.md). | | `stats_nz/national_population_estimates_2025` | [ ] | [ ] | [ ] | **Blocked:** no original-vintage single-year-age artifact acquired. The 19 August 2025 release points to Infoshare, not an XLSX. Landing-page evidence only is hash-pinned in `raw/nz`; it emits zero facts. [Evidence and access notes](pe-nz-population-source-notes.md). | | `stats_nz/census_2023_households_by_region` | [x] | [ ] | [x] | **Partial:** B5: 216 published 2023 Census counts, 12 per place (NZ, 16 regional councils and separate Area Outside Region): households, three tenure categories and eight usual-resident household sizes. Requested household composition and social-renting split are absent from this product. #176 named the S2 ArcGIS hub regional-council layer as the source; the published ArcGIS household and family layers are SA1/SA2 only, so regional household composition needs the Aotearoa Data Explorer (`CEN23_FHH_038`, API key or manual export) and stays open. Package-completion box stays unchecked until then. National root plus per-place members share the 18 unchanged receipt-pinned CSVs with the family package; the hub published the 18 objects to their `raw/nz/stats_nz/...` keys and verified each by SHA-256 on 2026-10-10. [Scope and evidence](../packages/stats_nz/census_2023_households_by_region/README.md). | diff --git a/packages/stats_nz/subnational_population_estimates_2025/README.md b/packages/stats_nz/subnational_population_estimates_2025/README.md index 825c54b6..cd150072 100644 --- a/packages/stats_nz/subnational_population_estimates_2025/README.md +++ b/packages/stats_nz/subnational_population_estimates_2025/README.md @@ -1,8 +1,9 @@ -# Stats NZ subnational population, 30 June 2025 — partial +# Stats NZ subnational population, 30 June 2023 and 2025 — partial -This package emits 85 publisher-backed population counts: the 16 regional +This package emits 102 publisher-backed population counts: the 16 regional council areas and the separately published New Zealand total, each for all ages -and the four source age bands. It does **not** complete the five-year-age × sex × +at 30 June 2023 and 2025, plus the four source age bands at 30 June 2025. +It does **not** complete the five-year-age × sex × region surface requested by [issue #176](https://github.com/PolicyEngine/chronicle/issues/176). The [29 October 2025 release](https://www.stats.govt.nz/information-releases/subnational-population-estimates-at-30-june-2025/) @@ -10,6 +11,7 @@ and its unchanged XLSX provide these cells: | Source sheet | Exact selection | Facts | |---|---|---:| +| Table 1 | B7:B22 and B25, headed numeric `2023` (no `P`) | 17 | | Table 1 | D7:D22 and D25, headed `2025 P` | 17 | | Table 3 | Columns D:G, rows 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 64 | 68 | @@ -19,35 +21,43 @@ worksheets are parsed over their complete used ranges (2,265 cells), including unselected years and explanatory footnotes. Territorial-authority all-age counts from Table 2 are ingested separately by [subnational_population_ta_2025](../subnational_population_ta_2025/README.md). -Local-board tables, island subtotals, percentages, median ages, and -population-change components are not ingested by this package. +The `2024 P` column, local-board tables, island subtotals, percentages, median +ages, and population-change components are not ingested by this package. ## Reference period and population definition -- Reference date: **30 June 2025**, expressed as `calendar_year: 2025` plus - equal start/end coverage dates `2025-06-30`. These are point-in-time counts, +- Reference dates: **30 June 2023** and **30 June 2025**, expressed as + `calendar_year: 2023` / `2025` plus equal start/end coverage dates + `2023-06-30` / `2025-06-30`, respectively. These are point-in-time counts, not annual average populations. -- Publication vintage: **29 October 2025**; all selected figures are - **provisional**. The source artifact remains pinned even if Stats NZ later - revises its historical estimates. +- Publication vintage: **29 October 2025**. The selected **2025** figures are + **provisional**. Table 1!B6 is numeric `2023` without a `P` marker, so the + 2023 facts are not marked provisional. The workbook's Contents and footnotes + do not explicitly label 2023 final or revised; neither status is added. + The source artifact remains pinned even if Stats NZ later revises its + historical estimates. - Population: estimated resident population (ERP), based on the 2023 census usually resident population and adjusted for net census under/overcount, residents temporarily overseas on census night, and subsequent births, deaths, and net migration. This is not the unadjusted census-night count. + The 2023 estimates are the release's 2023-base series, not the 7 March 2023 + census-night count (Table 1!A26:A28). - Geography: regional boundaries at **1 January 2025**, with REGC codes stored as strings and `geography_vintage: regc_2025`. The separately pinned official [REGC 2025 attributes](https://services2.arcgis.com/vKb0s8tBIA3bdocZ/arcgis/rest/services/Regional_Council_2025/FeatureServer) supply the codes and names. The classification names code `02` “Auckland”; the population workbook labels the same region “Auckland region”. -- National scope: Table 1!D25 publishes **5,324,700**, including areas outside - regions, such as Chatham Islands territory. The package reads the published +- National scope: Table 1!B25 publishes **5,200,000** for 2023 and D25 publishes + **5,324,700** for 2025, including areas outside regions, such as Chatham + Islands territory. The package reads the published national cells directly. It neither sums the 16 regions nor constructs a code-`99` outside-region residual. Published rounding also prevents exact reconciliation from being assumed. - Provenance: `census` for a census-controlled ERP estimate, and `assertion: observation` for the historical reference date. “Observation” does not mean final or error-free; the source's provisional status remains - explicit in the period and concept evidence. + explicit in the 2025 period and concept evidence. The 2023 evidence retains + its unmarked column header. - Concept relation: `source_label`, with Stats NZ evidence. No Axiom canonical equivalence, population calibration, uprating, interpolation, or target activation is claimed. @@ -58,6 +68,11 @@ footnote have guards. The tests compare every admitted value against the publisher workbook and every regional code/name against the independent publisher classification. +The 2023 selectors additionally guard Table 1!A3, A5 and B5, along with A4, +numeric B6, every selected row header, A29 and A34. Both dates use the same +source concept, REGC 2025 geographies, provenance class and assertion. No ratio +or growth rate is constructed. + Dimension labels follow the bundle's one-label-per-id rule. `geography` is "Geography" and `person.age_band` is "Age band", the labels the other packages that emit those ids already use. These two labels are Chronicle's wording. The @@ -76,8 +91,8 @@ keeps the package-completion checkbox unchecked. ```sh uv run chronicle validate-package stats-nz-subnational-population-estimates-2025 --year 2025 -uv run chronicle build-suite stats-nz-subnational-population-estimates-2025 --year 2025 --out /tmp/chronicle-nz-population-suite -uv run pytest -q tests/test_chronicle_nz_population.py tests/test_chronicle_artifacts.py tests/test_source_package_alias_drift.py +uv run --offline pytest -q tests/test_chronicle_nz_population.py -p no:cacheprovider --basetemp=.hub-scratch/pytest -o tmp_path_retention_policy=failed +python -c "import shutil; shutil.rmtree('.hub-scratch/pytest', ignore_errors=True)" ``` Source and consumer checks pass for this partial package. The concept-alignment diff --git a/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml b/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml index d984a9c9..24523fa2 100644 --- a/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml +++ b/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml @@ -1,10 +1,10 @@ -# Partial #176 source package: 85 directly published counts, not the missing +# Partial #176 source package: 102 directly published counts, not the missing # five-year-age x sex x region table. See README.md for scope and source access. -# The all-age national total is Table 1!D25, never the sum of the 16 regions. +# All-age national totals are Table 1!B25/D25, never sums of the 16 regions. # REGC codes/names come from the separately pinned Stats NZ REGC2025 attributes. schema_version: ledger.source_package.v1 package_id: stats-nz-subnational-population-estimates-2025 -label: 'Stats NZ 30 June 2025 provisional regional population: all ages and broad age groups (partial)' +label: 'Stats NZ October 2025 regional population release: 2023 all ages; 2025 provisional all ages and broad age groups (partial)' # Chronicle-owned labels (chronicle#261, #265). The age-band values are the # publisher's column headers at Table 3!D6:G6. The two dimension labels are # Chronicle's wording, not publisher text: the bundle allows one label per @@ -1082,3 +1082,269 @@ record_sets: column: G expected_column_header: "65+" rows: *broad_age_rows + +# Table 1's unmarked 2023 column is a separate reference period in this release. +# Inherit the same concept/geographies, but never the 2025 provisional status. +- <<: *population_set + record_set_id: stats_nz.cy2023.subnational_population.all_ages + record_set_spec_id: stats_nz.subnational_population.all_ages_2023.v1 + source_record_id_prefix: stats_nz.cy2023.subnational_population.all_ages + period: 2023 + period_coverage: + start_date: '2023-06-30' + end_date: '2023-06-30' + basis: calendar + source_period_label: At 30 June 2023 + notes: >- + Point-in-time estimated resident population in the release's 2023-base + series, not the 7 March 2023 census-night count; published 29 October 2025. + The numeric 2023 header has no P marker. Contents and Table 1 footnotes + do not explicitly label 2023 final or revised. + measures: + - <<: *population_measure + column: B + expected_column_header: 2023 + concept_evidence_notes: >- + Table 1 column B publishes all-age population counts at 30 June 2023, + using 1 January 2025 boundaries and the 2023 census base. The numeric + 2023 header has no P marker; Contents and Table 1 footnotes do not + explicitly label 2023 final or revised. ERP includes adjustments for net + census under/overcount, residents temporarily overseas on census night, + and births, deaths, and net migration between census night and the + estimate date. These are the release's 2023-base series, not the + 7 March 2023 census-night count. These counts include all sexes; there + is no sex-specific split in these worksheets. Only all ages are admitted + for 2023, not five-year or single-year ages. The New Zealand row includes + areas outside regions (for example Chatham Islands territory); it is + read directly, never constructed from regional counts. Published rounding + means component counts need not sum to totals. Source-only facts: no + uprating, interpolation, target activation, or calibration. + rows: + - value_id: regc_01 + label: Northland region + ordinal: 0 + row_number: 7 + expected_row_header_column: A + expected_row_header: Northland region + geography_id: '01' + geography_level: region + geography_name: Northland Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: &population_2023_guards + - column: A + row: 3 + expected_value: Regional council areas + label: regional population scope + - column: A + row: 4 + expected_value: At 30 June 2023–2025 + label: point-in-time release period + - column: A + row: 5 + expected_value: Regional council area(2) + label: regional row heading + - column: B + row: 5 + expected_value: At 30 June + label: point-in-time column heading + - column: A + row: 29 + expected_value: 2. Boundaries at 1 January 2025. + label: regional boundary vintage + - column: A + row: 34 + expected_value: P provisional + label: P symbol definition; 2023 header has no P + - value_id: regc_02 + label: Auckland region + ordinal: 1 + row_number: 8 + expected_row_header_column: A + expected_row_header: Auckland region + geography_id: '02' + geography_level: region + geography_name: Auckland + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_03 + label: Waikato region + ordinal: 2 + row_number: 9 + expected_row_header_column: A + expected_row_header: Waikato region + geography_id: '03' + geography_level: region + geography_name: Waikato Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_04 + label: Bay of Plenty region + ordinal: 3 + row_number: 10 + expected_row_header_column: A + expected_row_header: Bay of Plenty region + geography_id: '04' + geography_level: region + geography_name: Bay of Plenty Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_05 + label: Gisborne region + ordinal: 4 + row_number: 11 + expected_row_header_column: A + expected_row_header: Gisborne region + geography_id: '05' + geography_level: region + geography_name: Gisborne Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_06 + label: Hawke's Bay region + ordinal: 5 + row_number: 12 + expected_row_header_column: A + expected_row_header: Hawke's Bay region + geography_id: '06' + geography_level: region + geography_name: Hawke's Bay Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_07 + label: Taranaki region + ordinal: 6 + row_number: 13 + expected_row_header_column: A + expected_row_header: Taranaki region + geography_id: '07' + geography_level: region + geography_name: Taranaki Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_08 + label: Manawatū-Whanganui region + ordinal: 7 + row_number: 14 + expected_row_header_column: A + expected_row_header: Manawatū-Whanganui region + geography_id: 08 + geography_level: region + geography_name: Manawatū-Whanganui Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_09 + label: Wellington region + ordinal: 8 + row_number: 15 + expected_row_header_column: A + expected_row_header: Wellington region + geography_id: 09 + geography_level: region + geography_name: Wellington Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_16 + label: Tasman region + ordinal: 9 + row_number: 16 + expected_row_header_column: A + expected_row_header: Tasman region + geography_id: '16' + geography_level: region + geography_name: Tasman Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_17 + label: Nelson region + ordinal: 10 + row_number: 17 + expected_row_header_column: A + expected_row_header: Nelson region + geography_id: '17' + geography_level: region + geography_name: Nelson Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_18 + label: Marlborough region + ordinal: 11 + row_number: 18 + expected_row_header_column: A + expected_row_header: Marlborough region + geography_id: '18' + geography_level: region + geography_name: Marlborough Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_12 + label: West Coast region + ordinal: 12 + row_number: 19 + expected_row_header_column: A + expected_row_header: West Coast region + geography_id: '12' + geography_level: region + geography_name: West Coast Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_13 + label: Canterbury region + ordinal: 13 + row_number: 20 + expected_row_header_column: A + expected_row_header: Canterbury region + geography_id: '13' + geography_level: region + geography_name: Canterbury Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_14 + label: Otago region + ordinal: 14 + row_number: 21 + expected_row_header_column: A + expected_row_header: Otago region + geography_id: '14' + geography_level: region + geography_name: Otago Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: regc_15 + label: Southland region + ordinal: 15 + row_number: 22 + expected_row_header_column: A + expected_row_header: Southland region + geography_id: '15' + geography_level: region + geography_name: Southland Region + geography_vintage: regc_2025 + table_record_kind: total + guard_cells: *population_2023_guards + - value_id: nz + label: New Zealand + ordinal: 16 + row_number: 25 + expected_row_header_column: A + expected_row_header: New Zealand(3) + geography_id: NZ + geography_level: country + geography_name: New Zealand + geography_vintage: current + table_record_kind: total + guard_cells: *population_2023_guards diff --git a/tests/test_chronicle_bundle.py b/tests/test_chronicle_bundle.py index 77ff4fbf..359c7642 100644 --- a/tests/test_chronicle_bundle.py +++ b/tests/test_chronicle_bundle.py @@ -416,7 +416,8 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): # packages, and 91,982 net for the Stat-Xplore extracts (183,804 UC # area facts by Payment Indicator replace the 91,902 all-claims ones; # Move to UC adds 80). - "fact_count": 1579263, + # G8-d4 adds 17 CY2023 ERP cells to the existing Stats NZ package. + "fact_count": 1579280, # The same change adds 82,064 geography keys: 46,363 Scottish 2022 # output areas (UV404), 35,673 LSOA-level keys from the DESNZ LSOA # workbooks (Scotland's data zones already exist), the 26 Northern @@ -488,7 +489,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): if row_count == 1: assert row["aggregate_fact_key"].startswith("ledger.aggregate_fact.v2:") assert row["semantic_fact_key"].startswith("ledger.semantic_fact.v2:") - assert row_count == 1579263 + assert row_count == 1579280 assert provenance_classes <= { "administrative", "census", @@ -509,7 +510,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "jct-obbba-revenue-estimates-2025", "jct-tax-expenditures-2024", ] - assert coverage["fact_count"] == 1579263 + assert coverage["fact_count"] == 1579280 assert coverage["counts"]["by_source"] == { "bea": 446, "bfp_economic_outlook": 5, @@ -563,7 +564,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "statbel_fiscal_income_distribution": 14600, "statbel_population_structure": 36, "treasury": 139, - "stats_nz": 2015, + "stats_nz": 2032, "usda_snap": 852, "voa": 3001, "welshgov": 9567, @@ -619,7 +620,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "stats_nz:Subnational population estimates: At 30 June 2025 " "(provisional), Tables 1 and 3" ] - == 85 + == 102 ) assert ( table_counts[ @@ -1816,6 +1817,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): expected_period_counts["fiscal_year:2023"] += 1413 expected_period_counts["calendar_year:2023"] += 288 expected_period_counts["calendar_year:2023"] += 162 + expected_period_counts["calendar_year:2023"] += 17 # G8-d4 regional ERP. # chronicle#291: IRS SOI Historic Table 2 TY2023 state AGI-band facts. expected_period_counts["tax_year:2023"] += 1020 # The congressional-district file's Schedule A line 8b, 8c and 9 columns: @@ -1897,11 +1899,11 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): # (#321) + 3,522 IRD I1 (#318) + 510 IRD I2 (#210) + 103 MSD (#319) # + 84 MBIE (#320) + 112 Treasury Budget 2026 expenses (B7 part 2) # + 1,413 Stats NZ household net worth + 16 B5 national census - # + nine G8-d1 national weekly-rent facts = 6,131. - assert coverage["counts"]["by_geography"]["country:NZ"] == 6131 + # + nine G8-d1 national weekly-rent facts + one G8-d4 ERP fact = 6,132. + assert coverage["counts"]["by_geography"]["country:NZ"] == 6132 for regional_council in (*range(1, 10), *range(12, 19)): assert ( - coverage["counts"]["by_geography"][f"region:{regional_council:02d}"] == 30 + coverage["counts"]["by_geography"][f"region:{regional_council:02d}"] == 31 ) assert coverage["counts"]["by_geography"]["region:99"] == 25 assert coverage["counts"]["by_geography"]["local_authority:nz-ta-054"] == 57 @@ -1972,7 +1974,7 @@ def test_build_bundle_writes_merged_consumer_contract(tmp_path): "household": 676320, "institutional_sector": 6654, "pension_plan": 2, - "person": 521319, + "person": 521336, "return": 14952, "social_protection_scheme": 36, "tax_unit": 45268, diff --git a/tests/test_chronicle_nz_population.py b/tests/test_chronicle_nz_population.py index 77bfdd0c..b06dcaaa 100644 --- a/tests/test_chronicle_nz_population.py +++ b/tests/test_chronicle_nz_population.py @@ -1,17 +1,17 @@ -"""Source-only regression gates for the partial 30 June 2025 NZ population package.""" +"""Source-only gates for the 2023/2025 ERP in the pinned October 2025 release.""" from __future__ import annotations -from collections import Counter -from dataclasses import asdict, replace import hashlib import html -from io import BytesIO import json import os -from pathlib import Path import subprocess import sys +from collections import Counter +from dataclasses import asdict, replace +from io import BytesIO +from pathlib import Path import openpyxl import pytest @@ -160,8 +160,8 @@ def test_partial_package_counts_and_whole_sheet_preservation(population): report = validate_source_package(ALIAS, year=2025) assert report.valid assert not report.warnings - assert report.counts["record_set_count"] == 5 - assert report.counts["source_record_count"] == len(facts) == 85 + assert report.counts["record_set_count"] == 6 + assert report.counts["source_record_count"] == len(facts) == 102 assert "partial" in package.label.lower() assert package.artifact.sheets == ("Table 1", "Table 3") # Full used ranges, including unused years and source footnotes: 35x9 + 75x26. @@ -177,19 +177,24 @@ def test_partial_package_counts_and_whole_sheet_preservation(population): def test_every_fact_equals_one_published_cell_with_no_scaling(population, workbook): package, cells, facts = population actual = { - (fact.geography.id, fact.filters.get("person.age_band", "all")): fact + ( + fact.period.value, + fact.geography.id, + fact.filters.get("person.age_band", "all"), + ): fact for fact in facts } cell_keys = {build_source_cell_key(cell) for cell in cells} - assert len(actual) == 85 + assert len(actual) == 102 for geography_id, total_row, broad_age_row in REGIONS: - assert ( - actual[(geography_id, "all")].value - == workbook["Table 1"][f"D{total_row}"].value - ) + for year, column in ((2023, "B"), (2025, "D")): + assert ( + actual[(year, geography_id, "all")].value + == workbook["Table 1"][f"{column}{total_row}"].value + ) for age_band, (column, _lower, _upper) in AGE_BANDS.items(): assert ( - actual[(geography_id, age_band)].value + actual[(2025, geography_id, age_band)].value == workbook["Table 3"][f"{column}{broad_age_row}"].value ) for fact, spec in zip(facts, package.build_source_record_specs(2025), strict=True): @@ -206,6 +211,92 @@ def test_every_fact_equals_one_published_cell_with_no_scaling(population, workbo assert type(fact.value) is int +@pytest.mark.parametrize("geography_id,row_number,_age_row", REGIONS) +def test_2023_fact_preserves_unmarked_status_and_source_guards( + population, workbook, geography_id, row_number, _age_row +): + package, cells, facts = population + (fact,) = [ + fact + for fact in facts + if fact.period.value == 2023 and fact.geography.id == geography_id + ] + (comparison,) = [ + fact + for fact in facts + if fact.period.value == 2025 + and fact.geography.id == geography_id + and not fact.filters + ] + assert fact.geography == comparison.geography + # Evidence describes each reference year's column and status separately. + assert ( + replace( + fact.measure, + concept_evidence_notes=comparison.measure.concept_evidence_notes, + ) + == comparison.measure + ) + assert fact.entity == comparison.entity + assert fact.provenance_class == comparison.provenance_class == "census" + assert fact.assertion == comparison.assertion == "observation" + assert not fact.filters and not fact.constraints + assert fact.period_coverage.source_period_label == "At 30 June 2023" + assert "2023-base" in fact.period_coverage.notes + assert "7 March 2023 census-night count" in fact.period_coverage.notes + (spec,) = [ + spec + for spec in package.build_source_record_specs(2025) + if spec.source_record_id == fact.source_record_id + ] + selector = spec.selector + assert (selector.sheet_name, selector.address, selector.end_address) == ( + "Table 1", + f"B{row_number}", + None, + ) + assert selector.expected_row_header_address == f"A{row_number}" + assert selector.expected_row_header == workbook["Table 1"][f"A{row_number}"].value + assert selector.expected_column_header_address == "B6" + assert selector.expected_column_header == workbook["Table 1"]["B6"].value == 2023 + expected_guards = { + "A3": "Regional council areas", + "A4": "At 30 June 2023–2025", + "A5": "Regional council area(2)", + "B5": "At 30 June", + "A29": "2. Boundaries at 1 January 2025.", + "A34": "P provisional", + } + assert { + guard.address: guard.expected_value for guard in selector.guard_cells + } == expected_guards + lineage_addresses = { + cell.address + for cell in cells + if build_source_cell_key(cell) in fact.source_cell_keys + } + assert { + *expected_guards, + "B6", + f"B{row_number}", + } <= lineage_addresses + + +@pytest.mark.parametrize( + "address,expected", + [ + ("B7", 198_500), + ("B8", 1_755_200), + ("B15", 538_000), + ("B20", 676_600), + ("B22", 102_700), + ("B25", 5_200_000), + ], +) +def test_2023_sample_cells_retain_publisher_counts(workbook, address, expected): + assert workbook["Table 1"][address].value == expected + + def test_regc_codes_and_names_match_official_2025_classification(population): _package, _cells, facts = population content, _ = _artifact("regional_council_2025_codes") @@ -218,7 +309,7 @@ def test_regc_codes_and_names_match_official_2025_classification(population): assert len(codes) == 17 # Sixteen regions plus publisher code 99 (outside region). regional_facts = [fact for fact in facts if fact.geography.level == "region"] assert {fact.geography.id for fact in regional_facts} == codes.keys() - {"99"} - assert len(regional_facts) == 80 + assert len(regional_facts) == 96 assert all(fact.geography.vintage == "regc_2025" for fact in regional_facts) assert all( fact.geography.name == codes[fact.geography.id] for fact in regional_facts @@ -228,11 +319,20 @@ def test_regc_codes_and_names_match_official_2025_classification(population): ) # Workbook calls the same region "Auckland region". -def test_national_total_is_published_not_reconciled_from_regions(population, workbook): +@pytest.mark.parametrize( + "year,column,expected", [(2023, "B", 5_200_000), (2025, "D", 5_324_700)] +) +def test_national_total_is_published_not_reconciled_from_regions( + population, workbook, year, column, expected +): _package, _cells, facts = population - totals = {fact.geography.id: fact for fact in facts if not fact.filters} + totals = { + fact.geography.id: fact + for fact in facts + if not fact.filters and fact.period.value == year + } # Table 1 footnote 3 includes areas outside a region (for example Chatham). - assert totals["NZ"].value == workbook["Table 1"]["D25"].value == 5_324_700 + assert totals["NZ"].value == workbook["Table 1"][f"{column}25"].value == expected assert totals["NZ"].value != sum( fact.value for code, fact in totals.items() if code != "NZ" ) @@ -250,7 +350,7 @@ def test_each_fact_reads_its_own_cell_and_regions_never_feed_the_nation( value_cells = {} for fact, spec in zip(facts, specs, strict=True): # The first lineage key is the one value cell; the rest are its column - # header and guard cells, all of them text. + # header and guard cells. The 2023 column header is numeric. value_cell, *context_cells = ( cells_by_key[key] for key in fact.source_cell_keys ) @@ -259,13 +359,23 @@ def test_each_fact_reads_its_own_cell_and_regions_never_feed_the_nation( spec.selector.address, ) assert value_cell.raw_value == fact.value - assert all(type(cell.raw_value) is str for cell in context_cells) + for cell in context_cells: + if fact.period.value == 2023 and cell.address == "B6": + assert cell.raw_value == 2023 + else: + assert type(cell.raw_value) is str value_cells[fact.source_record_id] = fact.source_cell_keys[0] - assert len(set(value_cells.values())) == len(facts) == 85 + assert len(set(value_cells.values())) == len(facts) == 102 - for band, (sheet, column) in BAND_COLUMNS.items(): + selections = [(2023, "all", "Table 1", "B")] + [ + (2025, band, sheet, column) for band, (sheet, column) in BAND_COLUMNS.items() + ] + for year, band, sheet, column in selections: band_facts = [ - fact for fact in facts if fact.filters.get("person.age_band", "all") == band + fact + for fact in facts + if fact.period.value == year + and fact.filters.get("person.age_band", "all") == band ] regional = [fact for fact in band_facts if fact.geography.level == "region"] (national,) = [fact for fact in band_facts if fact.geography.id == "NZ"] @@ -295,11 +405,13 @@ def test_every_row_records_reference_date_and_regc_2025_boundaries(population): } == {"2. Boundaries at 1 January 2025."} assert len(footnotes) == 2 rows = consumer_fact_rows(facts) - assert len(rows) == 85 + assert len(rows) == 102 for row in rows: coverage = row["period_coverage"] - assert coverage["start_date"] == coverage["end_date"] == "2025-06-30" - assert row["period"] == {"type": "calendar_year", "value": 2025} + year = row["period"]["value"] + assert year in (2023, 2025) + assert coverage["start_date"] == coverage["end_date"] == f"{year}-06-30" + assert row["period"]["type"] == "calendar_year" # Each row's lineage carries its table's boundary footnote. assert len(footnotes & set(row["lineage"]["source_cell_keys"])) == 1 notes = row["concept_alignment"]["evidence_notes"] @@ -370,14 +482,15 @@ def test_requested_build_year_cannot_uprate_or_relabel_pinned_vintage( facts = package.build_facts(requested_year, cells=cells) assert facts == original assert package.artifact.artifact_year == 2025 + assert Counter(fact.period.value for fact in facts) == {2023: 17, 2025: 85} for fact in facts: assert fact.period.type == "calendar_year" - assert fact.period.value == 2025 - assert fact.period_coverage.start_date == "2025-06-30" - assert fact.period_coverage.end_date == "2025-06-30" + year = fact.period.value + assert fact.period_coverage.start_date == f"{year}-06-30" + assert fact.period_coverage.end_date == f"{year}-06-30" assert fact.period_coverage.basis == "calendar" - assert ( - fact.period_coverage.source_period_label == "At 30 June 2025 (provisional)" + assert fact.period_coverage.source_period_label == ( + "At 30 June 2023" if year == 2023 else "At 30 June 2025 (provisional)" ) assert fact.source.vintage == "stats_nz_subnational_population_2025_10_29" assert fact.assertion == "observation" @@ -387,6 +500,10 @@ def test_requested_build_year_cannot_uprate_or_relabel_pinned_vintage( @pytest.mark.parametrize( ("sheet", "address", "replacement"), [ + ("Table 1", "A3", "Territorial authority areas"), + ("Table 1", "A5", "Regional council area(3)"), + ("Table 1", "B5", "Year ended 30 June"), + ("Table 1", "B6", "2023 P"), ("Table 1", "D6", "2026 P"), ("Table 1", "A7", "Auckland region"), ("Table 1", "A4", "Year ended 30 June 2023–2025"), @@ -416,8 +533,8 @@ def test_consumer_rows_preserve_source_only_scope_and_lineage(population): _package, _cells, facts = population rows = consumer_fact_rows(facts) assert validate_consumer_fact_contract(facts).valid - assert len({row["aggregate_fact_key"] for row in rows}) == 85 - assert len({row["semantic_fact_key"] for row in rows}) == 85 + assert len({row["aggregate_fact_key"] for row in rows}) == 102 + assert len({row["semantic_fact_key"] for row in rows}) == 102 for row in rows: assert row["concept_alignment"]["relation"] == "source_label" notes = row["concept_alignment"]["evidence_notes"] @@ -455,10 +572,14 @@ def test_explicit_single_package_bundle_passes_all_source_gates(tmp_path): for line in (tmp_path / "bundle/consumer_facts.jsonl").read_text().splitlines() ] coverage = build_bundle_coverage(rows) - assert coverage["fact_count"] == 85 - assert coverage["counts"]["by_source"] == {"stats_nz": 85} - assert coverage["counts"]["by_period"] == {"calendar_year:2025": 85} - assert coverage["counts"]["by_geography"]["country:NZ"] == 5 + assert coverage["fact_count"] == 102 + assert coverage["counts"]["by_source"] == {"stats_nz": 102} + assert coverage["counts"]["by_period"] == { + "calendar_year:2023": 17, + "calendar_year:2025": 85, + } + assert set(coverage["counts"]["by_geography"].values()) == {6} + assert coverage["counts"]["by_geography"]["country:NZ"] == 6 assert len(coverage["counts"]["by_geography"]) == 17 assert not coverage["duplicates"]["aggregate_fact_keys"] assert not coverage["duplicates"]["semantic_fact_keys"] From f6d4c3976f3d4b5318cab42e35658138c7d1edb2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 22 Oct 2026 03:54:17 -0400 Subject: [PATCH 2/2] State the 2023 record set's provenance class and quote REGC 08/09 Review of #379 (GPT-6.1 Sol, head 8bcb66a8): - The new CY2023 record set inherited provenance_class from the YAML anchor. test_every_source_package_record_set_declares_provenance_class counts record_set_id and provenance_class lines and failed (14703 == 14702). Repeat `provenance_class: census` under the record_set_id, as the four sibling sets already do. Consumer rows are unchanged (the reviewer hashed all 102 rows with and without the line), so the bundle pins stand. - Quote geography_id '08' and '09' like every other REGC code in the file. - Restore the README's portable validation commands (build-suite and the three test files) in place of the lane's scratch-path commands. Co-Authored-By: Claude Opus 5.5 --- .../stats_nz/subnational_population_estimates_2025/README.md | 4 ++-- .../source_package.yaml | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/packages/stats_nz/subnational_population_estimates_2025/README.md b/packages/stats_nz/subnational_population_estimates_2025/README.md index cd150072..25ec9f06 100644 --- a/packages/stats_nz/subnational_population_estimates_2025/README.md +++ b/packages/stats_nz/subnational_population_estimates_2025/README.md @@ -91,8 +91,8 @@ keeps the package-completion checkbox unchecked. ```sh uv run chronicle validate-package stats-nz-subnational-population-estimates-2025 --year 2025 -uv run --offline pytest -q tests/test_chronicle_nz_population.py -p no:cacheprovider --basetemp=.hub-scratch/pytest -o tmp_path_retention_policy=failed -python -c "import shutil; shutil.rmtree('.hub-scratch/pytest', ignore_errors=True)" +uv run chronicle build-suite stats-nz-subnational-population-estimates-2025 --year 2025 --out /tmp/chronicle-nz-population-suite +uv run pytest -q tests/test_chronicle_nz_population.py tests/test_chronicle_artifacts.py tests/test_source_package_alias_drift.py ``` Source and consumer checks pass for this partial package. The concept-alignment diff --git a/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml b/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml index 24523fa2..09ab5f74 100644 --- a/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml +++ b/packages/stats_nz/subnational_population_estimates_2025/source_package.yaml @@ -1087,6 +1087,7 @@ record_sets: # Inherit the same concept/geographies, but never the 2025 provisional status. - <<: *population_set record_set_id: stats_nz.cy2023.subnational_population.all_ages + provenance_class: census record_set_spec_id: stats_nz.subnational_population.all_ages_2023.v1 source_record_id_prefix: stats_nz.cy2023.subnational_population.all_ages period: 2023 @@ -1234,7 +1235,7 @@ record_sets: row_number: 14 expected_row_header_column: A expected_row_header: Manawatū-Whanganui region - geography_id: 08 + geography_id: '08' geography_level: region geography_name: Manawatū-Whanganui Region geography_vintage: regc_2025 @@ -1246,7 +1247,7 @@ record_sets: row_number: 15 expected_row_header_column: A expected_row_header: Wellington region - geography_id: 09 + geography_id: '09' geography_level: region geography_name: Wellington Region geography_vintage: regc_2025