Refuse disclosure codes that match nothing, and key UK regions on GSS codes

changelog.d/625.added.md

@@ -0,0 +1 @@
+Added `UK_REGION_GSS_CODES` in `policyengine.outputs.uk_geography_disclosure`: the ONS GSS code of each UK region, under every spelling a household output may carry (GSS codes, the engine's region enum names and Microcosm's pseudo codes for the nations).

changelog.d/625.changed.md

@@ -0,0 +1 @@
+`compute_uk_geography_disclosure` now keys regions on ONS GSS codes (`E12000001` to `E12000009` for England's regions, `W92000004`, `S92000003` and `N92000002` for Wales, Scotland and Northern Ireland), whether the household output carries `region_code_oa` or only the stored `region`, so releases from national and local datasets publish the same codes. Microcosm's pseudo codes for the nations map to the country codes, `UNKNOWN` is unassigned, and any other region value raises. Region codes in `parent_of`, in `prior_public` and in region weights passed with `area_weights=` must be GSS codes. Region rows from national datasets used to carry the engine's enum names, and enum names in `parent_of` or `prior_public` were accepted; they now raise.

changelog.d/625.fixed.md

@@ -0,0 +1 @@
+The disclosure release check no longer passes in silence when area codes do not match. `Release.check()` now raises `DisclosureError` for a `parent_of` value that is not an area of the parent level, and for a `prior_public` cell at a level the release holds whose area code is not an area of that level: of the estimates for the cell's reform and year when the release holds them, and of any of the level's estimates otherwise. A prior table on older boundaries therefore raises even for a year the release does not hold, so filter those cells out first. `Release()` refuses a prior cell with an unknown outcome. Such codes used to leave the nesting and earlier-publication checks with nothing to check, so they returned no finding. Prior cells at levels the release does not hold are still accepted, but `compute_uk_geography_disclosure` raises for one at a level that is not a UK level, since that is a misspelling. The adapter also reads the stored `region` when `region_code_oa` gives no household a region, and raises a clear error when a requested level's column gives no household an area, instead of a `KeyError` from the check. The messages give counts, never codes or values.

docs/disclosure.md

@@ -117,11 +117,27 @@ are a reference: the release does not emit their rows, but it treats the rows
 that pass as public elsewhere and the withheld ones as secret, and checks
 them like its own.
 
+The check matches areas by their codes, and it raises on a code that matches
+nothing rather than leave areas unchecked. Each value of `parent_of` must be
+an area code of the parent level's estimates. Each cell of `prior_public` at
+a level the release holds must name an area of that level: of the estimates
+for the cell's reform and year when the release holds them, and of any of the
+level's estimates otherwise. So a prior table on older boundaries raises even
+for a year the release does not hold; filter those cells out first. Every
+cell needs one of the six outcomes. Cells at other levels belong to other
+publications: the release accepts them, but they take no part in the check.
+A prior cell counts only when its level, area code, reform label and year are
+the release's.
+
 `compute_uk_geography_disclosure` always checks constituencies and local
 authorities against their region, and regions against the nation, also when
 you publish one level only. A level given as a weight matrix needs
-`parent_of=` with the region of each area; without it the function raises,
-because a check against the nation alone nearly always passes.
+`parent_of=` with the GSS code of each area's region (see
+[UK region codes](#uk-region-codes)); without it the function raises,
+because a check against the nation alone nearly always passes. The function
+also raises for a prior cell at a level that is not a UK level (`national`,
+`region`, `constituency`, `local_authority`, or a level given in
+`area_weights=`): a release would accept a misspelt level and check nothing.
 
 The check reports; it does not withhold cells by itself. A blocking finding
 names the group and suggests a cell. Withhold it, then check again. When a
@@ -152,6 +168,43 @@ Two findings can be accepted after review, with `acknowledge=[...]`:
 - `noise_reference_public`: an earlier table shows the same cells under
   another noise setting.
 
+## UK region codes
+
+`compute_uk_geography_disclosure` keys regions on their ONS GSS codes,
+whichever column the household output carries. It reads `region_code_oa`
+when the output has it and the column gives some household a region, as
+Microcosm's local files do, and the stored `region` otherwise, as on the
+national datasets. Both give the same codes:
+
+| Region | Stored `region` | GSS code |
+|---|---|---|
+| North East | `NORTH_EAST` | `E12000001` |
+| North West | `NORTH_WEST` | `E12000002` |
+| Yorkshire and the Humber | `YORKSHIRE` | `E12000003` |
+| East Midlands | `EAST_MIDLANDS` | `E12000004` |
+| West Midlands | `WEST_MIDLANDS` | `E12000005` |
+| East of England | `EAST_OF_ENGLAND` | `E12000006` |
+| London | `LONDON` | `E12000007` |
+| South East | `SOUTH_EAST` | `E12000008` |
+| South West | `SOUTH_WEST` | `E12000009` |
+| Wales | `WALES` | `W92000004` |
+| Scotland | `SCOTLAND` | `S92000003` |
+| Northern Ireland | `NORTHERN_IRELAND` | `N92000002` |
+
+Wales, Scotland and Northern Ireland are one region each, under their country
+code. Microcosm writes the pseudo codes `W99999999`, `S99999999` and
+`N99999999` for them in `region_code_oa`, and the adapter reads those as the
+country codes. A household whose `region` is `UNKNOWN` counts in the nation
+but in no region. Any other value raises. The table is `UK_REGION_GSS_CODES`
+in `policyengine.outputs.uk_geography_disclosure`.
+
+Use the GSS codes wherever you name a region: in `parent_of`, in the region
+cells of `prior_public`, and in region weights passed with `area_weights=`,
+which the function uses as given. Names such as `London` or `Eastern` are not
+codes; map them first. Then a regional release from the national dataset and
+a seat release from a local file publish the same codes, and the regional
+table can be passed as `prior_public` to the seat release.
+
 ## Noise
 
 Noise is off by default. With `DisclosureSettings(noise=True)`, each survey

src/policyengine/disclosure/release.py

@@ -27,6 +27,7 @@
     FAMILY_LOSING,
     NOISE_REFERENCE_PUBLIC,
     OUTCOME_FAMILY,
+    OUTCOMES,
     PRIVATE_SECONDARY,
     REASON_DISCLOSURE_CONTROL,
     REASON_THIN_SUPPORT,
@@ -71,6 +72,17 @@ def __init__(
         if unknown:
             raise ValueError(f"These findings cannot be acknowledged: {unknown}")
         self._prior = tuple(prior_public)
+        for prior in self._prior:
+            # A cell with an unknown outcome matches no cell of any release.
+            unknown_outcomes = sum(
+                outcome not in OUTCOMES for *_, outcome in prior.cells
+            )
+            if unknown_outcomes:
+                raise DisclosureError(
+                    f"Cells of prior publication {prior.name!r} whose outcome is "
+                    f"unknown: {unknown_outcomes}. Outcomes are "
+                    f"{', '.join(OUTCOMES)}."
+                )
         self._estimates: dict[_Key, AreaEstimates] = {}
         self._published: dict[_Key, bool] = {}
         self._nestings: list[tuple[str, str, Optional[dict[str, str]]]] = []
@@ -107,6 +119,8 @@ def add_nesting(
 
         ``parent_of`` (child area code to parent area code) is needed only when
         the child level is a weight matrix and the parent has several areas.
+        Its parent codes must be area codes of the parent estimates: the check
+        raises on one that is not.
         """
         mapping = dict(parent_of) if parent_of is not None else None
         self._nestings.append((child_level, parent_level, mapping))
@@ -144,6 +158,7 @@ def findings(self) -> Optional[list[Finding]]:
 
     def check(self) -> list[Finding]:
         """Report what a reader could recover from the rows as they stand."""
+        self._require_prior_areas()
         tiers = {
             (key, family): self._family_tiers(key, family)
             for key in self._estimates
@@ -266,6 +281,44 @@ def _effective_rows(self, key: _Key) -> Iterable[tuple[dict, str, Optional[str]]
             else:
                 yield record, record["tier"], reason
 
+    def _require_prior_areas(self) -> None:
+        """Refuse prior cells that name no area of a level the release holds.
+
+        Prior cells match by code. A cell that matched nothing would be
+        ignored, so a cell an earlier publication showed would still count as
+        hidden. A cell at a level the release holds must name an area of the
+        estimates for its reform and year, or, when the release holds none,
+        of any of the level's estimates. Cells at other levels belong to other
+        publications.
+        """
+        areas: dict[str, set[str]] = {}
+        for (level, _, _), estimates in self._estimates.items():
+            areas.setdefault(level, set()).update(estimates.areas.area_codes)
+        for prior in self._prior:
+            unmatched = [
+                level
+                for level, area, reform, year, _ in prior.cells
+                if level in areas
+                and area not in self._prior_areas(level, reform, year, areas)
+            ]
+            if unmatched:
+                levels = " or ".join(sorted(set(unmatched)))
+                raise DisclosureError(
+                    f"Cells of prior publication {prior.name!r} that name no "
+                    f"{levels} area of the release: {len(unmatched)}. Prior "
+                    "cells match by code: use the area codes of the release's "
+                    "estimates."
+                )
+
+    def _prior_areas(
+        self, level: str, reform: str, year: int, areas: dict[str, set[str]]
+    ) -> set[str]:
+        """The areas a prior cell at a held level may name."""
+        estimates = self._estimates.get((level, reform, int(year)))
+        if estimates is None:
+            return areas[level]
+        return set(estimates.areas.area_codes)
+
     def _known_cells(self) -> set[tuple[str, str, str, int, str]]:
         """Family cells an earlier publication already made public."""
         known = set()
@@ -339,6 +392,15 @@ def _groups(
                 raise DisclosureError(
                     f"parent_of lacks {len(missing)} {child.level} areas."
                 )
+            # Groups match by code. A code that is no parent area would leave
+            # its child areas in no group, and the check would pass unseen.
+            unknown = {parent_of[c] for c in child.areas.area_codes} - set(parents)
+            if unknown:
+                raise DisclosureError(
+                    f"Codes in parent_of that are not {parent.level} areas: "
+                    f"{len(unknown)}. Map each {child.level} area to an area "
+                    f"code of the {parent.level} estimates."
+                )
             return {
                 code: (
                     [c for c in child.areas.area_codes if parent_of[c] == code],

src/policyengine/disclosure/schema.py

@@ -252,7 +252,11 @@ class Finding(BaseModel):
 
 
 class PriorPublic(BaseModel):
-    """Cells an earlier publication already made public."""
+    """Cells an earlier publication already made public.
+
+    Each cell is ``(level, area_code, reform, year, outcome)``. A release
+    matches cells by code, so they must use the area codes of its estimates.
+    """
 
     model_config = ConfigDict(frozen=True, extra="forbid")
 

src/policyengine/outputs/uk_geography_disclosure.py

@@ -7,6 +7,7 @@
 """
 
 from collections.abc import Iterable, Mapping, Sequence
+from enum import Enum
 from pathlib import Path
 from typing import TYPE_CHECKING, Optional, Union
 
@@ -16,6 +17,7 @@
 
 from policyengine.core import Output
 from policyengine.disclosure import (
+    UNASSIGNED,
     AreaWeights,
     DisclosureSettings,
     Finding,
@@ -39,6 +41,84 @@
     "local_authority": ("la_code_oa",),
 }
 UK_SOURCE_HOUSEHOLD_ID_COLUMN = "source_household_id"
+# The ONS GSS code of each region, by every spelling a household output may
+# carry. Regions are always keyed on these codes, so a release from a national
+# dataset and one from a local file publish the same codes, and publishers use
+# them in parent_of and prior_public. England has nine regions; Wales, Scotland
+# and Northern Ireland are one region each, under their country code.
+UK_REGION_GSS_CODES: dict[str, str] = {
+    # GSS codes, as Microcosm writes them in region_code_oa.
+    "E12000001": "E12000001",
+    "E12000002": "E12000002",
+    "E12000003": "E12000003",
+    "E12000004": "E12000004",
+    "E12000005": "E12000005",
+    "E12000006": "E12000006",
+    "E12000007": "E12000007",
+    "E12000008": "E12000008",
+    "E12000009": "E12000009",
+    "W92000004": "W92000004",
+    "S92000003": "S92000003",
+    "N92000002": "N92000002",
+    # The pseudo codes Microcosm writes in region_code_oa for the nations.
+    "W99999999": "W92000004",
+    "S99999999": "S92000003",
+    "N99999999": "N92000002",
+    # The names of the engine's region enum, as the stored region holds them.
+    "NORTH_EAST": "E12000001",
+    "NORTH_WEST": "E12000002",
+    "YORKSHIRE": "E12000003",
+    "EAST_MIDLANDS": "E12000004",
+    "WEST_MIDLANDS": "E12000005",
+    "EAST_OF_ENGLAND": "E12000006",
+    "LONDON": "E12000007",
+    "SOUTH_EAST": "E12000008",
+    "SOUTH_WEST": "E12000009",
+    "WALES": "W92000004",
+    "SCOTLAND": "S92000003",
+    "NORTHERN_IRELAND": "N92000002",
+    "UNKNOWN": UNASSIGNED,
+}
+
+
+def _uk_region_gss_codes(values: object, column: str) -> np.ndarray:
+    """The GSS code of each region value, blank where it is unassigned.
+
+    Values may be strings, bytes or members of the engine's region enum. A
+    missing value is unassigned. Any value outside the table raises: a region
+    under another spelling would match no code that a publisher passes.
+    """
+    index, spellings = pd.factorize(pd.Series(values, dtype=object))
+    codes: list[Optional[str]] = []
+    unknown = 0
+    for spelling in spellings:
+        if isinstance(spelling, Enum):
+            spelling = spelling.name
+        if isinstance(spelling, bytes):
+            spelling = spelling.decode("utf-8")
+        spelling = str(spelling).strip()
+        code = UK_REGION_GSS_CODES.get(spelling) if spelling else UNASSIGNED
+        if code is None:
+            unknown += 1
+        codes.append(code)
+    if unknown:
+        raise ValueError(
+            f"Values in {column} that name no UK region: {unknown}. A region is "
+            "a GSS code, a name of the engine's region enum, or a Microcosm "
+            "pseudo code for a nation; see UK_REGION_GSS_CODES."
+        )
+    # Factorize marks a missing value with -1, which picks the last entry.
+    return np.asarray(np.array([*codes, UNASSIGNED], dtype=object)[index])
+
+
+def _assigns_an_area(codes: np.ndarray) -> bool:
+    """Whether any household has an area code, rather than a blank."""
+    values = pd.Series(codes, dtype=object)
+    for code in values[values.notna()].unique():
+        text = code.decode("utf-8") if isinstance(code, bytes) else str(code)
+        if text.strip() != UNASSIGNED:
+            return True
+    return False
 
 
 class UKGeographyDisclosure(Output):
@@ -47,6 +127,7 @@ class UKGeographyDisclosure(Output):
     Every number passes the disclosure rules of
     :class:`~policyengine.disclosure.DisclosureSettings` and carries a 90%
     interval. ``public`` stays empty while a blocking finding remains.
+    Regions are keyed on their ONS GSS codes; see ``UK_REGION_GSS_CODES``.
     """
 
     model_config = ConfigDict(arbitrary_types_allowed=True)
@@ -124,6 +205,21 @@ def run(
         weights = baseline_hh["household_weight"].to_numpy(dtype=float)
         custom = dict(area_weights or {})
         parents = dict(parent_of or {})
+        # A release ignores prior cells at levels it does not hold, since they
+        # may belong to other publications. Every UK level is known here, so a
+        # cell at any other level is a misspelling that would match nothing.
+        known = {UK_NATIONAL_LEVEL, *UK_DISCLOSURE_LEVEL_COLUMNS, *custom}
+        stray = [
+            cell[0]
+            for prior in prior_public
+            for cell in prior.cells
+            if cell[0] not in known
+        ]
+        if stray:
+            raise ValueError(
+                f"Prior cells at levels that are not UK levels: {len(stray)}, "
+                f"at {sorted(set(stray))}. The levels are {sorted(known)}."
+            )
 
         def areas_for(level: str) -> Optional[AreaWeights]:
             if level in custom:
@@ -135,8 +231,15 @@ def areas_for(level: str) -> Optional[AreaWeights]:
             if columns is None:
                 raise ValueError(f"Unknown level {level!r}")
             for column in columns:
-                if column in baseline_hh.columns:
-                    return AreaWeights.from_labels(baseline_hh[column], weights)
+                if column not in baseline_hh.columns:
+                    continue
+                codes = baseline_hh[column].to_numpy(dtype=object)
+                if level == UK_REGION_LEVEL:
+                    codes = _uk_region_gss_codes(codes, column)
+                # A column that gives no household an area, such as an empty
+                # region_code_oa, says nothing: read the next one.
+                if _assigns_an_area(codes):
+                    return AreaWeights.from_labels(codes, weights)
             return None
 
         settings = self.settings or DisclosureSettings()
@@ -150,8 +253,8 @@ def areas_for(level: str) -> Optional[AreaWeights]:
                 if level in self.levels:
                     columns = UK_DISCLOSURE_LEVEL_COLUMNS[level]
                     raise ValueError(
-                        f"The household output has none of {list(columns)} for "
-                        f"level {level!r}"
+                        f"The household output gives no household an area in "
+                        f"any of {list(columns)} for level {level!r}"
                     )
                 continue
             estimated[level] = areas
@@ -242,11 +345,13 @@ def compute_uk_geography_disclosure(
             the dataset has no ``source_household_id`` column.
         area_weights: Weights for a level, in place of the level's area code
             column, for example an area x household weight matrix.
-        parent_of: For a level given as a weight matrix, the region code of
-            each of its areas. Required for such a level when the household
-            output carries regions.
+        parent_of: For a level given as a weight matrix, the GSS code of the
+            region of each of its areas (see ``UK_REGION_GSS_CODES``).
+            Required for such a level when the household output carries
+            regions.
         acknowledge: Findings the caller accepts after review.
         prior_public: Earlier publications that count as already public.
+            Their region cells use GSS codes too.
         audit_dir: Private directory, outside any repository, for the exact
             values and counts.