diff --git a/changelog.d/us-labelled-restated-age-band-filters.added.md b/changelog.d/us-labelled-restated-age-band-filters.added.md new file mode 100644 index 0000000000..7cc1f15b22 --- /dev/null +++ b/changelog.d/us-labelled-restated-age-band-filters.added.md @@ -0,0 +1 @@ +The US fiscal materializer now accepts `ledger_filter_age_lower_bound` / `ledger_filter_age_upper_bound` — the Ledger stamp of a fact's `age >=` / `age <` rows — beside the compiled `age_lower_bound` / `age_upper_bound`, but only where the two edges are equal and the spec's materializer (`population_age`, or the age-banded `policyengine_variable` branch the SSA SSI by-age counts use) actually slices on that band. On the pinned Chronicle feed this clears the 939 Census population-by-age and SSA SSI age-band targets the guard refused over the whole compiled registry. A restated edge that disagrees, has no compiled counterpart, sits on a spec whose materializer applies no age band, or comes from a fact with `age` as a dimension is refused and names both values; so is any restated edge the compile does not attest was stamped from a `>=` / `<` constraint row (the new `age_bound_stamp_source` metadata on every compiled age-band spec reads `constraint_rows` unless the fact has a dimension named `age` or `age` plus a bound suffix (`age_upper_bound`, …), whose operator-less value the Ledger stamp can write under the same key); an exact-age restatement is refused; and `<=` / `>` rows (stamped `_upper_bound_inclusive` / `_lower_bound_exclusive`) stay refused by bare key, since the compiled band drops the operator. diff --git a/experiments/us-labelled-filter-support/REPORT.md b/experiments/us-labelled-filter-support/REPORT.md index 4ff82371d4..114d56cfce 100644 --- a/experiments/us-labelled-filter-support/REPORT.md +++ b/experiments/us-labelled-filter-support/REPORT.md @@ -596,3 +596,87 @@ $ .venv/bin/python tools/ci_test_groups.py --verify verification=ok, exit 0 $ ruff check / ruff format --check on the four changed .py files All checks passed! / 4 files already formatted ``` + +## 9. 2026-09-22: the age-band follow-up (Max's ruling 2) + +Section 7's "still open" item is closed on branch +`us-restated-age-band-filters`, stacked on this branch at `01dc806f4`. It +adds `age` to `RESTATED_LEDGER_FILTER_CONCEPTS` under a third rule, +`age_band`, in `tools/build_us_fiscal_refresh_release.py`. + +**What the materializer applies, read at `01dc806f4`.** Both age paths in +`_materialize_target_frame` read `age_lower_bound` / `age_upper_bound` +through `_as_bound` into `lower <= age < upper`: `population_age` in +`_population_age_household_values`, and the `policyengine_variable` branch +entered whenever either key is present (the SSA SSI by-age counts). The +compiled bounds come from `us_runtime.fiscal_targets._age_bounds`, which +takes the value of any `>`/`>=` row as the lower edge and any `<`/`<=` row as +the upper edge, **dropping the operator**. The restated key keeps it: +`ledger_targets._constraint_bound_filters` stamps `age_lower_bound` only for +`>=` and `age_upper_bound` only for `<`, and `>`/`<=` as +`_lower_bound_exclusive` / `_upper_bound_inclusive`. + +**The rule.** A restated `age_{lower,upper}_bound` is accepted only when it +parses to the same number as the compiled edge on its own side, the spec's +materializer is `population_age` or `policyengine_variable`, and the compile +attests that the key was stamped from a `>=` / `<` constraint row +(`age_bound_stamp_source=constraint_rows`; see "The operator is attested at +compile" below). Refused, naming the values: a disagreeing edge, no compiled +counterpart, an exact-age key (`ledger_filter_age=`), any other +materializer, a spec carrying `ledger_filter_age` at all (age is then a +dimension, so `_constraint_bound_filters` stamped none of the age rows and the +bound came from the operator-less dimension stamp), and any bound without that +attestation. `_exclusive` / `_inclusive` keys +are not restated concepts and keep the bare-key refusal whatever their value. +A test that runs real facts through `_dynamic_us_fiscal_target_references` +and `compile_ledger_target_references` shows why that matters: an +`age <= 9` row compiles to `age_upper_bound=9`, which the materializer reads +as `age < 9`. The AGI and age rules share one edge comparison, +`_restated_band_edge_refusal`. + +**The pinned feeds, measured.** Both exports carry 1,493 facts with an age +constraint. Every row is `role: filter`, an integer value, `>=` or `<`, at +most one per side, and no fact has an age-like dimension key. Counted by a +throwaway scan of the two JSONL files, not committed. + +**Counts.** From `age_band_rule_receipt.json`, written by the committed +`measure_age_band_rule.py` (receipt `script_sha256` `db5dcb68…`). It ran at +`622486abb` (the operator-attestation commit) with a clean tree, took +14.1 s and 15.6 s to compile, and peaked at 2.9 GB RSS. The whole compiled registry +(`compile_us_fiscal_target_registry(..., age_targets=True)`) has 32,866 +targets on each feed: + +| Arm | Refused targets | Refusal entries | SOI silent skips | +|---|---:|---:|---:| +| `rule` (this follow-up) | **0** | 0 | 0 | +| `without_age_band` (#969 alone) | **939** | 1,825 | 0 | +| `reverted` (`main`'s behaviour) | 3,208 | 5,116 | 2,269 | + +The two feeds (`b8543739…` and `4d1dba8c…`) give identical counts. The 939 +are the targets carrying a restated age key: 936 `census_population` / +`population_age` / `population_age` and 3 `ssa` / +`ssa_ssi_age_band_recipients` / `policyengine_variable`. The only +`ledger_filter_age*` keys present are `_lower_bound` (939) and +`_upper_bound` (886). The `without_age_band` and `reverted` rows equal the +section 8 receipt's `rule` and `reverted` rows. + +### The operator is attested at compile + +A restated `ledger_filter_age_{lower,upper}_bound` key carries the operator of +the mask only when `_constraint_bound_filters` stamped it from a `>=` / `<` +constraint row. `_ledger_metadata` stamps the fact's dimensions first and the +constraint edges only through `setdefault`, so a dimension named `age`, or +`age` plus a bound suffix (`age_upper_bound`, …), can put an operator-less +value under the same key; and `age < 10` and `age <= 10` differ by everyone +aged exactly ten. The key name alone therefore does not settle the operator. + +The compile now settles it. `ledger_targets.constraint_bound_shadowing_dimensions` +names any such dimension on a fact, and `fiscal_targets` records the answer on +every compiled `population_age` and SSA SSI age-band spec as +`age_bound_stamp_source`: `constraint_rows` when no dimension could have +shadowed the age rows' stamp, otherwise `dimensions:`. The builder's +restated age-band rule accepts a restatement only when that key reads +`constraint_rows`; anything else is refused with the reason. Measured on both +pinned feeds (receipt above, `by_age_bound_stamp_source`): all 939 targets +carrying a restated age key read `constraint_rows`, so the attestation refuses +none of them today, and the `rule` row stays at 0. diff --git a/experiments/us-labelled-filter-support/age_band_rule_receipt.json b/experiments/us-labelled-filter-support/age_band_rule_receipt.json new file mode 100644 index 0000000000..62b530bc0a --- /dev/null +++ b/experiments/us-labelled-filter-support/age_band_rule_receipt.json @@ -0,0 +1,123 @@ +{ + "head": "622486abbe65c06ea348c5d4885193c073721344", + "worktree_status": [], + "script_sha256": "db5dcb680ac1af0c55dad996f0f1abe33b6ef78e5e9b19cb02f0b7551ec9a143", + "target_period": 2024, + "max_rss_bytes": 2875359232, + "results": [ + { + "feed": "/Users/maxghenis/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl", + "feed_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801", + "compile_seconds": 14.1, + "targets": 32866, + "age_key_census": { + "targets_carrying_restated_age_key": 939, + "by_family_role_materializer": { + "census_population|population_age|population_age": 936, + "ssa|ssa_ssi_age_band_recipients|policyengine_variable": 3 + }, + "by_age_bound_stamp_source": { + "constraint_rows": 939 + }, + "ledger_filter_age_keys": { + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + } + }, + "rule": { + "refused_targets": 0, + "refusal_entries": 0, + "refusal_entries_by_key": {}, + "refused_by_family_role": {}, + "soi_silently_skipped_targets": 0 + }, + "without_age_band": { + "refused_targets": 939, + "refusal_entries": 1825, + "refusal_entries_by_key": { + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + }, + "refused_by_family_role": { + "census_population|population_age": 936, + "ssa|ssa_ssi_age_band_recipients": 3 + }, + "soi_silently_skipped_targets": 0 + }, + "reverted": { + "refused_targets": 3208, + "refusal_entries": 5116, + "refusal_entries_by_key": { + "ledger_filter_us.tax.earned_income_credit_qualifying_children_lower_bound": 1137, + "ledger_filter_us:statutes/26/62#adjusted_gross_income_lower_bound": 1080, + "ledger_filter_us:statutes/26/62#adjusted_gross_income_upper_bound": 1074, + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + }, + "refused_by_family_role": { + "irs_soi|soi_fiscal_distribution": 2269, + "census_population|population_age": 936, + "ssa|ssa_ssi_age_band_recipients": 3 + }, + "soi_silently_skipped_targets": 2269 + } + }, + { + "feed": "/Users/maxghenis/PolicyEngine/_buildh-runtime/inputs/chronicle_us_b571381/artifact/consumer_facts.jsonl", + "feed_sha256": "4d1dba8c1b6274877bf184fa6de5d99b13fc61f34709ccab1487db2b5c64a79f", + "compile_seconds": 15.6, + "targets": 32866, + "age_key_census": { + "targets_carrying_restated_age_key": 939, + "by_family_role_materializer": { + "census_population|population_age|population_age": 936, + "ssa|ssa_ssi_age_band_recipients|policyengine_variable": 3 + }, + "by_age_bound_stamp_source": { + "constraint_rows": 939 + }, + "ledger_filter_age_keys": { + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + } + }, + "rule": { + "refused_targets": 0, + "refusal_entries": 0, + "refusal_entries_by_key": {}, + "refused_by_family_role": {}, + "soi_silently_skipped_targets": 0 + }, + "without_age_band": { + "refused_targets": 939, + "refusal_entries": 1825, + "refusal_entries_by_key": { + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + }, + "refused_by_family_role": { + "census_population|population_age": 936, + "ssa|ssa_ssi_age_band_recipients": 3 + }, + "soi_silently_skipped_targets": 0 + }, + "reverted": { + "refused_targets": 3208, + "refusal_entries": 5116, + "refusal_entries_by_key": { + "ledger_filter_us.tax.earned_income_credit_qualifying_children_lower_bound": 1137, + "ledger_filter_us:statutes/26/62#adjusted_gross_income_lower_bound": 1080, + "ledger_filter_us:statutes/26/62#adjusted_gross_income_upper_bound": 1074, + "ledger_filter_age_lower_bound": 939, + "ledger_filter_age_upper_bound": 886 + }, + "refused_by_family_role": { + "irs_soi|soi_fiscal_distribution": 2269, + "census_population|population_age": 936, + "ssa|ssa_ssi_age_band_recipients": 3 + }, + "soi_silently_skipped_targets": 2269 + } + } + ] +} diff --git a/experiments/us-labelled-filter-support/measure_age_band_rule.py b/experiments/us-labelled-filter-support/measure_age_band_rule.py new file mode 100644 index 0000000000..b4941d8906 --- /dev/null +++ b/experiments/us-labelled-filter-support/measure_age_band_rule.py @@ -0,0 +1,219 @@ +"""Count the restated-filter guard's refusals with and without the age-band rule. + +Sibling of ``measure_restated_filter_arms.py`` for the follow-up that adds +``age`` to ``RESTATED_LEDGER_FILTER_CONCEPTS`` under the ``age_band`` rule. +For each Chronicle consumer-facts feed given, compiles the whole US fiscal +target registry the way that script does +(``compile_us_fiscal_target_registry(..., age_targets=True)`` with the +default congressional-district vintage crosswalk) and surveys it under three +arms: + +* ``rule``: the tree as it stands (AGI, qualifying-child and age-band rules); +* ``without_age_band``: ``age`` removed from + ``RESTATED_LEDGER_FILTER_CONCEPTS``, which is #969's rule + (``us-labelled-filter-support``) and nothing else; +* ``reverted``: ``_restated_ledger_filter_refusal`` replaced by a function + that returns the bare key for every otherwise-unsupported + ``ledger_filter_*`` key, the pre-rule behaviour of both call sites and so + ``main``'s. + +Per arm it reports refused targets, refusal entries by key, refused targets by +(family, ``target_role``), and ``irs_soi`` targets the SOI loop would skip +silently. It also counts, independent of arm, the targets carrying a restated +age key (``ledger_filter_age_lower_bound`` / ``_upper_bound``) by +(family, ``target_role``, ``materializer``), those targets by the compile's +``age_bound_stamp_source`` attestation (``constraint_rows`` unless a +dimension could have shadowed the age rows' stamp), and every +``ledger_filter_age*`` key seen, so an operator-changed key (``_inclusive`` / +``_exclusive``) or an ``age`` dimension would show up by name. Output is +aggregate target-spec counts only; no microdata is read. It regenerates +``age_band_rule_receipt.json``:: + + uv run python experiments/us-labelled-filter-support/\\ + measure_age_band_rule.py [...] > receipt.json + +Needs no engine. The receipt records the SHA-256 of this script and of each +feed, the ``git rev-parse HEAD`` and ``git status --porcelain`` of the tree +it ran in, the compile time per feed and the process's peak RSS +(``max_rss_bytes``, bytes on macOS). +""" + +from __future__ import annotations + +import collections +import hashlib +import importlib.util +import json +import resource +import subprocess +import sys +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +RESTATED_AGE_KEYS = ("ledger_filter_age_lower_bound", "ledger_filter_age_upper_bound") + + +def _load_tool(name: str): + spec = importlib.util.spec_from_file_location(name, ROOT / "tools" / f"{name}.py") + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def _sha256(path: str) -> str: + digest = hashlib.sha256() + with open(path, "rb") as handle: + for block in iter(lambda: handle.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +def _survey(builder, specs: list) -> dict[str, object]: + unsupported = builder._unsupported_ledger_filter_metadata(specs) + entries: collections.Counter[str] = collections.Counter() + for spec_entries in unsupported.values(): + for entry in spec_entries: + entries[entry.split("=", 1)[0]] += 1 + by_name = {spec.name: spec for spec in specs} + by_role: collections.Counter[str] = collections.Counter() + for name in unsupported: + spec = by_name[name] + by_role[f"{spec.family}|{spec.metadata.get('target_role', '')}"] += 1 + skipped = sum( + 1 + for spec in specs + if getattr(spec, "family", "") == "irs_soi" + and builder._unsupported_soi_ledger_filters(spec.metadata) + ) + return { + "refused_targets": len(unsupported), + "refusal_entries": sum(entries.values()), + "refusal_entries_by_key": dict(entries.most_common()), + "refused_by_family_role": dict(by_role.most_common()), + "soi_silently_skipped_targets": skipped, + } + + +def _age_key_census(specs: list, stamp_source_key: str) -> dict[str, object]: + carrying: collections.Counter[str] = collections.Counter() + stamp_sources: collections.Counter[str] = collections.Counter() + keys: collections.Counter[str] = collections.Counter() + for spec in specs: + for key in spec.metadata: + if str(key).startswith("ledger_filter_age"): + keys[str(key)] += 1 + if any(key in spec.metadata for key in RESTATED_AGE_KEYS): + carrying[ + f"{spec.family}|{spec.metadata.get('target_role', '')}" + f"|{spec.metadata.get('materializer', '')}" + ] += 1 + stamp_sources[str(spec.metadata.get(stamp_source_key))] += 1 + return { + "targets_carrying_restated_age_key": sum(carrying.values()), + "by_family_role_materializer": dict(carrying.most_common()), + "by_age_bound_stamp_source": dict(stamp_sources.most_common()), + "ledger_filter_age_keys": dict(keys.most_common()), + } + + +def main(argv: list[str]) -> int: + if not argv: + print(__doc__) + return 2 + + builder = _load_tool("build_us_fiscal_refresh_release") + + import microcosm.build.ledger_targets as ledger_targets + from microcosm.build.ledger_artifact import load_ledger_consumer_artifact + from microcosm.build.us_runtime import ( + default_congressional_district_vintage_crosswalk_path, + load_congressional_district_vintage_crosswalk, + ) + from microcosm.build.us_runtime.fiscal_targets import ( + AGE_BOUND_STAMP_SOURCE_KEY, + compile_us_fiscal_target_registry, + ) + + # The compile must be this checkout's, not another installed tree's. + assert Path(ledger_targets.__file__).resolve().is_relative_to(ROOT), ( + ledger_targets.__file__ + ) + assert builder.RESTATED_LEDGER_FILTER_CONCEPTS.get("age") == "age_band" + + rule = builder._restated_ledger_filter_refusal + concepts = dict(builder.RESTATED_LEDGER_FILTER_CONCEPTS) + without_age = { + concept: name for concept, name in concepts.items() if concept != "age" + } + + def reverted(key: str, value: str, metadata: object) -> str: + return key + + arms = ( + ("rule", rule, concepts), + ("without_age_band", rule, without_age), + ("reverted", reverted, concepts), + ) + crosswalk = load_congressional_district_vintage_crosswalk( + default_congressional_district_vintage_crosswalk_path() + ) + + results = [] + for feed in argv: + started = time.time() + artifact = load_ledger_consumer_artifact(feed) + registry = compile_us_fiscal_target_registry( + artifact.facts, + target_period=builder.PERIOD, + congressional_district_vintage_crosswalk=crosswalk, + age_targets=True, + ) + specs = list(registry.specs) + del artifact, registry + result: dict[str, object] = { + "feed": feed, + "feed_sha256": _sha256(feed), + "compile_seconds": round(time.time() - started, 1), + "targets": len(specs), + "age_key_census": _age_key_census(specs, AGE_BOUND_STAMP_SOURCE_KEY), + } + try: + for arm, function, arm_concepts in arms: + builder._restated_ledger_filter_refusal = function + builder.RESTATED_LEDGER_FILTER_CONCEPTS = arm_concepts + result[arm] = _survey(builder, specs) + finally: + builder._restated_ledger_filter_refusal = rule + builder.RESTATED_LEDGER_FILTER_CONCEPTS = concepts + results.append(result) + print(f"done {feed}", file=sys.stderr, flush=True) + + head = subprocess.run( + ["git", "-C", str(ROOT), "rev-parse", "HEAD"], + capture_output=True, + text=True, + check=False, + ).stdout.strip() + status = subprocess.run( + ["git", "-C", str(ROOT), "status", "--porcelain"], + capture_output=True, + text=True, + check=False, + ).stdout.splitlines() + payload = { + "head": head, + "worktree_status": status, + "script_sha256": _sha256(__file__), + "target_period": builder.PERIOD, + "max_rss_bytes": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + "results": results, + } + print(json.dumps(payload, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/packages/microcosm-build/src/microcosm/build/ledger_targets.py b/packages/microcosm-build/src/microcosm/build/ledger_targets.py index 1d4d899b66..145aed30a3 100644 --- a/packages/microcosm-build/src/microcosm/build/ledger_targets.py +++ b/packages/microcosm-build/src/microcosm/build/ledger_targets.py @@ -3512,6 +3512,34 @@ def _constraint_bound_filters( return bounds +def constraint_bound_shadowing_dimensions( + fact: object, variable: str +) -> tuple[str, ...]: + """Dimensions that can put an operator-less value under *variable*'s bound keys. + + :func:`_constraint_bound_filters` keeps the operator in the key it stamps + (``>=`` as ``_lower_bound``, ``<`` as ``_upper_bound``, + and so on through ``_CONSTRAINT_BOUND_SUFFIXES``), but + :func:`_ledger_metadata` stamps the fact's dimensions first and the + constraint edges only through ``setdefault``. So a dimension named for one + of those keys is what ``ledger_filter_`` carries, with no + operator at all; and a dimension named *variable* makes + :func:`_constraint_bound_filters` skip every one of its rows, so any bound + key for it in the metadata is a dimension's. Returns the names of such + dimensions present on *fact*, by key whatever their value, sorted. Empty + means every ``ledger_filter__*bound*`` key the fact's metadata + carries was stamped from a constraint row, with the operator its suffix + names. + """ + + dimensions = _dimensions(fact) + names = ( + variable, + *(f"{variable}{suffix}" for suffix in _CONSTRAINT_BOUND_SUFFIXES.values()), + ) + return tuple(sorted(name for name in names if name in dimensions)) + + def _diagnostic_target_label_metadata( facts: tuple[object, ...], *, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py index bf8c84ab50..16d13d2d7e 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py @@ -21,6 +21,7 @@ LedgerTargetReference, apply_ledger_target_profile, compile_ledger_target_references, + constraint_bound_shadowing_dimensions, hierarchy_seed_from_catalog, ) from microcosm.build.us_runtime.congressional_district_vintage import ( @@ -2634,6 +2635,7 @@ def _population_age_reference_from_fact( "geography_scope": geography_scope, "age_lower_bound": lower, "age_upper_bound": upper, + AGE_BOUND_STAMP_SOURCE_KEY: _age_bound_stamp_source(fact), } groupby_value_id = _str_at(fact, "layout", "groupby_value_id") if groupby_value_id: @@ -2834,6 +2836,7 @@ def _ssa_ssi_reference_from_fact( "target_period": str(target_period), "age_lower_bound": lower, "age_upper_bound": upper, + AGE_BOUND_STAMP_SOURCE_KEY: _age_bound_stamp_source(fact), }, ) @@ -3196,6 +3199,30 @@ def _agi_bounds(fact: object) -> tuple[str, str]: return lower, upper +#: Compiled metadata key, on every age-banded spec, that says where the +#: ``ledger_filter_age_{lower,upper}_bound`` keys ``ledger_targets`` stamps on +#: the same spec came from. Those keys restate the band :func:`_age_bounds` +#: compiles, but only a key stamped from a constraint row carries a known +#: operator (``_lower_bound`` is ``>=``, ``_upper_bound`` is ``<``); a +#: dimension of the same name, or an ``age`` dimension, can put an +#: operator-less value there instead (see +#: :func:`microcosm.build.ledger_targets.constraint_bound_shadowing_dimensions`). +#: The builder's fatal Ledger-filter guard accepts a restated age bound only +#: where this key reads :data:`AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS`. +AGE_BOUND_STAMP_SOURCE_KEY = "age_bound_stamp_source" +#: :data:`AGE_BOUND_STAMP_SOURCE_KEY` value for a fact with no dimension that +#: could shadow its age constraint rows. Any other value is +#: ``dimensions:``, naming the dimensions that could. +AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS = "constraint_rows" + + +def _age_bound_stamp_source(fact: object) -> str: + shadowing = constraint_bound_shadowing_dimensions(fact, "age") + if shadowing: + return "dimensions:" + ",".join(shadowing) + return AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS + + def _age_bounds(fact: object) -> tuple[str, str]: lower = "-inf" upper = "inf" diff --git a/packages/microcosm-build/tests/test_ledger_targets.py b/packages/microcosm-build/tests/test_ledger_targets.py index cf2df190bc..9e7342a9c6 100644 --- a/packages/microcosm-build/tests/test_ledger_targets.py +++ b/packages/microcosm-build/tests/test_ledger_targets.py @@ -12,6 +12,7 @@ LedgerTargetReference, apply_ledger_target_profile, compile_ledger_target_references, + constraint_bound_shadowing_dimensions, hierarchy_seed_from_catalog, ledger_target_registry_parity_report, period_values_semantically_equal, @@ -4476,6 +4477,61 @@ def test__given_bound_already_a_dimension__then_constraint_does_not_restamp_it() assert "ledger_filter_ignored_lower_bound_exclusive" not in metadata +def test__given_dimension_named_for_a_bound_key__then_it_is_reported_as_shadowing() -> ( + None +): + # The dimension stamp runs first and the constraint edges only through + # ``setdefault``, so a dimension named for a bound key is what that key + # carries, with no operator; the helper names every dimension that can + # do that, or that makes the stamp skip the variable's rows outright. + fact = _consumer_fact_row_for_period(2024, value=10.0) + fact["dimensions"] = {"age_upper_bound": 10, "region": "north"} + fact["universe_constraints"] = { + "domain": "resident_population", + "constraints": [ + {"operator": ">=", "role": "filter", "value": 5, "variable": "age"}, + {"operator": "<", "role": "filter", "value": 9, "variable": "age"}, + ], + } + reference = LedgerTargetReference( + name="age band", + ledger_selector={"source_name": "irs_soi"}, + entity="person", + measure="person_count", + period=2024, + ) + + metadata = ( + compile_ledger_target_references([fact], [reference], country="us") + .specs[0] + .metadata + ) + # The ``< 9`` row lost its key to the dimension's operator-less value. + assert metadata["ledger_filter_age_upper_bound"] == "10" + assert metadata["ledger_filter_age_lower_bound"] == "5" + assert constraint_bound_shadowing_dimensions(fact, "age") == ("age_upper_bound",) + + fact["dimensions"] = {"region": "north"} + assert constraint_bound_shadowing_dimensions(fact, "age") == () + + # Presence of the key is what counts, whatever its value: an ``age`` + # dimension valued ``None`` stamps nothing yet still skips every age row. + fact["dimensions"] = { + "age": None, + "age_lower_bound": 5, + "age_upper_bound_inclusive": 9, + } + assert constraint_bound_shadowing_dimensions(fact, "age") == ( + "age", + "age_lower_bound", + "age_upper_bound_inclusive", + ) + + # ``filters`` is read in place of ``dimensions``, as the stamp reads it. + fact["filters"] = {"age_lower_bound": 5} + assert constraint_bound_shadowing_dimensions(fact, "age") == ("age_lower_bound",) + + def _ratio_fact( *, key: str, diff --git a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py index 98dcbf91b9..62822fe1ec 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py @@ -2868,12 +2868,585 @@ def test_restated_filters_clear_the_soi_skip_only_where_they_agree() -> None: ) == ("ledger_filter_new_dimension",) +_RESTATED_AGE_LOWER = "ledger_filter_age_lower_bound" +_RESTATED_AGE_UPPER = "ledger_filter_age_upper_bound" +_RESTATED_AGE_EXACT = "ledger_filter_age" +_AGE_STAMP_SOURCE = "age_bound_stamp_source" + + +def _population_age_metadata(**overrides: str | None) -> dict[str, str]: + """A compiled Census population-by-age spec's metadata, as compiled. + + Mirrors ``_population_age_reference_from_fact`` for a state 5-to-9 cell — + the half-open band the materializer slices on + (``age_lower_bound``/``age_upper_bound``) and the compile's attestation + that the fact has no dimension that could shadow its age rows — plus the + restatement ``ledger_targets._constraint_bound_filters`` stamps from that + fact's ``age >= 5`` and ``age < 10`` rows. An override of ``None`` drops + the key. + """ + + metadata = { + "materializer": "population_age", + "measure_mode": "indicator_sum", + "source_measure_id": "population", + "target_role": "population_age", + "geography_scope": "state", + "state_fips": "06", + "age_group": "5_to_9", + "age_lower_bound": "5", + "age_upper_bound": "10", + _AGE_STAMP_SOURCE: "constraint_rows", + _RESTATED_AGE_LOWER: "5", + _RESTATED_AGE_UPPER: "10", + } + for key, value in overrides.items(): + if value is None: + metadata.pop(key, None) + else: + metadata[key] = value + return metadata + + +def _ssa_age_band_metadata(**overrides: str | None) -> dict[str, str]: + """A compiled SSA SSI recipients-by-age spec's metadata, as compiled. + + Mirrors ``_ssa_ssi_reference_from_fact`` for the ``under_18`` row + (``age >= 0``, ``age < 18``): an age-banded ``policyengine_variable`` + indicator count, plus the stamped restatement. + """ + + metadata = { + "materializer": "policyengine_variable", + "measure_mode": "indicator_sum", + "base_variable": "ssi", + "target_role": "ssa_ssi_age_band_recipients", + "source_measure_id": "recipient_count", + "age_lower_bound": "0", + "age_upper_bound": "18", + _AGE_STAMP_SOURCE: "constraint_rows", + _RESTATED_AGE_LOWER: "0", + _RESTATED_AGE_UPPER: "18", + } + for key, value in overrides.items(): + if value is None: + metadata.pop(key, None) + else: + metadata[key] = value + return metadata + + +def test_restated_age_bounds_pass_the_guard_only_where_they_agree() -> None: + """A restated age bound equal to the compiled one is ignorable, so accepted. + + The Ledger stamp restates a fact's ``age >= a`` / ``age < b`` rows as + ``ledger_filter_age_{lower,upper}_bound`` beside the compiled + ``age_lower_bound``/``age_upper_bound`` that both age paths in + ``_materialize_target_frame`` slice on as ``a <= age < b``. Where the two + name the same edge, dropping the restatement changes no population. Both + materializers are covered, one side at a time, an open top (``85+``: no + upper row, compiled ``inf``), and float spellings of the same edge. The + SOI checker must not list them either, so no listing turns an accepted + restatement into a silent drop. + """ + + builder = _load_builder_module() + specs = ( + _spec( + "census_pep.v2024.cy2024.state_resident_population.06.5_to_9.population", + _population_age_metadata(), + ), + _spec("lower_only", _population_age_metadata(**{_RESTATED_AGE_UPPER: None})), + _spec("upper_only", _population_age_metadata(**{_RESTATED_AGE_LOWER: None})), + _spec( + "open_top", + _population_age_metadata( + age_group="85_plus", + age_lower_bound="85", + age_upper_bound="inf", + **{_RESTATED_AGE_LOWER: "85", _RESTATED_AGE_UPPER: None}, + ), + ), + _spec( + "float_spelling", + _population_age_metadata( + **{_RESTATED_AGE_LOWER: "5.0", _RESTATED_AGE_UPPER: "10.0"} + ), + ), + _spec( + "ssa_ssi_monthly.month2024_12.ssi_federal_payment_recipients." + "by_age.under_18.recipient_count", + _ssa_age_band_metadata(), + ), + ) + + assert builder._unsupported_ledger_filter_metadata(specs) == {} + builder._assert_supported_ledger_filter_metadata(specs) + for spec in specs: + assert builder._unsupported_soi_ledger_filters(spec.metadata) == () + + # Non-vacuity: with the concept unnamed, the same specs get the bare-key + # refusal every one of them got before this rule. + del builder.RESTATED_LEDGER_FILTER_CONCEPTS["age"] + refused = builder._unsupported_ledger_filter_metadata(specs) + assert set(refused) == {spec.name for spec in specs} + assert refused[specs[0].name] == (_RESTATED_AGE_LOWER, _RESTATED_AGE_UPPER) + + +def test_disagreeing_restated_age_bounds_are_refused_by_value() -> None: + """Negative control for the acceptance above: only an applied, equal edge. + + Each spec carries an age restatement the materializer would ignore while + slicing a different population, or no population at all: an edge a year + off on either side, a finite upper edge over a compiled open top, a bound + with no compiled counterpart, an exact-age restatement (the materializer + has no exact-age filter), a bound on a fact whose dimensions include age + (so the bound came from the operator-less dimension stamp), a bound on + specs whose materializer applies no age band, and agreeing bounds whose + operator the compile does not attest: no attestation at all, or one naming + a dimension that could have written the key with no operator. Every + refusal names the restated value and, where one exists, the compiled one + or the attestation. The unknown key is the control that the ordinary path + is untouched. + """ + + builder = _load_builder_module() + specs = ( + _spec( + "disagreeing_lower", + _population_age_metadata(**{_RESTATED_AGE_LOWER: "4"}), + ), + _spec( + "disagreeing_upper", + _population_age_metadata(**{_RESTATED_AGE_UPPER: "9"}), + ), + _spec( + "ssa_disagreeing_upper", + _ssa_age_band_metadata(**{_RESTATED_AGE_UPPER: "65"}), + ), + _spec( + "finite_over_open_top", + _population_age_metadata( + age_lower_bound="85", + age_upper_bound="inf", + **{_RESTATED_AGE_LOWER: "85", _RESTATED_AGE_UPPER: "100"}, + ), + ), + _spec( + "no_compiled_upper", + _population_age_metadata(age_upper_bound=None), + ), + _spec( + "exact_age", + _population_age_metadata( + **{ + _RESTATED_AGE_EXACT: "7", + _RESTATED_AGE_LOWER: None, + _RESTATED_AGE_UPPER: None, + } + ), + ), + _spec( + "age_dimension", + _population_age_metadata( + **{_RESTATED_AGE_EXACT: "all", _RESTATED_AGE_UPPER: None} + ), + ), + _spec( + "soi_materializer", + _soi_band_metadata( + age_lower_bound="5", + age_upper_bound="10", + **{_RESTATED_AGE_LOWER: "5"}, + ), + ), + _spec( + "no_materializer", + _population_age_metadata(materializer=None, **{_RESTATED_AGE_UPPER: None}), + ), + _spec( + "unattested", + _population_age_metadata( + **{_AGE_STAMP_SOURCE: None, _RESTATED_AGE_LOWER: None} + ), + ), + _spec( + "dimension_shadowed", + _ssa_age_band_metadata( + **{ + _AGE_STAMP_SOURCE: "dimensions:age_upper_bound", + _RESTATED_AGE_LOWER: None, + } + ), + ), + _spec( + "unknown_key", + _population_age_metadata(ledger_filter_novel_dimension="specific_slice"), + ), + ) + + assert builder._unsupported_ledger_filter_metadata(specs) == { + "disagreeing_lower": ( + f"{_RESTATED_AGE_LOWER}=4 disagrees with age_lower_bound=5", + ), + "disagreeing_upper": ( + f"{_RESTATED_AGE_UPPER}=9 disagrees with age_upper_bound=10", + ), + "ssa_disagreeing_upper": ( + f"{_RESTATED_AGE_UPPER}=65 disagrees with age_upper_bound=18", + ), + "finite_over_open_top": ( + f"{_RESTATED_AGE_UPPER}=100 disagrees with age_upper_bound=inf", + ), + "no_compiled_upper": ( + f"{_RESTATED_AGE_UPPER}=10 restates a bound the spec does not " + "compile: no age_upper_bound", + ), + "exact_age": ( + f"{_RESTATED_AGE_EXACT}=7 restates an exact age, but the " + "materializer slices a half-open age band and applies no " + "exact-age filter", + ), + "age_dimension": ( + f"{_RESTATED_AGE_LOWER}=5 restates an age bound on a fact whose " + f"dimensions include age ({_RESTATED_AGE_EXACT}=all), so the bound " + "came from the dimension stamp, which carries no operator", + ), + "soi_materializer": ( + f"{_RESTATED_AGE_LOWER}=5 restates an age bound on a spec whose " + "materializer ('irs_soi_slice') applies no age band", + ), + "no_materializer": ( + f"{_RESTATED_AGE_LOWER}=5 restates an age bound on a spec whose " + "materializer (None) applies no age band", + ), + "unattested": ( + f"{_RESTATED_AGE_UPPER}=10 restates an age bound whose operator is " + f"ambiguous: {_AGE_STAMP_SOURCE}=None does not attest it was stamped " + "from a constraint row (>= or <), and a dimension of that name " + "carries no operator", + ), + "dimension_shadowed": ( + f"{_RESTATED_AGE_UPPER}=18 restates an age bound whose operator is " + f"ambiguous: {_AGE_STAMP_SOURCE}=dimensions:age_upper_bound does not " + "attest it was stamped from a constraint row (>= or <), and a " + "dimension of that name carries no operator", + ), + "unknown_key": ("ledger_filter_novel_dimension",), + } + with pytest.raises(RuntimeError) as excinfo: + builder._assert_supported_ledger_filter_metadata(specs) + message = str(excinfo.value) + assert "Unsupported Ledger target filter metadata would be ignored" in message + assert f"{_RESTATED_AGE_UPPER}=9 disagrees with age_upper_bound=10" in message + assert "ledger_filter_novel_dimension" in message + # The SOI loop's checker lists the same refusal, so the fatal guard (which + # runs first) is what stops it rather than a silent skip. + assert builder._unsupported_soi_ledger_filters( + _soi_band_metadata( + age_lower_bound="5", age_upper_bound="10", **{_RESTATED_AGE_LOWER: "5"} + ) + ) == (_RESTATED_AGE_LOWER,) + + +def test_operator_changed_age_bounds_stay_refused_by_bare_key() -> None: + """A ``>`` or ``<=`` age row is refused even when its value matches. + + ``us_runtime.fiscal_targets._age_bounds`` drops the operator when it + compiles the band, so an ``age <= 9`` row compiles to + ``age_upper_bound=9`` and the materializer's ``age < 9`` leaves out the + nine-year-olds the published cell counts; ``age > 4`` compiles to + ``age_lower_bound=4`` and lets the four-year-olds in. The stamp keeps the + operator in the key (``_upper_bound_inclusive`` / + ``_lower_bound_exclusive``), which is not a restated concept, so those + keys keep the bare-key refusal whatever their value — the rule must not + swallow them by value agreement. + """ + + builder = _load_builder_module() + inclusive_upper = f"{_RESTATED_AGE_UPPER}_inclusive" + exclusive_lower = f"{_RESTATED_AGE_LOWER}_exclusive" + assert builder._restated_ledger_filter_concept(inclusive_upper) == ( + "age_upper_bound_inclusive", + None, + ) + specs = ( + _spec( + "inclusive_upper", + _population_age_metadata( + age_upper_bound="9", + **{_RESTATED_AGE_UPPER: None, inclusive_upper: "9"}, + ), + ), + _spec( + "exclusive_lower", + _population_age_metadata( + age_lower_bound="4", + **{_RESTATED_AGE_LOWER: None, exclusive_lower: "4"}, + ), + ), + ) + + assert builder._unsupported_ledger_filter_metadata(specs) == { + "inclusive_upper": (inclusive_upper,), + "exclusive_lower": (exclusive_lower,), + } + + +def _census_age_fact( + groupby_value_id: str, + constraints: list[tuple[str, int]], + *, + dimensions: dict[str, str] | None = None, +) -> dict[str, object]: + """A minimal Census PEP national population-by-age consumer fact.""" + + source_record_id = ( + f"census_pep.cy2024.national_resident_population_age.{groupby_value_id}" + ".population" + ) + fact_key = source_record_id.replace(".", "_") + dimensions = dict(dimensions or {}) + return { + "label": source_record_id, + "aggregate_fact_key": f"ledger.aggregate_fact.v2:{fact_key}", + "semantic_fact_key": f"ledger.semantic_fact.v2:{fact_key}", + "legacy_fact_key": f"ledger.fact.v1:{fact_key}", + "lineage": {"source_record_id": source_record_id}, + "value": 1_000.0, + "period": {"type": "calendar_year", "value": 2024}, + "entity": {"name": "person"}, + "aggregation": {"method": "sum"}, + "geography": {"level": "country", "id": "0100000US", "name": "United States"}, + "dimensions": dimensions, + "dimension_labels": {key: f"Label {key}" for key in dimensions}, + "dimension_value_labels": { + key: {str(value): f"Value {value}"} for key, value in dimensions.items() + }, + "universe_constraints": { + "constraints": [ + { + "variable": "age", + "operator": operator, + "value": value, + "role": "filter", + "unit": "years", + } + for operator, value in constraints + ] + }, + "layout": { + "record_set_id": "census_pep.cy2024.national_resident_population_age", + "groupby_value_id": groupby_value_id, + "measure_id": "population", + }, + "observed_measure": { + "source_name": "census_pep", + "source_measure_id": "population", + "unit": "persons", + }, + "source": {"source_name": "census_pep", "source_table": "PEP"}, + } + + +def test_restated_age_bounds_are_judged_on_the_real_compile() -> None: + """The rule meets the keys the production compile actually stamps. + + Facts run through ``_dynamic_us_fiscal_target_references`` and + ``compile_ledger_target_references`` — the path + ``compile_us_fiscal_target_registry`` takes — so the compiled bounds and + the restated keys are the real compiler's, not hand-written metadata. + The ``>=``/``<`` band and the open-topped ``>=`` band are accepted, and + the test first checks they really carry the restated keys; the ``<=`` and + ``>`` rows compile to the same numeric edges but keep the operator in the + stamped key and stay refused; a fact with ``age`` as a dimension gets no + bound stamp at all, only the exact restatement, which is refused. + + The last two facts are the dimension stamp's operator-less keys, which + agree with the compiled edge by value and carry no other refusable key: + an ``age_upper_bound`` dimension whose value the stamp writes over the + ``<`` row's (``setdefault``), and an ``age`` dimension valued ``None`` + beside an ``age_upper_bound`` dimension over an ``age <= 29`` row, where + no ``ledger_filter_age`` or ``_inclusive`` key is stamped at all and the + materializer's ``age < 29`` would drop the 29-year-olds. Only the + compile's attestation refuses them. + """ + + from microcosm.build.ledger_targets import compile_ledger_target_references + from microcosm.build.us_runtime.fiscal_targets import ( + _dynamic_us_fiscal_target_references, + ) + + builder = _load_builder_module() + facts = [ + _census_age_fact("0_to_4", [(">=", 0), ("<", 5)]), + _census_age_fact("85_plus", [(">=", 85)]), + _census_age_fact("5_to_9", [(">=", 5), ("<=", 9)]), + _census_age_fact("10_to_14", [(">", 9), ("<", 15)]), + _census_age_fact( + "15_to_19", [(">=", 15), ("<", 20)], dimensions={"age": "15_to_19"} + ), + _census_age_fact( + "20_to_24", [(">=", 20), ("<", 25)], dimensions={"age_upper_bound": 25} + ), + _census_age_fact( + "25_to_29", + [(">=", 25), ("<=", 29)], + dimensions={"age": None, "age_upper_bound": 29}, + ), + ] + registry = compile_ledger_target_references( + facts, + _dynamic_us_fiscal_target_references(facts, target_period=2024), + country="us", + ) + specs = {spec.name.split(".")[-2]: spec for spec in registry.specs} + assert set(specs) == { + "0_to_4", + "85_plus", + "5_to_9", + "10_to_14", + "15_to_19", + "20_to_24", + "25_to_29", + } + + def age_keys(name: str) -> dict[str, str]: + return { + key: value + for key, value in specs[name].metadata.items() + if key.startswith(("age_", "ledger_filter_age")) and key != "age_group" + } + + assert age_keys("0_to_4") == { + "age_lower_bound": "0", + "age_upper_bound": "5", + _AGE_STAMP_SOURCE: "constraint_rows", + _RESTATED_AGE_LOWER: "0", + _RESTATED_AGE_UPPER: "5", + } + assert age_keys("85_plus") == { + "age_lower_bound": "85", + "age_upper_bound": "inf", + _AGE_STAMP_SOURCE: "constraint_rows", + _RESTATED_AGE_LOWER: "85", + } + assert age_keys("5_to_9")["age_upper_bound"] == "9" + assert age_keys("10_to_14")["age_lower_bound"] == "9" + assert age_keys("15_to_19")[_AGE_STAMP_SOURCE] == "dimensions:age" + # The shadowed keys agree with the compiled edges by value, so value + # agreement alone would accept them. + assert age_keys("20_to_24") == { + "age_lower_bound": "20", + "age_upper_bound": "25", + _AGE_STAMP_SOURCE: "dimensions:age_upper_bound", + _RESTATED_AGE_LOWER: "20", + _RESTATED_AGE_UPPER: "25", + } + assert age_keys("25_to_29") == { + "age_lower_bound": "25", + "age_upper_bound": "29", + _AGE_STAMP_SOURCE: "dimensions:age,age_upper_bound", + _RESTATED_AGE_UPPER: "29", + } + + def ambiguous(key: str, value: str, source: str) -> str: + return ( + f"{key}={value} restates an age bound whose operator is ambiguous: " + f"{_AGE_STAMP_SOURCE}={source} does not attest it was stamped from " + "a constraint row (>= or <), and a dimension of that name carries " + "no operator" + ) + + refused = builder._unsupported_ledger_filter_metadata(registry.specs) + assert {name.split(".")[-2]: entries for name, entries in refused.items()} == { + "5_to_9": (f"{_RESTATED_AGE_UPPER}_inclusive",), + "10_to_14": (f"{_RESTATED_AGE_LOWER}_exclusive",), + "15_to_19": ( + f"{_RESTATED_AGE_EXACT}=15_to_19 restates an exact age, but the " + "materializer slices a half-open age band and applies no " + "exact-age filter", + ), + "20_to_24": ( + ambiguous(_RESTATED_AGE_LOWER, "20", "dimensions:age_upper_bound"), + ambiguous(_RESTATED_AGE_UPPER, "25", "dimensions:age_upper_bound"), + ), + "25_to_29": ( + ambiguous(_RESTATED_AGE_UPPER, "29", "dimensions:age,age_upper_bound"), + ), + } + + +def test_pinned_chronicle_feed_whole_registry_compiles_no_unsupported_filters() -> None: + """The whole compiled registry clears the guard, when this machine has the feed. + + Before the age-band rule, ``compile_us_fiscal_target_registry(..., + age_targets=True)`` over the pinned feed left 939 of 32,866 targets + refused, every one a ``census_population``/``population_age`` or + ``ssa``/``ssa_ssi_age_band_recipients`` target carrying + ``ledger_filter_age_{lower,upper}_bound`` + (``experiments/us-labelled-filter-support/REPORT.md`` section 7). This is + the whole-registry sibling of + :func:`test_pinned_chronicle_feed_state_surface_compiles_no_unsupported_filters` + and runs only when ``MICROCOSM_US_CHRONICLE_FACTS`` points at the feed. + The non-vacuity arm requires restated age keys to be present. + """ + + feed = os.environ.get("MICROCOSM_US_CHRONICLE_FACTS", "") + if not feed or not Path(feed).exists(): + pytest.skip( + "set MICROCOSM_US_CHRONICLE_FACTS to the pinned consumer-facts feed" + ) + + from microcosm.build.ledger_artifact import load_ledger_consumer_artifact + from microcosm.build.us_runtime import ( + default_congressional_district_vintage_crosswalk_path, + load_congressional_district_vintage_crosswalk, + ) + from microcosm.build.us_runtime.fiscal_targets import ( + compile_us_fiscal_target_registry, + ) + + builder = _load_builder_module() + registry = compile_us_fiscal_target_registry( + load_ledger_consumer_artifact(feed).facts, + target_period=builder.PERIOD, + congressional_district_vintage_crosswalk=( + load_congressional_district_vintage_crosswalk( + default_congressional_district_vintage_crosswalk_path() + ) + ), + age_targets=True, + ) + specs = tuple(registry.specs) + restated_age = [ + spec + for spec in specs + if _RESTATED_AGE_LOWER in spec.metadata or _RESTATED_AGE_UPPER in spec.metadata + ] + + assert restated_age + assert {spec.metadata.get("materializer") for spec in restated_age} <= ( + builder.RESTATED_AGE_BAND_MATERIALIZERS + ) + assert {spec.metadata.get(_AGE_STAMP_SOURCE) for spec in restated_age} == { + "constraint_rows" + } + assert builder._unsupported_ledger_filter_metadata(specs) == {} + assert not [ + spec + for spec in specs + if spec.family == "irs_soi" + and builder._unsupported_soi_ledger_filters(spec.metadata) + ] + + def test_restated_concept_rules_all_have_a_comparison() -> None: """Every named restated concept must route to an implemented comparison. Adding a concept to ``RESTATED_LEDGER_FILTER_CONCEPTS`` without a rule would raise at compile time on the first spec carrying it; this pins the - two rules that exist instead. + three rules that exist instead. """ builder = _load_builder_module() @@ -2881,7 +3454,19 @@ def test_restated_concept_rules_all_have_a_comparison() -> None: assert set(builder.RESTATED_LEDGER_FILTER_CONCEPTS.values()) == { "agi_band", "eitc_child_count", + "age_band", } + assert builder._restated_ledger_filter_concept(_RESTATED_AGE_UPPER) == ( + "age", + "upper", + ) + # The age rule reads the compile's attestation under the compile's names. + assert builder.AGE_BOUND_STAMP_SOURCE_KEY == _AGE_STAMP_SOURCE + assert builder.AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS == "constraint_rows" + assert builder._restated_ledger_filter_concept(_RESTATED_AGE_EXACT) == ( + "age", + None, + ) assert builder._restated_ledger_filter_concept(_RESTATED_AGI_LOWER) == ( "us:statutes/26/62#adjusted_gross_income", "lower", @@ -9321,9 +9906,23 @@ def _invalidate_all_caches(self): assert compilation["dropped_target_names"] == [] +@pytest.mark.parametrize("restatement", ["none", "agreeing", "disagreeing"]) def test_population_age_targets_materialize_person_age_counts( monkeypatch, + restatement, ) -> None: + """Age-banded targets materialize as ``lower <= age < upper`` person counts. + + ``restatement`` adds the ``ledger_filter_age_{lower,upper}_bound`` keys + ``ledger_targets._constraint_bound_filters`` stamps from a fact's + ``age >=`` / ``age <`` rows. Agreeing restatements must change nothing — + same columns, same values, no target dropped — on both age paths: the + ``population_age`` materializer and the age-banded + ``policyengine_variable`` branch the SSA SSI by-age counts use. A + restated upper edge one year off must stop the build at the guard, naming + both values, before anything is materialized. + """ + builder = _load_builder_module() _installed_variable_metadata_index(builder) frame = Frame( @@ -9391,6 +9990,8 @@ def population_age_spec( ), "age_lower_bound": str(lower), "age_upper_bound": str(upper), + _AGE_STAMP_SOURCE: "constraint_rows", + **restated_bounds(lower, upper), } if state_fips: metadata["state_fips"] = state_fips @@ -9406,6 +10007,37 @@ def population_age_spec( metadata=metadata, ) + def restated_bounds(lower, upper): + if restatement == "none": + return {} + restated_upper = upper + 1 if restatement == "disagreeing" else upper + restated = {"ledger_filter_age_lower_bound": str(lower)} + if upper != "inf": + restated["ledger_filter_age_upper_bound"] = str(restated_upper) + return restated + + def ssa_age_band_spec(name, lower, upper): + # The shape _ssa_ssi_reference_from_fact compiles for an SSA SSI + # by-age row (microcosm#470): an age-banded person indicator count. + return TargetSpec( + name=name, + entity="household", + measure=name, + value=1.0, + source="fixture", + family="ssa", + metadata={ + "materializer": "policyengine_variable", + "measure_mode": "indicator_sum", + "base_variable": "ssi", + "target_role": builder.SSA_SSI_AGE_BAND_RECIPIENTS_TARGET_ROLE, + "age_lower_bound": str(lower), + "age_upper_bound": str(upper), + _AGE_STAMP_SOURCE: "constraint_rows", + **restated_bounds(lower, upper), + }, + ) + targets = ( population_age_spec("national_age_0_to_4", 0, 5), population_age_spec("ca_age_0_to_4", 0, 5, state_fips="06"), @@ -9417,6 +10049,8 @@ def population_age_spec( state_fips="06", congressional_district_geoid="0601", ), + ssa_age_band_spec("ssi_under_18", 0, 18), + ssa_age_band_spec("ssi_age_8_to_64", 8, 65), ) class FakeVariable: @@ -9431,6 +10065,7 @@ class FakeSystem: "filing_status": FakeVariable("tax_unit"), "state_income_tax": FakeVariable("tax_unit"), "age": FakeVariable("person"), + "ssi": FakeVariable("person"), } def __init__(self, reform=None): @@ -9454,6 +10089,7 @@ def calculate(self, variable, *, period, **kwargs): "filing_status": np.asarray(["SINGLE", "SINGLE", "SINGLE"]), "state_income_tax": np.asarray([0.0, 0.0, 0.0]), "age": np.asarray([2.0, 7.0, 4.0, 11.0]), + "ssi": np.asarray([0.0, 300.0, 0.0, 50.0]), } return arrays[variable] @@ -9472,6 +10108,20 @@ def _invalidate_all_caches(self): monkeypatch.setattr(builder, "SOI_VARIABLE_MAP", {}) monkeypatch.setattr(builder, "US_JCT_TAX_EXPENDITURE_REFORMS", ()) + if restatement == "disagreeing": + with pytest.raises(RuntimeError) as excinfo: + builder._materialize_target_frame(frame, targets) + message = str(excinfo.value) + assert ( + "ca_age_5_to_9: ledger_filter_age_upper_bound=11 disagrees with " + "age_upper_bound=10" + ) in message + assert ( + "ssi_under_18: ledger_filter_age_upper_bound=19 disagrees with " + "age_upper_bound=18" + ) in message + return + target_frame, registry, compilation = builder._materialize_target_frame( frame, targets ) @@ -9481,7 +10131,11 @@ def _invalidate_all_caches(self): assert np.array_equal(household["ca_age_0_to_4"], np.asarray([1.0, 0.0])) assert np.array_equal(household["ca_age_5_to_9"], np.asarray([1.0, 0.0])) assert np.array_equal(household["ca_01_age_0_to_4"], np.asarray([1.0, 0.0])) - assert len(registry) == 4 + # SSI recipients are the age-7 person in household 1 and the age-11 person + # in household 2; the 8-to-64 band keeps only the latter. + assert np.array_equal(household["ssi_under_18"], np.asarray([1.0, 1.0])) + assert np.array_equal(household["ssi_age_8_to_64"], np.asarray([0.0, 1.0])) + assert len(registry) == 6 assert compilation["dropped_target_names"] == [] diff --git a/packages/microcosm-build/tests/test_us_fiscal_targets.py b/packages/microcosm-build/tests/test_us_fiscal_targets.py index 0252714f29..390c817229 100644 --- a/packages/microcosm-build/tests/test_us_fiscal_targets.py +++ b/packages/microcosm-build/tests/test_us_fiscal_targets.py @@ -4382,6 +4382,8 @@ def test_ssa_ssi_age_band_counts_bind_as_person_age_indicator_targets() -> None: assert spec.metadata["target_role"] == "ssa_ssi_age_band_recipients" assert spec.metadata["age_lower_bound"] == "0" assert spec.metadata["age_upper_bound"] == "18" + # No dimension of the fact can shadow its age rows' Ledger stamp. + assert spec.metadata["age_bound_stamp_source"] == "constraint_rows" assert spec.value == 1_001_922 assert all_ages_id not in specs @@ -4936,6 +4938,7 @@ def test_census_pep_population_age_facts_compile_to_count_targets() -> None: assert national.metadata["geography_scope"] == "national" assert national.metadata["age_lower_bound"] == "0" assert national.metadata["age_upper_bound"] == "5" + assert national.metadata["age_bound_stamp_source"] == "constraint_rows" assert national.value == 18_000_000 state = specs[state_source_record_id] diff --git a/tools/build_us_fiscal_refresh_release.py b/tools/build_us_fiscal_refresh_release.py index 00ca5df770..c6dda36861 100644 --- a/tools/build_us_fiscal_refresh_release.py +++ b/tools/build_us_fiscal_refresh_release.py @@ -231,6 +231,8 @@ exact_k_ladder_manifest_payload, ) from microcosm.build.us_runtime.fiscal_targets import ( + AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS, + AGE_BOUND_STAMP_SOURCE_KEY, SSA_SSI_AGE_BAND_RECIPIENTS_TARGET_ROLE, ) from microcosm.build.us_runtime.h5_io import ( @@ -672,6 +674,7 @@ def _automatic_gc_suspended(): RESTATED_LEDGER_FILTER_CONCEPTS = { "us:statutes/26/62#adjusted_gross_income": "agi_band", "us.tax.earned_income_credit_qualifying_children": "eitc_child_count", + "age": "age_band", } #: Bound-side suffixes a restated constraint key may carry. A key with no @@ -715,6 +718,63 @@ def _automatic_gc_suspended(): #: and exact restatements stay under the agreement rule. RESTATED_EITC_CHILD_COUNT_REFUSED_SIDES = frozenset({"upper"}) +#: Compiled metadata the materializer slices a person-age band on, by bound +#: side. Both age paths in :func:`_materialize_target_frame` read these +#: through :func:`_as_bound` into one half-open mask ``lower <= age < upper`` +#: — the ``population_age`` loop in :func:`_population_age_household_values` +#: and the age-banded ``policyengine_variable`` branch (the SSA SSI +#: recipients-by-age counts) — so a restated age bound is judged, like an AGI +#: bound, against its own side only. +#: +#: Numeric equality of the edges is equality of populations only for a +#: restated key whose operator is the mask's: ``age < 10`` and ``age <= 10`` +#: differ by everyone aged exactly ten, so the operator decides. +#: :func:`microcosm.build.ledger_targets._constraint_bound_filters` writes +#: ``age_lower_bound`` only for a ``>=`` row and ``age_upper_bound`` only for +#: a ``<`` row, the two operators of the materializer's mask. A ``>`` or +#: ``<=`` row is stamped ``age_lower_bound_exclusive`` / +#: ``age_upper_bound_inclusive``, which no restated concept names, so it stays +#: refused by bare key — as it must, because the compiled bound comes from +#: ``us_runtime.fiscal_targets._age_bounds``, which drops the operator: a +#: ``<= 4`` row compiles to ``age_upper_bound=4`` and the materializer's +#: ``age < 4`` would leave out the four-year-olds the published cell counts. +#: But the same ``ledger_filter_age_{lower,upper}_bound`` key can also be a +#: dimension's, which carries no operator (the ambiguity that keeps restated +#: qualifying-child upper bounds refused), so the key name alone does not +#: settle it; the compile's attestation does +#: (:data:`microcosm.build.us_runtime.fiscal_targets.AGE_BOUND_STAMP_SOURCE_KEY`). +RESTATED_AGE_BAND_COMPILED_KEYS = { + "lower": "age_lower_bound", + "upper": "age_upper_bound", +} + +#: Materializers that apply the compiled age band, and so the only ones for +#: which ignoring an agreeing age restatement changes nothing. Both are read +#: in :func:`_materialize_target_frame`: ``population_age`` through +#: :func:`_population_age_household_values`, and ``policyengine_variable`` +#: through its age-banded branch, entered whenever either compiled age key is +#: present. Every other materializer (the ``irs_soi`` slice, the direct +#: household-variable measures) never reads an age bound, so a restated age +#: bound there would be silently ignored however well it agreed with metadata +#: the materializer also ignores; it is refused instead. +RESTATED_AGE_BAND_MATERIALIZERS = frozenset({"population_age", "policyengine_variable"}) + +#: The exact-value age key the dimension stamp writes when ``age`` is itself +#: a dimension of the fact. +#: :func:`microcosm.build.ledger_targets._constraint_bound_filters` skips +#: every constraint row whose variable is already a dimension key, so on +#: such a fact an ``age_{lower,upper}_bound`` restatement cannot have come +#: from an operator-checked ``>=``/``<`` row — only from a dimension of that +#: name, which carries no operator at all. A restated age bound on a spec +#: carrying this key is refused, whatever its value, including ``all``. +#: ``_ledger_metadata`` stamps a dimension only when its value is not +#: ``None`` and then drops empty values, so an ``age`` dimension valued +#: ``None`` or ``""`` leaves no key here; the compile's attestation +#: (:data:`AGE_BOUND_STAMP_SOURCE_KEY`, which checks dimension keys whatever +#: their value) is what refuses that case. The pinned feeds have no age +#: dimension of any kind. +RESTATED_AGE_DIMENSION_KEY = "ledger_filter_age" + FISCAL_TARGET_SOURCE_KEYS = { "cbo": "Congressional Budget Office revenue projections", "cms_aca": "CMS ACA marketplace enrollment public use files", @@ -4527,19 +4587,23 @@ def _restated_count_value(value: str) -> float | None: return count -def _restated_agi_band_refusal( +def _restated_band_edge_refusal( key: str, value: str, metadata: Mapping[str, str], *, - side: str | None, + compiled_key: str, ) -> str | None: - if side is None: - return ( - f"{key}={value} restates an exact AGI, but the materializer " - "slices a half-open AGI band and applies no exact-value AGI filter" - ) - compiled_key = RESTATED_AGI_BAND_COMPILED_KEYS[side] + """Judge one restated half-open band edge against the edge it restates. + + Shared by the AGI and age rules, whose materializers both slice + ``lower <= x < upper`` on the compiled edge read through + :func:`_as_bound`, so a restated edge is ignorable exactly when it parses + to the same number. A missing compiled edge refuses: the age paths would + read it as the open end and the ``irs_soi`` loop indexes it directly, and + neither is the restated edge being checked. + """ + compiled = metadata.get(compiled_key) if compiled is None: return f"{key}={value} restates a bound the spec does not compile: no {compiled_key}" @@ -4550,6 +4614,61 @@ def _restated_agi_band_refusal( return None +def _restated_agi_band_refusal( + key: str, + value: str, + metadata: Mapping[str, str], + *, + side: str | None, +) -> str | None: + if side is None: + return ( + f"{key}={value} restates an exact AGI, but the materializer " + "slices a half-open AGI band and applies no exact-value AGI filter" + ) + return _restated_band_edge_refusal( + key, value, metadata, compiled_key=RESTATED_AGI_BAND_COMPILED_KEYS[side] + ) + + +def _restated_age_band_refusal( + key: str, + value: str, + metadata: Mapping[str, str], + *, + side: str | None, +) -> str | None: + if side is None: + return ( + f"{key}={value} restates an exact age, but the materializer " + "slices a half-open age band and applies no exact-age filter" + ) + if RESTATED_AGE_DIMENSION_KEY in metadata: + return ( + f"{key}={value} restates an age bound on a fact whose dimensions " + f"include age ({RESTATED_AGE_DIMENSION_KEY}=" + f"{metadata[RESTATED_AGE_DIMENSION_KEY]}), so the bound came from " + "the dimension stamp, which carries no operator" + ) + materializer = metadata.get("materializer") + if materializer not in RESTATED_AGE_BAND_MATERIALIZERS: + return ( + f"{key}={value} restates an age bound on a spec whose materializer " + f"({materializer!r}) applies no age band" + ) + stamp_source = metadata.get(AGE_BOUND_STAMP_SOURCE_KEY) + if stamp_source != AGE_BOUND_STAMP_FROM_CONSTRAINT_ROWS: + return ( + f"{key}={value} restates an age bound whose operator is ambiguous: " + f"{AGE_BOUND_STAMP_SOURCE_KEY}={stamp_source} does not attest it " + "was stamped from a constraint row (>= or <), and a dimension of " + "that name carries no operator" + ) + return _restated_band_edge_refusal( + key, value, metadata, compiled_key=RESTATED_AGE_BAND_COMPILED_KEYS[side] + ) + + def _restated_eitc_child_count_refusal( key: str, value: str, @@ -4613,12 +4732,13 @@ def _restated_ledger_filter_refusal( changes nothing. Every other key returns the entry to refuse with: the bare key for a filter the materializer does not model (unchanged from before this rule existed), or the key with both values when a restatement - disagrees with its compiled counterpart, or has none, or — for a + disagrees with its compiled counterpart, or has none, or is applied by + no materializer for this spec, or — for a qualifying-child upper bound, whose operator is unconfirmed (see :data:`RESTATED_EITC_CHILD_COUNT_REFUSED_SIDES`) — the key with that reason, whatever the spec compiles. - Both compiled counterparts — the AGI band in + Two compiled counterparts — the AGI band in :data:`RESTATED_AGI_BAND_COMPILED_KEYS` and :func:`_soi_eitc_child_count_filter` — are the ``irs_soi`` slice's, read in the loop :func:`_materialize_target_frame` runs over ``irs_soi`` @@ -4627,6 +4747,17 @@ def _restated_ledger_filter_refusal( ``ledger_filter_eitc_child_count`` is likewise a blanket supported key. A spec of another family carries neither counterpart, so it refuses on the "does not compile" arm rather than being accepted by accident. + + The third, the person-age band in :data:`RESTATED_AGE_BAND_COMPILED_KEYS`, + is read by two materializers (``population_age`` and the age-banded + ``policyengine_variable`` branch), so that rule is family-blind too but + names the materializers that apply the band + (:data:`RESTATED_AGE_BAND_MATERIALIZERS`) and refuses on every other; it + also refuses a bound on a fact whose dimensions include ``age`` + (:data:`RESTATED_AGE_DIMENSION_KEY`), and any bound the compile does not + attest was stamped from a ``>=`` / ``<`` constraint row + (:data:`AGE_BOUND_STAMP_SOURCE_KEY`): a dimension-stamped bound carries no + operator, and ``age < 10`` and ``age <= 10`` select different people. """ concept, side = _restated_ledger_filter_concept(key) @@ -4637,6 +4768,8 @@ def _restated_ledger_filter_refusal( return _restated_agi_band_refusal(key, value, metadata, side=side) if rule == "eitc_child_count": return _restated_eitc_child_count_refusal(key, value, metadata, side=side) + if rule == "age_band": + return _restated_age_band_refusal(key, value, metadata, side=side) raise ValueError( f"RESTATED_LEDGER_FILTER_CONCEPTS maps {concept!r} to rule {rule!r}, " "which has no comparison implemented."