Accept a restated age-band Ledger filter only where it agrees with the compiled bounds

changelog.d/us-labelled-restated-age-band-filters.added.md

@@ -0,0 +1 @@
+The US fiscal materializer now accepts `ledger_filter_age_lower_bound` / `ledger_filter_age_upper_bound` — the Ledger stamp of a fact's `age >=` / `age <` rows — beside the compiled `age_lower_bound` / `age_upper_bound`, but only where the two edges are equal and the spec's materializer (`population_age`, or the age-banded `policyengine_variable` branch the SSA SSI by-age counts use) actually slices on that band. On the pinned Chronicle feed this clears the 939 Census population-by-age and SSA SSI age-band targets the guard refused over the whole compiled registry. A restated edge that disagrees, has no compiled counterpart, sits on a spec whose materializer applies no age band, or comes from a fact with `age` as a dimension is refused and names both values; so is any restated edge the compile does not attest was stamped from a `>=` / `<` constraint row (the new `age_bound_stamp_source` metadata on every compiled age-band spec reads `constraint_rows` unless the fact has a dimension named `age` or `age` plus a bound suffix (`age_upper_bound`, …), whose operator-less value the Ledger stamp can write under the same key); an exact-age restatement is refused; and `<=` / `>` rows (stamped `_upper_bound_inclusive` / `_lower_bound_exclusive`) stay refused by bare key, since the compiled band drops the operator.

experiments/us-labelled-filter-support/REPORT.md

@@ -596,3 +596,87 @@ $ .venv/bin/python tools/ci_test_groups.py --verify     verification=ok, exit 0
 $ ruff check / ruff format --check on the four changed .py files
 All checks passed! / 4 files already formatted
 ```
+
+## 9. 2026-09-22: the age-band follow-up (Max's ruling 2)
+
+Section 7's "still open" item is closed on branch
+`us-restated-age-band-filters`, stacked on this branch at `01dc806f4`. It
+adds `age` to `RESTATED_LEDGER_FILTER_CONCEPTS` under a third rule,
+`age_band`, in `tools/build_us_fiscal_refresh_release.py`.
+
+**What the materializer applies, read at `01dc806f4`.** Both age paths in
+`_materialize_target_frame` read `age_lower_bound` / `age_upper_bound`
+through `_as_bound` into `lower <= age < upper`: `population_age` in
+`_population_age_household_values`, and the `policyengine_variable` branch
+entered whenever either key is present (the SSA SSI by-age counts). The
+compiled bounds come from `us_runtime.fiscal_targets._age_bounds`, which
+takes the value of any `>`/`>=` row as the lower edge and any `<`/`<=` row as
+the upper edge, **dropping the operator**. The restated key keeps it:
+`ledger_targets._constraint_bound_filters` stamps `age_lower_bound` only for
+`>=` and `age_upper_bound` only for `<`, and `>`/`<=` as
+`_lower_bound_exclusive` / `_upper_bound_inclusive`.
+
+**The rule.** A restated `age_{lower,upper}_bound` is accepted only when it
+parses to the same number as the compiled edge on its own side, the spec's
+materializer is `population_age` or `policyengine_variable`, and the compile
+attests that the key was stamped from a `>=` / `<` constraint row
+(`age_bound_stamp_source=constraint_rows`; see "The operator is attested at
+compile" below). Refused, naming the values: a disagreeing edge, no compiled
+counterpart, an exact-age key (`ledger_filter_age=<v>`), any other
+materializer, a spec carrying `ledger_filter_age` at all (age is then a
+dimension, so `_constraint_bound_filters` stamped none of the age rows and the
+bound came from the operator-less dimension stamp), and any bound without that
+attestation. `_exclusive` / `_inclusive` keys
+are not restated concepts and keep the bare-key refusal whatever their value.
+A test that runs real facts through `_dynamic_us_fiscal_target_references`
+and `compile_ledger_target_references` shows why that matters: an
+`age <= 9` row compiles to `age_upper_bound=9`, which the materializer reads
+as `age < 9`. The AGI and age rules share one edge comparison,
+`_restated_band_edge_refusal`.
+
+**The pinned feeds, measured.** Both exports carry 1,493 facts with an age
+constraint. Every row is `role: filter`, an integer value, `>=` or `<`, at
+most one per side, and no fact has an age-like dimension key. Counted by a
+throwaway scan of the two JSONL files, not committed.
+
+**Counts.** From `age_band_rule_receipt.json`, written by the committed
+`measure_age_band_rule.py` (receipt `script_sha256` `db5dcb68…`). It ran at
+`622486abb` (the operator-attestation commit) with a clean tree, took
+14.1 s and 15.6 s to compile, and peaked at 2.9 GB RSS. The whole compiled registry
+(`compile_us_fiscal_target_registry(..., age_targets=True)`) has 32,866
+targets on each feed:
+
+| Arm | Refused targets | Refusal entries | SOI silent skips |
+|---|---:|---:|---:|
+| `rule` (this follow-up) | **0** | 0 | 0 |
+| `without_age_band` (#969 alone) | **939** | 1,825 | 0 |
+| `reverted` (`main`'s behaviour) | 3,208 | 5,116 | 2,269 |
+
+The two feeds (`b8543739…` and `4d1dba8c…`) give identical counts. The 939
+are the targets carrying a restated age key: 936 `census_population` /
+`population_age` / `population_age` and 3 `ssa` /
+`ssa_ssi_age_band_recipients` / `policyengine_variable`. The only
+`ledger_filter_age*` keys present are `_lower_bound` (939) and
+`_upper_bound` (886). The `without_age_band` and `reverted` rows equal the
+section 8 receipt's `rule` and `reverted` rows.
+
+### The operator is attested at compile
+
+A restated `ledger_filter_age_{lower,upper}_bound` key carries the operator of
+the mask only when `_constraint_bound_filters` stamped it from a `>=` / `<`
+constraint row. `_ledger_metadata` stamps the fact's dimensions first and the
+constraint edges only through `setdefault`, so a dimension named `age`, or
+`age` plus a bound suffix (`age_upper_bound`, …), can put an operator-less
+value under the same key; and `age < 10` and `age <= 10` differ by everyone
+aged exactly ten. The key name alone therefore does not settle the operator.
+
+The compile now settles it. `ledger_targets.constraint_bound_shadowing_dimensions`
+names any such dimension on a fact, and `fiscal_targets` records the answer on
+every compiled `population_age` and SSA SSI age-band spec as
+`age_bound_stamp_source`: `constraint_rows` when no dimension could have
+shadowed the age rows' stamp, otherwise `dimensions:<names>`. The builder's
+restated age-band rule accepts a restatement only when that key reads
+`constraint_rows`; anything else is refused with the reason. Measured on both
+pinned feeds (receipt above, `by_age_bound_stamp_source`): all 939 targets
+carrying a restated age key read `constraint_rows`, so the attestation refuses
+none of them today, and the `rule` row stays at 0.

experiments/us-labelled-filter-support/age_band_rule_receipt.json

@@ -0,0 +1,123 @@
+{
+  "head": "622486abbe65c06ea348c5d4885193c073721344",
+  "worktree_status": [],
+  "script_sha256": "db5dcb680ac1af0c55dad996f0f1abe33b6ef78e5e9b19cb02f0b7551ec9a143",
+  "target_period": 2024,
+  "max_rss_bytes": 2875359232,
+  "results": [
+    {
+      "feed": "/Users/maxghenis/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl",
+      "feed_sha256": "b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801",
+      "compile_seconds": 14.1,
+      "targets": 32866,
+      "age_key_census": {
+        "targets_carrying_restated_age_key": 939,
+        "by_family_role_materializer": {
+          "census_population|population_age|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients|policyengine_variable": 3
+        },
+        "by_age_bound_stamp_source": {
+          "constraint_rows": 939
+        },
+        "ledger_filter_age_keys": {
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        }
+      },
+      "rule": {
+        "refused_targets": 0,
+        "refusal_entries": 0,
+        "refusal_entries_by_key": {},
+        "refused_by_family_role": {},
+        "soi_silently_skipped_targets": 0
+      },
+      "without_age_band": {
+        "refused_targets": 939,
+        "refusal_entries": 1825,
+        "refusal_entries_by_key": {
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        },
+        "refused_by_family_role": {
+          "census_population|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients": 3
+        },
+        "soi_silently_skipped_targets": 0
+      },
+      "reverted": {
+        "refused_targets": 3208,
+        "refusal_entries": 5116,
+        "refusal_entries_by_key": {
+          "ledger_filter_us.tax.earned_income_credit_qualifying_children_lower_bound": 1137,
+          "ledger_filter_us:statutes/26/62#adjusted_gross_income_lower_bound": 1080,
+          "ledger_filter_us:statutes/26/62#adjusted_gross_income_upper_bound": 1074,
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        },
+        "refused_by_family_role": {
+          "irs_soi|soi_fiscal_distribution": 2269,
+          "census_population|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients": 3
+        },
+        "soi_silently_skipped_targets": 2269
+      }
+    },
+    {
+      "feed": "/Users/maxghenis/PolicyEngine/_buildh-runtime/inputs/chronicle_us_b571381/artifact/consumer_facts.jsonl",
+      "feed_sha256": "4d1dba8c1b6274877bf184fa6de5d99b13fc61f34709ccab1487db2b5c64a79f",
+      "compile_seconds": 15.6,
+      "targets": 32866,
+      "age_key_census": {
+        "targets_carrying_restated_age_key": 939,
+        "by_family_role_materializer": {
+          "census_population|population_age|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients|policyengine_variable": 3
+        },
+        "by_age_bound_stamp_source": {
+          "constraint_rows": 939
+        },
+        "ledger_filter_age_keys": {
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        }
+      },
+      "rule": {
+        "refused_targets": 0,
+        "refusal_entries": 0,
+        "refusal_entries_by_key": {},
+        "refused_by_family_role": {},
+        "soi_silently_skipped_targets": 0
+      },
+      "without_age_band": {
+        "refused_targets": 939,
+        "refusal_entries": 1825,
+        "refusal_entries_by_key": {
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        },
+        "refused_by_family_role": {
+          "census_population|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients": 3
+        },
+        "soi_silently_skipped_targets": 0
+      },
+      "reverted": {
+        "refused_targets": 3208,
+        "refusal_entries": 5116,
+        "refusal_entries_by_key": {
+          "ledger_filter_us.tax.earned_income_credit_qualifying_children_lower_bound": 1137,
+          "ledger_filter_us:statutes/26/62#adjusted_gross_income_lower_bound": 1080,
+          "ledger_filter_us:statutes/26/62#adjusted_gross_income_upper_bound": 1074,
+          "ledger_filter_age_lower_bound": 939,
+          "ledger_filter_age_upper_bound": 886
+        },
+        "refused_by_family_role": {
+          "irs_soi|soi_fiscal_distribution": 2269,
+          "census_population|population_age": 936,
+          "ssa|ssa_ssi_age_band_recipients": 3
+        },
+        "soi_silently_skipped_targets": 2269
+      }
+    }
+  ]
+}