Pool each state's congressional-district rows of one concept into one national loss budget

docs/evidence/spec-engine/us-f0-coverage.json

@@ -6,7 +6,7 @@
   "country": "us",
   "documentation_sha256": "0931cbd3aae2abcd8c072b78eba40c1c006b1bf7fa23c6a6681f9a895e47070e",
   "field_usage": {
-    "authored_normative_field_count": 32404,
+    "authored_normative_field_count": 32407,
     "claim_count": 49,
     "claims": [
       {
@@ -108,8 +108,8 @@
         ],
         "mode": "legacy_behavior",
         "pointer_class": "all",
-        "pointer_count": 312,
-        "pointer_sha256": "9bad55b4945af1c4213c510942e7b9d22d104bf75b0d9804e4a183e1656c8393",
+        "pointer_count": 315,
+        "pointer_sha256": "1d49908235366d9dead0be600b44accc05148f027f3c56dde2c3413f3ade5cce",
         "rationale": null,
         "relative_sink_prefix": "/calibration_contract",
         "source_prefix": "/authored/spec~1calibration.yaml",
@@ -772,20 +772,20 @@
         "verifier": "vintages"
       }
     ],
-    "configuration_field_count": 42184,
-    "consumed_field_count": 42184,
+    "configuration_field_count": 42187,
+    "consumed_field_count": 42187,
     "generation0_effect_counts": {
-      "legacy_behavior": 38498,
+      "legacy_behavior": 38501,
       "no_generation0_effect": 3686
     },
     "mode_counts": {
       "compiler_semantic": 27717,
       "front_end_validation": 354,
       "identity_only": 103,
-      "legacy_behavior": 14010
+      "legacy_behavior": 14013
     },
     "multiple_primary_use_field_count": 0,
-    "pointer_inventory_sha256": "8f655670b9794adae905ede64bdceba6395eeb8d642b6c5896d5c516f1f6a6a7",
+    "pointer_inventory_sha256": "bc18b1e1715ecc9b3e9034d6dc1a969f87409c75cf346ff710fea06aa8bc94a3",
     "resolved_binding_field_count": 9780,
     "unused_field_count": 0
   },
@@ -2599,7 +2599,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "fbf25dbae68242198972f5bef4f1fc20f65ee0769615b8d94639e6a80121f31f"
+      "spec_sha256": "fe6d751d47cfdaf2541a9be48ae8bd026d21e0386d7644972f024f422c691e36"
     }
   },
   "report_schema_version": 3,
@@ -2609,7 +2609,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "fbf25dbae68242198972f5bef4f1fc20f65ee0769615b8d94639e6a80121f31f"
+    "spec_sha256": "fe6d751d47cfdaf2541a9be48ae8bd026d21e0386d7644972f024f422c691e36"
   },
   "status": "pass"
 }

experiments/us-acs-local-target-loss-weights-20261004/README.md

@@ -158,6 +158,18 @@ no default multiplier.
 
 ## A finding for the national release (not changed here)
 
+> **Fixed 2026-10-12** in `experiments/us-national-cd-concept-budget-20261012/`.
+> The national exclusions now list the compiler's per-row labels, and a guard
+> refuses a split district concept. On the current pinned feed (`965a29ac…`),
+> the 24,340 district rows form 3,125 groups and carry 11.0% of the loss
+> (45.6% before). The numbers below are from the 10-04 feed and the code at
+> that date. `national_cd_groups.py` now reports the fixed mapping under
+> `national_as_is`. On the `state_cd` surface, `distribution.py`'s
+> "national mapping as is" column would now be refused: the national guard
+> rejects the rebase's per-district `state_cd_cd_file_value`, which only the
+> ACS local key excludes. That section already refuses to run, because its
+> feed (`c5e5bf8`) is no longer the pin. The tables above are the record.
+
 `national_cd_groups.py` compiles the national registry from the pinned feed
 (`compile_us_fiscal_target_registry(age_targets=True)`) and keeps its 24,340
 district rows, all SOI. The national parser's default, `--target-surface

experiments/us-national-cd-concept-budget-20261012/README.md

@@ -0,0 +1,174 @@
+# National concept budget for congressional-district rows
+
+The national fiscal release weights each target's loss with
+`sqrt_value_concept_budget_weighted_mape_50_50_amount_count_target_scale_cap_100pct`
+(`microcosm.build.us_runtime.target_loss_weights`). Step 3 of that formula is
+the concept budget from microcosm#195 (June 2026): the congressional-district
+targets of one concept in one state form one group, which is rescaled to sum
+to its largest member's weight, so adding district geography cannot multiply
+a concept's weight.
+
+On the full surface it pooled nothing. The district key reads every metadata
+key outside `US_FISCAL_TARGET_CONCEPT_METADATA_EXCLUSIONS`. Commit efa6052c9
+(2026-09-05, "Normalize calibration hierarchy ownership") began stamping three
+per-row labels on every compiled row: `ledger_fact_label`,
+`ledger_layout_groupby_value_label` and `diagnostic_target_label`. The first
+two name the district ("WV congressional district 1"). The third repeats the
+fact label when the fact's period is the build's, which a district row on the
+pinned feed does not meet (its facts are TY2023, the build 2024), but a 2023
+build would. None of the three was excluded, so every compiled district row
+became its own group.
+
+The ACS local mapping (microcosm#1104) already excluded the first two for its
+own district rows. That PR measured the national gap and left the national
+mapping unchanged on purpose
+(`experiments/us-acs-local-target-loss-weights-20261004/`, "A finding for the
+national release").
+
+## The change
+
+- `LEDGER_PER_ROW_LABEL_METADATA` names the three labels, and the national
+  exclusions now include them. The ACS local key inherits them. Its own
+  addition is now only `state_cd_cd_file_value`, so its exclusion set is the
+  one it had plus `diagnostic_target_label`.
+- `validate_district_concept_groups` now guards the national mapping
+  (`us_fiscal_ledger_metadata.v2`), as the ACS local mapping's guard already
+  did for its rows. It refuses a surface where:
+  - the district rows of one concept in one state (`district_concept_identity`)
+    land in more than one group, naming the metadata keys that split them;
+  - a district group spans states;
+  - a district row has no `state_fips`.
+
+  So a new per-row key fails the weights loudly instead of silently
+  disabling the budget again.
+- The formula id (`US_FISCAL_TARGET_LOSS_WEIGHTING`) is unchanged. It names
+  the formula, and Route A's recorded weights (`national_state`, no district
+  rows) are reproduced bit for bit. On the full surface the weights move, so
+  the loss vector digest in `target_loss_basis` moves. The exact-k
+  incumbent gate compares that digest, so it refuses to compare a run
+  weighted the old way with one weighted the new way.
+
+## Results on the pinned feed
+
+`national_cd_concept_budget.py` compiles the pinned feed
+(`facts_sha256` `965a29ac…`, Chronicle `51aa40f`) at period 2024 with aging,
+which is what `--target-surface full` calibrates. It computes the weights with
+the pre-change code (the verbatim copy in `test_support`) and with the
+current module. Output: `results.json`.
+
+The surface has 33,249 compiled targets. 24,340 of them (73.2%) are district
+rows, all `irs_soi`. They form 3,125 concept-in-state identities:
+
+- 55 concepts from the district file in each state with districts;
+- 40 Historic Table 2 proxy concepts in each of eight states (the seven
+  single-district states and Montana).
+
+| District concept groups | Groups | Singletons | Largest |
+|---|---:|---:|---:|
+| Before | 24,340 | 24,340 | 1 |
+| After | 3,125 | 665 | 52 (California) |
+
+After the change, each group is exactly one identity: no identity is split
+and no group pools two. The 665 singletons are the seven single-district
+states (AK, DE, DC, ND, SD, VT, WY), 95 concepts each (55 from the district
+file and 40 Historic Table 2 proxies). Each of those concepts has one district
+row in its state, so it has nothing to pool.
+
+### Share of the loss (each row's weight over the total; equal misses)
+
+| Level | Targets | Equal weights | Before | After |
+|---|---:|---:|---:|---:|
+| Congressional district | 24,340 | 73.2% | 45.6% | 11.0% |
+| State | 8,409 | 25.3% | 42.1% | 68.8% |
+| Country | 500 | 1.5% | 12.3% | 20.2% |
+
+By basis, district amounts go from 23.0% to 5.8% and district counts from
+22.6% to 5.2%. Counts and amounts each still carry half of the total.
+
+By family, `irs_soi` goes from 90.5% to 84.5%. Every other family gains by
+the same factor as its rows, for example `census_population` 4.8% → 7.9% and
+`cms_medicaid` 1.2% → 1.9%.
+
+### What happens to each weight
+
+- **Every non-district target keeps its weight up to one rescale per basis:**
+  ×1.6373 for amounts and ×1.6332 for counts. Its group, its value weight and
+  its rank among the non-district rows of its basis are unchanged. The
+  district rows no longer crowd it out.
+- **A district row's weight is scaled by its group's budget over the group's
+  total, then by the basis rescale.**
+  - The median district weight is 0.157 of what it was. The smallest ratio,
+    0.033, falls on rows in California's 52-district groups.
+  - A single-district state's rows keep their weight up to the basis rescale
+    (ratio 1.633 to 1.637).
+  - The districts of one concept in one state now carry together what their
+    largest district would carry alone.
+- **Weights still have mean 1.** Their range moves from [2.0e-5, 263] to
+  [3.3e-5, 430]. The ten largest weights are all national rows (the largest
+  is the one `federal_reserve` target), which gain the basis rescale.
+
+### What it does not change
+
+- **Route A** and any `--target-surface national_state` run. Those surfaces
+  have no district rows, and a non-district row's key never reads metadata.
+  `test_route_a_weights_and_loss_basis_hash_are_reproduced_bit_for_bit` still
+  passes against the recorded weights and the loss basis hash `206ada09…`.
+- **The ACS local weights.** Its key already excluded the two district labels.
+  The one addition, `diagnostic_target_label`, cannot change its grouping:
+  - its guard already requires each district identity to be one group;
+  - two identities differ in metadata the key still reads;
+  - a non-district row's key reads no metadata.
+
+  Measured, not only argued: `acs_local_unchanged.py` compares the ACS local
+  weights from the pre-change module (51a263726) and from this one, bit for
+  bit, and finds them identical on:
+  - the 09-23 release registry, 4,459 targets (its digest is still the
+    recorded `ba36f887…`);
+  - each `--soi-mode` surface of the pinned feed, with the 09-23 ladder
+    population rows:
+
+    | Surface | Targets | District ledger rows |
+    |---|---:|---:|
+    | `state` | 4,867 | 0 |
+    | `totals` | 1,247 | 454 |
+    | `full` | 31,961 | 24,340 |
+    | `state_cd` | 26,912 | 21,743 |
+
+  No district row on those surfaces carries `diagnostic_target_label`
+  (`acs_local_unchanged.json`). The ACS local tests pass unchanged.
+- **Target values, the compiled surface, and any published release.** This
+  changes only the objective of future `--target-surface full` calibrations.
+
+## The CI fixture
+
+The script also writes
+`packages/microcosm-build/tests/fixtures/us_national_district_concept_rows.json`.
+It holds 31 real compiled district rows from the pinned feed, with the
+metadata the compiler stamped:
+
+- seven concepts in West Virginia (two districts), Montana (two districts and
+  Historic Table 2 proxies) and Wyoming (at large and proxies);
+- the concepts are AGI, return count, EITC two-children claims, itemized
+  deductions and net capital gains from the district file, plus return count
+  and net capital gains from Historic Table 2.
+
+The tests use it on CI, where the feed is not available.
+
+## Reproduce
+
+From the repository root, with the pinned feed at
+`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_51aa40f.jsonl`:
+
+```bash
+uv run python experiments/us-national-cd-concept-budget-20261012/national_cd_concept_budget.py
+uv run python experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.py \
+    experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.json
+```
+
+The second also needs the 09-23 registry that
+`experiments/us-acs-local-target-loss-weights-20261004/distribution.py` reads.
+
+The script refuses a feed whose sha256 is not the pin. Where the feed is on
+disk, `test_pinned_feed_district_rows_form_one_budget_per_concept_per_state`
+in `packages/microcosm-build/tests/engine_free/us/test_us_target_loss_weights.py`
+asserts these counts and shares on every run. Elsewhere it skips.

experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.json

@@ -0,0 +1,32 @@
+{
+ "release_20260923": {
+  "n": 4459,
+  "bit_identical": true,
+  "new_sha256": "ba36f887407181dc6641596b4a200e6b18017cd815d7806ac4f523da3204f1d1",
+  "recorded_sha256": "ba36f887407181dc6641596b4a200e6b18017cd815d7806ac4f523da3204f1d1"
+ },
+ "state": {
+  "n": 4867,
+  "district_ledger_rows": 0,
+  "district_rows_with_diagnostic_label": 0,
+  "bit_identical": true
+ },
+ "totals": {
+  "n": 1247,
+  "district_ledger_rows": 454,
+  "district_rows_with_diagnostic_label": 0,
+  "bit_identical": true
+ },
+ "full": {
+  "n": 31961,
+  "district_ledger_rows": 24340,
+  "district_rows_with_diagnostic_label": 0,
+  "bit_identical": true
+ },
+ "state_cd": {
+  "n": 26912,
+  "district_ledger_rows": 21743,
+  "district_rows_with_diagnostic_label": 0,
+  "bit_identical": true
+ }
+}

experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.py

@@ -0,0 +1,90 @@
+"""ACS local weights before and after this change, bit for bit, on real surfaces.
+
+Loads the pre-change ``target_loss_weights.py`` (at ``BASE``) as a separate
+module and compares ``us_acs_local_target_loss_weights`` with the current
+module's on the 09-23 release registry (whose recorded digest is
+``ba36f887…``) and on each ``--soi-mode`` surface of the pinned feed, with
+the 09-23 ladder population rows. Run from the repository root, with the
+pinned feed and the 09-23 registry on disk::
+
+    uv run python experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.py \
+        experiments/us-national-cd-concept-budget-20261012/acs_local_unchanged.json
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+import types
+from pathlib import Path
+
+import numpy as np
+
+REPO = Path.cwd()
+#: The commit before this change (origin/main when it was written).
+BASE = "51a263726"
+sys.path.insert(0, str(REPO / "experiments/us-acs-local-target-loss-weights-20261004"))
+import distribution as dist  # noqa: E402
+
+from microcosm.build.us_runtime import target_loss_weights as new  # noqa: E402
+from microcosm.calibrate import TargetRegistry  # noqa: E402
+
+source = subprocess.run(
+    [
+        "git",
+        "show",
+        f"{BASE}:packages/microcosm-build/src/microcosm/build/us_runtime/target_loss_weights.py",
+    ],
+    capture_output=True,
+    text=True,
+    check=True,
+    cwd=REPO,
+).stdout
+old = types.ModuleType("target_loss_weights_main")
+old.__dict__["__name__"] = "target_loss_weights_main"
+sys.modules["target_loss_weights_main"] = old
+exec(compile(source, "target_loss_weights_main.py", "exec"), old.__dict__)
+
+out = {}
+release = TargetRegistry.from_json(dist.DEFAULT_REGISTRY).specs
+a = old.us_acs_local_target_loss_weights(release)
+b = new.us_acs_local_target_loss_weights(release)
+names = [new.target_row_name(s) for s in release]
+out["release_20260923"] = {
+    "n": len(release),
+    "bit_identical": bool(np.array_equal(a, b)),
+    "new_sha256": new.target_loss_weights_sha256(names, b),
+    "recorded_sha256": "ba36f887407181dc6641596b4a200e6b18017cd815d7806ac4f523da3204f1d1",
+}
+print(json.dumps(out["release_20260923"]))
+
+tool = dist.load_tool()
+feed = (
+    Path.home() / "PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_51aa40f.jsonl"
+)
+population = [s for s in release if s.family == "census_population"]
+for mode in ("state", "totals", "full", "state_cd"):
+    surface = tool.state_admin_surface(feed, dist.FAMILIES, soi_mode=mode)
+    specs = [*surface.registry.specs, *population]
+    a = old.us_acs_local_target_loss_weights(specs)
+    b = new.us_acs_local_target_loss_weights(specs)
+    district = sum(
+        1
+        for s in specs
+        if s.metadata.get("ledger_geography_level") == "congressional_district"
+    )
+    diag = sum(
+        1
+        for s in specs
+        if s.metadata.get("ledger_geography_level") == "congressional_district"
+        and "diagnostic_target_label" in s.metadata
+    )
+    out[mode] = {
+        "n": len(specs),
+        "district_ledger_rows": district,
+        "district_rows_with_diagnostic_label": diag,
+        "bit_identical": bool(np.array_equal(a, b)),
+    }
+    print(mode, json.dumps(out[mode]))
+Path(sys.argv[1]).write_text(json.dumps(out, indent=1) + "\n")