diff --git a/changelog.d/259-binding-local-views.added.md b/changelog.d/259-binding-local-views.added.md new file mode 100644 index 0000000000..8c6873a899 --- /dev/null +++ b/changelog.d/259-binding-local-views.added.md @@ -0,0 +1,3 @@ +Rules bindings can declare binding-local views (S2b PR 1, microcosm#259). A view names an engine entity no frame entity maps to (rulespec-be's `Child`), one bool person selector column, and an outside value for each of its outputs: `{"logical_entity": "Child", "selector": {"column": "is_child"}, "outside": {"child_benefit": 0.0}}`. The Axiom adapter (`AxiomView`) gathers the selected person rows in table order, runs the module's program on the logical entity, and writes one value to every person row, the declared outside value on each row the selector leaves out. View outputs resolve to `person`, so composition, export, holdout and precalibration read them unedited. + +The registry refuses an empty list, views beside relations, a logical entity that names a mapped entity, two views on one logical entity, a selector with any field but `column`, and outside keys that differ from the binding's variables on the logical entity. Compose requires each selector on a whole person slice, and a view node's slice must carry every input the closure encodes or defaults on the logical entity. The input closure partitions each view's roots on its logical entity instead of refusing the binding. An absent `views` block adds no key to `axiom_engine_ref`, so no existing reference moves; a golden of NZ-shaped and relation references from the base checks that byte for byte. diff --git a/docs/transport-declared-units.md b/docs/transport-declared-units.md index 07b455a7af..1708affa70 100644 --- a/docs/transport-declared-units.md +++ b/docs/transport-declared-units.md @@ -81,7 +81,8 @@ relation declaration. The expected related input map must match the union of compiled schemas sharing that key. `one` means exactly one selected person per root; `many` permits zero. A role selector requires both its column and distinct role values. This interface supports person members of genuine group partitions; -binding-local views and arbitrary links remain separate work. +arbitrary links remain separate work. Binding-local views (below) run an engine +entity that has no frame table. Rules nodes declare whole-table slices for selector columns and every mapped related input. Tables always carry their IDs and memberships, and no slice can @@ -104,3 +105,63 @@ declaration remains absent; an explicitly supplied empty declaration is distinct Bindings without relations retain their existing engine-reference serialization and native execution call shape. Calibration roster counts use the same frame unit framework, with frozen-kernel differential tests preserving NZ results. + +## Binding-local views + +A binding on the person entity may run its module's program on an engine +entity that no frame entity maps to, such as rulespec-be's `Child`, by +declaring `views`: + +```json +"views": [ + { + "logical_entity": "Child", + "selector": {"column": "is_child"}, + "outside": {"child_benefit": 0.0} + } +] +``` + +The adapter (`AxiomView`) gathers the person rows whose bool `selector` +column holds true, in table order, runs the program on `logical_entity` over +them, and scatters each output back through the same mask. Every other row +gets the view's `outside` value for that output. Each binding variable on the +logical entity needs an outside value, with no implicit fill, and the adapter +refuses one that doesn't cast to the output's graph dtype without loss. A +view's root inputs read the person columns of the same names, and its outputs +resolve to the person entity, so downstream kernels read them as person +columns. An empty selection makes no engine call, and a null or non-bool +selector refuses. An input override of a selector changes which rows each +view selects; the adapter accepts one only when a program reads the selector +as a root input, as bridges require of any overridden column, and otherwise +refuses it by the selector's name. Gather and scatter assume that the engine computes each +selected row from that row's inputs alone. Views refuse programs with dense +relations, and a real-engine differential, skip-gated on the dense extension and a +rulespec-be tree, is written to test the assumption. + +The registry refuses `views: []`, views beside `relations`, views on a binding +on another entity than the person entity, a logical entity that a frame entity +maps to, two views on one logical entity, a selector that names a structural +column or a binding variable, and an outside key that names no binding +variable. Once the adapter pins its RuleSpec bytes, the registry also refuses +dense relations on a view's program or on the binding entity's program, outside +keys that differ from the binding's variables on the logical entity, and a +view input that shares its name with a root input of the binding entity's +program, since both would read one person column. Views need +`output_dtypes="graph"`, and an injected Python adapter can't run them. + +A rules node slices each view's selector over the whole person table, as it +does a relation's selector columns; composition and every rules kernel apply +the same check. When the spec carries an input closure, composition also +refuses a view node whose person slice omits an input that the closure encodes +(other than through a graph node) or defaults on the logical entity, and a +closure producer whose person slice omits a selector. The closure counts a +view's roots as live on its logical entity and checks them against the person +table; `unreached_roots` still excuses only roots on the binding's engine +entity. The support roster classes a defaulted root on a view's logical entity +as a person column. + +A binding without `views` keeps its semantics, engine reference and native +execution call. The engine reference carries `views` only when the binding +declares them, with the views list in declaration order and each +`outside` mapping in sorted key order. diff --git a/packages/microcosm-build/src/microcosm/build/transport/compose.py b/packages/microcosm-build/src/microcosm/build/transport/compose.py index d84167e60f..d167acb75f 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/compose.py +++ b/packages/microcosm-build/src/microcosm/build/transport/compose.py @@ -134,6 +134,7 @@ WEIGHTS_AUDIT_REF, WEIGHTS_AUDIT_TYPE, binding_relation_inputs, + binding_view_entities, ) from .money_kernels import ( MONEY_ANNUAL_TRANSFERS_REF, @@ -141,7 +142,7 @@ MONEY_NORMALIZE_REF, ) from .reference_selection import selection_resources, validate_exemptions -from .registry import binding_relation_dependencies +from .registry import binding_relation_dependencies, binding_view_dependencies from .target_kernels import TARGET_COMPILATION_KERNELS from .unit_declarations import ( CREATE_REFS, @@ -2274,6 +2275,41 @@ def _source_codecs(spec: Mapping[str, object]) -> dict[str, object]: return out +def _closure_defaults(spec: Mapping[str, object]) -> dict[tuple, str]: + """(frame entity, input) -> the closure entry of each defaulted root. + + A defaulted root on a binding-local view's logical entity is a person + column, because the view reads the person table; every other entity + maps through ``entity_names``. + """ + defaults: dict[tuple, str] = {} + if not ( + _has_resource(spec, _CLOSURE_RESOURCE) + and _has_resource(spec, _BINDINGS_RESOURCE) + ): + return defaults + names = transport_resource(spec, _BINDINGS_RESOURCE).get("entity_names", {}) + to_frame = {engine: frame for frame, engine in names.items()} + view_entities = { + (row.get("rulespec_path"), entity) + for row in _binding_modules(spec).values() + for entity in binding_view_entities(row) + } + person = _person_entity(spec) if view_entities else None + for entry in transport_resource(spec, _CLOSURE_RESOURCE).get("entries", ()): + if isinstance(entry, Mapping) and entry.get("class") == "defaulted": + frame_entity = ( + person + if (entry.get("rulespec_path"), entry.get("entity")) in view_entities + else to_frame.get(entry.get("entity")) + ) + defaults[(frame_entity, entry.get("input"))] = ( + f"{entry.get('rulespec_path')} {entry.get('entity')}." + f"{entry.get('input')}" + ) + return defaults + + def _support_roster(spec: Mapping[str, object]) -> dict[str, object]: """Every column the support gate covers, classed by its producer in lineage. @@ -2321,19 +2357,7 @@ def lineage(version: object) -> list[str]: _raw_transport_resource(spec, MONEY_NORMALIZATION_RESOURCE) ) ) - defaults: dict[tuple, str] = {} - if _has_resource(spec, _CLOSURE_RESOURCE) and _has_resource( - spec, _BINDINGS_RESOURCE - ): - names = transport_resource(spec, _BINDINGS_RESOURCE).get("entity_names", {}) - to_frame = {engine: frame for frame, engine in names.items()} - for entry in transport_resource(spec, _CLOSURE_RESOURCE).get("entries", ()): - if isinstance(entry, Mapping) and entry.get("class") == "defaulted": - frame_entity = to_frame.get(entry.get("entity")) - defaults[(frame_entity, entry.get("input"))] = ( - f"{entry.get('rulespec_path')} {entry.get('entity')}." - f"{entry.get('input')}" - ) + defaults = _closure_defaults(spec) reserved = [] for row in rows: for item in _sequence( @@ -2510,13 +2534,13 @@ def _person_entity(spec: Mapping[str, object]) -> str: def _check_closure_relation_slices(spec: Mapping[str, object], graph: Graph) -> None: - """A closure producer's person slice carries every declared relation column. + """A closure producer's person slice carries every relation and view column. A ``Slice`` delivers only the columns it names, and ``transport.input_closure@1`` reads each relation input's physical ``column`` and declared ``role_column`` on the person entity. A slice that omits one would report a ``gap`` the frame does not have, so composition - refuses it by name. + refuses it by name. Each binding-local view's selector is required too. """ nodes = [node for node in graph.nodes if node.kernel == INPUT_CLOSURE_REF] @@ -2538,6 +2562,15 @@ def _check_closure_relation_slices(spec: Mapping[str, object], graph: Graph) -> for column in (item.column, item.role_column) if column is not None } + | { + view["selector"]["column"] + for row in (rows if isinstance(rows, list) else ()) + if isinstance(row, Mapping) and isinstance(row.get("views"), list) + for view in row["views"] + if isinstance(view, Mapping) + and isinstance(view.get("selector"), Mapping) + and isinstance(view["selector"].get("column"), str) + } ) if not required: continue @@ -4222,11 +4255,20 @@ def transport_rules_node( def _require_relation_inputs(spec, binding_id, inputs) -> None: + """A rules node slices its binding's relation and view dependencies whole. + + A view binding's node must also slice, on the person entity, every input + the reviewed closure encodes (other than through a graph node) or + defaults on the view's logical entity: the engine would otherwise run + the view without a column the closure says the frame carries. + """ document = transport_resource(spec, "axiom_rules_bindings") bindings = [ row for row in document.get("bindings", ()) if row.get("id") == binding_id ] - if len(bindings) != 1 or "relations" not in bindings[0]: + if len(bindings) != 1 or ( + "relations" not in bindings[0] and "views" not in bindings[0] + ): return declaration = _versioned_create_units(spec) if declaration is None: @@ -4234,14 +4276,51 @@ def _require_relation_inputs(spec, binding_id, inputs) -> None: CREATE_V1_REF, transport_resource(spec, "benefit_unit_rule") ) schema = declaration.entity_schema()[0] - dependencies = binding_relation_dependencies( - bindings[0], schema=schema, entity_names=document["entity_names"] - ) + dependencies = { + *binding_relation_dependencies( + bindings[0], schema=schema, entity_names=document["entity_names"] + ), + *binding_view_dependencies( + bindings[0], schema=schema, entity_names=document["entity_names"] + ), + } missing = missing_relation_inputs(dependencies, inputs, schema) if missing: raise ValueError( f"Rules binding {binding_id!r} needs whole-table relation inputs {missing}." ) + logical = binding_view_entities(bindings[0]) + if not logical or not _has_resource(spec, _CLOSURE_RESOURCE): + return + path = bindings[0].get("rulespec_path") + closed = sorted( + { + entry.get("input") + for entry in transport_resource(spec, _CLOSURE_RESOURCE).get("entries", ()) + if isinstance(entry, Mapping) + and entry.get("rulespec_path") == path + and entry.get("entity") in logical + and ( + entry.get("class") == "defaulted" + or ( + entry.get("class") == "encoded" + and entry.get("encoded_by") != "graph_node" + ) + ) + } + ) + sliced = { + column + for item in inputs + if item.entity == schema.person_entity + for column in item.columns + } + absent = [name for name in closed if name not in sliced] + if absent: + raise ValueError( + f"Rules binding {binding_id!r} runs views on {sorted(logical)} whose " + f"closed inputs {absent} its {schema.person_entity!r} slice omits." + ) def _check_extension_checkpoints(spec, config, graph, compiled) -> None: diff --git a/packages/microcosm-build/src/microcosm/build/transport/evidence_contract.py b/packages/microcosm-build/src/microcosm/build/transport/evidence_contract.py index 3c27a2715f..49127385dc 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/evidence_contract.py +++ b/packages/microcosm-build/src/microcosm/build/transport/evidence_contract.py @@ -78,7 +78,6 @@ __all__ = [ "AXIOM_INPUT_CLOSURE", - "BINDING_LOCAL_VIEW_MESSAGE", "CLOSURE_CLASSES", "DONOR_ARTIFACT_RECEIPT", "DONOR_RECEIPT_REF", @@ -220,12 +219,6 @@ "gap", ) -BINDING_LOCAL_VIEW_MESSAGE = ( - "{module} {entity}.{input}: closure source 'binding_local_view' is reserved " - "for binding-local views (S2b, S2D-AGREED item 3) and is refused until that " - "extension lands." -) - _SHA256 = re.compile(r"[0-9a-f]{64}") _IMPORTS = re.compile(rb"^imports\s*:", re.MULTILINE) @@ -840,6 +833,30 @@ def binding_relation_inputs(row: Mapping[str, object]) -> tuple[RelationInput, . return tuple(sorted(inputs.values())) +def binding_view_entities(row: Mapping[str, object]) -> frozenset[str]: + """The logical engine entities a binding's ``views`` block declares. + + Reads the declaration fields only; the registry validates the views + themselves (``registry._binding_views``). + """ + + views = row.get("views") + if views is None: + return frozenset() + if isinstance(views, str | bytes) or not isinstance(views, Sequence): + raise TypeError(f"rules binding {row.get('id')!r}.views must be a list.") + entities = set() + for view in views: + if not isinstance(view, Mapping) or not isinstance( + view.get("logical_entity"), str + ): + raise ValueError( + f"rules binding {row.get('id')!r} view lacks a logical_entity." + ) + entities.add(view["logical_entity"]) + return frozenset(entities) + + def module_declares_imports(source: bytes) -> bool: """Whether a RuleSpec module has a top-level ``imports:`` key.""" @@ -997,7 +1014,9 @@ def closure_partition( module_sha256: Binding id -> SHA-256 of the module bytes the run binds. module_imports: Binding id -> whether the module declares ``imports:``. mapped_entities: The engine entities the schema maps; roots on any - other entity are reported as ``unmapped_entity`` and are not live. + other entity are reported as ``unmapped_entity`` and are not live, + except a binding-local view's logical entity on that view's + binding, whose roots are live and read person columns. frame_columns: Frame entity -> columns the producer's slice carries, or ``None`` to skip the frame checks (engine-free tests). entity_names: Frame entity -> engine entity, needed with @@ -1008,17 +1027,29 @@ def closure_partition( Returns: ``{"check": [...], "modules": [...]}``: ``check`` is :meth:`InputClosure.check` on the live surface (stamped with the - closure's commits) and the mapping without unreached refs; each + closure's commits) and the mapping without unreached refs. The + checker unions each path's checked roots per engine entity across all + bindings, independent of binding order. Each per-binding module row puts every live root in exactly one of :data:`CLOSURE_CLASSES`. Raises: - ValueError: Naming every refused ``unreached_roots`` declaration, or a - binding that declares a binding-local view. + ValueError: Naming every refused ``unreached_roots`` declaration, or + both bindings when one module path carries different SHA-256 values. """ mapped = frozenset(mapped_entities) to_frame = {engine: frame for frame, engine in (entity_names or {}).items()} + # Per binding: each view's logical entity is live there and reads the + # person table. + view_entities = {str(row["id"]): binding_view_entities(row) for row in bindings} + binding_to_frame = { + binding: { + **to_frame, + **{entity: person_entity for entity in view_entities[binding]}, + } + for binding in view_entities + } refusals: list[str] = [] excused: set[InputRef] = set() declared_unreached: dict[str, frozenset[str]] = {} @@ -1033,22 +1064,17 @@ def closure_partition( for row in bindings: binding = str(row["id"]) path = str(row["rulespec_path"]) - if "views" in row: - refusals.append( - BINDING_LOCAL_VIEW_MESSAGE.format( - module=path, entity=row.get("engine_entity"), input="*" - ) - ) + live_entities = mapped | view_entities[binding] by_entity = roots.get(binding, {}) live_roots[binding] = { entity: tuple(sorted(set(names))) for entity, names in sorted(by_entity.items()) - if entity in mapped + if entity in live_entities } unmapped_roots[binding] = sorted( _label(entity, name) for entity, names in by_entity.items() - if entity not in mapped + if entity not in live_entities for name in names ) declaration = _unreached_declaration(row) @@ -1136,6 +1162,20 @@ def closure_partition( } for row in bindings } + path_bindings: dict[str, str] = {} + path_roots: dict[str, dict[str, set[str]]] = {} + for row in bindings: + binding = str(row["id"]) + path = str(row["rulespec_path"]) + previous = path_bindings.setdefault(path, binding) + if module_sha256[previous] != module_sha256[binding]: + raise ValueError( + f"Rules bindings {previous!r} and {binding!r} give different " + f"module SHA-256 values for {path!r}." + ) + by_entity = path_roots.setdefault(path, {}) + for entity, names in checked_roots[binding].items(): + by_entity.setdefault(entity, set()).update(names) surface = AxiomInputSurface( country=closure.country, rulespec_commit=closure.surface_rulespec_commit, @@ -1143,19 +1183,27 @@ def closure_partition( engine_surface="live", modules=MappingProxyType( { - str(row["rulespec_path"]): AxiomModuleSurface( - path=str(row["rulespec_path"]), - sha256=module_sha256[str(row["id"])], + path: AxiomModuleSurface( + path=path, + sha256=module_sha256[path_bindings[path]], status="compiled", - inputs=MappingProxyType(checked_roots[str(row["id"])]), + inputs=MappingProxyType( + { + entity: tuple(sorted(names)) + for entity, names in sorted(by_entity.items()) + } + ), canonical_inputs=MappingProxyType({}), ) - for row in bindings + for path, by_entity in sorted(path_roots.items()) } ), ) checked_surface, checked_mapping = without_unreached( - surface, mapping, excused, mapped + surface, + mapping, + excused, + mapped.union(*view_entities.values()), ) problems = list(closure.check(checked_mapping, checked_surface)) @@ -1171,7 +1219,7 @@ def closure_partition( roots_out: list[str] = [] listed = path in closure.modules for entity, names in live_roots[binding].items(): - frame_entity = to_frame.get(entity) + frame_entity = binding_to_frame[binding].get(entity) carried = ( None if frame_columns is None diff --git a/packages/microcosm-build/src/microcosm/build/transport/evidence_kernels.py b/packages/microcosm-build/src/microcosm/build/transport/evidence_kernels.py index 1df800b22e..e06b094f81 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/evidence_kernels.py +++ b/packages/microcosm-build/src/microcosm/build/transport/evidence_kernels.py @@ -321,8 +321,10 @@ def engine_root_inputs( list. The engine keeps relation inputs apart from ``root_inputs``, and ``variables()`` adds them on the manifest's ``root_entity``, so each of the binding's declared relation inputs joins that entity's roots here. - An injected adapter (engine-free fixtures) reports one flat list, which - this places on the binding's declared engine entity. + Each binding-local view's roots join under its logical entity + (``Child``), read through ``view_root_inputs()``. An injected adapter + (engine-free fixtures) reports one flat list, which this places on the + binding's declared engine entity; it refuses a binding with views. """ if isinstance(engine, AxiomEngine): @@ -339,7 +341,15 @@ def engine_root_inputs( f"on {relation.root_entity!r}, which entity_names does not map." ) found.setdefault(engine_entity, set()).add(relation.input) + if "views" in binding: + for logical_entity, names in engine.view_root_inputs().items(): + found.setdefault(logical_entity, set()).update(names) return {entity: tuple(sorted(names)) for entity, names in found.items()} + if "views" in binding: + raise TypeError( + f"rules binding {binding.get('id')!r} declares views, which only an " + "AxiomEngine runs." + ) return {str(binding["engine_entity"]): tuple(sorted(engine.variables()))} diff --git a/packages/microcosm-build/src/microcosm/build/transport/registry.py b/packages/microcosm-build/src/microcosm/build/transport/registry.py index 77c1cf3fc6..85d64e2d82 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/registry.py +++ b/packages/microcosm-build/src/microcosm/build/transport/registry.py @@ -29,6 +29,8 @@ AxiomEngine, AxiomPeriod, AxiomRelation, + AxiomView, + _outside_values, assert_no_relations, axiom_engine_ref, rulespec_tree_digest, @@ -64,6 +66,7 @@ "PYTHON_ENGINE_REF_FORMAT", "TransportRegistry", "binding_relation_dependencies", + "binding_view_dependencies", "build_transport_registry", "register_transport_population_kernels", "transport_entity_schema", @@ -186,7 +189,9 @@ def _binding_semantics( "variables": variables, } if "views" in row: - raise NotImplementedError(f"{label}.views are not supported.") + semantics["views"] = tuple( + view.to_dict() for view in _binding_views(row, entity_names, schema) + ) if "relations" in row: semantics["relations"] = tuple( relation.to_dict() @@ -203,7 +208,8 @@ def _binding_relations( """Parse explicit partition relations without inventing absent declarations. These are person members of a declared group partition. Arbitrary links - and binding-local logical views require a separate declaration contract. + require a separate declaration contract; binding-local views have their + own (:func:`_binding_views`). Compiled key, slot orientation and input surfaces are checked after the adapter is pinned to its RuleSpec bytes. """ @@ -276,6 +282,101 @@ def _binding_relations( return tuple(relations) +def _binding_views( + row: Mapping, + entity_names: Mapping[str, str], + schema: EntitySchema, +) -> tuple[AxiomView, ...]: + """Parse a binding's binding-local views, in declaration order. + + Each view names its ``logical_entity``, a ``selector`` holding one bool + person ``column``, and an ``outside`` value for each of its outputs. The + binding keeps its person entity. Checks that need the compiled module + (the outside keys against the binding's variables on the logical entity, + and view inputs against the binding entity's root inputs) run after the + adapter is pinned to its RuleSpec bytes. + """ + label = f"rules binding {row['id']!r}.views" + values = row["views"] + if isinstance(values, str | bytes) or not isinstance(values, Sequence): + raise TypeError(f"{label} must be a list of view declarations.") + if not values: + raise ValueError(f"{label} must declare at least one view; omit it for none.") + if "relations" in row: + raise ValueError(f"{label} and relations cannot share a binding.") + if row["entity"] != schema.person_entity: + raise ValueError( + f"{label} require a binding on the person entity {schema.person_entity!r}." + ) + structural = {schema.person_id_column} | { + schema.membership_column(group) for group in schema.group_entities + } + variables = set(row.get("variables", ())) + views = [] + for value in values: + manifest = _mapping(value, f"{label} declaration") + if set(manifest) != {"logical_entity", "selector", "outside"}: + raise ValueError( + f"{label} declarations hold exactly logical_entity, selector and " + "outside." + ) + selector = _mapping(manifest["selector"], f"{label}.selector") + if set(selector) != {"column"}: + raise ValueError( + f"{label}.selector names exactly one bool person column, " + "as {'column': name}." + ) + logical = _text(manifest["logical_entity"], f"{label}.logical_entity") + if logical in set(entity_names.values()): + raise ValueError( + f"{label}.logical_entity {logical!r} names a mapped engine entity." + ) + column = _text(selector["column"], f"{label}.selector.column") + if column in structural: + raise ValueError(f"{label}.selector {column!r} names a structural column.") + if column in variables: + raise ValueError( + f"{label}.selector {column!r} names an output of the binding." + ) + outside = dict(_mapping(manifest["outside"], f"{label}.outside")) + unknown = sorted(set(outside) - variables) + if unknown: + raise ValueError(f"{label}.outside names non-binding variables {unknown}.") + views.append(AxiomView(logical, column, outside)) + logical_entities = [view.logical_entity for view in views] + if len(logical_entities) != len(set(logical_entities)): + raise ValueError(f"{label} repeats a logical entity.") + outputs = {name for view in views for name in view.outside} + if outputs & {view.selector for view in views}: + raise ValueError(f"{label} selects on a view output.") + return tuple(views) + + +def binding_view_dependencies( + binding: Mapping, + *, + entity_names: Mapping[str, str], + schema: EntitySchema, +) -> tuple[tuple[str, str], ...]: + """The whole-table person slices a binding's views read: each selector. + + Composition uses the same parser as preparation, and + ``AxiomEngine.relation_dependencies`` reports the same columns to the + rules kernels. A view's root inputs are not dependencies. No declaration + contributes no columns. + """ + if "views" not in binding: + return () + return tuple( + sorted( + { + (schema.person_entity, view.selector) + for view in _binding_views(binding, entity_names, schema) + } + ) + ) + + def binding_relation_dependencies( binding: Mapping, *, @@ -339,6 +440,13 @@ def _check_engine( ) if engine._relations != expected_relations: raise ValueError(f"{label} engine relations do not match the binding.") + expected_views = ( + None + if "views" not in semantics + else _binding_views(semantics, entity_names, schema) + ) + if engine._views != expected_views: + raise ValueError(f"{label} engine views do not match the binding.") else: declared_root = getattr(engine, "rulespec_root", None) if ( @@ -356,6 +464,52 @@ def _check_engine( raise TypeError(f"{label} injected engine must expose cache_identity().") if "relations" in semantics: raise TypeError(f"{label} relation manifests require an AxiomEngine.") + if "views" in semantics: + raise TypeError(f"{label} views require an AxiomEngine.") + + +def _check_views(engine: AxiomEngine, semantics: Mapping) -> None: + """Check a view binding against its compiled module. + + Each view's program must declare no relations, and the binding entity's + program, when the module has one, too. A view's outside keys must equal + the binding's variables on its logical entity, and no view input may + share its name with a root input of the binding entity's program: both + would read one person column. Each outside value must cast losslessly to + its output's graph dtype before a registry can pin it. + """ + label = f"rules binding {semantics['id']!r}.views" + entity = semantics["entity"] + binding_program = engine._program(entity, missing_ok=True) + if binding_program is not None: + assert_no_relations(engine, entity) + derived = engine._derived_metadata() + variables = semantics["variables"] + for view in engine._views: + assert_no_relations(engine, view.logical_entity) + on_view = { + name + for name in variables + if name in derived and derived[name].entity == view.logical_entity + } + if set(view.outside) != on_view: + raise ValueError( + f"{label} {view.logical_entity!r} outside keys " + f"{sorted(view.outside)} differ from the binding's variables on " + f"it {sorted(on_view)}." + ) + for name, value in view.outside.items(): + _outside_values(name, derived[name].dtype, value, 0) + if binding_program is not None: + shared = sorted( + {name for inputs in engine.view_root_inputs().values() for name in inputs} + & set(binding_program.root_inputs) + ) + if shared: + raise ValueError( + f"{label} inputs {shared} share their names with " + f"{semantics['engine_entity']!r} root inputs of the same module." + ) def build_transport_registry( @@ -464,6 +618,11 @@ def build_transport_registry( if "relations" in semantics else {} ), + **( + {"views": _binding_views(semantics, entity_names, schema)} + if "views" in semantics + else {} + ), ) if engines_by_binding is None else engines_by_binding[row["id"]] @@ -475,6 +634,8 @@ def build_transport_registry( reference = axiom_engine_ref(engine, rulespec_root=root, **pins) if "relations" in semantics: engine.assert_declared_relations(semantics["entity"]) + elif "views" in semantics: + _check_views(engine, semantics) else: assert_no_relations(engine, semantics["entity"]) else: diff --git a/packages/microcosm-build/tests/engine_free/shared/test_axiom_engine_ref_goldens.py b/packages/microcosm-build/tests/engine_free/shared/test_axiom_engine_ref_goldens.py new file mode 100644 index 0000000000..1d9aae1168 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/shared/test_axiom_engine_ref_goldens.py @@ -0,0 +1,92 @@ +"""Absent views leave every Axiom engine reference byte for byte unchanged. + +The golden holds references generated at the base of S2b PR 1 (main +``4a36edcf``) for the six NZ binding rows frozen in test support, +``unreached_roots`` included, over a synthetic tree, and for the registry +and adapter relation fixtures +(:mod:`test_support.microcosm_build.axiom_ref_goldens`). A reference that +gains ``"views": []`` or ``"views": null`` without a declared view fails +here. The real-tree check runs only with ``POPULACE_RULESPEC_NZ``. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest + +import test_support.microcosm_build.axiom_ref_goldens as golden_helpers +from microcosm.build.transport.registry import build_transport_registry +from test_support.microcosm_build.axiom_ref_goldens import ( + GOLDEN, + NZ_BINDINGS, + engine_refs, +) + + +def test_engine_refs_match_the_base_golden_byte_for_byte(tmp_path): + golden = json.loads(GOLDEN.read_text()) + assert engine_refs(tmp_path / "refs") == golden + for name, reference in golden.items(): + assert "views" not in json.loads(reference), name + + +def test_ci_golden_source_ignores_live_nz_bindings(tmp_path, monkeypatch): + expected = golden_helpers.nz_document(tmp_path / "before") + monkeypatch.setattr( + golden_helpers, "NZ_BINDINGS", tmp_path / "missing-live-bindings.json" + ) + assert golden_helpers.nz_document(tmp_path / "after") == expected + + +RULESPEC_NZ = os.environ.get("POPULACE_RULESPEC_NZ") +REAL_GOLDEN = GOLDEN.with_name("nz_real_axiom_engine_refs.json") + + +@pytest.mark.skipif( + not RULESPEC_NZ, + reason="needs a rulespec-nz export at the pinned commit at POPULACE_RULESPEC_NZ", +) +def test_nz_real_refs_match_their_frozen_copy(tmp_path): + """NZ's six real references against a frozen copy (the tree, not the engine). + + This check reads NZ's live binding resource and its declared RuleSpec + commit, unlike the synthetic CI golden. To regenerate intentionally, + export that pinned tree, set ``POPULACE_RULESPEC_NZ`` to the export, + remove ``REAL_GOLDEN``, and run this test. It writes the frozen file and + fails; review the result, then rerun to verify it. Keep the existing copy + when checking compatibility with a later source head. + """ + import microcosm.build.transport.registry as registry_module + from microcosm.frame import VariableMetadata + from microcosm.frame.adapters.axiom import AxiomEngine + from test_support.microcosm_build.transport_population import UNIT_RULE + + document = json.loads(NZ_BINDINGS.read_text()) + document["engine"] = {"commit": "a" * 40, "wheel_sha256": "b" * 64} + entities = { + variable: row["entity"] + for row in document["bindings"] + for variable in row["variables"] + } + with pytest.MonkeyPatch.context() as patch: + patch.setattr(registry_module, "assert_no_relations", lambda *args: None) + patch.setattr( + AxiomEngine, + "variable_metadata", + lambda engine, name: VariableMetadata( + name, entities[name], "float", "year" + ), + ) + patch.setattr(AxiomEngine, "graph_dtype", lambda engine, name: "float64") + registry = build_transport_registry( + document, Path(RULESPEC_NZ), unit_rule=UNIT_RULE + ) + refs = dict(sorted(registry.engine_refs.items())) + assert all("views" not in json.loads(ref) for ref in refs.values()) + if not REAL_GOLDEN.exists(): + REAL_GOLDEN.write_text(json.dumps(refs, indent=1, sort_keys=True) + "\n") + pytest.fail(f"Froze {REAL_GOLDEN.name}; commit it at the base and rerun.") + assert refs == json.loads(REAL_GOLDEN.read_text()) diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_evidence_kernels.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_evidence_kernels.py index 610fe829a6..451aaceea6 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_evidence_kernels.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_evidence_kernels.py @@ -943,12 +943,41 @@ def test_a_module_with_imports_refuses_its_unreached_declaration(self) -> None: ) assert not evidence_contract.module_declares_imports(b"rules:\n imports: 1\n") - def test_a_binding_local_view_is_refused_with_the_reserved_message(self) -> None: - _, bindings, *_ = _nz() + def test_a_binding_local_view_partitions_its_logical_entity_roots(self) -> None: + closure, bindings, surface, mapping = _nz() + base = _nz_partition() bindings = copy.deepcopy(bindings) - bindings["bindings"][0]["views"] = [{"name": "child"}] - with pytest.raises(ValueError, match="reserved for binding-local views"): - _nz_partition(bindings=bindings) + row = bindings["bindings"][0] + row["views"] = [ + { + "logical_entity": "Child", + "selector": {"column": "is_child"}, + "outside": {}, + } + ] + rows = bindings["bindings"] + roots = { + r["id"]: dict(surface.modules[r["rulespec_path"]].inputs) for r in rows + } + roots[row["id"]]["Child"] = ("child_age",) + result = closure_partition( + closure, + mapping, + rows, + roots, + module_sha256={ + r["id"]: surface.modules[r["rulespec_path"]].sha256 for r in rows + }, + module_imports={}, + mapped_entities=bindings["entity_names"].values(), + ) + # The view binding's Child root is live (an unclosed gap here); every + # other row partitions as without the view. + assert result["modules"][0]["gap"] == sorted( + [*base["modules"][0]["gap"], "Child.child_age"] + ) + assert result["modules"][0]["unmapped_entity"] == [] + assert result["modules"][1:] == base["modules"][1:] def test_be_multi_entity_modules_keep_every_root_on_its_entity(self) -> None: surface = load_axiom_input_surface(BE_SURFACE) diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_registry.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_registry.py index eb02d36193..d11c733007 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_registry.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_registry.py @@ -587,7 +587,7 @@ def test_relation_manifest_validation(tmp_path, monkeypatch, field, value, match ("scalar", TypeError, "list of relation manifests"), ("duplicate", ValueError, "repeats a compiled relation key"), ("missing", ValueError, "manifests require exactly"), - ("views", NotImplementedError, "views are not supported"), + ("views", ValueError, "at least one view"), ], ) def test_relation_declaration_refuses_ambiguous_or_future_shapes( diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_views.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_views.py new file mode 100644 index 0000000000..6a5ee4c593 --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_views.py @@ -0,0 +1,892 @@ +"""Binding-local views through the transport registry, compose and closure. + +A fake ``axiom_rules_engine`` compiles fake dense programs per engine +entity; it evaluates no RuleSpec formula. The registry builds real +:class:`AxiomEngine` adapters over it, so preparation, references and the +closure partition run their own code. +""" + +from __future__ import annotations + +import copy +import hashlib +import json +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pandas as pd +import pytest + +import microcosm.build.transport.compose as compose_module +from microcosm.build.transport.evidence_contract import ( + binding_view_entities, + closure_partition, +) +from microcosm.build.transport.evidence_kernels import engine_root_inputs +from microcosm.build.transport.registry import ( + binding_view_dependencies, + build_transport_registry, +) +from microcosm.frame import Frame, WeightKind, Weights +from microcosm.frame.adapters.axiom import AxiomEngine, AxiomPeriod, AxiomView +from microcosm.frame.concept_mapping import ConceptMapping +from microcosm.frame.concepts import CONCEPT_ENTITIES, CONCEPTS +from microcosm.frame.input_closure import ClosureEntry, InputClosure, UndeterminedPolicy +from microcosm.graph import Node, Owned, Slice +from test_support.microcosm_build.transport_population import UNIT_RULE + +PERIOD = "2026-27" +CHILD_OUTPUTS = ("child_amount", "child_test") + + +def _metadata(name, entity, dtype): + return SimpleNamespace(name=name, entity=entity, dtype=dtype, period="Year") + + +class _Program: + def __init__(self, entity, root_inputs, metadata, function, relations=()): + self.root_entity = entity + self.root_inputs = list(root_inputs) + self.relations = list(relations) + self.derived_metadata = list(metadata) + self.function = function + self.calls = [] + + def execute(self, **arguments): + self.calls.append(arguments) + values = self.function( + *(arguments["inputs"][name] for name in self.root_inputs) + ) + return {"outputs": {name: values[name] for name in arguments["outputs"]}} + + execute_f64 = execute + + +class _FakeEngine: + def __init__(self, programs_by_module): + self.programs_by_module = programs_by_module + self.CompiledDenseProgram = self + + def from_file(self, module, *, rulespec_roots, entity): + programs = self.programs_by_module[Path(module).name] + if entity not in programs: + raise ValueError( + f"dense compilation could not find derived outputs for entity `{entity}`" + ) + return programs[entity] + + +def _child_program(relations=()): + return _Program( + "Child", + ("age", "child_income"), + [ + _metadata("child_amount", "Child", "decimal"), + _metadata("child_test", "Child", "judgment"), + ], + lambda age, income: { + "child_amount": age * 10.0 + income, + "child_test": np.where(age < 12, 1, -1).astype(np.int8), + }, + relations, + ) + + +def _person_program(root_inputs=("income",), extra=()): + return _Program( + "Person", + root_inputs, + [_metadata("adult_amount", "Person", "decimal"), *extra], + lambda *columns: {"adult_amount": columns[0] + 0.5}, + ) + + +def _view(**updates): + view = { + "logical_entity": "Child", + "selector": {"column": "is_child"}, + "outside": {"child_amount": 0.0, "child_test": 0}, + } + view.update(updates) + return view + + +def _fixture(tmp_path, monkeypatch, *, programs=None, views=None, variables=None): + root = tmp_path / "rulespec" + root.mkdir(exist_ok=True) + payload = b"rules: []\n" + (root / "child.yaml").write_bytes(payload) + row = { + "id": "child_benefit", + "rulespec_path": "child.yaml", + "sha256": hashlib.sha256(payload).hexdigest(), + "entity": "person", + "engine_entity": "Person", + "period": PERIOD, + "variables": list(variables or CHILD_OUTPUTS), + "views": [_view()] if views is None else views, + } + document = { + "engine": {"commit": "a" * 40, "wheel_sha256": "b" * 64}, + "rulespec": {"commit": "c" * 40}, + "entity_names": { + "person": "Person", + "household": "Household", + "family": "Family", + }, + "periods": { + PERIOD: {"kind": "tax_year", "start": "2026-04-01", "end": "2027-03-31"} + }, + "bindings": [row], + } + engine = _FakeEngine( + {"child.yaml": {"Child": _child_program()} if programs is None else programs} + ) + monkeypatch.setattr(AxiomEngine, "_import_engine", lambda self: engine) + return root, document, engine + + +def _prepared(tmp_path, monkeypatch, **options): + root, document, engine = _fixture(tmp_path, monkeypatch, **options) + registry = build_transport_registry(document, root, unit_rule=UNIT_RULE) + adapter = registry.engines[registry.engine_refs["child_benefit"]] + return registry, adapter, document, engine + + +def _frame(selected, ages): + n = len(selected) + people = pd.DataFrame( + { + "person_id": np.arange(1, n + 1), + "person_household_id": np.ones(n, dtype=np.int64), + "person_family_id": np.ones(n, dtype=np.int64), + "is_child": np.asarray(selected, dtype=bool), + "age": np.asarray(ages, dtype=np.int64), + "child_income": np.zeros(n), + "income": np.full(n, 2.0), + } + ) + return Frame( + tables={ + "person": people, + "household": pd.DataFrame({"household_id": [1]}), + "family": pd.DataFrame({"family_id": [1]}), + }, + schema=UNIT_RULE_SCHEMA, + weights={"household": Weights(np.ones(1), WeightKind.DESIGN)}, + ) + + +UNIT_RULE_SCHEMA = None + + +@pytest.fixture(autouse=True) +def _schema(): + global UNIT_RULE_SCHEMA + from microcosm.build.transport.registry import transport_entity_schema + + UNIT_RULE_SCHEMA = transport_entity_schema(UNIT_RULE)[0] + + +# --------------------------------------------------------------------------- +# Registry +# --------------------------------------------------------------------------- + + +def test_a_child_only_module_prepares_through_the_registry(tmp_path, monkeypatch): + registry, adapter, document, _ = _prepared(tmp_path, monkeypatch) + reference = json.loads(registry.engine_refs["child_benefit"]) + assert reference["views"] == [_view()] + assert adapter._views == ( + AxiomView("Child", "is_child", {"child_amount": 0.0, "child_test": 0}), + ) + assert adapter.variable_metadata("child_amount").entity == "person" + result = adapter.materialize( + _frame([True, False, True], [3, 40, 15]), list(CHILD_OUTPUTS), PERIOD + ) + assert result["child_amount"].tolist() == [30.0, 0.0, 150.0] + assert result["child_test"].tolist() == [1, 0, -1] + + +def test_view_dependencies_agree_with_the_adapter(tmp_path, monkeypatch): + registry, adapter, document, _ = _prepared(tmp_path, monkeypatch) + binding = document["bindings"][0] + expected = (("person", "is_child"),) + assert ( + binding_view_dependencies( + binding, entity_names=document["entity_names"], schema=registry.schema + ) + == expected + ) + assert adapter.relation_dependencies("person") == expected + assert { + dependency + for entity in registry.schema.entities + for dependency in adapter.relation_dependencies(entity) + } == set(expected) + binding.pop("views") + assert ( + binding_view_dependencies( + binding, entity_names=document["entity_names"], schema=registry.schema + ) + == () + ) + + +def test_absent_views_stay_absent_from_the_reference(tmp_path, monkeypatch): + root, document, _ = _fixture( + tmp_path, + monkeypatch, + programs={"Person": _person_program()}, + variables=["adult_amount"], + ) + document["bindings"][0].pop("views") + registry = build_transport_registry(document, root, unit_rule=UNIT_RULE) + reference = json.loads(registry.engine_refs["child_benefit"]) + assert "views" not in reference + assert registry.engines[registry.engine_refs["child_benefit"]]._views is None + + +@pytest.mark.parametrize( + ("change", "error", "match"), + [ + (lambda row: row.update(views=[]), ValueError, "at least one view"), + (lambda row: row.update(views={}), TypeError, "list of view"), + ( + lambda row: row["views"][0]["outside"].pop("child_test"), + ValueError, + "outside keys", + ), + ( + lambda row: row["views"][0]["outside"].update(other=0.0), + ValueError, + "non-binding variables", + ), + ( + lambda row: row["views"][0].update(logical_entity="Person"), + ValueError, + "mapped engine entity", + ), + ( + lambda row: row["views"][0].update(logical_entity="Family"), + ValueError, + "mapped engine entity", + ), + ( + lambda row: row["views"].append(copy.deepcopy(row["views"][0])), + ValueError, + "repeats a logical entity", + ), + ( + lambda row: row["views"][0]["selector"].update(equals=True), + ValueError, + "exactly one bool person column", + ), + ( + lambda row: row["views"][0].update(selector={"expression": "age < 18"}), + ValueError, + "exactly one bool person column", + ), + ( + lambda row: row["views"][0].update(selector={"rows": [1, 2]}), + ValueError, + "exactly one bool person column", + ), + ( + lambda row: row["views"][0].update(selector={"column": "person_id"}), + ValueError, + "structural", + ), + ( + lambda row: row["views"][0].update(selector={"column": "person_family_id"}), + ValueError, + "structural", + ), + ( + lambda row: row["views"][0].update(selector={"column": "child_amount"}), + ValueError, + "output of the binding", + ), + ( + lambda row: row["views"][0].update(id="child"), + ValueError, + "exactly logical_entity, selector and outside", + ), + ( + lambda row: row.update(entity="family", engine_entity="Family"), + ValueError, + "person entity", + ), + ( + lambda row: row.update(relations=[]), + ValueError, + "relations cannot share", + ), + ], +) +def test_the_registry_refuses_an_unsupported_view( + tmp_path, monkeypatch, change, error, match +): + root, document, _ = _fixture(tmp_path, monkeypatch) + change(document["bindings"][0]) + with pytest.raises(error, match=match): + build_transport_registry(document, root, unit_rule=UNIT_RULE) + + +def test_the_registry_refuses_a_relation_on_the_view_program(tmp_path, monkeypatch): + relation = SimpleNamespace(name="toy_member", key="toy_member:0:1") + root, document, _ = _fixture( + tmp_path, monkeypatch, programs={"Child": _child_program([relation])} + ) + with pytest.raises(NotImplementedError, match="view entity 'Child'"): + build_transport_registry(document, root, unit_rule=UNIT_RULE) + + +def test_the_registry_refuses_a_view_input_named_like_a_person_root( + tmp_path, monkeypatch +): + root, document, _ = _fixture( + tmp_path, + monkeypatch, + programs={"Person": _person_program(("age",)), "Child": _child_program()}, + variables=["adult_amount", *CHILD_OUTPUTS], + ) + with pytest.raises(ValueError, match=r"inputs \['age'\] share their names"): + build_transport_registry(document, root, unit_rule=UNIT_RULE) + + +def test_a_mixed_module_prepares_with_distinct_person_and_view_inputs( + tmp_path, monkeypatch +): + registry, adapter, _, _ = _prepared( + tmp_path, + monkeypatch, + programs={"Person": _person_program(), "Child": _child_program()}, + variables=["adult_amount", *CHILD_OUTPUTS], + ) + result = adapter.materialize( + _frame([True, False], [5, 50]), ["adult_amount", "child_amount"], PERIOD + ) + assert result["adult_amount"].tolist() == [2.5, 2.5] + assert result["child_amount"].tolist() == [50.0, 0.0] + + +def test_a_child_output_without_a_view_is_refused(tmp_path, monkeypatch): + root, document, _ = _fixture( + tmp_path, + monkeypatch, + programs={ + "Person": _person_program(extra=_child_program().derived_metadata), + "Child": _child_program(), + }, + variables=["adult_amount", *CHILD_OUTPUTS], + ) + document["bindings"][0].pop("views") + with pytest.raises(ValueError, match="which no frame entity is mapped to"): + build_transport_registry(document, root, unit_rule=UNIT_RULE) + + +def test_an_injected_python_engine_with_views_is_refused(tmp_path, monkeypatch): + root, document, _ = _fixture(tmp_path, monkeypatch) + + class _Python: + rulespec_root = root + + def entity_schema(self): + return UNIT_RULE_SCHEMA + + def cache_identity(self): + return {} + + def assert_no_relations(self, entity): + return None + + def variable_metadata(self, name): + raise AssertionError("unused") + + def variables(self): + return () + + def materialize(self, bundle, variables, period): + raise AssertionError("unused") + + def export_contract(self): + raise AssertionError("unused") + + def write_dataset(self, bundle, path, period): + raise AssertionError("unused") + + with pytest.raises(TypeError, match="views require an AxiomEngine"): + build_transport_registry( + document, + root, + unit_rule=UNIT_RULE, + engines_by_binding={"child_benefit": _Python()}, + ) + with pytest.raises(TypeError, match="only an AxiomEngine runs"): + engine_root_inputs(_Python(), document["bindings"][0], document["entity_names"]) + + +@pytest.mark.parametrize( + "selector,outside", + [ + ("is_kid", {"child_amount": 0.0, "child_test": 0}), + ("is_child", {"child_amount": 0, "child_test": 0}), + ], +) +def test_an_injected_axiom_engine_must_carry_the_binding_views( + tmp_path, monkeypatch, selector, outside +): + root, document, _ = _fixture(tmp_path, monkeypatch) + schema = UNIT_RULE_SCHEMA + periods = {PERIOD: AxiomPeriod("2026-04-01", "2027-03-31", "tax_year")} + + def adapter(views): + return AxiomEngine( + root.resolve() / "child.yaml", + schema, + rulespec_roots=(root.resolve(),), + periods=periods, + entity_names=document["entity_names"], + nesting={"family": "household"}, + output_dtypes="graph", + views=views, + ) + + with pytest.raises(ValueError, match="views do not match"): + build_transport_registry( + document, + root, + unit_rule=UNIT_RULE, + engines_by_binding={ + "child_benefit": adapter((AxiomView("Child", selector, outside),)) + }, + ) + prepared = build_transport_registry( + document, + root, + unit_rule=UNIT_RULE, + engines_by_binding={ + "child_benefit": adapter( + ( + AxiomView( + "Child", "is_child", {"child_amount": 0.0, "child_test": 0} + ), + ) + ) + }, + ) + assert json.loads(prepared.engine_refs["child_benefit"])["views"] == [_view()] + + +# --------------------------------------------------------------------------- +# Compose +# --------------------------------------------------------------------------- + + +def _closure_document(document, entries): + row = document["bindings"][0] + return { + "entries": [ + { + "rulespec_path": row["rulespec_path"], + "entity": entity, + "input": name, + **fields, + } + for entity, name, fields in entries + ] + } + + +_ENTRIES = ( + ("Child", "age", {"class": "encoded", "encoded_by": "concept_binding"}), + ("Child", "child_income", {"class": "defaulted", "value": 0.0}), + ("Child", "child_supplement", {"class": "encoded", "encoded_by": "graph_node"}), + ("Person", "income", {"class": "encoded", "encoded_by": "concept_binding"}), +) + + +def _spec(document, entries=_ENTRIES): + return { + "country": "zz", + "resources": { + "axiom_rules_bindings": document, + "benefit_unit_rule": UNIT_RULE.to_dict(), + "axiom_input_closure": _closure_document(document, entries), + }, + } + + +def _rules_node(person_columns=("is_child", "age", "child_income"), rows="all"): + return Node( + "child_benefit_node", + "simulate.rules_by_ref@1", + population="create", + inputs=(Slice("person", tuple(person_columns), rows=rows),), + outputs=tuple(Owned("person", name, "float64") for name in CHILD_OUTPUTS), + params={}, + ) + + +def _config(registry): + return SimpleNamespace(engine_refs=dict(registry.engine_refs)) + + +def test_compose_binds_a_view_node_that_slices_its_selector_and_inputs( + tmp_path, monkeypatch +): + registry, _, document, _ = _prepared(tmp_path, monkeypatch) + node = compose_module.transport_rules_node( + _spec(document), _config(registry), "child_benefit", _rules_node() + ) + assert node.params["engine_ref"] == registry.engine_refs["child_benefit"] + assert node.params["variables"] == CHILD_OUTPUTS + + +@pytest.mark.parametrize( + ("node", "match"), + [ + (_rules_node(("age", "child_income")), "whole-table relation inputs"), + ( + _rules_node(("is_child", "age", "child_income"), rows="is_child"), + "whole-table relation inputs", + ), + (_rules_node(("is_child", "child_income")), r"closed inputs \['age'\]"), + (_rules_node(("is_child", "age")), r"closed inputs \['child_income'\]"), + ], + ids=("no-selector", "masked", "encoded-missing", "defaulted-missing"), +) +def test_compose_refuses_an_incomplete_view_node(tmp_path, monkeypatch, node, match): + registry, _, document, _ = _prepared(tmp_path, monkeypatch) + with pytest.raises(ValueError, match=match): + compose_module.transport_rules_node( + _spec(document), _config(registry), "child_benefit", node + ) + + +def test_compose_leaves_graph_node_inputs_to_the_node_that_writes_them( + tmp_path, monkeypatch +): + """``child_supplement`` is closed through a graph node, so the slice may omit it.""" + registry, _, document, _ = _prepared(tmp_path, monkeypatch) + compose_module._require_relation_inputs( + _spec(document), "child_benefit", _rules_node().inputs + ) + + +def _closure_node(person_columns): + return Node( + "closure", + compose_module.INPUT_CLOSURE_REF, + population="create", + inputs=(Slice("person", tuple(person_columns)),), + params={"bindings": json.dumps({"bindings": None})}, + ) + + +def test_the_closure_producer_refuses_a_person_slice_without_the_selector( + tmp_path, monkeypatch +): + _, _, document, _ = _prepared(tmp_path, monkeypatch) + spec = _spec(document) + + def graph(columns): + node = _closure_node(columns) + node = Node( + node.id, + node.kernel, + population=node.population, + inputs=node.inputs, + params={"bindings": json.dumps(document)}, + ) + return SimpleNamespace(nodes=(node,)) + + with pytest.raises(ValueError, match=r"\['is_child'\]"): + compose_module._check_closure_relation_slices(spec, graph(("age",))) + compose_module._check_closure_relation_slices(spec, graph(("is_child", "age"))) + + +# --------------------------------------------------------------------------- +# Closure +# --------------------------------------------------------------------------- + + +def _empty_mapping() -> ConceptMapping: + return ConceptMapping( + engine="none", + engine_version="none", + entity_correspondence={entity: entity for entity in CONCEPT_ENTITIES}, + input_declaration="usage_inferred", + bindings=(), + unmapped={item.id: "Synthetic: unmapped." for item in CONCEPTS}, + ) + + +def _closure(document, entries): + row = document["bindings"][0] + return InputClosure( + country="zz", + content_basis="transport", + mapping_engine="none", + mapping_engine_version="none", + surface_rulespec_commit="synthetic", + surface_engine_repository="engine", + surface_engine_commit="synthetic", + modules={row["rulespec_path"]: row["sha256"]}, + engine_optional_evidence="Synthetic.", + undetermined=UndeterminedPolicy( + action="refuse", reason="Synthetic.", alternatives=(), new_default=False + ), + knobs=(), + entries=tuple( + ClosureEntry( + module=row["rulespec_path"], + entity=entity, + input=name, + closure_class=closure_class, + value=value, + reason="Synthetic.", + citation=None if value is None else "Synthetic provision.", + ) + for entity, name, closure_class, value in entries + ), + ) + + +_DEFAULTED = ( + ("Child", "age", "defaulted", 4), + ("Child", "child_income", "engine_optional", None), +) + + +def _partition(document, entries=_DEFAULTED, *, roots=None, frame_columns=None): + rows = document["bindings"] + closure = _closure(document, entries) + return closure_partition( + closure, + _empty_mapping(), + rows, + roots or {row["id"]: {"Child": ("age", "child_income")} for row in rows}, + module_sha256={row["id"]: row["sha256"] for row in rows}, + module_imports={row["id"]: False for row in rows}, + mapped_entities=document["entity_names"].values(), + frame_columns=frame_columns, + entity_names=document["entity_names"], + person_entity="person", + ) + + +def test_the_closure_reads_view_roots_on_the_logical_entity(tmp_path, monkeypatch): + registry, adapter, document, _ = _prepared( + tmp_path, + monkeypatch, + programs={"Person": _person_program(), "Child": _child_program()}, + variables=["adult_amount", *CHILD_OUTPUTS], + ) + roots = engine_root_inputs( + adapter, document["bindings"][0], document["entity_names"] + ) + assert roots == {"Child": ("age", "child_income"), "Person": ("income",)} + assert binding_view_entities(document["bindings"][0]) == {"Child"} + assert binding_view_entities({"id": "x"}) == frozenset() + + +def test_a_view_binding_partitions_its_child_roots_on_child(tmp_path, monkeypatch): + _, document, _ = _fixture(tmp_path, monkeypatch) + (module,) = _partition(document)["modules"] + assert module["roots"] == ["Child.age", "Child.child_income"] + assert module["unmapped_entity"] == [] + assert [item["input"] for item in module["defaulted"]] == ["Child.age"] + assert module["engine_optional"] == ["Child.child_income"] + assert module["gap"] == [] + + +def test_without_the_view_child_roots_stay_unmapped(tmp_path, monkeypatch): + """The mutant that drops the per-binding view entity leaves Child unmapped.""" + _, document, _ = _fixture(tmp_path, monkeypatch) + document["bindings"][0].pop("views") + (module,) = _partition(document)["modules"] + assert module["roots"] == [] + assert module["unmapped_entity"] == ["Child.age", "Child.child_income"] + + +def test_child_roots_placed_on_person_do_not_close(tmp_path, monkeypatch): + """Child roots reported on Person miss their Child entries and become gaps.""" + _, document, _ = _fixture(tmp_path, monkeypatch) + result = _partition( + document, + roots={"child_benefit": {"Person": ("age", "child_income")}}, + ) + (module,) = result["modules"] + assert module["gap"] == ["Person.age", "Person.child_income"] + assert module["unexpected"] == ["Child.age", "Child.child_income"] + + +def test_an_unclosed_child_root_is_a_gap(tmp_path, monkeypatch): + _, document, _ = _fixture(tmp_path, monkeypatch) + result = _partition(document, entries=_DEFAULTED[:1]) + (module,) = result["modules"] + assert module["gap"] == ["Child.child_income"] + assert any("child_income" in str(problem) for problem in result["check"]) + + +def test_a_defaulted_child_root_on_the_person_table_is_not_frame_missing( + tmp_path, monkeypatch +): + """PR 2 writes Child defaults as person columns; the person slice carries them.""" + _, document, _ = _fixture(tmp_path, monkeypatch) + carried = _partition( + document, frame_columns={"person": frozenset({"is_child", "age"})} + ) + assert carried["modules"][0]["frame_missing"] == [] + missing = _partition(document, frame_columns={"person": frozenset({"is_child"})}) + assert missing["modules"][0]["frame_missing"] == ["Child.age"] + + +def test_several_view_bindings_keep_their_roots_in_order(tmp_path, monkeypatch): + _, document, _ = _fixture(tmp_path, monkeypatch) + second = copy.deepcopy(document["bindings"][0]) + second["id"] = "child_supplement" + document["bindings"].append(second) + result = _partition(document) + assert [module["binding"] for module in result["modules"]] == [ + "child_benefit", + "child_supplement", + ] + assert all( + module["roots"] == ["Child.age", "Child.child_income"] + for module in result["modules"] + ) + + +def test_the_closure_no_longer_refuses_a_view_binding(tmp_path, monkeypatch): + _, document, _ = _fixture(tmp_path, monkeypatch) + _partition(document) + + +# --------------------------------------------------------------------------- +# Downstream: a view output is an ordinary person column +# --------------------------------------------------------------------------- + + +def test_downstream_reads_a_view_output_as_a_person_variable(tmp_path, monkeypatch): + """Composition sites key on (binding entity, variable): both are person.""" + registry, adapter, document, _ = _prepared(tmp_path, monkeypatch) + row = document["bindings"][0] + assert { + (adapter.variable_metadata(name).entity, name) for name in row["variables"] + } == {(row["entity"], name) for name in row["variables"]} + assert adapter.graph_dtype("child_amount") == "float64" + assert adapter.graph_dtype("child_test") == "int64" + + +def test_the_support_roster_reads_a_view_default_as_a_person_column( + tmp_path, monkeypatch +): + """PR 2 writes Child closure defaults on the person table.""" + _, document, _ = _fixture(tmp_path, monkeypatch) + defaults = compose_module._closure_defaults(_spec(document)) + assert defaults == {("person", "child_income"): "child.yaml Child.child_income"} + document["bindings"][0].pop("views") + assert compose_module._closure_defaults(_spec(document)) == { + (None, "child_income"): "child.yaml Child.child_income" + } + + +def test_shared_path_closure_does_not_depend_on_binding_order(tmp_path, monkeypatch): + _, document, _ = _fixture(tmp_path, monkeypatch) + second = copy.deepcopy(document["bindings"][0]) + second["id"] = "no_view" + second.pop("views") + document["bindings"].append(second) + forward = _partition(document) + document["bindings"].reverse() + backward = _partition(document) + assert forward["check"] == backward["check"] == [] + assert {row["binding"]: row for row in forward["modules"]} == { + row["binding"]: row for row in backward["modules"] + } + + +@pytest.mark.parametrize("reverse", [False, True]) +def test_shared_path_refuses_different_module_hashes(tmp_path, monkeypatch, reverse): + _, document, _ = _fixture(tmp_path, monkeypatch) + second = copy.deepcopy(document["bindings"][0]) + second.update(id="no_view", sha256="d" * 64) + second.pop("views") + document["bindings"].append(second) + if reverse: + document["bindings"].reverse() + with pytest.raises( + ValueError, match=r"(?:child_benefit.*no_view|no_view.*child_benefit)" + ): + _partition(document) + + +def test_one_adapter_exposes_roots_from_two_logical_views(tmp_path, monkeypatch): + vehicle = _Program( + "Vehicle", + ("vehicle_value",), + [_metadata("vehicle_amount", "Vehicle", "decimal")], + lambda value: {"vehicle_amount": value * 0.1}, + ) + _, adapter, document, _ = _prepared( + tmp_path, + monkeypatch, + programs={"Child": _child_program(), "Vehicle": vehicle}, + views=[ + _view(), + _view( + logical_entity="Vehicle", + selector={"column": "has_vehicle"}, + outside={"vehicle_amount": 0.0}, + ), + ], + variables=[*CHILD_OUTPUTS, "vehicle_amount"], + ) + expected = {"Child": ("age", "child_income"), "Vehicle": ("vehicle_value",)} + assert adapter.view_root_inputs() == expected + assert {"age", "child_income", "vehicle_value"} <= set(adapter.variables()) + assert ( + engine_root_inputs(adapter, document["bindings"][0], document["entity_names"]) + == expected + ) + + +def test_outside_cannot_name_a_binding_output_on_person(tmp_path, monkeypatch): + root, document, _ = _fixture( + tmp_path, + monkeypatch, + programs={"Person": _person_program(), "Child": _child_program()}, + variables=["adult_amount", *CHILD_OUTPUTS], + ) + document["bindings"][0]["views"][0]["outside"]["adult_amount"] = 0.0 + with pytest.raises(ValueError, match="outside keys"): + build_transport_registry(document, root, unit_rule=UNIT_RULE) + + +@pytest.mark.parametrize( + "name,value,message", + [ + ( + "child_amount", + True, + "View outside value .* does not cast losslessly to its 'decimal' output", + ), + ("child_test", 2, "Judgment output 'child_test' holds codes outside -1/0/1"), + ( + "child_test", + 0.5, + "View outside value .* does not cast losslessly to its 'judgment' output", + ), + ], +) +def test_registry_refuses_outside_values_that_cannot_cast( + tmp_path, monkeypatch, name, value, message +): + root, document, _ = _fixture(tmp_path, monkeypatch) + document["bindings"][0]["views"][0]["outside"][name] = value + with pytest.raises(ValueError, match=message): + build_transport_registry(document, root, unit_rule=UNIT_RULE) diff --git a/packages/microcosm-build/tests/golden/axiom_engine_refs.json b/packages/microcosm-build/tests/golden/axiom_engine_refs.json new file mode 100644 index 0000000000..fae67de773 --- /dev/null +++ b/packages/microcosm-build/tests/golden/axiom_engine_refs.json @@ -0,0 +1,12 @@ +{ + "adapter_relations_absent": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"household\":\"Household\",\"person\":\"Person\",\"tax_unit\":\"TaxUnit\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"tax_unit\"],\"module\":\"zz/policies/tests/axiom_toy_relations.yaml\",\"module_sha256\":\"96cc3a89b8bcc25409ce5c3f3b4a5457ce758bb89593eef5eb87e7371da86c9d\",\"nesting\":{\"tax_unit\":\"household\"},\"output_dtypes\":\"native\",\"periods\":null,\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":5752,\"rulespec_tree_sha256\":\"7bafd19bdd6e329c7bbacd3540a98c26f31df4ed46cbf35f693936ccc39cf477\"}", + "adapter_relations_declared": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"household\":\"Household\",\"person\":\"Person\",\"tax_unit\":\"TaxUnit\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"tax_unit\"],\"module\":\"zz/policies/tests/axiom_toy_relations.yaml\",\"module_sha256\":\"96cc3a89b8bcc25409ce5c3f3b4a5457ce758bb89593eef5eb87e7371da86c9d\",\"nesting\":{\"tax_unit\":\"household\"},\"output_dtypes\":\"native\",\"periods\":null,\"person_entity\":\"person\",\"relations\":[{\"arguments\":[\"TaxUnit\",\"Person\"],\"cardinality\":\"many\",\"current_slot\":0,\"key\":\"toy_member:0:1\",\"membership_column\":\"person_tax_unit_id\",\"period\":\"2025\",\"predicate\":\"toy_member\",\"related_entity\":\"person\",\"related_inputs\":{\"toy_income\":\"income\"},\"related_slot\":1,\"root_entity\":\"tax_unit\"}],\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":5752,\"rulespec_tree_sha256\":\"7bafd19bdd6e329c7bbacd3540a98c26f31df4ed46cbf35f693936ccc39cf477\"}", + "adapter_relations_empty": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"household\":\"Household\",\"person\":\"Person\",\"tax_unit\":\"TaxUnit\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"tax_unit\"],\"module\":\"zz/policies/tests/axiom_toy_relations.yaml\",\"module_sha256\":\"96cc3a89b8bcc25409ce5c3f3b4a5457ce758bb89593eef5eb87e7371da86c9d\",\"nesting\":{\"tax_unit\":\"household\"},\"output_dtypes\":\"native\",\"periods\":null,\"person_entity\":\"person\",\"relations\":[],\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":5752,\"rulespec_tree_sha256\":\"7bafd19bdd6e329c7bbacd3540a98c26f31df4ed46cbf35f693936ccc39cf477\"}", + "nz_accommodation_supplement": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/accommodation_supplement/core.yaml\",\"module_sha256\":\"68e3590c81adc742c6de5845c2cb8f40c5e6fd0f76831e69e45251f8d30c1fff\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "nz_family_tax_credits": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/income_tax/family_scheme/tax_credits.yaml\",\"module_sha256\":\"19b25bcfb8ce392879d806795d980e814bf7aaee16f97c1764c785979e09fcba\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "nz_income_tax_schedule_1": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/income_tax/schedule_1/individual_income_tax.yaml\",\"module_sha256\":\"511e9657172f0bee5ca83cb1f4cef0a94cfc27f0b7346dc78e0996ecc006867f\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "nz_main_benefits_entitlement": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/main_benefits/entitlement.yaml\",\"module_sha256\":\"59fbc348788a7f9fbc044587ee700537fe81e8301a0fb36f5d72da73e5264d81\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "nz_main_benefits_rates": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/main_benefits/rates.yaml\",\"module_sha256\":\"7f3cf007271219aa556ca7e3e902d1bc61e730c302c662d8d0c044d0b117bdae\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "nz_nz_super": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/new_zealand_superannuation/core.yaml\",\"module_sha256\":\"7c7deabde0d97a94f045ee33addbbf8c3149af08ef64a777825422347d121d8e\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":669,\"rulespec_tree_sha256\":\"cc75602a3ec5858b8289b823c6b20c3e7bc40a980664dd128ab4c15b1d9c1fce\"}", + "registry_relation_housing": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"housing.yaml\",\"module_sha256\":\"17f2cc61ef6458a955dd26dbd9817c75ebb7251b52cd7857eadea6a2efa859b9\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"relations\":[{\"arguments\":[\"Family\",\"Person\"],\"cardinality\":\"many\",\"current_slot\":0,\"key\":\"family_member:0:1\",\"membership_column\":\"person_family_id\",\"period\":\"2026-27\",\"predicate\":\"family_member\",\"related_entity\":\"person\",\"related_inputs\":{\"member_income\":\"income\"},\"related_slot\":1,\"role_column\":\"person_family_role\",\"roles\":[\"head\",\"partner\"],\"root_entity\":\"family\"}],\"rulespec_commit\":\"cccccccccccccccccccccccccccccccccccccccc\",\"rulespec_tree_bytes\":21,\"rulespec_tree_sha256\":\"a095949c963e59c0aaeeac016989803e60ec6b83d2316428ea73eeb5021f7248\"}" +} diff --git a/packages/microcosm-build/tests/golden/nz_real_axiom_engine_refs.json b/packages/microcosm-build/tests/golden/nz_real_axiom_engine_refs.json new file mode 100644 index 0000000000..c71323c73f --- /dev/null +++ b/packages/microcosm-build/tests/golden/nz_real_axiom_engine_refs.json @@ -0,0 +1,8 @@ +{ + "accommodation_supplement": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/accommodation_supplement/core.yaml\",\"module_sha256\":\"f2feca3c2f1de71623ccd577cd2aed20e8822614aac0ccfc767b5faf4eaa83cc\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}", + "family_tax_credits": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/income_tax/family_scheme/tax_credits.yaml\",\"module_sha256\":\"d4bde4b4a72ec495dc516af1c95c9f438c0778d4f07cc02f64f242532a6efe05\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}", + "income_tax_schedule_1": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/income_tax/schedule_1/individual_income_tax.yaml\",\"module_sha256\":\"5ff8e7bf645b85e3d59d612e3c6eb70e09090fb13b40d49c3c5f8a4202e0b82a\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}", + "main_benefits_entitlement": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/main_benefits/entitlement.yaml\",\"module_sha256\":\"94332c778b3b5324798f67cb1b30f59c2af4131be8159455814f453bc814f8d0\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}", + "main_benefits_rates": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/social_security/main_benefits/rates.yaml\",\"module_sha256\":\"7c8ebbc3e62268722529ad790e3f561bbebf807d8a4d67bba24f93cb0809ec39\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}", + "nz_super": "{\"arithmetic\":\"decimal\",\"engine\":\"axiom-rules-engine\",\"engine_commit\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"engine_wheel_sha256\":\"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"entity_names\":{\"family\":\"Family\",\"household\":\"Household\",\"person\":\"Person\"},\"format\":\"microcosm.frame.axiom-engine-ref/1\",\"group_entities\":[\"household\",\"family\"],\"module\":\"nz/statutes/new_zealand_superannuation/core.yaml\",\"module_sha256\":\"d8ea0d15d97081031dcba242e66704c800bcf295e19d73bd173c56bd92d497b4\",\"nesting\":{\"family\":\"household\"},\"output_dtypes\":\"graph\",\"periods\":{\"2026-27\":{\"end\":\"2027-03-31\",\"kind\":\"tax_year\",\"start\":\"2026-04-01\"}},\"person_entity\":\"person\",\"rulespec_commit\":\"8dc2507d5f10a0467c285fed54d018ee37246798\",\"rulespec_tree_bytes\":3078546,\"rulespec_tree_sha256\":\"42ca3e76452a2ae8aaaf7c4407ae1e0f2915abb032039f73f2434f2a39f1203d\"}" +} diff --git a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py index b11c0f7070..c7078d9894 100644 --- a/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py +++ b/packages/microcosm-frame/src/microcosm/frame/adapters/axiom.py @@ -25,7 +25,18 @@ maps between them via ``entity_names`` (frame name -> engine name, default: capitalize). Engine entities outside the mapping (rulespec-be also defines ``Child``, ``Vehicle``, ...) are invisible to the kernel: their variables -resolve and materialize only once a frame entity is mapped to them. +resolve and materialize only once a frame entity is mapped to them, or a +binding-local view (:class:`AxiomView`) runs one over selected person rows. + +Binding-local views +------------------- +A view names an unmapped engine entity (its logical entity), one bool person +column (its selector) and an outside value per output. The adapter gathers +the selected person rows in table order, runs the module's program on the +logical entity over them, and scatters the outputs back through the same +mask; every unselected row holds the outside value. View outputs resolve to +the person entity, and the selector joins the person entity's whole-table +dependencies (:meth:`AxiomEngine.relation_dependencies`). RuleSpec authority roots ------------------------ @@ -123,6 +134,7 @@ "AxiomEntityTableDataset", "AxiomPeriod", "AxiomRelation", + "AxiomView", "BE_SCHEMA", "NZ_NESTING", "NZ_SCHEMA", @@ -326,6 +338,63 @@ def input_dependencies(self, schema: EntitySchema) -> tuple[tuple[str, str], ... return tuple(sorted(dependencies)) +@dataclass(frozen=True, eq=False) +class AxiomView: + """A binding-local view: one logical engine entity over selected persons. + + The engine entity ``logical_entity`` (rulespec-be's ``Child``) has no + frame table. The adapter gathers the person rows whose bool ``selector`` + column holds true, in table order, runs the module's program on + ``logical_entity`` over them, and writes one value to every person row: + the program's output on a selected row and the declared ``outside`` + value on every other row. ``outside`` maps each output on the logical + entity to that value; no output has an implicit fill. A view's root + inputs read the person columns of the same names. No rows or batches are + retained in this declaration. + """ + + logical_entity: str + selector: str + outside: Mapping[str, bool | int | float] + + def __post_init__(self) -> None: + for name in ("logical_entity", "selector"): + value = getattr(self, name) + if not isinstance(value, str) or not value: + raise ValueError(f"View {name} must be a non-empty string.") + if not isinstance(self.outside, Mapping) or not self.outside: + raise ValueError( + "View outside must map each output to its value outside the view." + ) + for name, value in self.outside.items(): + if not isinstance(name, str) or not name: + raise ValueError("View outside keys must be non-empty output names.") + if not isinstance(value, bool | int | float) or ( + isinstance(value, float) and not np.isfinite(value) + ): + raise ValueError( + f"View outside value for {name!r} must be a finite bool, " + "integer or decimal number." + ) + object.__setattr__(self, "outside", MappingProxyType(dict(self.outside))) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, AxiomView): + return NotImplemented + return canonical_json(self.to_dict()) == canonical_json(other.to_dict()) + + def __hash__(self) -> int: + return hash(canonical_json(self.to_dict())) + + def to_dict(self) -> dict[str, object]: + """Return the JSON declaration.""" + return { + "logical_entity": self.logical_entity, + "selector": {"column": self.selector}, + "outside": dict(sorted(self.outside.items())), + } + + @dataclass(frozen=True) class AxiomPeriod: """Explicit dense-execution bounds for one policy period. @@ -436,6 +505,15 @@ class AxiomEngine: output_dtypes: ``"native"`` (default) returns the dense surface's arrays unchanged; ``"graph"`` casts them to graph-ownable dtypes and refuses text and date outputs (see the module docstring). + relations: Optional explicit partition relations + (:class:`AxiomRelation`). + views: Optional binding-local views (:class:`AxiomView`), at most one + per logical entity. Each logical entity must be an engine entity + no frame entity maps to, and a view's outputs resolve to the + person entity. Views need ``output_dtypes="graph"`` (each outside + value casts through the graph dtype of its output) and exclude + ``relations``. Absent views leave every call and the engine + reference as they were without the field. The compiled dense programs (one per engine entity) and the module's variable metadata are loaded lazily and cached; constructing the adapter @@ -456,6 +534,7 @@ def __init__( nesting: Mapping[str, str] | None = None, output_dtypes: str = "native", relations: Sequence[AxiomRelation] | None = None, + views: Sequence[AxiomView] | None = None, ) -> None: if arithmetic not in ("decimal", "f64"): raise ValueError( @@ -536,7 +615,13 @@ def __init__( raise ValueError( f"Relation period {relation.period!r} is not declared." ) + self._views = None if views is None else tuple(views) + if self._views is not None: + self._validate_views(schema) self._programs: dict[str, Any] = {} + # View programs compile per logical entity under their own key, apart + # from the frame-entity programs above. + self._view_programs: dict[str, Any] = {} self._metadata: dict[str, Any] | None = None # Set by axiom_engine_ref: the RuleSpec tree digest this adapter's # reference names, and the file its module path resolved to. Every @@ -544,6 +629,77 @@ def __init__( self._pinned_tree_sha256: str | None = None self._pinned_module: Path | None = None + def _validate_views(self, schema: EntitySchema) -> None: + """Refuse a view declaration the adapter cannot execute. + + Raises: + TypeError: If an entry is not an :class:`AxiomView`. + ValueError: If the sequence is empty, a logical entity names a + mapped engine entity or repeats, a selector names a structural + column or a view output, views accompany relations, or the + adapter does not cast to graph dtypes. + """ + if not self._views: + raise ValueError("views must be None or name at least one view.") + if self._relations is not None: + raise ValueError("A binding declares views or relations, not both.") + if self._output_dtypes != "graph": + raise ValueError( + "Views require output_dtypes='graph': each outside value casts " + "through its output's graph dtype." + ) + if not all(isinstance(view, AxiomView) for view in self._views): + raise TypeError("views must contain AxiomView declarations.") + structural = {schema.person_id_column} | { + schema.membership_column(group) for group in schema.group_entities + } + outputs = {name for view in self._views for name in view.outside} + logical: set[str] = set() + for view in self._views: + if view.logical_entity in self._frame_entity_by_engine: + raise ValueError( + f"View logical entity {view.logical_entity!r} names an engine " + "entity a frame entity maps to; a view runs only an unmapped " + "engine entity." + ) + if view.logical_entity in logical: + raise ValueError( + f"Two views declare logical entity {view.logical_entity!r}." + ) + logical.add(view.logical_entity) + if view.selector in structural: + raise ValueError( + f"View selector {view.selector!r} names a structural column." + ) + if view.selector in outputs: + raise ValueError( + f"View selector {view.selector!r} names a view output." + ) + + def _view_for(self, engine_entity: str) -> AxiomView | None: + """The view whose logical entity is ``engine_entity``, if any.""" + for view in self._views or (): + if view.logical_entity == engine_entity: + return view + return None + + def view_root_inputs(self) -> dict[str, tuple[str, ...]]: + """Logical engine entity -> the root inputs its view's program reads. + + Each name reads the person column of the same name. An adapter with + no views returns an empty mapping. + + Raises: + ImportError: If ``axiom_rules_engine`` is not installed. + ValueError: If a view's logical entity has no derived rules. + """ + return { + view.logical_entity: tuple( + sorted(self._view_program(view.logical_entity).root_inputs) + ) + for view in self._views or () + } + # ------------------------------------------------------------------ # Variable metadata # ------------------------------------------------------------------ @@ -574,6 +730,9 @@ def variable_metadata(self, name: str) -> VariableMetadata: raise ValueError(f"Unknown Axiom variable {name!r}.") item = derived[name] frame_entity = self._frame_entity_by_engine.get(item.entity) + if frame_entity is None and self._view_for(item.entity) is not None: + # A view writes one value to every person row. + frame_entity = self._schema.person_entity if frame_entity is None: raise ValueError( f"Variable {name!r} lives on engine entity {item.entity!r}, " @@ -637,25 +796,49 @@ def variables(self) -> list[str]: for relation in self._relations: if relation.root_entity == frame_entity: names.update(relation.related_inputs) + for inputs in self.view_root_inputs().values(): + names.update(inputs) return sorted(names) def relation_dependencies(self, entity: str) -> tuple[tuple[str, str], ...]: - """Physical columns required to construct this root's relation batches.""" + """Physical columns required to construct this root's relation batches. + + A view's selector joins the person entity's dependencies: the view + gathers the rows it selects from the whole person table, so a node + must slice the selector over every row. A view's root inputs do not + join: the engine refuses a required input the slice lacks. + """ dependencies: set[tuple[str, str]] = set() for relation in self._relations or (): if relation.root_entity != entity: continue dependencies.update(relation.input_dependencies(self._schema)) + if entity == self._schema.person_entity: + dependencies.update((entity, view.selector) for view in self._views or ()) return tuple(sorted(dependencies)) def validate_input_override(self, entity: str, column: str) -> None: - """Refuse related overrides a bridge cannot apply to their source. - - Bridges write physical table columns after checking logical engine - inputs. An aliased relation input needs both names resolved together, - and only the related entity's table feeds a batch. Accepting either - override would silently leave the source intact. + """Refuse unread view selectors and related inputs without a source mapping. + + A view selector chooses batch rows. A selector that a program also + reads as a root input is an engine input: an override changes that + input and, through the column it writes, the rows each view selects. + Bridges refuse an override of any other selector as outside the + engine's inputs; this check names the selector instead. Bridges write + physical table columns after checking logical engine inputs. An + aliased relation input needs both names resolved together, and only + the related entity's table feeds a batch. Accepting either override + would silently leave the source intact. """ + if ( + entity == self._schema.person_entity + and any(view.selector == column for view in self._views or ()) + and column not in self.variables() + ): + raise ValueError( + f"Input overrides cannot change view selector {column!r}, which " + "no program reads as an input." + ) if not self._relations: return program = self._program(entity, missing_ok=True) @@ -886,10 +1069,17 @@ def materialize( self._require_schema(bundle) start, end, period_kind = self._materialization_period(period) + # Group requested names by execution root: a frame entity's program, + # or a view's program on its logical entity. by_entity: dict[str, list[str]] = {} + by_view: dict[str, list[str]] = {} for name in variables: metadata = self.variable_metadata(name) - by_entity.setdefault(metadata.entity, []).append(name) + view = self._view_for(self._derived_metadata()[name].entity) + if view is not None: + by_view.setdefault(view.logical_entity, []).append(name) + else: + by_entity.setdefault(metadata.entity, []).append(name) engine_dtypes: dict[str, str] = {} if self._output_dtypes == "graph": derived = self._derived_metadata() @@ -941,6 +1131,101 @@ def materialize( if self._output_dtypes == "graph": values = _graph_output(name, engine_dtypes[name], values) results[name] = values + for logical_entity, names in by_view.items(): + view = self._view_for(logical_entity) + results.update( + self._materialize_view( + bundle, + view, + names, + engine_dtypes, + dict(period_kind=period_kind, start=start, end=end), + ) + ) + return results + + def _view_selection(self, table: pd.DataFrame, view: AxiomView) -> np.ndarray: + """The boolean mask of the person rows ``view`` selects. + + Raises: + ValueError: If the person table lacks the selector, or the selector + is not a bool column or holds a null. + """ + if view.selector not in table.columns: + raise ValueError( + f"View {view.logical_entity!r} selector {view.selector!r} is not " + f"a column of the {self._schema.person_entity!r} table." + ) + column = table[view.selector] + if column.dtype.kind != "b": + raise ValueError( + f"View {view.logical_entity!r} selector {view.selector!r} has " + f"dtype {column.dtype}; a selector must be a bool column." + ) + if column.isna().any(): + raise ValueError( + f"View {view.logical_entity!r} selector {view.selector!r} holds a " + "null; a null selects neither way." + ) + return column.to_numpy(dtype=bool) + + def _materialize_view( + self, + bundle: Frame, + view: AxiomView, + names: Sequence[str], + engine_dtypes: Mapping[str, str], + bounds: Mapping[str, str], + ) -> dict[str, np.ndarray]: + """Run ``view``'s program on the rows it selects; fill every other row. + + Gather and scatter apply one boolean mask to one person table, so the + engine sees the selected rows in table order and each output lands on + the row it came from. An empty selection makes no engine call. + + Raises: + ValueError: As :meth:`_view_selection`; if an output has no outside + value, an outside value does not cast losslessly to the + output's graph dtype, or the engine's array length differs + from the selected count. + """ + table = bundle.table(self._schema.person_entity) + mask = self._view_selection(table, view) + n = len(table) + selected = int(mask.sum()) + missing = sorted(set(names) - set(view.outside)) + if missing: + raise ValueError( + f"View {view.logical_entity!r} declares no outside value for {missing}." + ) + results = { + name: _outside_values(name, engine_dtypes[name], view.outside[name], n) + for name in names + } + if not selected: + return results + program = self._view_program(view.logical_entity) + execute = program.execute_f64 if self._arithmetic == "f64" else program.execute + outputs = execute( + **bounds, + inputs=_batch_from_table(table.loc[mask], program.root_inputs), + outputs=list(names), + )["outputs"] + for name in names: + values = np.asarray(outputs[name]) + if values.shape != (selected,): + raise ValueError( + f"Materialized view variable {name!r} has shape " + f"{values.shape} but view {view.logical_entity!r} selects " + f"{selected} row(s)." + ) + out = results[name] + out[mask] = _graph_output(name, engine_dtypes[name], values) + if out.shape != (n,): + raise ValueError( + f"Scattered view variable {name!r} has shape {out.shape} but " + f"entity {self._schema.person_entity!r} has {n} row(s)." + ) return results def _materialization_period( @@ -1145,6 +1430,60 @@ def _program(self, frame_entity: str, *, missing_ok: bool = False) -> Any: self._metadata = {item.name: item for item in program.derived_metadata} return program + def _view_program(self, logical_entity: str) -> Any: + """The compiled dense program of the view on ``logical_entity``. + + Cached per logical entity, apart from the frame-entity programs. Like + :meth:`_program`, it re-checks the pinned RuleSpec tree before it + compiles, and it refuses a program with dense relations: a view + gathers no related rows. Its derived metadata joins the module's. + + Raises: + ImportError: If ``axiom_rules_engine`` is not installed. + ValueError: If no view declares ``logical_entity``, the module has + no derived rules on it, or the pinned tree moved. + NotImplementedError: If the program declares relations. + """ + if self._view_for(logical_entity) is None: + raise ValueError(f"No view declares logical entity {logical_entity!r}.") + if logical_entity in self._view_programs: + return self._view_programs[logical_entity] + if self._pinned_tree_sha256 is not None: + self._require_pinned_tree() + engine = self._import_engine() + try: + program = engine.CompiledDenseProgram.from_file( + self._module, + rulespec_roots=self._rulespec_roots, + entity=logical_entity, + ) + except ValueError as exc: + missing_entity = ( + "dense compilation could not find derived outputs for entity " + f"`{logical_entity}`" + ) + if str(exc) != missing_entity: + raise + raise ValueError( + f"Module {self._module.name!r} has no derived rules on the view's " + f"logical entity {logical_entity!r}." + ) from None + relations = list(program.relations) + if relations: + raise NotImplementedError( + f"Module {self._module_label()} declares dense relations " + f"{[item.name for item in relations]} on view entity " + f"{logical_entity!r}; a view gathers no related rows." + ) + self._view_programs[logical_entity] = program + metadata = {item.name: item for item in program.derived_metadata} + if self._metadata is None: + self._metadata = metadata + else: + for name, item in metadata.items(): + self._metadata.setdefault(name, item) + return program + def _require_pinned_tree(self) -> None: """Refuse to compile when the root no longer holds the referenced bytes. @@ -1180,11 +1519,20 @@ def _require_pinned_tree(self) -> None: ) def _derived_metadata(self) -> dict[str, Any]: - """Name -> authoring metadata for every derived rule in the module.""" - if self._metadata is None: - for frame_entity in self._schema.entities: - if self._program(frame_entity, missing_ok=True) is not None: - break + """Name -> authoring metadata for every derived rule in the module. + + With views, each view's program also contributes its metadata, so a + module whose only program is a view's still has metadata. + """ + if self._metadata is None or ( + self._views is not None and len(self._view_programs) < len(self._views) + ): + if self._metadata is None: + for frame_entity in self._schema.entities: + if self._program(frame_entity, missing_ok=True) is not None: + break + for view in self._views or (): + self._view_program(view.logical_entity) if self._metadata is None: raise ValueError( f"Module {self._module.name!r} has no derived rules on " @@ -1623,7 +1971,7 @@ def axiom_engine_ref( _require_clean_checkout(resolved_root, rulespec_commit) tree_sha256, tree_bytes = rulespec_tree_digest(resolved_root) if engine._pinned_tree_sha256 is None: - if engine._programs: + if engine._programs or engine._view_programs: raise ValueError( "axiom_engine_ref must name the adapter's RuleSpec tree before " "the adapter compiles: it has already compiled programs from " @@ -1673,6 +2021,9 @@ def axiom_engine_ref( } if engine._relations is not None: document["relations"] = [item.to_dict() for item in engine._relations] + # Absent views add no key, so every reference without them is unchanged. + if engine._views is not None: + document["views"] = [item.to_dict() for item in engine._views] return canonical_json(document).decode("utf-8") @@ -1753,7 +2104,9 @@ def assert_no_relations(engine: AxiomEngine, entity: str) -> None: Args: engine: The adapter to check. - entity: The frame entity the node materializes (``"family"``). + entity: The frame entity the node materializes (``"family"``), or + the logical engine entity of one of the adapter's views + (``"Child"``), whose own program is then checked. Raises: TypeError: If ``engine`` is not an :class:`AxiomEngine`. @@ -1764,6 +2117,9 @@ def assert_no_relations(engine: AxiomEngine, entity: str) -> None: """ if not isinstance(engine, AxiomEngine): raise TypeError(f"{engine!r} is not an AxiomEngine.") + if engine._view_for(entity) is not None: + engine._view_program(entity) + return relations = list(engine._program(entity).relations) if relations: raise NotImplementedError( @@ -1948,6 +2304,52 @@ def _graph_output(name: str, engine_dtype: str, values: np.ndarray) -> np.ndarra ) +def _outside_values( + name: str, engine_dtype: str, value: object, count: int +) -> np.ndarray: + """``count`` copies of a view's outside value, in its output's graph dtype. + + The value takes the numpy kind the engine returns for ``engine_dtype`` + (bool, int64 or float64) and passes through :func:`_graph_output`, so a + judgment code must be -1, 0 or 1. An integer stands for a decimal only + when float64 holds it exactly. + + Raises: + ValueError: If the value does not cast losslessly to the graph dtype. + """ + refused = ValueError( + f"View outside value {value!r} for {name!r} does not cast losslessly to " + f"its {engine_dtype!r} output." + ) + if engine_dtype == "bool": + if not isinstance(value, bool | np.bool_): + raise refused + array = np.full(max(count, 1), bool(value), dtype=np.bool_) + elif engine_dtype in ("integer", "judgment"): + if isinstance(value, bool | np.bool_) or not isinstance( + value, int | np.integer + ): + raise refused + if not np.iinfo(np.int64).min <= int(value) <= np.iinfo(np.int64).max: + raise refused + array = np.full(max(count, 1), int(value), dtype=np.int64) + elif engine_dtype == "decimal": + if isinstance(value, bool | np.bool_) or not isinstance( + value, int | float | np.integer | np.floating + ): + raise refused + cast = float(value) + if not np.isfinite(cast) or ( + isinstance(value, int | np.integer) and int(cast) != int(value) + ): + raise refused + array = np.full(max(count, 1), cast, dtype=np.float64) + else: + raise refused + # One element even for an empty table, so the cast always checks it. + return _graph_output(name, engine_dtype, array)[:count].copy() + + def _refuse_symlinks(root: Path) -> None: """Refuse a RuleSpec root holding symbolic links. diff --git a/packages/microcosm-frame/tests/engine_free/shared/test_axiom_views.py b/packages/microcosm-frame/tests/engine_free/shared/test_axiom_views.py new file mode 100644 index 0000000000..c84010455f --- /dev/null +++ b/packages/microcosm-frame/tests/engine_free/shared/test_axiom_views.py @@ -0,0 +1,1263 @@ +"""Binding-local view contract tests for the Axiom adapter, engine-free. + +A fake engine compiles fake dense programs per engine entity and evaluates a +fixed row-local function ``f``; it implements no RuleSpec formula. Real dense +execution of a view is the skip-gated differential (b) at the end. +""" + +import importlib.util +import json +import math +import os +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pandas as pd +import pytest +from hypothesis import example, given, settings +from hypothesis import strategies as st + +from microcosm.frame import Frame, WeightKind, Weights +from microcosm.frame.adapters.axiom import ( + AxiomEngine, + AxiomPeriod, + AxiomRelation, + AxiomView, + assert_no_relations, + axiom_engine_ref, +) +from microcosm.frame.kernels import SimulateRulesKernel +from microcosm.frame.rules_kernels import SimulateCounterfactualKernel +from microcosm.frame.schema import EntitySchema +from microcosm.graph import Node, Owned, Slice +from test_support.paths import paths_for + +ROOT = paths_for("microcosm-frame").tests / "fixtures/rulespec-zz" +MODULE = ROOT / "zz/policies/tests/axiom_toy_relations.yaml" +SCHEMA = EntitySchema(group_entities=("household",)) +NAMES = {"person": "Person", "household": "Household"} +PINS = dict( + engine_commit="a" * 40, + wheel_sha256="b" * 64, + rulespec_root=ROOT, + rulespec_commit="c" * 40, +) +OUTSIDE = {"child_amount": -7.5, "child_test": 0, "child_flag": False} + + +def f(age: np.ndarray, income: np.ndarray) -> dict[str, np.ndarray]: + """The fake program's row-local function; integer-valued decimals.""" + return { + "child_amount": age * 10.0 + income, + "child_test": np.where(age < 18, 1, -1).astype(np.int8), + "child_flag": age % 2 == 0, + "child_count": age // 3, + } + + +def _metadata(name, entity, dtype): + return SimpleNamespace(name=name, entity=entity, dtype=dtype, period="Year") + + +CHILD_METADATA = [ + _metadata("child_amount", "Child", "decimal"), + _metadata("child_test", "Child", "judgment"), + _metadata("child_flag", "Child", "bool"), + _metadata("child_count", "Child", "integer"), +] + + +class _Program: + """A fake dense program: records each call and evaluates ``f``.""" + + def __init__(self, entity, root_inputs, metadata, *, relations=(), function=f): + self.root_entity = entity + self.root_inputs = list(root_inputs) + self.relations = list(relations) + self.derived_metadata = list(metadata) + self.function = function + self.calls = [] + + def execute(self, **arguments): + self.calls.append(arguments) + inputs = arguments["inputs"] + values = self.function(*(inputs[name] for name in self.root_inputs)) + return {"outputs": {name: values[name] for name in arguments["outputs"]}} + + execute_f64 = execute + + +class _FakeEngine: + """Stands in for ``axiom_rules_engine``: compiles fake programs by entity.""" + + def __init__(self, programs): + self.programs = programs + self.compiled = [] + self.CompiledDenseProgram = self + + def from_file(self, module, *, rulespec_roots, entity): + self.compiled.append(entity) + if entity not in self.programs: + raise ValueError( + f"dense compilation could not find derived outputs for entity `{entity}`" + ) + return self.programs[entity] + + +def _child_program(**updates): + return _Program("Child", ("age", "income"), CHILD_METADATA, **updates) + + +def _view(**updates): + values = dict(logical_entity="Child", selector="is_child", outside=OUTSIDE) + values.update(updates) + return AxiomView(**values) + + +def _adapter(monkeypatch, programs=None, *, views=(), **updates): + values = dict( + module=MODULE, + rulespec_roots=(ROOT,), + schema=SCHEMA, + entity_names=NAMES, + output_dtypes="graph", + views=views if views != () else (_view(),), + ) + values.update(updates) + adapter = AxiomEngine(**values) + engine = _FakeEngine({"Child": _child_program()} if programs is None else programs) + monkeypatch.setattr(adapter, "_import_engine", lambda: engine) + return adapter, engine + + +def _frame(selected, age=None, income=None, ids=None): + n = len(selected) + people = pd.DataFrame( + { + "person_id": np.arange(1, n + 1) if ids is None else ids, + "person_household_id": np.ones(n, dtype=np.int64), + "is_child": np.asarray(selected, dtype=bool), + "age": np.arange(n, dtype=np.int64) if age is None else age, + "income": np.arange(n, dtype=np.float64) * 3.0 + if income is None + else income, + } + ) + return Frame( + tables={"person": people, "household": pd.DataFrame({"household_id": [1]})}, + schema=SCHEMA, + weights={"household": Weights(np.ones(1), WeightKind.DESIGN)}, + ) + + +def _materialize(adapter, selected, variables, age=None, income=None): + """Materialize through a Frame; an empty person table skips the Frame. + + A Frame refuses a table without rows (its weights cannot be empty), so a + zero-row person table reaches the view through ``_materialize_view`` + with the same graph dtypes ``materialize`` passes it. + """ + if len(selected): + return adapter.materialize(_frame(selected, age, income), variables, "2025") + table = pd.DataFrame( + { + "person_id": np.asarray([], dtype=np.int64), + "person_household_id": np.asarray([], dtype=np.int64), + "is_child": np.asarray([], dtype=bool), + "age": np.asarray([], dtype=np.int64), + "income": np.asarray([], dtype=np.float64), + } + ) + derived = adapter._derived_metadata() + return adapter._materialize_view( + SimpleNamespace(table=lambda entity: table), + adapter._views[0], + variables, + {name: derived[name].dtype for name in variables}, + dict(period_kind="calendar_year", start="2025-01-01", end="2025-12-31"), + ) + + +VARIABLES = ["child_amount", "child_test", "child_flag"] + + +# --------------------------------------------------------------------------- +# Declarations +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "updates,match", + [ + ({"logical_entity": ""}, "logical_entity"), + ({"selector": ""}, "selector"), + ({"outside": {}}, "outside"), + ({"outside": {"x": "zero"}}, "finite bool"), + ({"outside": {"x": float("nan")}}, "finite bool"), + ({"outside": {"": 0}}, "non-empty"), + ], +) +def test_view_declaration_shape_is_refused(updates, match): + with pytest.raises(ValueError, match=match): + _view(**updates) + + +def test_view_declaration_is_immutable_and_serializes_flat(): + source = {"child_amount": 1.0} + view = _view(outside=source) + source["child_amount"] = 2.0 + assert view.outside["child_amount"] == 1.0 + with pytest.raises(TypeError): + view.outside["child_amount"] = 3.0 + assert view.to_dict() == { + "logical_entity": "Child", + "selector": {"column": "is_child"}, + "outside": {"child_amount": 1.0}, + } + + +@pytest.mark.parametrize( + "updates,error,match", + [ + ({"views": []}, ValueError, "at least one view"), + ({"views": [{}]}, TypeError, "AxiomView"), + ({"views": [_view(logical_entity="Person")]}, ValueError, "maps to"), + ({"views": [_view(), _view()]}, ValueError, "Two views"), + ({"views": [_view(selector="person_id")]}, ValueError, "structural"), + ({"views": [_view(selector="person_household_id")]}, ValueError, "structural"), + ({"views": [_view(selector="child_flag")]}, ValueError, "view output"), + ({"output_dtypes": "native"}, ValueError, "output_dtypes='graph'"), + ], +) +def test_adapter_refuses_an_unexecutable_view(monkeypatch, updates, error, match): + with pytest.raises(error, match=match): + _adapter(monkeypatch, **updates) + + +def _selector_override(variables): + return { + "engine_ref": "fixture", + "variables": variables, + "period": "2025", + "input_overrides": [{"entity": "person", "column": "is_child", "value": True}], + } + + +def test_input_overrides_name_a_view_selector_in_the_refusal(monkeypatch): + adapter, engine = _adapter(monkeypatch) + with pytest.raises(ValueError, match="view selector 'is_child'"): + adapter.validate_input_override("person", "is_child") + kernel = SimulateCounterfactualKernel({"fixture": adapter}) + with pytest.raises(ValueError, match="view selector 'is_child'"): + kernel._evaluate( + _frame([True, False, False]), [_selector_override(["child_amount"])], [] + ) + assert engine.programs["Child"].calls == [] + + +def test_a_selector_that_a_program_reads_stays_overridable(monkeypatch): + """A Person program that reads the selector makes it an engine input. + + The override reaches that program's input and, through the column it + writes, the rows the view selects. + """ + person = _Program( + "Person", + ("is_child",), + [_metadata("adult_amount", "Person", "decimal")], + function=lambda selected: {"adult_amount": selected.astype(np.float64)}, + ) + child = _child_program() + adapter, _ = _adapter(monkeypatch, {"Person": person, "Child": child}) + assert "is_child" in adapter.variables() + adapter.validate_input_override("person", "is_child") + kernel = SimulateCounterfactualKernel({"fixture": adapter}) + kernel._evaluate( + _frame([True, False, False]), + [_selector_override(["adult_amount", "child_amount"])], + [], + ) + assert person.calls[-1]["inputs"]["is_child"].tolist() == [True, True, True] + assert child.calls[-1]["inputs"]["age"].tolist() == [0, 1, 2] + + +def test_adapter_refuses_views_with_relations(monkeypatch): + relation = AxiomRelation( + predicate="toy_member", + key="toy_member:0:1", + current_slot=0, + related_slot=1, + arguments=("Household", "Person"), + root_entity="household", + related_entity="person", + membership_column="person_household_id", + related_inputs={"toy_income": "income"}, + cardinality="many", + period="2025", + ) + with pytest.raises(ValueError, match="views or relations"): + _adapter(monkeypatch, relations=(relation,)) + + +# --------------------------------------------------------------------------- +# Materialization +# --------------------------------------------------------------------------- + + +def test_a_view_runs_the_logical_entity_on_selected_rows_only(monkeypatch): + adapter, engine = _adapter(monkeypatch) + frame = _frame([True, False, True, True, False]) + result = adapter.materialize(frame, VARIABLES, "2025") + program = engine.programs["Child"] + (call,) = program.calls + assert set(call) == {"period_kind", "start", "end", "inputs", "outputs"} + assert call["inputs"]["age"].tolist() == [0, 2, 3] + assert call["outputs"] == VARIABLES + assert result["child_amount"].tolist() == [0.0, -7.5, 26.0, 39.0, -7.5] + assert result["child_amount"].dtype == np.float64 + assert result["child_test"].tolist() == [1, 0, 1, 1, 0] + assert result["child_test"].dtype == np.int64 + assert result["child_flag"].tolist() == [True, False, True, False, False] + assert engine.compiled.count("Child") == 1 + + +def test_a_view_output_resolves_to_the_person_entity(monkeypatch): + adapter, _ = _adapter(monkeypatch) + metadata = adapter.variable_metadata("child_amount") + assert (metadata.entity, metadata.dtype, metadata.period) == ( + "person", + "float", + "year", + ) + assert adapter.graph_dtype("child_test") == "int64" + assert adapter.variables() == ["age", "income"] + assert adapter.view_root_inputs() == {"Child": ("age", "income")} + + +def test_a_child_output_without_a_view_keeps_todays_message(monkeypatch): + adapter = AxiomEngine( + MODULE, + SCHEMA, + rulespec_roots=(ROOT,), + entity_names=NAMES, + output_dtypes="graph", + ) + engine = _FakeEngine( + { + "Person": _Program( + "Person", + ("income",), + [_metadata("adult_amount", "Person", "decimal"), *CHILD_METADATA], + ) + } + ) + monkeypatch.setattr(adapter, "_import_engine", lambda: engine) + with pytest.raises(ValueError, match="which no frame entity is mapped to"): + adapter.variable_metadata("child_amount") + with pytest.raises(ValueError, match="which no frame entity is mapped to"): + adapter.materialize(_frame([True]), ["child_amount"], "2025") + + +def test_a_child_only_module_materializes(monkeypatch): + adapter, engine = _adapter(monkeypatch) + assert set(engine.compiled) <= {"Child"} + result = adapter.materialize(_frame([False, True]), ["child_amount"], "2025") + assert result["child_amount"].tolist() == [-7.5, 13.0] + assert "Person" in engine.compiled and "Household" in engine.compiled + + +@pytest.mark.parametrize( + "selector,match", + [ + (pd.array([True, None], dtype="boolean"), "null"), + (np.asarray([1, 0]), "bool column"), + (np.asarray(["y", "n"], dtype=object), "bool column"), + (None, "not a column"), + ], +) +def test_a_null_absent_or_non_bool_selector_refuses(monkeypatch, selector, match): + adapter, engine = _adapter(monkeypatch) + frame = _frame([True, False]) + if selector is None: + frame.table("person").drop(columns="is_child", inplace=True) + else: + frame.table("person")["is_child"] = selector + with pytest.raises(ValueError, match=match): + adapter.materialize(frame, VARIABLES, "2025") + assert engine.programs["Child"].calls == [] + + +def test_an_empty_selection_makes_no_engine_call(monkeypatch): + adapter, engine = _adapter(monkeypatch) + result = adapter.materialize(_frame([False, False, False]), VARIABLES, "2025") + assert engine.programs["Child"].calls == [] + assert result["child_amount"].tolist() == [-7.5] * 3 + assert result["child_test"].tolist() == [0] * 3 + empty = _materialize(adapter, [], VARIABLES) + assert {name: values.shape for name, values in empty.items()} == { + name: (0,) for name in VARIABLES + } + + +def test_the_batch_length_must_equal_the_selected_count(monkeypatch): + """A length-1 engine array would broadcast through ``out[mask] = values``.""" + + def one_value(age, income): + return {name: values[:1] for name, values in f(age, income).items()} + + adapter, engine = _adapter( + monkeypatch, {"Child": _child_program(function=one_value)} + ) + with pytest.raises(ValueError, match="selects 3 row"): + adapter.materialize(_frame([True, True, True]), ["child_amount"], "2025") + (call,) = engine.programs["Child"].calls + assert len(call["inputs"]["age"]) == 3 + + +@pytest.mark.parametrize( + "outside,match", + [ + ({"child_amount": 1.0}, "no outside value"), + ({**OUTSIDE, "child_test": 2}, "codes outside"), + ({**OUTSIDE, "child_test": 0.5}, "losslessly"), + ({**OUTSIDE, "child_test": True}, "losslessly"), + ({**OUTSIDE, "child_flag": 0}, "losslessly"), + ({**OUTSIDE, "child_amount": True}, "losslessly"), + ({**OUTSIDE, "child_amount": 2**60 + 1}, "losslessly"), + ], +) +def test_a_missing_or_uncastable_outside_value_refuses(monkeypatch, outside, match): + adapter, engine = _adapter(monkeypatch, views=(_view(outside=outside),)) + with pytest.raises(ValueError, match=match): + adapter.materialize(_frame([True, False]), VARIABLES, "2025") + assert engine.programs["Child"].calls == [] + + +def test_an_integer_outside_value_stands_for_an_exact_decimal(monkeypatch): + adapter, _ = _adapter( + monkeypatch, views=(_view(outside={**OUTSIDE, "child_amount": 4}),) + ) + result = adapter.materialize(_frame([False]), ["child_amount"], "2025") + assert result["child_amount"].dtype == np.float64 + assert result["child_amount"].tolist() == [4.0] + + +def test_a_relation_on_the_view_program_refuses(monkeypatch): + relation = SimpleNamespace(name="toy_member", key="toy_member:0:1") + adapter, _ = _adapter(monkeypatch, {"Child": _child_program(relations=[relation])}) + with pytest.raises(NotImplementedError, match="view entity 'Child'"): + adapter.materialize(_frame([True]), ["child_amount"], "2025") + adapter, _ = _adapter(monkeypatch, {"Child": _child_program(relations=[relation])}) + with pytest.raises(NotImplementedError, match="gathers no related rows"): + assert_no_relations(adapter, "Child") + + +def test_a_view_compile_refuses_after_the_pinned_tree_moves(monkeypatch, tmp_path): + root = tmp_path / "rulespec-zz" + module = root / "zz/policies/tests/axiom_toy_relations.yaml" + module.parent.mkdir(parents=True) + module.write_bytes(MODULE.read_bytes()) + adapter, engine = _adapter(monkeypatch, module=module, rulespec_roots=(root,)) + axiom_engine_ref(adapter, **{**PINS, "rulespec_root": root}) + (root / "zz/added.yaml").write_text("{}\n") + with pytest.raises(ValueError, match="changed after this adapter's engine_ref"): + adapter.view_root_inputs() + assert "Child" not in engine.compiled + + +def test_a_reference_refuses_an_adapter_whose_view_already_compiled(monkeypatch): + adapter, _ = _adapter(monkeypatch) + adapter.view_root_inputs() + with pytest.raises(ValueError, match="before the adapter compiles"): + axiom_engine_ref(adapter, **PINS) + + +def test_view_and_frame_programs_cache_under_separate_keys(monkeypatch): + """A module with a Person program and a Child view runs each on its own rows.""" + person = _Program( + "Person", + ("income",), + [_metadata("adult_amount", "Person", "decimal")], + function=lambda income: {"adult_amount": income + 0.5}, + ) + adapter, engine = _adapter( + monkeypatch, {"Person": person, "Child": _child_program()} + ) + frame = _frame([True, False, True]) + result = adapter.materialize(frame, ["adult_amount", "child_amount"], "2025") + assert result["adult_amount"].tolist() == [0.5, 3.5, 6.5] + assert result["child_amount"].tolist() == [0.0, -7.5, 26.0] + assert len(person.calls[-1]["inputs"]["income"]) == 3 + assert len(engine.programs["Child"].calls[-1]["inputs"]["income"]) == 2 + assert engine.compiled.count("Person") == 1 + assert engine.compiled.count("Child") == 1 + assert adapter._programs["person"] is person + assert adapter._view_programs["Child"] is engine.programs["Child"] + assert adapter.variable_metadata("adult_amount").entity == "person" + + +def test_the_selector_is_a_whole_table_person_dependency(monkeypatch): + adapter, _ = _adapter(monkeypatch) + assert adapter.relation_dependencies("person") == (("person", "is_child"),) + assert adapter.relation_dependencies("household") == () + + +def test_the_reference_carries_views_in_declaration_order(monkeypatch): + second = AxiomView("Vehicle", "has_vehicle", {"vehicle_tax": 0.0}) + adapter, _ = _adapter(monkeypatch, views=(_view(), second)) + document = json.loads(axiom_engine_ref(adapter, **PINS)) + assert document["views"] == [_view().to_dict(), second.to_dict()] + absent = AxiomEngine( + MODULE, + SCHEMA, + rulespec_roots=(ROOT,), + entity_names=NAMES, + output_dtypes="graph", + ) + assert "views" not in json.loads(axiom_engine_ref(absent, **PINS)) + + +# --------------------------------------------------------------------------- +# Properties P1 to P6 +# --------------------------------------------------------------------------- + + +@st.composite +def _population(draw, *, min_rows=0, max_rows=60): + n = draw(st.integers(min_value=min_rows, max_value=max_rows)) + selected = draw(st.lists(st.booleans(), min_size=n, max_size=n)) + ages = draw(st.lists(st.integers(0, 90), min_size=n, max_size=n)) + incomes = draw(st.lists(st.integers(-1000, 1000), min_size=n, max_size=n)) + outside = float(draw(st.integers(1, 1000)) * draw(st.sampled_from((-1, 1)))) + return ( + np.asarray(selected, dtype=bool), + np.asarray(ages, dtype=np.int64), + np.asarray(incomes, dtype=np.float64), + outside, + ) + + +_PROPERTY_OUTPUTS = { + "decimal": ("child_amount", np.dtype("float64")), + "integer": ("child_count", np.dtype("int64")), + "judgment": ("child_test", np.dtype("int64")), + "bool": ("child_flag", np.dtype("bool")), +} + + +def _property_outside(population, dtype): + outside = population[3] + return { + "decimal": float(outside), + "integer": int(outside), + "judgment": 1 if outside > 0 else -1, + "bool": outside > 0, + }[dtype] + + +def _edge_population(selected): + n = len(selected) + return ( + np.asarray(selected, dtype=bool), + np.arange(n, dtype=np.int64), + np.arange(n, dtype=np.float64) * 3.0, + 5.0, + ) + + +def _edge_examples(*, shuffled=False): + """Replay every graph dtype at zero rows, no rows selected and all selected.""" + + def decorate(test): + for dtype in _PROPERTY_OUTPUTS: + for selected in ([], [False, False], [True, True]): + arguments = dict(population=_edge_population(selected), dtype=dtype) + if shuffled: + arguments["seed"] = 0 + test = example(**arguments)(test) + return test + + return decorate + + +def _run(population, *, dtype="decimal", function=f): + selected, ages, incomes, _ = population + name, _ = _PROPERTY_OUTPUTS[dtype] + outside = _property_outside(population, dtype) + with pytest.MonkeyPatch.context() as patch: + adapter, engine = _adapter( + patch, + {"Child": _child_program(function=function)}, + views=(_view(outside={name: outside}),), + ) + result = _materialize(adapter, selected, [name], ages, incomes) + return result[name], engine.programs["Child"] + + +@settings(max_examples=60, deadline=None, database=None) +@_edge_examples() +@given(population=_population(), dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS))) +def test_p1_one_output_per_row(population, dtype): + """An exact oracle checks every row in each graph dtype.""" + selected, ages, incomes, _ = population + name, graph_dtype = _PROPERTY_OUTPUTS[dtype] + outside = _property_outside(population, dtype) + out, program = _run(population, dtype=dtype) + expected = np.full(len(selected), outside, dtype=graph_dtype) + expected[selected] = f(ages[selected], incomes[selected])[name] + assert out.dtype == graph_dtype + assert np.array_equal(out, expected) + assert len(program.calls) == int(selected.any()) + + +@settings(max_examples=60, deadline=None, database=None) +@_edge_examples() +@given(population=_population(), dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS))) +def test_p2_outside_rows_hold_the_outside_value(population, dtype): + selected, _, _, _ = population + _, graph_dtype = _PROPERTY_OUTPUTS[dtype] + outside = _property_outside(population, dtype) + out, _ = _run(population, dtype=dtype) + assert out.dtype == graph_dtype + assert (out[~selected] == outside).all() + + +@settings(max_examples=60, deadline=None, database=None) +@_edge_examples() +@given(population=_population(), dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS))) +def test_p3_conservation_is_exact(population, dtype): + """Integer-valued decimals and all discrete graph totals compare exactly.""" + selected, ages, incomes, _ = population + name, _ = _PROPERTY_OUTPUTS[dtype] + outside = _property_outside(population, dtype) + out, _ = _run(population, dtype=dtype) + view_total = f(ages[selected], incomes[selected])[name].sum() + assert out.sum() == view_total + outside * int((~selected).sum()) + + +_RELATION = AxiomRelation( + predicate="toy_member", + key="toy_member:0:1", + current_slot=0, + related_slot=1, + arguments=("Household", "Person"), + root_entity="household", + related_entity="person", + membership_column="person_household_id", + related_inputs={"toy_income": "income"}, + cardinality="many", + period="2025", +) + + +@settings(max_examples=40, deadline=None, database=None) +@given( + relations=st.sampled_from((None, (), (_RELATION,))), + arithmetic=st.sampled_from(("decimal", "f64")), + output_dtypes=st.sampled_from(("native", "graph")), + population=_population(min_rows=1, max_rows=12), +) +def test_p4_no_views_no_change(relations, arithmetic, output_dtypes, population): + """``views=None`` leaves the call, dependencies and variables as before.""" + selected, ages, incomes, _ = population + calls = [] + for spelled in (False, True): + program = _Program( + "Household", + ("scale",), + [_metadata("toy_total", "Household", "decimal")], + relations=[] + if not relations + else [ + SimpleNamespace( + name="toy_member", + key="toy_member:0:1", + current_slot=0, + related_slot=1, + related_inputs=("toy_income",), + ) + ], + function=lambda scale: {"toy_total": scale * 1.0}, + ) + options = dict( + relations=relations, arithmetic=arithmetic, output_dtypes=output_dtypes + ) + if spelled: + options["views"] = None + adapter = AxiomEngine( + MODULE, SCHEMA, rulespec_roots=(ROOT,), entity_names=NAMES, **options + ) + engine = _FakeEngine({"Household": program}) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(adapter, "_import_engine", lambda engine=engine: engine) + reference = json.loads(axiom_engine_ref(adapter, **PINS)) + patch.setattr( + "microcosm.frame.adapters.axiom._source_relation_arguments", + lambda *args: ("Household", "Person"), + ) + patch.setattr( + "microcosm.frame.adapters.axiom.AxiomEngine._relation_batches", + lambda self, bundle, entity, period: {"toy_member:0:1": "batch"}, + ) + frame = _frame(selected, ages, incomes) + frame.table("household")["scale"] = np.ones(1) + adapter.materialize(frame, ["toy_total"], "2025") + calls.append( + ( + sorted(program.calls[-1]), + {k: v.tolist() for k, v in program.calls[-1]["inputs"].items()}, + adapter.relation_dependencies("person"), + adapter.relation_dependencies("household"), + adapter.variables(), + reference, + ) + ) + keys = calls[-1][0] + assert "views" not in calls[-1][5] + assert ("relations" in keys) == bool(relations) + assert calls[0] == calls[1] + + +def _base_materialize(adapter, bundle, variables, period): + """A reference copy of main 4a36edcf's ``AxiomEngine.materialize``. + + The no-view path regroups requested names by execution root at PR 1's + head; this copy keeps the base grouping, so P4 compares outputs with it. + """ + from microcosm.frame.adapters.axiom import ( + _GRAPH_OUTPUT_DTYPES, + _batch_from_table, + _graph_output, + ) + + adapter._require_schema(bundle) + start, end, period_kind = adapter._materialization_period(period) + by_entity = {} + for name in variables: + metadata = adapter.variable_metadata(name) + by_entity.setdefault(metadata.entity, []).append(name) + engine_dtypes = {} + if adapter._output_dtypes == "graph": + derived = adapter._derived_metadata() + engine_dtypes = {name: derived[name].dtype for name in variables} + assert all(dtype in _GRAPH_OUTPUT_DTYPES for dtype in engine_dtypes.values()) + results = {} + for frame_entity, names in by_entity.items(): + program = adapter._program(frame_entity) + if program.relations or adapter._relations: + adapter.assert_declared_relations(frame_entity) + table = bundle.table(frame_entity) + inputs = _batch_from_table(table, program.root_inputs) + execute = ( + program.execute_f64 if adapter._arithmetic == "f64" else program.execute + ) + arguments = dict( + period_kind=period_kind, + start=start, + end=end, + inputs=inputs, + outputs=list(names), + ) + if program.relations: + arguments["relations"] = adapter._relation_batches( + bundle, frame_entity, period + ) + outputs = execute(**arguments)["outputs"] + expected = bundle.n(frame_entity) + for name in names: + values = np.asarray(outputs[name]) + assert values.shape == (expected,) + if adapter._output_dtypes == "graph": + values = _graph_output(name, engine_dtypes[name], values) + results[name] = values + return results + + +def _no_view_programs(): + person = _Program( + "Person", + ("age", "income"), + [ + _metadata("adult_amount", "Person", "decimal"), + _metadata("adult_test", "Person", "judgment"), + _metadata("adult_flag", "Person", "bool"), + _metadata("adult_count", "Person", "integer"), + _metadata("toy_total", "Household", "decimal"), + ], + function=lambda age, income: { + "adult_amount": age * 2.5 + income, + "adult_test": np.where(age > 40, 1, np.where(age > 20, 0, -1)).astype( + np.int8 + ), + "adult_flag": income > 0, + "adult_count": age // 3, + }, + ) + household = _Program( + "Household", + ("scale",), + person.derived_metadata, + function=lambda scale: {"toy_total": scale * 3.0}, + ) + return {"Person": person, "Household": household} + + +_NO_VIEW_NAMES = ("adult_amount", "adult_test", "adult_flag", "adult_count") + + +@settings(max_examples=40, deadline=None, database=None) +@given( + arithmetic=st.sampled_from(("decimal", "f64")), + output_dtypes=st.sampled_from(("native", "graph")), + names=st.lists( + st.sampled_from((*_NO_VIEW_NAMES, "toy_total")), min_size=1, unique=True + ), + population=_population(min_rows=1, max_rows=20), +) +def test_p4_no_views_outputs_equal_the_base_path( + arithmetic, output_dtypes, names, population +): + """``views=None`` returns the base adapter's tables: names, order, dtypes, values.""" + selected, ages, incomes, _ = population + returned = [] + for materialize in (AxiomEngine.materialize, _base_materialize): + adapter = AxiomEngine( + MODULE, + SCHEMA, + rulespec_roots=(ROOT,), + entity_names=NAMES, + arithmetic=arithmetic, + output_dtypes=output_dtypes, + views=None, + ) + engine = _FakeEngine(_no_view_programs()) + with pytest.MonkeyPatch.context() as patch: + patch.setattr(adapter, "_import_engine", lambda engine=engine: engine) + frame = _frame(selected, ages, incomes) + frame.table("household")["scale"] = np.full(1, 7.0) + result = materialize(adapter, frame, names, "2025") + returned.append( + ( + list(result), + [(values.dtype, values.tolist()) for values in result.values()], + [ + (sorted(call), call["outputs"]) + for program in engine.programs.values() + for call in program.calls + ], + ) + ) + assert returned[0] == returned[1] + + +@settings(max_examples=60, deadline=None, database=None) +@_edge_examples(shuffled=True) +@given( + population=_population(), + seed=st.integers(0, 2**32 - 1), + dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS)), +) +def test_p5_shuffling_rows_shuffles_outputs(population, seed, dtype): + selected, ages, incomes, outside = population + order = np.random.default_rng(seed).permutation(len(selected)) + out, _ = _run(population, dtype=dtype) + shuffled, _ = _run( + (selected[order], ages[order], incomes[order], outside), dtype=dtype + ) + assert np.array_equal(shuffled, out[order]) + + +@settings(max_examples=60, deadline=None, database=None) +@_edge_examples() +@given(population=_population(), dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS))) +def test_p6_each_selected_output_is_f_of_that_row_alone(population, dtype): + selected, ages, incomes, _ = population + name, _ = _PROPERTY_OUTPUTS[dtype] + out, _ = _run(population, dtype=dtype) + for index in np.flatnonzero(selected): + alone = f(ages[index : index + 1], incomes[index : index + 1])[name] + assert out[index] == alone[0] + + +def _batch_dependent(age, income): + """Varies with the batch: each output adds the batch's position.""" + values = f(age, income) + values["child_amount"] = values["child_amount"] + np.arange(len(age)) + return values + + +def test_p6_detects_an_engine_whose_output_depends_on_other_rows(): + population = ( + np.asarray([True, True]), + np.asarray([3, 4]), + np.asarray([1.0, 2.0]), + 5.0, + ) + out, _ = _run(population, function=_batch_dependent) + alone = f(np.asarray([4]), np.asarray([2.0]))["child_amount"][0] + assert out[1] != alone + + +# --------------------------------------------------------------------------- +# Differential (a): the mapped-entity path against the view path, engine-free +# --------------------------------------------------------------------------- + +CHILD_SCHEMA = EntitySchema(person_entity="child", group_entities=("household",)) + + +@settings(max_examples=40, deadline=None, database=None) +@example(population=_edge_population([True, True]), dtype="decimal") +@example(population=_edge_population([True, True]), dtype="integer") +@example(population=_edge_population([True, True]), dtype="judgment") +@example(population=_edge_population([True, True]), dtype="bool") +@given( + population=_population(min_rows=1, max_rows=30).filter(lambda item: item[0].any()), + dtype=st.sampled_from(tuple(_PROPERTY_OUTPUTS)), +) +def test_differential_a_mapped_child_entity_against_the_view(population, dtype): + """An adapter whose person entity maps to Child, and the view, agree per child.""" + selected, ages, incomes, _ = population + name, _ = _PROPERTY_OUTPUTS[dtype] + outside = _property_outside(population, dtype) + with pytest.MonkeyPatch.context() as patch: + view, _ = _adapter(patch, views=(_view(outside={name: outside}),)) + through_view = view.materialize(_frame(selected, ages, incomes), [name], "2025") + mapped = AxiomEngine( + MODULE, + CHILD_SCHEMA, + rulespec_roots=(ROOT,), + entity_names={"child": "Child", "household": "Household"}, + output_dtypes="graph", + ) + engine = _FakeEngine({"Child": _child_program()}) + patch.setattr(mapped, "_import_engine", lambda: engine) + count = int(selected.sum()) + children = Frame( + tables={ + "child": pd.DataFrame( + { + "child_id": np.arange(1, count + 1), + "child_household_id": np.ones(count, dtype=np.int64), + "age": ages[selected], + "income": incomes[selected], + } + ), + "household": pd.DataFrame({"household_id": [1]}), + }, + schema=CHILD_SCHEMA, + weights={"household": Weights(np.ones(1), WeightKind.DESIGN)}, + ) + direct = mapped.materialize(children, [name], "2025") + assert through_view[name][selected].tolist() == direct[name].tolist() + assert through_view[name].dtype == direct[name].dtype + + +# --------------------------------------------------------------------------- +# Placement: view code stays in axiom.py (source_hash does not follow imports) +# --------------------------------------------------------------------------- + + +def test_axiom_imports_are_pinned_everywhere_in_the_module(): + """Every import node, in function bodies and relative ones too, is pinned. + + A view helper moved to a new module, or into one axiom.py already imports + (``materialize``, ``concept_mapping``), would leave the simulate.* hashes, + which hash axiom.py's own file, unmoved by an edit to it. + """ + import ast + + source = ( + paths_for("microcosm-frame").package / "src/microcosm/frame/adapters/axiom.py" + ).read_text() + imported = set() + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.ImportFrom): + base = "." * node.level + (node.module or "") + if node.module is None: + imported.update(f"{base}{alias.name}" for alias in node.names) + else: + imported.add(base) + elif isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + first_party = { + name + for name in imported + if name.startswith("microcosm") or name.startswith(".") + } + assert first_party == { + "microcosm.frame.bundle", + "microcosm.frame.concept_mapping", + "microcosm.frame.materialize", + "microcosm.frame.rules", + "microcosm.frame.schema", + "microcosm.graph.canonical", + } + # Lazy engine imports in function bodies are third-party, not first-party. + assert {"axiom_rules_engine", "axiom_rules_engine.dense", "yaml"} <= imported + + +def test_view_code_is_defined_in_the_adapter_module(): + """Code re-exported from another module would carry that module's name.""" + from microcosm.frame.adapters import axiom + + view_code = ( + AxiomView, + AxiomEngine._validate_views, + AxiomEngine._view_for, + AxiomEngine.view_root_inputs, + AxiomEngine._view_program, + AxiomEngine._view_selection, + AxiomEngine._materialize_view, + AxiomEngine.materialize, + AxiomEngine.relation_dependencies, + AxiomEngine._derived_metadata, + axiom._outside_values, + axiom._graph_output, + axiom._batch_from_table, + ) + assert {item.__module__ for item in view_code} == {"microcosm.frame.adapters.axiom"} + + +# --------------------------------------------------------------------------- +# Differential (b): the real engine, skip-gated +# --------------------------------------------------------------------------- + +_ENGINE = importlib.util.find_spec("axiom_rules_engine") +if _ENGINE is not None: + from axiom_rules_engine.dense import NativeCompiledDenseProgram +else: + NativeCompiledDenseProgram = None + +#: rulespec-be at 530849e: ``be.json`` module 61 (0-based), 107 Child roots. +_BE_ROOT = os.environ.get("POPULACE_RULESPEC_BE") +_BE_MODULE = "be/statutes/family_benefits/regional_routing.yaml" +_BE_MODULE_SHA256 = "408e73587e60d91136f80a71612802cb8c111f7ea5baf5abaeecb772c43c2413" +_GRAPH_KINDS = ("bool", "integer", "decimal", "judgment") +#: The values of the Child roots whose names mislead :func:`_real_inputs`'s +#: suffix rule, typed as the written cases in rulespec-be's own tests at +#: 530849e type them. The routing region takes all four region codes, +#: Flanders 0 to the German-speaking Community 3. +_REAL_INPUT_VALUES = { + "belgium_family_benefits_region": (0, 1, 2, 3), + **dict.fromkeys( + ( + "brussels_family_benefits_child_domiciled_in_brussels_capital_region", + "flanders_family_benefits_child_domiciled_in_dutch_language_region", + "german_speaking_community_family_benefits_child_domiciled_in_german_language_region", + "german_speaking_community_family_benefits_foreign_stay_or_settlement_authorized_more_than_three_months", + "wallonia_family_benefits_child_legal_domicile_in_french_language_region", + "wallonia_family_benefits_child_without_legal_domicile_effectively_resides_in_french_language_region", + "wallonia_family_benefits_parents_eu_or_reg883_and_economic_activity_in_french_region", + ), + (False, True), + ), + "flanders_family_benefits_disability_tier": tuple(range(5)), + "flanders_family_benefits_orphan_status_code": tuple(range(3)), + "wallonia_family_benefits_disability_tier": tuple(range(8)), + "german_speaking_community_family_benefits_disability_category": tuple(range(4)), + "german_speaking_community_family_benefits_benefit_year": (2025, 2026, 2027, 2028), + "german_speaking_community_family_benefits_article_10_extension_months_used": tuple( + range(26) + ), + "german_speaking_community_family_benefits_months_since_disappearance": tuple( + range(26) + ), + **dict.fromkeys( + ( + "german_speaking_community_family_benefits_current_index_factor", + "german_speaking_community_family_benefits_pre_2025_index_factor", + ), + (1.0, 1.1, 1.2, 1.4), + ), +} + + +def _real_inputs(names, n, generator): + """Typed random inputs by name: RuleSpec inputs carry no dtype (axiom#62). + + A name in ``_REAL_INPUT_VALUES`` draws from its values there. Otherwise + counts, ages and years become integers, money becomes decimals, and + every other input a bool. The run on the pinned engine checks the + typing where a program needs a number: it reports a bool, text or date + column there as a type mismatch, though the message doesn't name the + input. Names draw in sorted order, because ``root_inputs`` comes back in + a different order in each process, and the seed fixes the batch only for + one order. + """ + columns = {} + for name in sorted(names): + if name in _REAL_INPUT_VALUES: + columns[name] = generator.choice(np.asarray(_REAL_INPUT_VALUES[name]), n) + elif name.endswith(("_years", "_count", "_months", "_rank")): + columns[name] = generator.integers(0, 26, n).astype(np.int64) + elif name.endswith(("_income", "_amount", "_eur")): + columns[name] = generator.integers(0, 60_000, n).astype(np.float64) + else: + columns[name] = generator.integers(0, 2, n).astype(bool) + return columns + + +def test_real_inputs_preserve_types_regions_and_seed_across_root_orders(): + """Misleading suffixes keep the types of the pinned RuleSpec examples.""" + kinds = { + "belgium_family_benefits_region": "i", + "flanders_family_benefits_child_domiciled_in_dutch_language_region": "b", + "flanders_family_benefits_disability_tier": "i", + "german_speaking_community_family_benefits_current_index_factor": "f", + "german_speaking_community_family_benefits_foreign_stay_or_settlement_authorized_more_than_three_months": "b", + } + forward = _real_inputs(kinds, 200, np.random.default_rng(530849)) + assert {name: values.dtype.kind for name, values in forward.items()} == kinds + assert set(forward["belgium_family_benefits_region"]) == {0, 1, 2, 3} + reverse = _real_inputs(reversed(kinds), 200, np.random.default_rng(530849)) + assert all(np.array_equal(forward[name], reverse[name]) for name in kinds) + + +@pytest.mark.skipif( + NativeCompiledDenseProgram is None or not _BE_ROOT, + reason=( + "Axiom dense native extension is not installed, or POPULACE_RULESPEC_BE " + "does not name a rulespec-be 530849e export" + ), +) +def test_differential_b_real_child_program_against_the_view(): + """The real Child program agrees with the view, and is row-local per batch. + + Shuffled, single-row and random-subset batches must give each row the + value the full batch gives it: the only test that can falsify + row-locality inside the engine. + """ + import hashlib + + from axiom_rules_engine import CompiledDenseProgram + + root = Path(_BE_ROOT).resolve() + module = root / _BE_MODULE + assert hashlib.sha256(module.read_bytes()).hexdigest() == _BE_MODULE_SHA256 + program = CompiledDenseProgram.from_file( + module, rulespec_roots=(root,), entity="Child" + ) + assert not list(program.relations) + assert "is_child" not in program.root_inputs + dtypes = { + item.name: item.dtype + for item in program.derived_metadata + if item.entity == "Child" and item.dtype in _GRAPH_KINDS + } + outputs = sorted(dtypes) + assert outputs + generator = np.random.default_rng(530849) + n = 40 + batch = _real_inputs(program.root_inputs, n, generator) + bounds = dict(period_kind="calendar_year", start="2026-01-01", end="2026-12-31") + full = program.execute(**bounds, inputs=batch, outputs=outputs)["outputs"] + assert any(len(set(np.asarray(full[name]).tolist())) > 1 for name in outputs) + for order in ( + generator.permutation(n), + np.asarray([0]), + np.flatnonzero(generator.integers(0, 2, n)), + ): + part = program.execute( + **bounds, + inputs={name: values[order] for name, values in batch.items()}, + outputs=outputs, + )["outputs"] + for name in outputs: + assert ( + np.asarray(part[name]).tolist() + == np.asarray(full[name])[order].tolist() + ) + # The view over persons: the children are the batch's rows, in order, + # among adults the selector leaves out. + selected = np.zeros(2 * n, dtype=bool) + selected[generator.choice(2 * n, n, replace=False)] = True + people = pd.DataFrame( + { + "person_id": np.arange(1, 2 * n + 1), + "person_household_id": np.ones(2 * n, dtype=np.int64), + "is_child": selected, + } + ) + for name, values in batch.items(): + column = np.zeros(2 * n, dtype=values.dtype) + column[selected] = values + people[name] = column + outside = { + name: {"bool": False, "integer": 0, "judgment": 0}.get(dtypes[name], 0.0) + for name in outputs + } + adapter = AxiomEngine( + module, + SCHEMA, + rulespec_roots=(root,), + entity_names=NAMES, + output_dtypes="graph", + periods={"2026": AxiomPeriod("2026-01-01", "2026-12-31", "calendar_year")}, + views=(AxiomView("Child", "is_child", outside),), + ) + frame = Frame( + tables={"person": people, "household": pd.DataFrame({"household_id": [1]})}, + schema=SCHEMA, + weights={"household": Weights(np.ones(1), WeightKind.DESIGN)}, + ) + through_view = adapter.materialize(frame, outputs, "2026") + for name in outputs: + expected = np.asarray(full[name]) + assert through_view[name][selected].tolist() == expected.tolist() + assert (through_view[name][~selected] == outside[name]).all() + if dtypes[name] == "decimal": + # P3 on the real engine. math.fsum rounds the exact total once, + # so neither the order of the rows nor the outside rows can move + # it, as float64 sums in numpy's order can. + assert math.fsum(through_view[name]) == math.fsum( + [ + *expected.astype(np.float64), + *[outside[name]] * int((~selected).sum()), + ] + ) + + +# --------------------------------------------------------------------------- +# Rules kernels: the selector is a whole-table dependency +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("person", "match"), + [ + (Slice("person", ("age", "income")), "whole-table relation inputs"), + ( + Slice("person", ("is_child", "age", "income"), rows="is_child"), + "whole-table relation inputs", + ), + (Slice("person", ("is_child", "age", "income")), None), + ], + ids=("no-selector", "masked", "whole"), +) +def test_every_rules_kernel_requires_the_whole_selector(monkeypatch, person, match): + adapter, _ = _adapter(monkeypatch) + node = Node( + "view", + "simulate.rules_by_ref@1", + population="create", + inputs=(person,), + outputs=(Owned("person", "child_amount", "float64"),), + params={"engine_ref": "fixture", "variables": ("child_amount",)}, + ) + if match is None: + SimulateRulesKernel._require_relation_inputs(node, adapter) + else: + with pytest.raises(ValueError, match=match): + SimulateRulesKernel._require_relation_inputs(node, adapter) + + +def test_view_equality_preserves_outside_value_types_and_hashes(): + left = AxiomView("Child", "s", {"a": 0.0, "b": 0}) + right = AxiomView("Child", "s", {"a": 0, "b": False}) + assert left != right + equal = AxiomView("Child", "s", {"b": 0, "a": 0.0}) + assert left == equal + assert hash(left) == hash(equal) + assert len({left, right, equal}) == 2 + assert left != left.to_dict() diff --git a/test_support/microcosm_build/axiom_ref_bindings.py b/test_support/microcosm_build/axiom_ref_bindings.py new file mode 100644 index 0000000000..dfd031822c --- /dev/null +++ b/test_support/microcosm_build/axiom_ref_bindings.py @@ -0,0 +1,189 @@ +"""NZ binding inputs frozen at ``ef398616`` for the synthetic reference golden. + +This copy includes the entity names, periods and six complete binding rows. +``axiom_ref_goldens.nz_document`` replaces module hashes for its synthetic +tree. Keep this snapshot independent of NZ's live binding resource; changes +require an explicit review of the base-reference golden. +""" + +NZ_BINDINGS_SNAPSHOT = { + "entity_names": {"person": "Person", "household": "Household", "family": "Family"}, + "periods": { + "2026-27": {"kind": "tax_year", "start": "2026-04-01", "end": "2027-03-31"} + }, + "bindings": [ + { + "id": "main_benefits_entitlement", + "rulespec_path": "nz/statutes/social_security/main_benefits/entitlement.yaml", + "sha256": "94332c778b3b5324798f67cb1b30f59c2af4131be8159455814f453bc814f8d0", + "entity": "person", + "engine_entity": "Person", + "period": "2026-27", + "role": "receipt_layer_eligibility", + "variables": [ + "entitled_to_jobseeker_support", + "entitled_to_sole_parent_support", + "entitled_to_supported_living_payment_restricted_or_blind", + "entitled_to_supported_living_payment_caring", + ], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 60, + "notes": "Entitlement predicates only; the weekly rates and income-test " + "reductions live in the rates module. Undetermined judgments " + "feed take-up through the G4 input-closure table.", + }, + { + "id": "main_benefits_rates", + "rulespec_path": "nz/statutes/social_security/main_benefits/rates.yaml", + "sha256": "7c8ebbc3e62268722529ad790e3f561bbebf807d8a4d67bba24f93cb0809ec39", + "entity": "person", + "engine_entity": "Person", + "period": "2026-27", + "role": "receipt_layer_rates_and_as_rate_bridge", + "variables": [ + "jobseeker_support_net_weekly_payment", + "sole_parent_support_net_weekly_payment", + "supported_living_payment_net_weekly_payment", + "jobseeker_support_gross_weekly_rate", + "jobseeker_support_weekly_income_test_reduction", + ], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 28, + "notes": "Also evaluated by the reg 17/18 bridge (as_rate_bridge.json) " + "under declared input overrides.", + }, + { + "id": "nz_super", + "rulespec_path": "nz/statutes/new_zealand_superannuation/core.yaml", + "sha256": "d8ea0d15d97081031dcba242e66704c800bcf295e19d73bd173c56bd92d497b4", + "entity": "person", + "engine_entity": "Person", + "period": "2026-27", + "role": "receipt_layer_eligibility", + "variables": [ + "entitled_to_new_zealand_superannuation", + "nz_super_weekly_amount_before_tax", + ], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 13, + "notes": "NZ Superannuation eligibility for the O4-calibrated receipt " + "layer (MC6).", + }, + { + "id": "accommodation_supplement", + "rulespec_path": "nz/statutes/social_security/accommodation_supplement/core.yaml", + "sha256": "f2feca3c2f1de71623ccd577cd2aed20e8822614aac0ccfc767b5faf4eaa83cc", + "entity": "family", + "engine_entity": "Family", + "period": "2026-27", + "role": "entitlement", + "variables": [ + "accommodation_supplement_rounded_weekly_payment", + "accommodation_supplement_weekly_amount_before_rounding", + "accommodation_supplement_non_rate_entitlement_requirements", + "accommodation_supplement_cash_assets_within_limit", + "accommodation_supplement_social_housing_exclusion_not_applies", + "accommodation_supplement_other_funding_exclusion_not_applies", + "accommodation_supplement_non_beneficiary_income_reduction", + "accommodation_supplement_weekly_qualifying_accommodation_costs", + "accommodation_supplement_maximum_weekly_rate", + ], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 39, + "notes": "The module takes the reg 17 base rate and reg 18 " + 'non-beneficiary cutout as inputs (its summary: "for later ' + 'cross-module wiring"); as_rate_bridge.json supplies them ' + "until rulespec-nz encodes regs 17 and 18. Calibration-tier " + "variants add their own AS nodes.", + }, + { + "id": "family_tax_credits", + "rulespec_path": "nz/statutes/income_tax/family_scheme/tax_credits.yaml", + "sha256": "d4bde4b4a72ec495dc516af1c95c9f438c0778d4f07cc02f64f242532a6efe05", + "entity": "family", + "engine_entity": "Family", + "period": "2026-27", + "role": "as_rate_bridge_only", + "variables": ["family_tax_credit_prescribed_amount"], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 21, + "notes": "Used only for the eldest-child family tax credit amount in " + "the reg 17 base rate. The module also declares 4 rules on an " + "engine Child entity that the NZ schema does not map; the " + "bridge requests none of them, and G1/G7 must show the adapter " + "compiles the module under that condition or stop. The G8-w4 " + "review ran the real engine, at the commit the input closure's " + "input_surface.engine names, against the pinned rulespec " + "commit above and the input-surface commit: the output " + "returned 7921 with only the two care-unit inputs at 1 and 0 " + "and the other 15 Family roots at arbitrary values, dropping " + "either input raised missing input, AxiomEngine.variables() " + "listed the 17 Family roots and no Child name, and " + "simulate.counterfactual@1 delivered both inputs as overrides " + "with neither column on the frame. A second walk over the " + "engine's parsed model reached exactly the two inputs.", + "unreached_roots": { + "sha256": "d4bde4b4a72ec495dc516af1c95c9f438c0778d4f07cc02f64f242532a6efe05", + "outputs": ["family_tax_credit_prescribed_amount"], + "names": [ + "child_tax_credit_for_entitlement_period", + "entitled_to_in_work_tax_credit", + "family_tax_credit_entitlement_days", + "in_work_tax_credit_allowed_children_count", + "in_work_tax_credit_weekly_periods", + "minimum_family_adjusted_income_tax_liability", + "minimum_family_amount_paid", + "minimum_family_amount_received", + "minimum_family_full_time_earner_weeks", + "minimum_family_scheme_income_attributable_to_full_time_weeks", + "minimum_family_tax_credit_weekly_periods", + "parental_tax_credit_additional_abatement", + "parental_tax_credit_for_entitlement_period", + "wff_family_credit_abatement_days", + "wff_family_scheme_income_for_relationship_period", + ], + "reason": "family_tax_credit_prescribed_amount " + "reads only the eldest and subsequent " + "dependent child care units, so no " + "request this binding makes reaches these " + "15 Family roots. The axiom:nz concept " + "mapping binds " + "in_work_tax_credit_allowed_children_count, " + "but no requested rule reads it, so the " + "input-closure producer drops that " + "binding from the mapping it checks. A " + "binding that requests another output, or " + "a module edit that moves the SHA-256 " + "above, makes the producer refuse until a " + "reviewer re-verifies this list.", + }, + }, + { + "id": "income_tax_schedule_1", + "rulespec_path": "nz/statutes/income_tax/schedule_1/individual_income_tax.yaml", + "sha256": "5ff8e7bf645b85e3d59d612e3c6eb70e09090fb13b40d49c3c5f8a4202e0b82a", + "entity": "person", + "engine_entity": "Person", + "period": "2026-27", + "role": "benefit_gross_up_tax_schedule", + "variables": ["individual_income_tax_before_credits"], + "relations_declared": False, + "imports_declared": False, + "root_input_count": 1, + "notes": "https://github.com/TheAxiomFoundation/rulespec-nz/blob/8dc2507d5f10a0467c285fed54d018ee37246798/nz/statutes/income_tax/schedule_1/individual_income_tax.yaml " + "(SHA-256 " + "5ff8e7bf645b85e3d59d612e3c6eb70e09090fb13b40d49c3c5f8a4202e0b82a); " + "https://www.workandincome.govt.nz/products/benefit-rates/benefit-rates-april-2026.html. " + "Each gross-up inverts annualised Schedule 1 before credits " + "over 52 weeks independently of other income, with " + "solver-owned taxable_income and no ACC earners levy. NZ Super " + "supplies before-tax amounts directly; the gross-up never " + "inverts it.", + }, + ], +} diff --git a/test_support/microcosm_build/axiom_ref_goldens.py b/test_support/microcosm_build/axiom_ref_goldens.py new file mode 100644 index 0000000000..ab54ac22b0 --- /dev/null +++ b/test_support/microcosm_build/axiom_ref_goldens.py @@ -0,0 +1,199 @@ +"""Engine-free ``axiom_engine_ref`` documents for NZ-shaped and relation rows. + +No NZ CI test builds an Axiom reference for NZ's rows: NZ's engine-free graph +injects Python toy engines. These helpers build native references through +``build_transport_registry`` over a synthetic tree that holds NZ's six +binding paths, with the six rows frozen in ``axiom_ref_bindings`` at +``ef398616``, ``unreached_roots`` included, and over the relation fixtures. +The live NZ bindings do not affect this synthetic tree. The golden +``axiom_engine_refs.json`` holds the documents +generated at the base of S2b PR 1 (main ``4a36edcf``); a reference that moves +fails the byte comparison. + +To regenerate this golden intentionally, call ``engine_refs`` on a fresh +scratch directory, then write its result to ``GOLDEN`` with +``json.dumps(refs, indent=1, sort_keys=True) + "\\n"``. Keep the frozen source +rows unchanged when NZ's live bindings move. The separate real-tree golden +uses the live bindings; its test documents that regeneration procedure. + +Preparation compiles nothing here: the module checks that would compile a +program (``assert_no_relations`` and ``assert_declared_relations``) and the +output metadata reads are replaced, because a reference needs the tree, not +the engine. +""" + +from __future__ import annotations + +import copy +import hashlib +from collections.abc import Mapping +from pathlib import Path + +import pytest + +import microcosm.build.transport.registry as registry_module +from microcosm.build.transport.registry import build_transport_registry +from microcosm.frame import VariableMetadata +from microcosm.frame.adapters.axiom import ( + AxiomEngine, + AxiomRelation, + axiom_engine_ref, +) +from microcosm.frame.schema import EntitySchema +from test_support.microcosm_build.axiom_ref_bindings import NZ_BINDINGS_SNAPSHOT +from test_support.microcosm_build.transport_population import UNIT_RULE +from test_support.paths import paths_for + +NZ_BINDINGS = ( + paths_for("microcosm-build").package + / "src/microcosm/build/nz/axiom_rules_bindings.json" +) +GOLDEN = paths_for("microcosm-build").tests / "golden/axiom_engine_refs.json" +PINS = { + "engine": {"commit": "a" * 40, "wheel_sha256": "b" * 64}, + "rulespec": {"commit": "c" * 40}, +} +RELATION_ROOT = paths_for("microcosm-frame").tests / "fixtures/rulespec-zz" +RELATION_MODULE = RELATION_ROOT / "zz/policies/tests/axiom_toy_relations.yaml" + + +def _module_bytes(path: str) -> bytes: + return f"# synthetic module for the reference golden: {path}\nrules: []\n".encode() + + +def nz_document(root: Path) -> dict: + """Frozen NZ binding rows over a synthetic tree written below ``root``.""" + source = copy.deepcopy(NZ_BINDINGS_SNAPSHOT) + rows = [] + for row in source["bindings"]: + payload = _module_bytes(row["rulespec_path"]) + module = root / row["rulespec_path"] + module.parent.mkdir(parents=True, exist_ok=True) + module.write_bytes(payload) + digest = hashlib.sha256(payload).hexdigest() + row = copy.deepcopy(row) + row["sha256"] = digest + if "unreached_roots" in row: + row["unreached_roots"]["sha256"] = digest + rows.append(row) + return { + **PINS, + "entity_names": source["entity_names"], + "periods": source["periods"], + "bindings": rows, + } + + +def relation_row(root: Path) -> dict: + """The registry relation fixture: one family module with a role selector.""" + payload = b'{"entity": "family"}\n' + (root / "housing.yaml").write_bytes(payload) + return { + "id": "housing", + "rulespec_path": "housing.yaml", + "sha256": hashlib.sha256(payload).hexdigest(), + "entity": "family", + "engine_entity": "Family", + "period": "2026-27", + "variables": ["toy_housing"], + "relations": [ + { + "predicate": "family_member", + "key": "family_member:0:1", + "current_slot": 0, + "related_slot": 1, + "arguments": ["Family", "Person"], + "root_entity": "family", + "related_entity": "person", + "membership_column": "person_family_id", + "related_inputs": {"member_income": "income"}, + "cardinality": "many", + "period": "2026-27", + "role_column": "person_family_role", + "roles": ["head", "partner"], + } + ], + } + + +def _adapter_relation_refs() -> dict[str, str]: + """The adapter-level relation fixture's references (frame relation tests).""" + relation = AxiomRelation( + predicate="toy_member", + key="toy_member:0:1", + current_slot=0, + related_slot=1, + arguments=("TaxUnit", "Person"), + root_entity="tax_unit", + related_entity="person", + membership_column="person_tax_unit_id", + related_inputs={"toy_income": "income"}, + cardinality="many", + period="2025", + ) + pins = dict( + engine_commit="a" * 40, + wheel_sha256="b" * 64, + rulespec_root=RELATION_ROOT, + rulespec_commit="c" * 40, + ) + refs = {} + for name, relations in ( + ("absent", None), + ("empty", ()), + ("declared", (relation,)), + ): + adapter = AxiomEngine( + RELATION_MODULE, + EntitySchema(group_entities=("household", "tax_unit")), + rulespec_roots=(RELATION_ROOT,), + entity_names={ + "person": "Person", + "household": "Household", + "tax_unit": "TaxUnit", + }, + nesting={"tax_unit": "household"}, + relations=relations, + ) + refs[f"adapter_relations_{name}"] = axiom_engine_ref(adapter, **pins) + return refs + + +def engine_refs(root: Path) -> dict[str, str]: + """Every golden reference, built engine-free below ``root``.""" + root.mkdir(parents=True, exist_ok=True) + nz_root = root / "nz-tree" + document = nz_document(nz_root) + relation_root = root / "relation-tree" + relation_root.mkdir() + relation_document = { + **copy.deepcopy(document), + "bindings": [relation_row(relation_root)], + } + entity_by_variable: Mapping[str, str] = { + variable: row["entity"] + for row in (*document["bindings"], *relation_document["bindings"]) + for variable in row["variables"] + } + with pytest.MonkeyPatch.context() as patch: + patch.setattr(registry_module, "assert_no_relations", lambda *args: None) + patch.setattr( + AxiomEngine, "assert_declared_relations", lambda engine, entity: None + ) + patch.setattr( + AxiomEngine, + "variable_metadata", + lambda engine, name: VariableMetadata( + name, entity_by_variable[name], "float", "year" + ), + ) + patch.setattr(AxiomEngine, "graph_dtype", lambda engine, name: "float64") + nz = build_transport_registry(document, nz_root, unit_rule=UNIT_RULE) + relation = build_transport_registry( + relation_document, relation_root, unit_rule=UNIT_RULE + ) + return { + **{f"nz_{binding}": ref for binding, ref in nz.engine_refs.items()}, + "registry_relation_housing": relation.engine_refs["housing"], + **_adapter_relation_refs(), + }