feat(sql-plan): port SQLPlanGenerator consumers of the production model fields

[dmautz1]

Summary

Second PR in the series porting production-hardened capabilities from the pulseflow fork of tablespec back upstream. Stacked on #18 (feat/port-umf-model-deltas) — the diff includes #18's commits until it merges; review this PR's own change as feat/port-umf-model-deltas...feat/port-sql-plan-consumers. Draft until #18 lands.

#18 landed the model fields as declarative stubs; this PR wires every SQL-generation field through its SQLPlanGenerator / RelationshipResolver consumers, per the series convention: nothing stays a model-only no-op. As with #18, everything is adapted to this repo's structure (single-statement views, CTE-mode contract, renderer seam, engine-agnostic SQL executed verbatim on DuckDB and Spark) rather than copied — and several capabilities the fork never actually had (per-branch row_filter on union branches, per-branch window dedup, target-schema branch projection) are new design pinned by a real downstream acceptance shape.

Per-field consumer wiring

base_table_strategy: 'union_branches' (new enum value — see model note below)
The base table plus each union_base_tables entry (falling back to source_tables) becomes one UNION branch inside a single disposition_base view. Unlike the fork's union_base_tables handling (which projects the base table's schema from every branch), each branch projects the target column set through that source's own derivation candidates — required for cutover shapes where sources have different columns:

• DerivationCandidate.row_filter → the branch WHERE clause (the single distinct value among a branch's candidates; conflicts raise ValueError). This is how generation cutovers are expressed.
• DerivationCandidate.union_value → CAST(<literal> AS <type>) per branch (source discriminators); native str/int/float/bool typing preserved end-to-end.
• Columns absent from a branch → CAST(NULL AS <type>), keeping the UNION column-aligned.
• dedup_strategy: latest + candidate order_by → per-branch ROW_NUMBER() OVER (PARTITION BY <target primary_key> ORDER BY <order_by> DESC NULLS LAST) ... WHERE __rn = 1 (NULLS LAST pinned — DuckDB/Spark default NULL placement diverges).
• union_type → UNION ALL (default) / UNION.
• union_exclude_base → per-union-branch NOT EXISTS anti-join on the target primary key against the base branch's post-filter, post-dedup rows (deliberate improvement over the fork's raw-base anti-join). No primary key → ValueError (the fork skipped silently — a correctness foot-gun).
• union_coalesce_base → 3-part union (base-only / COALESCE(b.c, u.c) overlap with base winning pk, meta, and union_value columns / union-only). Restricted to exactly one union table: the fork's per-table 3-part emission would double-count base rows with several union tables → ValueError.
• Joins after a union base keep working: join-source columns are projected into every branch (typed NULL where absent), and final assembly short-circuits branch columns to bare base.<col> (the branch already applied the candidate mapping under the target name).

base_table_filter → WHERE on the plain base view and on the union base branch (closing a fork gap — it never applied the filter on union paths). Warns and no-ops under unpivot/union_sources, which don't consume it.

base_join_column → overrides the inferred base join key in both resolver and base view and overwrites source_column on relationships declared outgoing from the base table (fork parity; the field exists precisely when the auto key is wrong and declared rels carry that same wrong key — documented contract).

final_filter / final_dedup → final assembly wraps as SELECT [DISTINCT *] FROM (<assembly>) _final WHERE <filter> so the filter can reference derived aliases; applies to the synthetic (no-base) path too; still one statement → CTE-mode safe.

ForeignKey.join_filter → new resolver pass fills JoinInfo.join_filter from FK metadata only where no candidate-level filter exists (candidate filters are (table, table_instance)-keyed and can disambiguate multi-instance joins, so they win). Emission machinery already existed and was conformance-tested.

OutgoingRelationship.alternative_joins → emitted as a UNION-of-joins, not ON (a = b OR c = d): Spark plans OR-joins as BroadcastNestedLoopJoin. One inner-join branch per path over the distinct base keys (primary = priority 1, alternatives in declared order), UNION + ROW_NUMBER by __branch_priority keeps one match per key, joined back null-safely via the portable (a = b OR (a IS NULL AND b IS NULL)) expansion (no <=>, no * EXCEPT, no engine hints). base_keys scans disposition_base when every key is base-sourced (the fork's documented lazy-view fan-out guard), else the previous step view with a warning. Resolver validates each entry's columns exist (ValueError otherwise). Non-direct strategies warn and use the primary path only.

Model change

UMFMetadata.base_table_strategy narrows from str | None to Literal["union_sources", "unpivot", "union_branches"] | None. Repo-wide only the two existing values were in use. union_base_tables present without the strategy logs a warning and no-ops (not warnings.warn — the repo's filterwarnings = error would hard-fail legacy-shaped fixtures; not a validation error — fork-authored UMFs must still load). Migration for fork specs: add base_table_strategy: union_branches.

Also in this PR

• Latent join bug fix: base views now project join-source columns (incl. alternative/join_via keys) that survive no derivation candidate — previously the emitted ON base.<key> = ... referenced columns disposition_base never selected.
• Excel round-trip: Union Value column on the Derivations sheet (native typing, appended header so older workbooks import unchanged); Metadata sheet now JSON-encodes list/dict values (str(list) previously broke union_base_tables — and source_tables — on re-import). F009-DERIV-02 losslessness test extended: a union_branches spec's CTE plan is byte-identical across an Excel round-trip.
• umf.schema.json synced for exactly the fields this PR consumes (the 8 UMFMetadata fields incl. the new enum, union_value, ForeignKey.join_filter, alternative_joins); the full drift regeneration remains AR-2026-03-16's follow-up.
• Docs: new docs/guide/sql-plans.md (strategies, filters, dedup, join controls, error philosophy), linked from happy-path §5; excel.md Union Value row; docs/api/generators.md gains generate_sql_plan/SQLPlanGenerator.

Deliberately NOT ported

• Fork's _rewrite_join_filter hardcoded client-specific rewrites (client_mbr_id → ClientMemberId)
• Fork's derived-column materialization CTE for alternative joins (no upstream driver yet)
• The naive OR-in-ON alternative-joins emission (perf hazard the fork itself warns about)
• Base-schema union projection (superseded by target-schema branch projection)
• Union-vs-union coalesce overlap semantics (pairwise base-vs-union only, matching the fork)
• Fork's silent skips on missing primary keys (replaced by plan-time ValueErrors)
• Next batch (unchanged from feat(models): port production UMF model extensions from pulseflow #18's roadmap): ingestion/upsert consumers — merge_strategy, update_mode, pre/post-upsert rules, effective_primary_key

Acceptance shape

The unit suite pins the real downstream cutover this port unblocks (synaptiq-northstar-idr's silver_fact_inventory_line): two feeds unioned at a DATE '2026-07-20' cutover via complementary row_filters, a source_generation discriminator via union_value, one-sided columns NULL-cast, per-branch ROW_NUMBER PARTITION BY arbit_id, cpt, dos, snapshot_date ORDER BY meta_load_dt DESC — compared via sqlglot normalization, not exact text. The conformance twin (gold_union_branches) executes the same shape end-to-end with a portable DATE filter and matches the committed Spark-oracle golden on both DuckDB and Spark (+ pairwise agreement).

Testing

• 50 new unit tests (tests/unit/test_sql_plan_consumers.py) covering every field's consumer wiring, every ValueError path, CTE-mode single-statement + both-dialect parseability, and the acceptance fixture; model Literal tests; 2 new Excel round-trip tests
• New conformance case gold_union_branches (corpus + Spark-generated golden), green on both engine legs
• Full local run post-change: 3,835 passed, 120 skipped under filterwarnings = error; conformance 226 passed; zero existing goldens changed (everything is opt-in); ruff clean; pyright 0 errors on touched modules

🤖 Generated with Claude Code