refactor(common): split DataArray conversion into a 3-rung strictness ladder

[FBumann]

Stacked on #732. Resolves the open questions from the #732 review thread by re-layering linopy.common's DataArray conversion. The design was developed in this PR's comment thread.

What

Two public entry points — replacing the as_dataarray-with-flag + private _as_dataarray_lax + align_to_coords trio:

as_dataarray(arr, coords, dims)                              convert: type dispatch + positional labeling
broadcast_to_coords(arr, coords, dims, *,                    + broadcast: project MI levels, reorder,
                    strict=True, label=...)                    expand missing dims, transpose to coords order

strict decides what happens to anything broadcasting alone cannot resolve (extra dims, disagreeing coord values, MI coverage gaps):

• strict=True (default): raise, naming label in the error (label is required in this mode — enforced via overloads and at runtime). Forgetting the flag adds safety instead of silently dropping validation.
• strict=False: pass through for downstream xarray alignment (the operator's join= owns reconciliation).

Backed by one private mechanics function (_broadcast_to_coords) that reports MultiIndex projections (_LevelProjection) instead of deciding what they mean; the public function applies policy per mode.

Caller profiles

Caller	Call
__matmul__ (2 sites) — broadcasting would contract a=(time,name) @ b=(name,location) to (location) instead of (time,location)	as_dataarray(other, coords=…, dims=…)
add_variables / add_constraints bounds + mask	broadcast_to_coords(lower, coords, label="lower bound")
Expression arithmetic, to_linexpr, as_expression (constraint lhs/rhs setters)	broadcast_to_coords(other, coords=…, strict=False)

MultiIndex policy (scenario B — decided in this thread)

Terminology: a stacked MultiIndex dim has levels (its component index names, e.g. period / timestep) and level combinations (its elements — one tuple per position, e.g. (2030, 't1')).

Implicit level projections are deprecated everywhere and will raise under the v1 convention — the MI check is the same in both modes:

MI situation	strict=False (arithmetic)	strict=True (bounds/mask)
input misses a whole level (per-period bounds)	EvolvingAPIWarning	EvolvingAPIWarning
coverage gap (some level combinations get no value)	EvolvingAPIWarning	ValueError — the error lists the missing combinations

The warning channel carries a TODO(#738): migrate EvolvingAPIWarning → LinopySemanticsWarning once #717 lands.

This removes the enforce_level_coverage flag and the cross-module use of a private helper — the two things the #732 thread flagged.

Behavior changes

One, deliberate (scenario B): add_variables / add_constraints with inputs indexed by a subset of a MultiIndex's levels (e.g. PyPSA's per-period bounds) now emit the EvolvingAPIWarning deprecation warning. Everything else keeps the semantics it has on #732. (Also caught in review and fixed: the constraint lhs/rhs setters go through as_expression, which now uses strict=False — with regression tests.)

Tests

• Strictness-contrast tests: as_dataarray doesn't expand / broadcast_to_coords(strict=False) passes mismatches through / strict=True rejects them with labeled errors, stays silent on partial-level bounds.
• test_matmul_contracts_only_shared_dims: pins (dim_0,dim_1) @ (dim_1,location) → (dim_0,location).
• Constraint rhs setter regressions: missing-dim broadcast + MultiIndex-level projection.
• Full suite: 3200 passed, 36 skipped. mypy + pre-commit clean.

Out of scope (agreed in the thread)

• Migration recipe documentation for the deprecated implicit projections — belongs with refactor: unify coords-as-truth handling in add_variables/add_constraints #732's MI follow-up; must avoid Cannot reindex onto a stacked MultiIndex via indexers — only reindex_like works pydata/xarray#11368 (reindex onto a stacked MI is broken upstream).
• reindex_to_coords — deferred until a coords-only caller exists; blocked on Cannot reindex onto a stacked MultiIndex via indexers — only reindex_like works pydata/xarray#11368.
• Warning channel unification (EvolvingAPIWarning → LinopySemanticsWarning) — Unify deprecation warning channels before v1: EvolvingAPIWarning vs LinopySemanticsWarning #738, blocked on feat: v1 semantic convention #717.

🤖 Generated with Claude Code

[FBumann]

@FabianHofmann For me the whole broadcasting and alignment in linopy is at least as difficult to undnerstand as the arithmetics. And arithmetics also rely on this broacasting. SO i think getting this right, with clear methods that have distinct, understandable roles is a really important step.

Thats why im putting this much effort into it.

And thanks for your work aout Multiindex. Im not really that good in that area

[FabianHofmann]

@FBumann thanks for taking another look, I read the code and the tests. it all makes sense. it is a complicated thing and the time spent here is definitely worth it. I am wondering whether we are already at the optimum.

still thinking about the arithmetics the generalization we have there is the join parameter. I wonder if we should/could have an equivalent for the coords alignment even though this is a asymetric case where the reference (coords) are immutable.

I would see that the final API could be

align_to_coords(arr, coords, join="exact")

equivalent to

align(expr, expr2, join="exact") # what we already have

most of the calls of as_dataarray + reindex/broadcase calls that we have could be replaced by one align_to_coords call. we could also make that a follow up, the only thing would be collapsing the broadcast/align function again. here is detailed plan from claude we we could add this here:

Details

Convergence target

as_dataarray(arr, coords, dims)                          # convert only — matmul
align_to_coords(arr, coords, *, join="defer",            # convert + broadcast + reconcile [+ validate]
                enforce_dims=False, fill_value=NA,
                dims=None, label=None)

Two public functions instead of three. broadcast_to_coords stops being a public name (or stays as a one-line deprecated alias for align_to_coords(join="defer")).

What to change in #737

1. Don't ship broadcast_to_coords as a public function. It's the thing we'd deprecate next PR. Instead, rename the private mechanics _broadcast_to_coords → _align_to_coords, and make align_to_coords the single public reconcile entry point with a join parameter. The three #737 presets become arguments:

#737 public name	becomes
broadcast_to_coords(arr, coords)	align_to_coords(arr, coords, join="defer")
align_to_coords(arr, coords, label=…)	align_to_coords(arr, coords, join="exact", enforce_dims="raise", label=…)
(new)	align_to_coords(arr, coords, join="left", fill_value=…)

2. Put join in the one reconcile block — nothing else moves. The only mechanic that varies is the per-dim loop #737 left at common.py ~425–435 ("Same values, different order → reindex; different value sets are left alone"). Make that block switch on join:

for dim, coord_values in expected.items():
    ...
    if actual_idx.equals(expected_idx):
        continue
    same_set = len(actual_idx) == len(expected_idx) and set(actual_idx) == set(expected_idx)
    if join == "defer":
        if same_set:                      # today's behavior verbatim
            arr = arr.reindex({dim: expected_idx})
    elif join == "left":
        arr = arr.reindex({dim: expected_idx}, fill_value=fill_value)   # NEW
    elif join == "override":
        if len(actual_idx) == len(expected_idx):
            arr = arr.assign_coords({dim: expected_idx})
    # join == "exact": leave values; the contract check raises (see #3)

join="defer" reproduces #737 byte-for-byte, so the arithmetic path is unchanged. left is the new capability; exact defers the raising to validation.

3. Make the policy a parameter keyed off enforce_dims (tri-state), folding in the _LevelProjection report. This is where the considerations from earlier land. One policy block in align_to_coords replaces the two preset blocks:

enforce_dims: Literal[False, "warn", "raise"] = False

concern	enforce_dims=False (arithmetic)	"warn" (to_constraint rhs)	"raise" (bounds/mask)
extra dims (arr.dims ⊄ coords)	ignore	warn	raise (validate_alignment)
MI partial-level projection	warn	warn	silent (documented bounds feature)
MI coverage gap	warn	warn	raise
shared-dim value mismatch	governed by join	join	join="exact" ⇒ raise

The "warn" rung is the must-have: to_constraint rhs (expressions.py:1109–1116) warns and proceeds on extra dims — a bare enforce_dims=True would wrongly raise there. The tri-state is what lets that warning move into the function instead of being hand-rolled at the call site.

4. Exclude HELPER_DIMS in validate_alignment's extra-dims check. #737's validate_alignment flags any dim not in coords. The constraint coeffs/vars setters carry _term, so without exclusion enforce_dims="raise" would reject them later. Mirror what align() already does (exclude=frozenset(...).union(HELPER_DIMS), common.py:1840). Zero behavior change today (bounds never carry _term); it just unblocks the setter migration.

5. Migrate call sites in #737 (atomically).

• Arithmetic (_add_constant :586, _apply_constant_op :612, as_expression): broadcast_to_coords → align_to_coords(..., join="defer"). These still feed _align_constant afterwards — leave that untouched (it owns the symmetric inner/outer join, which stays out of scope).
• Strict sites must change in the same commit (model.py:777/778/791/1060): today's bare align_to_coords(value, coords, label=…) must become align_to_coords(..., join="exact", enforce_dims="raise", label=…). If the new default is join="defer", leaving them bare would silently stop validating — that's the one correctness trap, so it has to be atomic.
• matmul: stays on as_dataarray (convert-only). The whole point — join can't express its no-broadcast need, and refactor(common): split DataArray conversion into a 3-rung strictness ladder #737 already got this right.

6. Demonstrate join="left" by collapsing two real dances now (behavior-preserving, since as_dataarray + reindex_like(fill) ≡ align_to_coords(join="left", fill_value=fill)):

• variables.py:330-332 to_linexpr coeff: 3 lines → align_to_coords(coeff, self.coords, dims=self.dims, join="left", fill_value=0).
• expressions.py:1107-1116 to_constraint rhs: → align_to_coords(rhs, self.coords, join="left", fill_value=NA, enforce_dims="warn").

This makes the join param used, not speculative — proving the design against the exact callers that motivated it.

What stays out of scope (and why — confirmed by the exploration)

• merge / _align_constant's explicit join — symmetric inner/outer, mutate the reference. Untouched.
• The per-field fill_value dict in Constraints.reindex — Dataset-level, not single-array. Untouched.
• The bare-DataArray(value).broadcast_like(...) setters (constraints.py, variables.py bounds) — candidates for a later join="override" migration, not refactor(common): split DataArray conversion into a 3-rung strictness ladder #737 (moving them to exact would tighten behavior).

Net effect on #737

Same diff size, same zero-behavior-change guarantee, same matmul/flag fixes — but the public surface lands as as_dataarray + align_to_coords(join=...) directly, so there's no introduce-then-deprecate of broadcast_to_coords, and the join="left" capability ships tested and in use. The PR's narrative shifts from "3-rung ladder" to "convert primitive + one reconcile function parameterized by join."

Two decisions for you

1. Drop broadcast_to_coords entirely, or keep it as a thin deprecated alias for readability at arithmetic sites (broadcast_to_coords(x, c) reads better than align_to_coords(x, c, join="defer") — and "align … join=defer" is mildly self-contradictory)?
2. join="left" migration in refactor(common): split DataArray conversion into a 3-rung strictness ladder #737, or as the immediate follow-up? Doing it in refactor(common): split DataArray conversion into a 3-rung strictness ladder #737 proves the param; deferring keeps refactor(common): split DataArray conversion into a 3-rung strictness ladder #737 purely structural.

Want me to check out refactor/dataarray-strictness-ladder and implement this?

[FBumann]

@FabianHofmann I think the important thing is to get the mental model right and to know what is done where in the codebase. This will be achieved with this PR i think. After that, refactoring to the proposed design seems much simpler.

However, Im not sure of going from 3 back to 2 methods is desireable. Its 3 very distinct methods. A join parameter in general is a good idea I think, but if we add an ambigous option like "defer", which is not unknown for a join, this creates new ambiguity i think.

My proposal would be: Lets merge this PR, then look if we need a refactoring back to 2 mehods and a join parameter after all.

[FBumann]

@FabianHofmann I thought about the join parameter. I think it's a nice addition — but not as align_to_coords(join=...).

The pattern: _like ↔ _to_coords

xarray's _like methods conform one already-built object to another object. The _to_coords family conforms raw user input (scalars, numpy, pandas, DataArrays) to explicitly passed coords — the thing add_variables / add_constraints / arithmetic must do before any xarray method can even apply, since the input isn't a DataArray yet and coords (a dict / Coordinates / list of indexes) isn't an alignable object.

Same operations, different reference type — and linopy already has most of the left column:

operation	reference = another object	reference = explicit coords
convert	—	as_dataarray(arr, coords, dims)
broadcast — make dims agree	Variable.broadcast_like(other) ✓	broadcast_to_coords(arr, coords)
reindex — make entries agree, fill gaps	reindex_like(other) on expressions / constraints ✓	reindex_to_coords(arr, coords, fill_value=…) ← NEW (your join="left")
exact check — entries must agree, else raise	align(a, b, join="exact") ✓	strict_broadcast_to_coords(arr, coords, label=…) (today's align_to_coords)
inner / outer join — both sides change	align(a, b, join=…) ✓, operator join= ✓	impossible

Three things the matrix shows:

1. The family completes what linopy already half-has. Each _to_coords function is the coords-flavored sibling of an existing method. Nothing exotic.

2. Why there's no join= parameter in the right column. A join that changes both sides (inner / outer) cannot be completed there: coords is frozen, and the expression it came from isn't in the function's hands — it can't be grown to match. The only place holding both sides is the operator (.add / .mul / .le (..., join=) → _align_constant), which is where inner / outer already live. What remains expressible against a frozen reference is exactly: check (exact) or conform (left) — two functions, not a parameter.

3. A purity note. xr.broadcast / broadcast_like quietly outer-align conflicting shared-dim entries before broadcasting (they can — they hold both objects). broadcast_to_coords deliberately doesn't: a half-completed join is worse than none, so entry conflicts pass through untouched to the operator's join=. This matches v1 §9, where broadcasting is dims-only and entry conflicts are §8's business.

The rename: align_to_coords → strict_broadcast_to_coords

The strict rung never aligns anything — it checks and raises. "Align" is also exactly the word that invited the join= idea: aligns take joins, broadcasts don't. Naming it as what it is — the same broadcast with a strict failure mode — makes the no-join design self-enforcing, and puts the required label argument on the one function that needs it.

Follow ups

The join="left" capability — follow-up PR as reindex_to_coords, migrating the two broadcast + reindex_like call sites (to_linexpr coefficient, to_constraint rhs) so it ships with real users. #737 stays purely structural.

---

Refined with Claude Code.

[FBumann]

I prototyped reindex_to_coords and I think we should NOT include it.

Reason 1:
Only 2 callers could need it, and they both can use arr.reindex_like() directly

Reason2:
arr.reindex() with multi-indexes is not properly supported by xarray.
Reported upstream with a reproducible example: pydata/xarray#11368

[FabianHofmann]

sounds good. we are moving in the right direction.

strict_broadcast and broadcast are differing in the MI checks. I would argue the latter should go to both. Second, and that is on me, the partial level coverage should be supported in future (I have to check compliance with v1 conventions again quickly).

that said, I still think the two should live in one function broadcast_to_coords with a flag strict: bool. they only differ in strictness checks and I find strict_broadcast_to_coords a bit too verbose as a function name.

[FBumann]

sounds good. we are moving in the right direction.

strict_broadcast and broadcast are differing in the MI checks. I would argue the latter should go to both. Second, and that is on me, the partial level coverage should be supported in future (I have to check compliance with v1 conventions again quickly).

that said, I still think the two should live in one function broadcast_to_coords with a flag strict: bool. they only differ in strictness checks and I find strict_broadcast_to_coords a bit too verbose as a function name.

I would leave the MI stuff up to you. I never use MI and therefore dont know whats needed. If it can be the same/extracted, thats a win i think!

And if the MI stuff is equal, both methods could be one. I'd suggest the signature:

broadcast_to_coords(arr, coords=None, dims=None, *, strict=True, label=None, **kwargs)

• bounds/mask: broadcast_to_coords(lower, coords, label="lower bound")
• arithmetic: broadcast_to_coords(other, coords=…, strict=False) — explicit opt-out
• Forgetting the flag adds safety instead of silently dropping validation.

That said, we can only unify the methods if we actually unify what strict means:

v1 says	Then
v1 governs add_variables inputs too	both modes warn on partial level → per-period bounds are deprecated usage
partial-level stays supported	both modes silent on partial level → the #732 arithmetic warning gets removed

So I'd wait for your v1 check — once strict means one thing, the merge is mechanical.

EDIT

[FabianHofmann]

great, let's go

[FBumann]

@FabianHofmann You where to fast. I edited it a bit. Did you check v1?

[FabianHofmann]

@FBumann let me reconcile my thoughts on the partial coverage. you do the signature change, use MI handling of today's strict version and I make some research again how MI is supported in upcoming xarray versions (which could change the picture). so don't mind the MI handling (perhaps don't deleted the checks as well), merge this one as soon as you feel ready and I take another look at MI in #732

[FBumann]

Note

Claude Code summary — structuring the open MI question so it's a one-table decision.

The strict flag and the v1 question are orthogonal:

• strict governs raise vs. tolerate (extra dims, value mismatches, coverage gaps) — same in both scenarios below.
• Your v1 check governs whether partial-level projection is a feature or deprecated — and that answer applies to both modes equally.

So the unified broadcast_to_coords(strict=...) is implementable either way; your check picks which table:

A — v1 allows partial-level projection (stays a feature)

MI situation	strict=False (arithmetic)	strict=True (bounds/mask)
partial level, full coverage	silent	silent
coverage gap	warn (implicit absence creation, §4)	raise

→ the #732 partial-level warning is removed; #717 needs an amendment legitimizing the projection (§9 extended to MI levels).

B — v1 forbids implicit projection (#717 as written, §8/§11)

MI situation	strict=False	strict=True
partial level	warn (deprecated)	warn (deprecated — also for bounds)
coverage gap	warn	raise

→ per-period bounds (PyPSA multi-investment) become deprecated usage with a migration path before v1; #717 stays as written.

Bottom line: pick A or B; the merge into one function is mechanical after that. A = amend the convention. B = deprecate what #732 just shipped.

---

@FabianHofmann this really helped me

[FabianHofmann]

we should learn from xarry community which struggled a lot with MI's it seems - let's pick B!

[FBumann]

Note

Claude Code reference note — the MultiIndex representation that underlies every MI issue in this stack. Useful background for the #732 MI follow-up.

Click to expand

A pandas MultiIndex has two xarray representations, and the friction between them is the root of all the MI complexity here:

Stacked: 1 dim + level coords (what linopy / PyPSA use)

<xarray.DataArray (snapshot: 4)>
Coordinates:
  * snapshot  (snapshot) MultiIndex          ← THE dim, holds tuples (2020,'t1')…
  * period    (snapshot) 2020 2020 2030 2030 ← level coord — NOT a dim
  * timestep  (snapshot) 't1' 't2' 't1' 't2' ← level coord — NOT a dim

One dimension; each level is a non-dimension (auxiliary) coordinate attached to it. Can represent sparse indexes (only the combinations that exist).

Unstacked: 2 dims (what .unstack() / Dataset.from_dataframe() produce)

<xarray.DataArray (period: 2, timestep: 2)>
array([[1., 2.],
       [3., 4.]])

Real dimensions, always the full cartesian product — sparse indexes become NaN holes.

Why every MI issue in this stack is this distinction

Issue	Cause
Partial-level broadcast (per-period bounds)	input has period as a dim; target has period as a level coord. Same name, different role → the projection translates between the two representations
v1 §11 applies (aux-coord conflict)	level coords are auxiliary coords — a level-indexed operand is exactly the §11 case
pydata/xarray#11368 (reindex fails)	reindexing the stacked dim must also rewrite the level coords; raw indexers don't know they exist
expand_dims workaround in #732	expanding a missing MI dim must create dim + all level coords, not just the dim
Coverage gaps	only exist when conforming a sparse stacked index to a full one — the unstacked form can't even express sparsity except as NaN

So for the #732 MI follow-up: the question "how is MI supported in upcoming xarray versions" is concretely "does xarray's indexer API learn to handle level coords" (#11368) — the stacked form itself is stable; it's every operation that crosses between the two forms that needs hand-holding.

[FabianHofmann]

Note

Claude Code reference note — the MultiIndex representation that underlies every MI issue in this stack. Useful background for the #732 MI follow-up.

Click to expand

yes, that is all correct but no blocker right? atm we support partial level coverage ie. allowing levels as indexes, but this will change in future (warn now, raise later)

[FBumann]

Note
Claude Code reference note — the MultiIndex representation that underlies every MI issue in this stack. Useful background for the #732 MI follow-up.
Click to expand

yes, that is all correct but no blocker right? atm we support partial level coverage ie. allowing levels as indexes, but this will change in future (warn now, raise later)

Yes, i just wanted to dump some context into this PR. Not blocking

[FBumann]

@FabianHofmann Ready