Align only the columns that carry a trajectory, and compute the DTW cost per row

[kgdunn]

Summary

The two follow-ups left open after #572: skip non-numeric columns when aligning, and fix the DTW cost matrix. #197, #198 and #199 are now closed.

---

Aligning only the columns that carry a trajectory

columns_to_align=None resolved to every column of the first batch, sweeping in whatever else the frame carried. Two kinds do not belong.

Not numeric. A label is not a trajectory. Skipped on the default path; named explicitly it now raises, giving the column and its dtype, since you asked for it by name and whether to carry, encode or discard it is yours to decide.

Flat through every batch. This is the case that prompted the change, and a dtype test alone does not catch it: the identifier melted_to_dict leaves in place is usually an integer, so dryer's batch_id sailed through select_dtypes(include="number") and was scaled as though it were a tag.

I checked whether that actually mattered before changing it. It is not the "identifier steals all the weight" story I half expected: batch_id takes almost no weight (0.000051), because its forced range of 1.0 leaves the raw identifiers deviating wildly from the average. But it joins the distance the alignment minimises and the normalisation that follows, and the real weights move a long way:

tag	tags only	tags + batch_id
AgitatorPower	0.051	0.081
AgitatorTorque	4.280	4.525
JacketTemperatureSP	0.132	0.472
JacketTemperature	0.257	0.494
DryerTemp	0.281	0.429

Every batch is checked for spread, not only the first, so a tag that happens to be flat in batch 1 but moves later is kept. The check is a min against a max rather than a distinct count: one pass, no hashing, and the question is only whether the column moves at all. A constant column named explicitly is left alone, since constant over one set of batches does not mean constant in general.

Side effect worth noting: the zero-range warning added in #572 now reports only the genuine case (DifferentialPressure, 16 of 71 batches) instead of leading with batch_id on every default call.

---

The DTW cost matrix

np.diag(A @ W @ A.T) builds an h-by-h product to keep its h diagonal entries: quadratic in the reference length per test sample, cubic over the whole matrix, plus a temporary to match. Summing (A @ W) * A along the tags computes only the diagonal, in h * J² work rather than h² * J.

samples	before	after	with a 10% band
100	2.08 ms	1.49 ms	0.33 ms
300	24.98 ms	11.69 ms	2.62 ms
700	272.65 ms	82.14 ms	18.49 ms
1200	885.63 ms	240.06 ms	55.41 ms

It also removes the numba contiguity problem the matmul had, where a slice taken with runtime bounds compiled to a slower kernel than the whole array. That was the only reason #571 split the banded and unconstrained cases into separate loops, so one loop now serves both.

I was wrong about this needing a pinned-value sweep

I told you this would shift pinned values by ~1e-15 and therefore needed its own careful PR. Measured, it does not. Cost values differ by about 5e-16 relative (one ulp), because the summation order changes, but:

• across nine combinations of length and tag count the warping path is identical;
• the dryer alignment reproduces its final weights exactly (max relative shift 0.000e+00);
• and the full suite confirms it: not one pinned value anywhere moved.

The path is chosen by comparisons between accumulated costs that sit far apart relative to one ulp, so the change does not reach anything downstream. Tests pin this rather than trusting it: the kernel is checked against the literal Mahalanobis expression, and three length/tag combinations are checked to produce the same warping path as a reference dynamic programme built from that expression. A fourth covers a tall reference against few tags (4000 × 2), which under the old form allocated a 4000-by-4000 product per test sample.

A correction to numbers I published in #571

The "6.55 s at 700 samples" in that PR was inflated by background test suites competing for the machine; uncontended it is 272 ms. The ratios there were measured back-to-back so they stand, but the absolute times were too high.

---

Test plan

• uv run pytest full suite: 3352 passed, 41 skipped, coverage 94.37%.
• CI green on this head: all 14 checks, including the test matrix over Python 3.10-3.13 on ubuntu / windows / macOS, test-under-dash-O, lint, typecheck, CodeQL, build and codecov.
• ruff check ., ruff format --check ., mypy src/process_improve all clean.
• Cost values agree with the literal Mahalanobis expression to 1e-12; warping paths identical over nine configurations; dryer weights bit-identical.

The first CI run caught one genuine failure, and it was the right one: a test from #572 asserted batch_id heads the zero-range warning, which is exactly the behaviour this PR removes. Split into two tests rather than deleted, so the original point is still pinned on DifferentialPressure and the new behaviour is pinned directly. Codecov then flagged five unreached lines in the new resolution; covered with eight tests, one per rule, rather than suppressed.

Checklist

• Version bumped (MINOR: 1.92.0 to 1.93.0; CITATION.cff in step in the same commit). MINOR rather than PATCH because columns_to_align=None now resolves to fewer columns, which changes the alignment for anyone relying on the old sweep, and an explicitly named non-numeric column now raises.
• Tests added
• ruff check . passes
• CHANGELOG.md updated

🤖 Generated with Claude Code

https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!