Close out #198, #197 and #199

[kgdunn]

Summary

Closes #198, the three remaining items of #197, and #199. All three land here because this session works on one branch; happy to split if you prefer.

No default behaviour changes. The scaling default is byte-for-byte what it was, the integer-identifier alignment path is verified bit-identical, and the new batch weighting defaults to exactly 1.0 per batch. The only new signals on existing call paths are a warning that was previously a silent substitution, and a ValueError where there was a bare assert.

---

#199

Downweighting badly aligned batches

Every batch contributed equally to the variable weights, so one badly aligned batch inflated the summed deviation of whichever variables it misfit and depressed their weights for every other batch. settings["batch_weighting"] is now "equal" (default) or "huber".

Huber weights each batch by its normalized_distance turned into a robust z-score against the median and MAD of the batch set: weight 1 inside the 1.345 cutoff, falling off as 1/|z| beyond it, rescaled to average 1.0.

Huber rather than a redescending function (Tukey's bisquare) because it never reaches zero. That is the feedback risk the old TODO recorded: a downweighted batch pulls the average trajectory away from itself, so it looks worse next iteration. A weight that could reach zero makes that a one-way door. Weights are recomputed from scratch each iteration and floored, so a batch that recovers is counted again.

Measured on the dryer fixture:

	equal	huber
batches at full weight	71 of 71	53 of 71
hardest downweighted	n/a	23, 48, 34
variable-weight spread	2.819	3.289
iterations	2	3

Those three batches are exactly the ones the distances output added in #570 reports as worst, so the diagnostic and the weighting agree without being wired together.

One result that looks wrong and is not: the worst normalized distance rises under Huber, 0.0714 to 0.0759. That is the feedback working as described and staying bounded, because the average moves away from the batch being downweighted. The aim is not to fit the worst batch better; it is to stop that batch setting the variable weights for the other seventy.

BatchScaler

A fit / transform wrapper over determine_scaling, apply_scaling and reverse_scaling, following the estimator contract the rest of the package uses, so batch scaling composes with Pipeline and survives clone. The three functions are unchanged and still public.

scaler = BatchScaler(columns_to_align=["Temperature"])
scaled = scaler.fit_transform(batches)
original = scaler.inverse_transform(scaled)

BatchScaler(batch_col="batch_id").fit_transform(melted_frame)   # the DataFrame input case

The "thesis page 181" item was stale

It referred to align_with_path: averaging the batch samples that compress onto one reference index. The original marker read "where more than 1 point in the target trajectory is aligned with the reference: compute the average". That is implemented (np.nanmean over those samples), pinned by tests/batch/test_dtw_align_with_path.py, and the marker was already replaced with a plain comment in #563. Nothing to do; the issue's list had not caught up.

---

#197

Batch identifiers may be of any type

A string identifier raised in two places, not the one the TODO predicted:

1. determine_scaling took its per-batch minimum over every column rather than over columns_to_align: TypeError: Cannot convert [...] to numeric. With integer identifiers it did not raise, but it still returned rows for columns never scaled, each with a real Minimum against a NaN Range.
2. align_with_path then averaged whole rows, so a compression in the warping path hit TypeError: unsupported operand type(s) for /: 'str' and 'int'. That is the predicted crash. With integer identifiers it wrote the mean of the identifier into the aligned frame, and pandas had already begun warning the string case would become a hard error.

Non-numeric columns are now carried through unaveraged, keeping their dtype. Integer path verified bit-identical; string identifiers give the same numbers per batch to 1e-12.

The resampled percentage axis accepts any resolution

maximum	delta	before	now
100	1	100 rows	100 rows
100	0.3	AssertionError	334 rows
100	7	AssertionError	15 rows

Worse than the bare assert: python -O strips asserts, and this repo runs a test-under-dash-O job, so under optimisation it extrapolated silently. The source axis now takes its endpoints from the target axis, so any delta works and there is nothing left to assert.

The parked termination assertion is restored

It had drifted twice: it indexed weight_history[4, :], a row that does not exist in the 3-row history, and positional indexing no longer works now the history is a DataFrame. New tests pin termination: tolerances 1.0 / 0.06 / 0.01 give 1, 3 and 11 iterations, every run starts from unit weights, and the looser run is a prefix of the tighter one.

---

#198

f_rupture now works

Never had a working body: it returned None, then raised NotImplementedError after #563. Now PELT from ruptures.

The default penalty is log(n), not the 100.0 in the old sketch. The rbf cost is bounded, so on a 100-sample signal a penalty of 100 is unreachable and the function finds nothing: a five-sigma step went undetected. log(n) recovers the step exactly at 60, 200 and 1000 samples under both rbf and l2, with at most one spurious point on pure noise.

The default model is rbf because its cost is scale-free. Multiplying a signal by 1000 leaves the points unchanged under rbf; under l2 at the same penalty one true change point became 37 spurious ones.

The first keyword is now tags, not columns, matching every other feature function. No working caller can be affected, but it is a signature change.

ruptures>=1.1.9 joins the optional batch extra via the _MissingExtra idiom.

q98 - q02 versus IQR

The IQR works but is not interchangeable: the ratio varies by tag (1.02 to 4.23 on dryer, 1.21 to 2.95 on nylon), so switching re-weights tags against each other rather than rescaling them together. Offered as settings["robust_range"], not substituted. It also collapses to zero more often (DifferentialPressure: 21 of 71 batches against 16).

The collapsed range was silent

rnge[rnge == 0] = 1.0 said nothing, which docs/development/error_handling.rst names as a warnings.warn case. One aggregated UserWarning per call now names each tag and its batch count.

Worth its own decision: it reports batch_id in 71 of 71 batches. With columns_to_align left None, the identifier column is swept in and scaled as though it were a tag. Pre-existing, unchanged here, but the warning is how you would notice.

---

Test plan

• uv run pytest full suite: 3343 passed, 41 skipped, coverage 94.37% (3261 before, plus 82 new tests).
• CI green on this head: all 14 checks, including the test matrix over Python 3.10-3.13 on ubuntu / windows / macOS, test-under-dash-O, lint, typecheck, CodeQL, build and codecov.
• ruff check ., ruff format --check ., mypy src/process_improve all clean locally.
• Scaling default identical; integer-identifier alignment bit-identical; "equal" batch weighting returns exactly 1.0.

Codecov first flagged two unreached lines in _batch_weights, both guards for distances that carry no information. Covered with three tests (every distance non-finite, one non-finite among usable ones, no batches at all) rather than suppressed; codecov is green again.

A note on one number, in case it shows up in a log: an intermediate full-suite run reported 40% coverage. That was my own doing, from running targeted pytest commands concurrently with the full suite, which clobbers the shared .coverage data. The clean serial runs read 94.37%.

Checklist

• Version bumped (MINOR: 1.89.0 to 1.92.0; CITATION.cff in step in the same commit). 1.90.0 and 1.91.0 were never released, so their entries are folded into one section in Keep a Changelog order.
• Tests added
• ruff check . passes
• CHANGELOG.md updated

After this merges

#198, #197 and #199 should all be closable. Their remaining list items are either delivered here or verified already done (f_robust_mad, the bare TODO markers cleared in #563, f_cross / f_elbow / the age column, and the thesis page 181 item above).

🤖 Generated with Claude Code

https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!