Report per-batch DTW distances and offer a deviation-metric setting (#199); document error_score for TPLS scoring (#565) by kgdunn · Pull Request #570 · kgdunn/process-improve

kgdunn · 2026-09-12T12:09:33Z

Summary

The diagnostics-first step on #199, plus the #565 doc follow-up. No existing behaviour changes: the default weighting is bit-identical to main, verified to fifteen digits.

batch_dtw now returns distances — a DataFrame indexed by batch identifier with Distance and Normalized distance. These were computed on every iteration and thrown away, so there was no way to see which batches aligned badly; the commented-out dist_df.hist(...) sketch was reaching for exactly this. Nothing extra is computed: each DTWresult already carries .distance and .normalized_distance, and batch_dtw already holds the final iteration's results. That also made the list one_iteration_dtw was building redundant, so it is gone.

outputs["distances"]["Normalized distance"].nlargest(5)
# batch_id
# 23    0.071427
# 48    0.059242
# 34    0.036997

settings["weighting"] selects "quadratic" (default) or "absolute". An unrecognised value raises ValueError rather than falling silently into the absolute branch.

Why the default stays quadratic, despite your preference for absolute

You asked for absolute; I've made it available but not the default, on a point I only found by reading the distance code. The weighted DTW distance is a Mahalanobis form, and the code says so (alignment_helpers.py:40):

# Mahalanobis distance:
dist[:, idx] = np.diag((row - ref) @ weight_matrix @ ((row - ref).T))

It is quadratic in the deviations, so the reciprocal of a sum of squares is an inverse-variance (precision) weight, which is exactly the weighting such a distance expects, and is the published Kassidas choice. 1/|deviation| is not a precision, so the distance loses that reading.

I was wrong about what absolute does, and the docs now say what it measures

I predicted absolute would compress the weights toward uniform. Measured on the bundled dryer data, it does the opposite:

	quadratic	absolute
weight spread (max/min)	2.819	5.996
iterations to converge	2	3

So both the fixed point and the path to it differ, and not in the direction I expected. The docstring now carries these measured numbers rather than my prediction, and tells the reader to measure on their own data.

Two other things worth knowing

Length normalisation would be a no-op. All synced frames are built on the reference grid (nr = md_path[:, 0].max() + 1), so every batch contributes the same row count to the weight accumulation. There is no length bias to correct. Normalized distance is still worth reporting because the distances do depend on path length, and on this fixture the two orderings genuinely differ, which a test pins.

The open half of #199 is left open, with a note in the code. Continuous downweighting of badly aligned batches is recorded as the remaining question, together with the feedback risk: a downweighted batch pulls the average away from itself and is then downweighted further, so any such weight must be recomputed per iteration and floored. The distances output is the prerequisite for choosing that weight function from data instead of guessing.

#565

TPLS.score now records that sklearn's error_score decides whether the named-scorer failure is visible. The default np.nan is what records the folds as NaN; error_score="raise" propagates the underlying TypeError. Since the failure is inside sklearn before TPLS is reached (0 calls, measured), this is the only route to a loud failure, and it is the closest available form of the "make it raise" option you preferred.

Test plan

uv run pytest full suite: 3257 passed, 41 skipped (3250 before, plus 7 new; skips are proxy-blocked dataset downloads, pre-existing).
uv run ruff check ., uv run ruff format --check ., uv run mypy src/process_improve all pass.
Default verified bit-identical to main: final weights and last_average_batch match to fifteen digits with the change stashed and restored.
The pinned reference values were captured from a run, not transcribed. I initially typed two of the five from a truncated print and the test caught it at 1e-7; they are now read from output, at rel=1e-6 (far tighter than the ~50% absolute moves them, loose enough for platform float noise across the CI matrix).

New coverage: TestBatchDtwDistancesOutput (4 tests: every batch reported, the reference batch at zero distance, agreement with the DTWresult objects, and that normalising changes the ranking) and TestBatchDtwWeightingSetting (3 tests: the default is quadratic and unchanged, absolute gives a different fixed point with a wider spread, and an invalid value is rejected).

Checklist

Version bumped in pyproject.toml (MINOR: 1.87.1 to 1.88.0, since a public setting and a new output key are API additions; CITATION.cff in step in the same commit)
Tests added or updated where relevant
ruff check . passes
CHANGELOG.md updated

Note for review

The deviation accumulation moved into _accumulate_deviations. That was needed to keep batch_dtw inside its branch budget without a new noqa (#307 already tracks 69), and it gives the setting a named, testable home.

🤖 Generated with Claude Code

https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

Generated by Claude Code

…re (#565) TPLS cannot raise on cross_val_score(..., scoring="r2"): the failure is inside sklearn's _Scorer.__call__ and TPLS.score is never called, measured at 0 calls against 3 of 3 folds under the default scoring. So the earlier note said only that the folds come back NaN. sklearn's own error_score decides whether that is visible. The default, np.nan, is what records the NaN; error_score="raise" propagates the underlying TypeError. Verified: cross_val_score(TPLS(...), X=blocks, cv=3, scoring="r2", error_score="raise") TypeError: _Scorer._score() missing 1 required positional argument: 'y_true' That is the closest thing to a loud failure available here, so the docstring now gives it, and says to use it whenever a silent NaN would be worse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

…199) Two changes to batch_dtw. They share preprocessing.py, so they share a commit. ---------------------------------------------------------------------- The per-batch DTW distances were collected on every iteration and thrown away, so there was no way to see which batches aligned badly; the commented-out sketch that wanted to histogram them was reaching for exactly this. batch_dtw now returns them under a `distances` key: a DataFrame indexed by batch identifier with `Distance` and `Normalized distance`. The normalized column divides by the summed path length, so it is the one that compares across batches of unequal duration, and on the dryer fixture the two orderings genuinely differ. Nothing extra is computed. Each DTWresult already carries .distance and .normalized_distance, and batch_dtw already holds the final iteration's results, so the frame is read off what was in hand. That also makes the list one_iteration_dtw was building redundant, and it is removed. No behaviour change: the alignment, the weights and the average trajectory are untouched. ---------------------------------------------------------------------- How a variable's deviation from the average trajectory is accumulated before the weight is taken as its reciprocal is now selectable: "quadratic" (the default) or "absolute". The default stays quadratic, and is bit-identical to previous releases: the final weights and the average trajectory match to fifteen digits on the dryer fixture. That is deliberate rather than conservative. The reciprocal of a sum of squares is an inverse-variance (precision) weight, and the weighted DTW distance in alignment_helpers.distance_matrix is a Mahalanobis form, quadratic in the deviations, which is the weighting such a distance expects. It is also the published Kassidas choice the docstring cites. "absolute" is less sensitive to a single badly aligned batch, but its reciprocal is not a precision, so the distance loses that reading. It does not simply flatten the weighting, which is what I expected before measuring: on the dryer data the ratio of largest to smallest weight rises from 2.8 to 6.0, and the iteration count from 2 to 3, so the fixed point and the path to it both differ. The docstring carries the measured numbers rather than the prediction, and tells the reader to measure on their own data. An unrecognised value raises ValueError rather than falling silently into the absolute branch. The accumulation moved into _accumulate_deviations, which gives the setting a named, testable home and keeps batch_dtw inside its branch budget without a new noqa (#307). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

A new public setting and a new key in the returned dict are API additions, so MINOR. No existing behaviour changes: the default weighting is bit-identical. CITATION.cff is kept in step in the same commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

codecov · 2026-09-12T12:25:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…imator (#199) **`settings["batch_weighting"]`.** Every batch contributed equally to the variable weights, so one badly aligned batch inflated the summed deviation of whichever variables it misfit and depressed their weights for every other batch. `"huber"` weights each batch by Huber's function on the robust z-score of its `normalized_distance`, against the median and MAD of the batch set: weight 1 inside the 1.345 cutoff, falling off as 1/|z| beyond it, rescaled to average 1.0. Huber rather than a redescending function (Tukey's bisquare) because it never reaches zero. That is the feedback risk the previous TODO recorded: a downweighted batch pulls the average trajectory away from itself, so it looks worse next iteration, and a weight that could reach zero would make that a one-way door. The weights are recomputed from scratch each iteration and floored, so a batch that recovers is counted again. The default stays `"equal"`, which returns exactly 1.0 per batch, so multiplying by it is a no-op and existing results cannot drift. Measured on the dryer fixture, Huber leaves 53 of 71 batches at full weight and downweights batches 23, 48 and 34 hardest. Those are precisely the three the `distances` output added in #570 reports as worst, so the diagnostic and the weighting agree without being wired together. The worst normalized distance rises slightly, from 0.0714 to 0.0759, which is the feedback working as described and staying bounded: the average moves away from the batch being downweighted. The point is not to fit the worst batch better, it is to stop that batch from setting the variable weights for the other seventy. **`BatchScaler`.** A fit / transform wrapper over `determine_scaling`, `apply_scaling` and `reverse_scaling`, following the estimator contract the rest of the package uses, so batch scaling composes with Pipeline and survives `clone`. The three functions are unchanged and still public. It also takes the melted-DataFrame input the functions reject: pass `batch_col` and it splits the frame itself, rather than telling the caller to do it. Also closes the "thesis page 181" item: it referred to averaging the batch samples that compress onto one reference index in `align_with_path`. That is implemented and pinned by tests/batch/test_dtw_align_with_path.py, and the marker was already replaced with a plain comment in #563; the issue's list was stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og

* Answer the q98-q02 versus IQR question in determine_scaling, and stop hiding collapsed ranges (#198) The TODO asked whether `f_iqr` would work as the robust range instead of q98 - q02. Measured on the bundled data, it works but is not interchangeable, so it is offered as `settings["robust_range"]` rather than substituted. The IQR spans the middle half of a batch; q98 - q02 spans nearly all of it. On a Gaussian tag the second is about 3.05 times the first, but a batch trajectory is not Gaussian, so the ratio varies by tag: per tag it runs 1.02 to 4.23 on the dryer data and 1.21 to 2.95 on nylon. Switching therefore re-weights the tags against each other rather than rescaling them together, which would silently change every existing alignment. The default is unchanged. The IQR also collapses to zero more often, because a tag holding one value for more than half a batch has no interquartile spread: `DifferentialPressure` collapses in 21 of the 71 dryer batches under the IQR against 16 under q98 - q02. That collapse was silent. `rnge[rnge == 0] = 1.0` leaves the tag unscaled and said nothing, which `docs/development/error_handling.rst` names as a `warnings.warn` case ("a constant column dropped"). One aggregated UserWarning per call now names the tags and how many batches each affected, rather than one warning per batch. On the dryer data with the default columns it reports `batch_id` in 71 of 71 batches, which is worth knowing on its own: the identifier column is swept in as a tag whenever `columns_to_align` is left None. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Implement f_rupture with change-point detection (#198) `f_rupture` has never had a working body: until #563 it returned None for a valid call, and since then it has raised NotImplementedError. It now detects change points with `ruptures`, added to the `batch` extra and imported at module level through the `_MissingExtra` idiom already used for plotly, so the module still imports without it. Detection uses PELT, which is exact and linear in the signal length and does not need to be told how many change points to look for. Three decisions worth recording: - **The parameter is `tags`, not `columns`.** Every other feature function in this module takes `tags`; `f_rupture` was the odd one out, and since it never returned a value there is no working caller to break. - **The default penalty is log(n), not the 100.0 from the old sketch.** The rbf cost is bounded, so on a 100-sample signal a penalty of 100 is unreachable and the function finds nothing: a five-sigma step went undetected. log(n) is the BIC-style choice and adapts to batch length. Measured on a single five-sigma step it recovers the change point exactly at 60, 200 and 1000 samples under both rbf and l2, and on pure noise of those lengths reports at most one spurious point. - **The default model is rbf because its cost is scale-free.** Multiplying a signal by 1000 leaves the detected points unchanged under rbf; under l2 at the same penalty one true change point became 37 spurious ones. The docstring says so, rather than leaving the reader to discover it. The return shape is one `<tag>_rupture` column per tag holding a tuple of positions, which is the only faithful shape for a result whose length varies per batch; the docstring shows `.map(len)` for a numeric feature. The trailing sentinel `ruptures` appends (the signal length, not a change point) is stripped, and a signal too short to split or carrying missing data returns an empty tuple rather than raising. The sketch's matplotlib call is not carried over: the function returns the breakpoints and leaves plotting to the caller. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Bump to 1.90.0 for the change-point feature and the scaling range setting MINOR: `f_rupture` becomes functional, `determine_scaling` gains a `robust_range` setting, and `ruptures` joins the optional `batch` extra. `determine_scaling` now warns where it used to substitute silently; nothing is removed and no default changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Let batch identifiers be any type, and the resampled axis any resolution (#197) Three items from #197, all in the alignment path. **Batch identifiers of string type crashed, in two places.** `melted_to_dict` never coerced its keys, but two functions downstream assumed every column was numeric: - `determine_scaling` took its per-batch minimum over every column rather than over `columns_to_align`, so a string identifier raised `TypeError: Cannot convert [...] to numeric`. With integer identifiers it did not raise, but it still emitted rows for columns that were never scaled, leaving a real `Minimum` against a NaN `Range` for each of them. - `align_with_path` then averaged whole rows, so a compression in the warping path reached `TypeError: unsupported operand type(s) for /: 'str' and 'int'`, which is the crash the TODO predicted. With integer identifiers it wrote the mean of the identifier into the aligned frame, and pandas had already started warning that assigning a string into the float frame would become an error. Non-numeric columns are now carried through unaveraged: they are constant within a batch, so the first value is taken and the column keeps its own dtype. The integer path is verified bit-identical (weights, average and aligned values), and string identifiers now produce exactly the same numbers as integer ones. **The resampled axis only accepted a delta that divided the maximum.** The target axis came from `np.arange(0, maximum, delta)` while the source axis was rebuilt as `maximum - delta`, which agree only when the delta divides evenly. A delta of 0.3 or 7 put the endpoints in different places, and two asserts turned that into a bare AssertionError. Worse, `python -O` strips asserts, so under optimisation it extrapolated silently instead. The source axis now takes its endpoints from the target axis, so any delta works and there is nothing left to assert. An axis that cannot be built at all (a non-positive value, or a delta no smaller than the maximum) is rejected up front with a message. **The commented-out termination assertion is restored.** It had been parked while DTW termination was in flux, and had drifted twice over: it indexed `weight_history[4, :]`, a row that does not exist in the 3-row history that tolerance produces, and positional indexing no longer works now that the history is a DataFrame. The values are captured from a run. New tests pin what termination now does: tolerances of 1.0, 0.06 and 0.01 give 1, 3 and 11 iterations, every run starts from unit weights, and the looser run is a prefix of the tighter one, so the tolerance decides only when to stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Bump to 1.91.0, folding the 1.90.0 entry into one unreleased section MINOR: a delta that does not divide the interpolation maximum now works rather than failing an assert, and non-numeric columns survive alignment instead of crashing it. Both change what the function does on inputs it previously rejected or corrupted, so this is not a pure patch. 1.90.0 was never released, so its entry is merged in rather than left as a separate heading, and the duplicate `### Fixed` blocks that the merge produced are combined into one in Keep a Changelog order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Downweight badly aligned batches, and wrap the scaling trio in an estimator (#199) **`settings["batch_weighting"]`.** Every batch contributed equally to the variable weights, so one badly aligned batch inflated the summed deviation of whichever variables it misfit and depressed their weights for every other batch. `"huber"` weights each batch by Huber's function on the robust z-score of its `normalized_distance`, against the median and MAD of the batch set: weight 1 inside the 1.345 cutoff, falling off as 1/|z| beyond it, rescaled to average 1.0. Huber rather than a redescending function (Tukey's bisquare) because it never reaches zero. That is the feedback risk the previous TODO recorded: a downweighted batch pulls the average trajectory away from itself, so it looks worse next iteration, and a weight that could reach zero would make that a one-way door. The weights are recomputed from scratch each iteration and floored, so a batch that recovers is counted again. The default stays `"equal"`, which returns exactly 1.0 per batch, so multiplying by it is a no-op and existing results cannot drift. Measured on the dryer fixture, Huber leaves 53 of 71 batches at full weight and downweights batches 23, 48 and 34 hardest. Those are precisely the three the `distances` output added in #570 reports as worst, so the diagnostic and the weighting agree without being wired together. The worst normalized distance rises slightly, from 0.0714 to 0.0759, which is the feedback working as described and staying bounded: the average moves away from the batch being downweighted. The point is not to fit the worst batch better, it is to stop that batch from setting the variable weights for the other seventy. **`BatchScaler`.** A fit / transform wrapper over `determine_scaling`, `apply_scaling` and `reverse_scaling`, following the estimator contract the rest of the package uses, so batch scaling composes with Pipeline and survives `clone`. The three functions are unchanged and still public. It also takes the melted-DataFrame input the functions reject: pass `batch_col` and it splits the frame itself, rather than telling the caller to do it. Also closes the "thesis page 181" item: it referred to averaging the batch samples that compress onto one reference index in `align_with_path`. That is implemented and pinned by tests/batch/test_dtw_align_with_path.py, and the marker was already replaced with a plain comment in #563; the issue's list was stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Bump to 1.92.0 for the batch weighting setting and BatchScaler MINOR: a new public estimator and a new setting. 1.91.0 was never released, so its entry is folded in rather than left as a separate heading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og * Cover the degenerate branches of _batch_weights (#199) Codecov flagged two lines of the batch-weighting helper as unreached. Both are the guards that decide what happens when the distances carry no information, so they are worth a test rather than a pragma: - every distance non-finite, where there is no basis to rank batches and none is penalised; - one distance non-finite among usable ones, where that batch is floored rather than dropped, so it still contributes; - no batches at all, where the `zip(..., strict=True)` in the caller needs the lengths to agree even at zero. The stand-in result object is factored into a helper, since three tests now need one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123j56Hk6jRz9zy91Doc1og --------- Co-authored-by: Claude <noreply@anthropic.com>

claude added 3 commits September 12, 2026 12:08

kgdunn merged commit 62b2907 into main Sep 12, 2026
14 checks passed

kgdunn deleted the claude/audit-todo-fixme-items-gy9vj6 branch September 12, 2026 14:06

kgdunn mentioned this pull request Sep 12, 2026

Close out #198, #197 and #199 #572

Merged

8 tasks

This was referenced Sep 12, 2026

Batch preprocessing: weighting options and API refactor #199

Closed

cross_val_score(TPLS, scoring="r2") returns all NaN behind a UserWarning #565

Open

Navigation Menu

Report per-batch DTW distances and offer a deviation-metric setting (#199); document error_score for TPLS scoring (#565) - #570

Report per-batch DTW distances and offer a deviation-metric setting (#199); document error_score for TPLS scoring (#565)#570
kgdunn merged 3 commits into
mainfrom
claude/audit-todo-fixme-items-gy9vj6

kgdunn commented Sep 12, 2026

Uh oh!

codecov Bot commented Sep 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kgdunn commented Sep 12, 2026

Summary

Why the default stays quadratic, despite your preference for absolute

I was wrong about what absolute does, and the docs now say what it measures

Two other things worth knowing

#565

Test plan

Checklist

Note for review

Uh oh!

codecov Bot commented Sep 12, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants