Single-apply feature kernels, numpy-exact parity and tsfresh fuzzing

[wilswer]

Summary

Makes extraction substantially faster and brings every feature to bit-for-bit parity with tsfresh's extract_features, checked by a new differential fuzzer that runs in CI.

• Speed: every feature except length is now computed in a single apply per value column (a Kernel per feature over a shared per-group context), instead of one apply per feature (~60 for Efficient). Earlier commits remove the plan serialisation for in-memory data and several per-feature copies.
• Parity: mean, variance, sums, median and quantile are computed exactly as numpy does them, and a Hypothesis fuzzer compares all 49 features against tsfresh. All pass.

⚠️ Behaviour changes

• Output column order: length first, then all features of each value column in turn, in tier order. Column names are unchanged. If you select columns by position (e.g. df.to_numpy() into a trained model), select by name or retrain.
• agg_linear_trend follows tsfresh's extract_features, which passes the calculator a numpy array. The var aggregator is now the population variance (ddof = 0, was ddof = 1), and a NaN in a chunk makes the result NaN instead of being skipped. tsfx had followed tsfresh's unit tests, which pass a pandas Series. The median aggregator (a tsfx addition) follows np.median.
• mean_n_absolute_max is NaN unless there are more than n values, and NaN if the series contains NaN (as tsfresh).
• median / quantile of a group without non-null values is NaN (was null), like every other feature.
• Last-bit changes wherever tsfx summed or interpolated differently from numpy: mean, variance, standard_deviation and everything built on them, sum_values, absolute_sum_of_changes, mean_absolute_change, root_mean_square, autocorrelation, c3, time_reversal_asymmetry_statistic, index_mass_quantile, symmetry_looking, sum_of_reoccurring_values, sum_of_reoccurring_data_points, median, quantile, linear_trend, agg_linear_trend. Threshold features can change where a value sits exactly on the threshold: ratio_beyond_r_sigma([1, 3, 3, -2, 7, 7, 0, 1], r=1.5) is now 0, as tsfresh (was 0.375).

Every change has a regression test that failed before the fix. See CHANGELOG.md.

Performance

Criterion, 2× Xeon Gold 6230 (80 threads), against the state after the in-place-read refactor (baseline views), measured at the final commit e6b5c72:

Bench	Before	After	Change
stocks Efficient (619k rows, 505 groups, 5 columns)	113.5 ms	44.6 ms	−61%
1BRC Efficient (100M rows, ~10k groups)	8.80 s	4.63 s	−47%
1BRC Minimal	4.99 s	4.31 s	−14%
stocks Minimal	19.2 ms	22.5 ms	+16%

The parity fixes are speed-neutral (within noise of the measurement before them). The stocks Minimal slowdown is accepted. Polars' lazy group-by alone costs ~12.6 ms there (computing the groups ~4.4 ms), and on short, already-sorted groups merging applys saves nothing to offset the per-apply overhead.

Earlier commits on this branch, measured on an M1 Max (hyperfine, Python end to end): in-memory Minimal 681 → 183 ms and Efficient 953 → 323 ms, by handing in-memory data to Rust without serialising the query plan; plus faster reoccurrence, number_peaks and agg_linear_trend.

Testing

• pytest: 384 passed.
• New: python_tests/test_fuzz_tsfresh.py, differential fuzzing of every feature against tsfresh with Hypothesis. Series include ones built to hit thresholds exactly (a block followed by permuted copies), where only the last bit decides a comparison. numpy-based features must match bit for bit; pandas/scipy/BLAS-based families use a 1e-9 tolerance; documented deviations are excluded. Opt-in (pytest -m fuzz, 200 examples per feature; HYPOTHESIS_PROFILE=thorough runs 5000 for targeted hunts); CI runs a 10-example pass with pinned tsfresh 0.21.2 / numpy 2.4.4 / pandas 3.0.2 / scipy 1.17.1 (49 passed).
• Each refactor commit was checked with an output snapshot (3 configs × 3 tiers + a dynamic group-by, Int64 column, edge-case groups) compared by column name. Values were bit-identical except where a commit says otherwise.
• cargo test, cargo clippy --all-targets -- -D warnings, rustdoc with -D warnings, cargo fmt --check.

Also in this PR

• benches/: a stocks criterion bench, a 1BRC bench on the first 100M rows (skips itself without the data), and benches/tools/ (output snapshot and comparison, samply hot-spot and before/after summaries).
• hypothesis joins the dev dependencies; ordered-float, ndarray-stats, noisy_float and num are no longer used.
• python_tests/TSFRESH_LICENSE: tsfresh's MIT licence for the test vectors copied from its test suite.

No version bump in this PR (still 0.3.3); the behaviour changes warrant a minor bump at release.

🤖 Generated with Claude Code