Batch case studies: DuPont batch PCA, SBR batch PLS, FMC multiblock batch PLS (#154, #155, #156)

[kgdunn]

Summary

• Re-implements the three legacy batch case studies from issues Case study: FMC batch process — multiblock batch PLS #154 (FMC batch dryer, multiblock batch PLS), Case study: DuPont Nylon batch process — batch PCA outlier diagnosis #155 (DuPont nylon batch, batch PCA outlier diagnosis) and Case study: SBR rubber batch process — batch PLS fault diagnosis (simulated) #156 (SBR reactor, batch PLS fault diagnosis) as plain Python scripts with narrative pages under docs/user_guide/case_studies/batch/. No notebooks: each script is a ladder of small functions that return the fitted models, and each page quotes the script section by section and states what it prints. The FMC script carries the clock time at each aligned sample as the eleventh trajectory of every batch model, as the original study did, and ends with a block-score reading of the four batches whose trajectories say off-specification while the product was on-specification.
• Adds remote loaders batch.load_dupont, batch.load_fmc and batch.load_sbr (data served from openmv.net: polymerization.csv, batch-dryer.xlsx and sbr-batch-reactor.xlsx, all three live), a shared bounded-timeout reader process_improve._remote_data with an Excel variant (the experiments loaders now use it, unchanged in behaviour), one plot helper batch.unfolded_contribution_plot, and a small additive convenience layer: BatchPCA.score_contributions / unfold_and_scale, and on BatchPLS the plot, limit and contribution forwards, predictions_, r2_per_variable_ and predictions_vs_observed_plot. time_varying_loading_plot now also accepts a BatchPLS or any multivariate model fitted on a dict_to_wide matrix.
• Brings the on-line (batch-so-far) API to BatchPLS: predict_online (scores, final-quality prediction, T2, cumulative and instantaneous SPE, residuals and the trajectory forecast of Wold et al. 2009 from a batch observed for upto_k samples, using the missing-data score estimators the library already had), predict_online_trace (the same at every sample of a complete batch) and online_rmse (the per-sample RMSE of the evolving prediction over a set of batches). BatchPCA.predict_online returns the same new keys. BatchMonitor now accepts a BatchPLS, standardises Hotelling's T2 by the reference batches' score scatter at each sample (the time-varying covariance Nomikos and MacGregor 1995 note the T2 chart needs, computed as Garcia-Munoz, Kourti and MacGregor 2004 do; the fixed end-of-batch covariance gave a normal batch a T2 of 220 after four samples), takes spe_statistic="instantaneous" for the per-interval SPE, and spe_window to pool the reference values of neighbouring samples into each per-sample SPE limit (default 0, the per-sample fit). The shared row-building and pattern-stacking code moved into a private batch/_online.py, with the BatchPCA traces bitwise unchanged.
• Contributions for rows with missing cells: spe_contributions, score_contributions and t2_contributions (and the PCA, PLS, BatchPCA and BatchPLS methods that forward to them) used to return an all-NaN row for any observation with a missing cell. The scores of such a row are now estimated from its observed cells with the same missing-data operators as project (method="scp" by default, which reproduces the NIPALS fit; "tsr" and "pmp" available), and each contribution is that row-specific operator times the cell, so the contributions are defined at every observed cell and NaN only at the missing ones. Complete rows take the exact complete-data path and are bitwise unchanged; a row with no observed feature raises. The FMC page uses it for batch 20, the batch above both limits of the batch PCA, whose record has 205 missing cells.
• Fixes PCA.r2_per_variable_ on the NIPALS path (taken whenever X has missing cells): it divided each column's residual by its sum of squares after the previous component, so column a held the share of what was left that component a explained rather than the cumulative R2 the docstring promises, and only the first column was right; the TSR path never filled the attribute and returned zeros. Found when the book's loading panels drew the R2 per cell (the collector-tank cells read 0.07 where the cumulative value is 0.78). A test checks the three PCA paths and PLS against the direct definition. The complete-data PCA path and PLS were already correct.
• Fixes the "scp" score estimator for PLS models. Running every missing-data path on data without missing values showed that single-component projection projected onto the observed part of each loading vector, where NIPALS computes a score by projecting onto the weight vector and deflating with the loading: with nothing missing the SCP estimate differed from the model's own scores by up to a score standard deviation, and an incomplete row did not reproduce the score the fit had stored for it. operator_for_pattern and project_rows take an optional x_weights, passed by PLS.project, projection_matrix, the contribution helpers and the mid-course corrector; PCA is unchanged (its weights are its loadings) and TSR, the default, already reduced to the score map exactly. PMP is the least-squares fit onto the loading plane and, for PLS, does not reduce to the model's scores even with nothing missing; the docstrings now say so. tests/test_missing_data_paths_on_complete_data.py runs the NIPALS and TSR fit paths of PCA and the three operators on complete data (the bundled dryer batches, the DuPont batches, the SBR batches) against the direct computation.
• Pins every number the legacy narratives quote in reference tests (tests/batch/test_case_studies.py), fills in the skipped TestFMCReference placeholder, vendors the legacy MATLAB batch-PCA scores and loadings on the SBR data as a fixture (tests/fixtures/sbr_batch_pca/) with a sign-agnostic cross-check, and adds tests/batch/test_online_prediction.py (exact identities of the on-line API on a thinned dryer set), tests/test_contributions_missing_cells.py (incomplete rows reproduce the stored diagnostics and the project estimates; complete rows bit-identical) and tests/test_r2_per_variable_cumulative.py, plus SBR alarm pins.

Closes #154, closes #155, closes #156.

Reference numbers reproduced (two components, tag-major unfolding, MCUV scaling):

Case	Model	Result
DuPont	batch PCA, 55 batches	R2 38.3% and 17.6% per component, 55.9% cumulative (narrative: 38.3, 17.6, 55.9); batch 49 largest in SPE; batches 50-55 in the scores; batch 49 residual in the heating, cooling and pressure tags at samples 57-63
DuPont	rebuilt models	second cluster 37, 39, 43-48 on t2-t3, read as a group against the model centre (TempC-1 and Press-3 lead; samples 0 to 25 carry 66% and 90% of the contribution); final 40-batch model leaves batches 38, 40, 41, 42 inside both limits (observability)
SBR	batch PLS, six tags	R2X 24.5% and 12.7%, R2Y 65.3% and 6.9% (narrative: 24.5, 12.7, 65.3, 6.9); batches 37 and 34 flagged in the scores, not in the SPE; batch 37 from conversion and latex density throughout, batch 34 from the heat balance from mid-batch (sustained departure of the temperatures and the energy released at samples 103-105, conversion and latex density at 123 and 129)
SBR	on-line, reference model on the 51 normal batches	batch 37 trips the per-sample T2 limit (10.5 at 99%) after 23 samples; batch 34 trips the instantaneous SPE after 105 samples, five after the impurity enters, with the reactor temperature carrying 40% of the residual at that sample; 0.2% of the reference batches' T2 values and 1.2% of their SPE values above the limits, unchanged in substance when neighbouring samples are pooled into the limits (spe_window=2: 1.3%, batch 34 still at 105); particle-size RMSEE / sd falls from 2.84 after 10 samples to 0.60 at the end (leave-one-batch-out RMSEP 3.10 to 0.78)
SBR	batch PCA, nine tags (legacy unit_tests.m)	R2 17.085% and 10.053%; scores and loadings match SBR-expected.mat sign-agnostically
FMC	the eight-model ladder, 46 batches, clock time as the eleventh trajectory	PCA on Y 0.500/0.703; PLS Zchem 0.163/0.222; PLS Zop 0.207/0.262; MBPLS Z 0.292/0.364; batch PCA 0.231/0.376; batch PLS 0.266/0.410; batch MBPLS 0.370/0.472 with super VIP Zchem 0.86, Zop 1.07, X 1.06 (regression pins; the FMC notes quote no numbers). Batch 20, above both limits of the batch PCA: 49% of its SPE in the dryer pressure, 58% in the first phase, its 205 missing cells (gaps between samples 34 and 109) blank. Batches 2, 3, 6 and 7, classed good, are placed with the abnormal batches by the trajectory block and with the good ones by both initial-condition blocks; against their nearest abnormal neighbours (42, 43, 44, 47, 50) the operating-condition contribution is carried by a longer cool-down, a shorter and steeper ramp and a shorter high-speed agitation, with the same peak set point
All three	the missing-data paths on complete data	PCA by NIPALS agrees with SVD to 1e-6 and by TSR to 1e-13 (scores, loadings, R2, R2 per variable, SPE, T2; also the DuPont batch PCA); with every cell observed the TSR, SCP and PMP operators return the PCA scores to 1e-13, and TSR and SCP the PLS scores (the SBR batch PLS included); project on complete rows equals the fit for every method; SCP reproduces the NIPALS scores of incomplete rows for PLS as it did for PCA

Consistency with the Nomikos and MacGregor papers. The SBR pages now state the simulated faults as the 1994 AIChE paper describes them (30% more impurity from the start of one batch, 50% more from halfway through the other) and that the feed tags carry the simulation's noise rather than being constant; the per-sample T2 covariance is credited to the 1995 Technometrics paper as a need it states, and to the 2004 paper for computing it; the DuPont page says the original reference set also left out the four poor-quality batches (36 batches, three components).

SBR hosting. The workbook sbr-batch-reactor.xlsx (sheets X_batch, Y_quality, the layout of batch-dryer.xlsx) is produced by tests/fixtures/sbr_batch_pca/prepare_fixture.py and is live at https://openmv.net/file/sbr-batch-reactor.xlsx, byte-identical to the fixture's output, with its description at https://openmv.net/info/sbr-batch-reactor. load_sbr() reads it in about two seconds, and the nine SBR reference tests, which skipped without a PROCESS_IMPROVE_SBR_URL override while the upload was pending, now run against it (the override stays for offline work).

Companion pull requests. The book chapter is kgdunn/pid-book#273 (three pages in the Applications chapter, each figure preceded by the equivalent Plotly code, with the two on-line sections on the SBR page and the block-score section on the FMC page), and its matplotlib figures are kgdunn/figures#84 (merge the figures PR before the book PR). The figures repository's script checks fail on the three batch scripts until this PR is merged (they import load_dupont, load_fmc, load_sbr and BatchMonitor from it) and, for the PyPI job, until a release carries it. The FMC loading-panel figure and the book's R2 echoes depend on the r2_per_variable_ fix.

Versioning note. Main took 1.79.0 (#539), 1.79.1 (#547) and 1.80.0 (#540) while this branch was open, so this branch's changelog section is 1.81.0, above main's 1.80.0. PR #541 also carries a 1.80.0 bump; if it merges before this one it needs a re-bump too.

Follow-ups, not in this PR. Converting the five existing case-study notebooks to the same script-plus-page form; MCUVScaler and PLS(scale=True) dropping multi-level column labels (the scripts re-attach wide.columns after scaling).

Test plan

• uv run ruff check . and uv run ruff format --check . pass
• uv run mypy src/process_improve passes (163 files)
• Full suite after the SCP fix, with the SBR workbook served locally: 3208 passed, 4 skipped; tests/batch tests/test_multiblock_reference.py after each merge of main and after the clock-time change (299 passed), plus the coverage gate
• Reference tests run online: DuPont and FMC against openmv.net; SBR against the local workbook while the upload was pending, then the nine SBR tests against the live openmv.net workbook with no override (9 passed)
• Each case-study script executed end to end and its printed numbers compared with the pages and the pin table; the BatchPCA on-line traces compared bitwise with the pre-refactor baseline; complete-row contributions compared bitwise with the previous implementation
• uv run sphinx-build docs docs/_build/html -b html -W succeeds after this round (pandoc installed for the existing notebooks)

Checklist

• Version bumped in pyproject.toml (PATCH for fixes/docs/config, MINOR for new features): 1.80.0 to 1.81.0, CITATION.cff synced
• Tests added or updated where relevant
• ruff check . passes
• CHANGELOG.md updated

🤖 Generated with Claude Code

https://claude.ai/code/session_01LXthGpHLQFfGubBiKFtGAE

[codecov]

Codecov Report

❌ Patch coverage is 92.37288% with 36 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/process_improve/batch/_online.py	78.64%	8 Missing and 14 partials ⚠️
src/process_improve/batch/_batch_pls.py	89.89%	6 Missing and 4 partials ⚠️
src/process_improve/batch/_batch_monitor.py	95.74%	1 Missing and 1 partial ⚠️
src/process_improve/multivariate/_projection.py	80.00%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!