fix(conformal): σ-model divergence silently masked as constant-width band — FastForest swap (un-reds macOS CI)

CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Fixed
+- **Normalized-conformal σ-model can no longer silently diverge into a constant-width band (also un-reds macOS CI)**: the heteroscedastic aux σ(x) model was a linear SDCA fit, and SDCA's async optimization trajectory is platform-/thread-sensitive on this small aux problem — it can outright *diverge* (observed: σ(x) ≈ −8.7×10⁷ on win-x64 single-threaded, and the same degeneracy on macos-arm64 CI's default fit, while win/linux multi-threaded happened to converge). Because the band formula floors σ at `max(σ,0)+β`, a diverged σ-model doesn't crash — every row silently gets the identical constant width while the experiment metadata still claims a heteroscedastic band, which is exactly why macos-latest CI had been red since 0.19.0 on the two per-row-band tests without anyone seeing a *wrong prediction*. The σ-model is now a FastForest (managed, no native deps): a forest predicts leaf averages of the |residual| target, so σ is structurally bounded to the training residual range — it cannot go negative or diverge — and it trains with `NumberOfThreads = 1` so conformal calibration is reproducible for the same data (as a bonus the aux fit dropped from ~90 s of grinding divergence to ~1 s in the affected tests). Alongside: `PredictionService`'s σ-scoring fallback now surfaces a warning naming the constant-width fallback instead of degrading silently (P-svc1's lesson), both band tests read the aux σ directly in their failure messages so a future regression reports whether the *fit* or the *serving path* degenerated, and `Predict_Multiclass_NumericLabel_MatchesTrainedSingleSchema` swaps its incidental LightGBM trainer for SdcaMaximumEntropy — GitHub's macos-latest runners are Apple silicon and lib_lightgbm ships no osx-arm64 native, which was the third red test (`DllNotFoundException`), and the pinned regression (serve-path label loading) is trainer-agnostic.
+
+- **API integration-test factory no longer mutates the process-global working directory**: `TestWebApplicationFactory` pinned the API's project discovery to its temp root by assigning `Environment.CurrentDirectory` — process-global state. That was latent while a single test class consumed the factory, but xUnit runs test classes in parallel, and D21-A's `ForecastingApiTests` added a second factory instance: one factory's dispose could delete the directory another factory's `Program` startup was using as its working directory, killing the entry point before it built an `IHost` ("The entry point exited without ever building an IHost", 37 API tests failing on every OS). The factory now injects a fixed-root `IProjectDiscovery` stub instead of touching the CWD, so parallel factories stay isolated.
+
+### Changed
+- **Object-detection training now forces libtorch to a single thread (D27 defensive mitigation, unverified)**: real-data OD training intermittently dies with a native access violation (0xC0000005) inside libtorch, with the crash location varying between runs (`THSNN_Linear_forward`, `Tensor.backward`) — a signature upstream (dotnet/TorchSharp#1292) traces to native heap corruption whose failure point shifts with memory/thread pressure. `RunObjectDetectionAsync` now calls `TorchSharp.torch.set_num_threads(1)` before fitting, removing that pressure source. This is a defensive, no-downside change (the only cost is slower CPU training) applied unconditionally rather than gated on a confirmed fix — this investigation's development environment had ambient resource contention from concurrent unrelated workloads that prevented reliably re-triggering or clearing the crash to verify the mitigation directly (see the D27 issue for the full investigation). Full elastic thread-count tuning or an upstream TorchSharp fix remain open; this is a stopgap.
+
+### Added
+- **`POST /predict` now serves forecasting models with a horizon-based contract instead of rejecting every request (D21-A)**: forecasting was the only task where `POST /predict` always failed — SSA forecasting is stateful (it forecasts a fixed horizon ahead of its training series), so the row-based contract every other task uses cannot apply, and D21 made that rejection actionable rather than a silent all-null 200. `POST /predict` for a forecasting model now accepts an optional JSON body `{"horizon": N}` (an empty object `{}` uses the model's trained horizon — a body is still required, matching every other task's contract, but the `horizon` field itself is optional) and returns the forecast on the exact same `PredictionRow` schema `mloop predict --json` already used (`score`/`scoreLowerBound`/`scoreUpperBound`/`intervalConfidence`), so structured consumers (mloop-mcp) get one shape regardless of which surface they call. Because the saved SSA model's horizon is fixed at train time (`variableHorizon` is not enabled), a `horizon` that doesn't match the trained value fails fast with an actionable 400 naming the trained horizon, instead of silently ignoring the request or truncating/padding the result — full elastic (train-time-variable) horizon support is tracked separately as a proposal, since it would change the training contract for all future forecasting models. The replay computation (load model → replay training series from the experiment config → extract native forecast/confidence-band columns) was extracted from CLI-only `PredictCommand` into a new shared `MLoop.Core.Prediction.ForecastReplayService`, so the CLI (`mloop predict`/`--json`) and the API now share one computation and cannot drift on how a forecast is produced (the same lesson as PRED-1/D-series). Pinned by new `PredictForecastingTests` (horizon-match/-mismatch) and a new `ForecastingApiTests` integration suite (empty body, matching horizon, mismatched horizon) that trains a real SSA model and seeds it as a production model end-to-end.
+
+### Fixed
+- **Label-less clustering's structured predict surfaces no longer hard-fail with a Features dimension mismatch (D24)**: `serve POST /predict` and `mloop predict --json` threw `"Schema mismatch for feature column 'Features': expected Vector<Single, 3>, got Vector<Single, 4>"` for every clustering model trained without a declared label (i.e. nearly all of them) — the CSV predict path worked by incidental shape-matching. Root cause: `CsvDataLoader` picks the first CSV column as a placeholder label so `InferColumns` can run on label-less data, excluding it from the merged `Features` vector at train time; the old `RunClusteringAsync` embedded a `Concatenate("Features", [placeholder, Features])` *inside* the fitted+saved pipeline, baking in that incidental shape. At predict time `PredictionService.EnsureFeaturesColumn` (D8) independently builds its own `Features` from every named non-excluded column (the placeholder included, since predict-time has no concept of "placeholder") — so the model's embedded `Concatenate` re-consumed that already-built 3-dim `Features` as one of *its own* two inputs, double-counting a dimension. `AutoMLRunner.RunClusteringAsync` now pre-featurizes before the K-search loop and fits only the trainer on the result, so the saved model's sole input contract is `Features` with no re-concatenation step left to collide with `EnsureFeaturesColumn`. The CLI CSV path (`PredictionEngine`) needed a matching, narrower fix: it now accepts the task type and, for label-less clustering specifically, re-concatenates `InferColumns`' placeholder-label column back into `Features` to match the new bare-trainer model's expectation. Non-destructive: pre-existing model.zip files trained under the old shape are unaffected (they keep their old embedded-concat behavior; only newly-trained clustering models get the corrected contract). Pinned by a new `ClusteringFeaturesContractTests` (RED confirmed against the pre-fix code — reproduces the exact reported error — then GREEN against the fix, covering both the row-based and CSV predict surfaces).
+- **`PredictionService` now rejects a task/model mismatch instead of silently returning an all-null result (P-svc1)**: D20 (zero column overlap), D21 (stateless forecast), and D22 (unread TS-anomaly vector) were each fixed as one-off guards, but they share a single failure shape — a schema/taskType mismatch (a renamed output column, a model trained for a different task than the caller declared, an unhandled model shape) leaves every row's task-defining field null while `ExtractResults` still returns a row per input and the caller still gets a 200/success. `RequireNonDegenerateOutput` generalizes the backstop: after extraction, if there is at least one row and the task's defining output field (`PredictedLabel` for classification/clustering, `Score` for regression/ranking/recommendation, `IsAnomaly`+`AnomalyScore` for anomaly/TS-anomaly) is null on *every* row, it throws an actionable `InvalidOperationException` naming the declared task instead of fabricating an empty-looking "nothing to report" answer. A genuine all-boring-but-real result (e.g. "no anomalies detected", "cluster 0 for everyone") has non-null values and passes through untouched — only total column absence trips the guard. Pinned by four new `PredictionServiceTests` forcing real task/model mismatches (regression model declared as multiclass, ranking model declared as clustering, a bare featurizer declared as anomaly-detection/time-series-anomaly).
+
 ## [0.19.3] - 2026-07-05
 
 ### Fixed

src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs

@@ -199,6 +199,16 @@ private async Task<AutoMLResult> RunObjectDetectionAsync(
                 .Append(_mlContext.Transforms.Conversion.MapKeyToValue(
                     outputColumnName: "PredictedLabel", inputColumnName: "PredictedLabel"));
 
+            // D27: real-data OD training intermittently dies with a native access violation
+            // (0xC0000005) inside libtorch, crash location varying between runs (Linear_forward,
+            // Tensor.backward). Upstream research (dotnet/TorchSharp#1292) traces this class of
+            // crash to native heap corruption whose exact failure point shifts with memory/thread
+            // pressure — forcing libtorch to a single thread removes that pressure source. This is
+            // a defensive mitigation, not a confirmed fix (unverified under this investigation's
+            // resource-constrained environment — see D27 issue); it carries no downside beyond
+            // slower CPU training, so it is applied unconditionally rather than gated on success.
+            TorchSharp.torch.set_num_threads(1);
+
             progress?.Report(new TrainingProgress { TrialNumber = 1, TrainerName = "ObjectDetection (AutoFormerV2)", MetricName = "accuracy", Metric = 0, ElapsedSeconds = 0 });
 
             var model = pipeline.Fit(trainSet);

src/MLoop.Core/AutoML/AutoMLRunner.cs

@@ -663,10 +663,22 @@ public static Dictionary<string, double> ComputeConformalIntervals(
         if (c1Residuals.Count < 5 || GetRowCount(ml, c2) < 5)
             return null; // too small to fit + calibrate meaningfully
 
-        // σ-model: raw numeric features → |residual|. Trained only on C1.
+        // σ-model: raw numeric features → |residual|. Trained only on C1. FastForest, not SDCA:
+        // SDCA's fit is platform-/thread-trajectory-sensitive and can silently DIVERGE on this tiny
+        // aux problem (observed: σ ≈ -8.7e7 single-threaded on win-x64, and the macOS-arm64 CI's
+        // multi-threaded fit degenerated the same way), and the max(σ,0)+β floor then masks the
+        // divergence as a constant-width band while the metadata still claims heteroscedastic.
+        // A forest predicts leaf averages of the |residual| target, so σ is structurally bounded to
+        // the training residual range (never negative, cannot diverge); NumberOfThreads = 1 keeps
+        // the conformal calibration reproducible for the same data.
         var auxPipeline = ml.Transforms.Concatenate("Features", feats)
             .Append(ml.Transforms.NormalizeMinMax("Features"))
-            .Append(ml.Regression.Trainers.Sdca(labelColumnName: "Target", featureColumnName: "Features"));
+            .Append(ml.Regression.Trainers.FastForest(new Microsoft.ML.Trainers.FastTree.FastForestRegressionTrainer.Options
+            {
+                LabelColumnName = "Target",
+                FeatureColumnName = "Features",
+                NumberOfThreads = 1,
+            }));
         var auxModel = auxPipeline.Fit(c1);
 
         // Positive floor β: a low quantile of the C1 residuals, so σ never collapses the band.
@@ -1145,7 +1157,23 @@ private async Task<AutoMLResult> RunClusteringAsync(
             if (featureColumns.Count == 0)
                 throw new InvalidOperationException("No numeric feature columns found for clustering.");
 
-            var concatenate = _mlContext.Transforms.Concatenate("Features", featureColumns.ToArray());
+            // D24: featurize BEFORE fitting and fit only the trainer on the result, instead of
+            // embedding Concatenate inside the saved pipeline. The embedded shape used to be
+            // fragile because CsvDataLoader picks a "dummy label" column for InferColumns on
+            // label-less data (clustering has none), so trainSet's schema mixed a leftover scalar
+            // (the dummy label) with an already-partially-merged "Features" vector that excluded
+            // it — the model's own Concatenate("Features", [dummy, Features]) baked in that exact,
+            // incidental shape. At predict time PredictionService.EnsureFeaturesColumn (D8)
+            // independently builds its own "Features" from every non-excluded named column
+            // (including the erstwhile dummy label, since predict-time has no concept of it) and
+            // the model's embedded Concatenate then re-consumed that already-built "Features" as
+            // one of *its* inputs, double-counting a dimension ("expected Vector<3>, got Vector<4>",
+            // cycle-154). Pre-featurizing here and fitting the trainer alone means the saved model's
+            // only input contract is "Features" — exactly what EnsureFeaturesColumn builds, once,
+            // with no re-concatenation to collide with it.
+            var featurizer = _mlContext.Transforms.Concatenate("Features", featureColumns.ToArray()).Fit(trainSet);
+            var featurizedTrainSet = featurizer.Transform(trainSet);
+            var featurizedTestSet = featurizer.Transform(testSet);
 
             // Determine K values to try
             int[] kValues;
@@ -1173,13 +1201,12 @@ private async Task<AutoMLResult> RunClusteringAsync(
                 cancellationToken.ThrowIfCancellationRequested();
                 var k = kValues[i];
 
-                var pipeline = concatenate
-                    .Append(_mlContext.Clustering.Trainers.KMeans(
-                        featureColumnName: "Features",
-                        numberOfClusters: k));
+                var pipeline = _mlContext.Clustering.Trainers.KMeans(
+                    featureColumnName: "Features",
+                    numberOfClusters: k);
 
-                var model = pipeline.Fit(trainSet);
-                var predictions = model.Transform(testSet);
+                var model = pipeline.Fit(featurizedTrainSet);
+                var predictions = model.Transform(featurizedTestSet);
 
                 // Evaluate. featureColumnName is REQUIRED for ML.NET to compute the Davies-Bouldin
                 // Index — without it DBI comes back 0, which made `useDbi` below always false so the
@@ -1227,7 +1254,7 @@ private async Task<AutoMLResult> RunClusteringAsync(
             // Add cluster distribution info from best model
             if (bestModel != null && bestMetrics != null)
             {
-                var bestPredictions = bestModel.Transform(testSet);
+                var bestPredictions = bestModel.Transform(featurizedTestSet);
                 var clusterCounts = new Dictionary<uint, long>();
                 var predictedLabelCol = bestPredictions.Schema.GetColumnOrNull("PredictedLabel");