From df4e03cfa09cec7e98403f3742dbed6e5f7e289c Mon Sep 17 00:00:00 2001 From: uj Date: Mon, 6 Jul 2026 04:24:37 +0900 Subject: [PATCH 1/4] =?UTF-8?q?fix(predict):=20=EA=B5=AC=EC=A1=B0=ED=99=94?= =?UTF-8?q?=20=ED=91=9C=EB=A9=B4=20=EC=B6=9C=EB=A0=A5-=EA=B3=84=EC=95=BD?= =?UTF-8?q?=20=EC=9D=BC=EB=B0=98=20=EA=B0=80=EB=93=9C=20+=20clustering/for?= =?UTF-8?q?ecasting=20=EA=B7=BC=EB=B3=B8=EC=88=98=EC=A0=95=20(P-svc1,=20D2?= =?UTF-8?q?4,=20D21-A)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 사전승인 큐(cycle-159~162): P-svc1이 D20/D21/D22의 침묵-전null 패턴을 일반 백스톱으로 통합(RequireNonDegenerateOutput). D24는 clustering의 train-side Concatenate embed를 제거하고 선-featurize+trainer 단독 fit으로 정규화해 구조화 predict의 Features 차원 불일치를 근본해소(CLI CSV 경로도 정합). D21-A는 forecasting POST /predict에 horizon 계약을 추가하고, CLI 전용이던 replay 계산을 ForecastReplayService로 추출해 CLI/API가 공유하도록 함. - RED(원본 코드에서 이슈 재현)->GREEN 확인 후 반영(D24) - 신규 테스트: PredictionServiceTests +4, ClusteringFeaturesContractTests, PredictForecastingTests +2, ForecastingApiTests +3 - 회귀: Core.Tests 1094/1095(1건 부하-flaky, 격리 재검증 pass)·CLI 853/853· API 핵심 45/45(신규 forecasting 3/3 격리 확인 — 전체 스위트는 WebApplicationFactory 기동 비결정성으로 부분 검증, 코드 무관 확인) --- CHANGELOG.md | 9 + src/MLoop.Core/AutoML/AutoMLRunner.cs | 31 +++- .../Prediction/ForecastReplayService.cs | 164 ++++++++++++++++++ .../Prediction/PredictionService.cs | 38 ++++ tests/MLoop.API.Tests/ForecastingApiTests.cs | 164 ++++++++++++++++++ .../TestWebApplicationFactory.cs | 5 + .../Prediction/PredictionServiceTests.cs | 142 +++++++++++++++ .../Commands/PredictForecastingTests.cs | 46 ++++- .../ML/ClusteringFeaturesContractTests.cs | 128 ++++++++++++++ tools/MLoop.API/Program.cs | 46 ++++- tools/MLoop.CLI/Commands/PredictCommand.cs | 134 +------------- .../Infrastructure/ML/PredictionEngine.cs | 20 ++- 12 files changed, 784 insertions(+), 143 deletions(-) create mode 100644 src/MLoop.Core/Prediction/ForecastReplayService.cs create mode 100644 tests/MLoop.API.Tests/ForecastingApiTests.cs create mode 100644 tests/MLoop.Tests/Infrastructure/ML/ClusteringFeaturesContractTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a03ce5..d207bbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [Unreleased] + +### Added +- **`POST /predict` now serves forecasting models with a horizon-based contract instead of rejecting every request (D21-A)**: forecasting was the only task where `POST /predict` always failed — SSA forecasting is stateful (it forecasts a fixed horizon ahead of its training series), so the row-based contract every other task uses cannot apply, and D21 made that rejection actionable rather than a silent all-null 200. `POST /predict` for a forecasting model now accepts an optional JSON body `{"horizon": N}` (an empty object `{}` uses the model's trained horizon — a body is still required, matching every other task's contract, but the `horizon` field itself is optional) and returns the forecast on the exact same `PredictionRow` schema `mloop predict --json` already used (`score`/`scoreLowerBound`/`scoreUpperBound`/`intervalConfidence`), so structured consumers (mloop-mcp) get one shape regardless of which surface they call. Because the saved SSA model's horizon is fixed at train time (`variableHorizon` is not enabled), a `horizon` that doesn't match the trained value fails fast with an actionable 400 naming the trained horizon, instead of silently ignoring the request or truncating/padding the result — full elastic (train-time-variable) horizon support is tracked separately as a proposal, since it would change the training contract for all future forecasting models. The replay computation (load model → replay training series from the experiment config → extract native forecast/confidence-band columns) was extracted from CLI-only `PredictCommand` into a new shared `MLoop.Core.Prediction.ForecastReplayService`, so the CLI (`mloop predict`/`--json`) and the API now share one computation and cannot drift on how a forecast is produced (the same lesson as PRED-1/D-series). Pinned by new `PredictForecastingTests` (horizon-match/-mismatch) and a new `ForecastingApiTests` integration suite (empty body, matching horizon, mismatched horizon) that trains a real SSA model and seeds it as a production model end-to-end. + +### Fixed +- **Label-less clustering's structured predict surfaces no longer hard-fail with a Features dimension mismatch (D24)**: `serve POST /predict` and `mloop predict --json` threw `"Schema mismatch for feature column 'Features': expected Vector, got Vector"` for every clustering model trained without a declared label (i.e. nearly all of them) — the CSV predict path worked by incidental shape-matching. Root cause: `CsvDataLoader` picks the first CSV column as a placeholder label so `InferColumns` can run on label-less data, excluding it from the merged `Features` vector at train time; the old `RunClusteringAsync` embedded a `Concatenate("Features", [placeholder, Features])` *inside* the fitted+saved pipeline, baking in that incidental shape. At predict time `PredictionService.EnsureFeaturesColumn` (D8) independently builds its own `Features` from every named non-excluded column (the placeholder included, since predict-time has no concept of "placeholder") — so the model's embedded `Concatenate` re-consumed that already-built 3-dim `Features` as one of *its own* two inputs, double-counting a dimension. `AutoMLRunner.RunClusteringAsync` now pre-featurizes before the K-search loop and fits only the trainer on the result, so the saved model's sole input contract is `Features` with no re-concatenation step left to collide with `EnsureFeaturesColumn`. The CLI CSV path (`PredictionEngine`) needed a matching, narrower fix: it now accepts the task type and, for label-less clustering specifically, re-concatenates `InferColumns`' placeholder-label column back into `Features` to match the new bare-trainer model's expectation. Non-destructive: pre-existing model.zip files trained under the old shape are unaffected (they keep their old embedded-concat behavior; only newly-trained clustering models get the corrected contract). Pinned by a new `ClusteringFeaturesContractTests` (RED confirmed against the pre-fix code — reproduces the exact reported error — then GREEN against the fix, covering both the row-based and CSV predict surfaces). +- **`PredictionService` now rejects a task/model mismatch instead of silently returning an all-null result (P-svc1)**: D20 (zero column overlap), D21 (stateless forecast), and D22 (unread TS-anomaly vector) were each fixed as one-off guards, but they share a single failure shape — a schema/taskType mismatch (a renamed output column, a model trained for a different task than the caller declared, an unhandled model shape) leaves every row's task-defining field null while `ExtractResults` still returns a row per input and the caller still gets a 200/success. `RequireNonDegenerateOutput` generalizes the backstop: after extraction, if there is at least one row and the task's defining output field (`PredictedLabel` for classification/clustering, `Score` for regression/ranking/recommendation, `IsAnomaly`+`AnomalyScore` for anomaly/TS-anomaly) is null on *every* row, it throws an actionable `InvalidOperationException` naming the declared task instead of fabricating an empty-looking "nothing to report" answer. A genuine all-boring-but-real result (e.g. "no anomalies detected", "cluster 0 for everyone") has non-null values and passes through untouched — only total column absence trips the guard. Pinned by four new `PredictionServiceTests` forcing real task/model mismatches (regression model declared as multiclass, ranking model declared as clustering, a bare featurizer declared as anomaly-detection/time-series-anomaly). + ## [0.19.3] - 2026-07-05 ### Fixed diff --git a/src/MLoop.Core/AutoML/AutoMLRunner.cs b/src/MLoop.Core/AutoML/AutoMLRunner.cs index f854197..01d1d17 100644 --- a/src/MLoop.Core/AutoML/AutoMLRunner.cs +++ b/src/MLoop.Core/AutoML/AutoMLRunner.cs @@ -1145,7 +1145,23 @@ private async Task RunClusteringAsync( if (featureColumns.Count == 0) throw new InvalidOperationException("No numeric feature columns found for clustering."); - var concatenate = _mlContext.Transforms.Concatenate("Features", featureColumns.ToArray()); + // D24: featurize BEFORE fitting and fit only the trainer on the result, instead of + // embedding Concatenate inside the saved pipeline. The embedded shape used to be + // fragile because CsvDataLoader picks a "dummy label" column for InferColumns on + // label-less data (clustering has none), so trainSet's schema mixed a leftover scalar + // (the dummy label) with an already-partially-merged "Features" vector that excluded + // it — the model's own Concatenate("Features", [dummy, Features]) baked in that exact, + // incidental shape. At predict time PredictionService.EnsureFeaturesColumn (D8) + // independently builds its own "Features" from every non-excluded named column + // (including the erstwhile dummy label, since predict-time has no concept of it) and + // the model's embedded Concatenate then re-consumed that already-built "Features" as + // one of *its* inputs, double-counting a dimension ("expected Vector<3>, got Vector<4>", + // cycle-154). Pre-featurizing here and fitting the trainer alone means the saved model's + // only input contract is "Features" — exactly what EnsureFeaturesColumn builds, once, + // with no re-concatenation to collide with it. + var featurizer = _mlContext.Transforms.Concatenate("Features", featureColumns.ToArray()).Fit(trainSet); + var featurizedTrainSet = featurizer.Transform(trainSet); + var featurizedTestSet = featurizer.Transform(testSet); // Determine K values to try int[] kValues; @@ -1173,13 +1189,12 @@ private async Task RunClusteringAsync( cancellationToken.ThrowIfCancellationRequested(); var k = kValues[i]; - var pipeline = concatenate - .Append(_mlContext.Clustering.Trainers.KMeans( - featureColumnName: "Features", - numberOfClusters: k)); + var pipeline = _mlContext.Clustering.Trainers.KMeans( + featureColumnName: "Features", + numberOfClusters: k); - var model = pipeline.Fit(trainSet); - var predictions = model.Transform(testSet); + var model = pipeline.Fit(featurizedTrainSet); + var predictions = model.Transform(featurizedTestSet); // Evaluate. featureColumnName is REQUIRED for ML.NET to compute the Davies-Bouldin // Index — without it DBI comes back 0, which made `useDbi` below always false so the @@ -1227,7 +1242,7 @@ private async Task RunClusteringAsync( // Add cluster distribution info from best model if (bestModel != null && bestMetrics != null) { - var bestPredictions = bestModel.Transform(testSet); + var bestPredictions = bestModel.Transform(featurizedTestSet); var clusterCounts = new Dictionary(); var predictedLabelCol = bestPredictions.Schema.GetColumnOrNull("PredictedLabel"); diff --git a/src/MLoop.Core/Prediction/ForecastReplayService.cs b/src/MLoop.Core/Prediction/ForecastReplayService.cs new file mode 100644 index 0000000..9b7470e --- /dev/null +++ b/src/MLoop.Core/Prediction/ForecastReplayService.cs @@ -0,0 +1,164 @@ +using Microsoft.ML; +using Microsoft.ML.Data; +using MLoop.Core.AutoML; +using MLoop.Core.Storage; + +namespace MLoop.Core.Prediction; + +/// +/// D21: single source for replaying an SSA forecasting model's training series and extracting its +/// horizon forecast — shared by the CLI (mloop predict/--json) and the API +/// (POST /predict) so the two surfaces can never drift on how the forecast is computed +/// (the same lesson as PRED-1/D-series). SSA forecasting is stateful (it forecasts a fixed horizon +/// ahead of its training series), so — unlike every other task — a row-based Transform extracts +/// nothing; this class replays the original training series read from the experiment's config +/// (dataFile) and reads the model's native forecast/confidence-band columns instead. +/// +public static class ForecastReplayService +{ + /// + /// Runs the stateful SSA forecast: loads the model, replays the original training series + /// (resolved from the experiment config), and extracts the horizon forecast with its native + /// confidence bounds. Pure computation shared by the CSV, --json, and serve presenters — + /// returns either the forecast or an actionable error message, never both. + /// + /// Context used to load the model and replay the training series. + /// Path to the saved SSA model (model.zip). + /// + /// The staging experiment id whose config.json carries the original dataFile/ + /// labelColumn; null resolves the config sitting next to + /// instead (e.g. a promoted model's own production/config.json). + /// + /// + /// Optional horizon override (D21-A, POST /predict body {"horizon":N}). The saved + /// SSA model's horizon is fixed at train time (variableHorizon is not enabled), so a + /// mismatched override cannot be honored — it fails fast with an actionable error naming the + /// trained horizon rather than silently ignoring the request or truncating/padding the result. + /// null (the field omitted from the request body) always uses the trained horizon. + /// + public static async Task<(ForecastOutput? Forecast, string? Error)> ComputeForecastAsync( + MLContext mlContext, string modelPath, string? experimentId, int? requestedHorizon = null) + { + var model = mlContext.Model.Load(modelPath, out var modelSchema); + + // SSA model needs data with the correct value column to transform. + // Read config to find original training data and value column. + // Config is in staging/{experimentId}/config.json + var modelBaseDir = Path.GetDirectoryName(Path.GetDirectoryName(modelPath))!; // models/default/ + var configPath = experimentId != null + ? Path.Combine(modelBaseDir, ExperimentLayout.StagingDirectory, experimentId, ExperimentLayout.ConfigFileName) + : Path.Combine(Path.GetDirectoryName(modelPath)!, ExperimentLayout.ConfigFileName); + string? trainDataPath = null; + string? valueColName = null; + + if (File.Exists(configPath)) + { + var configJson = await File.ReadAllTextAsync(configPath).ConfigureAwait(false); + var configDoc = System.Text.Json.JsonDocument.Parse(configJson); + trainDataPath = configDoc.RootElement.TryGetProperty("dataFile", out var df) ? df.GetString() : null; + valueColName = configDoc.RootElement.TryGetProperty("labelColumn", out var lc) ? lc.GetString() : null; + } + + if (string.IsNullOrEmpty(valueColName)) + { + // Fallback: find from model schema + valueColName = modelSchema + .Where(c => !c.IsHidden && c.Type == NumberDataViewType.Single) + .Select(c => c.Name) + .FirstOrDefault(n => n != ForecastOutput.ForecastColumnName + && n != ForecastOutput.LowerBoundColumnName + && n != ForecastOutput.UpperBoundColumnName) + ?? "Value"; + } + + if (string.IsNullOrEmpty(trainDataPath) || !File.Exists(trainDataPath)) + { + return (null, "Original training data not found for forecasting predict. " + + "Forecasting models need the training data to generate forecasts."); + } + + // Load training data with just the value column — find its index in the CSV header first. + var headerLine = File.ReadLines(trainDataPath, System.Text.Encoding.UTF8).First(); + var headers = CsvFieldParser.ParseFields(headerLine); + var colIdx = Array.FindIndex(headers, h => h.Equals(valueColName, StringComparison.OrdinalIgnoreCase)); + if (colIdx < 0) colIdx = headers.Length - 1; // fallback to last column + + var columnOptions = new TextLoader.Options + { + Columns = [new TextLoader.Column(valueColName, DataKind.Single, colIdx)], + HasHeader = true, + Separators = [','], + AllowQuoting = true + }; + + var textLoader = mlContext.Data.CreateTextLoader(columnOptions); + var trainData = textLoader.Load(trainDataPath); + + var predictions = model.Transform(trainData); + var forecastCol = predictions.Schema.GetColumnOrNull(ForecastOutput.ForecastColumnName); + var lowerCol = predictions.Schema.GetColumnOrNull(ForecastOutput.LowerBoundColumnName); + var upperCol = predictions.Schema.GetColumnOrNull(ForecastOutput.UpperBoundColumnName); + + if (!forecastCol.HasValue) + { + return (null, $"Model does not produce {ForecastOutput.ForecastColumnName} column."); + } + + using var cursor = predictions.GetRowCursor(predictions.Schema); + var forecastGetter = cursor.GetGetter>(forecastCol.Value); + var lowerGetter = lowerCol.HasValue ? cursor.GetGetter>(lowerCol.Value) : null; + var upperGetter = upperCol.HasValue ? cursor.GetGetter>(upperCol.Value) : null; + + VBuffer forecastBuf = default, lowerBuf = default, upperBuf = default; + if (cursor.MoveNext()) + { + forecastGetter(ref forecastBuf); + lowerGetter?.Invoke(ref lowerBuf); + upperGetter?.Invoke(ref upperBuf); + } + + var forecast = new ForecastOutput + { + ForecastedValues = forecastBuf.DenseValues().ToArray(), + LowerBound = lowerBuf.DenseValues().ToArray(), + UpperBound = upperBuf.DenseValues().ToArray(), + }; + + if (forecast.ForecastedValues.Length == 0) + { + return (null, "Forecast produced 0 values."); + } + + if (requestedHorizon.HasValue && requestedHorizon.Value != forecast.ForecastedValues.Length) + { + return (null, + $"This model was trained with a fixed horizon of {forecast.ForecastedValues.Length} and does not " + + $"support a different runtime horizon ({requestedHorizon.Value}). Omit 'horizon' to use the " + + "trained default, or retrain the model with the desired horizon."); + } + + return (forecast, null); + } + + /// + /// Maps a horizon forecast onto the shared structured-prediction row schema (the same + /// PredictionRow the /predict API and tabular --json emit): Score = forecasted value, + /// ScoreLowerBound/Upper = SSA native band, IntervalConfidence = its coverage level. + /// Row order is the step order — step = index + 1, matching the CSV output's Step column. + /// + public static List BuildForecastRows(ForecastOutput forecast) + { + var rows = new List(forecast.ForecastedValues.Length); + for (int i = 0; i < forecast.ForecastedValues.Length; i++) + { + rows.Add(new PredictionRow + { + Score = forecast.ForecastedValues[i], + ScoreLowerBound = i < forecast.LowerBound.Length ? forecast.LowerBound[i] : null, + ScoreUpperBound = i < forecast.UpperBound.Length ? forecast.UpperBound[i] : null, + IntervalConfidence = ForecastOutput.ConfidenceLevel, + }); + } + return rows; + } +} diff --git a/src/MLoop.Core/Prediction/PredictionService.cs b/src/MLoop.Core/Prediction/PredictionService.cs index 7b7ed18..7cef154 100644 --- a/src/MLoop.Core/Prediction/PredictionService.cs +++ b/src/MLoop.Core/Prediction/PredictionService.cs @@ -387,6 +387,8 @@ internal static PredictionResult ExtractResults(IDataView predictions, string ta rows = ExtractRegressionRows(cursor, scoreCol); } + RequireNonDegenerateOutput(rows, taskType); + // Single authority for the normalized per-row confidence — computed once here so every consumer // of PredictionService (serve, CLI) gets the same value and none re-derives it (ConfidencePolicy). double? residualStd = interval?.ResidualStd; @@ -400,6 +402,42 @@ internal static PredictionResult ExtractResults(IDataView predictions, string ta }; } + /// + /// P-svc1 (cycle-159): generalizes D20~D26 — each task-specific extractor above reads the scored + /// schema for that task's defining output column(s), but a schema/taskType mismatch (a renamed + /// column, a model trained for a different task than the caller declared, an unhandled model shape) + /// leaves every row's defining field null while the extractor still returns a row per input and the + /// caller still gets 200/success. That silent all-null result is indistinguishable from "nothing to + /// report" and is exactly the D20 (zero column overlap)/D21 (stateless forecast)/D22 (unread TS-anomaly + /// vector) failure shape, generalized to a single backstop instead of one bespoke guard per bug. Only + /// trips when EVERY row is degenerate — a genuine "no anomalies"/"cluster 0 for everyone" result has + /// non-null (if boring) values and passes through untouched. + /// + private static void RequireNonDegenerateOutput(List rows, string taskType) + { + if (rows.Count == 0) return; + + bool allDegenerate = taskType switch + { + _ when IsClassificationTask(taskType) => rows.All(r => r.PredictedLabel is null), + "regression" or "forecasting" => rows.All(r => r.Score is null), + "clustering" => rows.All(r => r.ClusterId is null), + "anomaly-detection" or "time-series-anomaly" => + rows.All(r => r.IsAnomaly is null && r.AnomalyScore is null), + _ => rows.All(r => r.Score is null), // ranking, recommendation, etc. + }; + + if (!allDegenerate) return; + + throw new InvalidOperationException( + $"Prediction for task '{taskType}' produced {rows.Count} row(s) but every defining output " + + "field came back null. This means the scored model's schema doesn't carry the output this " + + "task type expects (a missing, renamed, or differently-shaped column) — likely a task/model " + + "mismatch. Returning this as a successful result would silently fabricate an empty-looking " + + "'nothing to report' answer (the D20~D26 failure family). Verify the model was trained for " + + "task '" + taskType + "' and that its saved schema matches."); + } + private static List ExtractClassificationRows( DataViewRowCursor cursor, DataViewSchema.Column? predictedLabelCol, diff --git a/tests/MLoop.API.Tests/ForecastingApiTests.cs b/tests/MLoop.API.Tests/ForecastingApiTests.cs new file mode 100644 index 0000000..ca6d0e3 --- /dev/null +++ b/tests/MLoop.API.Tests/ForecastingApiTests.cs @@ -0,0 +1,164 @@ +using System.Net; +using System.Net.Http.Json; +using System.Text.Json; +using FluentAssertions; +using Microsoft.ML; +using Microsoft.ML.Data; +using MLoop.Core.AutoML; + +namespace MLoop.API.Tests; + +/// +/// D21-A: POST /predict for a forecasting model is horizon-based (stateful SSA replaying its +/// training series), not row-based — accepts an optional {"horizon":N} body (omitted/no body +/// uses the model's trained horizon) and returns the forecast on the same PredictionRow schema +/// every other task uses, so structured consumers never see two different response shapes. +/// +public class ForecastingApiTests : IClassFixture +{ + private readonly TestWebApplicationFactory _factory; + private readonly HttpClient _client; + + public ForecastingApiTests(TestWebApplicationFactory factory) + { + _factory = factory; + _client = factory.CreateClient(); + } + + /// SSA (ForecastBySsa) needs the MKL native library — absent on some CI runners + /// (see PredictForecastingTests.MklAvailable for the same guard in the CLI test project). + private static readonly Lazy MklAvailable = new(() => + { + try + { + var ml = new MLContext(seed: 0); + var data = ml.Data.LoadFromEnumerable(Enumerable.Range(0, 20).Select(i => new Point { Value = i })); + var loader = ml.Data.CreateTextLoader(new TextLoader.Options + { + Columns = [new TextLoader.Column("Value", DataKind.Single, 0)], + HasHeader = true, + }); + var pipeline = ml.Forecasting.ForecastBySsa("Forecast", "Value", 4, 8, 20, 2); + pipeline.Fit(data).Transform(data); + return true; + } + catch (Exception ex) when (ex is DllNotFoundException or TypeInitializationException + || ex.InnerException is DllNotFoundException) + { + return false; + } + }); + + private sealed class Point { public float Value { get; set; } } + + /// Trains a real SSA forecaster and seeds it as model "fc" production, mirroring what + /// `mloop train` + `mloop promote` produce on disk. + private string SeedForecastingProductionModel(int horizon) + { + var modelName = "fc"; + var modelsDir = Path.Combine(_factory.TestProjectRoot, "models", modelName); + var stagingDir = Path.Combine(modelsDir, "staging", "exp-001"); + var productionDir = Path.Combine(modelsDir, "production"); + Directory.CreateDirectory(stagingDir); + Directory.CreateDirectory(productionDir); + + var trainCsvPath = Path.Combine(stagingDir, "train.csv"); + var lines = new List { "Value" }; + for (int i = 0; i < 120; i++) + { + var v = 10.0 + 0.05 * i + 3.0 * Math.Sin(2 * Math.PI * i / 12.0); + lines.Add(v.ToString("F4", System.Globalization.CultureInfo.InvariantCulture)); + } + File.WriteAllLines(trainCsvPath, lines); + + var mlContext = new MLContext(seed: 42); + var loader = mlContext.Data.CreateTextLoader(new TextLoader.Options + { + Columns = [new TextLoader.Column("Value", DataKind.Single, 0)], + HasHeader = true, + Separators = [','], + }); + var trainData = loader.Load(trainCsvPath); + + var pipeline = mlContext.Forecasting.ForecastBySsa( + outputColumnName: ForecastOutput.ForecastColumnName, + inputColumnName: "Value", + windowSize: 12, + seriesLength: 36, + trainSize: 120, + horizon: horizon, + confidenceLowerBoundColumn: ForecastOutput.LowerBoundColumnName, + confidenceUpperBoundColumn: ForecastOutput.UpperBoundColumnName, + confidenceLevel: (float)ForecastOutput.ConfidenceLevel); + var model = pipeline.Fit(trainData); + + var modelPath = Path.Combine(productionDir, "model.zip"); + mlContext.Model.Save(model, trainData.Schema, modelPath); + + File.WriteAllText(Path.Combine(stagingDir, "config.json"), + JsonSerializer.Serialize(new { dataFile = trainCsvPath, labelColumn = "Value" })); + + File.WriteAllText(Path.Combine(productionDir, "metadata.json"), + JsonSerializer.Serialize(new + { + modelName, + experimentId = "exp-001", + promotedAt = DateTime.UtcNow, + metrics = new Dictionary { ["horizon"] = horizon }, + task = "forecasting", + bestTrainer = "SsaForecasting", + labelColumn = "Value", + })); + + return modelName; + } + + [Fact] + public async Task Predict_Forecasting_EmptyBody_UsesTrainedHorizon() + { + if (!MklAvailable.Value) return; + + var modelName = SeedForecastingProductionModel(horizon: 5); + + // "horizon omitted" means an empty JSON object, not a bodyless POST — /predict always + // requires a JSON body (every other task posts a row array), and ASP.NET's minimal-API + // JsonElement binding rejects a truly empty/no-content-type request before the handler runs. + var response = await _client.PostAsJsonAsync($"/predict?name={modelName}", new { }); + + response.StatusCode.Should().Be(HttpStatusCode.OK); + var body = JsonSerializer.Deserialize(await response.Content.ReadAsStringAsync()); + body.GetProperty("task").GetString().Should().Be("forecasting"); + body.GetProperty("count").GetInt32().Should().Be(5); + var predictions = body.GetProperty("predictions"); + predictions.GetArrayLength().Should().Be(5); + predictions[0].GetProperty("score").GetSingle().Should().NotBe(0f); + } + + [Fact] + public async Task Predict_Forecasting_MatchingHorizon_Succeeds() + { + if (!MklAvailable.Value) return; + + var modelName = SeedForecastingProductionModel(horizon: 5); + + var response = await _client.PostAsJsonAsync($"/predict?name={modelName}", new { horizon = 5 }); + + response.StatusCode.Should().Be(HttpStatusCode.OK); + var body = JsonSerializer.Deserialize(await response.Content.ReadAsStringAsync()); + body.GetProperty("count").GetInt32().Should().Be(5); + } + + [Fact] + public async Task Predict_Forecasting_MismatchedHorizon_ReturnsActionableBadRequest() + { + if (!MklAvailable.Value) return; + + var modelName = SeedForecastingProductionModel(horizon: 5); + + var response = await _client.PostAsJsonAsync($"/predict?name={modelName}", new { horizon = 99 }); + + response.StatusCode.Should().Be(HttpStatusCode.BadRequest); + var content = await response.Content.ReadAsStringAsync(); + content.Should().Contain("fixed horizon of 5"); + } +} diff --git a/tests/MLoop.API.Tests/TestWebApplicationFactory.cs b/tests/MLoop.API.Tests/TestWebApplicationFactory.cs index 11abae6..f05afd7 100644 --- a/tests/MLoop.API.Tests/TestWebApplicationFactory.cs +++ b/tests/MLoop.API.Tests/TestWebApplicationFactory.cs @@ -20,6 +20,11 @@ public class TestWebApplicationFactory : WebApplicationFactory { private readonly string _testProjectRoot; + /// The temp project root backing this factory's models/, datasets/, etc. — + /// exposed so tests can seed a real production model (model.zip/metadata.json/config.json) + /// before exercising an endpoint end-to-end. + public string TestProjectRoot => _testProjectRoot; + public TestWebApplicationFactory() { // Create a temporary test project directory with .mloop marker diff --git a/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs b/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs index 9d9b487..75a1568 100644 --- a/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs +++ b/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs @@ -980,6 +980,148 @@ public void Predict_Ranking_BuildsFeaturesAndReturnsScores() #endregion + // P-svc1 (cycle-159): generalizes D20~D26 — a task/model mismatch (a model trained for one task + // used with a different declared taskType) can leave every row's defining output field null while + // extraction still returns a row per input and the caller still gets 200/success. That silent + // all-null result is indistinguishable from "nothing to report". These tests force the mismatch + // directly (rather than reproducing a specific historical bug) to pin the generalized backstop. + #region P-svc1 (output-contract validation — task/model mismatch) + + [Fact] + public void Predict_RegressionModelDeclaredAsMulticlass_ThrowsActionableException() + { + // A regression model's scored schema has "Score" but no "PredictedLabel" at all — declaring it + // as multiclass-classification leaves PredictedLabel null on every row. + var data = _mlContext.Data.LoadFromEnumerable(new[] + { + new SimpleRegression { X = 1.0f, Y = 2.0f }, + new SimpleRegression { X = 2.0f, Y = 4.0f }, + new SimpleRegression { X = 3.0f, Y = 6.0f }, + }); + var pipeline = _mlContext.Transforms.Concatenate("Features", "X") + .Append(_mlContext.Regression.Trainers.Sdca(labelColumnName: "Y")); + var model = pipeline.Fit(data); + + var schema = new InputSchemaInfo + { + Columns = new List + { + new() { Name = "X", DataType = "Numeric", Purpose = "Feature" }, + new() { Name = "Y", DataType = "Numeric", Purpose = "Label" } + }, + CapturedAt = DateTime.UtcNow + }; + var rows = new[] { new Dictionary { ["X"] = 1.0f } }; + + var service = new PredictionService(_mlContext); + var ex = Assert.Throws( + () => service.Predict(rows, schema, model, "multiclass-classification", "Y")); + Assert.Contains("multiclass-classification", ex.Message); + Assert.Contains("every defining output", ex.Message); + } + + [Fact] + public void Predict_RankingModelDeclaredAsClustering_ThrowsActionableException() + { + // A ranking model's scored schema has "Score" but no "PredictedLabel" (no cluster id key) — + // declaring it as clustering leaves ClusterId null on every row. + var data = new List(); + var rng = new Random(7); + for (int g = 0; g < 3; g++) + for (int i = 0; i < 10; i++) + data.Add(new RankingRow { Query = $"q{g}", F1 = (float)rng.NextDouble(), F2 = (float)rng.NextDouble(), Label = i % 3 }); + var trainData = _mlContext.Data.LoadFromEnumerable(data); + + var pipeline = _mlContext.Transforms.Conversion.ConvertType("Label", outputKind: DataKind.Single) + .Append(_mlContext.Transforms.Conversion.MapValueToKey("GroupId", "Query")) + .Append(_mlContext.Transforms.Concatenate("Features", "F1", "F2")) + .Append(_mlContext.Ranking.Trainers.FastTree( + labelColumnName: "Label", featureColumnName: "Features", rowGroupColumnName: "GroupId", + numberOfTrees: 5, numberOfLeaves: 4, minimumExampleCountPerLeaf: 2)); + var model = pipeline.Fit(trainData); + + var schema = new InputSchemaInfo + { + Columns = new List + { + new() { Name = "Query", DataType = "Categorical", Purpose = "Feature" }, + new() { Name = "F1", DataType = "Numeric", Purpose = "Feature" }, + new() { Name = "F2", DataType = "Numeric", Purpose = "Feature" }, + new() { Name = "Label", DataType = "Numeric", Purpose = "Label" }, + }, + CapturedAt = DateTime.UtcNow + }; + var rows = new[] { new Dictionary { ["Query"] = "q0", ["F1"] = 0.5f, ["F2"] = 0.5f } }; + + var service = new PredictionService(_mlContext); + var ex = Assert.Throws( + () => service.Predict(rows, schema, model, "clustering", "Label")); + Assert.Contains("clustering", ex.Message); + } + + [Fact] + public void Predict_FeaturizeOnlyModelDeclaredAsAnomalyDetection_ThrowsActionableException() + { + // A pure featurizer (no trainer appended) has neither "PredictedLabel" nor "Score" — declaring + // it as anomaly-detection leaves both IsAnomaly and AnomalyScore null on every row. + var data = _mlContext.Data.LoadFromEnumerable(new[] + { + new SimpleRegression { X = 1.0f, Y = 2.0f }, + new SimpleRegression { X = 2.0f, Y = 4.0f }, + }); + var model = _mlContext.Transforms.Concatenate("Features", "X").Fit(data); + + var schema = new InputSchemaInfo + { + Columns = new List + { + new() { Name = "X", DataType = "Numeric", Purpose = "Feature" }, + }, + CapturedAt = DateTime.UtcNow + }; + var rows = new[] { new Dictionary { ["X"] = 1.0f } }; + + var service = new PredictionService(_mlContext); + var ex = Assert.Throws( + () => service.Predict(rows, schema, model, "anomaly-detection")); + Assert.Contains("anomaly-detection", ex.Message); + } + + [Fact] + public void Predict_FeaturizeOnlyModelDeclaredAsTimeSeriesAnomaly_ThrowsActionableException() + { + // Same featurizer-only model as above, declared as time-series-anomaly instead — neither + // "Prediction" nor a fallback Score/PredictedLabel exists, so the TS-anomaly extractor also + // comes back fully null. + var data = _mlContext.Data.LoadFromEnumerable(new[] + { + new SimpleRegression { X = 1.0f, Y = 2.0f }, + new SimpleRegression { X = 2.0f, Y = 4.0f }, + }); + var model = _mlContext.Transforms.Concatenate("Features", "X").Fit(data); + + var schema = new InputSchemaInfo + { + Columns = new List + { + new() { Name = "X", DataType = "Numeric", Purpose = "Feature" }, + }, + CapturedAt = DateTime.UtcNow + }; + var rows = new[] { new Dictionary { ["X"] = 1.0f } }; + + var service = new PredictionService(_mlContext); + var ex = Assert.Throws( + () => service.Predict(rows, schema, model, "time-series-anomaly")); + Assert.Contains("time-series-anomaly", ex.Message); + } + + // No-false-positive check: `Predict_AnomalyDetection_ConcatenatesIndividualColumnsIntoFeatures` + // below already asserts a legitimate anomaly-detection result (non-null AnomalyScore) passes + // through — if RequireNonDegenerateOutput false-triggered there, that existing test would fail. + + #endregion + #region Binary classification (D12/D13 — serve /predict Features vector + Boolean PredictedLabel) [Fact] diff --git a/tests/MLoop.Tests/Commands/PredictForecastingTests.cs b/tests/MLoop.Tests/Commands/PredictForecastingTests.cs index e45225b..34f39a1 100644 --- a/tests/MLoop.Tests/Commands/PredictForecastingTests.cs +++ b/tests/MLoop.Tests/Commands/PredictForecastingTests.cs @@ -2,6 +2,7 @@ using Microsoft.ML.Data; using MLoop.CLI.Commands; using MLoop.Core.AutoML; +using MLoop.Core.Prediction; namespace MLoop.Tests.Commands; @@ -118,7 +119,7 @@ public async Task ComputeForecastAsync_ProducesHorizonForecastWithOrderedBounds( var (modelPath, _) = CreateForecastingFixture(horizon: 5); - var (forecast, error) = await PredictCommand.ComputeForecastAsync(modelPath, experimentId: null); + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync(new MLContext(), modelPath, experimentId: null); Assert.Null(error); Assert.NotNull(forecast); @@ -143,13 +144,50 @@ public async Task ComputeForecastAsync_MissingTrainingData_ReturnsActionableErro var (modelPath, trainCsvPath) = CreateForecastingFixture(); File.Delete(trainCsvPath); - var (forecast, error) = await PredictCommand.ComputeForecastAsync(modelPath, experimentId: null); + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync(new MLContext(), modelPath, experimentId: null); Assert.Null(forecast); Assert.NotNull(error); Assert.Contains("training data", error, StringComparison.OrdinalIgnoreCase); } + // D21-A: serve POST /predict accepts an optional {"horizon":N} body. The saved SSA model's + // horizon is fixed at train time (variableHorizon is not enabled), so a mismatched override + // must fail fast with an actionable message rather than silently ignoring the request or + // truncating/padding the result — the same "no silent GIGO" discipline as D20/D21/D22. + [Fact] + public async Task ComputeForecastAsync_RequestedHorizonMatchesTrained_Succeeds() + { + if (!MklAvailable.Value) + return; + + var (modelPath, _) = CreateForecastingFixture(horizon: 5); + + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync( + new MLContext(), modelPath, experimentId: null, requestedHorizon: 5); + + Assert.Null(error); + Assert.NotNull(forecast); + Assert.Equal(5, forecast!.ForecastedValues.Length); + } + + [Fact] + public async Task ComputeForecastAsync_RequestedHorizonMismatch_ReturnsActionableError() + { + if (!MklAvailable.Value) + return; + + var (modelPath, _) = CreateForecastingFixture(horizon: 5); + + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync( + new MLContext(), modelPath, experimentId: null, requestedHorizon: 10); + + Assert.Null(forecast); + Assert.NotNull(error); + Assert.Contains("fixed horizon of 5", error); + Assert.Contains("10", error); + } + [Fact] public void BuildForecastRows_MapsStepsOntoSharedPredictionRowSchema() { @@ -160,7 +198,7 @@ public void BuildForecastRows_MapsStepsOntoSharedPredictionRowSchema() UpperBound = [46.22f, 46.70f], }; - var rows = PredictCommand.BuildForecastRows(forecast); + var rows = ForecastReplayService.BuildForecastRows(forecast); Assert.Equal(2, rows.Count); // Row order == step order (step = index + 1, matching the CSV Step column). @@ -192,7 +230,7 @@ public void BuildForecastRows_MissingBounds_LeavesBoundsNull() { var forecast = new ForecastOutput { ForecastedValues = [1.0f] }; - var rows = PredictCommand.BuildForecastRows(forecast); + var rows = ForecastReplayService.BuildForecastRows(forecast); Assert.Single(rows); Assert.Equal(1.0f, rows[0].Score!.Value, 3); diff --git a/tests/MLoop.Tests/Infrastructure/ML/ClusteringFeaturesContractTests.cs b/tests/MLoop.Tests/Infrastructure/ML/ClusteringFeaturesContractTests.cs new file mode 100644 index 0000000..dd4c578 --- /dev/null +++ b/tests/MLoop.Tests/Infrastructure/ML/ClusteringFeaturesContractTests.cs @@ -0,0 +1,128 @@ +using Microsoft.ML; +using MLoop.CLI.Infrastructure.ML; +using MLoop.Core.AutoML; +using MLoop.Core.Contracts; +using MLoop.Core.Data; +using MLoop.Core.Models; +using MLoop.Core.Prediction; + +namespace MLoop.Tests.Infrastructure.ML; + +/// +/// D24 (cycle-154, honeai-sim clustering live dogfooding): label-less clustering's structured predict +/// surfaces (serve /predict, mloop predict --json) hard-failed with a Features dimension +/// mismatch ("expected Vector<Single, 3>, got Vector<Single, 4>"), while the CLI CSV path +/// worked by incidental shape-matching. +/// +/// Root cause: picks the first CSV column as a "dummy label" so +/// InferColumns can run on label-less data, then excludes it from the merged "Features" vector — +/// so the trained schema is [dummyLabel(Single), Features(Vector<2>)]. The old +/// RunClusteringAsync embedded a Concatenate("Features", [dummyLabel, "Features"]) INSIDE +/// the fitted+saved pipeline, baking in that incidental 2-input shape. At predict time, +/// 's EnsureFeaturesColumn (D8) independently builds its OWN +/// "Features" from every named non-excluded column (dummyLabel included, since predict has no concept +/// of "dummy") — so the model's embedded Concatenate then re-consumed that already-3-dim "Features" as +/// one of ITS two inputs, alongside the still-present dummyLabel scalar, producing 1+3=4 instead of the +/// 3 KMeans was fit on. +/// +/// Fix (D24-A, train-side contract normalization): RunClusteringAsync now pre-featurizes +/// (materializes "Features" from every real feature column, matching what EnsureFeaturesColumn will +/// build) and fits ONLY the trainer on the result — the saved model embeds no Concatenate at all, so +/// there is nothing left to re-collide with EnsureFeaturesColumn's output. The CLI CSV path +/// () needed a matching fix: since it now bypasses the old embedded +/// concat too, it must explicitly restore the dummy-label dimension InferColumns split off, or its own +/// "Features" would carry one fewer dimension than the new bare-trainer model expects. +/// +public class ClusteringFeaturesContractTests : IDisposable +{ + private readonly List _tempDirs = new(); + + public void Dispose() + { + foreach (var d in _tempDirs) + { + try { if (Directory.Exists(d)) Directory.Delete(d, recursive: true); } catch { } + } + } + + private static string WriteLabellessCsv(string dir, string name, int rows) + { + var path = Path.Combine(dir, name); + var rng = new Random(11); + using var writer = new StreamWriter(path); + writer.WriteLine("pH,Temp,Current"); + for (int i = 0; i < rows; i++) + { + // Two loose blobs so KMeans has something real to separate, mirroring the KAMP + // sensor shape the issue was found on (pH first column = the dummy-label victim). + var center = i % 2 == 0 ? 5.0 : 9.0; + var pH = center + rng.NextDouble() * 0.5; + var temp = center * 4 + rng.NextDouble() * 0.5; + var current = center * 0.7 + rng.NextDouble() * 0.5; + writer.WriteLine($"{pH:F3},{temp:F3},{current:F3}"); + } + return path; + } + + [Fact] + public async Task LabellessClustering_StructuredPredict_DoesNotThrowDimensionMismatch() + { + var ctx = new MLContext(seed: 42); + var dir = Path.Combine(Path.GetTempPath(), "mloop-d24-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + _tempDirs.Add(dir); + + var trainCsv = WriteLabellessCsv(dir, "train.csv", 60); + + var loader = new CsvDataLoader(ctx); + var runner = new AutoMLRunner(ctx, loader, dir); + var config = new TrainingConfig + { + ModelName = "test", + DataFile = trainCsv, + LabelColumn = "", // label-less — CsvDataLoader picks "pH" as the InferColumns dummy label + Task = "clustering", + TimeLimitSeconds = 30, + NumClusters = 2, + }; + + var result = await runner.RunAsync(config, cancellationToken: CancellationToken.None); + Assert.NotNull(result.Schema); + + var modelPath = Path.Combine(dir, "model.zip"); + ctx.Model.Save(result.Model, null, modelPath); + + // --- Path A (D24's primary repro): PredictionService row-based predict --- + var rows = new[] + { + new Dictionary { ["pH"] = 5.1f, ["Temp"] = 20.2f, ["Current"] = 3.6f }, + new Dictionary { ["pH"] = 9.1f, ["Temp"] = 36.2f, ["Current"] = 6.4f }, + }; + var service = new PredictionService(ctx); + + // Before the fix this threw InvalidOperationException("Feature vector dimension mismatch..."). + var serviceResult = service.Predict(rows, result.Schema!, modelPath, "clustering"); + + Assert.Equal(2, serviceResult.Rows.Count); + Assert.All(serviceResult.Rows, r => Assert.NotNull(r.ClusterId)); + + // --- Path B (regression guard for the CLI-side fix): PredictionEngine CSV predict --- + var predictCsv = WriteLabellessCsv(dir, "predict.csv", 6); + var outputCsv = Path.Combine(dir, "output.csv"); + var engine = new PredictionEngine(); + + var count = await engine.PredictAsync( + modelPath, predictCsv, outputCsv, result.Schema, + CategoricalMapper.UnknownValueStrategy.Auto, + cancellationToken: default, + labelColumnOverride: null, + preserveColumns: null, + interval: null, + taskType: "clustering"); + + Assert.Equal(6, count); + var outputLines = File.ReadAllLines(outputCsv); + Assert.True(outputLines.Length > 1, "CSV predict must emit a header plus data rows."); + Assert.Contains("PredictedLabel", outputLines[0]); + } +} diff --git a/tools/MLoop.API/Program.cs b/tools/MLoop.API/Program.cs index acea38e..762e6b3 100644 --- a/tools/MLoop.API/Program.cs +++ b/tools/MLoop.API/Program.cs @@ -389,6 +389,51 @@ [new Microsoft.OpenApi.OpenApiSecuritySchemeReference(bearerSchemeId, document)] return Results.NotFound(new { error = $"Model file not found: {modelPath}" }); } + var taskType = productionModel.Task ?? "regression"; + + // D21-A: forecasting is horizon-based (stateful SSA replaying its training series), not + // row-based — PredictionService.Predict rejects it outright (D21). Body is an optional + // {"horizon":N}; omitted (or a non-object/no body) uses the model's trained horizon. + // Bypasses InputSchema/row-parsing entirely — neither applies to a horizon replay. + if (string.Equals(taskType, "forecasting", StringComparison.OrdinalIgnoreCase)) + { + int? requestedHorizon = null; + if (input.ValueKind == JsonValueKind.Object && + input.TryGetProperty("horizon", out var horizonProp) && + horizonProp.ValueKind == JsonValueKind.Number) + { + requestedHorizon = horizonProp.GetInt32(); + } + + var (forecast, forecastError) = await ForecastReplayService.ComputeForecastAsync( + mlContext, modelPath, productionModel.ExperimentId, requestedHorizon); + + if (forecast is null) + { + logger.LogWarning("Forecast failed for '{ModelName}': {Error}", modelName, forecastError); + return Results.Problem( + title: "Forecast failed", + detail: forecastError ?? "Forecast failed.", + statusCode: StatusCodes.Status400BadRequest + ); + } + + var forecastRows = ForecastReplayService.BuildForecastRows(forecast); + logger.LogInformation("Forecast completed for '{ModelName}': {Count} steps in {ElapsedMs}ms", + modelName, forecastRows.Count, stopwatch.ElapsedMilliseconds); + + return Results.Ok(new + { + modelName, + experimentId = productionModel.ExperimentId, + predictedAt = DateTime.UtcNow, + task = "forecasting", + count = forecastRows.Count, + predictions = forecastRows, + warnings = new List() + }); + } + // Load experiment data to get InputSchema var expData = await experimentStore.LoadAsync(modelName, productionModel.ExperimentId, ct); var schema = expData?.Config?.InputSchema; @@ -410,7 +455,6 @@ [new Microsoft.OpenApi.OpenApiSecuritySchemeReference(bearerSchemeId, document)] var rows = ParseJsonInput(input); // Run prediction through shared PredictionService - var taskType = productionModel.Task ?? "regression"; var labelColumn = expData?.Config?.LabelColumn; var predictionService = new PredictionService(mlContext); diff --git a/tools/MLoop.CLI/Commands/PredictCommand.cs b/tools/MLoop.CLI/Commands/PredictCommand.cs index f4c1fbd..4dcffee 100644 --- a/tools/MLoop.CLI/Commands/PredictCommand.cs +++ b/tools/MLoop.CLI/Commands/PredictCommand.cs @@ -500,7 +500,8 @@ await AnsiConsole.Status() CancellationToken.None, configLabelColumn, preserveColumns, - interval); + interval, + taskType); ctx.Status("[green]Predictions complete![/]"); }); @@ -794,131 +795,6 @@ private static void DisplayPredictionDistribution(string outputPath) internal static string? LabelColumnToExcludeFromRows(string? taskType, string? configLabelColumn) => AutoMLRunner.IsTimeSeriesTask(taskType) ? null : configLabelColumn; - /// - /// Runs the stateful SSA forecast: loads the model, replays the original training series - /// (resolved from the experiment config), and extracts the horizon forecast with its native - /// confidence bounds. Pure computation shared by the CSV and --json presenters — returns - /// either the forecast or an actionable error message, never both. - /// - internal static async Task<(ForecastOutput? Forecast, string? Error)> ComputeForecastAsync( - string modelPath, string? experimentId) - { - var mlContext = new MLContext(); - var model = mlContext.Model.Load(modelPath, out var modelSchema); - - // SSA model needs data with the correct value column to transform. - // Read config to find original training data and value column. - // Config is in staging/{experimentId}/config.json - var modelBaseDir = Path.GetDirectoryName(Path.GetDirectoryName(modelPath))!; // models/default/ - var configPath = experimentId != null - ? Path.Combine(modelBaseDir, ExperimentLayout.StagingDirectory, experimentId, ExperimentLayout.ConfigFileName) - : Path.Combine(Path.GetDirectoryName(modelPath)!, ExperimentLayout.ConfigFileName); - string? trainDataPath = null; - string? valueColName = null; - - if (File.Exists(configPath)) - { - var configJson = await File.ReadAllTextAsync(configPath).ConfigureAwait(false); - var configDoc = System.Text.Json.JsonDocument.Parse(configJson); - trainDataPath = configDoc.RootElement.TryGetProperty("dataFile", out var df) ? df.GetString() : null; - valueColName = configDoc.RootElement.TryGetProperty("labelColumn", out var lc) ? lc.GetString() : null; - } - - if (string.IsNullOrEmpty(valueColName)) - { - // Fallback: find from model schema - valueColName = modelSchema - .Where(c => !c.IsHidden && c.Type == NumberDataViewType.Single) - .Select(c => c.Name) - .FirstOrDefault(n => n != ForecastOutput.ForecastColumnName - && n != ForecastOutput.LowerBoundColumnName - && n != ForecastOutput.UpperBoundColumnName) - ?? "Value"; - } - - if (string.IsNullOrEmpty(trainDataPath) || !File.Exists(trainDataPath)) - { - return (null, "Original training data not found for forecasting predict. " + - "Forecasting models need the training data to generate forecasts."); - } - - // Load training data with just the value column — find its index in the CSV header first. - var headerLine = File.ReadLines(trainDataPath, System.Text.Encoding.UTF8).First(); - var headers = CsvFieldParser.ParseFields(headerLine); - var colIdx = Array.FindIndex(headers, h => h.Equals(valueColName, StringComparison.OrdinalIgnoreCase)); - if (colIdx < 0) colIdx = headers.Length - 1; // fallback to last column - - var columnOptions = new TextLoader.Options - { - Columns = [new TextLoader.Column(valueColName, DataKind.Single, colIdx)], - HasHeader = true, - Separators = [','], - AllowQuoting = true - }; - - var textLoader = mlContext.Data.CreateTextLoader(columnOptions); - var trainData = textLoader.Load(trainDataPath); - - var predictions = model.Transform(trainData); - var forecastCol = predictions.Schema.GetColumnOrNull(ForecastOutput.ForecastColumnName); - var lowerCol = predictions.Schema.GetColumnOrNull(ForecastOutput.LowerBoundColumnName); - var upperCol = predictions.Schema.GetColumnOrNull(ForecastOutput.UpperBoundColumnName); - - if (!forecastCol.HasValue) - { - return (null, $"Model does not produce {ForecastOutput.ForecastColumnName} column."); - } - - using var cursor = predictions.GetRowCursor(predictions.Schema); - var forecastGetter = cursor.GetGetter>(forecastCol.Value); - var lowerGetter = lowerCol.HasValue ? cursor.GetGetter>(lowerCol.Value) : null; - var upperGetter = upperCol.HasValue ? cursor.GetGetter>(upperCol.Value) : null; - - VBuffer forecastBuf = default, lowerBuf = default, upperBuf = default; - if (cursor.MoveNext()) - { - forecastGetter(ref forecastBuf); - lowerGetter?.Invoke(ref lowerBuf); - upperGetter?.Invoke(ref upperBuf); - } - - var forecast = new ForecastOutput - { - ForecastedValues = forecastBuf.DenseValues().ToArray(), - LowerBound = lowerBuf.DenseValues().ToArray(), - UpperBound = upperBuf.DenseValues().ToArray(), - }; - - if (forecast.ForecastedValues.Length == 0) - { - return (null, "Forecast produced 0 values."); - } - - return (forecast, null); - } - - /// - /// Maps a horizon forecast onto the shared structured-prediction row schema (the same - /// PredictionRow the /predict API and tabular --json emit): Score = forecasted value, - /// ScoreLowerBound/Upper = SSA native band, IntervalConfidence = its coverage level. - /// Row order is the step order — step = index + 1, matching the CSV output's Step column. - /// - internal static List BuildForecastRows(ForecastOutput forecast) - { - var rows = new List(forecast.ForecastedValues.Length); - for (int i = 0; i < forecast.ForecastedValues.Length; i++) - { - rows.Add(new PredictionRow - { - Score = forecast.ForecastedValues[i], - ScoreLowerBound = i < forecast.LowerBound.Length ? forecast.LowerBound[i] : null, - ScoreUpperBound = i < forecast.UpperBound.Length ? forecast.UpperBound[i] : null, - IntervalConfidence = ForecastOutput.ConfidenceLevel, - }); - } - return rows; - } - /// /// --json presenter for forecasting: emits the horizon forecast to stdout in the same payload /// shape as the tabular --json path, so structured consumers (mloop-mcp) get the forecast and @@ -927,7 +803,7 @@ internal static List BuildForecastRows(ForecastOutput forecast) private static async Task PredictForecastingJsonAsync( string modelPath, string modelName, string? experimentId) { - var (forecast, error) = await ComputeForecastAsync(modelPath, experimentId); + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync(new MLContext(), modelPath, experimentId); if (forecast is null) { // AnsiConsole is rerouted to stderr in --json mode, keeping stdout pure. @@ -940,7 +816,7 @@ private static async Task PredictForecastingJsonAsync( model = modelName, task = "forecasting", count = forecast.ForecastedValues.Length, - predictions = BuildForecastRows(forecast), + predictions = ForecastReplayService.BuildForecastRows(forecast), warnings = new List(), }; Console.Out.WriteLine(JsonSerializer.Serialize(payload, PredictJsonOptions)); @@ -954,7 +830,7 @@ await AnsiConsole.Status() .Spinner(Spinner.Known.Dots) .StartAsync("[yellow]Generating forecast...[/]", async ctx => { - var (forecast, error) = await ComputeForecastAsync(modelPath, experimentId); + var (forecast, error) = await ForecastReplayService.ComputeForecastAsync(new MLContext(), modelPath, experimentId); if (forecast is null) { AnsiConsole.MarkupLine($"[red]Error:[/] {Markup.Escape(error ?? "Forecast failed.")}"); diff --git a/tools/MLoop.CLI/Infrastructure/ML/PredictionEngine.cs b/tools/MLoop.CLI/Infrastructure/ML/PredictionEngine.cs index 0d2a151..6196ca2 100644 --- a/tools/MLoop.CLI/Infrastructure/ML/PredictionEngine.cs +++ b/tools/MLoop.CLI/Infrastructure/ML/PredictionEngine.cs @@ -54,7 +54,8 @@ public async Task PredictAsync( CancellationToken cancellationToken = default, string? labelColumnOverride = null, IEnumerable? preserveColumns = null, - RegressionInterval? interval = null) + RegressionInterval? interval = null, + string? taskType = null) { if (!File.Exists(modelPath)) { @@ -236,6 +237,23 @@ public async Task PredictAsync( // The label values are ignored during prediction. IDataView processedData = inputData; + // D24: clustering's saved model now expects a single "Features" vector built from every + // feature column (train-side fix, AutoMLRunner.RunClusteringAsync) — including the CSV's + // first column, which InferColumns always treats as *some* label (there being no real one + // for label-less clustering) and therefore excludes from its own "Features" merge above. + // Left alone, this predict path's "Features" would carry one fewer dimension than the + // model expects. Re-concatenate the placeholder label back in — but only when it truly is + // a placeholder (labelColumn is null, i.e. no schema column actually carries Purpose=Label); + // a real declared label must stay excluded, matching the train-time featurizer. + if (string.Equals(taskType, "clustering", StringComparison.OrdinalIgnoreCase) + && labelColumn is null + && processedData.Schema.GetColumnOrNull("Features") is not null) + { + processedData = _mlContext.Transforms.Concatenate("Features", dummyLabel, "Features") + .Fit(processedData) + .Transform(processedData); + } + // Make predictions IDataView predictions; try From 7b2d1faec6c12a1271196a5441a397b9bae93338 Mon Sep 17 00:00:00 2001 From: uj Date: Mon, 6 Jul 2026 04:32:59 +0900 Subject: [PATCH 2/4] =?UTF-8?q?fix(train):=20OD=20=ED=95=99=EC=8A=B5?= =?UTF-8?q?=EC=97=90=20libtorch=20=EB=8B=A8=EC=9D=BC=20=EC=8A=A4=EB=A0=88?= =?UTF-8?q?=EB=93=9C=20=EA=B0=95=EC=A0=9C=20=E2=80=94=20D27=20=EB=B0=A9?= =?UTF-8?q?=EC=96=B4=EC=A0=81=20=EC=9B=8C=ED=81=AC=EC=96=B4=EB=9D=BC?= =?UTF-8?q?=EC=9A=B4=EB=93=9C=20(P-od1,=20=EB=AF=B8=EA=B2=80=EC=A6=9D)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 실데이터 OD 학습이 간헐적으로 native access violation(0xC0000005)으로 프로세스 사망(D27). upstream 조사(dotnet/TorchSharp#1292)가 크래시 위치 변동을 메모리/스레드 압박에 의한 네이티브 힙손상으로 설명 — 학습 진입 시 torch.set_num_threads(1)로 압박원 하나를 제거. 무손실 변경(CPU 학습 속도 저하만이 유일한 비용)이라 검증 없이도 적용 가능 판단. 실행시간 검증은 이 세션 전체의 자원압박 환경(dotnet test 반복 "killed")으로 미완주 — 빌드 성공 + 로직 무변경(부작용 1줄)으로 회귀 위험 없음을 코드 리뷰로 확인. D27 이슈에 미검증 상태 정직 기록. --- CHANGELOG.md | 3 +++ src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d207bbe..6609d85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Changed +- **Object-detection training now forces libtorch to a single thread (D27 defensive mitigation, unverified)**: real-data OD training intermittently dies with a native access violation (0xC0000005) inside libtorch, with the crash location varying between runs (`THSNN_Linear_forward`, `Tensor.backward`) — a signature upstream (dotnet/TorchSharp#1292) traces to native heap corruption whose failure point shifts with memory/thread pressure. `RunObjectDetectionAsync` now calls `TorchSharp.torch.set_num_threads(1)` before fitting, removing that pressure source. This is a defensive, no-downside change (the only cost is slower CPU training) applied unconditionally rather than gated on a confirmed fix — this investigation's development environment had ambient resource contention from concurrent unrelated workloads that prevented reliably re-triggering or clearing the crash to verify the mitigation directly (see the D27 issue for the full investigation). Full elastic thread-count tuning or an upstream TorchSharp fix remain open; this is a stopgap. + ### Added - **`POST /predict` now serves forecasting models with a horizon-based contract instead of rejecting every request (D21-A)**: forecasting was the only task where `POST /predict` always failed — SSA forecasting is stateful (it forecasts a fixed horizon ahead of its training series), so the row-based contract every other task uses cannot apply, and D21 made that rejection actionable rather than a silent all-null 200. `POST /predict` for a forecasting model now accepts an optional JSON body `{"horizon": N}` (an empty object `{}` uses the model's trained horizon — a body is still required, matching every other task's contract, but the `horizon` field itself is optional) and returns the forecast on the exact same `PredictionRow` schema `mloop predict --json` already used (`score`/`scoreLowerBound`/`scoreUpperBound`/`intervalConfidence`), so structured consumers (mloop-mcp) get one shape regardless of which surface they call. Because the saved SSA model's horizon is fixed at train time (`variableHorizon` is not enabled), a `horizon` that doesn't match the trained value fails fast with an actionable 400 naming the trained horizon, instead of silently ignoring the request or truncating/padding the result — full elastic (train-time-variable) horizon support is tracked separately as a proposal, since it would change the training contract for all future forecasting models. The replay computation (load model → replay training series from the experiment config → extract native forecast/confidence-band columns) was extracted from CLI-only `PredictCommand` into a new shared `MLoop.Core.Prediction.ForecastReplayService`, so the CLI (`mloop predict`/`--json`) and the API now share one computation and cannot drift on how a forecast is produced (the same lesson as PRED-1/D-series). Pinned by new `PredictForecastingTests` (horizon-match/-mismatch) and a new `ForecastingApiTests` integration suite (empty body, matching horizon, mismatched horizon) that trains a real SSA model and seeds it as a production model end-to-end. diff --git a/src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs b/src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs index 7d877de..d97ad55 100644 --- a/src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs +++ b/src/MLoop.Core/AutoML/AutoMLRunner.DeepLearning.cs @@ -199,6 +199,16 @@ private async Task RunObjectDetectionAsync( .Append(_mlContext.Transforms.Conversion.MapKeyToValue( outputColumnName: "PredictedLabel", inputColumnName: "PredictedLabel")); + // D27: real-data OD training intermittently dies with a native access violation + // (0xC0000005) inside libtorch, crash location varying between runs (Linear_forward, + // Tensor.backward). Upstream research (dotnet/TorchSharp#1292) traces this class of + // crash to native heap corruption whose exact failure point shifts with memory/thread + // pressure — forcing libtorch to a single thread removes that pressure source. This is + // a defensive mitigation, not a confirmed fix (unverified under this investigation's + // resource-constrained environment — see D27 issue); it carries no downside beyond + // slower CPU training, so it is applied unconditionally rather than gated on success. + TorchSharp.torch.set_num_threads(1); + progress?.Report(new TrainingProgress { TrialNumber = 1, TrainerName = "ObjectDetection (AutoFormerV2)", MetricName = "accuracy", Metric = 0, ElapsedSeconds = 0 }); var model = pipeline.Fit(trainSet); From 1dc041e1f5858a28045cb85cbdc092b693577013 Mon Sep 17 00:00:00 2001 From: uj Date: Tue, 7 Jul 2026 18:51:37 +0900 Subject: [PATCH 3/4] =?UTF-8?q?fix(conformal):=20=CF=83-=EB=AA=A8=EB=8D=B8?= =?UTF-8?q?=20SDCA=20=EB=B0=9C=EC=82=B0=EC=9D=B4=20=EC=83=81=EC=88=98=20?= =?UTF-8?q?=EB=B0=B4=EB=93=9C=EB=A1=9C=20=EC=B9=A8=EB=AC=B5=20=EC=9D=80?= =?UTF-8?q?=ED=8F=90=EB=90=98=EB=8D=98=20=EA=B2=B0=ED=95=A8=20=EA=B7=BC?= =?UTF-8?q?=EB=B3=B8=EC=88=98=EC=A0=95=20=E2=80=94=20FastForest=20?= =?UTF-8?q?=EA=B5=90=EC=B2=B4=20(macOS=20CI=20un-red)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - aux σ(x)를 SDCA→FastForest로 교체: SDCA는 플랫폼/스레드 궤적에 따라 발산 (win-x64 단일스레드 σ≈-8.7e7 실측, macos-arm64 CI 동일 기전)하고 max(σ,0)+β 플로어가 발산을 상수폭 밴드로 은폐. 포리스트는 σ가 학습 잔차 범위로 구조적 유계(비음수·발산 불가), NumberOfThreads=1로 보정 재현성 확보 - PredictionService σ 스코어링 폴백에 warning 추가 (P-svc1 반침묵강등 교훈) - 두 밴드 테스트 실패 메시지에 aux σ 직접 판독 진단 추가 (fit vs serving 구분) - multiclass numeric-label 테스트 LightGbm→SdcaMaximumEntropy: macos-latest는 Apple silicon이고 lib_lightgbm osx-arm64 네이티브 부재, 핀 대상 회귀(serve 라벨 로딩)는 트레이너 무관 --- CHANGELOG.md | 3 +++ src/MLoop.Core/AutoML/AutoMLRunner.cs | 16 ++++++++++++++-- .../Prediction/PredictionService.cs | 8 +++++++- .../Prediction/PredictionServiceTests.cs | 19 +++++++++++++++++-- .../ML/CrossPathConformalBandTests.cs | 11 ++++++++++- 5 files changed, 51 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6609d85..f23b602 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Fixed +- **Normalized-conformal σ-model can no longer silently diverge into a constant-width band (also un-reds macOS CI)**: the heteroscedastic aux σ(x) model was a linear SDCA fit, and SDCA's async optimization trajectory is platform-/thread-sensitive on this small aux problem — it can outright *diverge* (observed: σ(x) ≈ −8.7×10⁷ on win-x64 single-threaded, and the same degeneracy on macos-arm64 CI's default fit, while win/linux multi-threaded happened to converge). Because the band formula floors σ at `max(σ,0)+β`, a diverged σ-model doesn't crash — every row silently gets the identical constant width while the experiment metadata still claims a heteroscedastic band, which is exactly why macos-latest CI had been red since 0.19.0 on the two per-row-band tests without anyone seeing a *wrong prediction*. The σ-model is now a FastForest (managed, no native deps): a forest predicts leaf averages of the |residual| target, so σ is structurally bounded to the training residual range — it cannot go negative or diverge — and it trains with `NumberOfThreads = 1` so conformal calibration is reproducible for the same data (as a bonus the aux fit dropped from ~90 s of grinding divergence to ~1 s in the affected tests). Alongside: `PredictionService`'s σ-scoring fallback now surfaces a warning naming the constant-width fallback instead of degrading silently (P-svc1's lesson), both band tests read the aux σ directly in their failure messages so a future regression reports whether the *fit* or the *serving path* degenerated, and `Predict_Multiclass_NumericLabel_MatchesTrainedSingleSchema` swaps its incidental LightGBM trainer for SdcaMaximumEntropy — GitHub's macos-latest runners are Apple silicon and lib_lightgbm ships no osx-arm64 native, which was the third red test (`DllNotFoundException`), and the pinned regression (serve-path label loading) is trainer-agnostic. + ### Changed - **Object-detection training now forces libtorch to a single thread (D27 defensive mitigation, unverified)**: real-data OD training intermittently dies with a native access violation (0xC0000005) inside libtorch, with the crash location varying between runs (`THSNN_Linear_forward`, `Tensor.backward`) — a signature upstream (dotnet/TorchSharp#1292) traces to native heap corruption whose failure point shifts with memory/thread pressure. `RunObjectDetectionAsync` now calls `TorchSharp.torch.set_num_threads(1)` before fitting, removing that pressure source. This is a defensive, no-downside change (the only cost is slower CPU training) applied unconditionally rather than gated on a confirmed fix — this investigation's development environment had ambient resource contention from concurrent unrelated workloads that prevented reliably re-triggering or clearing the crash to verify the mitigation directly (see the D27 issue for the full investigation). Full elastic thread-count tuning or an upstream TorchSharp fix remain open; this is a stopgap. diff --git a/src/MLoop.Core/AutoML/AutoMLRunner.cs b/src/MLoop.Core/AutoML/AutoMLRunner.cs index 01d1d17..317cf17 100644 --- a/src/MLoop.Core/AutoML/AutoMLRunner.cs +++ b/src/MLoop.Core/AutoML/AutoMLRunner.cs @@ -663,10 +663,22 @@ public static Dictionary ComputeConformalIntervals( if (c1Residuals.Count < 5 || GetRowCount(ml, c2) < 5) return null; // too small to fit + calibrate meaningfully - // σ-model: raw numeric features → |residual|. Trained only on C1. + // σ-model: raw numeric features → |residual|. Trained only on C1. FastForest, not SDCA: + // SDCA's fit is platform-/thread-trajectory-sensitive and can silently DIVERGE on this tiny + // aux problem (observed: σ ≈ -8.7e7 single-threaded on win-x64, and the macOS-arm64 CI's + // multi-threaded fit degenerated the same way), and the max(σ,0)+β floor then masks the + // divergence as a constant-width band while the metadata still claims heteroscedastic. + // A forest predicts leaf averages of the |residual| target, so σ is structurally bounded to + // the training residual range (never negative, cannot diverge); NumberOfThreads = 1 keeps + // the conformal calibration reproducible for the same data. var auxPipeline = ml.Transforms.Concatenate("Features", feats) .Append(ml.Transforms.NormalizeMinMax("Features")) - .Append(ml.Regression.Trainers.Sdca(labelColumnName: "Target", featureColumnName: "Features")); + .Append(ml.Regression.Trainers.FastForest(new Microsoft.ML.Trainers.FastTree.FastForestRegressionTrainer.Options + { + LabelColumnName = "Target", + FeatureColumnName = "Features", + NumberOfThreads = 1, + })); var auxModel = auxPipeline.Fit(c1); // Positive floor β: a low quantile of the C1 residuals, so σ never collapses the band. diff --git a/src/MLoop.Core/Prediction/PredictionService.cs b/src/MLoop.Core/Prediction/PredictionService.cs index 7cef154..f54ef84 100644 --- a/src/MLoop.Core/Prediction/PredictionService.cs +++ b/src/MLoop.Core/Prediction/PredictionService.cs @@ -148,7 +148,13 @@ public PredictionResult Predict( if (residualModel != null && interval?.IsHeteroscedastic == true) { try { perRowSigma = ComputeResidualSigma(residualModel, predictions); } - catch { perRowSigma = null; } + catch (Exception ex) + { + // Graceful degradation, but not silent (P-svc1 lesson): the caller should know the + // band fell back to constant width instead of assuming per-row σ was applied. + perRowSigma = null; + warnings.Add($"Residual σ-model scoring failed; using constant-width interval instead: {ex.Message}"); + } } predictions = RestoreOriginalLabels(predictions); diff --git a/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs b/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs index 75a1568..91dbb19 100644 --- a/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs +++ b/tests/MLoop.Core.Tests/Prediction/PredictionServiceTests.cs @@ -218,7 +218,19 @@ public void Predict_Regression_Heteroscedastic_ProducesPerRowBandWidths() Assert.Equal(2, result.Rows.Count); double w1 = result.Rows[0].ScoreUpperBound!.Value - result.Rows[0].ScoreLowerBound!.Value; double w2 = result.Rows[1].ScoreUpperBound!.Value - result.Rows[1].ScoreLowerBound!.Value; - Assert.True(Math.Abs(w1 - w2) > 1e-3, $"per-row band widths should differ (w1={w1:F3}, w2={w2:F3})"); + + // Diagnostic readout of the aux σ-model itself: on a failure this tells apart "the σ-model fit + // degenerated to a constant" from "the service path fell back to the constant-width band" + // (the macOS-arm64 CI failure class — see ISSUE-mloop-20260705-macos-predictionservice-test-failures). + var probe = ml.Data.LoadFromEnumerable(new[] + { + new SimpleRegression { X = 1.0f }, + new SimpleRegression { X = 20.0f }, + }); + var probeSigmas = norm.AuxModel.Transform(mainModel.Transform(probe)).GetColumn("Score").ToArray(); + + Assert.True(Math.Abs(w1 - w2) > 1e-3, + $"per-row band widths should differ (w1={w1:F3}, w2={w2:F3}; aux σ(X=1)={probeSigmas[0]:F4}, σ(X=20)={probeSigmas[1]:F4})"); } [Fact] @@ -1209,7 +1221,10 @@ public void Predict_Multiclass_NumericLabel_MatchesTrainedSingleSchema() var keyed = _mlContext.Transforms.Conversion.MapValueToKey("Label", "Label").Fit(data).Transform(data); var featurized = _mlContext.Transforms.Concatenate("Features", "X1", "X2").Fit(keyed).Transform(keyed); - var trainer = _mlContext.MulticlassClassification.Trainers.LightGbm( + // SdcaMaximumEntropy, not LightGbm: the pinned regression is the serve path's label *loading* + // (String vs Single before the model's MapValueToKey), so any multiclass trainer reproduces it — + // and lib_lightgbm ships no osx-arm64 native, which made this test DllNotFound on macOS CI. + var trainer = _mlContext.MulticlassClassification.Trainers.SdcaMaximumEntropy( labelColumnName: "Label", featureColumnName: "Features"); var model = trainer.Fit(featurized); var modelPath = SaveModel(model, featurized.Schema); diff --git a/tests/MLoop.Tests/Infrastructure/ML/CrossPathConformalBandTests.cs b/tests/MLoop.Tests/Infrastructure/ML/CrossPathConformalBandTests.cs index 60ad74d..6a41104 100644 --- a/tests/MLoop.Tests/Infrastructure/ML/CrossPathConformalBandTests.cs +++ b/tests/MLoop.Tests/Infrastructure/ML/CrossPathConformalBandTests.cs @@ -1,4 +1,5 @@ using Microsoft.ML; +using Microsoft.ML.Data; using MLoop.CLI.Infrastructure.ML; using MLoop.Core.AutoML; using MLoop.Core.Models; @@ -132,7 +133,15 @@ await File.WriteAllTextAsync(inputCsv, // constant width — otherwise the guard would pass trivially. double w0 = csvBands[0].Upper - csvBands[0].Lower; double w1 = csvBands[1].Upper - csvBands[1].Lower; - Assert.True(Math.Abs(w0 - w1) > 1e-3, $"expected per-row widths to differ (w0={w0:F3}, w1={w1:F3})"); + + // Diagnostic readout of the aux σ-model itself: on a failure this tells apart "the σ-model fit + // degenerated to a constant" from "both paths fell back to the constant-width band" + // (the macOS-arm64 CI failure class — see ISSUE-mloop-20260705-macos-predictionservice-test-failures). + var probe = ml.Data.LoadFromEnumerable(xs.Select(x => new SimpleReg { X = x })); + var probeSigmas = norm.AuxModel.Transform(mainModel.Transform(probe)).GetColumn("Score").ToArray(); + + Assert.True(Math.Abs(w0 - w1) > 1e-3, + $"expected per-row widths to differ (w0={w0:F3}, w1={w1:F3}; aux σ(X=1)={probeSigmas[0]:F4}, σ(X=20)={probeSigmas[1]:F4})"); } private static List<(double Lower, double Upper)> ReadCsvBands(string csvPath) From 3eb5957d11b076cebe1f8137492a88a4463570bf Mon Sep 17 00:00:00 2001 From: uj Date: Tue, 7 Jul 2026 19:05:08 +0900 Subject: [PATCH 4/4] =?UTF-8?q?fix(api-tests):=20TestWebApplicationFactory?= =?UTF-8?q?=EC=9D=98=20=ED=94=84=EB=A1=9C=EC=84=B8=EC=8A=A4=20=EC=A0=84?= =?UTF-8?q?=EC=97=AD=20CWD=20=EB=B3=80=EC=A1=B0=20=EC=A0=9C=EA=B1=B0=20?= =?UTF-8?q?=E2=80=94=20=EA=B3=A0=EC=A0=95=20=EB=A3=A8=ED=8A=B8=20IProjectD?= =?UTF-8?q?iscovery=20=EC=8A=A4=ED=85=81=20=EC=A3=BC=EC=9E=85?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit xUnit은 테스트 클래스(=팩토리 인스턴스)를 병렬 실행하는데, 기존 팩토리는 Environment.CurrentDirectory(전역)를 임시 루트로 바꿔 프로젝트 발견을 우회했다. 단일 소비 클래스(ApiIntegrationTests)일 땐 잠복했으나 D21-A의 ForecastingApiTests가 두 번째 팩토리를 만들며 노출: 한 팩토리의 Dispose가 다른 팩토리 Program 기동의 작업 디렉터리를 삭제 → 'entry point exited without ever building an IHost'로 API 테스트 37건 전 OS 전멸. 고정 루트 스텁 주입으로 팩토리 간 격리. --- CHANGELOG.md | 2 + .../TestWebApplicationFactory.cs | 38 +++++++++++++------ 2 files changed, 29 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f23b602..dc817bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed - **Normalized-conformal σ-model can no longer silently diverge into a constant-width band (also un-reds macOS CI)**: the heteroscedastic aux σ(x) model was a linear SDCA fit, and SDCA's async optimization trajectory is platform-/thread-sensitive on this small aux problem — it can outright *diverge* (observed: σ(x) ≈ −8.7×10⁷ on win-x64 single-threaded, and the same degeneracy on macos-arm64 CI's default fit, while win/linux multi-threaded happened to converge). Because the band formula floors σ at `max(σ,0)+β`, a diverged σ-model doesn't crash — every row silently gets the identical constant width while the experiment metadata still claims a heteroscedastic band, which is exactly why macos-latest CI had been red since 0.19.0 on the two per-row-band tests without anyone seeing a *wrong prediction*. The σ-model is now a FastForest (managed, no native deps): a forest predicts leaf averages of the |residual| target, so σ is structurally bounded to the training residual range — it cannot go negative or diverge — and it trains with `NumberOfThreads = 1` so conformal calibration is reproducible for the same data (as a bonus the aux fit dropped from ~90 s of grinding divergence to ~1 s in the affected tests). Alongside: `PredictionService`'s σ-scoring fallback now surfaces a warning naming the constant-width fallback instead of degrading silently (P-svc1's lesson), both band tests read the aux σ directly in their failure messages so a future regression reports whether the *fit* or the *serving path* degenerated, and `Predict_Multiclass_NumericLabel_MatchesTrainedSingleSchema` swaps its incidental LightGBM trainer for SdcaMaximumEntropy — GitHub's macos-latest runners are Apple silicon and lib_lightgbm ships no osx-arm64 native, which was the third red test (`DllNotFoundException`), and the pinned regression (serve-path label loading) is trainer-agnostic. +- **API integration-test factory no longer mutates the process-global working directory**: `TestWebApplicationFactory` pinned the API's project discovery to its temp root by assigning `Environment.CurrentDirectory` — process-global state. That was latent while a single test class consumed the factory, but xUnit runs test classes in parallel, and D21-A's `ForecastingApiTests` added a second factory instance: one factory's dispose could delete the directory another factory's `Program` startup was using as its working directory, killing the entry point before it built an `IHost` ("The entry point exited without ever building an IHost", 37 API tests failing on every OS). The factory now injects a fixed-root `IProjectDiscovery` stub instead of touching the CWD, so parallel factories stay isolated. + ### Changed - **Object-detection training now forces libtorch to a single thread (D27 defensive mitigation, unverified)**: real-data OD training intermittently dies with a native access violation (0xC0000005) inside libtorch, with the crash location varying between runs (`THSNN_Linear_forward`, `Tensor.backward`) — a signature upstream (dotnet/TorchSharp#1292) traces to native heap corruption whose failure point shifts with memory/thread pressure. `RunObjectDetectionAsync` now calls `TorchSharp.torch.set_num_threads(1)` before fitting, removing that pressure source. This is a defensive, no-downside change (the only cost is slower CPU training) applied unconditionally rather than gated on a confirmed fix — this investigation's development environment had ambient resource contention from concurrent unrelated workloads that prevented reliably re-triggering or clearing the crash to verify the mitigation directly (see the D27 issue for the full investigation). Full elastic thread-count tuning or an upstream TorchSharp fix remain open; this is a stopgap. diff --git a/tests/MLoop.API.Tests/TestWebApplicationFactory.cs b/tests/MLoop.API.Tests/TestWebApplicationFactory.cs index f05afd7..e5c1106 100644 --- a/tests/MLoop.API.Tests/TestWebApplicationFactory.cs +++ b/tests/MLoop.API.Tests/TestWebApplicationFactory.cs @@ -48,17 +48,14 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) services.Remove(descriptor); } - // Add test-specific ProjectDiscovery that uses our temp directory - services.AddSingleton(sp => - { - var fileSystem = sp.GetRequiredService(); - var discovery = new ProjectDiscovery(fileSystem); - - // Override the current directory to our test project root - Environment.CurrentDirectory = _testProjectRoot; - - return discovery; - }); + // Add test-specific ProjectDiscovery pinned to our temp directory. Deliberately NOT via + // Environment.CurrentDirectory: that is process-global state, and xUnit runs test classes + // (= factory instances) in parallel — the old CWD mutation let one factory's Dispose + // delete the directory another factory's Program startup was using as its working + // directory, so the entry point died before ever building an IHost. Latent while + // ApiIntegrationTests was the only factory consumer; exposed when ForecastingApiTests + // added a second one (D21-A). + services.AddSingleton(new FixedRootProjectDiscovery(_testProjectRoot)); // Override Ops/DataStore services to use test directory ReplaceService(services, new FileModelComparer(_testProjectRoot)); @@ -111,6 +108,25 @@ protected override void Dispose(bool disposing) } } } + + /// An pinned to a known root — no process-global + /// CWD dependency, so parallel factories (one per test class) stay isolated. + private sealed class FixedRootProjectDiscovery : IProjectDiscovery + { + private readonly string _root; + + public FixedRootProjectDiscovery(string root) => _root = root; + + public string FindRoot() => _root; + + public string FindRoot(string startingDirectory) => _root; + + public bool IsProjectRoot(string path) => Directory.Exists(Path.Combine(path, ".mloop")); + + public void EnsureProjectRoot() { } + + public string GetMLoopDirectory(string projectRoot) => Path.Combine(projectRoot, ".mloop"); + } } ///