Skip to content

feat: UI support for federated preprocessing steps - #73

Merged
KFilippopolitis merged 3 commits into
masterfrom
feat/federated-preprocessing-ui
Nov 9, 2026
Merged

KFilippopolitis merged 3 commits into
masterfrom
feat/federated-preprocessing-ui

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements the platform-ui tasks from the federated preprocessing spec (Obsidian: platform-ui should do). The proposal step names are used as-is — variable_transformer / variable_scaler; a rename follow-up exists for when exaflow fixes the names.

  • Missing values: most_frequent imputation for nominal variables only (mean/median remain numerical-only); wired into the editor, the valid-action check, validation, and the run summary. No category list is sent — the vocabulary stays on the worker side.
  • Transform step (variable_transformer): new per-variable station — log1p, sqrt, asinh, Yeo-Johnson, quantile→uniform, quantile→normal — with a batch "set all" menu and spec documentation panel. Domain-limit notes for log1p/sqrt check the observed minimum from the loaded describe (the plan's CDE minValue is not exposed to the UI; noted as a deviation — exaflow still enforces the real limit with a 400 that the apply footer shows as it comes).
  • Scaling step (variable_scaler): new per-variable station — standard, robust, minmax (with feature_range inputs), maxabs.
  • Shared work: both new steps are filtered to the algorithm's input variables at request time and dropped entirely when no strategy survives filtering; emitted in rail order (impute → clip → transform → scale, drop default first — an assumption to confirm with exaflow, which has not defined an allowed order yet); summarized in experiment detail / PDF / CSV (the minmax range rides only on the min-max lines); restored on reopen; removable per station.
  • Deliberately not included: column-creating indicator/flag creators and fit-diagnostics/recipe-reuse UI — both wait on exaflow shipping those steps/results.

Catalog gate (merge safety)

The transform/scaling stations, their rail nodes and add rows are shown only while the catalog lists the step (optimistically before the catalog loads, like the creators), and the steps are dropped from every request once the catalog loads without them. Exaflow rejects unknown steps with Transformer '<name>' does not exist., so:

This PR is safe to merge before exaflow ships the steps. Until the runtime integration lands, the stations stay hidden and no request carries the steps.

Files changed

  • src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.ts / .html — two new stations, rail sub-nodes + ghost add-rows, batch menus, hydration of saved experiments, most_frequent, catalog gating
  • src/app/services/experiment-studio.service.ts — summary entries, strategy labels, variable filtering (with the feature_range-only drop), request-level catalog gate
  • experiment-studio.service.spec.ts, statistic-analysis-panel.pipeline.spec.ts, statistic-analysis-panel.component.spec.ts — new coverage

Tests / checks

  • npm test -- --watch=false — 1160/1160 passing. New tests cover request shape, variable filtering, step order, summary text, restore, the catalog gate (hidden stations + dropped request steps), the scaler filter's feature_range case, and the most_frequent nominal-only gate; the component spec was extended (134 tests there).
  • npm run verify (typecheck + dead-code scan + build) — passing

Risk

  • Step names/parameter shapes are proposals; a rename follow-up is tracked in the Obsidian note. With the gate, a mis-named step is hidden rather than sent.
  • The "impute → clip → transform → scale" order is the UI's choice; flagged for confirmation against exaflow's allowed order.
  • A histogram override carrying transforms but no missing-values step is forwarded without a drop default (matches existing outlier-only override behavior; pinned by test).

const source = step === 'transform' ? transformStrategies : scalerStrategies;
const codes = this.outlierPreprocessingVariables.map((v) => v.code);
return JSON.stringify({
strategies: Object.fromEntries(codes.sort().map((code) => [code, source[code] ?? null])),
KFilippopolitis added a commit that referenced this pull request Nov 9, 2026
Review follow-ups on #73:

- The transform and scaling stations, their rail nodes and ghost add rows
  are shown only while the catalog lists the step (optimistically before
  the catalog loads, like the creators), and the steps are dropped from
  every request once the catalog loads without them — exaflow rejects
  unknown steps with 'Transformer does not exist.'
- The scaler step is dropped from a filtered request when no strategy
  survives, instead of a lone step-level feature_range keeping it
- The minmax feature range now rides on the min-max summary lines only
- Component spec coverage: most_frequent nominal-only gate, station
  availability against the catalog, pending/applied strategy lifecycle
Implements the platform-ui tasks from the federated preprocessing spec
(step names are the proposal: variable_transformer / variable_scaler;
replace them if exaflow ships different ones):

- most_frequent imputation for nominal variables (editor, validation,
  summary label, valid-action check); mean/median stay numerical-only
- New per-variable transform station (log1p, sqrt, asinh, yeo_johnson,
  quantile_uniform, quantile_normal) with domain-limit notes for
  log1p/sqrt read off the loaded describe
- New per-variable scaling station (standard, robust, minmax with
  feature_range, maxabs)
- Rail sub-nodes, ghost add rows, batch menus, doc panels, hydration of
  saved experiments, and Remove for both new stations
- Request building: variable filtering for the new steps, rail order
  (impute, clip, transform, scale), summary entries feeding experiment
  detail and PDF/CSV export
- Specs: request shape, filtering, step order, summary text, restore
Review follow-ups on #73:

- The transform and scaling stations, their rail nodes and ghost add rows
  are shown only while the catalog lists the step (optimistically before
  the catalog loads, like the creators), and the steps are dropped from
  every request once the catalog loads without them — exaflow rejects
  unknown steps with 'Transformer does not exist.'
- The scaler step is dropped from a filtered request when no strategy
  survives, instead of a lone step-level feature_range keeping it
- The minmax feature range now rides on the min-max summary lines only
- Component spec coverage: most_frequent nominal-only gate, station
  availability against the catalog, pending/applied strategy lifecycle
Current exaflow fills each dataset with its own local mode; no CDE
vocabulary is used yet. Say so in the gate comment instead of 'the
vocabulary comes from the CDE on the worker side'.
@KFilippopolitis
KFilippopolitis force-pushed the feat/federated-preprocessing-ui branch from cb6ae56 to 34c5859 Compare November 9, 2026 17:49
@KFilippopolitis
KFilippopolitis merged commit 3188b7b into master Nov 9, 2026
3 checks passed
KFilippopolitis added a commit that referenced this pull request Nov 9, 2026
Review follow-ups on #73:

- The transform and scaling stations, their rail nodes and ghost add rows
  are shown only while the catalog lists the step (optimistically before
  the catalog loads, like the creators), and the steps are dropped from
  every request once the catalog loads without them — exaflow rejects
  unknown steps with 'Transformer does not exist.'
- The scaler step is dropped from a filtered request when no strategy
  survives, instead of a lone step-level feature_range keeping it
- The minmax feature range now rides on the min-max summary lines only
- Component spec coverage: most_frequent nominal-only gate, station
  availability against the catalog, pending/applied strategy lifecycle
@KFilippopolitis
KFilippopolitis deleted the feat/federated-preprocessing-ui branch November 11, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants