Skip to content

refactor(data-model): Classify variables by the single CDE type - #57

Closed
KFilippopolitis wants to merge 231 commits into
masterfrom
refactor/cde-variable-types
Closed

KFilippopolitis wants to merge 231 commits into
masterfrom
refactor/cde-variable-types

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The MIP variable-type refactor removes sql_type and isCategorical from CDEs; type is one of continuous, discrete (numerical) or nominal, ordinal, binary (categorical).

  • core/data-model.utils.ts gains shared helpers: isNumericalVariableType, isCategoricalVariableType, and specificationAcceptsVariableType, which maps CDE types onto algorithm input types (continuous → real, discrete → int and real slots, categorical → text), matching Exaflow's request validation.
  • Algorithm availability, role validation, outlier eligibility, filter operators and inputs, histogram splits, statistics grouping and the metadata browser use these helpers instead of real/integer/text checks.
  • Categorical CDEs without enumerations (former free-text variables) are no longer offered as filter fields; Exaflow rejects them as algorithm inputs.
  • Data model interfaces drop sql_type/isCategorical; spec and e2e mock fixtures use the new types.

Tests

  • npm run verify (typecheck, dead-code scan, build): passed
  • npm test -- --watch=false: 1034 of 1036 passed. notebook.component.spec.ts (untouched) errors with "Cannot connect to the iframe" only in the full run and passes 9/9 alone.
  • Not run: npm run e2e, manual QA.

Compatibility

Breaking; requires the platform-backend and Exaflow PRs below. Rollback: revert together with the related PRs.

Related PRs

Same branch name refactor/cde-variable-types in every repo; merge and deploy together.

KFilippopolitis and others added 30 commits March 3, 2026 15:24
* docs: remove jupyter references from README

* feat: Replace the generic distribution graph with a dedicated histogram component, update CSV and PDF export services, and add warnings for empty datasets in the statistic analysis panel.

* feat: Implement tabbed filtering for 'My Experiments' and 'Shared' experiments and exclude 'all datasets' for numerical variables in statistic analysis.

* feat: Display a message prompting variable selection when group histogram data is present but no variable is chosen.

* feat(experiment-studio): share and reposition data exclusion warnings

Lift data-exclusion warnings into shared ExperimentStudioService state so they can be shown across the studio flow.

Render the warning inline between Descriptive Statistics and Algorithm sections with a softer style, and keep dismiss support.

Update warning copy to state exclusions apply to descriptive statistics and any experiment execution, and adjust descriptive-panel spec mocks.

* fix(experiment-studio): improve empty dataset handling and CV error metrics

- Updated Linear Regression CV renderer to correctly parse error metric arrays [mean, std].

- Added `excludedDatasetsSignal` to ExperimentStudioService to ensure datasets excluded due to empty variables are omitted from algorithm execution requests.

- Refined Data Exclusion Warning message to better explain Federated Learning complete-cases requirements, clarifying that records with missing values are dropped during preprocessing.

- Modified the Descriptive Statistics panel to still show excluded datasets in the Variables/Model tables while displaying the exclusion warnings.

- Added a clarifying note to the Model tab's descriptive text explaining N/A value exclusion.

* feat(ui): optimize experiment studio layout for responsive scaling

- Set explicit 55/45 grid split for the variables panel and histogram
  to ensure stable scaling across viewport sizes.
- Re-anchored the sidebar to a fixed 16px left position to prevent inward
  drifting and overlap over main content on ultra-wide screens.
- Updated the sidebar collapse breakpoint from 1440px to 1650px.
- Reduced global page padding to 16px to maximize horizontal canvas space.
- Added dynamic `.studio-main.sidebar-collapsed` padding adjustments
  so the main workspace aggressively reclaims horizontal space when the
  sidebar shrinks, instead of leaving empty padding.
- Added `min-width: 0` constraints to grid children to prevent responsive
  blowouts.

* feat: Introduce regression line charts for single-predictor linear and logistic models, update chart registry, and improve algorithm input type display.
Bumps the npm_and_yarn group with 1 update in the / directory: [jspdf](https://github.com/parallax/jsPDF).


Updates `jspdf` from 3.0.4 to 4.2.0
- [Release notes](https://github.com/parallax/jsPDF/releases)
- [Changelog](https://github.com/parallax/jsPDF/blob/master/RELEASE.md)
- [Commits](parallax/jsPDF@v3.0.4...v4.2.0)

---
updated-dependencies:
- dependency-name: jspdf
  dependency-version: 4.2.0
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ion (#6)

* fix(terms): resolve markdown rendering and Proceed button race condition

* fix(ui): avoid double proceed click on terms acceptance

* feat: Expect the MIP version from env variables.

* feat: centralize runtime environment variable access, including version information, into a new `RuntimeEnvService`
* feat: add guided on-boarding to the experiment workspace

* style: refresh platform branding and shared pages
…ated CHUV logo (#10)

* feat: update footer with interactive funding acknowledgements and updated CHUV logo

* feat: reorganize footer assets into dedicated directory and update component references
…d keys and adding robust preprocessing state management

- Rename payload keys in descriptive pipeline

- Add state management for pending and applied preprocessing

- Refine progress workflow UI in statistics analysis panel
…sources, multi-select parameters, and cross-validation state preservation

- Add dynamic enum sources and multi-select parameters support to config schema

- Implement cross-validation state preservation logic

- Add mapping utilities for algorithm results and enum values

- Enhance algorithm panel UI for complex configurations

- Expand chart and table renderers for advanced metrics
- Force ThemeService to always return 'light' theme

- Remove dynamic theme class toggling from document body

- Delete dark mode CSS variables and media queries from global styles

- Remove dark mode overrides from header component
…eview panel

- Replace third-party query builder with a custom recursive filter logic implementation

- Migrate filter management from variables panel to statistic analysis panel

- Simplify statistics analysis panel by removing accordion layouts

- Enhance experiment studio UI with flexible spinner and refined progress labels
- Enhance algorithm configuration schema to support dynamic enum sources, multi-select parameters, and cross-validation state preservation

- Add robust preprocessing state management and workflow visibility in algorithm panel

- Add custom visualisations and formatting for algorithm results in charts and tables

- Hardcode light theme globally and remove dark mode support
…hydration logic

- Ensure identical catalog models don't trigger unnecessary re-emits in data-model selector

- Rewrite dataset selector logic to properly handle auto-selection and hydration from inputs

- Add unit tests for both selector components
…st parameter normalization

- Expose readable preprocessing configuration formatting from studio service

- Map and display preprocessing steps and filter logic in dashboard experiment detail

- Add parameter type coercion and persistence to sessionStorage for algorithm state

- Fix backend filter typing in models
…ton loaders

- Introduce skeleton loaders for panel data loading states

- Implement unified panel header system with improved typography

- Add muted interaction states for unready studio rails and sections

- Add advanced configuration toggle to algorithm panel

- Apply modern aesthetic enhancements including dynamic shadows and refined scrollbars

- Update global fonts and animations
Add ontology tree and collapsible tree browser components as
alternative ways to browse the data model hierarchy alongside
the existing bubble chart.

- OntologyTreeBrowserComponent: flat indented tree with expand/collapse
- CollapsibleTreeBrowserComponent: D3 collapsible force layout
- Shared model (MetadataBrowserMode) and normalizer utility
- Unit tests for normalizer and both components
Extract variable/group metadata display into a focused component
used by the new Info tab in the variables panel.

- Shows variable type badge, hierarchy path, field grid
- Shows group summary (child groups, direct/total variable counts)
- Displays methodology and enumeration values
- Unit tests for computed fields and edge cases
Make the colorblind-friendly palette the default for the bubble chart
and CSS custom properties. Remove the color settings dropdown, the
DEFAULT_PALETTE constant, localStorage persistence, and the
body.colorblind-mode CSS class override.

- Variable: #ffba08 (golden yellow)
- Covariate: #bba66f (tan/beige)
- Filter: #483300 (dark brown)
- Remove FormsModule import (no longer needed)
- Update zoomable-circle-packing default colors to match
Wire up the ontology tree, collapsible tree, and bubble chart as
switchable metadata browser modes (persisted to localStorage).
Add Histogram / Info tab switcher to the right-side details panel.

- MetadataBrowserMode switcher replaces the fixed bubble chart
- Histogram tab: restructured layout with actions bar, bin selector,
  variant selector, and export buttons
- Info tab: delegates to MetadataInfoPanelComponent
- Reset to histogram tab on node selection and model change
- Polish: unified tab/header styling, responsive max-height,
  flex-fill for right column alignment, empty-state text scaling
- Tests for browser mode persistence and tab reset behavior
Replace 'Export' label with 'PDF' on all summary panel export
buttons to match the histogram panel's CSV/PDF naming convention.
Update corresponding spec assertion.
Set overflow to hidden on both axes (was scroll-y) so the chart
scales within its container instead of scrolling. Tweak padding
to 8px 10px 16px for better bottom spacing.

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contract review (CDEs JSON consumer)

core/data-model.utils.ts matches Exaflow: continuous→real, discrete→int (also accepted by real slots), nominal|ordinal|binary→text. Interfaces dropped sql_type/isCategorical. Filters correctly hide free-text categoricals (normalizeFilterVariables requires enumerations). Outlier eligibility is numerical-only.

Verdict: with fixes.

Must fix

1. Free-text categoricals still go to Exaflow as algorithm inputs

Nominal/ordinal without enumerations are valid CDEs (former text columns such as subjectid / notes). Filters drop them; describe, histogram splits, and algorithm roles do not. Exaflow rejects is_categorical and not enumerations (analysis_request_validator.py). Selecting one of those CDEs puts it in selectedVariables → preprocessingVariables → describe y, and isNominalHistogramNode is true for any categorical type.

Do not necessarily hide them from the metadata tree. Drop categorical-without-enumerations from describe y, histogram splits, and algorithmAssignableVariables / evaluateRole (same rule as normalizeFilterVariables).

export function isAnalysableCde(variable: { type?: unknown; enumerations?: unknown[] | null }): boolean {
  if (isNumericalVariableType(variable?.type)) return true;
  return isCategoricalVariableType(variable?.type)
    && Array.isArray(variable.enumerations)
    && variable.enumerations.length > 0;
}

2. E2E algorithm types still use CDE vocabulary nominal instead of spec text

specificationAcceptsVariableType maps a nominal CDE to text and checks membership in spec types. e2e/experiment-studio-api.ts still has types: ['nominal'] and ['real', 'nominal'] (linear_regression x, logistic, describe, histogram, …). After the mapper, Sex does not match those specs. Real Exaflow specs use InputDataType.TEXT. Keep nominal on CDE type and on spec stattypes only.

const yNominal = inputField(['text'], { required: true, max_count: 1 });
const xRealOrNominal = inputField(['real', 'text'], { required: true, min_count: 1, max_count: 1 });
const xNominal = inputField(['text'], { required: true, min_count: 1, max_count: 1 });

@KFilippopolitis

Copy link
Copy Markdown
Contributor Author

Both items addressed in a328897.

1. Free-text categoricals. Added isFreeTextCde (categorical type and no enumerations) in core/data-model.utils.ts, applied to:

  • algorithmAssignableVariables: free-text CDEs are no longer assignable.
  • evaluateRole: flags a free-text CDE (e.g. hydrated from a saved experiment) as invalid.
  • buildDescriptiveRequestBody: drops free-text codes from describe y; covers every describe caller.
  • buildHistogramSplitOptions: split candidates must have enumerations.
  • Variables panel: clicking a free-text CDE shows "no histogram available" instead of sending a histogram request Exaflow would reject.

Deviations from the suggested isAnalysableCde:

  • Used the narrower negation, so untyped nodes are unaffected.
  • Synthetic derived-column nodes (nominal, enumerations: [], isCreatedColumn) are exempt in evaluateRole; their categories come from the creator config.
  • isNominalHistogramNode is unchanged, otherwise a free-text CDE would be plotted as a numeric histogram.

2. E2E spec types. e2e/experiment-studio-api.ts input types now use text; nominal stays on CDE type only.

Unit fixtures that used categoricals without enumerations were given enumerations. npm run build, 1038 unit tests and 42 e2e tests pass.

KFilippopolitis and others added 19 commits October 27, 2026 14:02
… mip token

cli-token.ts runs an OAuth 2.0 authorization-code sign-in with PKCE as the
public Keycloak client named by GET /services/auth-config, asking for
offline_access. It remembers state and verifier in sessionStorage for the
return, checks the state, trades the one-time code for the tokens, and hands
back the refresh token. Nothing stores it. The account page uses it in the
next commit.
The "Use MIP from your own computer" guide now shows only when
/services/auth-config names the federation's public Keycloak client, and no
longer sends users to the JupyterHub token page. "Create access token" runs
the sign-in from cli-token.ts, takes the one-time code out of the address
bar, and shows the token once for `mip login --token` or MIP_API_TOKEN;
nothing stores it.

The setup snippet only installs the client and sets MIP_URL; the two ways to
sign in (access token, `mip login`) sit side by side in cards, and the guide
says a sign-in lasts at most 6 months, ends after 30 days without use, and
links to the Keycloak account page where it can be ended everywhere.
The PCA card opened on "From a PCA result" and, with no finished compatible
PCA experiment, only offered a dead end ("Run PCA first"). It now follows the
K-means card: when no compatible run exists it runs a transient PCA report
on the chosen numerical variables (current datasets and filters, nothing
saved), shows the components with their explained variance, and stores the
report's reusable_preprocessing as the pca_column_creator recipe. Reusing a
finished compatible run and "Fit on selected variables" still work.

- ExperimentStudioService: loadPcaReport, sharing the transient report path
  with loadKMeansReport.
- PcaColumnSourceComponent: in-place report flow (variable picks, run, stale
  report invalidation when inputs change, backend error display).
- Statistic analysis panel: how-to strip for the PCA card.
- Show a refused categorical edit on its card ("Column in use") instead of
  silently keeping the old rules when a later step reads the column.
- Type PCA output columns as numerical when the preprocessing spec has not
  loaded or failed to load.
- Ignore a PCA result response for an experiment the user moved away from.
- Generate step ids with crypto.getRandomValues so plain-HTTP origins work.
- Share the K-means/PCA fit-mode variable pool and column-code pattern.
- Clear all transformation steps in one write.
- Drop backwards compatibility: remove the name-keyed creator setters, the
  legacy categorical hydration path, the eigenvalue-only PCA table and the
  PCA label fallback for results without variables.
…ures (#60)

T-test key figures show the run's CI level (1 - alpha), the one-sided bound for less/greater runs, and what the mean difference subtracts (group A - group B; paired: x - y). The chart and table label the CI without a level, and the table trim uses the same run context as the figures.
Bumps the github-actions group with 3 updates: [actions/setup-python](https://github.com/actions/setup-python), [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) and [docker/login-action](https://github.com/docker/login-action).


Updates `actions/setup-python` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@5fda3b9...6d4fd22)

Updates `github/codeql-action/upload-sarif` from 4.39.0 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@6f2ac81...9b32540)

Updates `docker/login-action` from 4.6.0 to 4.7.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@dbcb813...81c6f93)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* Add staging image publish + bump workflow

Builds and pushes platform-ui to Docker Hub on push to staging-a/staging-b
and reuses the mip bump workflow to deploy into the matching federation.
The MIP variable-type refactor removes sql_type and isCategorical from
CDEs; `type` is one of continuous, discrete (numerical) or nominal,
ordinal, binary (categorical).

- core/data-model.utils gains the shared type helpers:
  isNumericalVariableType, isCategoricalVariableType and
  specificationAcceptsVariableType, which maps CDE types onto algorithm
  input `types` (continuous -> real, discrete -> int and real slots,
  categorical -> text).
- Algorithm availability, role validation, outlier eligibility, filter
  operators and inputs, histogram splits, statistics grouping and the
  metadata browser use these helpers instead of real/integer/text checks.
- Categorical CDEs without enumerations (former free-text variables) are
  not offered as filter fields.
- Data model interfaces drop sql_type/isCategorical; spec and e2e
  fixtures use the new types.

Compatibility: breaking; requires the backend and Exaflow from the same
refactor branch.
A categorical CDE without enumerations holds free text, and Exaflow's
request validator rejects it as an algorithm input. Drop such CDEs from
the algorithm role pool, describe y, histogram splits and the variables
panel histogram, and flag them in role availability. Derived columns are
exempt: their categories come from the creator config.

E2E algorithm specs now declare spec type text instead of the CDE type
nominal, matching Exaflow's InputDataType.
@KFilippopolitis
KFilippopolitis force-pushed the refactor/cde-variable-types branch from a328897 to 3d87f47 Compare November 3, 2026 16:08
@KFilippopolitis
KFilippopolitis force-pushed the master branch 4 times, most recently from 24aec72 to d790dd4 Compare November 5, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants