Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6960605
contracts: 2026-11-11_summary-preview-triage-ux
KFilippopolitis Nov 11, 2026
be2c981
task(01-step-picker-catalog): Add-step menu model: icons and purpose …
KFilippopolitis Nov 11, 2026
f803bac
task(02-step-picker-component): `app-step-picker`: the add row and it…
KFilippopolitis Nov 11, 2026
b7e5860
task(03-rail-add-step-menu): Collapsed stage rails: one add row with …
KFilippopolitis Nov 11, 2026
f80695f
task(03-list-quantities): List view: honest counts, size bars and poo…
KFilippopolitis Nov 11, 2026
4ac5a69
task(01-bubble-label-fit): Bubble map: labels that fit their circle a…
KFilippopolitis Nov 11, 2026
c2e1d0d
task(02-bubble-focus-trail): Bubble map: a location trail to see and …
KFilippopolitis Nov 11, 2026
5e3dcb4
task(04-tree-readable-view): Graph view: open readable, stay readable…
KFilippopolitis Nov 11, 2026
0ebc077
task(05-exploration-layout): Data exploration layout: named views, re…
KFilippopolitis Nov 11, 2026
17ec1b6
task(07-list-subtree-variables): List view: reach nested variables wi…
KFilippopolitis Nov 11, 2026
38481a6
task(06-exploration-columns): Data exploration columns: pool first, l…
KFilippopolitis Nov 11, 2026
1a8ecdc
task(04-stage-add-step-menu): Opened stages: stations only, plus one …
KFilippopolitis Nov 11, 2026
da20b60
task(01-summary-rail-utils): Pure helpers for the data summary rail a…
KFilippopolitis Nov 11, 2026
d27d7d8
task(02-summary-rail-triage): Variable rail of the data summary: coun…
KFilippopolitis Nov 11, 2026
68e5cbc
task(03-summary-table-columns): Statistics table: no duplicate "All d…
KFilippopolitis Nov 11, 2026
68f89bf
contracts: 2026-11-11_variable-browser-views-polish
KFilippopolitis Nov 11, 2026
4e4146b
fix: dead-code check ignores exports used only by spec helpers (*.tes…
KFilippopolitis Nov 11, 2026
33ddd90
test: e2e helper finds the study chip, not the bubble map's location …
KFilippopolitis Nov 11, 2026
915b8c1
test: fix spec mocks and align copy with merged content
KFilippopolitis Nov 11, 2026
93ae926
feat: schema-driven stations for every data transformation step in th…
KFilippopolitis Nov 11, 2026
a59a7ff
fix: match the engine's type rules and expose n_quantiles for quantil…
KFilippopolitis Nov 11, 2026
9d6ffe6
contracts: 2026-11-11_add-step-picker-ux
KFilippopolitis Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/context/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ flowchart TD
- The app uses Angular Signals heavily for feature state.
- `ExperimentStudioService` is the main orchestration layer for selected data models, datasets, variables, covariates, filters, algorithm configuration, run/save flows, preview calls, edit hydration, and sessionStorage persistence.
- Transformation (derived-column) state is an ordered `appliedTransformations: TransformationStep[]` signal on that service — not a keyed map. Singletons (`missing_values_handler`, outlier, longitudinal) stay in the preprocessing config; column creators (`categorical_column_creator`, `kmeans_cluster_creator`, `pca_column_creator`) live only in the ordered list and emit in list order after the singletons. Creator output codes come from `new_column_names` (never a legacy `code` field). There is no `pca_with_transformation` algorithm; PCA component scores are created with `pca_column_creator`.
- Pipeline stages follow what a step does to the data. **Preprocessing** edits a variable in place and keeps its meaning and range: `missing_values_handler`, `outlier_winsorizer`, `global_imputer`, `global_winsorizer`. **Data transformation** creates a column or changes what a variable means (range, scale, type, row layout): `kmeans_cluster_creator`, `pca_column_creator`, `categorical_column_creator`, `variable_transformer`, `variable_scaler`, `longitudinal_transformer`, `ordinal_encoder`, `one_hot_encoder`, `missing_indicator`, `outlier_detector` (default `flag` adds a column; `drop` and `set_missing` edit in place, noted on the station). `stepStage` in `core/data-transformation-step-form.utils.ts` is the single classification; a step the engine adds later lands in Data transformation until listed there. The endpoint (`/specifications/preprocessing`), the step machine ids and the request key `preprocessing` are the engine contract and keep their names.
- Catalog steps with no dedicated station (everything in `/specifications/preprocessing` except the missing-values, outlier winsorizer, longitudinal, transformer and scaler singletons and the three column creators) render through `generic-data-transformation-step`, a form built from the step's parameter schema by `core/data-transformation-step-form.utils.ts` (field kinds, payload building, trimming to the selection); the panel mounts each one in the stage `stepStage` names, so the component serves both families. They are singletons in the applied config like the others; the panel carries them over whenever another card applies, the service drops them from a request once the loaded catalog lacks them, and a missing indicator is ordered ahead of the missing-values step. Columns they add (`missing_indicator`, `outlier_detector` flag, `one_hot_encoder`) and variables they drop are derived from their parameters by `core/data-transformation-step-columns.utils.ts` and feed the algorithm role pool.
- `ExperimentsDashboardService` owns paged experiment list state, filtering, sharing, renaming, deletion, and result fetches.
- `ErrorService` shares global feature errors through a `BehaviorSubject`.

Expand Down
4 changes: 2 additions & 2 deletions docs/context/module-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ Notes: Stored historical experiment payloads may depend on legacy algorithm keys

## `src/app/pages/experiment-studio`
Purpose: Experiment composition workflow.
Key files: `experiment-studio.component.*`, `variables-panel/*`, `statistic-analysis-panel/*` (including `pca-column-source` / `kmeans-cluster-source`), `algorithm-panel/*`, `execution-panel/*` (result view + docked setup summary), `stepper/*`, `shared/*`, `guide/*`.
Key files: `experiment-studio.component.*`, `variables-panel/*`, `statistic-analysis-panel/*` (including `pca-column-source` / `kmeans-cluster-source` / `generic-data-transformation-step`), `algorithm-panel/*`, `execution-panel/*` (result view + docked setup summary), `stepper/*`, `shared/*`, `guide/*`.
Used by: `/experiment-studio` route and dashboard edit flows.
Rules: Keep page-specific UI here; use `ExperimentStudioService` for shared selection/run state. Transformation cards append to `appliedTransformations` in creation order; each creator step names outputs with `new_column_names`. Do not reintroduce `pca_with_transformation`.
Tests: Component specs live near components; e2e covers PCA then K-means then logistic regression payload order in `e2e/experiment-studio.flows.spec.ts`.
Notes: Browser validation often requires a backend with data models, algorithms, and authenticated user state. PCA/K-means creators need the aggregation server in the preprocessing catalog.
Notes: Browser validation often requires a backend with data models, algorithms, and authenticated user state. PCA/K-means creators need the aggregation server in the step catalog (`/specifications/preprocessing`).

## `src/app/pages/experiment-studio/visualisations`
Purpose: Chart/table rendering for algorithm outputs, histograms, metadata browsers, and auto-rendered results.
Expand Down
3 changes: 2 additions & 1 deletion e2e/experiment-studio.flows.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
applyLongitudinalStrategy,
applyMeanImputationToAll,
applyOutlierChoice,
pickStep,
assignRole,
clearRole,
collapsePreprocessing,
Expand Down Expand Up @@ -249,7 +250,7 @@ test.describe('preprocessing', () => {
test('batch actions set every numerical variable', async ({ page, api }) => {
await openAgeAndSex(page);
await applyMeanImputationToAll(page);
await page.locator('button:visible').filter({ hasText: /Add outlier/ }).first().click();
await pickStep(page, 'preprocessing', 'outlier_winsorizer');
const outlier = page.locator('app-station-card', { hasText: 'Outlier Handling' });
await outlier.getByRole('button', { name: 'Batch' }).click();
await page.getByRole('menuitem', { name: 'Enable for all numerical' }).click();
Expand Down
46 changes: 33 additions & 13 deletions e2e/studio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ export async function openNewStudio(page: Page) {
await page.goto('/experiments-dashboard');
await page.getByRole('button', { name: 'New experiment' }).click();
await expect(page).toHaveURL(/\/experiment-studio\/?$/);
await expect(page.getByRole('button', { name: /Dementia/ })).toBeVisible();
// The study chip, not the bubble map's "Dementia" location crumb.
await expect(page.locator('.study-context-chip', { hasText: /Dementia/ })).toBeVisible();
}

export async function addVariable(page: Page, label: string) {
Expand Down Expand Up @@ -125,6 +126,14 @@ export async function collapsePreprocessing(page: Page) {
if (await open.isVisible()) await open.click();
}

/** Same for the Data transformation step, where the longitudinal comparison lives. */
export async function collapseTransformation(page: Page) {
const section = page.locator('[data-guide="analysis-transformation"]');
const open = section.locator('.pipeline-node-toggle[aria-expanded="true"]');
await expect(open.or(section.getByRole('list', { name: 'Data transformation steps' }))).toBeVisible();
if (await open.isVisible()) await open.click();
}

export async function openMissingValues(page: Page) {
await page.getByRole('button', { name: 'Customize rules' }).click();
const missing = page.locator('app-station-card', { hasText: 'Missing Values' });
Expand All @@ -142,10 +151,21 @@ export async function applyMeanImputationToAll(page: Page) {
await expect(missing).toContainText(/Mean imputation|Applied/);
}

/**
* Pick a step from a stage's add-step menu: the rail's while the stage is collapsed, the stage's
* own once it is open. `step` is the engine's step name, e.g. `outlier_winsorizer`.
*/
export async function pickStep(page: Page, stage: 'preprocessing' | 'transformation', step: string) {
const picker = page.locator(`[data-guide="analysis-${stage}"] app-step-picker:visible`).first();
const trigger = picker.locator('.step-picker-trigger');
if ((await trigger.getAttribute('aria-expanded')) !== 'true') await trigger.click();
await picker.locator(`.step-picker-option[data-step="${step}"]`).click();
}

/** Enable outlier capping for every numerical variable, optionally picking a method and tail. */
export async function applyOutlierChoice(page: Page, method?: string, clip?: string) {
await page.getByRole('button', { name: 'Customize rules' }).click();
await page.locator('button:visible').filter({ hasText: /Add outlier/ }).first().click();
await pickStep(page, 'preprocessing', 'outlier_winsorizer');
const outlier = page.locator('app-station-card', { hasText: 'Outlier Handling' });
await outlier.getByRole('button', { name: 'Batch' }).click();
await page.getByRole('menuitem', { name: 'Enable for all numerical' }).click();
Expand All @@ -172,12 +192,13 @@ export async function openLongitudinalAge(page: Page) {
}

export async function applyLongitudinalStrategy(page: Page, label: string) {
await page.getByRole('button', { name: 'Customize rules' }).click();
// The longitudinal comparison changes what the variable means, so it is a data transformation.
await pickStep(page, 'transformation', 'longitudinal_transformer');
const card = page.locator('app-station-card', { hasText: 'Longitudinal Transformation' });
await card.getByRole('button', { name: /^Age/ }).click();
await card.locator('select').last().selectOption({ label });
await card.getByRole('button', { name: 'Apply', exact: true }).click();
await collapsePreprocessing(page);
await collapseTransformation(page);
await expect(page.getByText('Baseline vs Month 12')).toBeVisible();
}

Expand Down Expand Up @@ -206,8 +227,7 @@ export async function applyFilters(page: Page, { female = false } = {}) {
}

export async function addNominalColumn(page: Page, code: string, category: string, fallback: string) {
await page.getByTitle('Add and open a derived column').click();
await page.getByRole('button', { name: /Nominal column from rules/ }).click();
await pickStep(page, 'transformation', 'categorical_column_creator');
await page.getByPlaceholder('e.g. mrs_good_outcome').fill(code);
await page.getByRole('button', { name: /Add the first category/ }).click();
await page.getByPlaceholder('Enumeration value').fill(category);
Expand All @@ -220,10 +240,10 @@ export async function addNominalColumn(page: Page, code: string, category: strin
await expect(page.getByText('Sex is Female')).toBeVisible();
}

/** `stepAdded`: the transformation step is already open (another column was added first). */
/** `stepAdded` is kept for the callers: the add-step menu is the same whether the stage is open or not. */
export async function addKMeansColumn(page: Page, { stepAdded = false } = {}) {
if (!stepAdded) await page.getByTitle('Add and open a derived column').click();
await page.getByRole('button', { name: /K-means cluster column/ }).click();
void stepAdded;
await pickStep(page, 'transformation', 'kmeans_cluster_creator');
const source = page.locator('app-kmeans-cluster-source');
await source.getByRole('button', { name: 'Run K-means report' }).click();
await expect(source.getByText('Cluster 0')).toBeVisible();
Expand All @@ -238,8 +258,8 @@ export async function addKMeansColumn(page: Page, { stepAdded = false } = {}) {
* `stepAdded`: transformation stage already open with the type chooser visible.
*/
export async function addPcaColumns(page: Page, { stepAdded = false } = {}) {
if (!stepAdded) await page.getByTitle('Add and open a derived column').click();
await page.getByRole('button', { name: /PCA component columns/ }).click();
void stepAdded;
await pickStep(page, 'transformation', 'pca_column_creator');
const source = page.locator('app-pca-column-source');
await source.getByRole('button', { name: 'Fit on selected variables' }).click();
await source.getByRole('button', { name: 'Add component columns' }).click();
Expand All @@ -251,8 +271,8 @@ export async function addPcaColumns(page: Page, { stepAdded = false } = {}) {
* Deselects every other numerical option in the fit pool.
*/
export async function addKMeansOnColumns(page: Page, columns: string[], { stepAdded = false } = {}) {
if (!stepAdded) await page.getByTitle('Add and open a derived column').click();
await page.getByRole('button', { name: /K-means cluster column/ }).click();
void stepAdded;
await pickStep(page, 'transformation', 'kmeans_cluster_creator');
const source = page.locator('app-kmeans-cluster-source').last();
await source.getByRole('button', { name: 'Fit on selected variables' }).click();
const keep = new Set(columns);
Expand Down
3 changes: 2 additions & 1 deletion scripts/check-dead-code.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ const countIn = (haystack, name) => {
const DECL = /^\s*export\s+(?:declare\s+)?(?:abstract\s+)?(?:async\s+)?(const|let|var|function|class|interface|type|enum)\s+([A-Za-z0-9_$]+)/;
const deadExports = [];
const appSources = ts.filter((p) => !p.endsWith('.spec.ts'));
for (const p of appSources) {
// `*.testing.ts` files exist for specs, so an export only specs import is not dead.
for (const p of appSources.filter((q) => !q.endsWith('.testing.ts'))) {
const text = read(p);
for (const line of text.split('\n')) {
const m = DECL.exec(line);
Expand Down
40 changes: 40 additions & 0 deletions src/app/core/data-transformation-step-columns.utils.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { describe, expect, it } from 'vitest';
import { stepColumnEffect } from './data-transformation-step-columns.utils';

const sources = new Map([
['age', { code: 'age', label: 'Age' }],
['sex', { code: 'sex', label: 'Sex', enumerations: [{ code: 'M', label: 'Male' }, { code: 'F', label: 'Female' }, { code: 'X', label: 'Other' }] }],
]);

describe('stepColumnEffect', () => {
it('names indicator columns after the variable and the suffix, with the engine default', () => {
expect(stepColumnEffect('missing_indicator', { variables: ['age'] }, sources).created)
.toEqual([{ code: 'age_missing', label: 'Age (missing)' }]);
expect(stepColumnEffect('missing_indicator', { variables: ['age'], suffix: '_na' }, sources).created[0].code).toBe('age_na');
});

it('adds outlier flag columns only for the flag action', () => {
const strategies = { age: 'zscore' };
expect(stepColumnEffect('outlier_detector', { strategies }, sources).created).toEqual([{ code: 'age_outlier', label: 'Age (outlier)' }]);
expect(stepColumnEffect('outlier_detector', { strategies, action: 'flag', suffix: '_o' }, sources).created[0].code).toBe('age_o');
expect(stepColumnEffect('outlier_detector', { strategies, action: 'drop' }, sources).created).toEqual([]);
expect(stepColumnEffect('outlier_detector', { strategies, action: 'set_missing' }, sources).created).toEqual([]);
});

it('one-hot encodes sorted levels, dropping the reference level and the original by default', () => {
const effect = stepColumnEffect('one_hot_encoder', { variables: ['sex'] }, sources);
expect(effect.created.map((c) => c.code)).toEqual(['sex_M', 'sex_X']);
expect(effect.created[0].label).toBe('Sex: Male');
expect(effect.dropped).toEqual(['sex']);
});

it('keeps every level and the original when asked to', () => {
const effect = stepColumnEffect('one_hot_encoder', { variables: ['sex'], drop_first: false, drop_original: false }, sources);
expect(effect.created.map((c) => c.code)).toEqual(['sex_F', 'sex_M', 'sex_X']);
expect(effect.dropped).toEqual([]);
});

it('has no column effect for other steps', () => {
expect(stepColumnEffect('global_winsorizer', { variables: ['age'] }, sources)).toEqual({ created: [], dropped: [] });
});
});
78 changes: 78 additions & 0 deletions src/app/core/data-transformation-step-columns.utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/**
* Columns the schema-driven catalog steps add to, or remove from, the data. Their request
* parameters name them (`suffix`, the encoder's levels), so the studio can offer them as
* assignable variables and hide the ones a step drops.
*/

export interface CreatedStepColumn {
code: string;
label: string;
}

export interface StepColumnEffect {
created: CreatedStepColumn[];
/** Source variables the step removes from the data. */
dropped: string[];
}

interface SourceVariable {
code: string;
label?: string;
enumerations?: Array<{ code: string; label?: string }>;
}

const NO_EFFECT: StepColumnEffect = { created: [], dropped: [] };

function strings(value: unknown): string[] {
return Array.isArray(value) ? value.map((item) => String(item)) : [];
}

/** The columns a step adds and the variables it drops, from its applied parameters. */
export function stepColumnEffect(
name: string,
parameters: Record<string, unknown>,
sources: ReadonlyMap<string, SourceVariable>,
): StepColumnEffect {
const labelOf = (code: string) => sources.get(code)?.label || code;

if (name === 'missing_indicator') {
const suffix = typeof parameters['suffix'] === 'string' ? parameters['suffix'] : '_missing';
return {
created: strings(parameters['variables']).map((code) => ({
code: `${code}${suffix}`,
label: `${labelOf(code)} (missing)`,
})),
dropped: [],
};
}

if (name === 'outlier_detector') {
// Only the default `flag` action adds columns; `drop` removes rows and `set_missing` edits values.
if (parameters['action'] !== undefined && parameters['action'] !== 'flag') return NO_EFFECT;
const suffix = typeof parameters['suffix'] === 'string' ? parameters['suffix'] : '_outlier';
const strategies = (parameters['strategies'] ?? {}) as Record<string, unknown>;
return {
created: Object.keys(strategies).sort().map((code) => ({
code: `${code}${suffix}`,
label: `${labelOf(code)} (outlier)`,
})),
dropped: [],
};
}

if (name === 'one_hot_encoder') {
const dropFirst = parameters['drop_first'] !== false;
const dropOriginal = parameters['drop_original'] !== false;
const variables = strings(parameters['variables']);
const created: CreatedStepColumn[] = [];
for (const code of variables) {
const levels = [...(sources.get(code)?.enumerations ?? [])].sort((a, b) => (a.code < b.code ? -1 : a.code > b.code ? 1 : 0));
for (const level of dropFirst ? levels.slice(1) : levels) {
created.push({ code: `${code}_${level.code}`, label: `${labelOf(code)}: ${level.label || level.code}` });
}
}
return { created, dropped: dropOriginal ? variables : [] };
}

return NO_EFFECT;
}
Loading
Loading