Skip to content

refactor: Single CDE variable type with strict binary/ordinal inputs and integer covariate coding - #674

Open
KFilippopolitis wants to merge 5 commits into
masterfrom
refactor/cde-variable-types
Open

KFilippopolitis wants to merge 5 commits into
masterfrom
refactor/cde-variable-types

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Replaces the CDE sql_type/is_categorical pair with a single variable type (continuous, discrete, nominal, ordinal, binary). The type drives four things:

  • Inputs that expect a binary or an ordinal variable accept only that exact type.
  • Binary and ordinal covariates are coded as integers.
  • Every categorical CDE (nominal, ordinal, binary, including dataset and visitid) is stored as an integer code in DuckDB, with the readable value kept as the enumeration label.
  • Datasets are identified by label in the registry and in requests. The controller translates each label to the stored code before it calls the workers.

Supersedes #658.

Commits, in review order

Commit Content
ee0b01e6b Replace sql_type/is_categorical with the single CDE type.
a3d2bc25e exadata-validator: validate the type and the integer category codes.
36365089d Worker: store categorical CDEs as integer codes.
fdbe9eb47 Controller: match input slots to CDE types one-to-one.
322052802 Make the one-hot and ordinal encoders private.
b0ca068ab Algorithms: require binary and ordinal inputs, code covariates as integers.
2e39b5fa7 Test data models: declare variable types and integer codes.
a5b782e80 Standalone tests for types, integer codes and covariate coding.
92aa75330 Algorithm validation: regenerated expected outputs.
29af073f5 Prod env tests and fixtures for integer codes.
691e4f198 Docs.
1105d4292 Order integer level codes numerically.
e5095de21 Expected outputs for the numeric level order.
0e8238132 Identify datasets by label in the registry and in requests.
82070f118 Environment tests send dataset labels.

Changes

Single CDE variable type

  • CDEType in exaflow/datatypes.py.
  • CommonDataElement drops sql_type and is_categorical.
  • Validator rules, for example a binary variable must have exactly 2 enumerations.

One-to-one type matching

  • InputDataStatType gains binary and ordinal, and the controller validator requires an exact match. A nominal variable never satisfies a binary or ordinal input, so it is rejected instead of converted.
  • Each algorithm repeats the check at runtime (utils/cde_types.py).
  • Binary only:
    • logistic_regression, logistic_regression_cv, glmm_binary: y must be binary.
    • fisher_exact: x and y must be binary.
    • cox_regression_classical, cox_regression_stacked: event_var must be binary.
  • Ordinal only: in glmm_ordinal, y must be ordinal. Its level order comes from the data model, and category_order is optional and must match that order.
  • Generic categorical inputs (covariates, grouping variables, class labels) accept nominal, ordinal and binary.

Integer coding of binary and ordinal covariates

  • The shared helper is build_covariate_transformer (utils/covariate_encoding.py). It is used in linear regression (plus CV), logistic regression (plus CV), Cox (classical and stacked), lmm, glmm_binary and glmm_ordinal.
  • Binary covariates become one 0/1 column named var[level coded 1], with the first declared level coded 0.
  • Ordinal covariates become one integer column, 0..K-1, in declared order.
  • Nominal covariates keep one-hot encoding.
  • A missing value or an undeclared level raises an error.
  • A binary or ordinal covariate with only one observed value after filtering (across all workers, also in CV) raises an error and stops the analysis, instead of returning a wrong intercept.

Private encoders

  • The one-hot and ordinal encoders are now _FederatedOneHotEncoder and _FederatedOrdinalEncoder and are no longer exported. They are used only inside algorithms.

Integer storage of categorical CDEs

  • DuckDB types: continuous is DOUBLE and discrete is INTEGER. binary is INTEGER 0/1. nominal and ordinal are INTEGER codes starting at 1. dataset is an INTEGER code 1..N per data model.
  • visitid is an integer-coded nominal variable (BL=1, FL1=2, ...).
  • subjectid and subjectcode are discrete whole numbers without enumerations, so subject identifiers are not published as category levels.
  • The validator and the loader reject a categorical CDE that has no enumerations, or that has non-integer codes.
  • All test data, fixtures, examples and validator data were recoded, with the old text values kept as labels.

Numeric order of level codes

  • Level codes are integers presented as text, so they are sorted numerically (sort_levels), not as text. With 10 or more levels the text order (1, 10, 11, 2) gave a different one-hot reference level and a different class order.
  • Linear regression cases 52 and 91 and Gaussian Naive Bayes cases 12, 14 and 18 have new expected outputs. The reference design matrix of the test-case generator uses the same order.

Level order between workers

  • Categorical levels (nominal, ordinal and binary) are compared in declared order. A worker that declares the same levels in a different order loses that data model, and a log line records it.

Datasets identified by label

  • The Worker Landscape Aggregator keys dataset locations and dataset variables by the label of the dataset enumeration. GET /datasets, GET /datasets_locations and GET /datasets_variables return labels.
  • Requests send labels in inputdata.datasets and validation_datasets, for example "datasets": ["EDSD_0", "EDSD_2"].
  • The controller translates labels to codes in one helper (dataset_codes, WorkerLandscapeAggregator.get_dataset_codes). The request validator, the Exareme3 strategy and the Flower strategy use it. Workers and algorithms still filter on the stored integer code.
  • Results keep the dataset code, like every other categorical level (for example describe returns dataset: "1").
  • A dataset reported by two workers under the same label is a duplicate and is dropped. The worker CSV loader rejects a data model that gives one label to two dataset codes.
  • run_analysis -d takes labels.

Cox positive_class

  • Works like logistic regression. It is required, must be a declared level, and is validated before the UDFs run. A shared helper (utils/positive_class.py) serves logistic regression, logistic CV, glmm_binary and both Cox algorithms. The old 0/1/true/false fallback is removed.

Docs

  • New documentation/data-types-and-coding.md, and a "Variable types and coding" section on every algorithm page. The API specification, the decision log, the commands page, README.md and AGENTS.md use dataset labels.

Test data and fixtures

  • gender and gender_type are binary, mortality_gose is binary and agegroup is ordinal.
  • A new binary alzheimerstatus variable is added to the dementia and longitudinal data.
  • Logistic fixtures with multi-level nominal outcomes were removed.
  • Linear regression expected outputs were regenerated with statsmodels on the pooled data.
  • Environment test requests name datasets by label. Expected outputs were not changed by that step.

Contract changes

  • Spec stattypes may contain binary and ordinal. Clients must match the CDE type exactly.
  • A new parameter field variable_stattypes is added. It is set only for Cox event_var.
  • glmm_ordinal.category_order is now optional.
  • datasets and validation_datasets in requests are dataset labels. A label can contain spaces ("Dummy TBI_0"). /datasets, /datasets_locations and /datasets_variables return labels.
  • Level codes and filter values in requests are integers, also filters on dataset. Parameters that name a level (positive_class, groupA, groupB, category_order, visit1, visit2) take the code as a string.
  • Results are keyed by code, including dataset codes. Labels come from the CDE enumerations.
  • Cox requests without positive_class are rejected.
  • A categorical CDE with no enumerations, or with non-integer codes, fails to load. A data model with the same label on two dataset codes fails to load.
  • Coefficient names for binary covariates change to var[level coded 1]. Ordinal covariates give a single coefficient, which assumes equal spacing between levels.
  • Consumers to update: Platform UI, platform-backend and mip-jupyter (send dataset labels, read results by code), Data Catalog (integer enumeration codes for every categorical CDE) and the mip-infra data loading (data recoded to integers).

Testing

  • CI on 82070f118: all six checks pass, including the standalone suite, the algorithm validation tests (one and five nodes), the production env tests and ruff.
  • Cox and the mixed models were checked against the CI fixtures; there is no separate numerical-equivalence study.

Deployment

  • Upgrade all components at once. Controllers and workers must be upgraded together. The worker proto reuses field 3 (sql_type → type) with new values, so old and new controllers and workers cannot run side by side.
  • No migration of deployed data. Existing deployed data is not migrated, and only newly loaded data uses the new CDE types and integer codes.
  • Rollback is a revert of the whole PR plus a reload of the previous data. Clients must go back to dataset codes at the same time.

@KFilippopolitis
KFilippopolitis force-pushed the refactor/cde-variable-types branch 3 times, most recently from 82070f1 to dc7196b Compare November 17, 2026 07:34
@KFilippopolitis
KFilippopolitis force-pushed the refactor/cde-variable-types branch from dc7196b to 8ebdc6f Compare November 17, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant