Repository navigation
refactor: Single CDE variable type with strict binary/ordinal inputs and integer covariate coding - #674
Open
KFilippopolitis wants to merge 5 commits into
Open
refactor: Single CDE variable type with strict binary/ordinal inputs and integer covariate coding#674KFilippopolitis wants to merge 5 commits into
KFilippopolitis wants to merge 5 commits into
Conversation
KFilippopolitis
force-pushed
the
refactor/cde-variable-types
branch
3 times, most recently
from
November 17, 2026 07:34
82070f1 to
dc7196b
Compare
KFilippopolitis
force-pushed
the
refactor/cde-variable-types
branch
from
November 17, 2026 09:40
dc7196b to
8ebdc6f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the CDE
sql_type/is_categoricalpair with a single variable type (continuous,discrete,nominal,ordinal,binary). The type drives four things:datasetandvisitid) is stored as an integer code in DuckDB, with the readable value kept as the enumeration label.Supersedes #658.
Commits, in review order
ee0b01e6bsql_type/is_categoricalwith the single CDE type.a3d2bc25e36365089dfdbe9eb47322052802b0ca068ab2e39b5fa7a5b782e8092aa7533029af073f5691e4f1981105d4292e5095de210e823813282070f118Changes
Single CDE variable type
CDETypeinexaflow/datatypes.py.CommonDataElementdropssql_typeandis_categorical.binaryvariable must have exactly 2 enumerations.One-to-one type matching
InputDataStatTypegainsbinaryandordinal, and the controller validator requires an exact match. A nominal variable never satisfies a binary or ordinal input, so it is rejected instead of converted.utils/cde_types.py).logistic_regression,logistic_regression_cv,glmm_binary:ymust be binary.fisher_exact:xandymust be binary.cox_regression_classical,cox_regression_stacked:event_varmust be binary.glmm_ordinal,ymust be ordinal. Its level order comes from the data model, andcategory_orderis optional and must match that order.nominal,ordinalandbinary.Integer coding of binary and ordinal covariates
build_covariate_transformer(utils/covariate_encoding.py). It is used in linear regression (plus CV), logistic regression (plus CV), Cox (classical and stacked),lmm,glmm_binaryandglmm_ordinal.var[level coded 1], with the first declared level coded 0.Private encoders
_FederatedOneHotEncoderand_FederatedOrdinalEncoderand are no longer exported. They are used only inside algorithms.Integer storage of categorical CDEs
continuousis DOUBLE anddiscreteis INTEGER.binaryis INTEGER 0/1.nominalandordinalare INTEGER codes starting at 1.datasetis an INTEGER code 1..N per data model.visitidis an integer-coded nominal variable (BL=1, FL1=2, ...).subjectidandsubjectcodearediscretewhole numbers without enumerations, so subject identifiers are not published as category levels.Numeric order of level codes
sort_levels), not as text. With 10 or more levels the text order (1, 10, 11, 2) gave a different one-hot reference level and a different class order.Level order between workers
Datasets identified by label
datasetenumeration.GET /datasets,GET /datasets_locationsandGET /datasets_variablesreturn labels.inputdata.datasetsandvalidation_datasets, for example"datasets": ["EDSD_0", "EDSD_2"].dataset_codes,WorkerLandscapeAggregator.get_dataset_codes). The request validator, the Exareme3 strategy and the Flower strategy use it. Workers and algorithms still filter on the stored integer code.describereturnsdataset: "1").run_analysis -dtakes labels.Cox
positive_classutils/positive_class.py) serves logistic regression, logistic CV,glmm_binaryand both Cox algorithms. The old 0/1/true/false fallback is removed.Docs
documentation/data-types-and-coding.md, and a "Variable types and coding" section on every algorithm page. The API specification, the decision log, the commands page,README.mdandAGENTS.mduse dataset labels.Test data and fixtures
genderandgender_typeare binary,mortality_goseis binary andagegroupis ordinal.alzheimerstatusvariable is added to the dementia and longitudinal data.Contract changes
stattypesmay containbinaryandordinal. Clients must match the CDEtypeexactly.variable_stattypesis added. It is set only for Coxevent_var.glmm_ordinal.category_orderis now optional.datasetsandvalidation_datasetsin requests are dataset labels. A label can contain spaces ("Dummy TBI_0")./datasets,/datasets_locationsand/datasets_variablesreturn labels.dataset. Parameters that name a level (positive_class,groupA,groupB,category_order,visit1,visit2) take the code as a string.positive_classare rejected.var[level coded 1]. Ordinal covariates give a single coefficient, which assumes equal spacing between levels.Testing
82070f118: all six checks pass, including the standalone suite, the algorithm validation tests (one and five nodes), the production env tests and ruff.Deployment
sql_type→type) with new values, so old and new controllers and workers cannot run side by side.