Repository navigation
refactor: Replace CDE sql_type/isCategorical with a single variable type - #22
KFilippopolitis wants to merge 2 commits into
Conversation
Adopts the MIP variable-type refactor: a CDE's `type` alone carries the classification, one of continuous, discrete (numerical) or nominal, ordinal, binary (categorical). sql_type and isCategorical are no longer emitted or accepted. Data quality tool - Excel -> JSON accepts only the five types; categorical `values` become enumerations, numerical `values` a minValue/maxValue range (whole numbers for discrete). binary needs exactly two values; nominal and ordinal may omit them (free-text variables). Mixed enumeration/range values are rejected by the per-type parsers. - JSON validation requires code and type, rejects enumerations on numerical CDEs and minValue/maxValue on categorical CDEs, enforces the binary and discrete rules, and requires a nominal `dataset` CDE with enumerations. - Fixtures (JSON and xlsx) migrated: real -> continuous, integer -> discrete, text -> nominal. Backend - CommonDataElementDTO drops sql_type/isCategorical and validates the new types and per-type rules. - Flyway V1_1 Java migration rewrites stored data models: the new type is derived from the old isCategorical/sql_type, those fields are removed, and empty enumerations are dropped. Without it, stored rows would fail to deserialize. Frontend - Pathology interfaces drop sql_type/isCategorical; the tree shows enumerations for any variable that has them. Compatibility: breaking. Old Excel files and CDE JSON must use the new types. Rollback requires reverting this commit and restoring the data_model table from a pre-migration backup, since V1_1 rewrites rows.
KFilippopolitis
left a comment
There was a problem hiding this comment.
Contract review (CDEs JSON / POST /excel-to-json)
Checked against the new type-only contract: sql_type / isCategorical are no longer emitted; type is one of continuous | discrete | nominal | ordinal | binary; categorical values → enumerations; numerical values → minValue/maxValue; conceptPath builds the group tree then is stripped; empty lists are stripped. Flyway V1_1 maps the four old (isCategorical, sql_type) pairs correctly.
Verdict: with fixes (two producer bugs below). Not blocking on extra-key rejection: Spring/DTO deserialization already drops leftover sql_type/isCategorical, so they are not re-emitted on the store path.
Mixed Excel values cells (open question)
Rejected by the per-type parser. There is no third JSON shape with both enumerations and minValue/maxValue.
- Numerical
type+ a mixed cell:process_values_based_on_typeonlyrsplits on-andfloat()s both sides (data_quality_tool/converter/excel_to_json.py). Enumeration text fails that parse. - Categorical
type+ a mixed/range cell: onlyprocess_enumerationsruns; anything that is not{"code", "label"}pairs raisesNominal values format error. excel_validator.validate_variable_typeuses the same exclusive split.
Must fix
1. /validate-excel does not enforce whole-number bounds for discrete
validate_min_max only checks <min>-<max> and min < max (data_quality_tool/validator/excel_validator.py). type=discrete, values=1.5-2.5 passes POST /validate-excel and then fails in the converter. README / EXCEL_TO_CDES_JSON.md say both layers enforce this.
After validate_min_max, if type_val == "discrete", require both bounds float.is_integer() (same check as excel_to_json.py).
2. DTO Integer minValue/maxValue cannot store the converter’s continuous ranges
Converter tests emit floats (0.01 / 99.99). CommonDataElementDTO still uses Integer. Excel import (DataModelConverter.convertExcelToDataModelDTO) convertValues that JSON into the DTO, so a continuous range with a fractional bound fails or coerces. Platform-backend already stores these as String, so this is the catalog bottleneck.
Use Double (or Number), keep the discrete whole-number check in the compact constructor, and add a DTO test that continuous 0.01–99.99 is accepted and discrete 0.5 is rejected.
…ous ranges - /validate-excel now rejects a discrete row whose range bounds are not whole numbers, matching the converter and the JSON validator. - CommonDataElementDTO stores minValue/maxValue as Double, so continuous ranges such as 0.01-99.99 are no longer truncated to integers; the discrete whole-number check moves into the compact constructor. - JSON-to-Excel prints whole-number float bounds as integers (0-130, not 0.0-130.0) now that the backend sends doubles.
Summary
Adopts the MIP variable-type refactor: a CDE's
typealone carries the classification, one ofcontinuous,discrete(numerical) ornominal,ordinal,binary(categorical).sql_typeandisCategoricalare no longer emitted or accepted.Data quality tool
POST /excel-to-jsonaccepts only the five types. Categoricalvaluesbecomeenumerations; numericalvaluesbecome aminValue/maxValuerange (whole numbers fordiscrete).binaryneeds exactly two values;nominalandordinalmay omit values (free-text variables). Avaluescell mixing enumerations and a range is rejected by the per-type parsers.codeandtype, rejectsenumerationson numerical CDEs andminValue/maxValueon categorical CDEs, enforces thebinaryanddiscreterules, and requires anominaldatasetCDE with enumerations.real→continuous,integer→discrete,text→nominal). README andEXCEL_TO_CDES_JSON.mdupdated, including an old → new type mapping.Backend
CommonDataElementDTOdropssql_type/isCategoricaland validates the new types and per-type rules.V1_1__MigrateCdeVariableTypesrewrites stored data models: the new type is derived from the oldisCategorical/sql_type, those fields are removed, and empty enumerations are dropped. Without it, existing rows would fail to deserialize.Frontend
sql_type/isCategorical; the tree shows enumerations for any variable that has them.Commands run
cd data_quality_tool && poetry run pytest: 181 passed;poetry run black --check .cleancd backend && mvn test: passed (includes newV1_1__MigrateCdeVariableTypesTestandCommonDataElementDTOTest)cd frontend && npx tsc --noEmit -p tsconfig.app.json: cleanImpact
V1_1rewritesdata_model.variables/groupson deploy. No.env, Keycloak, Docker or Helm changes.data_modeltable from a pre-migration backup, sinceV1_1rewrites rows.Related PRs
Same branch name
refactor/cde-variable-typesin every repo; merge and deploy together.