Skip to content

refactor: Replace CDE sql_type/isCategorical with a single variable type - #22

Closed
KFilippopolitis wants to merge 2 commits into
masterfrom
refactor/cde-variable-types
Closed

KFilippopolitis wants to merge 2 commits into
masterfrom
refactor/cde-variable-types

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adopts the MIP variable-type refactor: a CDE's type alone carries the classification, one of continuous, discrete (numerical) or nominal, ordinal, binary (categorical). sql_type and isCategorical are no longer emitted or accepted.

Data quality tool

  • POST /excel-to-json accepts only the five types. Categorical values become enumerations; numerical values become a minValue/maxValue range (whole numbers for discrete). binary needs exactly two values; nominal and ordinal may omit values (free-text variables). A values cell mixing enumerations and a range is rejected by the per-type parsers.
  • JSON validation requires code and type, rejects enumerations on numerical CDEs and minValue/maxValue on categorical CDEs, enforces the binary and discrete rules, and requires a nominal dataset CDE with enumerations.
  • Excel and JSON fixtures migrated (real → continuous, integer → discrete, text → nominal). README and EXCEL_TO_CDES_JSON.md updated, including an old → new type mapping.

Backend

  • CommonDataElementDTO drops sql_type/isCategorical and validates the new types and per-type rules.
  • New Flyway Java migration V1_1__MigrateCdeVariableTypes rewrites stored data models: the new type is derived from the old isCategorical/sql_type, those fields are removed, and empty enumerations are dropped. Without it, existing rows would fail to deserialize.

Frontend

  • Pathology interfaces drop sql_type/isCategorical; the tree shows enumerations for any variable that has them.

Commands run

  • cd data_quality_tool && poetry run pytest: 181 passed; poetry run black --check . clean
  • cd backend && mvn test: passed (includes new V1_1__MigrateCdeVariableTypesTest and CommonDataElementDTOTest)
  • cd frontend && npx tsc --noEmit -p tsconfig.app.json: clean
  • Not run: frontend Karma tests (no spec covers the changed code).

Impact

  • Breaking: old Excel files and CDE JSON must use the new types.
  • Database: V1_1 rewrites data_model.variables/groups on deploy. No .env, Keycloak, Docker or Helm changes.
  • Rollback: revert this PR and restore the data_model table from a pre-migration backup, since V1_1 rewrites rows.

Related PRs

Same branch name refactor/cde-variable-types in every repo; merge and deploy together.

Adopts the MIP variable-type refactor: a CDE's `type` alone carries the
classification, one of continuous, discrete (numerical) or nominal,
ordinal, binary (categorical). sql_type and isCategorical are no longer
emitted or accepted.

Data quality tool
- Excel -> JSON accepts only the five types; categorical `values` become
  enumerations, numerical `values` a minValue/maxValue range (whole numbers
  for discrete). binary needs exactly two values; nominal and ordinal may
  omit them (free-text variables). Mixed enumeration/range values are
  rejected by the per-type parsers.
- JSON validation requires code and type, rejects enumerations on
  numerical CDEs and minValue/maxValue on categorical CDEs, enforces the
  binary and discrete rules, and requires a nominal `dataset` CDE with
  enumerations.
- Fixtures (JSON and xlsx) migrated: real -> continuous, integer ->
  discrete, text -> nominal.

Backend
- CommonDataElementDTO drops sql_type/isCategorical and validates the new
  types and per-type rules.
- Flyway V1_1 Java migration rewrites stored data models: the new type is
  derived from the old isCategorical/sql_type, those fields are removed,
  and empty enumerations are dropped. Without it, stored rows would fail
  to deserialize.

Frontend
- Pathology interfaces drop sql_type/isCategorical; the tree shows
  enumerations for any variable that has them.

Compatibility: breaking. Old Excel files and CDE JSON must use the new
types. Rollback requires reverting this commit and restoring the
data_model table from a pre-migration backup, since V1_1 rewrites rows.

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contract review (CDEs JSON / POST /excel-to-json)

Checked against the new type-only contract: sql_type / isCategorical are no longer emitted; type is one of continuous | discrete | nominal | ordinal | binary; categorical values → enumerations; numerical values → minValue/maxValue; conceptPath builds the group tree then is stripped; empty lists are stripped. Flyway V1_1 maps the four old (isCategorical, sql_type) pairs correctly.

Verdict: with fixes (two producer bugs below). Not blocking on extra-key rejection: Spring/DTO deserialization already drops leftover sql_type/isCategorical, so they are not re-emitted on the store path.

Mixed Excel values cells (open question)

Rejected by the per-type parser. There is no third JSON shape with both enumerations and minValue/maxValue.

  • Numerical type + a mixed cell: process_values_based_on_type only rsplits on - and float()s both sides (data_quality_tool/converter/excel_to_json.py). Enumeration text fails that parse.
  • Categorical type + a mixed/range cell: only process_enumerations runs; anything that is not {"code", "label"} pairs raises Nominal values format error.
  • excel_validator.validate_variable_type uses the same exclusive split.

Must fix

1. /validate-excel does not enforce whole-number bounds for discrete

validate_min_max only checks <min>-<max> and min < max (data_quality_tool/validator/excel_validator.py). type=discrete, values=1.5-2.5 passes POST /validate-excel and then fails in the converter. README / EXCEL_TO_CDES_JSON.md say both layers enforce this.

After validate_min_max, if type_val == "discrete", require both bounds float.is_integer() (same check as excel_to_json.py).

2. DTO Integer minValue/maxValue cannot store the converter’s continuous ranges

Converter tests emit floats (0.01 / 99.99). CommonDataElementDTO still uses Integer. Excel import (DataModelConverter.convertExcelToDataModelDTO) convertValues that JSON into the DTO, so a continuous range with a fractional bound fails or coerces. Platform-backend already stores these as String, so this is the catalog bottleneck.

Use Double (or Number), keep the discrete whole-number check in the compact constructor, and add a DTO test that continuous 0.01–99.99 is accepted and discrete 0.5 is rejected.

…ous ranges

- /validate-excel now rejects a discrete row whose range bounds are not
  whole numbers, matching the converter and the JSON validator.
- CommonDataElementDTO stores minValue/maxValue as Double, so continuous
  ranges such as 0.01-99.99 are no longer truncated to integers; the
  discrete whole-number check moves into the compact constructor.
- JSON-to-Excel prints whole-number float bounds as integers (0-130, not
  0.0-130.0) now that the backend sends doubles.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant