Skip to content

refactor: Replace CDE sql_type/is_categorical with a single variable type - #658

Closed
KFilippopolitis wants to merge 1 commit into
masterfrom
refactor/cde-variable-types
Closed

KFilippopolitis wants to merge 1 commit into
masterfrom
refactor/cde-variable-types

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adopts the MIP variable-type refactor of CDEsMetadata.json. CDEs no longer carry sql_type or isCategorical; type alone holds the classification, one of continuous, discrete (numerical) or nominal, ordinal, binary (categorical).

  • CDE model and transport: new CDEType enum and CATEGORICAL_CDE_TYPES in exaflow/datatypes.py. CommonDataElement stores type and derives is_categorical as a non-serialized property. Worker proto field 3 is now type; field 4 and the old names are reserved. GET /cdes_metadata returns type instead of sql_type/is_categorical.
  • Storage derived from type: the DuckDB loader stores continuous as DOUBLE, discrete as INTEGER and categorical types as VARCHAR. Column types for the test data models are unchanged.
  • Request validation: CDE types are mapped onto the unchanged algorithm input spec contract (continuous → real, discrete → int, categorical → text; real slots also accept discrete). Categorical CDEs without enumerations (former free-text text columns such as subjectid) are rejected as algorithm inputs, which keeps per-subject grouping blocked.
  • Algorithms and preprocessing: metadata dicts carry type; preprocessing output columns declare a CDE type instead of the type/stattype pair. promote_int_variables_to_real is now promote_discrete_variables_to_continuous.
  • exadata-validator: requires code/label/type with one of the five types; rejects enumerations on numerical CDEs and minValue/maxValue on categorical ones; requires whole-number bounds for discrete and exactly two enumerations for binary. dataset must be nominal with enumerations; visitid must be categorical with enumerations.
  • Fixtures and docs: all CDEsMetadata.json fixtures migrated (type derived from the old isCategorical/sql_type, text → nominal, empty enumerations dropped). API spec, algorithm docs, conventions and decision log updated.

Tests

  • uv run pytest -q -n 8 -m "not smpc" tests/standalone_tests: 2164 passed (includes slow tests)
  • cd exadata-validator && uv run pytest -q tests: 30 passed
  • Not run: prod/kind environment tests.

Risk and compatibility

Breaking by design (no compatibility path):

  • Data models must be migrated; old types (real, integer, text) are rejected.
  • Controller and workers must be upgraded together (proto change).
  • GET /cdes_metadata and the preprocessing spec outputs.columns[*] change shape; the platform and catalog PRs below consume the new shape.

Rollback

Revert this PR and redeploy controller and workers together with the previous data models.

Related PRs

Same branch name refactor/cde-variable-types in every repo; merge and deploy together.

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contract review (CDEs JSON consumer)

Checked that this engine PR consumes the new type-only CDE JSON: no sql_type / isCategorical on the CDE path; type ∈ {continuous, discrete, nominal, ordinal, binary}; DuckDB continuous→DOUBLE, discrete→INTEGER, categorical→VARCHAR; algorithm input specs stay real/int/text with continuous→real, discrete→int (also accepted by real slots), categorical→text; free-text categoricals (no enumerations) are rejected as algorithm inputs; worker proto field 3 is type, field 4 and the old names are reserved.

Leftover sql_type / is_categorical hits are the reserved proto names, DType.from_sql (SQL types, not CDE JSON), and the non-serialized CommonDataElement.is_categorical property. That is correct.

Mixed Excel values cells are a producer question (datacatalog#22): the converter rejects them per type. This repo only sees the resulting JSON (enumerations xor min/max).

Verdict: ready to merge with the coordinated catalog / backend / UI / client PRs. Mixed-version workers fail closed on old field-3 values.

…type

Adopts the MIP variable-type refactor of CDEsMetadata.json: CDEs no longer
carry sql_type or isCategorical, and `type` alone holds the classification,
one of continuous, discrete, nominal, ordinal, or binary.

CDE model and transport
- New CDEType enum and CATEGORICAL_CDE_TYPES in exaflow/datatypes.py.
  CommonDataElement stores `type` (plain string on dump) and derives
  is_categorical as a non-serialized property.
- Worker proto: field 3 is now `type`; field 4 and the old names are
  reserved. Regenerated worker_pb2.py.
- GET /cdes_metadata returns `type` instead of sql_type/is_categorical.

Storage and validation derived from type
- DuckDB loader: continuous -> DOUBLE, discrete -> INTEGER, categorical ->
  VARCHAR. DType.from_cde uses the same mapping for filter value checks.
- Request validation maps CDE types onto the unchanged algorithm input
  spec contract (continuous -> real, discrete -> int, categorical -> text)
  and rejects categorical CDEs without enumerations, which keeps former
  free-text columns (subjectid, DIAG_*) out of analyses.

Algorithms and preprocessing
- Metadata dicts carry `type`; all is_categorical/sql_type checks now test
  membership in CATEGORICAL_CDE_TYPES or equality with CDEType.DISCRETE.
- Preprocessing output columns declare a CDE `type` instead of the
  type/stattype pair; categorical outputs require enumerations.
- promote_int_variables_to_real is now
  promote_discrete_variables_to_continuous.

exadata-validator
- Requires code/label/type with type in the five values, rejects
  enumerations on numerical CDEs and minValue/maxValue on categorical
  CDEs, requires whole-number bounds for discrete and exactly two
  enumerations for binary. dataset must be nominal with enumerations;
  visitid must be categorical with enumerations.

Fixtures and docs
- Migrated all CDEsMetadata.json fixtures, deriving the new type from the
  old isCategorical/sql_type (text -> nominal) and dropping empty
  enumerations. Updated tests, testcase generators, API spec, algorithm
  docs, conventions, and the decision log.

Compatibility: breaking. Data models must be migrated, controller and
workers upgraded together, and platform-backend, platform-ui, and
mip-jupyter consumers of sql_type/is_categorical switched to `type`.
Rollback: revert this commit and redeploy controller and workers with the
previous data models.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant