Repository navigation
refactor: Replace CDE sql_type/is_categorical with a single variable type - #658
KFilippopolitis wants to merge 1 commit into
Conversation
KFilippopolitis
left a comment
There was a problem hiding this comment.
Contract review (CDEs JSON consumer)
Checked that this engine PR consumes the new type-only CDE JSON: no sql_type / isCategorical on the CDE path; type ∈ {continuous, discrete, nominal, ordinal, binary}; DuckDB continuous→DOUBLE, discrete→INTEGER, categorical→VARCHAR; algorithm input specs stay real/int/text with continuous→real, discrete→int (also accepted by real slots), categorical→text; free-text categoricals (no enumerations) are rejected as algorithm inputs; worker proto field 3 is type, field 4 and the old names are reserved.
Leftover sql_type / is_categorical hits are the reserved proto names, DType.from_sql (SQL types, not CDE JSON), and the non-serialized CommonDataElement.is_categorical property. That is correct.
Mixed Excel values cells are a producer question (datacatalog#22): the converter rejects them per type. This repo only sees the resulting JSON (enumerations xor min/max).
Verdict: ready to merge with the coordinated catalog / backend / UI / client PRs. Mixed-version workers fail closed on old field-3 values.
…type Adopts the MIP variable-type refactor of CDEsMetadata.json: CDEs no longer carry sql_type or isCategorical, and `type` alone holds the classification, one of continuous, discrete, nominal, ordinal, or binary. CDE model and transport - New CDEType enum and CATEGORICAL_CDE_TYPES in exaflow/datatypes.py. CommonDataElement stores `type` (plain string on dump) and derives is_categorical as a non-serialized property. - Worker proto: field 3 is now `type`; field 4 and the old names are reserved. Regenerated worker_pb2.py. - GET /cdes_metadata returns `type` instead of sql_type/is_categorical. Storage and validation derived from type - DuckDB loader: continuous -> DOUBLE, discrete -> INTEGER, categorical -> VARCHAR. DType.from_cde uses the same mapping for filter value checks. - Request validation maps CDE types onto the unchanged algorithm input spec contract (continuous -> real, discrete -> int, categorical -> text) and rejects categorical CDEs without enumerations, which keeps former free-text columns (subjectid, DIAG_*) out of analyses. Algorithms and preprocessing - Metadata dicts carry `type`; all is_categorical/sql_type checks now test membership in CATEGORICAL_CDE_TYPES or equality with CDEType.DISCRETE. - Preprocessing output columns declare a CDE `type` instead of the type/stattype pair; categorical outputs require enumerations. - promote_int_variables_to_real is now promote_discrete_variables_to_continuous. exadata-validator - Requires code/label/type with type in the five values, rejects enumerations on numerical CDEs and minValue/maxValue on categorical CDEs, requires whole-number bounds for discrete and exactly two enumerations for binary. dataset must be nominal with enumerations; visitid must be categorical with enumerations. Fixtures and docs - Migrated all CDEsMetadata.json fixtures, deriving the new type from the old isCategorical/sql_type (text -> nominal) and dropping empty enumerations. Updated tests, testcase generators, API spec, algorithm docs, conventions, and the decision log. Compatibility: breaking. Data models must be migrated, controller and workers upgraded together, and platform-backend, platform-ui, and mip-jupyter consumers of sql_type/is_categorical switched to `type`. Rollback: revert this commit and redeploy controller and workers with the previous data models.
6404ab0 to
ab31c2a
Compare
Summary
Adopts the MIP variable-type refactor of
CDEsMetadata.json. CDEs no longer carrysql_typeorisCategorical;typealone holds the classification, one ofcontinuous,discrete(numerical) ornominal,ordinal,binary(categorical).CDETypeenum andCATEGORICAL_CDE_TYPESinexaflow/datatypes.py.CommonDataElementstorestypeand derivesis_categoricalas a non-serialized property. Worker proto field 3 is nowtype; field 4 and the old names are reserved.GET /cdes_metadatareturnstypeinstead ofsql_type/is_categorical.continuousas DOUBLE,discreteas INTEGER and categorical types as VARCHAR. Column types for the test data models are unchanged.continuous→real,discrete→int, categorical →text;realslots also acceptdiscrete). Categorical CDEs without enumerations (former free-texttextcolumns such assubjectid) are rejected as algorithm inputs, which keeps per-subject grouping blocked.type; preprocessing output columns declare a CDEtypeinstead of thetype/stattypepair.promote_int_variables_to_realis nowpromote_discrete_variables_to_continuous.code/label/typewith one of the five types; rejects enumerations on numerical CDEs andminValue/maxValueon categorical ones; requires whole-number bounds fordiscreteand exactly two enumerations forbinary.datasetmust benominalwith enumerations;visitidmust be categorical with enumerations.CDEsMetadata.jsonfixtures migrated (type derived from the oldisCategorical/sql_type,text→nominal, empty enumerations dropped). API spec, algorithm docs, conventions and decision log updated.Tests
uv run pytest -q -n 8 -m "not smpc" tests/standalone_tests: 2164 passed (includes slow tests)cd exadata-validator && uv run pytest -q tests: 30 passedRisk and compatibility
Breaking by design (no compatibility path):
real,integer,text) are rejected.GET /cdes_metadataand the preprocessing specoutputs.columns[*]change shape; the platform and catalog PRs below consume the new shape.Rollback
Revert this PR and redeploy controller and workers together with the previous data models.
Related PRs
Same branch name
refactor/cde-variable-typesin every repo; merge and deploy together.