Skip to content

refactor(datamodel): Drop sql_type/is_categorical from the CDE DTO - #40

Closed
KFilippopolitis wants to merge 14 commits into
masterfrom
refactor/cde-variable-types
Closed

KFilippopolitis wants to merge 14 commits into
masterfrom
refactor/cde-variable-types

Conversation

@KFilippopolitis

@KFilippopolitis KFilippopolitis commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The MIP variable-type refactor removes sql_type and isCategorical from CDEs; type (continuous, discrete, nominal, ordinal, binary) alone carries the classification. Exaflow no longer sends either field, so DataModelDTO.CommonDataElementDTO stops declaring sql_type and is_categorical, and GET /services/data-models returns type only.

Tests

  • mvn test: 89 passed

Compatibility

Breaking for consumers that read sql_type or is_categorical from /data-models; the UI and notebook client PRs below switch to type. Deploy together with the Exaflow PR. Rollback: revert this PR together with the related PRs.

Related PRs

Same branch name refactor/cde-variable-types in every repo; merge and deploy together.

jdaln and others added 14 commits October 15, 2026 14:51
ci: run vulnerability scans on push to master
DataModelService now omits datasets whose variable list is null or [] and
drops null entries inside a list instead of failing the request, so the UI
only grays out variables for datasets whose variables are actually known.
ClaimUtils no longer adds an [] entry for an authorized dataset without a
variable list; it applies the same rule as DataModelService.
…available-variables

fix(datamodels): send only known dataset variable lists
… patches

- spring-boot-starter-parent 4.1.1 -> 4.1.2 (Spring Framework 7.0.10,
  Security 7.1.2, Data 2026.0.2, Tomcat 11.0.27)
- drop the tomcat.version 11.0.25 override: Boot 4.1.2 manages 11.0.27,
  which is the newest release and includes the CVE fixes it was added for
- hibernate-core 7.4.8.Final -> 7.4.13.Final (matches the Boot 4.1.2 BOM)
- postgresql 42.7.13 -> 42.7.14
- flyway-core / flyway-database-postgresql 13.6.0 -> 13.11.1
- nimbus-jose-jwt 10.9.1 -> 10.10
- jackson 2.22.3 -> 2.22.4, jackson 3.2.3 -> 3.2.4
GitHub Actions (SHA-pinned with a version comment):
- github/codeql-action/upload-sarif v4.31.2 -> v4.39.0; the gitleaks
  workflow was still on the deprecated v2 line (v2.25.6) and moves to v4.39.0
- actions/checkout v6.0.3 -> v7.0.1 in secrets-scan (others already on v7.0.1)
- actions/upload-artifact v7.0.1 -> v7.0.2
- docker/setup-buildx-action v4.3.0 -> v4.4.1
- docker/build-push-action v7.3.0 -> v7.4.0
- Python 3.14.7 -> 3.14.8 for the scan scripts

Scanners in ci/setup-tools.sh (SHA256 values checked against the
downloaded release assets):
- trivy v0.74.0 -> v0.76.0
- osv-scanner v2.5.1 -> v2.7.0
- opengrep v1.30.0 -> v1.31.0
- semgrep-rules develop ref -> 17e31e11 (2026-10-15)

Versions respect the repo's 7-day cooldown (dependabot.yml /
renovate.json): codeql-action v4.39.1, setup-python v7.1.0 / Python
3.14.9, trivy v0.76.1, opengrep v1.32.0 and newer semgrep-rules commits
are held back. hadolint v2.15.1, gitleaks v8.30.1 and cyclonedx-npm 6.0.1
are already latest.
Add a decision-log entry with versions, cooldown hold-backs, scan results
and validation; drop the resolved Tomcat-override risk; bump the Boot
version in AGENTS.md.
…-ci-latest

chore(deps): upgrade dependencies and CI to latest
…hon-3.x

chore(deps): update dependency python to v3.15.0
…ngrep-opengrep-1.x

chore(deps): update dependency opengrep/opengrep to v1.31.0
The MIP variable-type refactor removes sql_type and isCategorical from
CDEs; `type` (continuous, discrete, nominal, ordinal, binary) alone
carries the classification. Exaflow no longer sends either field, so the
CDE DTO stops declaring them and /data-models returns `type` only.

Compatibility: breaking for API consumers that read sql_type or
is_categorical from /data-models; the UI and the notebook client are
updated on the same branch name.

@KFilippopolitis KFilippopolitis left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contract review (CDEs JSON passthrough)

This is the only backend change needed: DataModelDTO.CommonDataElementDTO drops sql_type and is_categorical. type, enumerations, minValue, and maxValue still serialize. minValue/maxValue are String here, so continuous floats from the catalog are not truncated (unlike datacatalog’s Integer DTO — flagged on datacatalog#22).

Grep of this worktree: no remaining production reads of sql_type / is_categorical / isCategorical. Algorithm tests that mention real / text are Exaflow parameter types, not CDE type. Keep String type; this layer does not switch on it.

Verdict: ready to merge with UI#57, mip-jupyter#15, and Exaflow#658. Current UI/client still read the dropped fields from /data-models.

@KFilippopolitis
KFilippopolitis force-pushed the master branch 2 times, most recently from aab8297 to 03d18bf Compare November 5, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants