Skip to content

Add Databricks Snowflake and MongoDB attribute stores - #490

Merged
longquanzheng merged 1 commit into
mainfrom
codex/attribute-store-databricks-snowflake-mongodb
Sep 17, 2026
Merged

longquanzheng merged 1 commit into
mainfrom
codex/attribute-store-databricks-snowflake-mongodb

Conversation

@longquanzheng

@longquanzheng longquanzheng commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add Databricks and Snowflake Attribute Store dialects with startup schema validation and parameterized idempotent MERGE writes
  • add a MongoDB Attribute Store using existing collections, Flow IDs as _id, partial BSON upserts, and field-level filtering
  • preserve PostgreSQL/MySQL behavior while sharing SQL schema discovery, conversion, refresh, and lifecycle code
  • add configuration, SQL contract, MongoDB integration, Temporal/Cadence workflow, CLI, CI, and bilingual documentation coverage
  • make Attribute Sync integration label-gated in one workflow, so regular Server CI starts none of MySQL, PostgreSQL, or MongoDB

Rationale

Attribute Sync previously projected only into PostgreSQL and MySQL. Applications that use cloud warehouses or MongoDB needed a separate dual-write path. These backends keep Dex's existing asynchronous latest-state and at-least-once attempt semantics while making every destination write idempotent by Flow ID.

User impact

Server operators can configure databricks, snowflake, or mongodb stores without protobuf or SDK changes. SQL tables and MongoDB collections must already exist. Databricks and Snowflake require a logically unique string flowIdColumn; MongoDB writes one document per Flow. Dex does not create schemas, constraints, indexes, migrations, or automatic backfills.

Databricks and Snowflake are covered by local driver and SQL contract tests. This change does not claim live cloud warehouse wire-compatibility testing.

The Attribute Sync CI workflow contains independent relational and MongoDB jobs. A single ci:attribute-sync label runs attributeSyncIntegTests against MySQL/PostgreSQL and mongodbAttributeStoreIntegTests against MongoDB. Regular Server CI starts none of these databases.

Validation

  • make -C server unitTests
  • make -C cli test
  • make -C server attributeSyncIntegTests
  • make -C server mongodbAttributeStoreIntegTests
  • make copyright-check
  • cd docs && npm run check
  • go mod tidy -diff in server and cli
  • CI runner policy and actionlint validation for the Server and Attribute Sync workflows
  • companion skill-dex-developer 0.9.0 package, source-fidelity, skill, Codex plugin, and Claude plugin validation
  • all pull-request checks passing, including both label-gated Attribute Sync integration jobs

@longquanzheng
longquanzheng force-pushed the codex/attribute-store-databricks-snowflake-mongodb branch from 832230f to e239ba2 Compare September 17, 2026 19:18
@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

@longquanzheng
longquanzheng force-pushed the codex/attribute-store-databricks-snowflake-mongodb branch from e239ba2 to fd441b1 Compare September 17, 2026 19:34
@longquanzheng longquanzheng added the ci:mongodb Run the MongoDB Attribute Store integration workflow label Sep 17, 2026
@longquanzheng
longquanzheng force-pushed the codex/attribute-store-databricks-snowflake-mongodb branch 2 times, most recently from 993ac65 to d503bd5 Compare September 17, 2026 19:56
@longquanzheng longquanzheng added the ci:attribute-sync Run the Attribute Sync SQL integration job label Sep 17, 2026
@longquanzheng
longquanzheng force-pushed the codex/attribute-store-databricks-snowflake-mongodb branch from d503bd5 to 4f5ea64 Compare September 17, 2026 20:00
@longquanzheng longquanzheng removed the ci:mongodb Run the MongoDB Attribute Store integration workflow label Sep 17, 2026
@longquanzheng
longquanzheng marked this pull request as ready for review September 17, 2026 20:15
@longquanzheng
longquanzheng merged commit 239acde into main Sep 17, 2026
65 of 67 checks passed
@longquanzheng
longquanzheng deleted the codex/attribute-store-databricks-snowflake-mongodb branch September 17, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:attribute-sync Run the Attribute Sync SQL integration job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant