Repository navigation
feat(api): model BigQuery CDC settings and mappings - #1102
Merged
sdairs merged 1 commit intoOct 21, 2026
Merged
Conversation
sdairs
added this pull request to stack #1095
October 20, 2026 11:34
sdairs
changed the title
This was referenced Oct 20, 2026
sdairs
marked this pull request as ready for review
October 20, 2026 12:31
iskakaushik
approved these changes
Oct 20, 2026
sdairs
added a commit
that referenced
this pull request
Oct 21, 2026
…#1100) AWS backup-bucket payloads now include `iamRoleExternalId`, and the live OpenAPI adds organization notification listing and an additional service-create permission. Refresh the snapshot, regenerate operation metadata and the beta list, and model external IDs on the four AWS bucket types. The update request preserves omission, a replacement string, and explicit `null` (clear) independently. Its custom deserializer retains unknown-field tracking for strict CLI file input. Create/properties require the string under the repository's spec requiredness policy; response fields remain optional. The library README documents caller migration. Refs #1088. Stacked directly above #1099, preserving its Query API response-envelope fix and all earlier layers. BigQuery and notifications are separate API/CLI follow-ups; `spec_coverage_test` intentionally reports their remaining drift on this intermediate layer. Validation: three focused external-ID model tests and four backup client wiremock tests pass. No live resources or credentials were created. Stack dependency: #1099. Final API coverage: #1104; final CLI layer: #1105. Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`: `cargo fmt --all --check`; both CLI Clippy configurations; `cargo check -p clickhousectl --no-default-features`; API/analyzer all-target Clippy; `cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351 CLI tests; 96 Python/classifier tests. All passed. Live/resource-creating tests retain their existing ignore gates. No paid Cloud integration label or provisioning lifecycle was triggered. Exact-head CI at `ae6fffc5497229328a6f44458d4440a05cf72cf4`: format and MSRV pass; API build and Clippy pass. [The API test job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415838262) fails only in `vendored_openapi_snapshot_matches_rust_api`, with the expected 17 BigQuery/notifications findings covered by #1102/#1104. The dependent final API layer #1104 passes its full job. The rendering step is skipped after this expected test failure. Cloud integration decision requires authorization; no provisioning label was applied. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes AWS backup-bucket request shapes (required external ID on create) and PATCH semantics for IAM trust configuration; incorrect callers could break backup setup or fail API authorization checks after the new service-create permission is documented. > > **Overview** > Refreshes the vendored ClickHouse Cloud OpenAPI snapshot and regenerates operation metadata. **Service create** (`INSTANCE_CREATE`) now documents both `control-plane:organization:create-service` and `control-plane:service:manage`. The beta **organization notifications list** operation appears in the snapshot and in `meta` (`notification_get_list`); a full Rust client for it is left to a follow-up layer. > > **AWS backup buckets** gain `iam_role_external_id` (`iamRoleExternalId`) on response and request types. Responses accept missing or null external IDs; create/properties require a string. Patch uses `Option<Option<String>>` with a custom deserializer so callers can omit the field (preserve), send a string (replace), or send `null` (clear). The README describes migration for struct-literal callers. > > Adds focused **serde tests** and updates **wiremock backup-bucket client tests** for the new field and wire shapes. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ae6fffc. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
sdairs
added a commit
that referenced
this pull request
Oct 21, 2026
`cloud clickpipe create bigquery` now supports `cdc`/`cdc_only`, replication-method selection, and four timing/parallelism controls. Strict table-mapping JSON accepts the method-specific watermark or events function. Invalid mode/method combinations, missing per-table fields and invalid numeric values fail before reading credentials or sending HTTP; existing snapshot requests omit the new settings. Fixes #1107. Refs #1088. Depends on the immediately preceding BigQuery API layer in native stack #1095 above #1099. No new client method is omitted; BigQuery source updates remain unsupported by the existing API surface. Validation: 11 focused unit tests, nine focused subprocess tests, model write-back tests, and three command-tree checks passed. The ClickPipe configuration reference documents CDC inputs and constraints. No external BigQuery service or paid test service was provisioned. Stack dependency: #1102. Final API coverage: #1104; final CLI layer: #1105. Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`: `cargo fmt --all --check`; both CLI Clippy configurations; `cargo check -p clickhousectl --no-default-features`; API/analyzer all-target Clippy; `cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351 CLI tests; 96 Python/classifier tests. All passed. Live/resource-creating tests retain their existing ignore gates. No paid Cloud integration label or provisioning lifecycle was triggered. Exact-head CI at `65addfae0cb59bb91bbf879cf5a79039717009ac`: [full CLI job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415864220), format, MSRV and Cloud integration decision pass. Live suites were not selected for this CLI-only layer. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Adds new ingestion modes and validation on BigQuery pipe creation; mistakes are caught client-side, but misconfigured CDC could affect production replication behavior once deployed. > > **Overview** > **`cloud clickpipe create bigquery`** now supports **CDC** (`cdc` / `cdc_only`) alongside snapshot, using API-driven `--replication-mode` values instead of snapshot-only parsing. > > New create flags wire **replication method** (`query_based` / `events_based`) and **CDC tuning** (sync interval, pull parallelism, visibility delay, pull window). **`--table-mapping-json`** accepts **`queryCdcWatermarkColumn`** and **`eventsFunction`** (`appends` / `changes`), with validation that matches method to per-table fields and blocks CDC options in snapshot mode. > > Invalid combinations and out-of-range numbers **fail locally** before reading service-account files or calling the API; snapshot creates still omit the new settings. **Tests** cover parsing, request shape (both auth paths), and early rejection; **docs** add a BigQuery CDC section and update the source table blurb. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 65addfa. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BigQuery ClickPipe settings now preserve the live API's CDC replication method, sync interval, parallelism, delay and pull window. Table mappings preserve the watermark column and events function in both request and tolerant response types. Explicit fallible write-back conversions report absent required wire fields.
Add the spec-described
cdcandcdc_onlyreplication modes while retaining unknown future values. Keep the CLI's existing parser snapshot-only in this API layer; its CDC exposure is the next PR. The CLI settings literal is a compile-only adaptation, and a subprocess test pins omission of all newly added fields for existing snapshot requests. The library README documents Rust caller migration.Refs #1088. Depends on the preceding backup CLI layer in native stack #1095 above #1099. Notifications drift intentionally remains until its API follow-up.
Validation: four focused model tests, snapshot wire-omission coverage, and combined-stack repository gates. Third-party BigQuery provisioning and live CDC were not attempted.
Stack dependency: #1101. Final API coverage: #1104; final CLI layer: #1105.
Combined-stack validation at
dc45f645f20fa7b0d3639d732531d947726b68a5:cargo fmt --all --check; both CLI Clippy configurations;cargo check -p clickhousectl --no-default-features; API/analyzer all-target Clippy;cargo check --workspace --all-features; 817 API/analyzer tests; 2,351 CLI tests; 96 Python/classifier tests. All passed. Live/resource-creating tests retain their existing ignore gates. No paid Cloud integration label or provisioning lifecycle was triggered.Exact-head CI at
5bcc2976e5614040148dd848a207bd0286e8b1b8: format, MSRV and full CLI job pass. API build and Clippy pass; the API test job fails only invendored_openapi_snapshot_matches_rust_api, with the expected three notifications findings fixed by #1104. The rendering step is skipped after this expected test failure. Cloud integration decision requires authorization; no provisioning label was applied.Note
Medium Risk
Expands ClickPipe BigQuery request/response models and write-back conversions used by API clients; CLI behavior stays snapshot-only but struct-literal callers must adopt new fields or
Default.Overview
Extends the clickhouse-cloud-api BigQuery ClickPipe models to match the Cloud API’s CDC surface:
replicationModegainscdcandcdc_only, and settings/table mappings pick up optional CDC-related fields (replicationMethod, sync/pull tuning,queryCdcWatermarkColumn,eventsFunction, etc.) on both request and tolerant response types.Adds
TryFromconversions from BigQuery settings and table-mapping response types back to request types, withMissingRequiredFieldswhen required wire fields are absent. README documents migration (..Default::default()for new struct fields). Tests cover serde round-trip, null/absent handling, and enum values.clickhousectl is only adapted to compile: BigQuery create still accepts
snapshotonly for--replication-mode, settings use..Default::default()so CDC fields are not sent on snapshot creates, and CLI request-shape tests assert those fields stay omitted.Reviewed by Cursor Bugbot for commit 5bcc297. Bugbot is set up for automated code reviews on this repo. Configure here.