Repository navigation
feat(cli): expose BigQuery CDC settings and mappings - #1103
Merged
sdairs merged 1 commit intoOct 21, 2026
Merged
Conversation
sdairs
changed the title
This was referenced Oct 20, 2026
sdairs
marked this pull request as ready for review
October 20, 2026 12:31
iskakaushik
approved these changes
Oct 20, 2026
sdairs
added a commit
that referenced
this pull request
Oct 21, 2026
…1104) Add the beta `Client::notification_get_list` endpoint with all eight optional filters, opaque cursor forwarding, and `ApiResponse<Vec<Notification>>` pagination metadata. `Notification` tolerates missing/null fields and future type, severity and resource-kind strings. Export it through both public facade paths and add a read-only notification step to the organization integration lifecycle. Fixes #1088. Depends on the preceding BigQuery CLI layer in native stack #1095, preserving #1099 and the earlier drift work. Together the API layers cover all current findings: a fresh live dry run reports zero actionable drift. The three pre-existing API-role enum acknowledgements remain unchanged; no new exemption hides drift. Validation: nine focused API cases pass, covering filters, OAuth, cursor encoding, empty/sparse responses, metadata, facade paths and errors. The full combined API/analyzer suite, including vendored spec coverage, passes. The live organization lifecycle was not run because it also creates credentials/resources; the added step itself is read-only. Stack dependency: #1103. Final API coverage: #1104; final CLI layer: #1105. Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`: `cargo fmt --all --check`; both CLI Clippy configurations; `cargo check -p clickhousectl --no-default-features`; API/analyzer all-target Clippy; `cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351 CLI tests; 96 Python/classifier tests. All passed. Live/resource-creating tests retain their existing ignore gates. No paid Cloud integration label or provisioning lifecycle was triggered. Exact-head CI at `9431c677e0cf3f172dcbf48f81849c7191e81d38`: [full API/analyzer job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415873161), including vendored spec coverage and Python rendering/classifier tests, passes. Format and MSRV pass. Cloud integration decision requires authorization; the provisioning suites remain skipped. No provisioning label was applied. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Read-only beta client surface plus tolerant response models and tests; no auth, billing, or mutating API behavior. > > **Overview** > Adds the **beta** `Client::notification_get_list` endpoint so callers can page through an organization’s notification history as `ApiResponse<Vec<Notification>>`, with optional `limit`, opaque `cursor`, and filters (severity, type, resource id/type, RFC 3339 date range). > > Introduces the **`Notification`** response model (all fields optional; type/severity/resource kind stay open strings) and re-exports it on the public `models` facade. README documents pagination defaults (100, max 250) and retention/access rules. > > **Tests:** wiremock coverage for query params, envelope metadata, sparse/null pages, and API errors; serde round-trip tolerance for `Notification`; facade type parity; a read-only “list notifications” step in the org integration lifecycle (including optional cursor follow-up). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9431c67. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cloud clickpipe create bigquerynow supportscdc/cdc_only, replication-method selection, and four timing/parallelism controls. Strict table-mapping JSON accepts the method-specific watermark or events function. Invalid mode/method combinations, missing per-table fields and invalid numeric values fail before reading credentials or sending HTTP; existing snapshot requests omit the new settings.Fixes #1107. Refs #1088. Depends on the immediately preceding BigQuery API layer in native stack #1095 above #1099. No new client method is omitted; BigQuery source updates remain unsupported by the existing API surface.
Validation: 11 focused unit tests, nine focused subprocess tests, model write-back tests, and three command-tree checks passed. The ClickPipe configuration reference documents CDC inputs and constraints. No external BigQuery service or paid test service was provisioned.
Stack dependency: #1102. Final API coverage: #1104; final CLI layer: #1105.
Combined-stack validation at
dc45f645f20fa7b0d3639d732531d947726b68a5:cargo fmt --all --check; both CLI Clippy configurations;cargo check -p clickhousectl --no-default-features; API/analyzer all-target Clippy;cargo check --workspace --all-features; 817 API/analyzer tests; 2,351 CLI tests; 96 Python/classifier tests. All passed. Live/resource-creating tests retain their existing ignore gates. No paid Cloud integration label or provisioning lifecycle was triggered.Exact-head CI at
65addfae0cb59bb91bbf879cf5a79039717009ac: full CLI job, format, MSRV and Cloud integration decision pass. Live suites were not selected for this CLI-only layer.Note
Medium Risk
Adds new ingestion modes and validation on BigQuery pipe creation; mistakes are caught client-side, but misconfigured CDC could affect production replication behavior once deployed.
Overview
cloud clickpipe create bigquerynow supports CDC (cdc/cdc_only) alongside snapshot, using API-driven--replication-modevalues instead of snapshot-only parsing.New create flags wire replication method (
query_based/events_based) and CDC tuning (sync interval, pull parallelism, visibility delay, pull window).--table-mapping-jsonacceptsqueryCdcWatermarkColumnandeventsFunction(appends/changes), with validation that matches method to per-table fields and blocks CDC options in snapshot mode.Invalid combinations and out-of-range numbers fail locally before reading service-account files or calling the API; snapshot creates still omit the new settings. Tests cover parsing, request shape (both auth paths), and early rejection; docs add a BigQuery CDC section and update the source table blurb.
Reviewed by Cursor Bugbot for commit 65addfa. Bugbot is set up for automated code reviews on this repo. Configure here.