Skip to content

fix(api): decode Query API endpoint list envelopes - #1099

Merged
sdairs merged 1 commit into
claude/drift-cli-1088from
codex/query-api-endpoint-contract
Oct 21, 2026
Merged

sdairs merged 1 commit into
claude/drift-cli-1088from
codex/query-api-endpoint-contract

Conversation

@sdairs

@sdairs sdairs commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Query API endpoint lists return an array in result, with cursor metadata on the response envelope. The client decoded a nested items/pagination object instead, causing the scheduled Cloud Integration lifecycle and CLI list/name lookups to fail with invalid type: map, expected a sequence. A nonempty array is interpreted as positional struct fields; {"result":[{}]} reproduces the exact CI error at line 1, column 11.

Return ApiResponse<Vec<PublicQueryApiEndpointListItem>> from the library and consume its envelope pagination in the lifecycle test. The CLI adapts the response to its existing items/pagination output. Existing command flags, requests, authentication and name-selection safety checks are preserved. The legacy nested container remains exported for callers that use it independently; the library README documents the return-type migration.

The corrected client/model and subprocess fixtures cover nonempty and empty arrays, sparse/null responses, envelope metadata, opaque cursors, OAuth, multiple pages and unsafe name-selection failures.

Stacked directly above #1094 (claude/drift-cli-1088). Related release acceptance: #757; this PR does not close that tracker.

This is separate from #1088: neither its current gap list nor the normal dry-run at #1094 identifies the endpoint-list mismatch. The dry-run still reports the same 27 remaining findings after this change; no snapshot or analyzer policy is changed.

Validation:

  • The corrected client regression failed with the old decoder and passes with this fix.
  • Passed formatting, both CLI Clippy configurations, telemetry-disabled compilation, API/analyzer all-target Clippy and tests, and 96 Python/classifier tests.
  • Full CLI tests: all targets passed except failed_fresh_start_preserves_postgres_identity_without_polluting_clickhouse_selection, which reports docker_unavailable because the local Docker socket is absent. Endpoint subprocess and pagination/name-selection tests passed.
  • Read-only live smoke: the installed client reproduced the original decoding failure; the patched binary read two cursor pages and resolved an endpoint by name. No SQL ran and no Cloud resources changed.
  • The full provisioning lifecycle was not rerun.

Note

Medium Risk
Breaking return type for Rust callers of query_api_endpoint_list, but behavior matches the live API and fixes list/name-resolution failures; CLI output shape is preserved via an adapter layer.

Overview
Fixes Query API endpoint list decoding so the Cloud API client matches the real wire shape: result is an endpoint array and pagination lives on the envelope (next_cursor, limit, total_count), not a nested items/pagination object. That mismatch caused JSON decode failures (e.g. expected a sequence) for nonempty lists in integration and CLI name lookups.

query_api_endpoint_list now returns ApiResponse<Vec<PublicQueryApiEndpointListItem>>; QueryApiEndpointListResponse stays exported as a legacy helper type. clickhousectl rebuilds items/pagination from the envelope so human/JSON list output and --cursor behavior stay the same; missing/null result now fails the command instead of printing an empty list.

Docs note envelope vs CLI pagination fields; client, model, integration, and CLI subprocess tests use the corrected list envelopes and multi-page cursors.

Reviewed by Cursor Bugbot for commit b84d7b9. Bugbot is set up for automated code reviews on this repo. Configure here.

Read list entries and cursor metadata from the actual Cloud API envelope. Preserve CLI list output and name selection, and correct the client, model, subprocess and lifecycle fixtures.
@sdairs
sdairs merged commit 296eb38 into main Oct 21, 2026
11 of 12 checks passed
sdairs added a commit that referenced this pull request Oct 21, 2026
…#1100)

AWS backup-bucket payloads now include `iamRoleExternalId`, and the live
OpenAPI adds organization notification listing and an additional
service-create permission. Refresh the snapshot, regenerate operation
metadata and the beta list, and model external IDs on the four AWS
bucket types.

The update request preserves omission, a replacement string, and
explicit `null` (clear) independently. Its custom deserializer retains
unknown-field tracking for strict CLI file input. Create/properties
require the string under the repository's spec requiredness policy;
response fields remain optional. The library README documents caller
migration.

Refs #1088. Stacked directly above #1099, preserving its Query API
response-envelope fix and all earlier layers. BigQuery and notifications
are separate API/CLI follow-ups; `spec_coverage_test` intentionally
reports their remaining drift on this intermediate layer.

Validation: three focused external-ID model tests and four backup client
wiremock tests pass. No live resources or credentials were created.

Stack dependency: #1099. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `ae6fffc5497229328a6f44458d4440a05cf72cf4`: format and
MSRV pass; API build and Clippy pass. [The API test
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415838262)
fails only in `vendored_openapi_snapshot_matches_rust_api`, with the
expected 17 BigQuery/notifications findings covered by #1102/#1104. The
dependent final API layer #1104 passes its full job. The rendering step
is skipped after this expected test failure. Cloud integration decision
requires authorization; no provisioning label was applied.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes AWS backup-bucket request shapes (required external ID on
create) and PATCH semantics for IAM trust configuration; incorrect
callers could break backup setup or fail API authorization checks after
the new service-create permission is documented.
> 
> **Overview**
> Refreshes the vendored ClickHouse Cloud OpenAPI snapshot and
regenerates operation metadata. **Service create** (`INSTANCE_CREATE`)
now documents both `control-plane:organization:create-service` and
`control-plane:service:manage`. The beta **organization notifications
list** operation appears in the snapshot and in `meta`
(`notification_get_list`); a full Rust client for it is left to a
follow-up layer.
> 
> **AWS backup buckets** gain `iam_role_external_id`
(`iamRoleExternalId`) on response and request types. Responses accept
missing or null external IDs; create/properties require a string. Patch
uses `Option<Option<String>>` with a custom deserializer so callers can
omit the field (preserve), send a string (replace), or send `null`
(clear). The README describes migration for struct-literal callers.
> 
> Adds focused **serde tests** and updates **wiremock backup-bucket
client tests** for the new field and wire shapes.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ae6fffc. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
sdairs added a commit that referenced this pull request Oct 21, 2026
`cloud backup bucket create/update --file` now has explicit coverage and
documentation for AWS `iamRoleExternalId`. Update input distinguishes
omission (preserve), string (replace), and `null` (clear); unknown
fields and invalid types still fail before HTTP. The existing typed
file-input path carries the API field without adding a flag or
duplicating request serialization.

Fixes #1106. Refs #1088. Depends on the immediately preceding backup API
layer, itself above #1099 in native stack #1095. No new library method
is omitted. The AWS create example follows the live schema's required
string field.

Validation: builder tests cover create requiredness, all update states
and strict unknown-field checks; real-binary wiremock coverage pins
POST/PATCH bodies. The cloud-service guide includes clearing through
stdin. No paid provisioning was attempted.

Stack dependency: #1100. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `1b36b75c7b67bdc5ec7596c273b41f494770ca96`: [full CLI
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415845906),
format, MSRV and Cloud integration decision pass. Live suites were not
selected for this CLI-only layer.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Test and documentation only; no production CLI or API behavior changes
in this diff.
> 
> **Overview**
> Adds **tests and docs** for AWS **`iamRoleExternalId`** on `cloud
backup bucket create/update --file`, without changing CLI
request-building logic in this diff.
> 
> Unit tests now expect the field in AWS create/update JSON fixtures and
add
**`aws_backup_bucket_external_id_preserves_update_states_and_unknown_field_checks`**:
update accepts omit/string/`null`, rejects bad types and typos, and
create rejects missing or null external IDs. A **wiremock** test asserts
the real binary sends matching **POST/PATCH** bodies for create (with
external ID) and update (omit, replace, or clear).
> 
> The **cloud-service** guide documents `iamRoleExternalId` in the AWS
field table and explains preserve/replace/clear semantics with a stdin
example.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1b36b75. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
sdairs added a commit that referenced this pull request Oct 21, 2026
BigQuery ClickPipe settings now preserve the live API's CDC replication
method, sync interval, parallelism, delay and pull window. Table
mappings preserve the watermark column and events function in both
request and tolerant response types. Explicit fallible write-back
conversions report absent required wire fields.

Add the spec-described `cdc` and `cdc_only` replication modes while
retaining unknown future values. Keep the CLI's existing parser
snapshot-only in this API layer; its CDC exposure is the next PR. The
CLI settings literal is a compile-only adaptation, and a subprocess test
pins omission of all newly added fields for existing snapshot requests.
The library README documents Rust caller migration.

Refs #1088. Depends on the preceding backup CLI layer in native stack
#1095 above #1099. Notifications drift intentionally remains until its
API follow-up.

Validation: four focused model tests, snapshot wire-omission coverage,
and combined-stack repository gates. Third-party BigQuery provisioning
and live CDC were not attempted.

Stack dependency: #1101. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `5bcc2976e5614040148dd848a207bd0286e8b1b8`: format,
MSRV and [full CLI
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415854891)
pass. API build and Clippy pass; [the API test
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415854963)
fails only in `vendored_openapi_snapshot_matches_rust_api`, with the
expected three notifications findings fixed by #1104. The rendering step
is skipped after this expected test failure. Cloud integration decision
requires authorization; no provisioning label was applied.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Expands ClickPipe BigQuery request/response models and write-back
conversions used by API clients; CLI behavior stays snapshot-only but
struct-literal callers must adopt new fields or `Default`.
> 
> **Overview**
> Extends the **clickhouse-cloud-api** BigQuery ClickPipe models to
match the Cloud API’s CDC surface: `replicationMode` gains **`cdc`** and
**`cdc_only`**, and settings/table mappings pick up optional CDC-related
fields (`replicationMethod`, sync/pull tuning,
`queryCdcWatermarkColumn`, `eventsFunction`, etc.) on both request and
tolerant response types.
> 
> Adds **`TryFrom`** conversions from BigQuery settings and
table-mapping **response** types back to request types, with
**`MissingRequiredFields`** when required wire fields are absent.
**README** documents migration (`..Default::default()` for new struct
fields). **Tests** cover serde round-trip, null/absent handling, and
enum values.
> 
> **clickhousectl** is only adapted to compile: BigQuery create still
accepts **`snapshot`** only for `--replication-mode`, settings use
**`..Default::default()`** so CDC fields are not sent on snapshot
creates, and CLI request-shape tests assert those fields stay omitted.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
5bcc297. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
sdairs added a commit that referenced this pull request Oct 21, 2026
`cloud clickpipe create bigquery` now supports `cdc`/`cdc_only`,
replication-method selection, and four timing/parallelism controls.
Strict table-mapping JSON accepts the method-specific watermark or
events function. Invalid mode/method combinations, missing per-table
fields and invalid numeric values fail before reading credentials or
sending HTTP; existing snapshot requests omit the new settings.

Fixes #1107. Refs #1088. Depends on the immediately preceding BigQuery
API layer in native stack #1095 above #1099. No new client method is
omitted; BigQuery source updates remain unsupported by the existing API
surface.

Validation: 11 focused unit tests, nine focused subprocess tests, model
write-back tests, and three command-tree checks passed. The ClickPipe
configuration reference documents CDC inputs and constraints. No
external BigQuery service or paid test service was provisioned.

Stack dependency: #1102. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `65addfae0cb59bb91bbf879cf5a79039717009ac`: [full CLI
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415864220),
format, MSRV and Cloud integration decision pass. Live suites were not
selected for this CLI-only layer.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Adds new ingestion modes and validation on BigQuery pipe creation;
mistakes are caught client-side, but misconfigured CDC could affect
production replication behavior once deployed.
> 
> **Overview**
> **`cloud clickpipe create bigquery`** now supports **CDC** (`cdc` /
`cdc_only`) alongside snapshot, using API-driven `--replication-mode`
values instead of snapshot-only parsing.
> 
> New create flags wire **replication method** (`query_based` /
`events_based`) and **CDC tuning** (sync interval, pull parallelism,
visibility delay, pull window). **`--table-mapping-json`** accepts
**`queryCdcWatermarkColumn`** and **`eventsFunction`** (`appends` /
`changes`), with validation that matches method to per-table fields and
blocks CDC options in snapshot mode.
> 
> Invalid combinations and out-of-range numbers **fail locally** before
reading service-account files or calling the API; snapshot creates still
omit the new settings. **Tests** cover parsing, request shape (both auth
paths), and early rejection; **docs** add a BigQuery CDC section and
update the source table blurb.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
65addfa. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
sdairs added a commit that referenced this pull request Oct 21, 2026
…1104)

Add the beta `Client::notification_get_list` endpoint with all eight
optional filters, opaque cursor forwarding, and
`ApiResponse<Vec<Notification>>` pagination metadata. `Notification`
tolerates missing/null fields and future type, severity and
resource-kind strings. Export it through both public facade paths and
add a read-only notification step to the organization integration
lifecycle.

Fixes #1088. Depends on the preceding BigQuery CLI layer in native stack
#1095, preserving #1099 and the earlier drift work. Together the API
layers cover all current findings: a fresh live dry run reports zero
actionable drift. The three pre-existing API-role enum acknowledgements
remain unchanged; no new exemption hides drift.

Validation: nine focused API cases pass, covering filters, OAuth, cursor
encoding, empty/sparse responses, metadata, facade paths and errors. The
full combined API/analyzer suite, including vendored spec coverage,
passes. The live organization lifecycle was not run because it also
creates credentials/resources; the added step itself is read-only.

Stack dependency: #1103. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `9431c677e0cf3f172dcbf48f81849c7191e81d38`: [full
API/analyzer
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415873161),
including vendored spec coverage and Python rendering/classifier tests,
passes. Format and MSRV pass. Cloud integration decision requires
authorization; the provisioning suites remain skipped. No provisioning
label was applied.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Read-only beta client surface plus tolerant response models and tests;
no auth, billing, or mutating API behavior.
> 
> **Overview**
> Adds the **beta** `Client::notification_get_list` endpoint so callers
can page through an organization’s notification history as
`ApiResponse<Vec<Notification>>`, with optional `limit`, opaque
`cursor`, and filters (severity, type, resource id/type, RFC 3339 date
range).
> 
> Introduces the **`Notification`** response model (all fields optional;
type/severity/resource kind stay open strings) and re-exports it on the
public `models` facade. README documents pagination defaults (100, max
250) and retention/access rules.
> 
> **Tests:** wiremock coverage for query params, envelope metadata,
sparse/null pages, and API errors; serde round-trip tolerance for
`Notification`; facade type parity; a read-only “list notifications”
step in the org integration lifecycle (including optional cursor
follow-up).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
9431c67. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
sdairs added a commit that referenced this pull request Oct 21, 2026
Add read-only `cloud org notifications` (Beta), exposing limit, cursor,
severity, type, resource ID/type and inclusive RFC 3339 date filters.
JSON preserves page entries and cursor metadata, including an empty
opaque cursor; human output safely renders sparse entries. Route through
a domain-local `CloudClient` wrapper and declare the notification
workflow's permissions.

Fixes #1108. Refs #1088. Depends on the immediately preceding
notifications API layer, which closes the drift issue. This is the final
new layer in native stack #1095 above #1099. All parameters of the new
library method are exposed; no method is intentionally omitted.

Validation: five focused unit tests and five real-binary subprocess
tests pass, covering OAuth, filters, pagination, empty/sparse human
output, errors and local date/range validation. Parent/leaf help was
inspected and the authentication reference documents page-by-page reads.
No paid provisioning or credentials creation was attempted.

Stack dependency: #1104. Final API coverage: #1104; final CLI layer:
#1105.

Combined-stack validation at `dc45f645f20fa7b0d3639d732531d947726b68a5`:
`cargo fmt --all --check`; both CLI Clippy configurations; `cargo check
-p clickhousectl --no-default-features`; API/analyzer all-target Clippy;
`cargo check --workspace --all-features`; 817 API/analyzer tests; 2,351
CLI tests; 96 Python/classifier tests. All passed.
Live/resource-creating tests retain their existing ignore gates. No paid
Cloud integration label or provisioning lifecycle was triggered.

Exact-head CI at `dc45f645f20fa7b0d3639d732531d947726b68a5`: [full CLI
job](https://github.com/ClickHouse/clickhousectl/actions/runs/39415882754),
format, MSRV and Cloud integration decision pass. The final live-spec
dry run still reports zero actionable drift. Live suites were not
selected for this CLI-only layer.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Read-only CLI and documentation with local validation; no writes, auth
changes, or provisioning paths.
> 
> **Overview**
> Adds a read-only **Beta** `clickhousectl cloud org notifications`
command to page through organization notifications via the Cloud API.
> 
> The command exposes **limit**, **cursor**, **severity**, **--type**,
**resource-id**, **resource-type**, and inclusive **RFC 3339**
**from-date** / **to-date** filters. It is classified as a read
(OAuth-friendly), registers **`NOTIFICATION_GET_LIST`** in org
permissions, and calls a new **`CloudClient::list_notifications`**
wrapper. **JSON** returns `result` with optional `limit` and
`nextCursor`; **human** output prints a markdown table (sparse fields as
`-`) and prints the next cursor as a safely escaped JSON string.
> 
> Unit and subprocess tests cover flag parsing, validation, HTTP query
forwarding, pagination, errors, and table rendering.
**`docs/reference/authentication.md`** documents pagination and filter
behavior alongside **`notification-types`**.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
dc45f64. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants