Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions docs/design/google-sheets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Google Sheets integration design catalog

Status: **Proposed**

This catalog is the source of truth for the Google Sheets integration in Altinity SQL Browser. GitHub issues define executable implementation slices and link back to these documents.

## Product boundary

The integration connects a saved ClickHouse query to a Google spreadsheet without adding an application backend.

```text
Google Sheet hyperlink
ClickHouse-hosted SQL Browser SPA
ClickHouse query execution
Google Sheets API
```

SQL Browser owns query execution, typed parameters, schema validation, refresh orchestration, managed worksheet contents, and synchronization checkpoints. Google Sheets owns formulas, native pivots, charts, comments, sharing, and presentation sheets.

## Supported project scope

- One-time snapshot export.
- Create or attach a linked spreadsheet.
- Manual deep-link refresh.
- Failure-safe full refresh through staging.
- Large-result streaming and quota management.
- Append-only refresh for eligible ClickHouse 26.8+ MergeTree-family sources using `_block_number` and `_block_offset`.
- Refresh history, repair, and unlink workflows.

## Explicit non-goals

- Scheduled or background refresh.
- Refresh while SQL Browser is closed.
- `SELECT ... STREAM` integration.
- Updates or deletes in append mode.
- Google Docs integration.
- Apps Script or a dedicated application backend.
- Service-account impersonation.
- Strict distributed locking between spreadsheet collaborators.

## Documents

- [Product scope](product-scope.md)
- [Architecture and data flow](architecture.md)
- [Resource model](resource-model.md)
- [Authentication and authorization](authentication.md)
- [Spreadsheet layout](spreadsheet-layout.md)
- [Snapshot export](snapshot-export.md)
- [Full refresh](full-refresh.md)
- [Append refresh](append-refresh.md)
- [Large results and quotas](large-results.md)
- [Failure recovery](failure-recovery.md)
- [Sharing and security](sharing-security.md)
- [Deployment configuration](configuration.md)
- [Testing strategy](testing.md)
- [Architecture decisions](decisions.md)

## Interactive companion diagrams

- [Architecture](diagrams/architecture.html)
- [Refresh flow](diagrams/refresh-flow.html)
- [Append checkpoint](diagrams/append-checkpoint.html)

The Markdown documents and Mermaid diagrams are canonical. HTML diagrams are explanatory companions and must be updated in the same pull request when their corresponding design changes.

## Implementation tracking

The umbrella GitHub issue owns project sequencing. Child issues should contain only their implementation slice, acceptance criteria, dependencies, tests, and links to the relevant catalog sections. They must not duplicate the complete architecture.

## Change-management rule

Design precedence is:

```text
repository design catalog
implementation issue
pull request
```

A pull request that changes an agreed behavior must update the corresponding design document and, where appropriate, the decision log.
78 changes: 78 additions & 0 deletions docs/design/google-sheets/append-refresh.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Append refresh

Append refresh is an opt-in mode for immutable inserted rows. Updates and deletes are out of scope; run a full refresh when they must be reflected.

## Availability

Append mode is shown only when all checks pass:

- configured feature enabled;
- ClickHouse version is at least the configured minimum, initially `26.8`;
- direct `MergeTree` or `ReplicatedMergeTree` source;
- stable table UUID;
- persistent `_block_number` and `_block_offset` available;
- finite ordinary SELECT;
- validated append-compatible query shape;
- destination checkpoint valid.

Version gating is necessary but not sufficient. Runtime capability probes are authoritative.

`SELECT ... STREAM` is explicitly out of scope.

## Eligible query shape

Allow deterministic projection, aliases, typed parameters, optional filters, and WHERE predicates over one direct source table.

Reject joins, views, `FINAL`, aggregation, `GROUP BY`, `DISTINCT`, window functions, `UNION`, table functions, business `LIMIT`, changing top-N semantics, and arbitrary query rewriting.

The binding pins table UUID, query snapshot, query hash, schema hash, and filter parameter values. Changing the source subset requires a full refresh.

## Cursor

Commit position is per partition:

```json
{
"partitions": {
"202607": {
"blockNumber": "18401",
"blockOffset": "982"
}
}
}
```

All values are decimal strings. Rows are complete but not globally ordered across partitions. Process partitions in deterministic `partition_id` order and rows within a partition in `(_block_number, _block_offset)` order.

## Refresh procedure

```mermaid
flowchart TD
A[Read Google checkpoint] --> B[Revalidate server, table UUID, query, schema]
B --> C[Capture per-partition upper block boundaries]
C --> D[Fetch finite bounded page]
D --> E[Upload page to hidden delta staging]
E --> F[Atomically copy rows and advance partition checkpoint]
F --> G{Upper boundaries reached?}
G -- no --> D
G -- yes --> H[Record completion]
```

Each page query filters after the stored `(blockNumber, blockOffset)`, stops at the captured upper block, orders by commit position, and applies a bounded limit. Cursor columns are technical and need not appear in the visible sheet.

## Google commit invariant

For each page, copying rows to the exact live destination and advancing that partition checkpoint occur in the same final Google structural batch. Never advance the checkpoint independently of the corresponding rows.

## Initial population

- **Export existing rows, then append**: default. Use the cursor-aware bounded path for history.
- **Append future rows only**: capture current upper boundaries without exporting history.

## Invalidation

Require an explicit full refresh when server capability, table UUID, query hash, schema, parameters defining the source subset, managed row count, or checkpoint validity changes. Never silently fall back.

## Concurrency

Use an advisory lease and generation check. Re-read generation immediately before each commit and abort when another refresh advanced it. Strict distributed locking between collaborators is not promised.
74 changes: 74 additions & 0 deletions docs/design/google-sheets/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Architecture and data flow

## Runtime architecture

Altinity SQL Browser remains a static single-page application served by ClickHouse. It does not listen on a port, receive Google callbacks, or add an application server.

```mermaid
flowchart LR
U[User in Google Sheets] -->|clicks Refresh data| B[Browser navigation]
B -->|GET /sql| C[ClickHouse HTTP server]
C -->|serves static SPA| A[SQL Browser SPA]
A -->|authenticated SQL over HTTP| C
A -->|OAuth access token and REST| G[Google Sheets API]
C -->|query result stream| A
A -->|staged values and publish requests| G
```

The refresh URL uses an opaque binding identifier in the fragment:

```text
https://clickhouse.example.com/sql#sheet-refresh=<binding-id>
```

The fragment is interpreted by the SPA and is not sent to ClickHouse by the browser. It contains no SQL, parameters, credentials, or spreadsheet contents.

## Binding resolution

The SPA resolves the opaque identifier through a deployment-configured ClickHouse binding table using the current authenticated ClickHouse identity. The latest append-only binding version contains the spreadsheet identity, managed sheet IDs, pinned query snapshot, parameter policy, limits, and refresh mode.

## Authentication boundaries

```mermaid
sequenceDiagram
participant U as User
participant A as SQL Browser SPA
participant C as ClickHouse
participant G as Google Identity / Sheets

U->>A: Open refresh deep link
A->>C: Authenticate and resolve binding
C-->>A: Authorized binding definition
U->>A: Continue with Google
A->>G: Request short-lived browser access token
G-->>A: Access token callback
A->>C: Execute pinned query
C-->>A: Stream rows
A->>G: Upload staging values
A->>G: Publish and update checkpoint
```

ClickHouse authentication authorizes query execution and binding access. Google authentication authorizes spreadsheet modification. Neither identity implies the other.

## Data path

```text
ClickHouse response stream
SQL Browser JavaScript memory
Google Sheets REST API
```

Data does not pass through an Altinity application backend.

## Source-of-truth hierarchy

- ClickHouse binding table: durable binding definition and versions.
- Google hidden state sheet: destination commit state and append checkpoint.
- Managed data sheet: last committed exported rows.
- Local workspace: navigation convenience and editing source, not the durable refresh authority.

## Refresh entry and completion

When SQL Browser is closed, the spreadsheet remains usable with its last committed data. Nothing refreshes in the background. The control worksheet hyperlink launches a new SPA instance, which resolves, authenticates, executes, uploads, publishes, and terminates the refresh workflow. The completion screen links back to the spreadsheet and source query.
48 changes: 48 additions & 0 deletions docs/design/google-sheets/authentication.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Authentication and authorization

## Independent sessions

ClickHouse and Google authorization are independent:

```text
ClickHouse identity → execute query and access binding
Google identity → edit destination spreadsheet
```

A Google Sheet collaborator does not automatically receive ClickHouse access. A ClickHouse user does not automatically receive permission to edit the spreadsheet.

## ClickHouse authentication

The refresh deep link opens the normal SQL Browser SPA. If the ClickHouse session is absent or expired, the existing authentication flow runs. The pending refresh intent must survive the redirect through the URL fragment or tab-scoped session state.

After authentication, the SPA resolves the binding and executes the pinned query using the current user’s ClickHouse credentials. Binding visibility and query access are enforced by ClickHouse grants and row policies.

## Google authentication

Use the Google Identity Services browser token model. The user explicitly invokes a Continue with Google action, receives a short-lived access token in JavaScript, and the SPA calls Google APIs directly.

Rules:

- keep Google access tokens in browser memory only;
- never put tokens in URLs, logs, IndexedDB, workspace exports, or spreadsheet metadata;
- request the narrowest scopes that support Picker and editing files selected or created by the application;
- handle popup blocking and multiple signed-in Google accounts explicitly;
- request a new token after expiry through a user gesture;
- do not store a Google refresh token;
- do not use service accounts or impersonation in this project.

## Account mismatch

Before modification, show the active Google account and target spreadsheet identity. When the selected Google account lacks edit access, report the account and permission problem without changing the spreadsheet.

## Secrets and sensitive parameters

Spreadsheet developer metadata and hidden worksheets are not secret stores. Never write:

- ClickHouse credentials or JWTs;
- Google tokens;
- complete private SQL unless the design explicitly exposes it;
- password-like parameter values;
- secrets embedded in query parameters.

Sensitive parameters must use prompt-on-refresh mode. Stored non-sensitive parameters remain part of the versioned binding and are validated as typed ClickHouse parameters before execution.
43 changes: 43 additions & 0 deletions docs/design/google-sheets/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Deployment configuration

Google Sheets integration is enabled only when a valid configuration object is present.

```json
{
"google_sheets": {
"oauth_client_id": "...apps.googleusercontent.com",
"binding_table": "asb.google_sheet_bindings",
"refresh_log_table": "asb.google_sheet_refresh_log",
"append_refresh": {
"enabled": true,
"min_clickhouse_version": "26.8"
},
"limits": {
"max_rows_full_refresh": 100000,
"max_rows_per_append_run": 250000,
"max_managed_cells": 2000000,
"max_serialized_bytes": 200000000,
"google_request_target_bytes": 1000000,
"commit_batch_rows": 20000
}
}
}
```

## Rules

- Absence of `google_sheets` hides all Google actions and preserves existing behavior.
- Parse ClickHouse table names strictly as qualified identifiers and quote database/table components independently.
- The SPA does not create, alter, or migrate configured tables.
- Validate binding and refresh-log table conformance before first use per connection and cache successful checks for the session.
- Host-specific configuration may override top-level configuration for configured alternate ClickHouse hosts.
- Manual or unlisted connections have no Google Sheets capability unless explicitly configured.
- Limits are deployment policy and must be enforced before or during transfer without silent truncation.

## Required ClickHouse privileges

Binding operations require the deployment-selected combination of `SELECT` and `INSERT` on configured tables. Query execution requires the ordinary privileges of the pinned saved query. No application-specific `CREATE`, `ALTER`, `UPDATE`, or `DELETE` privilege is required.

## Feature compatibility

Full refresh does not require the append minimum version. Append mode additionally requires the configured minimum version and successful runtime probes for the source table and persistent commit-position columns.
Loading
Loading