Skip to content

The lakehouse is one protocol away: Trino, Databricks and Snowflake as query engines - #239

Merged
WaylandYang merged 2 commits into
devfrom
feat/lakehouse-engines
Sep 3, 2026
Merged

The lakehouse is one protocol away: Trino, Databricks and Snowflake as query engines#239
WaylandYang merged 2 commits into
devfrom
feat/lakehouse-engines

Conversation

@WaylandYang

@WaylandYang WaylandYang commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Three HTTP-protocol query engines beside Postgres, each behind the same QueryEngine trait:

  • Trino (trino://, also presto://) — POST /v1/statement + nextUri paging. One catalog away from Iceberg, Delta Lake, Hive and Hudi, and Starburst speaks the same protocol.
  • Databricks (databricks://) — SQL Statement Execution API with INLINE / JSON_ARRAY results and PENDING → SUCCEEDED polling; the path is the console's httpPath.
  • Snowflake (snowflake://) — SQL API v2 with 202 → handle polling; programmatic access token or OAuth (key-pair JWT deliberately not yet).

The scheme picks the engine (engine_from_conn), so the data-source page keeps its single connection-string field; the shape is validated at registration with the exact form in the error. The SQL gate parses with the engine's own dialect (guard_sql_for), LIMIT wrapping and JSON Lines are shared, and string-typed values from Databricks / Snowflake are restored to numbers and booleans from the column type. The engine client has an explicit proxy policy (loopback and NO_PROXY direct, HTTPS_PROXY / HTTP_PROXY otherwise) because reqwest's Windows system-proxy detection routed loopback through the proxy. Migration 0021 widens the engine CHECK.

Tests: 17 new unit tests (connection-string parsing, dialect gates, JSON assembly, type coercion) and wiremock replays of each protocol including their error bodies. End to end against a stand-in Trino on the isolated deployment: register → test → grant → mount (schema document says Engine: trino) → chat runs query_data through the Trino path and answers with the lake's numbers. None of the three has run against a real cluster; follow-up issues ask for that.

Decision record 0018 (English, per the new convention) explains why Trino rather than format readers, why MaxCompute waits, and what is missing. README's Ontology-driven querying row and roadmap updated (English only, by request).

🤖 Generated with Claude Code

…s query engines

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@WaylandYang
WaylandYang force-pushed the feat/lakehouse-engines branch from fca6e7b to a0a9949 Compare September 3, 2026 02:56
@WaylandYang
WaylandYang merged commit f18c560 into dev Sep 3, 2026
3 checks passed
@WaylandYang
WaylandYang deleted the feat/lakehouse-engines branch September 3, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant