Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release-binaries.yml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ jobs:
set -euo pipefail
uv run --python 3.12 test/release/cli_smoke.py build/release-host/duckdb dist/duckflight.duckdb_extension
uv run --python 3.12 test/release/pgwire_attached_database.py build/release-host/duckdb dist/duckflight.duckdb_extension
uv run --python 3.12 test/release/pgwire_cross_catalog.py build/release-host/duckdb dist/duckflight.duckdb_extension
uv run --python 3.12 test/release/http_quack_smoke.py build/release-host/duckdb dist/duckflight.duckdb_extension
uv run --python 3.12 test/release/host_sessions.py dist/duckflight.duckdb_extension --duckdb-cli build/release-host/duckdb
uv run --python 3.12 test/release/oidc_health.py dist/duckflight.duckdb_extension --duckdb-cli build/release-host/duckdb
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "duckflight"
version = "0.1.17"
version = "0.1.18"
edition = "2024"
license = "MIT"
description = "Public DuckDB extension shim for the DuckFlight PostgreSQL, Flight SQL, HTTP, and Quack runtime"
Expand Down
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,25 @@ Shutdown rejects new work while allowing admitted queries and result fetches to
within the grace period, then interrupts outstanding execution. Zero grace forces immediate
shutdown. Stopping one listener leaves other listeners running.

#### PostgreSQL cross-catalog reads

A PostgreSQL session follows PostgreSQL's rule against cross-database references: an
explicit catalog other than the connected database fails with SQLSTATE `3D000`. A
PostgreSQL listener can opt in to reading other attached catalogs:

```toml
[pgwire]
cross_catalog_reads = true
```

With the setting on, a read-only `SELECT` (including joins, subqueries, CTEs and set
operations) may name `catalog.schema.relation` in any attached, non-internal catalog, the
same set a client can select as its startup database. Two-part and unqualified names still
resolve in the connected database. Writes, DDL, writable CTEs, `COPY`, `EXPLAIN`, and
functions or types qualified with another catalog keep the default rule, and catalog
metadata still describes only the connected database. The setting requires DuckFlight
0.1.18 or newer: earlier versions reject a config file that contains a `[pgwire]` table.

#### Optional mTLS

mTLS is an advanced alternative for deployments that manage client certificates:
Expand Down
8 changes: 4 additions & 4 deletions core-assets.lock
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# platform sha256 immutable_url
#
# Each immutable release asset is fetched, checksum-verified, and embedded by `make release`.
linux_amd64 0877d3491b5d4aaa3c3231f53c3909ef465523667b6c6e3b0d406172401c765e https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.21/libduckflight_core_ffi-linux_amd64.so
linux_arm64 d86493d7293fcd66d317fc50e78a7895b0caf6ef415d8ce38066bfc0c1d87056 https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.21/libduckflight_core_ffi-linux_arm64.so
osx_amd64 7871c1c77be3be8ee87b6143acbc958072f1a20d13a142d067039c4ba6ef0f7a https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.21/libduckflight_core_ffi-osx_amd64.dylib
osx_arm64 850c633f81418e813b82023365bec9bad307d857bea2db7032cf642d30fb0290 https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.21/libduckflight_core_ffi-osx_arm64.dylib
linux_amd64 b02253e558ed394774f136d29905ff234f6dbcde98b7a099e3ec924689ec46f9 https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.22/libduckflight_core_ffi-linux_amd64.so
linux_arm64 ad7844e310c70c100c343b60a05645cb67b9d6c487e25dcc05ab92652d0870e0 https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.22/libduckflight_core_ffi-linux_arm64.so
osx_amd64 ad0e01f7728e2080260cee695bd31730d59130aae9022a10e3be0fe64cfb6049 https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.22/libduckflight_core_ffi-osx_amd64.dylib
osx_arm64 0fcf1a1ce6547a3a69a2ee0ab601a9fdd893698f0ba75666fb932863b391ccbe https://github.com/sidequery/duckflight-extension/releases/download/core-v0.1.22/libduckflight_core_ffi-osx_arm64.dylib
2 changes: 1 addition & 1 deletion description.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
extension:
name: duckflight
description: Query DuckDB from PostgreSQL, Arrow Flight SQL, HTTP, and Quack clients
version: 0.1.17
version: 0.1.18
language: Rust
build: cargo
license: MIT
Expand Down
4 changes: 2 additions & 2 deletions docs/BUNDLED_CORE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ build must download the exact platform library before Cargo runs, verify it, and
single `.duckdb_extension`; there is no runtime download or sidecar installation. Do not use a
`latest` URL.

`core-assets.lock` pins the four Linux and macOS `core-v0.1.21` payloads used by the standard
`core-assets.lock` pins the four Linux and macOS `core-v0.1.22` payloads used by the standard
`make release` invoked by DuckDB's reusable CI. The downloaded library stays under the ignored
`build/` tree and is embedded into the final extension; it is never committed or installed on the
user's machine as a sidecar. Release smoke tests load that embedded production core. Mock lifecycle
Expand All @@ -72,7 +72,7 @@ uv run test/release/observability.py build/release/duckflight.duckdb_extension
For DuckDB 2.0, build the native core against the final source pin
`d54cbbe3df17e8e5ce68312f483b4155836112aa`. The matching core accepts the initial
`v2.0.0-alpha44357` label and the final `v2.0.0` label. `core-assets.lock` selects
the 2.0 payloads from `core-v0.1.21`. The Docker image still runs DuckDB 1.5.6
the 2.0 payloads from `core-v0.1.22`. The Docker image still runs DuckDB 1.5.6
and selects its existing `core-v0.1.13` payloads from `docker/core-assets.lock`.
Preserve the old Community ref
under `repo.variegata` if an explicit 1.5.6 binary update is needed; `ref_next` no
Expand Down
145 changes: 145 additions & 0 deletions test/release/pgwire_cross_catalog.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["psycopg[binary]>=3.2,<4", "pyarrow>=18,<24"]
# ///
"""Verify the opt-in PgWire cross-catalog read setting in the actual bundled extension."""

from __future__ import annotations

import argparse
import hashlib
import secrets
import tempfile
from pathlib import Path

import psycopg
from http_quack_smoke import test_host

USER = "cross_catalog_test"
DATABASE_DOES_NOT_EXIST = "3D000"
CROSS_CATALOG_JOIN = (
"select v.id, t.id from lake_atm_views.events.revenue_view v "
"join analytics.revenue t on true"
)


def write_config(path: Path, password: str, cross_catalog_reads: bool) -> None:
salt = secrets.token_bytes(16)
digest = hashlib.pbkdf2_hmac(
"sha256", password.encode(), salt, 10_000, dklen=32
).hex()
setting = "[pgwire]\ncross_catalog_reads = true\n\n" if cross_catalog_reads else ""
path.write_text(
f"{setting}[users.{USER}]\n"
f'password_hash = "{digest}"\nsalt = {list(salt)}\niterations = 10000\n'
)
path.chmod(0o600)


def connect(address: str, password: str) -> psycopg.Connection:
server, port = address.rsplit(":", 1)
return psycopg.connect(
host=server,
port=int(port),
user=USER,
password=password,
dbname="lake_atm",
autocommit=True,
connect_timeout=5,
)


def assert_rejected(connection: psycopg.Connection, query: str) -> None:
try:
connection.execute(query).fetchall()
except psycopg.Error as error:
if error.sqlstate != DATABASE_DOES_NOT_EXIST:
raise AssertionError(
f"expected SQLSTATE {DATABASE_DOES_NOT_EXIST}, got {error.sqlstate}: {query}"
) from error
return
raise AssertionError(f"cross-catalog statement was not rejected: {query}")


def check_enabled(address: str, password: str) -> None:
with connect(address, password) as connection:
if connection.execute(CROSS_CATALOG_JOIN).fetchall() != [(42, 42)]:
raise AssertionError("cross-catalog join returned the wrong rows")
prepared = connection.execute(
CROSS_CATALOG_JOIN + " where v.id = %s", [42], prepare=True
).fetchall()
if prepared != [(42, 42)]:
raise AssertionError("prepared cross-catalog join returned the wrong rows")
# Two-part names still resolve in the connected database, not the host's.
if connection.execute("select id from analytics.revenue").fetchall() != [(42,)]:
raise AssertionError("two-part name left the connected database")
assert_rejected(connection, "select id from not_attached.events.revenue_view")
assert_rejected(
connection,
"create table analytics.copied as "
"select id from lake_atm_views.events.revenue_view",
)


def check_disabled(address: str, password: str) -> None:
with connect(address, password) as connection:
assert_rejected(connection, CROSS_CATALOG_JOIN)
if connection.execute("select id from analytics.revenue").fetchall() != [(42,)]:
raise AssertionError("default listener could not read its own database")


def run(cli: Path, extension: Path | None, repository: str | None) -> None:
with tempfile.TemporaryDirectory(
prefix="duckflight-pgwire-cross-catalog-"
) as directory:
fixtures = Path(directory)
password = secrets.token_urlsafe(24)
enabled_config = fixtures / "enabled.toml"
disabled_config = fixtures / "disabled.toml"
write_config(enabled_config, password, cross_catalog_reads=True)
write_config(disabled_config, password, cross_catalog_reads=False)
with test_host(cli, fixtures, extension, repository) as host:
if host.execute(
"select loaded, abi_version::bigint from duckflight_core_status()"
).fetchone() != (True, 1):
raise AssertionError("bundled core did not initialize")
host.execute(
"create schema analytics; "
"create table analytics.revenue as select 7 as id; "
"attach ':memory:' as lake_atm; "
"create schema lake_atm.analytics; "
"create table lake_atm.analytics.revenue as select 42 as id; "
"attach ':memory:' as lake_atm_views; "
"create schema lake_atm_views.events; "
"create view lake_atm_views.events.revenue_view as "
"select id from lake_atm.analytics.revenue"
)
for config, check in (
(enabled_config, check_enabled),
(disabled_config, check_disabled),
):
address = host.execute(
"select address from duckflight_pg_serve('127.0.0.1:0', ?)",
[str(config)],
).fetchone()[0]
try:
check(address, password)
finally:
host.execute(
"select * from duckflight_stop('pgwire', ?)", [address]
)
print(
"PASS cross-catalog reads: opt-in join, prepared parameters, and default rejection"
)


if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("cli", type=Path)
parser.add_argument("extension", type=Path, nargs="?")
parser.add_argument("--repository", help="signed extension repository prefix")
args = parser.parse_args()
if (args.extension is None) == (args.repository is None):
parser.error("provide either an extension path or --repository")
run(args.cli, args.extension, args.repository)
Loading