Skip to content

Run duckdb_arrow statements as relations on DuckDB 2.0 - #234

Merged
leonardovida merged 2 commits into
mainfrom
duckdb-2-arrow-relations
Nov 17, 2026
Merged

leonardovida merged 2 commits into
mainfrom
duckdb-2-arrow-relations

Conversation

@leonardovida

Copy link
Copy Markdown
Owner

Summary

DuckDB 2.0 copies every row of a DB-API result to export it as Arrow, builds that copy before the first batch of a reader, and emits a UserWarning. Through the dialect that meant result.batches() no longer streamed and result.arrow used about twice the memory.

On DuckDB 2.0, a statement executed with duckdb_arrow=True now runs as a DuckDB relation (connection.sql()), which exports Arrow directly. DuckDB 1.x keeps the DB-API path unchanged. This ports the approach of the earlier duckdb-2.0-arrow-and-streaming branch onto current main.

Peak memory for a 20 million row read through the dialect, measured locally:

DuckDB 1.5.6 DuckDB 2.0.0 before DuckDB 2.0.0 after
result.batches() 120 MB 933 MB 137 MB
result.arrow 789 MB 1573 MB 848 MB

Behavior on DuckDB 2.0 with duckdb_arrow=True:

  • A statement that only reads runs when its result is first read. Runtime errors are then raised by result.arrow or result.batches(), wrapped as the SQLAlchemy error execute() raises on 1.x.
  • Statements that write, including INSERT ... RETURNING, run once in execute(). DML without RETURNING keeps the DB-API path and its row count.
  • A result that is still unread when another statement, commit or rollback runs is read into Python rows first, like any other result.
  • Fetching rows from such a result reads all of it into Python rows. fetch_df_chunk() is not available.

The duckdb<2 requirement is unchanged in this PR.

Tests

New end-to-end tests in test_arrow_results.py. The warning and streaming tests fail on main under DuckDB 2.0.0. Full suite:

  • DuckDB 2.0.0, SQLAlchemy 2.1.9: 933 passed, 10 skipped. Upstream suite: 1545 passed
  • DuckDB 1.5.6, SQLAlchemy 2.0.52: 933 passed, 10 skipped. Upstream suite: 1331 passed
  • DuckDB 1.3.0, SQLAlchemy 2.0.0: 926 passed, 17 skipped

DuckDB 2.0 copies a DB-API result to export it as Arrow, builds the copy before the first batch and warns. Run statements executed with duckdb_arrow=True as relations there, so Arrow tables avoid the copy and batch readers stream again. Report errors raised by the deferred read as SQLAlchemy errors. DuckDB 1.x keeps the DB-API path.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T19:20:03.236198Z f9b61ec New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bdb226167

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +151 to +153
reader = DuckDBArrowBatchReader(
self._read(cursor, fetch_reader, batch_size), cursor
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wrap errors raised while consuming streamed batches

For a DuckDB 2.0 relation whose failure occurs after the reader is created—for example, a conversion error in a later batch—this _read call has already returned successfully, and DuckDBArrowBatchReader.read_next_batch() re-raises the later duckdb.Error unchanged. The same query on DuckDB 1.x fails through SQLAlchemy as DataError, so streamed results now expose a version- and batch-position-dependent exception API; route errors from read_next_batch() and read_all() through the same SQLAlchemy handler.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f9b61ecb7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/performance.md
Comment on lines +129 to +130
- When another statement, a commit or a rollback runs before the result is
read, the result is read first into Python rows, as for any other result.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the documented rollback behavior

On DuckDB 2.0, rolling back before this relation result is read does not materialize it into Python rows: ConnectionWrapper.rollback() calls pending.close(), and CursorWrapper.close() drops _relation. Consequently, the deferred statement never runs and the result cannot subsequently be read, which directly contradicts this guidance and matters for queries with Python-UDF side effects or runtime errors.

Useful? React with 👍 / 👎.

@leonardovida
leonardovida merged commit 33d3b6a into main Nov 17, 2026
32 checks passed
@leonardovida
leonardovida deleted the duckdb-2-arrow-relations branch November 17, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant