Repository navigation
Run duckdb_arrow statements as relations on DuckDB 2.0 - #234
Conversation
DuckDB 2.0 copies a DB-API result to export it as Arrow, builds the copy before the first batch and warns. Run statements executed with duckdb_arrow=True as relations there, so Arrow tables avoid the copy and batch readers stream again. Report errors raised by the deferred read as SQLAlchemy errors. DuckDB 1.x keeps the DB-API path.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bdb226167
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| reader = DuckDBArrowBatchReader( | ||
| self._read(cursor, fetch_reader, batch_size), cursor | ||
| ) |
There was a problem hiding this comment.
Wrap errors raised while consuming streamed batches
For a DuckDB 2.0 relation whose failure occurs after the reader is created—for example, a conversion error in a later batch—this _read call has already returned successfully, and DuckDBArrowBatchReader.read_next_batch() re-raises the later duckdb.Error unchanged. The same query on DuckDB 1.x fails through SQLAlchemy as DataError, so streamed results now expose a version- and batch-position-dependent exception API; route errors from read_next_batch() and read_all() through the same SQLAlchemy handler.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9b61ecb7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - When another statement, a commit or a rollback runs before the result is | ||
| read, the result is read first into Python rows, as for any other result. |
There was a problem hiding this comment.
Correct the documented rollback behavior
On DuckDB 2.0, rolling back before this relation result is read does not materialize it into Python rows: ConnectionWrapper.rollback() calls pending.close(), and CursorWrapper.close() drops _relation. Consequently, the deferred statement never runs and the result cannot subsequently be read, which directly contradicts this guidance and matters for queries with Python-UDF side effects or runtime errors.
Useful? React with 👍 / 👎.
Summary
DuckDB 2.0 copies every row of a DB-API result to export it as Arrow, builds that copy before the first batch of a reader, and emits a
UserWarning. Through the dialect that meantresult.batches()no longer streamed andresult.arrowused about twice the memory.On DuckDB 2.0, a statement executed with
duckdb_arrow=Truenow runs as a DuckDB relation (connection.sql()), which exports Arrow directly. DuckDB 1.x keeps the DB-API path unchanged. This ports the approach of the earlierduckdb-2.0-arrow-and-streamingbranch onto currentmain.Peak memory for a 20 million row read through the dialect, measured locally:
result.batches()result.arrowBehavior on DuckDB 2.0 with
duckdb_arrow=True:result.arroworresult.batches(), wrapped as the SQLAlchemy errorexecute()raises on 1.x.INSERT ... RETURNING, run once inexecute(). DML withoutRETURNINGkeeps the DB-API path and its row count.fetch_df_chunk()is not available.The
duckdb<2requirement is unchanged in this PR.Tests
New end-to-end tests in
test_arrow_results.py. The warning and streaming tests fail onmainunder DuckDB 2.0.0. Full suite: