Repository navigation
Handle abandoned results on DuckDB 2.0 and buffer before raw cursor calls - #230
Conversation
DuckDB 2.0 refuses every call except execute() while a result is unfinished. A result whose cursor was closed or dropped after a partial read stayed unfinished, so register(), large executemany INSERTs, insert_from_arrow and raw driver calls failed. Discard it with an empty statement first, and test DuckDB 2.0.0 in a required CI job.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a3389295d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| finished. An empty statement discards it the way the next statement | ||
| would, without ``abort()``, which can fail an open transaction. | ||
| """ | ||
| self.__c.execute("") |
There was a problem hiding this comment.
Clear abandoned results before commit and rollback
On DuckDB 2.0, if a partly read result is closed or dropped and the next operation is Connection.commit(), Connection.rollback(), or the pool's reset rollback, these paths still call the native commit()/rollback() directly without _discard_native_result(). Because closing the cursor clears _pending_cursor without finishing the native result, _buffer_pending_result() cannot help and the transaction boundary can still fail with "connection has an unfinished result"; apply the discard before these native transaction calls as well.
Useful? React with 👍 / 👎.
| self._buffer_pending_result() | ||
| self._discard_native_result() |
There was a problem hiding this comment.
Clear abandoned results before DBAPI executemany
When DuckDB 2.0 follows an abandoned result with a direct executemany operation, such as conn.exec_driver_sql("INSERT ... VALUES (?)", [(1,), (2,)]), SQLAlchemy reaches CursorWrapper.executemany(), which invokes the native self.__c.executemany() without this new discard hook. DuckDB rejects that call for the same unfinished-result reason, so the fix currently covers the register-based large-insert path but not ordinary DBAPI executemany.
Useful? React with 👍 / 👎.
|
|
||
| ### Bug Fixes | ||
|
|
||
| - On DuckDB 2.0, `exec_driver_sql("register", ...)`, a large `executemany` INSERT, `insert_from_arrow` and calls on `connection.driver_connection` such as `sql()` failed with "connection has an unfinished result" when an earlier result on the same connection had been closed or dropped after only part of it was read. The dialect now discards such a result first, as DuckDB 1.x does. DuckDB 2.0 remains outside the supported range of this package until the `duckdb<2` requirement is lifted. |
There was a problem hiding this comment.
Document the result-discard behavior in breaking changes
This entry explicitly describes a user-visible change in which several operations stop raising an unfinished-result error, but docs/breaking-changes.md still says "None yet" under Unreleased. Add the required entry there, including who is affected and any action needed, so the behavior-change index remains complete.
AGENTS.md reference: AGENTS.md:L39-L41
Useful? React with 👍 / 👎.
…sor calls An unconditional discard also dropped the result of a raw driver_connection.execute(). Track when a cursor is closed or dropped with unread rows and discard only then. Route raw calls on a DBAPI cursor through the connection wrapper so they buffer an open result instead of cutting it short.
Summary
DuckDB 2.0 refuses
register()and running a relation while the connection holds an unfinished result. A result whose cursor was closed or dropped after reading only part of it stayed unfinished, so these failed with "connection has an unfinished result":exec_driver_sql("register", ...)executemanyINSERT that reachesduckdb_copy_thresholdinsert_from_arrowregister()andsql()onconnection.driver_connectionThe connection wrapper now records when a cursor is closed or dropped with unread rows and discards that result with an empty statement before the next raw call, which is what the next ordinary statement already does.
abort()is avoided because it can fail an open transaction. A result the dialect does not track, such as one fromdriver_connection.execute(), is left alone.Raw calls on a DBAPI cursor (
connection.connection.cursor().register(...)) now go through the connection wrapper. They previously skipped result buffering, so on DuckDB 1.x a SQLAlchemy result still being read lost every row after its current 2048-row vector.The strict xfail on
test_register_returns_no_rowsis removed, and a requiredduckdb_2_0CI job installs DuckDB 2.0.0 over theduckdb<2cap. The cap itself is unchanged here.Tests
New end-to-end tests in
test_connection_lifecycle.py. Full suite locally: