Skip to content

Handle abandoned results on DuckDB 2.0 and buffer before raw cursor calls - #230

Merged
leonardovida merged 2 commits into
mainfrom
fix/duckdb-2-abandoned-results
Nov 17, 2026
Merged

leonardovida merged 2 commits into
mainfrom
fix/duckdb-2-abandoned-results

Conversation

@leonardovida

@leonardovida leonardovida commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

DuckDB 2.0 refuses register() and running a relation while the connection holds an unfinished result. A result whose cursor was closed or dropped after reading only part of it stayed unfinished, so these failed with "connection has an unfinished result":

  • exec_driver_sql("register", ...)
  • an executemany INSERT that reaches duckdb_copy_threshold
  • insert_from_arrow
  • register() and sql() on connection.driver_connection

The connection wrapper now records when a cursor is closed or dropped with unread rows and discards that result with an empty statement before the next raw call, which is what the next ordinary statement already does. abort() is avoided because it can fail an open transaction. A result the dialect does not track, such as one from driver_connection.execute(), is left alone.

Raw calls on a DBAPI cursor (connection.connection.cursor().register(...)) now go through the connection wrapper. They previously skipped result buffering, so on DuckDB 1.x a SQLAlchemy result still being read lost every row after its current 2048-row vector.

The strict xfail on test_register_returns_no_rows is removed, and a required duckdb_2_0 CI job installs DuckDB 2.0.0 over the duckdb<2 cap. The cap itself is unchanged here.

Tests

New end-to-end tests in test_connection_lifecycle.py. Full suite locally:

  • DuckDB 2.0.0, SQLAlchemy 2.1.9, Python 3.13: 921 passed, 10 skipped. Upstream suite: 1545 passed
  • DuckDB 1.5.6, SQLAlchemy 2.0.52, Python 3.11: 921 passed, 10 skipped. Upstream suite: 1331 passed
  • DuckDB 1.3.0, SQLAlchemy 2.0.0, Python 3.10: 914 passed, 17 skipped

DuckDB 2.0 refuses every call except execute() while a result is unfinished. A result whose cursor was closed or dropped after a partial read stayed unfinished, so register(), large executemany INSERTs, insert_from_arrow and raw driver calls failed. Discard it with an empty statement first, and test DuckDB 2.0.0 in a required CI job.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T13:21:06.707717Z a2f5c48 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a3389295d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread duckdb_sqlalchemy/__init__.py Outdated
finished. An empty statement discards it the way the next statement
would, without ``abort()``, which can fail an open transaction.
"""
self.__c.execute("")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear abandoned results before commit and rollback

On DuckDB 2.0, if a partly read result is closed or dropped and the next operation is Connection.commit(), Connection.rollback(), or the pool's reset rollback, these paths still call the native commit()/rollback() directly without _discard_native_result(). Because closing the cursor clears _pending_cursor without finishing the native result, _buffer_pending_result() cannot help and the transaction boundary can still fail with "connection has an unfinished result"; apply the discard before these native transaction calls as well.

Useful? React with 👍 / 👎.

Comment thread duckdb_sqlalchemy/__init__.py Outdated
Comment on lines +627 to +628
self._buffer_pending_result()
self._discard_native_result()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear abandoned results before DBAPI executemany

When DuckDB 2.0 follows an abandoned result with a direct executemany operation, such as conn.exec_driver_sql("INSERT ... VALUES (?)", [(1,), (2,)]), SQLAlchemy reaches CursorWrapper.executemany(), which invokes the native self.__c.executemany() without this new discard hook. DuckDB rejects that call for the same unfinished-result reason, so the fix currently covers the register-based large-insert path but not ordinary DBAPI executemany.

Useful? React with 👍 / 👎.

Comment thread CHANGELOG.md Outdated

### Bug Fixes

- On DuckDB 2.0, `exec_driver_sql("register", ...)`, a large `executemany` INSERT, `insert_from_arrow` and calls on `connection.driver_connection` such as `sql()` failed with "connection has an unfinished result" when an earlier result on the same connection had been closed or dropped after only part of it was read. The dialect now discards such a result first, as DuckDB 1.x does. DuckDB 2.0 remains outside the supported range of this package until the `duckdb<2` requirement is lifted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document the result-discard behavior in breaking changes

This entry explicitly describes a user-visible change in which several operations stop raising an unfinished-result error, but docs/breaking-changes.md still says "None yet" under Unreleased. Add the required entry there, including who is affected and any action needed, so the behavior-change index remains complete.

AGENTS.md reference: AGENTS.md:L39-L41

Useful? React with 👍 / 👎.

…sor calls

An unconditional discard also dropped the result of a raw driver_connection.execute(). Track when a cursor is closed or dropped with unread rows and discard only then. Route raw calls on a DBAPI cursor through the connection wrapper so they buffer an open result instead of cutting it short.
@leonardovida leonardovida changed the title Discard abandoned results before raw driver calls on DuckDB 2.0 Handle abandoned results on DuckDB 2.0 and buffer before raw cursor calls Nov 17, 2026
@leonardovida
leonardovida merged commit 6c82656 into main Nov 17, 2026
32 checks passed
@leonardovida
leonardovida deleted the fix/duckdb-2-abandoned-results branch November 17, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant