Skip to content

Keep SDK connections alive for streaming RPCs - #58184

Closed
mregrock wants to merge 2 commits into
ydb-platform:mainfrom
mregrock:codex/sdk-stream-connection-lifetime
Closed

mregrock wants to merge 2 commits into
ydb-platform:mainfrom
mregrock:codex/sdk-stream-connection-lifetime

Conversation

@mregrock

@mregrock mregrock commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by ydb-platform/ydb-python-sdk#938. The SDK fix and cleanup regression tests are now upstream; YDB will receive the fix through directory sync.

Bridge cleanup-test removal and removal of its SDK workaround are in #58181.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-04 20:49:08 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0c53b04 has started.
⚪ 2026-11-04 20:49:31 UTC Artifacts will be uploaded here
⚪ 2026-11-04 20:51:01 UTC ya make is running...
⚫ 2026-11-04 21:55:08 UTC Check cancelled

@mregrock
mregrock marked this pull request as ready for review November 4, 2026 21:55
@mregrock
mregrock requested a review from a team as a code owner November 4, 2026 21:55
Copilot AI balanced review requested due to automatic review settings November 4, 2026 21:55
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-04 21:56:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2b973bc has started.
⚪ 2026-11-04 21:56:53 UTC Artifacts will be uploaded here
⚪ 2026-11-04 21:58:25 UTC ya make is running...
🟡 2026-11-04 23:12:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15181 14975 0 5 184 17

⚪ 2026-11-04 23:12:46 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-11-04 23:15:49 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
142 (only retried tests) 141 0 0 0 1

🟢 2026-11-04 23:15:53 UTC Build successful.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Explicit iterator cancellation can leave calls retained and prevent channel closure, while unjoined daemon cleanup lets driver shutdown return before RPC cancellation and channel closure complete.

2 open issues
What changed in this PR

This PR updates the Python SDK’s connection lifecycle to preserve streaming RPCs during endpoint retirement and force-close current and retired connections at driver shutdown.

Changes:

  • Retain connections until streaming RPC completion and move completion-triggered and forced closure to background threads.
  • Track retired connections for shutdown and prevent stale removals from deleting replacement connections.
  • Add lifecycle regression tests and remove the bridge shutdown workaround.
File Description
ydb/​tests/​functional/​bridge/​ya.make Registers the streaming cleanup tests.
ydb/​tests/​functional/​bridge/​test_sdk_stream_cleanup.py Tests stream retention, cancellation, retirement, and shutdown.
ydb/​tests/​functional/​bridge/​test_sdk_cleanup.py Adapts shutdown checks for asynchronous channel closure.
ydb/​tests/​functional/​bridge/​common.py Removes the temporary driver cleanup workaround.
contrib/​python/​ydb/​py3/​ydb/​pool.py Tracks retired connections and makes cache removal identity-safe.
contrib/​python/​ydb/​py3/​ydb/​connection.py Retains streaming calls and adds asynchronous forced closure.
contrib/​python/​ydb/​py3/​patches/​08-retain-streaming-connections.patch Records the vendored SDK lifecycle changes.

Review effort: Balanced

Comment on lines +561 to +562
if isinstance(response, _SafeSyncIterator):
finish_call = not response.add_callback(lambda: self._finish_call(rpc_state))
Comment on lines +604 to +605
if force:
threading.Thread(target=self.destroy, daemon=True).start()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants