Skip to content

Make provider streaming lazy and reuse data sources - #10

Merged
lqlive merged 1 commit into
mainfrom
feature/lazy-provider-streaming
Jun 18, 2026
Merged

lqlive merged 1 commit into
mainfrom
feature/lazy-provider-streaming

Conversation

@lqlive

@lqlive lqlive commented Jun 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • Reuse shared database clients/data sources across MySQL, PostgreSQL, SQLite, ClickHouse, and MongoDB providers.
  • Defer provider connection/query execution until the first record batch is read.
  • Export native DataFusion streams lazily through Arrow C Data Interface instead of collecting all batches up front.

Test plan

  • dotnet build src/Apache.DataFusion.TableProviders.MySql/Apache.DataFusion.TableProviders.MySql.csproj -c Debug --nologo -v q
  • dotnet build src/Apache.DataFusion.TableProviders.PostgreSql/Apache.DataFusion.TableProviders.PostgreSql.csproj -c Debug --nologo -v q
  • dotnet build src/Apache.DataFusion.TableProviders.Sqlite/Apache.DataFusion.TableProviders.Sqlite.csproj -c Debug --nologo -v q
  • dotnet build src/Apache.DataFusion.TableProviders.ClickHouse/Apache.DataFusion.TableProviders.ClickHouse.csproj -c Debug --nologo -v q
  • dotnet test tests/Apache.DataFusion.Tests/Apache.DataFusion.Tests.csproj --nologo -v q

Defer provider query execution until batches are read and reuse shared database clients/data sources so stream setup stays lightweight and memory bounded.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lqlive
lqlive merged commit 4b82aa8 into main Jun 18, 2026
4 checks passed
@lqlive
lqlive deleted the feature/lazy-provider-streaming branch June 18, 2026 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant