Skip to content

[Feature Request]: Provide asynchronous file relations and Arrow analytics APIs #30066

Open

Description

@dengn

Existing issue check

  • Checked relevant Python SDK issues; this report concerns the SDK client path.

Environment

Repository main 3e50e1b32d8da61bbbf70c89989fb08b671809de, Python 3.12, SQLAlchemy 1.4/2.x. Initial findings were reproduced offline with synthetic inputs; real-service validation will be linked from the implementation PR.

Priority: P1. Related: #30049 and draft implementation #30054 (synchronous scope).

Problem and requirements

AsyncClient has no equivalent of the proposed synchronous analytics interface. Applications should not need blocking file/result operations on the event loop.

Provide async context-managed file relations, async Arrow batch iteration, DataFrame/Arrow upload and import with equivalent type/precision/commit-outcome contracts. Honor cancellation/deadlines and pool backpressure. Preserve optional dependencies and explicit shared-path limitations. No Flight or zero-copy promise is implied.

Design considerations

Use SQLAlchemy async streaming/aiomysql unbuffered retrieval; bound Arrow conversion and staging work and move blocking I/O off the event loop where appropriate. Retain transaction settings through COMMIT, distinguish unknown outcomes and define cancellation cleanup. Do not merely wrap complete synchronous result materialization in a thread.

Acceptance tests

Compare complete values and schemas with synchronous APIs; measure event-loop heartbeat during large transfers; cancel during staging/LOAD/read/COMMIT, exhaust a small pool, inject network stalls and verify outcome/resource ownership. Run dependency, same-host dual-CN and permissions matrices with no unexpected skips.

Implementation tracking

Design/implementation follow-up remains open. This requirement is not claimed as implemented by the current bug-fix/performance PRs. Related synchronous analytics draft: #30054.

Async connection pool configuration

Expose pool_size, max_overflow, pool_timeout and pool_recycle rather than retaining fixed testing defaults. Verify bounded backpressure and cancellation with a one-connection pool; preserve documented default behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions