[Feature Request]: Add DataFrame and Arrow interoperability to the Python SDK #30049

Is there an existing issue for the same feature request?

• Checked existing Python/Arrow SDK issues. [Refactoring]: Rebuild Python UDF on Arrow batches and an isolated runtime #28132 concerns the Python UDF runtime; this proposal concerns client-side uploads and query results.

Is your feature request related to a problem?

DataFrame/Arrow producers need a batch upload path instead of building row INSERTs. Analysis clients also need to consume large query results incrementally rather than materializing every MySQL row before constructing an Arrow table. Both paths need explicit type, transaction and cleanup guarantees.

Describe the feature you'd like

Add optional synchronous upload_dataframe, upload_arrow, and analytics.query APIs.

Requirements:

1. Accept pandas/Polars DataFrames and Arrow Table, RecordBatch and RecordBatchReader inputs.
2. Convert supported scalar types with checked precision, nullability and UTC timestamp handling; reject unsupported nested mappings.
3. Create a table or append atomically using native Arrow IPC LOAD through a bounded, shared staging path.
4. Keep strict SQL mode and TN primary-key checks active through COMMIT; restore session settings afterward.
5. Report unknown COMMIT outcomes separately from acknowledged commits followed by cleanup failures. Never automatically retry uncertain writes.
6. Stream query results as Arrow RecordBatches using an actual unbuffered cursor; derive types from protocol metadata for empty/all-NULL results.
7. Bound row-batch size and provide socket inactivity timeouts, checked result casts, early-close cleanup and connection-pool reuse.
8. Preserve the core SDK's optional-dependency behavior and existing APIs.

Describe implementation you've considered

Input: normalize Arrow batches, write an owned IPC file, issue native non-LOCAL LOAD DATA, then remove the staging file. Output: consume PyMySQL SSCursor rows in bounded batches and convert them to Arrow. Shared transaction/type helpers are also used by file exploration.

Acceptance scope is SDK interoperability over existing server interfaces. Arrow Flight, zero-copy MySQL transport, client-local upload without a shared path, asynchronous APIs and server-side query cancellation are future work.

Acceptance criteria

• Scalar edge cases round-trip accurately, including uint64 endpoints, Decimal128 precision 38, binary/Unicode, float special values and timezone-aware timestamps.
• Upload errors, late duplicates and concurrent primary-key conflicts do not partially commit.
• Dropping the COMMIT success reply reports committed=None; postcommit cleanup errors report the acknowledged state.
• Early-close, conversion and network failures release connections and retain reusable pool capacity.
• Million-row constrained-memory streaming, fault injection and crash/restart tests have inspectable results.
• Design, error-outcome documentation and reproducible tests accompany the implementation.

Documentation, Adoption, Use Case, Migration Strategy

client.analytics.upload_dataframe('features', df, staging_dir='/shared/staging')
with client.analytics.query('SELECT * FROM features', batch_size=4096) as batches:
    for batch in batches:
        consume(batch)

Install the optional analytics extra and the relevant DataFrame library. Native Arrow LOAD must be available on the server; this implementation was validated on a current source build, not certified against the released 4.2.5 binary.

Additional information

Existing buffered query APIs remain available. to_arrow() deliberately materializes the remaining result; iteration provides bounded row batches. A single very large cell can still dominate memory. Unknown commit outcomes require application reconciliation, and SIGKILL can leave caller-owned artifacts for explicit recovery.

Sub-features

• [Subtask]: Design and implement checked DataFrame/Arrow uploads with atomic commit outcomes #30052 — Design and implement checked DataFrame/Arrow uploads with atomic commit outcomes
• [Subtask]: Design and implement bounded Arrow query results, checked casts and stream cleanup #30053 — Design and implement bounded Arrow query results, checked casts and stream cleanup

Implementation PR

#30054 — draft implementation for design review, with the four sub-feature design documents, executable tests and recorded local evidence.

The parent feature remains open for acceptance; the PR closes its implementation sub-features on merge.