[Feature Request]: Add managed file exploration to the Python SDK #30048

Is there an existing issue for the same feature request?

• Checked existing issues for Python SDK file exploration and Parquet support.

Is your feature request related to a problem?

Users exploring Parquet, CSV, or JSON Lines data currently need to infer a schema, hand-write external-table DDL, assemble multi-file projections, and reclaim catalog objects themselves. A notebook workflow needs a relation that can be queried before importing the data, while retaining MatrixOne SQL execution and access controls.

Describe the feature you'd like

Provide optional synchronous Python SDK APIs client.analytics.read_parquet, read_csv, and read_json that return a managed file relation.

Requirements:

1. Infer supported scalar schemas or accept an explicit schema, and reject unsupported or potentially lossy mappings.
2. Resolve bounded local file lists/globs; support explicit remote paths through an Arrow filesystem and corresponding MatrixOne stage:// paths.
3. Support strict schema matching, union_by_name with nullable missing columns, and string-valued Hive partitions.
4. Allow SQL aggregates/filters through the relation without first importing files. Expose a checked relation SQL expression and bounded Arrow results.
5. Provide a transactional append into an existing table with strict conversion and primary-key checks through COMMIT.
6. Reclaim only relation-owned external tables on normal exit and failures; preserve primary failures and document crash recovery.
7. Detect missing sources, observed schema drift and malformed CSV records before scanning. Make the lack of an immutable source snapshot explicit.
8. Preserve existing SDK behavior when the analytics extra is not installed.

Describe implementation you've considered

A Python relation layer reuses MatrixOne native external tables, one per source file, and a UNION ALL projection. Schema discovery is performed client-side with optional PyArrow; MatrixOne performs SQL scanning and aggregation. This issue's acceptance scope is that SDK implementation. Native SQL read_parquet() table functions, remote glob discovery and file-snapshot guarantees are separate future work.

Acceptance criteria

• Query each supported format and obtain the same complete values/aggregates as an independent DuckDB oracle under the documented format conventions.
• Missing/unsupported/corrupt sources, unsafe schema promotion and precision loss fail explicitly.
• Tests cover many-file inputs, local S3-compatible sources, privileges, concurrent relations and cross-CN visibility.
• Import failures leave no partial append; normal cleanup leaves no owned external tables or checked-out connections.
• API documentation, design documents and reproducible adversarial tests accompany the implementation.

Documentation, Adoption, Use Case, Migration Strategy

with client.analytics.read_parquet('/shared/orders/*.parquet', union_by_name=True) as files:
    with files.query('SELECT count(*) AS n FROM ' + files.sql + ' AS f') as batches:
        for batch in batches:
            print(batch.to_pydict())

Install the optional analytics extra. Source files must be visible to both schema inference and the CN; remote credentials and stages remain caller-managed. No changes to existing load_data APIs are required.

Additional information

The implementation was locally validated on MatrixOne source commit cd8d5abf9665a97d13e50b04840fb74b877fd339, including two CN processes on one host and a local S3 emulator. These are not production-cloud, cross-host or published-release certifications. The implementation PR will contain the exact test matrix and limitations.

Sub-features

• [Subtask]: Design and implement file schema inference, multi-file relations and Hive partitions #30050 — Design and implement file schema inference, multi-file relations and Hive partitions
• [Subtask]: Design and implement file-relation ownership, strict import and recovery semantics #30051 — Design and implement file-relation ownership, strict import and recovery semantics

Implementation PR

#30054 — draft implementation for design review, with the four sub-feature design documents, executable tests and recorded local evidence.

The parent feature remains open for acceptance; the PR closes its implementation sub-features on merge.