Rebuild Python UDF with typed revisions and Arrow Flight execution

[iamlinjunhong]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Related to #28132. This delivers the external unisolated adapter stage; it does not close the issue's broader sandbox/production-isolation requirements.

What this PR does / why we need it:

Replace the demo's per-value protobuf transport with typed, revision-bound Python UDF execution over Arrow Flight. CN remains MO-vector-native and owns guarded expression evaluation; user code runs in external handler processes.

• Share immutable Catalog revisions, complete type descriptors and exact RoutineCall identities with SQL UDF. Validate syntax/handler contracts at CREATE/REPLACE, publish digest-addressed artifacts, and invalidate cached/prepared calls when the full overload namespace changes. DROP distinguishes exact decimal scales.
• Execute SCALAR and VECTOR handlers through ExternalRoutineEval, preserving CASE selection, NULL policy, zero-row/zero-argument calls, nested evaluation and DML error atomicity.
• Bound Gateway admission, stream windows (W=1), handler slots and terminal records. Close EndInput/result ACK/Finish semantics, independent cancellation, generation fencing and retryable cleanup after partial initialization. Container init reaps orphaned descendants.
• Enable the worker in the existing ordinary CI/BVT path, with one worker endpoint per CN in the multi-CN launch. Add correctness cases for Arrow compute, revision/overload changes and cross-account snapshot restore. Remove the old demo execution path.

Implementation and rollout contract: docs/design/python_udf.md. Namespace validation: docs/design/routine_namespace_validation.md. Local usage and cases: Python BVT README.

Current CI repair validation

The CI repair fixes the Arrow import ownership checks, preserves existing UDF SQL error codes/messages while adopting repository error wrapping, installs timezone data in both CI runtime image variants, and aligns the Python local launch FileServices with TN. Catalog result fixtures now assert the added revision table and six identity columns, including cross-account snapshot restore; Python error/result oracles remain unchanged. Real-worker test cleanup owns the process before readiness and records completion before repeated cleanup.

• macOS arm64: fresh native and service builds; owning frontend/colexec/planner/function/CN tests; UDF/protocol race tests and six real-worker race tests; SQL diagnostic regression controls.
• Fresh real SQL → Gateway → Flight → handler → MO result: 32 unique affected BVT scripts, 4004 passed, 0 failed, 22 pre-existing ignored (4026 total). Python subset 352/352. One script ran in both batches and is counted once. This current SQL run is single-CN macOS, not a claimed new Linux multi-CN run.
• Full affected-package lint: 0 issues; make err-check; CI actionlint; Python 3.12 coverage helper tests 6 passed.
• Actual CI prebuilt runtime image assembly and Python worker image both contain tzdata 2026c. The original runtime base had no timezone data. Runtime assembly validation used a cached executable; current SQL used the freshly built source binary.
• Linux current-source Go/Flight/race result: Linux amd64: final-source ordinary UDF/real-Flight and import-boundary tests passed; protocol race passed. Linux arm64 (local Docker, Go 1.26.4, Python 3.12/PyArrow 24): full Python package race 81 top-level tests passed, including all six real-worker tests, with no skips. The interrupted amd64 Python race is not counted as passed.

Existing host UT and coverage workflows use matrixorigin/CI#457 to install the checked-out worker requirements in a Python 3.12 venv and expose that interpreter to Go tests. CI #457 merged into matrixorigin/CI main at c1a30c45dfce0352d1d4fb9573dda2a0feedb4d9; this changes existing jobs only. At the pre-rebase head c560dc168944c6ed158c8db36284173c38d255b1, MatrixOne ALL CI run 35810950106 had not reached a terminal result at the last recorded check. It does not validate the rebased head. Per user request, hosted CI was not awaited; this PR update will trigger a fresh run, and no hosted result is claimed.

Latest-main rebase and current local validation

The branch was rebased onto main at ca71f51e2f962c0fbe0a92d9d01bb071bb1bb9c9 after nine new main commits made the previous PR head conflict. The merged MySQL grammar was regenerated with the repository goyacc target and its full parser package passed. Current-head validation also passed: Python UDF race, protocol race, full frontend, CN service and colexec packages; go vet; incremental golangci-lint (0 issues); and git diff --check. The official CI coverage parser, using the latest-main PR diff plus CI UT/BVT profiles and current local package profiles, reports 2,727/3,626 = 75.21%, strictly above the 75% gate. Hosted CI was not awaited, as requested.

Prior Linux integration evidence

Validated on Linux amd64 with the repository CI tester image, Go 1.26.4, Python 3.13.5 and PyArrow 24.0.0. Source included main merge 550ca2830d and repair 37889f08b7; these results predate the CI repair above and are retained as earlier deployment/semantic evidence.

• Native build and repository CGo-wrapper tests for UDF/protocol/Gateway; UDF packages with -race.
• Owning colexec, function, rule, MySQL parser and v4_0_7 upgrade packages; focused frontend Catalog/drop/revision/restore tests.
• Python worker suite: 142 passed, 29 subtests. The subsequently changed bounded orphan-reap assertion passed separately.
• Real SQL → CN Gateway → Flight → handler → MO result: Python BVT 352/352 on each of two CNs; SQL UDF 82/82; external-table clone 108/108, with no skips. Uses the repository CI's nometa mode.
• Cross-account snapshot restore executes revision A in a fresh target after replacing the source with B; checks exact restored identities/revisions/bodies, SQL UDF control and cleanup.
• Linux Supervisor worker SIGKILL: SQL error, handler/descendant reaping, no transparent replay, ordinary SQL preserved; replacement worker/lease executes a fresh call successfully.
• Actual worker image: three real Flight leader-exit cases leave no owned descendant PID; PID 1 is tini. Companion Operator command/render tests preserve init and loopback binding.

GPT-6 medium design and integrated mo-self-review covered the branch and corrected exact-DROP, partial-initialization ownership and orphan-reaping findings. Hosted CI is pending; local/remote test passes are not reported as hosted CI results.

Compatibility and remaining scope

Python is disabled in generic launch and requires explicit unisolated opt-in. Legacy demo definitions/plans are rejected and must be explicitly recreated; existing SQL UDFs remain supported. Worker/Gateway capability and tzdata/environment digests must match before execution. Operator integrations must use an init-capable worker image and preserve its init command.

This does not provide sandbox tenant isolation or authenticated/TLS Flight, imported dependency environments, W>1/cumulative ACK, or full production Operator rollout acceptance. Native Linux dual-CN tests and image tests do not claim all deployment topologies were exercised. Prior optional strict-metadata SQL checking found six header differences; those expectations were not changed to hide them, and this PR reports the existing CI comparison mode explicitly.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

mo-bot review · mo-bot-local · done

Commit: feba47948a79c8cd0f282260189716d0645aafbb

Review completed. Read the assessment.

[XuPeng-SH]

mo-bot review · mo-bot-local · review

Commit: feba47948a79c8cd0f282260189716d0645aafbb

Assessment: request_changes

The current r12 design still lacks mandatory independent Architecture and SQL/Planner approval. Keeping revision identity and namespace validation with existing catalog/planner owners is coherent, but the unresolved design gate prevents implementation acceptance. The accepted opt-in, unisolated development scope remains respected.

Model: gpt-6-astra / xhigh

Session: 01a12d8f-7dea-77c2-a565-38d45b5564e4 · slot 0 · generation 0

• P1 docs/design/python_udf_current_stage.md:8: Accepting the current r12 implementation while its independent Architecture and SQL/Planner decisions remain explicitly pending.
  Impact: This leaves the mandatory design gate unmet for shared catalog persistence, SQL compatibility, protocol admission and lifecycle changes. Disabling Python execution does not isolate these changes: bootstrap registers the 4.0.15 catalog repair unconditionally. Obtain approval identifying the exact current design revision, reviewers, scope and decisions, or a documented exception from the policy owner.
  Evidence: The current contract at line 8 and approval record at lines 211–213 explicitly mark r12 approval pending. Frozen comment 5809313638 approves historical r1 at 610d1e4; the supplied discussion contains no closing r12 decision. pkg/bootstrap/upgrade.go:79 registers v4_0_15.Handler independently of Python enablement; its tenant handler replays nine shared catalog entries after protocol 113 admission.

Validation:

• Verified the assigned head, merge-base and clean checkout; inventoried all 248 changed files. Confirmed upstream catalog 4.0.14 remains unchanged.

• Reviewed the four design documents, supplied comments/reviews and catalog, protocol and runtime activation owners. evidence.json.local_validation is empty.

• Static audit passed: checks/1791996406171198840-3.json matches design-gate-audit.log. Patch equivalence passed in checks/1791996471108559969-4.json and patch-terminal-newline.log: the retained patch differs only by one omitted final newline. The earlier literal-equality failure is retained and explained.

• Receipts, script/log hashes and scope are recorded in /home/xupeng/.local/state/mo-bot/jobs/49d1f7c98bf4c96eca541838/attempt-1/receipt-reconciliation.json.

Limitations:

• The applicable mo-self-review skill says an unapproved design “short-circuits full implementation polishing.” Exhaustive implementation, lifecycle, unhappy-path, performance and test-quality review remains outstanding; the complete inventory is not complete implementation approval.

• No runtime tests or performance measurements were executed. Frozen CI statuses were inspected, but execution logs were not reconciled and no product-test pass is claimed.

• The linked issue #28132 body is absent from the frozen evidence; its full behavioral contract was not independently verified.