Repository navigation

Rebuild Python UDF with typed revisions and Arrow Flight execution - #29152

Open

iamlinjunhong wants to merge 236 commits into

matrixorigin:mainfrom

iamlinjunhong:m-28132

Open

Rebuild Python UDF with typed revisions and Arrow Flight execution#29152
iamlinjunhong wants to merge 236 commits into
matrixorigin:mainfrom
iamlinjunhong:m-28132

Commits on Oct 20, 2026
- refactor(udf): replace legacy Python service with Arrow runtime
  iamlinjunhong
  committed
  5fb9c48
- fix(udf): tighten runtime naming and lifecycle
  iamlinjunhong
  committed
  759203a
- fix(udf): rename Python adapter and fence duplicate calls
  iamlinjunhong
  committed
  a3f9e75
- fix(udf): close remaining Python worker contracts
  iamlinjunhong
  committed
  ad81e73
- fix(udf): align SQL selection and system account fences
  iamlinjunhong
  committed
  1b4c9a0
- test(udf): add Python cases to CI BVT
  iamlinjunhong
  committed
  d2d04e2
- chore(udf): remove stale Python service ignore rules
  iamlinjunhong
  committed
  df36924
- ci(udf): harden Python BVT build fallback
  iamlinjunhong
  committed
  dff746c
- test(udf): run Python cases in ordinary BVT
  iamlinjunhong
  committed
  e8ec0af
- test(udf): expand Python boundary BVT coverage
  iamlinjunhong
  committed
  35a4eda
- test(udf): cover null multi-argument calls
  iamlinjunhong
  committed
  c60856a
- fix(udf): close Python execution and type contracts
  iamlinjunhong
  committed
  7d9ccc1
- fix(udf): separate active and terminal reservations
  iamlinjunhong
  committed
  7f3e143
- test(udf): update decimal normalization baseline
  iamlinjunhong
  committed
  a1f4e4e
- fix(udf): harden handler response ownership
  iamlinjunhong
  committed
  84274b9
- refactor(udf): clarify execution sequence ownership
  iamlinjunhong
  committed
  3263573
- fix(udf): encode UUID width in the ABI fingerprint
  iamlinjunhong
  committed
  19548b8
- fix(udf): close Python execution channel contracts
  iamlinjunhong
  committed
  31172f0
- fix(udf): close Python gateway response contracts
  iamlinjunhong
  committed
  d26854d
- fix(udf): make Python runtime shutdown terminal
  iamlinjunhong
  committed
  cecd783
- fix(udf): close Python SQL and runtime contracts
  iamlinjunhong
  committed
  1ecf7cd
- fix(udf): require complete Python type contracts
  iamlinjunhong
  committed
  9646b5a
- fix(udf): enforce one-shot group state transitions
  iamlinjunhong
  committed
  2faf020
- fix(udf): reject late Flight completion controls
  iamlinjunhong
  committed
  dc62118
- fix(udf): align Python JSON validation with SQL
  iamlinjunhong
  committed
  68209f0
- fix(udf): isolate Python handler environment
  iamlinjunhong
  committed
  83e39c6
- fix(udf): enforce complete Python routine bodies
  iamlinjunhong
  committed
  2e7914b
- fix(udf): reject non-finite handler deadlines
  iamlinjunhong
  committed
  2baa94f
- fix(udf): validate Python execution input shapes
  iamlinjunhong
  committed
  32a83f1
- fix(udf): validate Arrow physical descriptor limits
  iamlinjunhong
  committed
  463d191
- fix(udf): ignore null decimal payloads
  iamlinjunhong
  committed
  1c1c25e
- fix(udf): honor SQL integer type equivalence
  iamlinjunhong
  committed
  5b7821d
- fix(udf): reject duplicate invocation opens
  iamlinjunhong
  committed
  2920ed5
- fix(udf): reject unknown control envelope fields
  iamlinjunhong
  committed
  aa129a4
- fix(udf): reject duplicate control JSON fields
  iamlinjunhong
  committed
  3cdda23
- fix(udf): validate opens before ledger admission
  iamlinjunhong
  committed
  88de53c
- fix(udf): enforce Python descriptor shape
  iamlinjunhong
  committed
  8289afe
- fix(udf): reject Arrow bodies on control frames
  iamlinjunhong
  committed
  12153f5
- fix(udf): align Python fencing tuple validation
  iamlinjunhong
  committed
  bcdab6f
- fix(udf): reject empty Flight input frames
  iamlinjunhong
  committed
  129d85c
- fix(udf): encode zero sequence control fields
  iamlinjunhong
  committed
  19ebace
- fix(udf): enforce control field ownership
  iamlinjunhong
  committed
  7ee1fcc
- fix(udf): preserve decimal64 signed range
  iamlinjunhong
  committed
  6920ff7
- fix(udf): reject nonstandard control JSON
  iamlinjunhong
  committed
  c399060
- fix(udf): preserve fencing tuple encoding
  iamlinjunhong
  committed
  0ee5504
- fix(udf): validate IANA timezone identities
  iamlinjunhong
  committed
  ba7b6da
- fix(udf): validate input Arrow values
  iamlinjunhong
  committed
  01731f8
- fix(udf): reject empty constant inputs
  iamlinjunhong
  committed
  6f9e880
- fix(udf): classify malformed worker frames
  iamlinjunhong
  committed
  f484b47
- fix(udf): enforce descriptor value domains
  iamlinjunhong
  committed
  fb390eb
- fix(udf): reject invalid control UTF-8
  iamlinjunhong
  committed
  d2ed6d1
- fix(udf): validate every Python input batch
  iamlinjunhong
  committed
  9ecc6ad
- feat(udf): implement long-term Python UDF execution
  iamlinjunhong
  committed
  3b2f214
- feat(udf): validate Python definitions before publication
  iamlinjunhong
  committed
  51d804a
- fix(udf): read handler frames across pipe short reads
  iamlinjunhong
  committed
  46aeaab
- fix(udf): keep Arrow encoder cleanup retryable
  iamlinjunhong
  committed
  9e55ddd
- fix(udf): finalize input before terminal success
  iamlinjunhong
  committed
  8589482
- fix(udf): classify invalid handler buffers
  iamlinjunhong
  committed
  d61c737
- fix(udf): reject trailing handler batch bytes
  iamlinjunhong
  committed
  7c82519
- fix(udf): validate Python handler bindings at create
  iamlinjunhong
  committed
  b8c2e37
- fix(udf): validate empty and typed input boundaries
  iamlinjunhong
  committed
  ebafffd
- fix(udf): reject unsupported routine types before allocation
  iamlinjunhong
  committed
  2091c26
- fix(udf): enforce exact external routine selection domain
  iamlinjunhong
  committed
  fbcc227
- fix(udf): transfer external expression result ownership
  iamlinjunhong
  committed
  0c7d5a1
- fix(udf): verify frozen definitions before evaluating arguments
  iamlinjunhong
  committed
  f3f0529
- fix(udf): skip arguments outside the selected row domain
  iamlinjunhong
  committed
  c15c6ab
- fix(udf): preserve canonical JSON tokens across adapters
  iamlinjunhong
  committed
  4dcd266
- fix(udf): avoid re-expanding shared routine query dependencies
  iamlinjunhong
  committed
  1921c50
- fix(udf): own handler pipes before subsequent allocations
  iamlinjunhong
  committed
  f15d137
- docs(udf): describe CI workers without isolation claims
  iamlinjunhong
  committed
  9730cf8
- test(udf): cover canonical JSON through real Flight batches
  iamlinjunhong
  committed
  b870b7b
- perf(udf): reduce scalar batch conversion overhead
  iamlinjunhong
  committed
  33917e7
- refactor(udf): share scalar conversion contract
  iamlinjunhong
  committed
  61711c3
- perf(udf): trim handler process startup
  iamlinjunhong
  committed
  5ae9d24
- perf(udf): trim bounded handler IPC frames
  iamlinjunhong
  committed
  96b4e54
- perf(udf): avoid Arrow result stream copies
  iamlinjunhong
  committed
  3096ba3
- perf(udf): batch fixed-width input materialization
  iamlinjunhong
  committed
  5c9184e
- perf(udf): reuse variable-width input wrappers
  iamlinjunhong
  committed
  e2d0831
- fix(udf): guard empty variable-width inputs
  iamlinjunhong
  committed
  4c9b7f6
- perf(udf): reduce handler request copies
  iamlinjunhong
  committed
  7499a21
- refactor(udf): clarify varlena cache naming
  iamlinjunhong
  committed
  664d73f
- fix(udf): handle vector inputs in batch sizing
  iamlinjunhong
  committed
  9345317
- perf(udf): reuse result headers during decode
  iamlinjunhong
  committed
  d2471f3
- fix(udf): preserve Arrow metadata across input batches
  iamlinjunhong
  committed
  778a757
- fix(udf): fence supervisor restart during close
  iamlinjunhong
  committed
  ae71137
- fix(udf): require explicit Python invocation language
  iamlinjunhong
  committed
  a9bc0b3
- fix(udf): require explicit Python definition language
  iamlinjunhong
  committed
  fc0129b
- fix(udf): validate final Python handler binding
  iamlinjunhong
  committed
  f66ce0e
- fix(udf): fall back when handler group kill is denied
  iamlinjunhong
  committed
  d3c8a1f
- fix(udf): include timezone data in runtime image
  iamlinjunhong
  committed
  1ef31f7
- fix(udf): fail closed on broken liveness watches
  iamlinjunhong
  committed
  89d6fdd
- fix(udf): close gateway admission barrier
  iamlinjunhong
  committed
  0271d07
- fix(udf): validate empty selection domains
  iamlinjunhong
  committed
  0665568
- fix(udf): reject noncanonical JSON results
  iamlinjunhong
  committed
  f06c3fe
- fix(udf): reject opaque text descriptors
  iamlinjunhong
  committed
  635e20a
- fix(udf): serialize admission with gateway close
  iamlinjunhong
  committed
  85fd878
- fix(udf): enforce budget during handler writes
  iamlinjunhong
  committed
  30d9b55
- fix(udf): validate Python catalog signature before publish
  iamlinjunhong
  committed
  dfdc201
- fix(udf): refresh worker lease for readiness
  iamlinjunhong
  committed
  5d9bd7f
- fix(udf): validate temporal child validity
  iamlinjunhong
  committed
  0af9c2a
- fix(udf): scope worker group fences by account
  iamlinjunhong
  committed
  0316825
- fix(udf): scope gateway groups by account
  iamlinjunhong
  committed
  848b17e
- fix(udf): reject noncanonical descriptor fields
  iamlinjunhong
  committed
  1b559e9
- fix(udf): bound worker fencing identities
  iamlinjunhong
  committed
  7863fd6
- fix(udf): align Python security IDs with uint32
  iamlinjunhong
  committed
  cb43ace
- fix(udf): enforce required worker controls on encode
  iamlinjunhong
  committed
  b5fc66a
- fix(udf): bound Python statement timestamps
  iamlinjunhong
  committed
  ea762ab
- fix(udf): bound Python descriptor integers
  iamlinjunhong
  committed
  1f05f7c
- fix(udf): verify runtime result production
  iamlinjunhong
  committed
  7ce73ab
- fix(udf): reject boolean contract versions
  iamlinjunhong
  committed
  1f62ade
- fix(udf): require canonical timezone kinds
  iamlinjunhong
  committed
  2a384d7
- fix(udf): validate definition schema type
  iamlinjunhong
  committed
  959466b
- fix(udf): cancel exchanges during worker shutdown
  iamlinjunhong
  committed
  02b7a32
- fix(udf): require canonical routine language
  iamlinjunhong
  committed
  0d1fc74
- fix(udf): reject noncanonical plan languages
  iamlinjunhong
  committed
  d320a42
- test(udf): inspect scoped gateway group fences
  iamlinjunhong
  committed
  5b63c63
- fix(udf): align gateway timeout with worker limit
  iamlinjunhong
  committed
  764ee76
- fix(udf): preserve large JSON number tokens
  iamlinjunhong
  committed
  9e30903
- fix(udf): reject noncanonical routine language at bind
  iamlinjunhong
  committed
  770dc93
- fix(udf): enforce Arrow field nullability
  iamlinjunhong
  committed
  f704e86
- fix(udf): require object invocation payload
  iamlinjunhong
  committed
  b981349
- fix(udf): reject cancelled worker actions
  iamlinjunhong
  committed
  649e19e
- fix(udf): reject noncanonical catalog languages
  iamlinjunhong
  committed
  7d24c5d
- fix(udf): validate module handler bindings
  iamlinjunhong
  committed
  c47ce32
- fix(udf): bound resolved artifact sources
  iamlinjunhong
  committed
  3aa3923
- fix(udf): align statement timestamp bounds
  iamlinjunhong
  committed
  d1b7517
- fix(udf): handle bare except in handler validation
  iamlinjunhong
  committed
  4f65280
- fix(udf): bound artifact validation resolution
  iamlinjunhong
  committed
  6c14a38
- fix(udf): inspect mapping pattern handler captures
  iamlinjunhong
  committed
  554d646
- fix(udf): stop cleanup reaper on worker shutdown errors
  iamlinjunhong
  committed
  9e4d217
- fix(udf): reject null sql mode context
  iamlinjunhong
  committed
  1f7abb0
- fix(udf): validate IANA zones at CN boundary
  iamlinjunhong
  committed
  379b48d
- fix(udf): cap artifact store to wire contract
  iamlinjunhong
  committed
  24ae998
- fix(udf): bound control JSON nesting
  iamlinjunhong
  committed
  3828c84
- fix(udf): bound validation action JSON
  iamlinjunhong
  committed
  b6e7f4d
- fix(udf): reject sub-minute session offsets
  iamlinjunhong
  committed
  c9637df
- fix(frontend): stabilize Python replace validation
  iamlinjunhong
  committed
  549c6ae
- fix(frontend): reject incomplete Python identities
  iamlinjunhong
  committed
  8e0ca9d
- fix(udf): cancel exchange before releasing admission
  iamlinjunhong
  committed
  756fc25
- fix(udf): keep native Flight readers on RPC thread
  iamlinjunhong
  committed
  d145af5
- fix(udf): distinguish expected worker shutdown
  iamlinjunhong
  committed
  aded525
- fix(udf): complete handler response frame writes
  iamlinjunhong
  committed
  bd29947
- fix(udf): validate handler binding before loading
  iamlinjunhong
  committed
  eb8e249
- fix(udf): kill handler groups before reaping leaders
  iamlinjunhong
  committed
  7bb0302
- fix(udf): reject stale result acknowledgements
  iamlinjunhong
  committed
  e8db98b
- fix(udf): close Docker contract gaps
  iamlinjunhong
  committed
  2fc1c4c
- refactor(udf): centralize logical type mapping
  iamlinjunhong
  committed
  5c13d1b
- fix(udf): preserve legacy python heads on restore
  iamlinjunhong
  committed
  7654c31
- fix(udf): enforce descriptor validity at result boundaries
  iamlinjunhong
  committed
  d8c5979
- fix(udf): cancel gateway operations on close
  iamlinjunhong
  committed
  110ca0f
- fix(udf): reject incompatible vector descriptor casts
  iamlinjunhong
  committed
  561888f
- fix(udf): bound handler response buffering
  iamlinjunhong
  committed
  2a4b614
- udf: add runtime status bridge and finish fencing
  iamlinjunhong
  committed
  151942d
- test: add Arrow compute correctness cases for Python UDF
  iamlinjunhong
  committed
  01956b5
- test: make Arrow sort null placement explicit
  iamlinjunhong
  committed
  f8d602b
- test: cover null Arrow take indices
  iamlinjunhong
  committed
  59468a0
- test: make Arrow arithmetic scalar type explicit
  iamlinjunhong
  committed
  d50d123
- test: make Arrow string options explicit
  iamlinjunhong
  committed
  ccce352
- test: make Arrow selection safety explicit
  iamlinjunhong
  committed
  b4e0a0f
- test: cover Arrow vector cardinality rejection
  iamlinjunhong
  committed
  11375ae
- test: distinguish Arrow string option semantics
  iamlinjunhong
  committed
  6674769
- test: bind Arrow scalar arguments to SQL types
  iamlinjunhong
  committed
  90b2e72
- docs: describe Arrow cardinality coverage
  iamlinjunhong
  committed
  cbf5c56
- test: clean up BVT service processes
  iamlinjunhong
  committed
  c8d9f18
- test: reap BVT child services
  iamlinjunhong
  committed
  d8cbfe8
- test: wait for Python UDF capability handshake
  iamlinjunhong
  committed
  668deb5
- test: cover Arrow take bounds errors
  iamlinjunhong
  committed
  fd51981
- test: verify Arrow sort index positions
  iamlinjunhong
  committed
  bddc3cc
- test: cover Arrow float arithmetic
  iamlinjunhong
  committed
  62b7942
- fix: preserve vector size during Python UDF binding
  iamlinjunhong
  committed
  5d05271
- fix: keep BVT services alive for test runner
  iamlinjunhong
  committed
  5825468
- fix: clean BVT services after test completion
  iamlinjunhong
  committed
  ac1ff1d
- fix: harden BVT launcher argument handling
  iamlinjunhong
  committed
  ca8ebe7
- fix: freeze complete Arrow result frames
  iamlinjunhong
  committed
  46f3aff
- fix: verify BVT service liveness during startup
  iamlinjunhong
  committed
  fc4c661
- docs: fix Python UDF BVT generation example
  iamlinjunhong
  committed
  f810ef5
- test: make compose worker readiness verify capabilities
  iamlinjunhong
  committed
  e6408a9
- fix: clean BVT services on startup failure
  iamlinjunhong
  committed
  2c38620
- ci: preserve BVT logs when setup fails
  iamlinjunhong
  committed
  868647b
- fix: avoid waiting on exited BVT processes
  iamlinjunhong
  committed
  01f7a7e
- fix: keep handler quota ownership retryable
  iamlinjunhong
  committed
  89d32ad
- fix: degrade python runtime initialization failures
  iamlinjunhong
  committed
  95a56fe
- fix: clean up compose BVT services
  iamlinjunhong
  committed
  9912ae6
- test: execute scalar Python UDF BVT case
  iamlinjunhong
  committed
  e4c8046
- fix(udf): validate overload namespaces and cancel capability waits
  iamlinjunhong
  committed
  003a23c
- test(udf): supply NULL input for scalar control-flow assertion
  iamlinjunhong
  committed
  3a78bee
- fix(udf): preserve exact DROP and failed initialization ownership
  iamlinjunhong
  committed
  1d8f8c4
- docs(udf): document execution and rollout scope for review
  iamlinjunhong
  committed
  b8f7db9
- fix(ci): add missing license headers for Python UDF files
  iamlinjunhong
  committed
  5e41baa
- fix(udf): satisfy static checks and import ownership
  iamlinjunhong
  committed
  834a247
- fix(ci): align Python UDF runtime dependencies
  iamlinjunhong
  committed
  c5a67de
- fix(udf): integrate concurrent CI contract repair
  iamlinjunhong
  committed
  6f2f2af
- fix(udf): remove duplicate frontend import
  iamlinjunhong
  committed
  a4b8645
- fix(udf): preserve Python UDF test dependencies
  iamlinjunhong
  committed
  3797d03
- test(optools): isolate Python dependency probe in scheduler harness
  iamlinjunhong
  committed
  5a5c723
- test(bootstrap): preserve historical upgrade worker fixture
  iamlinjunhong
  committed
  a840613
- test(udf): cover CI contracts and failure paths
  iamlinjunhong
  committed
  dbacc0d
- fix(optools): retain Python runtime loader path
  iamlinjunhong
  committed
  e2a74d3
- docs(udf): clarify test-stage acceptance scope
  iamlinjunhong
  committed
  15c1e1b
- docs(udf): approve current-stage design contract
  iamlinjunhong
  committed
  adc39d3
- docs(udf): link design approval record
  iamlinjunhong
  committed
  c7a694f
- docs(udf): record current-stage design approvals
  iamlinjunhong
  committed
  a9b88a0
- fix(optools): keep watchdog diagnostics out of UT report
  iamlinjunhong
  committed
  3dd204b
- fix(udf): preserve recovered catalog and protocol repair
  iamlinjunhong
  committed
  8c12668
- fix(udf): close catalog and execution protocol admission
  iamlinjunhong
  committed
  9aebb63
- fix(udf): close catalog repair admission
  iamlinjunhong
  committed
  adfc39f
- fix(udf): regenerate merged mysql parser
  iamlinjunhong
  committed
  645293c
- test(bootstrap): pin historical compensation fixtures
  iamlinjunhong
  committed
  941b95a
- test(udf): clean rebase upgrade imports
  iamlinjunhong
  committed
  38952b6
- fix(udf): preserve routine catalogs across restore
  iamlinjunhong
  committed
  eb61670
- test(frontend): mock routine catalog restore schema
  iamlinjunhong
  committed
  82ebe9e
- docs(udf): clarify independent approval status
  iamlinjunhong
  committed
  118cd03
- fix(udf): assign fresh protocol after main rebase
  iamlinjunhong
  committed
  4289e57
- fix(udf): preserve official catalog migrations and add v4.0.14 repair
  iamlinjunhong
  committed
  4633434
- fix(udf): allocate protocol after upstream rebase
  iamlinjunhong
  committed
  11437a6
- optools: target embedded watchdog in stderr regression test
  iamlinjunhong
  committed
  e215100
- fix(udf): allocate MORPC 110 after upstream rebase
  iamlinjunhong
  committed
  bde3f6c
- test(optools): isolate embedded watchdog fixture by platform
  iamlinjunhong
  committed
  843eeea
- fix(udf): repair revision restore and preserve published upgrade handlers
  iamlinjunhong
  committed
  1570bc8
- fix(udf): preserve protocol and integer contracts after rebase
  iamlinjunhong
  committed
  4266491
- docs(udf): record r8 contract after upstream rebase
  iamlinjunhong
  committed
  596f137
- fix(udf): reconcile Python protocol with index search scans
  iamlinjunhong
  committed
  af47343
- docs(udf): align active protocol references with MORPC 112
  iamlinjunhong
  committed
  8f1e0ac
- test: isolate embedded admission release channels after main rebase
  iamlinjunhong
  committed
  5cd61c6
- fix(udf): advance Python protocol admission after calendar contracts
  iamlinjunhong
  committed
  6c23e52
- fix(udf): allocate MORPC 114 after main rebase
  iamlinjunhong
  committed
  97a1833
- fix(udf): allocate MORPC 115 and refresh planner fixtures
  iamlinjunhong
  committed
  946cfb4
- fix(udf): allocate MORPC 116 after upstream rebase
  iamlinjunhong
  committed
  108eef3
- fix(udf): allocate MORPC 117 after upstream rebase
  iamlinjunhong
  committed
  d540a15
- fix(udf): allocate MORPC 118 and isolate Python test helpers
  iamlinjunhong
  committed
  ffd3524
- fix(udf): preserve catalog 4.0.17 and add 4.0.18 repair
  iamlinjunhong
  committed
  8cd2cc9
- fix(udf): advance Python capability after session control rebase
  iamlinjunhong
  committed
  2acd01e
- test(ut): synchronize embedded admission with readiness barriers
  iamlinjunhong
  committed
  d8d9715
- fix(udf): allocate MORPC 122 after DECIMAL DIV rebase
  iamlinjunhong
  committed
  1e4ed38
- fix(udf): preserve temporal epoch and advance Python admission
  iamlinjunhong
  committed
  4609d98
- fix(udf): preserve nonblocking lock protocol during rebase
  iamlinjunhong
  committed
  11e57f5

Commits

Commits on Oct 20, 2026