[Feature Request]: Embed Sirius in MatrixOne through CGo and retire Flight offload #28966

Current scope and status

Embed Sirius and DuckDB in mo-service through C ABI/CGo, using the pinned third_party/sirius submodule and frozen Pixi GPU environment. MO #29547's bounded MO-reader/result bridge and Native A/B are merged. Exact semantic authority remains #29449 document blob 42a89f09a1d168d02b9583cb3ea7b4de6dbb5634; #29690 records the approved delivery sequence.

Status updated 2026-10-12: D #29862 merged as 7af631bdf79e7beb4bb3588e2bfd03ba4852b6cc. Its complete 744 timing/correctness and 264 resource executions retain their original MO/native/SDK provenance in the immutable evidence archive; generated captures remain outside the current source tree. The final CI fixture corrections are included in the merged D.

The approved batching follow-up is Sirius #31, merged as b953bb5a87788e17557929987208b3a0068d933b. It restores bounded fill-to-target scheduling with EOS/credit-pressure flushing and charged immutable summaries. Matched premerge development Q1 medians improve from 0.775129 to 0.185689 seconds at SF1 and 9.296148 to 2.217461 seconds at SF10; median input units fall from 600 to 12 and 7,264 to 117. Both binaries use the same MO base 64b7a43852, locked provider, datasets, canonical SQL and two streams, with one excluded warmup and five measured repetitions. All 22 SF1 public schema/value/order comparisons match native MO exactly. These targeted development results retain their original identities alongside the new merged-source release campaign.

Next: MO integration #30021 is ready for review, at 9101b0c9112d26faf768fb5779fad25fe11a753a, including authoritative main b09f2c6fc5. Local release acceptance is complete: all 744 candidate timing/correctness executions, the separate 264 resource executions and the matched 744 old-native executions passed; all six allocation profiles have zero unexplained warmed growth. Matched GPU suite medians improve from 4.216747 to 2.537705 seconds at SF1 and 49.884309 to 28.475628 seconds at SF10, passing the 5% regression gate. Fresh sanitizer controls preserve four raw statuses of 99; all ten matched numeric cases pass, with a maximum ratio of 1.021216. Current bridge/coexistence, writer/compiler normal/race and post-main owning/native-preparation checks pass. Generated captures remain outside Git and retain their actual source/binary/SDK identities. CI and reviewer approval, then the user merge, remain. The previous coverage setup failed on an HTTP504 CMake download before tests; aggregation had no complete profile. A normal newest-main sync, validated by five owning suites and all22 native preparations, started fresh CI run38262427688 at the current head. CI is pending; no tests or workflow gates were weakened. E/default cutover and numeric execution changes remain separate. F is excluded; the abandoned sidecar repository stays untouched.

The dated checkpoints below retain historical validation and review states; the current status and next action are stated above.

Merged native release delivery (2026-10-09)

Sirius #29 merged as 908ffc75a58b3be496b2172795e416326416e7ec; C now pins that merged revision and importer #5 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Clean release SDK regeneration, all 73 fingerprints and actual runtime-package identity checks pass. Default MO, Sirius-only and combined Sirius/cuVS release builds pass. Eight exporter and sixteen MO SDK-verifier tests pass.

At executable C head 59aaa5d234 on MO main 0eb746f37c22f57d2081331b25bf379c80168c81, all 22 native preparations and the complete public numeric fixture pass against the merged release SDK. All 31 terminal events confirm completed GPU tasks, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. Both default normal/race and Sirius-only owning suites pass for substrait, siriusbridge, cnservice and compile. The independent C99 GPU/credit/cleanup consumer, complete combined bridge suite and same-process cuVS-before/Sirius/cuVS-after pass. Native bridge data/cancellation passes 83 in-process race repetitions.

The latest main merge d7621ea017db4e43dbde9e5a855ba7b2ad920784 adds spill-cache batching and design documentation. Its changed spillio package passes; C source and unspilled fixture inputs are unchanged. Full mandatory pre-push SCA and self-review are required on the submitted C head.

Kernel #29775 now has passing required CI, including pessimistic Compose BVT, overall coverage, unit/SCA and Connector/J pool-reset checks, but remains open. C #29757 stays draft until it inherits the actual merged kernel fix and closes final integrated validation/CI. D implementation begins only after C merges, as one complete bounded public/resource/performance campaign PR. Charged-credit checks and this small fixture do not satisfy D's SF1/SF10, actual-memory/process or performance gates. E/F and both issue closures remain deferred.

Kernel metadata prerequisite and current CI (2026-10-09)

MO kernel #29775 is ready at 50847b7feb9a50545a4cb6a4056f61ad954526e4, based on freshly fetched main c2c1031d0ba4b1034849f6bd3d267252a11eb6c9. It fixes premature result-schema freezing for prepared SELECT when concurrent DDL commits after execute-time binding. An inert phase point independently reproduces the same 20631/HY000 error on clean main b6fa8ccfc7. Headers now use the executed generation before the first positive row, or after successful empty execution; incompatible retries after schema publication remain rejected. Saved results finalize after whole-query success.

The eight public cases pass: both prepared protocols across commit-before-execute and commit-after-binding, empty/nonempty exact BIGINT results, plus empty saved-result completion and exact BIGINT replay. Complete frontend/compiler owning suites pass normally and under race. Focused race validation passes 100 callback repetitions and three full public repetitions in one process. Full mandatory pre-push SCA and lifecycle self-review pass. Exact-head CI now passes, including pessimistic Compose BVT, coverage and CI Required; the kernel PR is still open.

C head 8284b77adad0a3d29881808c0893b51370a53124 has passing shared build, Ubuntu UT, arm64 SCA, UT coverage and Proxy BVT. Its pessimistic Compose BVT fails one statement at pessimistic_transaction/prepare.sql row 10 with the metadata retry error; overall coverage has no valid completed BVT producer and CI Required consequently fails. The earlier PR-size TLS failure passed on targeted retry. C remains draft and must inherit the merged kernel fix and merged #29 before final release/public/owning checks and required CI closure.

Approved D remains one complete PR after merged C: version-2 native-MO/embedded-MO public execution, bounded comparison (32 MiB RAM / 4 GiB scratch), immutable SF1/SF10 manifests, lifecycle/resource evidence and the full 744-execution schedule. Keep every performance gate, including fresh matched 10% numeric baselines and the Flight+MO ratio gate of 1.0 on equivalent available coverage. E/F and issue closure remain deferred.

Native consumer prerequisite and C validation (2026-10-09)

Sirius #29 was validated before merge at clean head 2a39339beb0c2312dc4b71fbdd472e822450637c. It fixes exact NULL predicates over private decimal carriers and condition-free joins left as ANY_JOIN when exact preparation skips DuckDB's ordinary optimizer. NULL predicates use the canonical validity mask; literal TRUE joins use equal constant keys in the existing GPU join, preserving join kind, multiplicity, projection maps and empty-side outer NULLs. FALSE/NULL controls still reject before work starts. No SQL benchmark rewrite, numeric narrowing or fallback is introduced.

Clean frozen-Pixi local validation passes 1,964 assertions / 10 production C ABI cases, 14,879 / 11 exact/ordinary GPU cases, and 852 / 81 binding cases. Tests cover all physical widths, offset slices, empty inputs, NULL literals, materialize/interpreted/JIT evaluation, INNER/LEFT/RIGHT/FULL TRUE joins, duplicate/NULL high-limb payloads and empty sides. Eight SDK exporter tests, all pinned changed-file hooks, an independent C99 GPU/credit/lifecycle consumer and all 73 clean SDK fingerprints pass. Lint, documentation and distribution CI pass; compiler jobs are queued as of this update. Local passes are reported separately from CI.

Joint C development validation passes all 22 native preparations (no reader/GPU work starts during preparation) and the complete public MySQL numeric fixture. The 31 terminal events confirm completed GPU work, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. The public fixture independently compares native-MO values/metadata/errors, including prepared division, masked errors and empty/all-NULL scalar joins. This uses an explicitly marked development SDK with #29's source, not a merged-pin release claim. Its MO base is 23d8f61252f3017dff218b530817a240206fd762; C now also merges main aee2a0b3764781a8ca908618548dfef75e336263, with exporter owning/vet and full mandatory SCA passing.

Merge order: #29 is merged and pinned in C. Merge kernel #29775, inherit that merged fix in C #29757, complete final integrated validation/CI, then merge C. D implementation begins only after C merges. All full-data/resource/process/performance gates remain open; E/F require separate recovery-readiness approval.

Merged root-name prerequisites and historical evidence

Importer #5 merged as 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Sirius #28 merged as af4dc60152b3e14f263c7fe863b29ac7e154de30 and is C's recorded pin. The original Q1 root-name crash is resolved: canonical decimal carriers are opaque SQL scalars and cannot consume private limb headings; actual name accesses are bounded while ordinary final-STRUCT compatibility is retained.

The final importer fix 6693c3a4d779fb0e15fa07e4ce4166f0572f3952 is tree-identical to its merge. Joint importer/native binding validation passed 1009 assertions / 84 cases, including 157 / 3 canonical importer cases. Clean Sirius head 433cadd43a10442b0825a23234fb6d859207e9d5 is tree-identical to merged #28; its binding 852 / 81, production numeric C ABI 1468 / 8, SDK exporter/independent C consumer/fingerprint and pinned-hook checks passed.

Historical importer distribution CI had seven failures: clean-base STRUCT field-selection failure/crash plus five later failures independently reproduced at clean importer 95d9ce8d78490db3991ab6145653716aa3ec42c9 and bundled DuckDB d8cdaa33fda8df955cc76ef58a280f68f4cd43fa. Those five selected cases reproduced DISTINCT_FROM, TPCH/TPCDS CTE, old literal expected-message mismatch and empty-plan root failures. No assertion was relaxed or skipped, and that distribution run was not green. These scoped historical results do not replace C/D acceptance.

Delivered prerequisites and Native A

• Sirius upstream synchronization: chore: sync upstream dev for embedded Sirius sirius#13 and build: sync embedding branch with upstream Sirius main sirius#23.
• MO backend/configuration contract: feat(sirius): introduce execution backend contract #28973.
• Sirius C ABI/lifecycle: feat(embedding): add native C ABI and lifecycle foundation sirius#14.
• Bounded MO-native ingestion: feat(embedding): add bounded MO-native GPU ingestion sirius#15.
• Native execution and incremental results: feat(embedding): execute native queries with bounded results sirius#19.
• Execution evidence and opaque query identity: feat(embedding): expose query execution statistics sirius#20 and fix(embedding): accept opaque binary query IDs sirius#21.
• MO read bindings and approved MO-reader-only design: feat: bind embedded Sirius MO reads #29209.
• Pixi-only GPU provider and combined SDK profile: build: use Pixi as the sole GPU provider #29254 and build: pin MatrixOne embedding to Pixi sirius#22.
• CGo/build/package/CN runtime bridge: feat: add bounded native Sirius CGo bridge #29203.
• Real bounded MO readers/results and terminal evidence: feat: execute embedded Sirius with bounded MO readers #29547. Includes preserved DOP>1 scans, correct EOF retirement without hiding caller cancellation or substantive reader errors, and first-row telemetry that accepts valid 0ns latency.
• Numeric inventory and approved exact-decimal semantics: docs: define exact Sirius numeric contract #29210 and docs: correct embedded Sirius numeric semantics #29449, semantic document blob 42a89f09a1d168d02b9583cb3ea7b4de6dbb5634.
• Query-scoped importer hooks and unity-build prerequisite: feat: add query-scoped extension import handlers duckdb-substrait#3 and fix(tests): qualify importer types in unity builds duckdb-substrait#4; merged importer revision 95d9ce8d78490db3991ab6145653716aa3ec42c9 is pinned by Native A.
• Exact-decimal coefficient representations/codecs and checked scalar GPU primitives: feat(numeric): add checked MO exact-decimal primitives sirius#25 (16135ebfbd1eb79d9a6e41c0858f69bd530f48ae).
• Native A: scoped exact type/literal import, immutable binding and cloning, masked GPU scalar execution, CASE/COALESCE and SDK literal schema: feat(numeric): bind MO exact-decimal plans and scalar expressions sirius#26 (c440013c89f1a54a6e5d934aabd0f9d3ef15ed3c). Production exact admission/capability stays disabled.
• Native B: aggregate/key integration, complete capability and typed statuses, feat(numeric): complete embedded MO aggregates and exact capability sirius#27, merged as 5ea60cd31955d0dced2adcbcd3df0772207b79ef ; opaque-carrier follow-up Add set union #28 is also merged.
• Owner-approved delivery sequence: docs(sirius): record remaining embedding PR sequence #29690. Native B/prerequisites, MO C feat(sirius): lower MO exact decimals for embedded GPU execution #29757 and D feat(sirius): add complete public SQL GPU acceptance campaign #29862 are merged; the current native batching and MO integration follow-up is next. E remains separate; F is excluded.

The scanner/Sirius TAE fixes already merged remain independently useful; they do not enable embedded direct-TAE input in this milestone. Importer hooks preserve identities and descriptors; Sirius owns numeric arithmetic and execution semantics.

Remaining approved implementation series

Native B/prerequisites, C #29757 and D #29862 are merged. The current implementation sequence is:

F is removed. The abandoned sidecar repository is left untouched; no customer compatibility or abandoned-service retirement machinery is required. Docker/image work remains separate.

The exact semantic and GPU/no-fallback contracts, unchanged canonical SQL, resource bounds and existing numeric regression policy remain in force. Missing required evidence cannot become a waiver. Original D captures retain their original source identities.

Delivery and completion

Use existing directories and dedicated branches in the user's forks. Sirius integration targets upstream-dev-merge; MO targets main. The abandoned sidecar repository is not a delivery target. Importer hooks target its compatible feature/26155-substrait-anti-joins branch pinned by Sirius; do not bundle unrelated importer synchronization.

Pin native consumers only to merged dependency commits. Sirius builds/tests always use frozen Pixi and incremental host builds, not image rebuilds. Intermediate PRs reference this issue and do not close it.

Predecessors #26154, #27586, #26159 and PR #27599 remain historical evidence, not this migration's closure. Preserve reusable tests and explicitly map replacements before deleting transport-only cases.

Close #28968 only after complete numeric/public/resource/performance acceptance. Close #28966 after the merged native/integration follow-up, E and verified release availability; F is not required. The opt-in supported-type milestone and Native A are distinct from migration completion.

CI/review correction (2026-10-10)

C #29757 now includes the explicit async histogram-export completion barrier at c4951901727116ed0c86806f772b215a845ca8fb. The previous exact-head Ubuntu CI failure named mometric.TestExporter; the test was observing two async sends before completion. All original batch/sample/counter/gauge/clock assertions remain. Exact normal/race, 100 focused race repetitions, complete metrics owning normal/race and full pre-push SCA pass. Fresh exact-head CI is running.

Kernel #29775's reviewer-found late saved-result failure cleanup is fixed at 6efb5fdfdaad60c2f458963ab143ec5ae5ec812f. The old-head red state regression fails; the corrected error/cancellation controls pass. The public same-connection test persists a batch, injects a producer error after saving, drains it, verifies connection identity, then saves/replays exact BIGINT data. All nine public cases, full frontend/compiler normal/race, three public race repetitions and full SCA pass. The reviewer thread is replied to and resolved, with re-review requested; approval and new CI are still pending. Inherit this prerequisite in C only after actual merge. D implementation remains gated on merged C, and E/F/issue closure stay deferred.

C merged-prerequisite integration (2026-10-10)

Kernel #29775 merged as af3f7232a3; C #29757 now inherits it and the merged MORPC CI-race correction from #29794. C is delivered at b39755ff4e2be20a3c9af464e9a46125ec14d9ea, normally merged with authoritative main 8f48d6fac0.

Final local release/native integration passes default/Sirius/combined builds, all 22 preparations, the public exact-numeric GPU fixture (31 healthy exact-profile/no-fallback events with zero retained credits), nine public prepared/save/replay cases, owning normal/race checks and combined Sirius/cuVS coexistence. Native cancellation passes 68 race repetitions; final planner checks and mandatory SCA pass. The detailed evidence is in C's versioned design record and PR description, with executable revisions stated explicitly.

C is now ready for review. Passing required CI, reviewer approval and the user merge remain before D implementation begins as one complete campaign PR. E/F remain deferred; this tracker stays open.

C deep-review corrections delivered (2026-10-10)

C #29757 is ready for review at a4e34ef05de6a6d02ceb0fa81219cacd452ae6c8, normally merged with authoritative main 62827a1c8b. Fix commit af4f342066 resolves persisted zero-DATE extraction, bound CASE nullability, and an admitted unchecked integer-overflow result. Ordinary embedded integer +/-/* now declines before execution; checked exact-decimal arithmetic and integral aggregates remain eligible.

Both human review findings have actual GPU red/green proofs. Their threads are replied to and resolved, and XuPeng-SH re-review is requested. Final all-22 native preparation, full public GPU normal/race (three repetitions), 34 healthy exact/no-fallback/zero-retained-credit events, owning normal/race, combined release/bridge and full SCA pass. Native pin/ABI and Flight behavior remain unchanged.

Fresh required CI and reviewer approval remain before the user merges C. D follows merged C as one complete campaign PR; E/F remain deferred and this tracker stays open.

D public campaign implementation delivered (2026-10-11)

D #29862 contains the public streaming runner, bounded exact schema/value/duplicate/order comparison, owned service/descendant cleanup, separate allocation profiling, lifecycle/sanitizer/performance evidence and fail-closed acceptance. Early writer/quota failures cancel the unread query before driver close. The shared-CTE compiler fix uses independent MO readers on the same statement snapshot; Sirius #30 preserves retained VARCHAR identity casts and validates destructive bindings. Canonical SQL is unchanged.

All recorded results retain actual source/binary/SDK/capture identities. The development GPU route was slower overall on this hardware; the fresh numeric regression maximum ratio was 1.007914, within the 1.1 policy. Raw sanitizer exit 99 is preserved with exact Native A CUB and no-GPU-work pinned-pool controls and six stable allocation profiles. Full final release acceptance remains incomplete. Merge Sirius #30 next, then advance the verified merged pin and complete the uniform final 744/264 release validation before promoting D. Required CI was inspected once after publication; no CI waiting or automatic merge is requested.

D uniform release acceptance (2026-10-12)

D #29862 now contains the complete merged-source release evidence. Native MO and embedded MO agree on exact schema, values, duplicates and ordering for every required canonical Q1-Q22 SF1/SF10 stream cell. Embedded queries completed actual GPU work with no fallback, healthy terminal status and zero retained input/result credit. All owned campaign processes joined before full publication. Timing and resource schedules use the same final release provenance; failed and mixed development captures remain separate.

All six warmed allocation profiles are stable. The sanitizer report preserves four raw statuses of 99 and exact matched Native A CUB / no-GPU-work 41,386,248-byte process-global cuDF pinned-resource controls. All ten matched numeric cases have 21 fresh samples and ratios <=1.1. Median full-suite times are native/GPU 1.804130/4.789524 seconds at SF1 and 27.947777/52.826967 seconds at SF10. No speedup or waiver is claimed.

The artifact acceptance command reproduces local_acceptance_complete: true, with only required CI and reviewer approval remaining. A normal main merge and source-qualified consumer checks preserve the measured source and binary identities. Finish those external D gates and the user merge, then design E (default/configuration/recovery migration) and F (sidecar retirement after verified MO release availability) separately. Docker remains a separate task.

D review handoff (2026-10-12)

Required CI at b9b434d1da passed, including changed-code coverage 77.92%
(247/317 lines, above 75%). There were no incoming review findings. Main's two
subsequent commits were merged normally; the affected compiler/frontend/CN/
transaction/runtime/message-board owners pass eight normal and six race suites.
Sirius, exact lowering, bridge, schema evidence and collectors are unchanged.
The source qualification
preserves the original timing/resource/native proof identities.

D #29862 is ready for review at eba41acd67, with XuPeng-SH requested. Complete
implementation and local evidence permit that handoff; reviewer approval is
still required before final acceptance and merge. Fresh required CI for the
new merged head must also pass. E/F design follows accepted and merged D;
Docker remains separate. Neither tracking issue is closed.

D P2 review corrections (2026-10-12)

Both substantive review gaps are closed in 78d87489da, delivered at
4230ce3d14 after normally merging main 035a5b61ba. Final retained summaries
reconnect native/GPU and declared Flight oracle cells; cardinality, physical
schema and identity must match. The CLI verifies the recorded archive manifest
and parses exactly the bytes hashed. Unordered/peer raw spool hashes need not
match; the live full-row comparator remains the value oracle.

Workspace checks now continue during readiness, worker execution, profile
finalization, owned export and shutdown. Quota/deadline failures kill and join
the owner before parsing/publication. The reduced live-writer tests reproduce
both former gaps on the reviewed source; their 1 KiB success control reaches
parsing after join. All 82 owning Python tests and affected normal/race consumer
suites pass. The original release archive still passes with identical output;
native Q1 count/OID mutations fail even after rehashing. Original captures,
contexts and raw sanitizer statuses remain unchanged.

D stays ready for review. Both threads are replied to and resolved, with XuPeng-SH
re-review requested. Fresh fix-head required CI, reviewer approval and user merge
remain before acceptance/closure. E/F follow merged and accepted D.

Generated evidence removed from the source tree (2026-10-12)

The approved cleanup removes 54 generated files and retains one concise
validation record. The original captures remain in contributor-fork tag
evidence/sirius-d-4230ce3d14 at commit
4230ce3d143e77a8b4be83feb47bd26858a8d459:
immutable release evidence.
The external tar digest and current-validator commands are documented in the
public-campaign design. All 82 existing Python tests pass without either evidence
directory, and archive acceptance matches the saved result. No measurements,
proof identities or raw sanitizer outcomes are rewritten; earlier history stays
intact. Fresh CI and reviewer approval remain required before merge/closure.