Repository navigation
[Bug]: CI shared deadline expires in retained-gate promotion test #30161
Description
Is there an existing issue for the same bug?
- I have checked the existing issues.
Searches covered the exact test name, 29400 context deadline, and retained-gate timeout. #29400 tracks the original DDL availability defect; closed #29881 tracks a transaction/allocator race, and #30129 tracks an ISCP startup stall in another test. None matches this failure signature.
Branch Name
issue-29974-main (PR #30115, targeting main)
Commit ID
d78564b6605fbca346fe82c4d0801f6f63ad8fab
Other Environment Information
- OS: Ubuntu/x86 GitHub Actions runner
- Toolchain: Go 1.27.1, CGo
- Mode:
-short -race -tags matrixone_test - CI entry:
make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all - Run: https://github.com/matrixorigin/matrixone/actions/runs/38509448204
- Failed UT job: https://github.com/matrixorigin/matrixone/actions/runs/38509448204/job/115674587476
Actual Behavior
TestIssue29400CopyAlterRetainedGatePromotionFastFails gives all sequential scenarios the same context.WithTimeout(context.Background(), 2*time.Minute) at pkg/tests/issues/issue_29400_test.go:41. Its deadline is exhausted before the final scenarios can run their assertions.
The downloaded ut-diagnostics-38509448204-1-all JSON event report establishes this UTC timeline:
| Event | Time | Result / elapsed |
|---|---|---|
| Parent starts | 05:27:44.945 | shared two-minute context created |
| replacement_revalidation ends | 05:28:17.280 | PASS, 32.33s |
| tenant_logical_snapshot_uses_broad_gate ends | 05:28:47.718 | PASS, 30.44s |
| fresh_truncate/commit | 05:29:06.301–13.680 | PASS, 7.36s |
| fresh_truncate/rollback | 05:29:13.680–28.194 | PASS, 14.57s |
| fresh_truncate/cancel | 05:29:28.194–33.803 | PASS, 5.61s |
| fresh_truncate/timeout | 05:29:33.804–39.956 | PASS, 6.15s |
| same_table_dml | 05:29:39.957–41.705 | PASS, 1.75s |
| fk_publication/forward_false | 05:29:41.706–44.296 | PASS, 2.59s |
| fk_publication/forward_true | 05:29:44.296–44.946 | FAIL, 0.65s |
| shared_gate / exclusive_gate | 05:29:44.947 | FAIL immediately, 0.00s each |
| Parent ends after cleanup | 05:29:54.988 | FAIL, 130.04s |
First failure is context deadline exceeded at line 621 (set foreign_key_checks=0). The last two fail at line 695 / execSQLRequire, while merely creating their databases:
exec failed: create database issue_29400_prior_dml_shared_gate
context deadline exceeded
exec failed: create database issue_29400_prior_dml_exclusive_gate
context deadline exceeded
All earlier named scenarios passed. These later failures do not exercise their intended retained-gate assertions: they inherit an already-expired parent context. The overall Go test timeout is 40 minutes, distinct from this internal two-minute fixture budget. CI Required fails as an aggregate consequence.
Expected Behavior
One scenario's accumulated setup/execution time should not consume another scenario's entire observation and cleanup budget. Preserve deterministic waiter/barrier assertions and bounded cancellation; do not mask a genuine wait-cycle failure by increasing a global timeout, sleeping, retrying or skipping scenarios. Investigate why the full shared-cluster/race lane is slower before selecting a fixture/budget-isolation fix.
Steps to Reproduce
Run the original CI workload above on Ubuntu/x86. For focused diagnosis with the repository CGo wrapper:
GOTOOLCHAIN=go1.27.1 .agents/skills/mo-dev/scripts/mo-cgo-test \
-v -short -race -tags=matrixone_test -count=1 -timeout=300s \
-run '^TestIssue29400CopyAlterRetainedGatePromotionFastFails$' ./pkg/tests/issuesAdditional information
Classification: non-PR flaky/test-budget candidate, not a proven product root cause or a claim that clean main reproduced this Linux failure.
The test, authenticated fixture and lockservice sources are unchanged against current main a53390c57d. The complete PR production delta is confined to LAG/LEAD default/result type reconciliation and semantic AST cache identity; it does not alter the shared context, DDL gate ownership, fixture timeout, or UT harness. The failing SQL contains no LAG/LEAD call. The event timeline proves deadline exhaustion and failure propagation, but does not establish the cause of the earlier successful scenarios' greater latency.
After merging current main, the full focused root test passed once on Darwin/arm64 at merge head a0739a43d5b02adf2087b5b0ff2851f5c8251cd4, retaining short/race/tags/CGo mode: package duration 25.425s, all named subtests passed, exit 0, no race report. The owning plan package also passed, 21.346s. This is a platform-limited diagnostic result, not proof of Linux/full-suite stability. No assertions/timeouts were weakened and no sleeps/retries/skips added.
A new-head CI execution after upstream synchronization is the controlled retry; do not simultaneously rerun the superseded SHA or describe the timeout as fixed before CI completes.
Activity
iamlinjunhong commentedon Oct 14, 2026
Additional occurrence from PR #30168 at exact head
a579075a8abdc954e93bd08216fca6e602532b1f(ABS signed-zero correction).-short -race -tags matrixone_test; command:make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all.The complete job log was inspected. The parent
TestIssue29400CopyAlterRetainedGatePromotionFastFailsstarts at 11:00:02 UTC and fails after 128.89s. The final scenarios inherit the single two-minute context atpkg/tests/issues/issue_29400_test.go:41:The first error occurs around 11:02:02, when the shared context expires. This occurrence first fails the competing-owner completion assertion, then the next scenario's database setup; it is not a setup-only failure. The preceding expected lock-conflict response is present in the log. The root and both failing subtests are the same normalized shared-deadline failure covered here.
Attribution uses the authorized known-open-defect comparison: this open issue already includes the independent Linux/x64 occurrence in unrelated PR #30115, run 38509448204 / job 115674587476, and repository-maintainer analysis of the shared two-minute fixture budget. The current PR's entire five-file diff changes only floating-point signed-zero normalization in AbsSigned and its math/array/SQL regression coverage. The failing SQL does not evaluate ABS. The test, CDC sources, lockservice, Makefile and native inputs are unchanged versus freshly fetched official main
fd4767e488dc111db02c9119cd46dfe78af1ee23. The PR's own momath and moarray owning packages passed locally with short/race/matrixone_test/CGo; exact-head CI SCA, UT coverage and both selected BVT lanes passed.No clean-main reproduction of this DDL timeout is claimed. An optional local DDL validation was stopped during dependency compilation, before this root test executed, once the existing independent evidence and review established the known-defect route. No code, timeout or assertion was changed.
The same job also has a distinct CDC duplicate-reader cancellation error-chain assertion, now tracked separately in #30205. That exact CDC case passed 3/3 on the PR head and 3/3 on clean official main in matching short/race/tags/CGo mode (Darwin/arm64), supporting a bounded non-PR flaky candidate classification rather than proven Linux stability. After both signatures are recorded, request one failed-jobs rerun of run 38535252841. A repeated failure requires renewed attribution, not an automatic retry loop.
iamlinjunhong commentedon Oct 14, 2026
Additional occurrence from PR #30184 at exact head
d48be22e128ef9f3de3e0ff924533ab9626d867e(floating-point signed-zero set operations and correlated UNION schemas).make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all; Go 1.27.1, CGo, short/race/matrixone_test mode.The complete failed-step logs for both jobs were inspected. The only failing root test is
TestIssue29400CopyAlterRetainedGatePromotionFastFails. The stable signature matches the shared two-minute parent context inpkg/tests/issues/issue_29400_test.go:41and the prior occurrence in comment 6169000489:The first assertion error is adjacent to the 15:54:02 shared deadline; the competing DROP database server operation finishes at 15:54:02.292. Earlier named scenarios passed (replacement revalidation 19.78s, tenant snapshot 30.14s, fresh truncate commit/rollback/cancel/timeout 6.44/8.00/6.51/14.38s, same-table DML 2.48s, FK publication 4.34/2.92s). The subsequent exclusive scenario inherits the expired context before database setup. CI Required reports
matrixone-ci: failureas an aggregate consequence. There is no additional failing test, build/type/static-check failure, panic or race detector report in the failed logs.Attribution uses the approved known-open-defect comparison: this open issue covers the exact root and normalized deadline class, includes repository-member analysis of the shared fixture budget, and links an independent Ubuntu/x86 occurrence in unrelated PR #30115, run 38509448204 / job 115674587476. Comment 6169000489 additionally records the exact shared_gate:797/exclusive_gate:695 pattern in unrelated PR #30168, run 38535252841 / job 115760279540.
The complete current PR diff contains 14 files: production changes bind float set-operation keys, retain UNION arms until their equality domain is known, preserve correlated UNION schemas, and refresh prepared set-operation keys; the remaining changes are corresponding tests. The appended function ID/overloads preserve existing IDs and are only selected for the new key function. The failing SQL exercises CREATE/ALTER/DROP/TRUNCATE/INSERT/UPDATE and scalar catalog reads, with no UNION/INTERSECT/EXCEPT and no new float-key function. The shared fixture, retained DDL gate, lockservice, native inputs and CI harness are outside the PR change and dependency closure. The failing test/fixture and lockservice are byte-unchanged against freshly fetched official main
ddb76ea12cc05ac7517c026fa237516865f86c2b.Current exact-head evidence for the PR's own changed closure passes: SCA (job 115889253264), UT Coverage (115889325171), both multi-CN BVT lanes PROXY (115894241030) and PESSIMISTIC (115894241314), and Coverage (115915267548), all in run 38574707638. Existing local validation at this same source revision also passed the complete planner package, public signed-zero/correlated/prepared SQL regressions and changed-package static checks. Independent attribution review found no material blocker.
Classification: matches the known shared-deadline defect; this does not claim a clean-main Linux reproduction or explain why the preceding successful scenarios consumed more time. No unrelated code, timeout, assertion, retry or skip is changed. Reproduction remains the full CI command above; the focused short/race/tags/CGo command in this issue is a diagnostic control, not an equivalent full-suite reproduction.
After recording this evidence, request one failed-jobs rerun of run 38574707638. A repeated failure requires renewed attribution rather than an automatic rerun loop.
XuPeng-SH commentedon Oct 17, 2026
Verified the reported shared-deadline ownership defect is corrected on latest official main b111224. Independent named scenarios/operations now receive the existing two-minute budget instead of consuming one setup-wide deadline. Assertions, bounded waiters, cleanup checks and the two-minute duration remain; no retry or skip was added.
The actual TestIssue29400CopyAlterRetainedGatePromotionFastFails root and its subtests ran successfully on Linux/amd64, Go1.27.1, CGo, short/race/matrixone_test (-count=1); root elapsed 20.91s, owning selection terminal0. The prior full successful CI39024356589 also executes this root with the same corrected ownership. Closing this test-budget defect. This does not claim the original storage/control-plane slowdown is understood or fixed; remaining startup and catalog-cleanup latency stays tracked by #30453 and #30454.