You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: isolated UT Coverage package exceeds 10-minute deadline during cluster shutdown #29368
The pkg/tests/issues/isolated package exhausts its 10-minute go test package deadline in UT Coverage across several unrelated open PRs. The package budget is shared by multiple embedded-cluster tests, so the final running test name is not by itself the cause.
Evidence
#29365 UT Coverage job timed out after about 601 seconds; the uploaded raw test report shows TestAutoIDCacheRestartAndDisabledNode taking about 302 seconds and TestIssue28234ExceptAllCluster about 157 seconds. TestSpeedupAbortAllTxn had run only 27 seconds when the package deadline fired.
At the timeout, TestSpeedupAbortAllTxn was cleaning up an embedded cluster. The main shutdown path was embed.cluster.Close -> cnservice.closeService -> Stopper.Stop; a heartbeat task was in createTaskService -> registerExecutorsLocked -> mysqlTaskStorage.PingContext -> net.Read.
Reproduce or compare against clean main using the same UT Coverage mode and package parallelism; collect per-test timings and the full shutdown goroutine graph.
Determine whether the heartbeat MySQL I/O can outlive CN shutdown and, if so, make the owner cancel/close it so Stopper.Stop terminates. Do not infer that network read is the root cause from one stack alone.
Reduce or isolate the expensive embedded-cluster fixture work so the package finishes with margin under its configured deadline, while preserving each distinct test contract. Avoid simply raising the timeout without resolving the shutdown path and budget.
Verify the affected lifecycle case and the owning package in coverage mode, then a complete UT Coverage producer run.
Root cause update: the Coverage workflow gives its whole test phase 3600 seconds, but invokes go test without -timeout, leaving Go's default 600-second deadline on each package. In the failed #29365 report, 11 completed isolated tests consumed 574.35 seconds before TestSpeedupAbortAllTxn began; the package alarm followed 27 seconds later. The final test's shutdown stack is not evidence that its MySQL read caused the 10-minute alarm.
The CI budget fix is now proposed in matrixorigin/CI#459 (-timeout=20m per package). The separate cancellation ownership concern visible in that stack is recorded as #29371. This issue remains open for validation of the CI change and for reducing the two expensive embedded-cluster fixtures (301.61s and 157.35s in the failed run).
Activity
XuPeng-SH commentedon Sep 24, 2026
Root cause update: the Coverage workflow gives its whole test phase 3600 seconds, but invokes
go testwithout-timeout, leaving Go's default 600-second deadline on each package. In the failed #29365 report, 11 completedisolatedtests consumed 574.35 seconds beforeTestSpeedupAbortAllTxnbegan; the package alarm followed 27 seconds later. The final test's shutdown stack is not evidence that its MySQL read caused the 10-minute alarm.The CI budget fix is now proposed in matrixorigin/CI#459 (
-timeout=20mper package). The separate cancellation ownership concern visible in that stack is recorded as #29371. This issue remains open for validation of the CI change and for reducing the two expensive embedded-cluster fixtures (301.61s and 157.35s in the failed run).fix(cn): cancel background registration and trace work on shutdown (#…