Repository navigation
[CI] TestDataBranchDiffAsFile cleanup timeout on Ubuntu/x86 #28638
Description
Summary
The Ubuntu/x86 unit-test job for PR #27716 failed in pkg/tests/dml with a cleanup timeout in TestDataBranchDiffAsFile. The PR changes information-schema view metadata parsing, persistence, and remote protocol handling; it does not change the failing test or the embedded-cluster cleanup lifecycle.
Source PR and failed checks
- PR: fix: correct information schema views metadata #27716
- Exact tested head:
43e05db0318855f13898bb348955a361d24feca6 - Workflow run: https://github.com/matrixorigin/matrixone/actions/runs/34453970444 (attempt 1)
- Failed job: UT Test shard embedded on Ubuntu/x86
- Related failed check: UT Test on Ubuntu/x86 (the job did not expose a downloadable log)
- Derivative failed check: CI Required
The failing job ran:
go test -mod=readonly -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race github.com/matrixorigin/matrixone/pkg/tests/dml
Failure log excerpt
The complete failed-job log contains the following stable signature:
dml_test.go:316: drop cleanup database db_TestDataBranchDiffAsFile_818329738: context deadline exceeded
--- FAIL: TestDataBranchDiffAsFile (155.27s)
--- PASS: TestDataBranchDiffAsFile/single_pk_with_base (3.75s)
--- PASS: TestDataBranchDiffAsFile/composite_pk_with_base (3.69s)
--- PASS: TestDataBranchDiffAsFile/single_pk_without_base (0.80s)
--- PASS: TestDataBranchDiffAsFile/composite_pk_without_base (1.82s)
steps.ut.conclusion: failure
FAIL_UT_CASES=TestDataBranchDiffAsFile,
The same job also logged repeated pre-clean attempts to reach the Docker socket:
failed to connect to the docker API at unix:///var/run/docker.sock;
connect: no such file or directory
Independence evidence
Using the repository mo-dev CGo test wrapper with the CI-equivalent race, tags, and CGO settings, the exact focused test passed three consecutive times at the repaired PR candidate e5f6e771e69574b531e5da34a98733d94362fdfb.
The same exact focused test also passed three consecutive times on a clean current official up/main control checkout at a9434ef1c692c661852fa3c418af9a4ce4dbf333. No changed file in PR #27716 is under pkg/tests/dml, pkg/embed, LogService, HAKeeper, or the shared embedded-cluster cleanup lifecycle; the changed planner/protocol paths do not implement the failing database cleanup mechanism.
The focused command used for both control sets was:
.agents/skills/mo-dev/scripts/mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml
An open-issue search found no existing issue covering this exact test-and-timeout signature. Closed issue #25929 covers an earlier nil-pointer cleanup panic with a different signature, and closed issue #26519 covers a broader embedded-cluster timeout family; neither is reused as the tracker for this occurrence.
Reproduction and impact
Run the command above on an Ubuntu/x86 CI worker with the MatrixOne matrixone_test race configuration and the normal embedded test-cluster setup. Under the observed timing/resource conditions, the test body can pass all subtests but fail while dropping its temporary database, causing the Ubuntu/x86 UT producer and the required aggregate check to fail.
This issue records the evidence before one failed-jobs rerun of workflow run 34453970444.
Activity
iamlinjunhong commentedon Sep 10, 2026
补充 run 34453970444 的 attempt 2 证据（本次重跑已按一次性规则执行）。
43e05db0318855f13898bb348955a361d24feca6。actions/github-script@v7汇总结果中matrixone-ci.result=failure，随后在.github/ci/change-scope.cjs:84抛出Error: matrixone-ci: failure；没有产品测试错误签名。runner_id=0、无 steps，GitHub 没有可下载日志，3 秒内失败，未执行测试命令。TestDataBranchDiffAsFile清理阶段context deadline exceeded证据仍记录在本 issue。结论：attempt 2 未提供新的代码失败证据，属于 CI 聚合/调度故障证据；本 run 不再重复 rerun。
iamlinjunhong commentedon Sep 16, 2026
本次为 PR #28987 的当前 exact head 复现证据，补充到本 issue：
完整 UT 诊断报告中，
pkg/tests/dml.TestDataBranchDiffAsFile的update_apply_special_columns/indexed_enum_conflict_accept和mixed_schema_ordinary_assignment在dml_test.go:1631收到context deadline exceeded；同一时间段伴随 HAKeeper/LogService timeout、heartbeat 延迟和事务提交状态未知。该测试的其他子用例通过，失败发生在嵌入集群控制面受压时，而不是 Data Branch 断言差异；随后全局 UT 预算又取消了仍在运行的 upgrade case。这是本 issue 所覆盖的TestDataBranchDiffAsFileUbuntu/x86 deadline/cleanup 故障类别的新复现。本 issue 已包含独立 Linux/x86 CI 发生记录以及 3/3 PR-head 和 3/3 clean-main focused controls。当前 PR 的 sort-rollup 改动没有触及
pkg/tests/dml、Data Branch hash-diff、嵌入集群生命周期、HAKeeper、LogService 或 UT harness；同一 exact head 的 sort-rollup 自身验证、SCA、UT Coverage、Proxy BVT、Pessimistic BVT、shared build 和 Coverage 均通过。请将 run
35086785091作为本 issue 的本次复现记录；在记录完成后按一次性失败 job 规则重跑该 run。iamlinjunhong commentedon Sep 22, 2026
补充 PR #29186 的当前 exact-head CI 证据，并复用本 issue 跟踪同类失败。
matrixone-ci: failure。稳定日志签名：
归因证据：
memory.max=17179869184、peak=17180041216、memory.events.max=2，并有多个并发 compile/test 进程；同时memory.events.oom=0、oom_kill=0，因此记录为 runner capacity pressure / embedded-cluster cleanup timeout，而不是已证明的 OOM kill。pkg/sql/compile的 role-rule rename admission、对应单测和zz_accesscontrol/role_ruleBVT；失败测试位于pkg/tests/dml/dml_test.go，本次TestDataBranchDiffAsFile不执行 rename，且角色规则 admission 在无 rename 时直接 no-op，依赖闭包不包含该 DML cleanup 机制。TestDataBranchDiffAsFilecleanup-timeout 类别一致。计划：在记录本次 run/job 证据后，按标准仅对 workflow run
35596394110执行一次gh run rerun 35596394110 --failed，并核验新 attempt；不修改 PR 源码或 reviewer。gouhongshen commentedon Sep 24, 2026
Automated AI triage
This issue requires human involvement and cannot currently be completed independently by the
automated fix loop.
Reason
A speculative fix is prohibited by the Handler skill and a red/green proof is not constructible from a healthy environment: after a repository audit of all three occurrences plus two fresh executable controls at base (focused race PASS 28.93s parent; GOMAXPROCS=2 starvation probe PASS 18.67s parent) and the issue's own 6/6 Linux focused controls, the timeout reproduces only under documented runner capacity pressure (cgroup memory.max/peak/memory.events in occurrence 3, control-plane heartbeat stalls in occurrence 2, docker socket loss in occurrence 1). No code-owned blocked cleanup was isolated by anyone in any occurrence, and every candidate owner path (DROP DATABASE waits #29393, logtail read barrier #30109, embedded fixture ownership #29048, fixture cost #29031/#29141/#29520/#30216) was reworked on main after the last occurrence, leaving no demonstrated defect to repair and no test that could fail before / pass after a change. Next human action: the Ubuntu UT / embedded-cluster owner captures the blocked DROP with goroutine stacks, in-flight transaction/lock state and control-plane logs from an affected run (or reproduces under a Linux cgroup matching the 16GiB CI memory ceiling with concurrent compile+race processes), and considers raising UT shard memory headroom. Evidence posted as issuecomment-6207109398; severity/s0 replaced with deferred.
Evidence
{
"ready_for_review": false,
"summary": "Issue #28638 fully re-investigated on base eb18dbf and deferred with the complete Handler-skill procedure executed: the TestDataBranchDiffAsFile cleanup-DROP timeout is an environment-only (runner capacity) trigger on all available evidence, no code-owned blocked cleanup was isolated in any of the three recorded CI occurrences, and the failing statement's owner paths were all reworked on main after the last occurrence. No speculative patch was made. The analysis was posted as issuecomment-6207109398 and the severity/s0 label was replaced with deferred (verified live: labels are now kind/bug, ai-easy, deferred).",
"reproduction": {
"method": "Attempted local reproduction of the CI flake signature (cleanup 'drop database if exists' exceeding its independent 30s client deadline) on a healthy macOS/arm64 host with a fresh in-tree make-cgo native build, using the CI-equivalent race command plus a bounded scheduler-starvation probe, combined with a repository audit of all three CI occurrences to bound the cause.",
"steps": [
"Fetched and inspected incident head 43e05db: incident-time cleanupTestDatabases already used an independent 30s context (dml_test.go:485-486, introduced by #27613 / commit 5bc051a merged 2026-08-26); the reported dml_test.go:316 line is the t.Helper()-attributed deferred call site of cleanupTestDatabases; the preceding 'use mo_catalog' on the same connection succeeded, so the CN frontend was responsive while the DROP ran past 30s server-side.",
"Built native artifacts with make cgo in this worktree (exit 0, about 16.5 minutes; cgo/libmo.dylib plus thirdparties/install).",
"Ran the focused control at base eb18dbf: .agents/skills/mo-dev/scripts/mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml.",
"Ran the identical command under GOMAXPROCS=2 as a bounded scheduler-starvation probe.",
"Audited occurrence chronology: runs 34453970444 (2026-09-10, PR #27716), 35086785091 (2026-09-16, PR #28987), 35596394110 (2026-09-21, PR #29186); fetched occurrence-3 head 0edfbce and verified its dml_test.go already contained the #29031 parallel-fixture rework (2-slot semaphore, per-subtest sessions)."
],
"baseline": "Clean worktree at base eb18dbf, macOS/arm64, Go 1.27.1, fresh native cgo build; CI reference command from the failing job: go test -mod=readonly -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race github.com/matrixorigin/matrixone/pkg/tests/dml.",
"expected": "If a deterministic or scheduler-starvation-sensitive product defect owned the blocked cleanup, the focused race run or the GOMAXPROCS=2 run should reproduce the signature 'drop cleanup database db_TestDataBranchDiffAsFile_: context deadline exceeded'.",
"observed": "No reproduction. Control: PASS with zero fails, parent TestDataBranchDiffAsFile elapsed 28.93s, package ok 32.944s, exit 0. GOMAXPROCS=2 probe: PASS, parent elapsed 18.67s, package ok 22.630s, exit 0 - two-thread scheduler oversubscription left the 30s cleanup deadline with margin. These match the issue-recorded 6/6 Linux focused controls (3x at PR head candidate e5f6e771e69574b531e5da34a98733d94362fdfb, 3x at clean up/main a9434ef). All three CI occurrences correlate with runner-level capacity evidence: occurrence 1 docker socket failures, occurrence 2 HAKeeper/LogService timeouts and heartbeat stalls in the same window, occurrence 3 cgroup memory.max=17179869184, peak=17180041216, memory.events.max=2 while already containing the #29031 fixture rework."
},
"root_cause": "Bounded unresolved cause: the cleanup DROP's independent, by-design 30s client deadline expired while the shared embedded cluster's control plane was externally starved on a degraded/oversubscribed Ubuntu CI runner. The distinguishing factor is kernel-level memory-reclaim stalls at the cgroup ceiling (occurrence-3 telemetry: memory.max events with peak at the 16GiB limit), not Go scheduler starvation (the GOMAXPROCS=2 probe passed with margin). A code-owned lock/transaction wait behind the DROP cannot be excluded with certainty from the available logs because no live stacks, transaction state, or control-plane dumps were captured in any occurrence; however none was isolated by anyone, the respective PR diffs are demonstrably unrelated to the failing mechanism, and the wait protocol that would own such a defect was replaced after the last occurrence: DROP DATABASE wait-protocol rework #29393 (b631e1d, merged 2026-09-29, postdates all occurrences), DROP read-barrier repair #30109 (c5208fc, 2026-10-13), embedded-fixture ownership release #29048 (24e66eb, 2026-09-17), plus fixture-pressure reductions #29031 (4ff27bb), #29141 (534e2f5), #29520 (19e01d2) and #30216 (f49bc6f, merged 2026-10-15). The incident-time code shape no longer exists on main.",
"changes": [
"No repository changes: source, tests, .github/workflows, .github/ci and mo-tester untouched; worktree verified clean (0 modifications) at base eb18dbf.",
"Posted the complete deferred analysis (verified evidence, executed controls, bounded cause, next human action) as github.com//issues/28638#issuecomment-6207109398 (verified live, created 2026-10-16T03:05:41Z).",
"Replaced label severity/s0 with deferred on issue #28638 (verified live: labels now kind/bug, ai-easy, deferred, updatedAt 2026-10-16T03:05:44Z)."
],
"tests": [
"CONTROL (fresh, this session): .agents/skills/mo-dev/scripts/mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml at base eb18dbf -> PASS, zero fails, parent elapsed 28.93s, package ok 32.944s, exit 0.",
"STARVATION PROBE (fresh, this session): identical command under GOMAXPROCS=2 -> PASS, zero fails, parent elapsed 18.67s, package ok 22.630s, exit 0; scheduler oversubscription alone insufficient to expire the 30s cleanup deadline.",
"Issue-recorded author controls (Linux): the same focused command passed 3/3 at PR head candidate e5f6e771e69574b531e5da34a98733d94362fdfb and 3/3 at clean up/main a9434ef.",
"CI occurrence audit: runs 34453970444 / 35086785091 / 35596394110 dated 2026-09-10 / 2026-09-16 / 2026-09-21, all predating the #29393 DROP rework (2026-09-29) and #30109 (2026-10-13); occurrence 3 at head 0edfbce verified in this session to already contain the #29031 fixture rework while still timing out under cgroup memory-ceiling events."
],
"proof": [
"Failing statement and deadline owner identified exactly at the incident commit: the deferred cleanupTestDatabases 'drop database if exists' with its own independent 30s context (dml_test.go:485-486 at 43e05db, from #27613 / 5bc051a); log lines dml_test.go:316 and :314 (recurrence) are the t.Helper()-attributed deferred call site; 'use mo_catalog' succeeded first, bounding the stall to the DROP's server-side execution rather than connection loss.",
"Occurrence chronology vs fix merges: all three occurrences (2026-09-10, 2026-09-16, 2026-09-21) predate #29393 (merged 2026-09-29) and #30109 (2026-10-13); occurrence-3 tree at 0edfbce verified to contain the #29031 rework, proving fixture-cost reduction alone does not gate the signature when the runner hits its memory ceiling (memory.events.max=2).",
"Fresh executable evidence at base eb18dbf: focused race control PASS (28.93s parent) and GOMAXPROCS=2 starvation probe PASS (18.67s parent), zero fails each; combined with the issue's 6/6 Linux focused controls, the failure does not reproduce without external capacity pressure.",
"Owner-path audit at base: cleanup helper and its 30s assertion preserved unchanged (dml_test.go:513-514); no remaining known code-owned wait to target (DROP waits #29393, read barrier #30109, fixture ownership #29048, fixture cost #29031/#29141/#29520/#30216), so any patch would be speculative.",
"Deferred-path governance completed and verified live: analysis comment issuecomment-6207109398 (2026-10-16T03:05:41Z) and label swap severity/s0 -> deferred (labels kind/bug, ai-easy, deferred)."
],
"risks": [
"The flake can recur on capacity-starved Ubuntu/x86 UT runners even on current main; any new occurrence now carries #29393/#30109, so captured live evidence from it would discriminate a residual product wait from pure capacity starvation.",
"A code-owned wait weakly correlated with control-plane starvation (for example deadlock-detector progress stalling when heartbeats stall) cannot be excluded from historical logs alone; it is bounded to paths since reworked, unproven either way.",
"GOMAXPROCS=2 is an imperfect proxy for Linux cgroup memory-reclaim stalls; the probe narrows but does not eliminate the trigger space.",
"No patch was produced: if a future occurrence captures live stacks disproving pure capacity starvation, a new fix cycle with red/green proof will be required; this deferral preserves, rather than weakens, the cleanup failure assertions and the 30s deadline."
],
"deferred_reason": "A speculative fix is prohibited by the Handler skill and a red/green proof is not constructible from a healthy environment: after a repository audit of all three occurrences plus two fresh executable controls at base (focused race PASS 28.93s parent; GOMAXPROCS=2 starvation probe PASS 18.67s parent) and the issue's own 6/6 Linux focused controls, the timeout reproduces only under documented runner capacity pressure (cgroup memory.max/peak/memory.events in occurrence 3, control-plane heartbeat stalls in occurrence 2, docker socket loss in occurrence 1). No code-owned blocked cleanup was isolated by anyone in any occurrence, and every candidate owner path (DROP DATABASE waits #29393, logtail read barrier #30109, embedded fixture ownership #29048, fixture cost #29031/#29141/#29520/#30216) was reworked on main after the last occurrence, leaving no demonstrated defect to repair and no test that could fail before / pass after a change. Next human action: the Ubuntu UT / embedded-cluster owner captures the blocked DROP with goroutine stacks, in-flight transaction/lock state and control-plane logs from an affected run (or reproduces under a Linux cgroup matching the 16GiB CI memory ceiling with concurrent compile+race processes), and considers raising UT shard memory headroom. Evidence posted as issuecomment-6207109398; severity/s0 replaced with deferred.",
"issue_reports": [
{
"number": 28638,
"reproduction": {
"method": "Local red/green-style reproduction attempts on a healthy host with the CI-equivalent race command (plus a GOMAXPROCS=2 starvation variant) at base eb18dbf, combined with a repository audit of the three CI occurrences and their version relationship to subsequently merged owner-path fixes.",
"steps": [
"Inspected the incident head 43e05db: cleanupTestDatabases uses an independent 30s context; dml_test.go:316 is the t.Helper()-attributed deferred DROP call site; 'use mo_catalog' succeeded before the DROP.",
"Built cgo/libmo.dylib and thirdparties via make cgo (exit 0).",
"Ran the focused control: mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml.",
"Ran the identical command under GOMAXPROCS=2 as a bounded starvation probe.",
"Dated the three occurrences (2026-09-10/16/21) against fix merges and verified occurrence-3 head 0edfbce already contained the #29031 parallel-fixture structure."
],
"baseline": "Clean worktree at base eb18dbf, macOS/arm64, Go 1.27.1, fresh native cgo build; CI job reference: go test -race -tags matrixone_test -p 2 -timeout 40m on pkg/tests/dml.",
"expected": "A product-owned blocked cleanup, if deterministic or scheduler-starvation-sensitive, should reproduce as 'drop cleanup database db_TestDataBranchDiffAsFile_: context deadline exceeded' in the focused or starved run.",
"observed": "No reproduction: control PASS (parent 28.93s, package ok 32.944s, exit 0); GOMAXPROCS=2 probe PASS (parent 18.67s, package ok 22.630s, exit 0); consistent with the issue's 6/6 Linux focused controls. The signature appears in CI only together with runner capacity evidence (docker socket loss; HAKeeper/LogService heartbeat stalls; cgroup memory.max=16GiB with peak at ceiling and memory.events.max=2)."
},
"root_cause": "External, environment-only control-plane starvation of the shared embedded cluster on the Ubuntu UT runner, expiring the cleanup DROP's independent 30s client deadline; kernel-level memory-reclaim stalls at the cgroup ceiling (occurrence-3 telemetry) are the distinguishing factor, not Go scheduler starvation (starvation probe passed). No code-owned lock/transaction wait was isolated in any occurrence, and the candidate owner paths (DROP wait protocol #29393, read barrier #30109, fixture ownership #29048, fixture cost #29031/#29141/#29520/#30216) were reworked on main after the last occurrence; a residual code-owned wait correlated with starvation remains weakly open only because no live-stack evidence was captured.",
"changes": [
"No repository changes; worktree verified clean at base.",
"Analysis posted as github.com//issues/28638#issuecomment-6207109398.",
"Label severity/s0 replaced with deferred (current labels: kind/bug, ai-easy, deferred)."
],
"tests": [
"mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml at base eb18dbf: PASS, parent elapsed 28.93s, package ok 32.944s, zero fails, exit 0.",
"Same command with GOMAXPROCS=2: PASS, parent elapsed 18.67s, package ok 22.630s, zero fails, exit 0.",
"Issue-recorded author controls: 3/3 focused PASS at PR head candidate e5f6e771e69574b531e5da34a98733d94362fdfb and 3/3 at clean up/main a9434ef.",
"Occurrence audit: 34453970444 / 35086785091 / 35596394110 (2026-09-10/16/21) all predate #29393 (2026-09-29) and #30109 (2026-10-13); occurrence 3 contained the #29031 rework and still timed out under documented memory-ceiling events."
],
"proof": [
"Incident-time failing statement and its independent 30s deadline identified exactly (dml_test.go:485-486 at 43e05db, introduced by #27613 / 5bc051a); log lines :316/:314 attributed via t.Helper() to the deferred cleanup DROP.",
"Two fresh executable controls at base with zero fails plus the issue's 6/6 Linux controls: failure not reproducible without external capacity pressure; scheduler starvation alone insufficient (GOMAXPROCS=2 PASS).",
"Version-bounded owner-path audit shows occurrences predate all relevant reworks (#29393, #30109, #29048, #29031, #29141, #29520, #30216) and occurrence 3 proves fixture-cost reduction does not gate the signature under cgroup ceiling events.",
"Deferred governance executed and verified live: comment 6207109398 (2026-10-16T03:05:41Z) with full evidence and next human action; severity/s0 replaced by deferred; no speculative patch submitted, cleanup assertions and the 30s deadline preserved."
]
}
]
}
Please use this analysis to provide the missing decision, environment, evidence, or ownership
needed before the issue is returned to the AI candidate pool.
gouhongshen commentedon Oct 16, 2026
Independent re-investigation (handler pass, base
eb18dbfa030b6309c5f5d0c2d5e7078a1466ac7f)This pass re-verified the failure mechanism, owner paths, and all three occurrences against the repository, and ran fresh controls locally. Conclusion unchanged: the signature is an environment-only (runner capacity) trigger; no code-owned blocked cleanup has been isolated, and the failing statement's owner paths have since been reworked on
main.Verified evidence (repository, this pass)
cleanupTestDatabasesDROP: at incident head43e05db0318855f13898bb348955a361d24feca6the helper already used an independent 30s context (dml_test.go:485-486, introduced by test: reduce high-cost integration fixtures deterministically #27613 /5bc051abc4, merged 2026-08-26).t.Helper()attributes itst.Errorfto the closure end atdml_test.go:316(recurrence 3::314, same pattern).use mo_catalogon the same connection succeeded first, so the CN frontend was responsive; the 30s client deadline expired whiledrop database if existswas executing server-side.b631e1d1a5, merged 2026-09-29) and the DROP read-barrier repair fix(disttae): retain read barriers through logtail reconnects #30109 (c5208fc54d, 2026-10-13): runs34453970444(2026-09-10, PR fix: correct information schema views metadata #27716),35086785091(2026-09-16, PR feat: add cost-based streaming sort rollup #28987),35596394110(2026-09-21, PR fix: prevent table rename from bypassing role row filters #29186).dml_test.goalready contained the test: reduce DML diff fixture work and scheduling tail #29031 parallel-fixture rework (2-slot semaphore, per-subtest sessions, verified at0edfbce70b74b7a6146d4aa292d50380ac95bdfc) and still exceeded the cleanup deadline - with runner telemetrymemory.max=17179869184,peak=17180041216,memory.events.max=2. Fixture-cost reduction lowers pressure sensitivity but cannot immunize the bounded deadline against control-plane starvation.eb18dbfa030b: cleanup helper 30s context preserved (dml_test.go:513-514); shared embedded-fixture ownership release fixed by fix: release embedded cluster ownership after terminal cleanup #29048 (24e66eba12, 2026-09-17); DROP DATABASE ordered cleanup with detector-resolved waits by fix(sql): reuse ordered cleanup for DROP DATABASE #29393; logtail read-barrier retention through reconnects by fix(disttae): retain read barriers through logtail reconnects #30109; continued fixture-pressure reductions test: reduce CPU UT fixture duplication #29141 (534e2f5335), test: reduce race fixture work and strengthen regression coverage #29520 (19e01d261b), perf(test): reduce race UT issues runtime and memory pressure #30216 (f49bc6fa53, merged 2026-10-15). The incident-time code shape no longer exists onmain.Executed controls (this pass, macOS/arm64, Go 1.27.1, fresh
make cgonative build in-tree).agents/skills/mo-dev/scripts/mo-cgo-test -vet=off -short -v -json -tags matrixone_test -p 2 -timeout 40m -race -count=1 -run '^TestDataBranchDiffAsFile$' ./pkg/tests/dml-> PASS, zero fails, parentTestDataBranchDiffAsFileelapsed 28.93s, packageok32.94s, exit 0.GOMAXPROCS=2-> PASS, parent elapsed 18.67s, packageok22.63s, exit 0. Two-thread Go scheduler oversubscription alone did not reproduce the cleanup deadline expiry (the 30s deadline held with margin). The distinguishing factor in the recurrences is kernel-level memory-ceiling reclaim stalls (cgroupmemory.maxevents recorded in occurrence 3), which cannot be faithfully or safely emulated on a shared non-Linux host.Together with the 6/6 focused controls already recorded in this issue (3x PR head candidate
e5f6e771e69574b531e5da34a98733d94362fdfb, 3x cleanup/main), the failure does not reproduce without external capacity pressure.Bounded unresolved cause
The cleanup DROP's independent 30s client deadline expired while the shared embedded cluster's control plane was externally starved (runner cgroup memory-ceiling events / HAKeeper-LogService heartbeat stalls documented in the recurrences). A code-owned lock/transaction wait cannot be excluded with certainty, but none was isolated in any occurrence, and the wait protocol that would own it was replaced by #29393/#30109 after the last occurrence.
Why deferred instead of a patch
Any code change here would be speculative: there is no demonstrated product wait to repair, the bounded-deadline assertions must be preserved, and weakening the cleanup (retry/swallow/deadline bump) would hide genuine product regressions. A red/green proof is not constructible from a healthy environment.
Next human action (unchanged, now with version bounds)
Ubuntu UT / embedded-cluster owner: capture the blocked DROP with goroutine stacks, in-flight transaction/lock state, and control-plane logs from an affected run (or reproduce under a Linux cgroup matching the CI memory ceiling of 16 GiB with concurrent
compile+-racetest processes). Runs onmain >= eb18dbfa030badditionally carry #29393/#30109; if the signature recurs there, the captured evidence discriminates a residual product wait from pure capacity starvation. Consider raising UT shard memory headroom (memory.events.max=2in occurrence 3 indicates the cgroup ceiling was hit).XuPeng-SH commentedon Oct 20, 2026
Consolidated audit: #30670
Closing as a superseded historical incident, rather than claiming an environment-only root cause was proven or every cleanup latency was fixed.
The September 10/16/21 occurrences predate the DROP wait-protocol and read-barrier corrections #29393/#30109. Current main
0fb05134c76148fe7fd42a6bd54d37038df9510eno longer has those incident-time wait paths. The currentTestDataBranchDiffAsFilenormal control passes (9.73s test body,-count=1, Go 1.27.1/Darwin arm64/CGo/short/matrixone_test); it does not recreate the original Linux cgroup pressure.Residual cleanup/post-DDL timeout investigation remains tracked in #30454, including the need for in-flight transaction/lock state and blocked stacks from an affected run. Earlier memory-pressure correlations and healthy focused passes do not exclude a code-owned wait and should not be treated as proof of an infrastructure-only cause.
test: fix transaction, context and resource ownership in UT fixtures (#…