Skip to content

[Bug]: Embedded-cluster UTs still lose HAKeeper connections under Ubuntu/x86 load #26522

Closed
#26752
Closed
[Bug]: Embedded-cluster UTs still lose HAKeeper connections under Ubuntu/x86 load#26522
#26752
Assignees
gouhongshen
Labels
kind/bugSomething isn't workingseverity/s0Active / top priority for current sprint. Owner has committed to working on it now.

Description

@gouhongshen
gouhongshen
opened on Jul 31, 2026

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

main

Commit ID

0b8b53d

Other Environment Information

  • Hardware parameters: amd64-mo-shanghai-8c16g GitHub Actions runner
  • OS type: Linux/amd64
  • Others: MatrixOne ALL CI, Ubuntu/x86 UT job 91130900423

Actual Behavior

Four embedded-cluster consumers failed in one current-main-based Ubuntu/x86 UT job after repeated HAKeeper/LogService heartbeat timeouts and TCP connection resets:

  • TestCreateAndDropPitr (pkg/tests/ddl): LogService heartbeat to 127.0.0.1:11020 was reset.
  • TestMultiClusterCanWork (pkg/embed): CN/TN/LogService heartbeats to 127.0.0.1:10740 repeatedly reset; the second cluster timed out in waitAnyShardReadyLocked at cluster_test.go:182.
  • TestIcebergSQLEngineEmbeddedCredentialSecurityAndRedaction (pkg/embed): LogService heartbeats to 127.0.0.1:11140 reset before the assertion at iceberg_sql_harness_test.go:796.
  • TestTableFeatures (pkg/tests/features): HAKeeper allocation on 127.0.0.1:12020 reset while creating the test database, failing at table_features_test.go:47.

The triggering PR #26505 changes only pkg/frontend/data_branch_output.go and its focused tests; the full pkg/frontend package passed in the same job. The failure is therefore outside that PR.

The job ran after #26448 (fb7dc32ee) was merged into the base. That fix stabilized the shared authenticated base cluster, but these failures show that other embedded-cluster consumers remain vulnerable to the same CI-load heartbeat starvation/reset pattern.

Expected Behavior

Embedded clusters used by UT packages should keep HAKeeper/LogService heartbeats alive and start deterministically under the Ubuntu/x86 CI load profile. Tests should not lose their control-plane connection or time out waiting for TN shards.

Steps to Reproduce

  1. Run the MatrixOne Ubuntu/x86 unit-test workflow at 0b8b53dff3869dcd5e3ab52992852c9ea6545026 or a descendant.
  2. Run the normal parallel UT package set on an amd64-mo-shanghai-8c16g runner.
  3. Observe the embedded clusters in pkg/embed, pkg/tests/ddl, and pkg/tests/features for HAKeeper/LogService heartbeat timeouts, connection resets, and waitAnyShardReadyLocked timeouts.

Additional information

Activity

  1. gouhongshen
    self-assigned this
    on Jul 31, 2026
  2. gouhongshen
    added
    kind/bugSomething isn't working
    needs-triage
    on Jul 31, 2026

  3. gouhongshen commented on Jul 31, 2026

    @gouhongshen
    gouhongshen
    on Jul 31, 2026
    ContributorAuthor

    Another occurrence on PR #26505 head d7a02bfaa1a976c996766a6862a7d236e31826a6 in the Ubuntu/x86 UT job:

    https://github.com/matrixorigin/matrixone/actions/runs/30638598838/job/91182528687

    Three tests failed with the embedded-cluster/HAKeeper signature tracked here:

    • TestCreateAndDropPitr: HAKeeper/LogService heartbeats timed out and the connection to 127.0.0.1:11020 was reset; database creation failed.
    • TestTableFeatures: HAKeeper ID allocation lost 127.0.0.1:12020 with connection reset by peer; database creation failed.
    • Test_UpgradeEntry: repeated HAKeeper proposal/checker-state timeouts and TN heartbeat failures ended in context deadline exceeded / internal error: waitAnyShardReadyLocked at pkg/tests/upgrade/upgrade_v1_2_3_test.go:32.

    Test_UpgradeEntry is a newly observed consumer of the same failure family. The fourth failed test, the independent ISCP TestStaleRead assertion, is documented in #26517. PR #26505 changes only data-branch file-output finalization in pkg/frontend and has no overlap with embedded-cluster, HAKeeper, DDL/features, or upgrade code.

  4. XuPeng-SH
    added a commit that references this issue on Jul 31, 2026

    fix(frontend): surface data branch file finalization errors (#26505)

    a7a04b4

  5. XuPeng-SH commented on Aug 1, 2026

    @XuPeng-SH
    XuPeng-SH
    on Aug 1, 2026
    Contributor

    Another two-attempt reproduction confirms that the embedded-cluster/HAKeeper startup failure is still present after the narrower fixes already included in the tested main base.

    Workflow: https://github.com/matrixorigin/matrixone/actions/runs/30639089405
    Tested PR: #26501
    Tested head: 939a675096f0cc9283a28bfd08304d9baecaf462
    Main parent merged into that head: e1a11841ec96c7fba3870dfa67ef86a6fb75533f

    Both Ubuntu/x86 UT attempts failed with the same control-plane signature, but in different embed tests:

    Attempt 2 shows the failure chain clearly:

    • the single-node HAKeeper Raft group repeatedly dropped proposals while it had no leader;
    • after leader election, initial-cluster-info proposals, HAKeeper ticks, checker-state reads, and LogService/TN heartbeats repeatedly timed out;
    • Dragonboat reported batches of 13–17 delayed LocalTick messages;
    • all three CN services remained at shard not ready;
    • cluster startup finally exhausted its deadline.

    PR #26501 changes recursive CTE planning/execution and has no overlap with pkg/embed, LogService, HAKeeper Raft startup, or TN shard readiness.

    Latest-main audit (2026-08-01):

    Conclusion: current main has no demonstrated fix for this startup failure. The two consecutive attempts make this stronger than a single flaky occurrence; keep this issue open until the relevant startup/control-plane path is fixed and repeatedly validated under the Ubuntu/x86 full-UT load profile.

  6. gouhongshen commented on Aug 3, 2026

    @gouhongshen
    gouhongshen
    on Aug 3, 2026
    ContributorAuthor

    Automated AI triage

    This issue requires human involvement and cannot currently be completed independently by the
    automated fix loop.

    Reason

    Requires a HAKeeper/CN/TN maintainer to define and approve the control-plane progress contract—command-delivery deadline, concurrency model, and stale-response behavior—before implementation. The human next action is to choose the design for #26566 and split #26522 into bounded implementation and CI-validation work.

    Evidence

    The failure recurred after merged mitigations #26507 and #26546. Current main still performs CN/TN heartbeat RPCs synchronously before handling schedule commands, while #26566 documents several competing architectural solutions and requires shutdown/restart-generation semantics across CN, TN, LogService, and embed startup.

    Please use this analysis to provide the missing decision, environment, evidence, or ownership
    needed before the issue is returned to the AI candidate pool.

  7. gouhongshen
    added
    deferred
    and removed
    needs-triage
    on Aug 3, 2026

  8. gouhongshen commented on Aug 3, 2026

    @gouhongshen
    gouhongshen
    on Aug 3, 2026
    ContributorAuthor

    Current-main root-cause audit

    I re-audited the original failures, the relevant code on current main (820ee12144439c5b63c03b261f4330ec0364fdb3), and CI after the merged mitigations.

    Conclusion

    This was a real load-sensitive embedded-cluster failure, not a bad assertion or a change-specific failure. It was not fixed by #26546 alone. However, the evidence after #26503 indicates that the original failure family is now substantially mitigated, while a separate production control-plane weakness remains tracked by #26566.

    Confirmed failure chain

    1. The Ubuntu/x86 job runs on an 8-core/16-GB runner with UT_PARALLEL=6. Several package processes can each keep a complete LogService/TN/3-CN cluster alive.
    2. fix: harden async lifecycle and CI test contracts #26546 serializes only embedded-cluster bootstrap. It releases the startup lease once services are ready, so already-started clusters still run concurrently. TestMultiClusterCanWork also intentionally keeps two clusters alive in one process.
    3. Under full-UT load, Dragonboat reported batches of 10-24 delayed LocalTick messages. At the configured 100-ms tick interval, that is direct evidence of roughly 1-2.4 seconds of delayed Raft/control-plane processing. HAKeeper proposals, ticks, checker-state reads, and heartbeats then timed out.
    4. CN and TN heartbeat loops are synchronous: SendCNHeartbeat / SendTNHeartbeat must return before the response commands are handled or another heartbeat can be sent. A slow request therefore also delays delivery of commands such as D/Start.
    5. The TN shard remained unavailable until waitAnyShardReadyLocked exhausted its 30-second deadline. Connection resets and broken pipes were largely downstream effects of failed startup/cleanup and, before fix(cnservice): return bootstrap failures #26503, a transient CN bootstrap error terminating the package process.

    The recurrence after #26546 is concrete:

    What the merged changes do and do not fix

    After #26503 merged, I checked 94 completed MatrixOne ALL CI workflows through 2026-08-03: 40 completed successfully, and the four Ubuntu/x86 UT job failures in that set did not reproduce the original waitAnyShardReadyLocked/HAKeeper-loss startup signature. This is good mitigation evidence, but not a deterministic proof of closure.

    A later current-main failure of TestClusterCanStartNewCNServices (run 30784347559, job 91594982538) spent about 39 minutes in cluster.Close -> CN Close -> txn trace Stopper.Stop. Its test body had already completed. That is a distinct shutdown-lifecycle defect and should not be used as evidence that the original startup failure recurred.

    Recommended scope

    1. Keep [Bug]: Embedded-cluster UTs still lose HAKeeper connections under Ubuntu/x86 load #26522 focused on the Ubuntu/x86 full-load test contract: embedded-cluster packages need a bounded runtime resource class, not only serialized bootstrap and not retry-based masking.
    2. Validate removing the shared base clusters explicit 30-second heartbeat RPC deadline and restoring the 3-second service default while retaining the independent 60-second store-liveness window. A 30-second serial RPC deadline delays retry and schedule-command delivery.
    3. Validate with repeated full Ubuntu/x86 UT runs under the real 8C/16G, UT_PARALLEL=6 profile; isolated package runs are insufficient.
    4. Keep the production heartbeat/command-progress redesign in [Bug]: Serial HAKeeper heartbeat RPC can starve schedule-command delivery #26566, with bounded in-flight work, cancellation, and stale-generation semantics.
    5. Track the 40-minute CN/txn-trace shutdown hang separately.

    I would not close #26522 solely from the recent green sample. Closure should require a bounded embedded-cluster runtime-concurrency contract plus repeated full-profile validation; #26566 can remain open independently for the production liveness contract.

  9. aptend commented on Aug 3, 2026

    @aptend
    aptend
    on Aug 3, 2026
    Contributor

    New reproduction on PR #26531 after merging current main into head 0349df7.\n\nFailed job: https://github.com/matrixorigin/matrixone/actions/runs/30817969769/job/91700405563\n\nThe Ubuntu/x86 UT job failed only TestTableFeatures after 69.90s. The failure signature matches this issue:\n- Dragonboat reported 12-16 delayed LocalTick messages.\n- HAKeeper proposals and checker-state reads timed out.\n- TN/LogService heartbeats timed out.\n- database setup finally failed with connection reset by peer while talking to the local HAKeeper endpoint.\n\nThe job reported no OOM or process-kill signature. The tested PR does not change the TestTableFeatures setup or HAKeeper/LogService control-plane path.

  10. gouhongshen
    added
    severity/s0Active / top priority for current sprint. Owner has committed to working on it now.
    and removed
    deferred
    on Aug 4, 2026

  11. gouhongshen commented on Aug 5, 2026

    @gouhongshen
    gouhongshen
    on Aug 5, 2026
    ContributorAuthor

    Three-day post-merge observation

    Updated: 2026-08-06. Merge commit: 295fb60070f394a9697a5544fa32f91f5494f7d9 from PR #26667.

    Scope and counts

    I enumerated all MatrixOne ALL CI workflow runs created after the merge, then kept only heads whose commit ancestry contains the merge commit.

    Category Count
    MatrixOne ALL CI workflow runs after merge 231
    Runs containing the merge commit 224
    Runs with the Ubuntu/x86 UT job 223
    Completed Unit Testing executions 169
    Completed Unit Testing successes 145
    Completed Unit Testing failures 24
    Still-running Unit Testing jobs 7

    The remaining jobs were cancelled, skipped, or failed before Unit Testing. Two pre-UT failures were infrastructure failures caused by TLS/network problems.

    #26522 recurrence

    I count a recurrence only when a failed Ubuntu/x86 Unit Testing job contains a runtime failure signature, not when a passing test name merely contains Timeout or waitAnyShardReadyLocked.

    7 recurrences were found among 169 completed Unit Testing executions:

    The other 17 Unit Testing failures did not contain this failure family. The two failures before Unit Testing were TLS/network infrastructure failures and are not counted as recurrences.

    Observation continues through the agreed three-day window; new completed jobs will be added to this same comment.

  12. gouhongshen commented on Aug 6, 2026

    @gouhongshen
    gouhongshen
    on Aug 6, 2026
    ContributorAuthor

    Root cause found and draft fix

    The post-merge recurrence is not only cross-process package parallelism. In the serial Ubuntu/x86 stage of run 31074714519, pkg/tests/issues ran TestAuthenticatedTestsReuseBaseCluster before TestHashBuildSharedBudgetRecoverySQL.

    The first retained the package's 3-CN shared base. The second started a bespoke 1-CN cluster. Admission was reentrant within one binary, so six services could coexist despite -p 1; the second cluster then hit the no-leader / dropped-proposal / failed-initial-cluster-info signature.

    Draft PR #26752 fixes the test lifecycle contract without rebuilding the shared base:

    • implicit second complete cluster in one binary now fails fast;
    • genuine multi-cluster tests explicitly opt in;
    • the three bespoke-configuration regressions now run in pkg/tests/issues/isolated, a separate test binary explicitly included in run_ut.sh's serial scope;
    • join-spill SQL helpers are shared through an internal test package.

    This avoids the extra 3-CN base-cluster restarts in the earlier draft approach. It does not change production HAKeeper heartbeat/RPC behavior; #26566 remains the separate production command-progress issue. CI duration is being measured on the final design.

  13. mergify
    added a commit that references this issue on Aug 7, 2026

    test(embed): isolate shared test-cluster lifecycles (#26522) (#26752)

    8824b6c
  14. gouhongshen
    closed this as completedon Aug 13, 2026

  15. iamlinjunhong commented on Sep 11, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 11, 2026
    Contributor

    New CI occurrence for PR #27716, exact original head 90a1ab95db305f486d99c541a33fbfdfde743838.

    Failed workflow/check: Matrixone CI execution / UT Test on Ubuntu/x86, run 34565278565, job 103156040935, attempt 1. The aggregate CI Required check failed on the same run.

    The second exact failure was pkg/logservice.TestHAKeeperClientSendCNHeartbeat at pkg/logservice/hakeeper_client_test.go:618:

    Received unexpected error: context deadline exceeded
memberlist: Failed to join 127.0.0.1:40275: connect: connection refused

    The PR changes view-metadata protocol/parser/planner/sysview paths and does not change pkg/logservice, HAKeeper heartbeat code, embedded-cluster startup, or the CI harness. With the repository CGo wrapper and -race -tags matrixone_test -short -p 1 -timeout 40m -count=3 -run ^TestHAKeeperClientSendCNHeartbeat$ ./pkg/logservice, the exact PR head passed 3/3 and clean detached current up/main (123b4c1f5fb6e0f0bc69d02eee5b0b7cca289bcf) passed 3/3. The test selection was non-empty in both trees. This is another occurrence of the Ubuntu/x86 HAKeeper/LogService control-plane timing failure family tracked by this issue, rather than a deterministic failure in PR #27716.

  16. iamlinjunhong commented on Sep 12, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 12, 2026
    Contributor

    New occurrence: PR #28608 / run 34600385547

    Source PR/head:

    Failed checks:

    Complete UT failure signature:

    • The only failed test package was github.com/matrixorigin/matrixone/pkg/tests/arrowload.
    • TestArrowLoadBVT failed after 404.80s while writing to HAKeeper at 127.0.0.1:11020: read tcp4 ...: connection reset by peer.
    • The same log contains HAKeeper/LogService broken-pipe and connection-reset heartbeat errors, invalid state no CN in the cluster, and a later 127.0.0.1:11141: connect: connection refused.
    • The following TestArrowLoadGateDisabled and TestArrowLoadDistributedDisabledSoftFallback failed at the admission guard with invalid state another complete test cluster is already active in this process.
    • The runner diagnostic recorded 23 embedded clusters, admission wait 18.31m total / 5.80m max, 7 observed unreleased admissions with partial release evidence, and cgroup memory peak 17179869184 with oom=0 and oom_kill=0. The job ended with make: *** [Makefile:513: ut] Error 1; the aggregate job only propagated this result.

    Attribution:

    • PR feat: enable cost-based window hash partitioning #28608 does not change pkg/tests/arrowload, pkg/embed, pkg/logservice, HAKeeper/LogService lifecycle code, or the UT runner/harness. Its changed planner/frontend/window-partition closure passed in the same run; the focused frontend, plan, partition, and window compile checks also pass locally through the repository CGo wrapper.
    • This is a recurrence of the Ubuntu/x86 embedded-cluster HAKeeper connection-loss/admission-lifecycle failure family tracked here, not a deterministic window-partition assertion or a PR-specific ArrowLoad change.

    The failed jobs from run 34600385547 will be rerun once under the approved CI policy after this evidence is recorded.

  17. iamlinjunhong commented on Sep 12, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 12, 2026
    Contributor

    Follow-up evidence for PR #28636 / run 34666156512

    Source:

    Complete UT log signature:

    • Command: make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all; the job is the normal CGO-enabled Ubuntu/x86 UT lane.
    • The only terminal product test failure was pkg/tests/arrowload.TestArrowLoadBVT: pkg/tests/arrowload/cluster_test.go:148 / arrow_load_test.go:45, context deadline exceeded, with HAKeeper heartbeat/propose/checker/address-refresh timeouts and connection-reset noise; the test failed after 52.36s.
    • The aggregate job only propagated matrixone-ci: failure; it has no independent product failure. The complete log contains no compile/type/static/security failure.
    • The PR changes planner/frontend SIGN numeric-domain handling and tests; it does not touch pkg/tests/arrowload, embedded-cluster/HAKeeper/LogService code, or the UT runner/harness.

    Exact-head and clean-main comparison:

    • PR head, repository CGo wrapper, -count=1 -timeout=120s -run '^TestArrowLoadBVT$' ./pkg/tests/arrowload: 3/3 passed (26.665s, 23.135s, 26.160s).
    • Clean official up/main efb2c5084957ac90273a3f625668fc6a784562c4, the identical selection and CGo mode in a detached worktree: 3/3 passed (27.998s, 27.537s, 26.528s).
    • The PR-specific prepared-SIGN planner regression and public issue [Bug]: SIGN loses runtime numeric type across prepared execution paths #28583 regression also pass locally through the same CGo wrapper.

    Attribution: this is a non-PR flaky candidate in the embedded-cluster/HAKeeper timing family tracked by this issue. All four required checks are present: complete failure log with no stable product/compiler/static failure; 3/3 exact selected-test passes on the PR head; no PR diff or dependency-closure overlap with the failed package/mechanism/harness; and 3/3 identical passes on clean official up/main. The failed jobs of run 34666156512 will be rerun once under the approved CI policy after this evidence is recorded.

  18. iamlinjunhong commented on Sep 12, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 12, 2026
    Contributor

    New CI occurrence: PR #28608 / run 34689614866

    Source:

    Complete failure signature:

    • The only failed UT package was github.com/matrixorigin/matrixone/pkg/tests/issues.
    • TestIssue26118DatabaseCopiesPreserveHashIdentifiers failed after 48.94s.
    • The business assertions were not reported as failing. During deferred fixture cleanup, the test received: 
      Received unexpected error:
context deadline exceeded
exec failed: drop database if exists \`issue_26118_clone\`
    • The failure was reported from pkg/tests/issues/issue_26118_test.go:62, through issue_test.go:182, and the run ended with make: *** [Makefile:513: ut] Error 1.
    • Surrounding service evidence in the same failure window includes repeated LogService heartbeat, checker-state, and proposal timeouts; broken-pipe and connection-reset errors; and WAL/commit latency around 7.2 to 7.5 seconds.
    • The runner diagnostics recorded embedded-cluster admission waits totaling 13.15 minutes with a 4.18 minute maximum, and cgroup memory peak 17183281152 with oom=0 and oom_kill=0.

    Attribution:

    • PR feat: enable cost-based window hash partitioning #28608 changes planner/frontend/window-partition code, tests, design text, and distributed window coverage. It does not change pkg/tests/issues, pkg/embed, LogService/HAKeeper lifecycle code, or the UT runner/harness.
    • The exact-head run completed all eight pkg/sql/plan race shards successfully; the failed-package summary contains only pkg/tests/issues. The failure occurred in teardown of an unrelated issue regression, not in the PR window closure.
    • The direct runner capacity and embedded control-plane timeout evidence is consistent with the Ubuntu/x86 embedded-cluster/HAKeeper timing and teardown failure family tracked by this issue, and is independent of the PR change closure.

    The failed job from run 34689614866 will be rerun once under the approved CI policy after this evidence is recorded.

  19. iamlinjunhong commented on Sep 12, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 12, 2026
    Contributor

    Follow-up evidence: PR #28608 / run 34689614866 attempt 2

    Source:

    Complete failure signature:

    • pkg/tests/issues.TestWWConflict remained active for the full 40-minute package timeout and ended with panic: test timed out after 40m0s; the stack contains the test goroutine waiting in the issue-test workflow.
    • pkg/tests/partition.TestAffectedRows failed during create database after context deadline exceeded from HAKeeper ID allocation.
    • The same failure window contains repeated HAKeeper/LogService heartbeat, proposal, checker-state, and schedule-command timeouts, MORPC broken-pipe errors, and TCP connection resets. The runner checkpoint showed a 16 GiB cgroup peak with 17 processes while the control plane was degraded.
    • CI Required has no independent product failure; its github-script step only propagated matrixone-ci: failure.

    Attribution:

    • The PR changes planner/frontend/window-partition code and tests. It does not change pkg/tests/issues, pkg/tests/partition, embedded-cluster, HAKeeper/LogService lifecycle, or the UT harness. The changed closure passed the exact-head focused race validation locally; the exact-head CI run also completed the changed pkg/sql/plan race shards and unrelated frontend/window packages successfully.
    • Two independent embedded-cluster consumers failed in the same Ubuntu/x86 run while the service/control-plane errors were active. This is a direct runner/service failure occurrence in the HAKeeper/LogService timing family tracked by this issue, not a deterministic window-partition failure.

    This is the already-authorized one rerun for run 34689614866; attempt 2 was accepted and completed with the same infrastructure signature. No further rerun is requested from this occurrence.

  20. VioletQwQ-0
    added 6 commits that reference this issue on Sep 17, 2026

    fix(frontend): surface data branch file finalization errors (matrixor…

    93918ce

    test(embed): isolate shared test-cluster lifecycles (matrixorigin#26522…

    a86f894

    fix(frontend): surface data branch file finalization errors (matrixor…

    f0bdca3

    test(embed): isolate shared test-cluster lifecycles (matrixorigin#26522…

    3cdbd06

    fix(frontend): surface data branch file finalization errors (matrixor…

    6157815

    test(embed): isolate shared test-cluster lifecycles (matrixorigin#26522…

    2097ac2

  21. iamlinjunhong commented on Oct 15, 2026

    @iamlinjunhong
    iamlinjunhong
    on Oct 15, 2026
    Contributor

    Additional CI occurrence for PR #30262, exact head 1fb117bedeb4ade795cc99cf505c340de40e609f, branch fix/csv-external-integer-whitespace.

    Run 38715428406, attempt 1 is complete. Recording this against the existing embedded-cluster/HAKeeper connection-loss tracker. This tracker is closed; the classification here is based on the fresh four-part comparison protocol below.

    Failed checks and complete-log evidence:

    • UT Test on Ubuntu/x86, job 116356905171: the only failed root test is pkg/tests/sqlintegration.TestLegacyPartitionColumnLimitCopy (49.39s).
    • CI Required, job 116412489549: verifyResults propagates matrixone-ci: failure, without an independent product failure.
    • CI entry: make ut UT_CONFIGURED=1 UT_PARALLEL=6 UT_TIMEOUT=40 UT_SHARD=all; Go 1.27.1, Ubuntu/x86, CGo, short/race, matrixone_test. The failed package ran in the embedded stage's four fresh batches with bounded process parallelism 2.

    Normalized failure:

    TestLegacyPartitionColumnLimitCopy
legacy_partition_column_limit_test.go:57 / :184
alter table wide drop c4097
Error 1105 (HY000): writeto tcp4 ...->127.0.0.1:11020:
read tcp4 ...->127.0.0.1:11020: read: connection reset by peer
--- FAIL: TestLegacyPartitionColumnLimitCopy (49.39s)

    The failure window also contains LogService heartbeat/checker/proposal timeouts and CN heartbeat connection resets. The cgroup checkpoint at 11:55:37 UTC reports memory current 11,637,600,256 bytes, peak 17,179,869,184, and oom=0,oom_kill=0; these capacity observations do not establish an infrastructure root cause. No OOM kill, panic, reported data race, compiler/type/static/security failure was found in the complete job logs. The raw diagnostic JSON artifact was size-truncated; the complete job log retains the failed-test output and assertion above.

    Fresh four-part comparison:

    1. Both failed jobs' complete logs were downloaded and inspected. The root failure is the runtime connection-reset assertion above.
    2. At the unchanged exact PR head, the exact failed test passed 3/3, root times 43.25s, 42.27s, 47.58s, with terminal package PASS (135.955s, exit 0).
    3. The complete 12-file PR delta changes external CSV numeric materialization, a new exact-decimal wrapper, whitespace record parsing, and their tests/fixtures. It does not change SQL integration fixtures, embedded-cluster startup/lifecycle, LogService/HAKeeper transport, the UT runner, or partitioned COPY DDL. The failing test seeds legacy catalog definitions through the typed engine API and executes COPY ALTER TABLE; it has no CSV source. The new decimal wrapper is reached through external CSV integer materialization, outside this DDL/RPC failure path. The existing assignment parser itself was not modified.
    4. Fresh official main eb18dbfa030b6309c5f5d0c2d5e7078a1466ac7f passed the same exact test 3/3, root times 41.38s, 39.68s, 57.24s, with terminal package PASS (141.075s, exit 0). To respect the managed-lane boundary, the official GitHub archive was exported into an ignored directory in the same lane; all 11,216 files were byte-verified against that archive before and after testing. No Git worktree, checkout or shared-config mutation was used. The failed test and fixture are identical between the PR and main. All 191 tracked native-source/wrapper inputs were also verified identical; both controls use the same read-only Darwin/arm64 native artifacts. Temporary native links were removed by the installed cleanup trap.

    Both local controls used Go 1.27.1 on Darwin/arm64, retaining the CI race/tags/CGo/short mode:

    .agents/skills/mo-dev/scripts/mo-cgo-test -v -race -short \
  -tags matrixone_test -vet=off -count=3 -p=1 -timeout=40m \
  -run '^TestLegacyPartitionColumnLimitCopy$' ./pkg/tests/sqlintegration

    For the exported main control, GIT_WORK_TREE scopes the wrapper's read-only root resolution to that source tree; GIT_OPTIONAL_LOCKS=0 prevents shared-index refresh. The archive SHA-256 is f5899980017a8c5569997caa78e7ce3e8a2c7c65c3f9e9afec5c2ae397dafa5f.

    Additional exact-head CI evidence: the failed race run's raw events contain terminal PASS for all changed owning packages: external 40.034s, function 23.683s, csvparser 1.265s. The same head's shared build, SCA, coverage producer/merge and both selected multi-CN BVT lanes also passed.

    Classification: non-PR flaky candidate under the fresh four-part protocol. These controls do not reproduce the Ubuntu/x86 full-load failure or prove its underlying mechanism is fixed. The absence of the signature in clean-main controls is the expected bounded flaky-candidate result. No unrelated production code or assertions were changed. After recording this evidence, only the failed jobs of run 38715428406 will be rerun once; another occurrence requires renewed attribution.

  22. iamlinjunhong commented on Oct 18, 2026

    @iamlinjunhong
    iamlinjunhong
    on Oct 18, 2026
    Contributor

    Additional occurrence for PR #30171, exact head e9955d8e2ac5c1d379f85d05bc2a0ed0d0dc78ab, branch fix/30158-decimal-div-domain.

    Run 39092472034, attempt 1 is complete. Reusing this embedded-cluster/HAKeeper tracker to retain the matching failure history. This issue is closed; the classification below relies on fresh four-part comparison evidence, rather than the known-open-defect exception.

    Complete-log evidence:

    • Ubuntu/x86 UT, job 117571954516: the only failed root is pkg/tests/issues.TestIssue26118DatabaseCopiesPreserveHashIdentifiers (59.06s).
    • CI Required, job 117582392464: verifyResults propagates matrixone-ci: failure; it has no independent product failure.
    • CI entry: time make ut UT_CONFIGURED=1 UT_PARALLEL=8 UT_TIMEOUT=40 UT_SHARD=all, Go 1.27.1, Linux/amd64, CGO enabled, short/race, matrixone_test, bounded Go package parallelism 2 and 40-minute timeout.

    Normalized failure:

    TestIssue26118DatabaseCopiesPreserveHashIdentifiers
issue_26118_test.go:62 -> issue_test.go:194
Received unexpected error: context deadline exceeded
exec failed: drop database if exists `issue#26118#source`
--- FAIL: TestIssue26118DatabaseCopiesPreserveHashIdentifiers (59.06s)

    The failure occurs in deferred cleanup: user/role removal and five database drops share one 30-second context, and the final source-database DROP times out. The preceding business assertions were not reported as failing. The surrounding window contains LogService/HAKeeper heartbeat/checker/proposal timeouts. Those messages and capacity observations are correlations, not proof of an infrastructure root cause. The full failed-step logs contain no compiler/type/static/security failure, panic, reported data race or OOM kill. The diagnostic JSON artifact was size-truncated; complete failure output remains in the downloaded job logs.

    Fresh four-part comparison:

    1. Both failed jobs' complete failed-step logs were read and the runtime cleanup timeout above was isolated.
    2. At the unchanged exact PR head, the exact failed test passed 3/3, root times 7.33s, 1.52s, 1.52s, with terminal package PASS (12.817s, exit 0).
    3. The complete 26-file PR delta and the failing call path were reviewed. The PR changes DECIMAL integer-DIV domain resolution, its protocol admission, DIV view-boundary provenance, unsigned DIV warnings and their regressions. It does not modify this test, the shared embedded fixture, HAKeeper/Raft transport, database-drop cleanup, or the CI harness. This test uses plain INT/VARCHAR columns with no DECIMAL or DIV; its simple view columns carry zero DecimalDivDisplayLength, so the new DIV view projection is not inserted. The failing deferred DROP DATABASE call does not exercise the changed DIV execution, view-boundary or warning paths. The failed test and shared fixture are byte-identical between the PR and current main.
    4. Fresh official main 3c07fac240311d6f2b706674614ed6034e97ff79 passed the same exact test 3/3, root times 7.20s, 1.57s, 1.54s, with terminal package PASS (12.733s, exit 0).

    Both local comparisons use Go 1.27.1 on Darwin/arm64, retaining the CI race/tags/CGO/short mode:

    .agents/skills/mo-dev/scripts/mo-cgo-test \
  -vet=off -short -v -race -tags matrixone_test -count=3 -p 2 -timeout=40m \
  -run '^TestIssue26118DatabaseCopiesPreserveHashIdentifiers$' ./pkg/tests/issues

    To respect the managed-lane boundary, official main was exported into an ignored directory inside the same lane. All 11,436 source files/symlinks were verified against the Git archive, including bytes/targets and modes, before and after testing. The archive SHA-256 is 9d5f7cfc093edad9ce338dd07520101419df9cd9369682bfb39585bf9e9c8b69. The main wrapper's root was verified to be this exported source tree using GIT_WORK_TREE; GIT_OPTIONAL_LOCKS=0 prevents shared-index refresh. Shared Git metadata remains read-only; main source identity is established by the archive verification. No Git worktree, checkout or shared-config mutation was used. All 193 tracked native/Makefile/wrapper/Go-module inputs match between PR and main; both controls use the same verified read-only Darwin/arm64 native artifacts. The cleanup trap removed both temporary native links, and the PR worktree remains clean.

    Additional evidence from this exact PR head's CI: raw UT events contain terminal PASS for pb/plan, sql/colexec, sql/compile, sql/plan/function, sql/plan/rule, and all eight sql/plan shards. Eleven PR-related DECIMAL integer-DIV/view/warning protocol regression roots found in the artifact passed. The same head's shared build, SCA, coverage producer/merge and both selected multi-CN BVT lanes also passed.

    Classification: non-PR flaky candidate under the fresh four-part protocol. The local controls do not reproduce the full-load Linux/amd64 failure or prove its underlying mechanism is fixed. Clean-main controls passing is the expected bounded flaky-candidate result. No production code, assertions, timeout or skip was changed. After recording this evidence, only the failed jobs of run 39092472034 will be rerun once; a further occurrence requires renewed attribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingseverity/s0Active / top priority for current sprint. Owner has committed to working on it now.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions