[Bug]: CI late_snapshot TRUNCATE test observes original table ID #30324

Is there an existing issue for the same bug?

• I have checked the existing issues.

Searches covered the exact root/subtest, late_snapshot, replacement revalidation, unchanged table ID, and related #29400 reports across open and closed issues. Open #30161 concerns the same root's shared two-minute context expiring; this failure is a table-ID assertion approximately 15 seconds into the root. Open #28317 concerns a View/SNAPSHOT lock-order deadlock, not this completed-TRUNCATE/catalog-read assertion. No matching issue was found.

Branch Name

codex/loom/develop/ed02c41e1648123295bbd560c4fb97f60da4539bd92bad6dd784f0b203b4a6e3 (PR #30307); official-main control main.

Commit ID

PR head: 104aa83d33c4f3618bd3f7f3400e31828091b5c6.
Clean current official-main control: 94da0308ad0c5d4fb1d9912293339218bf26dfa3.

Other Environment Information

• Original CI: Ubuntu/Linux x86_64, Go 1.27.1, CGo, short mode, matrixone_test tag, non-race coverage with -covermode=set, 506-package cover scope and package parallelism 8.
• Run 38750837210, attempt 1: https://github.com/matrixorigin/matrixone/actions/runs/38750837210
• Local controls: Darwin/arm64, Go 1.27.1, CGO_ENABLED=1, repository CGo test environment. The focused selection and platform differences are explicit limitations below.

Actual Behavior

The complete UT Coverage on Ubuntu/x86 job 116477066160 log was downloaded and inspected. Its only failing root is TestIssue29400CopyAlterRetainedGatePromotionFastFails, specifically replacement_revalidation/late_snapshot:

issue_29400_test.go:221:
Error: Should not be: 0x43759
Test: TestIssue29400CopyAlterRetainedGatePromotionFastFails/replacement_revalidation/late_snapshot
--- FAIL: .../late_snapshot (1.00s)
--- FAIL: TestIssue29400CopyAlterRetainedGatePromotionFastFails (15.33s)

The failed assertion expects idOf(table) to differ from the original physical ID after the truncate worker succeeds. idOf queries mo_catalog.mo_tables through CN0. The worker executes TRUNCATE through CN1, paused at the negative-preflight gate while CN0 creates a table snapshot, then resumes after the gate is released.

The same log records this ordering:

14:50:14.148 DataBranch-Alter-Lineage-Preserved old-table-id=276313 new-table-id=274504
14:50:14.180 tn.handle.drop.relation Id:276313 Name:late_snapshot
14:50:14.180 tn.handle.create.relation 274504-late_snapshot
14:50:14.204 tn.handle.commit.with.ddl
issue_29400_test.go:221: Should not be: 0x43759

0x43759 is 276313. Thus physical replacement is logged, while the following CN0 catalog assertion observes the original ID. This is the observed inconsistency, not proof of a particular cache or visibility root cause. It is not the shared-context deadline signature in #30161.

The failed producer exits 1. Coverage job 116505953698 receives prerequisites_ready=false and reports coverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated. CI Required job 116524676326 is an aggregate failure. The run also has a separately attributed heartbeat harness timeout in ordinary UT.

Expected Behavior

Once the TRUNCATE operation completes its late-protection replacement, the subsequent catalog observation must satisfy the test's replacement/lineage contract. Investigate whether the observation synchronization, catalog visibility, or replacement path violates that contract. Preserve the snapshot/history/lineage assertions and controlled gate; do not weaken the table-ID assertion or add timing sleeps/retries/skips to hide the failure.

Steps to Reproduce

The original coverage producer discovers packages with go list -mod=readonly ./..., excludes driver|engine/aoe|engine/memEngine|pkg/catalog, joins that scope for -coverpkg, then runs:

go test -mod=readonly -count=1 -json -short -v -tags matrixone_test \
  -timeout=20m -p 8 -covermode=set -coverprofile=ut-coverage.out \
  -coverpkg="$cover_pkgs" $test_scope

Focused controls used the same coverage-scope derivation (506 package names, identical for PR and main), non-race/short/tag/CGo mode, set instrumentation and parallelism. Three independent processes per source ran:

.agents/skills/mo-dev/scripts/mo-cgo-test -short -v \
  -tags=matrixone_test -count=1 -p=8 -timeout=20m \
  -covermode=set -coverprofile="$round_profile" -coverpkg="$cover_pkgs" \
  -run '^TestIssue29400CopyAlterRetainedGatePromotionFastFails$/^replacement_revalidation$/^late_snapshot$' \
  ./pkg/tests/issues

The Linux failure has not reproduced in these local controls. The baseline archive uses the same wrapper with only Git-root discovery replaced by its explicit verified archive root; Go flags and native linking/loading remain unchanged. All native inputs are copied byte-identically from verified Darwin/arm64 artifacts after confirming Makefile, CGo, thirdparty, go.mod and go.sum sources are identical to main. The adapter is outside exported tracked source.

Additional information

Classification: non-PR flaky candidate under the explicitly authorized three-round comparison rule, not a proven baseline reproduction or a claim that the underlying contract is fixed.

All four attribution conditions are recorded:

1. Complete logs identify only the table-ID assertion above in this producer, without compile/type/static/security failure, timeout or reported data race. Coverage and CI Required are downstream consequences, not a low changed-code coverage verdict.
2. The exact failing subtest passes three consecutive independent PR-head processes, each with terminal named PASS and exit 0. Package durations: 9.237s, 11.617s, 8.321s; subtest durations: 0.11s, 0.13s, 0.11s. These runs have real dependency coverage instrumentation. An earlier uninstrumented diagnostic pass is not counted in the three-round evidence.
3. The complete four-file PR changes only FLOAT/DOUBLE BIT_COUNT truncation and its unit/SQL regressions. The helper is called only by the float BIT_COUNT overloads. This test's snapshot/TRUNCATE, integer catalog ID query, retained gate, authenticated cluster, lockservice and transaction path do not evaluate BIT_COUNT. The PR changes neither those sources nor the fixture/CI harness. The failing case, fixture, compile/disttae/lockservice sources are byte-identical to current main. Main's newer changes concern ALTER rename/FK identity, NTILE, fulltext planning and temporary DELETE; the selected case creates a persistent table and exercises none of those changed branches.
4. Clean official-main source runs the identical focused instrumented command in three independent processes, all exit 0 with named subtest/package PASS. Package durations: 8.936s, 8.223s, 12.787s; subtest durations: 0.15s, 0.13s, 0.30s. The control uses a clean git archive source export to respect the managed-lane ban on another Git worktree/branch switch. All 11,233 exported tracked blobs are unchanged against official Git objects before and after testing; all 45 copied native-input files match SHA256 with the validated PR artifacts, and loader dependency/rpath configuration was inspected.

Limits: local Darwin/arm64 is not Linux/x64. The exact isolated subtest omits earlier sibling tests and full-suite contention; readable local output replaces CI's JSON sink. Passing controls establish the authorized bounded candidate classification, not full-CI equivalence, Linux stability or the precise source of the stale catalog observation.

The PR's own changed BIT_COUNT closure already passed focused/full owning-package tests, incremental vet/lint and two normal 129/129 SQL runs before delivery. Its patch remains unchanged, and exact-head shared build, SCA and both selected BVT lanes pass in this run. No product code, fixture timeout, assertion or skip was changed. After this signature and the separate ordinary-UT heartbeat signature are recorded, request one failed-jobs rerun of run 38750837210. A recurrence requires renewed attribution, not an unlimited rerun loop.