[Bug]: E2E image swap replaces shared Operator and restarts unrelated MO clusters #619

Description

A UDF E2E MatrixOne Operator started as the shared cluster-level operator and reconciled an unrelated, already-running big-data MatrixOne cluster in another namespace. During reconciliation it changed the existing MO configuration and triggered in-place restarts of LogService, DN, and both Proxy containers. The second Proxy restart terminated an active 100M-row UPDATE connection.

This is not a second MatrixOne cluster and not a query-triggered OOM. It is cross-test/operator isolation: a test Operator must not mutate workloads outside its own test namespace.

Environment

• Date: 2026-09-17
• Affected namespace: mo-big-data-commit-d2a3391-suite-1y
• MatrixOne commit: d2a3391 (4.2-dev)
• Nightly regression run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/35169294962
• Failed job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/35169294962/job/105058569478
• Topology: 3 LogService, 1 DN, 3 CN, 2 Proxy
• Operator pod: matrixone/moop-tke-guangzhou-matrixone-helm-operator-matrixone-operargkr6
• Operator build path printed in its stack: /tmp/mo-operator-udf-e2e-20260917
• Exact Operator commit was not printed in the runtime logs.

Steps to reproduce

1. Keep an existing MO cluster running in namespace mo-big-data-commit-d2a3391-suite-1y.

2. Start the UDF E2E Operator as a cluster-level controller rather than scoping it to the E2E namespace.

3. Run a long query through Proxy. In this incident the query was:

   UPDATE big_data_test.table_with_pk_index_for_write_100M_copy01
   SET col3 = col3 + 1;

4. Observe the Operator reconcile the unrelated namespace and update the existing ConfigMap.

5. Observe LogService, DN, and Proxy containers restart in place and the active client connection fail.

Actual behavior

The UDF E2E Operator started its controllers at 03:30:10Z and immediately reconciled the unrelated big-data namespace. Its logs included:

config key start.sh is not in use, will be deleted.
configmap mo-big-data-commit-d2a3391-suite-1y/nightly-regression-dis-log-config

config key logservice.toml-620d8e6 is not in use, will be deleted.
configmap mo-big-data-commit-d2a3391-suite-1y/nightly-regression-dis-log-config

The same Operator repeatedly reported that the existing cluster used a configuration incompatible with this Operator build:

LegacyPythonUdfSidecarUnsupported: recreate the function with spec.udfWorker

Component restart timeline:

The Proxy shutdown and client failure were approximately 6 ms apart. Pod UIDs remained unchanged while restart counts increased, consistent with OpenKruise in-place updates.

The UPDATE did not partially commit. The post-failure SUM differed from the expected value by exactly 100,000,000, showing that all 100M row modifications were rolled back.

Expected behavior

• An Operator deployed for an E2E test must only watch and mutate resources owned by that test.
• Starting or replacing an E2E Operator must not reconcile or restart MO components in unrelated namespaces.
• Existing client connections in unrelated clusters must not be interrupted.

Stability and controls

• The same LOG/DN/Proxy group accumulated a second restart later in the same run, while all three CNs remained at zero restarts.
• No container reported OOMKilled.
• DN peak working set was about 24.3 GiB with a 55 GiB limit; CN peaks were also below the 55 GiB limits.
• DN exit 137 occurred after it had received SIGTERM and exceeded the shutdown grace period; it was not a cgroup OOM kill.
• Queries executed after reconnection succeeded, confirming a transient infrastructure-induced connection interruption rather than persistent database failure.

Code analysis

SyncConfigMap documents that ConfigMap data changes cause a rolling update. For OperatorVersion >= 1.3, the generated configuration suffix is propagated to the Pod template; OpenKruise then restarts the container in place.

Relevant code:

• pkg/controllers/common/configmap.go: ConfigMap synchronization and deletion of no-longer-used keys
• pkg/controllers/common/podset.go: matrixorigin.io/config-suffix propagation
• LogSet/DNSet/ProxySet controllers: in-place ConfigMap update support

The immediate trigger in this incident was the UDF E2E Operator reconciling a pre-existing cluster outside its test scope. The configuration-diff rollout risk is related to #606, but this issue tracks the missing E2E/operator namespace isolation and the resulting real service interruption.

Proposed fix

1. Scope E2E Operator watches to the E2E namespace (WATCH_NAMESPACE or equivalent cache selector).
2. Do not replace or reuse the shared cluster-level Operator deployment for E2E binaries.
3. Give E2E Operator deployments unique leader-election IDs and resource ownership boundaries.
4. Add a pre-existing sentinel MO cluster to the Operator E2E environment and assert that its CR, ConfigMaps, pod UIDs, config suffixes, and restart counters do not change.
5. Add a guard that prevents a test Operator from starting with unrestricted cluster scope in a shared environment.

Related

• Actual nightly failure: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/35169294962/job/105058569478
• Related rollout risk: [Upgrade Risk]: MergeDeep configuration changes may trigger concurrent rolling updates across MO components #606