Skip to content

Make IoTConsensus peer-list reset retries best effort and fix Daily CI regressions - #18929

Merged
CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-stale-peers-after-restart-followup
Oct 20, 2026
Merged

CRZbulabula merged 1 commit into
apache:masterfrom
d-wang-commit:fix-stale-peers-after-restart-followup

Conversation

@d-wang-commit

@d-wang-commit d-wang-commit commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #18892: make peer-list reset retry exhaustion non-fatal after replica metadata removal has succeeded, and fix the affected regression fixtures used by Daily CI.

Make peer-list reconciliation best effort

After ConfigNode commits a Region peer removal, a retained DataNode can remain unavailable while ConfigNode retries resetting its peer list. Previously, exhausting these retries failed the child procedure and rolled back the enclosing DataNode removal even though the replica metadata had already been updated.

Allow IoTConsensus V1 peer-list reconciliation to finish without failing the procedure after its bounded retries are exhausted. Metadata removal failures remain fatal and retain the existing parent cleanup. Simplify retryStep to accept an optional caller-supplied failure, preserve the original cause of metadata exceptions, and remove duplicate retry logging while retaining the existing business failure logs.

Fix Daily CI regression fixtures

The targeted Region lookup API now queries the Region's database and then its replica set. Fixtures that only mocked getAllReplicaSets() returned no database and caused reconciliation to be skipped. Update the two affected regression fixtures to delegate getRegionDatabase() and getReplicaSets() to the live PartitionInfo, including the deleted-Region case.

Tests

  • Updated UTs: RestartDuringRemovePeerTest, RemoveDataNodesProcedureTest, and RemovePeerRestartConsensusTest. Cover reset exhaustion returning success, metadata exhaustion preserving failure and cleanup, and real IoTConsensus peer repair followed by continued writes.
  • Passed all 29 UT cases after rebasing: the three classes above plus RegionReplicaSetLookupTest (including DataRegion and SchemaRegion parameters). No failures, errors, or skipped tests.
  • No new or modified ITs. Passed all three existing cluster cases below, with no failures, errors, or skipped tests:
    • IoTDBRegionMigrateConfigNodeCrashIoTV1IT#cnCrashDuringRemoveRegionLocationCacheTest.
    • IoTDBRegionMigrateConfigNodeCrashForRatisIT#cnCrashDuringRemoveRegionLocationCacheTest.
    • IoTDBRemoveDataNodeNormalIT#success1C4DIoTTestUseTableSQL.
  • Spotless passed. Java compilation passed for both English and Chinese locales (52-module reactor, excluding distribution).

Side effects and risks

A retained DataNode that stays unavailable beyond the retry window may retain a stale peer list after the procedure completes; this change does not schedule further reconciliation. Metadata failure handling and parent rollback cleanup remain unchanged. There are no DataNode production-code changes or procedure serialization changes.


This PR has:

  • been self-reviewed.
  • been tested in a test IoTDB cluster.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • RemoveRegionPeerProcedure: distinguish fatal metadata failures from best-effort peer-list reconciliation through caller-supplied retry failures.
  • ProcedureMessages: remove redundant retry log messages in both locales.
  • RestartDuringRemovePeerTest, RemoveDataNodesProcedureTest, and RemovePeerRestartConsensusTest: validate exhaustion behavior and adapt fixtures to targeted Region lookups.

Finish node removal after bounded peer-list reset retries are exhausted.
Keep metadata failures fatal and simplify caller-controlled retry handling.
Cover exhaustion paths and adapt regression fixtures to targeted region lookups.

@CRZbulabula CRZbulabula left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CRZbulabula
CRZbulabula merged commit c462a97 into apache:master Oct 20, 2026
38 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants