fix(integration): SetpointDriftReconciler 透過 LeaderGate 阻擋 follower 寫#140
Open
csp0924 wants to merge 2 commits into
Open
fix(integration): SetpointDriftReconciler 透過 LeaderGate 阻擋 follower 寫#140csp0924 wants to merge 2 commits into
csp0924 wants to merge 2 commits into
Conversation
HA cluster 場景下，CommandRouter.try_write_single 無 leader_gate 守門， follower 節點的 SetpointDriftReconciler 仍會對共用設備發出寫入 → 破壞 single-writer invariant、產生 dual-write window。 修法（窄面 patch）： - SetpointDriftReconciler.__init__ 新增 leader_gate: LeaderGate | None = None 關鍵字參數（向後相容，patch bump）。 - _reconcile_work 在入口判斷 leader_gate.is_leader：非 leader 直接早退， detail['paused'] = 'not_leader'；不視為 unhealthy（被 gate 擋是正常運作）。 設計權衡： - CommandRouter 保持中性（strategy path 在 follower 仍可跑 shadow-mode compute）， 只在 reconciler 本身守門。對齊 WriteCommandManager.execute 的 gate 早期拒絕設計。 - 未注入 gate（單節點）行為不變。 Sandbox 量化驗證： - pre-fix: follower_writes=1（dual-write 確認存在） - post-fix: follower_writes=0, detail['paused']='not_leader'
There was a problem hiding this comment.
Pull request overview
此 PR 針對 HA/cluster 部署下的 single-writer invariant 問題，讓 SetpointDriftReconciler 在注入 LeaderGate 後，於 follower 節點直接暫停 reconcile-driven 寫入，避免 follower 與 leader 同時對共享實體設備雙寫；並補上對應的整合測試以驗證 follower/leader/無 gate/升格等情境。
Changes:
SetpointDriftReconciler新增leader_gate: LeaderGate | None = None參數，並在_reconcile_work開頭以is_leader做 follower 早退。- 新增
tests/integration/test_setpoint_drift_reconciler.py::TestLeaderGate，覆蓋 follower 不寫、leader 正常寫、向後相容、升格後恢復寫入。
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| csp_lib/integration/setpoint_drift_reconciler.py | 注入 LeaderGate 並在 follower 節點早退以阻止 reconciler-driven 寫入 |
| tests/integration/test_setpoint_drift_reconciler.py | 新增 LeaderGate 行為測試覆蓋，驗證 follower/leader/相容性/升格情境 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+133
to
+136
| # Leader gate：非 leader 直接早退，零寫入；不視為 unhealthy（被 gate 擋是正常運作） | ||
| if self._leader_gate is not None and not self._leader_gate.is_leader: | ||
| detail["paused"] = "not_leader" | ||
| return |
LeaderGate 早退分支原本在初始化 detail["drift_count"] / detail["devices_fixed"]（cooldown 開啟時還有 skipped_by_cooldown）前就 return， 導致 follower 節點的 status.detail 缺這些 key。 examples/16_operator_pattern.py 等呼叫端用 status.detail["drift_count"] 直接 index → follower 上會 KeyError。 修法：把預設值寫入提前到 gate 檢查之前，無論 follower 早退或 迴圈中途 raise，detail 都能保持 drift_count=0 / devices_fixed=() （cooldown 開啟時 skipped_by_cooldown=0）的契約。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
問題
Closed-loop probe 發現：
SetpointDriftReconciler._reconcile_work經CommandRouter.try_write_single寫設備，但CommandRouter完全沒接LeaderGate。HA cluster 下 follower-node 的 reconciler 會跟 leader 同時寫同一個共享物理設備（雙寫違反 single-writer invariant）。對照WriteCommandManager.execute是有 leader_gate 守護的，兩者不對稱。Sandbox 證據（
sandbox/manager_reconciler_race_leader_bypass_demo.py）：dual reconciler 並行 → pre-fix follower writes = 1（應為 0）。改動
SetpointDriftReconciler.__init__新增leader_gate: LeaderGate | None = None（kwarg、預設 None → 向後相容）。_reconcile_work開頭 early-return：CommandRouter保持中立（strategy path 仍需在 follower 跑 shadow compute），修在 reconciler 一層。驗證
tests/integration/test_setpoint_drift_reconciler.py::TestLeaderGate4 案（follower 不寫 / leader 正常寫 / leader_gate=None 向後相容 / 中途 promote 恢復寫）。Pre-fix 3 FAIL → post-fix 全 PASS。follower_writes == 0✓