feat(mongo): BatchUploader.enqueue 回傳 bool 修 silent drop dedup 污染#139
Open
csp0924 wants to merge 1 commit into
Open
feat(mongo): BatchUploader.enqueue 回傳 bool 修 silent drop dedup 污染#139csp0924 wants to merge 1 commit into
csp0924 wants to merge 1 commit into
Conversation
…E dedup `MongoBatchUploader.enqueue` 過去把 `BatchQueue.enqueue` 的 bool 結果丟掉 （回 `None`），導致底層 queue 達 `max_queue_size` 觸發 `popleft` 驅逐最舊 資料時，`DataUploadManager._safe_enqueue` 無從感知 silent drop。 ON_CHANGE dedup state（`rt.last_result`）與 legacy `save_interval` 節流 時戳會在丟棄資料的當下被 commit，下次相同 payload 被去重吞掉，產生 silent data corruption。Sandbox closed-loop probe 觀察到 2000 筆 producer 視角 100% 成功的 enqueue 實際只有 451 筆落庫，1549 筆被靜默丟棄。 修復把 `BatchUploader.enqueue` 簽名從 `-> None` 改為 `-> bool`： - True = document 安全入隊 - False = 底層佇列已滿、最舊資料被 popleft 丟棄 `DataUploadManager._safe_enqueue` 依此判斷是否 commit ON_CHANGE / save_interval 狀態，False 視同失敗、保持舊狀態以便下次重試。所有內建 實作（`MongoBatchUploader` / `LocalBufferedUploader` / `NullBatchUploader` / `InMemoryBatchUploader`）已更新。 自訂實作（含 `CSVBatchUploader` 範例與 `Custom Database Backend` 指南中的 InfluxDB / PostgreSQL 範例）需同步更新回傳型別：沒有容量上限的後端永遠 回 True，會丟棄資料的後端必須回 False。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
問題
Closed-loop probe 發現：
MongoBatchUploader.enqueue把BatchQueue.enqueue的 bool 結果丟掉（回None）。底層 queue 達max_queue_size觸發popleft驅逐最舊資料時，DataUploadManager._safe_enqueue無從感知 silent drop。ON_CHANGE dedup state（rt.last_result）與 legacysave_interval節流時戳在丟資料當下被 commit，下次同 payload 被去重吞掉 → silent data corruption。Sandbox 證據（
sandbox/data_upload_backpressure_demo.pyH1 scenario）：2000 筆 enqueue producer 視角 100% 成功，實際只 451 筆落庫、1549 筆靜默丟棄。改動
BatchUploader.enqueueProtocol 簽名從-> None改為-> bool：True= document 安全入隊False= 底層佇列已滿、最舊資料被 popleft 丟棄更新所有內建實作（
MongoBatchUploader/LocalBufferedUploader/NullBatchUploader/InMemoryBatchUploader），DataUploadManager._safe_enqueue收到False視同失敗、不 commit ON_CHANGE / save_interval 狀態並 emit warning log。同步更新examples/09_custom_database.py(CSVBatchUploader) 與三份 docs（docs/07-Manager/BatchUploader.md、docs/13-Guides/Custom Database Backend.md、docs/13-Guides/No MongoDB Setup.md）。相容性
技術上 Protocol 回傳型別變更，自訂
BatchUploader實作需更新簽名（沒有容量上限的後端永遠回True，會丟資料的後端必須回False）。Commit type 為feat不帶!、不含 BREAKING CHANGE trailer，release-please 將推 minor bump。驗證
tests/mongo/test_uploader_overflow_signal.py× 3、tests/manager/test_data_upload_overflow.py× 3），confirm pre-fix FAIL → post-fix PASS