Update get_history in HAHistory to take a copy of the history of an entity to avoid race conditions#3550
Open
AndrewColes wants to merge 1 commit intospringfall2008:mainfrom
Open
Update get_history in HAHistory to take a copy of the history of an entity to avoid race conditions#3550AndrewColes wants to merge 1 commit intospringfall2008:mainfrom
AndrewColes wants to merge 1 commit intospringfall2008:mainfrom
Conversation
…ntity, to avoid race conditions. get_history previously returned (a list containing) the history stored in HAHistory. This history could then be accessed concurrently with changes being made to it by update_entity, with no locking. At a minimum, because update_entity sorts the history -- which is not thread safe -- the history may appear transiently empty during these updates. This race condition caused glitches elsewhere in the code; at a minimum, with load data appearing to be transiently empty, leading to poor estimates. This commit fixes the issue by making get_history return a *copy* of the history; and to hold the lock while making this copy, to ensure thread safety. To militate against the performance impact of this, minute_data in utils.py is updated to take an extra argument -- can_modify_history, default to False. If set to True, this will stop it from taking a copy it would otherwise make of the history object provided to it. fetch.py is then updated to set it to true in some places, as appropriate. Hence the fix has almost no impact on performance. Fixes springfall2008#3511
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
get_history previously returned (a list containing) the history stored in HAHistory.
This history could then be accessed concurrently with changes being made to it by update_entity, with no locking.
At a minimum, because update_entity sorts the history -- which is not thread safe -- the history may appear transiently empty during these updates.
This race condition caused glitches elsewhere in the code; at a minimum, with load data appearing to be transiently empty, leading to poor estimates.
This commit fixes the issue by making get_history return a copy of the history; and to hold the lock while making this copy, to ensure thread safety.
To militate against the performance impact of this, minute_data in utils.py is updated to take an extra argument -- can_modify_history, default to False. If set to True, this will stop it from taking a copy it would otherwise make of the history object provided to it. fetch.py is then updated to set it to true in some places, as appropriate. Hence the fix has almost no impact on performance.
Fixes #3511 . Tested for two calendar days, see attached screenshot: