Repository navigation
Conversation
An asyncio.Lock belongs to the loop that first waits on it. One lock per key raised "is bound to a different event loop" when a second loop loaded the same key. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The lock map is keyed by (loop, key), and the Python interface runs each call on a new loop, so entries (and the closed loops they reference) accumulated. An entry is now removed when its load ends. Waiters keep the lock object and read the cache once they hold it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Congress bulk loaders can fail with
congress/utils/bulk.py::_memoizedkeeps oneasyncio.Lockper cache key in the module-level_LOAD_LOCKS. Anasyncio.Lockbinds to the event loop of its first contended wait. If a load of the same key is then waited on from another event loop while the first is still in flight,acquireraises instead of waiting.That needs commands running on more than one event loop in the same process, which the Python interface does when it is called from two threads: each call runs on its own loop. On a single loop the lock behaves as before.
Change
(running loop, key). Loads on one loop still share a single download; a load on another loop gets its own lock instead of an exception._LOAD_LOCKSholds only the loads in flight. The Python interface uses a new loop for every call; without this, the map would gain an entry per loop and keep each closed loop alive. Waiters keep the lock object itself and read the cache once they hold it. Ondevelopthe map also grew by one entry per key and was never pruned; that is gone too.The cost: two loops asking for the same cold key at the same moment can each run the loader once. Both store the same records, so the result is the same; only the download is repeated. The same can happen on one loop after a load fails, when a waiter retries while a new caller starts.
Tests
test_memoized_load_lock_is_per_event_loop: one loop holds a contended load in a thread while a second loop loads the same key. Ondevelopit fails with theRuntimeErrorabove. It also asserts that no lock for the key is left once both loads end; that assertion fails on this branch's first commit.test_memoized_dedupes_concurrent_loadsstill passes (one download for two concurrent loads on one loop).openbb_platform/providers/government_us/tests: 3,261 passed (Python 3.12).🤖 Generated with Claude Code