Skip to content

fix(government_us): Congress bulk load lock is per event loop - #7845

Open
artcashin wants to merge 2 commits into
openbq-org:developfrom
artcashin:fix/congress-load-lock-per-loop
Open

artcashin wants to merge 2 commits into
openbq-org:developfrom
artcashin:fix/congress-load-lock-per-loop

Conversation

@artcashin

@artcashin artcashin commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Congress bulk loaders can fail with

RuntimeError: <asyncio.locks.Lock object [locked, waiters:1]> is bound to a different event loop

congress/utils/bulk.py::_memoized keeps one asyncio.Lock per cache key in the module-level _LOAD_LOCKS. An asyncio.Lock binds to the event loop of its first contended wait. If a load of the same key is then waited on from another event loop while the first is still in flight, acquire raises instead of waiting.

That needs commands running on more than one event loop in the same process, which the Python interface does when it is called from two threads: each call runs on its own loop. On a single loop the lock behaves as before.

Change

  • The lock is keyed by (running loop, key). Loads on one loop still share a single download; a load on another loop gets its own lock instead of an exception.
  • An entry is removed when its load ends, so _LOAD_LOCKS holds only the loads in flight. The Python interface uses a new loop for every call; without this, the map would gain an entry per loop and keep each closed loop alive. Waiters keep the lock object itself and read the cache once they hold it. On develop the map also grew by one entry per key and was never pruned; that is gone too.

The cost: two loops asking for the same cold key at the same moment can each run the loader once. Both store the same records, so the result is the same; only the download is repeated. The same can happen on one loop after a load fails, when a waiter retries while a new caller starts.

Tests

  • test_memoized_load_lock_is_per_event_loop: one loop holds a contended load in a thread while a second loop loads the same key. On develop it fails with the RuntimeError above. It also asserts that no lock for the key is left once both loads end; that assertion fails on this branch's first commit.
  • test_memoized_dedupes_concurrent_loads still passes (one download for two concurrent loads on one loop).
  • openbb_platform/providers/government_us/tests: 3,261 passed (Python 3.12).

🤖 Generated with Claude Code

An asyncio.Lock belongs to the loop that first waits on it. One lock per key
raised "is bound to a different event loop" when a second loop loaded the same key.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The lock map is keyed by (loop, key), and the Python interface runs each call
on a new loop, so entries (and the closed loops they reference) accumulated.
An entry is now removed when its load ends. Waiters keep the lock object and
read the cache once they hold it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant