Repository navigation
fix: bug 005 (pilot_report OOM, poller DB reconnect, DB-gated dead-man) + urllib3 2.8.0 - #47
Merged
Merged
Conversation
The snapshot read joined raw_payload with select_related and materialized every row, so ~50 snapshots sharing one page payload each carried a copy of its body. A one-week production window joined 216 payloads into ~565 MB and the Python decode OOM-killed PostgreSQL's container on 2026-10-03. Snapshots are now read as narrow value tuples, streamed with iterator(), and only the newest row per bucket key is kept. The provider is classified in SQL, which also lets the freshness query group by provider instead of by every distinct raw endpoint. Run reads defer ProviderRun.import_listing_ids, run_output and ScraperRun.error, and stream. Report output is unchanged.
Django recycles connections only at HTTP request boundaries. The poller is not a request handler, so after PostgreSQL restarted on 2026-10-03 the dead connection stayed cached in the sync_to_async thread and every job raised 'the connection is closed' until the service was restarted. ConnectionHygieneExecutor, the scheduler's default executor, runs close_old_connections before and after each job on the thread that owns the job's ORM connection: through sync_to_async for coroutine jobs, inline in the worker for plain-function jobs. Regression tests sever the driver connection and require the following jobs to succeed, with CONN_MAX_AGE 0 and with persistent connections.
…able Spec 18.5 defines the push as a dead-man's switch that alerts on the absence of success, and ADR-0017 relies on it to surface a stalled poller off the box. deadman_job pushed on bare process liveness, so on 2026-10-03 the Uptime Kuma monitor stayed green while every collection job failed. The job now probes the database with SELECT 1 on the jobs' own sync_to_async thread and skips the push when the probe fails. The probe never raises, so pushing resumes as soon as the database is back.
…ery test The persistent-connection variant left the sync_to_async thread holding a connection with no close_at, so test_poller_deadman reused it and its unreachable-port probe never dialed; the failure appeared only in suite order.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pilot_reportjoined a shared ~28 KB raw page payload per snapshot (565 MB through the join) and OOM-killed PostgreSQL in the 4 GiB CT; the poller then failed every job on dead connections until restarted.982eebepilot_report reads narrow, streamed value rows; provider classified in SQL.8ac812aConnectionHygieneExecutorrecycles Django DB connections around every scheduled job.60a19b4dead-man push withheld while the database is unreachable (spec §18.5, ADR-0017).017d8f9test isolation fix for the shared sync_to_async thread.99b0942urllib3 2.7.0 -> 2.8.0 (PYSEC-2026-4175/4176/4177; runtime via scrapy).No migration, settings, or admission change. eBay x CPU stays the only admitted cell.
Verification
Local full gate (
scripts.check): ruff format/check, basedpyright, 3255 passed / 2 opt-in skips, 96% coverage, pip-audit clean, Actor gate 75 passed / 99%.