Log the error when Postgres closes an idle pool connection - #43
Merged
Merged
Conversation
Postgres can close an idle connection of the pg pool, for example in a restart, a failover, or maintenance, or with pg_terminate_backend. Then pg-pool emits "error" on the pool. db() attached no listener, so Node stopped the process with "Unhandled 'error' event" (57P01). The gateway and the workflows host both use this pool, so a restart of Postgres stopped both. db() now attaches an "error" listener that logs the error and does not throw it. The pool removes the client before it emits the event, and the next query gets a new connection. tests/store.test.ts emits "error" on the pool. Without the listener, the emit throws. The test connects to no Postgres. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anurag
added a commit
that referenced
this pull request
Sep 24, 2026
withAppLock() checks out a client from the pool for one transaction. claimRunApp (workflows host) and claimDelete (gateway) use it. While a client is checked out, pg-pool does not listen for "error" on it, and the pool listener of #43 gets only the errors of idle clients. If Postgres closes the connection during the transaction, for example in a restart, a failover, or with pg_terminate_backend, the query fails. Then the socket closes, and the client emits "error" ("Connection terminated unexpectedly"). No listener got the event, so Node stopped the process with exit code 1. withAppLock() now attaches an "error" listener after connect() and removes it before release(). The listener logs the error and does not throw it. The error of the query still goes to the caller, so claimRunApp and claimDelete fail as before. release() attaches the listener of the pool again, and the pool removes the closed client. tests/store.test.ts gives claimDelete a fake client. The lock query fails, and the client emits "error" while the rollback waits. Without the listener, the emit throws. The test connects to no Postgres. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
db()inapp/store.tsmade apg.Poolwith no"error"listener. Postgres can close an idle connection of the pool, for example in a restart, a failover, or maintenance, or withpg_terminate_backend. Then pg-pool emits"error"on the pool. With no listener, Node stops the process with exit code 1:The gateway (
app/server.ts) and the workflows host (app/host.ts) both use this pool. Thus a maintenance restart of Render Postgres can stop both processes.Change
db()attaches an"error"listener to the pool. The listener logs the error withconsole.errorand does not throw it. pg-pool removes the client before it emits the event, so the next query gets a new connection. The node-postgres documentation tells an application to always listen for this event.tests/store.test.tsemits"error"on the pool. It expects no throw and one log entry. Without the listener, the emit throws and the test fails. The test opens no connection to Postgres.AGENTS.mdaddsstoreto the list of tests.Verification
npm run checkpasses: lint, typecheck, and 394 tests.GET /ui/appseach second:pg_ctl -m fast stopstopped the gateway with exit code 1 and the stack above.Lost an idle Postgres connection: error: terminating connection due to administrator commandand continues.GET /ui/appsgives 503 while Postgres is stopped, and 200 after Postgres starts again.pg_terminate_backendon the idle connection gives the same result.DATABASE_URL, the logged error does not contain the password. pg keeps the password in a property that is not enumerable.console.errorprints it.Not in this change
A client that
withAppLock()checks out has no"error"listener until it goes back to the pool. If Postgres closes its connection during the transaction, the query fails, and then the client emitsConnection terminated unexpectedly. Node then stops the process. A local check shows this:pg_terminate_backendduringselect pg_sleep(5)in a transaction of the same shape. This change does not fix that path.🤖 Generated with Claude Code