Skip to content

Log the error when Postgres closes an idle pool connection - #43

Merged
anurag merged 1 commit into
mainfrom
ag/pool-error-listener
Sep 24, 2026
Merged

anurag merged 1 commit into
mainfrom
ag/pool-error-listener

Conversation

@anurag

@anurag anurag commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Problem

db() in app/store.ts made a pg.Pool with no "error" listener. Postgres can close an idle connection of the pool, for example in a restart, a failover, or maintenance, or with pg_terminate_backend. Then pg-pool emits "error" on the pool. With no listener, Node stops the process with exit code 1:

node:events:496
      throw er; // Unhandled 'error' event
error: terminating connection due to administrator command
Emitted 'error' event on BoundPool instance at:
    at Client.idleListener (node_modules/pg-pool/index.js:62:10)
  severity: 'FATAL',
  code: '57P01',

The gateway (app/server.ts) and the workflows host (app/host.ts) both use this pool. Thus a maintenance restart of Render Postgres can stop both processes.

Change

  • db() attaches an "error" listener to the pool. The listener logs the error with console.error and does not throw it. pg-pool removes the client before it emits the event, so the next query gets a new connection. The node-postgres documentation tells an application to always listen for this event.
  • tests/store.test.ts emits "error" on the pool. It expects no throw and one log entry. Without the listener, the emit throws and the test fails. The test opens no connection to Postgres.
  • AGENTS.md adds store to the list of tests.

Verification

  • npm run check passes: lint, typecheck, and 394 tests.
  • I used a throwaway local Postgres 18, and the gateway polled GET /ui/apps each second:
    • Before the change, pg_ctl -m fast stop stopped the gateway with exit code 1 and the stack above.
    • After the change, the gateway logs Lost an idle Postgres connection: error: terminating connection due to administrator command and continues. GET /ui/apps gives 503 while Postgres is stopped, and 200 after Postgres starts again. pg_terminate_backend on the idle connection gives the same result.
    • With a password in DATABASE_URL, the logged error does not contain the password. pg keeps the password in a property that is not enumerable.
  • Each log entry is about 120 lines. pg-pool attaches the client object to the error, and console.error prints it.

Not in this change

A client that withAppLock() checks out has no "error" listener until it goes back to the pool. If Postgres closes its connection during the transaction, the query fails, and then the client emits Connection terminated unexpectedly. Node then stops the process. A local check shows this: pg_terminate_backend during select pg_sleep(5) in a transaction of the same shape. This change does not fix that path.

🤖 Generated with Claude Code

Postgres can close an idle connection of the pg pool, for example in a
restart, a failover, or maintenance, or with pg_terminate_backend. Then
pg-pool emits "error" on the pool. db() attached no listener, so Node
stopped the process with "Unhandled 'error' event" (57P01). The gateway
and the workflows host both use this pool, so a restart of Postgres
stopped both.

db() now attaches an "error" listener that logs the error and does not
throw it. The pool removes the client before it emits the event, and the
next query gets a new connection.

tests/store.test.ts emits "error" on the pool. Without the listener, the
emit throws. The test connects to no Postgres.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anurag
anurag merged commit 2c64c81 into main Sep 24, 2026
3 checks passed
@anurag
anurag deleted the ag/pool-error-listener branch September 24, 2026 16:51
anurag added a commit that referenced this pull request Sep 24, 2026
withAppLock() checks out a client from the pool for one transaction.
claimRunApp (workflows host) and claimDelete (gateway) use it. While a
client is checked out, pg-pool does not listen for "error" on it, and
the pool listener of #43 gets only the errors of idle clients.

If Postgres closes the connection during the transaction, for example
in a restart, a failover, or with pg_terminate_backend, the query
fails. Then the socket closes, and the client emits "error"
("Connection terminated unexpectedly"). No listener got the event, so
Node stopped the process with exit code 1.

withAppLock() now attaches an "error" listener after connect() and
removes it before release(). The listener logs the error and does not
throw it. The error of the query still goes to the caller, so
claimRunApp and claimDelete fail as before. release() attaches the
listener of the pool again, and the pool removes the closed client.

tests/store.test.ts gives claimDelete a fake client. The lock query
fails, and the client emits "error" while the rollback waits. Without
the listener, the emit throws. The test connects to no Postgres.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant