[danielle-mustillo]

Why

When the controller can't log in to PostgreSQL, for example during a restart, a failover, or a serverless database resuming from a pause, every API call fails at once with Internal "internal server error". Nothing is logged, and clients can't tell that a retry would work.

What changed

• Retry (second commit). Store calls retry a failed login with backoff for up to database.pool.connectRetryTimeout (default 90s, 0s turns it off). No SQL has been sent when a login fails, so a retry never repeats a statement. The caller's context still ends the retry.
• Error code (first commit). A failed login returns Unavailable "database unavailable", and the cause is logged. A login PostgreSQL rejects (28P01, 28000, 3D000, 42501) stays Internal and isn't retried. A database failure during runtime auth returns Unavailable instead of Unauthenticated.
• A login's TCP dial stops after 10 s, so a host that never answers can't hold one attempt for 2 minutes.
• helm/README.md explains the retry window and how client timeouts cut it short. The web UI waits 30 s.

Scope

The two commits are separate on purpose. The first one (error code and log) is useful without the retry.

Blast radius

This touches every store call. A call that used to fail at once can now wait up to 90 s before it fails. Set connectRetryTimeout: 0s to get the old behavior back.

Known limits

• A proxy in front of PostgreSQL that replaces the server's error with a generic one hides a wrong password. The controller then retries it for the full window and reports Unavailable.
• A query that fails after login, such as one killed by a failover (57P01), isn't retried. Runtime auth still reports it as Unauthenticated.

Verification

• Unit tests put a proxy in front of a real PostgreSQL that drops, holds or rejects logins.
• Real Aurora Serverless v2 resuming from a pause, behind an IAM auth proxy that cuts each login at 15 s. Each attempt was the first request after CloudWatch showed 0 capacity. With the default 90s window, 2 of 2 session creates succeeded in 18.4 s and 20.4 s, after one retry each. With 0s, 2 of 2 failed with Unavailable at 15.3 s.
• Kind cluster, database cut off with a NetworkPolicy, runtime auth: before, Unauthenticated after 30 s with no cause logged. After, Unavailable after 20.5 s (20 s window) with the cause logged.
• Kind cluster, login role set to NOLOGIN: Internal in 0.3 s after one attempt, with SQLSTATE 28000 logged.
• Tests that fail on this branch (TestSessionExpiration*, TestEgressPolicyCatalog, TestSandboxTemplateCatalog) fail the same way on main.

[Copilot]

🟡 Changes recommended

The retry loop can start attempts after its time budget expires, while rejected database logins bypass logging and incorrectly return Unauthenticated instead of Internal.

2 open issues

• Recheck retry budget after backoff before starting another attempt · New
• Map and log rejected database login errors · New

1 file not reviewed

• go/api/gen/kagent/config/v1/config.pb.go: Generated file

What changed in this PR

This PR improves how the controller handles PostgreSQL login failures, adding configurable retries and clearer gRPC errors without retrying statements that have already run.

Changes:

• Adds login retries with backoff, a default 90-second retry window, and a 10-second default TCP dial timeout.
• Maps transient login failures to Unavailable and logs server-side error causes.
• Adds configuration validation, failure-path tests, and Helm documentation.

File	Description
proto/​kagent/​config/​v1/​config.proto	Defines and validates the retry timeout.
helm/​README.md	Explains retries and client timeout interactions.
helm/​kagent/​values.yaml	Adds a commented retry configuration example.
go/​core/​pkg/​postgres/​pool.go	Bounds TCP dialing when no connection timeout is configured.
go/​core/​pkg/​config/​load_test.go	Tests retry timeout validation and defaults.
go/​core/​pkg/​config/​helm_test.go	Tests Helm defaults and disabling retries.
go/​core/​pkg/​config/​defaults.go	Sets the default retry window to 90 seconds.
go/​core/​pkg/​app/​app.go	Passes retry configuration into the store.
go/​core/​internal/​grpcserver/​interceptors.go	Maps database failures and logs error causes.
go/​core/​internal/​grpcserver/​interceptors_test.go	Tests error mapping, logging, and runtime authentication failures.
go/​core/​internal/​dbtest/​dbtest.go	Adds a fake server that rejects PostgreSQL logins.
go/​core/​internal/​database/​sql.go	Wraps pool operations with login retries.
go/​core/​internal/​database/​session_deletion_test.go	Adapts client reconstruction to the pool wrapper.
go/​core/​internal/​database/​push_notifications_test.go	Uses the shared pool for direct test queries.
go/​core/​internal/​database/​errors.go	Classifies retryable login failures.
go/​core/​internal/​database/​errors_test.go	Tests login failure classification.
go/​core/​internal/​database/​connect_retry_test.go	Tests recovery, retry limits, and statement safety.
go/​core/​internal/​database/​client_postgres.go	Adds the retry option and integrates the pool wrapper.
go/​api/​gen/​kagent/​config/​v1/​config.pb.go	Regenerates protobuf bindings for the retry setting.

_{Review effort: Balanced}