Skip to content

Bound exporter gRPC queries with client keepalive and request deadlines - #11

Merged
masih merged 1 commit into
mainfrom
masih/1790088183-grpc-client-keepalive
Sep 25, 2026
Merged

masih merged 1 commit into
mainfrom
masih/1790088183-grpc-client-keepalive

Conversation

@masih

@masih masih commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

The gRPC-backed endpoints (/metrics/params, general, validators, wallet, validator, oracle) all share one grpc.ClientConn to seid, and every query on it used context.Background(). When that connection stops making progress, requests queue forever, each Prometheus scrape hits its 10s timeout, and nothing on the exporter side ever tears the transport down. On pacific-1 the only thing that eventually does is grpc-go's server-side keepalive default in seid (2h ping, 20s timeout), which is why the snapshotter exporters have been going up == 0 for exactly 2h00m15s at a time (Sept 8 13:51–15:51, Sept 15 01:07–03:07 and 03:16–05:17 UTC) and paging SnapshotOrGenesisMetricsEndpointUnreachable. The replace google.golang.org/grpc => v1.33.2 pin also kept the client on a 2020 transport; the module builds cleanly against the already-required v1.53.0 without it.

Each handler now derives its context via requestContext(r), which is context.WithTimeout(r.Context(), RequestTimeout) with a new --request-timeout flag defaulting to 8s, so a stuck query fails within the scrape window and is cancelled when the scraper disconnects. The dial adds keepalive.ClientParameters{Time: 5m, Timeout: 20s, PermitWithoutStream: true} (5m matches seid's keepalive-min-time, so the pings are not penalised) so a dead connection is detected and redialed within minutes instead of hours. grpc.WithInsecure() is replaced by grpc.WithTransportCredentials(insecure.NewCredentials()), and the grpc replace pin is dropped.

@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@masih
masih marked this pull request as ready for review September 22, 2026 14:49
@masih
masih requested a review from alexander-sei September 22, 2026 14:49
@cursor

cursor Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches all metrics endpoints' gRPC behavior and upgrades grpc-go; mis-tuned timeouts could cause partial scrapes, but the change targets production stuck-connection outages.

Overview
Metrics scrape handlers no longer use unbounded context.Background() for gRPC. Each handler derives a context from requestContext(r) (context.WithTimeout on the HTTP request, default 8s via new --request-timeout), so stuck node queries fail inside the scrape window and cancel when the client disconnects.

The shared gRPC dial now uses insecure.NewCredentials() instead of deprecated WithInsecure, adds client keepalive (5m ping, 20s timeout, permit without stream), and go.mod drops the grpc v1.33.2 replace so the build uses v1.53.0 already in require.

Reviewed by Cursor Bugbot for commit 3374d5d. Bugbot is set up for automated code reviews on this repo. Configure here.

@masih
masih enabled auto-merge September 22, 2026 14:53
@masih
masih added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 5c2a402 Sep 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants