Skip to content

Retry on request-interrupted watch failures#68

Merged
nkottary merged 2 commits into
masterfrom
retry-request-interrupted
Jul 15, 2026
Merged

Retry on request-interrupted watch failures#68
nkottary merged 2 commits into
masterfrom
retry-request-interrupted

Conversation

@nkottary

@nkottary nkottary commented Jul 15, 2026

Copy link
Copy Markdown
Member

Summary

  • Long-lived Pod watches can be interrupted by transient connection resets (idle LB/proxy timeouts, apiserver restarts, network blips), which OpenAPI.jl surfaces as InvocationException("request was interrupted").
  • k8s_retry_cond previously only retried ApiException with specific retryable status codes, so this case propagated straight to callers as an unrecoverable error.
  • This treats OpenAPI.Clients.is_request_interrupted(e) as retryable too, matching the existing is_longpoll_timeout handling used elsewhere for watch reconnects.
  • Bumped patch version (0.7.10 -> 0.7.11).

Context

Found while investigating a JobLoops production error: K8sReflector's pod-monitoring watch loop was hitting this exact InvocationException and falling through to a full restart (re-list + re-watch) instead of a quiet reconnect. A companion fix on the K8sReflector side (JuliaTeam repo) now also treats is_request_interrupted leniently; this Kuber.jl change closes the gap at the lower retry layer too.

Test plan

  • Pkg.test() in Kuber.jl
  • Manually verify k8s_retry_cond returns (s, true) for an OpenAPI.Clients.InvocationException("request was interrupted")

🤖 Generated with Claude Code

Long-lived Pod watches can be interrupted by transient connection
resets (idle LB/proxy timeouts, apiserver restarts, network blips),
which OpenAPI.jl surfaces as InvocationException("request was
interrupted"). k8s_retry_cond only retried ApiException with
retryable status codes, so this case fell straight through to
callers as an unrecoverable error. Treat it as retryable, matching
the existing is_longpoll_timeout handling.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@nkottary
nkottary requested a review from a team July 15, 2026 04:27
@nkottary
nkottary marked this pull request as ready for review July 15, 2026 04:27
@tanmaykm

Copy link
Copy Markdown
Member

One mechanism provided to the consumer of watch events to signal a termination of the watch loop is to close the watch stream/channel. Does that also throw an is_request_interrupted exception?

If so, then would this change prevent the loop from terminating as the event consumer would have wanted?

@tanmaykm pointed out on #68 that closing the watch event channel is the
documented way for a consumer to stop an in-progress watch (see the
`watch(streamprocessor, ctx, watched, ...)` composition), and that this
also surfaces as `is_request_interrupted` in the :downloads backend: the
interrupt-watcher task force-interrupts the download on channel close,
leaving `resp` nothing, which `exec()` reports identically to a genuine
dropped connection.

Thread the event/output stream through k8s_retry so k8s_retry_cond can
tell the two cases apart: only treat is_request_interrupted as retryable
while that channel is still open. A closed channel means the consumer
asked to stop, so let it propagate instead of retrying.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@nkottary

Copy link
Copy Markdown
Member Author

Good catch, thanks. Confirmed: in the :downloads backend, closing the watch channel is exactly the documented mechanism for a consumer to stop a watch, and it goes through the same interrupt path that leaves resp === nothing in exec() — so it does throw the same InvocationException("request was interrupted") as a genuine dropped connection.

Pushed a follow-up commit that threads the event/output stream through k8s_retry/k8s_retry_cond, so it only treats is_request_interrupted as retryable while that channel is still open. If the channel's already closed, that means the consumer asked to stop, so it's allowed to propagate instead of being retried.

@nkottary
nkottary merged commit 0d91a51 into master Jul 15, 2026
3 checks passed
@nkottary
nkottary deleted the retry-request-interrupted branch July 15, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants