Retry on request-interrupted watch failures#68
Conversation
Long-lived Pod watches can be interrupted by transient connection
resets (idle LB/proxy timeouts, apiserver restarts, network blips),
which OpenAPI.jl surfaces as InvocationException("request was
interrupted"). k8s_retry_cond only retried ApiException with
retryable status codes, so this case fell straight through to
callers as an unrecoverable error. Treat it as retryable, matching
the existing is_longpoll_timeout handling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
One mechanism provided to the consumer of watch events to signal a termination of the watch loop is to close the watch stream/channel. Does that also throw an If so, then would this change prevent the loop from terminating as the event consumer would have wanted? |
@tanmaykm pointed out on #68 that closing the watch event channel is the documented way for a consumer to stop an in-progress watch (see the `watch(streamprocessor, ctx, watched, ...)` composition), and that this also surfaces as `is_request_interrupted` in the :downloads backend: the interrupt-watcher task force-interrupts the download on channel close, leaving `resp` nothing, which `exec()` reports identically to a genuine dropped connection. Thread the event/output stream through k8s_retry so k8s_retry_cond can tell the two cases apart: only treat is_request_interrupted as retryable while that channel is still open. A closed channel means the consumer asked to stop, so let it propagate instead of retrying. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Good catch, thanks. Confirmed: in the Pushed a follow-up commit that threads the event/output stream through |
Summary
InvocationException("request was interrupted").k8s_retry_condpreviously only retriedApiExceptionwith specific retryable status codes, so this case propagated straight to callers as an unrecoverable error.OpenAPI.Clients.is_request_interrupted(e)as retryable too, matching the existingis_longpoll_timeouthandling used elsewhere for watch reconnects.Context
Found while investigating a JobLoops production error:
K8sReflector's pod-monitoring watch loop was hitting this exactInvocationExceptionand falling through to a full restart (re-list + re-watch) instead of a quiet reconnect. A companion fix on theK8sReflectorside (JuliaTeam repo) now also treatsis_request_interruptedleniently; this Kuber.jl change closes the gap at the lower retry layer too.Test plan
Pkg.test()in Kuber.jlk8s_retry_condreturns(s, true)for anOpenAPI.Clients.InvocationException("request was interrupted")🤖 Generated with Claude Code