Repository navigation
Conversation
A request-path text or ASS window on a virtual relay source had no bound of its own: the only ceiling was subtitleWindowExtractTimeout (2 min), applied after the leader detached from its request, and the client's 30 s subtitle fetch stall deadline expired first. The deployed build measured 25.9 s and 58.1 s of silence on that path with status 0 (client abandoned), against 24 subtitle requests in the window: p50 1011 ms, p90 49327 ms, max 58147 ms. Bound the extract with an elapsed budget of 20 s: on a breach the extraction is cancelled, the existing per-identity cooldown is armed, and the answer is 503 with Retry-After when nothing was written, or an aborted response when part of the slice had already streamed (never a completed partial 200). The handler starts the detached warm on that answer, and a detached prefetch is no longer paced by that cooldown, so the retry reads a committed slice or a window prefix instead of repeating the read. The bound covers only what a client waits on: not a whole-track extract, not a window read from a committed artifact, not a bitmap page, and not the detached prefetch, which may run to the window timeout because it is what eventually fills the window cache. Opening windows are attempted like any other rather than refused up front. An elapsed bound rather than a byte cap because ffmpeg reads an HTTP input with recv(2), which /proc/PID/io does not count in rchar: measured against a relay that served 24 MB, ffmpeg's rchar stayed at its ~181 KB of startup reads.
Collaborator
Author
|
Superseded by #459, which contains this change plus the rest of the playback reliability bundle. Closing to keep the review surface to one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Related issue: N/A
Validation tasks: none
Selecting an ASS track on a virtual source leaves the web player on "Loading
subtitles…" for 90+ seconds. In the deployed build on 2026-11-15, two
getPlaybackSubtitlerequests returned status 0 after 58,147 ms and25,856 ms — the client had already abandoned them — and two more answered
503 subtitle_source_unavailable. Across the 24 subtitle requests in thatwindow: p50 1,011 ms, p90 49,327 ms, max 58,147 ms. The font bundle is not
the blocker:
/fontsreturns a 2.5 MB bundle stored as 9.4 MB when it ispresent.
The windowed text extract has no bound below the client's own deadline. The
bounds in that path, as they exist on
mainplus #444 and #451:subtitleWindowExtractTimeout = 2 * time.Minute—internal/playback/subtitle_cache.go:282. It is the only ceiling on the extract, and it is longer than the client's 30 s subtitle fetch stall deadline (web/src/player/utils/subtitleWindows.ts:34), so the client always gives up first.context.WithoutCancel,internal/playback/subtitle_cache.go:807), so a client abandon does not stop it: the two minutes are spent either way.subtitleWindowWaitTimeout = 5 * time.Second—internal/playback/subtitle_cache.go:272. Bounds a duplicate that joins someone else's flight, not the leader's extract.internal/playback/subtitle_relay_admission.go:66,:448. Paces concurrency; no time bound.SubtitleProbeMax = 15 * time.Second—internal/playback/subtitle_layout.go:38. Bounds the drift probe that may run earlier in the same request, which spends part of the client's deadline before the extract starts.indexLoadTimeout = 30 * time.Second—internal/embeddedsub/cache.go:16. Bounds the kept-subtitle index read, which a virtual relay input never takes (SubtitleCache.sourceis wired for jellycompat only,internal/jellycompat/router.go:204).So a 120 s page on a relay delivering 3–7× realtime needs 17–40 s of source
read, the client's stall deadline expires first, and the server never answers.
Nothing was learned and the next attempt paid the same read again.
Approach
A text or ASS window a client is waiting on now has an elapsed budget of 20 s
(
subtitleTextWindowElapsedBudget,internal/playback/subtitle_window_budget.go:39),applied by
textWindowExtractBudget(:70) throughwindowExtractContext(
internal/playback/subtitle_cache.go:793). On a breach the extract iscancelled, the existing per-identity extraction cooldown is armed, and
windowBudgetBreachError(internal/playback/subtitle_cache.go:816) eitheranswers
503withRetry-Afteror, if part of the slice already reached theclient, keeps the error so the handler aborts the response — never a completed
partial
200.Retry-Afteris the remaining cooldown, floored at 1 s, and thecoverage headers are cleared by the existing retryable answer.
The bound covers only the request-path window: not a whole-track extract, not a
window read from a committed text artifact or from index packets (a local read),
not a bitmap page, and not the detached prefetch, which keeps the window timeout
because it is what fills the window cache. Opening windows are attempted like any
other rather than refused up front, so a fast relay still serves its first page.
On a breach the handler starts the detached warm the cache cannot start for a
virtual input (
internal/api/handlers/stream.go:4090), so the retry the 503 asksfor reads a committed slice or a warm prefix. A detached prefetch is exempt from
the extraction cooldown (
internal/playback/subtitle_cache.go:672,:702):without that, the breach armed the backoff that the prefetch also honours, the
warm was refused, and a mid-file page could never be filled.
An elapsed budget rather than a byte cap because ffmpeg reads an HTTP input with
recv(2), and/proc/PID/iocounts onlyread(2)/pread(2)inrchar.Measured against a local relay that served 24 MB, ffmpeg's
rcharstayed at the~181 KB it reads at startup, so a read cap cannot see the relay cost that matters
here.
20 ssits under the client's 30 s stall deadline, leaving room for ffmpegstartup, the response write, and a layout probe that ran earlier in the request.
docs/architecture/playback-protocol-v3.md§8 records the bound and its scope.Validation
303f3eccc, is still blocked in the extract at a 40 s test timeout — no answer inside the client's 30 s deadline. The same test on this branch answers at 20.001 s withsubtitle extraction retry after 9.99s.SubtitleExtractionRetryErrorwithWindowBudgetExceededinside the budget, writes no body, arms the cooldown, and a retry inside the cooldown costs no extract. A handler-level test gets503+Retry-Afterin 150 ms with a shortened budget, no slice in the body, and the detached warm started.go build ./...,go veton the two touched packages,gofmtclean.go test ./internal/playbackand theSubtitletests ininternal/api/handlers: all pass except two failures that already fail at303f3ecccin this environment —TestServeExtractDiscardsACappedDemuxCopy(kept output = 100717 bytes, want a cutoff near 4096) andTestOnDemandSubtitleSegmentsDoNotWaitForTheStream, which also fails on the base tree with-count=5.TestHandleStartPlaybackV3DoesNotWaitForSessionSyncis flaky on both trees (session sync did not starton repeated runs).-raceon the new playback and handler tests: pass.make verify-playback-fixtures,make verify-local-paths: pass.make lint-changed BASE_REF=kipsilabs/main: 0 issues. The pre-push hook runs the same script with its defaultorigin/main, which is a different fork's main and makes it analyse the whole tree; the documented base was run instead.Risks
answers
503instead. On a relay slower than ~6× realtime the opening 120 spage cannot finish inside the client's deadline anyway, and was previously
abandoned with no answer at all.
loop backs off rather than re-reading. Served paths are checked before that
backoff, so a cached window, a warm prefix, or a committed artifact still
serves during it.
503+Retry-Afterwhere there used to be a hung request.The web player already treats 503 as retryable (
RETRYABLE_WINDOW_STATUSES).A native client that fetches a whole track once on a virtual source now gets
that retryable answer at 20 s instead of silence; follow-up issues for
silo-appleandsilo-androidare not filed in this draft.sourced paths (
internal/jellycompat/streams.go), not this window path, sothere is no counterpart to change.
Checklist
AI Disclosure
StreamExtractOptsconstruction (handler serve, full-track warm, window prefetch, PGS, jellycompat), response shape, cooldown interaction, and context lifetime. It found one defect: the breach armed the backoff the detached prefetch also honoured, so the warm the handler started for a mid-file session was refused and the page could never be filled. The prefetch is now exempt from that backoff, with a test.Posted by an AI agent (Muse Spark, OpenCode harness).