Skip to content

fix(playback): bound a text subtitle window extract at its budget - #454

Closed
randrini wants to merge 1 commit into
mainfrom
fix/ass-extraction-bound
Closed

randrini wants to merge 1 commit into
mainfrom
fix/ass-extraction-bound

Conversation

@randrini

Copy link
Copy Markdown
Collaborator

Problem

Related issue: N/A
Validation tasks: none

Selecting an ASS track on a virtual source leaves the web player on "Loading
subtitles…" for 90+ seconds. In the deployed build on 2026-11-15, two
getPlaybackSubtitle requests returned status 0 after 58,147 ms and
25,856 ms — the client had already abandoned them — and two more answered
503 subtitle_source_unavailable. Across the 24 subtitle requests in that
window: p50 1,011 ms, p90 49,327 ms, max 58,147 ms. The font bundle is not
the blocker: /fonts returns a 2.5 MB bundle stored as 9.4 MB when it is
present.

The windowed text extract has no bound below the client's own deadline. The
bounds in that path, as they exist on main plus #444 and #451:

  • subtitleWindowExtractTimeout = 2 * time.Minute — internal/playback/subtitle_cache.go:282. It is the only ceiling on the extract, and it is longer than the client's 30 s subtitle fetch stall deadline (web/src/player/utils/subtitleWindows.ts:34), so the client always gives up first.
  • The extract runs detached from the request (context.WithoutCancel, internal/playback/subtitle_cache.go:807), so a client abandon does not stop it: the two minutes are spent either way.
  • subtitleWindowWaitTimeout = 5 * time.Second — internal/playback/subtitle_cache.go:272. Bounds a duplicate that joins someone else's flight, not the leader's extract.
  • Relay admission, 2 request-path slots per source — internal/playback/subtitle_relay_admission.go:66, :448. Paces concurrency; no time bound.
  • SubtitleProbeMax = 15 * time.Second — internal/playback/subtitle_layout.go:38. Bounds the drift probe that may run earlier in the same request, which spends part of the client's deadline before the extract starts.
  • indexLoadTimeout = 30 * time.Second — internal/embeddedsub/cache.go:16. Bounds the kept-subtitle index read, which a virtual relay input never takes (SubtitleCache.source is wired for jellycompat only, internal/jellycompat/router.go:204).
  • feat(playback): one progressive bitmap-subtitle demux per source, served behind a coverage watermark #444's read cap is bitmap-only, and fix(playback): deliver ASS and PGS subtitles on virtual sources #451's 120 s text page and header-only gate bound what the extract asks for, not how long it runs.

So a 120 s page on a relay delivering 3–7× realtime needs 17–40 s of source
read, the client's stall deadline expires first, and the server never answers.
Nothing was learned and the next attempt paid the same read again.

Approach

A text or ASS window a client is waiting on now has an elapsed budget of 20 s
(subtitleTextWindowElapsedBudget, internal/playback/subtitle_window_budget.go:39),
applied by textWindowExtractBudget (:70) through windowExtractContext
(internal/playback/subtitle_cache.go:793). On a breach the extract is
cancelled, the existing per-identity extraction cooldown is armed, and
windowBudgetBreachError (internal/playback/subtitle_cache.go:816) either
answers 503 with Retry-After or, if part of the slice already reached the
client, keeps the error so the handler aborts the response — never a completed
partial 200. Retry-After is the remaining cooldown, floored at 1 s, and the
coverage headers are cleared by the existing retryable answer.

The bound covers only the request-path window: not a whole-track extract, not a
window read from a committed text artifact or from index packets (a local read),
not a bitmap page, and not the detached prefetch, which keeps the window timeout
because it is what fills the window cache. Opening windows are attempted like any
other rather than refused up front, so a fast relay still serves its first page.

On a breach the handler starts the detached warm the cache cannot start for a
virtual input (internal/api/handlers/stream.go:4090), so the retry the 503 asks
for reads a committed slice or a warm prefix. A detached prefetch is exempt from
the extraction cooldown (internal/playback/subtitle_cache.go:672, :702):
without that, the breach armed the backoff that the prefetch also honours, the
warm was refused, and a mid-file page could never be filled.

An elapsed budget rather than a byte cap because ffmpeg reads an HTTP input with
recv(2), and /proc/PID/io counts only read(2)/pread(2) in rchar.
Measured against a local relay that served 24 MB, ffmpeg's rchar stayed at the
~181 KB it reads at startup, so a read cap cannot see the relay cost that matters
here. 20 s sits under the client's 30 s stall deadline, leaving room for ffmpeg
startup, the response write, and a layout probe that ran earlier in the request.

docs/architecture/playback-protocol-v3.md §8 records the bound and its scope.

Validation

  • Fail-before: a probe test whose extract never reaches the requested range, run at 303f3eccc, is still blocked in the extract at a 40 s test timeout — no answer inside the client's 30 s deadline. The same test on this branch answers at 20.001 s with subtitle extraction retry after 9.99s.
  • Pass-after: a slow ASS window answers SubtitleExtractionRetryError with WindowBudgetExceeded inside the budget, writes no body, arms the cooldown, and a retry inside the cooldown costs no extract. A handler-level test gets 503 + Retry-After in 150 ms with a shortened budget, no slice in the body, and the detached warm started.
  • SRT and WebVTT windows are covered by the same bound and unaffected in practice: a fast window is served byte-identical, and the scope table pins every excluded case.
  • go build ./..., go vet on the two touched packages, gofmt clean.
  • go test ./internal/playback and the Subtitle tests in internal/api/handlers: all pass except two failures that already fail at 303f3eccc in this environment — TestServeExtractDiscardsACappedDemuxCopy (kept output = 100717 bytes, want a cutoff near 4096) and TestOnDemandSubtitleSegmentsDoNotWaitForTheStream, which also fails on the base tree with -count=5. TestHandleStartPlaybackV3DoesNotWaitForSessionSync is flaky on both trees (session sync did not start on repeated runs).
  • -race on the new playback and handler tests: pass.
  • make verify-playback-fixtures, make verify-local-paths: pass.
  • make lint-changed BASE_REF=kipsilabs/main: 0 issues. The pre-push hook runs the same script with its default origin/main, which is a different fork's main and makes it analyse the whole tree; the documented base was run instead.

Risks

  • A page that would have finished between 20 s and the client's 30 s deadline now
    answers 503 instead. On a relay slower than ~6× realtime the opening 120 s
    page cannot finish inside the client's deadline anyway, and was previously
    abandoned with no answer at all.
  • The breach arms the existing backoff (10 s doubling to 60 s), so a client retry
    loop backs off rather than re-reading. Served paths are checked before that
    backoff, so a cached window, a warm prefix, or a committed artifact still
    serves during it.
  • Client-visible: a 503 + Retry-After where there used to be a hung request.
    The web player already treats 503 as retryable (RETRYABLE_WINDOW_STATUSES).
    A native client that fetches a whole track once on a virtual source now gets
    that retryable answer at 20 s instead of silence; follow-up issues for
    silo-apple and silo-android are not filed in this draft.
  • jellycompat parity: the compat surface extracts text through its own batch and
    sourced paths (internal/jellycompat/streams.go), not this window path, so
    there is no counterpart to change.
  • No migration, schema, or settings change.

Checklist

  • I read and can explain the complete diff.
  • This pull request addresses one concern.
  • Each test this pull request adds meets "Decide what to test" in CONTRIBUTING.md.
  • The Evidence section shows every change a user can see, or says there is none.

AI Disclosure

  • Harness: OpenCode
  • Tool(s): OpenCode agent (shell, file edit, git, gh CLI)
  • Model(s): Muse Spark
  • Involvement: Fully AI-generated, human review pending (draft)
  • Adversarial review: an agent review pass over the commit checked option scoping across every StreamExtractOpts construction (handler serve, full-track warm, window prefetch, PGS, jellycompat), response shape, cooldown interaction, and context lifetime. It found one defect: the breach armed the backoff the detached prefetch also honoured, so the warm the handler started for a mid-file session was refused and the page could never be filled. The prefetch is now exempt from that backoff, with a test.

Posted by an AI agent (Muse Spark, OpenCode harness).

A request-path text or ASS window on a virtual relay source had no bound of
its own: the only ceiling was subtitleWindowExtractTimeout (2 min), applied
after the leader detached from its request, and the client's 30 s subtitle
fetch stall deadline expired first. The deployed build measured 25.9 s and
58.1 s of silence on that path with status 0 (client abandoned), against 24
subtitle requests in the window: p50 1011 ms, p90 49327 ms, max 58147 ms.

Bound the extract with an elapsed budget of 20 s: on a breach the extraction
is cancelled, the existing per-identity cooldown is armed, and the answer is
503 with Retry-After when nothing was written, or an aborted response when
part of the slice had already streamed (never a completed partial 200). The
handler starts the detached warm on that answer, and a detached prefetch is
no longer paced by that cooldown, so the retry reads a committed slice or a
window prefix instead of repeating the read.

The bound covers only what a client waits on: not a whole-track extract, not
a window read from a committed artifact, not a bitmap page, and not the
detached prefetch, which may run to the window timeout because it is what
eventually fills the window cache. Opening windows are attempted like any
other rather than refused up front.

An elapsed bound rather than a byte cap because ffmpeg reads an HTTP input
with recv(2), which /proc/PID/io does not count in rchar: measured against a
relay that served 24 MB, ffmpeg's rchar stayed at its ~181 KB of startup
reads.
@randrini

Copy link
Copy Markdown
Collaborator Author

Superseded by #459, which contains this change plus the rest of the playback reliability bundle. Closing to keep the review surface to one PR.

@randrini randrini closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant