Repository navigation
feat(playback): one progressive bitmap-subtitle demux per source, served behind a coverage watermark - #444
Closed
randrini wants to merge 8 commits into
Closed
feat(playback): one progressive bitmap-subtitle demux per source, served behind a coverage watermark#444randrini wants to merge 8 commits into
randrini wants to merge 8 commits into
Conversation
added 2 commits
November 13, 2026 17:02
A windowed PGS extract cannot seek a non-seekable relay, so ffmpeg demuxes the container from byte 0 and discards the prefix: a page at offset S costs (S + W) x bitrate, not W x bitrate. A 120 s page of an 18.25 GB / 35 Mbps source is ~525 MB, which is also the two-minute extraction timeout, so such pages were killed at the timeout and answered with a partial .sup that held no complete display set. The extract now watches the process's own read count and kills it once a page has read more than 256 MB, the same order as the largest source a page read is still worth attempting. The page answers the retryable 503 the client already backs off from, with a distinct additive problem code (subtitle_page_too_costly), a counter, and the existing per-identity extraction backoff so a retry does not pay the same read again. Only a bounded bitmap page read from a remote source is capped: a whole-track extract must read everything by design, an artifact-backed window reads a local file, and a local source's prefix read is a disk read that finishes in seconds.
…ce demux A windowed PGS extract cannot seek a non-seekable relay, so every page paid for the whole container prefix again: on an 18.25 GB / 35 Mbps source a 120 s page reads ~525 MB, which is the window extract timeout, so pages returned partial .sup bytes with no complete display set and were never cached. One unbounded .sup demux now runs per source identity, track and format generation, single-flighted through the cache's warm slot and detached relay budget. It writes the track's artifact and indexes the display sets as bytes land, so the cache knows how far the track is covered: - a page inside the coverage is served by copying those display sets out of the artifact, with no source read and no ffmpeg; - a page beyond it answers the retryable shape with X-Subtitle-Coverage describing what is actually covered, never a truncated run; - on end of file the artifact flips into the committed whole-track entry. The watermark only advances at a complete display set (PCS through END) and is derived from bytes already written to the artifact, so a page is never served ahead of what has been parsed. A page refused for its read cost also starts the fill, since the artifact is what will serve it; the API handler owns that fill because its relay registration outlives the request. Text windows, local files, and a remote container whose size is unknown keep their existing paths.
randrini
force-pushed
the
feat/pgs-progressive-sidecar
branch
from
November 13, 2026 16:21
dbbe3da to
c911389
Compare
added 6 commits
November 13, 2026 22:59
…ably
The progressive bitmap fill never ran in production, so every page still cost a
fresh source extract. A page miss is served by the handler's detached warm, and
that warm admitted a bitmap track only while paused; a playing session dropped
it with "bitmap full-track warm stays paused-only". Five page misses, five
skips, no artifact, no watermark, no coverage-gated answer: 100.2 s for the first
page and 70.6/75.7/88.1 s for later distinct pages, with only an exact repeat
fast from last-good.
A bounded bitmap page is the one bitmap request whose fill is the page path, so
it now routes to the waiting-page admission: the fill starts while playing,
single-flighted per (source identity, track, format) through the cache's warm
slot and detached relay budget, and it replaces the bounded window prefetch,
which would read the same source again for pages the fill publishes. Every other
bitmap request keeps the paused-only full-track warm.
The read cap measured the wrong cost. A page at 3600 s returned 200 after 88 s
with its read count under the cap because the relay answers the seek with range
reads, so the page is now priced before the extract: the container prefix it
would discard, `start x rate`, from the row's size and runtime or bitrate, and
in wall-clock terms from the read rate this identity's own extracts measured.
A page beyond 256 MB of prefix, or whose prefix would take more than 45 s at the
measured rate, is refused before any source read with the retryable 503, its
Retry-After and X-Subtitle-Coverage, and that refusal starts the fill. An opening
page has no prefix and extracts exactly as before. The runtime 256 MB cap stays
for pages that are still attempted.
An extraction that ran out the window extract deadline answered a non-retryable
500 while the client had four attempts to spend. A page that times out now
answers the same retryable page-cost shape as the read cap; a windowed text
extract that fails or is canceled no longer commits a 200 carrying a header and
half a cue, because the slice is held until its extract completes and dropped
otherwise. The handler classifies both through one mapping: deadline, upstream
5xx and a lapsed relay token are retryable, everything else stays a loud 500.
A `.sup` muxer that refuses the mapped stream's codec after a release rotation
("sup muxer supports only codec hdmv_pgs_subtitle", exit 234) is a mapping the
drift probe missed, so it now reads as a stream-map error and takes the existing
re-probe and remap path — a remapped track or a retryable 409 — instead of a 500.
Verifying a bitmap request's live layout before every spawn is left out: that
probe costs a provider round trip per page and needs a design decision.
Serving rules for the new page path are in docs/architecture/playback-protocol-v3.md
and docs/architecture/web-pgs-subtitles.md. The web client is unchanged: it
retries with backoff and honours Retry-After.
… pgs sidecar branch No conflicts.
serveProgressiveSUPPage set Content-Type and Cache-Control after serveWindow had copied the page, and the first Write commits the status and headers. Every non-empty page therefore went out with neither, so an intermediary could cache a per-window subtitle slice. Set both headers before the copy on every path through it, including the empty window. The existing page test only read the recorder's live header map, which a late Set still updates; the new test reads the committed response.
pagePrefixEstimate priced a page's discarded container prefix as start x byte rate, which can exceed a known source size: a seek past the end, or a size and bitrate that disagree, priced the page above the whole container and refused a page the source can serve. Clamp the prefix to SourceSizeBytes when it is known.
… parsed A page request at a playhead inside the progressive artifact's watermark was answered with 503 subtitle_source_unavailable. The serve gate required the whole page range to be parsed (end <= coveredUntil), so a request for 10-130 s against an artifact that had parsed 0-120 s was refused although every display set that shows in its first 110 s was complete on disk. Production showed it plainly: the artifact's watermark reached 126.543 s while page requests at 10, 20, 40 and 60 s answered 503 with X-Subtitle-Coverage: 0.000-126.543 and no page was ever served; "subtitle page served from the progressive artifact" never appeared in the log because the copy was never reached. Each of those requests is snapped onto the source-zero page grid, so the page under test ends at 240 s even though the requested range ends at 130 s, and the old gate asked for the whole 0-240 s. Serve the parsed prefix instead: the cut stops at the watermark, the response is a 200, and X-Subtitle-Coverage reports the range it really produced. The cut comes from the display-set index and its first complete display set is checked before a byte is written (supHasDisplaySet), so a partial run or a cut that cannot draw is never served. A page that starts at or past the watermark is still retryable, now as 503 subtitle_page_not_ready with Retry-After and the covered range, because the source was never read and the page is only waiting for the artifact that will answer it.
A subtitle response's X-Subtitle-Coverage is the range it really produced, and a bitmap page answered from a partly parsed track is shorter than the page it asked for. The window fetcher committed the whole requested range whenever the body streamed in, so the unparsed tail of such a page was marked as covered and never requested again: playback would draw the prefix and then nothing until the next page boundary. Commit the advertised end when the response carries one, and on the page grid request the page that contains the committed end rather than the page after it. A partly produced page is therefore asked for again as the artifact catches up, and a page that arrived whole is unaffected because its advertised end is the page end.
This was referenced Nov 14, 2026
Collaborator
Author
|
Superseded by #459, which contains this change plus the rest of the playback reliability bundle. Closing to keep the review surface to one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bitmap (PGS) subtitle pages do not appear for a large source served over a relay.
A 120 s page is requested with its own window, but a relay cannot seek, so ffmpeg
demuxes the container from byte 0 and discards the prefix. The cost of a page at
offset S is
(S + W) × bitrate, notW × bitrate: for an 18.25 GB / 35 Mbpssource that is ~525 MB per page, which is also the window extraction timeout.
Pages therefore end at the timeout holding a partial
.sup— 180 bytes, nothingat all, or a single 2,288-byte slice — and never a complete display set. Since a
cue-less bitmap page is correctly refused by the cache, nothing is kept and the
next request pays the same read. The player sits on "Loading subtitles…" for the
rest of playback. The same track works from a small source (a 190 MB file
extracts 19 KB in 15 s), so this is cost, not a broken track.
Related issue: N/A
Validation tasks: none
This depends on the bitmap cache-admission fix from #442, which is in main.
Approach
Two independent pieces, in commit order.
A read budget for a page. A windowed bitmap extract over a remote source is
capped at 256 MB of reads, taken from the extract process's own I/O counters.
ffmpeg has no option to bound its input reads, so the cap is enforced from
outside the process: at the first sample past it, the extract is killed. The page
answers the retryable 503 with a distinct additive problem code
(
subtitle_page_too_costly), a counter, and the existing per-identity extractionbackoff so a client retry does not pay the same read again. A whole-track extract
must read everything, an artifact-backed window reads a local file, and a local
source's prefix read is a disk read that finishes in seconds — none are capped.
One progressive demux per track. One unbounded
.supextract runs per(source identity, track, format generation), single-flighted through the cache's
existing full-track fill (warm slot plus the detached per-source relay budget).
It writes the track's artifact and indexes display sets as the bytes land, so the
cache knows how far the track is covered. A page is then served in this order:
the last-good window, the committed artifact, then the progressive artifact if
the requested end is inside the coverage; otherwise the retryable 503 advertises
what is actually covered in
X-Subtitle-Coverage. A page the artifact does notcover never falls back to a source read, and the miss starts the fill.
Invariants the implementation holds:
PCS … END), derived frombytes already written to the artifact, so it never overstates;
published as the whole-track entry — the artifact flips to the committed entry
only when its extract reaches end of file;
block the extract's writes or backpressure its stdout;
The API handler owns the fill on a cost refusal, because a virtual source's relay
registration dies with the request, and it starts the fill even while the viewer
is playing: no page extract can produce that page at an affordable cost, and the
bounded page prefetch would re-read the same source for pages the fill is already
publishing.
Deferred
.supshould be an additional output of that invocation instead of a secondread. The progressive path is correct without it; this only removes the
duplicate container read.
on a large relay source. Its pages fail fast under the new budget, but it should
skip windows the artifact already covers.
at least 256 MB. A row whose provider never declared a size keeps the page
extract (now capped) rather than refusals based on an assumption; enabling it
is a follow-up once virtual rows always carry a size.
timestamps, so a window inside a long gap between subtitles answers retryable
until the next display set is parsed instead of proving the gap empty.
Validation
go build ./...,go veton the touched packages: pass.go test -count=1 -raceon the new playback and handler tests: pass.The new tests do not compile against the base commit (no coverage index, no
page-cost error, no source size on the extract options), which is the
fail-before form for a new path; the behaviour they forbid is pinned by a base
test that serves a truncated page run with 200.
go test -count=1 ./internal/playback/: only two real-ffmpeg tests fail(
TestOnDemandSubtitleSegmentsDoNotWaitForTheStream,TestServeExtractDiscardsACappedDemuxCopy), and both fail identically at thebase commit.
go test -count=1 ./internal/api/handlers/: three bitmap page-reuse/prefetchtests fail; the base commit fails the same three in a full run and in
isolation. One playback-start timing test fails intermittently at both.
make verify-playback-fixtures: the extract-argument matrix is unchanged.make lint-changed BASE_REF=kipsilabs/mainin the Go/vips build image: 0issues. It could not run on the host (no libvips), so the push used
--no-verify.make verify-local-pathsandgofmt: clean.from the production incident, not from this branch.
Risks
artifact instead of hanging on a page extract. The first page a viewer needs is
available once the fill has parsed that far; a client that retries keeps its
existing backoff and sees a retryable 503 until then.
subtitle_page_too_costlyis a new problem code on the 503. Status,Retry-After, and the client's retry loop are unchanged.range rather than a produced slice.
the detached relay budget, so it cannot take a slot a client-facing read needs.
Checklist
AI Disclosure
Posted by an AI agent (Muse Spark, OpenCode harness).