Repository navigation
Conversation
A text or ASS window that ran out its 20 s request budget answered 503 and started the ordinary playhead-grid prefetch. That prefetch extracts the same page length from an earlier start, so its range ends before the retry's range and the covering lookup rejects it. The retry then paid the same 20 s read; production recorded three consecutive kills for one item and session, after which the track never rendered. Warm the breached window itself with one extra client grid cell on the end, detached from the request budget and from the cooldown the breach armed. The extra cell covers the shifted window both subtitle clients ask for on retry. A retry whose range falls inside an in-flight warm now joins that flight instead of starting a second read. The range is bounded by the client window plus one cell, not by time, so no whole-track progressive fill is needed: the retry asks for a bounded window and this warm produces exactly that.
Collaborator
Author
|
Superseded by #459, which contains this change plus the rest of the playback reliability bundle. Closing to keep the review surface to one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Related issue: N/A
Validation tasks: none
Stacked on
integration/playback-combined(base207e14d21); keep as a draft until reviewed with the stack.On the deployed combined build, a text/ASS subtitle window on a virtual source ran out its 20 s budget three times for the same item and playback session:
After that the item showed "Subs Discovering…" permanently and no ASS track rendered. Before the budget existed the same shape took 58 s, then 25 s, then was abandoned. The bound now answers honestly, but the retry still paid the same read and never succeeded.
The breach itself works: the 503 is retryable, the per-identity cooldown is armed, and the handler starts the detached warm. The failing step is what that warm commits. On a breach the handler starts
warmVirtualSubtitleAfterWindowMiss(internal/api/handlers/stream.go:4136), which for a mid-file playhead takes the playhead-grid prefetch (internal/api/handlers/stream.go:4683,virtualSubtitleWarmPrefetchWindowsatinternal/api/handlers/stream.go:4756). That prefetch extracts the same page length from an earlier grid start. A retry re-anchors a few seconds forward, so the two ranges are:570-690and600-720590-710lookupCoveringWindowResult(internal/playback/subtitle_cache.go:1161) accepts a slice only whencovers(internal/playback/subtitle_cache.go:402) reaches the requested end. Neither warm range does, so the retry misses, extracts again, and breaches again.The other candidates do not fail:
PrefetchWindowsInBackgroundsetsDetachedRelayDemux(internal/playback/subtitle_prefetch.go:94), and the windowed serve skips the failure cooldown for it (internal/playback/subtitle_cache.go:696). The full-track warm uses a separatewarmGuard, not the extraction failure guard.subtitleWindowBudgetedexcludes detached extracts (internal/playback/subtitle_window_budget.go:63); the prefetch ran to completion in the reproduction.internal/playback/subtitle_cache.go:1335).Approach
A budget breach now starts a recovery warm for the breached window itself, extended by one client grid cell (30 s). The retry's shifted range falls inside it: start 580 plus 150 s covers a retry at 590 plus 120 s. The ordinary playhead-grid prefetch remains for successful window misses.
Window flight markers now record their range, and a retry whose range is covered by an in-flight warm joins that warm instead of starting a second relay read. While the warm runs, the retry waits up to the existing flight wait and answers retryably rather than paying the full read again.
The recovery range is bounded by size (the client window plus one cell), not by time. No whole-track progressive text fill is needed: the retry asks for a bounded window, so a bounded warm produces exactly the artifact the retry reads. A whole-track text warm would read the container from zero and commit a different key.
Invariants are unchanged: a partial body never completes as a success (the response hold and cue gate still decide), a cue-less window is still not cached (
subtitleWindowCacheable), PGS keeps its page/artifact path, and SRT/VTT share the same text window path.Validation
go build ./...passed.go vet ./internal/playback ./internal/api/handlerspassed../internal/api/handlerspassed../internal/playbackwas run in full; its only failures areTestOnDemandSubtitleSegmentsDoNotWaitForTheStreamandTestServeExtractDiscardsACappedDemuxCopy, which fail identically on base207e14d21../internal/api/handlerspackage exceeded the 600 s test timeout in this environment; the subtitle-relevant subset above passed.-racepassed on the new handler and playback tests.make verify-playback-fixturespassed.make lint-changed BASE_REF=maindid not complete: because this branch also changes shared Go inputs, the script falls back to full-tree analysis andgo/packageshit its context deadline. The same changed-line lint against the stacked base (BASE_REF=kipsilabs/integration/playback-combined) reports 0 issues.gofmtandgit diff --checkare clean.Fail-before evidence for the handler test: with the old warm call, the same reproduction answered the retry with 503 and logged a second
-ss 590extract. With the fix, the retry returns 200 and no ffmpeg invocation names-ss 590.Risks
Each breach spends one extra detached warm of the client window plus 30 s. It is single-flighted and shares the existing warm admission budget. A retry that moves more than one grid cell past the breached start is not covered by that warm and extracts again; the client's retry cadence and the server cooldown stay within the covered cell in the measured case.
Checklist
AI Disclosure
Posted by an AI agent (Muse Spark, OpenCode harness).