Repository navigation
Fix ffmpeg respawn churn and ~1s video latency on live H.264 streams - #25
Merged
Merged
Conversation
…est gaps Two live-stream (VideoStream) bugs in the video player, each costing a full ffmpeg process kill+respawn on the CLI decoder path: - enqueue_samples treated the playhead crossing into the next GOP as "skipping frames" and reset the decoder. At the live edge the next GOP can never be pre-enqueued (its samples don't exist yet), so this fired on every keyframe (~every 2s), spawning a fresh ffmpeg per GOP. Now the reset only happens when there is an actual gap between the enqueued samples and the requested keyframe's GOP; a contiguous rollover decodes straight through the new IDR. - end-of-video was signaled whenever a stream saw no new samples for 250ms (time_until_video_assumed_ended), a threshold the SDK's 200ms batch tick sits right under. Each misfire forced a decoder reset when the next batch arrived, which under load escalated into a self-reinforcing respawn storm (measured 45 spawns/s). Streams never signal end-of-video anymore; only static assets do. Tradeoff: during a stall the frozen frame is up to ~1s staler because trailing frames stay unflushed. Also adds DIAG debug! lines to every decoder-reset call site (the previously silent branches made this bug near-impossible to attribute) and promotes the out-of-order re-merge trace_once! to a countable debug!. Verified against a live Go2 zenoh H.264 feed (720p15, GOP 30): steady state went from one ffmpeg spawn per 2.1s (storming to ~45/s under load) to a single long-lived process with zero resets after startup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ffmpeg's frame-multithreading adds roughly one frame of output delay per thread and defaults to the core count: measured 17 frames of steady-state holdback (~1.1s at 15fps) with default threading vs 3 frames with -threads 1 on a 720p15 H.264 stream. That holdback was the visible native-viewer video delay vs the WebCodecs-based web viewer. Stream-delivered video now decodes with -threads 1 (one decode thread is ample for typical live-stream resolutions); static assets keep default threading for throughput. Also tried -fflags nobuffer / -flags low_delay: both measured *worse* (32 frames held), so they stay out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
paul-nechifor
approved these changes
Jul 22, 2026
leshy
added a commit
to dimensionalOS/dimos
that referenced
this pull request
Jul 22, 2026
Fixes ffmpeg respawn churn (per-GOP decoder resets + end-of-video misfires under load) and ~1s native-viewer video latency on live H.264 streams. dimensionalOS/dimos-viewer#25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running a Go2 zenoh blueprint pinned the machine (load ~12/20 cores) with the native viewer spawning a fresh ffmpeg per decoded frame, and video trailed ~1.2s behind live. Three causes, all in the video player:
1. Decoder reset on every live GOP rollover.
enqueue_samplestreats the playhead crossing into the next GOP as "skipping frames" and resets the decoder. At the live edge the next GOP can never be pre-enqueued (its samples don't exist yet), so this fired every keyframe — measured ffmpeg spawn intervals of 2.07–2.13s, exactly the stream's GOP period. Now a contiguous rollover decodes straight through the new IDR; reset only happens on an actual gap.2. End-of-video misfire on ingest gaps. A stream counts as live only if samples arrived within 250ms (
time_until_video_assumed_ended); the SDK batcher ticks at 200ms. Any jitter >50ms declared the stream ended, and the next batch forced a decoder reset ("Reset because new frames appeared…"). Under load this compounds into a respawn storm (measured ~45 spawns/s, self-reinforcing since each respawn adds load). Streams no longer signal end-of-video; only static assets do. Tradeoff: during a stall the frozen frame stays up to ~1s staler, since trailing frames remain unflushed in the decoder.3. ffmpeg frame-multithreading latency. ffmpeg holds back roughly one frame per decode thread and defaults to core count: measured 17 frames of steady-state holdback (~1.1s at 15fps) vs 3 frames with
-threads 1(720p15 H.264, feeding access units one at a time and counting outputs). This is also the story behind themin_num_samples_to_enqueue_ahead = 18constant. Streams now decode with-threads 1; static assets keep default threading for throughput.-fflags nobuffer/-flags low_delaymeasured worse (32 frames) and stay out.Also adds
debug!lines to every decoder-reset call site — they were all silent, which is why this took days to attribute (note: trace-level logging emits nothing in this viewer even withRUST_LOG=trace, separate mystery).Verified live against the Go2 720p15 feed: one long-lived ffmpeg for the whole session (vs one per 2.1s / storms), zero resets after startup, latency now comparable to the web viewer. Full investigation notes in dimos
video_issue.md.🤖 Generated with Claude Code