Skip to content

Video plays smoothly at 4K and at 60 fps - #137

Closed
shinyobjectz wants to merge 3 commits into
mainfrom
video-playback
Closed

shinyobjectz wants to merge 3 commits into
mainfrom
video-playback

Conversation

@shinyobjectz

@shinyobjectz shinyobjectz commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Video in a page (the app's players) lagged and dropped frames, most at 4K and at 60 fps. This makes four changes, each measured.

What changes

  • Frames are decoded ahead and shown when due. The decoder's thread keeps a quarter second of frames ahead of the clock (never more than 128 MB waiting, 200 nearing a loop's end, never fewer than two frames) and sends each frame with the moment it is due (Msg::Due). A decode that stalls for 40 to 85 ms (a reader starting again, a busy machine) no longer reaches the picture. At each of its own frames the page shows the frame due then. How long a frame takes to decode is no longer part of when it is shown, and a loop's first frame is ready before its last has gone.
  • BGRA to RGBA a word at a time (io/apple/src/convert.rs). A 4K frame takes 1.9 ms, was about 5.5: it was most of what decoding one cost.
  • A video drawn small is drawn from a halving of its frame. A video shown at under half its size (any 4K video in a player) was sampled across four or more of its pixels at a time, so it shimmered, and the whole 33 MB frame went to the GPU every frame. The halving it is drawn from (2x2 means, as an img already gets) is now made on the decoder's thread and is all that is sent: the frame comes as its size and that halving (Cmd::Halvings, blitz-paint preload_mips and video_level). A 4K frame waiting is then 8 or 2 MB, which is what makes the quarter second affordable. RAEA_MEDIA_HALVINGS=0 draws frames as they come, to compare.
  • A frame the size of the last one repaints its element and lays nothing out again.

Also: RAEA_PAGE_FPS sets a headless window's frame rate (60 as before), to measure a 60 fps video on a page that draws 120 times a second. RAEA_MEDIA_STATS=1 now also says how late frames were shown and how far apart the page's syncs were.

Measured

RAEA_MEDIA_STATS=1 in the built app's headless window, two videos side by side, 10 s each, the same reel at each size and rate.

clip before after
4K 30 fps 2 frames passed over, worst wait between frames 76 ms, decode 6.6 ms a frame 0 passed over, worst wait 39 ms (33 is even)
4K 60 fps, page at 60 12 passed over, 6 skipped by the decoder, worst wait 77 ms 0 skipped, worst wait 39 ms
4K 60 fps, page at 120 not measured 0 of 1214 passed over, each shown 2.4 ms after its moment on average, the page's longest frame 9 ms
1080p 60 fps, page at 120 not measured 0 of 1212 passed over, worst wait 21 ms
1600x900 30 fps worst wait 58 ms worst wait 44 to 57 ms, 0 passed over

With the page's loop uncapped, the halvings take the page's longest frame on two 4K60 videos from 11 to 12 ms down to 5 to 8 ms.

Not in this change

  • On a page drawing 60 times a second, a 60 fps video still loses about 1 frame in 14. A headless window's 60 is really about 53 (its loop sleeps a little long); that is the page's rate, not the video's. On screen the window draws at the display's rate; that is not measured here.
  • A frame is still CPU RGBA handed to Vello. The decoder's IOSurface path would skip the copy and the conversion.
  • The app's player re-renders its whole detail view at each timeupdate, which makes one page frame in fifteen about 9 ms longer. That is the app's to fix.
  • Before the quarter-second buffer, one run of 4K30 with the page at 120 skipped 28 frames while other heavy jobs ran on the machine; reruns were clean, and with the buffer every run since has skipped none.

Checked

  • cargo test -p raea-runtime-js media_play (each playing frame is due on the clock, through two laps of a loop; a loop's first frame comes before it is due; a paused picture asked for halvings comes again as its size and the one it is drawn from; a halving is the mean of four).
  • cargo test -p raea-runtime-dom-media (the frame shown is the latest one due).
  • cargo test -p raea-io-apple --release convert (the conversion, and its time).
  • tools/agents/check --land --lane video-playback: L1 ok.
  • Screenshots of the 4K clip with and without halvings: the same picture in the same place, less noise with.

…own when due, a faster BGRA conversion, halvings from the decoder's thread, no relayout for a same-size frame
@shinyobjectz

Copy link
Copy Markdown
Owner Author

Landed on main through the lane: d7a755c, fcd6e12, f9b6f22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant