Repository navigation
gpu: decode into the pool and encode from it on Linux, on the sidecar's Vulkan device - #15
Merged
Merged
Conversation
…ncode through it Work in progress, not yet built on Linux. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s written Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… its writer waits for A decoder ahead of a slower encoder or filter queued every picture it made on the reader's input, which on a 3 GB card ran the device out of memory in under two seconds at 1080p. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ffmpeg refuses hwupload in a Vulkan filtergraph with no device named, as it does on VideoToolbox; checked on a GTX 1050 with ffmpeg 8.0.1 and 9.0.2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the Vulkan decoder on it On the GTX 1050, decode into the pool, gpu_box and NVENC from the pool held 738 MiB at 1080p60: wgpu's default hint takes 128 to 256 MiB blocks, and libavcodec's Vulkan decoder on twelve frame threads kept sixteen to eighteen pictures. With blocks of 8 to 64 MiB and one thread it holds 423 to 487 MiB, runs at 268 to 274 frames a second against 247 to 256, and takes no more CPU at real time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ys so An encoder reading NV12 pictures of a pool on the device refused pix_fmt=yuv420p, which 0.31.0 accepted for h264_nvenc on Linux from host memory. yuv420p is the same 8-bit 4:2:0 sampling and the same coded stream, so it is coded as nv12, with the encode line saying so; any other format is refused naming yuv420p and nv12. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…on a device poll wgpu-core's poll holds the device's fence lock while it waits, and every submit waits for that lock: sampled on the box, the feed thread's copy of each decoded picture sat in queue_submit behind the signaller's wait for the picture before. The signaller now waits on wgpu's timeline semaphore directly, whose value is the submission index, and polls only where wgpu has no timeline semaphore. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s laid later once the clock is kept up with A 1080p programme on stdin starts about half a second behind real time: the sidecar is not reading while it starts, and then reads what queued faster than real time until it has caught up. A port feed arrives in real time, so a feed fixed during that catch-up is early by the backlog. Once the clock has caught up, each tick asks for sound that has not arrived yet, so the sound is silent, and the pictures show late. With the feed decoded on the device the first pictures come at once, so the feed is fixed early in the catch-up and only the first beep was heard. The software decoder's latency hid most of it, and main showed it as a -33 to -66 ms skew. The assembler now notes when each tick of an input clock is cut. The clock is kept up with when, over its last quarter second, it moved as far as the wall did between every one of those ticks and the newest, give or take 50 ms. The scheduler cannot see this backlog: it sits in the programme's pipe, and the reader is never waiting for room. A `first-frame` port feed that comes up while the clock is not kept up with is provisional. On the first tick that is, it moves later once, by whole ticks of the grid, until its newest frame is `lead` ahead of that tick, but by no more than the clock gained on the wall since the feed was fixed. The picture and the sound move together. A feed fixed while the clock is kept up with, a shared-clock feed and a feed bound in the plan are fixed as before. Measured on the GTX 1050 (1080p60 programme, lead 0.1, the flash and beep clip fed over a port, 36 flashes): - device decode: 1 of 36 beeps before; 36 of 36 after, skew medians +0.4, -0.3 and +0.0 ms; - software decode: 36 before (+0.4, +0.0 ms) and after (+0.4 ms); - main with this change: 36, -0.3, +0.4 and +0.4 ms, was -66 and -34 ms. A hold unit test scripts a node that starts behind and catches up. A live test sends the programme at half speed for its first 1.2 s and then all at once, with the feeder joining half a second behind. It checks the feed is laid later once, its sound is the feeder's own samples throughout, and each picture is within a frame of its sound. With the move turned off, that test finds silence. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d the sidecar's threads have names With FFRWD_TRACE_FEED naming a file, the sidecar appends one line to it per event, flushed as it is written. So a run under `ffrwd run -q`, or one that is killed, still leaves the trace. Each line carries the wall time, the milliseconds since start, the process id and the thread. The events: - each coded or raw picture and each run of sound a port feed brings; - each picture its decoder hands over, on the device or in host memory; - the feed lines the listener reports; - each offset a hold fixes, and each later move; - what every tick of a port-fed hold hands its node, a picture's and a sound's source time side by side; - each frame an output writes. Unset, each call is one check of a static. NODE-CLI.md says it in one sentence. The sidecar's threads are named: - `worker N` for the lane workers; - `feed PORT` for a port's listener, which also decodes its pictures; - `read N` for an input's reader, which also cuts the ticks of the lanes it feeds; - `write N` for an output; - `sink`, `demux`, `rows in` and `rows out`; - `vk signal` for the Vulkan semaphore signaller. On a 1080p feed into compose_av, top -H on the box shows the feed thread at 13-14%, each worker at 9-12%, and the readers at 2-4%. The row named ffrwd-wasm at about 70% is the process's main thread, and it was idle in every gdb sample. top -H reports the whole process's total on the leader's row. The CLI's FFRWD_DUMP_STDERR files are flushed per chunk. They were written through a buffer and only flushed on close, so a member killed with its run left an empty `.part`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ets it reuses, and the trace names the textures handed The Vulkan taker made a fresh pair of textures (luma and chroma) for every picture and dropped them with the frame. Now it keeps up to 32 pairs per decoder and copies each picture into the next free pair after the last one used, so the pairs come round in turn. A pair is free once no frame holds it. A later copy into it is queued on wgpu's queue after every read queued so far, and wgpu orders it after them. Free pairs of another size are let go, and past 32 a picture gets a pair of its own, dropped with it, as before. The textures keep their wgpu ids for the run, as the D3D12 route's wrapped pool and the uploads in host memory do. The feed trace moves into the runtime crate so the host functions can write to it. Three lines name textures: - `taker:` gives a picture's pts, its pool slot and the ids of its two targets; - the hold's tick line gives the texture ids of the picture it hands; - `planes:` gives the ids `frames.planes` hands the module for that tick. A mismatch between what the hold hands and what the module is handed shows on one tick's pair of lines. FFRWD_FEED_DECODE=software decodes a coded feed in software where it would be decoded on the device. It is a switch for telling the device's decode apart from the rest of a run, documented in NODE-CLI.md beside the trace. NODE-SHAPE.md, "Frames on the GPU", writes down the contract on texture identity. The textures `frames.planes` hands over are good for that call only. Which frame a texture holds is said by the frame (its stream, pts and feed start), not by the handle or the id, since the host reuses both. blitz's compose keys its pictures that way, by (stream, pts, feed-start.at, first-pts), and it asks for the planes again each tick. A new gpu_frames case holds a device-decoded feed of 250 distinct greys sent at twice real time, so a hundred and more pictures are held while others are shown and dropped and textures are made and reused throughout. It checks that every tick shows the grey of the picture that many ticks into the feed. It passes on the GTX 1050 with the pool and without it, and on Windows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ED=host, and rows in the trace A region whose node holds an input that a port serves is now told where libav is (`-av-runtime`) whatever the output's encoder. A feeder may send its picture coded: a run-time lateral copies an input's own H.264, and the sidecar decodes it with libavcodec. With `video_codec 'libx264'` the compiled query had no device codec, so the sidecar looked for libavutil.so.61 beside itself and refused the feed's decode. A test in test_node_world fails without the fix. FFRWD_ENCODE_FEED=host keeps NVENC on Linux but feeds it each frame read back to host memory, as 0.31.0 did, so the copy from the device into CUDA can be told apart from the encoder. It is documented beside FFRWD_FEED_DECODE. The feed trace adds rows: - `rows out:` for every row a module emits, with its lane, tick and stream, or the note that the lane has no rows stream; - `rows in:` for every message handed to a node's input; - `rows written:` for every row a rows output writes, and its queue's bytes when it waits; - `room:` the first time a producer waits for room, with each consumer's queue depth. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cond A run-time lateral's instance writes its input to a feeder's port, and its shape is declared, so ffmpeg's probe does nothing for the compile. It only holds the feed back: with ffmpeg's defaults it reads about 5 s of a live MPEG-TS input and then sends them at once (foundmeat's feeder, speed=10.1x over its first half second). Such an input now gets `-analyzeduration 500000 -probesize 500000` unless it sets either. Elsewhere a declared shape still hands ffmpeg nothing. A test in test_inputs covers the default, an input that sets one of the two, and a COPY that is not a feeder's. The default is noted in docs/dialect.md's paragraph on run-time laterals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the luma of small pictures The feed trace's tick line for a port-fed hold now gives each sound run's peak (s16 or f32, the member's own format) and the mean luma of each held picture of at most 64x64 rgba. A larger picture is left out, so the trace never reads one back. A new `emit:` line gives every frame a node writes: a run of samples with its peak, or a small picture with its luma. A 1 kHz beep or a white flash then shows on the tick it was handed and on the tick the node wrote it. trace_beats.py, beside trace_pairs.py on the box, matches them per feed. GpuFrame::size is public, for the luma. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the NUT output A frame a node emits on the device that is larger than 64x64 is made 32x32 there and read back, so the trace's `emit:` line gives its mean luma at the cost of a 4 KiB read-back. The line also gives its time base, so trace_beats.py can match a flash in the node's output against the flash the hold handed it, as it does for beeps. FFRWD_TEE_OUT=<file> appends a copy of every byte the sidecar writes to its NUT output to that file as it is written, the second NUT output to <file>.1 and so on. That is what the output's reader (the cli's output ffmpeg) was handed, so it can be measured apart from that reader. Documented in NODE-CLI.md beside the other switches. The device-only trace pieces (texture ids, the 32x32 read-back) are behind the gpu feature, so a build with --no-default-features compiles again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This reverts 1922469. With a half-second probe, a lateral that joins an MPEG-TS feed mid-stream sees no SPS before the probe ends when the GOP is longer (2 s in foundmeat's feed). ffmpeg then refuses the NUT header ("dimensions not set", "Could not find codec parameters for stream 0 ... unspecified size") and the instance exits, so every relaunch onto a running feed fails. ffmpeg's default probe reads past a keyframe at the cost of the backlog. A query that wants the short probe sets it on its `input()`, as foundmeat's does, knowing its feeds join at their start. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ead after it A port feed's connection is read only while its input has room. Behind an output slower than real time (libx264 at 1080p60), a feeder that had finished and closed could still have its last bytes unread in the socket, and the host had not seen it go. The feeder rejoining after its drop was then refused as a second one, six times on foundmeat's run, and the feed did not come back. The host now notes when a connection has waited for room since it last caught up with it. A feeder that connects then waits: it is read from its first byte once the one before ends. Other cases are as they were: a second feeder while the one before keeps up is refused, and a connection that has sent nothing is replaced. A live test sends 12 s of a feeder at once into a full input, connects a second at once, and checks both feeds show, one after the other, with nothing refused. With the wait turned off the test fails. The trace's `emit:` lines also give every message a node hands on a data output, the node-to-listener side of the rows trace. NODE-CLI.md says what a feeder that connects during a wait gets. The plan's Linux section has a dated paragraph on the lip-sync reports from foundmeat's box. They were a measurement artifact, and every build was within 5 ms measured with -copyts. The paragraph lists what that work did find. It also adds the next item: a lane's task queue seen past the scheduler's cap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
imbcmdth
force-pushed
the
linux-gpu-codec
branch
from
October 19, 2026 05:19
7628c15 to
912bf25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Linux half of the GPU-resident frames work, built and measured on foundmeat's GTX 1050 (Pascal, 3 GB, driver 580) on 2026-10-18. Before this a held 1080p60 H.264 feed decoded on the CPU there, NVDEC idle, the compose sidecar at 2.6 to 3.5 cores.
runtime/src/gpu/vulkan.rs(new): the sidecar opens its Vulkan device itself on the adapter wgpu chose, with the video queue, decode and encode extensions and one queue each of the transfer, compute, video decode and video encode families for libav, and hands it to wgpu-hal (device_from_raw) and wgpu-core. libav'sh264_vulkan,hevc_vulkanandav1_vulkandecoders run on it; each picture is copied once on the device into two pool textures (wgpu-core takes a raw image as uninitialised, so wrapping it is not safe), with a host wait on the picture's semaphore before and a host signal after.av::Shared::VulkanfillsAVVulkanDeviceContext; the structs are inav/sys.rswith offset asserts checked against gcc over the n9.0.1 headers.h264_nvenc,hevc_nvencorav1_nvenc. Nothing is read back.lanes.rs, every platform: a self-clocked reader's queued arrivals now count against the room its writer waits for. A decoder ahead of a slower encoder queued every picture it made, which filled the 3 GB card in under two seconds; outputs are unchanged, only memory is bounded.cli/ffrwd/hardware.py: the Vulkan card pass names the device itshwuploadgoes to.gpu_framesruns on the card (the device decode is byte-identical to the software decode); the D3D12-only cases skip. Docs in AV-EDGE.md and NODE-CLI.md; a "Linux" section inplans/gpu-resident-frames.mdwith the measurements and what is left (thefilternode's Linux routes, inference from the pool on CUDA, HEVC and AV1 untried, the L4).Measured, 1080p60 H.264 into a GPU module into h264_nvenc, 3600 frames: GPU route 268 to 274 fps unpaced at 1.2 cores, 0.53 of a core at real time with dec 9% and enc 20%, 0 bytes uploaded or downloaded; the CPU path 0.96 of a core at real time. Recipe 157 through
ffrwd run: 6.9 s and 600 uploads plus 600 downloads on main, 3.1 s and none here.🤖 Generated with Claude Code