Repository navigation
Draft: lossless 5K60 path (4:4:4 10-bit, tile-DPCM on the GPU) — seeking guidance on integration - #201
Draft
aalpgiray wants to merge 6 commits into
Draft
Draft: lossless 5K60 path (4:4:4 10-bit, tile-DPCM on the GPU) — seeking guidance on integration#201aalpgiray wants to merge 6 commits into
aalpgiray wants to merge 6 commits into
Conversation
…ence
Nothing is wired up yet. This commit is the format and the thing every later
commit is proven against, so it can be reviewed and verified on its own:
cc -O2 -o /tmp/test_dpcm \
TargetBridge-Receiver/TBReceiverC/tests/test_dpcm.c \
TargetBridge-Shared/codec/tb_dpcm.c -I TargetBridge-Shared/codec
/tmp/test_dpcm # 290 checks, no dependencies
WHY A CODEC AT ALL
A 5K frame at 4:4:4 10-bit is 59 MB, so 60 fps is 3.5 GB/s -- inside a 40 Gb/s
Thunderbolt link, so bandwidth is not the constraint. The receiver is. Measured
on a 2020 Retina 5K iMac: 23.4 ms of a 36 ms frame goes to pulling bytes off the
wire (single core, inside the TCP stack) and another 12.9 ms to pushing them
across PCIe. Either alone exceeds a 16.7 ms budget. Both costs are proportional
to frame size, so the cheapest way to raise the frame rate is to send fewer
bytes -- and losslessness rules out throwing any away.
WHAT IT DOES
DPCM: code each sample as its difference from a prediction rather than its
absolute value. Residuals are taken modulo the sample range and re-centred so
reconstruction wraps and stays exact; zigzagged, the widest possible residual
needs exactly the sample depth, so a tile can never expand beyond its header and
no escape hatch is needed for pathological content.
Measured on real content: 2.96x on a 36-megapixel photo (near worst case), 4.5x
on text-heavy UI, 13.8x on ordinary window chrome. LZFSE gets 3.48x on that same
photo, so this captures 85% of what a general-purpose compressor manages while
staying cheap enough to run per-tile on a GPU.
WHY TILES
Textbook DPCM is serial -- pixel N depends on N-1 -- which is fatal on a GPU,
and the GPU is the only spare compute this receiver has. So the frame is cut
into 8x8 tiles that are fully independent: each carries its own seed pixel and
bit widths and needs nothing from any neighbour. 230,400 threadgroups at 5K.
8x8 was measured, not chosen. 4x4 wins slightly on dense text (4.71x vs 4.46x)
and loses badly on flat content (9.7x vs 13.8x) by paying four times the
per-tile overhead where there is nothing to code. 16x16 loses everywhere: one
high-contrast edge sets the bit width for 256 pixels.
The predictor is the left neighbour, first column predicting from above.
JPEG-LS's median-edge predictor ties it to within 1% on every frame tested,
including the photo, and costs three extra loads and a clamp -- inside an 8x8
tile there is not enough vertical run for it to earn them.
The header carries the full format description, the rationale for each constant,
and the alternatives that were measured and rejected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Still not wired up. Depends only on Metal and tb_dpcm.h, and compiles standalone:
cc -c -fobjc-arc TargetBridge-Shared/codec/tb_dpcm_gpu.m \
-I TargetBridge-Shared/codec
The encoder has to keep up with 60 fps at 5K on the sender while the CPU is busy
capturing, so it runs as a Metal compute pass: one threadgroup per 8x8 tile,
230,400 of them per frame. That shape is the entire reason the format uses
independent tiles.
Groups of tiles are byte-aligned on the wire. That costs at most 7 bits per
group -- roughly 3 KB a frame -- and is what makes the ENCODER parallel: two
threads packing adjacent groups never share a byte, so groups can be written
concurrently. Within a group, residuals are packed with no padding at all.
Correctness is not argued, it is checked: the GPU output is compared against
tb_dpcm.c byte for byte. That is the rule this codec ships under -- a codec
change lands only when it is proven bit-exact against the CPU reference, because
"looks right" is not a property a lossless format can be verified by.
The header documents the kernel shape, the group/threadgroup sizing, and the
measurements behind them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a CAMetalLayer path alongside SDL's renderer, and the compute decoder that
feeds it. Not reachable yet -- the protocol and the sender come next -- so this
commit adds files and changes no behaviour.
WHY THE DECODE HAS TO BE ON THE GPU
Decoding a 5K frame costs roughly 44 million bit extractions. Measured on the
target hardware, a 2020 Retina 5K iMac: 166 ms per frame single-threaded on its
i5, against 6.5 ms on its GPU. The CPU was never an option -- it is the
bottleneck this codec exists to relieve -- and the GPU sits idle.
WHY SDL'S RENDERER IS REPLACED RATHER THAN USED
SDL2 cannot present 10-bit. The layer is driven at the frame's own resolution
and Core Animation scales it to the window; on a 5K panel fullscreen they match,
so the scaling is free.
The cursor is a separate CALayer child, not part of the video. While the cursor
was drawn into the frame it carried the whole pipeline cost -- capture, encode,
transfer, decode, present, vblank -- which is why vsync visibly affected pointer
movement. As a sibling layer, a pointer move reaches the panel without waiting
for a video frame.
tb_health.{h,m} comes along because the decoder reports into it: drawable wait,
upload copy, submit. It is 395 lines of counters and it is what makes the
numbers in this series checkable rather than asserted. The receiver's drawable
wait in particular is the only instrument that catches a bad send phase, which
holds a full 60 fps while every rate counter reads perfect.
Compiles standalone against this base:
cc -c -fobjc-arc TargetBridge-Receiver/TBReceiverC/src/tb_metal_plane.m \
-ITargetBridge-Receiver/TBReceiverC/src \
-ITargetBridge-Shared/codec $(pkg-config --cflags sdl2)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… them
Wires the decoder in. `make` in TargetBridge-Receiver/TBReceiverC builds and
links with this applied.
Four small changes:
proto.h TB_PKT_RAW_DPCM 0x25, adjacent to the other frame types. 0x25 was
unused here, so no renumbering.
main.c A handler beside handle_raw_frame, and "supportsDPCM" in the
display profile.
display.{c,h}
tb_disp_render_dpcm / tb_disp_dpcm_available, bridging to the
Metal plane the same way the NV12 path bridges to SDL.
Makefile The new sources, the codec include path, and -framework Metal
QuartzCore IOKit.
ADDITIVE BY CONSTRUCTION
The sender sends 0x25 only to a receiver that advertised "supportsDPCM", and
that flag is conditional on tb_disp_dpcm_available() -- which asks whether a
Metal plane can actually be created, not merely whether this build contains the
code. A receiver without a usable GPU path keeps receiving TB_PKT_RAW_FRAME, and
so does an unpatched one, which is silent on the capability. Nothing that works
today changes.
That conditional matters more than it looks: CPU decode is 166 ms per 5K frame
against 6.5 ms on the GPU, so a receiver that advertised the capability without a
working decoder would be far worse off than one that never advertised it.
A failed decode drops the frame rather than presenting a partial result. The
codec is lossless, so there is no graceful degradation to fall back on -- a
partial decode is garbage, not a softer picture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iver can
Completes the path. Both ends build against this base and the existing sender
suite passes unchanged (140 tests, 0 failures).
TBDisplaySenderService.swift
native5kRaw60, threaded through the preset switches. isRawPassthrough
diverts in encode() before the hardware encoder is consulted, and
sendDPCMFrame() encodes on the GPU and emits one TB_PKT_RAW_DPCM packet
with the same backpressure accounting sendRawFrame() uses.
TBMonitorProtocol.swift
rawDPCM = 0x25, and supportsDPCM on the display profile.
Bridging header / project.yml
The codec headers, and a search path to reach them.
Languages/*.json
One title key, untranslated in all five: DPCM is the codec's name.
THE PRESET IS 5120x2880 AT 60, 4:4:4 10-BIT, BIT-EXACT
10-bit is unconditional on this path rather than a toggle, because it is free:
packed 2-10-10-10 is 4 bytes/pixel, exactly like 8-bit BGRA, so it costs the same
bandwidth and the same per-frame work and only removes gradient banding. There
was never a reason to offer 8-bit 4:4:4 as a choice.
queueDepth is 8 here rather than the 5 the other high-rate presets use. Measured:
at 5K the capture callback runs 10-14 ms against a 16.7 ms period, so a shallow
queue leaves ScreenCaptureKit nowhere to put the next frame -- and a stalled
stream delivers NOTHING rather than a stale frame, which reads as "macOS did not
draw it". The reasoning behind 5 still holds at 4K and below.
EVERY FAILURE FALLS BACK RATHER THAN DROPPING
An older receiver (silent on supportsDPCM), a GPU encoder that will not create,
or an encode that returns nothing: each sends the frame uncompressed instead. The
fallback is a real picture, just a larger one on the wire. Nothing that works
today changes, because the lossless path is only ever entered by a preset that
did not previously exist.
NOT INCLUDED, DELIBERATELY
The async encode wrapper we run locally needs a framedPacket helper that does not
exist here, and the synchronous path above is enough to demonstrate the codec. It
is worth adding later: the encoder can reserve room for the packet header ahead
of the blob so the packet is built in place, and on a ~30 MB frame the second
copy measured ~3 ms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reasoning behind the previous five commits in one place: the receiver-side cost that forced a lossless codec, why the tiles are 8x8, why the decoder has to be on the GPU, and the capture and presentation findings that came with it. Also the instruments that lied, because they cost more time than the code did. A cadence histogram with a hardcoded reference period goes blind when that period changes and fails by looking perfect. A full 60 fps with zero drops is not health -- a bad send phase holds one while the receiver waits 16.6 ms for a drawable. `nc -z` proves the kernel completed a handshake, not that the app ever called accept(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
|
Thank you for sharing this experimental path. I am keeping it separate from the 3.6.1 maintenance candidate, in line with your integration-guidance request. The next review needs to cover protocol compatibility, bounds and resource limits, color correctness, fallback and measured performance on both supported Receiver architectures. I am not treating a lossless 5K60 target as a validated guarantee or merging this into the stable branch yet. |
swellweb
marked this pull request as draft
October 16, 2026 14:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. I am asking how you would like this shaped, not proposing it as ready
to merge. Glad to restructure, split, or drop any of it.
The one-sentence version
You already have a raw video path —
TB_PKT_RAW_FRAME, uncompressed NV12, behindthe
RAW=1environment variable. This adds a compressed one that is smaller onthe wire and better looking at the same time, which sounds impossible, so the
numbers are below.
The idea
Today's raw path sends NV12: brightness at full resolution, colour at quarter
resolution, 8 bits per sample. Throwing away three quarters of the colour is what
makes it small enough to move.
This sends 4:4:4 10-bit instead — full colour at every pixel, 10 bits per
sample rather than 8 — and gets it smaller by compressing losslessly instead.
Some terms, since they matter for the rest:
visible on text and UI edges as fringing.
banding in gradients.
"visually indistinguishable": identical.
At 5120×2880, 60 fps:
TB_PKT_RAW_FRAME(NV12)The worst case — a 36-megapixel photograph filling the screen, which is about as
incompressible as a desktop gets — still fits in 90% of the bytes NV12 uses,
while carrying roughly twice the colour information.
Why it is worth doing at all
The bottleneck is not the cable. A 40 Gb/s Thunderbolt link has room for 28 Gb/s
of uncompressed 5K. The bottleneck is the receiving machine.
Measured on a 2020 Retina 5K iMac, per frame: 23.4 ms spent purely pulling
bytes off the network (one core, inside the TCP stack) and another 12.9 ms
pushing those same bytes over PCIe into the GPU. The budget for 60 fps is
16.7 ms. Either number alone busts it.
Both scale with how many bytes arrive, so fewer bytes is the only lever — and it
has to be lossless, or there is no point sending 4:4:4 in the first place.
How the compression works, in plain terms
DPCM stands for differential pulse-code modulation, which is a long name for
a simple idea: instead of writing down each pixel's value, write down how much it
differs from the pixel to its left. Neighbouring pixels are usually similar, so
those differences are small numbers that need few bits. The decoder knows the same
rule, so it can rebuild the original exactly.
The catch is that this is serial — you cannot decode pixel 500 until you have
decoded pixel 499 — and a GPU is only fast because it does thousands of things at
once. Serial work is the one thing it is bad at.
So the frame is cut into 8×8 pixel tiles that are completely independent. Each
tile carries its own starting pixel and its own bit widths, and needs nothing from
its neighbours. At 5K that is 230,400 tiles, which is exactly the kind of
work a GPU wants: 230,400 small identical jobs, all at once.
8×8 was measured, not picked. 4×4 is slightly better on dense text (4.71× vs
4.46×) and much worse on flat areas (9.7× vs 13.8×), because it pays the per-tile
bookkeeping four times as often where there is nothing to compress. 16×16 is worse
everywhere: a single high-contrast edge forces a wide bit width across 256 pixels.
The decoder has to be on the GPU. Decoding is about 44 million bit extractions
per 5K frame: 166 ms on that iMac's CPU, 6.5 ms on its GPU. The CPU is
already the thing that is overloaded.
Nothing that works today changes
The sender only sends the new packet to a receiver that said it can decode it, and
that claim is conditional on a Metal decoder actually being creatable — not merely
on the build containing the code. An older receiver never mentions the capability,
so it keeps getting
TB_PKT_RAW_FRAMEexactly as now.Every failure falls back rather than dropping a frame: no GPU encoder, encoder
returns nothing, receiver too old — all send the uncompressed frame instead. A
bigger picture, never a missing one.
The commits
The first three add files and change no behaviour at all, so they can be read
without worrying about regressions:
f939666cc -O2 -o /tmp/t TargetBridge-Receiver/TBReceiverC/tests/test_dpcm.c TargetBridge-Shared/codec/tb_dpcm.c -ITargetBridge-Shared/codec && /tmp/t→ 290 checks, no dependencies, no hardware2639ab485750beed9bc56e9ff9402054b59What I checked, and what I did not
Checked on this base (
v3.5.2):makeinTargetBridge-Receiver/TBReceiverCbuilds and linksscripts/build_targetbridge_sender_app.shNot checked: two real machines streaming to each other on these commits.
This path runs daily on a fork, but this is a fresh port onto v3.5.2 and I have
not put hardware on it. Please treat the integration as unproven.
What I would like your view on
maint-3.5the right target? I retargeted after opening. Againstmainthe PR showed 64 commits, 58 of them yours, because
mainis at 3.3.0 and sitsbehind the
v3.5.2tag.an existing one?
TargetBridge-Shared/codec/and wired into both an Xcode target and a Makefile build, which may not be how
you want cross-end code organised.
queueDepthis 8 for the new preset where your other high-rate presets use5. At 5K the capture callback measured 10–14 ms against a 16.7 ms period, and a
stalled
SCStreamdelivers nothing rather than a stale frame. Your 5 looksright at 4K and below. Keep the split, or match your 5?
tb_healthcame along because the decoder reports timings into it. It is395 lines of counters and can be dropped if you would rather not carry it.
If reviewing 4.8k lines is the wrong starting point, I am happy to talk through
the design first instead.
🤖 Generated with Claude Code