DNI: DEBUG: PR to debug/fix CPU CI failures - #400
Draft
ksheth-dev wants to merge 8 commits into
Draft
Conversation
- VideoEncoder Protocol + ChunkDeliveryResult in
flashdreams/serving/webrtc/encoders.py, with two implementations:
DefaultRTCEncoder (adapter over aiortc's software encoder) and
PyNvHardwareEncoder (NVENC H.264 via PyNvVideoCodec, output as
av.Packet).
- NVENCVideoTrack (media.py) delivers pre-encoded packets on aiortc's
public av.Packet -> H264Encoder.pack() path -- no reach into
aiortc private attributes. Drop-oldest overflow with a bounded
Queue[av.Packet]; recv() paces at 1/fps like BufferedVideoTrack.
- select_encoder() factory with a two-stage capability probe:
Stage 1 GetEncoderCaps (silent fallback to DefaultRTCEncoder on
environmental "not supported"; loud EncoderInitError under
backend='nvenc'); Stage 2 CreateEncoder (hard error, never a
silent fallback -- masking a driver / session-pool / hardware
failure would hide real problems).
- NVENC configured for interactive low-latency H.264: fmt=ABGR
(NV_ENC_BUFFER_FORMAT_ABGR is word-ordered -> little-endian byte
layout [R,G,B,A]; see _chunk_to_abgr_cuda_frames docstring),
preset=P4, ULL tuning, CBR rate control, bf=0, lookahead=0,
repeatspspps=1 (aiortc's H264Encoder.pack does not synthesize SPS
and PPS). Packets carry pts on the RTP 90 kHz clock so
H264Encoder.pack rescales cleanly via convert_timebase.
- PyNvVideoCodec>=2.1,<3 added as a hard dep on
integrations/omnidreams -- omnidreams already requires CUDA at
runtime, so the install matrix is unchanged.
- Startup emits a single INFO log line per session,
'Video encoder ready: backend={pynvvideocodec|aiortc} ...',
giving one grep anchor regardless of which backend was selected.
- ci_cpu tests cover: Protocol conformance, factory branch coverage
(Stage-1 library / caps / bounds failures under both auto and nvenc,
Stage-2 hard-error regression guard under both), pre-encoded packet
plumbing on NVENCVideoTrack (pts / time_base / drop-oldest overflow),
cross-chunk packet-ordering invariant that guards the manager's
sequential-await pattern from a future create_task rewrite, and
compat guards for aiortc + PyNvVideoCodec public surfaces.
- ci_gpu smoke test encodes a 4-frame chunk on real hardware and
asserts Annex-B start code + IDR + SPS + PPS in the first packet,
plus pts=0 and time_base=1/90000.
No runtime path currently consumes the new abstraction; behavior
change lands in the follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… fallback
- OmnidreamsRuntimeConfig gains encoder_backend
(auto | nvenc | default), encoder_bitrate_bps, encoder_gop.
select_encoder() runs on the runtime executor thread inside
_initialize_video_encoder_sync; the encoder is closed cleanly in
_close_sync so its NVENC session slot is released promptly.
- _generate_one_chunk_sync drops the .cpu() D2H copy on the tensor
and replaces it with torch.cuda.current_stream().synchronize().
Same wall-clock cost as before (both wait for the compute stream to
drain), but the hardware encoder can now read the CUDA tensor
directly via DLPack; the software path picks up the D2H inside
BufferedVideoTrack's worker.
- omnidreams.webrtc.server exposes one new CLI flag,
--prefer_sw_encoder. Defaults false -> encoder_backend='auto'
(probe NVENC and fall back silently to aiortc's software encoder
on Stage-1 unsupported). Setting it maps to encoder_backend='default'
and skips the NVENC probe entirely -- useful for A/B profiling and
known-flaky NVENC hosts. The tri-state config field stays on
OmnidreamsRuntimeConfig for programmatic use and test coverage of
the 'nvenc' loud-on-failure branch.
- Manager wiring: addTransceiver + setCodecPreferences constrain the
SDP to H.264 when the encoder emits pre-encoded packets
(prefers_codec == 'h264'). Post-answer, transceiver._codecs is
inspected; if H.264 did not land (e.g. a browser that will not
offer it), _enforce_h264_or_fallback closes the NVENC session and
installs DefaultRTCEncoder + BufferedVideoTrack via
sender.replaceTrack -- a pre-stream swap, no renegotiation.
- Chunk delivery in the generation worker goes through
encoder.deliver_chunk(...). The await is load-bearing for
cross-chunk packet ordering (guarded by the regression test added
in the previous commit): a create_task rewrite here would allow
chunks to complete out of order and packets to land on the track
with non-monotonic pts.
- Test suite extends _FakeVideoEncoder to satisfy the VideoEncoder
Protocol, threads video_encoder through every ManagedWebRTCSession
construction, and adds ci_cpu tests for _enforce_h264_or_fallback
(swap on VP8-negotiated, keep on H.264-negotiated, swap on empty
codec list) plus a parametrized test that --prefer_sw_encoder
correctly maps to encoder_backend ('default' when set, 'auto' when
unset).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- encoders.py: guard the ``PyNvVideoCodec`` import under ``TYPE_CHECKING`` so ty sees a single ``Any``-typed ``nvc`` name. Without the guard, ty preserved a ``<module PyNvVideoCodec> | None | Any`` union and rejected the ``None`` fallback and every attribute access on ``GetEncoderCaps`` / ``FORCEIDR`` / ``CreateEncoder``. Runtime path (the ``else:`` branch) keeps the same guarded try / except behaviour. - encoders.py: narrow the ``VideoEncoder.create_track`` return type from ``MediaStreamTrack`` to ``BufferedVideoTrack | NVENCVideoTrack`` so the manager can assign the result to the ``ManagedWebRTCSession .video_track`` field without a widening error. - manager.py: change ``ManagedWebRTCSession.video_track`` annotation from ``MediaStreamTrack`` to ``BufferedVideoTrack | NVENCVideoTrack``. The aiortc base type does not advertise ``.close`` / ``.fps`` / ``.qsize``, which the generation worker and liveness watchdog need. Drop the now-unused ``MediaStreamTrack`` import. - test_encoders.py: swap mypy-style ``# type: ignore[misc|arg-type]`` to ty-style ``# ty:ignore[invalid-assignment|invalid-argument-type]`` on the frozen-dataclass and SimpleNamespace test sites. Add ``assert packet.pts is not None`` before ``int(packet.pts)`` in the ordering tests -- ``av.Packet.pts`` is nullable at the type level even though the fake encoder always sets it. - test_webrtc_manager.py: add a local ``_FakeVideoEncoder`` stub and thread ``video_encoder=`` through the ``ManagedWebRTCSession`` construction the shared base test uses. - integrations/lingbot/tests/test_webrtc_runtime.py: same treatment for the three ``_ManagedLingbotSession`` constructions -- the lingbot tests broke when we promoted ``video_encoder`` to a required field on the shared ``ManagedWebRTCSession`` base. - integrations/omnidreams/tests/test_webrtc_runtime.py: drop 8 unused ``# ty:ignore[invalid-argument-type]`` comments ty flagged (5 on ``video_encoder=…``, 3 on ``transceiver=transceiver``). - integrations/omnidreams/ludus-renderer/examples/render_mirror_augmented.py: drop the ``# ty:ignore[unresolved-import]`` on the ``PyNvVideoCodec`` import. The comment was pre-existing but became unused when this feature promoted PyNvVideoCodec from an optional extra to a hard dependency of ``integrations/omnidreams``. - Ruff format touched a few files in passing (test_nvenc_track.py, test_nvenc_smoke.py, test_encoders.py); those changes are formatting-only and included here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
``_enforce_h264_or_fallback`` had two ownership violations: 1. It overwrote ``managed_session.video_track`` without closing the pre-encoded NVENC track. ``ManagedWebRTCSession.close()`` then only saw the fallback track, leaving the original's ``readyState`` "live" and its packet queue undrained. 2. It called ``close()`` on ``managed_session.video_encoder``, which is the same object the runtime owns via ``runtime._video_encoder`` (created once in ``_initialize_video_encoder_sync`` and reused across sessions). Subsequent sessions would read a closed ``PyNvHardwareEncoder`` from ``runtime.video_encoder``. - ``_enforce_h264_or_fallback``: sync → async; await ``old_track.close()`` before installing the fallback. Do not close the encoder — the runtime keeps ownership and releases it at runtime shutdown. - Update the one caller in ``_create_answer_with_runtime_ready_locked`` to await the coroutine. - Tests: run the three cases under ``pytest-asyncio``; assert the orphaned track IS closed (session-scoped) and the runtime-owned encoder is NOT closed (survives session-scope fallback). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
``BaseWebRTCSessionManager._create_answer_with_runtime_ready_locked`` unconditionally read ``runtime.video_encoder``. Only ``OmnidreamsInferenceRuntime`` defines that property; ``LingbotInferenceRuntime`` does not, so a real Lingbot WebRTC session would raise ``AttributeError`` before returning the SDP answer. Lingbot's session-construction tests sidestep this because they build ``_ManagedLingbotSession`` directly with a fake encoder, bypassing the answer-creation path that triggers the read. - ``BaseWebRTCSessionManager``: add ``_resolve_video_encoder()`` that reads ``runtime.video_encoder`` if present, else constructs a session-scope ``DefaultRTCEncoder(fps=self.fps)``. - ``_create_answer_with_runtime_ready_locked`` routes through the new method instead of the direct attribute read. - ``test_webrtc_manager.py``: add ci_cpu tests for both branches (runtime provides an encoder, runtime does not). Extend ``_FakeVideoEncoder`` with an async ``deliver_chunk`` so the existing generation-worker test that drives one chunk end-to-end keeps passing after the manager started calling into the encoder. - ``integrations/lingbot/tests/test_webrtc_runtime.py``: add ``video_encoder=_FakeVideoEncoder()`` on three ``_ManagedLingbotSession`` construction sites that were missed when the field became required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… rank ``_initialize_video_encoder_sync`` ran on every rank via ``_initialize_sync_all_ranks`` (``distributed_op``), allocating an NVENC session on every worker even though only the master rank serves WebRTC media. Worker ranks then held a session slot they would never use, and if the local NVENC pool could not accommodate one allocation per rank, startup would fail on the ranks that were unable to allocate. - ``_initialize_video_encoder_sync``: early-return on non-master ranks. ``runtime.video_encoder`` is only ever read on the master, so leaving ``_video_encoder`` as ``None`` on workers is safe. - ci_cpu tests: worker rank never reaches ``select_encoder``; master rank still initializes normally. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split PyNvHardwareEncoder and its ABGR-frame / NAL-scan helpers out of encoders.py into a sibling nvenc.py that owns the top-level PyNvVideoCodec import. encoders.py no longer touches PyNvVideoCodec at module load; select_encoder probes availability via importlib.util.find_spec (no side effects) and imports nvenc only when a hardware backend is actually about to be constructed. The Lingbot WebRTC path resolves to DefaultRTCEncoder in the base manager and never enters the hardware branch, so its process no longer loads PyNvVideoCodec at all — sidestepping a silent early-exit observed during the Lingbot server launch when PyNvVideoCodec was importable. Tests updated to patch _pynvvideocodec_installed and inject a fake PyNvVideoCodec into sys.modules so nvenc's top-level import binds to the mock. The GPU smoke test's imports follow the split. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ksheth-dev
force-pushed
the
pull-request/999999
branch
from
July 29, 2026 09:09
3ce03df to
b65f4c8
Compare
ksheth-dev
force-pushed
the
dev/ksheth/pynvvideocodec_v2
branch
4 times, most recently
from
July 29, 2026 11:21
bec5ea2 to
6434238
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.