Skip to content

OBS mode holds 120 fps at 1440p, with an OBS Quality setting for size against picture (#1060) - #1066

Merged
ccoventry merged 5 commits into
devfrom
fix/obs-encoder-speed-1060
Oct 29, 2026
Merged

ccoventry merged 5 commits into
devfrom
fix/obs-encoder-speed-1060

Conversation

@ccoventry

Copy link
Copy Markdown
Owner

Release note: Fixed: OBS mode keeps up with 120 fps at 1440p instead of dropping frames, and a new OBS Quality setting trades file size for picture quality.

Closes #1060

What changes

  • Studio's [DoD-Studio] OBS profile now records in OBS's Advanced output mode on NVIDIA cards (OBS 31+), with NVENC H.264 preset P4, single pass, no lookahead, no B-frames, constant QP. Simple mode's fixed setup (P5, two-pass, lookahead 8, 2 B-frames) used the whole encoder at 1440p120 and dropped 32-95% of frames in five runs (OBS capture records the game's sound behind other windows, at full quality, in sync and smoother (part of #645) #1058's testing).
  • obs-websocket can't write Advanced mode's recordEncoder.json, so Studio writes it itself (native/src/obs/encoder.rs). It finds the profile folder by writing a fresh marker into basic.ini over obs-websocket (which saves at once) and looking for it under %APPDATA%\obs-studio and a portable OBS's config\obs-studio. The existing profile round trip then makes OBS read it.
  • Falls back to Simple mode (as before) when: the encoder isn't NVENC (AMD/Intel/x264 fast settings are unmeasured), OBS is older than 31, or the profile folder isn't found. Never Advanced mode on OBS's NVENC defaults (10 Mbps CBR).
  • New OBS Quality setting (Configuration → OBS, next to OBS Capture FPS): Small (QP 20), Standard (QP 16, default), High (QP 12), Max (4:4:4, QP 12), and "OBS's own" (the old Simple mode, the way back).
  • Colour pinned to OBS's defaults on every connect: NV12 (I444 for Max), Rec. 709, limited range.
  • Docs: docs/obs_alternate_capture.md has every number below.

Test done

All on POST, Brain account via game_lock, OBS 33.0.0, RTX 3070 Ti, 2560x1440, OBS Capture FPS 120. My own Studio build of this branch with its own settings copy (the user's settings.json hash unchanged: 67e8b2b8). Three clips (m00cat BAR multi-kills, wsod25_grp5_h1_m00cat x2, wsod25_grp2_h2_m00cat), game behind windows unless noted. Numbers from OBS's own log ("skipped frames due to encoding lag", "lagged frames due to rendering lag") and nvidia-smi.

OBS Quality Skipped (encoding lag) Missed (rendering lag) NVENC avg / peak GPU 3D MB per minute
OBS's own (Simple, before) 0 this run; 32-95% per clip in five runs on 32.2.2 0 96% / 100% 39% 2,880-3,340
Small 0 0 56% / 72% 40% 1,590-1,900
Standard (two runs) 0 0 57% / 75% 40% 2,770-3,170
High 0 0 64% / 94% 39% 4,170-4,740
Max (4:4:4) 2-5 per clip, all before the first written frame (no timestamp gaps) 0 55% / 79% 48% 5,030-5,720
Standard, game in front 0 0 57% / 69% 41% as Standard

Every take ran 119.9-120.0 fps start to end. OBS's log confirmed the settings it used (preset: p4, multipass: disabled, b-frames: 0, lookahead: false), and the switch from Simple to Advanced happened live on first connect.

Quality (offline, ffmpeg on the same NVENC, 720 frames of an HLAE lossless 1440p take at 120 fps, converted as OBS converts):

Setup Encoder fps VMAF VMAF worst 1% RGB PSNR
OBS's own Simple HQ 119 99.67 96.31 37.59
Small 247 99.42 95.47 37.06
Standard 244 99.79 96.76 37.66
High 236 99.89 97.46 38.08
Max (4:4:4) 172 99.86 97.19 41.98
H.264 lossless 4:2:0 (for scale) 309 99.96 98.32 38.53

4:2:0 caps RGB PSNR near 38.5 dB at any QP: red kill-feed names lose saturation in every 4:2:0 choice, only Max matches HLAE's. HEVC gave the same size at the same quality, slightly slower, so every choice is H.264. No AV1 encoder on a 3070 Ti.

Front vs behind: identical (0 skipped, NVENC 57%, same sizes).

Does the hook DLL cost OBS anything? POV demo ktps8w8-stealth_ih_saints_h1_p3, game in front, launched as (a) HLAE + hook, (b) HLAE only, (c) plain hl.exe:

(a) HLAE + hook (b) HLAE only (c) plain hl.exe
timedemo uncapped, avg fps (whole demo) 231.3 / 232.4 / 233.1 237.8 238.2 / 238.1 / 238.2
(a) with ds_hd_enabled 0 237.9
(a) with ds_sound_restore_gunshots 0 / ds_hud_show_scope 0 233.1 / 232.9
OBS at fps_max 240, 60 s: skipped / missed 0 / 0 0 / 0 0 / 0
same: NVENC / GPU 3D 66% / 33% 66% / 33% 67% / 33%
OBS at fps_max 1000, 60 s: skipped / missed 0 / 0 1 / 1 of 7,356 0 / 0

The game is CPU-bound near 238 fps at 1440p here. The hook costs ~2.5% of uncapped frame rate (~0.1 ms/frame), all of it HD textures; it is not a cause of OBS drops. No frame-time 1% low: PresentMon needs admin (ETW).

Other leads checked: OBS Game Capture uses gl shared texture for hl.exe (with and without HLAE), not the slow shmem path. fps_max 120 vs 240: GPU 3D 29% vs 40%, no difference to OBS. OBS as admin / process priority / Game Capture frame-rate limit not tested: there was no rendering lag left to fix. Windows Game Mode is off; HAGS at the driver default (no HwSchMode value).

Unit/CI checks: cargo test -p native -p desktop-studio (native 833 pass, studio 239), clippy 1.98.1 workspace -D warnings, fmt 1.98.1, npm run test:unit 914 pass, Playwright e2e 407 pass. New tests use fake_obs.rs: Advanced mode + encoder file written to our profile folder only, round trip happens, no folder found keeps Simple mode, "OBS's own" switches back.

Decisions

  • Default stays Standard, not High: High's NVENC peak hit 94% here, so on a weaker card Standard has the headroom. For this PC and the user's wish for top quality, High is my pick (or Max for clips that go through Studio's render step).
  • Max (4:4:4) is offered but not default: some editors can't decode 4:4:4 H.264, and OBS feeds I444 to NVENC through a slower system-memory path.
  • H.264 only (HEVC no better here).
  • Non-NVENC PCs keep Simple mode rather than guessed AMD/QSV settings.

Needs the user

  • Pick the OBS Quality you want (I'd say High; Max if VEGAS opens it). Short clips to compare: local\review-images\obs-encoder\live\clip_b1_11s_2s_{standard,high,max444}.mp4; stills in local\review-images\obs-encoder\quality\ and \live\.
  • Check whether VEGAS 21 opens the Max (4:4:4) clip; if not, Max is for the render step only.

🤖 Generated with Claude Code

ccoventry and others added 3 commits October 26, 2026 22:30
…lity choice (#1060)

Simple mode's NVENC setup (P5, two-pass, lookahead, B-frames) can't keep up
with 120 fps at 1440p. Studio now puts its own profile in Advanced mode and
writes recordEncoder.json itself (P4, single pass, no lookahead, no B-frames,
constant QP), found through a marker written over obs-websocket. Simple mode
stays for non-NVENC encoders, OBS before 31, or a profile folder it can't
find. A new OBS Quality setting picks Small/Standard/High/Max or OBS's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ladder, hook cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Merged dev in, and a live test on POST: passes

Merge (740359a): one conflict, pub mod drop_rate; (#1063) beside pub mod encoder;; both kept. fmt, clippy (1.98.1), cargo test -p native -p desktop-studio and npm run test:unit (991) pass locally.

Live test (POST-Anniversary, Brain account under game_lock, game behind windows, OBS 33.0.0, 2560x1440, OBS Capture FPS 120, debug build with its own settings copy; 3 clips of ktps8w3-dyelife_ih_anzio_h2.dem per run). With #1063 now on dev, its live drop line shows the result directly; the same 3 clips at 120 fps dropped 80-86% before this PR (#1063's test).

OBS Quality Profile change Studio logged Live line / Last Batch Files (ffprobe)
Max (4:4:4) Simple → Advanced, NV12 → I444 0% skipped, 0% missed on every clip, no warning or toast H.264, yuv444p, 120/1, 1,304-1,308 frames; 1.28 GB for 3 clips
Standard CQP 16, P4, single pass, no lookahead, no B-frames; I444 → NV12 0% skipped, 0% missed on every clip H.264, yuv420p, 120/1; 1.38 GB for 3 longer clips

Still open from the PR's "Needs the user": whether VEGAS 21 opens the Max (4:4:4) files.

@ccoventry

Copy link
Copy Markdown
Owner Author

VEGAS Pro 21 check: the user opened the Max (4:4:4) take video.mp4 and the 6 clips Studio finished from the test batches; all play fine. That closes the last "Needs the user" item.

@ccoventry
ccoventry merged commit 537c6ae into dev Oct 29, 2026
5 checks passed
@ccoventry
ccoventry deleted the fix/obs-encoder-speed-1060 branch October 29, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OBS mode's 120 fps default is past what the encoder holds at 1440p

1 participant