Skip to content

Render while capturing: each clip renders as soon as it's recorded, and a long batch waits for space instead of filling the drive (#861) - #885

Merged
ccoventry merged 19 commits into
devfrom
feat/render-while-capturing-861
Oct 28, 2026
Merged

ccoventry merged 19 commits into
devfrom
feat/render-while-capturing-861

Conversation

@ccoventry

Copy link
Copy Markdown
Owner

Release note: New: "Render while capturing" (Configuration → Render Output, off by default) renders each clip as soon as the batch has recorded it. With takes set to be deleted, a long batch waits between clips for the renders to free room instead of filling the drive.

Stacked on #716 (feat/auto-delete-takes-644), which isn't merged yet. This PR gets no CI until #716 merges and it retargets to dev; local results are below. Before stacking, I merged dev into #716's branch (no conflicts) and pushed it. That brings in #854's merging of scans into a running queue, which this PR needs.

Closes #861

Measurements

The full table is in #861 (comment).

Setup:

  • PRE install.
  • utvideo, 2560x1440, mirv_movie_fps 300.
  • 40 s of recording per run.
  • Studio's own ProRes 422 HQ arguments for the render.
Load during the capture Capture speed Render speed
none 121 / 114 fps (two runs) n/a
1 render, 4 threads 109 fps 8.9 fps (11.5 fps with nothing else running)
1 render, 16 threads 100 fps 24.5 fps (38.5 fps with nothing else running)
2 renders, 4 threads each 106 fps 8.1 + 8.1 fps

What the runs showed:

  • Video and audio lengths matched in every take (gap of 0.1 ms or less), with no dropped or doubled frames.
  • A capture writes about 4.2 MB per frame (about 480 MB/s). A render can't keep up with that, so this feature is about disk use, not speed.
  • The ProRes clip comes out at 41% of the take's size.
  • The planner's uncompressed estimate is about 2.6x the real take size.

What it does

  • The setting. Configuration → Render Output → Render while capturing, off by default. The hint under it follows "After a clip is finished" (Delete finished takes automatically: Recycle Bin or permanent, optionally only when space is needed (part of #644) #716).
  • Each clip renders as it lands.
  • One render at a time while recording. It uses the threads one of the usual concurrent renders gets (-threads = cores ÷ the setting).
    • While the batch is held, a render that starts gets the whole machine.
    • When the batch ends, renders go back to the usual number at once.
  • Takes go as soon as their clip verifies while the capture runs, using Delete finished takes automatically: Recycle Bin or permanent, optionally only when space is needed (part of #644) #716's rules: every stream Finished with the clip file there, plus the Delete Take guard. Those deletions are logged but don't toast, so a long batch doesn't toast once per clip.
  • The calculation (native::hlcr::capture_room, pure and tested). At every START_RECORD, it asks whether there is room for the clip being recorded plus the next clip plus 15 GB of headroom on the next clip's drive.
    • If not, it checks whether the takes on that drive that will be deleted after rendering free enough:
      • a take whose clip goes to another drive frees its whole size;
      • a take whose clip goes to the same drive frees its size minus the clip, priced by the ratio learned from this batch's finished renders.
    • Continue, Hold, or CannotFit.
    • A take that can't be priced yet means Hold rather than giving up.
    • Clip sizes are learned from the takes that have landed: the largest bytes per tick so far ×1.5, never above the planner's figure.
  • Resuming. While held, the batch goes on once there is room for the next clip plus one more clip, while renders can still free more. That stops it from holding again right after one clip. The time left is shown from how fast free space is coming back.
  • The pause. The hook's batch (Seek Between Clips) gets dodstudio_batch hold and stops at the next clip boundary:
    • a boundary is the seek to the next clip, or the move to the next demo, both after the clip's recording stopped;
    • the demo is paused and BATCH_HELD goes out on the events pipe;
    • go lets it carry on (BATCH_GOING).
    • capture_engine sends the change over the game's pipe and tries again every second if a send fails.
    • The Capture footer says "Will wait after this clip…", then "Waiting for renders to free X GB for the next clip (about N min)".
  • Stopping cleanly. The batch is cancelled at the boundary, keeping the clips it recorded, with an activity-log line and a toast, when either:
    • it is held and even rendering and deleting every pending take would leave the next clip short;
    • nothing has been queued or rendering for a minute.
  • Planning. With the setting on, deletion set to permanent and the hook running the batch, build_batch_queue still gives a drive to blocks that don't fit now, instead of refusing the batch. PatcherConfig::render_while_capturing controls this.

Decisions

  • Stacked on Delete finished takes automatically: Recycle Bin or permanent, optionally only when space is needed (part of #644) #716, with dev merged into it first. Render tab: Scan for Takes works at any time, even while rendering, and merges into the queue (#772) #854's queue merge was needed. That merge had no conflicts, and native, studio and unit tests passed after it.
  • Measured with game_probe.py, not a Studio batch. I sent HLAE commands at the same codec, resolution and frame rate as the capture settings. Driving the running Studio would have written into its configured WSOD25 export folders.
  • Off by default. This follows the "won't work on every PC" decision.
  • While recording, a render gets one share of threads, not every core. Every core costs the capture about 12-17% against 5-10% for one share, and the render barely keeps up either way.
  • The pause is a hook command, not a new console name. hold/go are new arguments to dodstudio_batch, so there is no name to add to console_commands_data.js. Commands.txt and docs/dodstudio_commands.md say what the arguments do.
  • A patched batch (no hook) renders as it goes but can't hold. For that reason, overcommitting the drive plan needs the hook route.
  • Keep or the Recycle Bin frees nothing, so the drive plan isn't overcommitted for them. With either choice, a batch that runs short stops cleanly at a clip boundary instead of filling the drive.

Tests (local)

  • cargo test -p native -p desktop-studio: pass (native 768, studio 163). New tests:
    • 11 in capture_room;
    • the overcommitted plan in builder/tests.rs;
    • the concurrency cap and the planner's share per block.
  • cargo test -p goldsrc-hooks --lib batch: 10 pass, including that a hold stops only at a clip boundary.
  • clippy 1.98.1 -D warnings: clean on the workspace, and on goldsrc-hooks for both i686 and the host. cargo fmt is clean.
  • npm run test:unit: 608 pass, 10 of them new in render_while_capturing.test.js.
  • Playwright: finish-clips, take-cleanup and render-studio specs pass (66 tests).
  • In game, the hold on both installs (game_probe.py with the test DLL local\test-861\dodstudio_goldsrc_hooks.dll), a 2-clip batch file on ktps8w8-stealth_ih_saints_h1_p1:
    • PRE: PASS. hold stopped it after clip 1's stop, before the seek. It stayed held for 10 s: BATCH_HELD arrived, the bare command reported "held between clips" and no seek ran. go sent BATCH_GOING, the seek ran, and the batch finished with BATCH_COMPLETE.
    • POST: PASS, same steps.

Not tested

The whole loop in Studio has not been run: a real batch with the setting on, a render queued per clip, takes deleted mid-batch, and a hold and resume on a drive that is nearly full. To check by hand:

  1. Turn the setting on with "Delete the take permanently", and capture 3 highlights.
    • Each clip should appear in the Render tab while the batch goes on.
    • Its take should go once its row finishes.
    • The activity log should have [render-while-capturing] lines.
  2. For a hold, capture to a drive with little free space, or a nearly full one: the footer should show the waiting line, the game should wait between clips, and it should carry on once renders have freed room.

🤖 Generated with Claude Code

ccoventry and others added 4 commits October 19, 2026 22:21
…between clips for space (#861)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…old while a take settles

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the whole machine

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…keep renders in the take index

Review fixes for #885:
- Record markers find their block by tick, so a skipped demo no longer
  shifts every later clip; no index panic in the hold watcher.
- The planner only overcommits when each clip fits a drive on its own.
- A held batch repeats BATCH_HELD and ends (game quits) once Studio has
  been gone 10 s; going on to the next demo needs no clock.
- go resets the OBS stall clock; hold gaps don't widen the deadline.
- The recording clip counts against the next only on the same drive.
- A take that never settles is not handed over.
- capture_clip_ready marks the block Captured and records the take before
  its render; batch-end verification counts rendered-and-deleted takes and
  keeps their render history.
- A HUD pair's take is cleaned up once per pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Correctness review

Fixed in 101eda3, pushed to this branch:

Clips matched to the wrong block. START_RECORD / STOP_RECORD were numbered by counting them. A demo the hook skips (load timeout, demo ended early) sends no markers for its clips, and a skip mid-clip sends a start with no stop. After that, every later clip was matched to the wrong block: the wrong folder was settled and queued, a skipped clip's partial take could be rendered, and sizes were learned from the wrong clips. The engine now passes the tick each marker names, and CaptureRoom finds the block with that record_start_tick / record_stop_tick, searching forward from the last match (resolve_block, tested).

A panic that would close Studio. The hold watcher indexed self.blocks[i] with the recording index. With more markers than blocks, that panics, and release builds abort. It now goes through get.

The planner let a batch start that can never finish. The overcommit added the whole demo's estimate to every drive's free space. So a clip bigger than any drive was planned (the PR's own test asserted that), and the first clip is never checked at run time, so it recorded into the full drive. Now each block only has to fit on its own, on the earliest drive whose free space at planning time holds it plus the 15 GB (allocate_blocks_one_at_a_time, tested). A block no drive can hold alone is still refused, with a message saying so. The test was split into "fits one at a time: planned" and "bigger than the drive: refused".

A hold that never gets a go. If Studio closed or crashed while the batch was held, HOLD stayed set and the game sat paused forever. #760's catch-up waits for hl.exe to exit, so it never ran either.

  • A held batch now repeats BATCH_HELD every 5 s. The events pipe only notices a reader has gone when a write fails, so this is what lets it notice.
  • Once Studio has been gone from the pipe for 10 s, the batch ends there and keeps its clips. batch_end::batch_ended() then quits the game after the usual grace period, the same as after a batch Studio stopped watching.
  • Without the events pipe (GOLDSRC_HOOKS_EVENTS=0), the hook can't tell whether Studio is still there, so the hold stays.
  • Studio only logs a change in game_held, so the repeats don't flood the activity log.

Going on to the next demo after a hold. That step needed the old demo's clock to still be active, or the demo was skipped with a BATCH_PROBLEM. Moving to the next demo needs nothing from that clock, so it now just goes.

OBS mode stopping right after a long hold. Once go was delivered, the stall check counted from the last marker before the hold. In the same loop pass, it would declare a stall. The gap spanning the hold also widened longest_marker_gap for the rest of the batch. Now go resets last_marker_at, and gaps while held aren't counted.

The clip being recorded counted against the next clip's drive. This happened even when the two clips go to different drives. It could Hold, or stop with CannotFit, a batch the planner had placed correctly, including with Keep. Now the clip only counts when both clips are on the same drive.

A take that may still be written was handed over. settle returned the last size it saw after the 120 s limit or on cancel, and the take was queued for rendering. That means it could later be deleted after rendering a partial file. Now settle returns None then, and the take is left for the finish step after the batch.

Render While Capturing's renders and the take index. The take index only learned a batch's takes at capture_takes_verified, at the batch's end. Because these renders ran mid-batch:

What changed:

  • capture_clip_ready now carries take_key and source_streak_indices. Before the render is queued, markClipCaptured (capture_pane.js) records the take in the index and marks its highlights Captured. The block logic moved to take_index.js as markBlockCaptured (tested), and the batch-end verification now calls the same function.
  • At the batch's end, a block whose take is gone but whose history shows a finished render is captured and rendered. rendered is a new field. Such a block counts toward renderable_count, and finishFolders still skips it.
  • record_capture_settings keeps the history it finds (tested).

A HUD pair checked twice. Both of a HUD pair's jobs finishing in one scheduler pass put the take into rendered_now twice. It now goes in once.

Checked and judged fine:

Not fixed, minor: after a resume, the footer keeps "Waiting for renders…" until the next progress line overwrites it, since null status clears nothing.

Tests (local; this PR is stacked, so it gets no CI):

  • cargo test -p native -p desktop-studio --locked: pass (native 771, studio 166).
  • cargo test -p goldsrc-hooks --lib batch: 10 pass.
  • clippy 1.98.1 -D warnings: clean on the workspace, and on goldsrc-hooks for both i686 and the host. cargo fmt --check: clean.
  • npm run test:unit: 609 pass.
  • Playwright finish-clips, take-cleanup and render-studio: 66 pass.

The test DLL in local\test-861\ is rebuilt with the hook change. The "Studio gone while held" path has not been run in the game.

ccoventry and others added 2 commits October 22, 2026 04:06
…use ds_capture_batch

hold_command sent the old dodstudio_batch name, which the hook no longer registers since #719, so hold and go did nothing; it now uses runtime_batch::BATCH_COMMAND (with a test). Commands.txt: dev's file with this branch's hold|go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Brought up to date with #716 (unattended pass). Head 5864f2e9.

In game. DLL local\test-861\, rebuilt from 5864f2e9. game_probe, Brain account, ktps8w1-m00cat_soul_lenn_h2, bare ds_capture_batch.

  • PRE, PASS (local\game-probe\uh-885-pre-r2\):
    • "no batch running; usage: ds_capture_batch start "" | away | hold | go | stop"
    • ds_debug_status: temp-entity fix 6 of 6, hull-trace guard on (pre-Anniversary on PRE, 25th Anniversary on POST)
  • POST, PASS (uh-885-post-r2\): the same.
  • A real batch with hold/go was not run.
  • Neither hook log shows a hook failing. The only "not installed" lines are modules that are build-specific by design: frame_esc on PRE, lightmap_gamma on POST.

🤖 Generated with Claude Code

…uring-861

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

In-game check of the "Studio gone while held" path, POST: it works.

DLL local\test-861\dodstudio_goldsrc_hooks.dll, game_probe, Brain account, POST, no Studio. A small script played Studio's part: it read the events pipe (\.\pipe\dodstudio-hl-<pid>-events) and then closed it 3 s after the first BATCH_HELD, as a Studio that crashed would. The batch file had one demo (k4-ktps9w1-gskill-axis from PRE's dod) with no recording: at 402 echo marker A, seek 406 500, at 505 echo marker B, next 510.

How the hook notices: BATCH_HELD is written every 5 s while held; the first write after the reader left fails, which marks Studio disconnected; 10 s after that the batch ends. So from the moment Studio dies to the batch ending takes 10-15 s, and the game quits 5 s later (batch_end's grace).

What happened (hook log, local\game-probe\probes-885-post-2\):

  • 16:31:08 ds_capture_batch hold → "will hold at the next clip boundary"
  • 16:31:20 "held before the next clip (demo 1, clock 406.0)": marker A had gone out, the seek did not run, marker B never ran. The demo is paused (two grabs 2 s apart show the same frame).
  • 16:31:25 bare ds_capture_batch → "demo 1 of 1: held between clips"; a console echo ran too, so the game took commands while held.
  • 16:31:25 the reader closed the pipe; 16:31:30 "events: Studio disconnected" (next beat)
  • 16:31:40 "Studio is gone while the batch was held for space: ending it here; the clips recorded so far are kept", "done: 1 demo(s) in 36 s". The demo unpaused and a console command still ran after this.
  • 16:31:45 "BATCH_COMPLETE and no Studio on the events pipe -- quitting the game (A capture batch outlives DoD Studio closing, and Studio catches up when it reopens #545)". The game exited on its own, with no crash.

So the game doesn't stay open after the timeout: as dodstudio_commands.md says, it ends the batch and quits, like a batch Studio stopped watching. It was alive and took commands right up to the quit.

A first run hit the "next demo" boundary instead: after the fast load this demo's clock starts at 397 (#893's gap), so my first batch file's times were all in the past. It held "before the next demo" and timed out the same way (Studio disconnected 16:29:55, batch ended 16:30:05, quit 16:30:10, local\game-probe\probes-885-post\). Both boundaries hold, and both time out.

Not tested: PRE; a real Studio (crash or close) mid-batch; and GOLDSRC_HOOKS_EVENTS=0, where by design the hold never times out.

Screenshots: local\review-images\885\post_held_paused_1.png, post_held_paused_2_two_s_later.png, post_after_studio_gone_batch_ended.png.

@ccoventry

Copy link
Copy Markdown
Owner Author

Heads-up from the #1025 (#952) review: git merges this PR and #1025 cleanly, but the build then fails. #1025 gives verify_capture_takes a second argument (a per-take callback), and this PR's new test calls verify_capture_takes(&manifest). Whichever merges second needs verify_capture_takes(&manifest, &mut |_| {}) there. I checked this with a trial merge plus cargo test --no-run. Nothing else clashed: check_takes picks up this PR's rendered counts as they are.

ccoventry and others added 2 commits October 27, 2026 05:58
…render-while-capturing-861

capture_manager.rs: verify_capture_takes keeps dev's on_checked progress
(#952) and this branch's rendered-while-capturing check; the #861 test
passes a no-op callback; both sides' tests kept. render_manager.rs and
settings_manager.rs: both sides' tests and asserts kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew form from #644

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Merged feat/auto-delete-takes-644 (#716, itself just brought up to date with dev) into this branch.

Conflicts: test modules in capture_manager.rs, render_manager.rs and settings_manager.rs (kept both sides' tests, each with its #[test]), and the #861 test now passes a no-op to verify_capture_takes, which takes dev's progress callback (#952). One semantic break from #716's "each job or block must fit on one drive" commit: SpaceNeed takes pieces instead of needed_bytes, so Render While Capturing's room-making now asks for one piece (the block being written plus the next, both on the same drive).

This PR is stacked on #716, so ci.yml doesn't run for it. Local results on the merged head:

  • rustup run 1.98.1 cargo fmt --all --check: clean
  • cargo test --workspace --locked --no-fail-fast: all pass
  • rustup run 1.98.1 cargo clippy --workspace --all-targets --locked -- -D warnings: clean
  • npm run test:unit: pass

🤖 Generated with Claude Code

Base automatically changed from feat/auto-delete-takes-644 to dev October 28, 2026 08:52
ccoventry and others added 2 commits October 28, 2026 03:18
Conflicts: index.html keeps dev's move of "After a clip is finished" to
Destinations (#1101), with Render while capturing left in Render Output;
the builder's allocation keeps #885's one-at-a-time fallback and uses
dev's no_room_message for both refusals (#1097).

allocate_blocks_one_at_a_time now walks drives and reads each one's
volume budget (#774), so with two capture folders on one disk it returns
folder indexes, not volume indexes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Merged dev in (237aa12). The two conflicts: "After a clip is finished" now lives in Destinations (#1101), with Render while capturing kept in Render Output; and both no-room refusals use #1097's plain message. Also fixed: allocate_blocks_one_at_a_time mixed up folder and volume indexes after #774's per-volume budgets, so with two capture folders on one disk a clip could be routed to the wrong folder. It now walks folders and reads each folder's volume budget; added a test for it. Local: workspace tests, clippy (1.98.1), 950 JS unit tests, and the take-cleanup/capture e2e specs all pass.

ccoventry and others added 2 commits October 28, 2026 14:44
…that Video mode plans at 2 B/px (#1103)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The hook's hold also stops at the fast-forward between two clips close
  together: with no seek between them, a hold asked for during clip 2 used
  to wait until after clip 3.
- Start is no longer greyed out for a batch short of room that will hold
  between clips for renders (same test as capture_manager.rs).
- The hold line stays on the Capture footer: it goes through the footer's
  own line, which redraws every second, instead of being overwritten by it.
- The automatic finish step stays out of a batch that rendered while
  capturing (`rendered_while_capturing` on capture_takes_verified), instead
  of failing with "Nothing queued to render".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccoventry

Copy link
Copy Markdown
Owner Author

Live test: Render while capturing in Studio (2026-10-28, POST only, Brain account)

Setup: Studio from this branch, run with its own copy of the settings so the real ones weren't touched (hash unchanged afterwards); test DLL local\test-885\ built from the branch. Demo ktps8w3-dyelife_ih_anzio_h2 (POST dod\), Video mode (utvideo, 2560x1440, 300 fps), ProRes 422. Takes went to F:, clips to D:, "Delete the take permanently" with "Only when space is needed" off.

Run Setup Result
1 3 clips, plenty of room Each clip rendered while the next one recorded, at about 11.5 fps (one render at a time). All 3 rendered and their takes were deleted. Bug A: the automatic finish step then showed "Could not finish the clips… Nothing queued to render"
2 4 clips, F: cut to 62 GB free by a placeholder file (needs ~88 GB at once) Bug B: Start was greyed out for "insufficient space", though the backend would plan it. After fixing B: hold, resume, all 4 rendered and deleted. Bug C: the hold was asked for during clip 2 but only stopped the batch after clip 3, because clips 2 and 3 have no seek between them, only a fast-forward. Bug D: while it waited, the footer flipped every second between "Capturing clip 4 of 4" and "Waiting for renders…"
3 Same as 2, with A–D fixed Hold in the right place, footer steady, but clip 2 came out 1.83 s long. HLAE's FFmpeg ran out of memory mid-clip, because my test Studio (a debug build with an empty analyzer cache) was re-analysing the whole library at 26 GB. That's a test-setup problem, but nothing caught the short take: filed as #1105
4 Same as 3, with the analyzer cache copied in PASS. "Will wait after this clip for renders to free 6.1 GB" during clip 2, then held after clip 2 ("Waiting for renders to free room"). Clips 1 and 2 rendered and their takes were deleted. It went on after 284 s with 52.7 GB free and finished clips 3 and 4. All four clips are 9.99 s long, with no FFmpeg errors; free space on F: never went below 37 GB

Fixed (bb03af1)

  • A: capture_takes_verified carries rendered_while_capturing, and the finish step stays out of such a batch (finishSkipReason).
  • B: rendersMakeRoom() (holdsForRoom, the same test capture_manager.rs makes) makes "short of room" an orange note instead of a block.
  • C: the hook also holds at a sys_fast_forward line with no seek: holds_at(then, lines, hold), plus a test for it.
  • D: the hold line goes through the footer's own line (captureProgress.setRoom). When held, it replaces the capture line; before that, it's added to it.

Local: workspace tests, clippy 1.98.1 (workspace, plus goldsrc-hooks for i686), fmt, 954 JS unit tests, 85 e2e tests (take-cleanup, finish-clips, capture-precheck, capture-progress, render-studio, app-boot).

Not run:

  • PRE. The hook change is schedule logic only, with no engine offsets; the earlier hold probe passed on both installs.
  • A drive that can't fit even after deleting (the stop at the boundary).
  • "Only when space is needed" together with this setting.
  • Frame-sequence and OBS modes.

Seen, not changed:

  • "All 4 takes verified on disk. 1 highlight(s) marked Captured." counts only the highlights still unmarked: the other 3 were marked as each clip landed.
  • The held line said "free room" without a size in run 4 (run 2 showed "10.9 GB").

@ccoventry

Copy link
Copy Markdown
Owner Author

PRE: PASS (2026-10-28, same setup as run 4, Half-Life - PRE-Anniversary for Movies, pre-Anniversary DemoPlayer.dll). The hold was asked for during clip 2, and the batch held after clip 2 at the fast-forward (clock 778.2). Clips 1 and 2 rendered and their takes were deleted. It went on after 256 s, and clips 3 and 4 were recorded, rendered and deleted. All four clips are 9.99 s long, with no FFmpeg errors. The footer showed "Will wait after this clip…", then "Waiting for renders…", steadily.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Render and clean up clips while a capture batch is still running, so a long batch never fills the drive

1 participant