[REA-6855] Save each reported step as a folder of files

[Dere-Wah]

Why

A caller that runs a session for its output, rather than to watch it, needs each step's result kept somewhere it can collect it, including the last step of a session that closed itself at its steps. Step reports and the manifest block are in place; this is the part that keeps what they describe.

What Changed

When runtime.step_results is on, every step the model reports is saved as a folder under the session's own id, the id /clips/chunks uses:

<temp>/<session id>/<step>/
├── output.mp4      every track of the step's output, one stream each
├── prompt.txt      an extra file the step carried
└── result.json     written last

{
  "step": 1,
  "session_id": "9f1e2d3c-0000-4000-8000-00000000c0de",
  "files": [{"name": "output.mp4", "content_type": "video/mp4", "size": 5170}],
  "messages": [{"type": "progress", "data": {"percent": 50}}],
  "error": null,
  "save_error": null,
  "timings": {"generate_s": 0.012, "encode_s": 0.015}
}

Each folder is written under a hidden name, result.json last, and renamed to its step number once complete, replacing any folder a reused session id left there; a reader therefore never sees a result.json beside files still being written. messages are the messages the model broadcast since the step before; they are taken on the model's thread when the step is reported, so each lands with the step it was sent before, a late report from an earlier session leaves them to the current one, and the buffer keeps the latest 256 for a model that broadcasts without reporting steps. Each step report now carries the rate its media was emitted at, which the saved video plays at: the measured throughput for a model that does not pin its rate, or the rate of the last emission for a step reported with complete_step().

Saving never makes the model wait. A step is offered to the store when it is reported, and one that finds the queue full is not saved. step_completed says which, so saved: true always means a folder follows. A step that failed is saved as result.json alone, whatever output or files its report carried, and so is a step that produced nothing. A step whose save fails still gets its result.json, with the reason in save_error and no files, so an admitted step never leaves a gap. A step reported after the session stopped at its steps is saved too.

One worker saves steps in order and journals step_result_ready with {session_id, step, files}. The session id is in the detail because a folder can finish after its session has ended, and the next session may have started. Each folder is deleted five minutes after its result.json is written, while the session runs and after it ends, so disk use follows the step rate rather than the session's length; a folder still being written is never touched. Admission and shutdown share a lock, so a step admitted while the store closes is saved and nothing is admitted after; on shutdown, pending saves get ten seconds to finish. The session descriptor reports "step_results": {"enabled": ...} from here, now that steps are saved. Only a session id that is a UUID names a folder, so a caller-chosen id cannot point outside the root.

[github-actions]

[codex-review] - [P2] runner.py: stale step reports can discard current-session buffered messages.

• [P2] store.py: shutdown can accept a step after the writer exits, leaving no result folder.

Scope: full (1c8fe61..4a3e9e0).

View workflow run.

[Dere-Wah]

• [REA-6855] Serve saved step folders over HTTP #239
• [REA-6855] Save each reported step as a folder of files #238 👈 (View in Graphite)
• [REA-6855] Write one step's output as a single MP4 file #237
• [REA-6855] Read the runtime.step_results block from the manifest #236
• [REA-6854] Close a session when it reaches its steps #235
• [REA-6854] Report each finished step to the runtime with complete_step() #234
• [REA-6854] Apply a session's starting input, sent by a system client #233
• [REA-6854] Read starting_input and steps from the start body #232
• [REA-6854] Deliver commands and reactor events to the model in posted order #231
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[tempusfrangit]

Requesting changes for result integrity: saved video should preserve the emitted media's playout rate, and an existing completion marker must not expose files while they are being rewritten. I also reproduced the existing stale-report message loss and admission/shutdown race. Please address these cases with regression tests. The error-step payload behavior is a separate contract clarification noted inline.

[Dere-Wah]

@tempusfrangit all addressed in 2333bb0, each with a regression test: the saved video takes the emission's rate (measured throughput for unpinned models); folders are written under a hidden name and swapped in whole, so a reused session id never exposes a half-rewritten step; error steps keep only result.json; admission and close share a lock; and a stale report no longer takes the current session's messages. Details on each inline thread.

review was addressed

[Dere-Wah]

Merge activity

• Oct 5, 6:07 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Oct 5, 6:30 PM UTC: Graphite rebased this pull request as part of a merge.
• Oct 5, 6:31 PM UTC: @Dere-Wah merged this pull request with Graphite.