From 1a754dd26857b03c3cb81ab61e9d8520913e9f99 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Wed, 28 Oct 2026 20:14:30 +0100 Subject: [PATCH] docs: clarify accounting for unscored posttrain rollouts --- docs/source/environments/posttrain.md | 10 +++++++--- envs/posttrain_env/README.md | 10 +++++++--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/source/environments/posttrain.md b/docs/source/environments/posttrain.md index b11c1a3af..463030403 100644 --- a/docs/source/environments/posttrain.md +++ b/docs/source/environments/posttrain.md @@ -126,9 +126,13 @@ and do not return the final step observation with reward and `service_spend`. If the score job itself fails, times out or is cancelled, the episode ends unscored: `reward` is `None` and `episode_end.reason` is `score_failed`, since -the fault is the grader's. Trainers should drop such a rollout, not count it as -zero, subject to acceptance of RFC 013 (#1256) and its final unscored-rollout -semantics. Ending the episode cancels every live job. `reset` cancels the previous +the fault is the grader's. Exclude such a rollout from reward-based training +and reward estimates, rather than treating it as zero, while retaining its +episode and spend for accounting. Under the current RFC 013 (#1256) proposal, +pass its `None` reward and spend to the sampler's `update` before excluding it +from reward-based training. RFC 013 remains a proposal until accepted; this +integration is subject to its acceptance and final unscored-rollout semantics. +Ending the episode cancels every live job. `reset` cancels the previous episode's jobs first and raises if HF Jobs does not confirm a cancellation, since that job may still be running and billing. diff --git a/envs/posttrain_env/README.md b/envs/posttrain_env/README.md index 309e24c18..3dba07d93 100644 --- a/envs/posttrain_env/README.md +++ b/envs/posttrain_env/README.md @@ -125,9 +125,13 @@ and do not return the final step observation with reward and `service_spend`. If the score job itself fails, times out or is cancelled, the episode ends unscored: `reward` is `None` and `episode_end.reason` is `score_failed`, since -the fault is the grader's. Trainers should drop such a rollout, not count it as -zero, subject to acceptance of RFC 013 (#1256) and its final unscored-rollout -semantics. Ending the episode cancels every live job. `reset` cancels the previous +the fault is the grader's. Exclude such a rollout from reward-based training +and reward estimates, rather than treating it as zero, while retaining its +episode and spend for accounting. Under the current RFC 013 (#1256) proposal, +pass its `None` reward and spend to the sampler's `update` before excluding it +from reward-based training. RFC 013 remains a proposal until accepted; this +integration is subject to its acceptance and final unscored-rollout semantics. +Ending the episode cancels every live job. `reset` cancels the previous episode's jobs first and raises if HF Jobs does not confirm a cancellation, since that job may still be running and billing.