RFC 006: Agentic RL through Harness Interception — token-faithful traces for TRL

[rycerzes]

Summary

Adds an RFC for training the policy driving a black-box agent harness (Pi, OpenCode, Claude Code, …) inside OpenEnv by intercepting its LLM calls at an OpenAI-compatible boundary and recording a token-faithful trace — (prompt_ids, response_ids, loss_mask, response_logprobs, reward) — that TRL consumes via rollout_func.

This turns RFC 005's CLIHarnessAdapter.run_white_box stub (currently NotImplementedError) into a real seam. It is the installed-agent training path — the counterpart to the external-agent pattern TRL's Harbor integration already covers.

Design-only RFC PR — no runtime behavior changes.

What's in it

• Ownership boundary with TRL (OpenEnv owns interception + trace; TRL owns generation, weight sync, advantages, IS correction).
• 15 design decisions with rationale/trade-offs, each backed by cross-framework evidence (NVIDIA Polar/ProRL, verifiers, AReaL, Agent Lightning, rLLM) and recent literature (TIM, SAO, TITO, loss-mask/provenance work).
• Token-level trace contract, TITO discipline via TRL's chat_template_utils, prefix-break forking, session-ID-as-API-key auth, synthetic SSE replay, weight-sync fencing, proxy-enforced budgets.
• Mermaid diagrams: architecture, prefix-break forking, request lifecycle.

Relationship to other work

• Tracking issue: Agentic RL with black-box harnesses in OpenEnv via LLM-call interception #940
• Extends RFC 005 (rfcs/005-agentic-harnesses.md) — reconciles with it, does not parallel it.
• Supersedes draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 (message-level contract) with the token-level contract.

Note on numbering

Filed as 006, but 006 is also used by open PRs #794 and #624 — happy to renumber to whatever's free at merge time.

cc @burtenshaw @adithya-s-k @sergiopaniego

---

Note

Low Risk
Documentation-only RFC with no production code, config, or behavioral changes.

Overview
Introduces RFC 006 (rfcs/006-agentic-rl-harness-interception.md) as a design document only—no runtime or API changes in this PR.

The RFC specifies how OpenEnv will train policies for black-box CLI harnesses (Pi, OpenCode, Claude Code, etc.) by hosting an OpenAI-compatible interception server that fronts TRL-controlled vLLM, records generation-time (prompt_ids, response_ids, loss_mask, response_logprobs, reward), and feeds TRL via run_white_box + build_harness_rollout_func. It targets the installed-agent gap left by TRL Harbor (external-agent only) and replaces the current CLIHarnessAdapter.run_white_box NotImplementedError stub from RFC 005.

The doc lays out an OpenEnv vs TRL ownership table (interception/trace/masking vs generation, weight sync, GRPO/IS), a HarnessTrace contract aligned with Polar/verifiers, 15 decisions (TITO via TRL chat_template_utils, prefix-break forking, session bearer auth, synthetic SSE, env verify() rewards, δ diagnostics, provenance, weight-sync fencing, rollout budgets), plus architecture/request-lifecycle mermaid diagrams, a GRPO wiring example, and references. It extends RFC 005 and supersedes the message-level draft in #864.

^{Reviewed by Cursor Bugbot for commit 08c1aeb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sergiopaniego]

linking huggingface/trl#6420 for considering it as it's directly related with this one.

[sergiopaniego]

@rycerzes solid RFC. One alignment point on the TRL seam: the harness-training path is being implemented in TRL as AsyncGRPOTrainer + HarnessRolloutWorker (huggingface/trl#6420), rather than GRPOTrainer's rollout_func as described here. I validated that branch end-to-end (OpenCode agents in remote HF sandboxes, in-sandbox proxy trace, GRPO), so it's working. Just flagging so you can take it into account and align the RFC's TRL integration section with it.

[Copilot]

Pull request overview

Adds a new design RFC describing an “installed-agent” RL training path for black-box CLI harnesses by intercepting OpenAI-compatible LLM traffic and emitting token-faithful traces consumable by TRL.

Changes:

• Introduces RFC 006 describing an interception server + trace recorder architecture for token-level rollouts.
• Defines the proposed trace contract (prompt_ids, response_ids, loss_mask, response_logprobs, reward) and key design decisions (TITO, prefix-break forking, session-as-API-key, etc.).
• Documents intended integration points with existing OpenEnv harness seams (CLIHarnessAdapter.run_white_box, build_harness_rollout_func) and TRL’s rollout_func.

Alignment Review Notes (per skill)

• Automated Checks
  • Lint: Not run (no facility to execute .claude/hooks/lint.sh in this review environment)
  • Debug code: Not run (no facility to execute .claude/hooks/check-debug.sh in this review environment)
• Open RFCs context (relevant)
  • RFC 005: In Review (rfcs/005-agentic-harnesses.md) — this RFC explicitly extends it.
  • Other RFCs in Draft/In Review exist but no direct conflict identified from this doc-only change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sergiopaniego]

Following up on my earlier note about the TRL seam. Now that huggingface/trl#6420 is merged, the installed-agent path in TRL is concretely AsyncGRPOTrainer + HarnessRolloutWorker (not GRPOTrainer.rollout_func / run_white_box as the RFC describes), so it would be worth aligning the integration section on that. Two more inputs from running it end to end (OpenCode + Pi, local subprocess and remote HF sandboxes):

1. Remote / managed-sandbox interception topology. The architecture assumes the harness can reach the trainer-side interception server. For agents in remote, network-isolated sandboxes (HF Sandbox, E2B) that needs spelling out: the sandbox can egress HTTPS but has no easy inbound. The shape we validated keeps the server trainer-side/colocated and exposes it to the sandbox through an egress tunnel. The harness only gets OPENAI_BASE_URL = tunnel/rollout/<sid>/v1 plus the per-rollout bearer, and nothing runs in the sandbox. Two properties. (a) Weight-sync fencing (D12) stays a local call (an in-sandbox recorder would make it distributed). (b) Less crosses the wire: the token-faithful data (ids, logprobs) is recorded trainer-side and never reaches the sandbox, only the OpenAI-dialect request and completion text do. Only the base-URL host differs between local and remote, so it generalizes cleanly. Happy to write this up as a subsection.

2. Where the TITO buffer should live. For context on the status quo: in the merged worker path, _turns_from_trace reconstructs each turn's prompt framework-side with apply_chat_template, while OpenEnv's proxy records only raw (messages, generation-time completion_token_ids, logprobs). That reconstruction is already faithful for prefix-preserving templates (vLLM re-renders per stateless call too, so both sides land on the same ids, consistent with the 18/19 you cite). TITO's real value is turning that into a guarantee and handling prefix-breaks (compaction, sub-agents) via forking, which is needed. The thing to weigh is placement. D3 puts TITO in openenv.core via trl.chat_template_utils, which couples core to a TRL internal. Since the reconstruction already lives framework-side today and OpenEnv stays a raw recorder (any trainer can consume the same trace), keeping the token-buffer on the framework side would preserve OpenEnv's framework-agnosticism, and the contract stays HarnessTrace. The counter-argument is buffer locality (verifiers/Polar assemble at the interception point), a real trade-off, just flagging it for a conscious call.

We have validated real OpenCode/Pi harnesses that could plug into the interception server, happy to help wire them up when it exists.

For reference, the in-sandbox vs trainer-side placement in a remote-sandbox run:

In-sandbox (what the merged path does today):
  [sandbox]  agent  →  in-sandbox proxy  →(tunnel)→  vLLM  [trainer host]
             the proxy records the trace inside the sandbox, trainer reads it back afterwards

Trainer-side (this RFC):
  [sandbox]  agent  →(tunnel)→  interception server  →  vLLM  [trainer host, localhost]
             the server records the trace on the trainer host, live, and owns the token buffer