Data Agent: reproducible TiTO training and evaluation across agent loops - #7
Open
adithya-s-k wants to merge 46 commits into
Open
adithya-s-k wants to merge 46 commits into
adithya-s-k wants to merge 46 commits into
Conversation
… tasks
An agent gets a question and a directory of real tables, works in a sandbox with
its own tools, and files an answer. The agent owns its loop, so everything needed
for training has to be recovered by observing model calls rather than by making
them -- and the obvious way to recover it is wrong, silently.
Rebuilding each turn's prompt with apply_chat_template produces a DIFFERENT string
from the one the engine scored: measured on Qwen3.5-4B, the rebuilt prompt matched
the engine on 0 of 28 turns. Nothing errors. The rollout log looks healthy and the
run collapses at its first weight update, because turns that should chain by exact
token prefix instead look like unrelated short rollouts and every fragment still
trains. So nothing is tokenised locally: a capture proxy keeps the engine's own
prompt_token_ids, and turn k+1's prompt IS the tokenisation of everything before
it, by construction.
envs/blackbox-opencode -- the full environment. Flat dataset (HuggingEnvs/
data-agent), opencode in an E2B or Hugging Face sandbox, verifier and reward
server-side, turns carrying prompt_token_ids, per-token logprobs and a loss mask.
rollout.py does not just print a reward: it asserts the rollout is train tier,
that every turn carries prompt ids, and that turn k+1's prompt equals turn k's
prompt plus its completion -- the three failures that are otherwise silent.
envs/blackbox-harbor -- the same tasks as a Harbor catalog (HuggingEnvs/
data-agent-harbor-{train,test,eval}), where each task ships its own Dockerfile,
healthcheck and grader and any Harbor harness can drive it. Not an environment
package: harbor_env already serves this shape, so it is a dataset plus a CLI
recipe plus the same trainability check.
The two are not independent and it is worth knowing before editing either: every
Harbor task directory carries a tests/grader.py that is the same grader as the
flat env's -- 120 of its 130 non-comment lines are identical, because the catalog
was baked from it.
Both depend on core changes still in review upstream: OpenEnv's TraceEntry
carrying prompt_token_ids and loss_mask, CaptureServer and the sandbox backends
promoted into openenv.core, the per-session model-call budget in the proxy, and
TRL's async_grpo consuming those ids instead of re-rendering. Nothing here is
verified end to end yet and no training results are published in this project;
status is wip in project.yaml and the root index says so.
Both rollout checkers asserted that turn k+1's prompt equals turn k's prompt plus its completion, exactly. That check is wrong and would fail legitimate rollouts. Measured live against Qwen3.5-2B through the capture proxy: a harness that re-sends a `messages` list gets the engine's tokenisation of the RECONSTRUCTED history, and the Qwen3.5 template does not round-trip. It emits `<|im_start|>assistant\n<think>\n\n</think>\n\n` as the generation prompt for the turn being produced, then renders that same turn from history as `<|im_start|>assistant\nOkay<|im_end|>` -- the think block is stripped. Three turns drifted 6-8 tokens per transition and the graph reported 3 roots for 3 turns. Disabling thinking does not remove it (drift 6 instead of 8); the residue is the turn boundary itself. The ids are still faithfully the engine's own, which is the actual contract. What decides whether a turn joins the rollout or starts a new one is the SIZE of the drift, against capture's fork_threshold_tokens. Fragmentation is the failure -- one conversation becoming several short rollouts, each still training -- and it shows up as large drift, not small drift. Real agent rollouts do chain, which is the comparison that settles it: 60 steps of opencode in real sandboxes gave drift_tokens_mean 0.26, fork_frac 0.0000, realign_frac 0.009, and 8.19 turns collapsing into 1.00 sample per rollout, with samples_per_rollout never once equal to turns_mean. The synthetic driver was the unrepresentative part, not the contract. So both checkers now report mean and max drift and fail only past the fork threshold, with the measured numbers written down next to the check.
… already knows Two changes, both from running the thing against a live engine over a real tunnel. PUBLISHING. The proxy being healthy on loopback proves nothing about whether a sandbox can reach it -- the sandbox is on another machine. The env now starts one of OpenEnv's own forwarders (DATA_AGENT_CAPTURE_EXPOSE / EXPOSE=gradio in serve.sh), once at startup rather than per rollout, and agent_base_url returns whatever that resolved to. An explicit CAPTURE_PUBLIC_URL still wins, so a deployment that is already published does not get a second tunnel stood up in front of it. Default stays `direct`, which is correct only for a sandbox on this host and wrong silently anywhere else -- opencode simply cannot reach the engine, makes zero model calls, and the rollout returns a flat zero that reads exactly like a policy that cannot do the task. gradio rather than cloudflare where a tunnel is wanted: cloudflared wedged for 32 minutes on this cluster, and a forwarder that hangs is worse than one that fails, because rollouts queue behind it looking healthy. shutdown() stops the forwarder BEFORE the server, or a live tunnel is left pointing at a closed port -- which from outside is indistinguishable from a healthy service. FINDINGS. fetch_turns now returns capture's findings alongside the turns, and the rollout carries them in metadata. This replaces a check I had written that was wrong: asserting turn k+1's prompt equals turn k's prompt plus its completion, byte for byte. Capture already answers that question properly and says so -- [WARN] per_turn_capture_only: every turn is its own root (2). This harness re-renders its prompt rather than appending, so rows are single-turn. Tokens and logprobs are exact; multi-turn credit assignment is not available. -- which is a structural fact about the graph, not an inference from token counts. Deriving it from drift instead gets it wrong: a messages-reconstructing harness against Qwen3.5 drifts 6-8 tokens per transition because the template emits <think>\n\n</think>\n\n for the current turn and strips it from history, and that drift is small, legitimate and indistinguishable by eye from real fragmentation. Verified end to end over a gradio tunnel against a live vLLM: public URL minted, an outside caller drove the proxy as a sandboxed agent would, the model-call budget cut it off at 2 of 3, both turns came back with engine prompt ids and aligned logprobs, and the session released cleanly.
base.py, e2b.py and hf.py are copies of OpenEnv's envs/opencode_env/sandbox/ again, and OpenEnv keeps its own where it was. An earlier pass promoted them into openenv.core and imported them from there; that is reverted. Two reasons it is the right way round here. `openenv-opencode-env` is not published to PyPI -- only `openenv` is -- so `opencode_env.sandbox` is not reachable from an installed environment at all, and importing it would mean depending on an OpenEnv source checkout. And an environment in this repo has to stand on its own: copy the directory, uv sync, get a working env, without also cloning OpenEnv. The cost is a copy that can drift. That is a snapshot, not a dependency, and the module docstring says so. The three files have no upward imports so they lift cleanly, and sandbox_home stays the single place that knows E2B runs the agent as `user` (/home/user) while Hugging Face sandboxes run as root (/root) -- a wrong home means opencode writes its provider config where it cannot read it back, starts with no model configured, and makes zero model calls, which arrives as a flat-zero reward indistinguishable from a policy that cannot do the task. This also shrinks what the env needs from upstream to one thing: the capture stack. The sandbox promotion was never required by this work -- it was a drive-by fix for a TODO pi_env had written down -- and bundling it made both changes harder to review.
…ndboxes
Every fix here came from running it rather than reading it. The env served the
Task API fine and could not complete a single rollout.
WRONG ROUTES. The client hit /splits; the Task API is registered under the
environment name (/data_agent_env/splits). Copied from HarborEnv without checking
it applied. A 404 {"detail":"Not Found"} reads as a dead server, not a wrong path.
MCP ENVELOPE NOT UNWRAPPED. Tool results arrive as {"content":[{"text": "..."}]}
and FastMCP wraps a non-object return in {"result": ...}, so the payload is a JSON
string two layers down. Returning the dict as-is handed callers the envelope, and
capabilities()["sandboxes"] was simply absent -- which reads as "no sandbox is
usable here" rather than as a parsing bug.
DOUBLE-TIERED SPLITS. list_splits appended tiers to an already-tiered configured
split, so `train:medium` generated `train:medium:easy` and friends; three error
entries buried the one real split.
difficulty_level IS AN INT (1/2/3), not the word form -- that is difficulty_tier.
Declaring it `str` made every get_task 500.
THE ROLLOUT PATH USED THE PUBLIC TASK. provider.get_task withholds the gold answer
on purpose, because a task spec travels to whoever asks, including the agent's side
of the wire. So the rollout could never grade. The withholding worked exactly as
designed and caught this; the fix is to take the full task from task_at().
E2B TAKES A TEMPLATE, NOT AN IMAGE, and its create() has no setup hook. Sizing is
baked at template build time. Staging is a separate exec, which is also what
opencode_env does, and it is retried once: 5 of 6 hard failures across 13,200
trials were a 60 s exec-stream timeout.
THE BUCKET IS A BUCKET. hf_bucket names hf://buckets/..., read with
list_bucket_tree / download_bucket_files. snapshot_download(repo_type="dataset")
404s in a way that looks like a permissions problem. Ported the working version
verbatim, including flattening at download time and distinct exit codes for
"nothing at prefix" versus "directory empty".
exec() TAKES timeout, create() TAKES timeout_s. They differ.
OPENCODE IS NOT IN THE HF IMAGE. The E2B template bakes it in; the HF image
carries the data stack only, so it is installed at runtime. Without that the agent
makes zero model calls and returns an empty answer that grades identically to a
model that could not do the task.
AND THE ONE THAT HID THE REST: `int(getattr(r, "exit_code", 1) or 1)`. `0 or 1` is
1, so every SUCCESSFUL command read as a failure. "Is opencode installed?" was a
permanent no, every rollout reinstalled it, and the installer exits non-zero on
"already installed" -- so a working sandbox failed with a message saying the thing
it needed was already there. Replaced by _exit_code(), which distinguishes a
missing code from a zero one.
Also: a rollout with ZERO model calls now returns UNGRADED rather than 0.0. That
is infrastructure failing, not the policy being wrong, and a zero would be counted
in the group baseline.
Verified against a live Qwen3.5-2B over a gradio tunnel, same task on both
backends:
e2b 8 trainable turns, 1029 completion tokens, prefixes chain exactly
hf 6 trainable turns, 1623 completion tokens, prefixes chain exactly
Both rollout_type=train, both graded (reward 0.0 -- the 2B wrote the literal
placeholder `<value>` from the instruction's shell example instead of the computed
value, so the grade is correct and the pipeline is not at fault), and exactly ONE
budget stop per rollout rather than 281.
stop_all.sh matches on argv[0] being a python interpreter, never on a command-line
substring: a substring match also hits every bash wrapper quoting the same command,
including the agent harness's own shells.
… rest
Three things stood between the Harbor path and a completed rollout. All were
found by running it; the CLI's no-server path (`openenv harbor rollout`) named
each one in a single line, which is exactly what it exists for.
rollout.py imported `harbor_env`, which lives in OpenEnv's envs/ tree and is not
published to PyPI -- unreachable from an installed environment, the same reason
this project vendors its sandbox backends. The client is `openenv.harbor.client`,
which harbor_env only re-exports anyway.
HF_TOKEN was absent. Harbor's task containers read it from the HOST environment
for the pull_bucket.py healthcheck, and refuse the rollout outright without it.
It lives in three different places here -- HF_TOKEN, experiments/.env's HF_API_KEY,
and huggingface_hub's cached login -- so logs/hf_token.sh tries all three and
prints `${HF_TOKEN:+present}` and nothing else. That form ALONE: combining it with
`${VAR:-...}` prints the value in exactly the case being tested for.
And the one that is not ours to fix here: every task in the suite declares
cpus = 1 and memory_mb = 1024, Harbor passes both into AsyncTemplate.build, and
_resource_value takes them from task.toml with no environment override. Task 0's
bucket is 0.31 GB and pandas wants 3-5x a file's size resident, so the sandbox is
OOM-killed. Measured, same task, same model:
cpus=1 memory_mb=1024 FAILED exit 137 after 517 s, capture 0/1 usable
cpus=2 memory_mb=4096 ok after 158 s, capture 1/1 usable, 11 turns, graded
Exit 137 is SIGKILL. The expensive part is that an OOM-killed rollout files no
answer and therefore scores identically to a model that could not do the task, so
a whole training run reads as a policy that never learns with nothing in the
reward to say otherwise. The README now carries the numbers and says the fix
belongs in the dataset -- patching the download cache survives exactly one run.
…b arm Reproducing the +0.2028 run needs the dataset it actually trained on, and two pieces of that were missing. THE CURRICULUM was not ported. The run used `warmup:125` -- 125 easy prompts first, then medium with hard sprinkled through -- and without it a run sees a shuffled mix from step 0. That is not cosmetic: a group whose num_generations rollouts ALL score zero contributes exactly zero gradient, and on the harder tiers early in training that is the likely outcome. It lives in the env rather than the trainer because order is a property of the task provider, and it is applied in prompt_rows() where the tiers are still known. PROMPT DEDUPLICATION, which is a correctness fix. The trainer forwards only the prompt, so create() resolves an instruction back to ONE task index; when two tasks share an instruction, every rollout for it is graded against whichever gold that lookup returns. Measured on `train`: 5000 tasks carry 4940 distinct instructions, 48 instructions are shared by 108 tasks, and 15 of those have CONFLICTING golds -- so a slice of the run was being scored against the answer to a different question, silently. Keeping the first occurrence also restores the exact pool the published run used: 4940 prompts, and under warmup:125 a 3678-prompt epoch of 125 easy + 2831 medium + 722 hard, which matches its tier counts exactly. Verified by resolving each prompt the way create() resolves it, not by a text-to-tier dict -- that shortcut reported one medium in the easy head, because colliding instructions make a raw-string lookup return the wrong task. The same class of mistake once degraded this whole curriculum to a plain shuffle while reporting success. train/ carries the launcher: vLLM with the capture flags on GPU 0, the env server and its published capture proxy on cpu, the trainer on GPU 1. The engine is the TRAINER's own vLLM, which is what keeps the rollouts on-policy -- the agent calls the same weights the optimizer is updating, through the proxy. Sandboxes reach CAPTURE, never vLLM, and capture is published with a gradio tunnel because E2B runs off-cluster; `direct` works only for a sandbox on this host and fails silently. Two interpreters, deliberately: .venv312 for torch/vLLM/TRL, the env's own light venv for the server. The trainer reaches the local `openenv` and `data_agent_env` through PYTHONPATH rather than an ad hoc pip install, since install.sh owns what is in .venv312. `data_agent_env` needs the _pypath symlink because the directory is named `blackbox-opencode`, which is not a legal Python identifier. And the trainer must not run from the repo root, where the `trl/` submodule DIRECTORY shadows the `trl` package and the import dies with "cannot import name '__version__' from trl (unknown location)". Also: the env warms its capture proxy at boot when an engine is configured, so binding the port and minting the tunnel happen once at startup instead of on the first rollout, where num_generations rollouts would queue behind whichever one holds the lock and a failed tunnel would read as a rollout timeout.
…started
The instruction reached opencode as `json.dumps(instruction)` -- a DOUBLE-quoted
shell word. These instructions end with a backticked example:
Write only that value to /workdir/answer.txt (e.g. `echo -n "<value>" > /workdir/answer.txt`)
Inside double quotes the shell runs backticks as COMMAND SUBSTITUTION. So before
opencode was even invoked, the shell executed that example and wrote the literal
string `<value>` into answer.txt -- and, because substitution replaces the text,
deleted the example from the instruction the agent received, which arrived as
"(e.g. ), then stop."
Measured on a full 144-task k=4 eval of base Qwen3.5-2B: 498 of 575 graded
rollouts (86.6%) filed the literal `<value>`, answer_source was `file` for 569 of
them, and pass@1 came out 0.0316. The same model under the old harness, which
passed the instruction through a file, scored 0.104 with ZERO placeholder answers
in 492 rollouts. The gap was entirely this.
It reads as a model that cannot follow instructions, which is the expensive kind
of wrong: the reward is plausible, the agent really does run tools (7.85 calls,
7.65 turns), a file really is written, and the grader really does score it. Every
layer reports success.
json.dumps was also escaping newlines to a literal two-character \n, so a
multi-paragraph instruction arrived as a single line of backslash-n.
Now written to task.md and passed as "$(cat .../task.md)", which is what the
working eval harness did. Verified directly: no answer.txt is created by the
shell, the backticked example survives in the text, and real newlines survive.
The 0.0316 baseline is void and the eval is being re-run.
Read out of experiments/temp_asyncgrpo_code's own runbooks, each backed there by the job number it cost. All three are real AsyncGRPOConfig fields and all three were wrong by default in this script. token_budget defaults to None, which makes it track the vLLM server's max_model_len -- 131072 here. That tripled the trained row and killed job 69906 with torch.OutOfMemoryError in fla/ops/gated_delta_rule/chunk.py BEFORE step 1. Pinned to 40960, which is not a safety margin: the measured rows already reach 40,870, i.e. 99.8% of it. heartbeat_stale_after_s defaults to 300 while agent_timeout_s is 600. Job 69319 died with "heartbeat stale: 302s > 300s; child is hung" on a worker that was not hung but BUSY -- the worker ticks its heartbeat at the top of the dispatch loop, which does not re-iterate while every max_inflight slot is full. 900. max_completion_length defaults to 2048; opencode asks for 32,000 and capture clamps to 8192. 16384. Also optim=paged_adamw_8bit and per_device_train_batch_size=4 from the same profile, and max_tokens on the worker so the rollout side agrees with the trainer. And the agent step cap goes 10 -> 25. The reward penalises tool calls beyond step_budget=30 and the cap must sit AT OR BELOW that: above it there is a band where the agent is allowed to act and punished for acting, and the policy escapes by not acting at all -- which under train_turn_fn=has_tool_call produces no rows, hence no gradient, and cannot recover. At 10 it was not a safety setting but a handicap: 197 of 224 eval rollouts (88%) were cut off mid-task, turns pinned at exactly 9. Verified by building the exact config and asserting each field rather than trusting the source.
… does not say so
Smoke job 76488 died at on_train_begin with
404 Client Error: Not Found for url: .../server_info?config_format=json
after vLLM was healthy, after the env server came up, after 3678 tasks loaded --
five minutes in, with everything upstream reporting success.
TRL's own vllm_client docstring names the cause: /server_info "sits behind
VLLM_SERVER_DEV_MODE=1, which this trainer already requires: /pause and
/init_weight_transfer_engine are gated by the same flag." The working launcher
sets it in four places and TRL's docs set it in every example. This launcher set
it nowhere.
Two more from the same source, both fatal later rather than sooner:
VLLM_USE_FLASHINFER_SAMPLER=0, because flashinfer JIT-compiles its sampling kernel
and needs nvcc; and --weight-transfer-config {"backend":"nccl"}, without which
there is no transfer engine for the trainer to initialise.
Confirmed empirically first: the standalone engine, launched by the working
serve script WITHOUT dev mode, also 404s on /server_info. So it is the environment
variable and not `vllm serve` versus `python -m ...`.
The launcher now checks /server_info itself right after /health and exits with a
named reason. /health passing says nothing about whether dev mode took effect, and
without this check the answer arrives five minutes later in a stack trace.
…austed threads Smoke 76494 got past weight transfer and then failed with two errors, one of which is the other's consequence: TypeError: DataAgentEnv.run_rollout() got an unexpected keyword argument 'timeout_s' RuntimeError: can't start new thread LoopOwningSession.wait_for_completion(timeout_s) passes its deadline straight through to run_rollout. The MCPToolClient rewrite of client.py dropped that parameter, so EVERY training rollout raised TypeError -- and because the trainer retries, each retry built another client with another thread pool, until the process could not start a thread at all. The thread exhaustion is what you see first and it looks like a resource limit. It is not: it is one missing keyword, multiplied by the retry loop. timeout_s is now accepted and threaded into the MCP step as its deadline. It is distinct from agent_timeout_s: one bounds this CALL from the client side, the other bounds the agent inside the sandbox. Checked statically against the caller rather than by eye -- parsing both files and comparing the keywords harness.py passes (plus the factory's rollout_kwargs) against what run_rollout accepts. They now agree with nothing left over.
…atio is biased AsyncGRPOConfig defaults dtype="float32" deliberately -- TRL prefers fp32 on the trainer because the training-inference mismatch is sensitive to it -- but its own docstring adds that closing that gap end to end "also requires serving the vLLM server in the same dtype", and a precision GAP BIASES THE IMPORTANCE RATIO (https://huggingface.co/papers/2510.26788). This is a correctness knob, not a performance one. TRL's preferred direction, serving fp32, is not available on this architecture, and that was measured rather than assumed: Qwen3.5 is hybrid Gated-DeltaNet and vLLM asserts ChunkGatedDeltaRuleFunction does not support float32. Please use bfloat16. (qwen_gdn_linear_attn.py:1165, job 72978) So the match is made on the trainer's side: dtype=bfloat16. Left at the default, job 72939 warned "serves in bfloat16 but the weights sent to it are float32" with embed_tokens.weight at 2.54 GB against a 1 GB transfer buffer -- and smoke 76494 reproduced that warning here. Halving the optimizer state is a side benefit on a card this work has already OOMed. This was the LAST of the nine config knobs the working launcher patches in. Listed them all out and compared rather than discovering them one crash at a time: heartbeat_stale_after_s, token_budget, run_name, dtype, num_generations, save_strategy, save_steps, save_total_limit, output_dir. Only dtype was missing.
eval_watcher.py polls experiments/temp_asyncgrpo_code/logs/ckpt-<train_job>/ for checkpoint-* . This run writes to the environment's own logs/job-<id>/run/, so the watcher would never have seen a checkpoint -- and the 400-step job would have finished with nothing evaluated. Silently, because "no checkpoints yet" and "watching the wrong directory" look identical to a poller. A symlink rather than writing there directly: the run's artifacts belong with the environment, and reusing the eval stack that produced the +0.2028 reference keeps the comparison apples-to-apples instead of introducing a second evaluator whose differences would be indistinguishable from the model's.
The 576-rollout baseline came back 44.6% EXCLUDED -- 257 of 576, dominated by ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout, with 1,625 server-side WebSocket errors. It is progressive rather than a constant rate: 1%, 3%, 35%, 89%, 40%, 100% across the run. One MCP client per rollout and a single-process uvicorn behind them. Longer rollouts hold their sessions longer, so the same 64 concurrent clients overlap far more at 16 turns than at 7 -- and the same concurrency that excluded 1 rollout in 576 at step-limit 10 excluded 257 at step-limit 25. A concurrency that was fine yesterday is not fine after any change that lengthens a rollout. Training is NOT affected and that was checked rather than assumed: the smoke runs at max_inflight 16 and has 0 WebSocket errors across ~450 rollouts, with reward, reward_std and turns flat between its first and last steps. So the 400-step run goes at 16, not the 32 the plan suggested -- that figure was measured on HF sandboxes at 8 prompts, and this is the constraint that actually binds here. The environment-validation gate already passed on clean data before the degradation: pass@1 0.1058 against the 0.104 reference at n=75 with 0 exclusions. The checkpoint evals run through eval_data_agent.py, which talks to E2B directly rather than through this server, so the paired comparison is unaffected. The 44.6% run is kept on disk as the record of the finding, not as a baseline. Its printed score -- pass@1 0.0638 -- looked entirely plausible beside 0.104, which is the reason exclusions are reported before the score.
… engine Job 76501 died at on_train_begin with NCCL error: unhandled cuda error RuntimeError: Failed to set up the NCCL weight-transfer group after every preflight check passed -- including the /server_info gate added an hour earlier specifically to catch a bad engine. It passed because it was talking to the WRONG ENGINE. The launcher used fixed ports (18100/18200/18300), the smoke 76499 and the 400-step run 76501 both landed on ip-10-53-93-25, and the second job's probes answered 200 from the FIRST job's vLLM. Reachability is not identity. The NCCL group then tried to attach to another job's engine and that is the error that finally surfaced. The failure mode had a worse sibling waiting: the two runs used the same model, so had the transfer succeeded it would have trained one job against the other's weights and reported perfect health throughout. The reference launcher derives ports per job for exactly this reason and says so: "two jobs on one node share localhost, so the default let job 71449's capture server attach to job 71450's ENGINE." Same scheme here -- (SLURM_JOB_ID % 200) * 4, stable per job and clear of the ephemeral range. The /server_info gate now also prints which model the engine on that port is actually serving, so "reachable" and "ours" are separate claims in the log.
… no nudges
Read the sweep's own launch line rather than inferring from the reward's step_budget:
STEP_PERSIST_CAP=0 STEP_SOFT_CAP=0 STEP_HARD_CAP=17 STEP_FORCE_TOOL=
Two things follow. The +0.2028 run used NO prompt nudges, so keeping them out of
core capture and out of this environment matches the reference rather than
departing from it. And its hard cap was 17, where the capture log shows it firing
4,627 times -- it bound constantly, not a ceiling sitting unused.
25 came from the reward's step_budget of 30, which is the right CONSTRAINT but not
the reference's VALUE. The difference shows up in dynamics, not as an error.
Measured over the first 40 steps:
reference 76353 samples/step 16.1 groups/step 2.2 turns 5.98
this stack @25 samples/step 9.8 groups/step 1.29 turns 10.81
Longer rollouts mean fewer complete per optimizer step, so the effective batch is
about half and the curriculum is consumed at a different rate -- at the same step
index the reference was still in its easy warmup (reward 0.73) while this run was
well past it (0.046). Comparing the two curves would have compared two schedules.
Caught at step 24 of 400, the cheapest point available: after that it would have
been a 400-step run answering a slightly different question, and the discrepancy
would have surfaced only as an unexplained gap against +0.2028.
…al per step At max_inflight 16 this run measured samples/step 7.2 against the reference's 16.1 over the same steady-state window -- less than half the training signal per optimizer step, which across 400 steps is a different experiment rather than a slower one. groups/step was 1.67 against 2.2. 16 was chosen after an eval collapsed at 64 concurrent MCP clients with ConnectionClosedError keepalive timeouts. But that eval ran ~16-turn rollouts, so 64 x 16 is roughly 1024 concurrent turn-slots; training runs ~8.5 turns, so 32 x 8.5 is roughly 272 -- about a quarter of the load that broke it. The comparison that mattered was turn-slots, not client count. It is also a schedule fix: 35 steps took ~26 minutes at 16, which puts 400 steps past 05:30 before any checkpoint eval starts.
…ained trace
The 400-step run collapsed -- reward 0.46 -> 0.00, entropy 0.195 -> 0.708, then reward_std 0 and
grad_norm 0, which is absorbing. The environment was not at fault: step-0 reward was HIGHER than the
reference's (0.459 vs 0.323) and the eval gate reproduced the reference within noise (pass@1 0.1058
vs 0.104, 0 exclusions).
The trainer was. `train_blackbox_opencode.py` passed no `agent_turn_fn`, on a comment asserting that
capture drops auxiliary calls structurally. It does not -- it DETECTS them and warns. Across the 576
recorded eval rollouts capture emitted, on 136 of them (24%):
[WARN] multiple_roots: 2 roots across 25 turns. Each root is a separate conversation
(subagent, aux call, or ...) -- 106 rollouts
[WARN] multiple_roots: 3 roots across 25 turns. -- 30 rollouts
and TRL's default `agent_turn_fn` (`_default_agent_entries`) keeps every one. The reference run
reaches `agent_turn_fn=opencode_agent_turns` via `data_agent_capture.py` -> `base.main()`
(opencode_hf_sandbox.py:398); ours kept opencode's conversation-title generator and context
summarizer.
Measured against reference run 76353 at matched steps:
ours 76557 reference 76353
rollout/fork_frac 0.021 .. 0.061 0 at every step
rollout/drift_tokens_max 5072, 32070, 32770 94, 26, 0, 0, 0
rollout/samples_per_rollout 1.04 .. 1.31 exactly 1.0
sample/forwarded_tokens 1.5e4 .. 2.3e4 7.7e3 .. 1.5e4
grad_norm 5.9 .. 9.2 3.4 .. 4.5
tools/failure_frequency 0.107 at step 1 0.022 at step 1
Each aux call carries its own system prompt, so it does not extend the previous turn: the prefix
chain breaks, one rollout stops being one sample, context doubles, and grad_norm doubles at an
unchanged 3e-6. Worst, those tokens are TRAINED, carrying the data-analysis task's advantage -- the
policy was being optimised to write conversation titles. The title call also fires LAST, so
`entries[-1]` was an aux call and `tools/failure_frequency` was reading the wrong conversation
entirely, which is why that metric looked like an environment regression.
`fork_threshold_tokens` was checked and needs no change: the reference passes 1024 explicitly and
that is already the TRL default.
…a `disabled_tools` list
`disabled_tools` is not a key in opencode's config schema. We were writing
"disabled_tools": ["webfetch", "question", "task"]
into opencode.json, which opencode ignores -- so all three tools were ENABLED while the config read
as if they were off. The reference never had this: `_build_tools_block`
(OpenEnv/envs/opencode_env/opencode_runtime.py:151) translates the same Python-side list into
`{"webfetch": false, "question": false, "task": false}`, which is the shape opencode reads.
`task` is the one that mattered. It spawns subagents, and a subagent is a separate conversation --
the same class of prefix-chain break as the aux calls fixed in the previous commit, and capture's own
warning names it FIRST:
[WARN] multiple_roots: 2 roots across 25 turns. Each root is a separate conversation
(subagent, aux call, or ...)
`webfetch` also gives a sandbox with no egress a tool that can only fail, and every failure is error
text in a tool result -- which is counted by `tools/failure_frequency` and fed to the policy.
Verified by evaluating the settings literal out of the real source with ast rather than by re-deriving
it: `tools` is the map, no `disabled_tools` key survives.
Also brought the provider block up to the reference's shape while here: `npm`
(@ai-sdk/openai-compatible) and `name`, plus `options.timeout = 600_000`. The reference sets that
timeout explicitly (OpenCodeConfig.request_timeout_ms) and we set none. Qwen3.5 is hybrid
linear-attention with no prefix caching, so every turn reprocesses the whole conversation and late
turns are slow; an undeclared client timeout converts slow into a failed tool result.
…able
The engine was sampling at temperature 1.0, untruncated, while the trainer divided logits by 0.8 to
recompute logprobs for the importance ratio. The gradient was therefore computed against a
distribution that never produced the samples.
Why it was silent: opencode sends NO sampling parameters, so whatever the engine defaults to is what
actually generates the actions -- and Qwen3.5-2B ships no generation_config.json, so vLLM falls back
to its own 1.0/1.0. Nothing errors; the run just samples from the full tail. Capture does not fill
these in either, it forwards what the client sent.
Measured, unpinned (job 76577, bands of 6 steps):
reward entropy turns/mean
step ours reference ours reference ours reference
0-6 0.592 0.259 0.229 0.172 6.75 5.87
6-12 0.248 0.386 0.291 0.137 9.62 6.37
12-18 0.194 0.412 0.380 0.187 10.44 5.56
18-24 0.216 0.600 0.587 0.207 8.99 7.43
Entropy climbs monotonically and reward collapses, while the reference holds entropy flat at
0.17-0.21 and reward RISES. Sampling the untruncated tail also makes the agent ramble: conversations
reached 49,887 tokens and were dropped outright against token_budget=40960.
This is a separate fault from the two prefix-chain bugs fixed in 0c3a401 and 0bd86c9. Those were
real -- rollout/fork_frac is now 0.0000 and samples_per_rollout exactly 1.000, matching the reference
-- but they were not what collapsed the run.
The fix follows the reference's own structure (run_cluster.py:165, "SAMPLING TEMPERATURE, for the
engine AND the trainer, from one variable so they cannot drift"): TEMPERATURE and TOP_P are declared
once and feed both `--override-generation-config` and the trainer's `--temperature`. Values are the
reference sweep's: 0.8 / 0.95 / top_k -1. top_k is -1 because top_k truncation is likewise unmodelled
by the recomputation.
1,265 files under `logs/` were committed across d9a1143 and 072f36a -- 1.2 GB, including a Harbor trial tree with an opencode sqlite database and its WAL/SHM files, a full git snapshot directory, and 577 files belonging to `eval-opencode-base-VOID-shellbug`, an eval that had already been VOIDED by the shell-substitution bug and so is not evidence of anything. On-disk the same tree is 22 GB. Untracked here with `git rm --cached`, so every file stays on disk and no measurement is lost; the eval `records.jsonl` files remain readable where they were written, and they are regenerable in any case. The five helper scripts under logs/ (start_server.sh, stop_all.sh, hf_token.sh, needle.txt) stay tracked -- they are source, not output. hf_token.sh was checked: it references $HF_TOKEN and contains no literal credential. Added `04-data-agent/**/logs/` so it cannot recur. NOTE: this removes the files from the INDEX, not from history -- the blobs are still in the 20 unpushed commits on this branch. Dropping them for real needs a history rewrite, which is safe here only because the branch has never been pushed. That should happen before any push, not after: force -pushing does not delete what GitHub has already counted.
…c trainer The two existing environments here are black box: opencode or a Harbor harness owns its own loop inside a sandbox, the trainer never drives a turn, and everything trainable has to be recovered by observing model calls through a capture proxy. This is the inverse. TRL's synchronous GRPOTrainer owns the loop, each tool call is one MCP call, and TRL masks the tool-result tokens itself -- so there is no capture proxy, no tunnel, and no token-id contract to get wrong. ONE ENVIRONMENT, NOT PARALLEL SERVERS bash, the Jupyter kernel and the SETA file tools are three views of ONE sandbox and share its filesystem. That is the semantics, not a convenience: a file written by `write` has to be visible to `bash` in the next call and importable from the notebook in the one after. Splitting them across servers would mean replicating state between sandboxes, and every divergence would present as an agent that wrote a file and then could not find it -- which reads as a model failure and is not one. What SETA would force apart is its SUITE, not its tools: 1,376 tasks graded by weighted pytest inside its own Ubuntu 24.04 image, over ORS. Its tool surface is implemented here under the same names (read/write/edit/grep/glob/ls) so a task written against SETA reads the same; the suite lands later as a split or a sibling env. TOOL SELECTION VARIES THE CLASS, NOT A FLAG TRL turns every public method of the environment instance into a tool (`grpo_trainer.py`: getmembers -> reset/get_reward special-cased, everything else public is a tool). A runtime flag therefore could not shrink the surface -- the model would still be offered every tool and would call ones the server does not serve. `white_box_bash_env()` composes a class from one mixin per toolset instead, so the schema and the served tools are the same set by construction. Three descriptions of that surface have to agree -- the registry in tools.py, the client's methods, and the server's registered FastMCP tools -- and tests/test_surface_matches.py asserts all three, in both directions. Drift there is silent: the model calls a tool nobody implements, gets an error, and the run reads as a policy that cannot use tools. Carried over from the black-box work, each learned expensively: sessions live at MODULE scope because the Task API discards the environment instance per request; difficulty is part of the split NAME because a filter shifts every index and the index is the task's identity; an ungraded rollout is `None` and never 0.0; the efficiency bonus is gated on a solve and is never a penalty; a submission that is really a shell command earns nothing; and `exit_code` 0 is falsy, so it is compared against None rather than truth-tested. Verified: `validate_env_structure` passes, 10 tests pass, and the three surfaces agree for every toolset selection. NOT verified: no GRPO run has used this yet, and no rollout has touched a real sandbox -- E2B is exercised only through the code path, not yet end to end.
The surface is now `bash` (just `bash`) and `seta` (`read`/`write`/`edit`/`grep`/`glob`/`ls`), with
`("bash", "seta")` the default -- full SETA parity.
The Jupyter kernel is gone. It meant two tools could do the same job under DIFFERENT state semantics
-- kernel names persist between calls, shell state does not -- which is an easy thing for a small
model to conflate, and one more unvalidated variable in an environment that has not trained yet. It
was also unnecessary for this task family: the black-box runs on the same domain reached +0.2343 with
bash/read/edit/grep and no notebook. `Sandbox.run_code` went with it rather than being left as dead
code.
The terminator is `submit_solution`, always present rather than belonging to `seta`. Always present
because a `bash`-only agent would otherwise have no way to finish and the step cap would be its only
terminator -- and a capped episode is indistinguishable from a stuck one. Named as SETA names it so a
task written against SETA reads unchanged here, which is the thing that makes its 1,376-task suite
portable later. The alternative, `submit` for bash-only and `submit_solution` with `seta`, would make
the terminator's NAME depend on the selection, which is worse than either name alone.
Verified after the change: validate_env_structure passes, 10 tests pass, and registry == client ==
server for every selection (`None`, `all`, `bash`, `bash,seta`, `seta`), with no unreachable server
tools in either direction.
…the terminator bug it found A LIVE ROLLOUT AGAINST A REAL E2B SANDBOX FOUND A BUG NO UNIT TEST COULD `submit_solution` went through the same step-budget check as every other tool, so once the budget was exhausted the agent was told "call submit_solution with your best answer" and then forbidden from doing it. Every capped episode scored 0.0 -- indistinguishable from a policy that cannot solve the task, and nothing in the metrics would have said otherwise. The terminator is now exempt from the budget (`_invoke(_counts=False)`), verified: a capped episode submits and grades 1.1. The same rollout exercised all eight tools end to end, including the error paths -- a missing `old` in `edit`, a read of a nonexistent file, and a command exiting 3 all return usable text with `ok=False` rather than raising. Grade came back 1.09/correct at 11 tool calls, and the session was released. TRAINING SCRIPT TRL forwards the WHOLE dataset row to `reset()` as kwargs (`grpo_trainer.py`: `reset_kwargs = x; environment.reset(**reset_kwargs)`), so a row carrying `split` and `index` starts exactly that episode. That is the entire task-selection mechanism and there is no side channel. The row deliberately does NOT carry the task text: that comes from `reset()` at rollout time, so the server stays the single source of truth and the trainer never holds what the agent should discover. `reward_funcs` is empty because the environment owns its reward through `get_reward()`; adding one would introduce a second unweighted reward source and quietly change the objective. ONE GPU vLLM runs colocate, in-process on the training card, following experiments/harbor_bash/smoke.slurm. `vllm_gpu_memory_utilization=0.3` is what makes it fit -- the default 0.9 leaves nothing for the optimizer states. Ports are derived from the job id rather than fixed: fixed ports are how one job ends up talking to another job's server, which happened on the async side. The launcher gates before training: E2B key present (by NAME, never value), server healthy, and the split actually has tasks -- a healthy server with an empty split would train on nothing and report clean numbers doing it. The server is killed on any exit, because a survivor holds the port and the next job's bind failure presents as "stuck loading forever". Also switched the dependency from e2b-code-interpreter to plain e2b: with no Jupyter toolset there is no kernel to drive, so the code-interpreter variant was a heavier dependency for nothing.
…bugs the live run found
Smoke 76701: 4/4 steps, Qwen3.5-2B, one H100, vLLM colocate.
step reward std loss grad_norm tool_calls tool_fail
1 0.825 0.55 -0.2837 10.82 7 0
2 1.35 0 0 0 4 0
3 1.013 0.675 0.4427 6.839 6 0
4 0.825 0.55 0.1395 22.5 4.5 0
TRL generates, parses the tool call, invokes a client method, that crosses HTTP to the server, runs
in an E2B sandbox, and the result comes back as the next turn -- then `get_reward()` scores it and
the column appears as `rewards/WhiteBoxBashEnv/mean`. Step 2's zero gradient is a uniform-reward
group, not a stall.
None of the four bugs below were visible to `validate_env_structure`, the surface tests, or an
in-process rollout. Each needed a real training run.
1. CONCURRENT SESSIONS. `create_app` defaults to ONE. TRL builds an environment client per batch
slot, so three of four were refused. Over the WebSocket transport the refusal is SILENT -- the
server accepts, immediately closes, and the client dies on its first call with
`ConnectionClosedOK: received 1000 (OK)`, naming no cause. Two smokes died at step 0 that way.
2. TRANSPORT. Switched to HTTP `/mcp`, which reports the same condition properly as
`Server at capacity: 1/1 sessions` -- that is how (1) was finally diagnosed. HTTP fits anyway:
the episode id already travels as `session_id`, so no per-connection session is needed, and there
is no long-lived socket for a GIL-blocked event loop to fail to keep alive.
3. `SUPPORTS_CONCURRENT_SESSIONS`. Required to raise the cap, and true here for a specific reason now
written down: no per-episode state on the instance, sessions in a module registry behind a lock.
4. PATH. Qwen3.5 is hybrid Gated-DeltaNet and vLLM routes prefill through flashinfer, which
JIT-COMPILES at first generation and shells out to `ninja` and `nvcc`. Invoking
.venv312/bin/python directly rather than activating left .venv312/bin off PATH, and the run died
at step 0 with a bare `FileNotFoundError: 'ninja'` thirty frames below flashinfer. The launcher
now exports it and GATES on it, so the next person gets a sentence instead of a dig.
And one found in the first training step itself: `submit_solution(answer: str)` rejected the model's
bare `3` with `Input should be a valid string [input_value=3, input_type=int]`. Tasks end in "submit
just the number", so the model emits a number -- every numeric answer failed to submit and would have
scored 0.0, indistinguishable from a model that could not finish. The annotation IS the schema the
model is validated against, so it now accepts `str | int | float` and coerces once.
… staging and grader The tasks are now HuggingEnvs/data-agent -- 5,000 train, 250 test -- loaded through the sibling black-box environment rather than reimplemented. A generated suite was written first and was the wrong call: this example already owns a real, verified task set, and building a second one threw away the thing that makes having three environments side by side worth anything. PARITY IS THE POINT, NOT CONVENIENCE white-box and black-box now run the SAME tasks, staged the SAME way, graded by the SAME code. The only remaining difference is who owns the agent loop, which turns white-box vs black-box into a controlled comparison instead of two numbers that cannot be put beside each other. Correctness comes from `data_agent_env.grader` (exact, then numeric with percent/fraction bridging, then order-insensitive list, then math-verify, at the task's own atol/rtol); only the efficiency shaping stays local, because the black-box reward was tuned for an agent whose tool calls can only be inferred from text whereas here every call is counted exactly. THE SHARED E2B TEMPLATE IS REQUIRED, NOT PREFERRED Measured on the default E2B base: `huggingface_hub` is absent so the bucket staging cannot run, and `/workdir` is not writable by `user` so setup dies on `mkdir: Permission denied` before the agent starts. Either one hands the agent an empty input directory, and it then scores 0 for a reason indistinguishable from a wrong answer. `data-agent-opencode` carries both, and its sizing (cpu=2, mem=4096) is baked in at build time -- which is why this is a template name, not sandbox kwargs. Verified live: sandbox up in 1s, huggingface_hub 1.19.0, pandas 3.0.3, staging in 2s, real CSV present, correct answer -> 1.1, wrong answer -> 0.0. THE SUBMISSION SENTENCE IS REWRITTEN AT LOAD TIME The data-agent instruction tells the agent to write `/workdir/answer.txt`, because that is how a loop-owning agent submits. Here it submits through a TOOL, so left alone that line would spend turns writing a file nobody reads. `_reword_submission` drops it and points at `submit_solution`, and leaves instructions that never mentioned a file untouched. Also carried: each task's HF_BUCKET / BUCKET_PREFIX / HF_TOKEN now reach the SANDBOX through the process environment, never interpolated into a command string, so a token cannot land in a log or a trace. The generated suite survives behind `WHITE_BOX_BASH_TASK_SOURCE=synthetic` as an escape hatch for running without HF credentials. Two things found while building it are worth keeping: seeding train and test differently does NOT make them disjoint (4 of 30 test tasks came out byte-identical to training tasks, so it is rejection-sampled against the training signatures now), and `bc` is not installed in the sandbox -- which first showed up as three "wrong" gold answers that were in fact a wrong reference command.
… and retry the bucket staging Four configuration faults, found one at a time by relaunching, and three of them were already written down in experiments/rollout_control/harbor_trl -- a previous SYNC GRPO run on this same dataset. Reading its FAILURE_MODES.md first would have saved all of it. E2 (theirs, reproduced here): `beta=0` with LR 2e-6 collapsed BOTH Qwen3.5-2B runs -- entropy 0.4 -> 0.09 by step 40, the policy degenerating into malformed repetition and reward to 0. Nothing anchors a small policy without a KL term; the 4B tolerated it and the 2B did not. We were running beta=0 at LR 3e-6 and reproduced it exactly: `tools/call_frequency` fell to 0 and the model began emitting `<tool_call><parameter=command>` with no `<function=...>` wrapper -- their E1, malformed tool calls. Fix is theirs: KL_BETA=0.04, LR=1e-6, WARMUP_STEPS=10. The other two were ours: GROUP SIZE HAS TO MATCH THE SUCCESS RATE. GRPO needs variance WITHIN a group; a group where every rollout scores alike contributes nothing. Base pass@1 is easy 0.214, medium 0.062, hard 0.060, so P(all rollouts fail) is easy g=8 -> 0.15 but medium g=4 -> 0.77. Training on `train` at g=4 produced reward 0 and grad_norm 0 on 7 of 7 steps. Default is now `train:easy` at g=8, with the arithmetic in the launcher. THE LOGITS TENSOR, NOT THE BATCH, IS WHAT OOMs. The loss materialises `per_device_batch x seq_len x vocab` and Qwen3.5's vocab is ~152k: batch 8 at a 4096 completion budget is ~23 GB and dies in `logits / temperature`. Now bs=2 x grad_accum=4 -- same group of 8, ~5 GB. `max_completion_length` also had to rise 1024 -> 3072, because it bounds the WHOLE multi-turn completion INCLUDING tool-result tokens (they sit in `completion_ids`, masked out of the loss and out of `completions/mean_length`, but still counted). One clipped CSV read consumed a 1024 budget and the agent could never take a second turn -- `call_frequency` pinned at exactly 1.0. Tool output is now clipped to 1,200 characters, which also pushes the agent toward computing with pandas rather than reading a 9.8 MB file into its context. BUCKET STAGING NOW RETRIES. Every episode lists and downloads from `hf://buckets/AdithyaSK/jupyter-agent-kaggle-all` -- the dataset row carries only metadata and a pointer, the tables live in the bucket -- so at g=8 that is ~800 tree listings over a 100-step run against one bucket. A single transient `504 Gateway Timeout` killed a run at step 6/100. Four attempts with linear backoff, in a subshell so the staging script's own `set -e` cannot abort the loop, and a loud failure at the end: an agent started against an empty input directory scores 0 for a reason indistinguishable from a wrong answer. Verified live: stages in 2s, and a re-run short-circuits in 1s because the script skips a populated input directory.
The existing harness drives `opencode` or `harbor`, both of which own their own agent loop, and it scores one model at a time. This environment has no agent -- TRL owns the loop during training -- so an evaluation has to supply one. The loop here is deliberately the same shape TRL uses (generate, parse tool calls, execute, append, repeat) because measuring the policy under different conditions than it trained under produces a number that is not comparable to anything. Paired on the common item set: base and checkpoint run the same task indices with the same seed, and only tasks where BOTH produced a grade are counted. Both models are served CONCURRENTLY on one GPU so the two arms see the same sandbox and the same hardware -- running arms hours apart lets a change in E2B health masquerade as a change in policy, which is exactly how two q3i evals reported a "collapse" that was 976 sandbox 404s. `--expect-base` is the guard for that: base is a fixed reference, so if it lands far from its known value the run measured infrastructure and the script fails loudly instead of publishing a plausible wrong number. Note the exclusion count does NOT catch this -- the broken q3i evals excluded FEWER rollouts (65-69) than the valid ones (135-136), because a starved sandbox returns an empty answer that gets scored zero rather than excluded. `--max-num-seqs` is required when two engines share a card: Qwen3.5 is hybrid Gated-DeltaNet, so every concurrent decode sequence needs its own Mamba cache block. vLLM defaults to 1024 and refuses to start with `max_num_seqs (1024) exceeds available Mamba cache blocks (417)` -- and confusingly only the SECOND engine dies, which reads as a bad checkpoint rather than a memory setting.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
`.gradio/certificate.pem` is the CA bundle gradio writes on first tunnel use -- generated per machine, not source, and it had been committed for both black-box environments. Untracked and `.gradio/` added to .gitignore so it cannot come back. Not worth a history rewrite: it is ~270 KB, unlike the 877 MB of trial trees and eval traces that were stripped from this branch's history before it was first pushed.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c6811fcf8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+53
to
+54
| packages = ["data_agent_env", "data_agent_env.server"] | ||
| package-dir = { "data_agent_env" = ".", "data_agent_env.server" = "server" } |
There was a problem hiding this comment.
Add data_agent_env.sandbox to the packaged modules (or switch to package discovery). The top-level data_agent_env import loads config.py, which imports .sandbox, while this explicit package list installs only the root and server; consequently the documented uv sync/pip install path produces a package that fails immediately with ModuleNotFoundError: data_agent_env.sandbox, before either the client or server can be used.
Useful? React with 👍 / 👎.
| CMD /app/.venv/bin/python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1 | ||
|
|
||
| EXPOSE 8000 | ||
| CMD ["sh", "-c", "cd /app/env && exec /app/.venv/bin/python -m uvicorn server.app:app --host 0.0.0.0 --port 8000"] |
There was a problem hiding this comment.
Change this target to data_agent_env.server.app:app. Launching server.app from /app/env makes server a top-level package, but server/app.py immediately executes from ..sandbox import describe; that relative import is then beyond the top-level package, so the committed container exits during import rather than starting the environment.
Useful? React with 👍 / 👎.
Comment on lines
+14
to
+15
| RUN --mount=type=cache,target=/root/.cache/uv \ | ||
| if [ -f uv.lock ]; then uv sync --frozen --no-editable; else uv sync --no-editable; fi |
There was a problem hiding this comment.
Include an extra such as --extra e2b in this sync command, or make the deployment backend a required dependency. The checked uv sync --help describes --extra <EXTRA> as “Include optional dependencies from the specified extra name”; this command supplies no extra even though e2b is optional in pyproject.toml and the server defaults to the E2B backend, so after the import blockers are fixed the default image still cannot construct any rollout sandbox.
Useful? React with 👍 / 👎.
| the rollout comes back with a flat zero that reads exactly like a policy that cannot do the task. | ||
| Falling back to loopback is therefore deliberate and only correct for a sandbox on this host. | ||
| """ | ||
| return _PUBLIC_URL or f"http://127.0.0.1:{server.port}" |
There was a problem hiding this comment.
Do not fall back to loopback when the configured backends are E2B or Hugging Face sandboxes. Both run on another machine, so the default DATA_AGENT_CAPTURE_EXPOSE=direct path advertises the sandbox's own 127.0.0.1, not this capture server; the README quickstart and default serve.sh invocation therefore produce zero captured model calls unless users discover and set a tunnel or CAPTURE_PUBLIC_URL themselves.
Useful? React with 👍 / 👎.
| # beside the two black-box ones: SAME tasks, SAME staging, SAME grader, so the only difference left | ||
| # between white box and black box is who owns the agent loop. `synthetic` is the generated suite, | ||
| # useful only for exercising the plumbing without network or HF credentials. | ||
| TASK_SOURCE = os.environ.get("WHITE_BOX_BASH_TASK_SOURCE", "data-agent") |
There was a problem hiding this comment.
Make data_agent_env an installed/copied dependency or default this standalone package to its bundled synthetic tasks. Every non-demo request follows this default into dataagent.load(), which imports data_agent_env, but neither whitebox-bash's project dependencies nor its Docker image contains that sibling package; thus the documented pip install -e '.[server]' and committed container pass health checks but fail their first num_tasks or start_episode request.
Useful? React with 👍 / 👎.
Comment on lines
+215
to
+217
| if verdict.ungraded: | ||
| logger.warning("episode ungraded (%s); reporting 0.0 to the trainer", verdict.note) | ||
| self._reward = 0.0 if verdict.reward is None else verdict.reward |
There was a problem hiding this comment.
Do not coerce an infrastructure failure to 0.0 here. The server and Grade type deliberately distinguish reward=None, and eval_whitebox_bash.py excludes records only when their reward remains None; this conversion therefore counts dead-sandbox episodes as model failures in evaluation and feeds them into GRPO as negative outcomes, directly biasing both reported comparisons and training baselines.
Useful? React with 👍 / 👎.
`logs/` held five tracked source files -- start_server.sh, stop_all.sh, hf_token.sh and needle.txt -- alongside gigabytes of generated output, and after `04-data-agent/**/logs/` was added to .gitignore they were tracked files sitting inside an ignored directory. That contradiction is the kind that bites later: a fresh clone looks complete and silently lacks them, and anyone adding a file beside them finds it ignored for no visible reason. They are now in each environment's `tools/`, matching the convention the rest of this project already uses (`experiments/*/tools/`). `logs/` holds only generated output and is ignored without exception. The two references were updated with them: `blackbox-opencode/tools/start_server.sh` sources `hf_token.sh` by relative path, and `train/launch.slurm` sources it too. Both still parse. Nothing else in the repo referenced these paths. (needle.txt is not a log either: it is the needle written to a file so process matching can scan /proc/*/cmdline instead of using `pkill -f`, which matches the searching process itself.)
`tools/build_index.py` renders the root README table from each project.yaml, and CI fails when the generated block is stale. Adding `whitebox-bash` to 04-data-agent's manifest moved its env count from 2 to 3 without regenerating, so the check was red. Regenerated, not hand-edited -- the block is generated output and editing it by hand would go stale again on the next manifest change.
…arbor A replication of train_blackbox_opencode.py -- the arm that reached +0.2343 (CI [+0.178,+0.291], p=0.0) -- with ONE variable changed: the environment is Harbor through the OpenEnv capture proxy instead of the bespoke blackbox-opencode env. Every training knob is the reference's value, so a difference in outcome is attributable to the environment and not to the recipe. Verified field by field: the worker matches on 14 of 15 arguments and AsyncGRPOConfig on every training knob. ONE DELIBERATE DEVIATION: top_p 1.0, not the reference's 0.95. Under --logprobs-mode processed_logprobs the captured logprob is taken AFTER truncation, so a truncating top_p renormalises it over the kept set while the trainer recomputes full-vocab; the step-0 importance ratio then lands at kept_mass rather than 1. TRL's AsyncGRPOConfig already defaults top_p to 1.0, so 0.95 on the engine is an ENGINE/TRAINER MISMATCH rather than a policy choice. Confirmed live: ratio reads 0.9985, against the 0.985-0.993 truncation signature the reference sat at. NO agent_turn_fn, and this is measured rather than assumed. The reference passes opencode_agent_turns to strip opencode's title/summariser calls. Harbor instead assigns roles structurally -- a path that never uses tools is AUXILIARY, and to_trace_entries skips anything not trainable -- so the aux calls are dropped before TRL sees them. Live at step 12: samples_per_rollout 1.000 and fork_frac 0.0004. --agent-turn-filter tools remains as a fallback, gated on measured fork_frac. The launcher owns its harbor server, capture proxy and tunnel on ports derived from the job id, so two jobs cannot collide and nothing outlives the run. It checks the split by IDENTITY, not just reachability, and probes the engine from the server's host -- if the server cannot reach the trainer's vLLM the tier grades `text` and every rollout returns zero trainable turns while looking healthy. multi_harness.py routes each GRPO GROUP to a harness off `seed` (= group_id, constant across all num_generations), so harness is constant WITHIN a group and varies between. That matters because measured pass@4 across harnesses on this suite spans 0.320 to 0.020; mixing within a group would make the advantage encode which harness rather than which action. pair_rows pads the task list so gcd(n_rows, n_harnesses) == 1 -- at 40 tasks and 2 harnesses, 0 of 40 tasks meet both and seed routing silently collapses into a disjoint partition.
…-solved groups Run 77284 (Qwen3.5-2B, opencode, 118 logged steps) measured two problems that one reward term addresses. The known one: nothing bounded turn count. tool calls per rollout drifted 13.1 -> 43.0 between the first and last 30 steps, p90 56.6, max 125.5, turns_max 235. Packed rows grow with the SQUARE of turn count and row_tokens_max reached 40,598 against a 40,960 budget. The one that was not being counted: 19 of those 118 steps logged reward_std == 0, and ALL NINETEEN were groups where every generation SOLVED the task. Under pure correctness an all-correct group has zero advantage for every member -- 8 sandboxes, no gradient, 16% of the run. Those generations were not identical; they differed in how long they took. Efficiency makes exactly those groups trainable, so this buys signal from rollouts already paid for rather than only suppressing behaviour. reward = correctness x (1 + W_EFF * B/(B + tool_calls)) W_EFF 0.3, B 15 MULTIPLICATIVE, not additive-with-a-gate. Added, zero tool calls scores the MAXIMUM efficiency, so inaction becomes the best move for a policy that cannot solve the task (0.300 vs 0.030 for a real attempt that fails); has_tool_call then yields no trainable turns and the group empties. Jobs 72452/72473 wedged at step 7 and 10 of 100 exactly that way, spending 4,076 E2B sandboxes on 11 productive groups. A gate patches it; multiplying removes it structurally -- reward is monotone non-decreasing in BOTH arguments, so efficiency can never be traded for correctness, and that survives a refactor where a conditional may not. B/(B+n), not the reference's clip(1 - n/B). Over a distribution spanning p10 8.4 to max 125.5 a linear clamp cannot be both sensitive at 10 and unsaturated at 100: clip(1-n/15) is zero above the MEDIAN, clip(1-n/60) is zero exactly where the drift lives. The reciprocal keeps a gradient across the whole range and keeps it strongest near B. B stays the reference's 15, now meaning half credit rather than a cliff. Efficiency is a tie-breaker, not a rival objective: it moves reward by at most 0.193 against correctness's 1.0, so a group that disagrees on correctness is still decided by correctness ~5:1. Keep W_EFF well under 1. Computed trainer-side because a tool-call count needs the TRACE, not the sandbox. The suite's grader tried to read it from /workdir/.n_tool_calls and $N_TOOL_CALLS; nothing writes either, it emitted null forever, Harbor's dict[str, float|int] rejected the whole dict and took correctness down with it -- 86 of 250 tasks silently unscored. None still means UNSCORABLE and drops the rollout from its group baseline; scoring a crashed sandbox 0.0 would teach the policy it is as bad as a wrong answer. --reward correctness keeps the pure-correctness arm of 76585/77284 reproducible. harbor_reward.py is symlinked into _pypath because rollout_reward_fn is pickled into the SPAWNED child, which must import it by name; PYTHONPATH is inherited, spawn's sys.path propagation is not something to rest it on -- that fails in the child only, after the sandboxes are paid for. This is a soft incentive, NOT a bound. It will not stop one rollout running 235 turns and blowing the row. The hard bound is max_model_calls in the capture proxy; ship both.
Every other knob in this launcher is env-overridable; --reward was not, so reproducing the pure-correctness arm of 76585/77284 meant editing the script. REWARD=correctness now does it, and the default stays efficiency.
adithya-s-k
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This example trains the same data-analysis tasks through Harbor harnesses, standalone OpenCode, or native bash/SETA. It adds portable local/Slurm and HF Jobs/Spaces recipes, exact-token training checks, and the recorded checkpoint results.
04-data-agent/reproduce.py, prepares pinned sources and dependencies, deploys environments, runs smokes, evaluates and trains. No external experiments checkout is required.reproduce.mdcontains the commands.results.mdlinks checkpoint/harness/difficulty tables, a plot and compact provenance. Historical runs and superseded local material are preserved; credentials, raw answers and checkpoints stay out of Git.Validation: 149 CPU tests passed, one dataset-dependent skip, 24 subtests. All 1,250 native grading configurations and 250 original baseline answers were checked. Harbor/OpenCode, native OpenCode and SETA each completed a deployed-Space GPU smoke with four optimizer steps, exact-token audits, changed weights, native optimizer state and checkpoint-2 remote restore. Exact versions, GPU allocations and receipts are in
04-data-agent/results/validation.md. These smokes verify execution and persistence, not learning improvement.Dependencies: huggingface/OpenEnv#1036, then huggingface/trl#6947. Reproduction pins the tested revisions. Recorded curves are observational; sync/async batching and harness weighting differences are documented.