Data Agent: reproducible TiTO training and evaluation across agent loops

.gitignore

@@ -227,3 +227,18 @@ dist/
 .astro/
 # Slide deck exports (PDF/PPTX) — published to the Hub, not tracked here
 export/
+
+# Generated run artifacts: training checkpoints, capture traces, Harbor trial trees, eval records.
+# These reach tens of GB -- 22 GB on disk here, of which 877 MB had actually been committed and had
+# to be stripped from this branch's history before its first push. Committing generated data is the
+# fastest way to make a repo unusable, and none of it is reproducible-by-review anyway.
+#
+# `logs/` now holds ONLY generated output. The operational scripts that used to live there
+# (start_server.sh, stop_all.sh, hf_token.sh, needle.txt) moved to each environment's `tools/`,
+# because tracked source inside an ignored directory is a contradiction waiting to bite -- a fresh
+# clone would look complete and silently lack them.
+04-data-agent/**/logs/
+
+# Gradio scratch. `certificate.pem` is the CA bundle gradio writes on first tunnel use and
+# `flagged/` is its UI capture dir -- both generated per machine, neither is source.
+.gradio/

04-data-agent/.gitignore

@@ -0,0 +1,8 @@
+/temp/
+/.env
+/.venv*/
+/__pycache__/
+**/__pycache__/
+/.pytest_cache/
+.gradio/
+*.pyc

04-data-agent/README.md

@@ -0,0 +1,52 @@
+# Data Agent: compare who owns the agent loop
+
+Train and evaluate **Qwen3.5-2B** on the same fixed data-analysis tasks, using three OpenEnv implementations.
+
+| Recipe | Agent loop | Trainer | Environment |
+| --- | --- | --- | --- |
+| `harbor-multi` | OpenCode, Claude Code, Codex and Mini-SWE-Agent via Harbor | AsyncGRPO | [Blackbox Harbor Space](https://huggingface.co/spaces/HuggingEnvs/data-agent-blackbox-harbor-env) |
+| `harbor-opencode` | OpenCode via Harbor | AsyncGRPO | Same Harbor Space |
+| `native-opencode` | Original standalone OpenCode adapter | AsyncGRPO | [Blackbox OpenCode Space](https://huggingface.co/spaces/HuggingEnvs/data-agent-blackbox-opencode-env) |
+| `seta` | Model calls native bash/SETA tools through TRL | Synchronous GRPO | [SETA Whitebox Space](https://huggingface.co/spaces/HuggingEnvs/data-agent-seta-whitebox-env) |
+
+Start with **[reproduce.md](reproduce.md)** for local/Slurm and HF Jobs instructions. See **[results.md](results.md)** for measured pass@1, checkpoint curves, difficulty breakdowns and the limits of the comparison.
+
+```bash
+cd 04-data-agent
+python reproduce.py prepare --recipe harbor-opencode --env-file .env
+python reproduce.py --help
+```
+
+Training uses 1,000 fixed tasks (150 easy, 600 medium, 250 hard), eight rollouts per selected task, LR `3e-6`, checkpoint saves every 50 optimizer steps and independent evaluations every 100. The four-harness recipe assigns one harness to each task per pass. Task count, rollout count and optimizer steps are different quantities.
+
+Checkpoint evaluation is pass@1 on 250 held-out tasks: 1,000 cells through the four Harbor harnesses for either async trainer; 250 native bash/SETA cells for the sync trainer. Hub evaluation defaults to concurrency **35**. Each Space serves both training and evaluation; reserved sandbox slots protect training. Evaluation uses separate inference GPUs and never swaps the trainer's active weights.
+
+Exact engine token IDs, processed log probabilities and loss masks are retained. Prompt rewrites can produce multiple training rows; the async recipe consumes complete rollout groups before updating. This prevents partial-group admission but does not establish that different harnesses receive identical gradient weighting.
+
+## Validated environments
+
+<!-- BEGIN:matrix -->
+| Env | Tools | Backend | `openenv` |
+|---|---|---|---|
+| **blackbox-opencode** | agent-owned | `e2b / hf / daytona` | ✅ |
+| **blackbox-harbor** | agent-owned | `Harbor / Daytona / E2B` | ✅ |
+| **whitebox-bash** | bash, read, write, edit, grep, glob, ls, submit_solution | `e2b / daytona` | ✅ |
+<!-- END:matrix -->
+
+## Folder map
+
+| Path | Purpose |
+| --- | --- |
+| `reproduce.py` | Main command: prepare, upload, deploy, smoke, evaluate, train |
+| `hf/configs/` | Shared recipe, immutable model/source/task pins |
+| `hf/locks/` | Separate hash-locked training and environment dependencies |
+| `hf/runtime/` | Shared Space/Job runtime, artifact upload, eval coordinator, TiTO audits |
+| `envs/` | Standalone OpenCode and whitebox implementations; Harbor uses OpenEnv |
+| `train/` | Trainer recipes, deterministic schedule, atomic rollout batching, save/resume |
+| `eval/` | Evaluation client, first-graded ledger and scoring checks |
+| `serve/` | Validated vLLM launcher |
+| `tools/` | Shared native evaluators, capture audits and logging utilities |
+| `results/` | Committed score tables, plots and dated provenance |
+| `temp/` | Ignored bundles, local runs and preserved superseded material |
+
+The baseline cohorts and recipe history differ in infrastructure; the recorded curves are observational, not a controlled causal comparison. Do not reinterpret incomplete evaluations as scores. Credentials, private grading data, raw captures, model weights and Trackio databases stay out of Git.

04-data-agent/envs/blackbox-harbor/README.md

@@ -0,0 +1,126 @@
+# blackbox-harbor
+
+The same data-analysis tasks as [`blackbox-opencode`](../blackbox-opencode), served the other way: as
+a **Harbor task catalog** rather than a flat dataset. There is no environment package here —
+OpenEnv's `harbor_env` already serves this shape, so this is a recipe for the CLI plus a script that
+proves the rollouts come back trainable.
+
+## Which one should I use?
+
+| | [`blackbox-opencode`](../blackbox-opencode) | this (`harbor_env` + the CLI) |
+| --- | --- | --- |
+| dataset | [`HuggingEnvs/data-agent`](https://huggingface.co/datasets/HuggingEnvs/data-agent) — flat, one row per task | `HuggingEnvs/data-agent-harbor-{train,test,eval}` — one directory + `task.toml` per task |
+| task setup | the env stages tables from a bucket | the task's own Dockerfile and healthcheck |
+| agent | opencode | any of Harbor's ~30 harnesses, per rollout |
+| grading | the env's verifier | the task's own verifier |
+| to change a task | edit a dataset row | edit a task directory |
+
+Neither replaces the other. Use the flat env when the task definition is data and you want to
+iterate on it quickly; use Harbor when the task ships its own container and verifier, or when you
+want to train one policy against several agent harnesses.
+
+## Serve it
+
+```bash
+openenv harbor serve \
+  --dataset HuggingEnvs/data-agent-harbor-train \
+  --llm-url http://127.0.0.1:8455/v1 \
+  --model Qwen/Qwen3.5-2B \
+  --port 8000 --capture-port 8100 \
+  --expose gradio
+```
+
+`--expose` is how the **sandbox** reaches the capture proxy, which is not the same as how the trainer
+reaches the env server. The sandbox is on another machine, so `direct` only works when this host is
+already routable from it; `gradio` and `cloudflare` mint a public URL.
+
+> **Serve the engine with `--return-tokens-as-token-ids --logprobs-mode processed_logprobs`.**
+> Without them the engine probes as eval tier and every rollout comes back with reward and a trace
+> but nothing trainable — silently. `openenv harbor info` reports the measured tier before you start.
+
+One engine is not a requirement of the deployment. `--llm-url` sets a *default*; each rollout may name
+its own, which is what lets a training run and an evaluation run share one server. The dataset and its
+prebuilt sandbox templates are the expensive things to host; the engine is the cheap, changing part.
+
+## One rollout, no trainer
+
+```bash
+openenv harbor rollout \
+  --dataset HuggingEnvs/data-agent-harbor-train \
+  --llm-url http://127.0.0.1:8455/v1 --model Qwen/Qwen3.5-2B \
+  --harness opencode --sandbox e2b --indices 0,1,2
+```
+
+This runs the whole path — engine, capture proxy, sandbox, harness, task verifier — with **no env
+server in the way**. When something breaks, that halves the search space immediately: if `rollout`
+works and `serve` does not, the fault is in the serving layer and nothing below it.
+
+## Check the rollouts are trainable
+
+```bash
+uv run python rollout.py --server http://127.0.0.1:8000 \
+  --llm-url http://127.0.0.1:8455/v1 --model Qwen/Qwen3.5-2B
+```
+
+It asserts the three things that are silent when wrong: `rollout_type == "train"`, every turn carries
+non-empty `prompt_token_ids`, and turn *k+1*'s prompt equals turn *k*'s prompt plus its completion.
+That last one is the contract — it is what makes turns link by exact token prefix instead of by a
+re-rendered string.
+
+## Train on it
+
+```python
+from harbor_env.harness import HarborSessionFactory
+
+factory = HarborSessionFactory(
+    server_url="http://127.0.0.1:8000",
+    split="HuggingEnvs/data-agent-harbor-train",
+    llm_url=VLLM_URL,          # the same engine the trainer syncs weights into
+    model=MODEL,
+    harness="opencode",
+    sandbox="e2b",
+    agent_step_limit=10,
+)
+```
+
+`agent_step_limit` is worth setting. AsyncGRPO packs every turn of a rollout into one training row and
+each turn re-sends the whole conversation, so packed length grows with the **square** of the turn
+count — an unbounded rollout can OOM the loss step while every rollout log line looks healthy.
+
+## The suite declares too little memory, and it OOM-kills rollouts
+
+**Every task in `HuggingEnvs/data-agent-harbor-{train,test,eval}` declares `cpus = 1` and
+`memory_mb = 1024`.** Harbor passes both straight into `AsyncTemplate.build`
+(`harbor/environments/e2b.py:192`), and `_resource_value` reads them from `task.toml` with no
+environment override (`harbor/environments/base.py:315`) -- so that is the size of the sandbox the
+agent actually gets.
+
+It is not enough. Task 0's bucket is 0.31 GB and pandas wants roughly 3-5x a file's size resident, so
+1 GB is marginal at best. Measured on `0000_324_324276_qa_3` with Qwen3.5-2B:
+
+| | `cpus=1, memory_mb=1024` (as shipped) | `cpus=2, memory_mb=4096` |
+| --- | --- | --- |
+| outcome | `FAILED (NonZeroAgentExitCodeError)`, **exit 137**, 517 s | `ok`, 158 s |
+| capture | 0/1 usable | **1/1 usable** |
+| turns / trainable tokens | 17 / 3252, discarded | 11 / 231 |
+| graded | no -- the verifier never ran | yes |
+
+Exit 137 is SIGKILL, i.e. the OOM killer. The expensive part is that an OOM-killed rollout files no
+answer, which scores **identically to a model that could not do the task** -- so a training run reads
+as a policy that never learns, with nothing in the reward to say otherwise.
+
+**The fix belongs in the dataset**, not here: bump `[environment] cpus` and `memory_mb` in the task
+files. Patching the downloaded cache works for one run and is undone the moment the server refetches.
+
+## Warm the sandbox templates first
+
+E2B builds one template per `env_hash`, from the task's **own Dockerfile** — `docker_image` in the
+task config is ignored whenever a Dockerfile exists. Concurrent first use races that build and fails
+with `404 tag default`, so run a single serial rollout once before any parallel run.
+
+## Concurrency
+
+The capture proxy is a single uvicorn process and is the real ceiling: `/health` starved at ~200
+concurrent sessions and the process crashed outright at 320 (3,525 fds, 542 threads, 6.7 GB). E2B
+allows 500 sandboxes per account, so capture gives out first. Size `MAX_CONCURRENT_ENVS` well under
+that, and remember a training run and an eval run share one deployment.

04-data-agent/envs/blackbox-harbor/pyproject.toml

@@ -0,0 +1,25 @@
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# NOT AN ENVIRONMENT PACKAGE, and deliberately so. There is no server and no environment code here:
+# OpenEnv's `harbor_env` already serves this task shape, so this variant is a dataset plus a recipe.
+# All this file does is pin what `rollout.py` needs to talk to a running one.
+
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "data-agent-blackbox-harbor"
+version = "0.1.0"
+description = "Data-agent tasks as a Harbor catalog, served through OpenEnv's harbor_env"
+requires-python = ">=3.12"
+dependencies = [
+    # Brings `openenv harbor` (the CLI that serves this), `harbor_env` and the capture stack.
+    "openenv>=0.3.1",
+    # The sandbox backends Harbor will drive. Listed individually rather than via `harbor[cloud]`,
+    # which cannot be installed at all: it pulls `langsmith[sandbox]` (websockets>=15) and
+    # `tensorlake` (websockets>=13,<14) together, and uv reports the pair as unsatisfiable.
+    "harbor[e2b,modal,daytona]>=0.20.0",
+    "huggingface_hub>=1.12",
+]

04-data-agent/envs/blackbox-harbor/rollout.py

@@ -0,0 +1,156 @@
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Run one Harbor rollout and check it is actually trainable.
+
+WHY THIS SCRIPT EXISTS RATHER THAN A GLANCE AT THE LOG
+Every failure it checks for is SILENT. An engine served without `--return-tokens-as-token-ids`
+produces rollouts with a reward, a transcript and a plausible turn count that carry nothing to train
+on; a consumer that re-renders prompts produces training rows that look fine and describe a
+conversation the model never had. Both read as healthy until the loss step, and one of them cost two
+production runs a night each.
+
+    uv run python rollout.py --server http://127.0.0.1:8000 \\
+        --llm-url http://127.0.0.1:8455/v1 --model Qwen/Qwen3.5-2B
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+# `openenv.harbor.client`, not the `harbor_env` env package. That package lives in OpenEnv's
+# envs/ tree and is not published to PyPI, so it is unreachable from an installed environment --
+# the same reason this project vendors its sandbox backends rather than importing opencode_env's.
+# `harbor_env/__init__.py` only re-exports this class anyway.
+from openenv.harbor.client import HarborEnv
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--server", default="http://127.0.0.1:8000", help="a running harbor_env server")
+    p.add_argument("--llm-url", default="", help="engine for this rollout; omit for the server's")
+    p.add_argument("--model", default="", help="served model id")
+    p.add_argument("--split", default="", help="dataset; omit for the server's first")
+    p.add_argument("--index", type=int, default=0)
+    p.add_argument("--harness", default="opencode")
+    p.add_argument("--sandbox", default="e2b")
+    p.add_argument("--step-limit", type=int, default=10)
+    args = p.parse_args()
+
+    env = HarborEnv(args.server)
+    try:
+        caps = env.capabilities()
+        llm = caps.get("llm") or {}
+        print(f"server     {args.server}")
+        print(f"datasets   {caps.get('datasets')}")
+        print(f"engine     {llm.get('url') or args.llm_url}  capture_level={llm.get('capture_level')}")
+        if llm.get("capture_level") not in ("tokens", None) and not args.llm_url:
+            print(
+                "\nWARNING: the server's default engine is EVAL TIER. Rollouts will carry a reward "
+                "and a trace but nothing trainable. Serve it with --return-tokens-as-token-ids "
+                "--logprobs-mode processed_logprobs, or name a train-tier engine with --llm-url.",
+                file=sys.stderr,
+            )
+
+        print(f"\nrunning {args.harness} on {args.sandbox}, task index {args.index} ...")
+        result = env.run_rollout(
+            split=args.split,
+            task_index=args.index,
+            harness=args.harness,
+            sandbox=args.sandbox,
+            llm_url=args.llm_url,
+            model=args.model,
+            agent_step_limit=args.step_limit,
+        )
+    finally:
+        env.close()
+
+    print(
+        f"\nreward {result.reward}  rollout_type {result.rollout_type}  "
+        f"turns {result.n_turns}  roots {result.n_roots}  capture_level {result.capture_level}"
+    )
+    if result.findings:
+        print("findings:")
+        for f in result.findings[:10]:
+            print(f"  {f}")
+
+    return _check(result)
+
+
+# Above this, the capture graph forks instead of realigning; it is its own `fork_threshold_tokens`.
+DRIFT_FORK_THRESHOLD = 1024
+
+
+def _check(result) -> int:
+    """The three properties that are silent when wrong. Returns a process exit code."""
+    failures: list[str] = []
+
+    # 1. Tier. `reward=None` is an UNGRADED rollout, not a zero, so it is reported separately: a
+    #    trainer drops an ungraded rollout from the group baseline rather than treating it as a loss.
+    if result.rollout_type != "train":
+        failures.append(
+            f"rollout_type is {result.rollout_type!r}: the engine did not return token ids, so this "
+            "rollout carries nothing to train on"
+        )
+    if result.reward is None:
+        print("\nnote: reward is None -- UNGRADED, not zero. The task's verifier did not run.")
+
+    turns = [t for t in result.turns if getattr(t, "trainable", False)]
+    if not turns:
+        failures.append("no trainable turns came back")
+
+    # 2. Every turn carries the ENGINE's tokenisation. Without it a consumer has to re-render the
+    #    prompt, which matched the engine on 0 of 28 measured turns on Qwen3.5-4B.
+    missing = [i for i, t in enumerate(turns) if not getattr(t, "prompt_token_ids", None)]
+    if missing:
+        failures.append(f"turns {missing[:5]} carried no prompt_token_ids")
+
+    # 3. CHAINING, measured rather than asserted as byte equality.
+    #
+    # The tempting check is `turn k+1's prompt == turn k's prompt + completion`, exactly. It fails
+    # legitimate rollouts: a harness that re-sends a `messages` list gets the engine's tokenisation
+    # of the RECONSTRUCTED history, and Qwen3.5's template does not round-trip -- it emits
+    # `<think>\n\n</think>\n\n` for the turn being generated and strips it from history. Measured
+    # live, that drifted 6-8 tokens per transition and produced 3 graph roots for 3 turns.
+    #
+    # The size of the drift is what decides realign-versus-fork, and fork is the real failure: one
+    # conversation becoming several short rollouts, each still training. Real opencode rollouts over
+    # 60 steps: drift_tokens_mean 0.26, fork_frac 0.0000, 8.19 turns into 1.00 sample.
+    drifts = []
+    for a, b in zip(turns, turns[1:]):
+        want = list(a.prompt_token_ids) + list(a.completion_token_ids)
+        got = list(b.prompt_token_ids)[: len(want)]
+        common = next((j for j, (x, y) in enumerate(zip(want, got)) if x != y), min(len(want), len(got)))
+        drifts.append(len(want) - common)
+    if drifts:
+        print(f"chaining   drift per transition: mean {sum(drifts)/len(drifts):.2f}, max {max(drifts)}")
+        if max(drifts) > DRIFT_FORK_THRESHOLD:
+            failures.append(
+                f"a transition drifted {max(drifts)} tokens (threshold {DRIFT_FORK_THRESHOLD}): the "
+                "capture graph forks rather than realigns, fragmenting one rollout into several"
+            )
+
+    if failures:
+        print("\nFAIL")
+        for f in failures:
+            print(f"  - {f}")
+        return 1
+
+    print(f"\nOK: {len(turns)} trainable turns, engine tokenisation intact, prefixes chain.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())