diff --git a/aorlta/README.md b/aorlta/README.md index 6f3b40fea..1f33dc7cb 100644 --- a/aorlta/README.md +++ b/aorlta/README.md @@ -128,6 +128,8 @@ flowchart TB ## Status +What is verified, with the commands to reproduce it, and what is not yet shown: [VALIDATION.md](VALIDATION.md). + On stand-ins the loop runs end to end on one node. The last stand-in training run (`configs/standin.yaml`'s settings, 3 iterations of 16 prompts × 16 samples) trained every iteration, landed every weight update on all 4 engines, and passed the gap check every step. From its iteration summaries (the command is under "Run outputs"): @@ -147,8 +149,9 @@ Not there yet: - No trained model worth sharing: only 3-iteration stand-in runs. The stand-in prompts show a kernel's original disassembly, with no bug planted, and ask for a HIP source, so they exercise the loop, not the task. - The stand-in checks are cheap, so the times above say little about a run with heavier checks. -- One prompt format, the stand-in one, and one node layout are wired; [docs/ADAPTING.md](docs/ADAPTING.md) says how - to add a prompt source, a reward function or a check. +- One node layout is wired. A task of your own (prompts, answer parsing, checks and reward) is a plugin, with no + edit to the package: [docs/ADAPTING.md](docs/ADAPTING.md) has the contract, and [docs/TUTORIAL.md](docs/TUTORIAL.md) + walks through `examples/hip-kernels/`, a kernel-writing task whose reward runs the kernel and checks its output. ## Layout @@ -170,7 +173,8 @@ and `configs/dev-standin.yaml` (one trainer rank and the CPU checks of `standin- compile and waitcheck failures, so a candidate that passes them stays unscored); `jobs/aorlta-run.sbatch` (a run), `jobs/aorlta-reward-standin.sbatch` (a test of the stand-in GPU checks) and two scripts the run job calls; `docker/` the trainer image; `tests/` the CPU suite; `tools/merge_base_adapter.py`, which merges a PEFT adapter into its base -checkpoint; `examples/standin-pool/`, a synthetic prompt pool; `docs/ADAPTING.md`. Code comments cite sections and +checkpoint; `examples/standin-pool/`, a synthetic prompt pool; `examples/hip-kernels/` with `configs/hipk-score.yaml` +and `configs/hipk-train.yaml`, the worked example of a task as a plugin; `docs/ADAPTING.md` and `docs/TUTORIAL.md`. Code comments cite sections and labels (Section 6.x, C1-C9, D7, ...) of our internal design notes, which are not included. ## Requirements diff --git a/aorlta/VALIDATION.md b/aorlta/VALIDATION.md new file mode 100644 index 000000000..d69dc856d --- /dev/null +++ b/aorlta/VALIDATION.md @@ -0,0 +1,207 @@ +# Is it working? What AORLTA's validation shows, and what it does not + +Each item says what is checked, the evidence, and how to reproduce it. Commands run from `aorlta/` in the +environment of [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md); `<...>` are your paths. GPU runs were on one node +of 8 MI350X (gfx950) with Qwen/Qwen3.5-4B at revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`. + +## 1. The CPU test suite + +About 1,900 tests over every component, run by CI on every pull request that touches `aorlta/` (the three +`pytest (AORLTA ..., py3.12)` jobs). What they prove, by directory of `tests/`: + +| tests | what they prove | +|---|---| +| `test_config.py`, `test_schemas.py` | every config key's type and range; unknown or duplicate keys refused; the run plan is written first and once | +| `rollout/` | turn-0 and repair-turn token sequences match the chat template exactly; compile errors are parsed from real clang output (`data/clang/`); chains end for the documented reasons; a sampler that ignores seeds is caught | +| `reward/` | the reward service under a fake checker with injected latency, stalls, crashes and dirty controls: an unsettled check never yields a clean reward, controls bracket results, the watchdog restarts a stalled worker, the cache keys on everything a result depends on; `standin.v11` by property tests (decide never contradicts compute) | +| `trainer/` | the GRPO loss and its aggregations, packing equivalence with unpacked samples, the merged-weight forward against PEFT's, the per-step checks (non-finite or zero gradients, frozen weights that moved, ranks that disagree) | +| `sync/` | the LoRA merge and its rounding, weight-slot writing and manifests, reload, the landing check's verdicts, the start-up round trip | +| `orchestrator/` | the loop end to end with fakes; kill-and-resume restores the trained state bit for bit; the wired loop on CPU with a tiny Qwen3.5 and in-process engines trains, publishes and lands (`test_wiring_e2e.py`); score runs; plugin tasks through the real wiring (`test_plugin_wired.py`) | +| `test_plugins*.py`, `examples/` | the plugin registry, three task shapes registered from a package outside `aorlta/`, and both worked examples (their checks, rewards, and one CPU training iteration through the real wiring each) | +| `tools/` | the adapter merge against PEFT's merge, the checkpoint export (merged, it is exactly what the engines were served), the before/after comparison | + +Reproduce (about 15 minutes on 8 cores): + +```bash +export OMP_NUM_THREADS=1 MALLOC_MMAP_THRESHOLD_=65536 MALLOC_ARENA_MAX=2 +python -m pytest -q -n 8 -m "not heavy and not cpu_run" +OMP_NUM_THREADS=2 python -m pytest -q -n 2 -m "heavy and not cpu_run" +OMP_NUM_THREADS=4 python -m pytest -q -m cpu_run +``` + +Expected: every test passes; a few skip and say what this machine lacks (hipcc, the tokenizer in `HF_HOME`, the +stand-in dataset, the stand-in check tools, a GPU). + +## 2. Mutation checks + +A test that cannot fail proves nothing, so the new code was mutated by hand and the tests rerun: each mutant must +make a test fail. For the plugin registry: dropping the registry lookup in `component_source`, in the check-adapter +factories, in the parse-adapter lookup, in the wiring's prompt pool, and the run plan's plugin record (5 of 5 +caught); the plugin self-test chats (1 of 1); the checkpoint export's tensors and its manifest check (2 of 2). The +components merged before them recorded their own mutation runs in their pull requests. Reproduce one: make the edit, +run the named test file, see it fail, revert. + +## 3. The checks every run makes on itself + +These run inside every real run, so a run that completes has passed them; the run stops otherwise. + +| check | what it refuses | evidence | +|---|---|---| +| engine start | an engine whose sampler, log-prob output, prefix cache or dtype read back wrong, or whose samples do not vary with the seed | every GPU run below started 4 engines (or 1) and passed | +| start-up round trip | weights that do not reach the engines: a synthetic perturbation must change every probe, restoring the base must bring it back bit for bit | PROVEN in every GPU run below | +| landing check | a weight update that did not land on every engine after each training step | LANDED on every engine, every iteration, in every training run below (66 updates) | +| trainer–engine gap gate | a sample whose trainer log-probs differ from the engine's by more than `trainer.max_generation_gap` is left out of the step; too many and the step is refused | p99 of the per-sample max \|Δ log-prob\|: 0.68–0.73 nats (stand-in, 4096-token replies), at most 0.19 (kernel example), 0.25–0.53 (arithmetic), against the 2.0 limit; no sample over it | +| step checks | a non-finite or zero gradient, a non-finite loss, a group trained twice, a LoRA tensor that did not move, a frozen weight that did | passed every step | +| scored fraction | training on an iteration whose checks did not settle for enough samples; an unscored sample is never counted clean | 100% scored in the example runs, 72–82% in the stand-in run (where ConSan often has nothing to watch, as documented) | + +## 4. GPU runs + +| run | config | commit | GPUs | result | +|---|---|---|---|---| +| stand-in training | `configs/dev-standin.yaml` | `21d7befb` | 4 engines, 1 rank | 3 iterations; 207, 184, 209 of 256 scored; every update landed | +| stand-in score | `configs/score-standin.yaml` | `21d7befb` | 4 engines, 4 checker GPUs | round trip PROVEN; 50 of 64 scored | +| kernel example, self-check | `examples/hip-kernels/hip_kernels.py selfcheck` | this branch | 1 at a time (GPUs 0, 4, 7) | all 8 reference kernels pass on each | +| kernel example, score | `configs/hipk-score.yaml` | this branch | 4 engines, 4 check GPUs | 64 of 64 scored, 56 correct; by task 8/8 on the six elementwise kernels, 5/8 transpose, 3/8 block sum | +| kernel example, train, export, score | `configs/hipk-train.yaml`, `tools/export_adapter.py`, a score run with `model.adapter_dir` | this branch | 4 engines, 1 rank | 3 iterations landed; the exported adapter merged, served and PROVEN; 59 of 64 correct on the same 8 prompts it trained on (base 56): that shows the pipeline, not learning | +| arithmetic, train | `configs/arith-train.yaml` | this branch | 1 GPU (engine and rank) | 20 iterations of 16 prompts x 8 samples, 13–49 s each; every update landed; correct on the training prompts 93.3% (mean of iterations 1–5) to 97.5% (16–20); mean reward 0.87 to 0.93 | +| arithmetic, held-out before/after | `configs/arith-score.yaml`, base and trained | this branch | 1 GPU | 64 held-out prompts x 4 samples: pass rate 94.9% (243/256) base, 98.4% (252/256) trained; mean reward 0.895 to 0.948; on the same prompt and seed, 12 samples went from wrong to right and 3 from right to wrong (sign test p = 0.035) | +| harder arithmetic, train | `configs/arith4-train.yaml` | this branch | 1 GPU | 40 iterations of 16 prompts x 8 samples, about 21 s each; every update landed; correct on the training prompts 74.8% (iterations 1–5) to 98.9% (36–40); mean reward 0.58 to 0.96 | +| harder arithmetic, held-out before/after | `configs/arith4-score.yaml`, base and trained | this branch | 1 GPU | 64 held-out prompts x 4 samples: pass rate **83.2% (213/256) base, 98.4% (252/256) trained**; mean reward 0.724 to 0.950; 30 prompts better, 1 worse; on the same prompt and seed, 42 samples went from wrong to right and 3 from right to wrong (sign test p < 1e-9) | + +Reproduce a run: `CONFIG=configs/.yaml sbatch jobs/aorlta-run.sbatch` with a site file (GETTING-STARTED.md, +"Set up the node"). The job's log ends with one line per iteration, or a score run's summary. + +## 5. Does it learn? The before/after evaluation + +The arithmetic example (`examples/qa/`, `arith4/` and `arith/`) asks for a product of two random 4-digit (or +3-digit) numbers and rewards an exact final answer: +1.0 correct, -0.5 wrong, -1.0 no `Answer:` line. Training +prompts and held-out prompts share no pair. To reproduce, train, export the last checkpoint, score the base and the +trained adapter on the held-out set under two run roots, and compare: + +```bash +CONFIG=configs/arith4-train.yaml sbatch jobs/aorlta-run.sbatch +python tools/export_adapter.py --ckpt /ckpt/iter-000040 --out +CONFIG=configs/arith4-score.yaml AORLTA_SITE= sbatch jobs/aorlta-run.sbatch # run.root: +CONFIG=configs/arith4-score.yaml AORLTA_SITE= sbatch jobs/aorlta-run.sbatch # adds model.adapter_dir +python tools/eval_compare.py --base /standin/arith4-score --trained /standin/arith4-score \ + --tokenizer --out +``` + +Result (commit of this branch, Qwen3.5-4B, one MI350X GPU for the engine and the LoRA rank together, 40 iterations +of 16 prompts x 8 samples at temperature 1.0, about 21 s an iteration, 20 minutes of run time with setup): + +| iterations | mean reward | correct (training prompts) | replies with no `Answer:` line | +|---|---|---|---| +| 1–5 | 0.58 | 74.8% | 29 of 640 | +| 6–10 | 0.73 | 84.1% | 12 of 640 | +| 11–20 | 0.89 | 94.5% | 16 of 1280 | +| 21–30 | 0.93 | 97.3% | 6 of 1280 | +| 31–40 | 0.95 | 98.4% | 2 of 1280 | + +Held out (64 multiplications never trained on, 4 samples each at temperature 0.7), from `tools/eval_compare.py`: + +| | base | trained | +|---|---|---| +| pass rate | 83.2% (213/256) | 98.4% (252/256) | +| mean reward | 0.724 | 0.950 | +| mean reply length | 412 tokens | 551 tokens | +| replies needing the repair turn for a missing `Answer:` line | 47 | 28 | + +What changed is visible in the rollouts: on `3961 × 2885` the base model answered `Answer: 11429685` with no working +(wrong), and the trained one wrote out the partial products and their sum before answering 11,427,485 (right). The +policy learned to work the product out rather than guess it. The 3-digit set shows the same direction with less +room to move (94.9% to 98.4% held out; 12 samples wrong to right, 3 right to wrong). + +What this shows and does not: the loop learns, the gain holds on prompts it never saw, and the checks that guard +each step (landing, gap, scored fraction) passed throughout. It is one run, one seed, on a toy task the base model +was already fair at; section 6 makes the same measurement on a real kernel task. + +## 6. Learning on a real kernel task + +The kernel example of section 4 has 8 tasks, far too few to learn from. This run used 93 HIP kernel-writing tasks in +eight families (elementwise, reduction, transpose, softmax, layernorm, scan, GEMM, gather/scatter), in float, double, +int and `__half`, with launches that include one thread per element, grid-stride loops with fewer threads than +elements, one block per row, 2-D tiles and a batch on `gridDim.z`. Each task gives a kernel's name, signature, launch +and what it computes, in the format of `examples/hip-kernels/tasks.json`. The model writes the kernel; the check +builds it, runs it on a GPU and compares every output element with a host reference loop, and also fails a write +past the end of an array or into an input. The reward is the `hipk.v1` table (`examples/hip-kernels/hip_kernels.py`): ++1.0 when the output matches, -0.5 wrong output, -0.6 crash or timeout, -0.8 bad signature, -0.9 compile error, -1.0 +no code block, minus a small length penalty. Each task has one correct reference kernel, used only to prove the check +(93 of 93 pass); no kernel carries a planted bug. + +The task set is not in this repository. It was generated by a script that writes each family's ops at several +shapes, with a reference kernel per task, and splits them by op and by shape: within each family the ops are divided +between 54 training and 39 held-out tasks, a held-out task uses no shape a training task of its family uses, and the +whole scan family is held out; the script refuses a split that breaks either rule. The check in +`examples/hip-kernels` handles float tasks only, so running this set needs that check extended to the other dtypes. +These runs used the check as it was before it compiled the candidate alone into a device code object, when the +candidate's host code was built into the harness. So every reply was searched afterwards: none of the trained model's +312 held-out replies prints the verdict line, defines `main` or runs host code of its own (`printf`, a constructor, +`exit`), and in the 5,120 training replies the only matches are in prose or a commented-out `main`. + +To reproduce, put the tasks in `/train/tasks.json` and `/heldout/tasks.json`, copy `configs/hipk-train.yaml` +and `configs/hipk-score.yaml` with `data.prompt_pools[0].path` set to those directories (and, for training, +`rollout.P: 16`, `rollout.temperature: 1.0`, `orchestrator.iterations: 40`, `trainer.lr: 2.0e-5`), then as in section 5: + +```bash +CONFIG= sbatch jobs/aorlta-run.sbatch +python tools/export_adapter.py --ckpt /ckpt/iter-000040 --out +CONFIG= AORLTA_SITE= sbatch jobs/aorlta-run.sbatch # run.root: +CONFIG= AORLTA_SITE= sbatch jobs/aorlta-run.sbatch # adds model.adapter_dir +python tools/eval_compare.py --base --trained --tokenizer \ + --out +``` + +Result (Qwen3.5-4B, 8 MI350X GPUs: 4 engines, one LoRA rank, kernels run on GPUs 4–7; a fresh LoRA, 40 iterations of +16 prompts × 8 samples at temperature 1.0, lr 2e-5, up to 2 compile-repair turns; 52 minutes). Every update LANDED +on all four engines, no sample came near the gap bound (largest 1.24 against 2.0, off-policy fraction 0), and the +scored fraction never fell below 0.984. On the training prompts the share of correct kernels rose from 59% in +iterations 1–5 to 88% in iterations 31–40, and the mean reward from 0.30 to 0.78. + +Held out (39 tasks never trained on, 4 samples × 2 passes each at temperature 0.7, up to 2048 new tokens), from +`tools/eval_compare.py`: + +| | base | trained | +|---|---|---| +| pass rate | 47.4% (148/312) | 76.6% (239/312) | +| mean reward | 0.130 | 0.633 | +| compiled | 84.0% | 97.1% | +| mean reply length | 966 tokens | 223 tokens | + +On the same task, pass and sample slot: 108 samples went from wrong to right and 17 from right to wrong (exact sign +test p = 2e-17). By task, 23 improved and 5 got worse (p = 0.0009). Part of the gain is shorter replies (the base hit +the 2048-token cap on 45 samples), but leaving out every slot where either model hit the cap, the pass rate still +rises from 56% to 77%, with 75 samples wrong to right against 17. The largest gains are on families the model saw +least or not at all: + +- scan, never trained on: 2% → 73%; +- transpose: 31% → 97%; +- softmax: 25% → 63%; +- layernorm: 54% → 92%. + +GEMM (43% → 43%) and gather/scatter (75% → 69%) did not improve. On a held-out running max, the base model spent its +budget in comments and then wrote a kernel that calls `hipMalloc` from device code; the trained model wrote a short +correct running-max loop. + +What this shows and does not: the loop teaches a real, checkable task, and the gain holds on ops, shapes and a +family it never trained on. **But the reward checks correctness only, not speed**, and the model learned exactly +that. On tasks launched with one block per row, none of the trained model's 101 correct kernels has the block's +threads share the row: one thread does the whole row. The base model already did this (45 of 48), and the reward +cannot tell the difference. It also learned one wrong habit: it drops the grid-stride loop where a launch has fewer +threads than elements (one task, 5/8 → 0/8). It is one run with one seed. Writing *fast* kernels needs a speed term +in the reward, which is not done yet. + +## Not yet shown + +- **Fast kernels.** Section 6 shows a model learning to write *correct* kernels on held-out tasks; its reward has no + speed term, the trained kernels leave most of a block idle, and GEMM and gather/scatter did not improve. Each + learning result here is one run with one seed. +- **The 93-task set of section 6 is not in the repository**, and the kernel example's check handles float tasks only. +- **The examples' checks are cheap stand-ins** for the checks a real vertical would run (a full test suite, a + sanitizer on a real workload, a simulator); section 6's check compares outputs and does not time them. +- **Only Qwen3.5-4B, only gfx950, only one node.** Other Qwen3.5 sizes should work and have not been run; Qwen3 + models (such as Qwen3-0.6B) need the changes listed under "Models" in GETTING-STARTED.md; fewer than 8 GPUs has run + only in the 1-GPU arithmetic and Q&A layouts; MI300X and MI355X have not been run. +- **Long runs**: the longest training run here is 40 iterations. Stability over hundreds of iterations, and resuming + one on real hardware beyond the kill-and-resume drill, are not shown. +- **Multi-turn episodes beyond compile-repair**, an evaluation exporter during a run, and runs outside Slurm. diff --git a/aorlta/configs/arith-score.yaml b/aorlta/configs/arith-score.yaml new file mode 100644 index 000000000..be7d71540 --- /dev/null +++ b/aorlta/configs/arith-score.yaml @@ -0,0 +1,38 @@ +# Score a model on the 64 held-out multiplications, 4 samples each (orchestrator.mode: score): the before/after evaluation of VALIDATION.md. +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/arith/heldout. +plugins: [examples/qa/qa_task.py] +run: + run_id: arith-score + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/arith/heldout + source: standin + task: qa +rollout: + engines: [0] + K: 4 + P: 16 + max_repair_turns: 1 + thinking: "off" + temperature: 0.7 + max_new_tokens: 512 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + mode: score + run_seed: 20261113 diff --git a/aorlta/configs/arith-train.yaml b/aorlta/configs/arith-train.yaml new file mode 100644 index 000000000..4bf587292 --- /dev/null +++ b/aorlta/configs/arith-train.yaml @@ -0,0 +1,42 @@ +# The validation run (VALIDATION.md): train on 512 generated 3-digit multiplications, 16 per iteration x 8 samples, 20 iterations. +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/arith/train. +plugins: [examples/qa/qa_task.py] +run: + run_id: arith-train + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/arith/train + source: standin + task: qa +rollout: + engines: [0] + K: 8 + P: 16 + max_repair_turns: 1 + thinking: "off" + temperature: 1.0 + max_new_tokens: 512 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + iterations: 20 + run_seed: 20261113 +trainer: + ranks: 1 + micro_batch_tokens: 20480 + lr: 2.0e-5 diff --git a/aorlta/configs/arith4-score.yaml b/aorlta/configs/arith4-score.yaml new file mode 100644 index 000000000..3b376c5e9 --- /dev/null +++ b/aorlta/configs/arith4-score.yaml @@ -0,0 +1,38 @@ +# Score a model on the 64 held-out multiplications, 4 samples each (orchestrator.mode: score): the before/after evaluation of VALIDATION.md. +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/arith4/heldout. +plugins: [examples/qa/qa_task.py] +run: + run_id: arith4-score + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/arith4/heldout + source: standin + task: qa +rollout: + engines: [0] + K: 4 + P: 16 + max_repair_turns: 1 + thinking: "off" + temperature: 0.7 + max_new_tokens: 1024 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + mode: score + run_seed: 20261113 diff --git a/aorlta/configs/arith4-train.yaml b/aorlta/configs/arith4-train.yaml new file mode 100644 index 000000000..a08e2891a --- /dev/null +++ b/aorlta/configs/arith4-train.yaml @@ -0,0 +1,42 @@ +# The validation run (VALIDATION.md): train on 512 generated 4-digit multiplications, 16 per iteration x 8 samples, 40 iterations. +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/arith4/train. +plugins: [examples/qa/qa_task.py] +run: + run_id: arith4-train + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/arith4/train + source: standin + task: qa +rollout: + engines: [0] + K: 8 + P: 16 + max_repair_turns: 1 + thinking: "off" + temperature: 1.0 + max_new_tokens: 1024 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + iterations: 40 + run_seed: 20261113 +trainer: + ranks: 1 + micro_batch_tokens: 20480 + lr: 2.0e-5 diff --git a/aorlta/configs/hipk-score.yaml b/aorlta/configs/hipk-score.yaml new file mode 100644 index 000000000..89c22f014 --- /dev/null +++ b/aorlta/configs/hipk-score.yaml @@ -0,0 +1,39 @@ +# Score the public Qwen3.5-4B on the example task of docs/TUTORIAL.md (examples/hip-kernels): 8 kernels to write, +# 4 samples each, 2 passes, up to 2 repair turns. 4 engines on GPUs 0-3; the kernels run on GPUs 4-7 (hipk-gpu.yaml). +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir and +# data.prompt_pools[0].path, which is this checkout's aorlta/examples/hip-kernels. +plugins: [examples/hip-kernels/hip_kernels.py] +run: + run_id: hipk-score + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/hip-kernels + source: standin + task: hipk +rollout: + engines: [0, 1, 2, 3] + K: 4 + P: 8 + max_repair_turns: 2 + thinking: "off" + temperature: 0.7 + max_new_tokens: 2048 + compile_adapter: standin.hipcc + parse_adapter: hipk.parse + feedback_template: hipk.feedback + # a short kernel can have one obvious answer: 8 identical answers to one prompt is not a broken sampler here + # (the engines' self-test checks that), and with 8 prompts one such group is 12.5% of them + identical_groups_stop: 0.5 +reward: + profile: hipk-gpu + reward_fn: hipk.v1 +orchestrator: + mode: score + run_seed: 20261112 + score: + passes: 2 diff --git a/aorlta/configs/hipk-train.yaml b/aorlta/configs/hipk-train.yaml new file mode 100644 index 000000000..057b10fdb --- /dev/null +++ b/aorlta/configs/hipk-train.yaml @@ -0,0 +1,41 @@ +# Train on the example task of docs/TUTORIAL.md (examples/hip-kernels): 3 iterations of the 8 kernels x 8 samples, +# one LoRA rank beside engine 0, the kernels run on GPUs 4-7 (hipk-gpu.yaml). A starting point to copy, not a result: +# 8 prompts are far too few to learn from. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir and +# data.prompt_pools[0].path, which is this checkout's aorlta/examples/hip-kernels. +plugins: [examples/hip-kernels/hip_kernels.py] +run: + run_id: hipk-train + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/hip-kernels + source: standin + task: hipk +rollout: + engines: [0, 1, 2, 3] + K: 8 + P: 8 + max_repair_turns: 2 + thinking: "off" + temperature: 0.7 + max_new_tokens: 2048 + compile_adapter: standin.hipcc + parse_adapter: hipk.parse + feedback_template: hipk.feedback + # a short kernel can have one obvious answer: 8 identical answers to one prompt is not a broken sampler here + # (the engines' self-test checks that), and with 8 prompts one such group is 12.5% of them + identical_groups_stop: 0.5 +reward: + profile: hipk-gpu + reward_fn: hipk.v1 +orchestrator: + iterations: 3 + run_seed: 20261112 +trainer: + ranks: 1 + micro_batch_tokens: 20480 diff --git a/aorlta/configs/qa-score.yaml b/aorlta/configs/qa-score.yaml new file mode 100644 index 000000000..a76566cd5 --- /dev/null +++ b/aorlta/configs/qa-score.yaml @@ -0,0 +1,38 @@ +# Score a model on the Q&A example's 8 held-out questions, 4 samples each (orchestrator.mode: score). +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/gpu-facts/heldout. +plugins: [examples/qa/qa_task.py] +run: + run_id: qa-score + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/gpu-facts/heldout + source: standin + task: qa +rollout: + engines: [0] + K: 4 + P: 8 + max_repair_turns: 1 + thinking: "off" + temperature: 0.7 + max_new_tokens: 512 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + mode: score + run_seed: 20261113 diff --git a/aorlta/configs/qa-train.yaml b/aorlta/configs/qa-train.yaml new file mode 100644 index 000000000..e2e819058 --- /dev/null +++ b/aorlta/configs/qa-train.yaml @@ -0,0 +1,42 @@ +# Train on the Q&A example of docs/TUTORIAL.md: 16 GPU-programming questions, 8 samples each, 3 iterations. +# One GPU: one TokenSpeed engine on GPU 0 and, for training, one LoRA rank beside it; the answer check runs on the CPU. +# Site-specific paths, to edit here or to set in a site file (AORLTA_SITE): run.root, model.base_dir (and +# model.adapter_dir to score a trained adapter) and data.prompt_pools[0].path, which is this checkout's +# aorlta/examples/qa/gpu-facts/train. +plugins: [examples/qa/qa_task.py] +run: + run_id: qa-train + source: standin + root: /scratch/aorlta/runs +model: + base_dir: /data/hf/hub/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a + source: standin +data: + prompt_pools: + - path: /opt/aorlta/examples/qa/gpu-facts/train + source: standin + task: qa +rollout: + engines: [0] + K: 8 + P: 8 + max_repair_turns: 1 + thinking: "off" + temperature: 1.0 + max_new_tokens: 512 + compile_adapter: none + parse_adapter: qa.parse + feedback_template: qa.feedback + # a question with one short answer can get K identical replies: that is not a broken sampler here (the engines' + # start-up self-test checks that), so only stop when most groups are + identical_groups_stop: 0.75 +reward: + profile: qa-match + reward_fn: qa.v1 +orchestrator: + iterations: 3 + run_seed: 20261113 +trainer: + ranks: 1 + micro_batch_tokens: 20480 + lr: 2.0e-5 diff --git a/aorlta/docs/TUTORIAL.md b/aorlta/docs/TUTORIAL.md new file mode 100644 index 000000000..45c2da22a --- /dev/null +++ b/aorlta/docs/TUTORIAL.md @@ -0,0 +1,190 @@ +# Tutorial: your own task, end to end + +This walks through `examples/hip-kernels/`, a small task built as a plugin: the model is asked to write a HIP kernel +from its description (vector add, SAXPY, ReLU, scale and shift, clamp, reverse, a per-block sum, a transpose), and +the reward builds its answer, runs it on a GPU and compares every output element with a host reference. Nothing in +`src/aorlta/` names this task. Copy the directory, change the four parts below, and you have your own. + +Paths are relative to `aorlta/`. The commands assume the environment from the README (a Python 3.12 venv with +`requirements-dev.txt` and `pip install -e . --no-deps`). + +## What you write + +| part | in the example | what it does | +|---|---|---| +| the prompts | `examples/hip-kernels/tasks.json`, and `records()` in `hip_kernels.py` | one prompt record per task: the chat, its prompt id, and a hash of what the checks read for it | +| how an answer is read | `HipkParse` and `HipkFeedback` | the last ```cpp block is the answer; no block, or no definition of the kernel asked for, gets a repair turn that says so; a compile error gets one with the compiler's errors | +| the checks | `HipkGate`, `HipkRun` and `examples/hip-kernels/hipk-gpu.yaml` | a parse gate with the rollout's own rule and the stand-in compile gate, then `ref.native`: compile the answer alone into a code object, run its kernel on a GPU from a harness built from the task, compare | +| the reward | `HipkV1` | -1.0 no parsable answer, -0.9 compile error, -0.8 wrong signature, -0.6 crash or hang, -0.5 wrong output, +1.0 correct, minus 0.05 x the share of the token budget used | + +The module registers each part under a name (`register()` at its end): + +```python +plugins.register_task(plugins.Task("hipk", records)) +plugins.register_parse_adapter("hipk.parse", HipkParse) +plugins.register_feedback_template("hipk.feedback", HipkFeedback) +plugins.register_profile("hipk-gpu", HERE / "hipk-gpu.yaml") +plugins.register_check_adapter("hipk.run", HipkRun) +plugins.register_check_adapter("hipk.gate", HipkGate) +plugins.register_reward_fn("hipk.v1", HipkV1) +``` + +and the run config lists the module and uses the names (`configs/hipk-score.yaml`): + +```yaml +plugins: [examples/hip-kernels/hip_kernels.py] +data: + prompt_pools: + - {path: /path/to/aorta/aorlta/examples/hip-kernels, source: standin, task: hipk} +rollout: + compile_adapter: standin.hipcc # a built-in part, mixed in by name + parse_adapter: hipk.parse + feedback_template: hipk.feedback +reward: + profile: hipk-gpu + reward_fn: hipk.v1 +``` + +[ADAPTING.md](ADAPTING.md) has the full contract of each part. The rules that matter most: + +- **Prompts.** `records(pool_dir, tokenizer, thinking)` returns `PromptRecord`s with `source="standin"` (a plugin's + data is handled as public stand-in data: its text may appear in logs). `task.task_sha256` must change whenever what + a check reads for that prompt changes: it is part of every cache key, and `hipk.run` finds the task by it. + Training takes `rollout.P` different prompts per iteration, so the pool needs at least `P` of them; the landing + check uses the first `sync.probes` (4) prompts' chats, which must share one system message. +- **A check never guesses.** Anything it cannot settle (no GPU, a tool missing, output it cannot read) is + `unsettled`, never `clean`; only the check may blame the candidate for a timeout. `verdict()` in the example is + the whole mapping, and `tests/examples/test_hip_kernels.py` tests every row of it on CPU. +- **The candidate is not trusted.** Its verdict must come from code it cannot change: the example never runs the + answer's host code and reads only the harness's own record file. A check's `version` must change whenever its + verdict could (the example's names its compiler and its own file), because cached legs are keyed by it. +- **The reward's parts.** `compute` returns `(reward, {"base", "broke_healthy", "length"})`, which add up to the + reward, within [-1.55, 1.0]; an unscored sample is `(None, {"reason": ...})` and is never counted clean. `decide` + answers early only when no pending leg could change the answer. The service recomputes and checks all of this, + and a reward that breaks it is unscored and raises an alarm. +- **The check's GPUs** are its own setting (`hipk.run.gpus` in the profile): keep them off `rollout.engines`. Each + live worker holds one of them, and a worker the service replaces takes the GPU its predecessor gave back. +- **Precise tasks.** An iteration stops when more than `rollout.identical_groups_stop` (5% by default) of its groups + have all `K` answers identical, which is how a broken sampler shows. A short kernel can have one obvious answer, + and with 8 prompts one such group is 12.5%, so the example's configs set it to 0.5; the engines' start-up + self-test still checks the sampler, on open-ended chats of its own. + +## 1. Check the tasks + +Each task in `tasks.json` comes with a correct reference kernel in `reference/.hip`. Build every task's harness +with its reference kernel, on CPU: + +```bash +python examples/hip-kernels/hip_kernels.py selfcheck --hipcc /opt/rocm/bin/hipcc --build-only +``` + +and run them on one GPU (one line per task, all `"pass": true`; on a cluster, inside a job): + +```bash +python examples/hip-kernels/hip_kernels.py selfcheck --hipcc /opt/rocm/bin/hipcc --gpu 0 +``` + +The candidate is compiled on its own into a device code object (`hipcc --genco`), so none of its host code is built +or run. The harness is a host program built from the task alone: it loads that code object, fills the inputs from a +fixed seed, runs the task's kernel once with the launch the prompt states, and compares every output element with +the task's `reference` loop (host C++), within `atol + rtol * |want|`. It writes its verdict to a file the reward +reads; nothing the candidate prints is read, and a failure before the launch (an allocation, the first copy) leaves +the check unsettled rather than blaming the candidate. Output elements start as a sentinel, so an element the kernel never +writes is a mismatch, and every array has a guard region past its end, so a write past the end is caught too. + +## 2. Run the tests + +```bash +OMP_NUM_THREADS=1 python -m pytest -q tests/examples/test_hip_kernels.py tests/test_plugins.py +``` + +They load the configs with the plugin, build the prompts and probes, run the parse and feedback, run `hipk.run` +against a stand-in `hipcc` (no GPU), and check the reward against the service's invariants for every combination of +leg outcomes. With hipcc installed, they also build every harness. + +## 3. The loop on CPU, with fake components + +`--components fake` loads the config and the plugin exactly as a real run does, then runs the loop with fake engines, +checks and trainer: it tells you the config is consistent and every name resolves, not that your task works (the +fakes use prompts and checks of their own). + +```bash +export SCRATCH=/path/to/scratch +cat > "$SCRATCH/site.yaml" </qa.jsonl`, one `{"id", "question", "answer", "aliases"?}` per line; the model is told to end with `Answer: ` | +| parse and feedback | a reply with an `Answer:` line is the answer; one without gets a repair turn that says so | +| compile | none: `rollout.compile_adapter: none`, so a parsed answer ends its chain | +| the check | one leg, `ref.answer`, on the CPU: the last `Answer:` line, normalised (case, spaces, quotes, a final full stop, digit commas), against the answer and its aliases; with a judge command in the profile's `qa.match.judge`, an answer that does not match is the judge's to decide, and no verdict leaves it unscored | +| the reward | `qa.v1`: +1.0 correct, -0.5 wrong, -1.0 no answer line, minus the length term | + +It comes with two pools, each split into `train/` and `heldout/`: `gpu-facts/`, 24 short questions on GPU +programming, and `arith/` and `arith4/`, generated multiplications (`make_arith.py`), which VALIDATION.md trains on +to show that the loop learns. Its configs run on **one GPU**: one engine on GPU 0 and one trainer rank beside it. + +```bash +OMP_NUM_THREADS=1 python -m pytest -q tests/examples/test_qa.py # on CPU, including a wired training run +CONFIG=configs/qa-score.yaml sbatch jobs/aorlta-run.sbatch # score the base model on the held-out set +CONFIG=configs/qa-train.yaml sbatch jobs/aorlta-run.sbatch # train +``` + +Then export the trained checkpoint, score it on the held-out set, and compare the two with +`tools/eval_compare.py` (VALIDATION.md, "Does it learn?"). Facts the base model does not know cannot be taught this +way; what training on such a pool changes is how the model answers. + +## Making it yours + +- **Your prompts.** Change `tasks.json` and `USER`, or write `records()` for your own data format. Keep the prompt + ids stable and unique, and hash into `task_sha256` everything your check reads. +- **Your answer format.** `HipkParse.check` returns `edits` (code to compile), `no_change`, or `rejected` with a + message for a repair turn. A task whose answers are not compiled can register a compile adapter that accepts + everything: each chain then ends after its first parsed answer (`rollout.max_repair_turns` still bounds the + repair turns a rejected answer gets). +- **Your check.** Keep `run` short and total: build, run, read, map to a state, and turn every exception into + `unsettled`. A slow check needs a pool timeout to match (`reward.timeouts`, named by the pool's `timeout`). +- **Your reward.** Keep the three parts and the range, so the service can check it; give every failure you can + name its own value, so that a group of answers to one prompt gets different rewards (GRPO learns only from the + difference). +- **The settings** are the run config's ([README](../README.md) and `src/aorlta/config.py`, which documents every + key): `rollout.K` samples per prompt, `rollout.P` prompts per iteration, `rollout.temperature`, + `rollout.max_repair_turns` (0 to 3), `rollout.max_new_tokens`, `orchestrator.min_scored_fraction` (the share of an + iteration's samples that must be scored before it trains), `orchestrator.deadlines`, `trainer.lr`. The LoRA is + r 16, alpha 32 on fixed target modules (`LORA_R`, `LORA_ALPHA` in `orchestrator/wiring.py`). diff --git a/aorlta/examples/hip-kernels/README.md b/aorlta/examples/hip-kernels/README.md new file mode 100644 index 000000000..2bd9839e8 --- /dev/null +++ b/aorlta/examples/hip-kernels/README.md @@ -0,0 +1,15 @@ +# hip-kernels: a worked example of a task of your own + +Write a HIP kernel from its description; the reward builds the answer, runs it on a GPU and compares every output +element with a host reference. It is a plugin (`hip_kernels.py`): nothing in `src/aorlta/` names it. +[docs/TUTORIAL.md](../../docs/TUTORIAL.md) walks through it. + +| file | what | +|---|---| +| `hip_kernels.py` | the plugin: prompts, parse and feedback, the `ref.native` check, the `hipk.v1` reward, and a `selfcheck` command | +| `tasks.json` | the 8 tasks: signature, what to compute, the launch, the inputs, a host reference loop and a tolerance | +| `reference/.hip` | a correct kernel for each task, used only by `selfcheck` | +| `hipk-gpu.yaml` | the check profile: parse gate, compile gate, `ref.native` on GPUs 4-7 | + +The run configs are `configs/hipk-score.yaml` and `configs/hipk-train.yaml`; the tests are +`tests/examples/test_hip_kernels.py`. diff --git a/aorlta/examples/hip-kernels/hip_kernels.py b/aorlta/examples/hip-kernels/hip_kernels.py new file mode 100644 index 000000000..0aa6d1823 --- /dev/null +++ b/aorlta/examples/hip-kernels/hip_kernels.py @@ -0,0 +1,632 @@ +"""A worked example of a task of your own, as a plugin (docs/TUTORIAL.md): write a HIP kernel from its description. + +Each task in `tasks.json` names a kernel, its exact signature, what it computes and how it is launched. The model +answers with a HIP source in one ```cpp block. The rollout compiles it with the stand-in compile check (hipcc -c) and +gives up to `rollout.max_repair_turns` repair turns with the compiler's errors; a reply that has no code block, or +whose code does not define the kernel, gets a repair turn that says so (this module's parse and feedback). The reward +service then runs three legs: this module's parse gate (the rollout's own parse rule) and the stand-in compile gate, +and `ref.native`, this module's check, which compiles the candidate alone into a device code object, runs its kernel +on a GPU from a host program built from the task alone, and compares every output element with a host reference +computed from the same inputs. The reward is `hipk.v1` (below). + +Registered names: task `hipk`; parse `hipk.parse`; feedback `hipk.feedback`; profile `hipk-gpu` (hipk-gpu.yaml); +check adapters `hipk.gate` and `hipk.run`; reward function `hipk.v1`. + +Run as a script, it checks the tasks themselves with the reference kernels in reference/: + python examples/hip-kernels/hip_kernels.py selfcheck --hipcc /opt/rocm/bin/hipcc [--gpu 0] [--build-only] +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import shutil +import sys +import tempfile +import threading +import time +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SYSTEM = "You are an expert AMD GPU kernel engineer." +USER = ( + "Write a HIP kernel with exactly this signature:\n\n```cpp\n{signature}\n```\n\n" + "It must compute {description}. It is launched {launch}.\n\n" + "Write a complete HIP C++ source file: include , define the kernel, and write no main " + "function. Reply with the source in a single ```cpp code block." +) +SENTINEL = "-7777.0f" # what an output element holds before the kernel runs: one it never writes is a mismatch +GUARD = 256 # elements past the end of every device array, which the kernel must leave alone +GUARD_VALUE = "31337.0f" + + +# --- the tasks ------------------------------------------------------------------------------------- + + +def load_tasks(pool_dir: str | Path) -> list[dict]: + """The pool's tasks, in file order: `/tasks.json`, a list of task specs.""" + tasks = json.loads((Path(pool_dir) / "tasks.json").read_text(encoding="utf-8")) + ids = [t["id"] for t in tasks] + if len(set(ids)) != len(ids): + raise ValueError(f"{pool_dir}/tasks.json lists a task id twice") + return tasks + + +def task_sha256(task: dict) -> str: + """What the check reads for a task, hashed: part of every check's cache key, and how the check finds the task.""" + return hashlib.sha256(json.dumps(task, sort_keys=True).encode("utf-8")).hexdigest() + + +def messages(task: dict) -> list[dict]: + user = USER.format(signature=task["signature"], description=task["description"], launch=task["launch"]) + return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}] + + +def records(pool_dir, tokenizer, thinking): + """The prompt records: one per task, prompt id `hipk:`.""" + from aorlta.ids import sha256_ids + from aorlta.rollout.templates import render_turn0 + from aorlta.schemas import PromptRecord, TaskRef + + out = [] + for task in load_tasks(pool_dir): + msgs = messages(task) + out.append(PromptRecord( + prompt_id=f"hipk:{task['id']}", turn0_ids=None, messages=msgs, + prompt_ids_sha256=sha256_ids(render_turn0(tokenizer, msgs, thinking)), + task=TaskRef(contract_id=f"hipk:{task['id']}", parent_id=None, parent_healthy=None, reference_kind=None, + task_sha256=task_sha256(task)), + source="standin")) + return out + + +# --- the rollout: parse and feedback ------------------------------------------------------------ + + +def defines_kernel(source: str, kernel: str) -> bool: + return re.search(r"__global__\s+(?:__launch_bounds__\s*\([^)]*\)\s*)?void\s+" + re.escape(kernel) + r"\s*\(", + source) is not None + + +def prompt_kernel(prompt) -> str | None: + """The kernel a prompt asks for, read from the signature in its user message: a task's id names the task and + need not be its kernel's name (shape variants of one kernel have their own ids).""" + for m in getattr(prompt, "messages", None) or (): + if m.get("role") == "user": + found = re.search(r"__global__\s+(?:__launch_bounds__\s*\([^)]*\)\s*)?void\s+(\w+)\s*\(", m["content"]) + return found.group(1) if found else None + return None + + +class HipkParse: + """`hipk.parse`: the last ```cpp block (the stand-in parse's rule). No block, or a block that does not define the + task's kernel, is rejected with a message, which `hipk.feedback` turns into a repair turn.""" + + name = "hipk.parse" + + def __init__(self): + from aorlta.rollout.templates import IM_END_ID + + self.reply_stops = (IM_END_ID,) + + def check_prompts(self, prompts, thinking, max_repair_turns=0): + for p in prompts: + if not p.prompt_id.startswith("hipk:"): + raise ValueError(f"hipk.parse reads hipk prompts, got {p.prompt_id!r}") + if prompt_kernel(p) is None: + raise ValueError(f"{p.prompt_id}: its user message has no __global__ signature to name the kernel") + + def check(self, text, thinking, prompt): + from aorlta.rollout.compile_check import ParseOutcome, parse_code + + code = parse_code(text, thinking) + if code is None: + return ParseOutcome("rejected", rejection=( + "Your reply has no complete ```cpp code block. Reply with the whole HIP source in one ```cpp block.", + "no_code_block")) + kernel = prompt_kernel(prompt) + if kernel is None: + raise ValueError(f"{prompt.prompt_id}: its user message has no __global__ signature to name the kernel") + if not defines_kernel(code, kernel): + return ParseOutcome("rejected", rejection=( + f"Your source does not define the kernel `{kernel}`. Define it as a __global__ function with exactly " + f"the signature asked for, and reply with the whole source in one ```cpp block.", "no_kernel")) + return ParseOutcome("edits", code=code) + + +class HipkFeedback: + """`hipk.feedback`: the stand-in compile feedback, and a rejected answer's message as its own repair turn.""" + + name = "hipk.feedback" + + def build(self, errors, max_chars): + from aorlta.rollout.feedback import build_feedback + + return build_feedback(errors, max_chars) + + def build_apply(self, message, reason): + from aorlta.rollout.feedback import Feedback + + return Feedback(message, 0, 0) + + +# --- the check: build, run, compare ------------------------------------------------------------ + + +def _f(x) -> str: + """A float as a C++ float literal.""" + return f"{float(x)!r}f" + + +def harness_source(task: dict) -> str: + """The task's host program, which holds none of the candidate's code: it loads the candidate's device code object + (argv[1]), fills the inputs, runs the task's kernel from it once and compares every output element with a host + reference. It writes its record to the file argv[2], replacing it whole each time: `{"stage": "setup"}` at the + start, `{"stage": "launched"}` just before the kernel runs, then one with a `status`: `ok`, `mismatch` (an output + element differs from the host reference beyond the tolerance), `out_of_bounds` (a write past the end of an + array), `no_kernel` (the code object has no kernel of the task's exact signature) or `hip_error` (with the + `stage` it failed in).""" + lines = ["#include ", "#include ", "#include ", "#include ", + "#include ", "#include ", "", + "// The task's kernel, defined here only to name it (__builtin_get_device_side_mangled_name): the code", + "// that runs is the candidate's, from the code object in argv[1].", + f"{task['signature']} {{}}", "", + "static const char* g_path = nullptr;", + "static void put(const char* json) { // written aside, then renamed over the record", + " std::string tmp = std::string(g_path) + \".partial\";", + " FILE* f = std::fopen(tmp.c_str(), \"w\");", + " if (f == nullptr) return;", + " std::fputs(json, f);", + " if (std::fclose(f) == 0) std::rename(tmp.c_str(), g_path);", + "}", "", + "static void fill(std::vector& v, uint32_t seed, float lo, float hi) {", + " for (auto& x : v) { seed = seed * 1664525u + 1013904223u; " + "x = lo + (hi - lo) * float(seed >> 8) / 16777216.0f; }", + "}", "", + "#define CHECK(stage, call) do { hipError_t e_ = (call); if (e_ != hipSuccess) { char b_[512]; " + "std::snprintf(b_, sizeof b_, \"{\\\"status\\\": \\\"hip_error\\\", \\\"stage\\\": \\\"%s\\\", " + "\\\"at\\\": \\\"%s\\\", \\\"error\\\": \\\"%s\\\"}\", stage, #call, hipGetErrorName(e_)); put(b_); " + "return 0; } } while (0)", "", + "int main(int argc, char** argv) {", + " if (argc != 3) return 2;", + " g_path = argv[2];", + " put(\"{\\\"stage\\\": \\\"setup\\\"}\");"] + for ctype, name, value in task["scalars"]: + lines.append(f" {ctype} {name} = {value};") + arrays = task["arrays"] + for i, a in enumerate(arrays): + n = a["size"] + lines.append(f" std::vector h_{a['name']}(({n}) + {GUARD}, {GUARD_VALUE});") + if a["role"] == "out": + lines.append(f" for (long i = 0; i < ({n}); ++i) h_{a['name']}[i] = {SENTINEL};") + else: + lines.append(f" {{ std::vector v({n}); fill(v, {1000 + 17 * i}u, {_f(a.get('lo', -1.0))}, " + f"{_f(a.get('hi', 1.0))}); for (long i = 0; i < ({n}); ++i) h_{a['name']}[i] = v[i]; }}") + lines.append(f" std::vector r_{a['name']} = h_{a['name']};") + lines.append(" {") + for a in arrays: + lines.append(f" float* {a['name']} = r_{a['name']}.data();") + lines.append(" " + task["reference"]) + lines.append(" }") + for a in arrays: + lines += [f" float* d_{a['name']} = nullptr;", + f" CHECK(\"setup\", hipMalloc(&d_{a['name']}, h_{a['name']}.size() * sizeof(float)));", + f" CHECK(\"setup\", hipMemcpy(d_{a['name']}, h_{a['name']}.data(), h_{a['name']}.size() * " + "sizeof(float), hipMemcpyHostToDevice));"] + names = {a["name"] for a in arrays} + args = ", ".join(f"&d_{x}" if x in names else f"&{x}" for x in task["args"]) + lines += [" hipModule_t mod_;", " CHECK(\"setup\", hipModuleLoad(&mod_, argv[1]));", " hipFunction_t fn_;", + " {", + # gf_, not e_: CHECK declares its own e_, and `hipError_t e_ = (e_)` reads itself (undefined) + f" hipError_t gf_ = hipModuleGetFunction(&fn_, mod_, __builtin_get_device_side_mangled_name(" + f"{task['kernel']}));", + " if (gf_ == hipErrorNotFound || gf_ == hipErrorInvalidDeviceFunction) {", + " char b_[256];", + " std::snprintf(b_, sizeof b_, \"{\\\"status\\\": \\\"no_kernel\\\", \\\"stage\\\": \\\"setup\\\", " + "\\\"error\\\": \\\"%s\\\"}\", hipGetErrorName(gf_));", + " put(b_);", + " return 0;", " }", " CHECK(\"setup\", gf_);", " }", + f" void* args_[] = {{{args}}};", + f" dim3 grid_ = {task['grid']};", f" dim3 block_ = {task['block']};", + " put(\"{\\\"stage\\\": \\\"launched\\\"}\");", + " CHECK(\"launch\", hipModuleLaunchKernel(fn_, grid_.x, grid_.y, grid_.z, block_.x, block_.y, block_.z, 0, " + "nullptr, args_, nullptr));", + " CHECK(\"run\", hipDeviceSynchronize());", + " long checked = 0, mismatches = 0, first = -1, oob = 0;"] + for a in arrays: + n = a["size"] + lines += [f" CHECK(\"run\", hipMemcpy(h_{a['name']}.data(), d_{a['name']}, h_{a['name']}.size() * " + "sizeof(float), hipMemcpyDeviceToHost));", + f" for (long i = ({n}); i < (long)h_{a['name']}.size(); ++i) if (h_{a['name']}[i] != {GUARD_VALUE}) " + "++oob;"] + if a["role"] != "in": + lines += [f" for (long i = 0; i < ({n}); ++i) {{", + f" float got = h_{a['name']}[i], want = r_{a['name']}[i];", + f" bool same = (std::isnan(got) && std::isnan(want)) || std::fabs(got - want) <= " + f"{_f(task['atol'])} + {_f(task['rtol'])} * std::fabs(want);", + " ++checked;", + " if (!same) { if (first < 0) first = checked - 1; ++mismatches; }", + " }"] + lines += [" const char* status = oob ? \"out_of_bounds\" : (mismatches ? \"mismatch\" : \"ok\");", + " char b_[512];", + " std::snprintf(b_, sizeof b_, \"{\\\"status\\\": \\\"%s\\\", \\\"checked\\\": %ld, " + "\\\"mismatches\\\": %ld, \\\"first\\\": %ld, \\\"out_of_bounds\\\": %ld}\", status, checked, " + "mismatches, first, oob);", + " put(b_);", " return 0;", "}", ""] + return "\n".join(lines) + + +def read_record(path: str | Path) -> dict | None: + """The harness's record file, or None (absent, unreadable, or not a record).""" + try: + out = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return out if isinstance(out, dict) and isinstance(out.get("status", out.get("stage")), str) else None + + +def verdict(run, record) -> tuple[str, str | None, list[dict]]: + """A leg's (state, reason, findings) from the harness's run (a ProcResult) and its record (read_record). The + verdict comes only from the record the harness writes, never from the candidate's output, and only what is the + candidate's is blamed on it: a code object without the task's kernel (bad_signature), a wrong output, and a crash + or a hang once the kernel was launched. A failure before the launch (an allocation, the first copy, loading the + code object) is the node's, and leaves the leg unsettled, never clean.""" + if run.launch_error or run.killed: + return "unsettled", "run_not_started", [] + record = record or {} + status, stage = record.get("status"), record.get("stage") + if status == "no_kernel": + return "fail", "bad_signature", [] + if status == "hip_error": + return ("unsettled", "setup_failed", []) if stage == "setup" else ("fail", "crash", []) + if status in ("mismatch", "out_of_bounds"): + return "fail", "wrong_output", [] + if status == "ok": + if run.returncode == 0 and not run.timed_out and isinstance(record.get("checked"), int) \ + and record["checked"] > 0: + return "clean", None, [] + return "unsettled", "no_result", [] + if stage == "launched": + if run.timed_out: + return "fail", "timeout", [] + if run.returncode not in (0, None): + return "fail", "crash", [] + if stage == "setup" or not record: + return "unsettled", ("setup_timeout" if run.timed_out else "setup_failed"), [] + return "unsettled", "no_result", [] + + +_LEASE_LOCK = threading.Lock() +_LEASED: dict[int, int] = {} # GPU -> live hipk.run adapters on it, in this process + + +def lease_gpu(gpus: list[int], preferred: int) -> int: + """A GPU of `gpus` for a new worker: `preferred` if no live worker has it, else the first free one, else the + least shared. The service retires and replaces workers under new ids, so an id says nothing about which GPUs + the live workers hold.""" + with _LEASE_LOCK: + free = [g for g in gpus if not _LEASED.get(g)] + gpu = preferred if preferred in free else (free[0] if free else min(gpus, key=lambda g: _LEASED.get(g, 0))) + _LEASED[gpu] = _LEASED.get(gpu, 0) + 1 + return gpu + + +def release_gpu(gpu: int) -> None: + with _LEASE_LOCK: + n = _LEASED.get(gpu, 0) - 1 + if n > 0: + _LEASED[gpu] = n + else: + _LEASED.pop(gpu, None) + + +class HipkRun: + """`hipk.run`, leg `ref.native`: the candidate compiled alone into a device code object (hipcc --genco, settings + `hipcc`, leg setting `target`), so none of its host code is built or run; the task's harness, built from the task + alone, loads it and runs its kernel on this worker's GPU (settings `gpus`, leased to live workers: see + `lease_gpu`; by HIP_VISIBLE_DEVICES) for at most `run_timeout_s`, and compares its outputs with the host reference. + Clean only when every output element matches and nothing was written past an array's end. `version` names the + compiler hipcc runs and this file, so a leg cached under another compiler or harness is not reused.""" + + legs = ("ref.native",) + resource = "own_gpu" + + def __init__(self, ctx): + from aorlta.reward.adapters.base import ProcessRunner + from aorlta.reward.adapters.toolchain import toolchain_identity + + self.name, self.ctx = ctx.name, ctx + s = ctx.settings + self.hipcc = s.get("hipcc") or "/opt/rocm/bin/hipcc" + self.version = f"2+{toolchain_identity(self.hipcc)}+{hashlib.sha256(Path(__file__).read_bytes()).hexdigest()[:16]}" + gpus = [int(g) for g in (s.get("gpus") or [0])] + tail = ctx.worker_id.rsplit("/", 1)[-1] + seq = int(tail) if tail.isdigit() else 1 + self.gpu = lease_gpu(gpus, gpus[(seq - 1) % len(gpus)]) + self._leased = True + self.tasks = {} + for pool in ctx.pools: + if (Path(pool) / "tasks.json").is_file(): + for task in load_tasks(pool): + self.tasks[task_sha256(task)] = task + self.runner = ProcessRunner() + self._harness_dir: Path | None = None + self._harnesses: dict[tuple[str, str], Path] = {} + + def _harness(self, task: dict, target: str, timeout_s: float, evidence: Path) -> Path | None: + """The task's harness program for `target`, built once per worker (its source is also written to + `evidence`); None if it does not build.""" + src = harness_source(task) + (evidence / "harness.hip").write_text(src, encoding="utf-8") + key = (task_sha256(task), target) + if key in self._harnesses: + return self._harnesses[key] + if self._harness_dir is None: + self._harness_dir = Path(tempfile.mkdtemp(prefix="hipk-harness-")) + d = self._harness_dir / f"{key[0][:16]}-{target}" + d.mkdir(exist_ok=True) + (d / "harness.hip").write_text(src, encoding="utf-8") + build = self.runner.run([self.hipcc, f"--offload-arch={target}", "-O2", "-o", str(d / "harness"), + str(d / "harness.hip")], timeout_s=timeout_s, cwd=str(d)) + (evidence / "harness-build.log").write_text(build.stderr[-20000:], encoding="utf-8") + if build.returncode != 0 or build.timed_out or build.killed or build.launch_error: + return None + self._harnesses[key] = d / "harness" + return d / "harness" + + def run(self, job): + from aorlta.reward.adapters.base import make_result, run_dir, unsettled + + t0 = self.ctx.clock() + try: + task = self.tasks.get(job.task_sha256) + source = job.inputs.get("source") + if job.leg != "ref.native" or task is None or not source or not job.output_dir: + return unsettled(job, "adapter_error", t_start=t0, t_end=self.ctx.clock()) + d = run_dir(job) + target = job.leg_settings.get("target", "gfx950") + budget = max(1.0, self.ctx.timeout_s) + t_build = time.monotonic() + co = self.runner.run([self.hipcc, "--genco", f"--offload-arch={target}", "-O2", "-o", + str(d / "candidate.co"), str(source)], timeout_s=budget, cwd=str(d)) + (d / "build.log").write_text(co.stderr[-20000:], encoding="utf-8") + if co.returncode != 0 or co.timed_out or co.killed or co.launch_error: # it compiled in gate.compile + return unsettled(job, "candidate_build_failed", t_start=t0, t_end=self.ctx.clock()) + harness = self._harness(task, target, max(1.0, budget - (time.monotonic() - t_build)), d) + if harness is None: + return unsettled(job, "harness_build_failed", t_start=t0, t_end=self.ctx.clock()) + left = budget - (time.monotonic() - t_build) + limit = min(float(job.leg_settings.get("run_timeout_s", 20.0)), left) + if limit <= 0: + return unsettled(job, "no_time_left", t_start=t0, t_end=self.ctx.clock()) + env = dict(os.environ, HIP_VISIBLE_DEVICES=str(self.gpu)) + record = d / "record.json" + run = self.runner.run([str(harness), str(d / "candidate.co"), str(record)], timeout_s=limit, env=env, + cwd=str(d)) + (d / "run.log").write_text((run.stdout + "\n" + run.stderr)[-20000:], encoding="utf-8") + state, reason, findings = verdict(run, read_record(record)) + return make_result(job, state, reason, findings=findings, t_start=t0, t_end=self.ctx.clock()) + except Exception: # noqa: BLE001 - an adapter never raises + return unsettled(job, "adapter_error", t_start=t0, t_end=self.ctx.clock()) + + def controls(self): + return [] + + def health(self): + return {"name": self.name, "responsive": True, "gpu": self.gpu, "tasks": len(self.tasks)} + + def close(self): + self.runner.close() + if self._leased: + self._leased = False + release_gpu(self.gpu) + if self._harness_dir is not None: + shutil.rmtree(self._harness_dir, ignore_errors=True) + + +class HipkGate: + """`hipk.gate`, leg `gate.parse`: the reward's parse gate with the rollout's own rule (`hipk.parse`, which is + the stand-in `parse_code`): the last ```cpp block, and none when a later block never closes. The stand-in gate + takes the last complete block, so a final reply whose last block was cut off would be scored on an earlier one + the rollout had rejected. Writes `source.hip` as the stand-in gate does.""" + + name = "hipk.gate" + version = "1" + legs = ("gate.parse",) + resource = "local_cpu" + + def __init__(self, ctx=None): + self._clock = ctx.clock if ctx is not None else time.time + + def run(self, job): + from aorlta.atomic import atomic_write_bytes + from aorlta.reward.adapters.base import make_result, unsettled + from aorlta.rollout.compile_check import parse_code + + t0 = self._clock() + try: + response = job.inputs.get("response") or (str(Path(job.package) / "response.txt") if job.package else None) + if job.leg != "gate.parse" or not response or not job.output_dir: + return unsettled(job, "adapter_error", t_start=t0, t_end=self._clock()) + source = parse_code(Path(response).read_text(encoding="utf-8"), "off") + if source is None: + return make_result(job, "fail", "parse", t_start=t0, t_end=self._clock()) + out = Path(job.output_dir) + out.mkdir(parents=True, exist_ok=True) + atomic_write_bytes(out / "source.hip", source.encode("utf-8")) + return make_result(job, "clean", t_start=t0, t_end=self._clock()) + except Exception: # noqa: BLE001 - an adapter never raises + return unsettled(job, "adapter_error", t_start=t0, t_end=self._clock()) + + def controls(self): + return [] + + def health(self): + return {"name": self.name, "responsive": True} + + +# --- the reward -------------------------------------------------------------------------------- + + +class HipkV1: + """`hipk.v1`: -1.0 when the parse gate fails, -0.9 for a compile error, -0.8 when the kernel's signature is not + the one asked for, -0.6 for a crash or a hang, -0.5 for a wrong output, +1.0 when every output matches; then + -0.05 x the share of the token budget the answer used. A leg that did not settle leaves the sample unscored: + an unscored sample is never counted clean. The parts are the service's (`base`, `broke_healthy` (always 0), + `length`), so the reward stays within its range.""" + + name = "hipk.v1" + version = "1" + legs = ("gate.parse", "gate.compile", "ref.native") + RUN = {"wrong_output": -0.5, "crash": -0.6, "timeout": -0.6, "bad_signature": -0.8} + + def __init__(self, profile=None): + from aorlta.reward.rewardfn.base import code_sha256 + + self.settings = {} + self.code_sha256 = code_sha256(__file__) + + def _base(self, legs) -> tuple[str, float | str]: + """("score", base), ("unscored", reason) or ("pending", leg), reading the legs in order.""" + for name in self.legs: + if name not in legs: + return "pending", name + leg = legs[name] + if leg.state in ("unsettled", "skipped_by_plan", "finding", "not_run"): + return "unscored", f"{name}:{leg.reason or leg.state}" + if leg.state == "fail": + if name == "gate.parse": + return "score", -1.0 + if name == "gate.compile": + return "score", -0.9 if leg.reason == "compile_error" else -1.0 + if leg.reason in self.RUN: + return "score", self.RUN[leg.reason] + return "unscored", f"{name}:{leg.reason}" + return "score", 1.0 + + @staticmethod + def _finish(base, cand): + length = -0.05 * (cand.response_tokens / cand.max_new_tokens) + parts = {"base": float(base), "broke_healthy": 0.0, "length": length} + return parts["base"] + parts["broke_healthy"] + parts["length"], parts + + def compute(self, legs, cand): + kind, value = self._base(legs) + if kind != "score": + return None, {"reason": str(value)} + return self._finish(value, cand) + + def decide(self, partial, cand): + from aorlta.reward.rewardfn.base import Decided, NeedMore, Unscored + + kind, value = self._base(partial) + if kind == "pending": + return NeedMore((value,)) + if kind == "unscored": + return Unscored(value) + return Decided(*self._finish(value, cand)) + + @staticmethod + def claims_clean(reward, parts): + return reward is not None and isinstance(parts, dict) and parts.get("base", 0.0) > 0.0 + + +# --- registration ------------------------------------------------------------------------------ + + +def register() -> None: + from aorlta import plugins + + plugins.register_task(plugins.Task("hipk", records)) + plugins.register_parse_adapter("hipk.parse", HipkParse) + plugins.register_feedback_template("hipk.feedback", HipkFeedback) + plugins.register_profile("hipk-gpu", HERE / "hipk-gpu.yaml") + plugins.register_check_adapter("hipk.run", HipkRun) + plugins.register_check_adapter("hipk.gate", HipkGate) + plugins.register_reward_fn("hipk.v1", HipkV1) + + +if __name__ != "__main__": + register() + + +# --- the self-check ---------------------------------------------------------------------------- + + +def selfcheck(argv=None) -> int: + """Each task's reference kernel (reference/.hip), or with --sources DIR the kernel DIR/.hip where there + is one, through the check's own steps: compiled alone into a code object, the task's harness built, and unless + --build-only run on the GPU and judged by `verdict`. Prints one JSON line per task; exit 0 when all are clean + (built, with --build-only). A task with no reference kernel fails; with --sources a task with no source is + skipped, and a run that checked no task at all fails.""" + import subprocess + from types import SimpleNamespace + + def run(argv, timeout_s, env=None): # what `verdict` reads of a ProcResult; standalone, so no aorlta import + try: + p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout_s, env=env) + return SimpleNamespace(returncode=p.returncode, stdout=p.stdout, stderr=p.stderr, timed_out=False, + killed=False, launch_error=None) + except subprocess.TimeoutExpired as exc: + out = exc.stdout.decode(errors="replace") if isinstance(exc.stdout, bytes) else (exc.stdout or "") + return SimpleNamespace(returncode=-9, stdout=out, stderr="", timed_out=True, killed=False, + launch_error=None) + + ap = argparse.ArgumentParser(prog="hip_kernels.py selfcheck") + ap.add_argument("--hipcc", default="/opt/rocm/bin/hipcc") + ap.add_argument("--arch", default="gfx950") + ap.add_argument("--gpu", type=int, default=0) + ap.add_argument("--pool", default=str(HERE)) + ap.add_argument("--sources", default=None, help="a directory of .hip kernels to check instead") + ap.add_argument("--build-only", action="store_true", help="compile the harnesses, run nothing (no GPU)") + args = ap.parse_args(argv) + failed = checked = 0 + work = Path(tempfile.mkdtemp(prefix="hipk-selfcheck-")) + try: + for task in load_tasks(args.pool): + src = Path(args.sources or HERE / "reference") / f"{task['id']}.hip" + if not src.is_file(): + if args.sources is None: + failed += 1 + print(json.dumps({"task": task["id"], "source": str(src), "pass": False, + "error": "no reference kernel"}), flush=True) + continue + checked += 1 + d = work / task["id"] + d.mkdir() + (d / "harness.hip").write_text(harness_source(task), encoding="utf-8") + co = run([args.hipcc, "--genco", f"--offload-arch={args.arch}", "-O2", "-o", str(d / "candidate.co"), + str(src)], 300) + hb = run([args.hipcc, f"--offload-arch={args.arch}", "-O2", "-o", str(d / "harness"), + str(d / "harness.hip")], 300) + row = {"task": task["id"], "source": str(src), "build_rc": [co.returncode, hb.returncode]} + built = co.returncode == 0 and hb.returncode == 0 + if built and not args.build_only: + res = run([str(d / "harness"), str(d / "candidate.co"), str(d / "record.json")], 60, + env=dict(os.environ, HIP_VISIBLE_DEVICES=str(args.gpu))) + row["record"] = read_record(d / "record.json") + row["stdout"] = res.stdout[-500:] + row["verdict"] = verdict(res, row["record"])[:2] + ok = row["verdict"][0] == "clean" + else: + ok = built + if not ok: + row["stderr"] = (co.stderr + hb.stderr)[-2000:] + row["pass"] = ok + failed += not ok + print(json.dumps(row), flush=True) + finally: + shutil.rmtree(work, ignore_errors=True) + if not checked: + print(f"selfcheck: no task of {args.pool} had a kernel to check", file=sys.stderr) + return 1 if failed or not checked else 0 + + +if __name__ == "__main__": + if len(sys.argv) > 1 and sys.argv[1] == "selfcheck": + sys.exit(selfcheck(sys.argv[2:])) + print(__doc__) + sys.exit(2) diff --git a/aorlta/examples/hip-kernels/hipk-gpu.yaml b/aorlta/examples/hip-kernels/hipk-gpu.yaml new file mode 100644 index 000000000..5d49a121d --- /dev/null +++ b/aorlta/examples/hip-kernels/hipk-gpu.yaml @@ -0,0 +1,27 @@ +# `hipk-gpu`: the example task's checks (examples/hip-kernels/hip_kernels.py). `hipk.gate` (the parse gate with the +# rollout's own rule) and the stand-in compile gate, then `ref.native`: the candidate compiled alone into a code object, +# run by its task's harness on one of the GPUs in `hipk.run.gpus`, its outputs compared with a host reference. Scored +# by hipk.v1. +name: hipk-gpu +reward_fn: hipk.v1 +clean_repeats: 1 +materialize: true +pools: + gates: {resource: local_cpu, adapter: hipk.gate, timeout: gate, workers: 4} + compile: {resource: local_cpu, adapter: standin.hipcc, timeout: compile, workers: 8} + run: {resource: own_gpu, adapter: hipk.run, timeout: sim_run, workers: 4} +stages: + - {name: S0, pool: gates, legs: [gate.parse]} + - {name: S1, pool: compile, legs: [gate.compile]} + - {name: S2, pool: run, legs: [ref.native]} +requires: + gate.compile: [gate.parse] + ref.native: [gate.compile] +leg_settings: + gate.compile: {offload_arch: gfx950, flags: ["-c", "-O1"]} + ref.native: {target: gfx950, run_timeout_s: 20} +# `${AORLTA_ROCM_ROOT}` is that environment variable, or /opt/rocm. Each live worker of the run pool holds one of +# `gpus` (a replaced worker's successor takes the one it gave back): keep these off the engines' GPUs (rollout.engines). +adapters: + standin.hipcc: {hipcc: "${AORLTA_ROCM_ROOT}/bin/hipcc", readelf: "${AORLTA_ROCM_ROOT}/llvm/bin/llvm-readelf"} + hipk.run: {hipcc: "${AORLTA_ROCM_ROOT}/bin/hipcc", gpus: [4, 5, 6, 7]} diff --git a/aorlta/examples/hip-kernels/reference/block_sum.hip b/aorlta/examples/hip-kernels/reference/block_sum.hip new file mode 100644 index 000000000..4607e3bf7 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/block_sum.hip @@ -0,0 +1,16 @@ +#include + +// One block of 256 threads per 256 elements: each thread loads one element (0 past the end), then a tree reduction +// in shared memory halves the active threads each step, with a barrier between steps. +__global__ void block_sum(const float* x, float* out, int n) { + __shared__ float s[256]; + int t = threadIdx.x; + int i = blockIdx.x * 256 + t; + s[t] = i < n ? x[i] : 0.0f; + __syncthreads(); + for (int stride = 128; stride > 0; stride >>= 1) { + if (t < stride) s[t] += s[t + stride]; + __syncthreads(); + } + if (t == 0) out[blockIdx.x] = s[0]; +} diff --git a/aorlta/examples/hip-kernels/reference/clamp.hip b/aorlta/examples/hip-kernels/reference/clamp.hip new file mode 100644 index 000000000..545e9f613 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/clamp.hip @@ -0,0 +1,6 @@ +#include + +__global__ void clamp(const float* x, float* y, float lo, float hi, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = fminf(fmaxf(x[i], lo), hi); +} diff --git a/aorlta/examples/hip-kernels/reference/relu.hip b/aorlta/examples/hip-kernels/reference/relu.hip new file mode 100644 index 000000000..f1bec822b --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/relu.hip @@ -0,0 +1,6 @@ +#include + +__global__ void relu(const float* x, float* y, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = x[i] > 0.0f ? x[i] : 0.0f; +} diff --git a/aorlta/examples/hip-kernels/reference/reverse.hip b/aorlta/examples/hip-kernels/reference/reverse.hip new file mode 100644 index 000000000..aaa2fbf43 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/reverse.hip @@ -0,0 +1,6 @@ +#include + +__global__ void reverse(const float* x, float* y, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = x[n - 1 - i]; +} diff --git a/aorlta/examples/hip-kernels/reference/saxpy.hip b/aorlta/examples/hip-kernels/reference/saxpy.hip new file mode 100644 index 000000000..ba56c006f --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/saxpy.hip @@ -0,0 +1,6 @@ +#include + +__global__ void saxpy(float alpha, const float* x, float* y, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = alpha * x[i] + y[i]; +} diff --git a/aorlta/examples/hip-kernels/reference/scale_shift.hip b/aorlta/examples/hip-kernels/reference/scale_shift.hip new file mode 100644 index 000000000..2e3e3a353 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/scale_shift.hip @@ -0,0 +1,6 @@ +#include + +__global__ void scale_shift(const float* x, float* y, float scale, float shift, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) y[i] = x[i] * scale + shift; +} diff --git a/aorlta/examples/hip-kernels/reference/transpose.hip b/aorlta/examples/hip-kernels/reference/transpose.hip new file mode 100644 index 000000000..83c7a4738 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/transpose.hip @@ -0,0 +1,7 @@ +#include + +__global__ void transpose(const float* in, float* out, int rows, int cols) { + int c = blockIdx.x * blockDim.x + threadIdx.x; + int r = blockIdx.y * blockDim.y + threadIdx.y; + if (r < rows && c < cols) out[c * rows + r] = in[r * cols + c]; +} diff --git a/aorlta/examples/hip-kernels/reference/vector_add.hip b/aorlta/examples/hip-kernels/reference/vector_add.hip new file mode 100644 index 000000000..967b8a668 --- /dev/null +++ b/aorlta/examples/hip-kernels/reference/vector_add.hip @@ -0,0 +1,6 @@ +#include + +__global__ void vector_add(const float* a, const float* b, float* c, int n) { + int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) c[i] = a[i] + b[i]; +} diff --git a/aorlta/examples/hip-kernels/tasks.json b/aorlta/examples/hip-kernels/tasks.json new file mode 100644 index 000000000..7174ea1da --- /dev/null +++ b/aorlta/examples/hip-kernels/tasks.json @@ -0,0 +1,99 @@ +[ + { + "id": "vector_add", + "kernel": "vector_add", + "signature": "__global__ void vector_add(const float* a, const float* b, float* c, int n)", + "description": "c[i] = a[i] + b[i] for every 0 <= i < n", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"]], + "arrays": [{"name": "a", "size": "n", "role": "in"}, {"name": "b", "size": "n", "role": "in"}, + {"name": "c", "size": "n", "role": "out"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["a", "b", "c", "n"], + "reference": "for (int i = 0; i < n; ++i) c[i] = a[i] + b[i];", + "rtol": 0.0, "atol": 0.0 + }, + { + "id": "saxpy", + "kernel": "saxpy", + "signature": "__global__ void saxpy(float alpha, const float* x, float* y, int n)", + "description": "y[i] = alpha * x[i] + y[i] for every 0 <= i < n, in place", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"], ["float", "alpha", "2.5f"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "y", "size": "n", "role": "inout"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["alpha", "x", "y", "n"], + "reference": "for (int i = 0; i < n; ++i) y[i] = alpha * x[i] + y[i];", + "rtol": 1e-05, "atol": 1e-06 + }, + { + "id": "relu", + "kernel": "relu", + "signature": "__global__ void relu(const float* x, float* y, int n)", + "description": "y[i] = max(x[i], 0) for every 0 <= i < n", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "y", "size": "n", "role": "out"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["x", "y", "n"], + "reference": "for (int i = 0; i < n; ++i) y[i] = x[i] > 0.0f ? x[i] : 0.0f;", + "rtol": 0.0, "atol": 0.0 + }, + { + "id": "scale_shift", + "kernel": "scale_shift", + "signature": "__global__ void scale_shift(const float* x, float* y, float scale, float shift, int n)", + "description": "y[i] = x[i] * scale + shift for every 0 <= i < n", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"], ["float", "scale", "-1.75f"], ["float", "shift", "0.25f"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "y", "size": "n", "role": "out"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["x", "y", "scale", "shift", "n"], + "reference": "for (int i = 0; i < n; ++i) y[i] = x[i] * scale + shift;", + "rtol": 1e-05, "atol": 1e-06 + }, + { + "id": "clamp", + "kernel": "clamp", + "signature": "__global__ void clamp(const float* x, float* y, float lo, float hi, int n)", + "description": "y[i] = min(max(x[i], lo), hi) for every 0 <= i < n", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"], ["float", "lo", "-0.5f"], ["float", "hi", "0.75f"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "y", "size": "n", "role": "out"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["x", "y", "lo", "hi", "n"], + "reference": "for (int i = 0; i < n; ++i) y[i] = x[i] < lo ? lo : (x[i] > hi ? hi : x[i]);", + "rtol": 0.0, "atol": 0.0 + }, + { + "id": "reverse", + "kernel": "reverse", + "signature": "__global__ void reverse(const float* x, float* y, int n)", + "description": "y[i] = x[n - 1 - i] for every 0 <= i < n", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one thread per element", + "scalars": [["int", "n", "100003"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "y", "size": "n", "role": "out"}], + "grid": "dim3((n + 255) / 256)", "block": "dim3(256)", "args": ["x", "y", "n"], + "reference": "for (int i = 0; i < n; ++i) y[i] = x[n - 1 - i];", + "rtol": 0.0, "atol": 0.0 + }, + { + "id": "block_sum", + "kernel": "block_sum", + "signature": "__global__ void block_sum(const float* x, float* out, int n)", + "description": "out[b] = the sum of x[i] over b * 256 <= i < min((b + 1) * 256, n), for every block b", + "launch": "with blockDim.x = 256 and gridDim.x = (n + 255) / 256, one block per 256 elements", + "scalars": [["int", "n", "100003"], ["int", "blocks", "(n + 255) / 256"]], + "arrays": [{"name": "x", "size": "n", "role": "in"}, {"name": "out", "size": "blocks", "role": "out"}], + "grid": "dim3(blocks)", "block": "dim3(256)", "args": ["x", "out", "n"], + "reference": "for (int b = 0; b < blocks; ++b) { double s = 0.0; for (int i = b * 256; i < (b + 1) * 256 && i < n; ++i) s += x[i]; out[b] = (float)s; }", + "rtol": 0.0001, "atol": 0.002 + }, + { + "id": "transpose", + "kernel": "transpose", + "signature": "__global__ void transpose(const float* in, float* out, int rows, int cols)", + "description": "out[c * rows + r] = in[r * cols + c] for every 0 <= r < rows and 0 <= c < cols: `in` is a rows x cols row-major matrix and `out` its cols x rows transpose", + "launch": "with blockDim = (16, 16) and gridDim = ((cols + 15) / 16, (rows + 15) / 16), where threadIdx.x and blockIdx.x run along the columns of `in`", + "scalars": [["int", "rows", "300"], ["int", "cols", "517"]], + "arrays": [{"name": "in", "size": "rows * cols", "role": "in"}, {"name": "out", "size": "rows * cols", "role": "out"}], + "grid": "dim3((cols + 15) / 16, (rows + 15) / 16)", "block": "dim3(16, 16)", "args": ["in", "out", "rows", "cols"], + "reference": "for (int r = 0; r < rows; ++r) for (int c = 0; c < cols; ++c) out[c * rows + r] = in[r * cols + c];", + "rtol": 0.0, "atol": 0.0 + } +] diff --git a/aorlta/examples/qa/README.md b/aorlta/examples/qa/README.md new file mode 100644 index 000000000..1ddf46da1 --- /dev/null +++ b/aorlta/examples/qa/README.md @@ -0,0 +1,15 @@ +# qa: questions with reference answers, nothing compiled + +The second worked example of a task as a plugin ([docs/TUTORIAL.md](../../docs/TUTORIAL.md)): a question, a reference +answer, and a reward for matching it, exactly or after normalising, with an optional judge command. It runs on one +GPU and needs no compile step and no sanitizer. + +| file | what | +|---|---| +| `qa_task.py` | the plugin: prompts, parse and feedback, the `ref.answer` check, the `qa.v1` reward | +| `qa-match.yaml` | the check profile: one CPU leg | +| `gpu-facts/{train,heldout}/qa.jsonl` | 24 questions on GPU programming | +| `arith/`, `arith4/` | 3-digit and 4-digit multiplications, written by `make_arith.py` | + +Configs: `configs/qa-{train,score}.yaml`, `configs/arith-{train,score}.yaml`, `configs/arith4-{train,score}.yaml`. +Tests: `tests/examples/test_qa.py`. diff --git a/aorlta/examples/qa/arith/heldout/qa.jsonl b/aorlta/examples/qa/arith/heldout/qa.jsonl new file mode 100644 index 000000000..1cb6d356e --- /dev/null +++ b/aorlta/examples/qa/arith/heldout/qa.jsonl @@ -0,0 +1,64 @@ +{"id": "mul-406x299", "question": "What is 406 \u00d7 299?", "answer": "121394"} +{"id": "mul-496x899", "question": "What is 496 \u00d7 899?", "answer": "445904"} +{"id": "mul-944x819", "question": "What is 944 \u00d7 819?", "answer": "773136"} +{"id": "mul-444x869", "question": "What is 444 \u00d7 869?", "answer": "385836"} +{"id": "mul-847x546", "question": "What is 847 \u00d7 546?", "answer": "462462"} +{"id": "mul-956x504", "question": "What is 956 \u00d7 504?", "answer": "481824"} +{"id": "mul-782x805", "question": "What is 782 \u00d7 805?", "answer": "629510"} +{"id": "mul-445x536", "question": "What is 445 \u00d7 536?", "answer": "238520"} +{"id": "mul-142x926", "question": "What is 142 \u00d7 926?", "answer": "131492"} +{"id": "mul-726x260", "question": "What is 726 \u00d7 260?", "answer": "188760"} +{"id": "mul-602x104", "question": "What is 602 \u00d7 104?", "answer": "62608"} +{"id": "mul-128x109", "question": "What is 128 \u00d7 109?", "answer": "13952"} +{"id": "mul-761x682", "question": "What is 761 \u00d7 682?", "answer": "519002"} +{"id": "mul-298x100", "question": "What is 298 \u00d7 100?", "answer": "29800"} +{"id": "mul-496x554", "question": "What is 496 \u00d7 554?", "answer": "274784"} +{"id": "mul-956x261", "question": "What is 956 \u00d7 261?", "answer": "249516"} +{"id": "mul-860x666", "question": "What is 860 \u00d7 666?", "answer": "572760"} +{"id": "mul-183x402", "question": "What is 183 \u00d7 402?", "answer": "73566"} +{"id": "mul-238x276", "question": "What is 238 \u00d7 276?", "answer": "65688"} +{"id": "mul-949x287", "question": "What is 949 \u00d7 287?", "answer": "272363"} +{"id": "mul-636x916", "question": "What is 636 \u00d7 916?", "answer": "582576"} +{"id": "mul-887x190", "question": "What is 887 \u00d7 190?", "answer": "168530"} +{"id": "mul-762x522", "question": "What is 762 \u00d7 522?", "answer": "397764"} +{"id": "mul-637x283", "question": "What is 637 \u00d7 283?", "answer": "180271"} +{"id": "mul-698x545", "question": "What is 698 \u00d7 545?", "answer": "380410"} +{"id": "mul-207x233", "question": "What is 207 \u00d7 233?", "answer": "48231"} +{"id": "mul-649x493", "question": "What is 649 \u00d7 493?", "answer": "319957"} +{"id": "mul-557x258", "question": "What is 557 \u00d7 258?", "answer": "143706"} +{"id": "mul-202x706", "question": "What is 202 \u00d7 706?", "answer": "142612"} +{"id": "mul-400x562", "question": "What is 400 \u00d7 562?", "answer": "224800"} +{"id": "mul-144x111", "question": "What is 144 \u00d7 111?", "answer": "15984"} +{"id": "mul-314x520", "question": "What is 314 \u00d7 520?", "answer": "163280"} +{"id": "mul-788x853", "question": "What is 788 \u00d7 853?", "answer": "672164"} +{"id": "mul-693x882", "question": "What is 693 \u00d7 882?", "answer": "611226"} +{"id": "mul-624x832", "question": "What is 624 \u00d7 832?", "answer": "519168"} +{"id": "mul-223x474", "question": "What is 223 \u00d7 474?", "answer": "105702"} +{"id": "mul-891x869", "question": "What is 891 \u00d7 869?", "answer": "774279"} +{"id": "mul-482x815", "question": "What is 482 \u00d7 815?", "answer": "392830"} +{"id": "mul-419x835", "question": "What is 419 \u00d7 835?", "answer": "349865"} +{"id": "mul-698x319", "question": "What is 698 \u00d7 319?", "answer": "222662"} +{"id": "mul-587x941", "question": "What is 587 \u00d7 941?", "answer": "552367"} +{"id": "mul-598x624", "question": "What is 598 \u00d7 624?", "answer": "373152"} +{"id": "mul-219x919", "question": "What is 219 \u00d7 919?", "answer": "201261"} +{"id": "mul-313x384", "question": "What is 313 \u00d7 384?", "answer": "120192"} +{"id": "mul-278x909", "question": "What is 278 \u00d7 909?", "answer": "252702"} +{"id": "mul-187x889", "question": "What is 187 \u00d7 889?", "answer": "166243"} +{"id": "mul-550x233", "question": "What is 550 \u00d7 233?", "answer": "128150"} +{"id": "mul-155x784", "question": "What is 155 \u00d7 784?", "answer": "121520"} +{"id": "mul-581x529", "question": "What is 581 \u00d7 529?", "answer": "307349"} +{"id": "mul-871x346", "question": "What is 871 \u00d7 346?", "answer": "301366"} +{"id": "mul-846x875", "question": "What is 846 \u00d7 875?", "answer": "740250"} +{"id": "mul-589x759", "question": "What is 589 \u00d7 759?", "answer": "447051"} +{"id": "mul-367x361", "question": "What is 367 \u00d7 361?", "answer": "132487"} +{"id": "mul-546x218", "question": "What is 546 \u00d7 218?", "answer": "119028"} +{"id": "mul-789x878", "question": "What is 789 \u00d7 878?", "answer": "692742"} +{"id": "mul-524x102", "question": "What is 524 \u00d7 102?", "answer": "53448"} +{"id": "mul-109x877", "question": "What is 109 \u00d7 877?", "answer": "95593"} +{"id": "mul-522x861", "question": "What is 522 \u00d7 861?", "answer": "449442"} +{"id": "mul-810x954", "question": "What is 810 \u00d7 954?", "answer": "772740"} +{"id": "mul-984x627", "question": "What is 984 \u00d7 627?", "answer": "616968"} +{"id": "mul-782x907", "question": "What is 782 \u00d7 907?", "answer": "709274"} +{"id": "mul-955x383", "question": "What is 955 \u00d7 383?", "answer": "365765"} +{"id": "mul-150x263", "question": "What is 150 \u00d7 263?", "answer": "39450"} +{"id": "mul-549x113", "question": "What is 549 \u00d7 113?", "answer": "62037"} diff --git a/aorlta/examples/qa/arith/train/qa.jsonl b/aorlta/examples/qa/arith/train/qa.jsonl new file mode 100644 index 000000000..389559b32 --- /dev/null +++ b/aorlta/examples/qa/arith/train/qa.jsonl @@ -0,0 +1,512 @@ +{"id": "mul-282x118", "question": "What is 282 \u00d7 118?", "answer": "33276"} +{"id": "mul-954x233", "question": "What is 954 \u00d7 233?", "answer": "222282"} +{"id": "mul-472x752", "question": "What is 472 \u00d7 752?", "answer": "354944"} +{"id": "mul-109x568", "question": "What is 109 \u00d7 568?", "answer": "61912"} +{"id": "mul-167x627", "question": "What is 167 \u00d7 627?", "answer": "104709"} +{"id": "mul-235x197", "question": "What is 235 \u00d7 197?", "answer": "46295"} +{"id": "mul-149x500", "question": "What is 149 \u00d7 500?", "answer": "74500"} +{"id": "mul-153x665", "question": "What is 153 \u00d7 665?", "answer": "101745"} +{"id": "mul-212x714", "question": "What is 212 \u00d7 714?", "answer": "151368"} +{"id": "mul-412x131", "question": "What is 412 \u00d7 131?", "answer": "53972"} +{"id": "mul-406x611", "question": "What is 406 \u00d7 611?", "answer": "248066"} +{"id": "mul-114x768", "question": "What is 114 \u00d7 768?", "answer": "87552"} +{"id": "mul-795x861", "question": "What is 795 \u00d7 861?", "answer": "684495"} +{"id": "mul-982x803", "question": "What is 982 \u00d7 803?", "answer": "788546"} +{"id": "mul-410x616", "question": "What is 410 \u00d7 616?", "answer": "252560"} +{"id": "mul-817x544", "question": "What is 817 \u00d7 544?", "answer": "444448"} +{"id": "mul-367x920", "question": "What is 367 \u00d7 920?", "answer": "337640"} +{"id": "mul-148x599", "question": "What is 148 \u00d7 599?", "answer": "88652"} +{"id": "mul-967x880", "question": "What is 967 \u00d7 880?", "answer": "850960"} +{"id": "mul-503x440", "question": "What is 503 \u00d7 440?", "answer": "221320"} +{"id": "mul-800x845", "question": "What is 800 \u00d7 845?", "answer": "676000"} +{"id": "mul-389x237", "question": "What is 389 \u00d7 237?", "answer": "92193"} +{"id": "mul-923x269", "question": "What is 923 \u00d7 269?", "answer": "248287"} +{"id": "mul-979x997", "question": "What is 979 \u00d7 997?", "answer": "976063"} +{"id": "mul-173x465", "question": "What is 173 \u00d7 465?", "answer": "80445"} +{"id": "mul-593x869", "question": "What is 593 \u00d7 869?", "answer": "515317"} +{"id": "mul-917x503", "question": "What is 917 \u00d7 503?", "answer": "461251"} +{"id": "mul-376x905", "question": "What is 376 \u00d7 905?", "answer": "340280"} +{"id": "mul-504x421", "question": "What is 504 \u00d7 421?", "answer": "212184"} +{"id": "mul-637x151", "question": "What is 637 \u00d7 151?", "answer": "96187"} +{"id": "mul-630x550", "question": "What is 630 \u00d7 550?", "answer": "346500"} +{"id": "mul-777x854", "question": "What is 777 \u00d7 854?", "answer": "663558"} +{"id": "mul-527x460", "question": "What is 527 \u00d7 460?", "answer": "242420"} +{"id": "mul-655x577", "question": "What is 655 \u00d7 577?", "answer": "377935"} +{"id": "mul-201x393", "question": "What is 201 \u00d7 393?", "answer": "78993"} +{"id": "mul-957x381", "question": "What is 957 \u00d7 381?", "answer": "364617"} +{"id": "mul-127x100", "question": "What is 127 \u00d7 100?", "answer": "12700"} +{"id": "mul-859x261", "question": "What is 859 \u00d7 261?", "answer": "224199"} +{"id": "mul-817x266", "question": "What is 817 \u00d7 266?", "answer": "217322"} +{"id": "mul-791x789", "question": "What is 791 \u00d7 789?", "answer": "624099"} +{"id": "mul-968x228", "question": "What is 968 \u00d7 228?", "answer": "220704"} +{"id": "mul-177x585", "question": "What is 177 \u00d7 585?", "answer": "103545"} +{"id": "mul-480x247", "question": "What is 480 \u00d7 247?", "answer": "118560"} +{"id": "mul-625x323", "question": "What is 625 \u00d7 323?", "answer": "201875"} +{"id": "mul-495x425", "question": "What is 495 \u00d7 425?", "answer": "210375"} +{"id": "mul-794x347", "question": "What is 794 \u00d7 347?", "answer": "275518"} +{"id": "mul-312x529", "question": "What is 312 \u00d7 529?", "answer": "165048"} +{"id": "mul-433x195", "question": "What is 433 \u00d7 195?", "answer": "84435"} +{"id": "mul-986x694", "question": "What is 986 \u00d7 694?", "answer": "684284"} +{"id": "mul-873x988", "question": "What is 873 \u00d7 988?", "answer": "862524"} +{"id": "mul-193x633", "question": "What is 193 \u00d7 633?", "answer": "122169"} +{"id": "mul-260x944", "question": "What is 260 \u00d7 944?", "answer": "245440"} +{"id": "mul-720x354", "question": "What is 720 \u00d7 354?", "answer": "254880"} +{"id": "mul-830x615", "question": "What is 830 \u00d7 615?", "answer": "510450"} +{"id": "mul-544x935", "question": "What is 544 \u00d7 935?", "answer": "508640"} +{"id": "mul-830x829", "question": "What is 830 \u00d7 829?", "answer": "688070"} +{"id": "mul-732x852", "question": "What is 732 \u00d7 852?", "answer": "623664"} +{"id": "mul-164x928", "question": "What is 164 \u00d7 928?", "answer": "152192"} +{"id": "mul-241x969", "question": "What is 241 \u00d7 969?", "answer": "233529"} +{"id": "mul-118x911", "question": "What is 118 \u00d7 911?", "answer": "107498"} +{"id": "mul-824x727", "question": "What is 824 \u00d7 727?", "answer": "599048"} +{"id": "mul-662x761", "question": "What is 662 \u00d7 761?", "answer": "503782"} +{"id": "mul-714x961", "question": "What is 714 \u00d7 961?", "answer": "686154"} +{"id": "mul-107x960", "question": "What is 107 \u00d7 960?", "answer": "102720"} +{"id": "mul-883x277", "question": "What is 883 \u00d7 277?", "answer": "244591"} +{"id": "mul-517x970", "question": "What is 517 \u00d7 970?", "answer": "501490"} +{"id": "mul-724x783", "question": "What is 724 \u00d7 783?", "answer": "566892"} +{"id": "mul-238x872", "question": "What is 238 \u00d7 872?", "answer": "207536"} +{"id": "mul-642x308", "question": "What is 642 \u00d7 308?", "answer": "197736"} +{"id": "mul-597x418", "question": "What is 597 \u00d7 418?", "answer": "249546"} +{"id": "mul-760x154", "question": "What is 760 \u00d7 154?", "answer": "117040"} +{"id": "mul-758x658", "question": "What is 758 \u00d7 658?", "answer": "498764"} +{"id": "mul-415x101", "question": "What is 415 \u00d7 101?", "answer": "41915"} +{"id": "mul-828x693", "question": "What is 828 \u00d7 693?", "answer": "573804"} +{"id": "mul-319x864", "question": "What is 319 \u00d7 864?", "answer": "275616"} +{"id": "mul-503x466", "question": "What is 503 \u00d7 466?", "answer": "234398"} +{"id": "mul-909x613", "question": "What is 909 \u00d7 613?", "answer": "557217"} +{"id": "mul-481x149", "question": "What is 481 \u00d7 149?", "answer": "71669"} +{"id": "mul-738x883", "question": "What is 738 \u00d7 883?", "answer": "651654"} +{"id": "mul-419x206", "question": "What is 419 \u00d7 206?", "answer": "86314"} +{"id": "mul-526x338", "question": "What is 526 \u00d7 338?", "answer": "177788"} +{"id": "mul-893x894", "question": "What is 893 \u00d7 894?", "answer": "798342"} +{"id": "mul-265x235", "question": "What is 265 \u00d7 235?", "answer": "62275"} +{"id": "mul-636x458", "question": "What is 636 \u00d7 458?", "answer": "291288"} +{"id": "mul-355x946", "question": "What is 355 \u00d7 946?", "answer": "335830"} +{"id": "mul-886x146", "question": "What is 886 \u00d7 146?", "answer": "129356"} +{"id": "mul-739x308", "question": "What is 739 \u00d7 308?", "answer": "227612"} +{"id": "mul-174x433", "question": "What is 174 \u00d7 433?", "answer": "75342"} +{"id": "mul-807x316", "question": "What is 807 \u00d7 316?", "answer": "255012"} +{"id": "mul-673x981", "question": "What is 673 \u00d7 981?", "answer": "660213"} +{"id": "mul-233x721", "question": "What is 233 \u00d7 721?", "answer": "167993"} +{"id": "mul-520x263", "question": "What is 520 \u00d7 263?", "answer": "136760"} +{"id": "mul-760x629", "question": "What is 760 \u00d7 629?", "answer": "478040"} +{"id": "mul-448x177", "question": "What is 448 \u00d7 177?", "answer": "79296"} +{"id": "mul-501x344", "question": "What is 501 \u00d7 344?", "answer": "172344"} +{"id": "mul-120x372", "question": "What is 120 \u00d7 372?", "answer": "44640"} +{"id": "mul-988x168", "question": "What is 988 \u00d7 168?", "answer": "165984"} +{"id": "mul-676x311", "question": "What is 676 \u00d7 311?", "answer": "210236"} +{"id": "mul-955x397", "question": "What is 955 \u00d7 397?", "answer": "379135"} +{"id": "mul-762x789", "question": "What is 762 \u00d7 789?", "answer": "601218"} +{"id": "mul-870x471", "question": "What is 870 \u00d7 471?", "answer": "409770"} +{"id": "mul-610x408", "question": "What is 610 \u00d7 408?", "answer": "248880"} +{"id": "mul-332x650", "question": "What is 332 \u00d7 650?", "answer": "215800"} +{"id": "mul-996x635", "question": "What is 996 \u00d7 635?", "answer": "632460"} +{"id": "mul-813x284", "question": "What is 813 \u00d7 284?", "answer": "230892"} +{"id": "mul-721x760", "question": "What is 721 \u00d7 760?", "answer": "547960"} +{"id": "mul-503x149", "question": "What is 503 \u00d7 149?", "answer": "74947"} +{"id": "mul-860x664", "question": "What is 860 \u00d7 664?", "answer": "571040"} +{"id": "mul-424x582", "question": "What is 424 \u00d7 582?", "answer": "246768"} +{"id": "mul-683x705", "question": "What is 683 \u00d7 705?", "answer": "481515"} +{"id": "mul-234x262", "question": "What is 234 \u00d7 262?", "answer": "61308"} +{"id": "mul-675x203", "question": "What is 675 \u00d7 203?", "answer": "137025"} +{"id": "mul-754x164", "question": "What is 754 \u00d7 164?", "answer": "123656"} +{"id": "mul-799x410", "question": "What is 799 \u00d7 410?", "answer": "327590"} +{"id": "mul-539x458", "question": "What is 539 \u00d7 458?", "answer": "246862"} +{"id": "mul-216x269", "question": "What is 216 \u00d7 269?", "answer": "58104"} +{"id": "mul-720x139", "question": "What is 720 \u00d7 139?", "answer": "100080"} +{"id": "mul-834x562", "question": "What is 834 \u00d7 562?", "answer": "468708"} +{"id": "mul-827x710", "question": "What is 827 \u00d7 710?", "answer": "587170"} +{"id": "mul-823x609", "question": "What is 823 \u00d7 609?", "answer": "501207"} +{"id": "mul-160x599", "question": "What is 160 \u00d7 599?", "answer": "95840"} +{"id": "mul-269x715", "question": "What is 269 \u00d7 715?", "answer": "192335"} +{"id": "mul-183x889", "question": "What is 183 \u00d7 889?", "answer": "162687"} +{"id": "mul-810x258", "question": "What is 810 \u00d7 258?", "answer": "208980"} +{"id": "mul-357x291", "question": "What is 357 \u00d7 291?", "answer": "103887"} +{"id": "mul-740x520", "question": "What is 740 \u00d7 520?", "answer": "384800"} +{"id": "mul-227x543", "question": "What is 227 \u00d7 543?", "answer": "123261"} +{"id": "mul-662x997", "question": "What is 662 \u00d7 997?", "answer": "660014"} +{"id": "mul-254x439", "question": "What is 254 \u00d7 439?", "answer": "111506"} +{"id": "mul-737x125", "question": "What is 737 \u00d7 125?", "answer": "92125"} +{"id": "mul-671x191", "question": "What is 671 \u00d7 191?", "answer": "128161"} +{"id": "mul-207x984", "question": "What is 207 \u00d7 984?", "answer": "203688"} +{"id": "mul-115x193", "question": "What is 115 \u00d7 193?", "answer": "22195"} +{"id": "mul-780x390", "question": "What is 780 \u00d7 390?", "answer": "304200"} +{"id": "mul-410x305", "question": "What is 410 \u00d7 305?", "answer": "125050"} +{"id": "mul-626x710", "question": "What is 626 \u00d7 710?", "answer": "444460"} +{"id": "mul-334x643", "question": "What is 334 \u00d7 643?", "answer": "214762"} +{"id": "mul-310x866", "question": "What is 310 \u00d7 866?", "answer": "268460"} +{"id": "mul-704x235", "question": "What is 704 \u00d7 235?", "answer": "165440"} +{"id": "mul-491x702", "question": "What is 491 \u00d7 702?", "answer": "344682"} +{"id": "mul-369x279", "question": "What is 369 \u00d7 279?", "answer": "102951"} +{"id": "mul-425x415", "question": "What is 425 \u00d7 415?", "answer": "176375"} +{"id": "mul-667x596", "question": "What is 667 \u00d7 596?", "answer": "397532"} +{"id": "mul-207x817", "question": "What is 207 \u00d7 817?", "answer": "169119"} +{"id": "mul-607x955", "question": "What is 607 \u00d7 955?", "answer": "579685"} +{"id": "mul-991x248", "question": "What is 991 \u00d7 248?", "answer": "245768"} +{"id": "mul-664x682", "question": "What is 664 \u00d7 682?", "answer": "452848"} +{"id": "mul-495x695", "question": "What is 495 \u00d7 695?", "answer": "344025"} +{"id": "mul-519x390", "question": "What is 519 \u00d7 390?", "answer": "202410"} +{"id": "mul-168x714", "question": "What is 168 \u00d7 714?", "answer": "119952"} +{"id": "mul-799x888", "question": "What is 799 \u00d7 888?", "answer": "709512"} +{"id": "mul-966x302", "question": "What is 966 \u00d7 302?", "answer": "291732"} +{"id": "mul-377x269", "question": "What is 377 \u00d7 269?", "answer": "101413"} +{"id": "mul-856x106", "question": "What is 856 \u00d7 106?", "answer": "90736"} +{"id": "mul-778x204", "question": "What is 778 \u00d7 204?", "answer": "158712"} +{"id": "mul-725x169", "question": "What is 725 \u00d7 169?", "answer": "122525"} +{"id": "mul-236x512", "question": "What is 236 \u00d7 512?", "answer": "120832"} +{"id": "mul-945x149", "question": "What is 945 \u00d7 149?", "answer": "140805"} +{"id": "mul-124x779", "question": "What is 124 \u00d7 779?", "answer": "96596"} +{"id": "mul-509x441", "question": "What is 509 \u00d7 441?", "answer": "224469"} +{"id": "mul-722x323", "question": "What is 722 \u00d7 323?", "answer": "233206"} +{"id": "mul-643x472", "question": "What is 643 \u00d7 472?", "answer": "303496"} +{"id": "mul-277x151", "question": "What is 277 \u00d7 151?", "answer": "41827"} +{"id": "mul-457x683", "question": "What is 457 \u00d7 683?", "answer": "312131"} +{"id": "mul-870x726", "question": "What is 870 \u00d7 726?", "answer": "631620"} +{"id": "mul-643x361", "question": "What is 643 \u00d7 361?", "answer": "232123"} +{"id": "mul-235x679", "question": "What is 235 \u00d7 679?", "answer": "159565"} +{"id": "mul-512x521", "question": "What is 512 \u00d7 521?", "answer": "266752"} +{"id": "mul-446x302", "question": "What is 446 \u00d7 302?", "answer": "134692"} +{"id": "mul-878x737", "question": "What is 878 \u00d7 737?", "answer": "647086"} +{"id": "mul-106x543", "question": "What is 106 \u00d7 543?", "answer": "57558"} +{"id": "mul-121x906", "question": "What is 121 \u00d7 906?", "answer": "109626"} +{"id": "mul-692x436", "question": "What is 692 \u00d7 436?", "answer": "301712"} +{"id": "mul-690x999", "question": "What is 690 \u00d7 999?", "answer": "689310"} +{"id": "mul-900x153", "question": "What is 900 \u00d7 153?", "answer": "137700"} +{"id": "mul-886x951", "question": "What is 886 \u00d7 951?", "answer": "842586"} +{"id": "mul-781x450", "question": "What is 781 \u00d7 450?", "answer": "351450"} +{"id": "mul-289x920", "question": "What is 289 \u00d7 920?", "answer": "265880"} +{"id": "mul-748x314", "question": "What is 748 \u00d7 314?", "answer": "234872"} +{"id": "mul-805x676", "question": "What is 805 \u00d7 676?", "answer": "544180"} +{"id": "mul-208x754", "question": "What is 208 \u00d7 754?", "answer": "156832"} +{"id": "mul-176x116", "question": "What is 176 \u00d7 116?", "answer": "20416"} +{"id": "mul-297x921", "question": "What is 297 \u00d7 921?", "answer": "273537"} +{"id": "mul-462x611", "question": "What is 462 \u00d7 611?", "answer": "282282"} +{"id": "mul-454x267", "question": "What is 454 \u00d7 267?", "answer": "121218"} +{"id": "mul-234x461", "question": "What is 234 \u00d7 461?", "answer": "107874"} +{"id": "mul-131x859", "question": "What is 131 \u00d7 859?", "answer": "112529"} +{"id": "mul-813x185", "question": "What is 813 \u00d7 185?", "answer": "150405"} +{"id": "mul-433x893", "question": "What is 433 \u00d7 893?", "answer": "386669"} +{"id": "mul-551x994", "question": "What is 551 \u00d7 994?", "answer": "547694"} +{"id": "mul-933x430", "question": "What is 933 \u00d7 430?", "answer": "401190"} +{"id": "mul-992x674", "question": "What is 992 \u00d7 674?", "answer": "668608"} +{"id": "mul-249x929", "question": "What is 249 \u00d7 929?", "answer": "231321"} +{"id": "mul-889x942", "question": "What is 889 \u00d7 942?", "answer": "837438"} +{"id": "mul-849x704", "question": "What is 849 \u00d7 704?", "answer": "597696"} +{"id": "mul-915x196", "question": "What is 915 \u00d7 196?", "answer": "179340"} +{"id": "mul-335x212", "question": "What is 335 \u00d7 212?", "answer": "71020"} +{"id": "mul-400x754", "question": "What is 400 \u00d7 754?", "answer": "301600"} +{"id": "mul-894x349", "question": "What is 894 \u00d7 349?", "answer": "312006"} +{"id": "mul-537x994", "question": "What is 537 \u00d7 994?", "answer": "533778"} +{"id": "mul-247x827", "question": "What is 247 \u00d7 827?", "answer": "204269"} +{"id": "mul-694x848", "question": "What is 694 \u00d7 848?", "answer": "588512"} +{"id": "mul-958x286", "question": "What is 958 \u00d7 286?", "answer": "273988"} +{"id": "mul-837x478", "question": "What is 837 \u00d7 478?", "answer": "400086"} +{"id": "mul-744x777", "question": "What is 744 \u00d7 777?", "answer": "578088"} +{"id": "mul-607x827", "question": "What is 607 \u00d7 827?", "answer": "501989"} +{"id": "mul-655x706", "question": "What is 655 \u00d7 706?", "answer": "462430"} +{"id": "mul-406x346", "question": "What is 406 \u00d7 346?", "answer": "140476"} +{"id": "mul-412x901", "question": "What is 412 \u00d7 901?", "answer": "371212"} +{"id": "mul-390x951", "question": "What is 390 \u00d7 951?", "answer": "370890"} +{"id": "mul-639x270", "question": "What is 639 \u00d7 270?", "answer": "172530"} +{"id": "mul-428x424", "question": "What is 428 \u00d7 424?", "answer": "181472"} +{"id": "mul-516x542", "question": "What is 516 \u00d7 542?", "answer": "279672"} +{"id": "mul-229x503", "question": "What is 229 \u00d7 503?", "answer": "115187"} +{"id": "mul-719x576", "question": "What is 719 \u00d7 576?", "answer": "414144"} +{"id": "mul-190x334", "question": "What is 190 \u00d7 334?", "answer": "63460"} +{"id": "mul-875x134", "question": "What is 875 \u00d7 134?", "answer": "117250"} +{"id": "mul-771x793", "question": "What is 771 \u00d7 793?", "answer": "611403"} +{"id": "mul-910x657", "question": "What is 910 \u00d7 657?", "answer": "597870"} +{"id": "mul-122x642", "question": "What is 122 \u00d7 642?", "answer": "78324"} +{"id": "mul-918x774", "question": "What is 918 \u00d7 774?", "answer": "710532"} +{"id": "mul-671x926", "question": "What is 671 \u00d7 926?", "answer": "621346"} +{"id": "mul-187x113", "question": "What is 187 \u00d7 113?", "answer": "21131"} +{"id": "mul-462x135", "question": "What is 462 \u00d7 135?", "answer": "62370"} +{"id": "mul-687x625", "question": "What is 687 \u00d7 625?", "answer": "429375"} +{"id": "mul-138x968", "question": "What is 138 \u00d7 968?", "answer": "133584"} +{"id": "mul-972x846", "question": "What is 972 \u00d7 846?", "answer": "822312"} +{"id": "mul-937x510", "question": "What is 937 \u00d7 510?", "answer": "477870"} +{"id": "mul-811x804", "question": "What is 811 \u00d7 804?", "answer": "652044"} +{"id": "mul-407x747", "question": "What is 407 \u00d7 747?", "answer": "304029"} +{"id": "mul-958x947", "question": "What is 958 \u00d7 947?", "answer": "907226"} +{"id": "mul-991x472", "question": "What is 991 \u00d7 472?", "answer": "467752"} +{"id": "mul-679x392", "question": "What is 679 \u00d7 392?", "answer": "266168"} +{"id": "mul-339x616", "question": "What is 339 \u00d7 616?", "answer": "208824"} +{"id": "mul-448x470", "question": "What is 448 \u00d7 470?", "answer": "210560"} +{"id": "mul-328x300", "question": "What is 328 \u00d7 300?", "answer": "98400"} +{"id": "mul-606x810", "question": "What is 606 \u00d7 810?", "answer": "490860"} +{"id": "mul-620x106", "question": "What is 620 \u00d7 106?", "answer": "65720"} +{"id": "mul-106x189", "question": "What is 106 \u00d7 189?", "answer": "20034"} +{"id": "mul-459x574", "question": "What is 459 \u00d7 574?", "answer": "263466"} +{"id": "mul-417x791", "question": "What is 417 \u00d7 791?", "answer": "329847"} +{"id": "mul-244x510", "question": "What is 244 \u00d7 510?", "answer": "124440"} +{"id": "mul-647x752", "question": "What is 647 \u00d7 752?", "answer": "486544"} +{"id": "mul-451x569", "question": "What is 451 \u00d7 569?", "answer": "256619"} +{"id": "mul-870x974", "question": "What is 870 \u00d7 974?", "answer": "847380"} +{"id": "mul-719x238", "question": "What is 719 \u00d7 238?", "answer": "171122"} +{"id": "mul-840x562", "question": "What is 840 \u00d7 562?", "answer": "472080"} +{"id": "mul-740x387", "question": "What is 740 \u00d7 387?", "answer": "286380"} +{"id": "mul-127x190", "question": "What is 127 \u00d7 190?", "answer": "24130"} +{"id": "mul-484x989", "question": "What is 484 \u00d7 989?", "answer": "478676"} +{"id": "mul-699x568", "question": "What is 699 \u00d7 568?", "answer": "397032"} +{"id": "mul-740x639", "question": "What is 740 \u00d7 639?", "answer": "472860"} +{"id": "mul-849x388", "question": "What is 849 \u00d7 388?", "answer": "329412"} +{"id": "mul-124x914", "question": "What is 124 \u00d7 914?", "answer": "113336"} +{"id": "mul-442x972", "question": "What is 442 \u00d7 972?", "answer": "429624"} +{"id": "mul-480x947", "question": "What is 480 \u00d7 947?", "answer": "454560"} +{"id": "mul-456x380", "question": "What is 456 \u00d7 380?", "answer": "173280"} +{"id": "mul-900x653", "question": "What is 900 \u00d7 653?", "answer": "587700"} +{"id": "mul-558x528", "question": "What is 558 \u00d7 528?", "answer": "294624"} +{"id": "mul-452x438", "question": "What is 452 \u00d7 438?", "answer": "197976"} +{"id": "mul-764x351", "question": "What is 764 \u00d7 351?", "answer": "268164"} +{"id": "mul-686x177", "question": "What is 686 \u00d7 177?", "answer": "121422"} +{"id": "mul-486x678", "question": "What is 486 \u00d7 678?", "answer": "329508"} +{"id": "mul-634x630", "question": "What is 634 \u00d7 630?", "answer": "399420"} +{"id": "mul-356x556", "question": "What is 356 \u00d7 556?", "answer": "197936"} +{"id": "mul-476x592", "question": "What is 476 \u00d7 592?", "answer": "281792"} +{"id": "mul-869x103", "question": "What is 869 \u00d7 103?", "answer": "89507"} +{"id": "mul-332x639", "question": "What is 332 \u00d7 639?", "answer": "212148"} +{"id": "mul-178x252", "question": "What is 178 \u00d7 252?", "answer": "44856"} +{"id": "mul-818x338", "question": "What is 818 \u00d7 338?", "answer": "276484"} +{"id": "mul-894x756", "question": "What is 894 \u00d7 756?", "answer": "675864"} +{"id": "mul-985x681", "question": "What is 985 \u00d7 681?", "answer": "670785"} +{"id": "mul-266x806", "question": "What is 266 \u00d7 806?", "answer": "214396"} +{"id": "mul-327x812", "question": "What is 327 \u00d7 812?", "answer": "265524"} +{"id": "mul-489x193", "question": "What is 489 \u00d7 193?", "answer": "94377"} +{"id": "mul-260x535", "question": "What is 260 \u00d7 535?", "answer": "139100"} +{"id": "mul-804x169", "question": "What is 804 \u00d7 169?", "answer": "135876"} +{"id": "mul-739x395", "question": "What is 739 \u00d7 395?", "answer": "291905"} +{"id": "mul-120x125", "question": "What is 120 \u00d7 125?", "answer": "15000"} +{"id": "mul-757x583", "question": "What is 757 \u00d7 583?", "answer": "441331"} +{"id": "mul-695x374", "question": "What is 695 \u00d7 374?", "answer": "259930"} +{"id": "mul-416x385", "question": "What is 416 \u00d7 385?", "answer": "160160"} +{"id": "mul-329x509", "question": "What is 329 \u00d7 509?", "answer": "167461"} +{"id": "mul-529x644", "question": "What is 529 \u00d7 644?", "answer": "340676"} +{"id": "mul-248x797", "question": "What is 248 \u00d7 797?", "answer": "197656"} +{"id": "mul-369x909", "question": "What is 369 \u00d7 909?", "answer": "335421"} +{"id": "mul-316x834", "question": "What is 316 \u00d7 834?", "answer": "263544"} +{"id": "mul-936x351", "question": "What is 936 \u00d7 351?", "answer": "328536"} +{"id": "mul-696x391", "question": "What is 696 \u00d7 391?", "answer": "272136"} +{"id": "mul-862x198", "question": "What is 862 \u00d7 198?", "answer": "170676"} +{"id": "mul-506x457", "question": "What is 506 \u00d7 457?", "answer": "231242"} +{"id": "mul-230x833", "question": "What is 230 \u00d7 833?", "answer": "191590"} +{"id": "mul-113x274", "question": "What is 113 \u00d7 274?", "answer": "30962"} +{"id": "mul-456x830", "question": "What is 456 \u00d7 830?", "answer": "378480"} +{"id": "mul-305x885", "question": "What is 305 \u00d7 885?", "answer": "269925"} +{"id": "mul-107x793", "question": "What is 107 \u00d7 793?", "answer": "84851"} +{"id": "mul-393x963", "question": "What is 393 \u00d7 963?", "answer": "378459"} +{"id": "mul-266x256", "question": "What is 266 \u00d7 256?", "answer": "68096"} +{"id": "mul-236x849", "question": "What is 236 \u00d7 849?", "answer": "200364"} +{"id": "mul-443x629", "question": "What is 443 \u00d7 629?", "answer": "278647"} +{"id": "mul-787x756", "question": "What is 787 \u00d7 756?", "answer": "594972"} +{"id": "mul-728x652", "question": "What is 728 \u00d7 652?", "answer": "474656"} +{"id": "mul-344x922", "question": "What is 344 \u00d7 922?", "answer": "317168"} +{"id": "mul-866x133", "question": "What is 866 \u00d7 133?", "answer": "115178"} +{"id": "mul-673x884", "question": "What is 673 \u00d7 884?", "answer": "594932"} +{"id": "mul-911x199", "question": "What is 911 \u00d7 199?", "answer": "181289"} +{"id": "mul-590x620", "question": "What is 590 \u00d7 620?", "answer": "365800"} +{"id": "mul-591x913", "question": "What is 591 \u00d7 913?", "answer": "539583"} +{"id": "mul-370x247", "question": "What is 370 \u00d7 247?", "answer": "91390"} +{"id": "mul-298x696", "question": "What is 298 \u00d7 696?", "answer": "207408"} +{"id": "mul-406x201", "question": "What is 406 \u00d7 201?", "answer": "81606"} +{"id": "mul-653x159", "question": "What is 653 \u00d7 159?", "answer": "103827"} +{"id": "mul-969x902", "question": "What is 969 \u00d7 902?", "answer": "874038"} +{"id": "mul-781x614", "question": "What is 781 \u00d7 614?", "answer": "479534"} +{"id": "mul-728x567", "question": "What is 728 \u00d7 567?", "answer": "412776"} +{"id": "mul-299x946", "question": "What is 299 \u00d7 946?", "answer": "282854"} +{"id": "mul-725x196", "question": "What is 725 \u00d7 196?", "answer": "142100"} +{"id": "mul-132x429", "question": "What is 132 \u00d7 429?", "answer": "56628"} +{"id": "mul-152x818", "question": "What is 152 \u00d7 818?", "answer": "124336"} +{"id": "mul-407x620", "question": "What is 407 \u00d7 620?", "answer": "252340"} +{"id": "mul-163x101", "question": "What is 163 \u00d7 101?", "answer": "16463"} +{"id": "mul-529x904", "question": "What is 529 \u00d7 904?", "answer": "478216"} +{"id": "mul-688x179", "question": "What is 688 \u00d7 179?", "answer": "123152"} +{"id": "mul-149x105", "question": "What is 149 \u00d7 105?", "answer": "15645"} +{"id": "mul-844x245", "question": "What is 844 \u00d7 245?", "answer": "206780"} +{"id": "mul-516x722", "question": "What is 516 \u00d7 722?", "answer": "372552"} +{"id": "mul-491x489", "question": "What is 491 \u00d7 489?", "answer": "240099"} +{"id": "mul-310x105", "question": "What is 310 \u00d7 105?", "answer": "32550"} +{"id": "mul-664x775", "question": "What is 664 \u00d7 775?", "answer": "514600"} +{"id": "mul-680x924", "question": "What is 680 \u00d7 924?", "answer": "628320"} +{"id": "mul-140x508", "question": "What is 140 \u00d7 508?", "answer": "71120"} +{"id": "mul-278x438", "question": "What is 278 \u00d7 438?", "answer": "121764"} +{"id": "mul-501x625", "question": "What is 501 \u00d7 625?", "answer": "313125"} +{"id": "mul-149x513", "question": "What is 149 \u00d7 513?", "answer": "76437"} +{"id": "mul-438x265", "question": "What is 438 \u00d7 265?", "answer": "116070"} +{"id": "mul-547x154", "question": "What is 547 \u00d7 154?", "answer": "84238"} +{"id": "mul-592x613", "question": "What is 592 \u00d7 613?", "answer": "362896"} +{"id": "mul-428x516", "question": "What is 428 \u00d7 516?", "answer": "220848"} +{"id": "mul-816x908", "question": "What is 816 \u00d7 908?", "answer": "740928"} +{"id": "mul-982x991", "question": "What is 982 \u00d7 991?", "answer": "973162"} +{"id": "mul-445x721", "question": "What is 445 \u00d7 721?", "answer": "320845"} +{"id": "mul-796x430", "question": "What is 796 \u00d7 430?", "answer": "342280"} +{"id": "mul-332x671", "question": "What is 332 \u00d7 671?", "answer": "222772"} +{"id": "mul-729x472", "question": "What is 729 \u00d7 472?", "answer": "344088"} +{"id": "mul-462x575", "question": "What is 462 \u00d7 575?", "answer": "265650"} +{"id": "mul-303x134", "question": "What is 303 \u00d7 134?", "answer": "40602"} +{"id": "mul-435x528", "question": "What is 435 \u00d7 528?", "answer": "229680"} +{"id": "mul-493x954", "question": "What is 493 \u00d7 954?", "answer": "470322"} +{"id": "mul-385x268", "question": "What is 385 \u00d7 268?", "answer": "103180"} +{"id": "mul-993x670", "question": "What is 993 \u00d7 670?", "answer": "665310"} +{"id": "mul-232x116", "question": "What is 232 \u00d7 116?", "answer": "26912"} +{"id": "mul-872x793", "question": "What is 872 \u00d7 793?", "answer": "691496"} +{"id": "mul-150x162", "question": "What is 150 \u00d7 162?", "answer": "24300"} +{"id": "mul-365x528", "question": "What is 365 \u00d7 528?", "answer": "192720"} +{"id": "mul-891x766", "question": "What is 891 \u00d7 766?", "answer": "682506"} +{"id": "mul-593x631", "question": "What is 593 \u00d7 631?", "answer": "374183"} +{"id": "mul-828x300", "question": "What is 828 \u00d7 300?", "answer": "248400"} +{"id": "mul-512x208", "question": "What is 512 \u00d7 208?", "answer": "106496"} +{"id": "mul-546x768", "question": "What is 546 \u00d7 768?", "answer": "419328"} +{"id": "mul-886x376", "question": "What is 886 \u00d7 376?", "answer": "333136"} +{"id": "mul-696x451", "question": "What is 696 \u00d7 451?", "answer": "313896"} +{"id": "mul-867x751", "question": "What is 867 \u00d7 751?", "answer": "651117"} +{"id": "mul-359x870", "question": "What is 359 \u00d7 870?", "answer": "312330"} +{"id": "mul-680x322", "question": "What is 680 \u00d7 322?", "answer": "218960"} +{"id": "mul-889x912", "question": "What is 889 \u00d7 912?", "answer": "810768"} +{"id": "mul-751x324", "question": "What is 751 \u00d7 324?", "answer": "243324"} +{"id": "mul-799x488", "question": "What is 799 \u00d7 488?", "answer": "389912"} +{"id": "mul-928x733", "question": "What is 928 \u00d7 733?", "answer": "680224"} +{"id": "mul-145x437", "question": "What is 145 \u00d7 437?", "answer": "63365"} +{"id": "mul-838x165", "question": "What is 838 \u00d7 165?", "answer": "138270"} +{"id": "mul-240x791", "question": "What is 240 \u00d7 791?", "answer": "189840"} +{"id": "mul-884x412", "question": "What is 884 \u00d7 412?", "answer": "364208"} +{"id": "mul-714x441", "question": "What is 714 \u00d7 441?", "answer": "314874"} +{"id": "mul-529x199", "question": "What is 529 \u00d7 199?", "answer": "105271"} +{"id": "mul-290x241", "question": "What is 290 \u00d7 241?", "answer": "69890"} +{"id": "mul-255x978", "question": "What is 255 \u00d7 978?", "answer": "249390"} +{"id": "mul-247x919", "question": "What is 247 \u00d7 919?", "answer": "226993"} +{"id": "mul-568x197", "question": "What is 568 \u00d7 197?", "answer": "111896"} +{"id": "mul-359x911", "question": "What is 359 \u00d7 911?", "answer": "327049"} +{"id": "mul-548x679", "question": "What is 548 \u00d7 679?", "answer": "372092"} +{"id": "mul-151x866", "question": "What is 151 \u00d7 866?", "answer": "130766"} +{"id": "mul-392x807", "question": "What is 392 \u00d7 807?", "answer": "316344"} +{"id": "mul-681x193", "question": "What is 681 \u00d7 193?", "answer": "131433"} +{"id": "mul-628x879", "question": "What is 628 \u00d7 879?", "answer": "552012"} +{"id": "mul-752x777", "question": "What is 752 \u00d7 777?", "answer": "584304"} +{"id": "mul-685x713", "question": "What is 685 \u00d7 713?", "answer": "488405"} +{"id": "mul-353x645", "question": "What is 353 \u00d7 645?", "answer": "227685"} +{"id": "mul-302x548", "question": "What is 302 \u00d7 548?", "answer": "165496"} +{"id": "mul-903x354", "question": "What is 903 \u00d7 354?", "answer": "319662"} +{"id": "mul-612x485", "question": "What is 612 \u00d7 485?", "answer": "296820"} +{"id": "mul-431x661", "question": "What is 431 \u00d7 661?", "answer": "284891"} +{"id": "mul-732x895", "question": "What is 732 \u00d7 895?", "answer": "655140"} +{"id": "mul-336x320", "question": "What is 336 \u00d7 320?", "answer": "107520"} +{"id": "mul-157x965", "question": "What is 157 \u00d7 965?", "answer": "151505"} +{"id": "mul-937x224", "question": "What is 937 \u00d7 224?", "answer": "209888"} +{"id": "mul-118x858", "question": "What is 118 \u00d7 858?", "answer": "101244"} +{"id": "mul-428x929", "question": "What is 428 \u00d7 929?", "answer": "397612"} +{"id": "mul-299x804", "question": "What is 299 \u00d7 804?", "answer": "240396"} +{"id": "mul-138x585", "question": "What is 138 \u00d7 585?", "answer": "80730"} +{"id": "mul-429x168", "question": "What is 429 \u00d7 168?", "answer": "72072"} +{"id": "mul-887x585", "question": "What is 887 \u00d7 585?", "answer": "518895"} +{"id": "mul-885x497", "question": "What is 885 \u00d7 497?", "answer": "439845"} +{"id": "mul-976x177", "question": "What is 976 \u00d7 177?", "answer": "172752"} +{"id": "mul-355x944", "question": "What is 355 \u00d7 944?", "answer": "335120"} +{"id": "mul-642x432", "question": "What is 642 \u00d7 432?", "answer": "277344"} +{"id": "mul-815x633", "question": "What is 815 \u00d7 633?", "answer": "515895"} +{"id": "mul-994x792", "question": "What is 994 \u00d7 792?", "answer": "787248"} +{"id": "mul-109x977", "question": "What is 109 \u00d7 977?", "answer": "106493"} +{"id": "mul-958x939", "question": "What is 958 \u00d7 939?", "answer": "899562"} +{"id": "mul-271x765", "question": "What is 271 \u00d7 765?", "answer": "207315"} +{"id": "mul-909x504", "question": "What is 909 \u00d7 504?", "answer": "458136"} +{"id": "mul-167x879", "question": "What is 167 \u00d7 879?", "answer": "146793"} +{"id": "mul-343x286", "question": "What is 343 \u00d7 286?", "answer": "98098"} +{"id": "mul-973x642", "question": "What is 973 \u00d7 642?", "answer": "624666"} +{"id": "mul-508x617", "question": "What is 508 \u00d7 617?", "answer": "313436"} +{"id": "mul-607x904", "question": "What is 607 \u00d7 904?", "answer": "548728"} +{"id": "mul-804x860", "question": "What is 804 \u00d7 860?", "answer": "691440"} +{"id": "mul-462x618", "question": "What is 462 \u00d7 618?", "answer": "285516"} +{"id": "mul-782x270", "question": "What is 782 \u00d7 270?", "answer": "211140"} +{"id": "mul-404x934", "question": "What is 404 \u00d7 934?", "answer": "377336"} +{"id": "mul-178x550", "question": "What is 178 \u00d7 550?", "answer": "97900"} +{"id": "mul-231x991", "question": "What is 231 \u00d7 991?", "answer": "228921"} +{"id": "mul-131x197", "question": "What is 131 \u00d7 197?", "answer": "25807"} +{"id": "mul-489x800", "question": "What is 489 \u00d7 800?", "answer": "391200"} +{"id": "mul-201x529", "question": "What is 201 \u00d7 529?", "answer": "106329"} +{"id": "mul-905x957", "question": "What is 905 \u00d7 957?", "answer": "866085"} +{"id": "mul-736x426", "question": "What is 736 \u00d7 426?", "answer": "313536"} +{"id": "mul-137x338", "question": "What is 137 \u00d7 338?", "answer": "46306"} +{"id": "mul-371x803", "question": "What is 371 \u00d7 803?", "answer": "297913"} +{"id": "mul-939x271", "question": "What is 939 \u00d7 271?", "answer": "254469"} +{"id": "mul-612x313", "question": "What is 612 \u00d7 313?", "answer": "191556"} +{"id": "mul-354x770", "question": "What is 354 \u00d7 770?", "answer": "272580"} +{"id": "mul-588x433", "question": "What is 588 \u00d7 433?", "answer": "254604"} +{"id": "mul-236x209", "question": "What is 236 \u00d7 209?", "answer": "49324"} +{"id": "mul-836x218", "question": "What is 836 \u00d7 218?", "answer": "182248"} +{"id": "mul-611x195", "question": "What is 611 \u00d7 195?", "answer": "119145"} +{"id": "mul-875x795", "question": "What is 875 \u00d7 795?", "answer": "695625"} +{"id": "mul-191x598", "question": "What is 191 \u00d7 598?", "answer": "114218"} +{"id": "mul-974x680", "question": "What is 974 \u00d7 680?", "answer": "662320"} +{"id": "mul-721x921", "question": "What is 721 \u00d7 921?", "answer": "664041"} +{"id": "mul-747x769", "question": "What is 747 \u00d7 769?", "answer": "574443"} +{"id": "mul-393x424", "question": "What is 393 \u00d7 424?", "answer": "166632"} +{"id": "mul-168x525", "question": "What is 168 \u00d7 525?", "answer": "88200"} +{"id": "mul-659x934", "question": "What is 659 \u00d7 934?", "answer": "615506"} +{"id": "mul-315x508", "question": "What is 315 \u00d7 508?", "answer": "160020"} +{"id": "mul-938x627", "question": "What is 938 \u00d7 627?", "answer": "588126"} +{"id": "mul-171x454", "question": "What is 171 \u00d7 454?", "answer": "77634"} +{"id": "mul-892x453", "question": "What is 892 \u00d7 453?", "answer": "404076"} +{"id": "mul-446x999", "question": "What is 446 \u00d7 999?", "answer": "445554"} +{"id": "mul-153x241", "question": "What is 153 \u00d7 241?", "answer": "36873"} +{"id": "mul-344x839", "question": "What is 344 \u00d7 839?", "answer": "288616"} +{"id": "mul-544x894", "question": "What is 544 \u00d7 894?", "answer": "486336"} +{"id": "mul-586x249", "question": "What is 586 \u00d7 249?", "answer": "145914"} +{"id": "mul-609x966", "question": "What is 609 \u00d7 966?", "answer": "588294"} +{"id": "mul-728x561", "question": "What is 728 \u00d7 561?", "answer": "408408"} +{"id": "mul-728x616", "question": "What is 728 \u00d7 616?", "answer": "448448"} +{"id": "mul-151x393", "question": "What is 151 \u00d7 393?", "answer": "59343"} +{"id": "mul-287x638", "question": "What is 287 \u00d7 638?", "answer": "183106"} +{"id": "mul-810x447", "question": "What is 810 \u00d7 447?", "answer": "362070"} +{"id": "mul-685x395", "question": "What is 685 \u00d7 395?", "answer": "270575"} +{"id": "mul-152x910", "question": "What is 152 \u00d7 910?", "answer": "138320"} +{"id": "mul-384x694", "question": "What is 384 \u00d7 694?", "answer": "266496"} +{"id": "mul-545x243", "question": "What is 545 \u00d7 243?", "answer": "132435"} +{"id": "mul-177x785", "question": "What is 177 \u00d7 785?", "answer": "138945"} +{"id": "mul-579x443", "question": "What is 579 \u00d7 443?", "answer": "256497"} +{"id": "mul-720x312", "question": "What is 720 \u00d7 312?", "answer": "224640"} +{"id": "mul-411x222", "question": "What is 411 \u00d7 222?", "answer": "91242"} +{"id": "mul-545x635", "question": "What is 545 \u00d7 635?", "answer": "346075"} +{"id": "mul-576x753", "question": "What is 576 \u00d7 753?", "answer": "433728"} +{"id": "mul-589x456", "question": "What is 589 \u00d7 456?", "answer": "268584"} +{"id": "mul-749x403", "question": "What is 749 \u00d7 403?", "answer": "301847"} +{"id": "mul-807x332", "question": "What is 807 \u00d7 332?", "answer": "267924"} +{"id": "mul-380x789", "question": "What is 380 \u00d7 789?", "answer": "299820"} +{"id": "mul-556x696", "question": "What is 556 \u00d7 696?", "answer": "386976"} +{"id": "mul-398x271", "question": "What is 398 \u00d7 271?", "answer": "107858"} +{"id": "mul-181x762", "question": "What is 181 \u00d7 762?", "answer": "137922"} +{"id": "mul-438x469", "question": "What is 438 \u00d7 469?", "answer": "205422"} +{"id": "mul-655x144", "question": "What is 655 \u00d7 144?", "answer": "94320"} +{"id": "mul-616x892", "question": "What is 616 \u00d7 892?", "answer": "549472"} +{"id": "mul-683x239", "question": "What is 683 \u00d7 239?", "answer": "163237"} +{"id": "mul-987x958", "question": "What is 987 \u00d7 958?", "answer": "945546"} +{"id": "mul-143x110", "question": "What is 143 \u00d7 110?", "answer": "15730"} +{"id": "mul-590x419", "question": "What is 590 \u00d7 419?", "answer": "247210"} +{"id": "mul-619x171", "question": "What is 619 \u00d7 171?", "answer": "105849"} +{"id": "mul-571x893", "question": "What is 571 \u00d7 893?", "answer": "509903"} +{"id": "mul-680x535", "question": "What is 680 \u00d7 535?", "answer": "363800"} +{"id": "mul-381x100", "question": "What is 381 \u00d7 100?", "answer": "38100"} +{"id": "mul-113x827", "question": "What is 113 \u00d7 827?", "answer": "93451"} +{"id": "mul-187x907", "question": "What is 187 \u00d7 907?", "answer": "169609"} +{"id": "mul-258x278", "question": "What is 258 \u00d7 278?", "answer": "71724"} +{"id": "mul-878x350", "question": "What is 878 \u00d7 350?", "answer": "307300"} +{"id": "mul-919x668", "question": "What is 919 \u00d7 668?", "answer": "613892"} +{"id": "mul-380x535", "question": "What is 380 \u00d7 535?", "answer": "203300"} +{"id": "mul-312x159", "question": "What is 312 \u00d7 159?", "answer": "49608"} +{"id": "mul-142x952", "question": "What is 142 \u00d7 952?", "answer": "135184"} +{"id": "mul-144x807", "question": "What is 144 \u00d7 807?", "answer": "116208"} +{"id": "mul-439x977", "question": "What is 439 \u00d7 977?", "answer": "428903"} +{"id": "mul-117x513", "question": "What is 117 \u00d7 513?", "answer": "60021"} +{"id": "mul-310x487", "question": "What is 310 \u00d7 487?", "answer": "150970"} +{"id": "mul-476x140", "question": "What is 476 \u00d7 140?", "answer": "66640"} +{"id": "mul-722x686", "question": "What is 722 \u00d7 686?", "answer": "495292"} +{"id": "mul-436x505", "question": "What is 436 \u00d7 505?", "answer": "220180"} +{"id": "mul-637x983", "question": "What is 637 \u00d7 983?", "answer": "626171"} +{"id": "mul-772x827", "question": "What is 772 \u00d7 827?", "answer": "638444"} +{"id": "mul-509x148", "question": "What is 509 \u00d7 148?", "answer": "75332"} +{"id": "mul-109x835", "question": "What is 109 \u00d7 835?", "answer": "91015"} +{"id": "mul-638x884", "question": "What is 638 \u00d7 884?", "answer": "563992"} +{"id": "mul-575x372", "question": "What is 575 \u00d7 372?", "answer": "213900"} +{"id": "mul-967x109", "question": "What is 967 \u00d7 109?", "answer": "105403"} +{"id": "mul-197x597", "question": "What is 197 \u00d7 597?", "answer": "117609"} +{"id": "mul-358x935", "question": "What is 358 \u00d7 935?", "answer": "334730"} +{"id": "mul-537x822", "question": "What is 537 \u00d7 822?", "answer": "441414"} diff --git a/aorlta/examples/qa/arith4/heldout/qa.jsonl b/aorlta/examples/qa/arith4/heldout/qa.jsonl new file mode 100644 index 000000000..a19399b47 --- /dev/null +++ b/aorlta/examples/qa/arith4/heldout/qa.jsonl @@ -0,0 +1,64 @@ +{"id": "mul-2970x1183", "question": "What is 2970 \u00d7 1183?", "answer": "3513510"} +{"id": "mul-9970x7366", "question": "What is 9970 \u00d7 7366?", "answer": "73439020"} +{"id": "mul-9121x8603", "question": "What is 9121 \u00d7 8603?", "answer": "78467963"} +{"id": "mul-6126x2237", "question": "What is 6126 \u00d7 2237?", "answer": "13703862"} +{"id": "mul-7350x3724", "question": "What is 7350 \u00d7 3724?", "answer": "27371400"} +{"id": "mul-1390x2804", "question": "What is 1390 \u00d7 2804?", "answer": "3897560"} +{"id": "mul-9699x3434", "question": "What is 9699 \u00d7 3434?", "answer": "33306366"} +{"id": "mul-3739x5267", "question": "What is 3739 \u00d7 5267?", "answer": "19693313"} +{"id": "mul-8104x8748", "question": "What is 8104 \u00d7 8748?", "answer": "70893792"} +{"id": "mul-3747x7584", "question": "What is 3747 \u00d7 7584?", "answer": "28417248"} +{"id": "mul-4837x6282", "question": "What is 4837 \u00d7 6282?", "answer": "30386034"} +{"id": "mul-3086x6219", "question": "What is 3086 \u00d7 6219?", "answer": "19191834"} +{"id": "mul-7047x7937", "question": "What is 7047 \u00d7 7937?", "answer": "55932039"} +{"id": "mul-3561x1702", "question": "What is 3561 \u00d7 1702?", "answer": "6060822"} +{"id": "mul-3660x7527", "question": "What is 3660 \u00d7 7527?", "answer": "27548820"} +{"id": "mul-4017x4736", "question": "What is 4017 \u00d7 4736?", "answer": "19024512"} +{"id": "mul-9163x2276", "question": "What is 9163 \u00d7 2276?", "answer": "20854988"} +{"id": "mul-9200x2377", "question": "What is 9200 \u00d7 2377?", "answer": "21868400"} +{"id": "mul-2209x5197", "question": "What is 2209 \u00d7 5197?", "answer": "11480173"} +{"id": "mul-6145x1794", "question": "What is 6145 \u00d7 1794?", "answer": "11024130"} +{"id": "mul-7912x7524", "question": "What is 7912 \u00d7 7524?", "answer": "59529888"} +{"id": "mul-7066x1861", "question": "What is 7066 \u00d7 1861?", "answer": "13149826"} +{"id": "mul-5887x5608", "question": "What is 5887 \u00d7 5608?", "answer": "33014296"} +{"id": "mul-5798x7150", "question": "What is 5798 \u00d7 7150?", "answer": "41455700"} +{"id": "mul-7649x7355", "question": "What is 7649 \u00d7 7355?", "answer": "56258395"} +{"id": "mul-5533x8559", "question": "What is 5533 \u00d7 8559?", "answer": "47356947"} +{"id": "mul-8358x7701", "question": "What is 8358 \u00d7 7701?", "answer": "64364958"} +{"id": "mul-6412x5953", "question": "What is 6412 \u00d7 5953?", "answer": "38170636"} +{"id": "mul-6387x3013", "question": "What is 6387 \u00d7 3013?", "answer": "19244031"} +{"id": "mul-8721x4405", "question": "What is 8721 \u00d7 4405?", "answer": "38416005"} +{"id": "mul-4869x6941", "question": "What is 4869 \u00d7 6941?", "answer": "33795729"} +{"id": "mul-9994x4436", "question": "What is 9994 \u00d7 4436?", "answer": "44333384"} +{"id": "mul-4984x2953", "question": "What is 4984 \u00d7 2953?", "answer": "14717752"} +{"id": "mul-3961x2885", "question": "What is 3961 \u00d7 2885?", "answer": "11427485"} +{"id": "mul-1020x9082", "question": "What is 1020 \u00d7 9082?", "answer": "9263640"} +{"id": "mul-6119x4166", "question": "What is 6119 \u00d7 4166?", "answer": "25491754"} +{"id": "mul-3192x1898", "question": "What is 3192 \u00d7 1898?", "answer": "6058416"} +{"id": "mul-7774x4707", "question": "What is 7774 \u00d7 4707?", "answer": "36592218"} +{"id": "mul-1403x6028", "question": "What is 1403 \u00d7 6028?", "answer": "8457284"} +{"id": "mul-9445x9085", "question": "What is 9445 \u00d7 9085?", "answer": "85807825"} +{"id": "mul-3183x3388", "question": "What is 3183 \u00d7 3388?", "answer": "10784004"} +{"id": "mul-1783x7575", "question": "What is 1783 \u00d7 7575?", "answer": "13506225"} +{"id": "mul-6165x3525", "question": "What is 6165 \u00d7 3525?", "answer": "21731625"} +{"id": "mul-1356x2277", "question": "What is 1356 \u00d7 2277?", "answer": "3087612"} +{"id": "mul-9082x3985", "question": "What is 9082 \u00d7 3985?", "answer": "36191770"} +{"id": "mul-1675x9759", "question": "What is 1675 \u00d7 9759?", "answer": "16346325"} +{"id": "mul-2267x6986", "question": "What is 2267 \u00d7 6986?", "answer": "15837262"} +{"id": "mul-1867x2014", "question": "What is 1867 \u00d7 2014?", "answer": "3760138"} +{"id": "mul-9827x1581", "question": "What is 9827 \u00d7 1581?", "answer": "15536487"} +{"id": "mul-8491x2147", "question": "What is 8491 \u00d7 2147?", "answer": "18230177"} +{"id": "mul-6511x3301", "question": "What is 6511 \u00d7 3301?", "answer": "21492811"} +{"id": "mul-6342x8639", "question": "What is 6342 \u00d7 8639?", "answer": "54788538"} +{"id": "mul-9300x3026", "question": "What is 9300 \u00d7 3026?", "answer": "28141800"} +{"id": "mul-1646x6099", "question": "What is 1646 \u00d7 6099?", "answer": "10038954"} +{"id": "mul-3529x5144", "question": "What is 3529 \u00d7 5144?", "answer": "18153176"} +{"id": "mul-2502x4392", "question": "What is 2502 \u00d7 4392?", "answer": "10988784"} +{"id": "mul-8282x7573", "question": "What is 8282 \u00d7 7573?", "answer": "62719586"} +{"id": "mul-4114x9991", "question": "What is 4114 \u00d7 9991?", "answer": "41102974"} +{"id": "mul-5600x4876", "question": "What is 5600 \u00d7 4876?", "answer": "27305600"} +{"id": "mul-9854x5784", "question": "What is 9854 \u00d7 5784?", "answer": "56995536"} +{"id": "mul-4447x4613", "question": "What is 4447 \u00d7 4613?", "answer": "20514011"} +{"id": "mul-6120x2506", "question": "What is 6120 \u00d7 2506?", "answer": "15336720"} +{"id": "mul-2291x9960", "question": "What is 2291 \u00d7 9960?", "answer": "22818360"} +{"id": "mul-9447x4547", "question": "What is 9447 \u00d7 4547?", "answer": "42955509"} diff --git a/aorlta/examples/qa/arith4/train/qa.jsonl b/aorlta/examples/qa/arith4/train/qa.jsonl new file mode 100644 index 000000000..6aad030ae --- /dev/null +++ b/aorlta/examples/qa/arith4/train/qa.jsonl @@ -0,0 +1,512 @@ +{"id": "mul-3927x1298", "question": "What is 3927 \u00d7 1298?", "answer": "5097246"} +{"id": "mul-3140x6965", "question": "What is 3140 \u00d7 6965?", "answer": "21870100"} +{"id": "mul-1157x8502", "question": "What is 1157 \u00d7 8502?", "answer": "9836814"} +{"id": "mul-2076x9443", "question": "What is 2076 \u00d7 9443?", "answer": "19603668"} +{"id": "mul-3170x2557", "question": "What is 3170 \u00d7 2557?", "answer": "8105690"} +{"id": "mul-1793x7413", "question": "What is 1793 \u00d7 7413?", "answer": "13291509"} +{"id": "mul-1853x2797", "question": "What is 1853 \u00d7 2797?", "answer": "5182841"} +{"id": "mul-5994x1509", "question": "What is 5994 \u00d7 1509?", "answer": "9044946"} +{"id": "mul-5907x9187", "question": "What is 5907 \u00d7 9187?", "answer": "54267609"} +{"id": "mul-1229x5973", "question": "What is 1229 \u00d7 5973?", "answer": "7340817"} +{"id": "mul-9262x8112", "question": "What is 9262 \u00d7 8112?", "answer": "75133344"} +{"id": "mul-5286x1768", "question": "What is 5286 \u00d7 1768?", "answer": "9345648"} +{"id": "mul-8992x7462", "question": "What is 8992 \u00d7 7462?", "answer": "67098304"} +{"id": "mul-6446x5632", "question": "What is 6446 \u00d7 5632?", "answer": "36303872"} +{"id": "mul-3199x3717", "question": "What is 3199 \u00d7 3717?", "answer": "11890683"} +{"id": "mul-2173x6855", "question": "What is 2173 \u00d7 6855?", "answer": "14895915"} +{"id": "mul-8894x7456", "question": "What is 8894 \u00d7 7456?", "answer": "66313664"} +{"id": "mul-5419x7475", "question": "What is 5419 \u00d7 7475?", "answer": "40507025"} +{"id": "mul-6140x9600", "question": "What is 6140 \u00d7 9600?", "answer": "58944000"} +{"id": "mul-1818x9482", "question": "What is 1818 \u00d7 9482?", "answer": "17238276"} +{"id": "mul-8202x7845", "question": "What is 8202 \u00d7 7845?", "answer": "64344690"} +{"id": "mul-6763x9895", "question": "What is 6763 \u00d7 9895?", "answer": "66919885"} +{"id": "mul-8641x2617", "question": "What is 8641 \u00d7 2617?", "answer": "22613497"} +{"id": "mul-5689x5506", "question": "What is 5689 \u00d7 5506?", "answer": "31323634"} +{"id": "mul-1434x1012", "question": "What is 1434 \u00d7 1012?", "answer": "1451208"} +{"id": "mul-3579x3665", "question": "What is 3579 \u00d7 3665?", "answer": "13117035"} +{"id": "mul-3052x2245", "question": "What is 3052 \u00d7 2245?", "answer": "6851740"} +{"id": "mul-8768x7087", "question": "What is 8768 \u00d7 7087?", "answer": "62138816"} +{"id": "mul-3367x9407", "question": "What is 3367 \u00d7 9407?", "answer": "31673369"} +{"id": "mul-4582x7324", "question": "What is 4582 \u00d7 7324?", "answer": "33558568"} +{"id": "mul-6207x4959", "question": "What is 6207 \u00d7 4959?", "answer": "30780513"} +{"id": "mul-4392x7864", "question": "What is 4392 \u00d7 7864?", "answer": "34538688"} +{"id": "mul-6335x2535", "question": "What is 6335 \u00d7 2535?", "answer": "16059225"} +{"id": "mul-2494x9534", "question": "What is 2494 \u00d7 9534?", "answer": "23777796"} +{"id": "mul-3564x5073", "question": "What is 3564 \u00d7 5073?", "answer": "18080172"} +{"id": "mul-9250x8118", "question": "What is 9250 \u00d7 8118?", "answer": "75091500"} +{"id": "mul-2038x3271", "question": "What is 2038 \u00d7 3271?", "answer": "6666298"} +{"id": "mul-1302x9992", "question": "What is 1302 \u00d7 9992?", "answer": "13009584"} +{"id": "mul-1119x3835", "question": "What is 1119 \u00d7 3835?", "answer": "4291365"} +{"id": "mul-7680x3216", "question": "What is 7680 \u00d7 3216?", "answer": "24698880"} +{"id": "mul-9674x4338", "question": "What is 9674 \u00d7 4338?", "answer": "41965812"} +{"id": "mul-8963x6096", "question": "What is 8963 \u00d7 6096?", "answer": "54638448"} +{"id": "mul-1864x9937", "question": "What is 1864 \u00d7 9937?", "answer": "18522568"} +{"id": "mul-6049x1030", "question": "What is 6049 \u00d7 1030?", "answer": "6230470"} +{"id": "mul-4505x7458", "question": "What is 4505 \u00d7 7458?", "answer": "33598290"} +{"id": "mul-6863x9222", "question": "What is 6863 \u00d7 9222?", "answer": "63290586"} +{"id": "mul-7110x1796", "question": "What is 7110 \u00d7 1796?", "answer": "12769560"} +{"id": "mul-6118x2705", "question": "What is 6118 \u00d7 2705?", "answer": "16549190"} +{"id": "mul-7821x4810", "question": "What is 7821 \u00d7 4810?", "answer": "37619010"} +{"id": "mul-3648x3161", "question": "What is 3648 \u00d7 3161?", "answer": "11531328"} +{"id": "mul-9583x6733", "question": "What is 9583 \u00d7 6733?", "answer": "64522339"} +{"id": "mul-5094x1741", "question": "What is 5094 \u00d7 1741?", "answer": "8868654"} +{"id": "mul-4330x2188", "question": "What is 4330 \u00d7 2188?", "answer": "9474040"} +{"id": "mul-6343x4471", "question": "What is 6343 \u00d7 4471?", "answer": "28359553"} +{"id": "mul-3140x7724", "question": "What is 3140 \u00d7 7724?", "answer": "24253360"} +{"id": "mul-3618x9478", "question": "What is 3618 \u00d7 9478?", "answer": "34291404"} +{"id": "mul-6576x2246", "question": "What is 6576 \u00d7 2246?", "answer": "14769696"} +{"id": "mul-7417x4917", "question": "What is 7417 \u00d7 4917?", "answer": "36469389"} +{"id": "mul-1328x5354", "question": "What is 1328 \u00d7 5354?", "answer": "7110112"} +{"id": "mul-2102x4391", "question": "What is 2102 \u00d7 4391?", "answer": "9229882"} +{"id": "mul-5752x6951", "question": "What is 5752 \u00d7 6951?", "answer": "39982152"} +{"id": "mul-9173x5938", "question": "What is 9173 \u00d7 5938?", "answer": "54469274"} +{"id": "mul-4727x9801", "question": "What is 4727 \u00d7 9801?", "answer": "46329327"} +{"id": "mul-9568x3948", "question": "What is 9568 \u00d7 3948?", "answer": "37774464"} +{"id": "mul-7461x1787", "question": "What is 7461 \u00d7 1787?", "answer": "13332807"} +{"id": "mul-6185x8719", "question": "What is 6185 \u00d7 8719?", "answer": "53927015"} +{"id": "mul-3157x3592", "question": "What is 3157 \u00d7 3592?", "answer": "11339944"} +{"id": "mul-2659x2030", "question": "What is 2659 \u00d7 2030?", "answer": "5397770"} +{"id": "mul-5970x8024", "question": "What is 5970 \u00d7 8024?", "answer": "47903280"} +{"id": "mul-6742x2856", "question": "What is 6742 \u00d7 2856?", "answer": "19255152"} +{"id": "mul-3716x1636", "question": "What is 3716 \u00d7 1636?", "answer": "6079376"} +{"id": "mul-8401x9153", "question": "What is 8401 \u00d7 9153?", "answer": "76894353"} +{"id": "mul-1965x8998", "question": "What is 1965 \u00d7 8998?", "answer": "17681070"} +{"id": "mul-3711x2328", "question": "What is 3711 \u00d7 2328?", "answer": "8639208"} +{"id": "mul-3529x5117", "question": "What is 3529 \u00d7 5117?", "answer": "18057893"} +{"id": "mul-4063x7727", "question": "What is 4063 \u00d7 7727?", "answer": "31394801"} +{"id": "mul-3033x8088", "question": "What is 3033 \u00d7 8088?", "answer": "24530904"} +{"id": "mul-3470x6437", "question": "What is 3470 \u00d7 6437?", "answer": "22336390"} +{"id": "mul-1405x2457", "question": "What is 1405 \u00d7 2457?", "answer": "3452085"} +{"id": "mul-2723x1244", "question": "What is 2723 \u00d7 1244?", "answer": "3387412"} +{"id": "mul-2488x5646", "question": "What is 2488 \u00d7 5646?", "answer": "14047248"} +{"id": "mul-5961x4290", "question": "What is 5961 \u00d7 4290?", "answer": "25572690"} +{"id": "mul-9428x4745", "question": "What is 9428 \u00d7 4745?", "answer": "44735860"} +{"id": "mul-9691x4371", "question": "What is 9691 \u00d7 4371?", "answer": "42359361"} +{"id": "mul-3169x7267", "question": "What is 3169 \u00d7 7267?", "answer": "23029123"} +{"id": "mul-5304x3873", "question": "What is 5304 \u00d7 3873?", "answer": "20542392"} +{"id": "mul-6211x6044", "question": "What is 6211 \u00d7 6044?", "answer": "37539284"} +{"id": "mul-8942x2713", "question": "What is 8942 \u00d7 2713?", "answer": "24259646"} +{"id": "mul-9112x3373", "question": "What is 9112 \u00d7 3373?", "answer": "30734776"} +{"id": "mul-7321x7709", "question": "What is 7321 \u00d7 7709?", "answer": "56437589"} +{"id": "mul-5641x2097", "question": "What is 5641 \u00d7 2097?", "answer": "11829177"} +{"id": "mul-4238x5432", "question": "What is 4238 \u00d7 5432?", "answer": "23020816"} +{"id": "mul-3704x1108", "question": "What is 3704 \u00d7 1108?", "answer": "4104032"} +{"id": "mul-2672x2115", "question": "What is 2672 \u00d7 2115?", "answer": "5651280"} +{"id": "mul-3184x7605", "question": "What is 3184 \u00d7 7605?", "answer": "24214320"} +{"id": "mul-1797x1394", "question": "What is 1797 \u00d7 1394?", "answer": "2505018"} +{"id": "mul-7549x6458", "question": "What is 7549 \u00d7 6458?", "answer": "48751442"} +{"id": "mul-4568x9699", "question": "What is 4568 \u00d7 9699?", "answer": "44305032"} +{"id": "mul-6967x3836", "question": "What is 6967 \u00d7 3836?", "answer": "26725412"} +{"id": "mul-1816x6715", "question": "What is 1816 \u00d7 6715?", "answer": "12194440"} +{"id": "mul-9700x5188", "question": "What is 9700 \u00d7 5188?", "answer": "50323600"} +{"id": "mul-3164x7601", "question": "What is 3164 \u00d7 7601?", "answer": "24049564"} +{"id": "mul-7746x6542", "question": "What is 7746 \u00d7 6542?", "answer": "50674332"} +{"id": "mul-4235x1104", "question": "What is 4235 \u00d7 1104?", "answer": "4675440"} +{"id": "mul-8088x1337", "question": "What is 8088 \u00d7 1337?", "answer": "10813656"} +{"id": "mul-6378x1848", "question": "What is 6378 \u00d7 1848?", "answer": "11786544"} +{"id": "mul-6615x4031", "question": "What is 6615 \u00d7 4031?", "answer": "26665065"} +{"id": "mul-4428x2743", "question": "What is 4428 \u00d7 2743?", "answer": "12146004"} +{"id": "mul-2220x1269", "question": "What is 2220 \u00d7 1269?", "answer": "2817180"} +{"id": "mul-4160x6799", "question": "What is 4160 \u00d7 6799?", "answer": "28283840"} +{"id": "mul-9182x6664", "question": "What is 9182 \u00d7 6664?", "answer": "61188848"} +{"id": "mul-3674x3144", "question": "What is 3674 \u00d7 3144?", "answer": "11551056"} +{"id": "mul-6784x1507", "question": "What is 6784 \u00d7 1507?", "answer": "10223488"} +{"id": "mul-2361x6333", "question": "What is 2361 \u00d7 6333?", "answer": "14952213"} +{"id": "mul-8231x6284", "question": "What is 8231 \u00d7 6284?", "answer": "51723604"} +{"id": "mul-3395x2539", "question": "What is 3395 \u00d7 2539?", "answer": "8619905"} +{"id": "mul-4768x2802", "question": "What is 4768 \u00d7 2802?", "answer": "13359936"} +{"id": "mul-5811x4993", "question": "What is 5811 \u00d7 4993?", "answer": "29014323"} +{"id": "mul-7997x3367", "question": "What is 7997 \u00d7 3367?", "answer": "26925899"} +{"id": "mul-3983x7051", "question": "What is 3983 \u00d7 7051?", "answer": "28084133"} +{"id": "mul-9121x9894", "question": "What is 9121 \u00d7 9894?", "answer": "90243174"} +{"id": "mul-5900x4946", "question": "What is 5900 \u00d7 4946?", "answer": "29181400"} +{"id": "mul-5992x5648", "question": "What is 5992 \u00d7 5648?", "answer": "33842816"} +{"id": "mul-9625x3722", "question": "What is 9625 \u00d7 3722?", "answer": "35824250"} +{"id": "mul-6262x6194", "question": "What is 6262 \u00d7 6194?", "answer": "38786828"} +{"id": "mul-7659x8084", "question": "What is 7659 \u00d7 8084?", "answer": "61915356"} +{"id": "mul-3070x7458", "question": "What is 3070 \u00d7 7458?", "answer": "22896060"} +{"id": "mul-8618x2447", "question": "What is 8618 \u00d7 2447?", "answer": "21088246"} +{"id": "mul-4758x1559", "question": "What is 4758 \u00d7 1559?", "answer": "7417722"} +{"id": "mul-9915x1354", "question": "What is 9915 \u00d7 1354?", "answer": "13424910"} +{"id": "mul-9683x2396", "question": "What is 9683 \u00d7 2396?", "answer": "23200468"} +{"id": "mul-1215x6795", "question": "What is 1215 \u00d7 6795?", "answer": "8255925"} +{"id": "mul-1560x9405", "question": "What is 1560 \u00d7 9405?", "answer": "14671800"} +{"id": "mul-1612x7572", "question": "What is 1612 \u00d7 7572?", "answer": "12206064"} +{"id": "mul-5927x6960", "question": "What is 5927 \u00d7 6960?", "answer": "41251920"} +{"id": "mul-5685x4829", "question": "What is 5685 \u00d7 4829?", "answer": "27452865"} +{"id": "mul-9270x6575", "question": "What is 9270 \u00d7 6575?", "answer": "60950250"} +{"id": "mul-6927x4651", "question": "What is 6927 \u00d7 4651?", "answer": "32217477"} +{"id": "mul-4204x9107", "question": "What is 4204 \u00d7 9107?", "answer": "38285828"} +{"id": "mul-9320x1097", "question": "What is 9320 \u00d7 1097?", "answer": "10224040"} +{"id": "mul-1098x2437", "question": "What is 1098 \u00d7 2437?", "answer": "2675826"} +{"id": "mul-6747x8596", "question": "What is 6747 \u00d7 8596?", "answer": "57997212"} +{"id": "mul-6087x3305", "question": "What is 6087 \u00d7 3305?", "answer": "20117535"} +{"id": "mul-7569x9755", "question": "What is 7569 \u00d7 9755?", "answer": "73835595"} +{"id": "mul-6619x8515", "question": "What is 6619 \u00d7 8515?", "answer": "56360785"} +{"id": "mul-3217x8393", "question": "What is 3217 \u00d7 8393?", "answer": "27000281"} +{"id": "mul-5597x1433", "question": "What is 5597 \u00d7 1433?", "answer": "8020501"} +{"id": "mul-2447x7156", "question": "What is 2447 \u00d7 7156?", "answer": "17510732"} +{"id": "mul-8501x9632", "question": "What is 8501 \u00d7 9632?", "answer": "81881632"} +{"id": "mul-5614x1395", "question": "What is 5614 \u00d7 1395?", "answer": "7831530"} +{"id": "mul-6476x7081", "question": "What is 6476 \u00d7 7081?", "answer": "45856556"} +{"id": "mul-6705x5482", "question": "What is 6705 \u00d7 5482?", "answer": "36756810"} +{"id": "mul-9859x8328", "question": "What is 9859 \u00d7 8328?", "answer": "82105752"} +{"id": "mul-7856x6642", "question": "What is 7856 \u00d7 6642?", "answer": "52179552"} +{"id": "mul-6418x5027", "question": "What is 6418 \u00d7 5027?", "answer": "32263286"} +{"id": "mul-2236x7182", "question": "What is 2236 \u00d7 7182?", "answer": "16058952"} +{"id": "mul-9550x9493", "question": "What is 9550 \u00d7 9493?", "answer": "90658150"} +{"id": "mul-5098x8300", "question": "What is 5098 \u00d7 8300?", "answer": "42313400"} +{"id": "mul-7030x8880", "question": "What is 7030 \u00d7 8880?", "answer": "62426400"} +{"id": "mul-1058x4714", "question": "What is 1058 \u00d7 4714?", "answer": "4987412"} +{"id": "mul-9631x2260", "question": "What is 9631 \u00d7 2260?", "answer": "21766060"} +{"id": "mul-3439x4816", "question": "What is 3439 \u00d7 4816?", "answer": "16562224"} +{"id": "mul-3657x4645", "question": "What is 3657 \u00d7 4645?", "answer": "16986765"} +{"id": "mul-7231x2503", "question": "What is 7231 \u00d7 2503?", "answer": "18099193"} +{"id": "mul-3572x7975", "question": "What is 3572 \u00d7 7975?", "answer": "28486700"} +{"id": "mul-2106x5732", "question": "What is 2106 \u00d7 5732?", "answer": "12071592"} +{"id": "mul-1328x1404", "question": "What is 1328 \u00d7 1404?", "answer": "1864512"} +{"id": "mul-8730x5399", "question": "What is 8730 \u00d7 5399?", "answer": "47133270"} +{"id": "mul-6065x5574", "question": "What is 6065 \u00d7 5574?", "answer": "33806310"} +{"id": "mul-4677x7556", "question": "What is 4677 \u00d7 7556?", "answer": "35339412"} +{"id": "mul-7875x9718", "question": "What is 7875 \u00d7 9718?", "answer": "76529250"} +{"id": "mul-3372x5308", "question": "What is 3372 \u00d7 5308?", "answer": "17898576"} +{"id": "mul-4467x5018", "question": "What is 4467 \u00d7 5018?", "answer": "22415406"} +{"id": "mul-5664x2576", "question": "What is 5664 \u00d7 2576?", "answer": "14590464"} +{"id": "mul-7511x6714", "question": "What is 7511 \u00d7 6714?", "answer": "50428854"} +{"id": "mul-3086x1214", "question": "What is 3086 \u00d7 1214?", "answer": "3746404"} +{"id": "mul-3796x6707", "question": "What is 3796 \u00d7 6707?", "answer": "25459772"} +{"id": "mul-4280x1116", "question": "What is 4280 \u00d7 1116?", "answer": "4776480"} +{"id": "mul-5690x3666", "question": "What is 5690 \u00d7 3666?", "answer": "20859540"} +{"id": "mul-3498x3191", "question": "What is 3498 \u00d7 3191?", "answer": "11162118"} +{"id": "mul-6496x9464", "question": "What is 6496 \u00d7 9464?", "answer": "61478144"} +{"id": "mul-9844x4904", "question": "What is 9844 \u00d7 4904?", "answer": "48274976"} +{"id": "mul-1541x2584", "question": "What is 1541 \u00d7 2584?", "answer": "3981944"} +{"id": "mul-8849x9323", "question": "What is 8849 \u00d7 9323?", "answer": "82499227"} +{"id": "mul-8867x5323", "question": "What is 8867 \u00d7 5323?", "answer": "47199041"} +{"id": "mul-3357x4178", "question": "What is 3357 \u00d7 4178?", "answer": "14025546"} +{"id": "mul-5902x2628", "question": "What is 5902 \u00d7 2628?", "answer": "15510456"} +{"id": "mul-9863x1947", "question": "What is 9863 \u00d7 1947?", "answer": "19203261"} +{"id": "mul-9225x8483", "question": "What is 9225 \u00d7 8483?", "answer": "78255675"} +{"id": "mul-4191x2540", "question": "What is 4191 \u00d7 2540?", "answer": "10645140"} +{"id": "mul-1527x6278", "question": "What is 1527 \u00d7 6278?", "answer": "9586506"} +{"id": "mul-1833x5912", "question": "What is 1833 \u00d7 5912?", "answer": "10836696"} +{"id": "mul-9329x2014", "question": "What is 9329 \u00d7 2014?", "answer": "18788606"} +{"id": "mul-1023x7875", "question": "What is 1023 \u00d7 7875?", "answer": "8056125"} +{"id": "mul-2269x1795", "question": "What is 2269 \u00d7 1795?", "answer": "4072855"} +{"id": "mul-1092x3330", "question": "What is 1092 \u00d7 3330?", "answer": "3636360"} +{"id": "mul-7664x7263", "question": "What is 7664 \u00d7 7263?", "answer": "55663632"} +{"id": "mul-7230x4360", "question": "What is 7230 \u00d7 4360?", "answer": "31522800"} +{"id": "mul-1087x1650", "question": "What is 1087 \u00d7 1650?", "answer": "1793550"} +{"id": "mul-7530x3858", "question": "What is 7530 \u00d7 3858?", "answer": "29050740"} +{"id": "mul-6413x7427", "question": "What is 6413 \u00d7 7427?", "answer": "47629351"} +{"id": "mul-9413x1788", "question": "What is 9413 \u00d7 1788?", "answer": "16830444"} +{"id": "mul-7612x6411", "question": "What is 7612 \u00d7 6411?", "answer": "48800532"} +{"id": "mul-3644x8155", "question": "What is 3644 \u00d7 8155?", "answer": "29716820"} +{"id": "mul-1872x8881", "question": "What is 1872 \u00d7 8881?", "answer": "16625232"} +{"id": "mul-9223x6262", "question": "What is 9223 \u00d7 6262?", "answer": "57754426"} +{"id": "mul-7666x6523", "question": "What is 7666 \u00d7 6523?", "answer": "50005318"} +{"id": "mul-6280x4723", "question": "What is 6280 \u00d7 4723?", "answer": "29660440"} +{"id": "mul-6963x6793", "question": "What is 6963 \u00d7 6793?", "answer": "47299659"} +{"id": "mul-8614x4250", "question": "What is 8614 \u00d7 4250?", "answer": "36609500"} +{"id": "mul-1546x6363", "question": "What is 1546 \u00d7 6363?", "answer": "9837198"} +{"id": "mul-7859x7295", "question": "What is 7859 \u00d7 7295?", "answer": "57331405"} +{"id": "mul-5561x3694", "question": "What is 5561 \u00d7 3694?", "answer": "20542334"} +{"id": "mul-3127x1262", "question": "What is 3127 \u00d7 1262?", "answer": "3946274"} +{"id": "mul-1804x1998", "question": "What is 1804 \u00d7 1998?", "answer": "3604392"} +{"id": "mul-5252x7860", "question": "What is 5252 \u00d7 7860?", "answer": "41280720"} +{"id": "mul-8902x9500", "question": "What is 8902 \u00d7 9500?", "answer": "84569000"} +{"id": "mul-4205x7595", "question": "What is 4205 \u00d7 7595?", "answer": "31936975"} +{"id": "mul-2735x8151", "question": "What is 2735 \u00d7 8151?", "answer": "22292985"} +{"id": "mul-5429x6631", "question": "What is 5429 \u00d7 6631?", "answer": "35999699"} +{"id": "mul-5159x4559", "question": "What is 5159 \u00d7 4559?", "answer": "23519881"} +{"id": "mul-4587x7218", "question": "What is 4587 \u00d7 7218?", "answer": "33108966"} +{"id": "mul-1729x6393", "question": "What is 1729 \u00d7 6393?", "answer": "11053497"} +{"id": "mul-2049x3249", "question": "What is 2049 \u00d7 3249?", "answer": "6657201"} +{"id": "mul-6000x6462", "question": "What is 6000 \u00d7 6462?", "answer": "38772000"} +{"id": "mul-7867x2589", "question": "What is 7867 \u00d7 2589?", "answer": "20367663"} +{"id": "mul-4048x3257", "question": "What is 4048 \u00d7 3257?", "answer": "13184336"} +{"id": "mul-3490x3357", "question": "What is 3490 \u00d7 3357?", "answer": "11715930"} +{"id": "mul-8490x2567", "question": "What is 8490 \u00d7 2567?", "answer": "21793830"} +{"id": "mul-5153x8180", "question": "What is 5153 \u00d7 8180?", "answer": "42151540"} +{"id": "mul-1823x5678", "question": "What is 1823 \u00d7 5678?", "answer": "10350994"} +{"id": "mul-2491x9454", "question": "What is 2491 \u00d7 9454?", "answer": "23549914"} +{"id": "mul-5063x9720", "question": "What is 5063 \u00d7 9720?", "answer": "49212360"} +{"id": "mul-4235x8180", "question": "What is 4235 \u00d7 8180?", "answer": "34642300"} +{"id": "mul-5076x9202", "question": "What is 5076 \u00d7 9202?", "answer": "46709352"} +{"id": "mul-7174x6309", "question": "What is 7174 \u00d7 6309?", "answer": "45260766"} +{"id": "mul-9981x4784", "question": "What is 9981 \u00d7 4784?", "answer": "47749104"} +{"id": "mul-4531x1926", "question": "What is 4531 \u00d7 1926?", "answer": "8726706"} +{"id": "mul-2988x1292", "question": "What is 2988 \u00d7 1292?", "answer": "3860496"} +{"id": "mul-6254x4197", "question": "What is 6254 \u00d7 4197?", "answer": "26248038"} +{"id": "mul-1609x8763", "question": "What is 1609 \u00d7 8763?", "answer": "14099667"} +{"id": "mul-6275x2092", "question": "What is 6275 \u00d7 2092?", "answer": "13127300"} +{"id": "mul-8763x7364", "question": "What is 8763 \u00d7 7364?", "answer": "64530732"} +{"id": "mul-2242x5086", "question": "What is 2242 \u00d7 5086?", "answer": "11402812"} +{"id": "mul-9672x6317", "question": "What is 9672 \u00d7 6317?", "answer": "61098024"} +{"id": "mul-9537x1151", "question": "What is 9537 \u00d7 1151?", "answer": "10977087"} +{"id": "mul-3747x7464", "question": "What is 3747 \u00d7 7464?", "answer": "27967608"} +{"id": "mul-2077x4898", "question": "What is 2077 \u00d7 4898?", "answer": "10173146"} +{"id": "mul-3976x9678", "question": "What is 3976 \u00d7 9678?", "answer": "38479728"} +{"id": "mul-7542x9287", "question": "What is 7542 \u00d7 9287?", "answer": "70042554"} +{"id": "mul-9112x6802", "question": "What is 9112 \u00d7 6802?", "answer": "61979824"} +{"id": "mul-9288x3727", "question": "What is 9288 \u00d7 3727?", "answer": "34616376"} +{"id": "mul-5878x2262", "question": "What is 5878 \u00d7 2262?", "answer": "13296036"} +{"id": "mul-8202x3105", "question": "What is 8202 \u00d7 3105?", "answer": "25467210"} +{"id": "mul-1511x2554", "question": "What is 1511 \u00d7 2554?", "answer": "3859094"} +{"id": "mul-7239x2619", "question": "What is 7239 \u00d7 2619?", "answer": "18958941"} +{"id": "mul-7870x6218", "question": "What is 7870 \u00d7 6218?", "answer": "48935660"} +{"id": "mul-1607x4823", "question": "What is 1607 \u00d7 4823?", "answer": "7750561"} +{"id": "mul-5346x3747", "question": "What is 5346 \u00d7 3747?", "answer": "20031462"} +{"id": "mul-9200x4410", "question": "What is 9200 \u00d7 4410?", "answer": "40572000"} +{"id": "mul-5070x8814", "question": "What is 5070 \u00d7 8814?", "answer": "44686980"} +{"id": "mul-6332x3188", "question": "What is 6332 \u00d7 3188?", "answer": "20186416"} +{"id": "mul-2744x2903", "question": "What is 2744 \u00d7 2903?", "answer": "7965832"} +{"id": "mul-9182x2521", "question": "What is 9182 \u00d7 2521?", "answer": "23147822"} +{"id": "mul-2467x8975", "question": "What is 2467 \u00d7 8975?", "answer": "22141325"} +{"id": "mul-5693x6187", "question": "What is 5693 \u00d7 6187?", "answer": "35222591"} +{"id": "mul-2096x7810", "question": "What is 2096 \u00d7 7810?", "answer": "16369760"} +{"id": "mul-9947x4451", "question": "What is 9947 \u00d7 4451?", "answer": "44274097"} +{"id": "mul-7535x9439", "question": "What is 7535 \u00d7 9439?", "answer": "71122865"} +{"id": "mul-2148x6671", "question": "What is 2148 \u00d7 6671?", "answer": "14329308"} +{"id": "mul-6652x6541", "question": "What is 6652 \u00d7 6541?", "answer": "43510732"} +{"id": "mul-1855x3266", "question": "What is 1855 \u00d7 3266?", "answer": "6058430"} +{"id": "mul-4916x8119", "question": "What is 4916 \u00d7 8119?", "answer": "39913004"} +{"id": "mul-8781x3394", "question": "What is 8781 \u00d7 3394?", "answer": "29802714"} +{"id": "mul-9159x8378", "question": "What is 9159 \u00d7 8378?", "answer": "76734102"} +{"id": "mul-9270x1825", "question": "What is 9270 \u00d7 1825?", "answer": "16917750"} +{"id": "mul-5688x3996", "question": "What is 5688 \u00d7 3996?", "answer": "22729248"} +{"id": "mul-9616x6560", "question": "What is 9616 \u00d7 6560?", "answer": "63080960"} +{"id": "mul-5720x1846", "question": "What is 5720 \u00d7 1846?", "answer": "10559120"} +{"id": "mul-5559x8123", "question": "What is 5559 \u00d7 8123?", "answer": "45155757"} +{"id": "mul-3294x2247", "question": "What is 3294 \u00d7 2247?", "answer": "7401618"} +{"id": "mul-8669x6495", "question": "What is 8669 \u00d7 6495?", "answer": "56305155"} +{"id": "mul-4398x5976", "question": "What is 4398 \u00d7 5976?", "answer": "26282448"} +{"id": "mul-2967x8132", "question": "What is 2967 \u00d7 8132?", "answer": "24127644"} +{"id": "mul-9562x8630", "question": "What is 9562 \u00d7 8630?", "answer": "82520060"} +{"id": "mul-8838x6697", "question": "What is 8838 \u00d7 6697?", "answer": "59188086"} +{"id": "mul-5863x4724", "question": "What is 5863 \u00d7 4724?", "answer": "27696812"} +{"id": "mul-5489x8299", "question": "What is 5489 \u00d7 8299?", "answer": "45553211"} +{"id": "mul-5778x3738", "question": "What is 5778 \u00d7 3738?", "answer": "21598164"} +{"id": "mul-2301x6417", "question": "What is 2301 \u00d7 6417?", "answer": "14765517"} +{"id": "mul-6911x9884", "question": "What is 6911 \u00d7 9884?", "answer": "68308324"} +{"id": "mul-1714x9257", "question": "What is 1714 \u00d7 9257?", "answer": "15866498"} +{"id": "mul-3225x1700", "question": "What is 3225 \u00d7 1700?", "answer": "5482500"} +{"id": "mul-1164x8841", "question": "What is 1164 \u00d7 8841?", "answer": "10290924"} +{"id": "mul-6105x9318", "question": "What is 6105 \u00d7 9318?", "answer": "56886390"} +{"id": "mul-2141x8544", "question": "What is 2141 \u00d7 8544?", "answer": "18292704"} +{"id": "mul-7974x5505", "question": "What is 7974 \u00d7 5505?", "answer": "43896870"} +{"id": "mul-1002x1218", "question": "What is 1002 \u00d7 1218?", "answer": "1220436"} +{"id": "mul-2396x3533", "question": "What is 2396 \u00d7 3533?", "answer": "8465068"} +{"id": "mul-3855x5004", "question": "What is 3855 \u00d7 5004?", "answer": "19290420"} +{"id": "mul-5491x7966", "question": "What is 5491 \u00d7 7966?", "answer": "43741306"} +{"id": "mul-4400x1957", "question": "What is 4400 \u00d7 1957?", "answer": "8610800"} +{"id": "mul-1680x1713", "question": "What is 1680 \u00d7 1713?", "answer": "2877840"} +{"id": "mul-6432x1275", "question": "What is 6432 \u00d7 1275?", "answer": "8200800"} +{"id": "mul-7609x4365", "question": "What is 7609 \u00d7 4365?", "answer": "33213285"} +{"id": "mul-7199x7017", "question": "What is 7199 \u00d7 7017?", "answer": "50515383"} +{"id": "mul-1646x6383", "question": "What is 1646 \u00d7 6383?", "answer": "10506418"} +{"id": "mul-7484x9606", "question": "What is 7484 \u00d7 9606?", "answer": "71891304"} +{"id": "mul-7554x1771", "question": "What is 7554 \u00d7 1771?", "answer": "13378134"} +{"id": "mul-1153x9202", "question": "What is 1153 \u00d7 9202?", "answer": "10609906"} +{"id": "mul-4409x9614", "question": "What is 4409 \u00d7 9614?", "answer": "42388126"} +{"id": "mul-8602x5362", "question": "What is 8602 \u00d7 5362?", "answer": "46123924"} +{"id": "mul-1154x2553", "question": "What is 1154 \u00d7 2553?", "answer": "2946162"} +{"id": "mul-8962x5139", "question": "What is 8962 \u00d7 5139?", "answer": "46055718"} +{"id": "mul-7994x5911", "question": "What is 7994 \u00d7 5911?", "answer": "47252534"} +{"id": "mul-4189x7345", "question": "What is 4189 \u00d7 7345?", "answer": "30768205"} +{"id": "mul-6507x8151", "question": "What is 6507 \u00d7 8151?", "answer": "53038557"} +{"id": "mul-7479x6533", "question": "What is 7479 \u00d7 6533?", "answer": "48860307"} +{"id": "mul-7976x1683", "question": "What is 7976 \u00d7 1683?", "answer": "13423608"} +{"id": "mul-3570x9044", "question": "What is 3570 \u00d7 9044?", "answer": "32287080"} +{"id": "mul-1072x1451", "question": "What is 1072 \u00d7 1451?", "answer": "1555472"} +{"id": "mul-1152x4176", "question": "What is 1152 \u00d7 4176?", "answer": "4810752"} +{"id": "mul-1009x7345", "question": "What is 1009 \u00d7 7345?", "answer": "7411105"} +{"id": "mul-8277x3591", "question": "What is 8277 \u00d7 3591?", "answer": "29722707"} +{"id": "mul-2333x5843", "question": "What is 2333 \u00d7 5843?", "answer": "13631719"} +{"id": "mul-3208x3828", "question": "What is 3208 \u00d7 3828?", "answer": "12280224"} +{"id": "mul-3998x9584", "question": "What is 3998 \u00d7 9584?", "answer": "38316832"} +{"id": "mul-2447x7754", "question": "What is 2447 \u00d7 7754?", "answer": "18974038"} +{"id": "mul-9605x3932", "question": "What is 9605 \u00d7 3932?", "answer": "37766860"} +{"id": "mul-8123x2716", "question": "What is 8123 \u00d7 2716?", "answer": "22062068"} +{"id": "mul-3133x9792", "question": "What is 3133 \u00d7 9792?", "answer": "30678336"} +{"id": "mul-7291x8315", "question": "What is 7291 \u00d7 8315?", "answer": "60624665"} +{"id": "mul-3536x2638", "question": "What is 3536 \u00d7 2638?", "answer": "9327968"} +{"id": "mul-5806x8400", "question": "What is 5806 \u00d7 8400?", "answer": "48770400"} +{"id": "mul-1710x1180", "question": "What is 1710 \u00d7 1180?", "answer": "2017800"} +{"id": "mul-4433x7733", "question": "What is 4433 \u00d7 7733?", "answer": "34280389"} +{"id": "mul-9396x2975", "question": "What is 9396 \u00d7 2975?", "answer": "27953100"} +{"id": "mul-6987x7122", "question": "What is 6987 \u00d7 7122?", "answer": "49761414"} +{"id": "mul-6116x4513", "question": "What is 6116 \u00d7 4513?", "answer": "27601508"} +{"id": "mul-8799x8974", "question": "What is 8799 \u00d7 8974?", "answer": "78962226"} +{"id": "mul-9392x2908", "question": "What is 9392 \u00d7 2908?", "answer": "27311936"} +{"id": "mul-4417x5557", "question": "What is 4417 \u00d7 5557?", "answer": "24545269"} +{"id": "mul-3853x2402", "question": "What is 3853 \u00d7 2402?", "answer": "9254906"} +{"id": "mul-8202x3142", "question": "What is 8202 \u00d7 3142?", "answer": "25770684"} +{"id": "mul-1882x8703", "question": "What is 1882 \u00d7 8703?", "answer": "16379046"} +{"id": "mul-7879x4940", "question": "What is 7879 \u00d7 4940?", "answer": "38922260"} +{"id": "mul-8825x5273", "question": "What is 8825 \u00d7 5273?", "answer": "46534225"} +{"id": "mul-5184x8147", "question": "What is 5184 \u00d7 8147?", "answer": "42234048"} +{"id": "mul-2896x7796", "question": "What is 2896 \u00d7 7796?", "answer": "22577216"} +{"id": "mul-1032x1152", "question": "What is 1032 \u00d7 1152?", "answer": "1188864"} +{"id": "mul-7759x9435", "question": "What is 7759 \u00d7 9435?", "answer": "73206165"} +{"id": "mul-5529x1813", "question": "What is 5529 \u00d7 1813?", "answer": "10024077"} +{"id": "mul-3617x8190", "question": "What is 3617 \u00d7 8190?", "answer": "29623230"} +{"id": "mul-1213x1502", "question": "What is 1213 \u00d7 1502?", "answer": "1821926"} +{"id": "mul-3265x3270", "question": "What is 3265 \u00d7 3270?", "answer": "10676550"} +{"id": "mul-6078x1345", "question": "What is 6078 \u00d7 1345?", "answer": "8174910"} +{"id": "mul-9912x1411", "question": "What is 9912 \u00d7 1411?", "answer": "13985832"} +{"id": "mul-1359x5185", "question": "What is 1359 \u00d7 5185?", "answer": "7046415"} +{"id": "mul-9204x5890", "question": "What is 9204 \u00d7 5890?", "answer": "54211560"} +{"id": "mul-8154x7618", "question": "What is 8154 \u00d7 7618?", "answer": "62117172"} +{"id": "mul-3357x4864", "question": "What is 3357 \u00d7 4864?", "answer": "16328448"} +{"id": "mul-9121x8131", "question": "What is 9121 \u00d7 8131?", "answer": "74162851"} +{"id": "mul-5049x3180", "question": "What is 5049 \u00d7 3180?", "answer": "16055820"} +{"id": "mul-4723x6163", "question": "What is 4723 \u00d7 6163?", "answer": "29107849"} +{"id": "mul-4978x4199", "question": "What is 4978 \u00d7 4199?", "answer": "20902622"} +{"id": "mul-5407x8855", "question": "What is 5407 \u00d7 8855?", "answer": "47878985"} +{"id": "mul-2274x2640", "question": "What is 2274 \u00d7 2640?", "answer": "6003360"} +{"id": "mul-4262x4964", "question": "What is 4262 \u00d7 4964?", "answer": "21156568"} +{"id": "mul-6488x2237", "question": "What is 6488 \u00d7 2237?", "answer": "14513656"} +{"id": "mul-4308x9393", "question": "What is 4308 \u00d7 9393?", "answer": "40465044"} +{"id": "mul-4201x3463", "question": "What is 4201 \u00d7 3463?", "answer": "14548063"} +{"id": "mul-9892x9450", "question": "What is 9892 \u00d7 9450?", "answer": "93479400"} +{"id": "mul-9151x4584", "question": "What is 9151 \u00d7 4584?", "answer": "41948184"} +{"id": "mul-4128x3187", "question": "What is 4128 \u00d7 3187?", "answer": "13155936"} +{"id": "mul-3118x8155", "question": "What is 3118 \u00d7 8155?", "answer": "25427290"} +{"id": "mul-8249x6170", "question": "What is 8249 \u00d7 6170?", "answer": "50896330"} +{"id": "mul-3774x8134", "question": "What is 3774 \u00d7 8134?", "answer": "30697716"} +{"id": "mul-9748x6879", "question": "What is 9748 \u00d7 6879?", "answer": "67056492"} +{"id": "mul-1298x5645", "question": "What is 1298 \u00d7 5645?", "answer": "7327210"} +{"id": "mul-1610x1601", "question": "What is 1610 \u00d7 1601?", "answer": "2577610"} +{"id": "mul-3452x4352", "question": "What is 3452 \u00d7 4352?", "answer": "15023104"} +{"id": "mul-9289x3075", "question": "What is 9289 \u00d7 3075?", "answer": "28563675"} +{"id": "mul-4771x7405", "question": "What is 4771 \u00d7 7405?", "answer": "35329255"} +{"id": "mul-6376x4317", "question": "What is 6376 \u00d7 4317?", "answer": "27525192"} +{"id": "mul-3014x5279", "question": "What is 3014 \u00d7 5279?", "answer": "15910906"} +{"id": "mul-4821x1864", "question": "What is 4821 \u00d7 1864?", "answer": "8986344"} +{"id": "mul-9968x1016", "question": "What is 9968 \u00d7 1016?", "answer": "10127488"} +{"id": "mul-5448x4200", "question": "What is 5448 \u00d7 4200?", "answer": "22881600"} +{"id": "mul-6677x4891", "question": "What is 6677 \u00d7 4891?", "answer": "32657207"} +{"id": "mul-7040x3273", "question": "What is 7040 \u00d7 3273?", "answer": "23041920"} +{"id": "mul-1302x4728", "question": "What is 1302 \u00d7 4728?", "answer": "6155856"} +{"id": "mul-7282x2200", "question": "What is 7282 \u00d7 2200?", "answer": "16020400"} +{"id": "mul-5121x6669", "question": "What is 5121 \u00d7 6669?", "answer": "34151949"} +{"id": "mul-6844x2411", "question": "What is 6844 \u00d7 2411?", "answer": "16500884"} +{"id": "mul-9486x4151", "question": "What is 9486 \u00d7 4151?", "answer": "39376386"} +{"id": "mul-6253x2083", "question": "What is 6253 \u00d7 2083?", "answer": "13024999"} +{"id": "mul-7216x3058", "question": "What is 7216 \u00d7 3058?", "answer": "22066528"} +{"id": "mul-7754x5559", "question": "What is 7754 \u00d7 5559?", "answer": "43104486"} +{"id": "mul-6276x7605", "question": "What is 6276 \u00d7 7605?", "answer": "47728980"} +{"id": "mul-9588x2787", "question": "What is 9588 \u00d7 2787?", "answer": "26721756"} +{"id": "mul-4775x5883", "question": "What is 4775 \u00d7 5883?", "answer": "28091325"} +{"id": "mul-5179x5241", "question": "What is 5179 \u00d7 5241?", "answer": "27143139"} +{"id": "mul-3915x4855", "question": "What is 3915 \u00d7 4855?", "answer": "19007325"} +{"id": "mul-5672x6724", "question": "What is 5672 \u00d7 6724?", "answer": "38138528"} +{"id": "mul-3482x9666", "question": "What is 3482 \u00d7 9666?", "answer": "33657012"} +{"id": "mul-1674x2708", "question": "What is 1674 \u00d7 2708?", "answer": "4533192"} +{"id": "mul-8438x9692", "question": "What is 8438 \u00d7 9692?", "answer": "81781096"} +{"id": "mul-6457x7847", "question": "What is 6457 \u00d7 7847?", "answer": "50668079"} +{"id": "mul-8536x1128", "question": "What is 8536 \u00d7 1128?", "answer": "9628608"} +{"id": "mul-5231x5623", "question": "What is 5231 \u00d7 5623?", "answer": "29413913"} +{"id": "mul-7034x8481", "question": "What is 7034 \u00d7 8481?", "answer": "59655354"} +{"id": "mul-6239x6302", "question": "What is 6239 \u00d7 6302?", "answer": "39318178"} +{"id": "mul-9656x6006", "question": "What is 9656 \u00d7 6006?", "answer": "57993936"} +{"id": "mul-7532x4560", "question": "What is 7532 \u00d7 4560?", "answer": "34345920"} +{"id": "mul-2144x7012", "question": "What is 2144 \u00d7 7012?", "answer": "15033728"} +{"id": "mul-9571x1205", "question": "What is 9571 \u00d7 1205?", "answer": "11533055"} +{"id": "mul-8782x8248", "question": "What is 8782 \u00d7 8248?", "answer": "72433936"} +{"id": "mul-5116x6730", "question": "What is 5116 \u00d7 6730?", "answer": "34430680"} +{"id": "mul-8619x2144", "question": "What is 8619 \u00d7 2144?", "answer": "18479136"} +{"id": "mul-3888x8669", "question": "What is 3888 \u00d7 8669?", "answer": "33705072"} +{"id": "mul-2098x6872", "question": "What is 2098 \u00d7 6872?", "answer": "14417456"} +{"id": "mul-4871x7583", "question": "What is 4871 \u00d7 7583?", "answer": "36936793"} +{"id": "mul-6001x6047", "question": "What is 6001 \u00d7 6047?", "answer": "36288047"} +{"id": "mul-8675x1308", "question": "What is 8675 \u00d7 1308?", "answer": "11346900"} +{"id": "mul-2201x4824", "question": "What is 2201 \u00d7 4824?", "answer": "10617624"} +{"id": "mul-1673x1912", "question": "What is 1673 \u00d7 1912?", "answer": "3198776"} +{"id": "mul-1085x8896", "question": "What is 1085 \u00d7 8896?", "answer": "9652160"} +{"id": "mul-2316x7091", "question": "What is 2316 \u00d7 7091?", "answer": "16422756"} +{"id": "mul-5625x7317", "question": "What is 5625 \u00d7 7317?", "answer": "41158125"} +{"id": "mul-7896x7068", "question": "What is 7896 \u00d7 7068?", "answer": "55808928"} +{"id": "mul-9775x7513", "question": "What is 9775 \u00d7 7513?", "answer": "73439575"} +{"id": "mul-1707x8088", "question": "What is 1707 \u00d7 8088?", "answer": "13806216"} +{"id": "mul-9748x3900", "question": "What is 9748 \u00d7 3900?", "answer": "38017200"} +{"id": "mul-5683x7456", "question": "What is 5683 \u00d7 7456?", "answer": "42372448"} +{"id": "mul-9354x6796", "question": "What is 9354 \u00d7 6796?", "answer": "63569784"} +{"id": "mul-1416x3998", "question": "What is 1416 \u00d7 3998?", "answer": "5661168"} +{"id": "mul-2261x1963", "question": "What is 2261 \u00d7 1963?", "answer": "4438343"} +{"id": "mul-1956x3317", "question": "What is 1956 \u00d7 3317?", "answer": "6488052"} +{"id": "mul-4476x2868", "question": "What is 4476 \u00d7 2868?", "answer": "12837168"} +{"id": "mul-4265x7017", "question": "What is 4265 \u00d7 7017?", "answer": "29927505"} +{"id": "mul-9024x4474", "question": "What is 9024 \u00d7 4474?", "answer": "40373376"} +{"id": "mul-4088x6738", "question": "What is 4088 \u00d7 6738?", "answer": "27544944"} +{"id": "mul-9811x8997", "question": "What is 9811 \u00d7 8997?", "answer": "88269567"} +{"id": "mul-1877x8965", "question": "What is 1877 \u00d7 8965?", "answer": "16827305"} +{"id": "mul-1980x3976", "question": "What is 1980 \u00d7 3976?", "answer": "7872480"} +{"id": "mul-9174x3673", "question": "What is 9174 \u00d7 3673?", "answer": "33696102"} +{"id": "mul-6288x6982", "question": "What is 6288 \u00d7 6982?", "answer": "43902816"} +{"id": "mul-6443x5373", "question": "What is 6443 \u00d7 5373?", "answer": "34618239"} +{"id": "mul-6402x8025", "question": "What is 6402 \u00d7 8025?", "answer": "51376050"} +{"id": "mul-1594x6480", "question": "What is 1594 \u00d7 6480?", "answer": "10329120"} +{"id": "mul-7784x9423", "question": "What is 7784 \u00d7 9423?", "answer": "73348632"} +{"id": "mul-3009x1325", "question": "What is 3009 \u00d7 1325?", "answer": "3986925"} +{"id": "mul-3597x7285", "question": "What is 3597 \u00d7 7285?", "answer": "26204145"} +{"id": "mul-6502x6639", "question": "What is 6502 \u00d7 6639?", "answer": "43166778"} +{"id": "mul-3242x8724", "question": "What is 3242 \u00d7 8724?", "answer": "28283208"} +{"id": "mul-4222x5201", "question": "What is 4222 \u00d7 5201?", "answer": "21958622"} +{"id": "mul-8847x6258", "question": "What is 8847 \u00d7 6258?", "answer": "55364526"} +{"id": "mul-3631x6889", "question": "What is 3631 \u00d7 6889?", "answer": "25013959"} +{"id": "mul-3835x4326", "question": "What is 3835 \u00d7 4326?", "answer": "16590210"} +{"id": "mul-7178x1213", "question": "What is 7178 \u00d7 1213?", "answer": "8706914"} +{"id": "mul-1875x2824", "question": "What is 1875 \u00d7 2824?", "answer": "5295000"} +{"id": "mul-5998x5784", "question": "What is 5998 \u00d7 5784?", "answer": "34692432"} +{"id": "mul-8129x1805", "question": "What is 8129 \u00d7 1805?", "answer": "14672845"} +{"id": "mul-4201x2454", "question": "What is 4201 \u00d7 2454?", "answer": "10309254"} +{"id": "mul-1620x2615", "question": "What is 1620 \u00d7 2615?", "answer": "4236300"} +{"id": "mul-3893x7375", "question": "What is 3893 \u00d7 7375?", "answer": "28710875"} +{"id": "mul-6100x7967", "question": "What is 6100 \u00d7 7967?", "answer": "48598700"} +{"id": "mul-2372x3464", "question": "What is 2372 \u00d7 3464?", "answer": "8216608"} +{"id": "mul-5905x3412", "question": "What is 5905 \u00d7 3412?", "answer": "20147860"} +{"id": "mul-7206x6349", "question": "What is 7206 \u00d7 6349?", "answer": "45750894"} +{"id": "mul-9248x4494", "question": "What is 9248 \u00d7 4494?", "answer": "41560512"} +{"id": "mul-8020x7028", "question": "What is 8020 \u00d7 7028?", "answer": "56364560"} +{"id": "mul-5899x6924", "question": "What is 5899 \u00d7 6924?", "answer": "40844676"} +{"id": "mul-3312x3530", "question": "What is 3312 \u00d7 3530?", "answer": "11691360"} +{"id": "mul-4475x3736", "question": "What is 4475 \u00d7 3736?", "answer": "16718600"} +{"id": "mul-7188x1482", "question": "What is 7188 \u00d7 1482?", "answer": "10652616"} +{"id": "mul-5789x6251", "question": "What is 5789 \u00d7 6251?", "answer": "36187039"} +{"id": "mul-5917x6010", "question": "What is 5917 \u00d7 6010?", "answer": "35561170"} +{"id": "mul-8534x9111", "question": "What is 8534 \u00d7 9111?", "answer": "77753274"} +{"id": "mul-6994x1141", "question": "What is 6994 \u00d7 1141?", "answer": "7980154"} +{"id": "mul-3792x7718", "question": "What is 3792 \u00d7 7718?", "answer": "29266656"} +{"id": "mul-8871x7380", "question": "What is 8871 \u00d7 7380?", "answer": "65467980"} +{"id": "mul-9102x6919", "question": "What is 9102 \u00d7 6919?", "answer": "62976738"} +{"id": "mul-2513x7731", "question": "What is 2513 \u00d7 7731?", "answer": "19428003"} +{"id": "mul-1957x4192", "question": "What is 1957 \u00d7 4192?", "answer": "8203744"} +{"id": "mul-5441x1117", "question": "What is 5441 \u00d7 1117?", "answer": "6077597"} +{"id": "mul-9981x3532", "question": "What is 9981 \u00d7 3532?", "answer": "35252892"} +{"id": "mul-4751x7290", "question": "What is 4751 \u00d7 7290?", "answer": "34634790"} +{"id": "mul-5709x6756", "question": "What is 5709 \u00d7 6756?", "answer": "38570004"} +{"id": "mul-4912x9847", "question": "What is 4912 \u00d7 9847?", "answer": "48368464"} +{"id": "mul-3179x4568", "question": "What is 3179 \u00d7 4568?", "answer": "14521672"} +{"id": "mul-6765x3698", "question": "What is 6765 \u00d7 3698?", "answer": "25016970"} +{"id": "mul-6301x7960", "question": "What is 6301 \u00d7 7960?", "answer": "50155960"} +{"id": "mul-6074x1575", "question": "What is 6074 \u00d7 1575?", "answer": "9566550"} +{"id": "mul-8900x9207", "question": "What is 8900 \u00d7 9207?", "answer": "81942300"} +{"id": "mul-2976x5099", "question": "What is 2976 \u00d7 5099?", "answer": "15174624"} +{"id": "mul-7890x9753", "question": "What is 7890 \u00d7 9753?", "answer": "76951170"} +{"id": "mul-7161x8229", "question": "What is 7161 \u00d7 8229?", "answer": "58927869"} +{"id": "mul-3754x8634", "question": "What is 3754 \u00d7 8634?", "answer": "32412036"} +{"id": "mul-2173x8150", "question": "What is 2173 \u00d7 8150?", "answer": "17709950"} +{"id": "mul-4337x7314", "question": "What is 4337 \u00d7 7314?", "answer": "31720818"} +{"id": "mul-1571x1949", "question": "What is 1571 \u00d7 1949?", "answer": "3061879"} +{"id": "mul-2903x5059", "question": "What is 2903 \u00d7 5059?", "answer": "14686277"} +{"id": "mul-5358x6112", "question": "What is 5358 \u00d7 6112?", "answer": "32748096"} +{"id": "mul-6607x6990", "question": "What is 6607 \u00d7 6990?", "answer": "46182930"} +{"id": "mul-7382x4839", "question": "What is 7382 \u00d7 4839?", "answer": "35721498"} +{"id": "mul-2192x1898", "question": "What is 2192 \u00d7 1898?", "answer": "4160416"} +{"id": "mul-6742x3188", "question": "What is 6742 \u00d7 3188?", "answer": "21493496"} +{"id": "mul-4569x7535", "question": "What is 4569 \u00d7 7535?", "answer": "34427415"} +{"id": "mul-1541x3492", "question": "What is 1541 \u00d7 3492?", "answer": "5381172"} +{"id": "mul-9873x7236", "question": "What is 9873 \u00d7 7236?", "answer": "71441028"} +{"id": "mul-2006x3324", "question": "What is 2006 \u00d7 3324?", "answer": "6667944"} diff --git a/aorlta/examples/qa/gpu-facts/heldout/qa.jsonl b/aorlta/examples/qa/gpu-facts/heldout/qa.jsonl new file mode 100644 index 000000000..fed2b0424 --- /dev/null +++ b/aorlta/examples/qa/gpu-facts/heldout/qa.jsonl @@ -0,0 +1,8 @@ +{"id": "warp-size", "question": "How many threads are in a warp on NVIDIA GPUs?", "answer": "32", "aliases": ["32 threads"]} +{"id": "free", "question": "Which HIP runtime function frees memory allocated with hipMalloc?", "answer": "hipFree", "aliases": ["hipFree()"]} +{"id": "blockdim", "question": "In a HIP kernel, which built-in variable holds the number of threads in each dimension of a block?", "answer": "blockDim"} +{"id": "rocm", "question": "What is the name of AMD's open software platform for GPU computing?", "answer": "ROCm"} +{"id": "stream", "question": "Which HIP runtime function creates a new stream?", "answer": "hipStreamCreate", "aliases": ["hipStreamCreate()"]} +{"id": "half-bytes", "question": "How many bytes does one 16-bit floating-point value occupy?", "answer": "2", "aliases": ["2 bytes", "two"]} +{"id": "griddim", "question": "In a HIP kernel, which built-in variable holds the number of blocks in each dimension of the grid?", "answer": "gridDim"} +{"id": "memset", "question": "Which HIP runtime function sets a range of device memory to a byte value?", "answer": "hipMemset", "aliases": ["hipMemset()"]} diff --git a/aorlta/examples/qa/gpu-facts/train/qa.jsonl b/aorlta/examples/qa/gpu-facts/train/qa.jsonl new file mode 100644 index 000000000..68397bd10 --- /dev/null +++ b/aorlta/examples/qa/gpu-facts/train/qa.jsonl @@ -0,0 +1,16 @@ +{"id": "global-kw", "question": "In HIP, which keyword marks a function that runs on the GPU and is launched from the host?", "answer": "__global__"} +{"id": "wave-cdna", "question": "How many threads make up one wavefront on AMD CDNA GPUs such as the MI300 series?", "answer": "64", "aliases": ["64 threads"]} +{"id": "lds", "question": "On AMD GPUs, what does the abbreviation LDS stand for?", "answer": "local data share"} +{"id": "memcpy", "question": "Which HIP runtime function copies memory between host and device?", "answer": "hipMemcpy", "aliases": ["hipMemcpy()"]} +{"id": "malloc", "question": "Which HIP runtime function allocates device memory?", "answer": "hipMalloc", "aliases": ["hipMalloc()"]} +{"id": "devsync", "question": "Which HIP runtime function blocks the host until all previously issued work on the current device has finished?", "answer": "hipDeviceSynchronize", "aliases": ["hipDeviceSynchronize()"]} +{"id": "syncthreads", "question": "Which HIP device function makes every thread of a block wait until all threads of that block reach it?", "answer": "__syncthreads", "aliases": ["__syncthreads()"]} +{"id": "threadidx", "question": "In a HIP kernel, which built-in variable holds a thread's index within its block?", "answer": "threadIdx"} +{"id": "shared-kw", "question": "Which HIP keyword declares a variable in memory shared by all threads of a block?", "answer": "__shared__"} +{"id": "hipcc", "question": "What is the name of ROCm's compiler driver for HIP source files?", "answer": "hipcc"} +{"id": "lasterror", "question": "Which HIP runtime function returns the last error produced by a HIP runtime call?", "answer": "hipGetLastError", "aliases": ["hipGetLastError()"]} +{"id": "atomicadd", "question": "Which HIP device function adds a value to a location in memory atomically?", "answer": "atomicAdd", "aliases": ["atomicAdd()"]} +{"id": "float-bytes", "question": "How many bytes does one 32-bit floating-point value occupy?", "answer": "4", "aliases": ["4 bytes", "four"]} +{"id": "gfx942", "question": "What is the LLVM target name (gfx number) of the AMD Instinct MI300X GPU?", "answer": "gfx942"} +{"id": "blockidx", "question": "In a HIP kernel, which built-in variable holds the index of the current block within the grid?", "answer": "blockIdx"} +{"id": "device-kw", "question": "Which HIP keyword marks a function that runs on the GPU and can be called only from GPU code?", "answer": "__device__"} diff --git a/aorlta/examples/qa/make_arith.py b/aorlta/examples/qa/make_arith.py new file mode 100644 index 000000000..409cea111 --- /dev/null +++ b/aorlta/examples/qa/make_arith.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +"""Write arith/{train,heldout}/qa.jsonl (3-digit by 3-digit multiplication) and arith4/{train,heldout}/qa.jsonl +(4-digit by 4-digit, harder), each from a fixed seed, with no pair in both halves. `python examples/qa/make_arith.py` +writes the committed files again, byte for byte.""" + +import json +import random +from pathlib import Path + +HERE = Path(__file__).resolve().parent + + +def write(name: str, digits: int, train: int = 512, heldout: int = 64, seed: int = 20261113) -> None: + rng = random.Random(seed) + lo, hi = 10 ** (digits - 1), 10 ** digits - 1 + pairs, seen = [], set() + while len(pairs) < train + heldout: + a, b = rng.randint(lo, hi), rng.randint(lo, hi) + if (a, b) not in seen and (b, a) not in seen: + seen.add((a, b)) + pairs.append((a, b)) + name_ = name + for part_name, part in (("train", pairs[:train]), ("heldout", pairs[train:])): + rows = [{"id": f"mul-{a}x{b}", "question": f"What is {a} × {b}?", "answer": str(a * b)} for a, b in part] + (HERE / name_ / part_name).mkdir(parents=True, exist_ok=True) + (HERE / name_ / part_name / "qa.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8") + + +def main() -> None: + write("arith", 3) + write("arith4", 4) + + +if __name__ == "__main__": + main() diff --git a/aorlta/examples/qa/qa-match.yaml b/aorlta/examples/qa/qa-match.yaml new file mode 100644 index 000000000..f962946e0 --- /dev/null +++ b/aorlta/examples/qa/qa-match.yaml @@ -0,0 +1,12 @@ +# `qa-match`: one leg, `ref.answer`, on the CPU: the reply's last `Answer:` line against the reference +# (examples/qa/qa_task.py). No compile and no sanitizer leg: a profile holds only the checks its task needs. +name: qa-match +reward_fn: qa.v1 +clean_repeats: 1 +materialize: true +pools: + match: {resource: local_cpu, adapter: qa.match, timeout: gate, workers: 4} +stages: + - {name: S0, pool: match, legs: [ref.answer]} +adapters: + qa.match: {} # {judge: [/path/to/judge, args...]} asks a command about answers that do not match diff --git a/aorlta/examples/qa/qa_task.py b/aorlta/examples/qa/qa_task.py new file mode 100644 index 000000000..29ef8ea81 --- /dev/null +++ b/aorlta/examples/qa/qa_task.py @@ -0,0 +1,267 @@ +"""A second worked example of a task as a plugin (docs/TUTORIAL.md): questions with reference answers, nothing +compiled. + +A pool is a directory holding `qa.jsonl`, one question per line: `{"id", "question", "answer", "aliases"?}`. The model +is asked the question and told to end its reply with a line `Answer: `. A reply without that line gets one +repair turn that says so (this module's parse and feedback); there is no compile step (`rollout.compile_adapter: +none`). The reward service runs one leg, `ref.answer`: the last `Answer:` line, normalised, against the reference and +its aliases. An optional judge (adapter setting `judge`: a command) is asked about an answer that does not match: it +reads `{"question", "reference", "answer"}` as JSON on stdin and prints `{"verdict": "correct"}` or `{"verdict": +"wrong"}`; anything else leaves the sample unscored. That is also the shape of a task whose reward is an external +tool's verdict. + +Registered names: task `qa`; parse `qa.parse`; feedback `qa.feedback`; profile `qa-match` (qa-match.yaml); check +adapter `qa.match`; reward function `qa.v1`. + +Two pools come with it: `gpu-facts/` (24 short questions on GPU programming, split train/held-out) and `arith/` +(3-digit multiplication, generated by make_arith.py, split train/held-out), the one the validation run trains on. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import unicodedata +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SYSTEM = "You are a helpful assistant." +USER = "{question}\n\nEnd your reply with a line of the form 'Answer: '." +ANSWER_RE = re.compile(r"^[ \t]*\**[ \t]*answer[ \t]*\**[ \t]*[:：][ \t]*(.*?)[ \t]*$", re.IGNORECASE) # one line + + +def load_rows(pool_dir) -> list[dict]: + rows = [json.loads(line) for line in (Path(pool_dir) / "qa.jsonl").read_text(encoding="utf-8").splitlines() + if line.strip()] + ids = [r["id"] for r in rows] + if len(set(ids)) != len(ids): + raise ValueError(f"{pool_dir}/qa.jsonl lists an id twice") + return rows + + +def row_sha256(row: dict) -> str: + return hashlib.sha256(json.dumps(row, sort_keys=True).encode("utf-8")).hexdigest() + + +def normalise(text: str) -> str: + """Lower case, NFKC, outer quotes, backticks, a trailing full stop, commas between digits and runs of space + removed: `**Answer:** "1,024."` and `answer: 1024` are one answer.""" + t = unicodedata.normalize("NFKC", text).strip().lower() + t = t.strip("*`'\" ") + t = re.sub(r"(?<=\d),(?=\d)", "", t) + t = t.rstrip(".").strip() + return re.sub(r"\s+", " ", t) + + +def final_answer(reply: str) -> str | None: + """The last `Answer:` line's answer, or None. Each line is read on its own, so an empty `Answer:` never takes the + next line as its answer, and the last marker decides, an empty one included.""" + found = [m.group(1) for line in reply.splitlines() if (m := ANSWER_RE.match(line))] + return (found[-1] or None) if found else None + + +def messages(row: dict) -> list[dict]: + return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER.format(question=row["question"])}] + + +def records(pool_dir, tokenizer, thinking): + from aorlta.ids import sha256_ids + from aorlta.rollout.templates import render_turn0 + from aorlta.schemas import PromptRecord, TaskRef + + out = [] + for row in load_rows(pool_dir): + msgs = messages(row) + out.append(PromptRecord( + prompt_id=f"qa:{row['id']}", turn0_ids=None, messages=msgs, + prompt_ids_sha256=sha256_ids(render_turn0(tokenizer, msgs, thinking)), + task=TaskRef(contract_id=f"qa:{row['id']}", parent_id=None, parent_healthy=None, reference_kind=None, + task_sha256=row_sha256(row)), + source="standin")) + return out + + +class QaParse: + """`qa.parse`: a reply with an `Answer:` line is an answer (nothing to compile); one without is rejected with a + message, which `qa.feedback` turns into a repair turn.""" + + name = "qa.parse" + + def __init__(self): + from aorlta.rollout.templates import IM_END_ID + + self.reply_stops = (IM_END_ID,) + + def check_prompts(self, prompts, thinking, max_repair_turns=0): + return None + + def check(self, text, thinking, prompt): + from aorlta.rollout.compile_check import ParseOutcome + + reply = text.rsplit("", 1)[-1] if thinking == "on" else text + answer = final_answer(reply) + if answer is None: + return ParseOutcome("rejected", rejection=( + "Your reply has no final line of the form 'Answer: '. Give your answer on such a line.", + "no_answer")) + return ParseOutcome("edits", code=answer) + + +class QaFeedback: + name = "qa.feedback" + + def build(self, errors, max_chars): + from aorlta.rollout.feedback import build_feedback + + return build_feedback(errors, max_chars) + + def build_apply(self, message, reason): + from aorlta.rollout.feedback import Feedback + + return Feedback(message, 0, 0) + + +class QaMatch: + """`qa.match`, leg `ref.answer`: clean when the final answer matches the reference or an alias after + `normalise`, or the judge (settings `judge`, a command as a list) says it is correct; `fail` `wrong_answer`, or + `no_answer` when the reply has no `Answer:` line; unsettled when the task is unknown, the reply cannot be read, + or the judge does not give a verdict.""" + + legs = ("ref.answer",) + resource = "local_cpu" + + def __init__(self, ctx): + from aorlta.reward.adapters.base import ProcessRunner + from aorlta.reward.adapters.toolchain import file_identity + + self.name, self.ctx = ctx.name, ctx + self.judge = ctx.settings.get("judge") + self.judge_timeout_s = float(ctx.settings.get("judge_timeout_s", 60.0)) + self.version = "2" # cached legs are keyed by it: the judge, which can turn a mismatch clean, is part of it + if self.judge: + files = {a: file_identity(a) for a in self.judge if isinstance(a, str) and Path(a).is_file()} + exe = shutil.which(self.judge[0]) if isinstance(self.judge[0], str) else None # as PATH finds it, if bare + if exe: + files[f"exe:{self.judge[0]}"] = f"{os.path.realpath(exe)}:{file_identity(exe)}" + ident = json.dumps({"judge": list(self.judge), "timeout_s": self.judge_timeout_s, "files": files}, + sort_keys=True) + self.version = f"2+judge-{hashlib.sha256(ident.encode()).hexdigest()[:16]}" + self.runner = ProcessRunner() # kills the judge's whole process group on a timeout or at close() + self.rows = {} + for pool in ctx.pools: + if (Path(pool) / "qa.jsonl").is_file(): + for row in load_rows(pool): + self.rows[row_sha256(row)] = row + + def run(self, job): + from aorlta.reward.adapters.base import make_result, unsettled + + t0 = self.ctx.clock() + try: + row = self.rows.get(job.task_sha256) + response = job.inputs.get("response") + if job.leg != "ref.answer" or row is None or not response: + return unsettled(job, "adapter_error", t_start=t0, t_end=self.ctx.clock()) + answer = final_answer(Path(response).read_text(encoding="utf-8").rsplit("", 1)[-1]) + if answer is None: + return make_result(job, "fail", "no_answer", t_start=t0, t_end=self.ctx.clock()) + wanted = {normalise(a) for a in [row["answer"], *row.get("aliases", [])]} + if normalise(answer) in wanted: + return make_result(job, "clean", t_start=t0, t_end=self.ctx.clock()) + if not self.judge: + return make_result(job, "fail", "wrong_answer", t_start=t0, t_end=self.ctx.clock()) + verdict = self._judge(row, answer) + if verdict == "correct": + return make_result(job, "clean", t_start=t0, t_end=self.ctx.clock()) + if verdict == "wrong": + return make_result(job, "fail", "wrong_answer", t_start=t0, t_end=self.ctx.clock()) + return unsettled(job, "judge_no_verdict", t_start=t0, t_end=self.ctx.clock()) + except Exception: # noqa: BLE001 - an adapter never raises + return unsettled(job, "adapter_error", t_start=t0, t_end=self.ctx.clock()) + + def _judge(self, row, answer) -> str | None: + proc = self.runner.run(list(self.judge), input=json.dumps({"question": row["question"], + "reference": row["answer"], "answer": answer}), + timeout_s=min(self.judge_timeout_s, self.ctx.timeout_s)) + if proc.timed_out or proc.killed or proc.launch_error or proc.returncode != 0 or not proc.stdout.strip(): + return None + try: + out = json.loads(proc.stdout.strip().splitlines()[-1]) + except ValueError: + return None + v = out.get("verdict") if isinstance(out, dict) else None + return v if v in ("correct", "wrong") else None + + def controls(self): + return [] + + def health(self): + return {"name": self.name, "responsive": True, "questions": len(self.rows)} + + def close(self): + self.runner.close() + + +class QaV1: + """`qa.v1`: +1.0 for a matching answer, -0.5 for a wrong one, -1.0 for no `Answer:` line; then -0.05 x the share + of the token budget used. An unsettled leg leaves the sample unscored.""" + + name = "qa.v1" + version = "1" + legs = ("ref.answer",) + BASE = {"wrong_answer": -0.5, "no_answer": -1.0} + + def __init__(self, profile=None): + from aorlta.reward.rewardfn.base import code_sha256 + + self.settings = {} + self.code_sha256 = code_sha256(__file__) + + def _base(self, legs): + leg = legs.get("ref.answer") + if leg is None: + return "pending", "ref.answer" + if leg.state == "clean": + return "score", 1.0 + if leg.state == "fail" and leg.reason in self.BASE: + return "score", self.BASE[leg.reason] + return "unscored", f"ref.answer:{leg.reason or leg.state}" + + @staticmethod + def _finish(base, cand): + parts = {"base": float(base), "broke_healthy": 0.0, + "length": -0.05 * (cand.response_tokens / cand.max_new_tokens)} + return parts["base"] + parts["broke_healthy"] + parts["length"], parts + + def compute(self, legs, cand): + kind, value = self._base(legs) + return (None, {"reason": str(value)}) if kind != "score" else self._finish(value, cand) + + def decide(self, partial, cand): + from aorlta.reward.rewardfn.base import Decided, NeedMore, Unscored + + kind, value = self._base(partial) + if kind == "pending": + return NeedMore((value,)) + return Unscored(value) if kind == "unscored" else Decided(*self._finish(value, cand)) + + @staticmethod + def claims_clean(reward, parts): + return reward is not None and isinstance(parts, dict) and parts.get("base", 0.0) > 0.0 + + +def register() -> None: + from aorlta import plugins + + plugins.register_task(plugins.Task("qa", records)) + plugins.register_parse_adapter("qa.parse", QaParse) + plugins.register_feedback_template("qa.feedback", QaFeedback) + plugins.register_profile("qa-match", HERE / "qa-match.yaml") + plugins.register_check_adapter("qa.match", QaMatch) + plugins.register_reward_fn("qa.v1", QaV1) + + +register() diff --git a/aorlta/src/aorlta/reward/adapters/base.py b/aorlta/src/aorlta/reward/adapters/base.py index ea89c1d41..426980ec4 100644 --- a/aorlta/src/aorlta/reward/adapters/base.py +++ b/aorlta/src/aorlta/reward/adapters/base.py @@ -218,12 +218,14 @@ def __init__(self) -> None: self._closed = threading.Event() def run(self, argv: Sequence[str], *, timeout_s: float, env: Mapping[str, str] | None = None, - cwd: str | None = None) -> ProcResult: + cwd: str | None = None, input: str | None = None) -> ProcResult: + """`input`, if given, is written to the child's stdin (else stdin is not redirected).""" t0 = time.monotonic() if self._closed.is_set(): return ProcResult(None, "", "", 0.0, False, True, "runner closed") try: - proc = subprocess.Popen(list(argv), stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + proc = subprocess.Popen(list(argv), stdin=subprocess.PIPE if input is not None else None, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, errors="replace", env=dict(env) if env is not None else None, cwd=cwd, start_new_session=True) except OSError as exc: @@ -234,7 +236,7 @@ def run(self, argv: Sequence[str], *, timeout_s: float, env: Mapping[str, str] | else: self._proc = proc try: - out, err = proc.communicate(timeout=timeout_s) + out, err = proc.communicate(input=input, timeout=timeout_s) timed_out = False except subprocess.TimeoutExpired: self._kill(proc) diff --git a/aorlta/src/aorlta/reward/evidence.py b/aorlta/src/aorlta/reward/evidence.py index 200a0319e..e97119aa4 100644 --- a/aorlta/src/aorlta/reward/evidence.py +++ b/aorlta/src/aorlta/reward/evidence.py @@ -1,7 +1,7 @@ """Keeping the GPU legs' evidence (run 2, step 4): the reward service's packages live in the run's node-local scratch, which the orchestrator removes at shutdown, so what explains an unsettled ConSan or ASan leg would go with it (run 1's -evidence archive came out empty). `archive_legs` copies the sanitizer legs' logs, generated harness sources and JSON, -and each candidate's parsed source, into one tarball in the run's sealed evidence store (Section 6.10 item 3) before +evidence archive came out empty). `archive_legs` copies the sanitizer and reference legs' logs, generated harness +sources and JSON, and each candidate's parsed source, into one tarball in the run's sealed evidence store (Section 6.10 item 3) before the scratch goes. Package layout: `pkg///-/...`. """ @@ -15,14 +15,14 @@ def kept(rel: Path) -> bool: - """Whether a file under the package root is evidence: a sanitizer leg's log, JSON or harness source, or the - source the parse gate wrote.""" + """Whether a file under the package root is evidence: a sanitizer leg's or a reference check's (`ref.*`, such as + a task plugin's own check) log, JSON or harness source, or the source the parse gate wrote.""" parts = rel.parts if len(parts) < 4: return False leg = parts[2] - return (leg.startswith("san.") and rel.suffix in SUFFIXES) or (leg.startswith("gate.parse-") - and rel.name == "source.hip") + return (leg.startswith(("san.", "ref.")) and rel.suffix in SUFFIXES) or (leg.startswith("gate.parse-") + and rel.name == "source.hip") def archive_legs(pkg_root: str | Path, dest: str | Path, *, max_file_bytes: int = MAX_FILE_BYTES) -> int: diff --git a/aorlta/tests/examples/test_hip_kernels.py b/aorlta/tests/examples/test_hip_kernels.py new file mode 100644 index 000000000..2e8760caf --- /dev/null +++ b/aorlta/tests/examples/test_hip_kernels.py @@ -0,0 +1,504 @@ +"""The worked example of docs/TUTORIAL.md (examples/hip-kernels), on CPU: its configs load with the plugin, its +prompts and probes build, its parse asks for a repair turn, its check maps each harness outcome to a leg state +without a GPU (a fake hipcc stands in for the compiler and the GPU run), its reward keeps the service's invariants, +and every task's harness compiles with its reference kernel where hipcc is installed.""" + +from __future__ import annotations + +import dataclasses +import hashlib +import itertools +import json +import os +import stat +import subprocess +import sys +from pathlib import Path + +import pytest + +from aorlta import plugins +from aorlta.config import config_from_dict, load_config, parse_yaml + +ROOT = Path(__file__).resolve().parents[2] +EXAMPLE = ROOT / "examples" / "hip-kernels" +sys.path.insert(0, str(ROOT / "tests")) +sys.path.insert(0, str(ROOT / "tests" / "orchestrator")) +import test_wiring_e2e as E # noqa: E402 - the tiny model and its engines, for the wired test +from test_plugins import CharChat, registry # noqa: E402,F401 - the probe tokenizer and the registry's restore + +tiny_base = E.tiny_base + +HIPCC = os.environ.get("HIPCC", "/opt/rocm/bin/hipcc") + + +@pytest.fixture +def hipk(registry): # noqa: F811 + """The example plugin, loaded as a run config loads it; its module.""" + plugins.load_plugins(["examples/hip-kernels/hip_kernels.py"]) + return sys.modules[plugins.identity()["examples/hip-kernels/hip_kernels.py"]["module"]] + + +def site_config(tmp_path, name: str): + site = tmp_path / "site.yaml" + site.write_text(json.dumps({"run": {"root": str(tmp_path / "runs")}, "model": {"base_dir": str(tmp_path / "base")}, + "data": {"prompt_pools": [str(EXAMPLE)]}})) + return load_config(ROOT / "configs" / f"{name}.yaml", site=site) + + +@pytest.mark.parametrize("name", ["hipk-score", "hipk-train"]) +def test_the_example_configs_load_with_the_plugin(tmp_path, registry, name): # noqa: F811 + cfg = site_config(tmp_path, name) + assert cfg.plugins == ("examples/hip-kernels/hip_kernels.py",) and cfg.data.prompt_pools[0].task == "hipk" + assert (cfg.reward.profile, cfg.reward.reward_fn, cfg.rollout.parse_adapter) == ("hipk-gpu", "hipk.v1", + "hipk.parse") + from aorlta.orchestrator import wiring + + wiring.check_supported(cfg) + + +def test_the_tasks_give_prompts_and_probes(tmp_path, hipk): + from aorlta.orchestrator import wiring + + cfg = site_config(tmp_path, "hipk-score") + tok = CharChat() + recs = wiring.TaskPromptPool(plugins.task("hipk"), tok, "off", EXAMPLE).records() + tasks = hipk.load_tasks(EXAMPLE) + assert [r.prompt_id for r in recs] == [f"hipk:{t['id']}" for t in tasks] and len(recs) == 8 + assert {r.task.task_sha256 for r in recs} == {hipk.task_sha256(t) for t in tasks} + for r, t in zip(recs, tasks): + assert t["signature"] in r.messages[1]["content"] + assert (EXAMPLE / "reference" / f"{t['id']}.hip").is_file() + probes = wiring.task_probes(cfg, tok, plugins.task("hipk"), EXAMPLE, recs) + assert len(probes.ids) == cfg.sync.probes + + +def prompt(hipk, kernel="saxpy"): + from aorlta.orchestrator import wiring + + return next(r for r in wiring.TaskPromptPool(plugins.task("hipk"), CharChat(), "off", EXAMPLE).records() + if r.prompt_id == f"hipk:{kernel}") + + +def test_the_parse_gives_a_repair_turn_for_no_block_or_no_kernel(hipk): + from aorlta.rollout.chain import make_feedback_template, make_parse_adapter + + parse, fb = make_parse_adapter("hipk.parse"), make_feedback_template("hipk.feedback") + p = prompt(hipk) + no_block = parse.check("I would write a loop.", "off", p) + assert no_block.kind == "rejected" and no_block.rejection[1] == "no_code_block" + other = parse.check("```cpp\n__global__ void axpy(float a, const float* x, float* y, int n) {}\n```", "off", p) + assert other.kind == "rejected" and other.rejection[1] == "no_kernel" and "`saxpy`" in other.rejection[0] + good = parse.check("x\n```cpp\n#include \n__global__ void saxpy(float a, const float* x, " + "float* y, int n) { }\n```\n", "off", p) + assert good.kind == "edits" and "__global__ void saxpy(" in good.code + turn = fb.build_apply(*other.rejection) + assert turn.text == other.rejection[0] and len(turn.text) <= 2000 + + +def test_the_parse_names_the_kernel_from_the_signature_not_the_task_id(hipk, tmp_path): + """Shape variants of one kernel have their own task ids; the kernel the reply must define is the signature's.""" + from aorlta.orchestrator import wiring + from aorlta.rollout.chain import make_parse_adapter + + task = dict(hipk.load_tasks(EXAMPLE)[1], id="saxpy_n1024") + (tmp_path / "tasks.json").write_text(json.dumps([task])) + (p,) = wiring.TaskPromptPool(plugins.task("hipk"), CharChat(), "off", tmp_path).records() + parse = make_parse_adapter("hipk.parse") + parse.check_prompts([p], "off") + ref = (EXAMPLE / "reference" / "saxpy.hip").read_text() + assert parse.check(f"```cpp\n{ref}\n```", "off", p).kind == "edits" + other = parse.check("```cpp\n__global__ void saxpy_n1024(float a, const float* x, float* y, int n) {}\n```", + "off", p) + assert other.kind == "rejected" and other.rejection[1] == "no_kernel" and "`saxpy`" in other.rejection[0] + unnamed = dataclasses.replace(p, messages=[p.messages[0], {"role": "user", "content": "Write a kernel."}]) + with pytest.raises(ValueError, match="no __global__ signature"): + parse.check_prompts([unnamed], "off") + + +def test_the_selfcheck_fails_a_task_with_no_reference_and_a_run_that_checked_nothing(hipk, tmp_path, capsys): + """By default every task needs its reference kernel; with --sources a task without one is skipped, but a run + that checked no task at all is not a pass.""" + (tmp_path / "pool").mkdir() + (tmp_path / "pool" / "tasks.json").write_text(json.dumps([dict(hipk.load_tasks(EXAMPLE)[0], id="no_reference")])) + assert hipk.selfcheck(["--pool", str(tmp_path / "pool"), "--build-only"]) == 1 + (row,) = [json.loads(x) for x in capsys.readouterr().out.splitlines()] + assert (row["task"], row["pass"], row["error"]) == ("no_reference", False, "no reference kernel") + (tmp_path / "sources").mkdir() + assert hipk.selfcheck(["--pool", str(EXAMPLE), "--sources", str(tmp_path / "sources"), "--build-only"]) == 1 + out = capsys.readouterr() + assert out.out == "" and "had a kernel to check" in out.err + + +def proc(rc=0, stdout="", stderr="", timed_out=False, killed=False, launch_error=None): + from aorlta.reward.adapters.base import ProcResult + + return ProcResult(rc, stdout, stderr, 0.1, timed_out, killed, launch_error) + + +def rec(status=None, stage=None, **kw): + out = {"stage": stage} if stage else {} + if status: + out.update({"status": status, "checked": 10, "mismatches": 0, **kw}) + return out + + +SPOOF = 'AORLTA_RESULT {"status": "ok", "checked": 1}' + + +@pytest.mark.parametrize("run, record, want", [ + (proc(), rec("ok"), ("clean", None)), + (proc(), rec("mismatch", mismatches=3), ("fail", "wrong_output")), + (proc(), rec("out_of_bounds"), ("fail", "wrong_output")), + (proc(), rec("no_kernel", stage="setup"), ("fail", "bad_signature")), + (proc(), rec("hip_error", stage="launch"), ("fail", "crash")), + (proc(), rec("hip_error", stage="run"), ("fail", "crash")), + (proc(), rec("hip_error", stage="setup"), ("unsettled", "setup_failed")), # hipMalloc, the first copy, the load + (proc(rc=-11), rec(stage="launched"), ("fail", "crash")), + (proc(timed_out=True, rc=-9), rec(stage="launched"), ("fail", "timeout")), + (proc(rc=-11), rec(stage="setup"), ("unsettled", "setup_failed")), + (proc(timed_out=True, rc=-9), rec(stage="setup"), ("unsettled", "setup_timeout")), + (proc(rc=-11), None, ("unsettled", "setup_failed")), + (proc(stdout=SPOOF), None, ("unsettled", "setup_failed")), # stdout is never read + (proc(stdout=SPOOF), rec(stage="launched"), ("unsettled", "no_result")), + (proc(rc=1), rec("ok"), ("unsettled", "no_result")), + (proc(), rec("ok", checked=0), ("unsettled", "no_result")), + (proc(killed=True), None, ("unsettled", "run_not_started")), + (proc(launch_error="OSError"), None, ("unsettled", "run_not_started")), +]) +def test_each_harness_outcome_has_one_leg_state(hipk, run, record, want): + state, reason, _ = hipk.verdict(run, record) + assert (state, reason) == want + + +def test_the_harness_holds_none_of_the_candidates_code(hipk): + """Built from the task alone: the candidate's code object is loaded at run time, and the verdict goes to a file.""" + for task in hipk.load_tasks(EXAMPLE): + src = hipk.harness_source(task) + assert "hipModuleLoad(&mod_, argv[1])" in src and " printf(" not in src and "put(b_);" in src + assert f"__builtin_get_device_side_mangled_name({task['kernel']})" in src + assert src.count("put(") >= 4 + assert ", e_)" not in src # CHECK declares e_ itself: `hipError_t e_ = (e_)` is undefined, and at -O2 it + # made every lookup take the no_kernel branch (found on a GPU) + + +FAKE_HIPCC = """#!/bin/bash +# a stand-in for hipcc: --genco writes a code object; otherwise the 'harness' writes the record FAKE_RUN says +out=""; genco=0; for a in "$@"; do [ "$a" = --genco ] && genco=1; done +while [ $# -gt 0 ]; do [ "$1" = -o ] && { out=$2; shift; }; shift; done +if [ "$genco" = 1 ]; then + [ "${FAKE_BUILD:-ok}" = co ] && { echo "error: cannot compile" >&2; exit 1; } + echo object > "$out"; exit 0 +fi +[ "${FAKE_BUILD:-ok}" = harness ] && { echo "error: harness" >&2; exit 1; } +printf '#!/bin/bash\necho "gpu $HIP_VISIBLE_DEVICES co $(cat $1)" >&2\necho %s\n' "'$SPOOF'" > "$out" +case "${FAKE_RUN:-ok}" in + ok) echo 'echo "{\\"status\\": \\"ok\\", \\"checked\\": 5}" > "$2"' >> "$out" ;; + mismatch) echo 'echo "{\\"status\\": \\"mismatch\\", \\"checked\\": 5, \\"mismatches\\": 2}" > "$2"' >> "$out" ;; + nokernel) echo 'echo "{\\"status\\": \\"no_kernel\\", \\"stage\\": \\"setup\\"}" > "$2"' >> "$out" ;; + setup) echo 'echo "{\\"status\\": \\"hip_error\\", \\"stage\\": \\"setup\\", \\"at\\": \\"hipMalloc\\"}" > "$2"' >> "$out" ;; + crash) echo 'echo "{\\"stage\\": \\"launched\\"}" > "$2"; kill -SEGV $$' >> "$out" ;; + hang) echo 'echo "{\\"stage\\": \\"launched\\"}" > "$2"; sleep 30' >> "$out" ;; + spoof) echo 'echo "{\\"stage\\": \\"launched\\"}" > "$2"' >> "$out" ;; +esac +chmod +x "$out" +""".replace("$SPOOF", SPOOF.replace('"', '\\"')) + + +@pytest.fixture +def fake_hipcc(tmp_path) -> Path: + p = tmp_path / "hipcc" + p.write_text(FAKE_HIPCC) + p.chmod(p.stat().st_mode | stat.S_IXUSR) + return p + + +def make_adapter(hipk, fake_hipcc, worker="run/2", gpus=(4, 6), timeout=5.0): + ctx = plugins.CheckContext(name="hipk.run", worker_id=worker, settings={"hipcc": str(fake_hipcc), + "gpus": list(gpus)}, + clock=lambda: 0.0, timeout_s=timeout, pools=(str(EXAMPLE),)) + return plugins.CHECK_ADAPTERS["hipk.run"](ctx) + + +def run_job(hipk, tmp_path, fake_hipcc, *, worker="run/2", task_id="saxpy", timeout=5.0, run_timeout_s=2.0): + from aorlta.schemas import LegJob + + task = next(t for t in hipk.load_tasks(EXAMPLE) if t["id"] == task_id) + src = tmp_path / "source.hip" + src.write_text("__global__ void saxpy(float a, const float* x, float* y, int n) {}\n") + out = tmp_path / "out" + out.mkdir(exist_ok=True) + adapter = make_adapter(hipk, fake_hipcc, worker=worker, timeout=timeout) + job = LegJob(job_id="j", leg="ref.native", candidate_id="c", control_id=None, source_sha256="s" * 64, + task_sha256=hipk.task_sha256(task), leg_settings={"target": "gfx950", "run_timeout_s": run_timeout_s}, + inputs={"source": str(src)}, output_dir=str(out)) + return adapter, adapter.run(job) + + +@pytest.mark.parametrize("build, run, want", [ + ("ok", "ok", ("clean", None)), ("ok", "mismatch", ("fail", "wrong_output")), ("ok", "crash", ("fail", "crash")), + ("ok", "hang", ("fail", "timeout")), ("ok", "nokernel", ("fail", "bad_signature")), + ("ok", "setup", ("unsettled", "setup_failed")), ("ok", "spoof", ("unsettled", "no_result")), + ("co", "ok", ("unsettled", "candidate_build_failed")), ("harness", "ok", ("unsettled", "harness_build_failed")), +]) +def test_the_check_runs_the_harness_on_its_workers_gpu(hipk, tmp_path, fake_hipcc, monkeypatch, build, run, want): + monkeypatch.setenv("FAKE_BUILD", build) + monkeypatch.setenv("FAKE_RUN", run) + adapter, res = run_job(hipk, tmp_path, fake_hipcc) + try: + assert (res.state, res.reason) == want and res.leg == "ref.native" + assert adapter.gpu == 6 # worker 2 of gpus [4, 6], no other worker live + if build == "ok" and run == "ok": + (log,) = tmp_path.glob("out/*/run.log") + assert "gpu 6 co object" in log.read_text() + assert {p.name for p in log.parent.iterdir()} >= {"build.log", "harness.hip", "record.json", "run.log"} + finally: + adapter.close() + + +def test_a_replaced_worker_takes_the_gpu_its_predecessor_gave_back(hipk, fake_hipcc): + """The service names a replacement by the next sequence number (run/5 for run/2), so a GPU from the id alone + would land run/5 on run/1's GPU and leave run/2's idle.""" + probe = make_adapter(hipk, fake_hipcc, worker="run/probe", gpus=(4, 5, 6, 7)) + probe.close() + workers = {n: make_adapter(hipk, fake_hipcc, worker=f"run/{n}", gpus=(4, 5, 6, 7)) for n in (1, 2, 3, 4)} + assert [w.gpu for w in workers.values()] == [4, 5, 6, 7] + workers.pop(2).close() + workers[5] = make_adapter(hipk, fake_hipcc, worker="run/5", gpus=(4, 5, 6, 7)) + assert workers[5].gpu == 5 + workers[6] = make_adapter(hipk, fake_hipcc, worker="run/6", gpus=(4, 5, 6, 7)) # more workers than GPUs: shared + assert workers[6].gpu in (4, 5, 6, 7) + for w in workers.values(): + w.close() + workers[1].close() # a second close gives nothing back twice + assert hipk._LEASED == {} + + +def test_the_checker_version_names_the_compiler_and_this_file(hipk, fake_hipcc, tmp_path): + """Cached ref.native legs are keyed by the checker's version: another compiler, or an edited harness, is a new + version, so an old verdict is not reused.""" + a = make_adapter(hipk, fake_hipcc) + other = tmp_path / "other" / "hipcc" + other.parent.mkdir() + other.write_text(FAKE_HIPCC + "# another build\n") + other.chmod(0o755) + b = make_adapter(hipk, other) + again = make_adapter(hipk, fake_hipcc) + try: + assert a.version == again.version != b.version + assert a.version.endswith(hashlib.sha256(Path(hipk.__file__).read_bytes()).hexdigest()[:16]) + finally: + for x in (a, b, again): + x.close() + + +def test_the_reward_parse_gate_reads_the_answer_as_the_rollout_does(hipk, tmp_path): + """A final reply whose last ```cpp block never closes is no answer to the rollout; the stand-in gate would score + the earlier, complete block in its place.""" + from aorlta.reward.adapters.standin_parse import StandinParse + from aorlta.rollout.compile_check import ParseOutcome # noqa: F401 + from aorlta.schemas import LegJob + + cut = ("Here:\n```cpp\n__global__ void saxpy(float a, const float* x, float* y, int n) {}\n```\n" + "Better:\n```cpp\n__global__ void saxpy(float a, const") + whole = "```cpp\n__global__ void saxpy(float a, const float* x, float* y, int n) {}\n```\n" + gate = plugins.CHECK_ADAPTERS["hipk.gate"](plugins.CheckContext(name="hipk.gate", worker_id="gates/1", + settings={}, clock=lambda: 0.0, timeout_s=5.0)) + for i, (text, want) in enumerate([(cut, ("fail", "parse")), (whole, ("clean", None))]): + (tmp_path / f"r{i}.txt").write_text(text) + job = LegJob(job_id="j", leg="gate.parse", candidate_id="c", control_id=None, source_sha256="s" * 64, + task_sha256="t" * 64, inputs={"response": str(tmp_path / f"r{i}.txt")}, + output_dir=str(tmp_path / f"o{i}")) + res = gate.run(job) + assert (res.state, res.reason) == want + assert (tmp_path / f"o{i}" / "source.hip").exists() == (want[0] == "clean") + assert StandinParse().run(dataclasses.replace(job, output_dir=str(tmp_path / f"s{i}"))).state == "clean" + assert hipk.HipkParse().check(cut, "off", None).kind == "rejected" + + +def test_a_task_the_check_does_not_know_is_unsettled(hipk, tmp_path, fake_hipcc): + from aorlta.schemas import LegJob + + ctx = plugins.CheckContext(name="hipk.run", worker_id="run/1", settings={"hipcc": str(fake_hipcc)}, + clock=lambda: 0.0, timeout_s=5.0, pools=(str(tmp_path),)) + adapter = plugins.CHECK_ADAPTERS["hipk.run"](ctx) + job = LegJob(job_id="j", leg="ref.native", candidate_id="c", control_id=None, source_sha256="s" * 64, + task_sha256="0" * 64, inputs={"source": str(tmp_path / "x.hip")}, output_dir=str(tmp_path)) + try: + res = adapter.run(job) + assert (res.state, res.reason) == ("unsettled", "adapter_error") + finally: + adapter.close() + + +def leg(name, state, reason=None): + from aorlta.schemas import LegResult + + return LegResult(leg=name, state=state, reason=reason, findings=[], repeats=1 if state != "unsettled" else 0, + attempts=1, worker_id=None, bracket=None, cache_hit=False, t_queued=0.0, t_start=0.0, t_end=0.0) + + +def cand(tokens=100): + from aorlta.schemas import Candidate, TaskRef + + return Candidate(candidate_id="c", run_id="r", iteration=1, prompt_id="hipk:saxpy", k=0, policy_version="v", + response_text="", response_tokens=tokens, max_new_tokens=2048, + task=TaskRef("hipk:saxpy", None, None, None, "t" * 64), check_profile="hipk-gpu", + rollout_compile=None) + + +OUTCOMES = { + "gate.parse": [("clean", None), ("fail", "parse"), ("unsettled", "timeout")], + "gate.compile": [("clean", None), ("fail", "compile_error"), ("unsettled", "adapter_error"), ("not_run", None)], + "ref.native": [("clean", None), ("fail", "wrong_output"), ("fail", "crash"), ("fail", "timeout"), + ("fail", "bad_signature"), ("fail", "other"), ("unsettled", "no_result"), ("not_run", None), + ("skipped_by_plan", None)], +} + + +def test_the_reward_ladder(hipk): + fn = plugins.REWARD_FNS["hipk.v1"](None) + c = cand(tokens=0) + + def score(**states): + legs = {n: leg(n, *states.get(n.replace(".", "_"), ("clean", None))) for n in fn.legs} + return fn.compute(legs, c)[0] + + assert score() == 1.0 + assert score(gate_parse=("fail", "parse")) == -1.0 + assert score(gate_compile=("fail", "compile_error")) == -0.9 + assert score(ref_native=("fail", "bad_signature")) == -0.8 + assert score(ref_native=("fail", "crash")) == score(ref_native=("fail", "timeout")) == -0.6 + assert score(ref_native=("fail", "wrong_output")) == -0.5 + assert score(ref_native=("unsettled", "no_result")) is None and score(ref_native=("fail", "other")) is None + + +def test_the_reward_keeps_the_services_invariants_and_decide_agrees_with_compute(hipk): + """Every combination of leg outcomes, and every prefix of it as a partial record: decide never contradicts + compute, and a score is a float in the service's range whose parts are base, broke_healthy and length and add + up to it (reward/service.py `_invariant_problems`).""" + from aorlta.reward.rewardfn.base import Decided, NeedMore, Unscored + from aorlta.reward.service import REWARD_MAX, REWARD_MIN + + fn = plugins.REWARD_FNS["hipk.v1"](None) + for tokens in (0, 1024, 2048): + c = cand(tokens) + for combo in itertools.product(*(OUTCOMES[n] for n in fn.legs)): + legs = {n: leg(n, *o) for n, o in zip(fn.legs, combo)} + reward, parts = fn.compute(legs, c) + if reward is not None: + assert type(reward) is float and REWARD_MIN <= reward <= REWARD_MAX + assert set(parts) == {"base", "broke_healthy", "length"} and all(type(v) is float for v in parts.values()) + assert parts["base"] + parts["broke_healthy"] + parts["length"] == reward + assert -0.05 <= parts["length"] <= 0.0 and parts["broke_healthy"] == 0.0 + assert fn.claims_clean(reward, parts) == all(legs[n].state == "clean" for n in fn.legs) + else: + assert isinstance(parts.get("reason"), str) and not fn.claims_clean(reward, parts) + for k in range(len(fn.legs) + 1): + d = fn.decide({n: legs[n] for n in fn.legs[:k]}, c) + if isinstance(d, Decided): + assert (d.reward, d.parts) == (reward, parts) + elif isinstance(d, Unscored): + assert reward is None + else: + assert isinstance(d, NeedMore) and d.legs == (fn.legs[k],) + + +@pytest.mark.skipif(not Path(HIPCC).exists(), reason=f"no hipcc at {HIPCC} (HIPCC)") +def test_every_harness_compiles_with_its_reference_kernel(tmp_path): + """The self-check's build step, CPU only: nothing is run.""" + proc_ = subprocess.run([sys.executable, str(EXAMPLE / "hip_kernels.py"), "selfcheck", "--hipcc", HIPCC, + "--build-only"], capture_output=True, text=True, env=dict(os.environ, TMPDIR=str(tmp_path))) + rows = [json.loads(x) for x in proc_.stdout.splitlines()] + assert proc_.returncode == 0 and len(rows) == 8 and all(r["pass"] for r in rows), proc_.stdout + proc_.stderr + + +def test_the_profile_names_only_what_the_plugin_and_the_package_provide(hipk): + from aorlta.reward.profiles import load_profile + + profile = load_profile("hipk-gpu") + assert profile.reward_fn == "hipk.v1" and profile.legs == ("gate.parse", "gate.compile", "ref.native") + assert {p.adapter for p in profile.pools.values()} == {"hipk.gate", "standin.hipcc", "hipk.run"} + raw = parse_yaml((EXAMPLE / "hipk-gpu.yaml").read_bytes()) + assert raw["adapters"]["hipk.run"]["gpus"] == [4, 5, 6, 7] + + +def test_the_score_config_keeps_the_checks_off_the_engines(tmp_path, registry): # noqa: F811 + cfg = site_config(tmp_path, "hipk-score") + from aorlta.reward.profiles import load_profile + + gpus = set(load_profile(cfg.reward.profile).adapters["hipk.run"]["gpus"]) + assert not gpus & set(cfg.rollout.engines) + + +def test_the_tutorial_names_files_that_exist(): + text = (ROOT / "docs" / "TUTORIAL.md").read_text() + for rel in ("examples/hip-kernels/hip_kernels.py", "examples/hip-kernels/tasks.json", + "examples/hip-kernels/hipk-gpu.yaml", "configs/hipk-score.yaml", "configs/hipk-train.yaml"): + assert rel in text and (ROOT / rel).exists(), rel + + +def test_config_from_dict_also_takes_the_plugin(tmp_path, registry): # noqa: F811 + data = parse_yaml((ROOT / "configs" / "hipk-score.yaml").read_bytes()) + data["run"]["root"] = str(tmp_path) + data["data"]["prompt_pools"][0]["path"] = str(EXAMPLE) + assert config_from_dict(data).reward.profile == "hipk-gpu" + + +class _CpuLeg: + """A CPU stand-in for one of the example's checks that need hipcc or a GPU (the wired test below): the compile + gate is clean, and `ref.native` passes half the sources by hash, so the rewards in a group differ.""" + + version, resource = "1", "local_cpu" + + def __init__(self, name, leg): + self.name, self.legs = name, (leg,) + + def run(self, job): + from aorlta.reward.adapters.base import make_result + + if job.leg == "gate.compile" or job.source_sha256[-1] in "02468ace": + return make_result(job, "clean", t_start=0.0, t_end=0.0) + return make_result(job, "fail", "wrong_output", t_start=0.0, t_end=0.0) + + def controls(self): + return [] + + def health(self): + return {"responsive": True} + + +@pytest.mark.heavy +def test_the_example_trains_on_cpu_through_the_real_wiring(tmp_path, hipk, tiny_base): + """The example's prompts, parse, feedback, profile and reward in the real loop, with test_wiring_e2e's tiny model + and engines. The fake engine never writes the kernel asked for, so every chain gets hipk.parse's repair turn.""" + from aorlta.orchestrator import wiring + from aorlta.rollout.engine_proc import InProcessEngine + + data = parse_yaml((ROOT / "configs" / "hipk-train.yaml").read_bytes()) + data["run"].update(root=str(tmp_path / "runs"), run_id="hipk-cpu") + data["model"]["base_dir"] = str(tiny_base) + data["data"]["prompt_pools"][0]["path"] = str(EXAMPLE) + data["rollout"].update(engines=[0, 1], K=4, P=4, max_new_tokens=512, request_timeout_s=300, + compile_adapter="fake.compile") + data["orchestrator"]["iterations"] = 1 + data["trainer"] = {"ranks": 1, "lr": 1e-2, "micro_batch_tokens": 2048} + data["sync"] = {"dir": str(tmp_path / "shm")} + cfg = config_from_dict(data) + engines = [E.TwinEngine(tiny_base, E.FAKE, i) for i in range(2)] + factories = {"hipk.gate": lambda w: hipk.HipkGate(), "standin.hipcc": lambda w: _CpuLeg("standin.hipcc", + "gate.compile"), + "hipk.run": lambda w: _CpuLeg("hipk.run", "ref.native")} + parts = wiring.Parts(tokenizer=E.ByteTokenizer(), handles=[InProcessEngine(i, e) for i, e in enumerate(engines)], + load_model=E.load_trainer_model, trainer_device="cpu", require_fast_path=False, + probes=E.PROBES, reward_factories=factories) + orch = wiring.build(cfg, parts=parts) + result = orch.run() + assert result["status"] == "completed", result + s = E.summary(cfg, 1) + assert s["trainer"]["step_taken"] and s["samples"]["trained"] > 0 and s["sync"]["landing"] == "LANDED" + eps = E.episodes(cfg, 1) + assert {e["prompt_id"].split(":")[0] for e in eps} == {"hipk"} + assert all(len(e["turns"]) == 3 and e["end_reason"] == "no_code_block" for e in eps if e["end_reason"] != "engine_error") + bases = {round(e["reward"] + 0.05, 1) for e in eps if e["status"] == "scored"} # base, give or take the length term + assert bases <= {1.0, -0.5, -1.0} and {1.0, -0.5} <= bases # correct, wrong output, and no code block at all diff --git a/aorlta/tests/examples/test_qa.py b/aorlta/tests/examples/test_qa.py new file mode 100644 index 000000000..b74c1e231 --- /dev/null +++ b/aorlta/tests/examples/test_qa.py @@ -0,0 +1,266 @@ +"""The Q&A example (examples/qa): a task with no compile step and no sanitizer, on CPU. Its configs load with the +plugin, its answers are read and normalised, its parse gives a repair turn to a reply with no answer line, its check +matches against the reference and asks the judge command, its reward keeps the service's invariants, and it trains +through the real wiring on CPU with test_wiring_e2e's tiny model and engines.""" + +from __future__ import annotations + +import json +import stat +import sys +from pathlib import Path + +import pytest + +from aorlta import plugins +from aorlta.config import config_from_dict, load_config, parse_yaml + +ROOT = Path(__file__).resolve().parents[2] +QA = ROOT / "examples" / "qa" +sys.path.insert(0, str(ROOT / "tests")) +sys.path.insert(0, str(ROOT / "tests" / "orchestrator")) +import test_wiring_e2e as E # noqa: E402 - the tiny model and its engines +from test_plugins import CharChat, registry # noqa: E402,F401 + +tiny_base = E.tiny_base + + +@pytest.fixture +def qa(registry): # noqa: F811 + plugins.load_plugins(["examples/qa/qa_task.py"]) + return sys.modules[plugins.identity()["examples/qa/qa_task.py"]["module"]] + + +def site_config(tmp_path, name, pool, **model): + site = tmp_path / "site.yaml" + site.write_text(json.dumps({"run": {"root": str(tmp_path / "runs")}, + "model": {"base_dir": str(tmp_path / "base"), **model}, + "data": {"prompt_pools": [str(QA / pool)]}})) + return load_config(ROOT / "configs" / f"{name}.yaml", site=site) + + +@pytest.mark.parametrize("name, pool", [("qa-train", "gpu-facts/train"), ("qa-score", "gpu-facts/heldout"), + ("arith-train", "arith/train"), ("arith-score", "arith/heldout")]) +def test_the_configs_load_and_run_on_one_gpu_with_no_compile(tmp_path, registry, name, pool): # noqa: F811 + from aorlta.orchestrator import wiring + + cfg = site_config(tmp_path, name, pool) + assert cfg.rollout.compile_adapter == "none" and cfg.rollout.engines == (0,) + assert cfg.reward.profile == "qa-match" and (cfg.trainer.ranks == 1 or cfg.orchestrator.mode == "score") + wiring.check_supported(cfg) + + +def test_the_profile_has_no_compile_or_sanitizer_leg(qa): + from aorlta.reward.profiles import load_profile + + assert load_profile("qa-match").legs == ("ref.answer",) + + +def test_the_held_out_questions_are_not_trained_on(qa): + for d in ("gpu-facts", "arith"): + train = {r["question"] for r in qa.load_rows(QA / d / "train")} + held = {r["question"] for r in qa.load_rows(QA / d / "heldout")} + assert train and held and not train & held + assert len(qa.load_rows(QA / "gpu-facts" / "train")) + len(qa.load_rows(QA / "gpu-facts" / "heldout")) == 24 + + +def test_the_arith_files_are_what_make_arith_writes(tmp_path, monkeypatch): + import importlib.util + + spec = importlib.util.spec_from_file_location("make_arith", QA / "make_arith.py") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + monkeypatch.setattr(mod, "HERE", tmp_path) + mod.main() + for part in ("train", "heldout"): + assert (tmp_path / "arith" / part / "qa.jsonl").read_bytes() == (QA / "arith" / part / "qa.jsonl").read_bytes() + for row in json.loads("[" + ",".join((QA / "arith" / "heldout" / "qa.jsonl").read_text().splitlines()) + "]"): + a, b = row["question"].removeprefix("What is ").removesuffix("?").split(" × ") + assert int(row["answer"]) == int(a) * int(b) + + +@pytest.mark.parametrize("reply, answer", [ + ("Some working.\nAnswer: 1,024.", "1024"), ("**Answer:** `hipMalloc`", "hipmalloc"), + ("answer: Local Data Share", "local data share"), ("Answer: 3\nthen\nAnswer: 42 ", "42"), +]) +def test_the_final_answer_is_read_and_normalised(qa, reply, answer): + assert qa.normalise(qa.final_answer(reply)) == answer + + +def test_no_answer_line_is_no_answer(qa): + assert qa.final_answer("The answer is 42.") is None + + +@pytest.mark.parametrize("reply", ["Answer:\n42", "Answer: \nthe rest of my working", "Answer: 7\nmore working\nAnswer:"]) +def test_an_answer_line_is_read_on_its_own(qa, reply): + """An empty `Answer:` does not take the next line's text, and it is the last marker that decides.""" + assert qa.final_answer(reply) is None + + +def test_the_parse_gives_a_repair_turn_to_a_reply_without_an_answer(qa): + from aorlta.rollout.chain import make_feedback_template, make_parse_adapter + from aorlta.rollout.compile_check import make_compiler + + parse, fb = make_parse_adapter("qa.parse"), make_feedback_template("qa.feedback") + out = parse.check("I think it is 64.", "off", None) + assert out.kind == "rejected" and out.rejection[1] == "no_answer" and "Answer:" in fb.build_apply(*out.rejection).text + good = parse.check("Working...\nAnswer: 64", "off", None) + assert good.kind == "edits" and good.code == "64" + assert make_compiler("none", Path("/nonexistent")).check(good.code, "c0-0-t0").ok + + +def job_for(qa, tmp_path, row, reply): + from aorlta.schemas import LegJob + + resp = tmp_path / "response.txt" + resp.write_text(reply) + return LegJob(job_id="j", leg="ref.answer", candidate_id="c", control_id=None, source_sha256="s" * 64, + task_sha256=qa.row_sha256(row), inputs={"response": str(resp)}, output_dir=str(tmp_path)) + + +def adapter(qa, settings=None): + ctx = plugins.CheckContext(name="qa.match", worker_id="match/1", settings=settings or {}, clock=lambda: 0.0, + timeout_s=30.0, pools=(str(QA / "gpu-facts" / "train"),)) + return plugins.CHECK_ADAPTERS["qa.match"](ctx) + + +@pytest.mark.parametrize("reply, want", [ + ("Answer: 64", ("clean", None)), ("Answer: 64 threads", ("clean", None)), ("Answer: 32", ("fail", "wrong_answer")), + ("sixty-four", ("fail", "no_answer")), +]) +def test_the_check_matches_the_reference_and_its_aliases(qa, tmp_path, reply, want): + row = next(r for r in qa.load_rows(QA / "gpu-facts" / "train") if r["id"] == "wave-cdna") + res = adapter(qa).run(job_for(qa, tmp_path, row, reply)) + assert (res.state, res.reason) == want + + +@pytest.mark.parametrize("judge_out, want", [ + ('{"verdict": "correct"}', ("clean", None)), ('{"verdict": "wrong"}', ("fail", "wrong_answer")), + ('not json', ("unsettled", "judge_no_verdict")), ('{"verdict": "maybe"}', ("unsettled", "judge_no_verdict")), +]) +def test_a_judge_command_decides_an_answer_that_does_not_match(qa, tmp_path, judge_out, want): + """The shape of a task whose reward is an external tool's verdict: the judge reads the question, the reference + and the answer on stdin and prints its verdict; no verdict is unsettled, never clean.""" + judge = tmp_path / "judge" + judge.write_text(f"#!/bin/bash\ncat > {tmp_path}/judge-input.json\necho '{judge_out}'\n") + judge.chmod(judge.stat().st_mode | stat.S_IXUSR) + row = next(r for r in qa.load_rows(QA / "gpu-facts" / "train") if r["id"] == "wave-cdna") + res = adapter(qa, {"judge": [str(judge)]}).run(job_for(qa, tmp_path, row, "Answer: a wave of sixty-four")) + assert (res.state, res.reason) == want + seen = json.loads((tmp_path / "judge-input.json").read_text()) + assert seen == {"question": row["question"], "reference": "64", "answer": "a wave of sixty-four"} + + +def test_the_reward_keeps_the_services_invariants(qa): + from aorlta.reward.service import REWARD_MAX, REWARD_MIN + from aorlta.schemas import Candidate, LegResult, TaskRef + + fn = plugins.REWARD_FNS["qa.v1"](None) + for state, reason, base in (("clean", None, 1.0), ("fail", "wrong_answer", -0.5), ("fail", "no_answer", -1.0), + ("fail", "other", None), ("unsettled", "x", None), ("not_run", None, None)): + for tokens in (0, 512): + leg = LegResult("ref.answer", state, reason, [], 1, 1, None, None, False, 0.0, 0.0, 0.0) + c = Candidate("c", "r", 1, "qa:x", 0, "v", "", tokens, 512, TaskRef("qa:x", None, None, None, "t"), + "qa-match", None) + reward, parts = fn.compute({"ref.answer": leg}, c) + d = fn.decide({"ref.answer": leg}, c) + if base is None: + assert reward is None and type(d).__name__ == "Unscored" + continue + assert parts["base"] == base and REWARD_MIN <= reward <= REWARD_MAX + assert parts["base"] + parts["broke_healthy"] + parts["length"] == reward and (d.reward, d.parts) == ( + reward, parts) + assert fn.claims_clean(reward, parts) == (state == "clean") + assert type(fn.decide({}, c)).__name__ == "NeedMore" + + +@pytest.mark.heavy +def test_the_qa_example_trains_on_cpu_through_the_real_wiring(tmp_path, qa, tiny_base): + """No compile step, no sanitizer, the real answer check: the fake engine never writes an `Answer:` line, so + every chain gets qa.parse's repair turn and is scored no_answer, and the iteration still trains and lands.""" + from aorlta.orchestrator import wiring + from aorlta.rollout.engine_proc import InProcessEngine + + data = parse_yaml((ROOT / "configs" / "qa-train.yaml").read_bytes()) + data["run"].update(root=str(tmp_path / "runs"), run_id="qa-cpu") + data["model"]["base_dir"] = str(tiny_base) + data["data"]["prompt_pools"][0]["path"] = str(QA / "gpu-facts" / "train") + data["rollout"].update(engines=[0, 1], K=4, P=4, request_timeout_s=300) + data["orchestrator"]["iterations"] = 1 + data["trainer"] = {"ranks": 1, "lr": 1e-2, "micro_batch_tokens": 2048} + data["sync"] = {"dir": str(tmp_path / "shm")} + cfg = config_from_dict(data) + engines = [E.TwinEngine(tiny_base, E.FAKE, i) for i in range(2)] + parts = wiring.Parts(tokenizer=E.ByteTokenizer(), handles=[InProcessEngine(i, e) for i, e in enumerate(engines)], + load_model=E.load_trainer_model, trainer_device="cpu", require_fast_path=False, + probes=E.PROBES) + result = wiring.build(cfg, parts=parts).run() + assert result["status"] == "completed", result + s = E.summary(cfg, 1) + assert s["samples"]["scored"] == s["samples"]["total"] - sum(1 for e in E.episodes(cfg, 1) + if e["end_reason"] == "engine_error") + assert s["sync"]["landing"] == "LANDED" + eps = [e for e in E.episodes(cfg, 1) if e["end_reason"] != "engine_error"] + assert all(len(e["turns"]) == 2 for e in eps) # the answer, and qa.parse's repair turn + scored = [e for e in eps if e["status"] == "scored"] + assert scored and all(e["legs"]["ref.answer"] == "fail" and -1.05 <= e["reward"] <= -1.0 for e in scored) + + +def test_the_judge_is_part_of_the_checkers_identity(qa, tmp_path): + """Cached ref.answer legs are keyed by the checker's version: turning the judge on, changing its command, its + timeout or its script is a new version, so a cached wrong_answer is not reused where the judge would decide.""" + judge = tmp_path / "judge" + judge.write_text("#!/bin/bash\necho '{\"verdict\": \"correct\"}'\n") + judge.chmod(0o755) + versions = [adapter(qa).version, adapter(qa, {"judge": [str(judge)]}).version, + adapter(qa, {"judge": [str(judge), "--strict"]}).version, + adapter(qa, {"judge": [str(judge)], "judge_timeout_s": 5}).version] + assert len(set(versions)) == 4 and versions[0] == "2" + assert adapter(qa, {"judge": [str(judge)]}).version == versions[1] + judge.write_text("#!/bin/bash\necho '{\"verdict\": \"wrong\"}'\n") + assert adapter(qa, {"judge": [str(judge)]}).version != versions[1] + + +def test_a_judge_that_times_out_or_is_retired_leaves_nothing_running(qa, tmp_path): + """The judge's whole process group is killed, on a timeout and when the worker is retired (close()), not only + the command itself: a wrapper's children went on running before.""" + import os + import threading + import time + + row = next(r for r in qa.load_rows(QA / "gpu-facts" / "train") if r["id"] == "wave-cdna") + for how in ("timeout", "close"): + pid = tmp_path / f"{how}.pid" + judge = tmp_path / f"judge-{how}" + judge.write_text(f"#!/bin/bash\nsleep 60 &\necho $! > {pid}\nwait\n") + judge.chmod(0o755) + a = adapter(qa, {"judge": [str(judge)], "judge_timeout_s": 1.0 if how == "timeout" else 60.0}) + if how == "close": + threading.Thread(target=lambda: (time.sleep(1.0), a.close()), daemon=True).start() + t0 = time.monotonic() + res = a.run(job_for(qa, tmp_path, row, "Answer: a wave of sixty-four")) + assert (res.state, res.reason) == ("unsettled", "judge_no_verdict") + assert time.monotonic() - t0 < 15, how # not the adapter's own 30 s, nor the child's 60 s + child = int(pid.read_text()) + deadline = time.monotonic() + 5 + while os.path.exists(f"/proc/{child}") and time.monotonic() < deadline: + time.sleep(0.05) + assert not os.path.exists(f"/proc/{child}"), how + a.close() + + +def test_a_judge_found_through_path_is_part_of_the_checkers_identity(qa, tmp_path, monkeypatch): + """A bare command name is resolved as PATH finds it: replacing that executable, or which one PATH selects, is a new + checker version.""" + bins = [tmp_path / "a", tmp_path / "b"] + for d in bins: + d.mkdir() + (d / "my-judge").write_text("#!/bin/bash\necho '{\"verdict\": \"correct\"}'\n") + (d / "my-judge").chmod(0o755) + monkeypatch.setenv("PATH", f"{bins[0]}:/usr/bin:/bin") + first = adapter(qa, {"judge": ["my-judge"]}).version + assert adapter(qa, {"judge": ["my-judge"]}).version == first + (bins[0] / "my-judge").write_text("#!/bin/bash\necho '{\"verdict\": \"wrong\"}'\n") + second = adapter(qa, {"judge": ["my-judge"]}).version + monkeypatch.setenv("PATH", f"{bins[1]}:/usr/bin:/bin") + assert len({first, second, adapter(qa, {"judge": ["my-judge"]}).version}) == 3 diff --git a/aorlta/tests/reward/test_evidence.py b/aorlta/tests/reward/test_evidence.py index 0ba995209..ef1a01071 100644 --- a/aorlta/tests/reward/test_evidence.py +++ b/aorlta/tests/reward/test_evidence.py @@ -23,6 +23,10 @@ "ab/abcd/san.asan-0123456789abcdef/asan/asan.log": True, "ab/abcd/san.waitcheck-0123456789abcdef/waitcheck-0.log": True, "ab/abcd/response.txt": False, + "ab/abcd/ref.native-0123456789abcdef/r1/run.log": True, # a task's own check, as the worked HIP example's + "ab/abcd/ref.native-0123456789abcdef/r1/harness.hip": True, + "ab/abcd/ref.native-0123456789abcdef/r1/record.json": True, + "ab/abcd/ref.native-0123456789abcdef/r1/candidate.co": False, } diff --git a/aorlta/tests/tools/test_eval_compare.py b/aorlta/tests/tools/test_eval_compare.py new file mode 100644 index 000000000..2a8f14653 --- /dev/null +++ b/aorlta/tests/tools/test_eval_compare.py @@ -0,0 +1,210 @@ +"""tools/eval_compare.py over two score runs made with the fake components: the headline numbers are the runs' own +totals, every prompt gets a row, samples are written, and runs that scored different prompts or sampled differently +are refused.""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[2] + + +def _load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + sys.modules[name] = mod + spec.loader.exec_module(mod) + return mod + + +X = _load("eval_compare", ROOT / "tools" / "eval_compare.py") + + +def score_run(tmp_path: Path, name: str, **over) -> Path: + from aorlta.orchestrator.run import main + + c = yaml.safe_load((ROOT / "configs" / "score-standin.yaml").read_text()) + c["run"]["root"] = str(tmp_path / name / "runs") + c["model"]["base_dir"] = str(tmp_path / name / "base") + c["data"]["prompt_pools"][0]["path"] = str(tmp_path / "pool") + c["sync"] = {"dir": str(tmp_path / name / "shm")} + for k, v in over.items(): + c["rollout"][k] = v + path = tmp_path / f"{name}.yaml" + path.write_text(yaml.safe_dump(c)) + assert main(["--config", str(path), "--components", "fake"]) == 0 + return tmp_path / name / "runs" / "standin" / c["run"]["run_id"] + + +def test_two_score_runs_compare_side_by_side(tmp_path, capsys): + base, trained = score_run(tmp_path, "base"), score_run(tmp_path, "trained") + assert X.main(["--base", str(base), "--trained", str(trained), "--out", str(tmp_path / "out"), "--samples", "3"]) == 0 + result = json.loads((tmp_path / "out" / "eval-compare.json").read_text()) + total = next(json.loads(x) for x in (base / "score" / "report.jsonl").read_text().splitlines() + if json.loads(x)["kind"] == "total") + assert result["base"]["pass_rate"] == total["clean"] / total["samples"] + assert result["base"]["mean_reward"] == total["reward"]["mean"] + assert len(result["prompts"]) == 16 and result["delta"]["pass_rate"] == 0.0 # the fakes are seeded alike + md = (tmp_path / "out" / "eval-compare.md").read_text() + assert md.count("**base**") == 3 and md.count("**trained**") == 3 and "pass rate" in md + assert "pass rate" in capsys.readouterr().out + + +def test_runs_that_sampled_differently_are_refused(tmp_path): + base, other = score_run(tmp_path, "base"), score_run(tmp_path, "other", temperature=1.0) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert not (tmp_path / "out" / "eval-compare.json").exists() + + +def test_a_directory_that_is_not_a_score_run_is_refused(tmp_path): + base = score_run(tmp_path, "base") + assert X.main(["--base", str(base), "--trained", str(tmp_path), "--out", str(tmp_path / "out")]) == 2 + + +def test_runs_with_another_batch_layout_or_reward_are_refused(tmp_path, capsys): + """Seeds follow the iteration number, which a different P changes; a different reward scores another thing.""" + base = score_run(tmp_path, "base") + other = score_run(tmp_path, "p8") + _rewrite(other / "score" / "report.jsonl", + lambda r: dict(r, options=dict(r["options"], prompts_per_batch=8)) if r["kind"] == "run" else r) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "differ in options" in capsys.readouterr().err + other = score_run(tmp_path, "rfn") + _rewrite(other / "score" / "report.jsonl", + lambda r: dict(r, reward=dict(r["reward"], reward_fn="other.v1")) if r["kind"] == "run" else r) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "differ in reward" in capsys.readouterr().err + + +def _rewrite(report: Path, edit) -> None: + rows = [json.loads(x) for x in report.read_text().splitlines()] + report.write_text("".join(json.dumps(edit(r)) + "\n" for r in rows)) + + +def test_runs_whose_tasks_order_or_sample_counts_differ_are_refused(tmp_path, capsys): + base = score_run(tmp_path, "base") + first = next(json.loads(x)["prompt_id"] for x in (base / "score" / "report.jsonl").read_text().splitlines() + if json.loads(x)["kind"] == "prompt") + for name, edit, what in ( + ("task", lambda r: dict(r, task_sha256="0" * 64) if r.get("prompt_id") == first else r, "task_sha256"), + ("count", lambda r: dict(r, samples=r["samples"] - 1) if r.get("prompt_id") == first else r, "samples")): + other = score_run(tmp_path, name) + _rewrite(other / "score" / "report.jsonl", edit) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert what in capsys.readouterr().err + other = score_run(tmp_path, "order") + rep = other / "score" / "report.jsonl" + rows = [json.loads(x) for x in rep.read_text().splitlines()] + prompts = [r for r in rows if r["kind"] == "prompt"] + rows = [rows[0], *reversed(prompts), rows[-1]] + rep.write_text("".join(json.dumps(r) + "\n" for r in rows)) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "same order" in capsys.readouterr().err + + +def test_a_sample_with_no_reply_is_shown_as_such(tmp_path): + """A chain that died before its first reply has no turns; the report shows it rather than crashing.""" + base, trained = score_run(tmp_path, "base"), score_run(tmp_path, "trained") + for f in trained.glob("iterations/iter-*/episodes.jsonl"): + eps = [json.loads(x) for x in f.read_text().splitlines()] + f.write_text("".join(json.dumps(dict(e, turns=[], final_turn=-1, end_reason="engine_error")) + "\n" + for e in eps)) + assert X.main(["--base", str(base), "--trained", str(trained), "--out", str(tmp_path / "out")]) == 0 + md = (tmp_path / "out" / "eval-compare.md").read_text() + assert "(no reply: the chain ended engine_error before its first turn)" in md + + +def _episode_file(run_dir: Path) -> Path: + return next(run_dir.glob("iterations/iter-*/episodes.jsonl")) + + +def _rewrite_episodes(run_dir: Path, edit) -> None: + f = _episode_file(run_dir) + eps = edit([json.loads(x) for x in f.read_text().splitlines()]) + f.write_text("".join(json.dumps(e) + "\n" for e in eps)) + + +def test_runs_without_the_evidence_the_checks_read_are_refused(tmp_path, capsys): + """A missing run plan, or episode records that do not cover the report one to one, would leave the fingerprint + checks nothing to compare; such a run is refused rather than passed.""" + base = score_run(tmp_path, "base") + + def drop_samples(eps): # three of the first prompt's four samples gone + pid = eps[0]["prompt_id"] + return [e for e in eps if e["prompt_id"] != pid or e["k"] == 0] + + def duplicate(eps): # one sample recorded twice in place of another, so the count still agrees + twin = next(e for e in eps[1:] if e["prompt_id"] == eps[0]["prompt_id"]) + return [e for e in eps if e is not twin] + [eps[0]] + + def renumber(eps): # one sample's k moved, so that sample has no partner in the other run + return [dict(eps[0], k=99), *eps[1:]] + + def unlisted(eps): + return [*eps, dict(eps[0], prompt_id="not-in-the-report")] + + for name, edit, what in (("dropped", drop_samples, "committed episode records; its report counts"), + ("duplicate", duplicate, "more than one episode record"), + ("unpaired", renumber, "do not pair"), + ("unlisted", unlisted, "does not list")): + other = score_run(tmp_path, name) + _rewrite_episodes(other, edit) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2, name + assert what in capsys.readouterr().err, name + assert X.main(["--base", str(other), "--trained", str(base), "--out", str(tmp_path / "out")]) == 2, name + assert what in capsys.readouterr().err, name + + other = score_run(tmp_path, "plan") + (other / "run_plan.json").unlink() + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "no run_plan.json" in capsys.readouterr().err + + other = score_run(tmp_path, "partial") # the batch never committed: only iter-.partial holds its records + d = _episode_file(other).parent + d.rename(d.with_name(d.name + ".partial")) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "no committed episode records" in capsys.readouterr().err + assert not (tmp_path / "out" / "eval-compare.json").exists() + + other = score_run(tmp_path, "leftover") # a leftover .partial beside the committed batch is not read at all + f = _episode_file(other) + stray = f.parent.with_name(f.parent.name + ".partial") + stray.mkdir() + (stray / f.name).write_text("".join(json.dumps(dict(json.loads(x), k=99)) + "\n" + for x in f.read_text().splitlines())) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 0 + + +def _edit_episodes(run_dir: Path, edit) -> None: + for f in run_dir.glob("iterations/iter-*/episodes.jsonl"): + eps = [json.loads(x) for x in f.read_text().splitlines()] + f.write_text("".join(json.dumps(edit(e)) + "\n" for e in eps)) + + +def test_runs_whose_prompts_seeds_reward_code_or_plugins_differ_are_refused(tmp_path, capsys): + """Names can agree while what they name changed: a Q&A template (the first prompt's token hash), the seeds, the + reward function's code, or a plugin's file.""" + base = score_run(tmp_path, "base") + + def first_turn(**kw): + return lambda e: dict(e, turns=[dict(e["turns"][0], **kw), *e["turns"][1:]]) if e["turns"] else e + + for name, edit, what in (("prompt", first_turn(prompt_ids_sha256="0" * 64), "first prompts or seeds"), + ("seed", first_turn(seed=7), "first prompts or seeds"), + ("code", lambda e: dict(e, reward_code_sha256="0" * 64) if e["status"] == "scored" + else e, "different reward code")): + other = score_run(tmp_path, name) + _edit_episodes(other, edit) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2, name + assert what in capsys.readouterr().err + other = score_run(tmp_path, "plugin") + plan = json.loads((other / "run_plan.json").read_text()) + plan["plugins"] = {"my_plugin.py": {"module": "m", "file": "f", "sha256": "1" * 64}} + (other / "run_plan.json").write_text(json.dumps(plan)) + assert X.main(["--base", str(base), "--trained", str(other), "--out", str(tmp_path / "out")]) == 2 + assert "different plugin files" in capsys.readouterr().err diff --git a/aorlta/tools/eval_compare.py b/aorlta/tools/eval_compare.py new file mode 100644 index 000000000..2bcc18b1b --- /dev/null +++ b/aorlta/tools/eval_compare.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Compare two score runs: a base model and a trained checkpoint, on the same held-out prompts. + +Run one score config twice (`orchestrator.mode: score`), once on the base model and once with `model.adapter_dir` +set to the trained adapter (`tools/export_adapter.py`), each under its own `run.root` (a site file sets both), then: + + eval_compare.py --base BASE_RUN_DIR --trained TRAINED_RUN_DIR [--tokenizer MODEL_DIR] [--samples 6] [--out DIR] + +It prints the pass rate (the share of samples whose reward claims clean: a correct answer), the mean reward and the +scored fraction of each, and the per-prompt change, and writes `/eval-compare.json` and `/eval-compare.md`. +The markdown holds, for the prompts that changed most, one sample of each model: its reward, its checks' states and, +with `--tokenizer`, its final reply decoded. It refuses two runs that did not score the same prompts in the same +order, with the same sampling settings, batch layout, task hashes, first prompts and seeds, reward code and plugin +files, since their numbers would not be comparable, and a run whose run plan is missing or whose committed episode +records do not cover every sample of its report, since those checks would have nothing to read. An unscored sample is +never counted as passing. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + + +class Refused(ValueError): + pass + + +def read_report(run_dir: Path) -> dict: + path = Path(run_dir) / "score" / "report.jsonl" + if not path.is_file(): + raise Refused(f"{run_dir}: no score/report.jsonl (is it a finished score run?)") + rows = [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()] + run = next((r for r in rows if r.get("kind") == "run"), None) + total = next((r for r in rows if r.get("kind") == "total"), None) + prompts = {r["prompt_id"]: r for r in rows if r.get("kind") == "prompt"} + if run is None or total is None or not prompts: + raise Refused(f"{path}: no run, total or prompt rows") + return {"run": run, "total": total, "prompts": prompts} + + +COMMITTED = re.compile(r"iter-\d+") # a batch is committed when iterations/iter-/ appears; iter-.partial is not + + +def episodes(run_dir: Path) -> dict[str, list[dict]]: + """The committed episode records, by prompt.""" + out: dict[str, list[dict]] = {} + for f in sorted((Path(run_dir) / "iterations").glob("iter-*/episodes.jsonl")): + if not COMMITTED.fullmatch(f.parent.name): + continue + for line in f.read_text(encoding="utf-8").splitlines(): + e = json.loads(line) + out.setdefault(e["prompt_id"], []).append(e) + return out + + +def paired_episodes(base_dir: Path, trained_dir: Path, base: dict, trained: dict) -> tuple[dict, dict]: + """Both runs' committed episodes, refused unless each report's every sample has exactly one record (its + iteration and k) and the two runs' records pair one to one. A record with no turns (a chain that ended before its + first reply) is legal.""" + runs = [] + for run_dir, rep in ((base_dir, base), (trained_dir, trained)): + eps = episodes(run_dir) + if not eps: + raise Refused(f"{run_dir}: no committed episode records (iterations/iter-/episodes.jsonl)") + if extra := sorted(set(eps) - set(rep["prompts"])): + raise Refused(f"{run_dir}: episode records for prompts its report does not list: {extra[:3]}") + keys = {} + for pid, row in rep["prompts"].items(): + got = [(e["iteration"], e["k"]) for e in eps.get(pid, [])] + if len(got) != len(set(got)): + raise Refused(f"{run_dir}: {pid} has more than one episode record for a sample") + if len(got) != row.get("samples"): + raise Refused(f"{run_dir}: {pid} has {len(got)} committed episode records; its report counts " + f"{row.get('samples')} samples") + keys[pid] = set(got) + runs.append((eps, keys)) + (eb, kb), (et, kt) = runs + for pid in base["prompts"]: + if kb[pid] != kt[pid]: + raise Refused(f"{pid}: the two runs' samples do not pair (iteration, k): " + f"{sorted(kb[pid] ^ kt[pid])[:3]} are in one run only") + return eb, et + + +def fingerprints(base_dir: Path, trained_dir: Path, base: dict, trained: dict) -> None: + """Refuse two runs whose names agree but whose inputs or scoring differ: the paired samples' first prompt (its + token hash, so a changed template shows) and seed, the reward function's version and code hash on every scored + sample, and the plugin files the run plans hashed (a task's own checks and reward live there). A run with no + run plan, or whose episode records do not cover its report, is refused: there is nothing to check it against.""" + pb, pt = (_plugin_hashes(d) for d in (base_dir, trained_dir)) + eb, et = paired_episodes(base_dir, trained_dir, base, trained) + for pid in sorted(eb): + first = {(e["k"], e["iteration"]): (e["turns"][0]["prompt_ids_sha256"], e["turns"][0]["seed"]) + for e in eb[pid] if e.get("turns")} + for e in et[pid]: + got = first.get((e["k"], e["iteration"])) + if e.get("turns") and got is not None and got != (e["turns"][0]["prompt_ids_sha256"], + e["turns"][0]["seed"]): + raise Refused(f"{pid} sample {e['k']}: the two runs' first prompts or seeds differ") + rb, rt = ({(e.get("reward_version"), e.get("reward_code_sha256")) for eps in run.values() for e in eps + if e.get("status") == "scored"} for run in (eb, et)) + if rb and rt and rb != rt: + raise Refused(f"the two runs were scored by different reward code: {sorted(rb)} against {sorted(rt)}") + if pb != pt: + raise Refused(f"the two runs loaded different plugin files: {pb} against {pt}") + + +def _plugin_hashes(run_dir: Path) -> dict: + path = Path(run_dir) / "run_plan.json" + if not path.is_file(): + raise Refused(f"{run_dir}: no run_plan.json, so its plugin files cannot be checked") + plan = json.loads(path.read_text(encoding="utf-8")) + return {k: (v or {}).get("sha256") for k, v in sorted((plan.get("plugins") or {}).items())} + + +def headline(rep: dict) -> dict: + t = rep["total"] + return {"samples": t["samples"], "scored": t["scored"], "scored_fraction": t["scored_fraction"], + "pass_rate": t["clean"] / t["samples"] if t["samples"] else 0.0, "clean": t["clean"], + "mean_reward": (t.get("reward") or {}).get("mean"), + "model": rep["run"].get("model", {}).get("kind")} + + +def compare(base_dir: Path, trained_dir: Path) -> dict: + base, trained = read_report(base_dir), read_report(trained_dir) + if list(base["prompts"]) != list(trained["prompts"]): + raise Refused("the two runs did not score the same prompts in the same order") + for key in ("sampling", "options", "reward"): # options: P and the passes, which decide each prompt's seeds + sb, st = base["run"].get(key), trained["run"].get(key) + if sb != st: + raise Refused(f"the two runs differ in {key}: {sb} against {st}") + for pid, b in base["prompts"].items(): + t = trained["prompts"][pid] + for key in ("task_sha256", "samples"): + if b.get(key) != t.get(key): + raise Refused(f"{pid}: the two runs differ in {key}: {b.get(key)} against {t.get(key)}") + fingerprints(base_dir, trained_dir, base, trained) + sb = base["run"].get("sampling") + rows = [] + for pid in sorted(base["prompts"]): + b, t = base["prompts"][pid], trained["prompts"][pid] + rb, rt = (b.get("reward") or {}).get("mean"), (t.get("reward") or {}).get("mean") + rows.append({"prompt_id": pid, "base_pass": b["clean"] / b["samples"], "trained_pass": t["clean"] / t["samples"], + "base_reward": rb, "trained_reward": rt, + "delta_pass": t["clean"] / t["samples"] - b["clean"] / b["samples"]}) + hb, ht = headline(base), headline(trained) + return {"base": {"run_dir": str(base_dir), **hb}, "trained": {"run_dir": str(trained_dir), **ht}, + "delta": {"pass_rate": ht["pass_rate"] - hb["pass_rate"], + "mean_reward": (None if hb["mean_reward"] is None or ht["mean_reward"] is None + else ht["mean_reward"] - hb["mean_reward"])}, + "prompts": rows, "sampling": sb, + "improved": sum(r["delta_pass"] > 0 for r in rows), "worse": sum(r["delta_pass"] < 0 for r in rows)} + + +def sample_text(ep: dict, tok) -> str: + if not ep.get("turns"): # a chain that ended before its first reply (engine_error) is a legal episode + return f"(no reply: the chain ended {ep.get('end_reason')} before its first turn)" + ids = ep["turns"][-1]["response_ids"] + if tok is None: + return f"({len(ids)} reply tokens; pass --tokenizer to decode)" + return tok.decode(ids, skip_special_tokens=True).strip() + + +def markdown(result: dict, base_dir: Path, trained_dir: Path, tok, samples: int) -> str: + b, t, d = result["base"], result["trained"], result["delta"] + fmt = lambda x: "-" if x is None else f"{x:.3f}" # noqa: E731 + dr = "-" if d["mean_reward"] is None else f"{d['mean_reward']:+.3f}" + lines = ["# Base against trained, on the same held-out prompts", "", + "| | base | trained | change |", "|---|---|---|---|", + f"| pass rate | {b['pass_rate']:.3f} ({b['clean']}/{b['samples']}) | {t['pass_rate']:.3f} " + f"({t['clean']}/{t['samples']}) | {d['pass_rate']:+.3f} |", + f"| mean reward | {fmt(b['mean_reward'])} | {fmt(t['mean_reward'])} | {dr} |", + f"| scored fraction | {b['scored_fraction']:.3f} | {t['scored_fraction']:.3f} | |", "", + f"Prompts improved: {result['improved']}; worse: {result['worse']}; of {len(result['prompts'])}.", "", + "## Samples, for the prompts that changed most", ""] + eb, et = episodes(base_dir), episodes(trained_dir) + picked = sorted(result["prompts"], key=lambda r: (-abs(r["delta_pass"]), r["prompt_id"]))[:samples] + for r in picked: + lines.append(f"### {r['prompt_id']}: pass {r['base_pass']:.2f} -> {r['trained_pass']:.2f}") + for name, eps in (("base", eb), ("trained", et)): + got = eps.get(r["prompt_id"]) or [] + if not got: + continue + e = next((x for x in got if x.get("turns")), got[0]) # a sample with a reply, if there is one + lines += ["", f"**{name}**: reward {fmt(e.get('reward'))}, {e['status']}, checks {e.get('legs')}", "", + "```text", sample_text(e, tok)[:3000], "```"] + lines.append("") + return "\n".join(lines) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="eval_compare.py", description=__doc__.split("\n\n")[0]) + ap.add_argument("--base", type=Path, required=True, help="the base model's score run directory") + ap.add_argument("--trained", type=Path, required=True, help="the trained model's score run directory") + ap.add_argument("--tokenizer", default=None, help="a model directory whose tokenizer decodes the sample replies") + ap.add_argument("--samples", type=int, default=6) + ap.add_argument("--out", type=Path, default=Path(".")) + args = ap.parse_args(argv) + try: + result = compare(args.base, args.trained) + except Refused as exc: + print(f"refused: {exc}", file=sys.stderr) + return 2 + tok = None + if args.tokenizer: + from transformers import AutoTokenizer + + tok = AutoTokenizer.from_pretrained(args.tokenizer, local_files_only=True) + args.out.mkdir(parents=True, exist_ok=True) + (args.out / "eval-compare.json").write_text(json.dumps(result, indent=1) + "\n", encoding="utf-8") + (args.out / "eval-compare.md").write_text(markdown(result, args.base, args.trained, tok, args.samples) + "\n", + encoding="utf-8") + b, t = result["base"], result["trained"] + print(f"pass rate {b['pass_rate']:.3f} -> {t['pass_rate']:.3f} ({result['delta']['pass_rate']:+.3f}); " + f"mean reward {b['mean_reward']} -> {t['mean_reward']}; improved {result['improved']}, worse " + f"{result['worse']} of {len(result['prompts'])} prompts; written to {args.out}/eval-compare.md") + return 0 + + +if __name__ == "__main__": + sys.exit(main())