Repository navigation
feat(aorlta): two worked examples (HIP kernels, Q&A), before/after eval, VALIDATION.md - #613
vivekkhandelwal1 wants to merge 26 commits into
Conversation
…m a plugin A run config lists plugin modules under `plugins:`. Each is imported before the config is checked and registers its components with aorlta.plugins: a task (the prompt pool's records), a parse adapter, a feedback template, a compile check, a check profile, check adapters and a reward function. Every place that looked a name up in a fixed list now also looks in the registry, so a new task needs no edit to the package. The run plan records each plugin file's sha256. A plugin's data is handled as the stand-in source's, and its reward keeps the service's range and parts; both are stated in ADAPTING.md.
Each engine's start-up self-test samples 4 chats with two seeds and needs 3 of the 4 pairs to differ. On the first batch's own prompts that fails a sound sampler when the task is precise: on MI350X, the hip-kernels example's prompts (write a vector add, a SAXPY, ...) gave the same 128 tokens for both seeds on 2 of 4 prompts, on every engine. A plugin task now self-tests on fixed open-ended chats (wiring.SELF_TEST_CHATS), or on its own (Task.selftest_messages). The stand-in path is unchanged.
examples/hip-kernels/ asks the model to write a HIP kernel from its description (vector add, SAXPY, ReLU, scale and shift, clamp, reverse, a per-block sum, a transpose). Its parse gives a repair turn when the reply has no code block or does not define the kernel asked for. Its check builds the answer with a generated host program, runs it on a GPU and compares every output element with a host reference; an element never written, and a write past an array's end, are caught. hipk.v1 scores the outcome. Each task comes with a correct reference kernel, which 'hip_kernels.py selfcheck' builds and runs. docs/TUTORIAL.md walks through it: what to write, where to register it, and how to check, test, score and train it. configs/hipk-score.yaml and hipk-train.yaml run it. It needs none of the stand-in sanitizer tools, only the run image's hipcc and GPUs.
There was a problem hiding this comment.
🟡 Changes recommended
The correctness verdict trusts candidate-controlled stdout, allowing incorrect kernels to forge success records and receive clean results.
4 open issues
What changed in this PR
Builds on the plugin support from #611 with a copyable HIP kernel-writing task and an end-to-end tutorial, without adding task-specific code to the package.
Changes:
- Adds eight kernel tasks, reference implementations, GPU correctness checks, and a graded reward function.
- Provides scoring and training configs, plus setup and adaptation guidance.
- Adds CPU-based behavior tests, reward-invariant checks, optional harness compilation, and a wired training test.
| File | Description |
|---|---|
aorlta/tests/examples/test_hip_kernels.py |
Tests plugin wiring, parsing, checks, rewards, and training. |
aorlta/README.md |
Introduces the example and links the tutorial. |
aorlta/examples/hip-kernels/tasks.json |
Defines eight tasks, inputs, references, and tolerances. |
aorlta/examples/hip-kernels/reference/vector_add.hip |
Adds the vector-add reference kernel. |
aorlta/examples/hip-kernels/reference/transpose.hip |
Adds the matrix-transpose reference kernel. |
aorlta/examples/hip-kernels/reference/scale_shift.hip |
Adds the scale-and-shift reference kernel. |
aorlta/examples/hip-kernels/reference/saxpy.hip |
Adds the SAXPY reference kernel. |
aorlta/examples/hip-kernels/reference/reverse.hip |
Adds the array-reversal reference kernel. |
aorlta/examples/hip-kernels/reference/relu.hip |
Adds the ReLU reference kernel. |
aorlta/examples/hip-kernels/reference/clamp.hip |
Adds the clamp reference kernel. |
aorlta/examples/hip-kernels/reference/block_sum.hip |
Adds the per-block reduction reference kernel. |
aorlta/examples/hip-kernels/README.md |
Summarizes the example’s files and entry points. |
aorlta/examples/hip-kernels/hipk-gpu.yaml |
Configures parse, compile, and GPU execution checks. |
aorlta/examples/hip-kernels/hip_kernels.py |
Implements the plugin, harness, rewards, and self-check. |
aorlta/docs/TUTORIAL.md |
Explains registration, validation, scoring, training, and adaptation. |
aorlta/configs/hipk-train.yaml |
Configures a three-iteration training example. |
aorlta/configs/hipk-score.yaml |
Configures two-pass base-model scoring. |
Review effort: Balanced
…al groups that stops a run The rollout stops an iteration when more than 5% of its groups have all K answers identical (sampler suspected). With few prompts and a precise task that is one group: the hip-kernels example's 3-iteration run on MI350X stopped in iteration 1 with 1 of 8 groups identical, on engines whose sampler had just passed the start-up self-test. The share is now a config key, default unchanged.
There was a problem hiding this comment.
🟡 Changes recommended
Candidate-controlled host code can forge successful validation results, and inconsistent parsing can reward rejected or truncated answers, leaving critical scoring-integrity issues unresolved.
7 open issues
Reward parser accepts code rejected by the rollout parser · New Candidate-controlled stdout can spoof a successful result Native leg cache omits toolchain and plugin identity · New Native check logs are discarded from retained evidence · New Worker restarts can assign the same GPU to multiple workers GPU setup failures are incorrectly scored as candidate crashes Documented submission path still requires aorta sanitizer modules
Review effort: Balanced
…ed package's entry point rollout.compile_adapter: none is a built-in for a task whose answers are not compiled (a question and answer, a verdict): every parsed answer passes, and the reward's checks decide. A plugins: entry may name an entry point in the aorlta.plugins group, so a task can live in a private package of its own and be registered without copying it here; AORLTA_PYTHONPATH puts that package on the run job's import path, on the host and in its containers.
… checklist for picking a vertical tests/test_plugins_out_of_tree.py installs a package of its own (an aorlta.plugins entry point) and registers from it: code rewarded by a compile gate and a reference check, a question with no compile step and no sanitizer, and a reward that is an external tool's verdict. Each loads as a run config, passes the wiring's checks and scores through its own check and reward. ADAPTING.md gains the checklist for picking a vertical (ending with the two-line summary), the three shapes, and the route for a task kept in a private package.
…d VALIDATION.md examples/qa/ is the second worked example: questions with reference answers, nothing compiled (rollout.compile_adapter: none). Its parse gives a reply with no Answer: line a repair turn, its check compares the normalised answer with the reference and its aliases and may ask a judge command, and qa.v1 scores it. It comes with 24 GPU-programming questions and generated 3- and 4-digit multiplications, each split train/held-out, and configs for one GPU. tools/eval_compare.py compares two score runs on the same held-out prompts (the base model, and the trained adapter via model.adapter_dir): pass rate, mean reward, per-prompt change, and decoded sample rollouts. It refuses runs that sampled differently. VALIDATION.md says what is verified (the CPU suite, mutation checks, the checks every run makes on itself, the GPU runs) with reproduce commands, the learning result, and what is not yet shown. On MI350X, 40 iterations on the 4-digit set took the held-out pass rate of Qwen3.5-4B from 83.2% to 98.4%. Merges feat/aorlta-plugins: tasks with no compile step, entry-point plugins.
There was a problem hiding this comment.
🟡 Changes recommended
Candidate HIP host code can modify or impersonate the trusted checker and obtain a clean reward without comparison, leaving a critical verification bypass unresolved.
11 open issues
Isolate candidate execution from trusted checker and storage · New Reward parser accepts code rejected by the rollout parser Candidate-controlled stdout can spoof a successful result Kill judge process groups on timeout and worker retirement · New Reject incompatible runs before comparing score deltas · New Handle episodes with no turns in Markdown reports · New Native check logs are discarded from retained evidence Native leg cache omits toolchain and plugin identity Worker restarts can assign the same GPU to multiple workers GPU setup failures are incorrectly scored as candidate crashes Documented submission path still requires aorta sanitizer modules
Review effort: Balanced
…what it receives and returns Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ard code or plugin files differ Names can agree while what they name changed: a Q&A template, a reward function's code, a plugin's check. The paired samples' first-prompt token hash and seed, the reward version and code hash of every scored sample, and the plugin files the run plans hashed must all agree, from what the runs already record.
…PATH finds it The judge's identity hashed only its arguments that were files under the working directory, so a judge given as a bare name (resolved through PATH) could be replaced, or another installation selected, without a new checker version. The executable PATH selects is now resolved and its path and content are part of the version (raised by review on the vertical starters, whose ISA tool had the same code).
There was a problem hiding this comment.
🟡 Changes recommended
The comparison checks silently accept missing plans or episode records, allowing runs with different templates, seeds, or plugin code to appear comparable without the evidence needed to establish compatibility.
1 open issue
1 resolved since last review
Review effort: Balanced
|
@amd-vivekag your seven points are addressed in cec0d31 and d673cbb, with #611's fixes merged in and a reply in each thread. The HIP check now loads the answer as a device-only code object, so none of its host code runs; checked on an MI350X node. adde2b4 and b17ccab cover two Copilot follow-ups. Could you take another look? |
amd-vivekag
left a comment
There was a problem hiding this comment.
Review summary
Verdict: fix-before-ship
The earlier findings are fixed on this head. The checker compiles the candidate to a code object and reads only the harness record, the judge is part of the cache key, live workers lease GPUs, a setup failure is unsettled, the checker version names the compiler and this file, and eval_compare refuses a different batch size or reward and does not crash when a chain has no turns.
Findings: 0 High · 1 Medium · 0 Low
Still open from last review: none.
The new fingerprint check passes when iterations/ or run_plan.json is missing, so two runs can still be reported as comparable without the prompt, seed, reward-code or plugin evidence the tool says it requires.
Copilot's note that plan_eval only accepted standin.v11 was already handled by passing the reward function, and the new --plugins path loads a plugin reward. No change asked there.
| eb, et = episodes(base_dir), episodes(trained_dir) | ||
| for pid in sorted(set(eb) & set(et)): | ||
| first = {(e["k"], e["iteration"]): (e["turns"][0]["prompt_ids_sha256"], e["turns"][0]["seed"]) | ||
| for e in eb[pid] if e.get("turns")} | ||
| for e in et[pid]: | ||
| got = first.get((e["k"], e["iteration"])) | ||
| if e.get("turns") and got is not None and got != (e["turns"][0]["prompt_ids_sha256"], | ||
| e["turns"][0]["seed"]): | ||
| raise Refused(f"{pid} sample {e['k']}: the two runs' first prompts or seeds differ") | ||
| rb, rt = ({(e.get("reward_version"), e.get("reward_code_sha256")) for eps in run.values() for e in eps | ||
| if e.get("status") == "scored"} for run in (eb, et)) | ||
| if rb and rt and rb != rt: | ||
| raise Refused(f"the two runs were scored by different reward code: {sorted(rb)} against {sorted(rt)}") | ||
| pb, pt = (_plugin_hashes(d) for d in (base_dir, trained_dir)) | ||
| if pb != pt: | ||
| raise Refused(f"the two runs loaded different plugin files: {pb} against {pt}") |
There was a problem hiding this comment.
[Medium] Missing episode logs or a missing run plan count as nothing differing.
episodes() is empty when iterations/ has no logs, and _plugin_hashes is empty when run_plan.json is absent, so both checks pass. The docstring says those runs are refused. Copilot flagged this too.
Fix: refuse when either run has no run_plan.json or no iterations/iter-*/episodes.jsonl, and require an episode for every prompt in the two reports. An empty turns list stays legal.
There was a problem hiding this comment.
Fixed in 3748c97. Your check as written still passed two runs I tried: one missing 3 of a prompt's 4 samples (every prompt still has an episode), and one whose only records were in an uncommitted iter-<n>.partial/, which the iter-* glob also matches. So it now reads only committed iter-<n>/ directories and needs a run_plan.json, exactly one record per sample the report counts, and the same (iteration, k) pairs in both runs; an empty turns stays legal.
* docs(aorlta): a getting-started guide, and the README facts a new user needs first docs/GETTING-STARTED.md puts the steps in order: what a run needs (and what is untested: fewer GPUs, other GPUs, other models), the CPU tests as CI runs them, the site file and the job's environment, a score run before a training run, measured times, reading the outputs, the settings people change, and what each of the job's exit codes and the common run stops mean. The README now says the test suite must not run with -n auto on a many-core machine, that only the Qwen3.5 family works as it stands, that the stand-in dataset is not published and the example pool is too small to run, and that the fake components do not read the configured pool or checks. * docs(aorlta): point the guide's checkpoint note at the README's run outputs * docs(aorlta): GPU counts and memory, models and checkpoints, getting help, and a pointer from the aorta README GETTING-STARTED.md: how to place engines, trainer ranks and checks on 1, 2, 4 or 8 GPUs of one node, measured peak memory per GPU (labelled estimates beside them), how to share a node, how to switch models and what limits that today, and where checkpoints and adapters go. The README gains a Getting help section (open an issue with the new AORLTA template, label aorlta; docs feedback as docs: issues). The top-level README points to aorlta/ as early and evolving. * docs(aorlta): diagrams of one iteration, of a score run and export, and of the GPU layout * docs(aorlta): wider boxes in the README diagrams, so they read without scrolling * docs(aorlta): getting started, where the guide disagreed with the code or with main - Every GPU run needs the aorta checkout (AORLTA_AORTA_ROOT): the import pre-flight loads two of its backends whatever the profile, and exits 99 without it. Its own row, and out of the stand-in-only environment line. - The score run and the training run are two submissions; give the training command. - min_scored_fraction is over the candidates handed to the reward service, not every chain generated (also in the README's correctness table). - A scored_fraction_below_threshold stop writes no summary for the failing iteration: read stop.detail.policy and stop.detail.unsettled in run_result.json. - Name the pull requests that add the worked examples (#613), plugins (#611) and the adapter exporter (#612) where a passage depends on them; drop hipk.run.gpus as if it were a config key; a checkpoint is not a PEFT adapter. * docs(aorlta): the stand-in parser ends a reply with no code block; the export and compare tools come with #612 and #613 - The README's iteration diagram drew every parse rejection with a turn left as a repair turn. A repair happens only when the parser sets a rejection reason (the worked examples' parsers do); standin.parse sets none, so its reply with no code block ends the chain (rollout/chain.py, no_code_block). - The score-run paragraph names tools/export_adapter.py (#612) and tools/eval_compare.py (#613) as additions. - The aorta checkout row says #611 makes the import pre-flight load aorta's backends only for profiles that use them.
…ecords that do not cover its report A missing run_plan.json or missing iterations/ used to make the plugin and fingerprint checks compare empty collections and pass. Now both runs need a run plan and committed episode records (iter-<n>/, never iter-<n>.partial/) with exactly one record per sample their report counts, and the two runs' (iteration, k) pairs must match. A record with no turns stays legal.
The pattern's \s could cross a newline, so an empty 'Answer:' took the next line's text as its answer, and a final empty marker was passed over for an earlier one. Each line is now matched alone and the last marker decides. The checker's version moves to 2, since cached verdicts were read with the old rule.
Qwen3.5-4B trained with GRPO on 54 HIP kernel-writing tasks: 47.4% to 76.6% on 39 held-out tasks (108 samples wrong to right against 17). The reward checks correctness only, so it learned correct kernels, not fast ones. The task set is described, not committed; Not yet shown is updated to match.
…s a missing reference The parse took the kernel name from the prompt id, so a task whose id is not its kernel's name (a shape variant) had its correct answers rejected as no_kernel. It now reads the name from the signature in the prompt's user message, and refuses at start a prompt that has none. The self-check skipped a task with no reference kernel and could exit 0 having checked nothing; a missing reference now fails, and with --sources a run that checked no task fails.
|
@amd-vivekag your points are addressed on 6db2f30 (now based on |



Stacked on #611. It'll retarget to
mainonce that merges.Two worked examples of a task built as a plugin, for teammates to copy, plus a tutorial, a before/after evaluation tool and
VALIDATION.md.Example 1, HIP kernels. In
examples/hip-kernels/, the model writes a HIP kernel from its description. There are eight tasks: vector add, SAXPY, ReLU, scale and shift, clamp, reverse, a per-block sum and a transpose.tasks.json: the signature, what to compute, the launch config, the inputs, a host reference loop and a tolerance.ref.native) builds the answer with a generated host program, runs it on one of GPUs 4-7, and compares every output element with the host reference. Outputs start as a sentinel and every array has a guard region, so an element the kernel never writes and a write past the end both count as wrong.hipk.v1scores: -1.0 no code, -0.9 compile error, -0.8 wrong signature, -0.6 crash or hang, -0.5 wrong output, +1.0 correct, minus a small length term.hip_kernels.py selfcheckbuilds and runs them, which checks the tasks themselves. No kernel in here has a planted bug.docs/TUTORIAL.mdcovers what to write, where to register it, and how to check, test, score and train it.configs/hipk-score.yamlandconfigs/hipk-train.yamlrun it. The example needs only the run image's hipcc and GPUs, none of the stand-in sanitizer tools.Tests (
tests/examples/test_hip_kernels.py, 32):GPU (MI350X):
configs/hipk-score.yamlon Qwen3.5-4B: 64 of 64 samples scored, 56 correct, 7 wrong output, 1 compile error, landing PROVEN on 4 engines, about 3.5 minutes of run time.block_sumis 3 of 8 andtransposeis 5 of 8.The first GPU tries found two things in #611, both fixed there: the engine self-test was sampling the task's own near-deterministic prompts, and the all-identical-groups stop tripped on one group in 8.
Example 2, Q&A (
examples/qa/). Questions with reference answers, nothing compiled (compile_adapter: none). A reply with noAnswer:line gets a repair turn. The check compares the normalised answer with the reference and its aliases, and can ask a judge command.qa.v1scores it. Pools: 24 GPU-programming questions, plus generated 3- and 4-digit multiplications, each split train/held-out. The configs run on one GPU (one engine plus one LoRA rank).tools/eval_compare.pycompares a base score run with a trained one on the same held-out prompts. It reports pass rate, mean reward, per-prompt change and decoded sample rollouts, and refuses runs that sampled differently.VALIDATION.mdlists what is verified, with reproduce commands, plus a "not yet shown" section. Linked from the README's Status section.The learning run, Qwen3.5-4B on one MI350X GPU, 40 iterations of 16 prompts × 8 samples, about 20 minutes:
It's one run on a toy task, and VALIDATION.md says so.
Merge order: after #611 (stacked) and #612 (the export tool the README and VALIDATION.md refer to).