Skip to content

feat(aorlta): two worked examples (HIP kernels, Q&A), before/after eval, VALIDATION.md - #613

Open
vivekkhandelwal1 wants to merge 26 commits into
mainfrom
feat/aorlta-example-hip-kernels
Open

vivekkhandelwal1 wants to merge 26 commits into
mainfrom
feat/aorlta-example-hip-kernels

Conversation

@vivekkhandelwal1

@vivekkhandelwal1 vivekkhandelwal1 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #611. It'll retarget to main once that merges.

Two worked examples of a task built as a plugin, for teammates to copy, plus a tutorial, a before/after evaluation tool and VALIDATION.md.

Example 1, HIP kernels. In examples/hip-kernels/, the model writes a HIP kernel from its description. There are eight tasks: vector add, SAXPY, ReLU, scale and shift, clamp, reverse, a per-block sum and a transpose.

  • Prompts come from tasks.json: the signature, what to compute, the launch config, the inputs, a host reference loop and a tolerance.
  • The parse takes the last cpp block. A reply with no block, or one that doesn't define the kernel asked for, gets a repair turn that says so. Compile errors get the usual compiler-feedback turn.
  • The check (ref.native) builds the answer with a generated host program, runs it on one of GPUs 4-7, and compares every output element with the host reference. Outputs start as a sentinel and every array has a guard region, so an element the kernel never writes and a write past the end both count as wrong.
  • hipk.v1 scores: -1.0 no code, -0.9 compile error, -0.8 wrong signature, -0.6 crash or hang, -0.5 wrong output, +1.0 correct, minus a small length term.
  • Each task ships a correct reference kernel. hip_kernels.py selfcheck builds and runs them, which checks the tasks themselves. No kernel in here has a planted bug.

docs/TUTORIAL.md covers what to write, where to register it, and how to check, test, score and train it. configs/hipk-score.yaml and configs/hipk-train.yaml run it. The example needs only the run image's hipcc and GPUs, none of the stand-in sanitizer tools.

Tests (tests/examples/test_hip_kernels.py, 32):

  • the configs load with the plugin;
  • prompts and probes build;
  • parse and feedback behave as described;
  • every harness outcome maps to one leg state, run against a stand-in hipcc on CPU;
  • the reward keeps the service's invariants, and decide agrees with compute, for every combination of leg outcomes;
  • every harness compiles with its reference kernel where hipcc is installed;
  • one heavy test trains an iteration of the example through the real wiring on CPU.

GPU (MI350X):

  • The self-check passes on all 8 tasks on GPUs 0, 4 and 7.
  • configs/hipk-score.yaml on Qwen3.5-4B: 64 of 64 samples scored, 56 correct, 7 wrong output, 1 compile error, landing PROVEN on 4 engines, about 3.5 minutes of run time.
  • The check separates good answers from bad ones. Every elementwise task is 8 of 8 correct, but block_sum is 3 of 8 and transpose is 5 of 8.

The first GPU tries found two things in #611, both fixed there: the engine self-test was sampling the task's own near-deterministic prompts, and the all-identical-groups stop tripped on one group in 8.

Example 2, Q&A (examples/qa/). Questions with reference answers, nothing compiled (compile_adapter: none). A reply with no Answer: line gets a repair turn. The check compares the normalised answer with the reference and its aliases, and can ask a judge command. qa.v1 scores it. Pools: 24 GPU-programming questions, plus generated 3- and 4-digit multiplications, each split train/held-out. The configs run on one GPU (one engine plus one LoRA rank).

tools/eval_compare.py compares a base score run with a trained one on the same held-out prompts. It reports pass rate, mean reward, per-prompt change and decoded sample rollouts, and refuses runs that sampled differently.

VALIDATION.md lists what is verified, with reproduce commands, plus a "not yet shown" section. Linked from the README's Status section.

The learning run, Qwen3.5-4B on one MI350X GPU, 40 iterations of 16 prompts × 8 samples, about 20 minutes:

  • Training prompts: 74.8% correct in iterations 1-5, 98.4% in iterations 31-40.
  • Held-out 4-digit multiplications: pass rate 83.2% (base) to 98.4% (trained), mean reward 0.724 to 0.950.
  • On the same prompts and seeds: 42 samples went from wrong to right and 3 from right to wrong.
  • The trained model writes out the partial products where the base guessed.

It's one run on a toy task, and VALIDATION.md says so.

Merge order: after #611 (stacked) and #612 (the export tool the README and VALIDATION.md refer to).

…m a plugin

A run config lists plugin modules under `plugins:`. Each is imported before the
config is checked and registers its components with aorlta.plugins: a task (the
prompt pool's records), a parse adapter, a feedback template, a compile check, a
check profile, check adapters and a reward function. Every place that looked a
name up in a fixed list now also looks in the registry, so a new task needs no
edit to the package. The run plan records each plugin file's sha256.

A plugin's data is handled as the stand-in source's, and its reward keeps the
service's range and parts; both are stated in ADAPTING.md.
Each engine's start-up self-test samples 4 chats with two seeds and needs 3 of the
4 pairs to differ. On the first batch's own prompts that fails a sound sampler
when the task is precise: on MI350X, the hip-kernels example's prompts (write a
vector add, a SAXPY, ...) gave the same 128 tokens for both seeds on 2 of 4
prompts, on every engine. A plugin task now self-tests on fixed open-ended chats
(wiring.SELF_TEST_CHATS), or on its own (Task.selftest_messages). The stand-in
path is unchanged.
examples/hip-kernels/ asks the model to write a HIP kernel from its description
(vector add, SAXPY, ReLU, scale and shift, clamp, reverse, a per-block sum, a
transpose). Its parse gives a repair turn when the reply has no code block or
does not define the kernel asked for. Its check builds the answer with a
generated host program, runs it on a GPU and compares every output element with
a host reference; an element never written, and a write past an array's end,
are caught. hipk.v1 scores the outcome. Each task comes with a correct reference
kernel, which 'hip_kernels.py selfcheck' builds and runs.

docs/TUTORIAL.md walks through it: what to write, where to register it, and how
to check, test, score and train it. configs/hipk-score.yaml and hipk-train.yaml
run it. It needs none of the stand-in sanitizer tools, only the run image's
hipcc and GPUs.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 17:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The correctness verdict trusts candidate-controlled stdout, allowing incorrect kernels to forge success records and receive clean results.

4 open issues
What changed in this PR

Builds on the plugin support from #611 with a copyable HIP kernel-writing task and an end-to-end tutorial, without adding task-specific code to the package.

Changes:

  • Adds eight kernel tasks, reference implementations, GPU correctness checks, and a graded reward function.
  • Provides scoring and training configs, plus setup and adaptation guidance.
  • Adds CPU-based behavior tests, reward-invariant checks, optional harness compilation, and a wired training test.
File Description
aorlta/​tests/​examples/​test_hip_kernels.py Tests plugin wiring, parsing, checks, rewards, and training.
aorlta/​README.md Introduces the example and links the tutorial.
aorlta/​examples/​hip-kernels/​tasks.json Defines eight tasks, inputs, references, and tolerances.
aorlta/​examples/​hip-kernels/​reference/​vector_add.hip Adds the vector-add reference kernel.
aorlta/​examples/​hip-kernels/​reference/​transpose.hip Adds the matrix-transpose reference kernel.
aorlta/​examples/​hip-kernels/​reference/​scale_shift.hip Adds the scale-and-shift reference kernel.
aorlta/​examples/​hip-kernels/​reference/​saxpy.hip Adds the SAXPY reference kernel.
aorlta/​examples/​hip-kernels/​reference/​reverse.hip Adds the array-reversal reference kernel.
aorlta/​examples/​hip-kernels/​reference/​relu.hip Adds the ReLU reference kernel.
aorlta/​examples/​hip-kernels/​reference/​clamp.hip Adds the clamp reference kernel.
aorlta/​examples/​hip-kernels/​reference/​block_sum.hip Adds the per-block reduction reference kernel.
aorlta/​examples/​hip-kernels/​README.md Summarizes the example’s files and entry points.
aorlta/​examples/​hip-kernels/​hipk-gpu.yaml Configures parse, compile, and GPU execution checks.
aorlta/​examples/​hip-kernels/​hip_kernels.py Implements the plugin, harness, rewards, and self-check.
aorlta/​docs/​TUTORIAL.md Explains registration, validation, scoring, training, and adaptation.
aorlta/​configs/​hipk-train.yaml Configures a three-iteration training example.
aorlta/​configs/​hipk-score.yaml Configures two-pass base-model scoring.

Review effort: Balanced

Comment thread aorlta/docs/TUTORIAL.md Resolved
…al groups that stops a run

The rollout stops an iteration when more than 5% of its groups have all K answers
identical (sampler suspected). With few prompts and a precise task that is one
group: the hip-kernels example's 3-iteration run on MI350X stopped in iteration 1
with 1 of 8 groups identical, on engines whose sampler had just passed the
start-up self-test. The share is now a config key, default unchanged.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 18:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Candidate-controlled host code can forge successful validation results, and inconsistent parsing can reward rejected or truncated answers, leaving critical scoring-integrity issues unresolved.

7 open issues

Review effort: Balanced

…ed package's entry point

rollout.compile_adapter: none is a built-in for a task whose answers are not
compiled (a question and answer, a verdict): every parsed answer passes, and the
reward's checks decide. A plugins: entry may name an entry point in the
aorlta.plugins group, so a task can live in a private package of its own and be
registered without copying it here; AORLTA_PYTHONPATH puts that package on the
run job's import path, on the host and in its containers.
… checklist for picking a vertical

tests/test_plugins_out_of_tree.py installs a package of its own (an aorlta.plugins
entry point) and registers from it: code rewarded by a compile gate and a
reference check, a question with no compile step and no sanitizer, and a reward
that is an external tool's verdict. Each loads as a run config, passes the
wiring's checks and scores through its own check and reward.

ADAPTING.md gains the checklist for picking a vertical (ending with the two-line
summary), the three shapes, and the route for a task kept in a private package.
…d VALIDATION.md

examples/qa/ is the second worked example: questions with reference answers,
nothing compiled (rollout.compile_adapter: none). Its parse gives a reply with no
Answer: line a repair turn, its check compares the normalised answer with the
reference and its aliases and may ask a judge command, and qa.v1 scores it. It
comes with 24 GPU-programming questions and generated 3- and 4-digit
multiplications, each split train/held-out, and configs for one GPU.

tools/eval_compare.py compares two score runs on the same held-out prompts (the
base model, and the trained adapter via model.adapter_dir): pass rate, mean
reward, per-prompt change, and decoded sample rollouts. It refuses runs that
sampled differently.

VALIDATION.md says what is verified (the CPU suite, mutation checks, the checks
every run makes on itself, the GPU runs) with reproduce commands, the learning
result, and what is not yet shown. On MI350X, 40 iterations on the 4-digit set
took the held-out pass rate of Qwen3.5-4B from 83.2% to 98.4%.

Merges feat/aorlta-plugins: tasks with no compile step, entry-point plugins.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 00:20
@vivekkhandelwal1 vivekkhandelwal1 changed the title feat(aorlta): worked example of your own task (HIP kernels) and a tutorial feat(aorlta): two worked examples (HIP kernels, Q&A), before/after eval, VALIDATION.md Nov 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread aorlta/examples/qa/qa_task.py Resolved
Comment thread aorlta/tools/eval_compare.py Resolved
Comment thread aorlta/tools/eval_compare.py Resolved
@vivekkhandelwal1
vivekkhandelwal1 marked this pull request as ready for review November 13, 2026 06:49
Copilot AI balanced review requested due to automatic review settings November 13, 2026 07:51
Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The comparison accepts runs with different reward implementations, checker settings, or prompt hashes, allowing evaluation changes to be misreported as model improvements.

1 open issue

Review effort: Balanced

…ard code or plugin files differ

Names can agree while what they name changed: a Q&A template, a reward function's code, a plugin's check. The
paired samples' first-prompt token hash and seed, the reward version and code hash of every scored sample, and the
plugin files the run plans hashed must all agree, from what the runs already record.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 10:16
…PATH finds it

The judge's identity hashed only its arguments that were files under the working directory, so a judge given as a
bare name (resolved through PATH) could be replaced, or another installation selected, without a new checker
version. The executable PATH selects is now resolved and its path and content are part of the version (raised by
review on the vertical starters, whose ISA tool had the same code).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The comparison checks silently accept missing plans or episode records, allowing runs with different templates, seeds, or plugin code to appear comparable without the evidence needed to establish compatibility.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread aorlta/tools/eval_compare.py Resolved
Copilot AI balanced review requested due to automatic review settings November 13, 2026 10:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Fingerprint validation fails open when episode records are missing, allowing unsupported comparisons and deltas without verifying prompt-token, seed, or reward-code evidence.

1 open issue

Review effort: Balanced

@vivekkhandelwal1

Copy link
Copy Markdown
Collaborator Author

@amd-vivekag your seven points are addressed in cec0d31 and d673cbb, with #611's fixes merged in and a reply in each thread. The HIP check now loads the answer as a device-only code object, so none of its host code runs; checked on an MI350X node. adde2b4 and b17ccab cover two Copilot follow-ups. Could you take another look?

@amd-vivekag amd-vivekag left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Verdict: fix-before-ship

The earlier findings are fixed on this head. The checker compiles the candidate to a code object and reads only the harness record, the judge is part of the cache key, live workers lease GPUs, a setup failure is unsettled, the checker version names the compiler and this file, and eval_compare refuses a different batch size or reward and does not crash when a chain has no turns.

Findings: 0 High · 1 Medium · 0 Low

Still open from last review: none.

The new fingerprint check passes when iterations/ or run_plan.json is missing, so two runs can still be reported as comparable without the prompt, seed, reward-code or plugin evidence the tool says it requires.

Copilot's note that plan_eval only accepted standin.v11 was already handled by passing the reward function, and the new --plugins path loads a plugin reward. No change asked there.

Comment thread aorlta/tools/eval_compare.py Outdated
Comment on lines +55 to +70
eb, et = episodes(base_dir), episodes(trained_dir)
for pid in sorted(set(eb) & set(et)):
first = {(e["k"], e["iteration"]): (e["turns"][0]["prompt_ids_sha256"], e["turns"][0]["seed"])
for e in eb[pid] if e.get("turns")}
for e in et[pid]:
got = first.get((e["k"], e["iteration"]))
if e.get("turns") and got is not None and got != (e["turns"][0]["prompt_ids_sha256"],
e["turns"][0]["seed"]):
raise Refused(f"{pid} sample {e['k']}: the two runs' first prompts or seeds differ")
rb, rt = ({(e.get("reward_version"), e.get("reward_code_sha256")) for eps in run.values() for e in eps
if e.get("status") == "scored"} for run in (eb, et))
if rb and rt and rb != rt:
raise Refused(f"the two runs were scored by different reward code: {sorted(rb)} against {sorted(rt)}")
pb, pt = (_plugin_hashes(d) for d in (base_dir, trained_dir))
if pb != pt:
raise Refused(f"the two runs loaded different plugin files: {pb} against {pt}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Missing episode logs or a missing run plan count as nothing differing.

episodes() is empty when iterations/ has no logs, and _plugin_hashes is empty when run_plan.json is absent, so both checks pass. The docstring says those runs are refused. Copilot flagged this too.

Fix: refuse when either run has no run_plan.json or no iterations/iter-*/episodes.jsonl, and require an episode for every prompt in the two reports. An empty turns list stays legal.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3748c97. Your check as written still passed two runs I tried: one missing 3 of a prompt's 4 samples (every prompt still has an episode), and one whose only records were in an uncommitted iter-<n>.partial/, which the iter-* glob also matches. So it now reads only committed iter-<n>/ directories and needs a run_plan.json, exactly one record per sample the report counts, and the same (iteration, k) pairs in both runs; an empty turns stays legal.

vivekkhandelwal1 added a commit that referenced this pull request Nov 13, 2026
* docs(aorlta): a getting-started guide, and the README facts a new user needs first

docs/GETTING-STARTED.md puts the steps in order: what a run needs (and what is
untested: fewer GPUs, other GPUs, other models), the CPU tests as CI runs them,
the site file and the job's environment, a score run before a training run,
measured times, reading the outputs, the settings people change, and what each
of the job's exit codes and the common run stops mean.

The README now says the test suite must not run with -n auto on a many-core
machine, that only the Qwen3.5 family works as it stands, that the stand-in
dataset is not published and the example pool is too small to run, and that
the fake components do not read the configured pool or checks.

* docs(aorlta): point the guide's checkpoint note at the README's run outputs

* docs(aorlta): GPU counts and memory, models and checkpoints, getting help, and a pointer from the aorta README

GETTING-STARTED.md: how to place engines, trainer ranks and checks on 1, 2, 4 or
8 GPUs of one node, measured peak memory per GPU (labelled estimates beside
them), how to share a node, how to switch models and what limits that today,
and where checkpoints and adapters go. The README gains a Getting help section
(open an issue with the new AORLTA template, label aorlta; docs feedback as
docs: issues). The top-level README points to aorlta/ as early and evolving.

* docs(aorlta): diagrams of one iteration, of a score run and export, and of the GPU layout

* docs(aorlta): wider boxes in the README diagrams, so they read without scrolling

* docs(aorlta): getting started, where the guide disagreed with the code or with main

- Every GPU run needs the aorta checkout (AORLTA_AORTA_ROOT): the import pre-flight loads two of its backends
  whatever the profile, and exits 99 without it. Its own row, and out of the stand-in-only environment line.
- The score run and the training run are two submissions; give the training command.
- min_scored_fraction is over the candidates handed to the reward service, not every chain generated (also in the
  README's correctness table).
- A scored_fraction_below_threshold stop writes no summary for the failing iteration: read stop.detail.policy and
  stop.detail.unsettled in run_result.json.
- Name the pull requests that add the worked examples (#613), plugins (#611) and the adapter exporter (#612) where a
  passage depends on them; drop hipk.run.gpus as if it were a config key; a checkpoint is not a PEFT adapter.

* docs(aorlta): the stand-in parser ends a reply with no code block; the export and compare tools come with #612 and #613

- The README's iteration diagram drew every parse rejection with a turn left as a repair turn. A repair happens only
  when the parser sets a rejection reason (the worked examples' parsers do); standin.parse sets none, so its reply
  with no code block ends the chain (rollout/chain.py, no_code_block).
- The score-run paragraph names tools/export_adapter.py (#612) and tools/eval_compare.py (#613) as additions.
- The aorta checkout row says #611 makes the import pre-flight load aorta's backends only for profiles that use them.
…ecords that do not cover its report

A missing run_plan.json or missing iterations/ used to make the plugin and
fingerprint checks compare empty collections and pass. Now both runs need a
run plan and committed episode records (iter-<n>/, never iter-<n>.partial/)
with exactly one record per sample their report counts, and the two runs'
(iteration, k) pairs must match. A record with no turns stays legal.
The pattern's \s could cross a newline, so an empty 'Answer:' took the next
line's text as its answer, and a final empty marker was passed over for an
earlier one. Each line is now matched alone and the last marker decides. The
checker's version moves to 2, since cached verdicts were read with the old
rule.
Qwen3.5-4B trained with GRPO on 54 HIP kernel-writing tasks: 47.4% to 76.6%
on 39 held-out tasks (108 samples wrong to right against 17). The reward
checks correctness only, so it learned correct kernels, not fast ones. The
task set is described, not committed; Not yet shown is updated to match.
@vivekkhandelwal1
vivekkhandelwal1 changed the base branch from feat/aorlta-plugins to main November 13, 2026 15:06
Copilot AI balanced review requested due to automatic review settings November 13, 2026 15:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Kernel-name resolution relies on task IDs and can reject valid shape variants, while default self-checks silently pass when reference kernels are missing.

2 open issues
1 resolved since last review

Review effort: Balanced

…s a missing reference

The parse took the kernel name from the prompt id, so a task whose id is not
its kernel's name (a shape variant) had its correct answers rejected as
no_kernel. It now reads the name from the signature in the prompt's user
message, and refuses at start a prompt that has none. The self-check skipped a
task with no reference kernel and could exit 0 having checked nothing; a
missing reference now fails, and with --sources a run that checked no task
fails.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 15:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the harness can accept kernels that produce correct outputs while modifying read-only input payloads without detection.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@vivekkhandelwal1

Copy link
Copy Markdown
Collaborator Author

@amd-vivekag your points are addressed on 6db2f30 (now based on main, all checks green): the eval_compare evidence check is in 3748c97, with a reply in your thread. Also in this round: the Q&A example reads each Answer: line on its own, hipk.parse takes the kernel name from the signature rather than the task id, the self-check fails a missing reference, and VALIDATION.md has a new section 6 on learning a real kernel task. Could you take another look?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants