helion-rag: four-arm head-to-head autotuning experiment (LFBO / LLM / hybrid / RAG-LLM)

[IshanAryendu]

What this is

We ran a head-to-head campaign comparing four Helion autotuning strategies over 33 GPU kernel shapes × 5 paired repetitions (660 runs) on a single H100, to answer one question: is retrieval-augmented LLM autotuning worth pursuing?

The answer is promising but modest, and the numbers are interesting enough that I'd like others to try to replicate or break them. This PR is the harness, the analysis, and the reference results — opened as a draft because it's an experiment to evaluate, not a feature to land.

The four arms

Arm	HELION_AUTOTUNER	Retrieval	LLM requests
lfbo	LFBOTreeSearch	off	0
llm	LLMGuidedSearch	off	1
hybrid_lfbo_llm	LLMSeededLFBOTreeSearch	off	1
contextual_rag_llm	LLMGuidedSearch	on	1

llm and contextual_rag_llm share identical provider, model, and round settings, so retrieval is the only difference between them. Everything else in the design exists to keep that one contrast clean.

Results

Arm	latency GM (ms)	readiness GM (s)	regret vs LFBO	correct	tokens
lfbo	0.0909	108.9	0%	164/165	0
llm	0.0825	62.8	−9.3%	164/165	746k
hybrid_lfbo_llm	0.0743	175.1	−18.3%	164/165	747k
contextual_rag_llm	0.0808	71.8	−11.1%	164/165	875k

There's no single winner — there's a frontier. Hybrid buys the best kernels with 2.8× LFBO's search time. LLM is by far the fastest to be ready and still beats LFBO on quality.

On the contrast the experiment was built for: retrieval beats context-free LLM search by 1.9% on selected latency (ratio 0.981, 95% CI 0.964–0.994, Holm-adjusted p=0.034), at 14.3% more readiness time and 17.3% more provider tokens.

That's a real effect but a small one. Whether 1.9% justifies operating a retrieval corpus is exactly the question I'd like the team's read on.

Design notes

• Paired and balanced. Same seed across all four arms per (workload, repetition) block; execution order is a balanced Latin square so machine drift can't favour an arm.
• Frozen controls. study_manifest.json hashes every control; a resume refuses to mix results from a different hash.
• Cold every time. Per-run Helion/Inductor/Triton/temp dirs; exact-cache reads, best-available reads, and cache writes all off.
• One shared budget for the hybrid. Its LLM and LFBO stages charge a single 80-attempt ledger and one continuous clock, so stage two can't quietly claim a second allowance. llm/contextual_rag_llm typically stop early at ~26 attempts; that realized effort is reported, not padded.
• Statistics. Per-kernel median matched-repetition ratios, geometric-mean aggregation, 200k-resample kernel bootstrap CIs, Wilcoxon on log ratios, all 12 contrasts Holm-corrected as one family.

Limitations

One host, one GPU (H100), one model (claude-opus-4-8 via Vertex), 5 seeds. 33 workloads is enough for the paired statistics used here, but family-level effects are noisy. The oracle is bounded by what the campaign observed, not exhaustive search, so absolute regret is a lower bound. Retrieval quality is capped by the corpus generation the index was built from.

Replicating

Start at docs/rag_autotuning_experiment.md. Reference numbers to diff against are checked into scripts/helion_rag/results/2026-07-26-h100/, including all 12 Holm-corrected contrasts, so a replication can be compared contrast by contrast rather than headline to headline. The claims worth checking are the relative ones — especially whether contextual_rag_llm still beats llm at all.

Scope

Stacked on #3045 (helion-rag: standalone retrieval over CI autotuning artifacts); this PR is only the experiment on top of it.

Deliberately not included: an earlier tiered-RAG confirmatory-study framework (~17k lines) that was built but never ran — it's blocked on pre-registered data stop-conditions and would only be noise here. The HELION_RAG_FROZEN_MANIFEST verification path in the adapter existed solely to serve that study and is removed too; it was never activated by any shipped driver. Both are recoverable from branch history if we come back to them.

Test plan

• PYTHONPATH=scripts/helion_rag pytest scripts/helion_rag/tests -q — 167 passed
• pytest test/test_rag_*.py test/test_candidate_attempt_budget.py test/test_llm_*.py -q — 235 passed, 65 subtests
• ruff check / ruff format --check clean; pyrefly check at 124 errors, unchanged from the base branch
• Full re-analysis of the 660-run campaign reproduces every published number
• Live 4-arm single-workload pilot on an H100: all four arms completed and correct, RAG arm reaching lookup_tier: 1 / decision: ContextualSearch with no fallback