Clarification on reported 1.56x speedup aggregation method

[XAheli]

The blog post KernelAgent: Hardware-Guided GPU Kernel Optimization states:

We evaluate the kernels generated by KernelAgent on all 100 L1 KernelBench tasks. Overall, it achieved 2.02x speedup over generated kernels from earlier versions. On average, KernelAgent generated 1.56x speedup when compared to default torch.compile, outperforming 65 of 100 KernelBench L1 tasks and achieving 89% of the hardware roofline efficiency on the H100.

Could you please clarify the aggregation method used for the "1.56x speedup" figure?

1. Is this geometric mean or arithmetic mean?
2. What timing method was used? The benchmark.py in the artifacts repo uses triton.testing.do_bench with warmup=500, rep=100, return_mode="mean". Was this the same configuration used for the reported numbers?
3. What dtype was used? The artifacts benchmark.py casts models and inputs to bfloat16 before benchmarking. Was the 1.56x measured in BF16 or FP32?
4. Are the per-kernel speedup numbers for all 100 L1 tasks available? The artifacts repo only contains 5 curated kernels.

This information would help the community make accurate comparisons when benchmarking other kernel optimization tools.

Thank you for open-sourcing this work :)

cc @kaiming-cheng @Laurawly @vishalgoyal316 @RohitRathore1