[Perf][Qwen3-8B TP=2 CP=2] Final P/P-aligned vs R/R-aligned trace: E2E parity and remaining adapter gaps

[inaniloquentee]

Summary

This issue compares the final aligned baselines for Qwen3-8B Dense with TP=2 and CP=2 on 8x H100. It separates the user-visible no-profiler result from Nsight evidence, because the two answer different questions:

• User-visible result: R/R-aligned is already at end-to-end parity with P/P-aligned: 22.8104 s versus 22.8091 s per step.
• Correctness result: R/R-aligned is strict bitwise (torch.equal=true, 0/128 mismatches, zero mean/max logp diff); P/P-aligned is not.
• Remaining compute gap: R/R-aligned still spends about +0.1208 s in rollout and +0.2921 s in training. Disk weight update and framework wait dominate the full step and hide this gap.
• Trace result: R/R launches 4.55x as many training kernels and 2.11x as many rollout kernels. The useful optimization targets are adapter-level GEMM/data movement and deterministic-collective scheduling, not kernel math micro-tuning.
• Realistic ceiling inside the thin RL-Kernel adapter: recover about 0.31-0.41 s/step, or roughly 1.4-1.8% of the current full step. A 2x full-step target is impossible without changing Vime's disk weight-update and orchestration path, which is explicitly out of scope.

No Vime, Megatron, or vLLM native implementation should be modified for the work proposed here. RL-Kernel must remain a thin, fail-closed operator backend.

Related: #335, #341, #343, #344.

Compared runs

Item	P/P-aligned	R/R-aligned
Run tag	fair-pp-aligned-final-fulltrace-20260826T131445Z	fair-rr-aligned-cublaslt-trace-20260826T1536Z
GPUs	8x H100 80 GB	8x H100 80 GB
Model	Qwen3-8B Dense, BF16	Qwen3-8B Dense, BF16
Training topology	4 GPUs, TP=2, CP=2	4 GPUs, TP=2, CP=2
Rollout topology	4 GPUs, 2 engines x TP=2	4 GPUs, 2 engines x TP=2
Rollout workload	1 prompt, 2 samples, response length 128	Same
Global batch	2	2
Weight update	Full, disk transport	Same
Attention / FFN / logp	Native/native	RL-Kernel/RL-Kernel
Fallback policy	Native baseline	Strict fail-closed, no fallback observed

The no-profiler table below is the performance result. The Nsight run is used only to explain composition and launch behavior. For the profiled timing table, iteration 0 is excluded because it includes warmup/compilation; values are means of iterations 1 and 2.

Kernel times later in this issue are multi-process, multi-rank sums over the whole trace. They are not additive wall-clock time. Wait kernels, especially NCCL and peer-wait kernels, cannot be interpreted as recoverable E2E time without rank/critical-path normalization.

Correctness and user-visible E2E

Metric	P/P-aligned	R/R-aligned	R/R delta	Interpretation
Rollout	0.8965 s	1.0173 s	+0.1208 s (+13.5%)	Small remaining adapter/collective gap
Actor train	0.5611 s	0.8536 s	+0.2925 s (+52.1%)	Main RL-Kernel compute gap
Train	0.7178 s	1.0099 s	+0.2921 s (+40.7%)	Same actor-side gap
Update weights	15.5491 s	14.9063 s	-0.6428 s (-4.1%)	Disk variation; not an RL-Kernel speedup
Full step	22.8091 s	22.8104 s	+0.0013 s (+0.006%)	End-to-end parity
Mean selected-logp diff	0.0162428282	0	-0.0162428282	R/R exact
Max selected-logp diff	0.3142096102	0	-0.3142096102	R/R exact
Mismatch count	99.5 / 128	0 / 128	-99.5	R/R exact
torch.equal	false	true	-	R/R exact

The apparent 0.64 s R/R advantage in disk weight update is run-to-run storage noise. It almost exactly masks the 0.41 s compute disadvantage in this sample and must not be credited to RL-Kernel. Multiple no-profiler repeats are required for future E2E claims.

Profiled steady-state timing

Metric	P/P-aligned	R/R-aligned	R/R delta
Rollout	0.8696 s	1.0451 s	+0.1755 s (+20.2%)
Rollout throughput	73.60 tokens/GPU/s	61.24 tokens/GPU/s	-16.8%
Actor train	0.5766 s	0.9688 s	+0.3922 s (+68.0%)
Actor throughput	520.28 tokens/s	309.91 tokens/s	-40.4%
Train	0.7356 s	1.1260 s	+0.3904 s (+53.1%)
Update weights	14.8634 s	15.0464 s	+0.1830 s (+1.2%)
Full step	22.4360 s	23.2831 s	+0.8471 s (+3.8%)

Nsight amplifies the R/R gap because R/R records substantially more short kernels. This is useful evidence of launch/scheduling overhead, but the profiled +0.85 s step delta must not replace the no-profiler +0.0013 s result.

Kernel launch overview

Scope	P/P-aligned records	R/R-aligned records	Ratio
Training workers	131,408	598,276	4.55x
Rollout workers	624,905	1,316,815	2.11x
Combined relevant workers	756,313	1,915,091	2.53x

This launch multiplication is more important than small differences in FA4, SwiGLU, or selected-logp kernel throughput.

Training-side breakdown

Counts and GPU times are aggregated across the four training ranks and the complete trace.

Family	P/P-aligned	R/R-aligned	Finding
Projection GEMM	cuBLAS nvjet: 5,220 calls / 155.6 ms	det_gemm_sm90: 4,344 / 1,557.8 ms; cuBLAS nvjet: 2,184 / 51.3 ms	Dominant actor-side delta
Direct-copy kernels	5,776 / 100.4 ms	71,296 / 648.9 ms	12.3x call growth; wrapper/layout/materialization cost
Fill kernels	50,834 / 704.7 ms	209,224 / 641.0 ms	4.1x call growth; launch overhead rather than throughput
RL collective wait + publish	0	51,936 / 1,274.5 ms	Coordination/scheduling cost; wait time is not fully additive
RL AG/RS/AR payload	0	12,984 / 97.6 ms	Payload math is not a tuning target
FA4 core	Native training Attention is not directly comparable	5,184 / 33.9 ms	FA4 math is small
Strict linear-logp	Native path	24 / 4.2 ms	Negligible

The multi-rank extra SM90 GEMM time relative to the native cuBLAS family is about 1.40 s; divided across four parallel training ranks it is approximately 0.35 s of critical-path opportunity. That agrees with the observed no-profiler actor gap of 0.2925 s. This makes GEMM routing/dataflow the only P0 training target.

There is also an observability inconsistency that must be fixed before performance work: readback reports rlkernel.det_gemm.cublaslt_nosplitk.v1 for the training FFN contract while Nsight records 4,344 det_gemm_sm90_kernel launches. Backend provenance must distinguish the arithmetic contract from the actual executed kernel/library.

Rollout-side breakdown

Counts and GPU times are aggregated across the rollout workers and the complete trace.

Family	P/P-aligned	R/R-aligned	Finding
cuBLAS nvjet GEMM	153,910 / 3,574.6 ms	158,488 / 3,676.0 ms	Only +2.8%; GEMM math is not the rollout gap
Attention core	30,750 / 418.4 ms	31,882 / 358.9 ms	R/R FA4 is already 14.2% lower in aggregate time
NCCL AllReduce	77,686 / 10,737.8 ms	40,546 / 10,471.7 ms	Includes waiting and framework collectives; not evidence of fallback
RL collective wait + publish	0	160,624 / 1,621.3 ms	Largest R/R-specific rollout overhead
RL AR/AG payload	0	40,156 / 215.4 ms	Payload kernel is small; optimize scheduling, not math
Native fused pointwise	Triton: 195,014 / 458.2 ms	-	Native fused baseline
R/R QK norm	-	generic layer norm: 158,863 / 705.5 ms	Candidate for proven fused-library reuse
R/R SwiGLU	Native Triton included above	39,441 / 159.3 ms	Not large enough for kernel tuning
R/R direct copy	3,343 / 12.1 ms	84,039 / 208.9 ms	Adapter materialization/graph-boundary cost
Query metadata search	Native path	31,882 / 61.7 ms	One search per Attention invocation; low-priority cache target
Selected-logp probability	Native full log-softmax: 675 / 108.9 ms	strict linear-logp: 715 / 58.3 ms	RL-Kernel path is already cheaper; do not tune

R/R replaces roughly half of the vLLM NCCL TP reductions with the self-owned deterministic collective. The remaining NCCL kernels belong to other framework/runtime work and do not imply an RL-Kernel fallback; readbacks report zero fallbacks.

The key collective result is not "optimize the 215 ms payload kernel." The actionable cost is the much larger number of wait/publish launches and the synchronization schedule around the payload. The multi-rank aggregate must be normalized, but it is consistent with the observed 0.12-0.18 s rollout wall-time gap.

Prioritized work

Priority	Work	Expected phase recovery	Expected full-step recovery
P0	Training GEMM route/dataflow	0.23-0.29 s actor train	1.0-1.3%
P1	Rollout collective coordination and adapter graph boundaries	0.08-0.12 s rollout	0.35-0.53%
P2	Backend provenance and automated trace gates	No direct claim	Enables trustworthy regressions
Combined realistic ceiling	Thin adapter only	0.31-0.41 s compute	1.4-1.8%

P0: make training projection GEMM use the fastest proven strict route

This is routing and dataflow work, not SM90 tile-level micro-tuning.

1. Make the readback report both the arithmetic contract and actual executor, for example contract=cublaslt_nosplitk plus executor=det_gemm_sm90 or executor=cublaslt.
2. Through the RL-Kernel integration only, evaluate the deterministic cuBLASLt no-split-K route already used by the aligned rollout for the observed Megatron projection shapes.
3. Keep the route only if the full TP=2, CP=2 run remains strict bitwise. Strict mode must fail closed if the requested executor is unavailable.
4. If SM90 remains required, remove adapter-level weight transpose, padding, temporary allocation, and output-copy work; cache weight descriptors/workspaces by weight version. Do not start with MMA/kernel-internal tuning.
5. Do not modify Megatron, vLLM, or Vime native files.

Acceptance target: recover at least 0.23 s of the 0.2925 s no-profiler actor gap and bring R/R actor time within 10% of P/P, while keeping exact parity and zero fallback.

P1: reduce rollout orchestration without touching native vLLM

1. Keep the self-owned deterministic collective as the formal path and NCCL only as an explicit fallback outside strict runs.
2. Cache peer plans, rank validation, descriptors, and stable graph metadata once per worker/weight version.
3. Reduce separate wait/publish/validation launches around each TP collective. Prefer capture/replay or a consolidated adapter dispatch; do not micro-tune the payload kernel.
4. Preserve vLLM CUDA-graph boundaries across the RL-Kernel custom op. Avoid adapter-created materializations that explain the 84,039 direct-copy launches.
5. Evaluate a proven fused QK normalization implementation through the RL-Kernel adapter. It must use one arithmetic contract on training and rollout and must be rejected if bitwise parity changes.
6. Cache layer-invariant query/KV metadata so searchsorted is not launched once per Attention invocation. This is P2 within this work item because its aggregate time is only 61.7 ms.

Acceptance target: recover 0.08-0.12 s of rollout wall time, keep R/R Attention/FFN/logp and collective readbacks fallback-free, and do not modify vLLM native logic.

P2: make trace provenance mechanically trustworthy

1. Report contract_backend, executor_backend, and fallback separately for GEMM, Attention, FFN, logp, and collectives.
2. Add low-volume NVTX ranges around adapter preparation, executor call, collective wait/payload/publish, and output materialization.
3. Print one route line per worker/operator after first real execution; keep full detail in readback JSON.
4. Add automated trace assertions for the requested executor and zero fallback. A label/readback must never claim cuBLASLt when the trace executed SM90 custom GEMM, or vice versa.

Explicit non-targets

Non-target	Reason
FA4 math micro-tuning	R/R rollout FA4 is already lower than P/P in aggregate GPU time; training FA4 is only 33.9 ms across ranks
linear_logp probability kernel	58.3 ms aggregated in rollout and 4.2 ms in training; already cheaper than native full log-softmax
SwiGLU kernel internals	159.3 ms aggregated; launch/dataflow dominates
AG/RS/AR payload kernel internals	Payload is 97.6 ms training and 215.4 ms rollout across ranks; coordination is much larger
NCCL removal	Formal runs use self-owned collectives and strict mode disables fallback; unrelated framework NCCL remains valid
Vime disk weight update	About 15 s/step and outside the RL-Kernel thin-backend boundary
Vime/Megatron/vLLM native refactors	Integration must remain a thin RL-Kernel adapter

Performance claim and measurement gate

Future PRs should report two separate results:

1. No-profiler performance: at least five steady iterations after warmup, median and spread, with identical workload and disk-update mode.
2. Profile composition: kernel/range counts and multi-rank sums, explicitly labeled as non-wall-clock metrics.

Required correctness/routing gates:

TP=2, CP=2
selected-logp mismatch = 0 / 128
mean_abs_diff = 0
max_abs_diff = 0
torch.equal = true
Attention fallback = false
FFN fallback = false
linear_logp fallback = false
collective fallback = false
executor backend in readback == executor observed in trace

Performance definition of done:

• R/R actor train is within 10% of P/P-aligned without changing native framework code.
• R/R rollout is within 5% of P/P-aligned.
• Training kernel-record ratio is materially reduced from 4.55x; rollout ratio is materially reduced from 2.11x.
• Full-step performance has no statistically significant regression.
• No claim of 2x full-step speedup is made from RL-Kernel-only work. With the current disk-update path, the measured thin-adapter ceiling is approximately 1.4-1.8%.