[Docs/Repro][8xH100][Qwen3-8B TP=2 CP=2] Strict R/R trace and operator ablation runbook

[inaniloquentee]

Goal

Provide one reproducible runbook for the Qwen3-8B strict train/rollout validation used by #335 and the performance trace analyzed in #341. The runbook must cover:

• one 8xH100, TP=2, CP=2 strict R/R post-training step;
• a full-stack Nsight Systems trace started before Ray;
• proof that Attention/FFN/logprob execute RL-Kernel CUDA operators without fallback;
• proof that RL-Kernel collectives execute the self-owned CUDA IPC AG/RS/AllReduce payload kernels;
• the operator ablation matrix, including single-case, resume, and report workflows.

Source and environment used for the successful trace

Item	Exact value
Host	one node, 8x NVIDIA H100 80GB HBM3
Training layout	GPUs 0-3, Megatron TP=2, CP=2, PP=1
Rollout layout	GPUs 4-7, two vLLM engines, TP=2 per engine
Model/dtype	Qwen3-8B Dense, BF16
RL-Kernel	PR #339 head 5a1ad576c9a4f85b1154c4692bf92b2d491d10a7, plus the local SM90 fail-closed route guard described in #341
RL-Align/vime	b2c9b02b25529bd500fb49cc9f070971685ee548 on main
CUDA runtime	CUDA 12.8 / PyTorch runtime at /home/ellm/workspace/ljj/.conda/envs/rlk-attention-engines
Trace artifacts	/home/ellm/ljj/vime-debug-main/outputs/rlkernel/rr-bitwise-perf-trace-20260825T093418Z

At the time of writing, origin/test is 7207ebda0447d48f2cd85091864aa22c4edcd40b and PR #339 is still open. Do not claim that a plain origin/test checkout reproduces the successful trace until #339 and the SM90 fail-closed guard are merged. After merge, replace the PR pin with the exact resulting test SHA and retain it in every report.

The successful run used these host paths:

export VIME_ROOT=/home/ellm/ljj/vime-debug-main
export RL_KERNEL_ROOT=/home/ellm/ljj/RL-Kernel-issue335-fix
export MEGATRON_ROOT=/home/ellm/ljj/Megatron-LM
export PYTHON_BIN=/home/ellm/workspace/ljj/.conda/envs/rlk-attention-engines/bin/python
export RAY_BIN=/home/ellm/workspace/ljj/.conda/envs/rlk-attention-engines/bin/ray

export MODEL_ROOT=/home/ellm/ljj/checkpoints/Qwen3-8B_vime_rlkernel_tp2_cp2
export TORCH_DIST_ROOT=/home/ellm/ljj/checkpoints/Qwen3-8B_torch_dist
export VIME_CKPT=/home/ellm/ljj/checkpoints/Qwen3-8B_vime_rlkernel_tp2_cp2
export PROMPT_DATA=/home/ellm/ljj/qwen3_tp2_cp2_smoke_valid.jsonl

1. Checkout and build

Use a clean checkout for a new reproduction. Do not reset the shared validation checkout because it may contain the local fail-closed instrumentation.

git -C "${VIME_ROOT}" fetch origin main
git -C "${VIME_ROOT}" checkout --detach b2c9b02b25529bd500fb49cc9f070971685ee548

git -C "${RL_KERNEL_ROOT}" fetch origin pull/339/head
git -C "${RL_KERNEL_ROOT}" checkout --detach 5a1ad576c9a4f85b1154c4692bf92b2d491d10a7

cd "${RL_KERNEL_ROOT}"
KERNEL_ALIGN_FORCE_SM90=1 \
KERNEL_ALIGN_DET_GEMM_SM90=1 \
"${PYTHON_BIN}" -m pip install --no-build-isolation -e .

The extension must be built in the same Python environment used by Ray workers. The runner must fail before Ray starts unless all of the following are true:

• Python imports rl_engine and rl_engine._C from ${RL_KERNEL_ROOT};
• the rl_kernel vLLM general plugin entry point is installed;
• FA4 initializes;
• deterministic GEMM, SwiGLU, and strict linear-logp symbols exist;
• det_gemm_sm90_compiled() is true;
• a real CUDA probe routes to SM90 and cannot fall back to det_gemm_naive.

The exact validation runner used on the host is:

${VIME_ROOT}/scripts/codex-debug-qwen3-8B-rlkernel-tp2-cp2.sh

It is currently an untracked validation-host helper. Before this issue is closed, it must be upstreamed or replaced by an equivalent tracked entry point. The portable owner should be RL-Align/vime for the Vime launch glue, with numerical providers and artifact validation remaining in RL-Kernel.

2. Run one strict R/R step without Nsight

This is the correctness gate to run before collecting a trace:

cd "${VIME_ROOT}"

RUN_ID="rr-bitwise-$(date -u +%Y%m%dT%H%M%SZ)"
ARTIFACT_DIR="${VIME_ROOT}/outputs/rlkernel/${RUN_ID}"
mkdir -p "${ARTIFACT_DIR}"

env \
  RL_KERNEL_ROOT="${RL_KERNEL_ROOT}" \
  PYTHON_BIN="${PYTHON_BIN}" \
  RAY_BIN="${RAY_BIN}" \
  MEGATRON_ROOT="${MEGATRON_ROOT}" \
  MODEL_ROOT="${MODEL_ROOT}" \
  TORCH_DIST_ROOT="${TORCH_DIST_ROOT}" \
  VIME_CKPT="${ARTIFACT_DIR}/unused-checkpoint" \
  PROMPT_DATA="${PROMPT_DATA}" \
  RL_KERNEL_RUN_ID="${RUN_ID}" \
  RL_KERNEL_ARTIFACT_DIR="${ARTIFACT_DIR}" \
  RL_KERNEL_ATTENTION_CASE=R/R \
  RL_KERNEL_FFN_CASE=R/R \
  RL_KERNEL_LOGP_CASE=R/R \
  TP_SIZE=2 CP_SIZE=2 \
  ACTOR_GPUS=4 ROLLOUT_GPUS=4 NUM_GPUS=8 \
  ROLLOUT_GPUS_PER_ENGINE=2 \
  ROLLOUT_BATCH_SIZE=1 \
  N_SAMPLES_PER_PROMPT=2 \
  MAX_RESPONSE_LEN=128 \
  GLOBAL_BATCH_SIZE=2 \
  MAX_TOKENS_PER_GPU=2048 \
  VLLM_GPU_MEMORY_UTILIZATION=0.4 \
  VLLM_ENFORCE_EAGER=1 \
  CI_TEST=1 \
  CI_LOGP_ABS_DIFF_THRESHOLD=0 \
  SAVE_OPTIM=0 SAVE_CHECKPOINT=0 \
  VALIDATE_ARTIFACTS=1 \
  TRACE_MODE=0 \
  bash scripts/codex-debug-qwen3-8B-rlkernel-tp2-cp2.sh \
  2>&1 | tee "${ARTIFACT_DIR}/run.log"

The minimal workload is one prompt, two samples, response length 128, and global batch 2. At CP=2, the reported selected-logprob comparison owns 128 response rows on the reporting CP shard.

Required result:

train_current_rollout_logprob_numel          128
train_current_rollout_logprob_mismatch_count 0
train_current_rollout_logprob_max_abs_diff   0.0

Also require:

• bitwise-validation.json reports torch_equal=true and fallback_failures=0;
• Megatron and vLLM readbacks contain executed Attention, FFN, and logprob records;
• every R side reports an rlkernel.* backend and fallback=false;
• the low-noise route line is printed once per worker/operator, not once per invocation;
• SM90-only preflight succeeds and det_gemm_naive is unavailable in strict mode.

3. Capture the full-stack trace

The analyzed trace was started before Ray so CUDA activity from the Ray CLI, submitted train process, Megatron actors, and vLLM workers was captured. Use the same environment as section 2, but set CI_TEST=0, VALIDATE_ARTIFACTS=0, and launch the runner under one Nsight session:

RUN_ID="rr-bitwise-perf-trace-$(date -u +%Y%m%dT%H%M%SZ)"
ARTIFACT_DIR="${VIME_ROOT}/outputs/rlkernel/${RUN_ID}"
TRACE_BASE="${ARTIFACT_DIR}/trace/full-stack"
RUN_LOG="${ARTIFACT_DIR}/run.log"
SESSION="rrbitwiseperf"
mkdir -p "${ARTIFACT_DIR}/trace"

cleanup() {
  nsys stop --session="${SESSION}" >/dev/null 2>&1 || true
  "${RAY_BIN}" stop --force >/dev/null 2>&1 || true
}
trap cleanup EXIT INT TERM

export RL_KERNEL_RUN_ID="${RUN_ID}"
export RL_KERNEL_ARTIFACT_DIR="${ARTIFACT_DIR}"
export RL_KERNEL_ATTENTION_CASE=R/R
export RL_KERNEL_FFN_CASE=R/R
export RL_KERNEL_LOGP_CASE=R/R
export TP_SIZE=2 CP_SIZE=2
export ACTOR_GPUS=4 ROLLOUT_GPUS=4 NUM_GPUS=8
export ROLLOUT_GPUS_PER_ENGINE=2
export ROLLOUT_BATCH_SIZE=1 N_SAMPLES_PER_PROMPT=2
export MAX_RESPONSE_LEN=128 GLOBAL_BATCH_SIZE=2 MAX_TOKENS_PER_GPU=2048
export VIME_CKPT="${ARTIFACT_DIR}/unused-checkpoint"
export CI_TEST=0 SAVE_OPTIM=0 SAVE_CHECKPOINT=0 VALIDATE_ARTIFACTS=0
export TRACE_MODE=0 VLLM_ENFORCE_EAGER=1

nsys launch \
  --session-new="${SESSION}" \
  --trace=cuda,nvtx \
  --trace-fork-before-exec=true \
  --wait=primary \
  --show-output=true \
  bash "${VIME_ROOT}/scripts/codex-debug-qwen3-8B-rlkernel-tp2-cp2.sh" \
  >"${RUN_LOG}" 2>&1 &
launch_pid=$!

sleep 1
nsys start \
  --session="${SESSION}" \
  --sample=none \
  --cpuctxsw=none \
  --force-overwrite=true \
  --output="${TRACE_BASE}"

set +e
wait "${launch_pid}"
run_status=$?
set -e
nsys stop --session="${SESSION}" || true
trap - EXIT INT TERM
cleanup
exit "${run_status}"

After the run, validate the saved tensors/readbacks separately:

"${PYTHON_BIN}" \
  "${RL_KERNEL_ROOT}/examples/vime_qwen3_8b_tp2_cp2/validate_artifacts.py" \
  --readback-dir "${ARTIFACT_DIR}/readbacks" \
  --train-data-dir "${ARTIFACT_DIR}/train-data" \
  --output "${ARTIFACT_DIR}/bitwise-validation.json"

The trace must contain all RL-Kernel payload families:

det_gemm_sm90_kernel > 0
det_gemm_naive == 0
FA4 forward/backward kernels > 0
SwiGLU CUDA kernels > 0
linear_logp CUDA kernels > 0
deterministic_all_gather_kernel > 0
deterministic_reduce_scatter_kernel > 0
deterministic_all_reduce_kernel > 0

The successful trace recorded 3,312 self-owned AG kernels, 2,160 self-owned RS kernels, and 9,585 self-owned AllReduce kernels. These prove that the payload path was RL-Kernel CUDA IPC rather than NCCL.

The full application may still contain NCCL from Megatron, vLLM, weight synchronization, and collective control/validation. NCCL is also retained as the explicit p2p_nccl_reference fallback backend. For this strict trace, fallback must be disabled: readback must report communication_backend=cuda_ag_rs, fallback_allowed=false, and fallback=false; the trace must contain self-owned payload kernels and zero p2p_nccl_reference payload calls. Global NCCL counts cannot decide whether the Attention AG/RS payload routed correctly.

4. Ablation-matrix semantics

P means the framework's production implementation and R means the RL-Kernel implementation. The first letter is Megatron training; the second is vLLM rollout:

Case	Training	Rollout	Purpose
R/R	RL-Kernel	RL-Kernel	strict control
P/R	production	RL-Kernel	isolate a rollout-only implementation difference
R/P	RL-Kernel	production	isolate a training-only implementation difference
P/P	production	production	native baseline for the selected module

These are full post-training runs, not isolated unit kernels. Every cell performs model initialization/weight application, vLLM rollout, selected-logprob transfer, Megatron training, and artifact readback. A P side is an explicitly selected route, not an unplanned fallback. Any R side that falls back is a failure.

The operator-isolation matrix changes one module at a time while holding the other two at R/R:

Run	Attention	FFN	logprob
attention-PR	P/R	R/R	R/R
attention-RP	R/P	R/R	R/R
attention-PP	P/P	R/R	R/R
ffn-PR	R/R	P/R	R/R
ffn-RP	R/R	R/P	R/R
ffn-PP	R/R	P/P	R/R
logp-PR	R/R	R/R	P/R
logp-RP	R/R	R/R	R/P
logp-PP	R/R	R/R	P/P

Run the R/R control first. The nine cells above then explain which module/side reintroduces a mismatch without changing the rest of the user-visible workflow. An optional all-production baseline sets all three variables to P/P, but it is not one of the nine operator-isolation cells.

5. Run one matrix cell

Example: production Attention in training and RL-Kernel Attention in rollout, with strict R/R FFN and logprob:

MODULE=attention
CASE_ID=P/R
RUN_NAME=attention-PR
ARTIFACT_DIR="${VIME_ROOT}/outputs/rlkernel/ablation-${RUN_NAME}-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "${ARTIFACT_DIR}"

env \
  RL_KERNEL_ROOT="${RL_KERNEL_ROOT}" \
  PYTHON_BIN="${PYTHON_BIN}" \
  RAY_BIN="${RAY_BIN}" \
  RL_KERNEL_RUN_ID="ablation-${RUN_NAME}" \
  RL_KERNEL_ARTIFACT_DIR="${ARTIFACT_DIR}" \
  VIME_CKPT="${ARTIFACT_DIR}/unused-checkpoint" \
  RL_KERNEL_ATTENTION_CASE=P/R \
  RL_KERNEL_FFN_CASE=R/R \
  RL_KERNEL_LOGP_CASE=R/R \
  CI_TEST=0 SAVE_OPTIM=0 SAVE_CHECKPOINT=0 VALIDATE_ARTIFACTS=0 \
  MAX_RESPONSE_LEN=128 ROLLOUT_BATCH_SIZE=1 N_SAMPLES_PER_PROMPT=2 \
  GLOBAL_BATCH_SIZE=2 MAX_TOKENS_PER_GPU=2048 \
  VLLM_GPU_MEMORY_UTILIZATION=0.4 \
  bash "${VIME_ROOT}/scripts/codex-debug-qwen3-8B-rlkernel-tp2-cp2.sh" \
  2>&1 | tee "${ARTIFACT_DIR}/run.log"
printf '%s\n' "${PIPESTATUS[0]}" >"${ARTIFACT_DIR}/exit-code"

Change only the three RL_KERNEL_*_CASE variables to run another cell. The runner derives selected-logprob provider mode from RL_KERNEL_LOGP_CASE: training P/* uses the Vime production provider path, while training R/* requires the strict RL-Kernel provider.

6. Run and resume the complete matrix

The validation host currently has:

${VIME_ROOT}/scripts/run-ablation-matrix.sh
${VIME_ROOT}/scripts/analyze-ablation-matrix.py

Run all nine cells sequentially:

cd "${VIME_ROOT}"
export MATRIX_STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
export MATRIX_ROOT="${VIME_ROOT}/outputs/rlkernel/ablation-matrix-${MATRIX_STAMP}"
bash scripts/run-ablation-matrix.sh

The current host helper contains host-specific defaults for VIME_ROOT, RL_KERNEL_ROOT, and PYTHON_BIN. Before upstreaming, parameterize all three as ${VARIABLE:-default}, record both repository SHAs in every case, and add:

• --case module:case to run one cell;
• --resume to skip a cell only when exit-code, run.log, and readbacks are complete;
• --rerun-failed to rerun failed/incomplete cells;
• cleanup that always stops Ray between cells;
• a unique timestamp by default instead of a fixed date.

Until --resume exists, resume manually by running the single-cell command from section 5 for each missing directory. Do not rerun completed cells into the same directory because that mixes readbacks from different processes.

Generate or regenerate the report:

"${PYTHON_BIN}" "${VIME_ROOT}/scripts/analyze-ablation-matrix.py" \
  --matrix-root "${MATRIX_ROOT}" \
  --output "${MATRIX_ROOT}/matrix-report.json"

The analyzer must check, per cell:

• runner exit code and absence of traceback;
• the exact Attention/FFN/logprob plan inherited by Megatron and vLLM;
• executed operator records for both framework sides;
• expected production/RL-Kernel implementation and backend identity;
• zero unplanned fallback on every R side;
• train_current_rollout_logprob_numel == 128;
• mismatch count and max absolute difference recorded as diagnostic output.

A mixed or P/P cell is not required to be bitwise zero. Its mismatch is the diagnostic signal. matrix-report.json.passed=true means the requested routes executed and evidence is complete; only the R/R control has the strict mismatch_count=0 correctness requirement.

7. Artifact layout

Every run should be self-contained:

outputs/rlkernel/<run-id>/
  run.log
  exit-code
  matrix-case.env                 # matrix cells only
  readbacks/
    megatron-training-*.json
    vllm-rollout-*.json
  train-data/
    <rollout-id>.rank<rank>.pt
  bitwise-validation.json         # R/R validation
  matrix-report.json              # matrix root only
  trace/
    full-stack.nsys-rep
    full-stack.sqlite

Do not declare success from console text alone. Archive the source SHAs, full environment plan, readbacks, tensor comparison, and trace gate in machine-readable JSON.

Definition of done

• The strict runner, matrix runner, and analyzer are tracked and use configurable paths.
• A clean 8xH100 checkout can execute the section 2 command without local source edits.
• R/R remains 0/128, max_abs_diff=0.0, with zero Attention/FFN/logprob fallback.
• The trace proves SM90 GEMM and self-owned CUDA IPC AG/RS/AllReduce execution.
• The strict trace selects cuda_ag_rs / rlkernel.cuda_ipc.*, sets fallback_allowed=false, reports fallback=false, and executes zero p2p_nccl_reference payload calls.
• All nine full-stack matrix cells can run, resume, and produce one machine-readable report.
• Documentation and scripts refer only to RL-Align/vime, never vllm-project/vime.

Refs #335, #341.

---

Reproducibility audit: current host state versus a clean checkout

This section is normative and supersedes any earlier wording that implies the
current host can already be recreated from clean test and main checkouts.
It records the state audited after the strict R/R and production P/P traces.

Current answer

Scenario	Reproducible now?	Reason
Rerun R/R in the existing host directories	Yes	The dirty RL-Kernel checkout, untracked Vime runner, runtime, checkpoints, and compatibility files are present.
R/R from clean RL-Kernel/Vime checkouts	No	Required RL-Kernel changes and the Vime runner are not all tracked upstream.
P/P production trace from clean checkouts	No	The TE 2.11/CUDA 12 isolation, production runner, compatibility shim, and lazy production integration are not fully documented or tracked upstream.
Full nine-cell matrix from clean checkouts	No	The matrix runner and analyzer are untracked in Vime.

Do not report a clean reproduction until every blocker below is resolved and
the clean-room gate at the end of this section passes. A source hash for a
dirty diff is evidence of host identity, not a substitute for publishing the
changes.

Exact source snapshot found on the validation host

RL-Align/RL-Kernel origin/test:
  7207ebda0447d48f2cd85091864aa22c4edcd40b
RL-Align/RL-Kernel PR #339 head and current checkout HEAD:
  5a1ad576c9a4f85b1154c4692bf92b2d491d10a7
RL-Align/vime origin/main and checkout HEAD:
  b2c9b02b25529bd500fb49cc9f070971685ee548
Megatron-LM checkout HEAD:
  1dcf0dafa884ad52ffb243625717a3471643e087

origin was verified as https://github.com/RL-Align/RL-Kernel.git; the
remote test ref really is 7207ebd, rather than a stale local remote-tracking
ref.

The RL-Kernel checkout has a 467-line, 20,231-byte dirty diff over PR #339.
The binary-safe patch stream has this identity:

sha256(git diff --binary) =
bd29e676c4ed41a8521bbf434460831aa7d6bd53274181503cb35d7e2ec3b5ec

Modified files:

csrc/cuda/gemm/det_gemm_kernel.cu
csrc/ops.cpp
rl_engine/integrations/framework_operators.py
rl_engine/integrations/megatron_runtime.py
rl_engine/integrations/runtime.py
rl_engine/integrations/vime/logp.py
rl_engine/integrations/vllm_runtime.py
rl_engine/kernels/ops/cuda/matmul/det_gemm.py
tests/test_framework_runtime_adapters.py
tests/test_vime_logprob_provider.py

The delta contains the SM90 fail-closed dispatch/compiled probe, low-noise
route/readback behavior, production-side lazy strict-operator construction,
the vLLM flash-attn compatibility namespace, and tests. These changes must be
published as reviewed commits on the intended RL-Kernel branch. A clean user
cannot reconstruct them from the SHA above.

The following Vime files are untracked at b2c9b02:

b4387b9460d5e9330645e4aadb68958132e56c76db5a710a777ecf33816ef455  scripts/codex-debug-qwen3-8B-rlkernel-tp2-cp2.sh
0b3f8906ffdfeb3ade35d2239d25665d8216f2a604b2d037a5227d7596191a67  scripts/codex-debug-qwen3-8B-production-pp-tp2-cp2.sh
212f26b6ff9918c66e424db56a12327c7729d277e3d5736a8b024531c39a9eb7  scripts/run-ablation-matrix.sh
0569f2819a8cf260f0adf5914ef368645568e90476fe1e7de5ca834eb15930df  scripts/analyze-ablation-matrix.py

These four files must be upstreamed to RL-Align/vime; do not use or modify
vllm-project/vime. Until they are tracked, sections that invoke them are
host-only instructions.

Hardware and base runtime manifest

GPU                         8x NVIDIA H100 80GB HBM3
NVIDIA driver               595.71.05
Nsight Systems              2025.3.2.474-253236389321v0
Python                      3.11.15
PyTorch                     2.9.1+cu128
vLLM                        0.16.0
Ray                         2.57.0
NumPy                       1.26.4
nvidia-cudnn-cu12           9.10.2.21
nvidia-cublas-cu12          12.8.4.1
flash-attn-4                0.0.0
base cuda-python            13.3.1

Record pip freeze, nvidia-smi -q, all repository SHAs, build flags, and
nsys --version into every new artifact directory. The extension must be
rebuilt with the same PyTorch/CUDA ABI used by Ray workers.

Production P/P runtime overlay

P/P is a Vime production-operator baseline, but the successful run is not a
vanilla uninstrumented Vime checkout. It uses RL-Kernel route/readback wrappers
with every selected side set to production. The P/P-only overlay is:

transformer_engine          2.11.0
transformer_engine_torch    2.11.0
transformer_engine_cu12     2.11.0
cuda-python                 12.8.0
cuda-bindings               12.8.0
Transformer Engine source   c188b533cc3721ca9c6bbfd26148f5cf60108c25
TE torch wheel SHA-256      5e277c95c8df38d4b3621f320702a6a36dc1396283ea8f5e2afc037721d633fa

Host overlay paths:

export TE_SITE=/home/ellm/ljj/.te-2.11.0-site-packages
export CUDA_PYTHON_SITE=/home/ellm/ljj/.cuda-python-12.8-site-packages
export CUDA12_COMPAT_LIB=/home/ellm/ljj/.cuda12-compat-lib

NVIDIA_LIBS="$({ find "$(dirname "$(dirname "${PYTHON_BIN}")")/lib/python3.11/site-packages/nvidia" \
  -type d -name lib -print 2>/dev/null || true; } | paste -sd: -)"
export PYTHONPATH="${CUDA_PYTHON_SITE}:${TE_SITE}:${PYTHONPATH:-}"
export LD_LIBRARY_PATH="${CUDA12_COMPAT_LIB}:${TE_SITE}/transformer_engine/wheel_lib${NVIDIA_LIBS:+:${NVIDIA_LIBS}}:${LD_LIBRARY_PATH:-}"

.cuda12-compat-lib resolves unversioned libcudart, libcurand, and
libnvrtc to the CUDA 12 runtime packaged in the Python environment. It also
contains an intentionally invalid libcudart.so.13 sentinel: TE 2.11's cuDNN
frontend probes both libcudart major versions and fails when CUDA 12 and 13 are
both loadable. This overlay hides the system CUDA 13 runtime only from the P/P
process; do not change system libraries.

Required P/P plan:

export RL_KERNEL_ATTENTION_CASE=P/P
export RL_KERNEL_FFN_CASE=P/P
export RL_KERNEL_LOGP_CASE=P/P
export TRANSFORMER_IMPL=transformer_engine
export ATTENTION_BACKEND=auto
export CP_COMM_TYPE=p2p
export TP_SIZE=2
export CP_SIZE=2

TE 2.11 all_gather CP rejects packed THD padding_causal; the successful
production run therefore uses TE P2P CP. This differs from R/R, whose Attention
payload reports RL-Kernel cuda_ag_rs. Any performance comparison must retain
this caveat.

Successful P/P artifact:

/home/ellm/ljj/vime-debug-main/outputs/rlkernel/
  pp-production-perf-trace-20260825T122901Z/

P/P acceptance is route correctness, not bitwise equality:

Attention/FFN/logprob plan       P/P on Megatron and vLLM
fallbacks                        []
strict det_gemm/FA4/logp calls   0 in P/P operator payload
mismatch count                   82 / 128 (diagnostic, allowed)
max_abs_diff                     0.17086082696914673

Model, converted checkpoint, tokenizer, and prompt identity

The current host input sizes are approximately 31 GiB for
Qwen3-8B_vime_rlkernel_tp2_cp2 and 16 GiB for Qwen3-8B_torch_dist.
Complete manifests were generated at:

/home/ellm/ljj/Qwen3-8B_vime_rlkernel_tp2_cp2.sha256
/home/ellm/ljj/Qwen3-8B_torch_dist.sha256

Manifest identities:

bb0b394a6016b0ce8c1dda7c2137aa3812fed1e6d696cb906877908872d30c1f  Qwen3-8B_vime_rlkernel_tp2_cp2.sha256
9187aadade73857146fc3355396bc25d6790dbb9dd27067613ee88f0b941cb1e  Qwen3-8B_torch_dist.sha256
cd59c54aca26fb326d8dd5a2a0545d43313b0527170c3a77b9d270eff552b9b4  qwen3_tp2_cp2_smoke_valid.jsonl

Important model metadata identities:

f7c4eadfbbf522470667b797a3c89be2524832d2d599797248dc304fff447c30  config.json
2325da0f15bb848e018c5ae071b7943332e9f871d6b60e2ed22ca97d4cb993d2  generation_config.json
f9fdbcb91c23971c13ec5d5f2573d2349e8f61f2f049371ec699281748fdb1bc  model.safetensors.index.json
d5d09f07b48c3086c508b30d1c9114bd1189145b74e982a265350c923acd8101  tokenizer_config.json
aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4  tokenizer.json

Copy the complete manifest files with the checkpoints and validate them before
starting Ray:

sha256sum --check Qwen3-8B_vime_rlkernel_tp2_cp2.sha256
sha256sum --check Qwen3-8B_torch_dist.sha256
printf '%s  %s\n' \
  cd59c54aca26fb326d8dd5a2a0545d43313b0527170c3a77b9d270eff552b9b4 \
  "${PROMPT_DATA}" | sha256sum --check

Paths alone are not reproducibility evidence. A run with different converted
weights, tokenizer files, generation config, or prompt data must use a new
artifact identity and cannot be compared as the same bitwise experiment.

Required upstream work before claiming clean reproduction

• Merge or otherwise publish PR fix(integrations): restore strict Qwen3 bitwise parity #339 plus every required local RL-Kernel
  delta; update this issue to the resulting exact test SHA.
• Make both R/R and P/P runners tracked files in RL-Align/vime main and
  replace the host file hashes above with a commit SHA.
• Track the matrix runner and analyzer in RL-Align/vime main with
  --case, --resume, and --rerun-failed support.
• Replace manually assembled TE/CUDA overlay directories with a locked,
  scripted environment build or container image and publish its digest.
• Store complete source/environment/input manifests inside every artifact
  directory before execution.
• Ensure production-only imports construct no strict operator instances;
  P/P trace/readback must contain no strict operator payload.
• Keep route output once per worker/operator and remove per-call filesystem
  persistence before using P/P as an uninstrumented throughput claim.

Clean-room acceptance gate

Run this gate on a fresh directory or a second node. Reusing the dirty host
checkout does not count.

1. Clone only the published RL-Kernel test SHA, RL-Align/vime main SHA, and
   pinned Megatron SHA. git status --porcelain must be empty in all three.
2. Build/install RL-Kernel in the exact Ray Python environment and run the
   SM90 fail-closed CUDA probe. Require det_gemm_sm90_compiled=true and reject
   every naive/fallback route.
3. Validate both complete checkpoint manifests and the prompt hash before Ray
   starts.
4. Run the targeted integration tests and archive their output.
5. Run R/R without Nsight. Require 0/128 mismatch, max diff 0, all six framework
   routes executed, fallbacks=[], and self-owned CUDA IPC collective payload.
6. Capture the full R/R trace before Ray. Require SM90 GEMM, FA4, SwiGLU,
   linear-logp, deterministic AG/RS/AllReduce, and zero strict fallback.
7. Build the locked P/P overlay and run P/P separately. Require every route to
   be production and fallbacks=[]; do not require bitwise equality.
8. Run each matrix cell into a unique directory, resume only complete cells,
   and produce the machine-readable report.
9. Save repository SHAs, dirty-state checks, package lock/container digest,
   environment variables, input manifests, readbacks, run log, exit code, and
   trace gates in the artifact directory.

Only after this gate passes may this issue change the clean-checkout rows in
the status table from No to Yes.