[WS2][Attention] H100 validation and reproduction guide for #235 PR1-PR8

[inaniloquentee]

Status

The strict Attention operator gate passed on 8xH100. The production path uses rlkernel.attention.deterministic_core.v1, disables Split-KV, accumulates in FP32, merges in fixed global_block_index order, and downcasts only at final write.

Evidence	Result
Hardware	8x NVIDIA H100 80 GB
Runtime	PyTorch 2.13.0+cu130; FlashInfer 0.6.17
Attention regression	1056 passed, 84 skipped
Required acceptance	Passed; failed_required_cases=[]
Strict forward/backward	Out/LSE/dQ/dK/dV max abs 0.0 at 2/4/8 ranks
Strict decode/prefill	Out/LSE/dlogp max abs 0.0
Fixed Split-K diagnostic	Native FlashInfer runtime plans verified; no fallback
Pre-overlap Ring default	rlkernel.attention.strict_ring_state.v1; decoupled compute/communication; fixed merge order; overlap disabled; full-KV deterministic arithmetic retained

The H100 artifacts were produced from integrated validation commit 5e90e7b. The current PR7 head is 4784c9b (strict Ring schedule plus the formatting-only CI follow-up).

H100 result

PR stack

All PRs target test. Record the exact fetched head for every rerun; do not copy stale SHAs from this issue.

Order	PR	Owned validation boundary
1	#236	Attention contract, TP/CP ownership, RoPE/cache metadata, strict dispatch
2	#253	Single-GPU full/chunked/paged-KV attribution
3	#238	Deterministic CP reference and fixed logical (Out, LSE) merge
4	#284	Prefill/chunked-prefill backward reference (dQ/dK/dV)
5	#260	Decode KV-cache replay and cache/position/RoPE identity
6	#285	Rank-aware aggregate gate and machine-readable artifacts
7	#263	TP=2/CP=2 engine binding and provenance; not an engine execution claim
8	#279	Shared deterministic core, P2P NCCL reference, self-owned AG/RS, paged-KV adapter

Logical dependencies remain: #236/#238 before #263; #238 before #284; #284 before #285; #253 before #260; #253/#260 before #279.

Clean H100 setup

For a clean rerun, use a CUDA 12.4 devel image. The recommended image family is:

runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04

Keep the checkout, Conda environment, caches, and artifacts under a user-owned ljj directory. Do not install into (base) or another user's checkout.

export RLK_ROOT="${RLK_ROOT:-$HOME/workspace/ljj}"
export RLK_REPO="${RLK_REPO:-$RLK_ROOT/RL-Kernel}"
export RLK_ENV="${RLK_ENV:-$RLK_ROOT/.conda/envs/rlk-attention-py311}"
export RLK_ARTIFACTS="${RLK_ARTIFACTS:-$RLK_ROOT/results/issue294}"
mkdir -p "$RLK_ROOT" "$RLK_ARTIFACTS"

nvidia-smi -L
nvcc --version

if [ ! -d "$RLK_REPO/.git" ]; then
  git clone https://github.com/RL-Align/RL-Kernel.git "$RLK_REPO"
fi
cd "$RLK_REPO"
[ "$(stat -c %U .)" = "$(id -un)" ] || { echo "checkout is not user-owned"; exit 1; }

source "$(conda info --base)/etc/profile.d/conda.sh"
if [ ! -x "$RLK_ENV/bin/python" ]; then
  conda create -p "$RLK_ENV" python=3.11 -y
fi
conda activate "$RLK_ENV"

NVCC_PATH="$(readlink -f "$(command -v nvcc)")"
export CUDA_HOME="$(dirname "$(dirname "$NVCC_PATH")")"
export TORCH_CUDA_ARCH_LIST="9.0+PTX"
export FORCE_CUDA=1
export MAX_JOBS=8

nvidia-smi -L must show eight H100s and nvcc --version must show CUDA 12.4. If the provider image is runtime-only or resolves a different compiler, recreate the instance with the devel image instead of changing a shared host.

Install the pinned core stack

cd "$RLK_REPO"
python -m pip install --upgrade pip setuptools wheel ninja packaging
python -m pip install --no-cache-dir torch==2.4.1 --index-url https://download.pytorch.org/whl/cu124
python -m pip install --no-cache-dir numpy tabulate pytest
python -m pip install --no-cache-dir --no-deps flashinfer-python==0.6.17

for pr in 236 253 238 284 260 285 263 279; do
  git fetch origin "pull/${pr}/head:refs/remotes/origin/pr/${pr}"
done
git switch --detach refs/remotes/origin/pr/279

export TORCH_LIB="$(python -c 'import os,torch; print(os.path.join(os.path.dirname(torch.__file__), "lib"))')"
export LD_LIBRARY_PATH="$TORCH_LIB:$CUDA_HOME/lib64:$CUDA_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export LIBRARY_PATH="$CUDA_HOME/lib64:$CUDA_HOME/lib:$CUDA_HOME/lib64/stubs${LIBRARY_PATH:+:$LIBRARY_PATH}"
python -m pip install --no-build-isolation --no-deps -e .
python -m pip check

python - <<'PY'
import torch
from rl_engine import _C

assert torch.__version__.startswith("2.4.1")
assert torch.version.cuda == "12.4"
assert torch.cuda.is_available() and torch.cuda.device_count() >= 8
assert torch.distributed.is_nccl_available()
assert all(torch.cuda.get_device_capability(i) == (9, 0) for i in range(8))
print("environment OK")
PY

FlashInfer 0.6.17 is required for the PR7 paged-KV adapter and fixed-Split-K diagnostic. Transformer Engine, Megatron, vLLM, and a Qwen3 checkpoint remain optional engine-integration dependencies; keep them in a separate compatible environment and never upgrade the pinned Torch stack silently.

Test order

1. Focused regression

cd "$RLK_REPO"
git switch --detach refs/remotes/origin/pr/279
python -m pytest tests/test_flashinfer_pr7_attention.py -q
# record the actual output; this is a local regression check only

git switch --detach refs/remotes/origin/pr/285
python -m pytest tests/test_ws2_attention_gpu_acceptance.py \
  tests/test_ws2_cp_attention_drift_benchmark.py -q
# record the actual output; this is a local acceptance harness check only

git switch --detach refs/remotes/origin/pr/279

2. Retained P2P NCCL reference

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc-per-node=2 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport p2p_nccl_reference --repeats 3 \
  --output "$RLK_ARTIFACTS/p2p-nccl-reference-2rank.json"

This checks transport semantics and fixed global_block_index merge order. It is not the production strict arithmetic path.

3. Strict shared-core AG/RS matrix

CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nproc-per-node=2 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-2rank-pr279-head.json"

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --standalone --nproc-per-node=4 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-tp2-cp2-4rank-pr279-head.json"

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc-per-node=8 \
  scripts/ws2_p2p_nccl_attention_reference_check.py \
  --transport cuda_ag_rs --strict-shared-core --repeats 3 \
  --output "$RLK_ARTIFACTS/strict-shared-core-agrs-tp2-cp2-replica2-8rank-pr279-head.json"

Acceptance

Each required JSON must report:

• the exact commit, world size, TP/CP topology, replica index, dtype, and logical block manifest;
• core_id=rlkernel.attention.deterministic_core.v1;
• Split-KV disabled, native_attention_arithmetic=false, and fallback=false;
• compute_schedule=rlkernel.attention.strict_ring_state.v1, communication_overlap=disabled, ring_schedule_default=true, and ring_partial_arithmetic=false;
• communication_autograd=true and global_failure_count=0;
• bitwise-equal Out, attention-domain LSE, dQ, dK, and dV with max abs 0.0;
• repeat-run Out/LSE bitwise equality on every rank.

Artifact	Topology	Result
ws2-p2p_nccl_reference.json	P2P reference, 2 ranks	Passed
ws2-p2p_nccl_reference_tp2_cp2.json	P2P reference, 4 ranks	Passed
ws2-p2p_nccl_reference_tp2_cp2_replica2.json	P2P reference, 8 ranks	Passed
ws2-custom_cuda_ag_rs.json	TP=1, CP=2	Passed; all strict tensors 0.0
ws2-custom_cuda_ag_rs_tp2_cp2.json	TP=2, CP=2	Passed; all strict tensors 0.0
ws2-custom_cuda_ag_rs_tp2_cp2_replica2.json	TP=2, CP=2, two replicas	Passed; all strict tensors 0.0

Boundary

Strict operator acceptance is complete for the tested H100 shapes and topologies. The following remain separate claims and must not be inferred from these artifacts:

• end-to-end Megatron/vLLM Qwen3 checkpoint replay and active-token model-level dlogp;
• native TE Attention arithmetic and auto Split-K execution; fixed FlashInfer Split-K passed only as a diagnostic and is not the strict production path;
• performance acceptance for communication/computation overlap;
• decode backward or a production fused backward claim.

Related: #235, #108.