[WS2][Logprob] H100 validation and reproduction guide for Issue #241 PR1-PR5

[hihaluemen]

Status

The Issue #241 PR1-PR5 integration stack passed the complete single-host H100 validation matrix.

Tested integration commit:

9aca0596ab4c3670361f4464e2a3f56b0240f332

Integrated stack:

PR1 #259 -> PR2 #262 -> PR3 #265 -> PR4 #289 -> PR5 #291

H100 validation matrix

Evidence	Result
Hardware	8 x NVIDIA H100 80GB HBM3
Runtime	Python 3.11.15; PyTorch 2.11.0+cu128; NCCL
Candidate	BF16 pytorch-vocab-parallel-logp-ws2 with FP32 accumulation
Reference	Independent full-vocabulary FP32 PyTorch oracle
Topology	TP=1/2/4 x CP=1/2; 1/2/4/8 ranks
Focused regression	179 passed, 1 skipped
Required matrix	Six of six topology reports passed
Cross-TP determinism	TP=1/2/4 candidate hashes identical under CP=1
Fallback	None
Invalid JSON values	None; strict JSON serialization

TP	CP	Ranks	LSE max abs	Active dlogp max abs	Result
1	1	1	3.3379e-05	5.8489e-03	Passed
1	2	2	3.3379e-05	5.8489e-03	Passed
2	1	2	3.3379e-05	5.8489e-03	Passed
2	2	4	3.3379e-05	5.8489e-03	Passed
4	1	4	3.3379e-05	5.8489e-03	Passed
4	2	8	3.3379e-05	5.8489e-03	Passed

Validation Boundary

This validation covers deterministic TP-aware selected-token logprob and the PR5 GRPO contract tests.

It validates:

• BF16 candidate logits with FP32 accumulation;
• deterministic vocab-parallel TP merge;
• explicit backend materialization and fail-closed dispatch;
• the TP=1/2/4 x CP=1/2 matrix;
• comparison with an independent full-vocabulary FP32 PyTorch oracle;
• bitwise equality within each TP group;
• cross-topology bitwise equality of TP=1/2/4 candidate tensors;
• structured JSON artifacts and reproducible launch commands.

It does not claim end-to-end Megatron/vLLM replay, rollout-versus-training integration, PP/DP model execution, or the optional PR6 fused kernel.

Hardware and Runtime

GPU: 8 x NVIDIA H100 80GB HBM3
Compute capability: 9.0 (SM90)
Driver: 595.71.05
Driver-reported CUDA compatibility: 13.2
nvcc: CUDA 13.0
Python: 3.11.15
PyTorch: 2.11.0+cu128
PyTorch CUDA runtime: 12.8
NCCL: available

The tested backend is pytorch-vocab-parallel-logp-ws2. The optional PR2 cuda-sm90 extension was not compiled and is not required for the PR4 PyTorch/NCCL path.

Reproduction

export RLK_ROOT="${RLK_ROOT:-$HOME/workspace/lhh}"
export RLK_REPO="${RLK_REPO:-$RLK_ROOT/RL-Kernel}"
export RLK_RESULTS_ROOT="${RLK_RESULTS_ROOT:-$RLK_ROOT/results/issue241}"

cd "$RLK_REPO"
git fetch origin test/issue-241-logprob-integration
git switch test/issue-241-logprob-integration
git pull --ff-only origin test/issue-241-logprob-integration

test "$(git rev-parse HEAD)" = "9aca0596ab4c3670361f4464e2a3f56b0240f332"

export RUN_NAME="$(git rev-parse --short HEAD)-$(date +%Y%m%d-%H%M%S)"
export PR4_OUTPUT_DIR="$RLK_RESULTS_ROOT/$RUN_NAME"
mkdir -p "$PR4_OUTPUT_DIR"
export PR4_OUTPUT_BASE="$RLK_RESULTS_ROOT"
set -o pipefail

Record the environment before testing:

{
  echo "commit: $(git rev-parse HEAD)"
  git log -1 --oneline
  echo "branch: $(git branch --show-current)"
  python --version
  python -c 'import torch; import torch.distributed as dist; print("torch:", torch.__version__); print("torch_cuda:", torch.version.cuda); print("cuda_available:", torch.cuda.is_available()); print("gpu_count:", torch.cuda.device_count()); print("nccl_available:", dist.is_nccl_available())'
  nvidia-smi
  nvcc --version
} > "$PR4_OUTPUT_DIR/environment.txt" 2>&1

Test Order

1. Focused Regression

python -m pytest -q -rs \
  tests/test_logprob_contract.py \
  tests/test_vocab_parallel_logp.py \
  tests/test_logprob_comparison.py \
  tests/test_distributed_logprob_comparison.py \
  tests/test_grpo_loss_contract.py \
  tests/test_distributed_grpo_loss.py \
  2>&1 | tee "$PR4_OUTPUT_DIR/pytest-focused.log"

Recorded result:

179 passed, 1 skipped

The single skipped test is the optional PR2 cuda-sm90 diagnostic because _C.batch_invariant_logp_sm90 was not compiled.

2. Six-Case TP/CP Matrix

TP=1 CP=1, ranks=1
TP=1 CP=2, ranks=2
TP=2 CP=1, ranks=2
TP=2 CP=2, ranks=4
TP=4 CP=1, ranks=4
TP=4 CP=2, ranks=8

Use BF16, real/padded vocabulary 151936, 64 vocabulary tiles, batch 2, sequence 16, prompt tokens 8, seed 123, explicit pytorch-vocab-parallel-logp-ws2, and NCCL. Generate the exact commands and plan:

python rl_engine/testing/distributed_logprob_comparison.py \
  --plan --device cuda --dtype bf16 \
  --backend pytorch-vocab-parallel-logp-ws2 \
  --real-vocab 151936 --padded-vocab 151936 \
  --num-vocab-tiles 64 --batch 2 --seq 16 \
  --prompt-tokens 8 --seed 123 \
  --output "$PR4_OUTPUT_DIR/plan.json" \
  2>&1 | tee "$PR4_OUTPUT_DIR/plan.log"

Run the six printed torchrun commands and write each report to $PR4_OUTPUT_DIR/tp<tp>-cp<cp>.json.

3. Report Acceptance

Every report must satisfy:

passed=true
requested_backend=pytorch-vocab-parallel-logp-ws2
actual_backend=pytorch-vocab-parallel-logp-ws2
fallback=false
tp_outputs_bitwise_replicated=true

Reports must contain finite LSE and active-token dlogp statistics, topology/reduction provenance, and the exact launch command.

4. Cross-Topology Bitwise Check

The runner records SHA-256 fingerprints of the complete aggregate candidate logprob and LSE tensors. Compare the CP=1 reports:

python - <<'PY'
import json
import os
from pathlib import Path

root = Path(os.environ["PR4_OUTPUT_DIR"])
reports = {
    tp: json.loads((root / f"tp{tp}-cp1.json").read_text())
    for tp in (1, 2, 4)
}
fingerprints = {tp: reports[tp]["bitwise_fingerprints"] for tp in reports}
assert len({item["dtype"] for item in fingerprints.values()}) == 1
assert len({tuple(item["shape"]) for item in fingerprints.values()}) == 1
assert len({item["candidate_logp_sha256"] for item in fingerprints.values()}) == 1
assert len({item["candidate_lse_sha256"] for item in fingerprints.values()}) == 1
print("TP=1/2/4 CP=1 candidate outputs are bitwise identical.")
print(fingerprints[1])
PY

For this fixed seeded input, CP=2 must produce the same reconstructed aggregate fingerprints.

Acceptance Results

All six topologies passed:

TP	CP	Ranks	Result	LSE max abs	Active dlogp max abs
1	1	1	Passed	3.337860107421875e-05	0.005848884582519531
1	2	2	Passed	3.337860107421875e-05	0.005848884582519531
2	1	2	Passed	3.337860107421875e-05	0.005848884582519531
2	2	4	Passed	3.337860107421875e-05	0.005848884582519531
4	1	4	Passed	3.337860107421875e-05	0.005848884582519531
4	2	8	Passed	3.337860107421875e-05	0.005848884582519531

The BF16 contract used atol=0.05 and rtol=0.0.

Cross-topology fingerprints:

candidate_logp_sha256:
12885c01b0c0d953667652dd9e3f20dffa60ec0a57a5c1a138de1440f6c9b865

candidate_lse_sha256:
1bf5ceddb7610e0215e06998bc572dae65c5548d4f395a7afdbb78620141a159

The same fingerprints were observed in all six reports. No fallback, traceback, NCCL timeout, NaN, or Infinity was observed.

Artifacts

Archive: issue241-9aca059-20260819-171035.tar.gz
SHA256: 69378708d82997a70b5aa638076d6569febf1df8f1d8d418f1feaf150393fc57

The archive contains the environment record, focused-test log, plan, six JSON reports, and six per-case logs.

Boundary

This result satisfies the Issue #241 PR1-PR5 deterministic logprob acceptance boundary. PR6 remains optional. BF16 candidate versus FP32 oracle drift is expected because the candidate receives BF16 logits and uses a fixed-order vocab-parallel reduction path; the requirement is tolerance compliance, not candidate/oracle bitwise equality.