feat(experimental): add DAPO training support

[CharlesXu-HQ]

What does this PR do?

Adds opt-in, experimental DAPO training support, following the DAPO paper and the maintained verl DAPO recipe.

The implementation includes:

• asymmetric Clip-Higher policy clipping
• dynamic sampling that collects complete informative prompt groups
• response-token-mean policy-gradient normalization across uneven microbatches, gradient accumulation, and data parallelism
• soft overlong reward shaping
• rollout-policy logprobs as the old-policy reference, so importance ratios can evolve across optimizer updates
• additive experimental registry, CLI, and documentation surfaces; existing stable algorithms remain unchanged
• data-integrity checks for rollout token/logprob alignment, finite old logprobs, exact n_samples, and reward/group cardinality

Related issue

Fixes #485

A follow-up numerical clarification is documented in this issue comment: AReno's current on-policy surrogate has forward value exp(new - new.detach()) = 1, which leaves Clip-Higher numerically inactive. This PR therefore keeps the real rollout-policy ratio scoped to experimental DAPO while leaving stable GRPO/GSPO behavior unchanged.

Type of change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

How was it tested?

Local CPU, static, and documentation validation:

• python -m pytest tests/test_dapo_cpu.py tests/test_train_cli_config_cpu.py -q — 129 passed
• python -m pytest tests/*_cpu.py -q — 492 passed, 11 skipped
• ARENO_LOG_COMPLETIONS=1 python -m pytest tests/test_dapo_cpu.py::DAPODynamicSamplingTest::test_real_multi_prompt_generation_filters_and_truncates_whole_groups -q — passed
• ruff check areno/experimental/dapo/trainer.py tests/test_dapo_cpu.py — passed
• python -m py_compile areno/experimental/dapo/trainer.py tests/test_dapo_cpu.py — passed
• python -m sphinx -W --keep-going -b html -D html_copy_source=0 docs <output> — 39 documents built without warnings
• git diff --check — passed

Algorithm and data correctness coverage includes:

• hand-derived Clip-Higher loss, clip fractions, and analytical gradients across all clipped/unclipped sign branches
• central finite-difference gradient agreement
• exact rollout → reward record → overlong shaping → group advantage → TrainSequence → packed action tensor field alignment
• packed/padded objective parity
• real split_data_pack_by_dp loss/gradient equivalence for an uneven 3-to-2 split
• combined gradient-accumulation and DP equivalence, including a microbatch smaller than the DP world that is replicated by the runtime
• real multi-prompt scoring order through filtering, accepted-group selection, and whole-group overflow truncation
• exact n_samples enforcement before prompt-group scoring
• mutation checks confirming the DP scale, prompt index, hand-derived Clip-Higher, and packed-data alignment tests fail when their respective production paths are intentionally corrupted

Real-model validation

Environment: Python 3.12.13, PyTorch 2.11.0+cu128, CUDA 12.8, one NVIDIA GeForce RTX 5090. Source archive SHA-256: f0f0278f8b83c14a545a58139a1066a4284d22db7d50f6fa873ce72ae7d8faf8 (ead49f4).

The following bounded Qwen3-0.6B run deliberately makes the first prompt group uninformative and the second informative, so it exercises filter → retry → train rather than only a direct training smoke:

{"prompt":"Continue this short sequence: 1, 2,"}
{"prompt":"Name one primary color:"}

# /tmp/areno-dapo-dynamic-reward.py
def reward_fn(record):
    prompt = record.source_record.get("prompt", "")
    if prompt.startswith("Continue this short sequence"):
        return 1.0
    return float(record.metadata["sample_index"])

python -m areno.cli.main train \
  --algo dapo \
  --ckpt Qwen/Qwen3-0.6B \
  --model-hub modelscope \
  --dataset-path /tmp/areno-dapo-validation.jsonl \
  --reward-fn-path /tmp/areno-dapo-dynamic-reward.py \
  --epochs 1 \
  --max-steps 1 \
  --world-size 1 \
  --tp-size 1 \
  --train-devices 0 \
  --batch-size 1 \
  --dapo-gen-batch-size 1 \
  --dapo-max-num-gen-batches 2 \
  --n-samples 2 \
  --mini-bs 1 \
  --gradient-accumulation-steps 1 \
  --max-prompt-tokens 64 \
  --max-new-tokens 4 \
  --temperature 1.0 \
  --attn-backend native \
  --eager-decode \
  --no-activation-checkpointing

Observed result (exit code 0):

• after candidate 1: gen_batches=1 generated_groups=1 qualified_groups=0 filtered_groups=1
• after candidate 2: gen_batches=2 generated_groups=2 qualified_groups=1 filtered_groups=1
• final sampling metrics: dapo_sampling_efficiency=0.5, dapo_discarded_qualified_groups=0
• finite training metrics: loss=2.3841858e-07, grad_norm=0.7063947
• real ratio metrics: ratio_mean=1.0000181, ratio_std=2.9320003e-05
• logged policy values: rollout_logprobs_mean=-0.00108719, train_logprobs_mean=-0.00106907

The real-model run is a bounded integration test, not a convergence or benchmark reproduction. I did not run a paper-scale AIME experiment or a multi-GPU end-to-end training job; distributed response-token normalization is covered by hand-derived CPU references and tests that invoke the runtime's real DP split function.

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description (if any).
• Existing tests pass (pytest tests/*_cpu.py).
• New behavior is covered by tests.
• Described the test commands run and hardware scope.
• Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).