feat(experimental): add Dr. GRPO

[CharlesXu-HQ]

What does this PR do?

Adds Dr. GRPO as an opt-in experimental policy-training algorithm, following Understanding R1-Zero-Like Training: A Critical Perspective.

The implementation keeps AReno's existing rollout and policy-only training flow while applying the two Dr. GRPO objective changes:

• computes group advantages as reward - group_mean, without standard-deviation normalization;
• normalizes the token policy objective by num_sequences * max_completion_length instead of the number of sampled response tokens.

The change also:

• registers drgrpo lazily under areno.experimental and keeps it out of the stable algorithm listing;
• specializes only the group-advantage hook while reusing PolicyOnlyTrainer;
• binds grpo_clip_eps and max_new_tokens to the experimental loss;
• preserves the global fixed-sequence denominator across packed/padded layouts, uneven microbatches, gradient accumulation, and data-parallel sharding;
• rejects agentic rollouts explicitly because this implementation assumes fixed prompt groups;
• keeps AReno's current GRPO ratio semantics unchanged. Stored rollout logprobs remain diagnostic in this PR; changing the old-policy ratio behavior is intentionally out of scope.

Existing GRPO behavior and defaults are unchanged. Users opt in with --algo drgrpo.

Related issue

Fixes #483

Type of change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

How was it tested?

Local static and CPU validation on the final rebased commit:

python -m pytest tests/ -k cpu -q

Result: 474 passed, 11 skipped, with 12 existing FastAPI deprecation warnings.

ruff check .
ruff format --check .
python -m compileall -q areno
git diff --check upstream/main...HEAD

Result: all checks passed; 267 files were already formatted.

The new CPU coverage includes:

• exact centered advantages and fixed-normalizer loss/gradient values;
• prompt and explicit loss-mask gradient blocking;
• finite zero loss/gradients for fully masked batches;
• packed/padded loss, statistic, and gradient equivalence;
• full-batch versus gradient-accumulation equivalence;
• uneven data-parallel shard equivalence;
• combined uneven microbatch, accumulation, and DP scaling equivalence;
• registry metadata, experimental visibility, loss binding, trainer dispatch, and agentic rejection.

Target-side validation used the final commit with Qwen3-0.6B, PyTorch 2.9.1+cu130, native attention, TP 1, world size 1, four samples per prompt, mini-batch size 2, gradient accumulation 2, and 64 generated tokens:

• tests/test_drgrpo_cpu.py: 21 passed;
• Dr. GRPO: 5/5 training steps completed with finite statistics, reward variation in every group, and positive gradient norms in every step;
• Dr. GRPO gradient norms: 2.4697, 6.2598, 2.8107, 5.6674, 7.0344;
• existing GRPO baseline: 2/2 training steps completed under the same configuration with finite statistics and positive gradients.

Hardware limitation: GPU validation was single-device only (one NVIDIA RTX 5090). Multi-GPU CUDA execution was not available; DP denominator/scaling behavior is covered by deterministic CPU tests, including uneven real pack sharding.

Checklist

• The PR title summarizes the contribution.
• Linked the related issue in the description.
• Existing tests pass (pytest tests/ -k cpu).
• New behavior is covered by tests.
• Described the test commands run and any hardware limitations.
• Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

[xsuler]

@CharlesXu-HQ Thanks for your contribution, I will give this implementation a full test, after that your pr will be merged.