Repository navigation
Fix PMPP v2 sort reward hack and harden correctness checks - #173
Merged
Merged
Conversation
msaroufim
marked this pull request as ready for review
November 5, 2026 09:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sort checker accepted invented increasing values because input magnitude depended on the seed and correctness used relative tolerance. At seed 10^12, the old 4096-element generator collapses to one float32 value, while an unrelated sequence spanning ±4,000,000 passes its checker.
Require exact sorted values and float16 addition; generate sort inputs independently of seed magnitude with varied distributions. Reject malformed shapes, dtypes, devices and non-finite results across all eight PMPP v2 tasks. Fix histogram scalar broadcasting and replace prefix-sum tolerance that grew to ~16% with fixed relative tolerance and an input-energy roundoff allowance. Calibrate the scan allowance against legitimate PyTorch scans through the largest benchmark. Bound mismatch diagnostics for large incorrect outputs and preserve reference determinism settings.
Validation: 118 CPU regression tests, Ruff, diff checks and GitHub CI pass. All 32 GPU positive controls pass: every PMPP v2 task on A100, B200, H100 and L4, including every benchmark shape. Fabricated sort, histogram and scan negative controls are rejected. Private historical revalidation also applies the exact sort checker to the original generator, catching input-detecting approximations that fall back to real sorting on the repaired generator. Contestant source is excluded from this PR; leaderboard remediation is tracked separately. All eight production task definitions were synced and their embedded files, correctness cases and benchmarks verified against the merged suite.