Skip to content

Fix PMPP v2 sort reward hack and harden correctness checks - #173

Merged
msaroufim merged 2 commits into
mainfrom
fix-pmpp-v2-checkers
Nov 5, 2026
Merged

msaroufim merged 2 commits into
mainfrom
fix-pmpp-v2-checkers

Conversation

@msaroufim

@msaroufim msaroufim commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

The sort checker accepted invented increasing values because input magnitude depended on the seed and correctness used relative tolerance. At seed 10^12, the old 4096-element generator collapses to one float32 value, while an unrelated sequence spanning ±4,000,000 passes its checker.

Require exact sorted values and float16 addition; generate sort inputs independently of seed magnitude with varied distributions. Reject malformed shapes, dtypes, devices and non-finite results across all eight PMPP v2 tasks. Fix histogram scalar broadcasting and replace prefix-sum tolerance that grew to ~16% with fixed relative tolerance and an input-energy roundoff allowance. Calibrate the scan allowance against legitimate PyTorch scans through the largest benchmark. Bound mismatch diagnostics for large incorrect outputs and preserve reference determinism settings.

Validation: 118 CPU regression tests, Ruff, diff checks and GitHub CI pass. All 32 GPU positive controls pass: every PMPP v2 task on A100, B200, H100 and L4, including every benchmark shape. Fabricated sort, histogram and scan negative controls are rejected. Private historical revalidation also applies the exact sort checker to the original generator, catching input-detecting approximations that fall back to real sorting on the repaired generator. Contestant source is excluded from this PR; leaderboard remediation is tracked separately. All eight production task definitions were synced and their embedded files, correctness cases and benchmarks verified against the merged suite.

@msaroufim
msaroufim marked this pull request as ready for review November 5, 2026 09:43
@msaroufim
msaroufim merged commit 3dda50a into main Nov 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant