Motivation
Reward hacking / reward-model overoptimization is a common failure mode in RL post-training, and there's currently no standard TorchRL transform for monitoring it during a run.
Solution
An optional transform that tracks signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — and logs them alongside existing metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors already (currently for GRPO/TRL-style training).
This comes from work on two reward-hacking-detection benchmarks (RHOB, Flight Recorder). One relevant finding: these signals reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking") — worth knowing before assuming such a transform gives early warning rather than post-hoc detection.
Proposing before attempting a PR — happy to share methodology/code if there's interest.
Motivation
Reward hacking / reward-model overoptimization is a common failure mode in RL post-training, and there's currently no standard TorchRL transform for monitoring it during a run.
Solution
An optional transform that tracks signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — and logs them alongside existing metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors already (currently for GRPO/TRL-style training).
This comes from work on two reward-hacking-detection benchmarks (RHOB, Flight Recorder). One relevant finding: these signals reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking") — worth knowing before assuming such a transform gives early warning rather than post-hoc detection.
Proposing before attempting a PR — happy to share methodology/code if there's interest.