Fix eos issues in sglang / vllm samplers during on-policy rollout

[yixinw]

The existing implementation that leverages sglang/vllm samplers during rollout has some issues with the eos (e.g., <|im_end|> in Qwen models) which can make it problematic. This PR addresses this by removing the unnecessary (sometimes wrong) eos tokens in the input, to make RL more stable and effective.

On one benchmark that we experimented, we found this change has lead to significance in terms of the learning efficiency and quality.

Checklist

• I have added all the necessary unit tests for my change.
• I have verified that my change does not break existing code and all unit tests pass.
• I have added all appropriate doc-strings/documentation.
• My PR is based on the latest changes of the main branch (if unsure, rebase the code).
• I have signed the Contributor License Agreement.
• I have followed Contribution Guidelines.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[tianshub]

ah wow! thanks for the fix Bethany, this is huge improvement.