Ask about (i) positional embedding; (ii) [EOS] truncation during training.

[keyu-wang-2002]

Hi! I’ve been following LightThinker and making some modifications. I have two questions:

(i) I noticed that you seem to have implemented rolling_rope, but the default is "false". May I ask whether the "true" track is already implemented (considering whether it corresponds only to inference or to both training and inference)? If it is implemented, did your results show that "false" performed better than "true"? If not implemented yet, I plan to implement it myself and see.

(ii) For bs17k, more than half of the data has a length exceeding 4096. The approach in the code is to truncate directly. it seems truncating [EOS] as well. Could this introduce bias in how vanilla and LightThinker handle the closing after solving the questions? I’m also considering other suitable training datasets, etc.

BTW, I wanted to share that in LightThinker (token), using step6_comp2, I found through my experiments that step6 can actually be much much larger, and the results can even be better. e.g., based on my training and testing results, step32_comp2 outperformed step6_comp.

[MikeDean2367]

Hi there,

Thank you so much for your interest in LightThinker and for sharing your findings!

Here are the answers to your questions:

• Regarding rolling_rope: Yes, the true setting is implemented, but only for the inference stage. As you noticed, the default is false. We did experiment with this and found that false actually yielded better performance than true in our tests.
• Regarding bs17k truncation and [EOS]: You are correct that for sequences exceeding 4096 tokens, we truncate directly, which means the [EOS] token is also dropped. We do this intentionally. If we were to forcibly append an [EOS] token at the cut-off point, it would distort the model's training distribution. Essentially, we would be forcing $P(\text{EOS}|x_{1 \dots 4095})$ instead of the true $P(x_{4096}|x_{1 \dots 4095})$. This would inadvertently teach the model that it must stop generating at exactly 4096 tokens, even when it still has more to output. By omitting it, we rely on the model's inherent generalization capabilities to handle the closing properly. Another viable workaround for this would simply be to increase the maximum training length.
• Regarding your step32_comp2 finding: This is a fantastic discovery! We didn't test this specific hyperparameter configuration during our experiments, as it pushes the compression rate to nearly 94%. If this setup outperforms step6_comp without an increase in the overall inference length (meaning the number of model compressions hasn't significantly grown), that is a truly exciting result. Congratulations on finding this out!

We are really looking forward to seeing your new work. If you run into anything else or have more questions, please feel free to let me know! :)

[keyu-wang-2002]

thanks a lot! my question are all solved!

• Regarding bs17k truncation and [EOS], your response is consistent with my observations. I make a subset of bs17k by dropping out the samples longer than 8k, and increase the training context to 8k. The results show no significant improvements.

• Regarding increasing steps, my assumption is that the lower compression frequency and later compression help preserve intermediate reasoning information more effectively, reducing information loss and reasoning drift in long-chain reasoning. As a result, the gain on MMLU is limited, while the improvement on GSM8K is much more significant.

btw, it's normal that my results show differences. During training, I split the train data and eval data at a ratio of 9:1. I found that with the original paper's settings, overfitting occurred in less than half of whole process. Therefore, I set the learning rate to 1e-5 and selected the ckpt with the smallest eval loss for testing. Anyway, on the test task, there was no significant difference between the selected ckpt and the final overtrained ckpt.

[MikeDean2367]

OK! Thanks for your insightful findings! We are also really looking forward to seeing your new work. If you run into anything else or have more questions, please feel free to let me know! :)