Questions about QAT Data Composition

[xyuany]

Hi, thanks for sharing this impressive work.

I have a few questions regarding the 2-bit QAT data setup and would appreciate some clarification:

1. SFT data usage in QAT

• Is the 2-bit QAT training conducted entirely on SFT data?

• When you mention “50% of SFT corpus,” does this mean 50% of the full SFT corpus?

• Could you clarify the scale of the full SFT corpus (e.g., total tokens or samples)?

2. Relationship with the 89B curated dataset

• You mentioned curating an optimized dataset of 89B tokens — is this dataset derived from the full SFT corpus, or is it a separate hybrid dataset (e.g., including pretraining or other sources)?
• Is this 89B dataset actually used during QAT, or only for earlier stages?

3. Training data scale

• The paper mentions using only 10% of the data compared to BitNet-2B. Does the 2-bit QAT training use approximately ~400B tokens in total?

Thanks in advance for the clarification!

[WOODchen7]

1.1: Yes. In the future, we will employ additional training techniques to further enhance model accuracy.
1.2: Yes.
1.3: Since the base model has not been publicly released, we are likewise unable to provide any information regarding it.
2.1: Dataset derived from the full SFT corpus.
2.2: Dataset actually used during QAT.
3: The training data volume for BitNet-2B mentioned in the paper refers to the amount of data used for training from scratch; therefore, we utilized approximately 400 billion tokens.

[xyuany]

Thank you so much for the clarification! This is very helpful!

I still have a few follow-up questions to better understand the data setup:

1. Relationship between SFT corpus and the 89B dataset. Since the 89B dataset is derived from the full SFT corpus and used in QAT, does the “50% of SFT corpus” correspond to this 89B dataset?
2. QAT data usage. Was the QAT training conducted using only the optimized 89B dataset?
3. Relationship between SFT corpus and the 400B data comparison. When you mentioned 400B tokens, did you mean that the 89B dataset was fed multiple times during the QAT phase, totaling approximately 400B?

Thanks again!