Clarification on how OCR annotations are used during training

[JerryPW]

Hi, thank you for releasing this excellent work.

While reading the paper, there seems to be one point that is still unclear: how the OCR annotations are actually incorporated into training.

From the paper, the following part is understood:

PaddleOCR is applied to images from OBELICS and Zero250M
the recognized text is tokenized
100 fine-grained tags are constructed for each image
OCR data is introduced in Stage 2 together with video supervision

However, the paper does not seem to explicitly describe how these OCR-derived tags are optimized in the training objective.

[anxiangsir]

Thank you very much for your attention and for raising this question.

We have prepared a brief principle-level explanation to answer it. Please see the following link:

https://evolvinglmms-lab.github.io/OneVision-Encoder/issues/issue-105.html

If you have any further questions, please feel free to leave them below, and we will continue to respond and clarify.

[JerryPW]

Hi, thank you very much for the detailed reply. It makes sense now.

The OCR part is much clearer after your explanation. Thanks again for the clarification and for the helpful explainer page.