Some questions about the design of the loss function in the article

[TJ12342]

Dear author, hello!

Seeing this loss function, I have some questions. S uses the value at time t0, and the value of Q at time t is copied from time t0. So the part of the time that actually has an effect is v_t_i. So, is the purpose of introducing the "track" just to weight the loss?

[SMSD75]

Hi,

Our loss function forces the similarity of the student features at time step t0 with the teacher features at further time steps for corresponding patches along each motion trajectory. In other words, what you say is true if we suppose that the weight of loss for patches in the same motion trajectory is 1 and the rest are 0. Please let me know if you have any more questions.

Best regards.

[TJ12342]

Dear Author,

Thank you very much for your reply. Regarding the issue of the marks in the article

From Sec. 3.2 :

• Patch features: $Z^{stu} \in \mathbb{R}^{n_s \times d \times T}$
• OT plans / assignment matrices: $M^{stu} \in \mathbb{R}^{n_s \times K}$, $M^{teach} \in \mathbb{R}^{n \times K}$

Eqs. (6)(7) and especially Eq. (9) involve terms like $M^{stu*}\cdot z^{stu,i}_{t_0}$ (and the text says “softmax on similarity between $z$ and cluster prototypes $M^{stu*}$”).

I'm not quite sure how to multiply them together.

Looking forward to your reply!

[SMSD75]

Right, I think I understand your question now. We assign the student's unmasked patches to the prototypes and leave the masked ones as 0. So they are not involved in the final loss function, yet the matrix shape stays similar to the teacher.