Unable to reproduce nano droid policy post-training

[Randdl]

I followed your post-training configs and used your released dataset, but I can get 0.1783 SR.
I noticed that there is a f_max=0.4 that restricts the lr to 8e-5 instead of 2e-4, and the lr scheduler cycle is 100k instead of 10k.
Are these parameters intended?
Did you test your released toml configs?
(model.rectified_flow_training_config and dataloader_train.dataloader are not registered in SFTExperimentConfig, so it contradicts the extra="forbid" and raises errors.)

[fwd4]

f_max=0.4 that restricts the lr to 8e-5 instead of 2e-4, and the lr scheduler cycle is 100k instead of 10k

Both are intended, and those are used our Nano model training. @ychao-nvidia

model.rectified_flow_training_config and dataloader_train.dataloader are not registered in SFTExperimentConfig, so it contradicts the extra="forbid" and raises errors.

This is a real issue, resolved in PR139

[Randdl]

Thanks a lot! Can you also verify the configs with your internal post-training configs? Are there any silently missing values or values overwritten by defaults? The two unregistered values suggest there might be other misalignments.
Also, I think the robolab eval with your checkpoint silently falls back to plain text prompt; however, the post-training config you provided uses JSON format. Could you confirm if this is the case or if I used it wrong? This is the only reason I can find that might explain my bad reproduction result right now.

[fwd4]

Thanks a lot! Can you also verify the configs with your internal post-training configs? Are there any silently missing values or values overwritten by defaults?

Thank you for bring this up. Everything else in the config is a match. One thing you might want to handle carefully is the range filter, make sure you are following the steps in the doc.

Could you confirm if this is the case or if I used it wrong?

Yes, we need post-trained Nano and Edge policy model both having json prompt. But normally it would be sr drop of less than 10pp even if you have not configured json/dense format properly. @ychao-nvidia can you confirm on this on the server script?

[Randdl]

I used the range filter, so I don't think this is my root cause.

What I found is that your Nano checkpoint uses plain text, but edge uses json format. During evaluation on Robolab (not training), when I force nano to use json format, the sr rate actually drops from 0.3975 to 0.2050.

[fwd4]

@Randdl noted. We should take a look on the released checkpoint. Can you kindly evaluate your trained ckpt with forced json prompt?

[ychao-nvidia]

@Randdl

What I found is that your Nano checkpoint uses plain text, but edge uses json format.

This is intended for our released checkpoints. The released Nano-Policy-DROID was trained from an earlier experiment that used dense prompt. We switched all our training to json prompt later.

[Randdl]

I trained my ckpt with json prompt. When I evaluate it with dense prompt, the SR drops. I think other than these, I am following the post-training configs faithfully.

[Randdl]

Two more distinctions I found.

The nano droid checkpoint's reasoner weights deviate from the nano checkpoint, and is identical to the vanilla Qwen weights.

The other action heads (untrained) in nano droid checkpoints are identical to the nano checkpoint. However, if following the post-training config, all the action heads will be re-initialized. It suggests the nano droid checkpoint did not re-initialize the action heads.

[Randdl]

keephead and vanilla qwen did not help. Do you have any updates?
Your released checkpoint apparently does not align with the config you released. Can you provide the real config you used for the checkpoint or provide a checkpoint that you reproduce using the config?