Synthetic Data Generation Pipeline

[gemblerz]

Goal: Create a pipeline that takes a few episodes of OpenArm manipulation and generates over a few K episodes.

Main Tasks:

• Obtain Doga's OpenARM robot manipulation dataset as human-guided episode
• Try tools like DexMimicGen and design the pipeline
• Generate few K episodes for a couple of robot tasks (e.g., pick-and-place cubic and open closet door)
• Replay episodes in Isaac Sim

Optional Tasks:

• Augment other sensor data (e.g., tactile) in generated synthetic dataset
• Analyze # of episodes VS training accuracy (if we can finetune a VLA model with this dataset)

Success Criteria:

• End-to-end workflow document showing an example workflow from putting episodes for a task and getting x1000 dataset
• Getting thousands of episode for the two robot tasks

[gemblerz]

Things to consider,

• How much can tele-operation data help in rsl_rl training
• How much accuracy can we gain in AI training from the synthetic data generation
• Compare how well is AI performance against tele-operation data
• (Robustness) Do we need separated tele-operation data for any changes in the environment? For example, if we put an obstacle on the table for pick-and-place task, can AI still do that without further training? What policies work the best in this case?