RMBench: Memory-Dependent Manipulation Benchmark

RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design. Under Review, PDF | arXiv | Website | Join our Community 🔥

Tianxing Chen*, Yuran Wang*, Mingleyang Li*, Yan Qin*, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang, Renjing Xu, Ruihai Wu, Yao Mu, Yaodong Yang, Hao Dong†, Ping Luo†

📰 Updates

2026.07.14 — Since the previously trained Mem-0 checkpoints were not backed up before our development machine was recycled, we have re-organized the training and now publicly release the retrained model weights:

• M(1) tasks: due to limited computational resources, all M1 tasks were trained jointly into a single multi-task m1_mix model. The complete model, the processed m1_mix dataset, training/inference configs, and all evaluation logs and videos are available at qiuly/Mem-0-m1mix-RMBench and qiuly/Mem-0-m1mix-dataset-RMBench.
• M(n) tasks: per-task execution-module checkpoints for battery_try, blocks_ranking_try, cover_blocks and press_button, together with per-task normalization stats and evaluation results, are available at qiuly/Mem-0-mn-RMBench.

Detailed evaluation results can be found in the Hugging Face model cards above.

🧑🏻‍💻 RMBench Usage

This project is built upon RoboTwin 2.0. Policy evaluation can go through the original policy/ adapters or through XPolicyLab.

1. Installation

First, prepare a conda environment.

conda create -n RMBench python=3.10 -y
conda activate RMBench

RMBench Repo: https://github.com/RoboTwin-Platform/RMBench

git clone --recurse-submodules https://github.com/RoboTwin-Platform/RMBench.git
cd RMBench

For an existing checkout, initialize the pinned XPolicyLab submodule:

git submodule update --init --recursive XPolicyLab

Then, run scripts/_install.sh to install basic conda envs, CuRobo, and XPolicyLab:

bash scripts/_install.sh

To refresh the XPolicyLab pin later:

bash scripts/update_xpolicylab.sh

2. Download Assets

To download the assets, run the following command. If you encounter any rate-limit issues, please log in to your Hugging Face account by running huggingface-cli login:

bash scripts/_download_assets.sh

3. Download Data

Please run the following command to download all data.

bash scripts/_download_data.sh

If you need to collect the data (we actually recommend downloading it directly)

In RMBench, we always use demo_clean setting.

Running the following command will first search for a random seed for the target collection quantity, and then replay the seed to collect data.

Please strictly follow our tutorial in RoboTwin 2.0 Doc - Collect Data.

bash collect_data.sh ${task_name} ${task_config} ${gpu_id}
# Example: bash collect_data.sh cover_blocks demo_clean 0

New collections write XPolicyLab-format HDF5 under data/<task_config>/<task_name>/<embodiment>/:

data/demo_clean/cover_blocks/aloha_agilex/data/episode_0000000.hdf5

Always decode through decode_image_bit, and always encode through encode_image_bit. Camera frames are stored as JPEG bits in two formats (legacy channel-reversed JPEGs and standard XPL-RGB1-marked JPEGs). Both decode to RGB. A local copy is in data/decode_image_bit.py; prefer XPolicyLab.utils.process_data when that package is installed:

from XPolicyLab.utils.process_data import decode_image_bit, encode_image_bit
rgb = decode_image_bit(image_bits)  # RGB for every data version

Do not add cv2.cvtColor(..., COLOR_BGR2RGB) after decode_image_bit. At eval time the policy server already receives decoded RGB arrays.

4. Run Policies

Two evaluation paths are available.

Original adapters (still in policy/):

python scripts/eval_policy.py --config policy/<Name>/deploy_policy.yml ...
# or
bash policy/<Name>/eval.sh ...

1. Mem-0 (ours): See Mem-0 Document
2. DP: See DP Document
3. ACT: See ACT Document
4. Pi 0.5: See Pi 0.5 Document
5. X-VLA: See X-VLA Document
6. Other Policies (Pi0, RDT, etc): See Document and See Folder
7. Configure your policy: See Tutorial Here

XPolicyLab adapters (recommended for new work):

bash XPolicyLab/policy/Mem_0/eval.sh RMBench <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <env_gpu_id> <policy_env> <eval_env>

# or schedule the RMBench task list
bash scripts/eval_policy.sh multitask \
  --config env_cfg/eval/all_tasks.yml \
  --policy-name Mem_0 \
  --env-cfg-type arx_x5 \
  --action-type joint \
  --ckpt-name <ckpt_name> \
  --eval-env-conda-env RMBench

See XPolicyLab/policy/Mem_0/ for the Mem-0 adapter. Task configs live in env_cfg/task_config/.

👍 Citations

If you find our work useful, please consider citing:

@article{chen2026rmbench,
  title={RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design},
  author={Chen, Tianxing and Wang, Yuran and Li, Mingleyang and Qin, Yan and Shi, Hao and Li, Zixuan and Hu, Yifan and Zhang, Yingsheng and Wang, Kaixuan and Chen, Yue and others},
  journal={arXiv preprint arXiv:2603.01229},
  year={2026}
}

🏷️ License

This repository is released under the MIT license. See LICENSE for additional details.