L-CAMP: Language-Conditioned Axis and Motion Prediction for Articulated Object Manipulation

Ikechukwu Daniel Adebi, Mitchell Pryor

Setup

1. Create the conda/micromamba environment from environment_lcamp.yml:

   micromamba create -f environment_lcamp.yml   # or: conda env create -f environment_lcamp.yml
   micromamba activate lcamp

2. Build the LCAMP dataset (only needed once per dataset version), see process_lcamp_dataset.py / process_lcamp_dataset.slurm. This produces a dataset directory containing train_dataset.json, test_dataset.json, and intrinsics.npy, referenced below as --data_root/--data_dir.

All commands below (train.py, eval.py) assume the lcamp environment is active.

Download Model

A pretrained checkpoint is available on HuggingFace: danieladebi/lcamp-model. Download it into checkpoints/ to skip training and go straight to evaluation.

Training

Training is done with train.py via torchrun (DDP). The dataset must already be built (see process_lcamp_dataset.py / process_lcamp_dataset.slurm) as a directory containing train_dataset.json, test_dataset.json, and intrinsics.npy.

Single-GPU / local run:

torchrun --nproc_per_node=1 train.py \
    --save_path checkpoints/my_lcamp_model \
    --data_root ./lcamp_dataset_limited_dataset \
    --model_type resnet \
    --epochs 50 \
    --batch_size 64 \
    --lr 1e-3 \
    --min_lr 1e-5 \
    --use_camera_frame \
    --use_depth \
    --use_text_instructions \
    --emphasize_part_mask \
    --lambda_axis 8 \
    --lambda_anchor 2 \
    --lambda_joint 1

Multi-GPU (cluster) example:

torchrun --nproc_per_node=8 train.py \
    --save_path "$SCRATCH/lcamp_model_RGB_log_norm_DEPTH_NO_AXIS_SYMMETRY" \
    --data_root "$SCRATCH/lcamp_dataset_off_target_depth_LANGUAGE_EXPLICIT_OBJ_CMDS" \
    --model_type resnet \
    --epochs 50 \
    --warmup_epochs 5 \
    --batch_size 256 \
    --lr 1e-3 \
    --min_lr 1e-5 \
    --use_camera_frame \
    --use_depth \
    --use_text_instructions \
    --emphasize_part_mask \
    --randomize_background \
    --lambda_axis 8 \
    --lambda_anchor 4 \
    --lambda_joint 1 \
    --lambda_plucker 0 \
    --log_dir "$SCRATCH/train_logs" \
    --cluster_loc "stampede3"

See train.slurm / train_h100.slurm for full Slurm submission scripts (note: those scripts currently hold older --lr/--lambda_anchor/--lambda_joint values, update them if you want the Slurm jobs to match the above).

Notable flags (train.py --help for the complete list):

• --model_type: resnet (default LCAMP backbone) or dino.
• --use_text_instructions: enables the language-conditioned model (LCAMPModelResnetLanguage); required if you plan to run the lcamp backend with instructions in lcamp_service_node.py.
• --use_depth: adds depth as an input channel.
• --bbox_loc: predict axis location as (u,v,z) decoded via the part bounding box instead of raw XYZ, must match --bbox_loc in eval.py and use_bbox_loc in lcamp_service_node.py/lcamp.launch.py for the same checkpoint.
• --resume / --resume_from <path>: resume from checkpoint_best.pt in --save_path, or from an explicit checkpoint path.
• --save_freq, --eval_freq: checkpoint/eval cadence in epochs.

Checkpoints are written to --save_path as checkpoint_best.pt, checkpoint_last.pt, and periodic checkpoint_{epoch}.pt files.

Evaluation

Evaluation is done with eval.py (single process, no DDP):

python eval.py \
    --model_path checkpoints/my_lcamp_model/checkpoint_best.pt \
    --data_dir ./lcamp_dataset_limited_dataset \
    --split test \
    --model_type resnet \
    --use_camera_frame \
    --use_depth \
    --use_text_instructions \
    --visualize

This must use the same architecture flags used at training time (--model_type, --use_depth, --use_text_instructions, --exclude_object_mask, --emphasize_part_mask/--emphasize_object_mask, --bbox_loc, etc.), mismatches will load state dict weights incorrectly or error, and --use_camera_frame/--cluster_loc must match how the dataset was built.

Notable flags:

• --split: train or test.
• --visualize: saves per-sample prediction images.
• --random_preds: baseline using random axis/location predictions.
• --free_motion_query_prob: fraction of eval samples using a free-motion (zero-vector) query instead of the GT axis, to measure reliance on visual cues vs. the GT axis itself.

Results (per-object/per-category success rates, axis/location error, joint type accuracy) are written to results/<model_name>/, where <model_name> is derived from --model_path.

Once you have a checkpoint you're happy with, drop it into checkpoints/ and point the model_name launch argument in launch/lcamp.launch.py at it (see the launch file for details on use_bbox_loc and other runtime flags).

Deploying on a Robot

Running predictions on a real Boston Dynamics Spot (via the ROS2 service node in scripts/lcamp_service_node.py) is outside the scope of this README, see ros2_setup/ROS2_SETUP.md for that setup.