Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices
Chenfeng Gao*, Taeyoung Yeon*, Sungheon Park, Vasco Xu, Anish Prabhu, Mar Gonzalez-Franco, Karan Ahuja (* equal contribution)
IEEE ISMAR 2026 (IEEE Transactions on Visualization and Computer Graphics)
Paper (arXiv) · Video · Project Page · SPICE Lab
FlowAvatar is a unified system for real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression and eye gaze tracking from sparse egocentric inputs (head and hands) on consumer XR devices (Meta Quest Pro). A recurrent pose model predicts the SMPL-X body from the headset and hand tracking, either on a PC that streams the pose to the headset or directly on the headset.
| Directory | Content | Start here |
|---|---|---|
unity/ |
Unity client for Meta Quest: avatar, tracking inputs, streaming and on-device (Unity Sentis) modes | unity/README.md |
streaming/ |
Python server for the streaming mode: receives the Quest tracking over TCP, streams the predicted pose back | Streaming demo |
avatar-pipeline/ |
Personalized avatar from a phone video: UV texture and body shape for the Unity avatar | avatar-pipeline/README.md |
training/ |
Training and evaluation on AMASS, reproducing the paper's benchmark tables | training/README.md |
checkpoints/ |
Paper (benchmark) and demo (deployment) checkpoints | checkpoints/README.md |
flowavatar/ |
Shared Python package: network, checkpoint loading, rotation and kinematics utilities | |
configs/, docs/ |
Streaming configuration; Unity ⇄ Python protocol |
The pose model runs on a PC; the Quest sends its tracking and receives the pose.
All commands run from the repository root.
-
Install (Python ≥ 3.10, git):
pip install -r requirements.txt -
SMPL-X model: download the SMPL-X model in NPZ format (v1.1 or v1.0) from the SMPL-X download page (free registration) and place
SMPLX_NEUTRAL.npz(in the zip'smodels/smplx/) atbody_models/smplx/SMPLX_NEUTRAL.npz(or setSMPLX_MODEL_PATH). -
Run the server (the checkpoints are in the repository):
python -m streaming.live_demo --model gru
It waits on TCP ports 8888 (tracking in) and 8889 (pose out), exits if no client connects within 30 seconds, and serves one session (restart it for the next). Ports, model choice and input-feature flags are set in configs/streaming.yaml (
--configto use another file). -
Start the Unity client on the headset, pointed at this machine's IP (unity/README.md); allow Python through the firewall.
Models: gru (default, best accuracy, 60 Hz input) and lstm (the on-device model,
30 Hz input: set the Unity client's send rate to 30); both take a 40-frame window.
The LSTM (30 Hz input, 40-frame window) runs on the headset CPU with Unity Sentis (Sentis has no GRU operator); no PC is needed. The Unity client ships it as ONNX; see unity/README.md. To export an LSTM checkpoint:
pip install onnx onnxruntime
python -m streaming.export_onnx --model_path checkpoints/LSTM/baseline.ptthen copy converted_onnx/flowavatar_sentis_final.onnx over
unity/Assets/Scripts/FlowAvatar/OnDevice/Models/flowavatar_lstm.onnx (keep its .meta).
avatar-pipeline/ turns a short phone video into a texture in the Unity avatar's UV layout and its ten SMPL-X shape parameters. It runs offline on a Linux machine with an NVIDIA GPU.
python -m training.prepare_data --amass_dir data/AMASS --save_dir data/amass_smplh_60fps --fps 60
python -m training.train --config training/configs/gru_smplh.yaml
python -m training.evaluate --config training/configs/gru_smplh.yaml \
--checkpoint checkpoints/benchmark/gru_smplh.ptgru_smplh.yaml / lstm_smplh.yaml train the benchmark models on SMPL+H (comparable
with prior work and the paper tables), training/configs/ablations/ the loss ablations,
and gru_smplx.yaml / lstm_smplx.yaml the SMPL-X models of the demos. The released
paper checkpoints reproduce Tables 2 and 3; see training/README.md
(body models and AMASS download).
@article{gao2026flowavatar,
title={FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices},
author={Gao, Chenfeng and Yeon, Taeyoung and Park, Sungheon and Xu, Vasco and Prabhu, Anish and Gonzalez-Franco, Mar and Ahuja, Karan},
journal={arXiv preprint arXiv:2610.16277},
year={2026}
}This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. For commercial use, a separate commercial license is required. Please contact kahuja@northwestern.edu at Northwestern University for licensing inquiries.
Third-party components keep their own licenses (THIRD_PARTY_NOTICES.md);
in particular the files in avatar-pipeline/patches/ modify third-party projects and are provided
under the licenses of those projects. The SMPL-X, SMPL+H and SMPL body models, the AMASS dataset
and the human_body_prior package are licensed separately by MPI-IS for non-commercial research
and are not covered by this license.