Add Meta IOBT body tracking: emit BodyMeta from the Quest client

[zerenluo123]

Meta IOBT body tracking: emit BodyMeta from the Quest client

Replaces the "Coming soon ......" placeholder in the Body Tracking panel with a working
implementation: 70 upper-body joints inferred from the headset's own cameras
(Meta Inside-Out Body Tracking), published to PC-Service under a new BodyMeta key.

No extra hardware — no base stations, no wearable trackers.

The Mode dropdown replaces the placeholder text, and the Status line below is a live readout
(refreshed a few times a second) so an operator wearing the headset can confirm data is flowing.

What's in the box

• Mode dropdown: Off / Body (IOBT) / Motion (PICO only). Selecting Body (IOBT)
  starts publishing; Motion is rejected up front with a notice rather than silently producing
  nothing, since PICO's motion-tracker mode has no Quest counterpart.
• BodyMeta wire format — 70 joint poses, per-joint validity flags, plus jointSet /
  fidelity / calib / confidence / isActive / count and the tracking-space origin.
  Specified in Docs/BodyMeta.md.
• BodyTrackingSetup.cs — scripts the five project settings body tracking needs
  (Tools > Body Tracking > Configure Project Settings).
• Live status line in the panel, replacing text that was previously written once at
  dropdown-change time.

Why a new key instead of reusing Body

PICO's Body is a fixed 24-joint layout with velocity and acceleration per joint. Meta IOBT gives
70 (or 84 for FullBody) and reports no rates at all — OVRPlugin.BodyJointLocation carries only
LocationFlags and Pose. Sharing the key would mean either truncating 46 joints or redefining a
format existing PICO consumers already parse, so Quest publishes BodyMeta and old consumers are
unaffected. Consumers read the layout from jointSet rather than assuming one.

⚠️ After cloning, run the setup once

Tools > Body Tracking > Configure Project Settings

Or headless:

Unity -batchmode -projectPath <proj> -buildTarget Android \
      -executeMethod BodyTrackingSetup.Configure -quit

This exists because all five settings fail silently when wrong — the app runs, every Inspector
checkbox looks right, and no data arrives with nothing in logcat to say why:

Setting	Location	Required
bodyTrackingSupport	OculusProjectConfig	1
bodyTrackingFidelity	OculusRuntimeSettings	2 (High) — 1 silently degrades to IK-only
bodyTrackingJointSet	OculusRuntimeSettings	0 (UpperBody)
requestBodyTrackingPermissionOnStartup	OculusProjectConfig	true — otherwise com.oculus.permission.BODY_TRACKING is never requested
OVRBody component	scene	present — nothing requests a tracking session without it

Worth flagging: the Meta XR SDK resets bodyTrackingFidelity to 1 when the project is opened
without running this, so a fresh clone builds an IK-only APK unless the step is run. Happy to
restructure this if you'd prefer the settings committed differently.

Two behaviours consumers need to know

Both are documented in Docs/BodyMeta.md; noting them here because they surprised us.

calib is not a gate. Joints stream complete from the first frame. Calibration runs inside the
runtime, refining the skeleton's scale; Valid only means it stopped refining. Measured on a
Quest 3:

    t  mounted  isActive  calib          count
 61.5        1         1  Calibrating       70   <- put on: immediately 70 joints
 73.9        1         1  Valid             70   <- 12 s later it turns Valid
 78.0        0         0  Valid             70   <- taken off: stale values, isActive says so

All 70 joints flowed for 12 s before Valid arrived — and Unity-Movement's sample scene likewise
drives its avatar throughout Calibrating. Calibration can't be disabled or pre-seeded
(BodyTrackingCalibrationInfo exposes only BodyHeight, and SuggestBodyTrackingCalibrationOverride
is a suggestion the runtime may ignore) and it restarts on every re-don, so gating on it would stall
the stream 30–60 s each time for no gain.

isActive must be checked first. When the headset leaves the head the joints array keeps its
last values — the JSON objects are reused rather than cleared, and calib/count go stale too
(row three above). Nothing in the numbers reveals it. Body tracking requires the headset actually
worn; hanging it on the neck stops the data, which differs from controller tracking.

Also included

LogWindow.Info is made thread-safe. Body tracking logs from a non-main thread and the previous
static-instance access could crash there — hit while developing this, so it's fixed here rather
than left as a trap.

Scope

Controller, head and hand paths are untouched. Verified by extracting the body of each
controller/head/hand method and diffing against main: OnControllerTog, OnHeadTog,
OnHandTrackingTog, OnSendTog, OnReconnectBtn, TcpConnect, SetControllerOn, SetHeadOn,
SetHandTrackingOn, SetTrackingType, GetControllerPose, IsControllerActive,
GetHeadsetPose, GetJoints, GetActiveInputDevice, IsHandTrackingActive — all 16 byte-identical.

The two code deletions are OnHighAccuracy and the HighAccuracy toggle's SetActive call: Meta
has no per-session fidelity switch (it's a build-time project setting, not a call like PICO's
StartBodyTracking(mode)), so the toggle is hidden rather than left implying an inactive control.

App identity (applicationIdentifier, productName, bundleVersion) is deliberately left at
upstream's values.

Testing

Verified end-to-end on a Quest 3 with PC-Service and the Python SDK:

joints=70   calib=Valid   fidelity=High   confidence=1.00
pose shape (70, 7)      ids 0..69 contiguous
valid: pos 70/70  orient 70/70      quaternion norms: min 1.0  max 1.0
timestamps advancing: True          max joint delta over 0.5 s: 0.0676 m
PICO "Body" path: unaffected

The joint parent table was validated geometrically rather than by eye: of 69 bones, 62 vary by
under 5 mm in length; upper arm 23.8 cm, forearm 23.8 cm, shoulder width 33.3 cm, and left/right
match to the hundredth of a centimetre — a wrong parent link could not produce that symmetry.

Built with Unity 2021.3.45f2 (the version this repo's README specifies), 0 errors.

---

Happy to split this up, rename the key, or move the docs if you'd prefer a different shape.

[zerenluo123]

The PC-side counterpart that parses this key is XR-Robotics/XRoboToolkit-PC-Service-Pybind#16

[luffy-yu]

Thank you for the PR!

I have a question. It seems that the full-body mode (84 joints) is not included in this PR, right?

[zerenluo123]

Thank you for the PR!

I have a question. It seems that the full-body mode (84 joints) is not included in this PR, right?

Correct — this PR is UpperBody (70 joints) only.

The reason is that FullBody's extra 14 joints are predicted rather than measured: the legs come from Generative Legs, inferred from head and upper-body motion, since the cameras can't see your legs while the headset is on your head.

For teleoperation we want ground truth for every joint we publish, and we want it without any external hardware — no base stations, no strap-on trackers. There's no official Meta leg tracker that would give real leg data (the SDK exposes no external tracker input at all), and the third-party full-body setups for Quest route through SteamVR. UpperBody at fidelity=High is the set that is both camera-measured and hardware-free.

[zerenluo123]

We just added the Full Body feature — the Mode dropdown is now
Off / Upper Body (70) / Full Body (84), switching the joint set at runtime via OVRBody.SetRequestedJointSet rather than fixing it at build time. Pushed to this branch.

Upper Body — 70 joints

Full Body — 84 joints

Full Body appends ids 70–83 and leaves 0–69 unchanged, so a consumer that only understands the first 70 keeps working. count and jointSet change in the same frame as the selection.

Those 14 joints come from Generative Legs — inferred from head and upper-body motion, since the cameras cannot see the operator's legs while the headset is worn. Different in kind from 0–69, so Docs/BodyMeta.md says so and Upper Body stays the startup default.

One note on project settings: BodyTrackingSetup sets Body Tracking Fidelity to High, which is required rather than a preference — [the Unity docs prescribe it for IOBT]:

If you want to use Inside-Out Body Tracking (IOBT), select High for Body Tracking Fidelity in the Movement Tracking section.

Verified on a Quest 3: switches both directions, all 84 joints position-valid, legs move with the operator. Also bumped bundleVersion to 1.0.2 and refreshed Docs/ui.png.