Add the PhyAI embodied rollout backend and the PPO policy-correction path used by PI0.5.

[Caspian443]

Description

Add the PhyAI embodied rollout backend and the PPO policy-correction path used by PI0.5.

• Register phyai as an embodied rollout backend alongside Hugging Face.
• Add the PhyAI worker integration, streamed actor-weight synchronization, and rollout lifecycle handling.
• Recompute actor-side log probabilities and apply configurable importance sampling correction for rollout-policy mismatch.
• Validate the model action horizon independently from the executed action chunk.
• Add the PI0.5 benchmark configuration and focused worker, config, and importance-sampling tests.

Motivation and Context

The PhyAI fused/BF16 rollout engine and the RLinf actor can produce slightly different behavior distributions. The correction path makes that mismatch explicit in PPO while keeping the existing Hugging Face backend unchanged. Backend selection remains centralized and opt-in through rollout.rollout_backend, so existing embodied configurations keep their current behavior.

How has this been tested?

• In the existing rlinf-phyai:dev container: python3 -m pytest -q tests/unit_tests/test_phyai_worker.py tests/unit_tests/test_embodied_importance_sampling.py tests/unit_tests/test_openpi_rlinf_config.py passed: 19 passed.
• The same container ran the PhyAI focused suite python3 -m pytest -q phyai/tests/test_pi05_rollout_schedule.py phyai/tests/test_engine_updates.py phyai/tests/weights/test_loader.py with 36 passed.
• An 8-GPU H800 smoke run with CUDA Graph enabled completed successfully (Slurm job 2511, exit code 0) for two PPO steps using the same SFT checkpoint and examples/pi05/kernel_policy_rlinf_bf16.yaml. success_once was 0.8261719 at step 1 and 0.8300781 at step 2; actor ratios were 1.001 and 1.000.
• The H800 run captured and recaptured the fixed-shape PI05 CUDA Graph after the hot weight update, with loaded=819, missing=0, optional_missing=0, and unexpected=0.
• A full local model smoke was not run because all local A100 devices were occupied; the local verification above is unit-level and container-level only.

Additional information (optional, e.g., figures and logs):

• The benchmark configuration is examples/embodiment/config/libero_spatial_ppo_openpi_pi05_rlinf_backend_benchmark.yaml. It selects rollout.rollout_backend: phyai, the RLinf BF16 operator policy, the RLinf-Pi05-LIBERO-SFT checkpoint, pi05_base, and the PaliGemma tokenizer.
• Keep examples/pi05/kernel_policy_rlinf_bf16.yaml and CUDA Graph enabled when reproducing the H800 result; changing the operator policy changes the numerical and performance comparison.
• The PhyAI runtime is not installed by the generic RLinf install target. Use the project image or a separate Python 3.12 environment containing the PhyAI dependencies; keep the actor and rollout interpreter paths explicit when they use different environments.
• Upload these figures to the PR description from the benchmark workspace: /home/phyai/doc/hf_vs_phyai_all8_offpolicy_success_rate.png and /home/phyai/doc/figures/predict_occupancy.png.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Documentation update (Document-only update)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.