async-pbt

Asynchronous population-based training for RL-Games, with checkpoint inheritance, bounded hyperparameter mutation, and auditable selection.

async-pbt extracts training infrastructure developed for dexterous-manipulation experiments into a small standalone package. It is research-grade software, not an exact DexPBT paper reproduction, a cluster scheduler, or evidence that PBT outperforms independent training.

What it does

• Runs independent training processes against a shared population workspace.
• Compares completed-episode evidence with freshness and condition-compatibility checks.
• Copies a selected donor's training checkpoint and restarts the recipient process.
• Preserves recipient progress and mutation RNG while inheriting donor training state.
• Applies bounded mutations to supported PPO hyperparameters.
• Retains historical best checkpoints and donor-lineage records.
• Offers optional bounded Gaussian exploration for compatible RL-Games policies.

The core imports without Isaac Sim, Torch, or RL-Games. Training integration is optional. Robot assets, simulator tasks, pretrained checkpoints, and remote deployment scripts are not included.

Install

Use a fresh Python 3.12 environment. For a CPU-only setup:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
python -m pip install 'async-pbt[rl-games,examples] @ git+https://github.com/LCK-Andy/async-pbt.git@v0.1.0'

This release is distributed through GitHub, not PyPI. The initial trainer integration targets RL-Games 1.6.5. See validation for the combinations actually checked.

For development from a clone:

python -m pip install -e '.[rl-games,examples,test]'
python -m pytest -q

Run a CPU population

python -m async_pbt.examples.cpu_population \
  --members 2 --smoke --workspace ./runs/cpu-smoke

The example uses a small continuous-control task, real PPO updates, and separate CPU processes. It needs no simulator, GPU, or Ray. Use a new workspace for an independent experiment. A short learning run is an execution check: it does not guarantee exploitation will occur or establish learning quality. Automated end-to-end tests use a separately identified deterministic selection fixture to exercise actual process replacement.

Architecture

The package separates population configuration, mutation, and episode evidence from RL-Games observers and checkpoint restoration. An environment supplies completed-episode summaries; a trainer supplies the model, optimizer, progress counters, and restart entrypoint.

• Configuration and episode evidence
• Checkpoint and restart semantics
• Isaac Lab integration and extraction boundaries
• Validation and limitations
• Source provenance

Important boundaries

• One writer per member; all members need a trusted shared filesystem with the required atomic replacement semantics.
• Checkpoints can contain executable pickle data. Load only trusted checkpoints and use a trusted population workspace.
• Resuming trainer and curriculum state does not imply restoring the complete simulator state or bitwise deterministic training.
• Constant/adaptive learning-rate schedules are supported; linear schedules are rejected. Raw checkpoints without PBT metadata cannot reconstruct unsaved entropy, gradient-norm, KL, or mutation-RNG state; see restore limits.
• No reward-weight evolution, PB2 proposals, general resource scheduling, or multi-GPU distributed PPO within a member is claimed.
• Online selection scores are not held-out evaluations. Population members interact and are not independent statistical replications.

License and attribution

BSD-3-Clause for original contributions, with retained third-party notices. See LICENSE, THIRD_PARTY_NOTICES.md, and provenance.

Please use CITATION.cff when citing the software. Contributions and reproducible bug reports are welcome; see CONTRIBUTING.md.