PolyVision is a reinforcement-learning research environment for The Battle of Polytopia. It connects a Java implementation of the game to Python through Py4J, exposes a constrained task through Gymnasium, and trains legality-aware PPO policies with a heavily adapted CleanRL implementation.
The project does not currently solve the full game. Its active curriculum is an economy-first Bardur task ending after Turn 10, with combat attacks excluded from the policy interface. This narrower setting is used to study reliable action representation, exploration, city growth, research, and evaluation before expanding toward full-game play.
PolyVision originated from ClaireBookworm/polytopia_rl and has since been substantially extended with a new experimental curriculum, action representation, map corpus, training infrastructure, evaluation system, and research documentation. See Attribution and licensing.
PolyVision runs on genuine Polytopia initial-map data. Compressed save states are converted into canonical, schema-versioned JSON and then into validated Tribes CSV maps. Phase 1 contains 5,517 genuine 11×11 Bardur Drylands maps in a frozen identity-based split: 5,000 train, 250 validation, 250 test, and 17 human benchmark maps.
The policy interacts through stable global action IDs rather than state-local Java indices. Each state also carries the current legal IDs, a fixed-capacity validity mask, and optional semantic/economic action features. Strict validation checks that every policy-visible legal action has one collision-free global identity and that selected IDs execute the intended Java action.
- Java Tribes engine controlled from Python through Py4J
- Gymnasium
Tribes-v0environment with deterministic reset and cleanup behavior - 11×11 genuine Bardur corpus with a reproducible, hash-verified experimental split
- geometry-derived observations and global action catalog
- legality-aware PPO actor modes:
legal_features(default),legal_only(ablation/baseline), anddense_debug - 47-dimensional semantic/economic legal-action features
- shaped Phase 1 rewards for economy, expansion, exploration, and core Terminal-SPT bonus
- strict pre-training action-interface validation with content-aware caching
- checkpoint sidecars that enforce environment/interface compatibility
- asynchronous multi-JVM training, TensorBoard/W&B telemetry, diagnostics, and SPS profiling
- canonical deterministic/sampled validation evaluation with paired policy-visible baselines and machine-readable reports
- persistent first-attempt human benchmarking with model-interface and information parity
flowchart LR
A["Genuine Polytopia .state data"] --> B["Canonical JSON and validated CSV"]
B --> C["Java Tribes engine"]
C <--> D["Py4J bridge"]
D <--> E["Gymnasium environment"]
E --> F["Stable IDs and legal-action interface"]
F <--> G["Adapted CleanRL PPO"]
G --> H["Evaluation and diagnostics"]
The Java engine remains authoritative for game state and raw legal actions. The Python wrapper defines the research curriculum, observation, stable action interface, reward, horizon, and validation contract. See Architecture for the full control flow.
The policy controls Bardur after the fail-closed v2_guaranteed_two_unit opening: two animals, Workshop, two original-warrior moves, a required second-warrior spawn, and handoff at Turn 2. Turn-1 opening selection alone excludes the capital; ordinary policy gameplay is unchanged. The corrected audit passed 5,517/5,517 maps. Historical Seed-1 used v1_mixed_capital_regression (55.07% two-unit) and is preserved but shelved before pristine capability test. See the scripted-opening audit and reflection.
ATTACK is deliberately unavailable to the Phase 1 policy. The environment prioritizes early village-expansion lines through active action filters and trains on shaped SPT, city capture, village discovery/progress, and fog-clearing signals. Final T10 SPT and other economy metrics are reported separately from shaped return.
On the active maps, the observation has 6,424 values (under the v5_human_information_parity contract) and the global action space has 63,913 IDs. Only a small legal subset is selectable in a state; by default it is represented in 256 slots with 47 legal action features (v1_4_parity_spatial_and_cost). See Actions, Observations, and Rewards for the exact current contract.
Requirements are Python 3.10+, a JDK with java and javac, and PowerShell for the commands below.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements-lock.txt
Set-Location pol_env/Tribes
$sources = Get-ChildItem -Path src -Recurse -Filter *.java | ForEach-Object FullName
javac -cp "lib/json.jar" -d out -sourcepath src $sources
Set-Location ../..Verify the frozen split and live training-pool contract:
python tools/split_phase1_map_pool.py
$env:POLYVISION_LEVEL_POOL_GLOB = 'levels/phase1_pool_bardur_real/train/*.csv'
$env:POLYVISION_SOLO_NO_OPPONENT_MODE = '1'
python tools/validate_environment_contract.py --expected-width 11 --expected-height 11The wrapper defaults to the training pool only; the explicit setting above makes the experimental role visible in commands and run records. Static split verification checks all 5,517 hashes and assignments, while the live validator resets the selected pool and checks geometry, observation, catalog, and interface consistency. For detailed installation notes and a smaller smoke test, use Getting started.
The primary PPO entrypoint is:
py_rl/cleanrl/cleanrl/ppo.py
A representative development training run (using default legal_features actor and Terminal-SPT reward):
$env:POLYVISION_LEVEL_POOL_GLOB = 'levels/phase1_pool_bardur_real/train/*.csv'
$env:POLYVISION_SOLO_NO_OPPONENT_MODE = '1'
$env:POLYVISION_INFO_MODE = 'fast'
python py_rl/cleanrl/cleanrl/ppo.py `
--total-timesteps 500000 `
--num-envs 12 `
--num-steps 128 `
--save-model `
--save-frequency 100000Standard Phase-1 training automatically uses legal_features and Terminal-SPT without requiring explicit flags or reward environment variables. Strict 10,000-state interface validation runs before training by default and is cached by code, configuration, and map-pool identity. TensorBoard output and models are written under runs/; W&B is optional with --track. Each model is accompanied by an action-interface JSON sidecar required by current evaluators. See Training for ablations, full options, and reproducibility commands.
Use evaluate_brain.py for compatible one-episode policy inspection:
python evaluate_brain.py `
--model-path runs/<run>/ppo.cleanrl_model `
--level-pool-glob 'levels/phase1_pool_bardur_real/validation/*.csv' `
--seed 42The repository also includes visible-information scripted baselines, privileged diagnostics, action/feature audits, and a pool contract validator. These tools have different visibility and protocol assumptions; Evaluation identifies their intended use.
Run the canonical 3,000-episode Phase 1 validation suite with:
python tools/evaluate_phase1.py `
--model-path runs/<run>/ppo.cleanrl_model `
--pool validation `
--suite full `
--repeats-per-map 5 `
--seed 42This compares deterministic PPO, sampled PPO, visible greedy, and random legal on the same 250 validation maps. Final T10 SPT is primary; stochastic uncertainty is computed across per-map replicate means. Results and provenance are retained under outputs/evaluations/. Test evaluation is separately guarded and is not part of ordinary development evaluation.
Run a permanent human challenge attempt with:
python tools/human_benchmark.pyThe command selects an unplayed human-benchmark map and presents exactly the wrapper-filtered stable global IDs available to PPO. Results remain separate from pristine test evidence; see Human benchmark.
The current frozen reference result is the Phase 1 v5 PARITY002 Seed3 16M Terminal-SPT frozen reference benchmark: on 250 fixed held-out test maps, deterministic PPO achieved 20.17 mean Turn-10 SPT (95% CI 19.67–20.67, median 20.00), compared with 7.96 for the policy-visible greedy baseline and 20.52 on the 250 validation maps. PPO beat visible greedy on all 250 test maps (250 W / 0 T / 0 L), with a mean paired advantage of +12.22 SPT. See Phase 1 v5 PARITY002 Seed3 16M Terminal-SPT Reference Run for full metrics, validation/test distributions, paired comparisons, and scientific interpretation.
Phase 1 is not complete; active optimization remains ongoing and 20.17 test argmax SPT serves as the current frozen reference benchmark to beat. This reference remains a single-seed training result (Seed 3), and the fixed test pool has been used for prior reference benchmarks (and is thus a fixed held-out benchmark rather than newly pristine evidence).
Historical Seed-1, Seed-2, the baseline 16M reference run, the superseded v3 Terminal-SPT reference run, and the superseded v4 PARITY001 reference run remain preserved for their respective historical task, interface, and reward definitions. See Phase 1 v4 PARITY001 Seed3 16M Terminal-SPT Reference Run (Superseded), Phase 1 v3 Seed3 16M Terminal-SPT Reference Run (Superseded), Phase 1 v3 Seed3 16M Reference Run, Phase 1 Seed-1 mixed-opening reflection, and Phase 1 scripted-opening audit.
| Path | Purpose |
|---|---|
pol_env/Tribes/src/ |
Java Tribes engine and Py4J-facing environment |
pol_env/Tribes/py/ |
Bridge, Gymnasium wrapper, validators, and utilities |
pol_env/Tribes/levels/phase1_pool_bardur_real/ |
Frozen train/validation/test/human Phase 1 split and manifest |
py_rl/cleanrl/cleanrl/ppo.py |
Primary PolyVision PPO trainer |
tools/polytopia_state_converter/ |
Compressed save to canonical JSON |
tools/polytopia_map_converter/ |
Canonical JSON to validated Tribes CSV |
tools/evaluate_phase1.py |
Canonical Phase 1 validation evaluator |
tools/ |
Additional evaluation and environment-contract utilities |
docs/ |
Current project documentation |
- Getting started
- Architecture
- Environment
- Actions
- Observations
- Rewards
- Training
- Evaluation
- Phase 1 v3 Seed3 16M Terminal-SPT Reference Run
- Phase 1 v3 Seed3 16M Reference Run (Historical)
- Phase 1 v3 Seed3 16M Behavioral Failure Analysis
- First Phase 1 validation results (Historical)
- Phase 1 Seed-1 mixed-opening reflection
- Phase 1 scripted-opening audit
- Human benchmark
- Human–AI Information Parity Audit
- Maps
- Reproducibility
- Troubleshooting
Focused manuals remain beside the state converter, map converter, and optional map harvester. CHANGELOG.md records implementation history; historical benchmark evidence is explicitly non-normative.
- The task ends after Turn 10 and does not represent full-game strategy.
- Combat attacks are excluded from policy control.
- The active corpus covers one tribe and one 11×11 Drylands distribution.
- The observation is a compact engineered vector, not the complete game state.
- Reward and action filtering encode a Phase 1 curriculum and can bias learned behavior.
- Checkpoints are tightly coupled to geometry and action-interface metadata.
- The trainer saves weights and interface metadata, not full optimizer/RNG state for exact resume.
- Current-interface multi-training-seed validation/test benchmark results are not yet established.
Current-interface multi-seed Phase 1 training and evidence are deferred for the constrained Bardur Turn-10 task. They may begin only after the canonical human-benchmark gate is established under GitHub issue #3 and the applicable model or candidate passes it; see Human benchmark for the canonical workflow. Later phases can broaden map distributions and tribes, relax curriculum filters, add combat and opponents, extend observations and action semantics, and evaluate full-game policies. Those items are future work, not current capabilities.
PolyVision originated as a fork of ClaireBookworm/polytopia_rl and has since developed into a substantially expanded research project.
The repository retains and modifies upstream code from polytopia_rl, including portions of the Tribes game engine, Python/Java bridge, environment code, and related utilities. Credit for that upstream work belongs to its original authors and contributors. Git history is retained to preserve authorship and provenance.
PolyVision does not currently have a repository-wide open-source license.
The upstream ClaireBookworm/polytopia_rl repository does not provide a root license covering the project as a whole. Because PolyVision contains substantial code derived from that repository, PolyVision cannot simply relicense the entire codebase under a new license without first resolving the licensing status of the inherited work.
Some components within the repository are separately licensed:
- RL code under
py_rl/cleanrlcontains code derived from CleanRL and other projects. The applicable notices and licenses are preserved inpy_rl/cleanrl/LICENSE. - Vendored dependencies and third-party components retain their own license and attribution notices where applicable.
- Third-party game assets, trademarks, and other intellectual property are not relicensed by PolyVision.
Unless a file or component explicitly states otherwise, the presence of source code in this repository should not be interpreted as granting additional rights beyond those provided by the applicable copyright holders and licenses.
I intend to clarify the licensing of PolyVision's original contributions once the licensing status of the inherited upstream code has been resolved.
The Battle of Polytopia and related names, assets, and intellectual property belong to their respective rights holders. PolyVision is an independent research project and is not affiliated with or endorsed by Midjiwan AB.