Add Wordle GRPO that classifies dead groups as cliff vs collapse

[aljojoby9]

GeoGuesser already tells you to watch frac_reward_zero_std. If it is near 1, the step taught nothing. TRL still has no dynamic sampling, so those groups still get a backward pass. This folder treats that as a training failure rather than a chart.

What it is

04-wordle/ — the 2,309-word answer list as a training env (project 00 is 50 words), an estimator that splits a dead GRPO group into a cliff (different trajectories, same reward) and collapse (the same trajectory, G times), and two CPU measurements.

The two kinds need opposite responses. Resampling a cliff can recover a gradient. Resampling a collapse draws the same action; that is the 750 steps after GeoGuesser run 1 plateaued.

What it is not

A Qwen run. train/grpo_wordle.py is the TRL recipe (in-process, no Space). It has not been spent on a GPU. The numbers below are from a 32-d pointer over the answer list and from a Monte Carlo of GeoGuesser's published medians.

Numbers

Wordle, groups of 8, 200 steps, one seed, greedy eval on a 200-word holdout:

arm	best solve	at step	final	dead groups at end
sparse + GRPO	0.770	75	0.710	0.19
sparse + alive	0.790	50	0.700	0.05
process + GRPO	0.815	200	0.815	0.07
process + alive	0.820	175	0.760	0.03

Sparse GRPO peaked at step 75 and gave the points back as dead groups went 0.02 → 0.19. That is run 1's plateau, on CPU. Process reward is the +0.10; skipping dead groups is the dead-group column, not a substitute for a ranking the reward refused to give.

GeoGuesser, 20,000 within-task groups of 8, from the medians in 03-geoguesser/LEARNINGS.md:

• Independent P(eight zeros) from the 29.5% episode-zero rate is 5.7e-5. Measured within-task under subtract-and-floor: 6.43% dead, almost all cliffs.
• One resample of a dead group: 0.39%.
• The mixture they already train with: 0.01% dead.
• Eight identical city guesses: 100% collapse, 10,000× GRPO scale.
• 15 km of jitter around the published 662 km median: 0% dead, 277× median scale.

Population zeros are not group zeros. ACCUM=2 is the setting where this number is the within-task std.

How to check

cd 04-wordle
python -m pytest -q          # 24 tests
python envs/wordle/rollout.py zonal
python train/analyse_geoguesser.py
python train/tiny_grpo.py --steps 200 --seed 0

The observation never names the answer. rollout.py zonal is the trajectory a person can read.

train/geoguesser_run4.py is the classifier plus the env I would actually launch on GeoGuesser (SCALE_REWARDS=none, COST_SCALE=0.2, MAX_STEPS=250). It is not a fourth GeoGuesser run until someone launches it.

If 04 is taken by an in-flight PR, renumber. The next free folder on main was 04.

[Copilot]

🟡 Changes recommended

Unresolved moderate issues remain in the training recipes and GeoGuesser analysis/configuration.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a Wordle GRPO project that classifies dead groups as reward cliffs or policy collapse, with a 2,309-word environment, CPU experiments, GeoGuesser analysis, training recipes, tests, and reproducibility artifacts.

Changes:

• Adds Wordle gameplay, rewards, information-gain scoring, and dataset splits.
• Adds alive-group classification, GRPO experiments, and GeoGuesser simulations.
• Adds documentation, test coverage, result artifacts, and project metadata.

File summaries

File	Reviewed change / final review note
README.md	Registers the Wordle project.
04-wordle/train/tiny_grpo.py	Implements CPU GRPO training. Moderate (2 votes): non-skipping arms drop zero-advantage rollouts, so they do not match the claimed vanilla GRPO baseline.
04-wordle/train/test_tiny.py	Tests the tiny policy trainer.
04-wordle/train/test_geoguesser_run4.py	Tests GeoGuesser helpers.
04-wordle/train/test_alive.py	Tests classification and estimators.
04-wordle/train/README.md	Documents training scripts. Nit (1 vote): the dead-group metric should correspond to frac_dead, not frac_cliff.
04-wordle/train/grpo_wordle.py	Provides the TRL recipe. Moderate (3 votes): recorded cliff tasks are never replayed. Moderate (3 votes): the recipe does not apply the documented centered-rank estimator.
04-wordle/train/geoguesser_run4.py	Provides GeoGuesser classification helpers. Moderate (1 vote): advertised skip/stop settings are not wired through. Moderate (1 vote): collapsed groups can still receive a train recommendation.
04-wordle/train/analyse_geoguesser.py	Simulates GeoGuesser dead groups. Moderate (2 votes): the no-submission model is not calibrated to the cited zero-scoring data. Moderate (2 votes): resampling changes the task median instead of preserving the same task.
04-wordle/train/alive.py	Implements group classification, advantages, and replay weighting.
04-wordle/results/tiny-ablation.json	Stores CPU ablation results.
04-wordle/results/tiny-ablation.csv	Stores ablation curves.
04-wordle/results/README.md	Documents generated result artifacts.
04-wordle/results/geoguesser-dead-groups.json	Stores GeoGuesser simulation results.
04-wordle/REPRODUCE.md	Provides reproduction commands.
04-wordle/README.md	Documents the environment and findings. Nit (1 vote): the dead-group metric should correspond to frac_dead, not frac_cliff.
04-wordle/project.yaml	Defines project metadata.
04-wordle/LEARNINGS.md	Records experiment conclusions.
04-wordle/envs/wordle/rollout.py	Provides a readable rollout. Nit (1 vote): the docstring describes candidate scoring that the implementation does not perform.
04-wordle/envs/wordle/core/words.py	Loads the answer vocabulary.
04-wordle/envs/wordle/core/test_core.py	Tests Wordle domain behavior.
04-wordle/envs/wordle/core/tasks.py	Defines training and evaluation splits.
04-wordle/envs/wordle/core/rewards.py	Defines reward shapes.
04-wordle/envs/wordle/core/information.py	Computes remaining candidates and information gain.
04-wordle/envs/wordle/core/game.py	Implements Wordle gameplay and feedback.
04-wordle/envs/wordle/core/answers.txt	Adds the 2,309-word answer list.
04-wordle/envs/wordle/core/__init__.py	Exports core APIs.

Review details

Suppressed comments (5)

04-wordle/README.md:102

• frac_reward_zero_std counts every dead group, so it corresponds to alive/frac_dead (frac_cliff + frac_collapse), not alive/frac_cliff. These metrics will diverge as soon as collapse is nonzero, causing the suggested check to report a false mismatch.

Watch `alive/frac_cliff` against TRL's `frac_reward_zero_std`. They should agree. Watch `alive/frac_collapse`, which TRL does not report. If collapse is high and solve rate has been flat for 50 steps, stop. That is the $70.

04-wordle/envs/wordle/rollout.py:5

• The docstring says the rollout chooses the remaining answer that keeps the most greens, but the implementation simply uses left[0] and never scores candidate guesses. The description should match the deterministic first-candidate behavior, or the selection algorithm should be implemented.

Uses the frequency opener CRANE, then picks the remaining answer that keeps
the most greens. Not a policy — a trajectory, so you can see what the
environment actually returns before anyone trains against it.

04-wordle/train/README.md:20

• frac_reward_zero_std counts every dead group, so it corresponds to alive/frac_dead (frac_cliff + frac_collapse), not alive/frac_cliff. These metrics will diverge as soon as collapse is nonzero, causing the suggested check to report a false mismatch.

Step 3 has not been run in this contribution. The numbers in the project README are from steps 1 and 2. If you run step 3, the metric to watch is `alive/frac_cliff` against TRL's `frac_reward_zero_std` — they should agree — and `alive/frac_collapse`, which TRL does not report.

04-wordle/train/geoguesser_run4.py:102

• The recommended launch settings above advertise ALIVE_SKIP_CLIFF=1 and ALIVE_STOP_COLLAPSE=0.8, but this helper omits both and the existing GeoGuesser trainer does not consume them. Calling run4_env() therefore cannot enable the promised skip/stop behavior; add the settings and wire them into the caller, or remove them from the launch recommendation.

        "NUM_GENERATIONS": "8",
        "SAVE_STEPS": "25",
        "MAX_TURNS": "12",
    }

04-wordle/train/geoguesser_run4.py:80

• For a collapsed group whose mean turn count is above the 1.5 threshold, skip remains true by default but recommendation is still train. A consumer following the recommendation can run the backward pass that this report says to skip; handle all collapse groups explicitly and reserve train for live groups.

    else:
        recommendation = "train"

• Files reviewed: 27/27 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.