docs(experiments): add PyTorch video poker example with a full W&B run lifecycle

examples/pytorch/pytorch-video-poker-bot/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.py[cod]
+.DS_Store
+checkpoints/
+wandb/
+.env
+*.pt
+data/*.npz
+artifacts/

examples/pytorch/pytorch-video-poker-bot/README.md

@@ -0,0 +1,136 @@
+# Train a video poker bot with PyTorch and W&B
+
+This example trains a small PyTorch network to play video poker and tracks
+it with [Weights & Biases](https://wandb.ai). The poker is just a stand-in
+task. The point is to see a complete W&B run:
+
+1. **`wandb.init()`** starts a run, puts it in a group (`train` or `eval`),
+   and records its config (hyperparameters and dataset size).
+2. **`run.log()`** sends metrics every epoch, so you can watch training live.
+3. **`run.log_artifact()`** uploads the trained checkpoint as a versioned
+   model artifact.
+4. **Finish**: `train.py` opens the run with `with wandb.init(...) as run:`,
+   so the run finishes automatically when the block exits. Without the
+   `with` block, you would call `run.finish()` yourself.
+
+`evaluate.py` then starts a second run (`job_type="evaluation"`). It fetches
+the model artifact from W&B with **`run.use_artifact()`**, which links the
+two runs, and logs its score every 100 hands as it plays.
+
+## How video poker works
+
+You are dealt five cards. You choose which of them to hold: none, all five,
+or any mix in between (32 possible combinations). The cards you don't hold
+are replaced with new ones from the deck, and that is your final hand. Its
+reward is looked up in the paytable.
+
+This example plays **9/6 Jacks or Better**, betting 5 credits per hand:
+
+| Final hand | Reward (credits) |
+| --- | --- |
+| Royal flush | 4000 |
+| Straight flush | 250 |
+| Four of a kind | 125 |
+| Full house | 45 |
+| Flush | 30 |
+| Straight | 20 |
+| Three of a kind | 15 |
+| Two pair | 10 |
+| Pair of jacks or better | 5 |
+| Anything else | 0 |
+
+The name "9/6" comes from the full house paying 9 and the flush paying 6 for
+each credit bet. With optimal holds, the rewards average about 99.5% of the
+credits wagered.
+
+The dataset gives each dealt hand the exact expected reward of all 32 hold
+choices. The network learns to predict those values, and at play time it
+makes the hold with the highest prediction.
+
+## Setup
+
+```bash
+cd examples/pytorch/pytorch-video-poker-bot
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+wandb login
+```
+
+## Run it
+
+### 1. Generate the dataset
+
+```bash
+python generate_dataset.py --output data/hands.npz
+```
+
+This deals 1,000,000 random hands and labels each one. It takes about two
+minutes and writes a ~44 MB file. For a quick test, add `--hands 20000`.
+
+### 2. Train
+
+```bash
+python train.py \
+  --dataset data/hands.npz \
+  --checkpoint checkpoints/jacks_or_better_network.pt \
+  --artifact-name jacks-or-better-network \
+  --project video-poker \
+  --run-name train-baseline
+```
+
+This takes under a minute on a laptop CPU. After each epoch it saves the
+checkpoint if validation improved, and at the end it uploads the checkpoint
+to W&B as the model artifact `jacks-or-better-network`. Optional flags:
+`--entity` (a W&B team or user; defaults to your default entity),
+`--epochs` (default 20), `--lr`, `--batch-size`, `--hidden-size`, `--seed`.
+
+Without `--seed`, each run picks a random seed. The seed it used is saved
+in the run's config in W&B, so you can repeat any run exactly by passing
+that value.
+
+### 3. Evaluate
+
+```bash
+python evaluate.py \
+  --artifact jacks-or-better-network:latest \
+  --project video-poker \
+  --run-name eval-baseline
+```
+
+This downloads the latest version of the model artifact and plays 100,000
+hands with it (change with `--hands`), which takes about 10 seconds. It logs
+to a separate run in the same project. Like training, it deals with a random
+seed unless you pass `--seed`, so each evaluation plays different hands.
+If you trained under a team with `--entity`, pass the same `--entity` here.
+
+## What you will see in W&B
+
+- **Training run**: config on the Overview tab, and these charts per epoch:
+  - `train_loss` and `val_loss`
+  - `val_optimal_action_pct`: how often the network makes the optimal hold
+  - `val_mean_regret`: the expected reward per credit bet that the network's
+    holds give up compared with the optimal holds, on average. Lower is
+    better, and the saved checkpoint is the epoch with the lowest value.
+  - `val_expected_return_pct`: expected rewards as a percentage of credits
+    wagered
+- **Evaluation run**: running totals logged every 100 hands, so you can
+  watch the score settle as more hands are played: `wagered` (credits bet),
+  `payout` (credits rewarded), `profit` (the difference) and `return_pct`
+  (rewards as a percentage of credits wagered).
+- **Model artifact**: every training run adds a new version of
+  `jacks-or-better-network`. Its Lineage tab shows which training run
+  produced each version and which evaluation runs used it.
+
+## Files
+
+| File | What it does |
+| --- | --- |
+| `generate_dataset.py` | Builds the labeled training dataset. |
+| `train.py` | Trains the network and logs the run to W&B. |
+| `evaluate.py` | Fetches the trained model from W&B, plays hands with it, and logs the score. |
+| `dataset.py` | Loads a generated dataset for training. |
+| `model.py` | The network, hand encoding, training and validation steps, and checkpoint save/load. |
+| `game.py` | Cards, deck, dealing and drawing. |
+| `jacks_or_better.py` | Hand rankings and the paytable. |
+| `ev.py` | Calculates the exact expected reward of each hold choice. |

examples/pytorch/pytorch-video-poker-bot/dataset.py

@@ -0,0 +1,48 @@
+"""Load a dataset written by generate_dataset.py."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from game import CARDS_PER_HAND, NUM_HOLDS
+from model import encode_cards
+
+
+@dataclass(frozen=True)
+class HoldDataset:
+    cards: np.ndarray  # (N, 5) card codes, sorted
+    targets: np.ndarray  # (N, 32) expected profit per credit bet for each hold
+
+    @property
+    def states(self) -> np.ndarray:
+        return encode_cards(self.cards)
+
+    def split_samples(
+        self,
+        *,
+        seed: int,
+        validation_fraction: float = 0.1,
+    ) -> tuple[np.ndarray, np.ndarray]:
+        """Shuffle into train / validation (90/10)."""
+        rng = np.random.default_rng(seed)
+        order = rng.permutation(len(self.cards))
+        val_start = len(order) - int(len(order) * validation_fraction)
+        return order[:val_start], order[val_start:]
+
+
+def load_dataset(path: Path) -> HoldDataset:
+    if not path.exists():
+        raise FileNotFoundError(f"{path} not found; create it with generate_dataset.py --output {path}")
+    with np.load(path, allow_pickle=False) as payload:
+        cards = payload["cards"]
+        targets = payload["targets"]
+
+    if cards.ndim != 2 or cards.shape[1] != CARDS_PER_HAND:
+        raise ValueError(f"invalid cards shape: {cards.shape}")
+    if targets.shape != (len(cards), NUM_HOLDS):
+        raise ValueError(f"invalid targets shape: {targets.shape}")
+
+    return HoldDataset(cards=cards, targets=targets)

examples/pytorch/pytorch-video-poker-bot/ev.py

@@ -0,0 +1,134 @@
+"""Exact expected profit of every hold choice, found by counting, not simulating.
+
+Once per process, `build_tables` ranks all 2,598,960 five-card hands with the
+Jacks or Better classifier. For every group of 0 to 4 cards, it counts how many
+five-card hands containing that group land in each hand rank.
+
+For a dealt hand, `hold_expected_values` looks up those counts for the hand's
+32 subsets of cards. The counts include hands that reuse discarded cards, which
+can't be drawn again, so inclusion-exclusion over the discards removes them.
+Averaging the rewards over every possible draw and subtracting the bet gives
+the expected profit of each hold.
+"""
+
+from __future__ import annotations
+
+from itertools import chain, combinations
+from math import comb
+from typing import Sequence
+
+import numpy as np
+
+from game import BET, CARDS_PER_HAND, DECK_SIZE, NUM_HOLDS, Card, Paytable, hand_to_codes
+from jacks_or_better import NUM_RANK_CLASSES, RANK_CLASSES, classify
+
+TOTAL_HANDS = comb(DECK_SIZE, CARDS_PER_HAND)
+ALL_POSITIONS = NUM_HOLDS - 1  # every card position set
+CARDS_LEFT_IN_DECK = DECK_SIZE - CARDS_PER_HAND
+
+_BINOMIAL = np.array(
+    [[comb(n, k) for k in range(CARDS_PER_HAND + 1)] for n in range(DECK_SIZE + 1)],
+    dtype=np.int64,
+)
+
+
+def colex_index(sorted_codes: np.ndarray, size: int) -> np.ndarray:
+    index = np.zeros(sorted_codes.shape[0], dtype=np.int64)
+    for position in range(size):
+        index += _BINOMIAL[sorted_codes[:, position], position + 1]
+    return index
+
+
+def colex_index_scalar(sorted_codes: Sequence[int]) -> int:
+    return sum(comb(code, position + 1) for position, code in enumerate(sorted_codes))
+
+
+class EVTables:
+    def __init__(self, subset_counts: Sequence[np.ndarray], hand_ranks: np.ndarray) -> None:
+        self._subset_counts = tuple(subset_counts)
+        self._hand_ranks = hand_ranks
+
+    def counts_for(self, sorted_codes: Sequence[int]) -> np.ndarray:
+        size = len(sorted_codes)
+        index = colex_index_scalar(sorted_codes)
+        if size == CARDS_PER_HAND:
+            counts = np.zeros(NUM_RANK_CLASSES, dtype=np.int64)
+            counts[int(self._hand_ranks[index])] = 1
+            return counts
+        return self._subset_counts[size][index].astype(np.int64)
+
+
+def build_tables() -> EVTables:
+    flat = np.fromiter(
+        chain.from_iterable(combinations(range(DECK_SIZE), CARDS_PER_HAND)),
+        dtype=np.int8,
+        count=TOTAL_HANDS * CARDS_PER_HAND,
+    )
+    hands = flat.reshape(TOTAL_HANDS, CARDS_PER_HAND)
+    ranks = classify(hands).astype(np.int64)
+
+    hand_ranks = np.zeros(TOTAL_HANDS, dtype=np.uint8)
+    hand_ranks[colex_index(hands, CARDS_PER_HAND)] = ranks.astype(np.uint8)
+
+    subset_counts: list[np.ndarray] = []
+    for size in range(CARDS_PER_HAND):
+        rows = comb(DECK_SIZE, size)
+        table = np.zeros(rows * NUM_RANK_CLASSES, dtype=np.int64)
+        for positions in combinations(range(CARDS_PER_HAND), size):
+            if size == 0:
+                index = np.zeros(TOTAL_HANDS, dtype=np.int64)
+            else:
+                index = colex_index(hands[:, positions], size)
+            table += np.bincount(
+                index * NUM_RANK_CLASSES + ranks,
+                minlength=rows * NUM_RANK_CLASSES,
+            )
+        subset_counts.append(table.reshape(rows, NUM_RANK_CLASSES).astype(np.uint32))
+    return EVTables(subset_counts, hand_ranks)
+
+
+_TABLES: EVTables | None = None
+
+
+def get_tables() -> EVTables:
+    """Build EV tables once per process; reuse in memory afterward."""
+    global _TABLES
+    if _TABLES is None:
+        _TABLES = build_tables()
+    return _TABLES
+
+
+def rank_payouts(paytable: Paytable) -> np.ndarray:
+    """Reward for each hand rank, in the order `classify` numbers them."""
+    return np.array([paytable.payout_for(rank) for rank in RANK_CLASSES], dtype=np.int64)
+
+
+def hold_expected_values(hand: Sequence[Card], payouts: np.ndarray) -> list[float]:
+    """Exact expected profit (in credits) for every hold mask.
+
+    `payouts` comes from `rank_payouts`; build it once and reuse it.
+    """
+    tables = get_tables()
+    codes = hand_to_codes(hand)
+
+    subset_payout = [0] * NUM_HOLDS
+    for subset in range(NUM_HOLDS):
+        sorted_codes = sorted(
+            codes[position] for position in range(CARDS_PER_HAND) if subset & (1 << position)
+        )
+        subset_payout[subset] = int(tables.counts_for(sorted_codes) @ payouts)
+
+    expected_values: list[float] = []
+    for hold_mask in range(NUM_HOLDS):
+        discards = ALL_POSITIONS ^ hold_mask
+        total = 0
+        subset = discards
+        while True:
+            sign = -1 if bin(subset).count("1") % 2 else 1
+            total += sign * subset_payout[hold_mask | subset]
+            if subset == 0:
+                break
+            subset = (subset - 1) & discards
+        draws = comb(CARDS_LEFT_IN_DECK, bin(discards).count("1"))
+        expected_values.append(total / draws - BET)
+    return expected_values