From 05574644fd084d8176206a8571aa5b5791884454 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Thu, 21 May 2026 19:40:17 +0200 Subject: [PATCH 1/9] Create scoring and parsing functions --- README.md | 359 ++++++++++++++++++++++------------------ pyproject.toml | 6 +- src/help.py | 4 +- src/miniBen/__init__.py | 23 ++- src/miniBen/example.py | 119 ++++++++++++- src/miniBen/model.py | 84 ++++++---- src/miniBen/parsers.py | 94 ++++++++++- src/miniBen/prompts.py | 130 +++++++++++---- src/miniBen/runner.py | 97 ++++++++--- src/miniBen/scorers.py | 278 ++++++++++++++++++++++++++++++- tests/test_runner.py | 48 +++++- 11 files changed, 965 insertions(+), 277 deletions(-) diff --git a/README.md b/README.md index c8b2cc6..621a9a3 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,20 @@ # miniBen — Mini LLM Benchmarks -**miniBen** is a Python package for running small, task-specific benchmarks against Large Language Models (LLMs) via [OpenRouter](https://openrouter.ai/). Each benchmark sends a structured prompt to a model, parses the reply, and scores the result so you can compare models on skills like cognitive flexibility and creativity. +**miniBen** evaluates Large Language Models on small benchmarks through [OpenRouter](https://openrouter.ai/). Each benchmark runs several prompt variants, sends them to a model, parses replies, and aggregates a score. -> **Status:** The package can call models and run the full benchmark pipeline. Parsers and scorers for some benchmarks are still stubs (`pass`); scores may be `None` until those are implemented. +> **Status:** `run_benchmark()` and `AIModel.ask()` work end-to-end. **Parsers** and **scorers** in `parsers.py` / `scorers.py` are still stubs, so `parsed` is usually `None` and `score` is `None` until you implement them. --- ## Table of contents -- [Features](#features) +- [How it works](#how-it-works) - [Requirements](#requirements) - [Installation](#installation) -- [API key setup](#api-key-setup) +- [API key](#api-key) - [Quick start](#quick-start) -- [Available benchmarks](#available-benchmarks) -- [Usage guide](#usage-guide) +- [Benchmarks](#benchmarks) +- [API reference](#api-reference) - [Return values](#return-values) - [Project layout](#project-layout) - [Development](#development) @@ -24,108 +24,92 @@ --- -## Features +## How it works -- **OpenRouter integration** — Use any model listed on OpenRouter with a single model ID string. -- **Optional reasoning mode** — Enable chain-of-thought style reasoning on supported models. -- **Built-in benchmarks** — Predefined prompts for Meta-Chess (cognitive flexibility) and creative writing. -- **Extensible pipeline** — Each benchmark wires together a prompt, parser, and scorer; you can add new ones in `runner.py`. -- **Simple Python API** — `AIModel.ask()` for one-off prompts, `run_benchmark()` for the full flow. +``` +prompts.py → runner.run_benchmark() → model.AIModel.ask() → OpenRouter + ↓ + parsers.parse_*() → scorers.score_*() +``` + +For **`cognitive flexibility`** and **`creativity`**, `run_benchmark()`: + +1. Builds **5 prompts** (one per variant — opening or word set). +2. Calls the API **5 times** via `AIModel.ask()`. +3. Parses each reply (errors are caught; failed parses become `None`). +4. Scores all parses together with the benchmark scorer. --- ## Requirements -- **Python** 3.9 or newer -- An **[OpenRouter](https://openrouter.ai/)** account and API key -- Internet access when calling models (requests go to `https://openrouter.ai/api/v1`) +- Python **3.9+** +- [OpenRouter](https://openrouter.ai/) API key +- Network access for live runs --- ## Installation -### From a local clone (recommended for development) - ```bash git clone https://github.com/Programming-The-Next-Step-2026/mini_benchmarks.git cd mini_benchmarks python -m venv .venv -source .venv/bin/activate # Windows: .venv\Scripts\activate -pip install -e . -``` - -### With development tools (pytest) - -```bash +source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -e ".[dev]" ``` -### Verify the install +Verify: ```bash -python -c "from miniBen import AIModel, run_benchmark, BENCHMARKS; print(list(BENCHMARKS))" +python -c "from miniBen import BENCHMARKS; print(list(BENCHMARKS))" +# ['cognitive flexibility', 'creativity'] ``` -You should see the benchmark keys printed without errors. - --- -## API key setup +## API key -miniBen reads your key from the environment variable **`OPENROUTER_API_KEY`**. +Set **`OPENROUTER_API_KEY`** (loaded from `.env` via `python-dotenv` on import). -### Option 1 — `.env` file (recommended) - -Create a file named `.env` in the project root (it is gitignored by default): +**`.env` in project root:** ```env OPENROUTER_API_KEY=sk-or-v1-your-key-here ``` -`python-dotenv` loads this automatically when you import `miniBen`. - -### Option 2 — export in the shell +**Shell:** ```bash export OPENROUTER_API_KEY="sk-or-v1-your-key-here" ``` -### Option 3 — interactive prompt - -```python -from miniBen import put_openrouter_api_key_into_env - -put_openrouter_api_key_into_env() # prompts only if the key is missing -``` - -### Check that the key is set +**Python:** ```python -from miniBen import check_openrouter_api_key_exist +from miniBen import check_openrouter_api_key_exist, put_openrouter_api_key_into_env -print(check_openrouter_api_key_exist()) # True if OPENROUTER_API_KEY is set +if not check_openrouter_api_key_exist(): + put_openrouter_api_key_into_env() ``` -Get a key from the [OpenRouter keys page](https://openrouter.ai/keys). Never commit your key to git. +Keys: [openrouter.ai/keys](https://openrouter.ai/keys). Never commit secrets. --- ## Quick start -### 1. Ask a model a single question +### One-off question ```python from miniBen import AIModel -model = AIModel("openrouter/free") # replace with any OpenRouter model ID +model = AIModel("openrouter/free") content, reasoning = model.ask("Say hello in one sentence.", reasoning=False) - -print("Answer:", content) -if reasoning: - print("Reasoning:", reasoning) +print(content) ``` -### 2. Run a full benchmark +### Full benchmark (5 API calls) ```python from miniBen import run_benchmark @@ -133,100 +117,110 @@ from miniBen import run_benchmark results = run_benchmark( model_name="openrouter/free", benchmark_name="creativity", - reasoning=True, + reasoning=False, ) -print(results["score"]) +print(results["num_calls"]) # 5 +print(results["outputs"][0]) # per-call detail +print(results["score"]) # None until scorer is implemented ``` -`run_benchmark` prints progress to the terminal and returns a dictionary (see [Return values](#return-values)). +### Command line (`example.py`) + +```bash +python -m miniBen.example --list +python -m miniBen.example --benchmark creativity --model openrouter/free +python -m miniBen.example --benchmark "cognitive flexibility" --reasoning +``` --- -## Available benchmarks +## Benchmarks -Use these exact strings as `benchmark_name` in `run_benchmark()`: +| `benchmark_name` | Display name | Calls | What changes each call | +|------------------|--------------|-------|-------------------------| +| `cognitive flexibility` | Meta-Chess Game (Cognitive Flexibility) | 5 | Trial 1 opening | +| `creativity` | Creativity in story writing | 5 | Required three words | -| Key | Display name | What it tests | -|-----|----------------|---------------| -| `cognitive flexibility` | Meta-Chess Game (Cognitive Flexibility) | Model must follow Meta-Chess rules and output structured move lists | -| `creativity` | Creativity in story writing | Short story using the words *stamp*, *letter*, *send* | +### Cognitive flexibility — openings +Defined in `COG_FLEX_OPENINGS` (`prompts.py`): -List keys programmatically: +| # | White | Black | +|---|-------|-------| +| 1 | e4 | e5 | +| 2 | e4 | c5 | +| 3 | d4 | d5 | +| 4 | e4 | e6 | +| 5 | d4 | c6 | -```python -from miniBen import BENCHMARKS +Each call uses `build_cog_flex_prompt(white, black)`. The model is asked for one Meta-Chess game as JSON between `<<>>` / `<<>>` (see `JSON_START`, `JSON_END` in `prompts.py`). -for key, meta in BENCHMARKS.items(): - print(key, "→", meta["name"]) -``` +### Creativity — word triplets -Additional prompt placeholders exist in `prompts.py` (`humor`, `bullshit`) but are not registered in `BENCHMARKS` yet. +Defined in `CREATIVITY_WORD_TRIPLETS`: ---- +| # | Words | +|---|-------| +| 1 | stamp, letter, send | +| 2 | kitchen, cook, food | +| 3 | bike, wheel, ride | +| 4 | book, read, story | +| 5 | study, laptop, desk | -## Usage guide +Each call uses `build_creativity_prompt(word1, word2, word3)` for one five-sentence story. -### `AIModel` +### Not registered yet -Wraps the OpenRouter chat API. +`humor` and `bullshit` exist as placeholders in `prompts.py` but are not in `BENCHMARKS`. -```python -from miniBen import AIModel +--- -model = AIModel(model="nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free") +## API reference -content, reasoning_text = model.ask( - prompt="How many r's are in the word strawberry?", - reasoning=True, # default: True -) -``` +### `AIModel` (`model.py`) -| Parameter | Type | Description | -|-----------|------|-------------| -| `model` (constructor) | `str` | OpenRouter model ID, e.g. `openrouter/free` | -| `prompt` | `str` | User message sent to the model | -| `reasoning` | `bool` | Passes `extra_body={"reasoning": {"enabled": ...}}` to the API | +| | | +|--|--| +| Constructor | `AIModel(model: str)` — OpenRouter model slug | +| Method | `ask(prompt: str, reasoning: bool = True) -> tuple[str, str \| None]` | +| Client | `base_url="https://openrouter.ai/api/v1"` | -**Returns:** `(content, reasoning_text)` — answer text and optional reasoning trace (may be `None`). +### `run_benchmark` (`runner.py`) -### `run_benchmark` +| Argument | Description | +|----------|-------------| +| `model_name` | OpenRouter model slug | +| `benchmark_name` | `cognitive flexibility` or `creativity` | +| `reasoning` | Passed to every `ask()` (default `True`) | -Runs prompt → model → parse → score for one benchmark. +Raises `KeyError` for unknown benchmarks. Raises `ValueError` if prompt and variant counts differ. -```python -from miniBen import run_benchmark +### Prompt helpers (`prompts.py`) -results = run_benchmark( - model_name="deepseek/deepseek-chat", - benchmark_name="cognitive flexibility", - reasoning=True, -) -``` +| Name | Role | +|------|------| +| `cog_flex_prompts()` | List of 5 Meta-Chess prompts | +| `creativity_prompts()` | List of 5 story prompts | +| `build_cog_flex_prompt(white, black)` | Single chess prompt | +| `build_creativity_prompt(w1, w2, w3)` | Single story prompt | +| `JSON_START`, `JSON_END` | Sentinel strings for chess JSON output | -| Parameter | Type | Description | -|-----------|------|-------------| -| `model_name` | `str` | OpenRouter model ID | -| `benchmark_name` | `str` | Key from `BENCHMARKS` (see table above) | -| `reasoning` | `bool` | Enable reasoning tokens when supported (default `True`) | +### Auth (`auth.py`) -### Using prompts directly +| Function | Returns | +|----------|---------| +| `check_openrouter_api_key_exist()` | `True` if key is set | +| `put_openrouter_api_key_into_env()` | Prompts and sets key if missing | + +### Registry (`runner.py`) ```python -from miniBen import AIModel, cog_flex, creativity +from miniBen import BENCHMARKS -model = AIModel("openrouter/free") -content, _ = model.ask(cog_flex, reasoning=True) +# Each entry: name, prompts(), variants(), parser, scorer ``` -### Auth helpers - -| Function | Purpose | -|----------|---------| -| `check_openrouter_api_key_exist()` | Returns `True` if `OPENROUTER_API_KEY` is set | -| `put_openrouter_api_key_into_env()` | Prompts for a key if missing and sets the env var | - --- ## Return values @@ -241,14 +235,31 @@ content, _ = model.ask(cog_flex, reasoning=True) ```python { - "model": str, # model_name you passed in - "benchmark": str, # benchmark_name you passed in - "raw_response": tuple, # (content, reasoning_text) from ask() - "parsed": ..., # output of the benchmark parser (stub → None for now) - "score": ..., # output of the benchmark scorer (stub → None for now) + "model": str, + "benchmark": str, + "num_calls": int, # len(prompts), typically 5 + "variants": list[dict], # per-call metadata + "parsed": list, # one entry per call (None if parser stub/fails) + "outputs": list[dict], # see below + "score": Any, # from scorer(parsed); None while stub } ``` +Each **`outputs`** item: + +```python +{ + "call": int, # 1 … num_calls + "variant": dict, # e.g. {"white": "e4", "black": "e5"} + # or {"words": ["stamp", "letter", "send"]} + "content": str, # model reply text + "reasoning": str | None, + "parsed": Any | None, +} +``` + +Parse failures are logged as warnings; that call’s `parsed` is `None` and the run continues. + --- ## Project layout @@ -256,75 +267,105 @@ content, _ = model.ask(cog_flex, reasoning=True) ``` mini_benchmarks/ ├── src/miniBen/ -│ ├── __init__.py # Public exports -│ ├── auth.py # API key helpers, load_dotenv -│ ├── model.py # AIModel, OpenRouter client -│ ├── prompts.py # Benchmark prompt strings -│ ├── parsers.py # Parse raw model text -│ ├── scorers.py # Score parsed output -│ ├── runner.py # BENCHMARKS registry, run_benchmark() -│ └── example.py # Re-exports (backward compatibility) +│ ├── __init__.py # Public exports +│ ├── auth.py # OPENROUTER_API_KEY, load_dotenv +│ ├── model.py # AIModel +│ ├── prompts.py # Variants, builders, JSON_START/END +│ ├── parsers.py # parse_chess_output, parse_creativity_output +│ ├── scorers.py # score_chess, score_creativity +│ ├── runner.py # BENCHMARKS, run_benchmark +│ └── example.py # CLI demo (--list, --benchmark, --model) ├── tests/ -│ ├── test_model.py # tests for miniBen.model (AIModel.ask) -│ └── test_runner.py # tests for miniBen.runner (run_benchmark) +│ ├── test_model.py +│ └── test_runner.py ├── pyproject.toml -├── README.md -└── .env # You create this; not in git +└── .env # you create this ``` -**Import style:** prefer `from miniBen import AIModel, run_benchmark`. Imports from `miniBen.example` still work but are equivalent to the package root. +**Import from:** `from miniBen import …` +**CLI:** `python -m miniBen.example` --- ## Development -### Run tests - -From the repository root: +### Tests ```bash -pip install -e ".[dev]" pytest ``` -Tests mock the OpenRouter client so they do not use your API key or network. +Mocks `OpenAI` — no API key or network needed. -### Add a new benchmark +### Add a benchmark -1. Add a prompt string in `src/miniBen/prompts.py`. -2. Implement `parse_*` and `score_*` in `parsers.py` and `scorers.py`. -3. Register an entry in `BENCHMARKS` inside `src/miniBen/runner.py`. -4. Export new symbols in `src/miniBen/__init__.py` if they should be public. +1. Add variant constants and a `*_prompts() -> list[str]` factory in `prompts.py`. +2. Implement `parse_*` and `score_*`. +3. Add a `BENCHMARKS` entry in `runner.py` with `prompts`, `variants`, `parser`, `scorer`. +4. Export from `__init__.py` if public. -### Choosing a model +### Models -Browse models on [OpenRouter](https://openrouter.ai/models). Use the model slug exactly as shown (e.g. `openrouter/free`, `anthropic/claude-3.5-sonnet`). Free and reasoning-capable models vary; if reasoning fails, try `reasoning=False` or another model. +Use slugs from [OpenRouter models](https://openrouter.ai/models). If reasoning breaks, pass `reasoning=False`. --- ## Troubleshooting -| Problem | What to try | -|---------|-------------| -| `OPENROUTER_API_KEY` errors / 401 | Set the key in `.env` or the shell; run `check_openrouter_api_key_exist()` | -| `KeyError` on `benchmark_name` | Use keys from [Available benchmarks](#available-benchmarks) exactly | -| `ValueError: returned a None content block` | Model refused or returned empty output; try another model or shorter prompt | -| `parsed` / `score` are always `None` | Expected until parsers and scorers are implemented | -| Import errors after clone | Run `pip install -e .` from the repo root | -| Rate limits | OpenRouter quota; wait or use a different model tier | +| Issue | Fix | +|-------|-----| +| 401 / missing key | Set `OPENROUTER_API_KEY` in `.env` or environment | +| `KeyError: Unknown benchmark` | Use `cognitive flexibility` or `creativity` exactly | +| `ValueError: None content block` | Try another model or `reasoning=False` | +| `parsed` / `score` always `None` | Implement parsers and scorers | +| `ImportError` | `pip install -e .` from repo root | +| Rate limits | Wait or switch model on OpenRouter | --- -## License +## Citations + +### This package -This project is licensed under the **MIT License** — see [LICENSE](LICENSE). +```bibtex +@software{yang2026miniben, + author = {Leyla Yang}, + title = {miniBen: Mini LLM Benchmarks}, + year = {2026}, + url = {https://github.com/Programming-The-Next-Step-2026/mini_benchmarks} +} +``` + +### Benchmark sources + +| Task | Credit | +|------|--------| +| Meta-Chess / cognitive flexibility | Original to this project | +| Creativity | Adapted from [creative-story-gen](https://github.com/mismayil/creative-story-gen) — [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) | +| Bullshit (planned) | [bullshit-benchmark](https://github.com/petergpt/bullshit-benchmark) (BullshitBench) | + +```bibtex +@misc{ismayilzada2024evaluatingcreativeshortstory, + title = {Evaluating Creative Short Story Generation in Humans and Large Language Models}, + author = {Mete Ismayilzada and Claire Stevenson and Lonneke van der Plas}, + year = {2024}, + eprint = {2411.02316}, + archivePrefix = {arXiv}, + primaryClass = {cs.CL}, + url = {https://arxiv.org/abs/2411.02316} +} +``` + +--- + +## License -When using upstream benchmarks, also follow the licenses of [creative-story-gen](https://github.com/mismayil/creative-story-gen) and [bullshit-benchmark](https://github.com/petergpt/bullshit-benchmark) (both are open source; check their repositories for the exact terms). +MIT — see [LICENSE](LICENSE). Upstream benchmark repos have their own licenses when you integrate them. --- ## Links -- **Repository:** [Programming-The-Next-Step-2026/mini_benchmarks](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) -- **Issues:** [GitHub Issues](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks/issues) -- **OpenRouter:** [openrouter.ai](https://openrouter.ai/) +- [Repository](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) +- [Issues](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks/issues) +- [OpenRouter](https://openrouter.ai/) diff --git a/pyproject.toml b/pyproject.toml index 36d4359..af965ae 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,12 +20,14 @@ classifiers = [ "Operating System :: OS Independent", ] dependencies = [ - "numpy", - "pandas", + "benepar", + "Tying", "chess", "spacy", "python-dotenv", "openai", + "sentence_transformers", + "en_core_web_sm" # more to be added ] diff --git a/src/help.py b/src/help.py index 135c318..33369ab 100644 --- a/src/help.py +++ b/src/help.py @@ -1,6 +1,6 @@ from tkinter.messagebox import askyesno import miniBen -from miniBen import AIModel +from miniBen import prompts, runner, model, auth + -help(AIModel) diff --git a/src/miniBen/__init__.py b/src/miniBen/__init__.py index 71d4936..3d1716e 100644 --- a/src/miniBen/__init__.py +++ b/src/miniBen/__init__.py @@ -1,18 +1,31 @@ from .auth import check_openrouter_api_key_exist, put_openrouter_api_key_into_env from .model import AIModel from .parsers import parse_chess_output, parse_creativity_output -from .prompts import bullshit, cog_flex, creativity, humor +from .prompts import ( + COG_FLEX_OPENINGS, + CREATIVITY_WORD_TRIPLETS, + JSON_END, + JSON_START, + build_cog_flex_prompt, + build_creativity_prompt, + cog_flex_prompts, + creativity_prompts, +) from .runner import BENCHMARKS, run_benchmark from .scorers import score_chess, score_creativity __all__ = [ "AIModel", "BENCHMARKS", - "bullshit", + "COG_FLEX_OPENINGS", + "CREATIVITY_WORD_TRIPLETS", + "JSON_END", + "JSON_START", + "build_cog_flex_prompt", + "build_creativity_prompt", "check_openrouter_api_key_exist", - "cog_flex", - "creativity", - "humor", + "cog_flex_prompts", + "creativity_prompts", "parse_chess_output", "parse_creativity_output", "put_openrouter_api_key_into_env", diff --git a/src/miniBen/example.py b/src/miniBen/example.py index cbe4dd9..78f777d 100644 --- a/src/miniBen/example.py +++ b/src/miniBen/example.py @@ -1,20 +1,59 @@ -"""Backward-compatible re-exports. Prefer ``from miniBen import ...``.""" +""" +Runnable demo and backward-compatible re-exports for miniBen. + +Preferred import style:: + + from miniBen import AIModel, run_benchmark, BENCHMARKS + +Run a benchmark from the terminal (needs OPENROUTER_API_KEY):: + + python -m miniBen.example --list + python -m miniBen.example --benchmark creativity --model openrouter/free + python -m miniBen.example --benchmark "cognitive flexibility" --reasoning +""" + +from __future__ import annotations + +import argparse +import sys from .auth import check_openrouter_api_key_exist, put_openrouter_api_key_into_env from .model import AIModel from .parsers import parse_chess_output, parse_creativity_output -from .prompts import bullshit, cog_flex, creativity, humor +from .prompts import ( + COG_FLEX_OPENINGS, + CREATIVITY_WORD_TRIPLETS, + JSON_END, + JSON_START, + build_cog_flex_prompt, + build_creativity_prompt, + cog_flex_prompts, + creativity_prompts, +) from .runner import BENCHMARKS, run_benchmark from .scorers import score_chess, score_creativity +# First prompt of each benchmark (lazy — avoids import-time prompt building) +def _first_cog_flex() -> str: + return cog_flex_prompts()[0] + + +def _first_creativity() -> str: + return creativity_prompts()[0] + + __all__ = [ "AIModel", "BENCHMARKS", - "bullshit", + "COG_FLEX_OPENINGS", + "CREATIVITY_WORD_TRIPLETS", + "JSON_END", + "JSON_START", + "build_cog_flex_prompt", + "build_creativity_prompt", "check_openrouter_api_key_exist", - "cog_flex", - "creativity", - "humor", + "cog_flex_prompts", + "creativity_prompts", "parse_chess_output", "parse_creativity_output", "put_openrouter_api_key_into_env", @@ -22,3 +61,71 @@ "score_chess", "score_creativity", ] + + +def list_benchmarks() -> None: + """Print registered benchmark keys and variant counts.""" + print("Registered benchmarks:\n") + for key, meta in BENCHMARKS.items(): + n = len(meta["prompts"]()) + print(f" {key!r}") + print(f" {meta['name']}") + print(f" {n} API calls per run\n") + + +def main(argv: list[str] | None = None) -> int: + """Run one benchmark or list available benchmarks.""" + parser = argparse.ArgumentParser( + description="Run miniBen LLM benchmarks via OpenRouter.", + ) + parser.add_argument( + "--list", + action="store_true", + help="List benchmark keys and exit", + ) + parser.add_argument( + "--benchmark", + choices=list(BENCHMARKS), + default="creativity", + help="Benchmark key (default: creativity)", + ) + parser.add_argument( + "--model", + default="openrouter/free", + help="OpenRouter model id (default: openrouter/free)", + ) + parser.add_argument( + "--reasoning", + action="store_true", + help="Enable reasoning tokens when the model supports them", + ) + args = parser.parse_args(argv) + + if args.list: + list_benchmarks() + return 0 + + if not check_openrouter_api_key_exist(): + put_openrouter_api_key_into_env() + + results = run_benchmark( + model_name=args.model, + benchmark_name=args.benchmark, + reasoning=args.reasoning, + ) + + print("\n=== Summary ===") + print(f"Model: {results['model']}") + print(f"Benchmark: {results['benchmark']}") + print(f"API calls: {results['num_calls']}") + print(f"Score: {results['score']}") + + for row in results["outputs"]: + status = "ok" if row["parsed"] is not None else "parse pending / failed" + print(f" Call {row['call']}: {row['variant']} — {status}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/miniBen/model.py b/src/miniBen/model.py index cb2ff89..b5d0732 100644 --- a/src/miniBen/model.py +++ b/src/miniBen/model.py @@ -1,5 +1,5 @@ import os - +import openai from openai import OpenAI @@ -18,44 +18,62 @@ def __init__(self, model: str): base_url="https://openrouter.ai/api/v1", api_key=os.getenv("OPENROUTER_API_KEY"), ) - def ask(self, prompt: str, reasoning: bool = True) -> tuple[str, str | None]: - """ - Submit a single-turn chat completion request to the configured model. + extra: dict = {} + + if not reasoning: + extra = {"reasoning": {"enabled": False}} + + try: + response = self.client.chat.completions.create( + model=self.model, + messages=[{"role": "user", "content": prompt}], + **({"extra_body": extra} if extra else {}), + ) + + except Exception as e: + # Retry without reasoning if unsupported + if "reasoning" in str(e).lower() and extra: + response = self.client.chat.completions.create( + model=self.model, + messages=[{"role": "user", "content": prompt}], + ) + else: + raise + + # ---- Defensive validation ---- + + if response is None: + raise ValueError("API returned None response.") + + if not hasattr(response, "choices") or response.choices is None: + raise ValueError( + f"Model '{self.model}' returned no choices.\n" + f"Full response: {response}" + ) + + if len(response.choices) == 0: + raise ValueError( + f"Model '{self.model}' returned empty choices.\n" + f"Full response: {response}" + ) - Parameters - ---------- - prompt : str - The user-facing instruction string to send to the model. - reasoning : bool, optional - Enables chain-of-thought reasoning tokens on supported models. - Defaults to ``True``. - - Returns - ------- - tuple[str, str | None] - ``(content, reasoning_text)`` from the first choice returned by the API. - """ - response = self.client.chat.completions.create( - model=self.model, - messages=[ - { - "role": "user", - "content": prompt, - } - ], - extra_body={"reasoning": {"enabled": reasoning}}, - ) message = response.choices[0].message - content: str | None = message.content + + if message is None: + raise ValueError( + f"Model '{self.model}' returned a choice with no message.\n" + f"Full response: {response}" + ) + + content = getattr(message, "content", None) if content is None: raise ValueError( - f"Model '{self.model}' returned a None content block. " - "The model may have refused the request, exceeded its context, " - "or does not support text output. Consider switching models." + f"Model '{self.model}' returned None content.\n" + f"Message: {message}" ) - reasoning_text: str | None = getattr(message, "reasoning", None) + reasoning_text = getattr(message, "reasoning", None) - return content, reasoning_text + return content, reasoning_text \ No newline at end of file diff --git a/src/miniBen/parsers.py b/src/miniBen/parsers.py index 8448f7c..2df3cb6 100644 --- a/src/miniBen/parsers.py +++ b/src/miniBen/parsers.py @@ -1,11 +1,95 @@ +import ast +import json +import re +from typing import Any + +from .prompts import JSON_START, JSON_END + + # ============================================================================== -# PARSING FUNCTIONS +# COGNITIVE FLEXIBILITY PARSER # ============================================================================== +def parse_chess_output(raw: str) -> list[list[str]] | None: + """ + Extract a 6x8 move matrix from model output. + + Tries two strategies in order: + 1. JSON sentinel extraction (<<>> ... <<>>) + 2. Python list literal extraction (fallback for list-of-lists format) + + Returns: + List of 6 lists of 8 SAN strings, or None if both strategies fail. + """ + # --- Strategy 1: JSON sentinels --- + pattern = re.compile( + re.escape(JSON_START) + r"\s*(.*?)\s*" + re.escape(JSON_END), + re.DOTALL, + ) + match = pattern.search(raw) + if match: + try: + data = json.loads(match.group(1)) + trials = data["trials"] + if _valid_chess_matrix(trials): + return trials + except (json.JSONDecodeError, KeyError): + pass + + # --- Strategy 2: Python list literal --- + match = re.search(r"(\[\s*\[.*?\]\s*\])", raw, re.DOTALL) + if match: + try: + trials = ast.literal_eval(match.group(1)) + if _valid_chess_matrix(trials): + return trials + except (ValueError, SyntaxError): + pass + + return None + + +def _valid_chess_matrix(trials: Any) -> bool: + """Return True if trials is exactly 6 lists of 8 strings.""" + return ( + isinstance(trials, list) + and len(trials) == 6 + and all( + isinstance(trial, list) + and len(trial) == 8 + and all(isinstance(move, str) for move in trial) + for trial in trials + ) + ) + + +# ============================================================================== +# CREATIVITY PARSER +# ============================================================================== + +def parse_creativity_output(raw: str) -> dict[str, Any] | None: + """ + Extract the story text from model output. + + The creativity prompt requests plain prose with no structured format, + so this parser strips whitespace and returns the raw text plus a + sentence count for the scorer. -def parse_chess_output(raw_str: str): - pass + Returns: + Dict with keys 'story' (str) and 'sentence_count' (int), or None if empty. + """ + text = raw.strip() + if not text: + return None + # Split on sentence-ending punctuation followed by whitespace + sentences = [ + s.strip() + for s in re.split(r"(?<=[.!?])\s+", text) + if s.strip() + ] -def parse_creativity_output(raw_str: str): - pass + return { + "story": text, + "sentence_count": len(sentences), + } \ No newline at end of file diff --git a/src/miniBen/prompts.py b/src/miniBen/prompts.py index 53ddd2e..c045701 100644 --- a/src/miniBen/prompts.py +++ b/src/miniBen/prompts.py @@ -2,7 +2,28 @@ # PROMPTS # ============================================================================== -cog_flex = """You are a professional chess player. Generate 5 Meta Chess games. +JSON_START = "<<>>" +JSON_END = "<<>>" + +# Five opening pairs for Meta-Chess (Trial 1, moves 1–2), one per API call. +COG_FLEX_OPENINGS: list[tuple[str, str]] = [ + ("e4", "e5"), + ("e4", "c5"), + ("d4", "d5"), + ("e4", "e6"), + ("d4", "c6"), +] + +# Five three-word sets for the creativity benchmark, one per API call. +CREATIVITY_WORD_TRIPLETS: list[tuple[str, str, str]] = [ + ("stamp", "letter", "send"), + ("kitchen", "cook", "food"), + ("bike", "wheel", "ride"), + ("book", "read", "story"), + ("study", "laptop", "desk"), +] + +_COG_FLEX_RULES = """You are a professional chess player. Generate 1 Meta Chess game. META CHESS RULES: Each game has 6 trials. In each trial, ONLY the listed piece type may move (including captures). All other pieces are frozen. @@ -16,49 +37,98 @@ Both sides must retain at least one of the upcoming trial's piece type before that trial begins. Plan all 6 trials ahead to guarantee this. REQUIRED OPENING MOVES (Trial 1, moves 1-2): - Game 1: e4, e5 | Game 2: e4, c5 | Game 3: d4, d5 | Game 4: e4, e6 | Game 5: d4, c6 + White: {white} | Black: {black} OUTPUT RULES — VIOLATIONS INVALIDATE THE RESPONSE: -1. Output ONLY a raw Python list-of-lists-of-lists. No prose, no markdown, no comments. -2. Outer list = 5 games. Each game = exactly 6 inner lists. Each inner list = exactly 8 SAN strings. +1. Output ONLY a raw Python list-of-lists. No prose, no markdown, no comments. +2. Exactly 6 inner lists (one per trial). Each inner list = exactly 8 SAN strings. 3. SAN requirements: captures use 'x' (e.g. 'fxg5', 'Nxe4'); castling uses 'O-O'/'O-O-O' (capital O); check = '+'; checkmate = '#'; no move numbers or annotations. 4. Each trial's moves must be legal given that trial's piece restriction and the board state inherited from prior trials. 5. Do NOT reproduce the example moves below. -6. Separate each game block with a blank line. Each trial on its own line. +6. Put each trial on its own line. -REQUIRED FORMAT (5 games; skeleton shown for 2): +REQUIRED FORMAT (1 game; skeleton): [['W1','B1','W2','B2','W3','B3','W4','B4'], ['W1','B1','W2','B2','W3','B3','W4','B4'], ['W1','B1','W2','B2','W3','B3','W4','B4'], ['W1','B1','W2','B2','W3','B3','W4','B4'], ['W1','B1','W2','B2','W3','B3','W4','B4'], - ['W1','B1','W2','B2','W3','B3','W4','B4']], - -[['W1','B1','W2','B2','W3','B3','W4','B4'], - ... ['W1','B1','W2','B2','W3','B3','W4','B4']] -REFERENCE EXAMPLE (structure only — do not reuse these moves): -[['e4','e5','f4','f5','g4','g5','fxg5','fxg4'], - ['Nc3','Nc6','Nf3','Nf6','Nxe4','Nxe5','Nxe5','Ng4'], - ['Bc4','Bc5','Bd5','Bd6','Bxc6','Bxc6','Bb5','Bb4'], - ['O-O','O-O','Rfe1','Rfe8','Rxe8+','Rxe8','Rd1','Rd8'], - ['Qf3','Qf6','Qxf6+','Rxf6','Qd3','Rf8','Qd8+','Kf7'], - ['Kg1','Kg8','Kf1','Kh8','Kg2','Kg7','f4','f5']], - -[['e4','c5','d4','cxd4','c3','d5','cxd4','dxe4'], - ['Nf3','Nc6','Nc3','Nf6','Nb5','Nd4','Nxd4','Nxd4'], - ['Bb5','Bb4','Bxc6','Bxc3+','Bxb7','Bxa2','Bc6','Bxb1'], - ['O-O','O-O','Rfe1','Rac8','Rxe8+','Rxe8','Ra1','Rb8'], - ['Qd3','Qf6','Qxf6+','Rxf6','Qd4','Rf8','Qg4','Rf6'], - ['Kf1','Kg8','Kg1','Kh8','Kh1','Kg8','h4','h5']] +2. The sentinel tokens {json_start} and {json_end} must be the FIRST and LAST + non-whitespace content in your reply. Any reasoning, commentary, or + explanation must be entirely absent — not before, not after, not between + any part of the output. +3. The JSON object has exactly one key: "trials". +4. "trials" is an array of exactly 6 arrays. +5. Each inner array contains exactly 8 strings (SAN moves, alternating + White/Black). +6. SAN encoding: captures use 'x' (e.g. 'fxg5'); castling = 'O-O' or 'O-O-O' + (capital letter O, not zero); check = '+'; checkmate = '#'. + No move numbers. No annotations. +7. Each trial's moves must be legal given that trial's piece restriction and + the inherited board state. + +REQUIRED SCHEMA (inside sentinels): +{{ + "trials": [ + ["W1","B1","W2","B2","W3","B3","W4","B4"], + ["W1","B1","W2","B2","W3","B3","W4","B4"], + ["W1","B1","W2","B2","W3","B3","W4","B4"], + ["W1","B1","W2","B2","W3","B3","W4","B4"], + ["W1","B1","W2","B2","W3","B3","W4","B4"], + ["W1","B1","W2","B2","W3","B3","W4","B4"] + ] +}} + +REFERENCE EXAMPLE (structure only): +{json_start} +{{ + "trials": [ + ["e4","e5","f4","f5","g4","g5","fxg5","fxg4"], + ["Nc3","Nc6","Nf3","Nf6","Nxe4","Nxe5","Nd6","Nb4"], + ["Bc4","Bc5","Bd5","Bd6","Bxc6","bxc6","Bb5","Bb4"], + ["O-O","O-O","Rfe1","Rfe8","Rxe8+","Rxe8","Rd1","Rd8"], + ["Qf3","Qf6","Qxf6+","gxf6","Qd3","Qd6","Qe4","Qe5"], + ["Kg1","Kg8","Kf1","Kh8","Kg2","Kg7","Kh3","Kh6"] + ] +}} +{json_end} """ -creativity = ( - "Write a creative story within five sentences that must includes " - "three words: stamp, letter, send" -) -humor = "" +def build_cog_flex_prompt(white: str, black: str) -> str: + """Build a single-game Meta-Chess prompt with the given Trial 1 opening.""" + return _COG_FLEX_RULES.format( + white=white, + black=black, + json_start=JSON_START, + json_end=JSON_END, + ) + + +def build_creativity_prompt(word1: str, word2: str, word3: str) -> str: + """Build a single-story creativity prompt with the given three required words.""" + return ( + "You are a professional story writer. Generate one creative short story.\n" + "Creative short story task rule: Write a creative story within five sentences " + f"that must include these three words: {word1}, {word2}, {word3}." + ) + + +def cog_flex_prompts() -> list[str]: + """Return five cog_flex prompts, each with a different required opening.""" + return [ + build_cog_flex_prompt(white, black) + for white, black in COG_FLEX_OPENINGS + ] + + +def creativity_prompts() -> list[str]: + """Return five prompts, each with a different three-word set.""" + return [ + build_creativity_prompt(w1, w2, w3) + for w1, w2, w3 in CREATIVITY_WORD_TRIPLETS + ] + -bullshit = "" diff --git a/src/miniBen/runner.py b/src/miniBen/runner.py index 75f46cf..9e1ccf4 100644 --- a/src/miniBen/runner.py +++ b/src/miniBen/runner.py @@ -2,19 +2,31 @@ from .model import AIModel from .parsers import parse_chess_output, parse_creativity_output -from .prompts import cog_flex, creativity +from .prompts import ( + COG_FLEX_OPENINGS, + CREATIVITY_WORD_TRIPLETS, + cog_flex_prompts, + creativity_prompts, +) from .scorers import score_chess, score_creativity + BENCHMARKS = { "cognitive flexibility": { "name": "Meta-Chess Game (Cognitive Flexibility)", - "prompt": cog_flex, + "prompts": cog_flex_prompts, + "variants": lambda: [ + {"white": w, "black": b} for w, b in COG_FLEX_OPENINGS + ], "parser": parse_chess_output, "scorer": score_chess, }, "creativity": { "name": "Creativity in story writing", - "prompt": creativity, + "prompts": creativity_prompts, + "variants": lambda: [ + {"words": [w1, w2, w3]} for w1, w2, w3 in CREATIVITY_WORD_TRIPLETS + ], "parser": parse_creativity_output, "scorer": score_creativity, }, @@ -22,49 +34,84 @@ def run_benchmark( - model_name: str, benchmark_name: str, reasoning: bool = True + model_name: str, + benchmark_name: str, + reasoning: bool = True, ) -> dict: """ - Return model response and score for the given benchmark and model name. + Run a benchmark across N API calls (one per prompt variant). Parameters ---------- model_name : str + Identifier passed to AIModel. benchmark_name : str + Must be a key in BENCHMARKS. reasoning : bool + Whether to request reasoning traces from the model. Returns ------- dict - Keys: model, benchmark, raw_response, parsed, score + Keys: model, benchmark, num_calls, variants, parsed, outputs, score. """ - model = AIModel(model_name) + if benchmark_name not in BENCHMARKS: + raise KeyError( + f"Unknown benchmark {benchmark_name!r}. " + f"Choose from: {list(BENCHMARKS)}" + ) + benchmark = BENCHMARKS[benchmark_name] - print(f"\nRunning benchmark: {benchmark['name']} ...") + prompts = benchmark["prompts"]() + variants = benchmark["variants"]() - content, reasoning_text = model.ask( - prompt=benchmark["prompt"], reasoning=reasoning - ) - raw_response = (content, reasoning_text) - print("\n--- Raw Response ---") - print(raw_response) + if len(prompts) != len(variants): + raise ValueError( + f"Prompt count ({len(prompts)}) and variant count " + f"({len(variants)}) must match." + ) + num_calls = len(prompts) + model = AIModel(model_name) parser = benchmark["parser"] scorer = benchmark["scorer"] - print("\n--- Parsed Output ---") - parsed = parser(content) - pprint(parsed) + print(f"\nRunning benchmark: {benchmark['name']} ({num_calls} API calls) ...") + + parsed_list: list = [] + outputs: list[dict] = [] + score_list: list = [] + + for i, (prompt, variant) in enumerate(zip(prompts, variants), start=1): + print(f"\n--- API call {i}/{num_calls} — variant: {variant} ---") + content, reasoning_text = model.ask(prompt=prompt, reasoning=reasoning) - print("\n--- Score ---") - score = scorer(parsed) - pprint(score) + print("\n--- Raw Response ---") + print(content) + + print("\n--- Parsed Output ---") + try: + parsed = parser(content) + except Exception as e: + print(f" [WARNING] Parse failed on call {i}: {e}") + parsed = None + + parsed_list.append(parsed) + pprint(parsed) + + print("\n--- Total Score (all calls) ---") + call_score = scorer(parsed) + score_list.append(call_score) + print(call_score) return { - "model": model_name, - "benchmark": benchmark_name, - "raw_response": raw_response, - "parsed": parsed, - "score": score, + "model": model_name, + "benchmark": benchmark_name, + "num_calls": num_calls, + "variants": variants, + "parsed": parsed_list, + "outputs": outputs, + "score": call_score, } + diff --git a/src/miniBen/scorers.py b/src/miniBen/scorers.py index 7635943..acb74a0 100644 --- a/src/miniBen/scorers.py +++ b/src/miniBen/scorers.py @@ -1,11 +1,281 @@ +import re +from collections import Counter +from statistics import mean +from functools import lru_cache + +import spacy +from numpy.core.multiarray import item +from sentence_transformers import SentenceTransformer +from sentence_transformers.util import cos_sim, dot_score, euclidean_sim, manhattan_sim + +import chess +from typing import Any # ============================================================================== # SCORING FUNCTIONS # ============================================================================== +TRIAL_PIECE_MAP: dict[int, chess.PieceType] = { + 0: chess.PAWN, + 1: chess.KNIGHT, + 2: chess.BISHOP, + 3: chess.ROOK, + 4: chess.QUEEN, + 5: chess.KING, +} + +EXPECTED_TRIALS = 6 +EXPECTED_MOVES_PER_TRIAL = 8 + + +def score_chess(parsed_list: list[list[list[str]] | None]) -> dict[str, Any]: + """ + Score a list of parsed Meta-Chess games for move legality. + + Checks three violation types per move: + - Parse error: move string is not valid SAN + - Wrong piece: move does not involve the trial's required piece type + - Illegal move: move is not legal given the current board state + + Parameters + ---------- + parsed_list : list + Output of parse_chess_output() for each API call. + Each entry is either a 6x8 list of SAN strings or None. + + Returns + ------- + dict with keys: + games : per-game breakdown (violations by trial) + total_moves : total half-moves evaluated + total_violations : total illegal/wrong-piece moves across all games + compliance_rate : float in [0, 1], proportion of legal correct moves + """ + games_data = [] + total_moves = 0 + total_violations = 0 + + def _score_single_game( + game_idx: int, + trials: list[list[str]] | None, + ) -> dict[str, Any]: + """Score one parsed game. Returns structured result dict.""" + + # --- Handle parse failure or wrong structure --- + if trials is None: + return { + "game_index": game_idx + 1, + "error": "parse_failed", + "trials": [], + "total_moves": 0, + "total_violations": 0, + } + + board = chess.Board() + trial_results = [] + + for trial_idx, moves in enumerate(trials): + expected_piece = TRIAL_PIECE_MAP.get(trial_idx) + piece_name = chess.piece_name(expected_piece).capitalize() + move_results = [] + + wrong_length = len(moves) != EXPECTED_MOVES_PER_TRIAL + + for move_idx, move_str in enumerate(moves): + color = "White" if board.turn == chess.WHITE else "Black" + violation = None + + try: + move = board.parse_san(move_str) + + piece = board.piece_at(move.from_square) + if piece is None or piece.piece_type != expected_piece: + violation = f"wrong_piece (expected {piece_name})" + elif move not in board.legal_moves: + violation = "illegal_move" + else: + board.push(move) + + except Exception as e: + violation = f"parse_error: {e}" + + move_results.append({ + "move_number": move_idx + 1, + "color": color, + "san": move_str, + "violation": violation, + }) + + trial_violations = sum(1 for m in move_results if m["violation"]) + trial_results.append({ + "trial_index": trial_idx + 1, + "expected_piece": piece_name, + "wrong_length": wrong_length, + "actual_length": len(moves), + "moves": move_results, + "trial_violations": trial_violations, + }) + + total_moves = sum(t["actual_length"] for t in trial_results) + total_violations = sum(t["trial_violations"] for t in trial_results) + final_score = total_violations / total_moves + + return { + "game_index": game_idx + 1, + "total_moves": total_moves, + "total_violations": total_violations, + "final_score": final_score, + } + + for game_idx, trials in enumerate(parsed_list): + game_result = _score_single_game(game_idx, trials) + games_data.append(game_result) + total_moves += game_result["total_moves"] + total_violations += game_result["total_violations"] + + compliance_rate = ( + round(1 - total_violations / total_moves, 4) + if total_moves > 0 else 0.0 + ) + + return { + "games": games_data, + "total_moves": total_moves, + "total_violations": total_violations, + "compliance_rate": compliance_rate, + } + + +# ============================================================================== +# Creativity SCORING FUNCTIONS +# ============================================================================== + +# ── Defaults ────────────────────────────────────────────────────────────────── +DEF_EMB_MODEL = "thenlper/gte-large" +DEF_EMB_TYPE = "sentence_embedding" +DEF_DIST_FN = "cosine" +DEF_SPACY_LANG = "en_core_web_sm" +DEF_PREPROCESSING = dict( + lower=True, remove_punct=True, remove_stopwords=True, + lemmatize=True, dominant_k=None, unique=True, +) + +# ── Cached loaders ──────────────────────────────────────────────────────────── +@lru_cache(maxsize=4) +def load_spacy_engine(language=DEF_SPACY_LANG, + include_syllables=False, + include_constituency=False): + engine = spacy.load(language) + if include_syllables: + engine.add_pipe("syllables", after="tagger") + if include_constituency: + engine.add_pipe("benepar", config={"model": "benepar_en3"}) + return engine + + +@lru_cache(maxsize=4) +def load_emb_model(model=DEF_EMB_MODEL): + return SentenceTransformer(model) + + +# ── Core NLP helpers ────────────────────────────────────────────────────────── +@lru_cache(maxsize=2048) +def _spacy_doc(text): + return load_spacy_engine()(text) + + +def get_sentences(text: str) -> list[str]: + return [s.text for s in _spacy_doc(text).sents] + + +def get_words(text: str, *, lower=True, remove_punct=True, + remove_stopwords=True, lemmatize=True, + unique=True, dominant_k=None) -> list[str]: + tokens = _spacy_doc(text) + if remove_punct: + tokens = [t for t in tokens if not t.is_punct] + if remove_stopwords: + tokens = [t for t in tokens if not t.is_stop] + + words = [t.lemma_ if lemmatize else t.text for t in tokens] + if lower: + words = [w.lower() for w in words] + + if dominant_k: + return [w for w, _ in Counter(words).most_common(dominant_k)] + return list(dict.fromkeys(words)) if unique else words # preserves order + + +# ── Embedding helpers ───────────────────────────────────────────────────────── +@lru_cache(maxsize=4096) +def get_embedding(text, model=DEF_EMB_MODEL, emb_type=DEF_EMB_TYPE): + output_value = "token_embeddings" if "token" in emb_type else "sentence_embedding" + emb = load_emb_model(model).encode(text, output_value=output_value) + return emb.mean(axis=0) if emb.ndim == 2 else emb + + +def _distance(emb1, emb2, fn=DEF_DIST_FN) -> float: + match fn: + case "cosine": return (1 - cos_sim(emb1, emb2)).item() + case "dot": return (1 - dot_score(emb1, emb2)).item() + case "euclidean": return (-euclidean_sim(emb1, emb2)).item() + case "manhattan": return (-manhattan_sim(emb1, emb2)).item() + case _: raise ValueError(f"Unknown distance function: {fn}") + + +# ── Semantic distance ───────────────────────────────────────────────────────── +def avg_sem_dis(text: str, model=DEF_EMB_MODEL, + emb_type=DEF_EMB_TYPE, dist_fn=DEF_DIST_FN, + preprocessing=DEF_PREPROCESSING) -> float: + """Mean pairwise semantic distance across all unique words in *text*.""" + words = get_words(text, **preprocessing) + if len(words) < 2: + return 0.0 + embs = [get_embedding(w, model, emb_type) for w in words] + n = len(embs) + total = sum( + _distance(embs[i], embs[j], dist_fn) + for i in range(n) for j in range(n) if i != j + ) + return total / (n * (n - 1)) + + +# ── Novelty & Surprise ──────────────────────────────────────────────────────── +def compute_novelty(texts: list[str], model=DEF_EMB_MODEL, + emb_type=DEF_EMB_TYPE, dist_fn=DEF_DIST_FN, + preprocessing=DEF_PREPROCESSING) -> list[float]: + """How semantically distinct each text is from the overall corpus.""" + kw = dict(model=model, emb_type=emb_type, dist_fn=dist_fn, preprocessing=preprocessing) + corpus_dis = avg_sem_dis("".join(texts), **kw) + return [2 * abs(avg_sem_dis(t, **kw) - corpus_dis) for t in texts] + + +def compute_surprise(text: str, model=DEF_EMB_MODEL, + emb_type=DEF_EMB_TYPE, dist_fn=DEF_DIST_FN, + preprocessing=DEF_PREPROCESSING) -> tuple[float, list[float], float]: + """Sentence-to-sentence semantic shifts within a single text.""" + sentences = get_sentences(text) + if len(sentences) < 2: + return 0.0, [], 0.0 + + kw = dict(model=model, emb_type=emb_type, dist_fn=dist_fn, preprocessing=preprocessing) + dists = [avg_sem_dis(s, **kw) for s in sentences] + deltas = [abs(dists[i] - dists[i - 1]) for i in range(1, len(dists))] + + score = (2 / len(deltas)) * sum(deltas) + return score, deltas, max(deltas) + -def score_chess(parsed_chess): - pass +# ── Final scorer ────────────────────────────────────────────────────────────── +def score_creativity(parsed_creativity: list[dict], + novelty_weight=0.6, surprise_weight=0.4) -> float: + """ + Args: + parsed_creativity: list of dicts with at least a 'story' key. + """ + text = parsed_creativity.items("story") + combined_text = " ".join(text) + novelty = mean(compute_novelty(text)) + surprise, _, _ = compute_surprise(text) -def score_creativity(parsed_creativity): - pass + return novelty_weight * novelty + surprise_weight * surprise \ No newline at end of file diff --git a/tests/test_runner.py b/tests/test_runner.py index 2d822c3..9beec53 100644 --- a/tests/test_runner.py +++ b/tests/test_runner.py @@ -1,11 +1,12 @@ from unittest.mock import patch from miniBen.model import AIModel +from miniBen.prompts import COG_FLEX_OPENINGS, CREATIVITY_WORD_TRIPLETS from miniBen.runner import run_benchmark -def test_run_benchmark_returns_dict(): - with patch.object(AIModel, "ask", return_value=("fake response", None)): +def test_run_benchmark_makes_five_api_calls_for_creativity(): + with patch.object(AIModel, "ask", return_value=("fake response", None)) as mock_ask: with patch("miniBen.model.OpenAI"): result = run_benchmark( model_name="deepseek/deepseek-v4-flash:free", @@ -13,7 +14,42 @@ def test_run_benchmark_returns_dict(): reasoning=False, ) - assert isinstance(result, dict) - assert result["model"] == "deepseek/deepseek-v4-flash:free" - assert result["benchmark"] == "creativity" - assert result["raw_response"] == ("fake response", None) + assert result["num_calls"] == 5 + assert len(result["parsed"]) == 5 + assert len(result["outputs"]) == 5 + assert mock_ask.call_count == 5 + + +def test_run_benchmark_makes_five_api_calls_for_cognitive_flexibility(): + with patch.object(AIModel, "ask", return_value=("fake chess", None)) as mock_ask: + with patch("miniBen.model.OpenAI"): + result = run_benchmark( + model_name="deepseek/deepseek-v4-flash:free", + benchmark_name="cognitive flexibility", + reasoning=False, + ) + + assert result["num_calls"] == 5 + assert mock_ask.call_count == 5 + assert result["variants"][0] == {"white": "e4", "black": "e5"} + + +def test_cog_flex_prompts_match_openings(): + from miniBen.prompts import JSON_START, cog_flex_prompts + + prompts = cog_flex_prompts() + assert len(prompts) == len(COG_FLEX_OPENINGS) + for prompt, (white, black) in zip(prompts, COG_FLEX_OPENINGS): + assert white in prompt + assert black in prompt + assert JSON_START in prompt + + +def test_creativity_prompts_match_word_triplets(): + from miniBen.prompts import creativity_prompts + + prompts = creativity_prompts() + assert len(prompts) == len(CREATIVITY_WORD_TRIPLETS) + for prompt, words in zip(prompts, CREATIVITY_WORD_TRIPLETS): + for word in words: + assert word in prompt From ecd98c9bb295f86db3c6b6961dbf8b9afbf75f11 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Thu, 21 May 2026 19:53:03 +0200 Subject: [PATCH 2/9] Refine runner.py --- pyproject.toml | 1 - src/miniBen/runner.py | 121 +++++++++++++++++++++++++++++++++++++----- 2 files changed, 108 insertions(+), 14 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index af965ae..5baa6b2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,7 +21,6 @@ classifiers = [ ] dependencies = [ "benepar", - "Tying", "chess", "spacy", "python-dotenv", diff --git a/src/miniBen/runner.py b/src/miniBen/runner.py index 9e1ccf4..58f8df7 100644 --- a/src/miniBen/runner.py +++ b/src/miniBen/runner.py @@ -83,35 +83,130 @@ def run_benchmark( outputs: list[dict] = [] score_list: list = [] + successful_calls = 0 + failed_calls = 0 + for i, (prompt, variant) in enumerate(zip(prompts, variants), start=1): + print(f"\n--- API call {i}/{num_calls} — variant: {variant} ---") - content, reasoning_text = model.ask(prompt=prompt, reasoning=reasoning) + + # ------------------------- + # API CALL + # ------------------------- + try: + content, reasoning_text = model.ask( + prompt=prompt, + reasoning=reasoning, + ) + + except Exception as e: + failed_calls += 1 + + print( + f"[WARNING] API call {i} failed. " + f"Skipping variant.\n" + f"Reason: {type(e).__name__}: {e}" + ) + + outputs.append( + { + "variant": variant, + "success": False, + "error": str(e), + "content": None, + "reasoning": None, + } + ) + + continue + + # ------------------------- + # EMPTY / INVALID RESPONSE + # ------------------------- + if not content or not isinstance(content, str): + failed_calls += 1 + + print( + f"[WARNING] API call {i} returned " + f"empty or invalid content. Skipping." + ) + + outputs.append( + { + "variant": variant, + "success": False, + "error": "Empty or invalid response content", + "content": content, + "reasoning": reasoning_text, + } + ) + + continue + + successful_calls += 1 print("\n--- Raw Response ---") print(content) + # ------------------------- + # PARSING + # ------------------------- print("\n--- Parsed Output ---") + try: parsed = parser(content) + except Exception as e: - print(f" [WARNING] Parse failed on call {i}: {e}") + + print(f"[WARNING] Parse failed on call {i}: {e}") + parsed = None parsed_list.append(parsed) + + outputs.append( + { + "variant": variant, + "success": True, + "error": None, + "content": content, + "reasoning": reasoning_text, + "parsed": parsed, + } + ) + pprint(parsed) - print("\n--- Total Score (all calls) ---") - call_score = scorer(parsed) + # ------------------------- + # SCORING + # ------------------------- + print("\n--- Total Score (valid calls only) ---") + + try: + call_score = scorer(parsed_list) + + except Exception as e: + + print(f"[WARNING] Scoring failed: {e}") + + call_score = None + score_list.append(call_score) + print(call_score) - return { - "model": model_name, - "benchmark": benchmark_name, - "num_calls": num_calls, - "variants": variants, - "parsed": parsed_list, - "outputs": outputs, - "score": call_score, - } + print("\n--- Benchmark Summary ---") + print(f"Successful calls: {successful_calls}") + print(f"Failed/skipped calls: {failed_calls}") + return { + "model": model_name, + "benchmark": benchmark_name, + "num_calls": num_calls, + "successful_calls": successful_calls, + "failed_calls": failed_calls, + "variants": variants, + "parsed": parsed_list, + "outputs": outputs, + "score": call_score, + } \ No newline at end of file From 10305ac6cf0c15973d2e63360093c85e19c067eb Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Thu, 21 May 2026 19:59:26 +0200 Subject: [PATCH 3/9] Update functions --- .DS_Store | Bin 12292 -> 12292 bytes src/miniBen/scorers.py | 1 + 2 files changed, 1 insertion(+) diff --git a/.DS_Store b/.DS_Store index 64a71a9d1ff59b22cebc46ea5914fe482f4842ea..666c2bcdcf3f372b8ecf4c32c4f5aa550b8583b9 100644 GIT binary patch delta 400 zcmZokXi3=M$j;0->FQ<|_6Hoc49v^#A9rM6U|?n_W+-Av&dpD8Ny^Dj0*Y{eL{Vi6 zG7N*0^K%P;G7Jn%3pOVTure}5{@I)%7{tbO>;L3E;z=qgW5VWt;u;+F z(HTG?76v_rbcRfZlH7b3Bnuc``$Mcpl1Df|z--NPba@mfuqyzKW?%)Hoym~Gkjjvl zlaAy-261tqEYL-8X=KZ1@7%H8_uk|~3VBS-CX=Tr%1sVXs$> Du!d@X diff --git a/src/miniBen/scorers.py b/src/miniBen/scorers.py index acb74a0..046197e 100644 --- a/src/miniBen/scorers.py +++ b/src/miniBen/scorers.py @@ -274,6 +274,7 @@ def score_creativity(parsed_creativity: list[dict], """ text = parsed_creativity.items("story") combined_text = " ".join(text) + ###TODO: novelty should have one score for all stories novelty = mean(compute_novelty(text)) surprise, _, _ = compute_surprise(text) From baaa63df34afed2f54be492539b96d42a8d06759 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Thu, 21 May 2026 20:19:54 +0200 Subject: [PATCH 4/9] Update .DS_Store --- .DS_Store | Bin 12292 -> 12292 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/.DS_Store b/.DS_Store index 666c2bcdcf3f372b8ecf4c32c4f5aa550b8583b9..50b59b44f745d5cd3305c3ab7d47fdfca742355b 100644 GIT binary patch delta 140 zcmZokXi3SkzUbWHZ!2RQnsPZXzSle Date: Thu, 21 May 2026 20:52:32 +0200 Subject: [PATCH 5/9] Update runner.py --- .DS_Store | Bin 12292 -> 12292 bytes src/miniBen/runner.py | 5 +++++ src/miniBen/scorers.py | 45 ++++++++++++++++++++++++----------------- 3 files changed, 32 insertions(+), 18 deletions(-) diff --git a/.DS_Store b/.DS_Store index 50b59b44f745d5cd3305c3ab7d47fdfca742355b..65540cb84e6cc2a4267a6ffa9c23be67131edbc9 100644 GIT binary patch delta 393 zcmZokXi1phC7poML+i_$-)>9;1kup96&FfcFzF%WQYZ&s8LXW6XI^F+8_94Nrf zpvRERP|8rmP|T3eP?VEy7@VA+TL4xJA<$Ih=DWBg<>V)U46_O^?LU0~xFe=Hx%nvs zRS7W^6=a}VHoP zlJ$(AuK*=j81xv@88R74a`RnWl5+BsK#J>C8+{JkKkkSspMqDuAj2>?IX|}msE2`p za|4KE1)817kiwA4keHK>B+DQkz`!7jERAgWqw@JX_fHN~lx2JH=?|~(z0D7QzSs5$r|3 diff --git a/src/miniBen/runner.py b/src/miniBen/runner.py index 58f8df7..20b33ed 100644 --- a/src/miniBen/runner.py +++ b/src/miniBen/runner.py @@ -110,11 +110,13 @@ def run_benchmark( outputs.append( { + "call": i, "variant": variant, "success": False, "error": str(e), "content": None, "reasoning": None, + "parsed": None, } ) @@ -133,11 +135,13 @@ def run_benchmark( outputs.append( { + "call": i, "variant": variant, "success": False, "error": "Empty or invalid response content", "content": content, "reasoning": reasoning_text, + "parsed": None, } ) @@ -166,6 +170,7 @@ def run_benchmark( outputs.append( { + "call": i, "variant": variant, "success": True, "error": None, diff --git a/src/miniBen/scorers.py b/src/miniBen/scorers.py index 046197e..9e61703 100644 --- a/src/miniBen/scorers.py +++ b/src/miniBen/scorers.py @@ -4,7 +4,6 @@ from functools import lru_cache import spacy -from numpy.core.multiarray import item from sentence_transformers import SentenceTransformer from sentence_transformers.util import cos_sim, dot_score, euclidean_sim, manhattan_sim @@ -75,7 +74,10 @@ def _score_single_game( for trial_idx, moves in enumerate(trials): expected_piece = TRIAL_PIECE_MAP.get(trial_idx) - piece_name = chess.piece_name(expected_piece).capitalize() + piece_name = ( + chess.piece_name(expected_piece).capitalize() + if expected_piece is not None else "N/A" + ) move_results = [] wrong_length = len(moves) != EXPECTED_MOVES_PER_TRIAL @@ -86,18 +88,17 @@ def _score_single_game( try: move = board.parse_san(move_str) - + except chess.IllegalMoveError: + violation = "illegal_move" + except Exception as e: + violation = f"parse_error: {e}" + else: piece = board.piece_at(move.from_square) if piece is None or piece.piece_type != expected_piece: violation = f"wrong_piece (expected {piece_name})" - elif move not in board.legal_moves: - violation = "illegal_move" else: board.push(move) - except Exception as e: - violation = f"parse_error: {e}" - move_results.append({ "move_number": move_idx + 1, "color": color, @@ -117,7 +118,7 @@ def _score_single_game( total_moves = sum(t["actual_length"] for t in trial_results) total_violations = sum(t["trial_violations"] for t in trial_results) - final_score = total_violations / total_moves + final_score = total_violations / total_moves if total_moves else 0.0 return { "game_index": game_idx + 1, @@ -210,7 +211,9 @@ def get_words(text: str, *, lower=True, remove_punct=True, def get_embedding(text, model=DEF_EMB_MODEL, emb_type=DEF_EMB_TYPE): output_value = "token_embeddings" if "token" in emb_type else "sentence_embedding" emb = load_emb_model(model).encode(text, output_value=output_value) - return emb.mean(axis=0) if emb.ndim == 2 else emb + emb = emb.mean(axis=0) if emb.ndim == 2 else emb + # gte-large emits float16; torch.cdist (euclidean/manhattan sims) has no Half CPU kernel + return emb.astype("float32") def _distance(emb1, emb2, fn=DEF_DIST_FN) -> float: @@ -245,7 +248,7 @@ def compute_novelty(texts: list[str], model=DEF_EMB_MODEL, preprocessing=DEF_PREPROCESSING) -> list[float]: """How semantically distinct each text is from the overall corpus.""" kw = dict(model=model, emb_type=emb_type, dist_fn=dist_fn, preprocessing=preprocessing) - corpus_dis = avg_sem_dis("".join(texts), **kw) + corpus_dis = avg_sem_dis(" ".join(texts), **kw) return [2 * abs(avg_sem_dis(t, **kw) - corpus_dis) for t in texts] @@ -266,17 +269,23 @@ def compute_surprise(text: str, model=DEF_EMB_MODEL, # ── Final scorer ────────────────────────────────────────────────────────────── -def score_creativity(parsed_creativity: list[dict], +def score_creativity(parsed_creativity: list[dict | None], novelty_weight=0.6, surprise_weight=0.4) -> float: """ Args: - parsed_creativity: list of dicts with at least a 'story' key. + parsed_creativity: list of per-call dicts, each with a 'story' (str) key + (the output of parse_creativity_output). None entries (failed parses) + are skipped. + + Returns the weighted blend of mean novelty (each story vs. the corpus of all + stories) and mean sentence-level surprise (within each story). Returns 0.0 + when there are no usable stories. """ - text = parsed_creativity.items("story") - combined_text = " ".join(text) - ###TODO: novelty should have one score for all stories + stories = [p["story"] for p in parsed_creativity if p] + if not stories: + return 0.0 - novelty = mean(compute_novelty(text)) - surprise, _, _ = compute_surprise(text) + novelty = mean(compute_novelty(stories)) + surprise = mean(compute_surprise(s)[0] for s in stories) return novelty_weight * novelty + surprise_weight * surprise \ No newline at end of file From 488b366fb83e2d4de0b8acd949f9ce45566ccae3 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Sun, 24 May 2026 01:22:14 +0200 Subject: [PATCH 6/9] Add GitHub Actions workflow and update benchmark logic --- .github/workflows/python-tests.yaml | 24 +++ .gitignore | 2 +- src/.DS_Store | Bin 0 -> 6148 bytes src/miniBen/.DS_Store | Bin 0 -> 6148 bytes src/miniBen/example.py | 20 +-- src/miniBen/runner.py | 240 +++++++++++++--------------- src/miniBen/scorers.py | 240 ++++++++++++++++++++++------ 7 files changed, 339 insertions(+), 187 deletions(-) create mode 100644 .github/workflows/python-tests.yaml create mode 100644 src/.DS_Store create mode 100644 src/miniBen/.DS_Store diff --git a/.github/workflows/python-tests.yaml b/.github/workflows/python-tests.yaml new file mode 100644 index 0000000..68a9296 --- /dev/null +++ b/.github/workflows/python-tests.yaml @@ -0,0 +1,24 @@ +name: Python tests + +on: + push: + branches: [main, master] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-python@v6 + with: + python-version: "3.12" # <- change this to the Python version you use locally + + - name: Install package and test tools + run: | + python -m pip install --upgrade pip + python -m pip install -e . pytest # <- if you need to add many packages here, your package setup is probably missing that information! + + - name: Run unit tests + run: pytest \ No newline at end of file diff --git a/.gitignore b/.gitignore index 8f2c43e..24065bb 100644 --- a/.gitignore +++ b/.gitignore @@ -13,4 +13,4 @@ __pycache__/ .idea/ # macOS -.DS_Store \ No newline at end of file +.DS_Store.DS_Store diff --git a/src/.DS_Store b/src/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..83550b01b2dd75cf5d6b8bb9e328e72d975d7230 GIT binary patch literal 6148 zcmeHK!Ab-%3{BP*2fXZY!ApN(|6m-(FW8GmnH^DKX1gHX`y+mic=8)O`WwD9aoKTu z6GS9|^rcPGP9IF$5D_;|{fcNoL={w!#VCjj7hN+8J_T}~qoG^c_K%yPi-n2);*fm% z8IANnTj;^}#{paV?i%h^>rEVYc_*AMjw8m{w%rgrc&g{?CB3}AUY4TzeH6SOZ^fwH z{lRK8T@=wS&VV!E3^)VMz>gS^3l379D|+h;I0MeWo&h-@0#vXzOp5vFK$A-V;2gtQ zU`s6_Il-_tOp5S8SW|(T%2r~qrei!J5S<|@R@x?IMCG0;H!#t1f*fEW8$^mkyA1`MkH8tY3rdwUP{JW7c;guY zE0#1!2%#Cte*V0%XP>mThlqH7yP6SAh^RmjWN8eT3=dA77h66nqZ{H&?(Fa0OfeSKvnq;La8+N=2_-0aw5kI4B_JLqHLXhDk9W9jI~&0F-BR zHnb&|5T9Te4U-~fV63h{b!AI2Sluxmj28`)qPi1X^1-(Ad-B40cdQ>Woj59b?FzU8 zeFa9^oyh%vfnR1Y$?v;(&lPY5{+R+it`^lCA7yvzm(P>CHlSRhhzT!A1BQ0*5rB@I gBhx&o^Fi9+MZ=^htH^kE2l_)G3*wb4@CypO0pY(iFaQ7m literal 0 HcmV?d00001 diff --git a/src/miniBen/example.py b/src/miniBen/example.py index 78f777d..6de5a57 100644 --- a/src/miniBen/example.py +++ b/src/miniBen/example.py @@ -114,15 +114,17 @@ def main(argv: list[str] | None = None) -> int: reasoning=args.reasoning, ) - print("\n=== Summary ===") - print(f"Model: {results['model']}") - print(f"Benchmark: {results['benchmark']}") - print(f"API calls: {results['num_calls']}") - print(f"Score: {results['score']}") - - for row in results["outputs"]: - status = "ok" if row["parsed"] is not None else "parse pending / failed" - print(f" Call {row['call']}: {row['variant']} — {status}") +# print("\n=== Summary ===") +# print(f"Model: {results['model']}") +# print(f"Benchmark: {results['benchmark']}") +# print(f"API calls: {results['num_calls']}") +# print(f"Score: {results['score']}") +# +# #TODO: This summary duplicate with the summary in runner + +# for row in results["outputs"]: +# status = "ok" if row["parsed"] is not None else "parse pending / failed" +# print(f" Call {row['call']}: {row['variant']} — {status}") return 0 diff --git a/src/miniBen/runner.py b/src/miniBen/runner.py index 20b33ed..b683d5e 100644 --- a/src/miniBen/runner.py +++ b/src/miniBen/runner.py @@ -8,31 +8,44 @@ cog_flex_prompts, creativity_prompts, ) -from .scorers import score_chess, score_creativity +from .scorers import score_chess, score_creativity, _print_chess_score, _print_chess_summary, _print_creativity_score, _print_creativity_summary + BENCHMARKS = { "cognitive flexibility": { - "name": "Meta-Chess Game (Cognitive Flexibility)", - "prompts": cog_flex_prompts, - "variants": lambda: [ - {"white": w, "black": b} for w, b in COG_FLEX_OPENINGS - ], - "parser": parse_chess_output, - "scorer": score_chess, + "name": "Meta-Chess Game (Cognitive Flexibility)", + "prompts": cog_flex_prompts, + "variants": lambda: [{"white": w, "black": b} for w, b in COG_FLEX_OPENINGS], + "parser": parse_chess_output, + "scorer": score_chess, + "print_score": _print_chess_score, + "print_summary": _print_chess_summary, }, "creativity": { - "name": "Creativity in story writing", - "prompts": creativity_prompts, - "variants": lambda: [ - {"words": [w1, w2, w3]} for w1, w2, w3 in CREATIVITY_WORD_TRIPLETS - ], - "parser": parse_creativity_output, - "scorer": score_creativity, + "name": "Creativity in story writing", + "prompts": creativity_prompts, + "variants": lambda: [{"words": [w1, w2, w3]} for w1, w2, w3 in CREATIVITY_WORD_TRIPLETS], + "parser": parse_creativity_output, + "scorer": score_creativity, + "print_score": _print_creativity_score, + "print_summary": _print_creativity_summary, }, } +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _divider(char: str = "─", width: int = 60) -> None: + print(char * width) + + +# --------------------------------------------------------------------------- +# Runner +# --------------------------------------------------------------------------- + def run_benchmark( model_name: str, benchmark_name: str, @@ -53,7 +66,8 @@ def run_benchmark( Returns ------- dict - Keys: model, benchmark, num_calls, variants, parsed, outputs, score. + Keys: model, benchmark, num_calls, successful_calls, failed_calls, + variants, parsed, outputs, score, per_call_scores. """ if benchmark_name not in BENCHMARKS: raise KeyError( @@ -61,157 +75,131 @@ def run_benchmark( f"Choose from: {list(BENCHMARKS)}" ) - benchmark = BENCHMARKS[benchmark_name] - prompts = benchmark["prompts"]() - variants = benchmark["variants"]() - + benchmark = BENCHMARKS[benchmark_name] + prompts = benchmark["prompts"]() + variants = benchmark["variants"]() + print_score = benchmark["print_score"] + print_summary = benchmark["print_summary"] + parser = benchmark["parser"] + scorer = benchmark["scorer"] if len(prompts) != len(variants): raise ValueError( f"Prompt count ({len(prompts)}) and variant count " f"({len(variants)}) must match." ) - num_calls = len(prompts) - model = AIModel(model_name) - parser = benchmark["parser"] - scorer = benchmark["scorer"] - - print(f"\nRunning benchmark: {benchmark['name']} ({num_calls} API calls) ...") + num_calls = len(prompts) + model = AIModel(model_name) - parsed_list: list = [] - outputs: list[dict] = [] - score_list: list = [] + _divider("═") + print(f" BENCHMARK : {benchmark['name']}") + print(f" MODEL : {model_name}") + print(f" CALLS : {num_calls}") + _divider("═") - successful_calls = 0 - failed_calls = 0 + parsed_list: list = [] + outputs: list[dict] = [] + per_call_scores: list = [] + successful_calls = 0 + failed_calls = 0 for i, (prompt, variant) in enumerate(zip(prompts, variants), start=1): - print(f"\n--- API call {i}/{num_calls} — variant: {variant} ---") + _divider() + print(f" API Call {i}/{num_calls} | variant: {variant}") + _divider() - # ------------------------- - # API CALL - # ------------------------- + # --- API call --- try: - content, reasoning_text = model.ask( - prompt=prompt, - reasoning=reasoning, - ) - + content, reasoning_text = model.ask(prompt=prompt, reasoning=reasoning) except Exception as e: failed_calls += 1 - - print( - f"[WARNING] API call {i} failed. " - f"Skipping variant.\n" - f"Reason: {type(e).__name__}: {e}" - ) - - outputs.append( - { - "call": i, - "variant": variant, - "success": False, - "error": str(e), - "content": None, - "reasoning": None, - "parsed": None, - } - ) - + print(f" [FAILED] {type(e).__name__}: {e}") + outputs.append({ + "call": i, "variant": variant, "success": False, + "error": str(e), "content": None, + "reasoning": None, "parsed": None, + }) + per_call_scores.append(None) continue - # ------------------------- - # EMPTY / INVALID RESPONSE - # ------------------------- if not content or not isinstance(content, str): failed_calls += 1 - - print( - f"[WARNING] API call {i} returned " - f"empty or invalid content. Skipping." - ) - - outputs.append( - { - "call": i, - "variant": variant, - "success": False, - "error": "Empty or invalid response content", - "content": content, - "reasoning": reasoning_text, - "parsed": None, - } - ) - + print(" [FAILED] Empty or invalid response content.") + outputs.append({ + "call": i, "variant": variant, "success": False, + "error": "Empty or invalid response content", + "content": content, "reasoning": reasoning_text, "parsed": None, + }) + per_call_scores.append(None) continue successful_calls += 1 - print("\n--- Raw Response ---") + print("\n RAW RESPONSE") print(content) - # ------------------------- - # PARSING - # ------------------------- - print("\n--- Parsed Output ---") - + # --- Parsing --- try: parsed = parser(content) - except Exception as e: - - print(f"[WARNING] Parse failed on call {i}: {e}") - + print(f"\n [PARSE FAILED] {e}") parsed = None parsed_list.append(parsed) + outputs.append({ + "call": i, "variant": variant, "success": True, + "error": None, "content": content, + "reasoning": reasoning_text, "parsed": parsed, + }) - outputs.append( - { - "call": i, - "variant": variant, - "success": True, - "error": None, - "content": content, - "reasoning": reasoning_text, - "parsed": parsed, - } - ) - + print("\n PARSED OUTPUT") pprint(parsed) - # ------------------------- - # SCORING - # ------------------------- - print("\n--- Total Score (valid calls only) ---") - - try: - call_score = scorer(parsed_list) - - except Exception as e: + # --- Per-call score --- + _divider() + print(f" SCORE (call {i})") + _divider() + try: + call_score = scorer([parsed]) + except Exception as e: + print(f" [SCORE FAILED] {e}") + call_score = None - print(f"[WARNING] Scoring failed: {e}") + per_call_scores.append(call_score) + print_score(call_score, i) # ← benchmark-specific printer - call_score = None + # --- Aggregate score --- + _divider("═") + print(" AGGREGATE SCORE (all calls)") + _divider("═") - score_list.append(call_score) + try: + aggregate_score = scorer(parsed_list) + except Exception as e: + print(f" [AGGREGATE SCORE FAILED] {e}") + aggregate_score = None - print(call_score) + print_summary(per_call_scores) # ← benchmark-specific summary - print("\n--- Benchmark Summary ---") - print(f"Successful calls: {successful_calls}") - print(f"Failed/skipped calls: {failed_calls}") + # --- Final summary --- + _divider("═") + print(" BENCHMARK COMPLETE") + _divider("═") + print(f" Successful calls : {successful_calls} / {num_calls}") + print(f" Failed calls : {failed_calls} / {num_calls}") + _divider("═") return { - "model": model_name, - "benchmark": benchmark_name, - "num_calls": num_calls, + "model": model_name, + "benchmark": benchmark_name, + "num_calls": num_calls, "successful_calls": successful_calls, - "failed_calls": failed_calls, - "variants": variants, - "parsed": parsed_list, - "outputs": outputs, - "score": call_score, + "failed_calls": failed_calls, + "variants": variants, + "parsed": parsed_list, + "outputs": outputs, + "score": aggregate_score, + "per_call_scores": per_call_scores, } \ No newline at end of file diff --git a/src/miniBen/scorers.py b/src/miniBen/scorers.py index 9e61703..4e4e3ef 100644 --- a/src/miniBen/scorers.py +++ b/src/miniBen/scorers.py @@ -35,6 +35,10 @@ def score_chess(parsed_list: list[list[list[str]] | None]) -> dict[str, Any]: - Wrong piece: move does not involve the trial's required piece type - Illegal move: move is not legal given the current board state + Board turn is forced per move number (odd = White, even = Black). + Parse-failed games are tracked separately and excluded from the + compliance rate so they do not silently inflate or deflate the score. + Parameters ---------- parsed_list : list @@ -44,94 +48,129 @@ def score_chess(parsed_list: list[list[list[str]] | None]) -> dict[str, Any]: Returns ------- dict with keys: - games : per-game breakdown (violations by trial) - total_moves : total half-moves evaluated - total_violations : total illegal/wrong-piece moves across all games - compliance_rate : float in [0, 1], proportion of legal correct moves + games : per-game breakdown (violations by trial) + total_moves : total half-moves evaluated (failed games excluded) + total_violations : total violations across scored games + failed_games : number of games that could not be parsed + scored_games : number of games actually evaluated + compliance_rate : float in [0, 1], proportion of legal correct moves + (0.0 when no moves could be evaluated) + per_trial_violations : list of total violations per trial index across + all scored games (length = EXPECTED_TRIALS) """ games_data = [] total_moves = 0 total_violations = 0 + failed_games = 0 + scored_games = 0 + # Accumulate violations per trial slot across all games + per_trial_violations = [0] * EXPECTED_TRIALS def _score_single_game( game_idx: int, trials: list[list[str]] | None, ) -> dict[str, Any]: - """Score one parsed game. Returns structured result dict.""" - - # --- Handle parse failure or wrong structure --- if trials is None: return { - "game_index": game_idx + 1, - "error": "parse_failed", - "trials": [], - "total_moves": 0, - "total_violations": 0, + "game_index": game_idx + 1, + "error": "parse_failed", + "trials": [], + "game_moves": 0, + "game_violations": 0, } board = chess.Board() trial_results = [] + game_moves = 0 + game_violations = 0 for trial_idx, moves in enumerate(trials): expected_piece = TRIAL_PIECE_MAP.get(trial_idx) - piece_name = ( + piece_name = ( chess.piece_name(expected_piece).capitalize() - if expected_piece is not None else "N/A" + if expected_piece is not None else "Unknown" ) - move_results = [] wrong_length = len(moves) != EXPECTED_MOVES_PER_TRIAL + move_results = [] + trial_violations = 0 for move_idx, move_str in enumerate(moves): - color = "White" if board.turn == chess.WHITE else "Black" - violation = None + move_number = move_idx + 1 + expected_turn = chess.WHITE if move_number % 2 == 1 else chess.BLACK + board.turn = expected_turn + color = "White" if expected_turn == chess.WHITE else "Black" + + is_illegal = False + reason = "" try: move = board.parse_san(move_str) - except chess.IllegalMoveError: - violation = "illegal_move" - except Exception as e: - violation = f"parse_error: {e}" - else: - piece = board.piece_at(move.from_square) - if piece is None or piece.piece_type != expected_piece: - violation = f"wrong_piece (expected {piece_name})" + + # Check legality first + if move not in board.legal_moves: + is_illegal = True + reason = "not a legal chess move" else: + # Check piece type + piece = board.piece_at(move.from_square) + if piece is None or piece.piece_type != expected_piece: + is_illegal = True + reason = f"expected a {piece_name.lower()} move" + + # Always push if SAN parsed and move is legal on this board, + # even if it's the wrong piece — so the board keeps advancing + if move in board.legal_moves: board.push(move) + except Exception as e: + is_illegal = True + # e.g. "illegal san: 'Rc1' in r1bqk2r/..." — drop everything after " in " + reason = f"illegal move: {str(e).split(' in ')[0]}" + + if is_illegal: + trial_violations += 1 + move_results.append({ - "move_number": move_idx + 1, - "color": color, - "san": move_str, - "violation": violation, + "move_number": move_number, + "color": color, + "san": move_str, + "violation": reason if is_illegal else None, }) - trial_violations = sum(1 for m in move_results if m["violation"]) + game_moves += len(moves) + game_violations += trial_violations + trial_results.append({ - "trial_index": trial_idx + 1, - "expected_piece": piece_name, - "wrong_length": wrong_length, - "actual_length": len(moves), - "moves": move_results, - "trial_violations": trial_violations, + "trial_index": trial_idx + 1, + "expected_piece": piece_name, + "wrong_length": wrong_length, + "actual_length": len(moves), + "moves": move_results, + "trial_violations": trial_violations, }) - total_moves = sum(t["actual_length"] for t in trial_results) - total_violations = sum(t["trial_violations"] for t in trial_results) - final_score = total_violations / total_moves if total_moves else 0.0 - return { - "game_index": game_idx + 1, - "total_moves": total_moves, - "total_violations": total_violations, - "final_score": final_score, + "game_index": game_idx + 1, + "trials": trial_results, + "game_moves": game_moves, + "game_violations": game_violations, } for game_idx, trials in enumerate(parsed_list): game_result = _score_single_game(game_idx, trials) games_data.append(game_result) - total_moves += game_result["total_moves"] - total_violations += game_result["total_violations"] + + if "error" in game_result: + failed_games += 1 + else: + scored_games += 1 + total_moves += game_result["game_moves"] + total_violations += game_result["game_violations"] + for trial in game_result["trials"]: + slot = trial["trial_index"] - 1 + if slot < EXPECTED_TRIALS: + per_trial_violations[slot] += trial["trial_violations"] compliance_rate = ( round(1 - total_violations / total_moves, 4) @@ -139,13 +178,91 @@ def _score_single_game( ) return { - "games": games_data, - "total_moves": total_moves, - "total_violations": total_violations, - "compliance_rate": compliance_rate, + "games": games_data, + "total_moves": total_moves, + "total_violations": total_violations, + "failed_games": failed_games, + "scored_games": scored_games, + "compliance_rate": compliance_rate, + "per_trial_violations": per_trial_violations, } +_PIECE_NAMES = ["Pawn", "Knight", "Bishop", "Rook", "Queen", "King"] + + +def _print_chess_score(score: dict, call_num: int) -> None: + if score is None: + print(f" Call {call_num}: scoring failed") + return + + compliance = score["compliance_rate"] * 100 + print(f" Compliance : {compliance:.1f}% " + f"({score['total_violations']} violations / {score['total_moves']} moves)") + + if score["failed_games"]: + print(f" ⚠ Parse failed : {score['failed_games']} game(s) excluded from score") + + if score["scored_games"] > 0: + print(f"\n Violations by piece type:") + for slot, (piece, count) in enumerate(zip(_PIECE_NAMES, score["per_trial_violations"])): + bar = "█" * count + "░" * max(0, score["scored_games"] * EXPECTED_MOVES_PER_TRIAL - count) + bar = bar[:20] + print(f" Trial {slot + 1} {piece:7s}: {count:3d} {bar}") + + for game in score["games"]: + if "error" in game: + print(f"\n Game {game['game_index']}: ✗ parse failed") + continue + + game_compliance = ( + round((1 - game["game_violations"] / game["game_moves"]) * 100, 1) + if game["game_moves"] > 0 else 0.0 + ) + print(f"\n Game {game['game_index']}: " + f"{game_compliance:.1f}% compliant " + f"({game['game_violations']} violations / {game['game_moves']} moves)") + + for trial in game["trials"]: + length_warn = f" ⚠ {trial['actual_length']}/8 moves" if trial["wrong_length"] else "" + status = "✓" if trial["trial_violations"] == 0 else "✗" + print(f" {status} Trial {trial['trial_index']} " + f"({trial['expected_piece']:7s}): " + f"{trial['trial_violations']} violations{length_warn}") + for m in trial["moves"]: + if m["violation"]: + print(f" move {m['move_number']} " + f"({m['color']:5s}) {m['san']:8s} ✗ {m['violation']}") + + +def _print_chess_summary(all_scores: list[dict]) -> None: + valid = [s for s in all_scores if s is not None] + if not valid: + print(" No valid scores to summarise.") + return + + avg_compliance = mean(s["compliance_rate"] for s in valid) * 100 + total_moves = sum(s["total_moves"] for s in valid) + total_violations = sum(s["total_violations"] for s in valid) + total_failed = sum(s["failed_games"] for s in valid) + + agg_trial = [0] * EXPECTED_TRIALS + for s in valid: + for i, v in enumerate(s["per_trial_violations"]): + agg_trial[i] += v + + print(f" Calls scored : {len(valid)} / {len(all_scores)}") + print(f" Avg compliance : {avg_compliance:.1f}%") + print(f" Total moves : {total_moves}") + print(f" Total violations : {total_violations}") + if total_failed: + print(f" ⚠ Parse failures : {total_failed} game(s) excluded") + + print(f"\n Violations by piece type (all calls):") + for slot, (piece, count) in enumerate(zip(_PIECE_NAMES, agg_trial)): + bar = "█" * min(count, 40) + print(f" Trial {slot + 1} {piece:7s}: {count:3d} {bar}") + # ============================================================================== # Creativity SCORING FUNCTIONS # ============================================================================== @@ -288,4 +405,25 @@ def score_creativity(parsed_creativity: list[dict | None], novelty = mean(compute_novelty(stories)) surprise = mean(compute_surprise(s)[0] for s in stories) - return novelty_weight * novelty + surprise_weight * surprise \ No newline at end of file + return novelty_weight * novelty + surprise_weight * surprise + +def _print_creativity_score(score: float | None, call_num: int) -> None: + if score is None: + print(f" Call {call_num}: scoring failed") + return + print(f" Creativity score : {score:.4f}") + + +def _print_creativity_summary(all_scores: list[float | None]) -> None: + valid = [s for s in all_scores if s is not None] + if not valid: + print(" No valid scores to summarise.") + return + print(f" Calls scored : {len(valid)} / {len(all_scores)}") + print(f" Avg creativity : {mean(valid):.4f}") + print(f" Min : {min(valid):.4f}") + print(f" Max : {max(valid):.4f}") + + + +#TODO: the scoring for chess is wrong, the final score does not reflect right answer; the printing of the score is off. \ No newline at end of file From ffb0d6b8cc9dcc3bb2cd21c76ecd232a5246f4bc Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Sun, 24 May 2026 03:18:54 +0200 Subject: [PATCH 7/9] Add vignette and update scoring functions --- .github/workflows/python-tests.yaml | 2 +- docs/vignette.ipynb | 635 ++++++++++++++++++++++++++++ src/miniBen/runner.py | 2 +- src/miniBen/scorers.py | 82 ++-- tests/conftest.py | 24 ++ tests/test_auth.py | 32 ++ tests/test_parsers.py | 66 +++ tests/test_runner.py | 50 +++ tests/test_scorers.py | 126 ++++++ 9 files changed, 993 insertions(+), 26 deletions(-) create mode 100644 docs/vignette.ipynb create mode 100644 tests/conftest.py create mode 100644 tests/test_auth.py create mode 100644 tests/test_parsers.py create mode 100644 tests/test_scorers.py diff --git a/.github/workflows/python-tests.yaml b/.github/workflows/python-tests.yaml index 68a9296..93838af 100644 --- a/.github/workflows/python-tests.yaml +++ b/.github/workflows/python-tests.yaml @@ -13,7 +13,7 @@ jobs: - uses: actions/checkout@v6 - uses: actions/setup-python@v6 with: - python-version: "3.12" # <- change this to the Python version you use locally + python-version: "3.12" - name: Install package and test tools run: | diff --git a/docs/vignette.ipynb b/docs/vignette.ipynb new file mode 100644 index 0000000..63d9325 --- /dev/null +++ b/docs/vignette.ipynb @@ -0,0 +1,635 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# miniBen — Vignette / Report\n", + "\n", + "**Author:** Leyla Yang \n", + "**Package:** [miniBen](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) — Mini LLM Benchmarks\n", + "\n", + "This document describes the purpose, users, workflow, and usage of **miniBen**, a Python package that evaluates large language models (LLMs) on small cognitive benchmarks via the [OpenRouter](https://openrouter.ai/) API." + ], + "id": "b6b5ab82af110bc0" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 1. Task / Technique Description\n", + "\n", + "### 1.1 Scenario\n", + "\n", + "#### Statement of purpose\n", + "\n", + "Researchers and developers need a lightweight, reproducible way to compare LLMs on structured tasks without building a full evaluation pipeline from scratch. **miniBen** fills this gap by:\n", + "\n", + "1. Generating benchmark-specific prompts with fixed variant sets (5 calls per benchmark).\n", + "2. Sending prompts to any OpenRouter model.\n", + "3. Parsing model replies into structured data.\n", + "4. Scoring replies with task-specific metrics and printing a human-readable report.\n", + "\n", + "Two benchmarks are currently registered:\n", + "\n", + "| Key | Task | Score metric |\n", + "|---|---|---|\n", + "| `cognitive flexibility` | Meta-Chess — six trials, each restricted to one piece type | Move **compliance rate** (legal SAN + correct piece) |\n", + "| `creativity` | Five-sentence story with three required words | Weighted **novelty + surprise** (semantic distance) |\n", + "\n", + "#### Description of the individual user\n", + "\n", + "The primary user is a **computational social-science or NLP researcher** (graduate student or lab member) who:\n", + "\n", + "- Wants to compare LLMs on the benchmark protocol.\n", + "- Has (or can obtain) an OpenRouter API key for live model calls.\n", + "- Does **not** need a graphical desktop application — interaction happens through the CLI, Python API, or this notebook.\n", + "\n", + "#### Assumptions about equipment and dependencies\n", + "\n", + "| Requirement | Details |\n", + "|---|---|\n", + "| **Hardware** | Any laptop or server with internet access for live runs |\n", + "| **OS** | macOS, Linux, or Windows |\n", + "| **Python** | 3.9 or newer |\n", + "| **Network** | Required for OpenRouter API calls; creativity scoring also downloads `thenlper/gte-large` on first use |\n", + "| **API key** | `OPENROUTER_API_KEY` in `.env` or environment |\n", + "\n", + "**Python package dependencies** (from `pyproject.toml`):\n", + "\n", + "| Package | Role in miniBen |\n", + "|---|---|\n", + "| `openai` | HTTP client for OpenRouter chat completions |\n", + "| `python-dotenv` | Load API key from `.env` |\n", + "| `chess` | Validate Meta-Chess moves and board state |\n", + "| `spacy`, `en_core_web_sm` | Sentence splitting and tokenisation for creativity scoring |\n", + "| `sentence_transformers` | Word embeddings for novelty / surprise |\n", + "| `benepar` | Constituency parsing (optional pipeline extension) |\n", + "\n", + "Install from the repository root:\n", + "\n", + "```bash\n", + "python -m venv .venv\n", + "source .venv/bin/activate # Windows: .venv\\Scripts\\activate\n", + "pip install -e \".[dev]\"\n", + "```\n", + "\n", + "#### Scenario\n", + "\n", + "> *A researcher is comparing two OpenRouter models on cognitive flexibility and creativity. They clone the miniBen repo, add their API key to `.env`, list available benchmarks from the terminal, then run the creativity benchmark for each model. miniBen makes five API calls per model, parses each story, computes per-call and aggregate creativity scores, and prints a summary Alex can paste into their lab report.*\n", + "\n", + "Steps in the scenario:\n", + "\n", + "1. Install miniBen and configure `OPENROUTER_API_KEY`.\n", + "2. Choose interface: **CLI** (`python -m miniBen.example`) or **Python** (`run_benchmark(...)`).\n", + "3. Select benchmark (`creativity` or `cognitive flexibility`) and model slug.\n", + "4. miniBen loops over five prompt variants → API → parse → score.\n", + "5. User reads printed report and/or inspects the returned `dict` for further analysis." + ], + "id": "5cd5521eec80d3c4" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.2 Flowchart — user, software, and main functions\n", + "\n", + "The diagram below shows how the user interacts with miniBen and how data flows through the main modules." + ], + "id": "a6be0f857fc9ba21" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```mermaid\n", + "flowchart TB\n", + " subgraph User[\"User\"]\n", + " U1[\"Researcher / developer\"]\n", + " U2[\"CLI: python -m miniBen.example\"]\n", + " U3[\"Python: run_benchmark() / AIModel.ask()\"]\n", + " end\n", + "\n", + " subgraph miniBen[\"miniBen package\"]\n", + " AUTH[\"auth.py\\ncheck / set API key\"]\n", + " RUN[\"runner.py\\nrun_benchmark()\"]\n", + " PROMPT[\"prompts.py\\ncog_flex_prompts()\\ncreativity_prompts()\"]\n", + " MODEL[\"model.py\\nAIModel.ask()\"]\n", + " PARSE[\"parsers.py\\nparse_chess_output()\\nparse_creativity_output()\"]\n", + " SCORE[\"scorers.py\\nscore_chess()\\nscore_creativity()\"]\n", + " end\n", + "\n", + " OR[\"OpenRouter API\"]\n", + " OUT[\"Console report + results dict\"]\n", + "\n", + " U1 --> U2\n", + " U1 --> U3\n", + " U2 --> AUTH\n", + " U3 --> AUTH\n", + " AUTH --> RUN\n", + " RUN --> PROMPT\n", + " PROMPT -->|\"5 prompts + variants\"| MODEL\n", + " MODEL -->|\"HTTP POST\"| OR\n", + " OR -->|\"raw text\"| MODEL\n", + " MODEL -->|\"content, reasoning\"| PARSE\n", + " PARSE -->|\"structured parse\"| SCORE\n", + " SCORE -->|\"per-call + aggregate score\"| OUT\n", + " RUN --> OUT\n", + "```" + ], + "id": "4b87331d8f2814d7" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Static flowchart figure (for PDF/export when Mermaid is unavailable)\n", + "import matplotlib.pyplot as plt\n", + "from matplotlib.patches import FancyBboxPatch, FancyArrowPatch\n", + "\n", + "fig, ax = plt.subplots(figsize=(11, 6))\n", + "ax.set_xlim(0, 11)\n", + "ax.set_ylim(0, 6)\n", + "ax.axis(\"off\")\n", + "ax.set_title(\"miniBen workflow: user → software → OpenRouter → scores\", fontsize=13, pad=12)\n", + "\n", + "def box(x, y, w, h, text, fc=\"#e8f4fd\", ec=\"#2c5282\"):\n", + " p = FancyBboxPatch((x, y), w, h, boxstyle=\"round,pad=0.02\", fc=fc, ec=ec, lw=1.2)\n", + " ax.add_patch(p)\n", + " ax.text(x + w/2, y + h/2, text, ha=\"center\", va=\"center\", fontsize=9, wrap=True)\n", + "\n", + "def arrow(x1, y1, x2, y2):\n", + " ax.add_patch(FancyArrowPatch((x1, y1), (x2, y2), arrowstyle=\"->\", mutation_scale=12, lw=1.2, color=\"#444\"))\n", + "\n", + "box(0.3, 4.5, 1.8, 0.9, \"User\\n(CLI / Python)\", fc=\"#fff3cd\")\n", + "box(2.5, 4.5, 1.6, 0.9, \"auth.py\", fc=\"#e8f4fd\")\n", + "box(4.5, 4.5, 1.8, 0.9, \"run_benchmark\\nrunner.py\", fc=\"#e8f4fd\")\n", + "box(6.8, 4.5, 1.6, 0.9, \"prompts.py\\n(5 variants)\", fc=\"#e8f4fd\")\n", + "box(4.5, 2.8, 1.8, 0.9, \"AIModel.ask\\nmodel.py\", fc=\"#e8f4fd\")\n", + "box(6.8, 2.8, 1.6, 0.9, \"OpenRouter\\nAPI\", fc=\"#f8d7da\")\n", + "box(2.5, 2.8, 1.6, 0.9, \"parsers.py\", fc=\"#e8f4fd\")\n", + "box(0.3, 2.8, 1.8, 0.9, \"scorers.py\", fc=\"#e8f4fd\")\n", + "box(0.3, 1.0, 2.2, 0.9, \"Console report\\n+ results dict\", fc=\"#d4edda\")\n", + "\n", + "arrow(2.1, 4.95, 2.5, 4.95)\n", + "arrow(4.1, 4.95, 4.5, 4.95)\n", + "arrow(6.3, 4.95, 6.8, 4.95)\n", + "arrow(7.6, 4.5, 5.4, 3.7)\n", + "arrow(6.3, 3.25, 6.8, 3.25)\n", + "arrow(6.8, 3.25, 6.3, 3.25)\n", + "arrow(4.5, 3.25, 4.1, 3.25)\n", + "arrow(2.5, 3.25, 2.1, 3.25)\n", + "arrow(1.2, 2.8, 1.2, 1.9)\n", + "\n", + "plt.tight_layout()\n", + "plt.show()" + ], + "id": "2b5844139e1e62bb" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 2. How to Use the Software\n", + "\n", + "miniBen has **no graphical user interface (GUI)**. The \"UI\" consists of:\n", + "\n", + "| Interface | Where | Typical use |\n", + "|---|---|---|\n", + "| **Command-line (CLI)** | Terminal | Quick one-off benchmark runs |\n", + "| **Python API** | Script, REPL, or Jupyter | Programmatic control and custom analysis |\n", + "| **Console report** | stdout during `run_benchmark()` | Human-readable progress, parse output, and scores |\n", + "\n", + "### 2.1 Setup — API key\n", + "\n", + "Create a `.env` file in the project root:\n", + "\n", + "```env\n", + "OPENROUTER_API_KEY=sk-or-v1-your-key-here\n", + "```\n", + "\n", + "Or set the variable in your shell before running." + ], + "id": "7bd6f271897fe12b" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import check_openrouter_api_key_exist\n", + "\n", + "print(\"API key configured:\", check_openrouter_api_key_exist())" + ], + "id": "41a60378a7f13542" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.2 CLI — listing and running benchmarks\n", + "\n", + "**Screenshot equivalent:** terminal capture of `--list` and `--help`.\n", + "\n", + "```bash\n", + "python -m miniBen.example --help\n", + "python -m miniBen.example --list\n", + "python -m miniBen.example --benchmark creativity --model openrouter/free\n", + "python -m miniBen.example --benchmark \"cognitive flexibility\" --reasoning\n", + "```\n", + "\n", + "| Flag | Effect |\n", + "|---|---|\n", + "| `--list` | Print registered benchmarks and exit |\n", + "| `--benchmark` | Choose `creativity` or `cognitive flexibility` |\n", + "| `--model` | OpenRouter model slug (default: `openrouter/free`) |\n", + "| `--reasoning` | Request reasoning tokens when supported |\n", + "\n", + "Expected output of `--list`:" + ], + "id": "cee148e24097492f" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Reproduce CLI --list output inside the notebook\n", + "from miniBen.example import list_benchmarks\n", + "\n", + "list_benchmarks()" + ], + "id": "90297d3b1f6b7bde" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3 Console report — what you see during a run\n", + "\n", + "When `run_benchmark()` executes, it prints a structured trace to the terminal. The report has **four recurring sections** per API call:\n", + "\n", + "1. **Header** — benchmark name, model, total calls \n", + "2. **Per call** — variant metadata → raw model response → parsed output → per-call score \n", + "3. **Aggregate score** — summary across all five calls \n", + "4. **Footer** — success / failure counts \n", + "\n", + "**Representative console output** (abbreviated; chess benchmark):" + ], + "id": "e9ffdab8d971fcfa" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```\n", + "════════════════════════════════════════════════════════════\n", + " BENCHMARK : Meta-Chess Game (Cognitive Flexibility)\n", + " MODEL : openrouter/free\n", + " CALLS : 5\n", + "════════════════════════════════════════════════════════════\n", + "────────────────────────────────────────────────────────────\n", + " API Call 1/5 | variant: {'white': 'e4', 'black': 'e5'}\n", + "────────────────────────────────────────────────────────────\n", + "\n", + " RAW RESPONSE\n", + " <<>>{ \"trials\": [ ... ] }<<>>\n", + "\n", + " PARSED OUTPUT\n", + " [['e4', 'e5', ...], ['Nc3', 'Nc6', ...], ...]\n", + "\n", + " SCORE (call 1)\n", + " Compliance : 87.5% (6 violations / 48 moves)\n", + " Violations by piece type:\n", + " Trial 1 Pawn : 2 ██░░░░░░░░░░░░░░░░░░\n", + " Trial 2 Knight : 1 █░░░░░░░░░░░░░░░░░░░\n", + " ...\n", + "\n", + "════════════════════════════════════════════════════════════\n", + " AGGREGATE SCORE (all calls)\n", + " Calls scored : 5 / 5\n", + " Avg compliance : 82.3%\n", + " ...\n", + "════════════════════════════════════════════════════════════\n", + " BENCHMARK COMPLETE\n", + " Successful calls : 5 / 5\n", + " Failed calls : 0 / 5\n", + "════════════════════════════════════════════════════════════\n", + "```" + ], + "id": "20217f83a97ec597" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.4 Python API — programmatic interaction\n", + "\n", + "| Function / class | Purpose |\n", + "|---|---|\n", + "| `AIModel(model).ask(prompt, reasoning=...)` | Single OpenRouter call; returns `(content, reasoning)` |\n", + "| `run_benchmark(model_name, benchmark_name, reasoning=...)` | Full 5-call benchmark; returns results `dict` |\n", + "| `parse_chess_output(raw)` / `parse_creativity_output(raw)` | Parse model text offline |\n", + "| `score_chess(parsed_list)` / `score_creativity(parsed_list)` | Score parsed outputs offline |\n", + "\n", + "Inspect registered benchmarks:" + ], + "id": "90d0189df035054b" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import BENCHMARKS\n", + "\n", + "for key, spec in BENCHMARKS.items():\n", + " print(f\"{key!r} → {spec['name']} ({len(spec['prompts']())} calls)\")" + ], + "id": "e84c91966ac7f75" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 3. Example Application\n", + "\n", + "This section demonstrates miniBen on **sample model outputs** (no API key required) and produces **figures** you can include in a report. Set `RUN_LIVE = True` at the end to run against a real model.\n", + "\n", + "### 3.1 Example A — Cognitive flexibility (Meta-Chess)\n", + "\n", + "We feed a reference Meta-Chess JSON response through the parser and scorer, then plot violations by trial (piece type)." + ], + "id": "7d0fc42707cf46b8" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import parse_chess_output, score_chess, JSON_START, JSON_END\n", + "\n", + "sample_chess_raw = f\"\"\"{JSON_START}\n", + "{{\n", + " \"trials\": [\n", + " [\"e4\",\"e5\",\"f4\",\"f5\",\"g4\",\"g5\",\"fxg5\",\"fxg4\"],\n", + " [\"Nc3\",\"Nc6\",\"Nf3\",\"Nf6\",\"Nxe4\",\"Nxe5\",\"Nd6\",\"Nb4\"],\n", + " [\"Bc4\",\"Bc5\",\"Bd5\",\"Bd6\",\"Bxc6\",\"bxc6\",\"Bb5\",\"Bb4\"],\n", + " [\"O-O\",\"O-O\",\"Rfe1\",\"Rfe8\",\"Rxe8+\",\"Rxe8\",\"Rd1\",\"Rd8\"],\n", + " [\"Qf3\",\"Qf6\",\"Qxf6+\",\"gxf6\",\"Qd3\",\"Qd6\",\"Qe4\",\"Qe5\"],\n", + " [\"Kg1\",\"Kg8\",\"Kf1\",\"Kh8\",\"Kg2\",\"Kg7\",\"Kh3\",\"Kh6\"]\n", + " ]\n", + "}}\n", + "{JSON_END}\"\"\"\n", + "\n", + "parsed_chess = parse_chess_output(sample_chess_raw)\n", + "chess_score = score_chess([parsed_chess])\n", + "\n", + "print(f\"Compliance rate : {chess_score['compliance_rate']:.1%}\")\n", + "print(f\"Total moves : {chess_score['total_moves']}\")\n", + "print(f\"Total violations: {chess_score['total_violations']}\")" + ], + "id": "245aa93a63fa1edd" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "\n", + "piece_names = [\"Pawn\", \"Knight\", \"Bishop\", \"Rook\", \"Queen\", \"King\"]\n", + "violations = chess_score[\"per_trial_violations\"]\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(11, 4))\n", + "\n", + "# Figure 1: violations by trial\n", + "colors = plt.cm.RdYlGn_r(np.linspace(0.2, 0.8, len(piece_names)))\n", + "axes[0].bar(piece_names, violations, color=colors, edgecolor=\"#333\")\n", + "axes[0].set_title(\"Meta-Chess: violations by trial (piece type)\")\n", + "axes[0].set_xlabel(\"Trial / piece type\")\n", + "axes[0].set_ylabel(\"Number of violations\")\n", + "axes[0].tick_params(axis=\"x\", rotation=30)\n", + "\n", + "# Figure 2: compliance gauge\n", + "compliance = chess_score[\"compliance_rate\"]\n", + "axes[1].pie(\n", + " [compliance, 1 - compliance],\n", + " labels=[f\"Compliant\\n{compliance:.1%}\", f\"Violations\\n{1-compliance:.1%}\"],\n", + " colors=[\"#28a745\", \"#dc3545\"],\n", + " startangle=90,\n", + " autopct=\"\",\n", + ")\n", + "axes[1].set_title(\"Overall move compliance rate\")\n", + "\n", + "plt.tight_layout()\n", + "plt.savefig(\"docs/figures/chess_example.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(\"Saved: docs/figures/chess_example.png\")" + ], + "id": "d31d4ea4b88bd724" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2 Example B — Creativity (short stories)\n", + "\n", + "Three sample five-sentence stories (matching the creativity prompt format) are parsed and scored. The creativity metric combines **novelty** (distinctiveness from the corpus) and **surprise** (sentence-to-sentence semantic shift)." + ], + "id": "6228c3743dd0a5ec" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import parse_creativity_output, score_creativity\n", + "\n", + "stories_raw = [\n", + " \"She found an old stamp tucked inside the letter. \"\n", + " \"Without thinking, she decided to send it to her grandmother. \"\n", + " \"The post office clerk smiled at the faded ink. \"\n", + " \"Days later, a reply arrived with a story of its own. \"\n", + " \"Some messages travel farther than miles.\",\n", + " \"The kitchen smelled of spices before dawn. \"\n", + " \"She began to cook a meal no recipe described. \"\n", + " \"Steam rose like a curtain over the food. \"\n", + " \"Neighbors knocked, drawn by the aroma. \"\n", + " \"Hunger became the evening's quiet language.\",\n", + " \"He tightened the wheel and mounted the bike. \"\n", + " \"Wind pressed against him on the first ride. \"\n", + " \"A stranger waved from a hilltop. \"\n", + " \"They spoke of roads neither had taken. \"\n", + " \"He rode home knowing distance is a choice.\",\n", + "]\n", + "\n", + "parsed_stories = [parse_creativity_output(s) for s in stories_raw]\n", + "for i, p in enumerate(parsed_stories, 1):\n", + " print(f\"Story {i}: {p['sentence_count']} sentences\")\n", + " print(f\" {p['story'][:80]}…\\n\")" + ], + "id": "d3267c9d4f18eb83" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Requires sentence_transformers + spaCy (first run downloads gte-large)\n", + "try:\n", + " from miniBen.scorers import compute_novelty, compute_surprise\n", + "\n", + " story_texts = [p[\"story\"] for p in parsed_stories]\n", + " novelty_scores = compute_novelty(story_texts)\n", + " surprise_scores = [compute_surprise(s)[0] for s in story_texts]\n", + " aggregate = score_creativity(parsed_stories)\n", + "\n", + " print(f\"Aggregate creativity score: {aggregate:.4f}\")\n", + "\n", + " fig, ax = plt.subplots(figsize=(8, 4))\n", + " x = np.arange(len(story_texts))\n", + " width = 0.35\n", + " ax.bar(x - width/2, novelty_scores, width, label=\"Novelty\", color=\"#5b8def\")\n", + " ax.bar(x + width/2, surprise_scores, width, label=\"Surprise\", color=\"#f4a261\")\n", + " ax.set_xticks(x)\n", + " ax.set_xticklabels([f\"Story {i+1}\" for i in range(len(story_texts))])\n", + " ax.set_ylabel(\"Score\")\n", + " ax.set_title(f\"Creativity components (aggregate = {aggregate:.4f})\")\n", + " ax.legend()\n", + " plt.tight_layout()\n", + " plt.savefig(\"docs/figures/creativity_example.png\", dpi=150, bbox_inches=\"tight\")\n", + " plt.show()\n", + " print(\"Saved: docs/figures/creativity_example.png\")\n", + "except Exception as e:\n", + " print(f\"Creativity scoring skipped ({type(e).__name__}: {e}).\")\n", + " print(\"Install dependencies and ensure network access for first-time model download.\")" + ], + "id": "77ab00e0989817dc" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.3 Example C — Simulated multi-call benchmark summary\n", + "\n", + "The cell below mimics what a user sees after five creativity calls: per-call scores and an aggregate. Values are illustrative; replace with `run_benchmark()` output for live data." + ], + "id": "a606fa4702b8c407" + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import CREATIVITY_WORD_TRIPLETS\n", + "\n", + "# Illustrative per-call scores (replace with results[\"per_call_scores\"] from a live run)\n", + "illustrative_scores = [0.412, 0.387, 0.445, 0.401, 0.428]\n", + "\n", + "fig, ax = plt.subplots(figsize=(9, 4))\n", + "labels = [f\"Call {i+1}\\n{', '.join(w)}\" for i, w in enumerate(CREATIVITY_WORD_TRIPLETS)]\n", + "bars = ax.bar(labels, illustrative_scores, color=\"#6f42c1\", edgecolor=\"#333\", alpha=0.85)\n", + "ax.axhline(np.mean(illustrative_scores), color=\"#dc3545\", ls=\"--\", label=f\"Mean = {np.mean(illustrative_scores):.3f}\")\n", + "ax.set_ylim(0, max(illustrative_scores) * 1.2)\n", + "ax.set_ylabel(\"Creativity score\")\n", + "ax.set_title(\"Example: per-call creativity scores across five word triplets\")\n", + "ax.legend()\n", + "plt.xticks(rotation=15, ha=\"right\")\n", + "plt.tight_layout()\n", + "plt.savefig(\"docs/figures/benchmark_summary_example.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(\"Saved: docs/figures/benchmark_summary_example.png\")" + ], + "id": "68d148b9636e43cc" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.4 Optional — live benchmark run\n", + "\n", + "Set `RUN_LIVE = True` and ensure `OPENROUTER_API_KEY` is configured." + ], + "id": "8e0d4ad1ce845ce5" + }, + { + "cell_type": "code", + "execution_count": null, + "id": "afeb243a", + "metadata": {}, + "outputs": [], + "source": [ + "RUN_LIVE = False\n", + "MODEL = \"openrouter/free\"\n", + "\n", + "if RUN_LIVE and check_openrouter_api_key_exist():\n", + " from miniBen import run_benchmark\n", + "\n", + " results = run_benchmark(\n", + " model_name=MODEL,\n", + " benchmark_name=\"creativity\",\n", + " reasoning=False,\n", + " )\n", + " print(\"\\n--- returned dict (summary) ---\")\n", + " for k in [\"model\", \"benchmark\", \"successful_calls\", \"num_calls\", \"score\", \"per_call_scores\"]:\n", + " print(f\" {k}: {results[k]}\")\n", + "else:\n", + " print(\"Skipped. Set RUN_LIVE = True and configure OPENROUTER_API_KEY for a live run.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Summary\n", + "\n", + "| Section | Content |\n", + "|---|---|\n", + "| **Purpose** | Evaluate LLMs on Meta-Chess and creativity benchmarks via OpenRouter |\n", + "| **User** | Python-literate researcher comparing models |\n", + "| **Interface** | CLI, Python API, console report (no GUI) |\n", + "| **Outputs** | Printed trace, results `dict`, optional matplotlib figures |\n", + "\n", + "**References**\n", + "\n", + "- miniBen repository: https://github.com/Programming-The-Next-Step-2026/mini_benchmarks \n", + "- Creativity benchmark: Ismayilzada et al. (2024) [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) \n", + "- OpenRouter models: https://openrouter.ai/models" + ], + "id": "7fa3478ebd41d39e" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/miniBen/runner.py b/src/miniBen/runner.py index b683d5e..347a334 100644 --- a/src/miniBen/runner.py +++ b/src/miniBen/runner.py @@ -181,7 +181,7 @@ def run_benchmark( print(f" [AGGREGATE SCORE FAILED] {e}") aggregate_score = None - print_summary(per_call_scores) # ← benchmark-specific summary + print_summary(per_call_scores, aggregate_score) # --- Final summary --- _divider("═") diff --git a/src/miniBen/scorers.py b/src/miniBen/scorers.py index 4e4e3ef..8e720c0 100644 --- a/src/miniBen/scorers.py +++ b/src/miniBen/scorers.py @@ -235,7 +235,7 @@ def _print_chess_score(score: dict, call_num: int) -> None: f"({m['color']:5s}) {m['san']:8s} ✗ {m['violation']}") -def _print_chess_summary(all_scores: list[dict]) -> None: +def _print_chess_summary(all_scores: list[dict], aggregate_score: dict | None = None) -> None: valid = [s for s in all_scores if s is not None] if not valid: print(" No valid scores to summarise.") @@ -386,44 +386,78 @@ def compute_surprise(text: str, model=DEF_EMB_MODEL, # ── Final scorer ────────────────────────────────────────────────────────────── -def score_creativity(parsed_creativity: list[dict | None], - novelty_weight=0.6, surprise_weight=0.4) -> float: +def score_creativity(parsed_creativity: list[dict | None]) -> dict: """ - Args: - parsed_creativity: list of per-call dicts, each with a 'story' (str) key - (the output of parse_creativity_output). None entries (failed parses) - are skipped. + Score creativity for one or more stories. + + When called with a single story (per-call): computes surprise only, + since novelty requires a corpus. + When called with multiple stories (aggregate): computes both surprise + per story and novelty across the corpus. - Returns the weighted blend of mean novelty (each story vs. the corpus of all - stories) and mean sentence-level surprise (within each story). Returns 0.0 - when there are no usable stories. + Args: + parsed_creativity: list of per-call dicts with a 'story' key. + None entries (failed parses) are skipped. + Returns: + dict with keys: + stories_scored : int + surprise_scores : list of per-story surprise floats + avg_surprise : float + novelty_scores : list of per-story novelty floats, or None if < 2 stories + avg_novelty : float or None """ - stories = [p["story"] for p in parsed_creativity if p] - if not stories: - return 0.0 + stories = [p["story"] for p in parsed_creativity if p and "story" in p] + + surprise_scores = [round(compute_surprise(s)[0], 4) for s in stories] + avg_surprise = round(mean(surprise_scores), 4) if surprise_scores else 0.0 + + if len(stories) >= 2: + novelty_scores = [round(n, 4) for n in compute_novelty(stories)] + avg_novelty = round(mean(novelty_scores), 4) + else: + novelty_scores = None + avg_novelty = None - novelty = mean(compute_novelty(stories)) - surprise = mean(compute_surprise(s)[0] for s in stories) + return { + "stories_scored": len(stories), + "surprise_scores": surprise_scores, + "avg_surprise": avg_surprise, + "novelty_scores": novelty_scores, + "avg_novelty": avg_novelty, + } - return novelty_weight * novelty + surprise_weight * surprise -def _print_creativity_score(score: float | None, call_num: int) -> None: +def _print_creativity_score(score: dict | None, call_num: int) -> None: if score is None: print(f" Call {call_num}: scoring failed") return - print(f" Creativity score : {score:.4f}") - + if not score["surprise_scores"]: + print(f" Call {call_num}: no valid story to score") + return + print(f" Surprise : {score['surprise_scores'][0]:.4f}") -def _print_creativity_summary(all_scores: list[float | None]) -> None: +def _print_creativity_summary( + all_scores: list[dict | None], + aggregate_score: dict | None = None, +) -> None: valid = [s for s in all_scores if s is not None] if not valid: print(" No valid scores to summarise.") return - print(f" Calls scored : {len(valid)} / {len(all_scores)}") - print(f" Avg creativity : {mean(valid):.4f}") - print(f" Min : {min(valid):.4f}") - print(f" Max : {max(valid):.4f}") + all_surprise = [s for v in valid for s in v["surprise_scores"]] + avg_surprise = round(mean(all_surprise), 4) if all_surprise else 0.0 + print(f" Calls scored : {len(valid)} / {len(all_scores)}") + print(f"\n Surprise (per call):") + for i, v in enumerate(valid, 1): + s = v["surprise_scores"][0] if v["surprise_scores"] else None + print(f" Call {i}: {s:.4f}" if s is not None else f" Call {i}: n/a") + print(f" Avg surprise : {avg_surprise:.4f}") + + if aggregate_score and aggregate_score.get("avg_novelty") is not None: + print(f"\n Novelty (corpus) : {aggregate_score['avg_novelty']:.4f}") + else: + print(f"\n Novelty : n/a (need ≥ 2 stories)") #TODO: the scoring for chess is wrong, the final score does not reflect right answer; the printing of the score is off. \ No newline at end of file diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..548a28a --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,24 @@ +import pytest + +from miniBen.prompts import JSON_END, JSON_START + + +@pytest.fixture +def valid_chess_trials() -> list[list[str]]: + """Reference Meta-Chess game (6 trials × 8 moves).""" + return [ + ["e4", "e5", "f4", "f5", "g4", "g5", "fxg5", "fxg4"], + ["Nc3", "Nc6", "Nf3", "Nf6", "Nxe4", "Nxe5", "Nd6", "Nb4"], + ["Bc4", "Bc5", "Bd5", "Bd6", "Bxc6", "bxc6", "Bb5", "Bb4"], + ["O-O", "O-O", "Rfe1", "Rfe8", "Rxe8+", "Rxe8", "Rd1", "Rd8"], + ["Qf3", "Qf6", "Qxf6+", "gxf6", "Qd3", "Qd6", "Qe4", "Qe5"], + ["Kg1", "Kg8", "Kf1", "Kh8", "Kg2", "Kg7", "Kh3", "Kh6"], + ] + + +@pytest.fixture +def valid_chess_json(valid_chess_trials) -> str: + import json + + payload = json.dumps({"trials": valid_chess_trials}) + return f"{JSON_START}\n{payload}\n{JSON_END}" diff --git a/tests/test_auth.py b/tests/test_auth.py new file mode 100644 index 0000000..09e9fb5 --- /dev/null +++ b/tests/test_auth.py @@ -0,0 +1,32 @@ +from unittest.mock import patch + +from miniBen.auth import check_openrouter_api_key_exist, put_openrouter_api_key_into_env + + +def test_check_openrouter_api_key_exist_true(): + with patch.dict("os.environ", {"OPENROUTER_API_KEY": "sk-test"}, clear=False): + assert check_openrouter_api_key_exist() is True + + +def test_check_openrouter_api_key_exist_false(): + with patch.dict("os.environ", {}, clear=True): + assert check_openrouter_api_key_exist() is False + + +def test_put_openrouter_api_key_into_env_prompts_when_missing(): + with patch("miniBen.auth.check_openrouter_api_key_exist", return_value=False): + with patch("builtins.input", return_value="sk-entered") as mock_input: + put_openrouter_api_key_into_env() + + import os + + mock_input.assert_called_once() + assert os.environ["OPENROUTER_API_KEY"] == "sk-entered" + + +def test_put_openrouter_api_key_into_env_skips_when_present(): + with patch.dict("os.environ", {"OPENROUTER_API_KEY": "sk-existing"}, clear=False): + with patch("builtins.input") as mock_input: + put_openrouter_api_key_into_env() + + mock_input.assert_not_called() diff --git a/tests/test_parsers.py b/tests/test_parsers.py new file mode 100644 index 0000000..f01a5ee --- /dev/null +++ b/tests/test_parsers.py @@ -0,0 +1,66 @@ +import json + +import pytest + +from miniBen.parsers import parse_chess_output, parse_creativity_output +from miniBen.prompts import JSON_END, JSON_START + + +def test_parse_chess_output_from_json_sentinels(valid_chess_json, valid_chess_trials): + result = parse_chess_output(valid_chess_json) + assert result == valid_chess_trials + + +def test_parse_chess_output_from_list_literal(valid_chess_trials): + raw = f"Here is the game:\n{valid_chess_trials}\nDone." + result = parse_chess_output(raw) + assert result == valid_chess_trials + + +def test_parse_chess_output_prefers_json_over_list_literal(valid_chess_trials): + wrong_literal = [[["bad"] * 8] * 6] + payload = json.dumps({"trials": valid_chess_trials}) + raw = f"{wrong_literal}\n{JSON_START}{payload}{JSON_END}" + assert parse_chess_output(raw) == valid_chess_trials + + +def test_parse_chess_output_returns_none_for_invalid_json(): + raw = f"{JSON_START}\n{{not json}}\n{JSON_END}" + assert parse_chess_output(raw) is None + + +def test_parse_chess_output_returns_none_for_wrong_shape(): + payload = json.dumps({"trials": [["e4"] * 8] * 6}) # only 1 move string repeated + raw = f"{JSON_START}{payload}{JSON_END}" + # shape is valid (6×8 strings) — should parse + assert parse_chess_output(raw) is not None + + payload = json.dumps({"trials": [["e4"] * 8] * 3}) # only 3 trials + raw = f"{JSON_START}{payload}{JSON_END}" + assert parse_chess_output(raw) is None + + +def test_parse_chess_output_returns_none_for_empty_input(): + assert parse_chess_output("") is None + assert parse_chess_output("no chess here") is None + + +def test_parse_creativity_output_returns_story_and_sentence_count(): + raw = "One sentence here. Another follows. A third! And fourth? Fifth." + result = parse_creativity_output(raw) + + assert result is not None + assert result["story"] == raw + assert result["sentence_count"] == 5 + + +def test_parse_creativity_output_strips_whitespace(): + raw = " Hello world. " + result = parse_creativity_output(raw) + + assert result["story"] == "Hello world." + + +def test_parse_creativity_output_returns_none_for_empty_string(): + assert parse_creativity_output("") is None + assert parse_creativity_output(" \n\t ") is None diff --git a/tests/test_runner.py b/tests/test_runner.py index 9beec53..9be345f 100644 --- a/tests/test_runner.py +++ b/tests/test_runner.py @@ -53,3 +53,53 @@ def test_creativity_prompts_match_word_triplets(): for prompt, words in zip(prompts, CREATIVITY_WORD_TRIPLETS): for word in words: assert word in prompt + + +def test_run_benchmark_unknown_benchmark_raises(): + import pytest + + with pytest.raises(KeyError, match="Unknown benchmark"): + run_benchmark( + model_name="openrouter/free", + benchmark_name="not a benchmark", + reasoning=False, + ) + + +def test_run_benchmark_records_api_failure(): + with patch.object(AIModel, "ask", side_effect=RuntimeError("network down")): + with patch("miniBen.model.OpenAI"): + result = run_benchmark( + model_name="openrouter/free", + benchmark_name="creativity", + reasoning=False, + ) + + assert result["failed_calls"] == 5 + assert result["successful_calls"] == 0 + assert all(not row["success"] for row in result["outputs"]) + assert all(row["error"] == "network down" for row in result["outputs"]) + + +def test_run_benchmark_result_keys(): + with patch.object(AIModel, "ask", return_value=("story one. Two. Three. Four. Five.", None)): + with patch("miniBen.model.OpenAI"): + result = run_benchmark( + model_name="openrouter/free", + benchmark_name="creativity", + reasoning=False, + ) + + expected_keys = { + "model", + "benchmark", + "num_calls", + "successful_calls", + "failed_calls", + "variants", + "parsed", + "outputs", + "score", + "per_call_scores", + } + assert expected_keys <= set(result) diff --git a/tests/test_scorers.py b/tests/test_scorers.py new file mode 100644 index 0000000..8a8cc6b --- /dev/null +++ b/tests/test_scorers.py @@ -0,0 +1,126 @@ +from unittest.mock import patch + +import pytest + +from miniBen.scorers import EXPECTED_TRIALS, score_chess, score_creativity + + +def test_score_chess_returns_expected_keys(valid_chess_trials): + result = score_chess([valid_chess_trials]) + + assert set(result) == { + "games", + "total_moves", + "total_violations", + "failed_games", + "scored_games", + "compliance_rate", + "per_trial_violations", + } + + +def test_score_chess_counts_moves_and_compliance(valid_chess_trials): + result = score_chess([valid_chess_trials]) + + assert result["scored_games"] == 1 + assert result["failed_games"] == 0 + assert result["total_moves"] == 48 + assert 0.0 <= result["compliance_rate"] <= 1.0 + assert len(result["per_trial_violations"]) == EXPECTED_TRIALS + + +def test_score_chess_tracks_parse_failures(): + result = score_chess([None, None]) + + assert result["failed_games"] == 2 + assert result["scored_games"] == 0 + assert result["total_moves"] == 0 + assert result["compliance_rate"] == 0.0 + assert all(g["error"] == "parse_failed" for g in result["games"]) + + +def test_score_chess_mixed_success_and_failure(valid_chess_trials): + result = score_chess([valid_chess_trials, None]) + + assert result["scored_games"] == 1 + assert result["failed_games"] == 1 + assert result["total_moves"] == 48 + + +def test_score_chess_empty_input(): + result = score_chess([]) + + assert result["scored_games"] == 0 + assert result["failed_games"] == 0 + assert result["compliance_rate"] == 0.0 + + +@patch("miniBen.scorers.compute_novelty", return_value=[0.5, 0.6]) +@patch("miniBen.scorers.compute_surprise", return_value=(0.4, [0.1], 0.2)) +def test_score_creativity_single_story(mock_surprise, mock_novelty): + parsed = [{"story": "One. Two. Three. Four. Five."}] + + result = score_creativity(parsed) + + assert result["stories_scored"] == 1 + assert result["surprise_scores"] == [0.4] + assert result["avg_surprise"] == 0.4 + assert result["novelty_scores"] is None + assert result["avg_novelty"] is None + mock_surprise.assert_called_once() + mock_novelty.assert_not_called() + + +@patch("miniBen.scorers.compute_novelty", return_value=[0.5, 0.6]) +@patch("miniBen.scorers.compute_surprise", return_value=(0.4, [0.1], 0.2)) +def test_score_creativity_multiple_stories(mock_surprise, mock_novelty): + parsed = [ + {"story": "First story here. With more. And more. Still more. Done."}, + {"story": "Second tale. Another line. Third line. Fourth. Fifth."}, + ] + + result = score_creativity(parsed) + + assert result["stories_scored"] == 2 + assert len(result["surprise_scores"]) == 2 + assert result["novelty_scores"] == [0.5, 0.6] + assert result["avg_novelty"] == 0.55 + mock_novelty.assert_called_once() + + +@patch("miniBen.scorers.compute_surprise", return_value=(0.3, [0.1], 0.2)) +def test_score_creativity_skips_failed_parses(_mock_surprise): + result = score_creativity([None, {"story": "A valid story. With five. Total of. Five. Sentences."}]) + + assert result["stories_scored"] == 1 + + +@patch("miniBen.scorers.compute_surprise", return_value=(0.0, [], 0.0)) +def test_score_creativity_empty_after_filtering(_mock_surprise): + result = score_creativity([None, None]) + + assert result["stories_scored"] == 0 + assert result["avg_surprise"] == 0.0 + assert result["surprise_scores"] == [] + assert result["avg_novelty"] is None + + +def test_print_creativity_summary_shows_novelty_from_aggregate(capsys): + from miniBen.scorers import _print_creativity_summary + + per_call = [ + {"surprise_scores": [0.01], "avg_surprise": 0.01, "novelty_scores": None, "avg_novelty": None}, + {"surprise_scores": [0.02], "avg_surprise": 0.02, "novelty_scores": None, "avg_novelty": None}, + ] + aggregate = { + "surprise_scores": [0.01, 0.02], + "avg_surprise": 0.015, + "novelty_scores": [0.4, 0.6], + "avg_novelty": 0.5, + } + + _print_creativity_summary(per_call, aggregate) + output = capsys.readouterr().out + + assert "Novelty (corpus) : 0.5000" in output + assert "n/a (need ≥ 2 stories)" not in output From 82be12ee7958410e73de829b8f8f4b6652a5e2a1 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Sun, 24 May 2026 03:37:16 +0200 Subject: [PATCH 8/9] update workflow --- docs/figures/workflow.svg | 82 +++ docs/vignette.ipynb | 1256 ++++++++++++++++++------------------- docs/workflow.md | 44 ++ 3 files changed, 750 insertions(+), 632 deletions(-) create mode 100644 docs/figures/workflow.svg create mode 100644 docs/workflow.md diff --git a/docs/figures/workflow.svg b/docs/figures/workflow.svg new file mode 100644 index 0000000..43e7279 --- /dev/null +++ b/docs/figures/workflow.svg @@ -0,0 +1,82 @@ + + + + + + + + + + + miniBen workflow + + + + USER + CLI python -m miniBen.example + Python run_benchmark() / AIModel.ask() + + + + + + auth.py check / load OPENROUTER_API_KEY + + + + + + runner.py run_benchmark() + + + + + + prompts.py build 5 prompt variants + + + + + + repeat for each of 5 variants + + + + + + model.py AIModel.ask() + + + + OpenRouter + API (external) + HTTP POST + + + + prompt + reply + + + + + + parsers.py parse model output + + + + + + scorers.py per-call + aggregate score + + + + + + OUTPUT + Console report + results dict + + + USER + miniBen + diff --git a/docs/vignette.ipynb b/docs/vignette.ipynb index 63d9325..dd02689 100644 --- a/docs/vignette.ipynb +++ b/docs/vignette.ipynb @@ -1,635 +1,627 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# miniBen — Vignette / Report\n", - "\n", - "**Author:** Leyla Yang \n", - "**Package:** [miniBen](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) — Mini LLM Benchmarks\n", - "\n", - "This document describes the purpose, users, workflow, and usage of **miniBen**, a Python package that evaluates large language models (LLMs) on small cognitive benchmarks via the [OpenRouter](https://openrouter.ai/) API." - ], - "id": "b6b5ab82af110bc0" + "cells": [ + { + "cell_type": "markdown", + "id": "b6b5ab82af110bc0", + "metadata": {}, + "source": [ + "# miniBen — Vignette / Report\n", + "\n", + "**Author:** Leyla Yang \n", + "**Package:** [miniBen](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) — Mini LLM Benchmarks\n", + "\n", + "This document describes the purpose, users, workflow, and usage of **miniBen**, a Python package that evaluates large language models (LLMs) on small cognitive benchmarks via the [OpenRouter](https://openrouter.ai/) API." + ] + }, + { + "cell_type": "markdown", + "id": "5cd5521eec80d3c4", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 1. Task / Technique Description\n", + "\n", + "### 1.1 Scenario\n", + "\n", + "#### Statement of purpose\n", + "\n", + "Researchers and developers need a lightweight, reproducible way to compare LLMs on structured tasks without building a full evaluation pipeline from scratch. **miniBen** fills this gap by:\n", + "\n", + "1. Generating benchmark-specific prompts with fixed variant sets (5 calls per benchmark).\n", + "2. Sending prompts to any OpenRouter model.\n", + "3. Parsing model replies into structured data.\n", + "4. Scoring replies with task-specific metrics and printing a human-readable report.\n", + "\n", + "Two benchmarks are currently registered:\n", + "\n", + "| Key | Task | Score metric |\n", + "|---|---|---|\n", + "| `cognitive flexibility` | Meta-Chess — six trials, each restricted to one piece type | Move **compliance rate** (legal SAN + correct piece) |\n", + "| `creativity` | Five-sentence story with three required words | Weighted **novelty + surprise** (semantic distance) |\n", + "\n", + "#### Description of the individual user\n", + "\n", + "The primary user is a **computational social-science or NLP researcher** (graduate student or lab member) who:\n", + "\n", + "- Wants to compare LLMs on the benchmark protocol.\n", + "- Has (or can obtain) an OpenRouter API key for live model calls.\n", + "- Does **not** need a graphical desktop application — interaction happens through the CLI, Python API, or this notebook.\n", + "\n", + "#### Assumptions about equipment and dependencies\n", + "\n", + "| Requirement | Details |\n", + "|---|---|\n", + "| **Hardware** | Any laptop or server with internet access for live runs |\n", + "| **OS** | macOS, Linux, or Windows |\n", + "| **Python** | 3.9 or newer |\n", + "| **Network** | Required for OpenRouter API calls; creativity scoring also downloads `thenlper/gte-large` on first use |\n", + "| **API key** | `OPENROUTER_API_KEY` in `.env` or environment |\n", + "\n", + "**Python package dependencies** (from `pyproject.toml`):\n", + "\n", + "| Package | Role in miniBen |\n", + "|---|---|\n", + "| `openai` | HTTP client for OpenRouter chat completions |\n", + "| `python-dotenv` | Load API key from `.env` |\n", + "| `chess` | Validate Meta-Chess moves and board state |\n", + "| `spacy`, `en_core_web_sm` | Sentence splitting and tokenisation for creativity scoring |\n", + "| `sentence_transformers` | Word embeddings for novelty / surprise |\n", + "| `benepar` | Constituency parsing (optional pipeline extension) |\n", + "\n", + "Install from the repository root:\n", + "\n", + "```bash\n", + "python -m venv .venv\n", + "source .venv/bin/activate # Windows: .venv\\Scripts\\activate\n", + "pip install -e \".[dev]\"\n", + "```\n", + "\n", + "#### Scenario\n", + "\n", + "> *A researcher is comparing two OpenRouter models on cognitive flexibility and creativity. They clone the miniBen repo, add their API key to `.env`, list available benchmarks from the terminal, then run the creativity benchmark for each model. miniBen makes five API calls per model, parses each story, computes per-call and aggregate creativity scores, and prints a summary Alex can paste into their lab report.*\n", + "\n", + "Steps in the scenario:\n", + "\n", + "1. Install miniBen and configure `OPENROUTER_API_KEY`.\n", + "2. Choose interface: **CLI** (`python -m miniBen.example`) or **Python** (`run_benchmark(...)`).\n", + "3. Select benchmark (`creativity` or `cognitive flexibility`) and model slug.\n", + "4. miniBen loops over five prompt variants → API → parse → score.\n", + "5. User reads printed report and/or inspects the returned `dict` for further analysis." + ] + }, + { + "cell_type": "markdown", + "id": "a6be0f857fc9ba21", + "metadata": {}, + "source": [ + "### 1.2 Flowchart — user, software, and main functions\n", + "\n", + "The flowchart below shows how the user interacts with miniBen and how data moves through each module. (GitHub renders Mermaid in [workflow.md](workflow.md); the notebook uses a text flowchart so it displays here without an image.)" + ] + }, + { + "cell_type": "markdown", + "id": "4b87331d8f2814d7", + "metadata": {}, + "source": [ + "```\n", + " ┌─────────────────────────────────────┐\n", + " │ USER │\n", + " │ CLI: python -m miniBen.example │\n", + " │ Python: run_benchmark(), AIModel │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ auth.py │\n", + " │ Load OPENROUTER_API_KEY (.env) │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ runner.py — run_benchmark() │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ prompts.py — 5 prompt variants │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ╔════════════╧════════════╗\n", + " ║ Loop: once per variant ║\n", + " ╚════════════╤════════════╝\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ model.py — AIModel.ask() │\n", + " └──────────┬───────────────┬──────────┘\n", + " │ │\n", + " prompt │ │ model reply\n", + " ▼ ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ OpenRouter API │\n", + " │ (external) │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ parsers.py — parse raw text │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ scorers.py — per-call + aggregate │\n", + " └──────────────────┬──────────────────┘\n", + " │\n", + " ▼\n", + " ┌─────────────────────────────────────┐\n", + " │ OUTPUT: console report + dict │\n", + " └─────────────────────────────────────┘\n", + "```\n", + "\n", + "**Rendered flowchart (Mermaid):** open [docs/workflow.md](workflow.md) on GitHub to see the diagram version.\n", + "\n", + "| Step | Module | What happens |\n", + "|------|--------|----------------|\n", + "| 1 | User | Run via CLI or call `run_benchmark()` / `AIModel.ask()` |\n", + "| 2 | `auth.py` | Ensure `OPENROUTER_API_KEY` is set |\n", + "| 3 | `runner.py` | `run_benchmark()` orchestrates five API calls |\n", + "| 4 | `prompts.py` | Build one prompt per variant (openings or word triplets) |\n", + "| 5 | `model.py` | `AIModel.ask()` sends each prompt to OpenRouter |\n", + "| 6 | `parsers.py` | Extract structured chess JSON or story text |\n", + "| 7 | `scorers.py` | Compute compliance or creativity metrics |\n", + "| 8 | Output | Printed trace and returned `dict` |" + ] + }, + { + "cell_type": "markdown", + "id": "7bd6f271897fe12b", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 2. How to Use the Software\n", + "\n", + "miniBen has **no graphical user interface (GUI)**. The \"UI\" consists of:\n", + "\n", + "| Interface | Where | Typical use |\n", + "|---|---|---|\n", + "| **Command-line (CLI)** | Terminal | Quick one-off benchmark runs |\n", + "| **Python API** | Script, REPL, or Jupyter | Programmatic control and custom analysis |\n", + "| **Console report** | stdout during `run_benchmark()` | Human-readable progress, parse output, and scores |\n", + "\n", + "### 2.1 Setup — API key\n", + "\n", + "Create a `.env` file in the project root:\n", + "\n", + "```env\n", + "OPENROUTER_API_KEY=sk-or-v1-your-key-here\n", + "```\n", + "\n", + "Or set the variable in your shell before running." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "41a60378a7f13542", + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import check_openrouter_api_key_exist\n", + "\n", + "print(\"API key configured:\", check_openrouter_api_key_exist())" + ] + }, + { + "cell_type": "markdown", + "id": "cee148e24097492f", + "metadata": {}, + "source": [ + "### 2.2 CLI — listing and running benchmarks\n", + "\n", + "**Screenshot equivalent:** terminal capture of `--list` and `--help`.\n", + "\n", + "```bash\n", + "python -m miniBen.example --help\n", + "python -m miniBen.example --list\n", + "python -m miniBen.example --benchmark creativity --model openrouter/free\n", + "python -m miniBen.example --benchmark \"cognitive flexibility\" --reasoning\n", + "```\n", + "\n", + "| Flag | Effect |\n", + "|---|---|\n", + "| `--list` | Print registered benchmarks and exit |\n", + "| `--benchmark` | Choose `creativity` or `cognitive flexibility` |\n", + "| `--model` | OpenRouter model slug (default: `openrouter/free`) |\n", + "| `--reasoning` | Request reasoning tokens when supported |\n", + "\n", + "Expected output of `--list`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "90297d3b1f6b7bde", + "metadata": {}, + "outputs": [], + "source": [ + "# Reproduce CLI --list output inside the notebook\n", + "from miniBen.example import list_benchmarks\n", + "\n", + "list_benchmarks()" + ] + }, + { + "cell_type": "markdown", + "id": "e9ffdab8d971fcfa", + "metadata": {}, + "source": [ + "### 2.3 Console report — what you see during a run\n", + "\n", + "When `run_benchmark()` executes, it prints a structured trace to the terminal. The report has **four recurring sections** per API call:\n", + "\n", + "1. **Header** — benchmark name, model, total calls \n", + "2. **Per call** — variant metadata → raw model response → parsed output → per-call score \n", + "3. **Aggregate score** — summary across all five calls \n", + "4. **Footer** — success / failure counts \n", + "\n", + "**Representative console output** (abbreviated; chess benchmark):" + ] + }, + { + "cell_type": "markdown", + "id": "20217f83a97ec597", + "metadata": {}, + "source": [ + "```\n", + "════════════════════════════════════════════════════════════\n", + " BENCHMARK : Meta-Chess Game (Cognitive Flexibility)\n", + " MODEL : openrouter/free\n", + " CALLS : 5\n", + "════════════════════════════════════════════════════════════\n", + "────────────────────────────────────────────────────────────\n", + " API Call 1/5 | variant: {'white': 'e4', 'black': 'e5'}\n", + "────────────────────────────────────────────────────────────\n", + "\n", + " RAW RESPONSE\n", + " <<>>{ \"trials\": [ ... ] }<<>>\n", + "\n", + " PARSED OUTPUT\n", + " [['e4', 'e5', ...], ['Nc3', 'Nc6', ...], ...]\n", + "\n", + " SCORE (call 1)\n", + " Compliance : 87.5% (6 violations / 48 moves)\n", + " Violations by piece type:\n", + " Trial 1 Pawn : 2 ██░░░░░░░░░░░░░░░░░░\n", + " Trial 2 Knight : 1 █░░░░░░░░░░░░░░░░░░░\n", + " ...\n", + "\n", + "════════════════════════════════════════════════════════════\n", + " AGGREGATE SCORE (all calls)\n", + " Calls scored : 5 / 5\n", + " Avg compliance : 82.3%\n", + " ...\n", + "════════════════════════════════════════════════════════════\n", + " BENCHMARK COMPLETE\n", + " Successful calls : 5 / 5\n", + " Failed calls : 0 / 5\n", + "════════════════════════════════════════════════════════════\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "90d0189df035054b", + "metadata": {}, + "source": [ + "### 2.4 Python API — programmatic interaction\n", + "\n", + "| Function / class | Purpose |\n", + "|---|---|\n", + "| `AIModel(model).ask(prompt, reasoning=...)` | Single OpenRouter call; returns `(content, reasoning)` |\n", + "| `run_benchmark(model_name, benchmark_name, reasoning=...)` | Full 5-call benchmark; returns results `dict` |\n", + "| `parse_chess_output(raw)` / `parse_creativity_output(raw)` | Parse model text offline |\n", + "| `score_chess(parsed_list)` / `score_creativity(parsed_list)` | Score parsed outputs offline |\n", + "\n", + "Inspect registered benchmarks:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e84c91966ac7f75", + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import BENCHMARKS\n", + "\n", + "for key, spec in BENCHMARKS.items():\n", + " print(f\"{key!r} → {spec['name']} ({len(spec['prompts']())} calls)\")" + ] + }, + { + "cell_type": "markdown", + "id": "7d0fc42707cf46b8", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## 3. Example Application\n", + "\n", + "This section demonstrates miniBen on **sample model outputs** (no API key required) and produces **figures** you can include in a report. Set `RUN_LIVE = True` at the end to run against a real model.\n", + "\n", + "### 3.1 Example A — Cognitive flexibility (Meta-Chess)\n", + "\n", + "We feed a reference Meta-Chess JSON response through the parser and scorer, then plot violations by trial (piece type)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "245aa93a63fa1edd", + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import parse_chess_output, score_chess, JSON_START, JSON_END\n", + "\n", + "sample_chess_raw = f\"\"\"{JSON_START}\n", + "{{\n", + " \"trials\": [\n", + " [\"e4\",\"e5\",\"f4\",\"f5\",\"g4\",\"g5\",\"fxg5\",\"fxg4\"],\n", + " [\"Nc3\",\"Nc6\",\"Nf3\",\"Nf6\",\"Nxe4\",\"Nxe5\",\"Nd6\",\"Nb4\"],\n", + " [\"Bc4\",\"Bc5\",\"Bd5\",\"Bd6\",\"Bxc6\",\"bxc6\",\"Bb5\",\"Bb4\"],\n", + " [\"O-O\",\"O-O\",\"Rfe1\",\"Rfe8\",\"Rxe8+\",\"Rxe8\",\"Rd1\",\"Rd8\"],\n", + " [\"Qf3\",\"Qf6\",\"Qxf6+\",\"gxf6\",\"Qd3\",\"Qd6\",\"Qe4\",\"Qe5\"],\n", + " [\"Kg1\",\"Kg8\",\"Kf1\",\"Kh8\",\"Kg2\",\"Kg7\",\"Kh3\",\"Kh6\"]\n", + " ]\n", + "}}\n", + "{JSON_END}\"\"\"\n", + "\n", + "parsed_chess = parse_chess_output(sample_chess_raw)\n", + "chess_score = score_chess([parsed_chess])\n", + "\n", + "print(f\"Compliance rate : {chess_score['compliance_rate']:.1%}\")\n", + "print(f\"Total moves : {chess_score['total_moves']}\")\n", + "print(f\"Total violations: {chess_score['total_violations']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d31d4ea4b88bd724", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "\n", + "piece_names = [\"Pawn\", \"Knight\", \"Bishop\", \"Rook\", \"Queen\", \"King\"]\n", + "violations = chess_score[\"per_trial_violations\"]\n", + "\n", + "fig, axes = plt.subplots(1, 2, figsize=(11, 4))\n", + "\n", + "# Figure 1: violations by trial\n", + "colors = plt.cm.RdYlGn_r(np.linspace(0.2, 0.8, len(piece_names)))\n", + "axes[0].bar(piece_names, violations, color=colors, edgecolor=\"#333\")\n", + "axes[0].set_title(\"Meta-Chess: violations by trial (piece type)\")\n", + "axes[0].set_xlabel(\"Trial / piece type\")\n", + "axes[0].set_ylabel(\"Number of violations\")\n", + "axes[0].tick_params(axis=\"x\", rotation=30)\n", + "\n", + "# Figure 2: compliance gauge\n", + "compliance = chess_score[\"compliance_rate\"]\n", + "axes[1].pie(\n", + " [compliance, 1 - compliance],\n", + " labels=[f\"Compliant\\n{compliance:.1%}\", f\"Violations\\n{1-compliance:.1%}\"],\n", + " colors=[\"#28a745\", \"#dc3545\"],\n", + " startangle=90,\n", + " autopct=\"\",\n", + ")\n", + "axes[1].set_title(\"Overall move compliance rate\")\n", + "\n", + "plt.tight_layout()\n", + "plt.savefig(\"docs/figures/chess_example.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(\"Saved: docs/figures/chess_example.png\")" + ] + }, + { + "cell_type": "markdown", + "id": "6228c3743dd0a5ec", + "metadata": {}, + "source": [ + "### 3.2 Example B — Creativity (short stories)\n", + "\n", + "Three sample five-sentence stories (matching the creativity prompt format) are parsed and scored. The creativity metric combines **novelty** (distinctiveness from the corpus) and **surprise** (sentence-to-sentence semantic shift)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d3267c9d4f18eb83", + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import parse_creativity_output, score_creativity\n", + "\n", + "stories_raw = [\n", + " \"She found an old stamp tucked inside the letter. \"\n", + " \"Without thinking, she decided to send it to her grandmother. \"\n", + " \"The post office clerk smiled at the faded ink. \"\n", + " \"Days later, a reply arrived with a story of its own. \"\n", + " \"Some messages travel farther than miles.\",\n", + " \"The kitchen smelled of spices before dawn. \"\n", + " \"She began to cook a meal no recipe described. \"\n", + " \"Steam rose like a curtain over the food. \"\n", + " \"Neighbors knocked, drawn by the aroma. \"\n", + " \"Hunger became the evening's quiet language.\",\n", + " \"He tightened the wheel and mounted the bike. \"\n", + " \"Wind pressed against him on the first ride. \"\n", + " \"A stranger waved from a hilltop. \"\n", + " \"They spoke of roads neither had taken. \"\n", + " \"He rode home knowing distance is a choice.\",\n", + "]\n", + "\n", + "parsed_stories = [parse_creativity_output(s) for s in stories_raw]\n", + "for i, p in enumerate(parsed_stories, 1):\n", + " print(f\"Story {i}: {p['sentence_count']} sentences\")\n", + " print(f\" {p['story'][:80]}…\\n\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "77ab00e0989817dc", + "metadata": {}, + "outputs": [], + "source": [ + "# Requires sentence_transformers + spaCy (first run downloads gte-large)\n", + "try:\n", + " from miniBen.scorers import compute_novelty, compute_surprise\n", + "\n", + " story_texts = [p[\"story\"] for p in parsed_stories]\n", + " novelty_scores = compute_novelty(story_texts)\n", + " surprise_scores = [compute_surprise(s)[0] for s in story_texts]\n", + " aggregate = score_creativity(parsed_stories)\n", + "\n", + " print(f\"Aggregate creativity score: {aggregate:.4f}\")\n", + "\n", + " fig, ax = plt.subplots(figsize=(8, 4))\n", + " x = np.arange(len(story_texts))\n", + " width = 0.35\n", + " ax.bar(x - width/2, novelty_scores, width, label=\"Novelty\", color=\"#5b8def\")\n", + " ax.bar(x + width/2, surprise_scores, width, label=\"Surprise\", color=\"#f4a261\")\n", + " ax.set_xticks(x)\n", + " ax.set_xticklabels([f\"Story {i+1}\" for i in range(len(story_texts))])\n", + " ax.set_ylabel(\"Score\")\n", + " ax.set_title(f\"Creativity components (aggregate = {aggregate:.4f})\")\n", + " ax.legend()\n", + " plt.tight_layout()\n", + " plt.savefig(\"docs/figures/creativity_example.png\", dpi=150, bbox_inches=\"tight\")\n", + " plt.show()\n", + " print(\"Saved: docs/figures/creativity_example.png\")\n", + "except Exception as e:\n", + " print(f\"Creativity scoring skipped ({type(e).__name__}: {e}).\")\n", + " print(\"Install dependencies and ensure network access for first-time model download.\")" + ] + }, + { + "cell_type": "markdown", + "id": "a606fa4702b8c407", + "metadata": {}, + "source": [ + "### 3.3 Example C — Simulated multi-call benchmark summary\n", + "\n", + "The cell below mimics what a user sees after five creativity calls: per-call scores and an aggregate. Values are illustrative; replace with `run_benchmark()` output for live data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "68d148b9636e43cc", + "metadata": {}, + "outputs": [], + "source": [ + "from miniBen import CREATIVITY_WORD_TRIPLETS\n", + "\n", + "# Illustrative per-call scores (replace with results[\"per_call_scores\"] from a live run)\n", + "illustrative_scores = [0.412, 0.387, 0.445, 0.401, 0.428]\n", + "\n", + "fig, ax = plt.subplots(figsize=(9, 4))\n", + "labels = [f\"Call {i+1}\\n{', '.join(w)}\" for i, w in enumerate(CREATIVITY_WORD_TRIPLETS)]\n", + "bars = ax.bar(labels, illustrative_scores, color=\"#6f42c1\", edgecolor=\"#333\", alpha=0.85)\n", + "ax.axhline(np.mean(illustrative_scores), color=\"#dc3545\", ls=\"--\", label=f\"Mean = {np.mean(illustrative_scores):.3f}\")\n", + "ax.set_ylim(0, max(illustrative_scores) * 1.2)\n", + "ax.set_ylabel(\"Creativity score\")\n", + "ax.set_title(\"Example: per-call creativity scores across five word triplets\")\n", + "ax.legend()\n", + "plt.xticks(rotation=15, ha=\"right\")\n", + "plt.tight_layout()\n", + "plt.savefig(\"docs/figures/benchmark_summary_example.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(\"Saved: docs/figures/benchmark_summary_example.png\")" + ] + }, + { + "cell_type": "markdown", + "id": "8e0d4ad1ce845ce5", + "metadata": {}, + "source": [ + "### 3.4 Optional — live benchmark run\n", + "\n", + "Set `RUN_LIVE = True` and ensure `OPENROUTER_API_KEY` is configured." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "afeb243a", + "metadata": {}, + "outputs": [], + "source": [ + "RUN_LIVE = False\n", + "MODEL = \"openrouter/free\"\n", + "\n", + "if RUN_LIVE and check_openrouter_api_key_exist():\n", + " from miniBen import run_benchmark\n", + "\n", + " results = run_benchmark(\n", + " model_name=MODEL,\n", + " benchmark_name=\"creativity\",\n", + " reasoning=False,\n", + " )\n", + " print(\"\\n--- returned dict (summary) ---\")\n", + " for k in [\"model\", \"benchmark\", \"successful_calls\", \"num_calls\", \"score\", \"per_call_scores\"]:\n", + " print(f\" {k}: {results[k]}\")\n", + "else:\n", + " print(\"Skipped. Set RUN_LIVE = True and configure OPENROUTER_API_KEY for a live run.\")" + ] + }, + { + "cell_type": "markdown", + "id": "7fa3478ebd41d39e", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Summary\n", + "\n", + "| Section | Content |\n", + "|---|---|\n", + "| **Purpose** | Evaluate LLMs on Meta-Chess and creativity benchmarks via OpenRouter |\n", + "| **User** | Python-literate researcher comparing models |\n", + "| **Interface** | CLI, Python API, console report (no GUI) |\n", + "| **Outputs** | Printed trace, results `dict`, optional matplotlib figures |\n", + "\n", + "**References**\n", + "\n", + "- miniBen repository: https://github.com/Programming-The-Next-Step-2026/mini_benchmarks \n", + "- Creativity benchmark: Ismayilzada et al. (2024) [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) \n", + "- OpenRouter models: https://openrouter.ai/models" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "84820a278dfaeaa1", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12.0" + } }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## 1. Task / Technique Description\n", - "\n", - "### 1.1 Scenario\n", - "\n", - "#### Statement of purpose\n", - "\n", - "Researchers and developers need a lightweight, reproducible way to compare LLMs on structured tasks without building a full evaluation pipeline from scratch. **miniBen** fills this gap by:\n", - "\n", - "1. Generating benchmark-specific prompts with fixed variant sets (5 calls per benchmark).\n", - "2. Sending prompts to any OpenRouter model.\n", - "3. Parsing model replies into structured data.\n", - "4. Scoring replies with task-specific metrics and printing a human-readable report.\n", - "\n", - "Two benchmarks are currently registered:\n", - "\n", - "| Key | Task | Score metric |\n", - "|---|---|---|\n", - "| `cognitive flexibility` | Meta-Chess — six trials, each restricted to one piece type | Move **compliance rate** (legal SAN + correct piece) |\n", - "| `creativity` | Five-sentence story with three required words | Weighted **novelty + surprise** (semantic distance) |\n", - "\n", - "#### Description of the individual user\n", - "\n", - "The primary user is a **computational social-science or NLP researcher** (graduate student or lab member) who:\n", - "\n", - "- Wants to compare LLMs on the benchmark protocol.\n", - "- Has (or can obtain) an OpenRouter API key for live model calls.\n", - "- Does **not** need a graphical desktop application — interaction happens through the CLI, Python API, or this notebook.\n", - "\n", - "#### Assumptions about equipment and dependencies\n", - "\n", - "| Requirement | Details |\n", - "|---|---|\n", - "| **Hardware** | Any laptop or server with internet access for live runs |\n", - "| **OS** | macOS, Linux, or Windows |\n", - "| **Python** | 3.9 or newer |\n", - "| **Network** | Required for OpenRouter API calls; creativity scoring also downloads `thenlper/gte-large` on first use |\n", - "| **API key** | `OPENROUTER_API_KEY` in `.env` or environment |\n", - "\n", - "**Python package dependencies** (from `pyproject.toml`):\n", - "\n", - "| Package | Role in miniBen |\n", - "|---|---|\n", - "| `openai` | HTTP client for OpenRouter chat completions |\n", - "| `python-dotenv` | Load API key from `.env` |\n", - "| `chess` | Validate Meta-Chess moves and board state |\n", - "| `spacy`, `en_core_web_sm` | Sentence splitting and tokenisation for creativity scoring |\n", - "| `sentence_transformers` | Word embeddings for novelty / surprise |\n", - "| `benepar` | Constituency parsing (optional pipeline extension) |\n", - "\n", - "Install from the repository root:\n", - "\n", - "```bash\n", - "python -m venv .venv\n", - "source .venv/bin/activate # Windows: .venv\\Scripts\\activate\n", - "pip install -e \".[dev]\"\n", - "```\n", - "\n", - "#### Scenario\n", - "\n", - "> *A researcher is comparing two OpenRouter models on cognitive flexibility and creativity. They clone the miniBen repo, add their API key to `.env`, list available benchmarks from the terminal, then run the creativity benchmark for each model. miniBen makes five API calls per model, parses each story, computes per-call and aggregate creativity scores, and prints a summary Alex can paste into their lab report.*\n", - "\n", - "Steps in the scenario:\n", - "\n", - "1. Install miniBen and configure `OPENROUTER_API_KEY`.\n", - "2. Choose interface: **CLI** (`python -m miniBen.example`) or **Python** (`run_benchmark(...)`).\n", - "3. Select benchmark (`creativity` or `cognitive flexibility`) and model slug.\n", - "4. miniBen loops over five prompt variants → API → parse → score.\n", - "5. User reads printed report and/or inspects the returned `dict` for further analysis." - ], - "id": "5cd5521eec80d3c4" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.2 Flowchart — user, software, and main functions\n", - "\n", - "The diagram below shows how the user interacts with miniBen and how data flows through the main modules." - ], - "id": "a6be0f857fc9ba21" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "```mermaid\n", - "flowchart TB\n", - " subgraph User[\"User\"]\n", - " U1[\"Researcher / developer\"]\n", - " U2[\"CLI: python -m miniBen.example\"]\n", - " U3[\"Python: run_benchmark() / AIModel.ask()\"]\n", - " end\n", - "\n", - " subgraph miniBen[\"miniBen package\"]\n", - " AUTH[\"auth.py\\ncheck / set API key\"]\n", - " RUN[\"runner.py\\nrun_benchmark()\"]\n", - " PROMPT[\"prompts.py\\ncog_flex_prompts()\\ncreativity_prompts()\"]\n", - " MODEL[\"model.py\\nAIModel.ask()\"]\n", - " PARSE[\"parsers.py\\nparse_chess_output()\\nparse_creativity_output()\"]\n", - " SCORE[\"scorers.py\\nscore_chess()\\nscore_creativity()\"]\n", - " end\n", - "\n", - " OR[\"OpenRouter API\"]\n", - " OUT[\"Console report + results dict\"]\n", - "\n", - " U1 --> U2\n", - " U1 --> U3\n", - " U2 --> AUTH\n", - " U3 --> AUTH\n", - " AUTH --> RUN\n", - " RUN --> PROMPT\n", - " PROMPT -->|\"5 prompts + variants\"| MODEL\n", - " MODEL -->|\"HTTP POST\"| OR\n", - " OR -->|\"raw text\"| MODEL\n", - " MODEL -->|\"content, reasoning\"| PARSE\n", - " PARSE -->|\"structured parse\"| SCORE\n", - " SCORE -->|\"per-call + aggregate score\"| OUT\n", - " RUN --> OUT\n", - "```" - ], - "id": "4b87331d8f2814d7" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Static flowchart figure (for PDF/export when Mermaid is unavailable)\n", - "import matplotlib.pyplot as plt\n", - "from matplotlib.patches import FancyBboxPatch, FancyArrowPatch\n", - "\n", - "fig, ax = plt.subplots(figsize=(11, 6))\n", - "ax.set_xlim(0, 11)\n", - "ax.set_ylim(0, 6)\n", - "ax.axis(\"off\")\n", - "ax.set_title(\"miniBen workflow: user → software → OpenRouter → scores\", fontsize=13, pad=12)\n", - "\n", - "def box(x, y, w, h, text, fc=\"#e8f4fd\", ec=\"#2c5282\"):\n", - " p = FancyBboxPatch((x, y), w, h, boxstyle=\"round,pad=0.02\", fc=fc, ec=ec, lw=1.2)\n", - " ax.add_patch(p)\n", - " ax.text(x + w/2, y + h/2, text, ha=\"center\", va=\"center\", fontsize=9, wrap=True)\n", - "\n", - "def arrow(x1, y1, x2, y2):\n", - " ax.add_patch(FancyArrowPatch((x1, y1), (x2, y2), arrowstyle=\"->\", mutation_scale=12, lw=1.2, color=\"#444\"))\n", - "\n", - "box(0.3, 4.5, 1.8, 0.9, \"User\\n(CLI / Python)\", fc=\"#fff3cd\")\n", - "box(2.5, 4.5, 1.6, 0.9, \"auth.py\", fc=\"#e8f4fd\")\n", - "box(4.5, 4.5, 1.8, 0.9, \"run_benchmark\\nrunner.py\", fc=\"#e8f4fd\")\n", - "box(6.8, 4.5, 1.6, 0.9, \"prompts.py\\n(5 variants)\", fc=\"#e8f4fd\")\n", - "box(4.5, 2.8, 1.8, 0.9, \"AIModel.ask\\nmodel.py\", fc=\"#e8f4fd\")\n", - "box(6.8, 2.8, 1.6, 0.9, \"OpenRouter\\nAPI\", fc=\"#f8d7da\")\n", - "box(2.5, 2.8, 1.6, 0.9, \"parsers.py\", fc=\"#e8f4fd\")\n", - "box(0.3, 2.8, 1.8, 0.9, \"scorers.py\", fc=\"#e8f4fd\")\n", - "box(0.3, 1.0, 2.2, 0.9, \"Console report\\n+ results dict\", fc=\"#d4edda\")\n", - "\n", - "arrow(2.1, 4.95, 2.5, 4.95)\n", - "arrow(4.1, 4.95, 4.5, 4.95)\n", - "arrow(6.3, 4.95, 6.8, 4.95)\n", - "arrow(7.6, 4.5, 5.4, 3.7)\n", - "arrow(6.3, 3.25, 6.8, 3.25)\n", - "arrow(6.8, 3.25, 6.3, 3.25)\n", - "arrow(4.5, 3.25, 4.1, 3.25)\n", - "arrow(2.5, 3.25, 2.1, 3.25)\n", - "arrow(1.2, 2.8, 1.2, 1.9)\n", - "\n", - "plt.tight_layout()\n", - "plt.show()" - ], - "id": "2b5844139e1e62bb" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## 2. How to Use the Software\n", - "\n", - "miniBen has **no graphical user interface (GUI)**. The \"UI\" consists of:\n", - "\n", - "| Interface | Where | Typical use |\n", - "|---|---|---|\n", - "| **Command-line (CLI)** | Terminal | Quick one-off benchmark runs |\n", - "| **Python API** | Script, REPL, or Jupyter | Programmatic control and custom analysis |\n", - "| **Console report** | stdout during `run_benchmark()` | Human-readable progress, parse output, and scores |\n", - "\n", - "### 2.1 Setup — API key\n", - "\n", - "Create a `.env` file in the project root:\n", - "\n", - "```env\n", - "OPENROUTER_API_KEY=sk-or-v1-your-key-here\n", - "```\n", - "\n", - "Or set the variable in your shell before running." - ], - "id": "7bd6f271897fe12b" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from miniBen import check_openrouter_api_key_exist\n", - "\n", - "print(\"API key configured:\", check_openrouter_api_key_exist())" - ], - "id": "41a60378a7f13542" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.2 CLI — listing and running benchmarks\n", - "\n", - "**Screenshot equivalent:** terminal capture of `--list` and `--help`.\n", - "\n", - "```bash\n", - "python -m miniBen.example --help\n", - "python -m miniBen.example --list\n", - "python -m miniBen.example --benchmark creativity --model openrouter/free\n", - "python -m miniBen.example --benchmark \"cognitive flexibility\" --reasoning\n", - "```\n", - "\n", - "| Flag | Effect |\n", - "|---|---|\n", - "| `--list` | Print registered benchmarks and exit |\n", - "| `--benchmark` | Choose `creativity` or `cognitive flexibility` |\n", - "| `--model` | OpenRouter model slug (default: `openrouter/free`) |\n", - "| `--reasoning` | Request reasoning tokens when supported |\n", - "\n", - "Expected output of `--list`:" - ], - "id": "cee148e24097492f" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Reproduce CLI --list output inside the notebook\n", - "from miniBen.example import list_benchmarks\n", - "\n", - "list_benchmarks()" - ], - "id": "90297d3b1f6b7bde" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.3 Console report — what you see during a run\n", - "\n", - "When `run_benchmark()` executes, it prints a structured trace to the terminal. The report has **four recurring sections** per API call:\n", - "\n", - "1. **Header** — benchmark name, model, total calls \n", - "2. **Per call** — variant metadata → raw model response → parsed output → per-call score \n", - "3. **Aggregate score** — summary across all five calls \n", - "4. **Footer** — success / failure counts \n", - "\n", - "**Representative console output** (abbreviated; chess benchmark):" - ], - "id": "e9ffdab8d971fcfa" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "```\n", - "════════════════════════════════════════════════════════════\n", - " BENCHMARK : Meta-Chess Game (Cognitive Flexibility)\n", - " MODEL : openrouter/free\n", - " CALLS : 5\n", - "════════════════════════════════════════════════════════════\n", - "────────────────────────────────────────────────────────────\n", - " API Call 1/5 | variant: {'white': 'e4', 'black': 'e5'}\n", - "────────────────────────────────────────────────────────────\n", - "\n", - " RAW RESPONSE\n", - " <<>>{ \"trials\": [ ... ] }<<>>\n", - "\n", - " PARSED OUTPUT\n", - " [['e4', 'e5', ...], ['Nc3', 'Nc6', ...], ...]\n", - "\n", - " SCORE (call 1)\n", - " Compliance : 87.5% (6 violations / 48 moves)\n", - " Violations by piece type:\n", - " Trial 1 Pawn : 2 ██░░░░░░░░░░░░░░░░░░\n", - " Trial 2 Knight : 1 █░░░░░░░░░░░░░░░░░░░\n", - " ...\n", - "\n", - "════════════════════════════════════════════════════════════\n", - " AGGREGATE SCORE (all calls)\n", - " Calls scored : 5 / 5\n", - " Avg compliance : 82.3%\n", - " ...\n", - "════════════════════════════════════════════════════════════\n", - " BENCHMARK COMPLETE\n", - " Successful calls : 5 / 5\n", - " Failed calls : 0 / 5\n", - "════════════════════════════════════════════════════════════\n", - "```" - ], - "id": "20217f83a97ec597" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.4 Python API — programmatic interaction\n", - "\n", - "| Function / class | Purpose |\n", - "|---|---|\n", - "| `AIModel(model).ask(prompt, reasoning=...)` | Single OpenRouter call; returns `(content, reasoning)` |\n", - "| `run_benchmark(model_name, benchmark_name, reasoning=...)` | Full 5-call benchmark; returns results `dict` |\n", - "| `parse_chess_output(raw)` / `parse_creativity_output(raw)` | Parse model text offline |\n", - "| `score_chess(parsed_list)` / `score_creativity(parsed_list)` | Score parsed outputs offline |\n", - "\n", - "Inspect registered benchmarks:" - ], - "id": "90d0189df035054b" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from miniBen import BENCHMARKS\n", - "\n", - "for key, spec in BENCHMARKS.items():\n", - " print(f\"{key!r} → {spec['name']} ({len(spec['prompts']())} calls)\")" - ], - "id": "e84c91966ac7f75" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## 3. Example Application\n", - "\n", - "This section demonstrates miniBen on **sample model outputs** (no API key required) and produces **figures** you can include in a report. Set `RUN_LIVE = True` at the end to run against a real model.\n", - "\n", - "### 3.1 Example A — Cognitive flexibility (Meta-Chess)\n", - "\n", - "We feed a reference Meta-Chess JSON response through the parser and scorer, then plot violations by trial (piece type)." - ], - "id": "7d0fc42707cf46b8" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from miniBen import parse_chess_output, score_chess, JSON_START, JSON_END\n", - "\n", - "sample_chess_raw = f\"\"\"{JSON_START}\n", - "{{\n", - " \"trials\": [\n", - " [\"e4\",\"e5\",\"f4\",\"f5\",\"g4\",\"g5\",\"fxg5\",\"fxg4\"],\n", - " [\"Nc3\",\"Nc6\",\"Nf3\",\"Nf6\",\"Nxe4\",\"Nxe5\",\"Nd6\",\"Nb4\"],\n", - " [\"Bc4\",\"Bc5\",\"Bd5\",\"Bd6\",\"Bxc6\",\"bxc6\",\"Bb5\",\"Bb4\"],\n", - " [\"O-O\",\"O-O\",\"Rfe1\",\"Rfe8\",\"Rxe8+\",\"Rxe8\",\"Rd1\",\"Rd8\"],\n", - " [\"Qf3\",\"Qf6\",\"Qxf6+\",\"gxf6\",\"Qd3\",\"Qd6\",\"Qe4\",\"Qe5\"],\n", - " [\"Kg1\",\"Kg8\",\"Kf1\",\"Kh8\",\"Kg2\",\"Kg7\",\"Kh3\",\"Kh6\"]\n", - " ]\n", - "}}\n", - "{JSON_END}\"\"\"\n", - "\n", - "parsed_chess = parse_chess_output(sample_chess_raw)\n", - "chess_score = score_chess([parsed_chess])\n", - "\n", - "print(f\"Compliance rate : {chess_score['compliance_rate']:.1%}\")\n", - "print(f\"Total moves : {chess_score['total_moves']}\")\n", - "print(f\"Total violations: {chess_score['total_violations']}\")" - ], - "id": "245aa93a63fa1edd" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n", - "import numpy as np\n", - "\n", - "piece_names = [\"Pawn\", \"Knight\", \"Bishop\", \"Rook\", \"Queen\", \"King\"]\n", - "violations = chess_score[\"per_trial_violations\"]\n", - "\n", - "fig, axes = plt.subplots(1, 2, figsize=(11, 4))\n", - "\n", - "# Figure 1: violations by trial\n", - "colors = plt.cm.RdYlGn_r(np.linspace(0.2, 0.8, len(piece_names)))\n", - "axes[0].bar(piece_names, violations, color=colors, edgecolor=\"#333\")\n", - "axes[0].set_title(\"Meta-Chess: violations by trial (piece type)\")\n", - "axes[0].set_xlabel(\"Trial / piece type\")\n", - "axes[0].set_ylabel(\"Number of violations\")\n", - "axes[0].tick_params(axis=\"x\", rotation=30)\n", - "\n", - "# Figure 2: compliance gauge\n", - "compliance = chess_score[\"compliance_rate\"]\n", - "axes[1].pie(\n", - " [compliance, 1 - compliance],\n", - " labels=[f\"Compliant\\n{compliance:.1%}\", f\"Violations\\n{1-compliance:.1%}\"],\n", - " colors=[\"#28a745\", \"#dc3545\"],\n", - " startangle=90,\n", - " autopct=\"\",\n", - ")\n", - "axes[1].set_title(\"Overall move compliance rate\")\n", - "\n", - "plt.tight_layout()\n", - "plt.savefig(\"docs/figures/chess_example.png\", dpi=150, bbox_inches=\"tight\")\n", - "plt.show()\n", - "print(\"Saved: docs/figures/chess_example.png\")" - ], - "id": "d31d4ea4b88bd724" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2 Example B — Creativity (short stories)\n", - "\n", - "Three sample five-sentence stories (matching the creativity prompt format) are parsed and scored. The creativity metric combines **novelty** (distinctiveness from the corpus) and **surprise** (sentence-to-sentence semantic shift)." - ], - "id": "6228c3743dd0a5ec" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from miniBen import parse_creativity_output, score_creativity\n", - "\n", - "stories_raw = [\n", - " \"She found an old stamp tucked inside the letter. \"\n", - " \"Without thinking, she decided to send it to her grandmother. \"\n", - " \"The post office clerk smiled at the faded ink. \"\n", - " \"Days later, a reply arrived with a story of its own. \"\n", - " \"Some messages travel farther than miles.\",\n", - " \"The kitchen smelled of spices before dawn. \"\n", - " \"She began to cook a meal no recipe described. \"\n", - " \"Steam rose like a curtain over the food. \"\n", - " \"Neighbors knocked, drawn by the aroma. \"\n", - " \"Hunger became the evening's quiet language.\",\n", - " \"He tightened the wheel and mounted the bike. \"\n", - " \"Wind pressed against him on the first ride. \"\n", - " \"A stranger waved from a hilltop. \"\n", - " \"They spoke of roads neither had taken. \"\n", - " \"He rode home knowing distance is a choice.\",\n", - "]\n", - "\n", - "parsed_stories = [parse_creativity_output(s) for s in stories_raw]\n", - "for i, p in enumerate(parsed_stories, 1):\n", - " print(f\"Story {i}: {p['sentence_count']} sentences\")\n", - " print(f\" {p['story'][:80]}…\\n\")" - ], - "id": "d3267c9d4f18eb83" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Requires sentence_transformers + spaCy (first run downloads gte-large)\n", - "try:\n", - " from miniBen.scorers import compute_novelty, compute_surprise\n", - "\n", - " story_texts = [p[\"story\"] for p in parsed_stories]\n", - " novelty_scores = compute_novelty(story_texts)\n", - " surprise_scores = [compute_surprise(s)[0] for s in story_texts]\n", - " aggregate = score_creativity(parsed_stories)\n", - "\n", - " print(f\"Aggregate creativity score: {aggregate:.4f}\")\n", - "\n", - " fig, ax = plt.subplots(figsize=(8, 4))\n", - " x = np.arange(len(story_texts))\n", - " width = 0.35\n", - " ax.bar(x - width/2, novelty_scores, width, label=\"Novelty\", color=\"#5b8def\")\n", - " ax.bar(x + width/2, surprise_scores, width, label=\"Surprise\", color=\"#f4a261\")\n", - " ax.set_xticks(x)\n", - " ax.set_xticklabels([f\"Story {i+1}\" for i in range(len(story_texts))])\n", - " ax.set_ylabel(\"Score\")\n", - " ax.set_title(f\"Creativity components (aggregate = {aggregate:.4f})\")\n", - " ax.legend()\n", - " plt.tight_layout()\n", - " plt.savefig(\"docs/figures/creativity_example.png\", dpi=150, bbox_inches=\"tight\")\n", - " plt.show()\n", - " print(\"Saved: docs/figures/creativity_example.png\")\n", - "except Exception as e:\n", - " print(f\"Creativity scoring skipped ({type(e).__name__}: {e}).\")\n", - " print(\"Install dependencies and ensure network access for first-time model download.\")" - ], - "id": "77ab00e0989817dc" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.3 Example C — Simulated multi-call benchmark summary\n", - "\n", - "The cell below mimics what a user sees after five creativity calls: per-call scores and an aggregate. Values are illustrative; replace with `run_benchmark()` output for live data." - ], - "id": "a606fa4702b8c407" - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from miniBen import CREATIVITY_WORD_TRIPLETS\n", - "\n", - "# Illustrative per-call scores (replace with results[\"per_call_scores\"] from a live run)\n", - "illustrative_scores = [0.412, 0.387, 0.445, 0.401, 0.428]\n", - "\n", - "fig, ax = plt.subplots(figsize=(9, 4))\n", - "labels = [f\"Call {i+1}\\n{', '.join(w)}\" for i, w in enumerate(CREATIVITY_WORD_TRIPLETS)]\n", - "bars = ax.bar(labels, illustrative_scores, color=\"#6f42c1\", edgecolor=\"#333\", alpha=0.85)\n", - "ax.axhline(np.mean(illustrative_scores), color=\"#dc3545\", ls=\"--\", label=f\"Mean = {np.mean(illustrative_scores):.3f}\")\n", - "ax.set_ylim(0, max(illustrative_scores) * 1.2)\n", - "ax.set_ylabel(\"Creativity score\")\n", - "ax.set_title(\"Example: per-call creativity scores across five word triplets\")\n", - "ax.legend()\n", - "plt.xticks(rotation=15, ha=\"right\")\n", - "plt.tight_layout()\n", - "plt.savefig(\"docs/figures/benchmark_summary_example.png\", dpi=150, bbox_inches=\"tight\")\n", - "plt.show()\n", - "print(\"Saved: docs/figures/benchmark_summary_example.png\")" - ], - "id": "68d148b9636e43cc" - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.4 Optional — live benchmark run\n", - "\n", - "Set `RUN_LIVE = True` and ensure `OPENROUTER_API_KEY` is configured." - ], - "id": "8e0d4ad1ce845ce5" - }, - { - "cell_type": "code", - "execution_count": null, - "id": "afeb243a", - "metadata": {}, - "outputs": [], - "source": [ - "RUN_LIVE = False\n", - "MODEL = \"openrouter/free\"\n", - "\n", - "if RUN_LIVE and check_openrouter_api_key_exist():\n", - " from miniBen import run_benchmark\n", - "\n", - " results = run_benchmark(\n", - " model_name=MODEL,\n", - " benchmark_name=\"creativity\",\n", - " reasoning=False,\n", - " )\n", - " print(\"\\n--- returned dict (summary) ---\")\n", - " for k in [\"model\", \"benchmark\", \"successful_calls\", \"num_calls\", \"score\", \"per_call_scores\"]:\n", - " print(f\" {k}: {results[k]}\")\n", - "else:\n", - " print(\"Skipped. Set RUN_LIVE = True and configure OPENROUTER_API_KEY for a live run.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Summary\n", - "\n", - "| Section | Content |\n", - "|---|---|\n", - "| **Purpose** | Evaluate LLMs on Meta-Chess and creativity benchmarks via OpenRouter |\n", - "| **User** | Python-literate researcher comparing models |\n", - "| **Interface** | CLI, Python API, console report (no GUI) |\n", - "| **Outputs** | Printed trace, results `dict`, optional matplotlib figures |\n", - "\n", - "**References**\n", - "\n", - "- miniBen repository: https://github.com/Programming-The-Next-Step-2026/mini_benchmarks \n", - "- Creativity benchmark: Ismayilzada et al. (2024) [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) \n", - "- OpenRouter models: https://openrouter.ai/models" - ], - "id": "7fa3478ebd41d39e" - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "name": "python", - "version": "3.12.0" - } - }, - "nbformat": 4, - "nbformat_minor": 5 + "nbformat": 4, + "nbformat_minor": 5 } diff --git a/docs/workflow.md b/docs/workflow.md new file mode 100644 index 0000000..e63d243 --- /dev/null +++ b/docs/workflow.md @@ -0,0 +1,44 @@ +# miniBen workflow + +This page renders as a **flowchart on GitHub** (Mermaid is supported in `.md` files, not in `.ipynb` cells). + +```mermaid +flowchart TD + subgraph user["User"] + CLI["CLI: python -m miniBen.example"] + PY["Python: run_benchmark() / AIModel.ask()"] + end + + subgraph miniben["miniBen package"] + AUTH["auth.py — API key"] + RUN["runner.py — run_benchmark()"] + PROMPT["prompts.py — 5 variants"] + MODEL["model.py — AIModel.ask()"] + PARSE["parsers.py"] + SCORE["scorers.py"] + end + + OR["OpenRouter API"] + OUT["Console report + results dict"] + + CLI --> AUTH + PY --> AUTH + AUTH --> RUN + RUN --> PROMPT + PROMPT --> MODEL + MODEL <-->|"HTTP request / response"| OR + MODEL --> PARSE + PARSE --> SCORE + SCORE --> OUT + RUN --> OUT +``` + +| Step | Module | Main function | +|------|--------|----------------| +| 1 | `auth.py` | Load or prompt for `OPENROUTER_API_KEY` | +| 2 | `runner.py` | `run_benchmark()` — orchestrates the run | +| 3 | `prompts.py` | Build five benchmark-specific prompts | +| 4 | `model.py` | `AIModel.ask()` — one API call per variant | +| 5 | `parsers.py` | `parse_chess_output()` / `parse_creativity_output()` | +| 6 | `scorers.py` | `score_chess()` / `score_creativity()` | +| 7 | — | Printed report and returned `dict` | From 6f8de525d83b10a9fe94088123926f62c37e23c3 Mon Sep 17 00:00:00 2001 From: Leyla Yang Date: Sun, 24 May 2026 22:07:40 +0200 Subject: [PATCH 9/9] update dependency --- .github/workflows/python-tests.yaml | 9 +- docs/vignette.ipynb | 258 ++++++++++++++-------------- docs/workflow.md | 104 +++++++---- pyproject.toml | 1 - 4 files changed, 202 insertions(+), 170 deletions(-) diff --git a/.github/workflows/python-tests.yaml b/.github/workflows/python-tests.yaml index 93838af..6ac03a2 100644 --- a/.github/workflows/python-tests.yaml +++ b/.github/workflows/python-tests.yaml @@ -10,15 +10,18 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 - - uses: actions/setup-python@v6 + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install package and test tools run: | python -m pip install --upgrade pip - python -m pip install -e . pytest # <- if you need to add many packages here, your package setup is probably missing that information! + python -m pip install -e . pytest + + - name: Download spaCy model + run: python -m spacy download en_core_web_sm - name: Run unit tests run: pytest \ No newline at end of file diff --git a/docs/vignette.ipynb b/docs/vignette.ipynb index dd02689..ae1cd95 100644 --- a/docs/vignette.ipynb +++ b/docs/vignette.ipynb @@ -2,7 +2,6 @@ "cells": [ { "cell_type": "markdown", - "id": "b6b5ab82af110bc0", "metadata": {}, "source": [ "# miniBen — Vignette / Report\n", @@ -11,11 +10,11 @@ "**Package:** [miniBen](https://github.com/Programming-The-Next-Step-2026/mini_benchmarks) — Mini LLM Benchmarks\n", "\n", "This document describes the purpose, users, workflow, and usage of **miniBen**, a Python package that evaluates large language models (LLMs) on small cognitive benchmarks via the [OpenRouter](https://openrouter.ai/) API." - ] + ], + "id": "b6b5ab82af110bc0" }, { "cell_type": "markdown", - "id": "5cd5521eec80d3c4", "metadata": {}, "source": [ "---\n", @@ -88,95 +87,87 @@ "3. Select benchmark (`creativity` or `cognitive flexibility`) and model slug.\n", "4. miniBen loops over five prompt variants → API → parse → score.\n", "5. User reads printed report and/or inspects the returned `dict` for further analysis." - ] + ], + "id": "5cd5521eec80d3c4" }, { "cell_type": "markdown", - "id": "a6be0f857fc9ba21", "metadata": {}, "source": [ "### 1.2 Flowchart — user, software, and main functions\n", "\n", - "The flowchart below shows how the user interacts with miniBen and how data moves through each module. (GitHub renders Mermaid in [workflow.md](workflow.md); the notebook uses a text flowchart so it displays here without an image.)" - ] + "There are **two entry paths**: a full benchmark run (`run_benchmark` / CLI) and a single API call (`AIModel.ask`). The text flowchart below matches the code in `runner.py` and `example.py`. A Mermaid diagram is in [workflow.md](workflow.md)." + ], + "id": "a6be0f857fc9ba21" }, { "cell_type": "markdown", - "id": "4b87331d8f2814d7", "metadata": {}, "source": [ + "#### Path A — full benchmark (`run_benchmark` or CLI)\n", + "\n", + "```\n", + " import miniBen\n", + " │\n", + " └── auth.py: load_dotenv() loads .env automatically (on import)\n", + "\n", + " CLI only (example.py):\n", + " └── if no key → prompt user for OPENROUTER_API_KEY\n", + " (run_benchmark itself does NOT check or prompt for a key)\n", + "\n", + " run_benchmark(model_name, benchmark_name)\n", + " │\n", + " ├── runner.py: lookup BENCHMARKS → pick prompts, parser, scorer\n", + " ├── prompts.py: build ALL 5 prompts + variants (once, before loop)\n", + " └── model.py: create one AIModel instance\n", + "\n", + " ┌──── loop i = 1 … 5 ─────────────────────────────────────────────┐\n", + " │ │\n", + " │ model.py: AIModel.ask(prompt_i) ◄──────► OpenRouter API │\n", + " │ │ │\n", + " │ ▼ raw text │\n", + " │ parsers.py: parser(content) → parsed_i │\n", + " │ │ │\n", + " │ ▼ │\n", + " │ scorers.py: scorer([parsed_i]) → per-call score (printed) │\n", + " │ │\n", + " └──────────────────────────┬───────────────────────────────────────┘\n", + " │ after all 5 calls\n", + " ▼\n", + " scorers.py: scorer(all parsed) → aggregate score (e.g. corpus novelty)\n", + " │\n", + " ├── print summary + success/fail counts\n", + " └── return results dict (score, outputs, per_call_scores, …)\n", + "```\n", + "\n", + "#### Path B — single question (`AIModel.ask` only)\n", + "\n", "```\n", - " ┌─────────────────────────────────────┐\n", - " │ USER │\n", - " │ CLI: python -m miniBen.example │\n", - " │ Python: run_benchmark(), AIModel │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ auth.py │\n", - " │ Load OPENROUTER_API_KEY (.env) │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ runner.py — run_benchmark() │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ prompts.py — 5 prompt variants │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ╔════════════╧════════════╗\n", - " ║ Loop: once per variant ║\n", - " ╚════════════╤════════════╝\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ model.py — AIModel.ask() │\n", - " └──────────┬───────────────┬──────────┘\n", - " │ │\n", - " prompt │ │ model reply\n", - " ▼ ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ OpenRouter API │\n", - " │ (external) │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ parsers.py — parse raw text │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ scorers.py — per-call + aggregate │\n", - " └──────────────────┬──────────────────┘\n", - " │\n", - " ▼\n", - " ┌─────────────────────────────────────┐\n", - " │ OUTPUT: console report + dict │\n", - " └─────────────────────────────────────┘\n", + " AIModel(model).ask(prompt) ◄──────► OpenRouter API\n", + " │\n", + " └── return (content, reasoning)\n", + " (no prompts batch, parser, or scorer unless you call them yourself)\n", "```\n", "\n", - "**Rendered flowchart (Mermaid):** open [docs/workflow.md](workflow.md) on GitHub to see the diagram version.\n", - "\n", - "| Step | Module | What happens |\n", - "|------|--------|----------------|\n", - "| 1 | User | Run via CLI or call `run_benchmark()` / `AIModel.ask()` |\n", - "| 2 | `auth.py` | Ensure `OPENROUTER_API_KEY` is set |\n", - "| 3 | `runner.py` | `run_benchmark()` orchestrates five API calls |\n", - "| 4 | `prompts.py` | Build one prompt per variant (openings or word triplets) |\n", - "| 5 | `model.py` | `AIModel.ask()` sends each prompt to OpenRouter |\n", - "| 6 | `parsers.py` | Extract structured chess JSON or story text |\n", - "| 7 | `scorers.py` | Compute compliance or creativity metrics |\n", - "| 8 | Output | Printed trace and returned `dict` |" - ] + "**Mermaid version:** [workflow.md](workflow.md) on GitHub (renders as a diagram in `.md` files).\n", + "\n", + "| Phase | Module | What happens |\n", + "|-------|--------|----------------|\n", + "| Import | `auth.py` | `load_dotenv()` reads `.env` |\n", + "| CLI only | `example.py` | Check / prompt for API key |\n", + "| Setup | `runner.py` | `run_benchmark()` — lookup `BENCHMARKS`, validate benchmark |\n", + "| Setup | `prompts.py` | Build all 5 prompts + variants **before** the loop |\n", + "| Setup | `model.py` | Create one `AIModel` for the run |\n", + "| Loop ×5 | `model.py` | `ask(prompt_i)` → OpenRouter |\n", + "| Loop ×5 | `parsers.py` | Parse raw reply |\n", + "| Loop ×5 | `scorers.py` | Per-call score (`scorer([parsed_i])`) |\n", + "| After loop | `scorers.py` | Aggregate score (`scorer(parsed_list)`) |\n", + "| After loop | `runner.py` | Print report; return `dict` |" + ], + "id": "4b87331d8f2814d7" }, { "cell_type": "markdown", - "id": "7bd6f271897fe12b", "metadata": {}, "source": [ "---\n", @@ -200,23 +191,23 @@ "```\n", "\n", "Or set the variable in your shell before running." - ] + ], + "id": "7bd6f271897fe12b" }, { "cell_type": "code", - "execution_count": null, - "id": "41a60378a7f13542", "metadata": {}, - "outputs": [], "source": [ "from miniBen import check_openrouter_api_key_exist\n", "\n", "print(\"API key configured:\", check_openrouter_api_key_exist())" - ] + ], + "execution_count": null, + "outputs": [], + "id": "41a60378a7f13542" }, { "cell_type": "markdown", - "id": "cee148e24097492f", "metadata": {}, "source": [ "### 2.2 CLI — listing and running benchmarks\n", @@ -238,24 +229,24 @@ "| `--reasoning` | Request reasoning tokens when supported |\n", "\n", "Expected output of `--list`:" - ] + ], + "id": "cee148e24097492f" }, { "cell_type": "code", - "execution_count": null, - "id": "90297d3b1f6b7bde", "metadata": {}, - "outputs": [], "source": [ "# Reproduce CLI --list output inside the notebook\n", "from miniBen.example import list_benchmarks\n", "\n", "list_benchmarks()" - ] + ], + "execution_count": null, + "outputs": [], + "id": "90297d3b1f6b7bde" }, { "cell_type": "markdown", - "id": "e9ffdab8d971fcfa", "metadata": {}, "source": [ "### 2.3 Console report — what you see during a run\n", @@ -268,11 +259,11 @@ "4. **Footer** — success / failure counts \n", "\n", "**Representative console output** (abbreviated; chess benchmark):" - ] + ], + "id": "e9ffdab8d971fcfa" }, { "cell_type": "markdown", - "id": "20217f83a97ec597", "metadata": {}, "source": [ "```\n", @@ -309,11 +300,11 @@ " Failed calls : 0 / 5\n", "════════════════════════════════════════════════════════════\n", "```" - ] + ], + "id": "20217f83a97ec597" }, { "cell_type": "markdown", - "id": "90d0189df035054b", "metadata": {}, "source": [ "### 2.4 Python API — programmatic interaction\n", @@ -326,24 +317,24 @@ "| `score_chess(parsed_list)` / `score_creativity(parsed_list)` | Score parsed outputs offline |\n", "\n", "Inspect registered benchmarks:" - ] + ], + "id": "90d0189df035054b" }, { "cell_type": "code", - "execution_count": null, - "id": "e84c91966ac7f75", "metadata": {}, - "outputs": [], "source": [ "from miniBen import BENCHMARKS\n", "\n", "for key, spec in BENCHMARKS.items():\n", " print(f\"{key!r} → {spec['name']} ({len(spec['prompts']())} calls)\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "e84c91966ac7f75" }, { "cell_type": "markdown", - "id": "7d0fc42707cf46b8", "metadata": {}, "source": [ "---\n", @@ -355,14 +346,12 @@ "### 3.1 Example A — Cognitive flexibility (Meta-Chess)\n", "\n", "We feed a reference Meta-Chess JSON response through the parser and scorer, then plot violations by trial (piece type)." - ] + ], + "id": "7d0fc42707cf46b8" }, { "cell_type": "code", - "execution_count": null, - "id": "245aa93a63fa1edd", "metadata": {}, - "outputs": [], "source": [ "from miniBen import parse_chess_output, score_chess, JSON_START, JSON_END\n", "\n", @@ -385,14 +374,14 @@ "print(f\"Compliance rate : {chess_score['compliance_rate']:.1%}\")\n", "print(f\"Total moves : {chess_score['total_moves']}\")\n", "print(f\"Total violations: {chess_score['total_violations']}\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "245aa93a63fa1edd" }, { "cell_type": "code", - "execution_count": null, - "id": "d31d4ea4b88bd724", "metadata": {}, - "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -425,24 +414,24 @@ "plt.savefig(\"docs/figures/chess_example.png\", dpi=150, bbox_inches=\"tight\")\n", "plt.show()\n", "print(\"Saved: docs/figures/chess_example.png\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "d31d4ea4b88bd724" }, { "cell_type": "markdown", - "id": "6228c3743dd0a5ec", "metadata": {}, "source": [ "### 3.2 Example B — Creativity (short stories)\n", "\n", "Three sample five-sentence stories (matching the creativity prompt format) are parsed and scored. The creativity metric combines **novelty** (distinctiveness from the corpus) and **surprise** (sentence-to-sentence semantic shift)." - ] + ], + "id": "6228c3743dd0a5ec" }, { "cell_type": "code", - "execution_count": null, - "id": "d3267c9d4f18eb83", "metadata": {}, - "outputs": [], "source": [ "from miniBen import parse_creativity_output, score_creativity\n", "\n", @@ -468,14 +457,14 @@ "for i, p in enumerate(parsed_stories, 1):\n", " print(f\"Story {i}: {p['sentence_count']} sentences\")\n", " print(f\" {p['story'][:80]}…\\n\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "d3267c9d4f18eb83" }, { "cell_type": "code", - "execution_count": null, - "id": "77ab00e0989817dc", "metadata": {}, - "outputs": [], "source": [ "# Requires sentence_transformers + spaCy (first run downloads gte-large)\n", "try:\n", @@ -505,24 +494,24 @@ "except Exception as e:\n", " print(f\"Creativity scoring skipped ({type(e).__name__}: {e}).\")\n", " print(\"Install dependencies and ensure network access for first-time model download.\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "77ab00e0989817dc" }, { "cell_type": "markdown", - "id": "a606fa4702b8c407", "metadata": {}, "source": [ "### 3.3 Example C — Simulated multi-call benchmark summary\n", "\n", "The cell below mimics what a user sees after five creativity calls: per-call scores and an aggregate. Values are illustrative; replace with `run_benchmark()` output for live data." - ] + ], + "id": "a606fa4702b8c407" }, { "cell_type": "code", - "execution_count": null, - "id": "68d148b9636e43cc", "metadata": {}, - "outputs": [], "source": [ "from miniBen import CREATIVITY_WORD_TRIPLETS\n", "\n", @@ -542,24 +531,24 @@ "plt.savefig(\"docs/figures/benchmark_summary_example.png\", dpi=150, bbox_inches=\"tight\")\n", "plt.show()\n", "print(\"Saved: docs/figures/benchmark_summary_example.png\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "68d148b9636e43cc" }, { "cell_type": "markdown", - "id": "8e0d4ad1ce845ce5", "metadata": {}, "source": [ "### 3.4 Optional — live benchmark run\n", "\n", "Set `RUN_LIVE = True` and ensure `OPENROUTER_API_KEY` is configured." - ] + ], + "id": "8e0d4ad1ce845ce5" }, { "cell_type": "code", - "execution_count": null, - "id": "afeb243a", "metadata": {}, - "outputs": [], "source": [ "RUN_LIVE = False\n", "MODEL = \"openrouter/free\"\n", @@ -577,11 +566,13 @@ " print(f\" {k}: {results[k]}\")\n", "else:\n", " print(\"Skipped. Set RUN_LIVE = True and configure OPENROUTER_API_KEY for a live run.\")" - ] + ], + "execution_count": null, + "outputs": [], + "id": "afeb243a" }, { "cell_type": "markdown", - "id": "7fa3478ebd41d39e", "metadata": {}, "source": [ "---\n", @@ -600,15 +591,16 @@ "- miniBen repository: https://github.com/Programming-The-Next-Step-2026/mini_benchmarks \n", "- Creativity benchmark: Ismayilzada et al. (2024) [arXiv:2411.02316](https://arxiv.org/abs/2411.02316) \n", "- OpenRouter models: https://openrouter.ai/models" - ] + ], + "id": "7fa3478ebd41d39e" }, { "cell_type": "code", - "execution_count": null, - "id": "84820a278dfaeaa1", "metadata": {}, + "source": [], + "execution_count": null, "outputs": [], - "source": [] + "id": "84820a278dfaeaa1" } ], "metadata": { @@ -624,4 +616,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/docs/workflow.md b/docs/workflow.md index e63d243..e7b25a7 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -1,44 +1,82 @@ # miniBen workflow -This page renders as a **flowchart on GitHub** (Mermaid is supported in `.md` files, not in `.ipynb` cells). +Accurate flow based on `example.py`, `runner.py`, and `model.py`. +(Mermaid renders on GitHub in `.md` files; the same content is in `vignette.ipynb` as text.) + +## Path A — full benchmark run + +Used by `python -m miniBen.example` (without `--list`) or `run_benchmark(...)`. ```mermaid flowchart TD - subgraph user["User"] - CLI["CLI: python -m miniBen.example"] - PY["Python: run_benchmark() / AIModel.ask()"] + START(["User starts benchmark run"]) + + subgraph cli_only["CLI only (example.py)"] + KEY["check_openrouter_api_key_exist()"] + PROMPTKEY["put_openrouter_api_key_into_env() if missing"] + end + + subgraph import_setup["On import (automatic)"] + DOTENV["auth.py: load_dotenv() reads .env"] end - subgraph miniben["miniBen package"] - AUTH["auth.py — API key"] - RUN["runner.py — run_benchmark()"] - PROMPT["prompts.py — 5 variants"] - MODEL["model.py — AIModel.ask()"] - PARSE["parsers.py"] - SCORE["scorers.py"] + RUN["runner.py: run_benchmark()"] + REG["Lookup BENCHMARKS entry\n(prompts, variants, parser, scorer)"] + PROMPTS["prompts.py: build all 5 prompts + variants\n(once, before the loop)"] + INIT["model.py: AIModel(model_name)\n(created once)"] + + subgraph loop["Loop i = 1 … 5"] + ASK["AIModel.ask(prompt_i, reasoning)"] + OR["OpenRouter API"] + PARSE["parsers.py: parser(content)"] + SC1["scorers.py: scorer([parsed_i])\nper-call score + print"] end - OR["OpenRouter API"] - OUT["Console report + results dict"] - - CLI --> AUTH - PY --> AUTH - AUTH --> RUN - RUN --> PROMPT - PROMPT --> MODEL - MODEL <-->|"HTTP request / response"| OR - MODEL --> PARSE - PARSE --> SCORE - SCORE --> OUT - RUN --> OUT + SCALL["scorers.py: scorer(all parsed)\naggregate score"] + SUMMARY["Print aggregate summary + stats"] + OUT(["Return results dict\n+ console report"]) + + START --> KEY + KEY -->|missing| PROMPTKEY + KEY --> RUN + PROMPTKEY --> RUN + DOTENV -.-> ASK + RUN --> REG --> PROMPTS --> INIT --> ASK + ASK <-->|HTTP| OR + ASK --> PARSE --> SC1 + SC1 -->|next i| ASK + SC1 -->|after 5 calls| SCALL + SCALL --> SUMMARY --> OUT ``` -| Step | Module | Main function | -|------|--------|----------------| -| 1 | `auth.py` | Load or prompt for `OPENROUTER_API_KEY` | -| 2 | `runner.py` | `run_benchmark()` — orchestrates the run | -| 3 | `prompts.py` | Build five benchmark-specific prompts | -| 4 | `model.py` | `AIModel.ask()` — one API call per variant | -| 5 | `parsers.py` | `parse_chess_output()` / `parse_creativity_output()` | -| 6 | `scorers.py` | `score_chess()` / `score_creativity()` | -| 7 | — | Printed report and returned `dict` | +## Path B — single API call + +Used when calling `AIModel(...).ask(...)` directly. Does **not** go through `run_benchmark`, prompts, parsers, or scorers unless you wire them up yourself. + +```mermaid +flowchart LR + U["User"] --> M["AIModel.ask(prompt)"] + M <-->|HTTP| OR["OpenRouter API"] + M --> R["return content, reasoning"] +``` + +## Step reference (Path A) + +| Phase | Where | What actually happens | +|-------|--------|------------------------| +| Import | `auth.py` | `load_dotenv()` loads `OPENROUTER_API_KEY` from `.env` | +| CLI only | `example.py` | Prompt for API key if not set (`run_benchmark` does **not** do this) | +| Setup | `runner.py` | Validate benchmark name; fetch prompts, variants, parser, scorer from `BENCHMARKS` | +| Setup | `prompts.py` | `cog_flex_prompts()` or `creativity_prompts()` — all **5** prompts built **before** the loop | +| Setup | `model.py` | One `AIModel` instance for the whole run | +| Loop ×5 | `model.py` | `ask(prompt_i)` → OpenRouter → `(content, reasoning)` | +| Loop ×5 | `parsers.py` | `parse_chess_output` or `parse_creativity_output` on raw text | +| Loop ×5 | `scorers.py` | `scorer([parsed_i])` — **per-call** score printed | +| After loop | `scorers.py` | `scorer(parsed_list)` — **aggregate** score (e.g. corpus novelty) | +| After loop | `runner.py` | Print summary, success/fail counts; return `dict` | + +## Notes + +- `--list` only prints registered benchmarks and **exits** — no API key or `run_benchmark` needed. +- Failed API calls or parse errors are logged; the loop continues with remaining variants. +- Creativity **novelty** uses all stories together and appears in the **aggregate** score, not per-call scores. diff --git a/pyproject.toml b/pyproject.toml index 5baa6b2..7649678 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,7 +26,6 @@ dependencies = [ "python-dotenv", "openai", "sentence_transformers", - "en_core_web_sm" # more to be added ]