Skip to content
google-deepmindPublic

About

Evaluation, serving performance, and release-readiness benchmark harness for Gemma and open-weights LLMs.

Resources

Code of conduct

Contributing

Stars

14 stars

Watchers

1 watching

Forks

Repository files navigation

gBench logo: a jaguar in sunglasses relaxing with a coffee in front of a speedometer

gBench: performance and capability benchmarks for open models

gBench is an open-source benchmarking and evaluation suite for open large language models (LLMs) and vision-language models (VLMs). It measures serving performance on the hardware it runs on, and model capability in three tiers: academic eval suites, the Golden Set smoke tests and the agentic quality suite.

It works with any OpenAI-compatible /v1 server through --remote-endpoint (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls), or starts and stops a local vLLM server itself.


Table of contents


Key features

  • OpenAI API compatibility: measure any OpenAI-compatible /v1 endpoint with --remote-endpoint <URL>/v1 (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls; see Remote Endpoint Mode).
  • Three capability tiers:
    1. Academic and domain evals (--evals): 86 canonical Python suites in --evals all (plus ui_control_osworld and swe_lancer, which run only when named) across 6 core capability pillars (General Knowledge & Scientific Reasoning, Mathematics & Proofs, Coding & Algorithmic Design, Long Context & Retrieval, Tool Use & Agentic Workflows, Multimodal Vision & Grounding) with the model's reasoning mode on request (--eval-thinking), images sent as their original bytes where the dataset stores them (no re-encoding), plugin discovery (--eval-plugins-dir) and custom JSONL datasets (--eval-custom-jsonl).
    2. Golden Set smoke tests (--golden-only): 16 deterministic capability checks (code_canonical, function_call_single, structured_json, etc.) with two-sided presence assertions and python_exec unit-test matching, including a labelled chat template subset that separates a broken chat template from a weak model.
    3. Agentic quality suite (--quality-only): the GemmaClaw / OpenClaw QA suite (the QA scenarios of the pinned GemmaClaw commit, 60 at d8bc6989; scenarios that need another model provider are listed apart and not scored) testing multi-turn tool calling, session memory and plugin routing, run in gBench's local gbench-quality Docker image (built once per GemmaClaw commit).
  • Model discovery and GPU sizing: model properties read from the HuggingFace config.json (parameter count, MoE topology, vision/audio tokens), a recommended GPU tier per model size (1 GPU up to 20B, 2 GPUs up to 80B, 8 GPUs above) that is checked against the required --num-gpus (a warning when off the tier, a refusal when the model will not fit), and a server startup timeout that scales with model size (max(600, min(3600, (300 + params_b * 10) * 3 * (1.3 if MoE)))).
  • Repeatability checks: several iterations per configuration with mean, median, standard deviation, P50, P95, P99 and the coefficient of variation, checked against --max-cv-percent (default 5.0).
  • Open-loop stress sweep (--stress-test): one merged Poisson schedule per rate point at increasing rates (x2 per step) until a point fails; each of --stress-reps repeats gives a bracket [pass, fail], and the median repeat's limit is then certified with probes sized for the P99 SLO (a cohort of >= 1099 requests, about 1400 sent, the same rule on every device) to +-9% when that resolution is reached (otherwise a certified lower bound with its resolution, a lower bound censored at --stress-max-qps, or no limit when certification refutes the sweep), or reported UNCERTIFIED with the time certifying would need (--stress-certify-budget). A point passes when, on the requests planned after the point's ramp-up: the queue is not growing, >= 90% finish within the end-to-end time the SLOs imply, P99 TTFT and P99 ITL meet their SLOs, failures stay under 2% and the client kept to its schedule. The queue-growth test reads TTFT; overload that shows only as longer end-to-end times (for example KV-cache preemption) is left to the completion-fraction and ITL gates. SLOs are per campaign (5000 ms TTFT and 200 ms ITL without --campaign), overridden by --stress-threshold and --itl-slo-ms.

Documentation

Using gBench

What gBench measures

Background


Installation

python3 -m venv venv && source venv/bin/activate

# Install from PyPI (Python 3.11+):
pip install gbench               # base package: laptops and remote endpoints such as Ollama

# optional, per use:
pip install transformers         # --remote-endpoint: prompt sizing (--tokenizer), Golden Set chat-template
                                 # cases and the mmlu, wmdp, ruler and longbench_v2_code suites
pip install vllm                 # local GPU serving, and remote real-dataset workloads (--campaign chat-like,
                                 # --dataset custom/hf); gBench never installs vllm/torch itself
pip install "gbench[evals]"      # the eval suites' client, scorer and harness dependencies

# Or install from source:
git clone https://github.com/google-deepmind/gbench.git
cd gbench && pip install -e ".[evals]"

The installation guide covers the requirements, conda, the optional installs per suite and the test environment.


Quick start

# 1. Quick local performance smoke test (auto-starts local vLLM server)
gbench --models google/gemma-4-E4B-it --num-gpus 1 --preset quick

# 2. Run a specific workload performance campaign (e.g. agentic long-context)
gbench --models google/gemma-4-12B-it --num-gpus 1 --campaign agentic

# 3. Run the Golden Set capability smoke tests on a local vLLM server
#    gbench launches; its tool cases need the model's tool-call parser (Gemma 4 example)
gbench --golden-only --models google/gemma-4-12B-it --num-gpus 1 --tool-call-parser gemma4 --reasoning-parser gemma4

# 4. Run GemmaClaw agentic quality suite over remote endpoint
gbench --quality-only --models google/gemma-4-12B-it --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-12B-it

# 5. Run academic evaluations with Chain-of-Thought reasoning against remote server
gbench --evals-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --evals gpqa mmlu --eval-thinking --max-output-tokens 65536

# 6. Stage model artifacts from HuggingFace to Google Cloud Storage
gbench --models google/gemma-4-12B-it --stage-to-gcs gs://my-gbench-bucket/models/

# 7. Dry run: the plan, each server's vllm serve command, port, GPUs and base URL (validated)
gbench --models google/gemma-4-12B-it --num-gpus 1 --preset default --dry-run

Three capability evaluation tiers

+-------------------------------------------------------------------------------+
|                         gbench capability evaluation                          |
+-------------------------------------------------------------------------------+
|  Tier 1: academic evals         |  Tier 2: Golden Set    |  Tier 3: agentic QA|
|  (--evals)                      |  (--golden-only)       |  (--quality-only)  |
|  - bfcl, gpqa, gsm8k, mmlu,     |  - 16 cases            |  - GemmaClaw suite |
|    mmmu_pro, mrcr, screenspot   |  - Two-sided presence  |  - QA scenarios    |
|  - Reasoning mode (--thinking)  |  - Chat template pairs |  - Multi-turn QA   |
|  - Vision soft-token budgets    |  - N/A-aware scoring   |  - Tools, memory   |
+---------------------------------+------------------------+--------------------+
                                         |
                                         v
+-------------------------------------------------------------------------------+
|                      OpenAI-compatible REST API (/v1)                         |
+-------------------------------------------------------------------------------+
|  Mode A: local vLLM server  <---- OR ---->  Mode B: remote endpoint           |
|  (Auto start / stop / VRAM poll)            (--remote-endpoint <URL>/v1)      |
+-------------------------------------------------------------------------------+

Tier 1: academic and custom evals (--evals)

The full catalog of the 86 suites in 6 pillars, with each suite's protocol and options, custom plugins and JSONL suites: docs/eval-suites.md. Each suite has a setup and scoring guide in docs/evals/.

Tier 2: Golden Set capability smoke test (--golden-only)

16 deterministic capability checks (code, math, tool calling, structured output, chat templates, factual knowledge, translation, safety, vision and audio) with two-sided presence assertions and python_exec unit-test matching, including a chat-template subset that separates a broken chat template from a weak model. Details: docs/golden-set.md.

Tier 3: agentic quality suite (--quality-only)

Evaluates multi-turn agent behavior with the GemmaClaw / OpenClaw QA suite: the QA scenarios of the pinned GemmaClaw commit (60 at d8bc6989); scenarios that need a model provider other than the model under test and the Gemini API are listed apart and not scored.

  • Image build: clones GemmaClaw (~/.cache/gbench/gemmaclaw, no working tree) and builds a local gbench-quality image from the commit's git archive (the TypeScript workspace installed and built inside it), then runs the suite in a container of that image against /v1.
  • Build cache: the image is tagged with the commit and a digest of its Dockerfile, helpers and gBench's edits, so it is built once per commit (several minutes on the first run).
  • Configuration: generates a throwaway openclaw.json that disables the third-party plugins the suite does not use, to keep memory use down.
  • Gemini-only helper models: the QA suite's alternate model (google/gemini-3.1-pro-preview, via --alt-model, in place of openai/gpt-5.4), image-generation model (google/gemini-3.1-flash-image-preview, via a recorded edit applied in the image's build context, in place of openai/gpt-image-1) and memory embeddings (google/gemini-embedding-001, in place of openai/text-embedding-3-small) run on the Gemini API, and every Gemini call is recorded in gemini_calls; GEMINI_API_KEYS or GEMINI_API_KEY is required. No key reaches the model under test: the suite runs in gBench's own Docker container (no network, cloud metadata servers unreachable, a read-only checkout, a user without capabilities, no host path writable), and its only way out is a forwarder in the container that hands its model-endpoint and Gemini calls to a relay in gBench over a unix socket; the relay alone holds the keys and forwards only the Gemini API calls the suite makes. Harness errors (relay, endpoint, Gemini or container failures) are not scored; a model-endpoint 4xx other than 400 and 422 on the suite's own requests (for example the 404 of an endpoint that serves the model under another name) is one, and a run with no scenario left to score withholds its pass rate (null, printed withheld), never 0. Details: docs/quality.md.
# Run specific GemmaClaw QA scenarios (by scenario id) from a local checkout
gbench --quality-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --scenarios memory-recall mcp-plugin-tools-call --gemmaclaw-path /path/to/gemmaclaw

Developer guide and contribution workflow

  1. Install development environment (in a test environment, see tests/README.md): pip install -e ".[dev,evals,omnidocbench]" transformers and python -m nltk.downloader punkt punkt_tab; the system tools the tests exercise are bubblewrap, a Java runtime and Docker (the Unit Tests workflow installs exactly this)
  2. Run code style check: ruff check gbench/
  3. Run type checker: mypy gbench/
  4. Run unit test suite: pytest tests/ (the Gemma 4 chat-template artifact check is not a unit test; it lives in tools/template_checks/, see its README)
  5. The pull-request gate is the unit-test suite (pytest tests/, see CONTRIBUTING.md). A change to the Golden Set keeps its two-sided presence assertions; the Golden Set itself measures a model and never gates a gBench merge.

Authors


License

Copyright 2026 Google LLC. Licensed under the Apache License, Version 2.0.

About

Evaluation, serving performance, and release-readiness benchmark harness for Gemma and open-weights LLMs.

Resources

Code of conduct

Contributing

Stars

14 stars

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages