kernel_zoo is a git-repository-as-database leaderboard and benchmark
platform for LLM GPU kernels: every operator implementation is a
kernel_package directory in a plain kernel_data git repository,
bench_result.yaml is the scoreboard, and git log is the full attempt
history. This repository is the platform that serves kernels, runs
benchmarks, and builds the WebUI — it is not the kernel repo itself.
There are two ways to contribute:
- Add a kernel — contribute a
kernel_package(an operator implementation with its own benchmark) that the platform evaluates. - Improve the platform — server, runner, WebUI, schemas, CI, or docs.
If your goal is only to write a kernel_package, jump straight to
Contributing a kernel.
├── kernel_zoo/ # the Python package (the platform)
│ ├── server/ # FastAPI app + HTTP route modules
│ │ ├── app.py # app factory, lifespan, static /webui/ hosting
│ │ ├── deps.py # FastAPI dependencies (auth, kernel_id checks)
│ │ ├── routes_kernels.py # kernel listing / detail / bench-history / claim
│ │ ├── routes_submissions.py # submit, long-poll status, report result
│ │ ├── routes_edits.py # edit rebase
│ │ ├── routes_runner.py # runner register / claim / heartbeat
│ │ ├── routes_health.py # /api/health + queue depth
│ │ ├── routes_webui.py # /webui/ SPA hosting
│ │ └── runner_registry.py # in-memory runner register/claim state
│ ├── runner/ # benchmark runner CLI + container backends
│ │ ├── runner.py # claim → build env → benchmark → report loop
│ │ ├── container_backend.py # pluggable Docker / stub backends
│ │ ├── client.py # HTTP client used by the runner
│ │ └── config.py # runner env config
│ ├── cli/ # console entry points (server, validate, indexer)
│ ├── config.py # env/YAML server config
│ ├── indexer.py # debounced/periodic index.json builder
│ ├── queue_manager.py # pending/running/done queue + sweeps + LRU
│ ├── repo_writer.py # two-phase git commit accept path (atomic rollback)
│ ├── git_repo.py # thin git plumbing wrapper
│ ├── system_status.py # health + queue-depth stats
│ ├── validation.py # schema validation for packages/tarballs
│ ├── _normalize.py # shared status normalization helpers
│ └── errors.py # error types
├── api/openapi.yaml # normative HTTP API contract
├── schemas/ # JSON Schemas for every artifact
├── kernel_zoo/server/static/webui/ # WebUI SPA (vanilla ES modules, no build step)
├── tests/ # pytest suite (contract / server / runner / unit)
├── tests/webui/ # WebUI JS tests (node:test)
├── tools/ # coverage gate + schema-doc generator
├── docs/ # user + contributor docs (bilingual)
├── README.md / README.zh-CN.md # bilingual project overview
└── pyproject.toml # packaging, lint, type-check, test config
A kernel_package is an operator implementation with four contract files under
.kernel_package/ — desc.yaml, bench_result.yaml, build_test_env.py,
benchmark.py — plus a source-authored src/ area. Kernel repositories and
submission tarballs must be generated-evidence-free, not merely binary-free. Do
not include model weights, generated tensors, golden data, compiled objects,
runtime traces, profiler exports, census or environment dumps, logs, captured
stdout/stderr, generated reports, or summaries derived from them in any format.
Package authors must create temporary artifacts deterministically under the runner
workdir in build_test_env.py; they are consumed for that run and discarded. See
docs/kernel_package_format.md. It lives in the
kernel_data repo (default <platform>/../kernel_data), not in this
repository. The server validates it against the JSON Schemas, a runner executes
it in a container, and benchmark.py decides whether the result is accepted.
Read the authoring guides before writing a package:
docs/kernel_package_guide_en.md(English) — rewrite an existing GPU kernel project (code + tests) into akernel_package.docs/kernel_package_guide.md(简体中文) — create a package from a blank slate.docs/kernel_package_format.md— on-disk format and every schema (normative source:schemas/).docs/api_reference.md— the HTTP API for submission, long-poll status, and result reporting.docs/examples/— a complete, schema-validating example package you can copy as a skeleton.
Validate a package directory before submitting it:
kernel_zoo-validate-package <kernel_package directory> # exit 0 = validThen package the directory as <kernel_id>.tar.gz and submit it with
POST /api/submissions (see the guides and docs/api_reference.md for the
exact contract).
- Python 3.11 or 3.12 (both are exercised in CI).
python -m pip install -e '.[dev]'— thedevextra brings pytest, pytest-cov, ruff, mypy, the OpenAPI validator, pytest-asyncio, and httpx.- Node 24 — required for the WebUI JS tests.
- Create a branch off
main(e.g.feat/<short-description>). CI runs onmainandfeat/**, and on every pull request. - Keep each commit small and focused: one logical change per commit, with its tests in the same commit.
- Add or update tests with every change, and keep the quality gates in §3.4 green before pushing.
- Ruff — line length 100, double quotes; run
ruff formaton your edits. - mypy
--strict— annotate everything, including tests. - Schema-driven contracts — new artifacts get a JSON Schema in
schemas/; new or changed endpoints get an entry inapi/openapi.yaml. The platform validates every artifact (desc, bench_result, benchmark output, submission envelope) against these schemas. - No external database — the kernel git repository is the database.
- Test hygiene — unit/integration tests belong in
tests/; end-to-end tests (real server + runner subprocesses) are tagged@pytest.mark.e2e.
Run these before pushing; CI enforces the same set on Python 3.11 and 3.12:
# Lint and type-check
ruff check kernel_zoo tests tools
ruff format --check kernel_zoo tests tools
mypy kernel_zoo tests tools
# Docs in sync with schemas/openapi (L5 gate)
python tools/gen_schema_docs.py --check
# Unit + integration with the layered coverage gate
pytest -m "not e2e" --cov=kernel_zoo --cov-report=json:coverage.json
python tools/check_coverage.py --cov-json coverage.json # core ≥90%, total ≥80%
# WebUI JS tests
node --test 'tests/webui/**/*.test.mjs'End-to-end tests run in a separate CI job (they spawn real subprocesses):
pytest -m e2e -vtests/contract/— JSON Schemas validate, the OpenAPI spec is coherent, and the example packages validate against the schemas.tests/server/— route handlers, app smoke tests, and the OpenAPI contract.tests/runner/andtests/test_runner_*.py— runner loop, container backends, runner client and CLI.tests/test_*.py— unit tests per platform module.tests/webui/*.test.mjs— WebUI logic undernode:test.
- The README is bilingual:
README.md(English) andREADME.zh-CN.md(简体中文) mirror each other. Keep both in sync, and keep the top language toggle linking correctly. docs/api_reference.mdanddocs/kernel_package_format.mdare partly auto-generated fromapi/openapi.yamlandschemas/*.json. Do not edit the autogen blocks by hand — edit the source and runpython tools/gen_schema_docs.pyto regenerate (the L5 gate fails if they drift).- Keep the "Related documentation" cross-link blocks in
docs/up to date when you add or rename a page.
Use Conventional Commits, scoped by subsystem where useful:
feat(scope): <summary> # new capability, e.g. feat(webui): detail page
fix(scope): <summary> # bug fix, e.g. fix(repo_writer): atomic rollback
test(scope): <summary> # new/updated tests
docs: <summary> # documentation only
ci: <summary> # CI / workflow changes
refactor(scope): <summary> # behavior-preserving change
style(scope): <summary> # formatting, no behavior change
chore: <summary> # maintenance
Agent-authored commits in this repo's history also carry a Co-Authored-By
trailer; please preserve it when continuing an agent-authored branch.
- Open an issue. Include: reproduction steps, expected vs. actual behavior, and
for submission problems the
kernel_idplus any runner/server logs. - For HTTP API questions, consult
docs/api_reference.mdand the normativeapi/openapi.yamlfirst.
MIT — see [project].license in pyproject.toml.