diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..deb83cb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,133 @@ +name: ๐Ÿž Bug report +description: Something is broken, crashes, or produces wrong/garbled output. +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Thanks for reporting a bug! Please fill in the environment fields โ€” most issues we see + (garbled output, install errors, collapse-to-`!` loops) are pinned down by the platform, + chip, mlx version, and whether the prerouter/LoRA adapters are loaded. + + **Before filing:** search [existing issues](https://github.com/Edge0-AI/Edge0/issues?q=is%3Aissue) + and check the [known mlx version pin](https://github.com/Edge0-AI/Edge0/blob/main/python/pyproject.toml) + (A18 / A18 Pro garbled output โ†’ `mlx==0.30.6`, see #8). + + - type: dropdown + id: platform + attributes: + label: Platform / runtime + description: Which runtime are you using? + options: + - Python (MLX, Apple Silicon macOS) + - macOS app / CLI (Rust) + - iOS app + - Android app / engine + - Windows app / engine + validations: + required: true + + - type: dropdown + id: tier + attributes: + label: Model tier + options: + - edge0-35b + - edge0-8b + - both + - other / custom checkpoint + validations: + required: true + + - type: input + id: hardware + attributes: + label: Hardware / chip + description: e.g. "Mac mini M4 Pro", "MacBook Air M1", "iPhone 16 Pro (A18 Pro)", "RTX 4090", "Pixel 10 Pro". + placeholder: Apple M1 Pro + validations: + required: true + + - type: input + id: ram + attributes: + label: RAM / unified memory + description: Total memory of the device. Many "slow" / OOM reports come from low-memory machines. + placeholder: 16 GB + validations: + required: true + + - type: input + id: os + attributes: + label: OS version + placeholder: macOS 15.1 / Windows 11 23H2 / Android 15 / iOS 18.1 + validations: + required: true + + - type: input + id: mlx_version + attributes: + label: mlx / mlx-metal / mlx-lm version (Python runtime only) + description: "Output of: pip show mlx mlx-metal mlx-lm | grep -i version โ€” pin is mlx==0.30.6, mlx-lm==0.31.0." + placeholder: mlx 0.30.6, mlx-metal 0.30.6, mlx-lm 0.31.0 + validations: + required: false + + - type: dropdown + id: install_method + attributes: + label: How did you install / get the model? + options: + - pip install -e '.[dev,fetch]' from source + - scripts/fetch_models.py + - huggingface-cli download + - ModelScope + - Prebuilt app (macOS/iOS/Android/Windows) + - Other (describe below) + validations: + required: false + + - type: dropdown + id: adapters + attributes: + label: Adapters in use + description: Does the bug still happen with the prerouter disabled? (`--no-prerouter`) This isolates a whole class of bugs (see #16). + options: + - Default (prerouter + LoRA enabled) + - "--no-prerouter" + - "--no-lora" + - Both disabled (bare base model) + validations: + required: false + + - type: textarea + id: what_happened + attributes: + label: What happened? + description: What went wrong, and what did you expect instead? + placeholder: After the 3rd request in the same process, every output collapses to "!!!!". + validations: + required: true + + - type: textarea + id: repro + attributes: + label: Steps to reproduce + description: Exact commands / code. Include the prompt if output is wrong. + placeholder: | + 1. edge0 serve edge0-35b + 2. curl -X POST http://127.0.0.1:8000/v1/chat/completions -d '...' + 3. Repeat 3 times + render: shell + validations: + required: true + + - type: textarea + id: logs + attributes: + label: Logs / stack trace / output + description: Paste any error output or the garbled text. This will be rendered as code, no need for backticks. + render: shell + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..1f2c7a0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,11 @@ +blank_issues_enabled: false +contact_links: + - name: ๐Ÿ’ฌ Questions & Discussions + url: https://github.com/Edge0-AI/Edge0/discussions + about: Usage questions, "how do I run it on X", show-and-tell, and general Q&A. Please ask here instead of opening an issue. + - name: ๐Ÿ“– Documentation + url: https://github.com/Edge0-AI/Edge0#readme + about: Setup, quickstart, benchmarks, and the per-platform READMEs (python/ ยท macos/ ยท ios/ ยท android/ ยท windows/). + - name: ๐Ÿ“ง Security / private contact + url: mailto:samuel@edge0.ai + about: Report security issues or anything you cannot share publicly. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..a037dcc --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,48 @@ +name: ๐Ÿ’ก Feature request +description: Suggest a capability, backend, platform, or API improvement. +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + Before filing, please check the [roadmap in the README](https://github.com/Edge0-AI/Edge0#roadmap) + and existing issues โ€” several popular asks are already tracked (CUDA backend #107/#20/#19, + non-macOS support #18, tool/function calling #115/#119). + + - type: dropdown + id: area + attributes: + label: Area + options: + - New backend (CUDA / other accelerator) + - New platform support + - Server / OpenAI-compatible API + - CLI / Python API + - Quantization / adapters (LoRA / prerouter) + - Documentation + - Other + validations: + required: true + + - type: textarea + id: problem + attributes: + label: What problem are you trying to solve? + description: Describe the use case and who it helps. "I have an NVIDIA GPU and want to run edge0-8b for ..." + validations: + required: true + + - type: textarea + id: proposal + attributes: + label: Proposed solution + description: What would you like to see? If you're willing to contribute it, say so โ€” we love PRs (see docs/adding-a-model.md for model work). + validations: + required: false + + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/model_request.yml b/.github/ISSUE_TEMPLATE/model_request.yml new file mode 100644 index 0000000..a17a204 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/model_request.yml @@ -0,0 +1,56 @@ +name: ๐Ÿงฉ New model support request +description: Request (or offer to add) support for another MoE model / checkpoint. +labels: ["model-support"] +body: + - type: markdown + attributes: + value: | + edge0 is built to take new models through a documented path โ€” see + [`docs/adding-a-model.md`](https://github.com/Edge0-AI/Edge0/blob/main/docs/adding-a-model.md). + The best model requests come with the config below. **Contributions welcome** โ€” if you + want to add it yourself, say so and we'll help review. + + - type: input + id: model_name + attributes: + label: Model name + placeholder: Qwen3.6-Next-80B-A3B + validations: + required: true + + - type: input + id: hf_link + attributes: + label: Hugging Face / ModelScope link + placeholder: https://huggingface.co/... + validations: + required: true + + - type: input + id: model_type + attributes: + label: config.json `model_type` + description: The architecture id from the checkpoint's config.json โ€” edge0 resolves tiers via this (see TYPE_ALIASES). + placeholder: qwen3_5_moe / bailing_hybrid / ... + validations: + required: false + + - type: textarea + id: arch + attributes: + label: Architecture summary + description: Sparse MoE? Number of layers / experts / top-k routing width? Attention type (MHA / MLA / linear)? Anything unusual. + placeholder: Sparse MoE, 48 layers, 128 experts, top-8 routing, MLA attention. + validations: + required: true + + - type: dropdown + id: contribute + attributes: + label: Are you willing to contribute this? + options: + - "Yes โ€” I'd like to open a PR (happy to get guidance)" + - "Maybe, with help" + - "No, this is a request" + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/performance_report.yml b/.github/ISSUE_TEMPLATE/performance_report.yml new file mode 100644 index 0000000..8d5e30c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/performance_report.yml @@ -0,0 +1,89 @@ +name: ๐Ÿข Performance report +description: Decode/prefill speed, time-to-first-token, or memory usage differs from the documented benchmarks. +labels: ["performance"] +body: + - type: markdown + attributes: + value: | + Use this for "slower than the README numbers" or "high memory / OOM" reports + (e.g. #17, #110). The documented benchmarks were measured on a **Mac mini M4 Pro, 24 GB**. + Low-memory machines (8โ€“16 GB) can be dramatically slower because expert pages are + continuously evicted and re-read from SSD โ€” please include your total RAM. + + - type: dropdown + id: tier + attributes: + label: Model tier + options: + - edge0-35b + - edge0-8b + validations: + required: true + + - type: input + id: hardware + attributes: + label: Hardware / chip + placeholder: Apple M1 Pro + validations: + required: true + + - type: input + id: ram + attributes: + label: Total RAM / unified memory + placeholder: 16 GB + validations: + required: true + + - type: input + id: disk + attributes: + label: Storage type + description: Internal SSD vs external/USB matters a lot for streaming expert reads. + placeholder: Internal NVMe SSD + validations: + required: false + + - type: input + id: decode_observed + attributes: + label: Observed decode speed (tok/s) + placeholder: "0.6 tok/s (doc says ~15)" + validations: + required: false + + - type: input + id: ttft + attributes: + label: Observed time-to-first-token + placeholder: 11 s + validations: + required: false + + - type: input + id: memory + attributes: + label: Observed peak memory + description: Note whether this is MLX allocator peak or process RSS (they differ โ€” see #106). + placeholder: 4.2 GB (process RSS) + validations: + required: false + + - type: textarea + id: flags + attributes: + label: Flags / settings tried + description: e.g. --prefill-ondemand, full_layer_prefill, warm_willneed, context length, max_tokens. + placeholder: Tried warm_willneed and full_layer_prefill=False, no change. + validations: + required: false + + - type: textarea + id: command + attributes: + label: Exact command used to measure + description: Ideally the output of examples/bench.py so numbers are comparable. + render: shell + validations: + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..3de612b --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,28 @@ + + +## What & why + + + +Fixes # + +## Area + +- [ ] Python framework (`python/`) +- [ ] macOS (`macos/`) +- [ ] iOS (`ios/`) +- [ ] Android (`android/`) +- [ ] Windows (`windows/`) +- [ ] Docs / other + +## How was it tested? + + + +## Checklist + +- [ ] `pytest -m 'not slow'` passes locally (Python changes; CI runs this on macOS + MLX) +- [ ] Repo hygiene passes: `pytest tests/test_repo_hygiene.py` (no hardcoded paths, no secrets, **no MLX imports outside `backends/mlx/`**) +- [ ] Added/updated tests for new behavior +- [ ] Did **not** bump `mlx` past the pinned `0.30.6` (see the note in `python/pyproject.toml` / issue #8) unless this PR is specifically about that +- [ ] Updated relevant docs / READMEs (incl. translations if user-facing) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..b173305 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,125 @@ +# Contributing to edge0 + +Thanks for your interest in contributing! edge0 is an open-source streaming MoE +inference framework, and contributions of all kinds are welcome โ€” bug reports, +performance measurements, documentation fixes, new model support, and new +platform/backend work. + +This is a multi-platform monorepo. Please read the section for the part you're +touching. + +## Ways to contribute + +- **Report a bug / performance issue / request a feature or model** โ€” open an + issue using one of the [issue forms](.github/ISSUE_TEMPLATE/). The forms ask + for the platform, chip, memory, and (for the Python runtime) the `mlx` + version โ€” these fields resolve the large majority of reports, so please fill + them in. +- **Ask a question or share results** โ€” please use + [GitHub Discussions](https://github.com/Edge0-AI/Edge0/discussions) rather + than an issue. +- **Send a pull request** โ€” see the workflow below. + +## Repository layout + +| Path | What | Stack | +|---|---|---| +| [`python/`](python/) | Reference framework + CLI + OpenAI-compatible server | Python + MLX (Apple Silicon) | +| [`macos/`](macos/) | Desktop app / CLI / daemon | Rust | +| [`ios/`](ios/) | iPhone on-device app | Swift + MLX Swift | +| [`android/`](android/) | On-device app + native engine | Kotlin + native | +| [`windows/`](windows/) | Desktop app + native engine | C++ + Vulkan | +| [`docs/`](docs/) | Architecture & design docs | โ€” | + +Each platform directory has its own README with build and test instructions. +The sections below focus on the **Python framework**, which is the easiest place +to start. + +## Development setup (Python framework) + +Requires macOS on Apple Silicon (the MLX backend has no wheels on other +platforms yet โ€” a CUDA backend slot is reserved under +`python/src/edge0/backends/cuda/`). + +```bash +cd python +python3.12 -m venv .venv +.venv/bin/pip install -e '.[dev,fetch]' +``` + +> **Do not bump `mlx` past the pinned `0.30.6`.** `0.30.4` and older mis-gate +> the NAX matmul kernels on Apple A18 / A18 Pro (silently wrong numbers โ†’ +> garbled output, [#8](https://github.com/Edge0-AI/Edge0/issues/8)), and +> `mlx>=0.31.2` breaks the streaming pool threads. `mlx-lm` stays `0.31.0`. +> See the note in [`python/pyproject.toml`](python/pyproject.toml). Only touch +> these pins in a PR that is specifically about the upgrade. + +## Running the tests + +```bash +cd python +pytest -m 'not slow' # unit tests (no real weights) โ€” what CI runs +pytest tests/test_repo_hygiene.py # repo hygiene (pure Python, runs anywhere) + +# Real-weight tests (need a downloaded checkpoint; skipped if absent): +EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q +.venv/bin/python scripts/e2e_smoke.py \ + --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b +``` + +## What CI checks (and how to pass it) + +Every PR runs two jobs ([`.github/workflows/ci.yml`](.github/workflows/ci.yml)): + +1. **Unit tests (macOS + MLX)** โ€” `pytest -m 'not slow'`. +2. **Repo hygiene (Linux, pure Python)** โ€” `tests/test_repo_hygiene.py`, which + enforces the "no dependency on the author's machine" rules across the + *entire* repo: + - **No hardcoded local paths** โ€” nothing that points at a specific machine's + filesystem (absolute home directories, network mounts, cluster node names, + etc.). Use env vars, fixtures, or relative paths. + - **Backend boundary** โ€” `import mlx` / `from mlx ...` may appear **only** + under `python/src/edge0/backends/mlx/`. Framework code must go through the + `edge0.backends` facade (`core` / `nn` / `io` / `quant`). This keeps the + core backend-agnostic so other backends can plug in. + - **No secrets** โ€” no API keys or private keys in the tree. + - **Docs links resolve** โ€” e.g. the README `NOTICE` link must point to a real + file. + +You can run both locally before pushing (see above). + +## Adding a new model + +edge0 is designed to take new MoE checkpoints through a documented path. Start +with [`docs/adding-a-model.md`](docs/adding-a-model.md): implement the +three-part adapter contract (`Config` / `build_model` / `build_engine`), +register it, and map the checkpoint's `model_type` via `TYPE_ALIASES`. Please +include a short note on the architecture (layers / experts / routing width / +attention type) in your PR. + +## Pull request workflow + +1. **Fork** the repo and create a feature branch off `main` + (e.g. `fix/streaming-oob`, `docs/clarify-memory`, `feat/tool-calling`). +2. Make your change; **add or update tests** for new behavior. +3. Run `pytest -m 'not slow'` and `pytest tests/test_repo_hygiene.py` locally. +4. Open a PR against `Edge0-AI/Edge0:main` and fill in the + [PR template](.github/PULL_REQUEST_TEMPLATE.md). Link the issue it closes + (`Fixes #NN`). +5. Keep PRs focused โ€” one logical change per PR is easier to review and merge. + +### Commit / PR style + +This repo uses conventional-commit-style prefixes, e.g. +`fix(streaming): ...`, `feat(server): ...`, `docs(readme): ...`, +`perf(streaming): ...`, `chore: ...`. Please follow the same convention. + +## Reporting security issues + +Do not open a public issue for security problems. Email **samuel@edge0.ai** +instead. + +## License + +By contributing, you agree that your contributions will be licensed under the +project's [Apache-2.0](LICENSE) license. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md new file mode 100644 index 0000000..3c583fb --- /dev/null +++ b/docs/benchmarks/README.md @@ -0,0 +1,59 @@ +# Community Hardware Benchmarks + +This directory collects real-world performance numbers for edge0 across different +hardware, contributed by the community. If you run edge0 on your machine, your +numbers are welcome here โ€” **every reporter is credited by handle**. + +## How to contribute a result + +1. Run the benchmark on your device (macOS / Apple Silicon is supported today): + + ```bash + cd python + python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]' + export EDGE0_8B_MODEL=$PWD/models/edge0-8b # or EDGE0_35B_MODEL + BENCH_LONG=1 .venv/bin/python examples/bench.py edge0-8b + ``` + +2. Post your numbers in the pinned + [Hardware Benchmarks discussion](https://github.com/Edge0-AI/Edge0/discussions/142), + using the submission block there. + +3. Either a maintainer curates your reply into [`results.csv`](./results.csv), + or you open a PR adding your own row โ€” we'll help review it. + +> Other platforms (Windows / iOS / Android) don't have a one-command benchmark +> yet. Help build them! See the good-first-issues linked from the discussion: +> [#139](https://github.com/Edge0-AI/Edge0/issues/139) (Windows), +> [#140](https://github.com/Edge0-AI/Edge0/issues/140) (iOS), +> [#141](https://github.com/Edge0-AI/Edge0/issues/141) (Android). + +## Data + +Results live in [`results.csv`](./results.csv) (machine-readable; GitHub renders +it as a table). Columns: + +| column | meaning | +| --- | --- | +| `date` | submission date (YYYY-MM-DD) | +| `tier` | `edge0-8b` or `edge0-35b` | +| `runtime` | backend used, e.g. `python-mlx` | +| `device` | device / machine name | +| `chip` | SoC, e.g. `Apple M2` | +| `ram_gb` | unified / system memory in GiB | +| `os` | OS version | +| `storage` | `internal SSD` / `external USB-SSD` (matters for expert paging) | +| `mlx_version` | `pip show mlx` version | +| `decode_tok_s` | decode throughput (tokens/sec) | +| `prefill_tok_s` | prefill throughput, prefer `BENCH_LONG` long-context | +| `peak_active_gib` | MLX peak active memory (GiB) | +| `on_battery` | `yes` / `no` | +| `reporter` | GitHub handle, e.g. `@you` | +| `notes` | anything unusual (throttling, short vs long, background load) | + +## Why this matters + +edge0 streams experts from SSD on demand, so throughput depends heavily on RAM +and storage speed โ€” low-memory machines evict and re-read expert pages more +often. Collecting numbers across real hardware helps us (and users) understand +the practical performance envelope on each device. diff --git a/docs/benchmarks/results.csv b/docs/benchmarks/results.csv new file mode 100644 index 0000000..3b3c1de --- /dev/null +++ b/docs/benchmarks/results.csv @@ -0,0 +1,2 @@ +date,tier,runtime,device,chip,ram_gb,os,storage,mlx_version,decode_tok_s,prefill_tok_s,peak_active_gib,on_battery,reporter,notes +2026-10-15,edge0-8b,python-mlx,MacBook Pro,Apple M5,16,macOS 26.3.2,internal SSD,0.30.6,11.8,314,3.19,no,@shuk777,"decode 13.5 short / 11.8 long; prefill 231 cold / 314 warm (BENCH_LONG)"