Skip to content
BranchesTags

Repository files navigation

Bloombee Logo
Run large language models in a heterogeneous decentralized environment over internet.

The rapid rise of generative AI has boosted demand for large language model (LLM) inference services. While proprietary models are still favored, advancements in open-source LLMs have made them competitive. However, high costs and limited GPU resources hinder deployment. BloomBee is a decentralized offline serving system that leverages idle GPU resources to provide cost-effective access to LLMs.

Instead of requiring a single powerful machine, BloomBee splits a model's transformer blocks across multiple peers in a P2P network. If your GPU can only hold a small portion of a large model like LLaMA 3.1 (405B), you can join a network of servers each hosting different layers and collaboratively serve inference requests.

📄  Read the paper  |  🚀  Try now in Colab

News

  • 2026/04/22 : Released our paper on arXiv.
  • 2026/02/23 : Improve documentation, CI, and developer tooling (PR #41 by @dadaism).
  • 2026/02/19 : Support micro batching and lossless compression (PR #39 by @JiuChen0).
  • 2026/02/05 : Add batch support for speculative decoding and its pruning (PR #38 by @xiongxu1998).
  • 2026/01/13 : Spec dec (PR #37 by @xiongxu1998).

More News

  • 2025/11/29 : Update a new template to support weight cache and batch (PR #36 by @TomekWei).
  • 2025/11/21 : Remove O(prompt_len) prompt copies (PR #35 by @JiuChen0).
  • 2025/11/12 : Optimize shared memory usage, clean up legacy quantization, and remove unused modules (PR #34 by @JiuChen0).
  • 2025/11/01 : Add multi-batch inference support, fix hivemind dependency, and improve installation process (PR #27 by @JiuChen0).

Key Features

Running an LLM across decentralized GPUs is bottlenecked by inter-node bandwidth and per-node memory. BloomBee addresses both, with a focus on multi-dimensional communication optimization to reduce or hide communication overhead.

  • Tensor offloading — reduces per-node memory consumption so each peer can host more layers, shrinking the total number of network hops.
  • Speculative decoding over internet — reduces communication frequency by sending multiple draft tokens per round-trip.
  • Lossless activation-compression — shrinks the bytes transferred per activation, without accuracy loss.
  • Micro-batch pipelining — overlaps communication with computation to hide network latency.

Table of Contents

How It Works

BloomBee distributes a model's transformer layers across a peer-to-peer network:

┌─────────────────────────────────────────────────────────┐
│  CLIENT (you)                                           │
│  • Runs word embeddings and the LM head locally         │
│  • Routes through remote layers via DHT                 │
└──────────────────────┬──────────────────────────────────┘
                       │ P2P (libp2p)
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
   ┌──────────┐  ┌──────────┐  ┌──────────┐
   │ Worker A │  │ Worker B │  │ Worker C │
   │ Layers   │  │ Layers   │  │ Layers   │
   │  0 – 15  │  │ 16 – 31  │  │ 32 – 47  │
   └──────────┘  └──────────┘  └──────────┘
         Peers registered in DHT

A Distributed Hash Table (DHT) keeps track of which server hosts which layers. The client automatically discovers and routes through available peers. Servers are decentralized — anyone with a compatible GPU can join and contribute capacity.

Supported Models

Model Family Example HuggingFace IDs
LLaMA / LLaMA 2 / LLaMA 3 meta-llama/Llama-2-7b-hf, meta-llama/Meta-Llama-3-8B
BLOOM bigscience/bloom-7b1, bigscience/bloom
Falcon tiiuae/falcon-7b, tiiuae/falcon-40b
Mixtral mistralai/Mixtral-8x7B-v0.1
Qwen3 Qwen/Qwen3-0.6B, Qwen/Qwen3-4B, Qwen/Qwen3-14B
Gemma-4 google/gemma-4-31B-it

Any HuggingFace model with a matching architecture can be served. Use AutoDistributedModelForCausalLM to load a model automatically.

Note: Qwen3 (152k vocab) and Gemma-4 (262k vocab) have large vocabularies, so the client-side LM head matmul dominates decode latency on CPU. Use a GPU client for these families (model.to("cuda"), or --client_device cuda in benchmarks/benchmark_inference.py).

Prerequisites

  • Python 3.8 or later
  • PyTorch 1.12 or later (with CUDA for GPU support)
  • A GPU with at least ~4 GB VRAM to serve a portion of a model (workers)
  • A machine with internet access and, optionally, a public IP or port forwarding for external peer discovery

Note: The client machine does not need a GPU — only worker servers do.

Installation

From PyPI

pip install bloombee

From Source

git clone https://github.com/ai-decentralized/BloomBee.git
cd BloomBee
pip install .

Heterogeneous NVIDIA / Chameleon Hosts

BloomBee is intended to run across heterogeneous GPU fleets, so avoid relying on pip install torch to pick a CUDA wheel. If nvidia-smi sees the GPU but torch.cuda.is_available() is False, or before installing BloomBee on a fresh host, let BloomBee choose a PyTorch wheel from the local NVIDIA driver CUDA version and GPU compute capability:

conda activate bb
python scripts/install_compatible_torch.py
pip install -e .

For example, Tesla P100 / Pascal hosts are kept on a CUDA 12.1 PyTorch wheel that supports SM 6.0, while newer GPUs with newer drivers can use newer CUDA wheels. You can preview the decision with:

python scripts/install_compatible_torch.py --dry-run

Manual overrides are available for unusual clusters:

BLOOMBEE_TORCH_SPEC='torch==2.4.1+cu121' \
BLOOMBEE_TORCH_INDEX_URL='https://download.pytorch.org/whl/cu121' \
python scripts/install_compatible_torch.py

Quick Start

(Try in Colab)

1. Start a Bootstrap Node

A bootstrap node is a lightweight DHT peer that helps other nodes discover each other. Start one first:

python -m bloombee.cli.run_dht \
  --host_maddrs /ip4/0.0.0.0/tcp/31340 \
  --identity_path bootstrap.id

You will see a line like:

Mon 00 01:23:45.678 [INFO] Running a DHT instance. To connect other peers to this one, use:
  --initial_peers /ip4/YOUR_IP/tcp/31340/p2p/QmefxzDL1DaJ7TcrZjLuz7Xs9sUVKpufyg7f5276ZHFjbQ

Copy this address — you'll pass it as --initial_peers to all workers and clients.

If you want your swarm accessible from outside your local network, make sure you have a public IP address or have port forwarding configured correctly.

2. Start Worker Servers

Export the bootstrap address for convenience:

export BBSERVER=/ip4/YOUR_IP/tcp/31340/p2p/QmefxzDL1DaJ7TcrZjLuz7Xs9sUVKpufyg7f5276ZHFjbQ

Start workers, each hosting a slice of the model. For a 32-layer model, you might split it across two servers:

# Worker 1: hosts 16 transformer layers
python -m bloombee.cli.run_server meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --num_blocks 16 \
  --identity_path worker_1.id

# Worker 2: hosts the remaining 16 layers
python -m bloombee.cli.run_server meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --num_blocks 16 \
  --identity_path worker_2.id

Workers will automatically download their assigned model layers from HuggingFace on first run.

3. Run Inference

python benchmarks/benchmark_inference.py \
  --model meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --torch_dtype float32 \
  --seq_len 128

CLI Reference

bloombee.cli.run_dht — Bootstrap Node

Starts a lightweight DHT peer for peer discovery. Does not load any model.

Argument Default Description
--host_maddrs — Multiaddresses to listen on (e.g. /ip4/0.0.0.0/tcp/31340)
--identity_path — Path to store/load this node's persistent identity key
--announce_maddrs — Public multiaddresses to announce to other peers (useful behind NAT)

bloombee.cli.run_server — Worker Server

Loads and serves transformer blocks on a peer in the swarm.

Argument Default Description
model (required) HuggingFace model name or local path
--initial_peers — Multiaddresses of bootstrap nodes to connect to
--num_blocks auto Number of transformer layers to serve
--block_indices auto Specific layer index range to serve (e.g. 0:16)
--identity_path — Path to store/load this peer's persistent identity key
--quant_type none Quantization: int8 (LLM.int8) or nf4 (QLoRA 4-bit)
--torch_dtype auto Model precision: float32, float16, bfloat16
--throughput auto Reported throughput in tokens/sec; use eval to measure or dry_run to skip
--cache_dir — Directory to cache downloaded model weights
--max_batch_size 2048 Maximum number of tokens per forward batch

Environment Switches

README.environment-switches.md contains the full BLOOMBEE_* switch reference, including:

  • micro-batching / overlap / server-to-server push
  • KV cache and offload flags
  • lossless compression and profiling flags
  • debug groups and log-channel toggles
  • speculative decoding and EAGLE-2 flags
  • activation dumping and runtime helpers

If you add a new switch later, the quickest rescan command is:

rg -n -o "BLOOMBEE_[A-Z0-9_]+" README*.md src benchmarks tests | sort -u

Python API

BloomBee integrates with HuggingFace Transformers. Use the Auto classes to load a distributed model:

from transformers import AutoTokenizer
from bloombee import AutoDistributedModelForCausalLM

model = AutoDistributedModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    initial_peers=["/ip4/YOUR_IP/tcp/31340/p2p/Qm..."],
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

inputs = tokenizer("The quick brown fox", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

For efficient multi-turn generation, reuse an inference session to avoid reprocessing past tokens:

with model.transformer.h.inference_session(max_length=512) as sess:
    for _ in range(20):
        outputs = model.generate(max_new_tokens=1, session=sess)

Speculative Decoding (EAGLE-2)

EAGLE-2 speculative decoding is available for LLaMA-family targets through the speculative auto class and EAGLE drafter:

from transformers import AutoTokenizer
from bloombee import AutoDistributedSpeculativeModel
from bloombee.models.llama.eagle_drafter import EAGLEDrafter

model_name = "lmsys/vicuna-33b-v1.3"
model = AutoDistributedSpeculativeModel.from_pretrained(
    model_name,
    initial_peers=["/ip4/YOUR_IP/tcp/31340/p2p/Qm..."],
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(model_name)
drafter = EAGLEDrafter.for_target(model, device="cuda")

input_ids = tokenizer("The quick brown fox", return_tensors="pt")["input_ids"].to("cuda")
outputs = model.generate(
    input_ids,
    drafter=drafter,
    max_new_tokens=128,
    max_tree_depth=5,
)

By default, EAGLE-2 auto-selection is conservative and only picks known LLaMA-family drafter checkpoints. Pass ea_model_path explicitly for a custom compatible checkpoint. Runtime generation uses a compact tree budget by default; pass tree_budget=59, topk_per_step=10 to reproduce the paper's total-token=60 tree. In current benchmarks, EAGLE-2 shows positive throughput on 13B-class targets and is recommended most strongly for 33B-class and larger targets, where target verification cost amortizes the drafter/tree overhead better.

Measuring acceptance correctly. EAGLE drafters are trained on the target's conversation distribution, so the prompt format and domain dominate the measured acceptance length. Use the model's chat template (for Vicuna: "A chat between a curious user ... USER: {q} ASSISTANT:") with instruction-style prompts. On Vicuna-13B + EAGLE-Vicuna-13B-v1.3 we measure steady-state accept length 5.3–5.8 on code/math prompts, ~3.3 on free-form chat, ~5.0 averaged (matching EAGLE-2's reported ≈4). Generic non-chat text (e.g. "Topic 1: ...") is out-of-distribution and collapses acceptance to ~2.4, which silently understates speculative-decoding throughput. Always report the prompt set used alongside acceptance numbers.

Available auto classes:

Class Use case
AutoDistributedModelForCausalLM Text generation
AutoDistributedModelForSequenceClassification Classification
AutoDistributedModel Raw transformer (no LM head)

Benchmarking

Three benchmark scripts are provided in benchmarks/:

# Measure autoregressive inference throughput (tokens/sec)
python benchmarks/benchmark_inference.py \
  --model meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --seq_len 256

# Measure forward pass throughput (tokens/sec)
python benchmarks/benchmark_forward.py \
  --model meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --batch_size 4 \
  --seq_len 128

# Measure training (forward + backward) throughput
python benchmarks/benchmark_training.py \
  --model meta-llama/Llama-2-7b-hf \
  --initial_peers $BBSERVER \
  --batch_size 4 \
  --seq_len 128 \
  --n_steps 20

Examples

Jupyter notebook examples are in the examples/ directory:

Notebook Description
prompt-tuning-sst2.ipynb Prompt-tune LLaMA for sentiment classification (SST-2)
prompt-tuning-personachat.ipynb Prompt-tune BLOOM for dialogue generation (PersonaChat)

Troubleshooting

Workers cannot find each other

  • Ensure all workers use the same --initial_peers address.
  • If running across machines, verify the bootstrap node has a public IP and the port is open.
  • Use --announce_maddrs to explicitly advertise your public address if behind NAT.

ModuleNotFoundError: No module named 'bloombee'

  • Run pip install bloombee or pip install -e . from the repository root.

Out of GPU memory on a worker

  • Reduce --num_blocks to serve fewer layers.
  • Enable quantization: --quant_type int8 or --quant_type nf4.
  • Use a smaller --max_batch_size.

transformers version mismatch

  • Current source installs and checks for Transformers 5.x: 
    pip install -e .
# or, if repairing an existing environment:
pip install "transformers>=5.5.0"

Slow inference / high latency

  • Latency increases with the number of network hops between layers. Place workers on the same local network when possible.
  • Ensure workers report accurate throughput: use --throughput eval on the first run.

Contribution

Bloombee is mainly developed by PASA Lab at University of California Merced with significant supports from Yotta Labs and College of William&Mary. We welcome and appreciate any contribution to this open-source project.

Citation

If you find BloomBee useful in your research, please cite our paper:

@misc{bloombee2026,
      title={Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization}, 
      author={Jiu Chen and Shuangyan Yang and Xu Xiong and Hexiao Duan and Xinran Zhang and Jie Ren and Dong Li},
      year={2026},
      eprint={2604.21072},
      archivePrefix={arXiv},
      primaryClass={cs.DC},
      url={https://arxiv.org/abs/2604.21072}, 
}

Acknowledgements

BloomBee is built upon the following open-source projects:

  • Hivemind - A PyTorch library for decentralized deep learning across the Internet.
  • FlexLLMGen - An offloading-based system running on weak GPUs.
  • Petals - A library for decentralized LLMs fine-tuning and inference without offloading.

About

Decentralized LLMs fine-tuning and inference with offloading

Topics

deep-learningdistributed-systemsllamamachine-learningpipeline-parallelismpytorchtensor-parallelism

Resources

Readme
Apache-2.0 license

Contributing

Contributing
Activity
Custom properties

Stars

123 stars

Watchers

11 watching

Forks

31 forks
Report repository

Releases

Contributors

Languages