Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
647 lines (564 loc) · 40.7 KB

ARCHITECTURE.md

File metadata and controls

647 lines (564 loc) · 40.7 KB
Raw

E2B Infrastructure Architecture

This document explains what this repository implements, what each service does, and how the services interact. It is the fastest way to build a mental model of the codebase before diving into the code.

Keep this document updated. If a code change alters anything described here (service responsibilities, ports, protocols, data stores, flows, deployment topology), update this file in the same PR.

What this repository implements

E2B provides sandboxes: isolated Linux VMs that start almost instantly (they resume pre-booted snapshots instead of cold-booting), run arbitrary code (typically generated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole backend: the control-plane REST API, the data-plane VM orchestration built on Firecracker microVMs, the in-VM agent, the edge routing layer, and template building.

Two ideas drive the design:

  1. A sandbox is a resumed snapshot. Templates are pre-booted VM snapshots (memory + disk + VM state) stored in object storage. "Creating" a sandbox means restoring a snapshot, which is why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root filesystem is a copy-on-write overlay, so only touched data is ever fetched.
  2. Control plane and data plane are separate. The API decides where a sandbox runs and tracks that it runs (Postgres/Redis); the orchestrator on each node owns how it runs (Firecracker, networking, storage). Sandbox traffic never passes through the API.

System overview

flowchart TB
    subgraph clients["Clients"]
        SDK["SDK / CLI"]
        Browser["Browser / HTTP clients"]
    end

    LB["Load balancer<br/>api.* | *.domain wildcard"]

    VC["volume-content API (belt)<br/>api.&lt;domain&gt;"]

    subgraph controlplane["Control plane (API node pool)"]
        API["API<br/>REST :80, gRPC :5009/:5109"]
        DashAPI["dashboard-api :3010"]
        CP["client-proxy<br/>:3002"]
    end

    subgraph datastores["State"]
        PG[("PostgreSQL<br/>teams, templates, builds, snapshots")]
        RD[("Redis<br/>running sandboxes, routing catalog, caches")]
        CH[("ClickHouse<br/>metrics, events, optional logs")]
        OS[("Object storage GCS/S3<br/>template + snapshot artifacts")]
    end

    subgraph clientnode["Sandbox nodes (one orchestrator per node)"]
        ORCH["orchestrator<br/>gRPC :5008, proxy :5007"]
        subgraph vm["Firecracker microVM (per sandbox)"]
            ENVD["envd :49983"]
            USERPROC["user processes"]
        end
    end

    subgraph buildnode["Build nodes"]
        TM["template-manager<br/>(orchestrator binary, gRPC :5008)"]
    end

    SDK -->|REST| LB --> API
    Browser -->|"port-sandboxid.domain"| LB --> CP
    API -.->|"mint content token + domain"| SDK
    SDK -->|"volume content (token-authed)<br/>api.&lt;BYOC or default domain&gt;"| VC
    API -->|"gRPC Create/Delete/Pause"| ORCH
    API -->|"gRPC TemplateCreate"| TM
    CP -->|"lookup sandbox → node"| RD
    CP -->|"forward :5007"| ORCH
    CP -.->|"gRPC auto-resume"| API
    ORCH --> ENVD
    ENVD --> USERPROC
    API --> PG & RD & CH
    DashAPI --> PG & CH
    ORCH --> OS & CH
    TM --> OS
Loading

Services

Service Package Runs on Purpose
API packages/api API nodes Public REST API; sandbox lifecycle, placement, auth, quotas
Orchestrator packages/orchestrator every sandbox node Runs Firecracker VMs; sandbox create/pause/resume/kill
Template manager packages/orchestrator (role) build nodes Builds templates from Docker images
Client proxy packages/client-proxy API nodes Edge router: sandbox URL → correct node
Envd packages/envd inside every VM In-VM agent: process/filesystem API for SDKs
Dashboard API packages/dashboard-api API nodes Backend for the web dashboard (teams, builds, admin)

Supporting packages: packages/shared (protos, telemetry, storage clients, feature flags), packages/auth (authentication library), packages/db (Postgres migrations + sqlc queries), packages/clickhouse (ClickHouse schema + clients), packages/otel-collector (collector config), packages/nomad-nodepool-apm (autoscaler plugin), packages/local-dev (local stack).

API (packages/api)

The control-plane entry point (Gin, OpenAPI-generated from spec/openapi.yml, port 80).

  • Resources: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs), templates and builds, teams, volumes, API keys, secrets, admin operations.
  • Auth (via packages/auth): team API keys (X-API-Key, e2b_ prefix), auth-provider JWTs (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS. Backed by an auth DB (Postgres) with a Redis team cache.
  • Workload identity: sandbox create accepts an optional iam.tokens map of caller-named workload-token definitions (each an exact audience and tokenType). A non-empty, validated map enables workload identity, whose identity the orchestrator derives from the sandbox's already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the orchestrator in SandboxConfig.iam. The API mints no credential and delivers nothing into the sandbox; file-based delivery is rejected at admission. Definitions are persisted in the running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and orchestrator re-sync; a fork starts a new workload and does not inherit them.
  • Placement: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a static list). Chooses a node per sandbox with a best-of-K algorithm (internal/orchestrator/placement/): sample K ready nodes, score by CPU commitment/usage, pick the lowest; retry on exhausted nodes. Tunable live via feature flags.
  • State: writes sandbox records to Redis (source of truth for running sandboxes) and the sandbox→node routing catalog (sandbox:catalog:{id}) in Redis that client-proxy reads. This API-written record is the default routing source; the orchestrator-written sandbox:routing:{id} is a flag-gated alternative (see "Sandbox routing records"). Persistent entities (templates, builds, snapshots, teams) live in Postgres.
  • Secrets: /secrets is the only public surface for secret management (create, list, get, update, delete). The API authenticates the caller with the customer alternatives above, converts the authenticated team UUID to the project UUID the backend knows, checks the customer-secrets feature flag, and forwards a metadata-only request over a unary gRPC contract (e2b.secretsstore.management.v1) to the secrets store backend named by SECRETS_STORE_BACKEND_GRPC_ADDRESS. No caller credential, header or client-supplied tenant crosses that hop, and no response - here or in a log, a span or an error - carries a secret value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves a marker to a value at sandbox egress, never here. Without a configured address, or with the flag off, the routes stay registered and answer 403. Responses are Cache-Control: no-store, request bodies are capped at 512 KiB, and values at 64 KiB.
  • Rig passthrough: admin endpoints under /clusters/{clusterID}/rigs manage a cluster's orchestrator node pools ("rigs"). They list rigs, change rig capacity, list and terminate instances, and read recent scaling errors. Every handler delegates to the cluster's edge service (/v1/rigs/...) through the per-cluster HTTP client that the cluster sync keeps fresh. The API holds no cloud credentials and makes no scaling decisions. The caller authenticates with an admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the cluster, not that the caller's token is invalid. A cluster with no rig management configured returns an empty list. The local cluster has no edge deployment and answers 501 for every rig operation.
  • Extra listeners: internal gRPC :5009 and edge gRPC :5109 expose ResumeSandbox so client-proxy can wake paused sandboxes on incoming traffic.
  • Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to Loki, with a LaunchDarkly-gated ClickHouse read path (logs-read-config) for local-cluster logs during the log storage migration. LOKI_URL is optional: without it the api has no Loki client and those reads fail until the flag routes them to ClickHouse. LOGS_READ_CONFIG=true|false sets the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement parameters, rate limits, and rollouts.

Orchestrator (packages/orchestrator)

A single Go binary running on every sandbox node (as root). ORCHESTRATOR_SERVICES selects its roles: orchestrator (run sandboxes) and/or template-manager (build templates). Code lives under pkg/, almost all Linux-only.

gRPC services on :5008 (pkg/server/, pkg/service/, pkg/template/server/, pkg/volumes/):

  • SandboxService — Create, Update, List, Delete, Pause, Checkpoint.
  • TemplateService — TemplateCreate, TemplateBuildStatus, TemplateBuildDelete (template-manager role only).
  • InfoService — node identity, roles, capacity, health status (used by API node discovery).
  • ChunkService / VolumeService — peer-to-peer template chunk serving; persistent volumes.

Process shutdown, triggered by SIGINT, SIGTERM, SIGUSR1, or a service failure, moves the node to ShuttingDown before draining builds and sandboxes. Like Draining, this excludes new placement while existing work stays reachable, and /health returns HTTP 200 with status draining. Unlike reversible Draining, ShuttingDown is terminal for the process; status overrides cannot enter or leave it. FORCE_STOP skips waiting for sandboxes to exit.

After sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP server shutdown with a separate deadline, then forces server closure if needed. Successful forced closure recovers graceful deadline expiry; other server errors still propagate. Pprof stays available through service teardown; its final shutdown also has a deadline and forced-close fallback.

InfoService.ServiceInfo reports optional outstanding_work for sandbox, template-builder, and mixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes tracked background persistence and cleanup. Reporting nodes send an explicit zero when idle; an absent field means unknown. The API caches this report and exposes it as optional top-level outstandingWork in admin node list and detail responses, preserving explicit zero and omitting unknown counts. This observational count does not authorize node deletion.

Key mechanisms (all under pkg/sandbox/):

  • Firecracker (fc/): each sandbox is one Firecracker process in its own cgroup and network namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots. Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.
  • Lazy memory / UFFD (uffd/): on resume, Firecracker restores the VM without loading memory; a userfaultfd handler serves page faults directly from the template's memfile, so only touched pages are read. An optional prefetcher warms known-hot pages.
  • Copy-on-write rootfs (rootfs/, nbd/, block/): the template rootfs stays read-only; writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.
  • Template cache (template/): templates are fetched lazily from object storage and cached on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other nodes before upload completes).
  • Networking (network/): each sandbox gets a slot — a network namespace with a veth pair and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and reused; slot indexes are allocated locally against the node's netns state (leftover namespaces from a previous run are torn down by startup reclaim).
  • Sandbox proxy (:5007, pkg/proxy/): reverse-proxies incoming traffic from client-proxy to the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox traffic access tokens. HTTPS backends may use self-signed certificates.
  • Writes sandbox lifecycle events and cgroup host stats to ClickHouse; exports metrics via OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy collector remains the fallback primary destination, and configured shadow destinations can mirror writes during collector/storage migrations without changing sandbox behavior.

Detached sandbox-event publication holds node work until the publisher returns. This covers delivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory batcher, whose flushing remains part of shutdown.

Envd (packages/envd)

The agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.

  • Process service (spec/process/process.proto): start/list/connect to processes, stream stdout/stderr, stdin, signals, PTYs — this is what SDKs use to "run code".
  • Filesystem service (spec/filesystem/filesystem.proto): stat/list/make/move/remove/watch.
  • REST: /health, /metrics, /envs, /files upload/download, /init (orchestrator pushes env vars, access token, metadata after boot/resume), /upgrade (live self-upgrade, below), freeze/thaw hooks used during pause.
  • Public vs. control-plane routes: the control routes (/init, /upgrade, and the freeze/thaw hooks) are marked x-internal: true in spec/envd.yaml, and /upgrade — which the spec does not describe — is listed alongside them in the orchestrator's pkg/sandbox/envd. The orchestrator reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a 404, for every method, so they are not reachable through a sandbox URL. Adding a control route therefore means marking it in the spec (go generate carries the marker into the proxy's rejection list) — otherwise it ships reachable from the internet.
  • Auth: X-Access-Token header checked against a token delivered via Firecracker MMDS; signed URLs for file endpoints. /init is exempt (it is what delivers the token), which is the main reason the proxy refuses it outright.
  • Live upgrade (internal/services/process/upgrade.go): an authenticated POST /upgrade lets the orchestrator swap envd inside a running sandbox at resume. It streams the new binary in the request body and envd syscall.Execs into it with the same PID, carrying the workload's stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade /init restores the access token (so no re-adopted process runs unauthenticated), and the handover outcome (procs/watchers re-adopted, plus any failures) rides back on that /init's X-Envd-Handover header for fleet visibility.
  • Scans guest ports and forwards them so any port a user process opens becomes reachable through sandbox URLs. pkg/version.go must be bumped on every behavioral change — the API and the orchestrator gate features on the envd version recorded in each template build.

Client proxy (packages/client-proxy)

The stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates https://<port>-<sandboxID>.<domain> requests (host parsing in packages/shared/pkg/proxy/host.go), looks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to that node's orchestrator proxy on :5007 by default. ORCHESTRATOR_PROXY_PORT selects a different downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls the API's ResumeSandbox gRPC and retries — paused sandboxes wake transparently on traffic. By default the record is the API-owned sandbox:catalog:{id}; the orchestrator-routing-prioritized flag switches the read to the orchestrator-owned sandbox:routing:{id} (see "Sandbox routing records" below).

Dashboard API (packages/dashboard-api)

A separate REST service (port 3010, spec spec/openapi-dashboard.yml) consumed by the web dashboard, not the SDK: legacy team management/provisioning, template tags, build listings, admin bootstrap. Its admin routes accept either a shared admin token or a short-lived service JWT verified against the configured admin JWKS. The disable-legacy-team-mutations LaunchDarkly flag rejects legacy lifecycle writes with 412 after authentication; it leaves reads and the workspace API's management projection writes available. Team-scoped template and build read routes accept either dashboard user auth or team API key auth (X-API-Key). Its workspace-agnostic /v1/management operations are defined in the same dashboard OpenAPI contract and registered on the existing router. Their AdminJWTAuth OpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api /.well-known/jwks.json endpoint, with accepted signing methods derived from each JWK's required alg metadata. Issuers and audiences are configured through the JSON ADMIN_AUTH_PROVIDER_CONFIG value — the same config shape as AUTH_PROVIDER_CONFIG. Talks to Postgres and ClickHouse; never talks to orchestrators.

An issuer normally configures at least one accepted audience, binding a JWT to its intended target. An issuer may omit audiences only with no audienceMatchPolicy; this deliberately disables audience matching while issuer, signature, and temporal-claim verification remain required.

The /v1/management operations are the cluster's half of a contract the workspace residency owns: project upsert (a project is a public.teams row created from a caller-supplied UUID; the tier is assigned once at creation from a local default and no push moves it; a changed slug renames the project, and nothing else follows it), per-member projection, and limit sync (into project_limits, which team_limits reads in preference to tiers). All are idempotent, because the caller is level-triggered and retries. PUT /v1/management/projects/{projectID}/members/{userID} applies the desired presence for one user, gated by a monotonic per-project/user revision stored in projection.project_members; duplicate or older revisions succeed without changing target state. PUT /v1/management/projects/{projectID}/limits is gated the same way, by a monotonic per-project revision in projection.project_limits: the caller raises it whenever the limits it resolved for a project change, and a delivery at or below the recorded revision is dropped and still answers 204. The ledger and the values in public.project_limits advance in one transaction, so a revision is never recorded without the values it admitted. Both fences are the target's, and both exist for the same reason — the caller can only fence what it sends, so two deliveries in flight arrive in whichever order the network gives them and the older one has to be refused where it lands. A present projection includes that User's OIDC issuer/subject identities. Every projected user has at least one identity, and an identity already owned by a different user returns 409. A revocation removes only that User's users_teams row; projected Users and identities are retained. Membership writes live in internal/management with their post-commit cache eviction rather than in the handlers: auth caches member authorization, so each accepted command invalidates that User's authorization for the Project after commit.

The management surface also owns a replay-safe cluster lifecycle. A caller registers a stable cluster UUID with immutable connection details, assigns it only to the named project, detaches that exact assignment before provider cleanup, and deletes the cluster only after no project references it. Replaying the same registration or assignment succeeds, while a changed descriptor, a different assignment, an inexact detach, or deletion of a referenced cluster returns a conflict. A new or replacement assignment requires the project's tier identifier to contain enterprise, case-insensitively; an identical assignment remains replayable after a later tier change.

DELETE /v1/management/projects/{teamID} is declared and answers 501. envs, snapshots and volumes reference teams with ON DELETE NO ACTION and templates are only soft-deleted, so a project that ever built one pins its team row — and releasing it needs the API service's orchestrator connections, which this service does not have. Projects are not deleted from control planes today.

Data stores

Store Owner packages What lives there
PostgreSQL packages/db (goose migrations, sqlc) Durable control-plane state: teams, users, tiers (quota defaults), project_limits (per-team quota overrides pushed in by the owning service; the team_limits view reads it in preference to tiers), envs (templates), env_builds (build rows: vcpu, ram_mb, status, versions), env_aliases, snapshots (paused sandboxes), team_api_keys, volumes, clusters
Redis API, client-proxy, orchestrator Ephemeral runtime state: running-sandbox store (source of truth), sandbox→node routing catalog, team/template/snapshot caches, rate limiting, P2P chunk peer registry
ClickHouse packages/clickhouse Time-series/analytics: metrics_gauge/metrics_sum (written by the OTel collector), sandbox_events, sandbox_host_stats (written by orchestrator), team metrics, and optionally sandbox_logs during the log migration. Read by API and dashboard-api
Object storage (GCS/S3/local, packages/shared/pkg/storage) orchestrator, template-manager Template & snapshot artifacts, keyed by build ID: {buildID}/memfile, {buildID}/rootfs.ext4, {buildID}/snapfile, {buildID}/metadata.json + .header index files

A template and a paused-sandbox snapshot have the same artifact shape — a snapshot is just a new build whose memfile/rootfs are stored as diffs against the template it came from (diff chains are resolved through the .header files).

Core flows

Sandbox creation

sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant R as Redis
    participant O as Orchestrator (chosen node)
    participant FC as Firecracker
    participant E as envd (in VM)

    C->>API: POST /sandboxes {templateID}
    API->>API: auth team, resolve template alias → ready build (Postgres/cache)
    API->>API: best-of-K placement → pick node
    API->>O: gRPC SandboxService.Create(SandboxConfig)
    O->>O: fetch template (local cache / NFS / object storage)
    O->>O: acquire network slot + NBD rootfs overlay + uffd memory
    O->>FC: load snapshot, resume VM
    O->>E: POST /init (env vars, access token) — retried until ready
    E-->>O: 204
    O-->>API: Create OK
    API->>R: store running sandbox + routing catalog entry
    API-->>C: 201 sandbox {sandboxID, domain}
Loading

The API blocks on the gRPC Create, which itself blocks on envd's /init — when the client gets a response, the sandbox is fully usable. Fresh creates are internally a resume of the template's base snapshot (cold boots happen for filesystem-only templates and builds, or when an explicit resume requests one — see pause and resume below; template creates never do).

Sandbox traffic

sequenceDiagram
    autonumber
    participant U as Client
    participant CP as client-proxy :3002
    participant R as Redis catalog
    participant API as API
    participant OP as orchestrator proxy :5007
    participant E as envd / user process

    U->>CP: https://3000-i7fa3.domain
    CP->>CP: parse host → port 3000, sandbox i7fa3
    CP->>R: GetSandbox(i7fa3)
    alt running
        R-->>CP: node IP
    else paused / unknown
        CP->>API: gRPC ResumeSandbox(i7fa3)
        API-->>CP: node IP (after resume)
    end
    CP->>OP: forward to http://nodeIP:5007
    OP->>OP: lookup sandbox, check traffic access token
    OP->>E: http://slotIP:3000 (via veth/tap into VM)
    E-->>U: response
Loading

Sandbox routing records

client-proxy resolves the node IP of a sandbox from a routing record in Redis. Two records exist today. Both have the same JSON shape (sandbox_catalog.SandboxInfo in packages/shared/pkg/sandbox-catalog): orchestrator_id, orchestrator_ip, execution_id, sandbox_started_at, sandbox_max_length_in_hours.

Record Key Writer Written Deleted
API-owned (default) sandbox:catalog:{sandboxID} API (cloud) or the cluster edge from gRPC metadata (BYOC) after Create returns before Pause/Kill is sent to the node
Orchestrator-owned (v1, flag-gated) sandbox:routing:{sandboxID} orchestrator, packages/orchestrator/pkg/routing on MarkRunning (sandbox enters the live map, envd is ready) on MarkStopping (kill, pause, checkpoint, crash)

The API-owned record is still the source of truth. client-proxy reads sandbox:catalog:{id} unless the orchestrator-routing-prioritized flag is on. The orchestrator-owned record is a v1 test path. It runs next to the API path and does not replace it yet.

Two feature flags in packages/shared/pkg/featureflags control the new path:

  • orchestrator-routing-publish (orchestrator): write sandbox:routing:{id} on MarkRunning and delete it on MarkStopping. A failed write is logged and counted (orchestrator.routing.publish.total{result=error}); the sandbox keeps running. Build sandboxes are skipped. The delete is guarded by execution_id in a Lua script, so a stale lifecycle never removes the record of a newer execution.
  • orchestrator-routing-prioritized (client-proxy): resolve the node from sandbox:routing:{id} instead of sandbox:catalog:{id}. There is no fallback to the API-owned record on a miss. A miss goes to the auto-resume path (ResumeSandbox gRPC to the API), same as today.

Rollout order: turn on orchestrator-routing-publish first and wait one maximum sandbox length, so every live sandbox has a record. Then turn on orchestrator-routing-prioritized. To roll back, turn off orchestrator-routing-prioritized; the API path is untouched.

The TTL of both records is sandbox_max_length_in_hours from the write time. The record is deleted earlier in every normal stop path.

Volume content

Persistent volumes (packages/orchestrator/pkg/volumes/) are managed through the control-plane API (POST/GET /volumes), but their content — reading and writing files — is served by a separate volume-content API (belt, e2b-dev/belt) that the SDK talks to directly, not through the control-plane API. The API's role is to mint the credential and tell the SDK where to send content traffic.

sequenceDiagram
    autonumber
    participant U as SDK
    participant API as API
    participant PG as PostgreSQL
    participant VC as volume-content API (belt)

    U->>API: POST /volumes (create) or GET /volumes/{id}
    API->>PG: persist / load volume row
    API->>API: mint JWT (aud = https://api.&lt;domain&gt;)<br/>resolve domain
    API-->>U: { volumeID, name, token, domain? }
    Note over U: domain is returned only for BYOC teams;<br/>SDK stores it and falls back to api.&lt;E2B_DOMAIN&gt; otherwise
    U->>VC: /volumecontent/{id}/... at api.&lt;domain&gt;<br/>Authorization: Bearer token
    VC->>VC: verify token (audience must match its own origin)
    VC-->>U: file content
Loading
  • Domain selection. The token's audience and the content host are the same origin, https://api.<domain>. For teams on a custom (BYOC) cluster (team.ClusterID set), the API returns that cluster's domain (cluster.SandboxDomain, resolved in handlers.volumeContentDomain) so content traffic goes to the BYOC cluster's edge instead of the control-plane host. For teams on the default cluster the response omits domain and the SDK uses its configured default (api.<E2B_DOMAIN>); the audience then uses the deployment's DOMAIN_NAME.
  • Token. A short-lived JWT (handlers.generateVolumeContentToken, config in cfg.VolumesTokenConfig) signed by the API, scoped to the team and volume, presented as a bearer token on every content request. Its aud claim is https://api.<domain>, so a token minted for one cluster's origin is not accepted by another.

Pause and resume

  • Pause: API records a snapshot row in Postgres, then gRPC Pause to the node. The orchestrator pauses the VM, snapshots it, diffs memory (dirty-page tracking) and rootfs (COW cache) against the template, caches the snapshot locally, and uploads asynchronously to object storage (with a retry budget). The sandbox leaves the Redis catalog.

    • Deferred rootfs export (gated by the deferred-rootfs-export flag in packages/shared/pkg/featureflags): instead of diffing the rootfs on the pause critical path, the orchestrator ejects the writable COW cache during pause and returns, then seals it into the rootfs diff (reflink) in the background. This moves the rootfs-diff latency off the pause, but the local snapshot's rootfs body isn't materialized until the seal finishes, so the async upload — and any origin-node resume/prefetch that reads the rootfs diff — waits on the seal. A seal failure is permanent (it never re-runs), so the upload fails fast rather than retrying.

  • Resume: same path as creation, but placement prefers the origin node — if the snapshot is still in its local cache, resume avoids any object-storage reads. Checkpoint is a pause+resume in place used to persist state while keeping the sandbox running.

  • Explicit filesystem-only resume: memory: false on resume/connect demands a cold boot (RebootSandbox) even when the snapshot includes memory, as a self-serve rescue when the restored memory state is unusable. Gated per team by the fs-only-resume-api flag; when off the request is rejected with an explicit error, never silently downgraded to a memory restore. The disk has crash-recovery semantics (unflushed pre-pause writes are lost), nothing durable is mutated (the memory snapshot survives untouched), and auto-resume never takes this path — traffic always memory-resumes. The one way a memory-preferring sandbox ends up with a filesystem-only snapshot is at pause time, not resume time: when the node keeps refusing a timeout auto-pause (its parent memfile header still deduplicating after the admission grace) for longer than the retry budget counted from the first refusal (auto-pause-overstay-budget-milliseconds, 120 s by default; 0 degrades at the first refusal, negative never degrades), the evictor requests a filesystem-only snapshot instead, so the sandbox stops overstaying its expiry; the next resume of that snapshot is a cold boot. The degrade is only ever decided on a refusal in the same sweep, so eviction lag alone never degrades anything, and refusals only survive with pause-refusal-restore on. On a BYOC cluster that flag also needs every edge replica on a release that restores the route after a refusal: roll the edge fully first, and turn the flag off for the cluster before any edge rollback — nothing in the cluster model can check the edge's version.

  • Pre-boot filesystem recovery: every cold boot of a rootfs that was not frozen at pause (fs_quiesced false/absent) runs a jailed e2fsck -p -E journal_only before the VM starts — journal replay only, the same recovery the guest kernel would do at mount — so a memory: false rescue and a legacy sync-fallback filesystem-only snapshot both mount a consistent disk. It replays and exits without a full consistency scan, so the cost is bounded by journal content, not filesystem size. Runs under the same confinement as the offline envd swap (unprivileged transient unit, device access pinned to the sandbox's own NBD node). A clean replay boots; anything else fails the start with the snapshot untouched but stays retryable. Journal replay never condemns a snapshot: its exit codes cannot tell an unmountable filesystem apart from a transient device fault, so every non-replayed outcome — an operational failure (timeout, I/O, device error) or an e2fsck exit that is not a clean replay — is retryable, never a permanent customer-facing verdict. In-file corruption that still mounts is likewise not condemned — replay does not scan, so it boots. Whole-filesystem repair and condemning a rootfs are left to a separate opt-in full-filesystem repair path. Gated by the preboot-fs-recovery flag, separate from fs-only-resume-api because it also changes the behavior of existing filesystem-only cold boots.

  • Envd live-upgrade on resume: the orchestrator can upgrade the sandbox's envd to a newer node-local build during resume (gated by the envd-upgrade-target flag in packages/shared/pkg/featureflags), via envd's POST /upgrade (see the envd section). It is best-effort — a delivery failure before the exec leaves the old envd serving — except an unrecoverable post-exec failure (the new envd never re-initializes), which fails the resume rather than return a permanently unusable sandbox.

  • Envd offline-upgrade on cold-boot resume: reaches envd too old for the live /upgrade handover (below MinEnvdVersionForUpgrade). When a filesystem-only snapshot cold-boots (RebootSandbox), the orchestrator rewrites /usr/bin/envd in the rootfs before the VM boots (PreBootFn → pkg/sandbox/rootfs.SwapEnvdBinary), entirely in userspace via a jailed debugfs — never a host-kernel mount of the tenant image. The old envd never participates, so the method is version-agnostic. Gated by the envd-offline-upgrade-target flag (a sibling of envd-upgrade-target sharing the same version-remap resolver), and applied only when the snapshot's rootfs was captured frozen (fs_quiesced, so it is crash-consistent); best-effort (a swap failure boots the original envd). Because the swap keys on the snapshot's built-with version, which it does not advance, it re-fires idempotently on each cold-boot resume until a re-pause re-bakes the running version.

  • Both upgrade paths read the host envd binary through a node-local cache (gated by the envd-binary-cache flag): the version probe, the live delivery and the offline swap's staging copy all read a local copy rather than the read-only artifact mount, which an exec would otherwise demand-page in small random reads. The cache is why neither path performs bulk I/O against that mount on the resume path.

    It adds a third condition to both paths, so an eligible snapshot is not unconditionally upgraded: an upgrade whose binary is not cached on this node is deferred to a later resume rather than served from the mount, since reading it there would put tens of seconds on a path a customer is waiting on. A node whose cache is cold — any miss, and in particular a ramp keyed on sandbox or team, where the boot-time warm does not evaluate — defers that upgrade while a background warm populates the cache; on a node-scoped rule the boot warm normally lands before the first resume. Both upgrades are idempotent and re-fire per resume, so a deferral costs one cycle. Ramp the flag on a node-scoped context kind — the cache is shared by every sandbox on the node.

  • Auto-pause/auto-resume make sandboxes effectively serverless: idle sandboxes pause, traffic resumes them (see traffic flow above).

Template build

sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant TM as template-manager (build node)
    participant FC as Firecracker build VMs
    participant OS as Object storage

    C->>API: POST /v3/templates (register build: cpu, ram, free-disk target) → Postgres env_builds
    C->>API: POST /v2/templates/{id}/builds/{buildID} (recipe: steps, start/ready cmd)
    API->>TM: gRPC TemplateCreate(TemplateConfig)
    TM->>TM: pull image → inject envd/provisioning → extract ext4 rootfs
    TM->>FC: boot VM per phase: provision → user steps
    TM->>TM: resize disk on host
    TM->>FC: boot VM per phase: finalize → optimize
    TM->>OS: upload layers + final {buildID}/memfile, rootfs.ext4, snapfile, metadata
    API->>TM: poll TemplateBuildStatus
    API->>API: mark build ready in Postgres
Loading

Builds are layered (pkg/template/build/phases/): base → user → one layer per recipe step → resize disk → finalize → optimize. Each layer is hashed and cached, so rebuilds only re-run changed steps. Resize disk grows the quiescent rootfs on the host; the other non-cached phases run in a real Firecracker VM and their pause-diffs become layers. The optimize phase records which memory pages a fresh resume touches, producing prefetch hints that speed up future sandbox starts.

Template creation holds node work through foreground setup and asynchronous build completion, including layer uploads, synchronous cleanup, and final status publication. Cancellation can mark a build failed before execution finishes; the work hold remains until execution unwinds. Template deletion owns a separate hold until artifact cleanup returns.

Deployment topology

The services are scheduler-agnostic binaries and containers; the supported way to run them is the Kubernetes-based distribution. The roles below hold regardless of how the nodes are provisioned.

flowchart TB
    LB["Load balancer + TLS<br/>api.* → API | *.domain → client-proxy"]

    subgraph control["control plane"]
        AJ["api, dashboard-api, client-proxy,<br/>redis, otel-collector"]
    end
    subgraph sandboxnodes["sandbox nodes (autoscaled)"]
        OJ["orchestrator<br/>+ Firecracker sandboxes"]
    end
    subgraph buildnodes["build nodes (autoscaled)"]
        TJ["template-manager"]
    end
    subgraph ch["analytics"]
        CJ["clickhouse"]
    end

    LB --> control
    AJ -->|gRPC| OJ & TJ
Loading
  • Control-plane services (api, dashboard-api, client-proxy) are stateless containers and the only LB backends. The api discovers orchestrator nodes through packages/shared/pkg/servicediscovery (Kubernetes, DNS, a static list, or the legacy Nomad backend the code still carries).
  • Sandbox nodes run the orchestrator directly on the host (it needs root for Firecracker, namespaces, NBD, cgroups), configured with hugepages and local template caches. Autoscaled.
  • Build nodes run the same binary in template-manager mode.
  • PostgreSQL is external (connection string via secrets); Redis is a managed service or a single in-cluster instance; ClickHouse runs on its own nodes.
  • Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics) and Grafana Cloud/stack. Logs default to the legacy Vector → Loki path; dynamic log routing can select a primary collector and shadow collectors, and local-cluster log reads can be switched to ClickHouse with logs-read-config after sandbox_logs is populated (LOGS_READ_CONFIG is the flag's fallback where LaunchDarkly has no value). Once reads are on ClickHouse, Loki can be left out of a deployment and the api started without LOKI_URL.

Repository layout

packages/
  api/                  Control-plane REST API
  orchestrator/         Sandbox runtime + template builder (one binary, per-node)
  client-proxy/         Edge router for sandbox traffic
  envd/                 In-VM agent (bump pkg/version.go on behavior change!)
  dashboard-api/        Web-dashboard backend
  shared/               Protos, telemetry, storage clients, proxy engine, feature flags
  auth/                 AuthN library (API keys, JWT/OIDC) used by api + dashboard-api
  db/                   Postgres migrations (goose) + queries (sqlc)
  clickhouse/           ClickHouse schema, batching writers, query clients
  otel-collector/       Collector config
  nomad-nodepool-apm/   Nomad autoscaler metric and deployment-aware target plugins
  local-dev/            docker-compose local stack + DB seeding
spec/                   OpenAPI specs (public, edge, dashboard) — codegen sources
tests/integration/      Integration tests against a live deployment

Cross-service contracts are all generated: OpenAPI specs in spec/, gRPC protos in packages/orchestrator/*.proto and packages/envd/spec/, SQL in packages/db/queries/. Run make generate after changing any of them.