Skip to content

Repository files navigation

jailmachine

jailmachine (jm) is docker-machine / podman machine for FreeBSD. One command brings up a FreeBSD 15.1 virtual machine on your Mac, provisions it with podman and bastille, and hands your host's podman or docker client an endpoint pointing at it — so you can run native FreeBSD OCI images, Linux images through the Linuxulator, and jails, from a macOS terminal, without keeping a FreeBSD box around.

Status: MVP / working demo. This proves the idea end to end and is usable for real work: jm init && jm start, then build and run FreeBSD and Linux images, publish ports to the host, create jails in the guest. Since v0.1.0 it also shares host directories at identical paths, resolves names exactly as your Mac does, starts the machine on demand from jpodman / jdocker, and ships a jdocker wrapper for the docker CLI. It is still not a Docker Desktop replacement: see what works, and what does not yet.

Install

macOS on Apple Silicon only. qemu (for HVF) and podman (which also ships gvproxy) are required; the cask declares both and creates the jpodman and jdocker symlinks, otherwise they are yours to arrange.

brew install --cask gabrielbelli/tap/jailmachine

# or:
brew install qemu podman
go install github.com/gabrielbelli/jailmachine/cmd/jm@latest
ln -sf "$(go env GOPATH)/bin/jm" "$(go env GOPATH)/bin/jpodman"
ln -sf "$(go env GOPATH)/bin/jm" "$(go env GOPATH)/bin/jdocker"

# or, from source (PREFIX defaults to /opt/homebrew):
git clone https://github.com/gabrielbelli/jailmachine && cd jailmachine && make install

jm doctor checks every tool and machine and prints a fix per failure. Details in docs/INSTALL.md.

60-second quickstart

jm init      # SSH key, download and verify the guest image, grow the disk, write the seed
jm start     # boot, provision, connect podman, share host paths, start the forwarder and resolver

jm init takes 60–115 s. It downloads roughly 800 MiB and checks its SHA256, but the download is the smaller half — about 31 s on this link, against 59–113 s for an init from an image already cached on disk. Writing disk.raw out is what dominates, and that is a bug of ours: see the machine itself. On the prebaked image a cold first boot takes about 22 s (32 s was observed once with two other VMs already running on the host) and a warm start 12–25 s on an idle Mac; --image official:<release> provisions a stock FreeBSD cloud image on first boot instead, taking about 2 minutes.

jpodman is podman pointed at the machine and jdocker is the docker CLI pointed at the same engine, whatever your default connection or docker context is; jm never repoints a default you already had (jm start --set-default opts in, and on a Mac with no podman connections at all, podman itself promotes the first one jm registers). Both wrappers start a stopped machine for you, printing one line on stderr while it boots — JM_AUTOSTART=0, or --no-autostart as the first argument, makes them fail instead. A machine left idle for 30 minutes is suspended to disk and gives its memory back to macOS; the wrappers wake it again in seconds, whatever JM_AUTOSTART says.

The guest is FreeBSD, so Linux images need --os=linux with podman:

jpodman run --rm --os=linux docker.io/alpine echo hi              # Linux, via the Linuxulator
jpodman run --rm ghcr.io/freebsd/freebsd-runtime:15.1 uname -srm  # native FreeBSD

jdocker needs no flag — the docker CLI has none, so the wrapper defaults DOCKER_DEFAULT_PLATFORM=linux/arm64 and a plain jdocker run pulls the Linux image as Docker Desktop would. Set DOCKER_DEFAULT_PLATFORM yourself, or pass --platform, for native FreeBSD images.

jdocker run --rm docker.io/alpine echo hi
jdocker compose up -d          # a compose.yaml, driving the guest's podman
jpodman kube play pod.yaml     # or Kubernetes YAML, FreeBSD's native route

Those two and jpodman compose are all covered in Compose and Kubernetes YAML.

Build a native FreeBSD image:

cat > Containerfile <<'EOF'
FROM ghcr.io/freebsd/freebsd-runtime:15.1
RUN env ASSUME_ALWAYS_YES=yes pkg bootstrap -f && pkg install -y curl && pkg clean -ay
CMD ["uname", "-srm"]
EOF
jpodman build -t jm-demo . && jpodman run --rm jm-demo

Publish a port and reach it from the Mac (the forwarder reconciles a second or two after the container starts, hence the retry):

jpodman run -d --rm --os=linux -p 8080:80 --name web docker.io/busybox \
  sh -c 'echo hello from the FreeBSD VM > /tmp/index.html && httpd -f -p 80 -h /tmp'
curl --retry 10 --retry-connrefused http://localhost:8080/   # hello from the FreeBSD VM
jm ports              # what is mapped, where it binds, and why something is not
jpodman rm -f web

For plain podman or a docker client you would rather point yourself: eval "$(jm env)" (fish: eval (jm env --shell fish)). A Linux image in a compose file or a Pod manifest needs its platform naming — see Compose and Kubernetes YAML.

Sharing host directories

Host directories appear in the guest at the same absolute path, so a volume argument written on the Mac resolves inside the VM unchanged — from any directory, with no rewriting and no /host_mnt prefix:

jpodman run --rm --os=linux -v ~/code:/app docker.io/alpine ls /app
jpodman run --rm --os=linux -v ~/code:"$HOME/code" docker.io/alpine ls "$HOME/code"

A new machine shares your home directory, /Volumes, /private/tmp and the per-user temporary directory $TMPDIR lives in (/var/folders/<hash>), so -v $(mktemp -d):/work and -v $TMPDIR/x:/work both work. jm inspect lists the set, and jm doctor checks that a file written on the host really is visible to a container at the same path.

jm init --mount /work --mount /srv/data:ro   # on top of the defaults
jm init --no-mounts                          # share nothing

# the share set changes only on a stopped machine, and applies on the next start
jm stop && jm set --mount /work --unmount /Volumes && jm start
jm stop && jm set --no-mounts && jm start    # drop every share

jm inspect | grep -i share

Every container can read and write everything shared — by default that is your whole home directory, including ~/.ssh, ~/.aws and ~/.jailmachine itself. That is the same posture as Docker Desktop's default, and it is a deliberate one. If you run images you do not trust, narrow it on a stopped machine:

jm stop
jm set --no-mounts                                    # drop every share
jm set --mount ~/code --mount "/srv/data:ro"          # add back only these
jm start

--no-mounts cannot be combined with --mount or --unmount in one call, hence the two commands.

/tmp is the one path that cannot follow the rule. On macOS it is a symlink to /private/tmp, and a share mounted at the guest's own /tmp would shadow it — so jm shares /private/tmp instead and never rewrites your -v argument. Write -v /private/tmp/x:/app, not -v /tmp/x:/app; the latter fails with source path does not exist — or, if that path happens to exist in the guest as well, silently binds the guest's own empty /tmp/x.

zsh users: brace a :ro suffix — quoting does not help. In zsh, :ro at the end of a word carrying a parameter expansion is a history modifier: :r strips the extension and the o is left behind, so both $P:ro and "$P:ro" become /Users/you/codeo. Only braces or a backslash survive:

jm set --mount "${P}:ro"                  # or $P\:ro
jpodman run --rm --os=linux -v "${P}:${P}:ro" docker.io/alpine ls "$P"

Nothing warns you. jm set --mount $P:ro on a stopped machine exits 0 and changes nothing: the mangled path falls inside a root that is already shared, so it is absorbed and the :ro is lost. The only signal is what is missing — no ==> share: lines and no "attached on the next start" notice, just the machine summary. A correct "${P}:ro" prints both. bash is unaffected in every form.

Shares are for source trees and data. utimes is a silent no-op, an inotify watch cannot be created on a share at all — inotify_add_watch fails with Bad file descriptor, where it works on an engine-managed volume and on the container's own filesystem, so use a polling watcher (#4) — and 9p is far slower than the guest's ZFS (~70 MB/s, and worse on metadata: 1000 small files in 3.6 s against 0.76 s) — keep build output in an engine-managed volume.

A file a container creates shows up on the Mac as 0600, with its real mode and owner in user.virtfs.* xattrs. Shares use the 9p mapped-xattr security model, which keeps guest ownership and modes in xattrs instead of on the host file — the only way a container running as root can rewrite a file it has just made read-only, which git clone needs and macOS refuses when the modes are host-native. JM_9P_SECURITY=none trades it back. See Modes and ownership on a share.

Name resolution

Whatever resolves on the Mac resolves in the guest and in containers, with the same answer. jm start runs a small resolver on the host and points the guest at it, so queries go through macOS's own resolution API: split-horizon VPN resolvers, per-domain nameservers, search domains from scutil --dns, /etc/hosts entries, .local mDNS names, and the special-use development TLDs .test, .invalid, .home.arpa and .onion that a stock DNS resolver would answer NXDOMAIN on its own. Joining or leaving a VPN converges without a restart.

The Mac itself is host.docker.internal and host.containers.internal from inside a container, and answers to its own hostname and .local name; a host answer of 127.0.0.1 is rewritten to the address that means "the host" in the guest, so a service on your loopback is reachable.

jpodman run --rm --os=linux docker.io/alpine ping -c1 host.docker.internal
jm doctor            # asserts the guest resolves a host-only name to the right address

If the guest's resolver cannot be brought up, jm start warns and carries on with the resolution the guest already had rather than failing; the check in jm doctor is what reports the loss, and resolver.log in the machine directory says why.

Publishing ports

jpodman run -p works as on any other machine: a forwarder started by jm start watches podman events and converges gvproxy's mapping table onto the guest's container state.

Published ports bind every host interface by default, as docker run -p does on Linux: 127.0.0.1, ::1, localhost and your LAN address, so anyone on your network can reach the container. Confine them to the loopback with jm init --publish-addr 127.0.0.1 or, on an existing machine, jm set --publish-addr 127.0.0.1 (it applies from the next jm stop + jm start). The address is stored on the machine and shown by jm inspect and jm ports; JM_PUBLISH_ADDR is an override read at jm start time and written onto the record.

Naming a host address in the flag itself works and wins over the default: -p 127.0.0.1:8080:80 binds that address on the Mac and nothing else, whatever --publish-addr says.

How it works

QEMU (-M virt,accel=hvf, EDK2 firmware, virtio) boots a FreeBSD 15.1 arm64 guest; networking is gvproxy, the same userspace stack podman machine uses. Host to guest is SSH only, as root with a dedicated ed25519 key: FreeBSD has no vsock driver, so the engine socket is tunnelled over SSH rather than exposed directly. Host directories are exported over virtio-9p (there is no virtiofs driver either) and mounted declaratively by the guest before the engine starts.

flowchart LR
  subgraph mac["macOS host"]
    jm["jm (lifecycle)"]
    cli["jpodman / jdocker / podman / docker"]
    gv["gvproxy"]
    fwd["port forwarder"]
    res["host resolver"]
  end
  subgraph vm["FreeBSD guest (QEMU + HVF)"]
    sshd["sshd"]
    api["podman system service<br/>/var/run/podman/podman.sock"]
    unb["local_unbound"]
    work["containers + bastille jails"]
    shr["9p shares at identity paths"]
  end
  jm -->|SSH control channel| sshd
  cli -->|unix socket| gv
  gv -->|ssh -L| api
  fwd -->|events + ps| api
  fwd -->|expose / unexpose| gv
  gv -.->|virtio-net 192.168.127.2| sshd
  unb -->|forward to 192.168.127.254| res
  api --> work
  jm -.->|virtio-9p| shr
  work --> shr
  work --> unb
Loading

Published ports are reconciled, not requested: the forwarder watches the guest's container state and converges gvproxy's mapping table onto it. State lives in ~/.jailmachine/machines/<name>/ (JM_HOME or --state-root move it), so deleting it uninstalls completely. More in docs/ARCHITECTURE.md.

Commands

[name] defaults to jailmachine. Exit codes: 0 ok, 1 failure, 2 usage. Every flag and environment variable is in docs/USAGE.md.

Command Does
jm init [name] Create a machine: SSH key, image download and SHA256 check, grow disk, NoCloud seed. --cpus, --memory (2048 MiB by default), --arc (the guest's ZFS cache cap, 512 MiB by default), --idle-suspend (30 min by default, 0 never), --disk, --image, --ssh-port, --mount, --no-mounts, --publish-addr
jm start [name] Boot, provision on first boot, connect podman, mount the shares, start the port forwarder, the host resolver and the sleeper; wakes a suspended machine instead of booting it; idempotent
jm stop [name] Stop the sleeper, forwarder and resolver, ask the guest to power off, then the hypervisor and the network provider. A suspended machine is restored first; --force discards its saved state
jm suspend [name] Save a running machine to disk now and give its memory back; the first use wakes it. Refuses while containers, jails, engine clients or sessions are active unless --force, and always while a shared directory is in use
jm ssh [name] [-- cmd] Root shell, or a command, in the guest
jm podman / jpodman Run the host podman against the machine, whatever your default connection is; starts it if stopped, wakes it if suspended
jm docker / jdocker Run the host docker CLI (and compose) against the machine's engine, leaving your docker contexts alone; starts it if stopped, wakes it if suspended
jm env [name] Shell exports (CONTAINER_HOST, DOCKER_HOST) for podman and docker clients
jm ports [name] Published container ports, where they bind, and the error per mapping
jm list / jm inspect Machines and their computed state (running, stopped, suspended or broken), shares, publish address and what keeps an idle machine awake (--json on both)
jm set [name] Change --cpus, --memory, --ssh-port, --disk (grows only, live if running), --mount/--unmount/--no-mounts, --publish-addr, --arc (live if running), --idle-suspend (any state, applies at once)
jm console [name] Guest serial console log (-f to follow)
jm rm [name] Remove the machine, its directory and its podman connections
jm doctor Check qemu, HVF, EDK2 firmware, gvproxy, podman, ssh, the state root, share parity, resolver parity and every machine
jm version / jm image build Build identity (--json); and, for maintainers, sealing a prebaked guest image (about 800 MiB as a .zst)

Image providers

jm init --image <source>. The contract an image must satisfy is in docs/guest-contract.md and ADR 0003; JM_IMAGE_BASEURL points the prebaked source elsewhere, to test an unpublished image.

Source What you get Verification
prebaked (default), prebaked:<ver> Already-provisioned guest from this repo's guest-<ver> GitHub release; first boot is a boot, nothing more Mandatory .sha256 sidecar, checked by jm init
official, official:<release> Stock FreeBSD BASIC-CLOUDINIT-zfs cloud image, provisioned on first boot Mandatory CHECKSUM.SHA256
Path or http(s) URL to .raw (or .img), .raw.xz, .raw.zst Your own disk satisfying the contract; jm applies the seed, you own provisioning Sibling .sha256 if present, else image_trusted=false in jm inspect

Jails

bastille is installed and configured in the guest (ZFS, bastille0 loopback, NAT through pf); managing jails from the host (jm jail ...) is out of MVP scope (ADR 0006).

jm ssh -- bastille bootstrap 15.1-RELEASE
jm ssh -- bastille create demo 15.1-RELEASE 10.17.89.10
jm ssh -- bastille cmd demo pkg install -y curl
jm ssh -- bastille list all          # "-a" is deprecated in bastille 1.4.4
jm ssh -- bastille destroy -a -y demo

bastille destroy -f is not the non-interactive form — -f only forces unmounting datasets, so the command still prompts and then dies on the empty answer jm ssh gives it ([ERROR]: Invalid input. Please answer 'y' or 'n'). -y is "assume yes" and -a stops the jail first.

Docker Hub compatibility

Verified on this Mac against a running machine (podman 6.1.0, guest FreeBSD 15.1-RELEASE-p2 arm64, compat.linux.osrelease=5.15.0). Linux images need --os=linux for pull and build; FreeBSD images — which come from Docker Hub and GHCR — need no flag at all.

Image Flag Result
alpine, debian:trixie-slim, ubuntu:24.04, python:3-alpine, golang:alpine, hello-world --os=linux Works
postgres:17-alpine, caddy:alpine --os=linux Works
nginx:1.31-alpine --os=linux Works with one config line: accept_mutex on;
redis:alpine --os=linux Works with redis-server --ignore-warnings ARM64-COW-BUG
node:22-alpine --os=linux No — the one known-bad image, details
dougrabson/freebsd15-minimal, dougrabson/freebsd14-minimal, ghcr.io/freebsd/freebsd-runtime:15.1 and :14.3 none Work (native FreeBSD)

The full matrix, both workarounds, and the script that produced it (demo/hub-matrix.sh) are in docs/USAGE.md.

What works, and what does not yet

Capability State
Native FreeBSD OCI images Works — run and build, e.g. ghcr.io/freebsd/freebsd-runtime:15.1
Linux images Works through the Linuxulator, with --os=linux (podman) or the wrapper's default platform (jdocker)
Host directory sharing Works — host paths appear in the guest at the same absolute path over 9p; defaults are your home tree, /Volumes, /private/tmp and $TMPDIR's root. Slow (~70 MB/s), utimes is a no-op, an inotify watch cannot be created on a share at all, and guest ownership and modes live in host xattrs
Container DNS matching the host Works for external names — the host's own resolver answers for the guest, so VPN, split-horizon, /etc/hosts and .local names all match, and the Mac is host.docker.internal
Resolving another container by its name No. The guest's podman uses the CNI backend and netavark is not packaged for FreeBSD, so the podman network has dns_enabled: false and nc: bad address 'redis' is what you get. Use a Pod (localhost), network_mode: "service:<name>", or --add-host/extra_hosts (#5)
Autostart Works on demand: jpodman and jdocker start a stopped machine. There is deliberately no login agent — JM_AUTOSTART=0 opts out. A suspended machine is woken regardless: by the wrappers, jm start and jm ssh, and by a client of podman.sock or the SSH port while its sleeper runs
Idle memory Works — a machine idle for 30 minutes (--idle-suspend) with no containers or jails is suspended to disk and its memory goes back to macOS; the restore measured 4.06 s to SSH on a 2048 MiB guest. Published ports do not wake it, and after a Mac restart only jm commands do. See idle suspend
docker.io/nginx (Linux) Works with one config line: accept_mutex on; in the events block. Stock nginx registers its listening socket with EPOLLEXCLUSIVE when worker_processes > 1, which FreeBSD's linux_epoll rejects. A ready-made image is in demo/
Publishing ports (-p 8080:80) Works — reconciled onto the host by the forwarder, binding every interface by default (--publish-addr to change the default)
-p 127.0.0.1:8080:80, -p [::1]:…, ranges, /udp Works — a host address in the flag binds that address on the Mac and nothing else, as under Docker Desktop. -p localhost:… is rejected by podman itself
Docker CLI and compose Works via jdocker, or eval "$(jm env)" for a client you point yourself. Compose pulls FreeBSD variants under plain podman, so a Linux image needs jpodman pull --os=linux <image> first plus pull_policy: missing on the service
Jails Works — bastille bootstrap, create, cmd, pkg install in the guest
Several machines at once Works, but each needs its own SSH port — jm init --ssh-port 2223 dev, then JM_MACHINE=dev jpodman ps
UDP from a Linux container Works — binding, sending, receiving, DNS-over-UDP and publishing with -p 5354:53/udp, verified from the Mac's loopback and its LAN address. One idiom fails: busybox's nc -u -l reports Address family not supported by protocol. Use apk add netcat-openbsd, socat, or any real UDP server. Datagrams are capped at 8972 bytes by the gvproxy link, which does not fragment: it is MTU 9000 (the virtio-net jumbo frame) by default, and JM_MTU at jm start moves the ceiling — JM_MTU=1500 restores Docker's link size and its 1472-byte cap. See UDP in a Linux container
docker.io/node (Linux) Partly. node --version prints and exits 0; every other invocation hangs, node -e '' and node -p 1+1 included — they never reach your script. FreeBSD's linux_mremap cannot grow a mapping, which node's allocator retries forever. Workaround: node:22-bookworm-slim (glibc), verified working. See node
Routable VM IP No. gvproxy is NAT; vmnet/bridged is a later step
Intel Macs, Linux and Windows hosts No. Only darwin/arm64 has a backend; the Linux release binaries are build-only. Apple Virtualization.framework cannot boot FreeBSD/arm64, hence QEMU

Troubleshooting

Symptom Do
Anything at all jm doctor — checks every tool and machine and prints a fix per failure
start hangs or fails at a stage The error names the stage and the log to read; jm console shows the guest's serial console (-f to follow the boot)
Provisioning failed jm ssh -- cat /var/log/jm-provision.log; the marker /var/db/jm-provision-failed means the script aborted
Port not reachable jm ports lists each mapping with its error (host port busy, loopback bind, forwarder down)
-v fails source path does not exist, or mounts an empty directory The host path is outside the shared set (jm inspect), or you wrote /tmp/... instead of /private/tmp/.... A source that exists nowhere in the guest is an error; one that happens to exist there anyway binds the guest's own copy, which looks empty
A container cannot resolve another container by name (nc: bad address 'redis') Container DNS is off — the guest's podman runs the CNI backend and netavark is not packaged for FreeBSD (#5). Put the containers in a Pod and use localhost, or network_mode: "service:<name>", or --add-host/extra_hosts
A name resolves on the Mac but not in a container jm doctor, then resolver.log
nc -u -l in a Linux container says Address family not supported Only busybox's UDP listener is affected — apk add netcat-openbsd, or use socat. UDP itself works
UDP datagrams over 8972 bytes never arrive The gvproxy link does not fragment, so its MTU (9000 by default) is a hard ceiling; jm doctor states the limit per machine, and JM_MTU at jm start changes it (576–16384)
Stale state after a crash or reboot jm stop repairs "broken" (pid file without process); jm rm && jm init is always a clean slate
A machine never suspends, or a client did not wake it jm inspect lists Idle suspend: held awake by: …; sleeper.log and wake.log say what the sleeper and the wake did

Host-side logs, all under ~/.jailmachine/machines/<name>/:

File From
console.log guest serial console (jm console)
qemu.log QEMU's own stdout/stderr
gvproxy.log network provider
forwarder.log port-publishing loop
resolver.log host DNS resolver (name-resolution parity)
sleeper.log the helper that suspends an idle machine and holds its endpoints while it is suspended
wake.log wakes started by a connection to a suspended machine

More symptoms and fixes in docs/TROUBLESHOOTING.md.

Building from source

git clone https://github.com/gabrielbelli/jailmachine && cd jailmachine
make build && ./jm version
make test lint
JM_E2E=1 make e2e     # full init -> start -> podman run -> stop -> rm, needs qemu + podman

bin/jm is the original shell proof of concept, kept as legacy reference; the Go binary is the product.

Documentation

Licence

BSD-2-Clause. See LICENSE.

Known issues

Tracked, with measurements, in the issue tracker:

Issue Effect Workaround today
#2 UDP datagrams larger than the link MTU are dropped gvproxy does not fragment: the ceiling is 8972 bytes at the default MTU, where Linux delivers 65507. Native FreeBSD containers hit the same wall, so it is the link, not the Linuxulator JM_MTU (576–16384) moves the ceiling; jm doctor states it per machine
#3 Healthchecks never run A podman-on-FreeBSD gap: healthchecks are scheduled with systemd transient timers and there is no systemd, so --health-interval never fires and the status sits at starting with zero log entries. A bare-metal FreeBSD container host behaves the same. Restart policies do work — the issue's title still says otherwise and needs amending jm ssh -- podman healthcheck run <name>, or a cron entry in the guest
#4 an inotify watch cannot be created on a 9p share, which also runs at ~70 MB/s Not missing events — inotify_add_watch on a shared path fails outright with Bad file descriptor, so the watch is never created; it works on an engine-managed volume and on the container's own filesystem. Reads are coherent immediately. Metadata is the bigger cost: 1000 small files take 3.6 s on a share against 0.76 s on the guest's own disk Use polling watchers (CHOKIDAR_USEPOLLING=1, nodemon --legacy-watch, --watch.usePolling); keep build output in an engine-managed volume
#5 containers cannot resolve each other by name A podman-on-FreeBSD gap: the guest runs podman's CNI backend because netavark is not packaged for FreeBSD, and the CNI dnsname plugin is not packaged either, so podman network inspect podman reports dns_enabled: false and a sibling lookup fails with nc: bad address 'redis'. aardvark-dns is in the repo but is useless without netavark. This breaks the default shape of most compose files Put the containers in a Pod — they share a network namespace, so localhost works, which is what kube play gives you. (Podman also writes pod mates' container names into /etc/hosts, but under kube play that is <pod>-<container>, not the manifest's name: — so use localhost.) Otherwise network_mode: "service:<name>" in compose, or extra_hosts/--add-host with the container's bridge IP

About

docker-machine-style FreeBSD VM for jails and OCI containers (FreeBSD + Linux images) on macOS

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages