From 27da21965d2f6219036a339f54fc6a7dd475617e Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Mon, 24 Aug 2026 21:55:43 -0500 Subject: [PATCH 1/8] feat: add ceph dev env --- .gitignore | 3 + .local/ceph/README.md | 243 ++++++++++++++++++++++++++++++++++++++++++ .local/ceph/init.sh | 230 +++++++++++++++++++++++++++++++++++++++ .local/ceph/reset.sh | 80 ++++++++++++++ .local/ceph/start.sh | 47 ++++++++ .local/ceph/stop.sh | 40 +++++++ 6 files changed, 643 insertions(+) create mode 100644 .local/ceph/README.md create mode 100644 .local/ceph/init.sh create mode 100644 .local/ceph/reset.sh create mode 100644 .local/ceph/start.sh create mode 100644 .local/ceph/stop.sh diff --git a/.gitignore b/.gitignore index 0e257a6..0034635 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,9 @@ dev *.pdb *.yaml +# Local Ceph credentials and loopback OSD state. +.local/ceph/generated/ +.local/ceph/state/ # Generated by cargo mutants # Contains mutation testing data diff --git a/.local/ceph/README.md b/.local/ceph/README.md new file mode 100644 index 0000000..28066c8 --- /dev/null +++ b/.local/ceph/README.md @@ -0,0 +1,243 @@ +# Local Ceph with cephadm + +These scripts let you run a single host ceph cluster via cephadm (uses podman and quay.io/ceph/ceph internally). +It creates a 10gb sparse raw file that is then used as a `/dev/nbdN` block device on your host system. +You can then run odorobo on your host against this ceph cluster. + +## Prerequisites + +On Fedora: + +```bash +sudo dnf install -y cephadm ceph-common podman cloud-hypervisor qemu-img qemu-nbd kmod iproute socat +sudo modprobe rbd nbd +``` +## Usage + +There are 4 scripts for controlling the cluster. + +- `init.sh` creates a new cluster if needed, provisions the virtual OSD, and creates the Odorobo pool/client/image. +- `start.sh` starts an existing stopped cluster without provisioning or deleting data. +- `stop.sh` stops an existing cluster while preserving its data. +- `reset.sh` destructively removes the cluster and local virtual-disk state. + +Use `init.sh` once for a new checkout or after `reset.sh`. + +Then export the following environment variables so odorobo can connect to it. + +```bash +export CEPH_CONFIG="$PWD/.local/ceph/generated/ceph.conf" +export CEPH_ID=odorobo +export CEPH_KEYFILE="$PWD/.local/ceph/generated/client.odorobo.key" +export CEPH_CLUSTER=ceph +``` + + + + + + +## Reference + +### What `init.sh` does + +`init.sh` is the provisioning command. On a new machine, it: + +1. Bootstraps a single-host Ceph cluster with `cephadm` if `/etc/ceph/ceph.conf` does not exist. +2. Runs the Ceph MON, MGR, and OSD daemons in rootful Podman containers. +3. Creates or reattaches the sparse OSD image at `.local/ceph/state/osd.raw`. +4. Exposes that image to the host as a `/dev/nbdN` block device using `qemu-nbd`. +5. Creates an OSD on that virtual block device. +6. Creates the `odorobo-blockpool` RBD pool with single-node development settings. +7. Creates the restricted `client.odorobo` client. +8. Creates the `dev-disk` RBD image. +9. Writes the host-side Ceph configuration and client key under `.local/ceph/generated/`. + +`init.sh` does not reuse or wipe existing virtual-disk state. If a previous attempt leaves `.local/ceph/state/osd.raw` or `.local/ceph/state/osd.device`, it stops and directs you to `reset.sh`. This keeps initialization deterministic and destructive cleanup in the reset lifecycle. + +The OSD image is sparse, so its virtual size defaults to 10 GiB but it does not immediately consume 10 GiB of physical storage. + +The virtual disk and cluster are intentionally suitable only for local development. This setup does not provide production-level redundancy, quorum, or failure recovery. + +### Verify the cluster + +After `init.sh` or `start.sh`, check that the Ceph containers and daemons are running: + +```bash +sudo podman ps \ + --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' + +sudo ceph -s +sudo ceph osd tree +``` + +A one-node development cluster may report `HEALTH_WARN`. Warnings about a single monitor, low monitor disk space, or reduced redundancy are expected. The OSD should be `up` and `in`. + +Check the virtual OSD device: + +```bash +cat .local/ceph/state/osd.device +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS +``` + +The device recorded in `osd.device` should be the device containing the `ceph_bluestore` mapping. + +### Verify the Odorobo client and RBD image + +Export the generated credentials if you have not already done so: + +```bash +export CEPH_CONFIG="$PWD/.local/ceph/generated/ceph.conf" +export CEPH_ID=odorobo +export CEPH_KEYFILE="$PWD/.local/ceph/generated/client.odorobo.key" +export CEPH_CLUSTER=ceph +``` + +List the pool contents: + +```bash +sudo -E rbd \ + --conf="$CEPH_CONFIG" \ + --id="$CEPH_ID" \ + --keyfile="$CEPH_KEYFILE" \ + ls --pool odorobo-blockpool +``` + +Expected output: + +```text +dev-disk +``` + +Inspect the client permissions: + +```bash +sudo ceph auth get client.odorobo +``` + +The client should have read-only monitor access and read/write access limited to `odorobo-blockpool`. + +### Test host-side RBD mapping + +Odorobo currently invokes the host's `rbd` command and expects the Linux kernel RBD module to create a local block device. The host therefore needs `ceph-common`, the `rbd` kernel module, and the udev rule described in `docs/storage.md`. + +If the udev rule is not already installed by the distribution's Ceph packages: + +```bash +sudo tee /etc/udev/rules.d/50-rbd.rules >/dev/null <<'EOF' +KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", PROGRAM=="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c" +KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="partition", PROGRAM=="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c-part%n" +EOF + +sudo udevadm control --reload-rules +sudo udevadm trigger +``` + +Map the test image: + +```bash +sudo -E rbd device map odorobo-blockpool/dev-disk \ + --conf="$CEPH_CONFIG" \ + --id="$CEPH_ID" \ + --keyfile="$CEPH_KEYFILE" +``` + +Verify the stable device path: + +```bash +ls -l /dev/rbd/odorobo-blockpool/dev-disk +sudo rbd device list +``` + +When finished, unmap the image before stopping or resetting Ceph: + +```bash +sudo -E rbd device unmap odorobo-blockpool/dev-disk \ + --conf="$CEPH_CONFIG" \ + --id="$CEPH_ID" \ + --keyfile="$CEPH_KEYFILE" +``` + +The helper scripts also unmap RBD devices automatically: +Stop the cluster while preserving its data: + +```bash +bash .local/ceph/stop.sh +``` + +This unmaps RBD devices and stops the cephadm systemd target for the whole cluster without deleting its data. + +### Recover from an interrupted initialization + +The initializer intentionally does not recover partial virtual-OSD state. If it fails or is interrupted, run the destructive reset before retrying: + +```bash +bash .local/ceph/reset.sh +``` + +If you need to inspect a failed attempt before resetting: + +```bash +ps auxww | grep '[q]emu-nbd' +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS +``` + +The initializer does not recover partial virtual-OSD state or reuse an existing backing file. If it fails or is interrupted, run `reset.sh` before retrying. If you need to inspect a failed run first, check for leftover `qemu-nbd` processes: + +```bash +ps auxww | grep '[q]emu-nbd' +``` + +Only disconnect a duplicate process whose command references this repository's file: + +```text +.local/ceph/state/osd.raw +``` + +Do not disconnect the device that `lsblk` shows as containing `ceph_bluestore`; that is the active OSD device. + +Disconnect a confirmed duplicate with: + +```bash +sudo qemu-nbd --disconnect /dev/nbdN +``` + +Then rerun: + +```bash +bash .local/ceph/init.sh +``` + +### Configuration overrides + +The initializer accepts these environment variables: + +```bash +# Defaults to the host's routable IPv4 address. Override if needed. +# CEPH_MON_IP=192.168.40.123 +CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3 +CEPH_POOL=odorobo-blockpool +CEPH_CLIENT=odorobo +CEPH_IMAGE_NAME=dev-disk +CEPH_IMAGE_SIZE=1G +CEPH_OSD_SIZE=10G +``` + +To use an intentionally disposable real block device instead of the virtual NBD disk: + +```bash +CEPH_OSD_DEVICE=/dev/sdb bash .local/ceph/init.sh +``` + +Do not use a device containing data. Ceph will format it. + +The Ceph bootstrap state is system-wide under `/etc/ceph` and `/var/lib/ceph`. Repository-local state is stored as follows: + +```text +.local/ceph/state/osd.raw sparse virtual OSD disk +.local/ceph/state/osd.device attached NBD device name +.local/ceph/generated/ceph.conf generated client configuration +.local/ceph/generated/client.odorobo.key generated client credential +``` + +The generated files and virtual disk state are ignored by git. diff --git a/.local/ceph/init.sh b/.local/ceph/init.sh new file mode 100644 index 0000000..29c2d4e --- /dev/null +++ b/.local/ceph/init.sh @@ -0,0 +1,230 @@ +#!/usr/bin/env bash +# Run with: bash .local/ceph/init.sh +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +GENERATED_DIR="$ROOT_DIR/generated" +CEPH_IMAGE="${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3}" +CEPH_OSD_DEVICE="${CEPH_OSD_DEVICE:-}" +CEPH_OSD_SIZE="${CEPH_OSD_SIZE:-10G}" +OSD_STATE_DIR="$ROOT_DIR/state" +OSD_IMAGE="$OSD_STATE_DIR/osd.raw" +if [[ -n "${CEPH_MON_IP:-}" ]]; then + MON_IP="$CEPH_MON_IP" +else + MON_IP=$(ip -4 route get 1.1.1.1 2>/dev/null \ + | awk '{for (i=1; i<=NF; i++) if ($i == "src") {print $(i+1); exit}}') + MON_IP="${MON_IP:-127.0.0.1}" +fi +POOL_NAME="${CEPH_POOL:-odorobo-blockpool}" +CLIENT_NAME="${CEPH_CLIENT:-odorobo}" +IMAGE_NAME="${CEPH_IMAGE_NAME:-dev-disk}" +IMAGE_SIZE="${CEPH_IMAGE_SIZE:-1G}" + +if [[ $EUID -eq 0 ]]; then + SUDO=() +else + SUDO=(sudo) +fi + +cephadm() { "${SUDO[@]}" cephadm "$@"; } +# Run cluster-control commands inside the pinned Ceph container so the CLI +# version and authentication behavior match the daemons. +ceph() { "${SUDO[@]}" cephadm shell -- ceph "$@"; } +rbd() { "${SUDO[@]}" rbd "$@"; } +qemu_img() { "${SUDO[@]}" qemu-img "$@"; } +qemu_nbd() { "${SUDO[@]}" qemu-nbd "$@"; } + + +mkdir -p "$GENERATED_DIR" "$OSD_STATE_DIR" + +if ! command -v podman >/dev/null 2>&1; then + echo "Podman is required because cephadm runs the Ceph daemons in Podman containers." >&2 + echo "For Fedora: sudo dnf install -y podman" >&2 + exit 1 +fi + +if ! command -v cephadm >/dev/null 2>&1; then + echo "cephadm is required. Install it with your distribution package manager." >&2 + echo "For Fedora: sudo dnf install -y cephadm ceph-common" >&2 + exit 1 +fi + +if ! command -v ceph >/dev/null 2>&1; then + echo "ceph CLI is required. Install ceph-common or use cephadm shell." >&2 + echo "For Fedora: sudo dnf install -y ceph-common" >&2 + exit 1 +fi + +if [[ -z "$CEPH_OSD_DEVICE" ]]; then + for command in qemu-img qemu-nbd; do + if ! command -v "$command" >/dev/null 2>&1; then + echo "$command is required for the virtual OSD device." >&2 + echo "For Fedora: sudo dnf install -y qemu-img" >&2 + exit 1 + fi + done +fi + +if [[ ! -f /etc/ceph/ceph.conf ]]; then + echo "Bootstrapping single-host Ceph with $CEPH_IMAGE..." + BOOTSTRAP_ARGS=( + --mon-ip "$MON_IP" + --single-host-defaults + --output-dir /etc/ceph + --skip-monitoring-stack + --allow-fqdn-hostname + ) + if [[ "$MON_IP" == "127.0.0.1" || "$MON_IP" == "::1" ]]; then + BOOTSTRAP_ARGS+=(--skip-mon-network) + fi + cephadm --image "$CEPH_IMAGE" bootstrap "${BOOTSTRAP_ARGS[@]}" +fi + +if ! ceph -s >/dev/null 2>&1; then + echo "Ceph did not become available after bootstrap." >&2 + cephadm shell -- ceph -s >&2 || true + exit 1 +fi + +OSD_EXISTS=0 +if [[ -n "$(ceph osd ls 2>/dev/null)" ]]; then + OSD_EXISTS=1 +fi +VIRTUAL_OSD=0 +if [[ "$OSD_EXISTS" -eq 0 && -z "$CEPH_OSD_DEVICE" ]]; then + VIRTUAL_OSD=1 + "${SUDO[@]}" modprobe nbd max_part=8 + + if [[ -f "$OSD_IMAGE" || -f "$OSD_STATE_DIR/osd.device" ]]; then + echo "Local virtual OSD state already exists." >&2 + echo "Run reset.sh before reinitializing:" >&2 + echo " bash .local/ceph/reset.sh" >&2 + exit 1 + fi + + echo "Creating a ${CEPH_OSD_SIZE} virtual OSD disk at $OSD_IMAGE..." + qemu_img create -f raw "$OSD_IMAGE" "$CEPH_OSD_SIZE" + + OSD_DEVICE="" + echo "Attaching virtual OSD disk..." + for candidate in /dev/nbd{0..15}; do + if [[ -b "$candidate" ]] && (( $("${SUDO[@]}" blockdev --getsize64 "$candidate" 2>/dev/null || echo 0) > 0 )); then + continue + fi + attach_log="$OSD_STATE_DIR/qemu-nbd-${candidate##*/}.log" + echo "Trying qemu-nbd on $candidate..." + if ! "${SUDO[@]}" qemu-nbd --persistent --fork --verbose \ + --connect="$candidate" --format=raw "$OSD_IMAGE" \ + >"$attach_log" 2>&1; then + cat "$attach_log" >&2 || true + continue + fi + candidate_size=0 + for _ in {1..60}; do + "${SUDO[@]}" udevadm settle || true + candidate_size=$("${SUDO[@]}" blockdev --getsize64 "$candidate" 2>/dev/null || echo 0) + if (( candidate_size >= 5368709120 )); then + break + fi + sleep 1 + done + if (( candidate_size >= 5368709120 )); then + OSD_DEVICE="$candidate" + break + fi + echo "qemu-nbd did not expose a usable block device on $candidate after 60 seconds (size: ${candidate_size} bytes)." >&2 + cat "$attach_log" >&2 || true + "${SUDO[@]}" qemu-nbd --disconnect "$candidate" >/dev/null 2>&1 || true + done + + if [[ -z "$OSD_DEVICE" ]]; then + echo "Could not attach a virtual OSD disk of at least 5 GiB." >&2 + echo "qemu-img info:" + qemu_img info "$OSD_IMAGE" >&2 || true + echo "qemu-nbd attach logs:" + for attach_log in "$OSD_STATE_DIR"/qemu-nbd-*.log; do + [[ -f "$attach_log" ]] || continue + echo "--- $attach_log" >&2 + cat "$attach_log" >&2 + done + exit 1 + fi + + echo "$OSD_DEVICE" > "$OSD_STATE_DIR/osd.device" + CEPH_OSD_DEVICE="$OSD_DEVICE" +fi + +# `ceph osd ls` is deliberately used instead of matching formatted JSON output. +if [[ "$OSD_EXISTS" -eq 0 ]]; then + if [[ ! -b "$CEPH_OSD_DEVICE" ]]; then + echo "CEPH_OSD_DEVICE is not a block device: $CEPH_OSD_DEVICE" >&2 + exit 1 + fi + + if [[ "$VIRTUAL_OSD" -eq 1 ]]; then + echo "Zapping virtual OSD device $CEPH_OSD_DEVICE..." + "${SUDO[@]}" wipefs --all --force "$CEPH_OSD_DEVICE" >/dev/null 2>&1 || true + "${SUDO[@]}" pvremove --force --force --yes "$CEPH_OSD_DEVICE" >/dev/null 2>&1 || true + "${SUDO[@]}" udevadm settle || true + fi + + echo "Creating an OSD on $CEPH_OSD_DEVICE..." + if ! timeout --foreground 120 "${SUDO[@]}" cephadm shell -- ceph orch daemon add osd "$(hostname -s):$CEPH_OSD_DEVICE"; then + echo "cephadm could not create an OSD on $CEPH_OSD_DEVICE." >&2 + ceph orch device ls --wide >&2 || true + ceph orch ps --daemon-type osd >&2 || true + exit 1 + fi +fi + +# Wait for the OSD to register. The cluster may remain HEALTH_WARN due to +# being a one-node cluster; that is expected. +for _ in {1..60}; do + if ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then + break + fi + sleep 2 +done + +if ! ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then + echo "The OSD did not become available." >&2 + ceph orch ps --daemon-type osd >&2 || true + exit 1 +fi + +if ! ceph osd pool ls --format=json | grep -qF "\"$POOL_NAME\""; then + ceph osd pool create "$POOL_NAME" 8 +fi +# A one-OSD development cluster needs a replicated pool size of one. Ceph +# requires this explicit opt-in instead of allowing it by default. +ceph config set mon mon_allow_pool_size_one true +ceph config set osd osd_pool_default_size 1 +ceph config set osd osd_pool_default_min_size 1 +ceph osd pool set "$POOL_NAME" size 1 --yes-i-really-mean-it +ceph osd pool set "$POOL_NAME" min_size 1 +rbd pool init "$POOL_NAME" + +if ! ceph auth get "client.$CLIENT_NAME" >/dev/null 2>&1; then + ceph auth get-or-create "client.$CLIENT_NAME" \ + mon "allow r" \ + osd "allow rwx pool=$POOL_NAME" >/dev/null +fi + +ceph config generate-minimal-conf > "$GENERATED_DIR/ceph.conf" +ceph auth get-key "client.$CLIENT_NAME" > "$GENERATED_DIR/client.$CLIENT_NAME.key" +chmod 600 "$GENERATED_DIR/client.$CLIENT_NAME.key" + +if ! rbd info "$POOL_NAME/$IMAGE_NAME" >/dev/null 2>&1; then + rbd create "$POOL_NAME/$IMAGE_NAME" --size "$IMAGE_SIZE" +fi + +echo "Ceph is ready." +echo "Image: $POOL_NAME/$IMAGE_NAME" +echo "Config: $GENERATED_DIR/ceph.conf" +echo "Key: $GENERATED_DIR/client.$CLIENT_NAME.key" +echo +echo "Export for host-side Odorobo:" +echo " export CEPH_CONFIG=$GENERATED_DIR/ceph.conf" +echo " export CEPH_ID=$CLIENT_NAME" +echo " export CEPH_KEYFILE=$GENERATED_DIR/client.$CLIENT_NAME.key" diff --git a/.local/ceph/reset.sh b/.local/ceph/reset.sh new file mode 100644 index 0000000..895388c --- /dev/null +++ b/.local/ceph/reset.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Destructively remove the local Ceph cluster and virtual OSD state. +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +GENERATED_DIR="$ROOT_DIR/generated" +STATE_DIR="$ROOT_DIR/state" +OSD_IMAGE="$STATE_DIR/osd.raw" + +if [[ $EUID -eq 0 ]]; then + SUDO=() +else + SUDO=(sudo) +fi + +cephadm() { "${SUDO[@]}" cephadm "$@"; } +rbd() { "${SUDO[@]}" rbd "$@"; } +qemu_nbd() { "${SUDO[@]}" qemu-nbd "$@"; } + +nbd_attached() { + local candidate="$1" + local proc cmdline + for proc in /proc/[0-9]*; do + [[ -r "$proc/cmdline" ]] || continue + cmdline=$(tr '\0' ' ' < "$proc/cmdline" 2>/dev/null || true) + if [[ "$cmdline" == *qemu-nbd* \ + && "$cmdline" == *"--connect=$candidate"* \ + && "$cmdline" == *"$OSD_IMAGE"* ]]; then + return 0 + fi + done + return 1 +} + +if ! command -v cephadm >/dev/null 2>&1; then + echo "cephadm is required. Install cephadm first." >&2 + exit 1 +fi + +FSID=$(cephadm ls 2>/dev/null | python3 -c \ + 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') + +# Unmap any host RBD devices before removing the cluster. +rbd device unmap odorobo-blockpool/dev-disk >/dev/null 2>&1 || true +while read -r _ _ _ _ _ device; do + [[ -n "${device:-}" ]] || continue + rbd device unmap "$device" >/dev/null 2>&1 || true +done < <(rbd device list 2>/dev/null | tail -n +2) + +if [[ -n "$FSID" ]]; then + echo "Removing Ceph cluster $FSID..." + cephadm rm-cluster --force --zap-osds --fsid "$FSID" +else + echo "No active Ceph cluster found." +fi + +# Disconnect only NBD devices whose qemu-nbd process references this project's +# backing file. Never disconnect unrelated NBD devices. +while read -r device; do + [[ -n "$device" ]] || continue + if nbd_attached "$device"; then + echo "Disconnecting $device..." + qemu_nbd --disconnect "$device" >/dev/null 2>&1 || true + fi +done < <(ps -eo args= | sed -n 's/.*--connect=\(\/dev\/nbd[0-9][0-9]*\).*/\1/p' | sort -u) + +# Remove the virtual OSD image only after the cluster and NBD device are gone. +if [[ -f "$OSD_IMAGE" ]]; then + echo "Clearing virtual OSD image..." + OSD_BYTES=$("${SUDO[@]}" stat -c '%s' "$OSD_IMAGE") + "${SUDO[@]}" dd if=/dev/zero of="$OSD_IMAGE" bs=1M count=16 conv=notrunc status=none || true + if (( OSD_BYTES >= 33554432 )); then + "${SUDO[@]}" dd if=/dev/zero of="$OSD_IMAGE" bs=1M seek=$(( (OSD_BYTES / 1048576) - 16 )) count=16 conv=notrunc status=none || true + fi +fi + +rm -rf "$GENERATED_DIR" "$STATE_DIR" +"${SUDO[@]}" rm -rf /etc/ceph/* + +echo "Local Ceph cluster, credentials, and virtual OSD state removed." diff --git a/.local/ceph/start.sh b/.local/ceph/start.sh new file mode 100644 index 0000000..0f5f6fc --- /dev/null +++ b/.local/ceph/start.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Start an existing local Ceph cluster without provisioning or deleting data. +set -euo pipefail + +if [[ $EUID -eq 0 ]]; then + SUDO=() +else + SUDO=(sudo) +fi + +cephadm() { "${SUDO[@]}" cephadm "$@"; } +ceph() { "${SUDO[@]}" cephadm shell -- ceph "$@"; } + +if ! command -v cephadm >/dev/null 2>&1; then + echo "cephadm is required. Install cephadm first." >&2 + exit 1 +fi + +if [[ ! -f /etc/ceph/ceph.conf ]]; then + echo "No local Ceph cluster exists. Run init.sh first:" >&2 + echo " bash .local/ceph/init.sh" >&2 + exit 1 +fi + +FSID=$(cephadm ls 2>/dev/null | python3 -c \ + 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') +if [[ -z "$FSID" ]]; then + echo "Could not determine the local Ceph cluster FSID." >&2 + exit 1 +fi + +TARGET="ceph-$FSID.target" +echo "Starting Ceph cluster $FSID..." +"${SUDO[@]}" systemctl start "$TARGET" + +for _ in {1..30}; do + if ceph -s >/dev/null 2>&1; then + echo "Local Ceph cluster is available." + exit 0 + fi + sleep 2 +done + +echo "Ceph did not become available after starting $TARGET." >&2 +"${SUDO[@]}" systemctl status "$TARGET" --no-pager >&2 || true +ceph -s >&2 || true +exit 1 diff --git a/.local/ceph/stop.sh b/.local/ceph/stop.sh new file mode 100644 index 0000000..7396414 --- /dev/null +++ b/.local/ceph/stop.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Stop the local Ceph cluster without deleting its data. +set -euo pipefail + +if [[ $EUID -eq 0 ]]; then + SUDO=() +else + SUDO=(sudo) +fi + +cephadm() { "${SUDO[@]}" cephadm "$@"; } +rbd() { "${SUDO[@]}" rbd "$@"; } + +# Unmap the known test image if it is mapped. Ignore the expected error when it +# is already unmapped. +rbd device unmap odorobo-blockpool/dev-disk >/dev/null 2>&1 || true + +# Unmap any remaining RBD devices. This matters if a test used another image. +while read -r _ _ _ _ _ device; do + [[ -n "${device:-}" ]] || continue + rbd device unmap "$device" >/dev/null 2>&1 || true +done < <(rbd device list 2>/dev/null | tail -n +2) + +if ! command -v cephadm >/dev/null 2>&1; then + echo "cephadm is required. Install cephadm first." >&2 + exit 1 +fi + +FSID=$(cephadm ls 2>/dev/null | python3 -c \ + 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') +if [[ -z "$FSID" ]]; then + echo "No local Ceph cluster found; nothing to stop." + exit 0 +fi + +TARGET="ceph-$FSID.target" +echo "Stopping Ceph cluster $FSID..." +"${SUDO[@]}" systemctl stop "$TARGET" + +echo "Local Ceph daemons stopped. Data and virtual OSD state were preserved." From 18baff2b2eef4dd7694151aa081cdaa73c44ab6f Mon Sep 17 00:00:00 2001 From: Cypress Reed Date: Mon, 31 Aug 2026 13:58:29 -0600 Subject: [PATCH 2/8] create dev cluster with containers --- .gitignore | 10 +- .local/ceph/README.md | 243 ------------------------------- .local/ceph/init.sh | 230 ----------------------------- .local/ceph/reset.sh | 80 ---------- .local/ceph/start.sh | 47 ------ .local/ceph/stop.sh | 40 ----- .local/dev/README.md | 118 +++++++++++++++ .local/dev/ceph/Containerfile | 9 ++ .local/dev/ceph/entrypoint.sh | 71 +++++++++ .local/dev/compose.yml | 61 ++++++++ .local/dev/init.sh | 31 ++++ .local/dev/odorobo/Containerfile | 18 +++ .local/dev/reset.sh | 19 +++ .local/dev/start.sh | 16 ++ .local/dev/stop.sh | 19 +++ 15 files changed, 370 insertions(+), 642 deletions(-) delete mode 100644 .local/ceph/README.md delete mode 100644 .local/ceph/init.sh delete mode 100644 .local/ceph/reset.sh delete mode 100644 .local/ceph/start.sh delete mode 100644 .local/ceph/stop.sh create mode 100644 .local/dev/README.md create mode 100644 .local/dev/ceph/Containerfile create mode 100644 .local/dev/ceph/entrypoint.sh create mode 100644 .local/dev/compose.yml create mode 100644 .local/dev/init.sh create mode 100644 .local/dev/odorobo/Containerfile create mode 100644 .local/dev/reset.sh create mode 100644 .local/dev/start.sh create mode 100644 .local/dev/stop.sh diff --git a/.gitignore b/.gitignore index 0034635..544c488 100644 --- a/.gitignore +++ b/.gitignore @@ -12,9 +12,15 @@ dev *.pdb *.yaml +# The local development container stack is checked in, despite the broad `dev` +# scratch-directory rule above. +!.local/ +!.local/dev/ +!.local/dev/** + # Local Ceph credentials and loopback OSD state. -.local/ceph/generated/ -.local/ceph/state/ +.local/dev/ceph/generated/ +.local/dev/ceph/state/ # Generated by cargo mutants # Contains mutation testing data diff --git a/.local/ceph/README.md b/.local/ceph/README.md deleted file mode 100644 index 28066c8..0000000 --- a/.local/ceph/README.md +++ /dev/null @@ -1,243 +0,0 @@ -# Local Ceph with cephadm - -These scripts let you run a single host ceph cluster via cephadm (uses podman and quay.io/ceph/ceph internally). -It creates a 10gb sparse raw file that is then used as a `/dev/nbdN` block device on your host system. -You can then run odorobo on your host against this ceph cluster. - -## Prerequisites - -On Fedora: - -```bash -sudo dnf install -y cephadm ceph-common podman cloud-hypervisor qemu-img qemu-nbd kmod iproute socat -sudo modprobe rbd nbd -``` -## Usage - -There are 4 scripts for controlling the cluster. - -- `init.sh` creates a new cluster if needed, provisions the virtual OSD, and creates the Odorobo pool/client/image. -- `start.sh` starts an existing stopped cluster without provisioning or deleting data. -- `stop.sh` stops an existing cluster while preserving its data. -- `reset.sh` destructively removes the cluster and local virtual-disk state. - -Use `init.sh` once for a new checkout or after `reset.sh`. - -Then export the following environment variables so odorobo can connect to it. - -```bash -export CEPH_CONFIG="$PWD/.local/ceph/generated/ceph.conf" -export CEPH_ID=odorobo -export CEPH_KEYFILE="$PWD/.local/ceph/generated/client.odorobo.key" -export CEPH_CLUSTER=ceph -``` - - - - - - -## Reference - -### What `init.sh` does - -`init.sh` is the provisioning command. On a new machine, it: - -1. Bootstraps a single-host Ceph cluster with `cephadm` if `/etc/ceph/ceph.conf` does not exist. -2. Runs the Ceph MON, MGR, and OSD daemons in rootful Podman containers. -3. Creates or reattaches the sparse OSD image at `.local/ceph/state/osd.raw`. -4. Exposes that image to the host as a `/dev/nbdN` block device using `qemu-nbd`. -5. Creates an OSD on that virtual block device. -6. Creates the `odorobo-blockpool` RBD pool with single-node development settings. -7. Creates the restricted `client.odorobo` client. -8. Creates the `dev-disk` RBD image. -9. Writes the host-side Ceph configuration and client key under `.local/ceph/generated/`. - -`init.sh` does not reuse or wipe existing virtual-disk state. If a previous attempt leaves `.local/ceph/state/osd.raw` or `.local/ceph/state/osd.device`, it stops and directs you to `reset.sh`. This keeps initialization deterministic and destructive cleanup in the reset lifecycle. - -The OSD image is sparse, so its virtual size defaults to 10 GiB but it does not immediately consume 10 GiB of physical storage. - -The virtual disk and cluster are intentionally suitable only for local development. This setup does not provide production-level redundancy, quorum, or failure recovery. - -### Verify the cluster - -After `init.sh` or `start.sh`, check that the Ceph containers and daemons are running: - -```bash -sudo podman ps \ - --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' - -sudo ceph -s -sudo ceph osd tree -``` - -A one-node development cluster may report `HEALTH_WARN`. Warnings about a single monitor, low monitor disk space, or reduced redundancy are expected. The OSD should be `up` and `in`. - -Check the virtual OSD device: - -```bash -cat .local/ceph/state/osd.device -lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS -``` - -The device recorded in `osd.device` should be the device containing the `ceph_bluestore` mapping. - -### Verify the Odorobo client and RBD image - -Export the generated credentials if you have not already done so: - -```bash -export CEPH_CONFIG="$PWD/.local/ceph/generated/ceph.conf" -export CEPH_ID=odorobo -export CEPH_KEYFILE="$PWD/.local/ceph/generated/client.odorobo.key" -export CEPH_CLUSTER=ceph -``` - -List the pool contents: - -```bash -sudo -E rbd \ - --conf="$CEPH_CONFIG" \ - --id="$CEPH_ID" \ - --keyfile="$CEPH_KEYFILE" \ - ls --pool odorobo-blockpool -``` - -Expected output: - -```text -dev-disk -``` - -Inspect the client permissions: - -```bash -sudo ceph auth get client.odorobo -``` - -The client should have read-only monitor access and read/write access limited to `odorobo-blockpool`. - -### Test host-side RBD mapping - -Odorobo currently invokes the host's `rbd` command and expects the Linux kernel RBD module to create a local block device. The host therefore needs `ceph-common`, the `rbd` kernel module, and the udev rule described in `docs/storage.md`. - -If the udev rule is not already installed by the distribution's Ceph packages: - -```bash -sudo tee /etc/udev/rules.d/50-rbd.rules >/dev/null <<'EOF' -KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", PROGRAM=="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c" -KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="partition", PROGRAM=="/usr/bin/ceph-rbdnamer %k", SYMLINK+="rbd/%c-part%n" -EOF - -sudo udevadm control --reload-rules -sudo udevadm trigger -``` - -Map the test image: - -```bash -sudo -E rbd device map odorobo-blockpool/dev-disk \ - --conf="$CEPH_CONFIG" \ - --id="$CEPH_ID" \ - --keyfile="$CEPH_KEYFILE" -``` - -Verify the stable device path: - -```bash -ls -l /dev/rbd/odorobo-blockpool/dev-disk -sudo rbd device list -``` - -When finished, unmap the image before stopping or resetting Ceph: - -```bash -sudo -E rbd device unmap odorobo-blockpool/dev-disk \ - --conf="$CEPH_CONFIG" \ - --id="$CEPH_ID" \ - --keyfile="$CEPH_KEYFILE" -``` - -The helper scripts also unmap RBD devices automatically: -Stop the cluster while preserving its data: - -```bash -bash .local/ceph/stop.sh -``` - -This unmaps RBD devices and stops the cephadm systemd target for the whole cluster without deleting its data. - -### Recover from an interrupted initialization - -The initializer intentionally does not recover partial virtual-OSD state. If it fails or is interrupted, run the destructive reset before retrying: - -```bash -bash .local/ceph/reset.sh -``` - -If you need to inspect a failed attempt before resetting: - -```bash -ps auxww | grep '[q]emu-nbd' -lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS -``` - -The initializer does not recover partial virtual-OSD state or reuse an existing backing file. If it fails or is interrupted, run `reset.sh` before retrying. If you need to inspect a failed run first, check for leftover `qemu-nbd` processes: - -```bash -ps auxww | grep '[q]emu-nbd' -``` - -Only disconnect a duplicate process whose command references this repository's file: - -```text -.local/ceph/state/osd.raw -``` - -Do not disconnect the device that `lsblk` shows as containing `ceph_bluestore`; that is the active OSD device. - -Disconnect a confirmed duplicate with: - -```bash -sudo qemu-nbd --disconnect /dev/nbdN -``` - -Then rerun: - -```bash -bash .local/ceph/init.sh -``` - -### Configuration overrides - -The initializer accepts these environment variables: - -```bash -# Defaults to the host's routable IPv4 address. Override if needed. -# CEPH_MON_IP=192.168.40.123 -CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3 -CEPH_POOL=odorobo-blockpool -CEPH_CLIENT=odorobo -CEPH_IMAGE_NAME=dev-disk -CEPH_IMAGE_SIZE=1G -CEPH_OSD_SIZE=10G -``` - -To use an intentionally disposable real block device instead of the virtual NBD disk: - -```bash -CEPH_OSD_DEVICE=/dev/sdb bash .local/ceph/init.sh -``` - -Do not use a device containing data. Ceph will format it. - -The Ceph bootstrap state is system-wide under `/etc/ceph` and `/var/lib/ceph`. Repository-local state is stored as follows: - -```text -.local/ceph/state/osd.raw sparse virtual OSD disk -.local/ceph/state/osd.device attached NBD device name -.local/ceph/generated/ceph.conf generated client configuration -.local/ceph/generated/client.odorobo.key generated client credential -``` - -The generated files and virtual disk state are ignored by git. diff --git a/.local/ceph/init.sh b/.local/ceph/init.sh deleted file mode 100644 index 29c2d4e..0000000 --- a/.local/ceph/init.sh +++ /dev/null @@ -1,230 +0,0 @@ -#!/usr/bin/env bash -# Run with: bash .local/ceph/init.sh -set -euo pipefail - -ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) -GENERATED_DIR="$ROOT_DIR/generated" -CEPH_IMAGE="${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3}" -CEPH_OSD_DEVICE="${CEPH_OSD_DEVICE:-}" -CEPH_OSD_SIZE="${CEPH_OSD_SIZE:-10G}" -OSD_STATE_DIR="$ROOT_DIR/state" -OSD_IMAGE="$OSD_STATE_DIR/osd.raw" -if [[ -n "${CEPH_MON_IP:-}" ]]; then - MON_IP="$CEPH_MON_IP" -else - MON_IP=$(ip -4 route get 1.1.1.1 2>/dev/null \ - | awk '{for (i=1; i<=NF; i++) if ($i == "src") {print $(i+1); exit}}') - MON_IP="${MON_IP:-127.0.0.1}" -fi -POOL_NAME="${CEPH_POOL:-odorobo-blockpool}" -CLIENT_NAME="${CEPH_CLIENT:-odorobo}" -IMAGE_NAME="${CEPH_IMAGE_NAME:-dev-disk}" -IMAGE_SIZE="${CEPH_IMAGE_SIZE:-1G}" - -if [[ $EUID -eq 0 ]]; then - SUDO=() -else - SUDO=(sudo) -fi - -cephadm() { "${SUDO[@]}" cephadm "$@"; } -# Run cluster-control commands inside the pinned Ceph container so the CLI -# version and authentication behavior match the daemons. -ceph() { "${SUDO[@]}" cephadm shell -- ceph "$@"; } -rbd() { "${SUDO[@]}" rbd "$@"; } -qemu_img() { "${SUDO[@]}" qemu-img "$@"; } -qemu_nbd() { "${SUDO[@]}" qemu-nbd "$@"; } - - -mkdir -p "$GENERATED_DIR" "$OSD_STATE_DIR" - -if ! command -v podman >/dev/null 2>&1; then - echo "Podman is required because cephadm runs the Ceph daemons in Podman containers." >&2 - echo "For Fedora: sudo dnf install -y podman" >&2 - exit 1 -fi - -if ! command -v cephadm >/dev/null 2>&1; then - echo "cephadm is required. Install it with your distribution package manager." >&2 - echo "For Fedora: sudo dnf install -y cephadm ceph-common" >&2 - exit 1 -fi - -if ! command -v ceph >/dev/null 2>&1; then - echo "ceph CLI is required. Install ceph-common or use cephadm shell." >&2 - echo "For Fedora: sudo dnf install -y ceph-common" >&2 - exit 1 -fi - -if [[ -z "$CEPH_OSD_DEVICE" ]]; then - for command in qemu-img qemu-nbd; do - if ! command -v "$command" >/dev/null 2>&1; then - echo "$command is required for the virtual OSD device." >&2 - echo "For Fedora: sudo dnf install -y qemu-img" >&2 - exit 1 - fi - done -fi - -if [[ ! -f /etc/ceph/ceph.conf ]]; then - echo "Bootstrapping single-host Ceph with $CEPH_IMAGE..." - BOOTSTRAP_ARGS=( - --mon-ip "$MON_IP" - --single-host-defaults - --output-dir /etc/ceph - --skip-monitoring-stack - --allow-fqdn-hostname - ) - if [[ "$MON_IP" == "127.0.0.1" || "$MON_IP" == "::1" ]]; then - BOOTSTRAP_ARGS+=(--skip-mon-network) - fi - cephadm --image "$CEPH_IMAGE" bootstrap "${BOOTSTRAP_ARGS[@]}" -fi - -if ! ceph -s >/dev/null 2>&1; then - echo "Ceph did not become available after bootstrap." >&2 - cephadm shell -- ceph -s >&2 || true - exit 1 -fi - -OSD_EXISTS=0 -if [[ -n "$(ceph osd ls 2>/dev/null)" ]]; then - OSD_EXISTS=1 -fi -VIRTUAL_OSD=0 -if [[ "$OSD_EXISTS" -eq 0 && -z "$CEPH_OSD_DEVICE" ]]; then - VIRTUAL_OSD=1 - "${SUDO[@]}" modprobe nbd max_part=8 - - if [[ -f "$OSD_IMAGE" || -f "$OSD_STATE_DIR/osd.device" ]]; then - echo "Local virtual OSD state already exists." >&2 - echo "Run reset.sh before reinitializing:" >&2 - echo " bash .local/ceph/reset.sh" >&2 - exit 1 - fi - - echo "Creating a ${CEPH_OSD_SIZE} virtual OSD disk at $OSD_IMAGE..." - qemu_img create -f raw "$OSD_IMAGE" "$CEPH_OSD_SIZE" - - OSD_DEVICE="" - echo "Attaching virtual OSD disk..." - for candidate in /dev/nbd{0..15}; do - if [[ -b "$candidate" ]] && (( $("${SUDO[@]}" blockdev --getsize64 "$candidate" 2>/dev/null || echo 0) > 0 )); then - continue - fi - attach_log="$OSD_STATE_DIR/qemu-nbd-${candidate##*/}.log" - echo "Trying qemu-nbd on $candidate..." - if ! "${SUDO[@]}" qemu-nbd --persistent --fork --verbose \ - --connect="$candidate" --format=raw "$OSD_IMAGE" \ - >"$attach_log" 2>&1; then - cat "$attach_log" >&2 || true - continue - fi - candidate_size=0 - for _ in {1..60}; do - "${SUDO[@]}" udevadm settle || true - candidate_size=$("${SUDO[@]}" blockdev --getsize64 "$candidate" 2>/dev/null || echo 0) - if (( candidate_size >= 5368709120 )); then - break - fi - sleep 1 - done - if (( candidate_size >= 5368709120 )); then - OSD_DEVICE="$candidate" - break - fi - echo "qemu-nbd did not expose a usable block device on $candidate after 60 seconds (size: ${candidate_size} bytes)." >&2 - cat "$attach_log" >&2 || true - "${SUDO[@]}" qemu-nbd --disconnect "$candidate" >/dev/null 2>&1 || true - done - - if [[ -z "$OSD_DEVICE" ]]; then - echo "Could not attach a virtual OSD disk of at least 5 GiB." >&2 - echo "qemu-img info:" - qemu_img info "$OSD_IMAGE" >&2 || true - echo "qemu-nbd attach logs:" - for attach_log in "$OSD_STATE_DIR"/qemu-nbd-*.log; do - [[ -f "$attach_log" ]] || continue - echo "--- $attach_log" >&2 - cat "$attach_log" >&2 - done - exit 1 - fi - - echo "$OSD_DEVICE" > "$OSD_STATE_DIR/osd.device" - CEPH_OSD_DEVICE="$OSD_DEVICE" -fi - -# `ceph osd ls` is deliberately used instead of matching formatted JSON output. -if [[ "$OSD_EXISTS" -eq 0 ]]; then - if [[ ! -b "$CEPH_OSD_DEVICE" ]]; then - echo "CEPH_OSD_DEVICE is not a block device: $CEPH_OSD_DEVICE" >&2 - exit 1 - fi - - if [[ "$VIRTUAL_OSD" -eq 1 ]]; then - echo "Zapping virtual OSD device $CEPH_OSD_DEVICE..." - "${SUDO[@]}" wipefs --all --force "$CEPH_OSD_DEVICE" >/dev/null 2>&1 || true - "${SUDO[@]}" pvremove --force --force --yes "$CEPH_OSD_DEVICE" >/dev/null 2>&1 || true - "${SUDO[@]}" udevadm settle || true - fi - - echo "Creating an OSD on $CEPH_OSD_DEVICE..." - if ! timeout --foreground 120 "${SUDO[@]}" cephadm shell -- ceph orch daemon add osd "$(hostname -s):$CEPH_OSD_DEVICE"; then - echo "cephadm could not create an OSD on $CEPH_OSD_DEVICE." >&2 - ceph orch device ls --wide >&2 || true - ceph orch ps --daemon-type osd >&2 || true - exit 1 - fi -fi - -# Wait for the OSD to register. The cluster may remain HEALTH_WARN due to -# being a one-node cluster; that is expected. -for _ in {1..60}; do - if ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then - break - fi - sleep 2 -done - -if ! ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then - echo "The OSD did not become available." >&2 - ceph orch ps --daemon-type osd >&2 || true - exit 1 -fi - -if ! ceph osd pool ls --format=json | grep -qF "\"$POOL_NAME\""; then - ceph osd pool create "$POOL_NAME" 8 -fi -# A one-OSD development cluster needs a replicated pool size of one. Ceph -# requires this explicit opt-in instead of allowing it by default. -ceph config set mon mon_allow_pool_size_one true -ceph config set osd osd_pool_default_size 1 -ceph config set osd osd_pool_default_min_size 1 -ceph osd pool set "$POOL_NAME" size 1 --yes-i-really-mean-it -ceph osd pool set "$POOL_NAME" min_size 1 -rbd pool init "$POOL_NAME" - -if ! ceph auth get "client.$CLIENT_NAME" >/dev/null 2>&1; then - ceph auth get-or-create "client.$CLIENT_NAME" \ - mon "allow r" \ - osd "allow rwx pool=$POOL_NAME" >/dev/null -fi - -ceph config generate-minimal-conf > "$GENERATED_DIR/ceph.conf" -ceph auth get-key "client.$CLIENT_NAME" > "$GENERATED_DIR/client.$CLIENT_NAME.key" -chmod 600 "$GENERATED_DIR/client.$CLIENT_NAME.key" - -if ! rbd info "$POOL_NAME/$IMAGE_NAME" >/dev/null 2>&1; then - rbd create "$POOL_NAME/$IMAGE_NAME" --size "$IMAGE_SIZE" -fi - -echo "Ceph is ready." -echo "Image: $POOL_NAME/$IMAGE_NAME" -echo "Config: $GENERATED_DIR/ceph.conf" -echo "Key: $GENERATED_DIR/client.$CLIENT_NAME.key" -echo -echo "Export for host-side Odorobo:" -echo " export CEPH_CONFIG=$GENERATED_DIR/ceph.conf" -echo " export CEPH_ID=$CLIENT_NAME" -echo " export CEPH_KEYFILE=$GENERATED_DIR/client.$CLIENT_NAME.key" diff --git a/.local/ceph/reset.sh b/.local/ceph/reset.sh deleted file mode 100644 index 895388c..0000000 --- a/.local/ceph/reset.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash -# Destructively remove the local Ceph cluster and virtual OSD state. -set -euo pipefail - -ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) -GENERATED_DIR="$ROOT_DIR/generated" -STATE_DIR="$ROOT_DIR/state" -OSD_IMAGE="$STATE_DIR/osd.raw" - -if [[ $EUID -eq 0 ]]; then - SUDO=() -else - SUDO=(sudo) -fi - -cephadm() { "${SUDO[@]}" cephadm "$@"; } -rbd() { "${SUDO[@]}" rbd "$@"; } -qemu_nbd() { "${SUDO[@]}" qemu-nbd "$@"; } - -nbd_attached() { - local candidate="$1" - local proc cmdline - for proc in /proc/[0-9]*; do - [[ -r "$proc/cmdline" ]] || continue - cmdline=$(tr '\0' ' ' < "$proc/cmdline" 2>/dev/null || true) - if [[ "$cmdline" == *qemu-nbd* \ - && "$cmdline" == *"--connect=$candidate"* \ - && "$cmdline" == *"$OSD_IMAGE"* ]]; then - return 0 - fi - done - return 1 -} - -if ! command -v cephadm >/dev/null 2>&1; then - echo "cephadm is required. Install cephadm first." >&2 - exit 1 -fi - -FSID=$(cephadm ls 2>/dev/null | python3 -c \ - 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') - -# Unmap any host RBD devices before removing the cluster. -rbd device unmap odorobo-blockpool/dev-disk >/dev/null 2>&1 || true -while read -r _ _ _ _ _ device; do - [[ -n "${device:-}" ]] || continue - rbd device unmap "$device" >/dev/null 2>&1 || true -done < <(rbd device list 2>/dev/null | tail -n +2) - -if [[ -n "$FSID" ]]; then - echo "Removing Ceph cluster $FSID..." - cephadm rm-cluster --force --zap-osds --fsid "$FSID" -else - echo "No active Ceph cluster found." -fi - -# Disconnect only NBD devices whose qemu-nbd process references this project's -# backing file. Never disconnect unrelated NBD devices. -while read -r device; do - [[ -n "$device" ]] || continue - if nbd_attached "$device"; then - echo "Disconnecting $device..." - qemu_nbd --disconnect "$device" >/dev/null 2>&1 || true - fi -done < <(ps -eo args= | sed -n 's/.*--connect=\(\/dev\/nbd[0-9][0-9]*\).*/\1/p' | sort -u) - -# Remove the virtual OSD image only after the cluster and NBD device are gone. -if [[ -f "$OSD_IMAGE" ]]; then - echo "Clearing virtual OSD image..." - OSD_BYTES=$("${SUDO[@]}" stat -c '%s' "$OSD_IMAGE") - "${SUDO[@]}" dd if=/dev/zero of="$OSD_IMAGE" bs=1M count=16 conv=notrunc status=none || true - if (( OSD_BYTES >= 33554432 )); then - "${SUDO[@]}" dd if=/dev/zero of="$OSD_IMAGE" bs=1M seek=$(( (OSD_BYTES / 1048576) - 16 )) count=16 conv=notrunc status=none || true - fi -fi - -rm -rf "$GENERATED_DIR" "$STATE_DIR" -"${SUDO[@]}" rm -rf /etc/ceph/* - -echo "Local Ceph cluster, credentials, and virtual OSD state removed." diff --git a/.local/ceph/start.sh b/.local/ceph/start.sh deleted file mode 100644 index 0f5f6fc..0000000 --- a/.local/ceph/start.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env bash -# Start an existing local Ceph cluster without provisioning or deleting data. -set -euo pipefail - -if [[ $EUID -eq 0 ]]; then - SUDO=() -else - SUDO=(sudo) -fi - -cephadm() { "${SUDO[@]}" cephadm "$@"; } -ceph() { "${SUDO[@]}" cephadm shell -- ceph "$@"; } - -if ! command -v cephadm >/dev/null 2>&1; then - echo "cephadm is required. Install cephadm first." >&2 - exit 1 -fi - -if [[ ! -f /etc/ceph/ceph.conf ]]; then - echo "No local Ceph cluster exists. Run init.sh first:" >&2 - echo " bash .local/ceph/init.sh" >&2 - exit 1 -fi - -FSID=$(cephadm ls 2>/dev/null | python3 -c \ - 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') -if [[ -z "$FSID" ]]; then - echo "Could not determine the local Ceph cluster FSID." >&2 - exit 1 -fi - -TARGET="ceph-$FSID.target" -echo "Starting Ceph cluster $FSID..." -"${SUDO[@]}" systemctl start "$TARGET" - -for _ in {1..30}; do - if ceph -s >/dev/null 2>&1; then - echo "Local Ceph cluster is available." - exit 0 - fi - sleep 2 -done - -echo "Ceph did not become available after starting $TARGET." >&2 -"${SUDO[@]}" systemctl status "$TARGET" --no-pager >&2 || true -ceph -s >&2 || true -exit 1 diff --git a/.local/ceph/stop.sh b/.local/ceph/stop.sh deleted file mode 100644 index 7396414..0000000 --- a/.local/ceph/stop.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env bash -# Stop the local Ceph cluster without deleting its data. -set -euo pipefail - -if [[ $EUID -eq 0 ]]; then - SUDO=() -else - SUDO=(sudo) -fi - -cephadm() { "${SUDO[@]}" cephadm "$@"; } -rbd() { "${SUDO[@]}" rbd "$@"; } - -# Unmap the known test image if it is mapped. Ignore the expected error when it -# is already unmapped. -rbd device unmap odorobo-blockpool/dev-disk >/dev/null 2>&1 || true - -# Unmap any remaining RBD devices. This matters if a test used another image. -while read -r _ _ _ _ _ device; do - [[ -n "${device:-}" ]] || continue - rbd device unmap "$device" >/dev/null 2>&1 || true -done < <(rbd device list 2>/dev/null | tail -n +2) - -if ! command -v cephadm >/dev/null 2>&1; then - echo "cephadm is required. Install cephadm first." >&2 - exit 1 -fi - -FSID=$(cephadm ls 2>/dev/null | python3 -c \ - 'import json, sys; items=json.load(sys.stdin); print(items[0]["fsid"] if items else "")') -if [[ -z "$FSID" ]]; then - echo "No local Ceph cluster found; nothing to stop." - exit 0 -fi - -TARGET="ceph-$FSID.target" -echo "Stopping Ceph cluster $FSID..." -"${SUDO[@]}" systemctl stop "$TARGET" - -echo "Local Ceph daemons stopped. Data and virtual OSD state were preserved." diff --git a/.local/dev/README.md b/.local/dev/README.md new file mode 100644 index 0000000..6750327 --- /dev/null +++ b/.local/dev/README.md @@ -0,0 +1,118 @@ +# Local Ceph and Odorobo with Compose + +This directory runs the local development stack in containers: + +- `ceph` provides a single-node Ceph cluster and a file-backed raw OSD. +- `odorobo` runs the agent in the same network, PID, and device namespaces as Ceph. + +Odorobo must run in the container for the `rbd://` storage path. It invokes `rbd device map`, which creates a kernel block device, and then passes that device to Cloud Hypervisor. A host-side process would not see the container's `/dev/rbd*` device or have the required device and privilege context. + +This is intended for Linux development with a rootful container engine. The stack uses privileged containers because kernel RBD mapping, Cloud Hypervisor, networking, and Ceph's daemon management require host kernel access. + +## Prerequisites + +Install Podman with a working Compose provider. Docker Compose v2 is supported as a fallback. + +For Fedora, the host needs the container engine and kernel modules: + +```bash +sudo dnf install -y podman podman-compose kmod +sudo modprobe rbd +``` + +The container image installs `ceph-common`, Rust tooling, and Cloud Hypervisor tooling. The host does not need `cephadm`, `ceph-common`, `qemu-nbd`, or systemd Ceph units. + +## Usage + +Initialize Ceph and start Odorobo: + +```bash +bash .local/dev/init.sh +``` + +This builds both images, starts both services, provisions the `odorobo-blockpool/dev-disk` RBD image, and starts Odorobo with manager mode enabled. + +Start and stop the complete stack without deleting data: + +```bash +bash .local/dev/start.sh +bash .local/dev/stop.sh +``` + +Destructively remove the containers and all local Ceph state: + +```bash +bash .local/dev/reset.sh +``` + +The scripts prefer `podman compose` and fall back to `docker compose` when Podman Compose is unavailable. + +Useful direct commands: + +```bash +podman compose -f .local/dev/compose.yml ps +podman compose -f .local/dev/compose.yml logs -f ceph odorobo +podman compose -f .local/dev/compose.yml exec ceph ceph -s +``` + +Because `odorobo` uses Ceph's network namespace, the generated Ceph config intentionally uses `127.0.0.1` for the monitor. The application and monitor share that namespace. + +## Application development + +The repository is mounted at `/workspace` in the Odorobo container. Rebuild and restart the application after source changes: + +```bash +podman compose -f .local/dev/compose.yml build odorobo +podman compose -f .local/dev/compose.yml up -d odorobo +podman compose -f .local/dev/compose.yml logs -f odorobo +``` + +The agent runs as: + +```text +cargo run --release -p odorobo -- --manager-enabled +``` + +Its runtime directory is shared through `/run/odorobo`, and Cloud Hypervisor processes and RBD devices are visible in the same namespaces as the agent. + +## Verify the image + +Run Ceph commands inside the Ceph container: + +```bash +podman compose -f .local/dev/compose.yml exec ceph rbd \ + --conf=/etc/ceph/ceph.conf --id=odorobo \ + --keyfile=/var/lib/odorobo-ceph/client.odorobo.key \ + ls --pool odorobo-blockpool +``` + +Expected output includes `dev-disk`. A one-node cluster may report `HEALTH_WARN`; reduced redundancy and a single monitor are expected for local development. + +Do not map the image from the host. To test the exact application path, use an Odorobo manifest with `rbd://odorobo-blockpool/dev-disk`; the `odorobo` service will execute `rbd device map` and pass the resulting device to Cloud Hypervisor. + +## Configuration + +The following environment variables can be set before `init.sh` or passed through Compose: + +```bash +CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3 +CEPH_MON_IP=127.0.0.1 +CEPH_POOL=odorobo-blockpool +CEPH_CLIENT=odorobo +CEPH_IMAGE_NAME=dev-disk +CEPH_IMAGE_SIZE=1G +CEPH_OSD_SIZE=10G +``` + +`CEPH_MON_IP` should remain `127.0.0.1` with the provided Compose topology. If you change the network topology, it must be an address reachable from both services. + +## Layout + +- `compose.yml` — Ceph and Odorobo services, shared namespaces, privilege, mounts, and ports. +- `ceph/Containerfile` — pinned Ceph image. +- `ceph/entrypoint.sh` — bootstrap, OSD, pool, client, and image provisioning. +- `odorobo/Containerfile` — runnable Odorobo development image. +- `ceph/generated/` — generated Ceph credentials shared read-only with Odorobo; ignored by git. +- `ceph/state/` — Ceph configuration, daemons, logs, and file-backed OSD; ignored by git. + +This setup is intentionally not production-ready: it has one monitor, one OSD, no redundancy, and privileged containers. diff --git a/.local/dev/ceph/Containerfile b/.local/dev/ceph/Containerfile new file mode 100644 index 0000000..8afb6fe --- /dev/null +++ b/.local/dev/ceph/Containerfile @@ -0,0 +1,9 @@ +ARG CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3 +FROM ${CEPH_IMAGE} + +USER root + +COPY entrypoint.sh /usr/local/bin/odorobo-ceph +RUN chmod 0755 /usr/local/bin/odorobo-ceph + +ENTRYPOINT ["/usr/local/bin/odorobo-ceph"] diff --git a/.local/dev/ceph/entrypoint.sh b/.local/dev/ceph/entrypoint.sh new file mode 100644 index 0000000..b24578f --- /dev/null +++ b/.local/dev/ceph/entrypoint.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${CEPH_IMAGE:=quay.io/ceph/ceph:v20.2.3}" +: "${CEPH_MON_IP:=127.0.0.1}" +: "${CEPH_POOL:=odorobo-blockpool}" +: "${CEPH_CLIENT:=odorobo}" +: "${CEPH_IMAGE_NAME:=dev-disk}" +: "${CEPH_IMAGE_SIZE:=1G}" +: "${CEPH_OSD_SIZE:=10G}" + +mkdir -p /etc/ceph /var/lib/ceph /var/log/ceph /run/ceph /var/lib/odorobo-ceph + +if [[ ! -f /etc/ceph/ceph.conf ]]; then + cephadm --image "$CEPH_IMAGE" bootstrap \ + --mon-ip "$CEPH_MON_IP" \ + --single-host-defaults \ + --output-dir /etc/ceph \ + --skip-monitoring-stack \ + --allow-fqdn-hostname \ + --skip-pull +fi + +# cephadm has created and started the daemons. Keep this container alive while +# allowing compose stop/restart to control the complete development cluster. +while ! ceph -s >/dev/null 2>&1; do + sleep 2 +done + +if ! ceph osd ls 2>/dev/null | grep -q '[0-9]'; then + truncate -s "$CEPH_OSD_SIZE" /var/lib/odorobo-ceph/osd.raw + OSD_DEVICE=$(losetup --find --show /var/lib/odorobo-ceph/osd.raw) + echo "$OSD_DEVICE" > /var/lib/odorobo-ceph/osd.device + cephadm shell -- ceph-volume raw prepare --data "$OSD_DEVICE" + cephadm shell -- ceph-volume raw activate --device "$OSD_DEVICE" --no-systemd & +fi + +for _ in {1..60}; do + if ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then + break + fi + sleep 2 +done + +ceph config set mon mon_allow_pool_size_one true +ceph config set osd osd_pool_default_size 1 +ceph config set osd osd_pool_default_min_size 1 +if ! ceph osd pool ls --format=json | grep -qF "\"$CEPH_POOL\""; then + ceph osd pool create "$CEPH_POOL" 8 +fi +ceph osd pool set "$CEPH_POOL" size 1 --yes-i-really-mean-it +ceph osd pool set "$CEPH_POOL" min_size 1 +rbd pool init "$CEPH_POOL" + +if ! ceph auth get "client.$CEPH_CLIENT" >/dev/null 2>&1; then + ceph auth get-or-create "client.$CEPH_CLIENT" \ + mon 'allow r' osd "allow rwx pool=$CEPH_POOL" >/dev/null +fi + +if ! rbd info "$CEPH_POOL/$CEPH_IMAGE_NAME" >/dev/null 2>&1; then + rbd create "$CEPH_POOL/$CEPH_IMAGE_NAME" --size "$CEPH_IMAGE_SIZE" +fi + +ceph config generate-minimal-conf > /var/lib/odorobo-ceph/ceph.conf +ceph auth get-key "client.$CEPH_CLIENT" > /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key +chmod 600 /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key +cp /var/lib/odorobo-ceph/ceph.conf /generated/ceph.conf +cp /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key /generated/client.$CEPH_CLIENT.key +chmod 600 /generated/client.$CEPH_CLIENT.key + +tail -f /dev/null diff --git a/.local/dev/compose.yml b/.local/dev/compose.yml new file mode 100644 index 0000000..31cf426 --- /dev/null +++ b/.local/dev/compose.yml @@ -0,0 +1,61 @@ +services: + ceph: + build: + context: ./ceph + dockerfile: Containerfile + args: + CEPH_IMAGE: ${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3} + image: odorobo-ceph:local + container_name: odorobo-ceph + hostname: ceph + privileged: true + restart: unless-stopped + environment: + CEPH_IMAGE: ${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3} + CEPH_MON_IP: ${CEPH_MON_IP:-127.0.0.1} + CEPH_POOL: ${CEPH_POOL:-odorobo-blockpool} + CEPH_CLIENT: ${CEPH_CLIENT:-odorobo} + CEPH_IMAGE_NAME: ${CEPH_IMAGE_NAME:-dev-disk} + CEPH_IMAGE_SIZE: ${CEPH_IMAGE_SIZE:-1G} + CEPH_OSD_SIZE: ${CEPH_OSD_SIZE:-10G} + volumes: + - ./ceph/state/etc-ceph:/etc/ceph + - ./ceph/state/lib-ceph:/var/lib/ceph + - ./ceph/state/log-ceph:/var/log/ceph + - ./ceph/state/run-ceph:/run/ceph + - ./ceph/state/odorobo-ceph:/var/lib/odorobo-ceph + - ./ceph/generated:/generated + healthcheck: + test: ["CMD", "ceph", "-s"] + interval: 5s + timeout: 5s + retries: 30 + ports: + - "3300:3300" + - "6789:6789" + - "8443:8443" + - "9283:9283" + - "6800-7300:6800-7300" + + odorobo: + build: + context: ../.. + dockerfile: .local/dev/odorobo/Containerfile + privileged: true + depends_on: + ceph: + condition: service_healthy + network_mode: service:ceph + pid: service:ceph + volumes: + - ../../:/workspace + - ./ceph/generated:/workspace/.local/dev/ceph/generated:ro + - /run/odorobo:/run/odorobo + - /dev:/dev + working_dir: /workspace + environment: + CEPH_CONFIG: /workspace/.local/dev/ceph/generated/ceph.conf + CEPH_ID: ${CEPH_CLIENT:-odorobo} + CEPH_KEYFILE: /workspace/.local/dev/ceph/generated/client.${CEPH_CLIENT:-odorobo}.key + CEPH_CLUSTER: ceph + command: ["cargo", "run", "--release", "-p", "odorobo", "--", "--manager-enabled"] diff --git a/.local/dev/init.sh b/.local/dev/init.sh new file mode 100644 index 0000000..40310b5 --- /dev/null +++ b/.local/dev/init.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +cd "$ROOT_DIR" + +if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then + COMPOSE=(podman compose) +elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +else + echo "Docker Compose or Podman Compose is required." >&2 + exit 1 +fi + +mkdir -p ceph/generated ceph/state/{etc-ceph,lib-ceph,log-ceph,run-ceph,odorobo-ceph} +"${COMPOSE[@]}" up --build -d ceph odorobo +"${COMPOSE[@]}" exec -T ceph ceph -s + +cat </dev/null 2>&1 && podman compose version >/dev/null 2>&1; then + COMPOSE=(podman compose) +elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +else + echo "Docker Compose or Podman Compose is required." >&2 + exit 1 +fi + +"${COMPOSE[@]}" down --remove-orphans --volumes +rm -rf ceph/generated ceph/state + +echo "Local Ceph container, credentials, and state removed." diff --git a/.local/dev/start.sh b/.local/dev/start.sh new file mode 100644 index 0000000..1ca440a --- /dev/null +++ b/.local/dev/start.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +cd "$ROOT_DIR" + +if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then + COMPOSE=(podman compose) +elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +else + echo "Docker Compose or Podman Compose is required." >&2 + exit 1 +fi + +"${COMPOSE[@]}" start ceph odorobo diff --git a/.local/dev/stop.sh b/.local/dev/stop.sh new file mode 100644 index 0000000..62d42d8 --- /dev/null +++ b/.local/dev/stop.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +cd "$ROOT_DIR" + +if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then + COMPOSE=(podman compose) +elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +elif command -v docker >/dev/null 2>&1; then + echo "Docker Compose is required." >&2 + exit 1 +else + echo "Docker Compose or Podman Compose is required." >&2 + exit 1 +fi + +"${COMPOSE[@]}" stop odorobo ceph From 9e6cb4afe8d42933d51b1a5e024993a5fd6c6a5a Mon Sep 17 00:00:00 2001 From: Willow C Reed Date: Mon, 7 Sep 2026 22:56:08 -0600 Subject: [PATCH 3/8] i don't remember what i was doing but i'm committing it --- .gitignore | 1 + .local/dev/README.md | 13 ++- .local/dev/ceph/entrypoint.sh | 203 +++++++++++++++++++++++++++++----- .local/dev/compose.yml | 18 ++- .local/dev/init.sh | 42 ++++++- .local/dev/reset.sh | 13 ++- 6 files changed, 247 insertions(+), 43 deletions(-) diff --git a/.gitignore b/.gitignore index 544c488..39715bb 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ debug target # ignore dev scratch files dev +images # These are backup files generated by rustfmt **/*.rs.bk diff --git a/.local/dev/README.md b/.local/dev/README.md index 6750327..81d3bd4 100644 --- a/.local/dev/README.md +++ b/.local/dev/README.md @@ -2,7 +2,7 @@ This directory runs the local development stack in containers: -- `ceph` provides a single-node Ceph cluster and a file-backed raw OSD. +- `ceph` provides a single-node Ceph cluster and a file-backed OSD. - `odorobo` runs the agent in the same network, PID, and device namespaces as Ceph. Odorobo must run in the container for the `rbd://` storage path. It invokes `rbd device map`, which creates a kernel block device, and then passes that device to Cloud Hypervisor. A host-side process would not see the container's `/dev/rbd*` device or have the required device and privilege context. @@ -17,10 +17,11 @@ For Fedora, the host needs the container engine and kernel modules: ```bash sudo dnf install -y podman podman-compose kmod -sudo modprobe rbd +sudo modprobe rbd loop +sudo losetup -f ``` -The container image installs `ceph-common`, Rust tooling, and Cloud Hypervisor tooling. The host does not need `cephadm`, `ceph-common`, `qemu-nbd`, or systemd Ceph units. +The container image installs Ceph directly and starts the MON and OSD daemons itself; it intentionally does not start MGR because the MGR's optional Python modules require host udev/system services unavailable in this container. It does not use `cephadm`, nested Podman, or systemd. The OSD uses a persistent raw file attached through a host loop device, initialized directly with `ceph-osd` rather than `ceph-volume`. ## Usage @@ -30,7 +31,7 @@ Initialize Ceph and start Odorobo: bash .local/dev/init.sh ``` -This builds both images, starts both services, provisions the `odorobo-blockpool/dev-disk` RBD image, and starts Odorobo with manager mode enabled. +This builds both images, starts Ceph and waits up to two minutes for it to become healthy, provisions the `odorobo-blockpool/dev-disk` RBD image, and then starts Odorobo with manager mode enabled. On a bootstrap failure, it prints the last 200 Ceph log lines instead of waiting indefinitely. Start and stop the complete stack without deleting data: @@ -69,6 +70,8 @@ podman compose -f .local/dev/compose.yml logs -f odorobo The agent runs as: +By default, the container limits Cargo to two concurrent build jobs to reduce CPU and memory pressure during the initial release build. Override it when starting the stack, for example `CARGO_BUILD_JOBS=4 bash .local/dev/init.sh`. + ```text cargo run --release -p odorobo -- --manager-enabled ``` @@ -110,7 +113,7 @@ CEPH_OSD_SIZE=10G - `compose.yml` — Ceph and Odorobo services, shared namespaces, privilege, mounts, and ports. - `ceph/Containerfile` — pinned Ceph image. -- `ceph/entrypoint.sh` — bootstrap, OSD, pool, client, and image provisioning. +- `ceph/entrypoint.sh` — direct MON bootstrap, filesystem-backed OSD initialization, pool/client/image provisioning, and daemon lifecycle. - `odorobo/Containerfile` — runnable Odorobo development image. - `ceph/generated/` — generated Ceph credentials shared read-only with Odorobo; ignored by git. - `ceph/state/` — Ceph configuration, daemons, logs, and file-backed OSD; ignored by git. diff --git a/.local/dev/ceph/entrypoint.sh b/.local/dev/ceph/entrypoint.sh index b24578f..5b1dc60 100644 --- a/.local/dev/ceph/entrypoint.sh +++ b/.local/dev/ceph/entrypoint.sh @@ -1,7 +1,6 @@ #!/usr/bin/env bash set -euo pipefail -: "${CEPH_IMAGE:=quay.io/ceph/ceph:v20.2.3}" : "${CEPH_MON_IP:=127.0.0.1}" : "${CEPH_POOL:=odorobo-blockpool}" : "${CEPH_CLIENT:=odorobo}" @@ -9,39 +8,184 @@ set -euo pipefail : "${CEPH_IMAGE_SIZE:=1G}" : "${CEPH_OSD_SIZE:=10G}" -mkdir -p /etc/ceph /var/lib/ceph /var/log/ceph /run/ceph /var/lib/odorobo-ceph +CLUSTER=ceph +MON_ID=ceph +MGR_ID=ceph +CEPH_CONF=/etc/ceph/ceph.conf +CEPH_DATA_DIR=/var/lib/ceph +OSD_STATE_DIR=/var/lib/odorobo-ceph +OSD_DATA_DIR="$OSD_STATE_DIR/osd" +OSD_IMAGE="$OSD_STATE_DIR/osd.raw" +OSD_ID_FILE="$OSD_STATE_DIR/osd.id" +OSD_UUID_FILE="$OSD_STATE_DIR/osd.uuid" +LOOP_DEVICE="" +MON_PID="" +MGR_PID="" +OSD_PID="" -if [[ ! -f /etc/ceph/ceph.conf ]]; then - cephadm --image "$CEPH_IMAGE" bootstrap \ - --mon-ip "$CEPH_MON_IP" \ - --single-host-defaults \ - --output-dir /etc/ceph \ - --skip-monitoring-stack \ - --allow-fqdn-hostname \ - --skip-pull +cleanup() { + local pid + trap - EXIT INT TERM + for pid in "$OSD_PID" "$MGR_PID" "$MON_PID"; do + [[ -n "$pid" ]] || continue + kill -TERM "$pid" 2>/dev/null || true + done + for pid in "$OSD_PID" "$MGR_PID" "$MON_PID"; do + [[ -n "$pid" ]] || continue + wait "$pid" 2>/dev/null || true + done + [[ -n "$LOOP_DEVICE" ]] && losetup --detach "$LOOP_DEVICE" 2>/dev/null || true + } +trap cleanup EXIT INT TERM + +mkdir -p /etc/ceph "$CEPH_DATA_DIR" /var/log/ceph /run/ceph "$OSD_STATE_DIR" /generated +chown ceph:ceph /run/ceph + +# These paths are bind-mounted from the host and may retain ownership from a +# previous interrupted or root-run initialization. +for directory in \ + "$CEPH_DATA_DIR/mon" \ + "$CEPH_DATA_DIR/mgr" \ + "$CEPH_DATA_DIR/bootstrap-osd" \ + "$CEPH_DATA_DIR/osd" \ + "$OSD_STATE_DIR"; do + [[ -d "$directory" ]] && chown -R ceph:ceph "$directory" +done + +echo "[odorobo-ceph] initializing direct Ceph daemons" +if [[ ! -f "$CEPH_CONF" ]]; then + echo "[odorobo-ceph] creating monitor configuration and keyrings" + FSID=$(uuidgen) + cat >"$CEPH_CONF" </dev/null 2>&1; do - sleep 2 +echo "[odorobo-ceph] starting monitor" +ceph-mon -f -i "$MON_ID" --setuser ceph --setgroup ceph & +MON_PID=$! + +for _ in {1..60}; do + if timeout 5 ceph -s >/dev/null 2>&1; then + break + fi + if ! kill -0 "$MON_PID" 2>/dev/null; then + wait "$MON_PID" + fi + sleep 1 done +if ! timeout 5 ceph -s; then + echo "Monitor did not become ready within 60 seconds." >&2 + exit 1 +fi + +echo "[odorobo-ceph] monitor ready; preparing OSD" -if ! ceph osd ls 2>/dev/null | grep -q '[0-9]'; then - truncate -s "$CEPH_OSD_SIZE" /var/lib/odorobo-ceph/osd.raw - OSD_DEVICE=$(losetup --find --show /var/lib/odorobo-ceph/osd.raw) - echo "$OSD_DEVICE" > /var/lib/odorobo-ceph/osd.device - cephadm shell -- ceph-volume raw prepare --data "$OSD_DEVICE" - cephadm shell -- ceph-volume raw activate --device "$OSD_DEVICE" --no-systemd & +if [[ -f "$OSD_ID_FILE" ]]; then + OSD_ID=$(<"$OSD_ID_FILE") + OSD_UUID=$(<"$OSD_UUID_FILE") +else + OSD_UUID=$(uuidgen) + OSD_SECRET=$(ceph-authtool --gen-print-key) + OSD_ID=$(printf '{"cephx_secret": "%s"}\n' "$OSD_SECRET" \ + | ceph osd new "$OSD_UUID" -i - -n client.bootstrap-osd \ + -k "$CEPH_DATA_DIR/bootstrap-osd/$CLUSTER.keyring") + [[ -n "$OSD_ID" ]] || { echo "Ceph did not register the OSD." >&2; exit 1; } + mkdir -p "$OSD_DATA_DIR" + ceph-authtool --create-keyring "$OSD_DATA_DIR/keyring" \ + --name "osd.$OSD_ID" --add-key "$OSD_SECRET" + truncate -s "$CEPH_OSD_SIZE" "$OSD_IMAGE" + LOOP_DEVICE=$(losetup --find --show "$OSD_IMAGE") || { + echo "Unable to attach the OSD image to a loop device." >&2 + exit 1 + } + # The host-created loop node is normally root:disk; the OSD daemon drops to + # the ceph user and therefore needs direct access to this block device. + chown ceph:ceph "$LOOP_DEVICE" + chmod 0660 "$LOOP_DEVICE" + ln -sfn "$LOOP_DEVICE" "$OSD_DATA_DIR/block" + ceph-osd -i "$OSD_ID" --mkfs --osd-uuid "$OSD_UUID" \ + --osd-data "$OSD_DATA_DIR" + printf '%s\n' "$OSD_ID" >"$OSD_ID_FILE" + printf '%s\n' "$OSD_UUID" >"$OSD_UUID_FILE" fi +if [[ -z "$LOOP_DEVICE" ]]; then + LOOP_DEVICE=$(losetup --associated --noheadings --output NAME "$OSD_IMAGE" | awk 'NR == 1 { print; exit }') + [[ -n "$LOOP_DEVICE" ]] || LOOP_DEVICE=$(losetup --find --show "$OSD_IMAGE") || { + echo "Unable to attach the OSD image to a loop device." >&2 + exit 1 + } + chown ceph:ceph "$LOOP_DEVICE" + chmod 0660 "$LOOP_DEVICE" + ln -sfn "$LOOP_DEVICE" "$OSD_DATA_DIR/block" +fi + +chown -R ceph:ceph "$OSD_DATA_DIR" +ceph-osd -f -i "$OSD_ID" --osd-data "$OSD_DATA_DIR" \ + --setuser ceph --setgroup ceph & +OSD_PID=$! + +echo "[odorobo-ceph] preparing and starting OSD" for _ in {1..60}; do - if ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* osds:'; then + if timeout 5 ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* up'; then break fi - sleep 2 + if ! kill -0 "$OSD_PID" 2>/dev/null; then + wait "$OSD_PID" + fi + sleep 1 done +if ! timeout 5 ceph osd stat 2>/dev/null | grep -q '[1-9][0-9]* up'; then + echo "OSD did not become up within 60 seconds." >&2 + exit 1 +fi +echo "[odorobo-ceph] OSD ready; provisioning RBD" ceph config set mon mon_allow_pool_size_one true ceph config set osd osd_pool_default_size 1 ceph config set osd osd_pool_default_min_size 1 @@ -56,16 +200,15 @@ if ! ceph auth get "client.$CEPH_CLIENT" >/dev/null 2>&1; then ceph auth get-or-create "client.$CEPH_CLIENT" \ mon 'allow r' osd "allow rwx pool=$CEPH_POOL" >/dev/null fi - if ! rbd info "$CEPH_POOL/$CEPH_IMAGE_NAME" >/dev/null 2>&1; then rbd create "$CEPH_POOL/$CEPH_IMAGE_NAME" --size "$CEPH_IMAGE_SIZE" fi -ceph config generate-minimal-conf > /var/lib/odorobo-ceph/ceph.conf -ceph auth get-key "client.$CEPH_CLIENT" > /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key -chmod 600 /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key -cp /var/lib/odorobo-ceph/ceph.conf /generated/ceph.conf -cp /var/lib/odorobo-ceph/client.$CEPH_CLIENT.key /generated/client.$CEPH_CLIENT.key -chmod 600 /generated/client.$CEPH_CLIENT.key +ceph config generate-minimal-conf >"$OSD_STATE_DIR/ceph.conf" +ceph auth get-key "client.$CEPH_CLIENT" >"$OSD_STATE_DIR/client.$CEPH_CLIENT.key" +chmod 600 "$OSD_STATE_DIR/client.$CEPH_CLIENT.key" +cp "$OSD_STATE_DIR/ceph.conf" /generated/ceph.conf +cp "$OSD_STATE_DIR/client.$CEPH_CLIENT.key" /generated/client."$CEPH_CLIENT".key +chmod 600 /generated/client."$CEPH_CLIENT".key -tail -f /dev/null +wait "$MON_PID" "$MGR_PID" "$OSD_PID" diff --git a/.local/dev/compose.yml b/.local/dev/compose.yml index 31cf426..d7421e5 100644 --- a/.local/dev/compose.yml +++ b/.local/dev/compose.yml @@ -9,7 +9,9 @@ services: container_name: odorobo-ceph hostname: ceph privileged: true - restart: unless-stopped + # Bootstrap errors must leave the container stopped so `init.sh` can report + # them instead of silently repeating a failed initialization. + restart: "no" environment: CEPH_IMAGE: ${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3} CEPH_MON_IP: ${CEPH_MON_IP:-127.0.0.1} @@ -25,6 +27,9 @@ services: - ./ceph/state/run-ceph:/run/ceph - ./ceph/state/odorobo-ceph:/var/lib/odorobo-ceph - ./ceph/generated:/generated + # Direct BlueStore initialization uses a host loop-backed block device. + - /dev:/dev + healthcheck: test: ["CMD", "ceph", "-s"] interval: 5s @@ -45,8 +50,10 @@ services: depends_on: ceph: condition: service_healthy - network_mode: service:ceph - pid: service:ceph + # podman-compose supports container namespace targets, but not the + # Compose `service:ceph` shorthand. + network_mode: container:odorobo-ceph + pid: container:odorobo-ceph volumes: - ../../:/workspace - ./ceph/generated:/workspace/.local/dev/ceph/generated:ro @@ -58,4 +65,7 @@ services: CEPH_ID: ${CEPH_CLIENT:-odorobo} CEPH_KEYFILE: /workspace/.local/dev/ceph/generated/client.${CEPH_CLIENT:-odorobo}.key CEPH_CLUSTER: ceph - command: ["cargo", "run", "--release", "-p", "odorobo", "--", "--manager-enabled"] + # Release compilation is memory-intensive; keep the default usable on + # development laptops while allowing an override for faster machines. + CARGO_BUILD_JOBS: ${CARGO_BUILD_JOBS:-2} + command: ["cargo", "run", "--release", "-p", "odorobo", "--", "--manager-enabled", "true"] diff --git a/.local/dev/init.sh b/.local/dev/init.sh index 40310b5..70232cd 100644 --- a/.local/dev/init.sh +++ b/.local/dev/init.sh @@ -6,16 +6,54 @@ cd "$ROOT_DIR" if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then COMPOSE=(podman compose) + ENGINE=(podman) elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then COMPOSE=(docker compose) + ENGINE=(docker) else echo "Docker Compose or Podman Compose is required." >&2 exit 1 fi + +if ! losetup -f >/dev/null 2>&1; then + echo "No loop device is available. Run: sudo modprobe loop && sudo losetup -f" >&2 + exit 1 +fi + mkdir -p ceph/generated ceph/state/{etc-ceph,lib-ceph,log-ceph,run-ceph,odorobo-ceph} -"${COMPOSE[@]}" up --build -d ceph odorobo -"${COMPOSE[@]}" exec -T ceph ceph -s + +# Start Ceph independently: `odorobo` depends on its health check, and starting +# both at once hides a Ceph bootstrap failure behind Compose's dependency wait. +"${COMPOSE[@]}" up --build -d ceph + +echo "Waiting for Ceph to become healthy..." +for attempt in {1..60}; do + # Inspect the engine directly; the external podman-compose provider's `ps` + # command is not a reliable readiness API. + container_state=$(timeout 5 "${ENGINE[@]}" inspect --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}' odorobo-ceph 2>/dev/null || true) + if [[ "$container_state" == running\ healthy* ]]; then + break + elif [[ "$container_state" != running* && -n "$container_state" ]]; then + echo "Ceph stopped before becoming healthy. Recent logs:" >&2 + "${COMPOSE[@]}" logs --tail=200 ceph >&2 || true + exit 1 + fi + + if (( attempt % 10 == 0 )); then + echo "Ceph is still not ready; recent logs:" >&2 + "${COMPOSE[@]}" logs --tail=40 ceph >&2 || true + fi + sleep 2 +done + +if [[ $(timeout 5 "${ENGINE[@]}" inspect --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}' odorobo-ceph 2>/dev/null || true) != running\ healthy* ]]; then + echo "Ceph did not become healthy within 120 seconds. Recent logs:" >&2 + "${COMPOSE[@]}" logs --tail=200 ceph >&2 || true + exit 1 +fi + +"${COMPOSE[@]}" up -d odorobo cat </dev/null 2>&1; then + SUDO=(sudo) +else + echo "reset.sh must run as root or have sudo available to remove Ceph-owned state." >&2 + exit 1 +fi + if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then COMPOSE=(podman compose) elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then @@ -13,7 +22,7 @@ else exit 1 fi -"${COMPOSE[@]}" down --remove-orphans --volumes -rm -rf ceph/generated ceph/state +"${COMPOSE[@]}" down --remove-orphans --volumes || true +"${SUDO[@]}" rm -rf ceph/generated ceph/state echo "Local Ceph container, credentials, and state removed." From 43518fa77b4131c45830c28ed5a3dbd85a235536 Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Sun, 20 Sep 2026 20:26:13 -0500 Subject: [PATCH 4/8] chore: remove overengineered util scripts --- .local/dev/README.md | 22 +++++++++++----------- .local/dev/reset.sh | 28 ---------------------------- .local/dev/start.sh | 16 ---------------- .local/dev/stop.sh | 19 ------------------- 4 files changed, 11 insertions(+), 74 deletions(-) delete mode 100644 .local/dev/reset.sh delete mode 100644 .local/dev/start.sh delete mode 100644 .local/dev/stop.sh diff --git a/.local/dev/README.md b/.local/dev/README.md index 81d3bd4..02fac17 100644 --- a/.local/dev/README.md +++ b/.local/dev/README.md @@ -31,23 +31,22 @@ Initialize Ceph and start Odorobo: bash .local/dev/init.sh ``` -This builds both images, starts Ceph and waits up to two minutes for it to become healthy, provisions the `odorobo-blockpool/dev-disk` RBD image, and then starts Odorobo with manager mode enabled. On a bootstrap failure, it prints the last 200 Ceph log lines instead of waiting indefinitely. +This builds both images, starts Ceph and waits up to two minutes for it to become healthy, provisions the `odorobo-blockpool/dev-disk` RBD image, and then starts Odorobo with manager mode enabled. On a bootstrap failure, it prints the last 200 Ceph log lines instead of waiting indefinitely. It prefers `podman compose` and falls back to `docker compose` when Podman Compose is unavailable. -Start and stop the complete stack without deleting data: +Start and stop the complete stack without deleting data. `start` only works on existing containers; after a reset, run `init.sh` again: ```bash -bash .local/dev/start.sh -bash .local/dev/stop.sh +podman compose -f .local/dev/compose.yml start ceph odorobo +podman compose -f .local/dev/compose.yml stop odorobo ceph ``` -Destructively remove the containers and all local Ceph state: +Destructively remove the containers and all local Ceph state. ```bash -bash .local/dev/reset.sh +podman compose -f .local/dev/compose.yml down --remove-orphans --volumes +sudo rm -rf .local/dev/ceph/generated .local/dev/ceph/state ``` -The scripts prefer `podman compose` and fall back to `docker compose` when Podman Compose is unavailable. - Useful direct commands: ```bash @@ -68,12 +67,12 @@ podman compose -f .local/dev/compose.yml up -d odorobo podman compose -f .local/dev/compose.yml logs -f odorobo ``` -The agent runs as: - By default, the container limits Cargo to two concurrent build jobs to reduce CPU and memory pressure during the initial release build. Override it when starting the stack, for example `CARGO_BUILD_JOBS=4 bash .local/dev/init.sh`. +The agent runs as: + ```text -cargo run --release -p odorobo -- --manager-enabled +cargo run --release -p odorobo -- --manager-enabled true ``` Its runtime directory is shared through `/run/odorobo`, and Cloud Hypervisor processes and RBD devices are visible in the same namespaces as the agent. @@ -111,6 +110,7 @@ CEPH_OSD_SIZE=10G ## Layout +- `init.sh` — builds and starts the stack, waits for Ceph health, and reports bootstrap failures with logs. - `compose.yml` — Ceph and Odorobo services, shared namespaces, privilege, mounts, and ports. - `ceph/Containerfile` — pinned Ceph image. - `ceph/entrypoint.sh` — direct MON bootstrap, filesystem-backed OSD initialization, pool/client/image provisioning, and daemon lifecycle. diff --git a/.local/dev/reset.sh b/.local/dev/reset.sh deleted file mode 100644 index 9fbdf85..0000000 --- a/.local/dev/reset.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) -cd "$ROOT_DIR" - -if [[ $EUID -eq 0 ]]; then - SUDO=() -elif command -v sudo >/dev/null 2>&1; then - SUDO=(sudo) -else - echo "reset.sh must run as root or have sudo available to remove Ceph-owned state." >&2 - exit 1 -fi - -if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then - COMPOSE=(podman compose) -elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then - COMPOSE=(docker compose) -else - echo "Docker Compose or Podman Compose is required." >&2 - exit 1 -fi - -"${COMPOSE[@]}" down --remove-orphans --volumes || true -"${SUDO[@]}" rm -rf ceph/generated ceph/state - -echo "Local Ceph container, credentials, and state removed." diff --git a/.local/dev/start.sh b/.local/dev/start.sh deleted file mode 100644 index 1ca440a..0000000 --- a/.local/dev/start.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) -cd "$ROOT_DIR" - -if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then - COMPOSE=(podman compose) -elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then - COMPOSE=(docker compose) -else - echo "Docker Compose or Podman Compose is required." >&2 - exit 1 -fi - -"${COMPOSE[@]}" start ceph odorobo diff --git a/.local/dev/stop.sh b/.local/dev/stop.sh deleted file mode 100644 index 62d42d8..0000000 --- a/.local/dev/stop.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) -cd "$ROOT_DIR" - -if command -v podman >/dev/null 2>&1 && podman compose version >/dev/null 2>&1; then - COMPOSE=(podman compose) -elif command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then - COMPOSE=(docker compose) -elif command -v docker >/dev/null 2>&1; then - echo "Docker Compose is required." >&2 - exit 1 -else - echo "Docker Compose or Podman Compose is required." >&2 - exit 1 -fi - -"${COMPOSE[@]}" stop odorobo ceph From 26b5ed5290a4afde83e7ac25f446e137a80c2b1b Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Sun, 20 Sep 2026 21:20:47 -0500 Subject: [PATCH 5/8] various small improvements --- .gitignore | 1 - .local/dev/README.md | 8 ++++---- .local/dev/ceph/.dockerignore | 2 ++ .local/dev/ceph/entrypoint.sh | 19 ++++++------------- .local/dev/compose.yml | 11 ++--------- .local/dev/init.sh | 4 ++-- .local/dev/odorobo/Containerfile | 2 +- 7 files changed, 17 insertions(+), 30 deletions(-) create mode 100644 .local/dev/ceph/.dockerignore diff --git a/.gitignore b/.gitignore index 39715bb..544c488 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,6 @@ debug target # ignore dev scratch files dev -images # These are backup files generated by rustfmt **/*.rs.bk diff --git a/.local/dev/README.md b/.local/dev/README.md index 02fac17..a3f5c7d 100644 --- a/.local/dev/README.md +++ b/.local/dev/README.md @@ -3,9 +3,9 @@ This directory runs the local development stack in containers: - `ceph` provides a single-node Ceph cluster and a file-backed OSD. -- `odorobo` runs the agent in the same network, PID, and device namespaces as Ceph. +- `odorobo` runs the agent, sharing Ceph's network and PID namespaces and the host's `/dev`. -Odorobo must run in the container for the `rbd://` storage path. It invokes `rbd device map`, which creates a kernel block device, and then passes that device to Cloud Hypervisor. A host-side process would not see the container's `/dev/rbd*` device or have the required device and privilege context. +Odorobo must run in the container for the `rbd://` storage path. It invokes `rbd device map` using the generated Ceph credentials, which creates a kernel block device, and then passes that device to Cloud Hypervisor. The container provides the credential files, the privileged device access, and the shared namespaces that a host-side process would have to replicate. This is intended for Linux development with a rootful container engine. The stack uses privileged containers because kernel RBD mapping, Cloud Hypervisor, networking, and Ceph's daemon management require host kernel access. @@ -21,7 +21,7 @@ sudo modprobe rbd loop sudo losetup -f ``` -The container image installs Ceph directly and starts the MON and OSD daemons itself; it intentionally does not start MGR because the MGR's optional Python modules require host udev/system services unavailable in this container. It does not use `cephadm`, nested Podman, or systemd. The OSD uses a persistent raw file attached through a host loop device, initialized directly with `ceph-osd` rather than `ceph-volume`. +The Ceph image is based on the official `quay.io/ceph/ceph` image and starts the MON and OSD daemons itself; it intentionally does not start MGR because the MGR's optional Python modules require host udev/system services unavailable in this container. It does not use `cephadm`, nested Podman, or systemd. The OSD uses a persistent raw file attached through a host loop device, initialized directly with `ceph-osd` rather than `ceph-volume`. ## Usage @@ -88,7 +88,7 @@ podman compose -f .local/dev/compose.yml exec ceph rbd \ ls --pool odorobo-blockpool ``` -Expected output includes `dev-disk`. A one-node cluster may report `HEALTH_WARN`; reduced redundancy and a single monitor are expected for local development. +Expected output includes `dev-disk`. A one-node cluster may report `HEALTH_WARN`; reduced redundancy, a single monitor, and no running MGR are expected for local development. Do not map the image from the host. To test the exact application path, use an Odorobo manifest with `rbd://odorobo-blockpool/dev-disk`; the `odorobo` service will execute `rbd device map` and pass the resulting device to Cloud Hypervisor. diff --git a/.local/dev/ceph/.dockerignore b/.local/dev/ceph/.dockerignore new file mode 100644 index 0000000..888513e --- /dev/null +++ b/.local/dev/ceph/.dockerignore @@ -0,0 +1,2 @@ +state/ +generated/ diff --git a/.local/dev/ceph/entrypoint.sh b/.local/dev/ceph/entrypoint.sh index 5b1dc60..f460ead 100644 --- a/.local/dev/ceph/entrypoint.sh +++ b/.local/dev/ceph/entrypoint.sh @@ -10,7 +10,6 @@ set -euo pipefail CLUSTER=ceph MON_ID=ceph -MGR_ID=ceph CEPH_CONF=/etc/ceph/ceph.conf CEPH_DATA_DIR=/var/lib/ceph OSD_STATE_DIR=/var/lib/odorobo-ceph @@ -20,17 +19,16 @@ OSD_ID_FILE="$OSD_STATE_DIR/osd.id" OSD_UUID_FILE="$OSD_STATE_DIR/osd.uuid" LOOP_DEVICE="" MON_PID="" -MGR_PID="" OSD_PID="" cleanup() { local pid trap - EXIT INT TERM - for pid in "$OSD_PID" "$MGR_PID" "$MON_PID"; do + for pid in "$OSD_PID" "$MON_PID"; do [[ -n "$pid" ]] || continue kill -TERM "$pid" 2>/dev/null || true done - for pid in "$OSD_PID" "$MGR_PID" "$MON_PID"; do + for pid in "$OSD_PID" "$MON_PID"; do [[ -n "$pid" ]] || continue wait "$pid" 2>/dev/null || true done @@ -45,7 +43,6 @@ chown ceph:ceph /run/ceph # previous interrupted or root-run initialization. for directory in \ "$CEPH_DATA_DIR/mon" \ - "$CEPH_DATA_DIR/mgr" \ "$CEPH_DATA_DIR/bootstrap-osd" \ "$CEPH_DATA_DIR/osd" \ "$OSD_STATE_DIR"; do @@ -68,16 +65,9 @@ osd pool default size = 1 osd pool default min size = 1 osd crush chooseleaf type = 0 osd objectstore = bluestore - -[mgr] -# This stack runs daemons directly and does not use cephadm orchestration. -# The direct-daemon container does not provide the host udev/system services used -# by optional MGR Python modules. RBD and Ceph CLI operations do not require them. -mgr disabled modules = alerts,balancer,cephadm,crash,dashboard,devicehealth,diskprediction_local,dynatrace,influx,insights,iostat,k8sevents,loki,nfs,orchestrator,pg_autoscaler,prometheus,restful,selftest,snap_schedule,stats,telemetry,telegraf,volumes,zabbix EOF install -d -o ceph -g ceph "$CEPH_DATA_DIR/mon/$CLUSTER-$MON_ID" - install -d -o ceph -g ceph "$CEPH_DATA_DIR/mgr/$CLUSTER-$MGR_ID" install -d -o ceph -g ceph "$CEPH_DATA_DIR/bootstrap-osd" ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. @@ -211,4 +201,7 @@ cp "$OSD_STATE_DIR/ceph.conf" /generated/ceph.conf cp "$OSD_STATE_DIR/client.$CEPH_CLIENT.key" /generated/client."$CEPH_CLIENT".key chmod 600 /generated/client."$CEPH_CLIENT".key -wait "$MON_PID" "$MGR_PID" "$OSD_PID" +for pid in "$MON_PID" "$OSD_PID"; do + [[ -n "$pid" ]] || continue + wait "$pid" +done diff --git a/.local/dev/compose.yml b/.local/dev/compose.yml index d7421e5..19b357e 100644 --- a/.local/dev/compose.yml +++ b/.local/dev/compose.yml @@ -13,7 +13,6 @@ services: # them instead of silently repeating a failed initialization. restart: "no" environment: - CEPH_IMAGE: ${CEPH_IMAGE:-quay.io/ceph/ceph:v20.2.3} CEPH_MON_IP: ${CEPH_MON_IP:-127.0.0.1} CEPH_POOL: ${CEPH_POOL:-odorobo-blockpool} CEPH_CLIENT: ${CEPH_CLIENT:-odorobo} @@ -35,17 +34,11 @@ services: interval: 5s timeout: 5s retries: 30 - ports: - - "3300:3300" - - "6789:6789" - - "8443:8443" - - "9283:9283" - - "6800-7300:6800-7300" odorobo: build: - context: ../.. - dockerfile: .local/dev/odorobo/Containerfile + context: ./odorobo + dockerfile: Containerfile privileged: true depends_on: ceph: diff --git a/.local/dev/init.sh b/.local/dev/init.sh index 70232cd..34e5602 100644 --- a/.local/dev/init.sh +++ b/.local/dev/init.sh @@ -62,8 +62,8 @@ Config: $ROOT_DIR/ceph/generated/ceph.conf Key: $ROOT_DIR/ceph/generated/client.${CEPH_CLIENT:-odorobo}.key Generated credentials (for tools running inside the Odorobo container): - export CEPH_CONFIG=$ROOT_DIR/ceph/generated/ceph.conf + export CEPH_CONFIG=/workspace/.local/dev/ceph/generated/ceph.conf export CEPH_ID=${CEPH_CLIENT:-odorobo} - export CEPH_KEYFILE=$ROOT_DIR/ceph/generated/client.${CEPH_CLIENT:-odorobo}.key + export CEPH_KEYFILE=/workspace/.local/dev/ceph/generated/client.${CEPH_CLIENT:-odorobo}.key export CEPH_CLUSTER=ceph EOF diff --git a/.local/dev/odorobo/Containerfile b/.local/dev/odorobo/Containerfile index ed4298b..06c800d 100644 --- a/.local/dev/odorobo/Containerfile +++ b/.local/dev/odorobo/Containerfile @@ -15,4 +15,4 @@ RUN dnf do -y --action=install \ WORKDIR /workspace -CMD ["cargo", "run", "--release", "-p", "odorobo", "--", "--manager-enabled"] +CMD ["cargo", "run", "--release", "-p", "odorobo", "--", "--manager-enabled", "true"] From 07548180bb64d46870f4df97dbdfd99d8664d3af Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Mon, 21 Sep 2026 03:14:55 -0500 Subject: [PATCH 6/8] slowly making progress. big things here are the noudev and msgr2 options --- .local/dev/README.md | 19 ++++++---- .local/dev/ceph/entrypoint.sh | 8 +++- .local/dev/compose.yml | 5 +++ .local/dev/init.sh | 4 +- .local/dev/vm-test.json | 18 +++++++++ .local/dev/vm-test.sh | 28 ++++++++++++++ odorobo/src/ch_driver/manifest.rs | 37 ++++++++++++++++++- .../src/ch_driver/transform/storage/rbd.rs | 4 ++ 8 files changed, 112 insertions(+), 11 deletions(-) create mode 100644 .local/dev/vm-test.json create mode 100644 .local/dev/vm-test.sh diff --git a/.local/dev/README.md b/.local/dev/README.md index a3f5c7d..504fee2 100644 --- a/.local/dev/README.md +++ b/.local/dev/README.md @@ -21,6 +21,10 @@ sudo modprobe rbd loop sudo losetup -f ``` +`podman compose` prefers the `docker-compose` plugin when it is installed, and that provider talks to Podman's API socket, which must be running (`systemctl --user enable --now podman.socket` for rootless, `sudo systemctl enable --now podman.socket` for rootful). If you hit `failed to connect to the docker API at unix:///run/user//podman/podman.sock`, either enable that socket or select the standalone tool as the provider: `compose_providers = ["podman-compose"]` in `~/.config/containers/containers.conf` (or `export PODMAN_COMPOSE_PROVIDER=podman-compose`). + +The stack must run on a **rootful** engine. Rootless Podman cannot work: the Ceph container attaches the OSD file through a host loop device, and the kernel's loop driver requires `CAP_SYS_ADMIN` in the initial user namespace, which a rootless container never has (even `privileged` + a `/dev` bind mount do not help). Run everything rootful, e.g. `sudo bash .local/dev/init.sh`, and prefix the `podman compose` commands below with `sudo` accordingly. + The Ceph image is based on the official `quay.io/ceph/ceph` image and starts the MON and OSD daemons itself; it intentionally does not start MGR because the MGR's optional Python modules require host udev/system services unavailable in this container. It does not use `cephadm`, nested Podman, or systemd. The OSD uses a persistent raw file attached through a host loop device, initialized directly with `ceph-osd` rather than `ceph-volume`. ## Usage @@ -28,7 +32,7 @@ The Ceph image is based on the official `quay.io/ceph/ceph` image and starts the Initialize Ceph and start Odorobo: ```bash -bash .local/dev/init.sh +sudo bash .local/dev/init.sh ``` This builds both images, starts Ceph and waits up to two minutes for it to become healthy, provisions the `odorobo-blockpool/dev-disk` RBD image, and then starts Odorobo with manager mode enabled. On a bootstrap failure, it prints the last 200 Ceph log lines instead of waiting indefinitely. It prefers `podman compose` and falls back to `docker compose` when Podman Compose is unavailable. @@ -36,23 +40,24 @@ This builds both images, starts Ceph and waits up to two minutes for it to becom Start and stop the complete stack without deleting data. `start` only works on existing containers; after a reset, run `init.sh` again: ```bash -podman compose -f .local/dev/compose.yml start ceph odorobo -podman compose -f .local/dev/compose.yml stop odorobo ceph +sudo podman compose -f .local/dev/compose.yml start ceph odorobo +sudo podman compose -f .local/dev/compose.yml stop odorobo ceph ``` Destructively remove the containers and all local Ceph state. ```bash -podman compose -f .local/dev/compose.yml down --remove-orphans --volumes +sudo podman compose -f .local/dev/compose.yml down --remove-orphans --volumes sudo rm -rf .local/dev/ceph/generated .local/dev/ceph/state ``` Useful direct commands: ```bash -podman compose -f .local/dev/compose.yml ps -podman compose -f .local/dev/compose.yml logs -f ceph odorobo -podman compose -f .local/dev/compose.yml exec ceph ceph -s +sudo podman compose -f .local/dev/compose.yml ps +sudo podman compose -f .local/dev/compose.yml logs -f ceph odorobo +sudo podman compose -f .local/dev/compose.yml exec ceph ceph -s +sudo podman compose -f .local/dev/compose.yml exec -it odorobo sh ``` Because `odorobo` uses Ceph's network namespace, the generated Ceph config intentionally uses `127.0.0.1` for the monitor. The application and monitor share that namespace. diff --git a/.local/dev/ceph/entrypoint.sh b/.local/dev/ceph/entrypoint.sh index f460ead..2dae03a 100644 --- a/.local/dev/ceph/entrypoint.sh +++ b/.local/dev/ceph/entrypoint.sh @@ -113,7 +113,13 @@ if ! timeout 5 ceph -s; then exit 1 fi -echo "[odorobo-ceph] monitor ready; preparing OSD" +echo "[odorobo-ceph] monitor ready; enabling msgr2" +# Ceph v20's kernel-RBD client defaults to msgr2 and rejects a monmap that +# advertises only the legacy v1 endpoint. This also makes the generated minimal +# client config contain an address usable by `rbd device map`. +ceph mon enable-msgr2 + +echo "[odorobo-ceph] preparing OSD" if [[ -f "$OSD_ID_FILE" ]]; then OSD_ID=$(<"$OSD_ID_FILE") diff --git a/.local/dev/compose.yml b/.local/dev/compose.yml index 19b357e..532055a 100644 --- a/.local/dev/compose.yml +++ b/.local/dev/compose.yml @@ -52,6 +52,11 @@ services: - ./ceph/generated:/workspace/.local/dev/ceph/generated:ro - /run/odorobo:/run/odorobo - /dev:/dev + # The rbd CLI runs `modprobe rbd` before mapping; the container's own + # /lib/modules won't match the host's running kernel, so bind the host's + # module tree in. modprobe finds rbd, sees it's already loaded on the + # shared kernel, and skips the insmod. + - /lib/modules:/lib/modules:ro working_dir: /workspace environment: CEPH_CONFIG: /workspace/.local/dev/ceph/generated/ceph.conf diff --git a/.local/dev/init.sh b/.local/dev/init.sh index 34e5602..453b480 100644 --- a/.local/dev/init.sh +++ b/.local/dev/init.sh @@ -16,8 +16,8 @@ else fi -if ! losetup -f >/dev/null 2>&1; then - echo "No loop device is available. Run: sudo modprobe loop && sudo losetup -f" >&2 +if [[ -z "$(losetup -f 2>/dev/null)" ]]; then + echo "No free loop device is available. Run: sudo modprobe loop" >&2 exit 1 fi diff --git a/.local/dev/vm-test.json b/.local/dev/vm-test.json new file mode 100644 index 0000000..16612e1 --- /dev/null +++ b/.local/dev/vm-test.json @@ -0,0 +1,18 @@ +{ + "vm": { + "api_version": 1, + "id": "01J0TESTVM0000000000000001", + "desired": { + "metadata": { "name": "ceph-test", "labels": {}, "annotations": {} }, + "compute": { "vcpus": 2, "memory_bytes": 2147483648 }, + "storage": [ { "id": "root", "uri": "rbd://odorobo-blockpool/dev-disk" } ], + "networks": [], + "placement": {}, + "boot": { + "start": true, + "kernel": "/tmp/vmlinuz-virt", + "cmdline": "console=ttyS0 root=/dev/vda init=/init" + } + } + } +} diff --git a/.local/dev/vm-test.sh b/.local/dev/vm-test.sh new file mode 100644 index 0000000..027b476 --- /dev/null +++ b/.local/dev/vm-test.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Start the ceph-backed test VM. +# +# Run this from INSIDE the odorobo container, where 127.0.0.1:3000 is the +# agent (the container shares Ceph's network namespace): +# sudo podman compose -f .local/dev/compose.yml exec odorobo \ +# bash .local/dev/vm-test.sh +# +# Prerequisites (already done manually this session): +# - /tmp/vmlinuz-virt present in the container (Alpine virt kernel) +# - rbd image mapped: /dev/rbd/odorobo-blockpool/dev-disk -> /dev/nbd0 +# - rootfs dd'd into that device +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +echo ">> health check" +curl -s http://127.0.0.1:3000/health; echo + +echo ">> POST /vms" +curl -s -X POST http://127.0.0.1:3000/vms \ + -H 'Content-Type: application/json' \ + -d "@$DIR/vm-test.json" +echo + +echo ">> waiting for boot, then dumping console history" +sleep 15 +curl -s http://127.0.0.1:3000/vms/01J0TESTVM0000000000000001/console/history \ + | strings | tail -40 diff --git a/odorobo/src/ch_driver/manifest.rs b/odorobo/src/ch_driver/manifest.rs index 6791b77..aac0c33 100644 --- a/odorobo/src/ch_driver/manifest.rs +++ b/odorobo/src/ch_driver/manifest.rs @@ -76,7 +76,18 @@ pub fn to_vm_config(manifest: &VmManifest) -> Result { .boot .firmware .clone() - .or_else(|| Some("/var/lib/odorobo/CLOUDHV.fd".to_owned())), + .or_else(|| { + // Only default to the firmware when doing a firmware boot + // (no kernel specified). Direct kernel boot must not set a + // firmware, or Cloud Hypervisor rejects the config with + // "Specifying a kernel is not supported when a firmware is + // provided". + if desired.boot.kernel.is_some() { + None + } else { + Some("/var/lib/odorobo/CLOUDHV.fd".to_owned()) + } + }), kernel: desired.boot.kernel.clone(), cmdline: desired.boot.cmdline.clone(), ..Default::default() @@ -166,6 +177,30 @@ mod tests { ); } + #[test] + fn firmware_default_only_for_firmware_boot() { + // No kernel, no explicit firmware -> defaults to the firmware. + let config = to_vm_config(&minimal()).expect("minimal manifest converts"); + assert_eq!( + config.payload.firmware.as_deref(), + Some("/var/lib/odorobo/CLOUDHV.fd") + ); + + // Kernel set, no explicit firmware -> no firmware (direct kernel boot). + let mut manifest = minimal(); + manifest.desired.boot.kernel = Some("/tmp/vmlinuz".to_owned()); + let config = to_vm_config(&manifest).expect("kernel manifest converts"); + assert_eq!(config.payload.firmware, None); + assert_eq!(config.payload.kernel.as_deref(), Some("/tmp/vmlinuz")); + + // Explicit firmware wins even when a kernel is also set. + let mut manifest = minimal(); + manifest.desired.boot.kernel = Some("/tmp/vmlinuz".to_owned()); + manifest.desired.boot.firmware = Some("/custom/fw.fd".to_owned()); + let config = to_vm_config(&manifest).expect("explicit firmware manifest converts"); + assert_eq!(config.payload.firmware.as_deref(), Some("/custom/fw.fd")); + } + #[test] fn converts_networks_for_the_transform_pipeline() { let mut manifest = minimal(); diff --git a/odorobo/src/ch_driver/transform/storage/rbd.rs b/odorobo/src/ch_driver/transform/storage/rbd.rs index c7f90b9..de75192 100644 --- a/odorobo/src/ch_driver/transform/storage/rbd.rs +++ b/odorobo/src/ch_driver/transform/storage/rbd.rs @@ -105,6 +105,8 @@ impl RbdImage { .args(rbd_extra_args()) .arg("device") .arg("map") + .arg("--options") + .arg("noudev") .arg(&rbd_path) .output() .await @@ -125,6 +127,8 @@ impl RbdImage { .args(rbd_extra_args()) .arg("device") .arg("unmap") + .arg("--options") + .arg("noudev") .arg(&rbd_path) .output() .await From 39543f7f0eaacddb5ec41850b019d6e2d7b69692 Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Mon, 21 Sep 2026 03:36:49 -0500 Subject: [PATCH 7/8] cargo fmt --- odorobo/src/ch_driver/manifest.rs | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/odorobo/src/ch_driver/manifest.rs b/odorobo/src/ch_driver/manifest.rs index aac0c33..5cf4715 100644 --- a/odorobo/src/ch_driver/manifest.rs +++ b/odorobo/src/ch_driver/manifest.rs @@ -72,22 +72,18 @@ pub fn to_vm_config(manifest: &VmManifest) -> Result { ..Default::default() }), payload: PayloadConfig { - firmware: desired - .boot - .firmware - .clone() - .or_else(|| { - // Only default to the firmware when doing a firmware boot - // (no kernel specified). Direct kernel boot must not set a - // firmware, or Cloud Hypervisor rejects the config with - // "Specifying a kernel is not supported when a firmware is - // provided". - if desired.boot.kernel.is_some() { - None - } else { - Some("/var/lib/odorobo/CLOUDHV.fd".to_owned()) - } - }), + firmware: desired.boot.firmware.clone().or_else(|| { + // Only default to the firmware when doing a firmware boot + // (no kernel specified). Direct kernel boot must not set a + // firmware, or Cloud Hypervisor rejects the config with + // "Specifying a kernel is not supported when a firmware is + // provided". + if desired.boot.kernel.is_some() { + None + } else { + Some("/var/lib/odorobo/CLOUDHV.fd".to_owned()) + } + }), kernel: desired.boot.kernel.clone(), cmdline: desired.boot.cmdline.clone(), ..Default::default() From 54e92e4c240c1f8d72f940c66a1071ae77150cf0 Mon Sep 17 00:00:00 2001 From: Caleb Jones Date: Mon, 21 Sep 2026 03:39:10 -0500 Subject: [PATCH 8/8] e2e test script --- .local/dev/TEST.md | 103 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 .local/dev/TEST.md diff --git a/.local/dev/TEST.md b/.local/dev/TEST.md new file mode 100644 index 0000000..b44439b --- /dev/null +++ b/.local/dev/TEST.md @@ -0,0 +1,103 @@ +# End-to-end test: boot a VM from a Ceph RBD disk + +Boots a minimal VM through Odorobo whose root disk is an `rbd://` image in the +in-container Ceph cluster. It proves the `rbd://` storage path end to end: the +agent maps the RBD to a kernel block device, resolves the URI, and hands the disk +to Cloud Hypervisor. + +Step 0 runs from the host; steps 1–6 run inside the `odorobo` container. + +## Step 0 — Start the stack and enter the container + +```sh +sudo bash .local/dev/init.sh +sudo podman compose -f .local/dev/compose.yml exec -it odorobo sh +``` + +## Step 1 — Install required packages + +```sh +dnf install -y busybox tar +``` + +## Step 2 — Re-fetch the Alpine kernel + +```sh +cd /tmp +curl -O https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/linux-virt-6.12.110-r0.apk +tar xOf linux-virt-6.12.110-r0.apk boot/vmlinuz-virt > /tmp/vmlinuz-virt +ls -l /tmp/vmlinuz-virt +``` + +## Step 3 — Build the rootfs + +```sh +mkdir -p /tmp/rootfs/{bin,dev,etc,proc,sys} +cp /usr/bin/busybox /tmp/rootfs/bin/ +printf '%s\n' \ + '#!/bin/sh' \ + 'mount -t proc proc /proc' \ + 'mount -t sysfs sys /sys' \ + 'echo "=== odorobo VM: booted from ceph-backed rootfs ==="' \ + 'ls /dev | grep vda' \ + 'echo "=== dropping to shell ==="' \ + 'exec /bin/busybox ash' > /tmp/rootfs/init +chmod +x /tmp/rootfs/init +for a in ash sh ls cat mount echo grep; do ln -s bin/busybox /tmp/rootfs/$a; done +fallocate -l 512M /tmp/rootfs.img +mkfs.ext4 /tmp/rootfs.img +mkdir -p /tmp/mnt && mount /tmp/rootfs.img /tmp/mnt +cp -a /tmp/rootfs/. /tmp/mnt/ +umount /tmp/mnt +``` + +## Step 4 — Map the RBD + +A reset re-creates the image with a new object id, so any mapping that survived in +the kernel is stale. Unmap it first, or the agent's "already mapped?" check will +reuse a broken device. + +```sh +RBD="rbd --conf=/workspace/.local/dev/ceph/generated/ceph.conf --id=odorobo --keyfile=/workspace/.local/dev/ceph/generated/client.odorobo.key" + +# Clear any stale mappings that survived the reset +$RBD device unmap /dev/rbd0 --options noudev 2>/dev/null +$RBD device unmap /dev/rbd1 --options noudev 2>/dev/null +$RBD device ls # confirm the table is empty + +# Map the fresh image +$RBD device map odorobo-blockpool/dev-disk --options noudev; echo "map-exit=$?" +ls -l /dev/rbd* 2>/dev/null + +# Gate: the agent's map() runs `rbd device list` first, so it must succeed +$RBD device list; echo "list-exit=$?" +``` + +Expect `map-exit=0`, a `brw-... /dev/rbd0` line, and `list-exit=0`. Confirm before +continuing. + +## Step 5 — Write the rootfs to the RBD + +```sh +dd if=/tmp/rootfs.img of=/dev/rbd/odorobo-blockpool/dev-disk bs=1M +sync +``` + +## Step 6 — Boot the VM + +```sh +bash /workspace/.local/dev/vm-test.sh +``` + +The script health-checks the agent, posts the VM manifest, waits, and dumps the +console history. + +Expected console: kernel boot log → `=== odorobo VM: booted from ceph-backed +rootfs ===` → `vda` listed → shell prompt. Then run `cat /proc/mounts` and confirm +`/dev/vda / ext4` — that is the VM reading from the Ceph RBD pool. + +> **Note:** the stock Alpine `linux-virt` kernel ships `ext4` and `virtio_blk` as +> modules, so a direct kernel boot (no initramfs) panics at +> `VFS: Unable to mount root fs on "/dev/vda"`. That is a test-kernel limitation, +> not a Ceph issue — the `rbd://` resolution and disk attach are proven. To get a +> full shell, use a kernel with `ext4`/`virtio_blk` built in.