feat: add ceph dev env

.gitignore

@@ -12,6 +12,15 @@ dev
 *.pdb
 
 *.yaml
+# The local development container stack is checked in, despite the broad `dev`
+# scratch-directory rule above.
+!.local/
+!.local/dev/
+!.local/dev/**
+
+# Local Ceph credentials and loopback OSD state.
+.local/dev/ceph/generated/
+.local/dev/ceph/state/
 
 # Generated by cargo mutants
 # Contains mutation testing data

.local/dev/README.md

@@ -0,0 +1,126 @@
+# Local Ceph and Odorobo with Compose
+
+This directory runs the local development stack in containers:
+
+- `ceph` provides a single-node Ceph cluster and a file-backed OSD.
+- `odorobo` runs the agent, sharing Ceph's network and PID namespaces and the host's `/dev`.
+
+Odorobo must run in the container for the `rbd://` storage path. It invokes `rbd device map` using the generated Ceph credentials, which creates a kernel block device, and then passes that device to Cloud Hypervisor. The container provides the credential files, the privileged device access, and the shared namespaces that a host-side process would have to replicate.
+
+This is intended for Linux development with a rootful container engine. The stack uses privileged containers because kernel RBD mapping, Cloud Hypervisor, networking, and Ceph's daemon management require host kernel access.
+
+## Prerequisites
+
+Install Podman with a working Compose provider. Docker Compose v2 is supported as a fallback.
+
+For Fedora, the host needs the container engine and kernel modules:
+
+```bash
+sudo dnf install -y podman podman-compose kmod
+sudo modprobe rbd loop
+sudo losetup -f
+```
+
+`podman compose` prefers the `docker-compose` plugin when it is installed, and that provider talks to Podman's API socket, which must be running (`systemctl --user enable --now podman.socket` for rootless, `sudo systemctl enable --now podman.socket` for rootful). If you hit `failed to connect to the docker API at unix:///run/user/<uid>/podman/podman.sock`, either enable that socket or select the standalone tool as the provider: `compose_providers = ["podman-compose"]` in `~/.config/containers/containers.conf` (or `export PODMAN_COMPOSE_PROVIDER=podman-compose`).
+
+The stack must run on a **rootful** engine. Rootless Podman cannot work: the Ceph container attaches the OSD file through a host loop device, and the kernel's loop driver requires `CAP_SYS_ADMIN` in the initial user namespace, which a rootless container never has (even `privileged` + a `/dev` bind mount do not help). Run everything rootful, e.g. `sudo bash .local/dev/init.sh`, and prefix the `podman compose` commands below with `sudo` accordingly.
+
+The Ceph image is based on the official `quay.io/ceph/ceph` image and starts the MON and OSD daemons itself; it intentionally does not start MGR because the MGR's optional Python modules require host udev/system services unavailable in this container. It does not use `cephadm`, nested Podman, or systemd. The OSD uses a persistent raw file attached through a host loop device, initialized directly with `ceph-osd` rather than `ceph-volume`.
+
+## Usage
+
+Initialize Ceph and start Odorobo:
+
+```bash
+sudo bash .local/dev/init.sh
+```
+
+This builds both images, starts Ceph and waits up to two minutes for it to become healthy, provisions the `odorobo-blockpool/dev-disk` RBD image, and then starts Odorobo with manager mode enabled. On a bootstrap failure, it prints the last 200 Ceph log lines instead of waiting indefinitely. It prefers `podman compose` and falls back to `docker compose` when Podman Compose is unavailable.
+
+Start and stop the complete stack without deleting data. `start` only works on existing containers; after a reset, run `init.sh` again:
+
+```bash
+sudo podman compose -f .local/dev/compose.yml start ceph odorobo
+sudo podman compose -f .local/dev/compose.yml stop odorobo ceph
+```
+
+Destructively remove the containers and all local Ceph state. 
+
+```bash
+sudo podman compose -f .local/dev/compose.yml down --remove-orphans --volumes
+sudo rm -rf .local/dev/ceph/generated .local/dev/ceph/state
+```
+
+Useful direct commands:
+
+```bash
+sudo podman compose -f .local/dev/compose.yml ps
+sudo podman compose -f .local/dev/compose.yml logs -f ceph odorobo
+sudo podman compose -f .local/dev/compose.yml exec ceph ceph -s
+sudo podman compose -f .local/dev/compose.yml exec -it odorobo sh
+```
+
+Because `odorobo` uses Ceph's network namespace, the generated Ceph config intentionally uses `127.0.0.1` for the monitor. The application and monitor share that namespace.
+
+## Application development
+
+The repository is mounted at `/workspace` in the Odorobo container. Rebuild and restart the application after source changes:
+
+```bash
+podman compose -f .local/dev/compose.yml build odorobo
+podman compose -f .local/dev/compose.yml up -d odorobo
+podman compose -f .local/dev/compose.yml logs -f odorobo
+```
+
+By default, the container limits Cargo to two concurrent build jobs to reduce CPU and memory pressure during the initial release build. Override it when starting the stack, for example `CARGO_BUILD_JOBS=4 bash .local/dev/init.sh`.
+
+The agent runs as:
+
+```text
+cargo run --release -p odorobo -- --manager-enabled true
+```
+
+Its runtime directory is shared through `/run/odorobo`, and Cloud Hypervisor processes and RBD devices are visible in the same namespaces as the agent.
+
+## Verify the image
+
+Run Ceph commands inside the Ceph container:
+
+```bash
+podman compose -f .local/dev/compose.yml exec ceph rbd \
+  --conf=/etc/ceph/ceph.conf --id=odorobo \
+  --keyfile=/var/lib/odorobo-ceph/client.odorobo.key \
+  ls --pool odorobo-blockpool
+```
+
+Expected output includes `dev-disk`. A one-node cluster may report `HEALTH_WARN`; reduced redundancy, a single monitor, and no running MGR are expected for local development.
+
+Do not map the image from the host. To test the exact application path, use an Odorobo manifest with `rbd://odorobo-blockpool/dev-disk`; the `odorobo` service will execute `rbd device map` and pass the resulting device to Cloud Hypervisor.
+
+## Configuration
+
+The following environment variables can be set before `init.sh` or passed through Compose:
+
+```bash
+CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3
+CEPH_MON_IP=127.0.0.1
+CEPH_POOL=odorobo-blockpool
+CEPH_CLIENT=odorobo
+CEPH_IMAGE_NAME=dev-disk
+CEPH_IMAGE_SIZE=1G
+CEPH_OSD_SIZE=10G
+```
+
+`CEPH_MON_IP` should remain `127.0.0.1` with the provided Compose topology. If you change the network topology, it must be an address reachable from both services.
+
+## Layout
+
+- `init.sh` — builds and starts the stack, waits for Ceph health, and reports bootstrap failures with logs.
+- `compose.yml` — Ceph and Odorobo services, shared namespaces, privilege, mounts, and ports.
+- `ceph/Containerfile` — pinned Ceph image.
+- `ceph/entrypoint.sh` — direct MON bootstrap, filesystem-backed OSD initialization, pool/client/image provisioning, and daemon lifecycle.
+- `odorobo/Containerfile` — runnable Odorobo development image.
+- `ceph/generated/` — generated Ceph credentials shared read-only with Odorobo; ignored by git.
+- `ceph/state/` — Ceph configuration, daemons, logs, and file-backed OSD; ignored by git.
+
+This setup is intentionally not production-ready: it has one monitor, one OSD, no redundancy, and privileged containers.

.local/dev/TEST.md

@@ -0,0 +1,103 @@
+# End-to-end test: boot a VM from a Ceph RBD disk
+
+Boots a minimal VM through Odorobo whose root disk is an `rbd://` image in the
+in-container Ceph cluster. It proves the `rbd://` storage path end to end: the
+agent maps the RBD to a kernel block device, resolves the URI, and hands the disk
+to Cloud Hypervisor.
+
+Step 0 runs from the host; steps 1–6 run inside the `odorobo` container.
+
+## Step 0 — Start the stack and enter the container
+
+```sh
+sudo bash .local/dev/init.sh
+sudo podman compose -f .local/dev/compose.yml exec -it odorobo sh
+```
+
+## Step 1 — Install required packages
+
+```sh
+dnf install -y busybox tar
+```
+
+## Step 2 — Re-fetch the Alpine kernel
+
+```sh
+cd /tmp
+curl -O https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/linux-virt-6.12.110-r0.apk
+tar xOf linux-virt-6.12.110-r0.apk boot/vmlinuz-virt > /tmp/vmlinuz-virt
+ls -l /tmp/vmlinuz-virt
+```
+
+## Step 3 — Build the rootfs
+
+```sh
+mkdir -p /tmp/rootfs/{bin,dev,etc,proc,sys}
+cp /usr/bin/busybox /tmp/rootfs/bin/
+printf '%s\n' \
+  '#!/bin/sh' \
+  'mount -t proc proc /proc' \
+  'mount -t sysfs sys /sys' \
+  'echo "=== odorobo VM: booted from ceph-backed rootfs ==="' \
+  'ls /dev | grep vda' \
+  'echo "=== dropping to shell ==="' \
+  'exec /bin/busybox ash' > /tmp/rootfs/init
+chmod +x /tmp/rootfs/init
+for a in ash sh ls cat mount echo grep; do ln -s bin/busybox /tmp/rootfs/$a; done
+fallocate -l 512M /tmp/rootfs.img
+mkfs.ext4 /tmp/rootfs.img
+mkdir -p /tmp/mnt && mount /tmp/rootfs.img /tmp/mnt
+cp -a /tmp/rootfs/. /tmp/mnt/
+umount /tmp/mnt
+```
+
+## Step 4 — Map the RBD
+
+A reset re-creates the image with a new object id, so any mapping that survived in
+the kernel is stale. Unmap it first, or the agent's "already mapped?" check will
+reuse a broken device.
+
+```sh
+RBD="rbd --conf=/workspace/.local/dev/ceph/generated/ceph.conf --id=odorobo --keyfile=/workspace/.local/dev/ceph/generated/client.odorobo.key"
+
+# Clear any stale mappings that survived the reset
+$RBD device unmap /dev/rbd0 --options noudev 2>/dev/null
+$RBD device unmap /dev/rbd1 --options noudev 2>/dev/null
+$RBD device ls   # confirm the table is empty
+
+# Map the fresh image
+$RBD device map odorobo-blockpool/dev-disk --options noudev; echo "map-exit=$?"
+ls -l /dev/rbd* 2>/dev/null
+
+# Gate: the agent's map() runs `rbd device list` first, so it must succeed
+$RBD device list; echo "list-exit=$?"
+```
+
+Expect `map-exit=0`, a `brw-... /dev/rbd0` line, and `list-exit=0`. Confirm before
+continuing.
+
+## Step 5 — Write the rootfs to the RBD
+
+```sh
+dd if=/tmp/rootfs.img of=/dev/rbd/odorobo-blockpool/dev-disk bs=1M
+sync
+```
+
+## Step 6 — Boot the VM
+
+```sh
+bash /workspace/.local/dev/vm-test.sh
+```
+
+The script health-checks the agent, posts the VM manifest, waits, and dumps the
+console history.
+
+Expected console: kernel boot log → `=== odorobo VM: booted from ceph-backed
+rootfs ===` → `vda` listed → shell prompt. Then run `cat /proc/mounts` and confirm
+`/dev/vda / ext4` — that is the VM reading from the Ceph RBD pool.
+
+> **Note:** the stock Alpine `linux-virt` kernel ships `ext4` and `virtio_blk` as
+> modules, so a direct kernel boot (no initramfs) panics at
+> `VFS: Unable to mount root fs on "/dev/vda"`. That is a test-kernel limitation,
+> not a Ceph issue — the `rbd://` resolution and disk attach are proven. To get a
+> full shell, use a kernel with `ext4`/`virtio_blk` built in.

.local/dev/ceph/.dockerignore

@@ -0,0 +1,2 @@
+state/
+generated/

.local/dev/ceph/Containerfile

@@ -0,0 +1,9 @@
+ARG CEPH_IMAGE=quay.io/ceph/ceph:v20.2.3
+FROM ${CEPH_IMAGE}
+
+USER root
+
+COPY entrypoint.sh /usr/local/bin/odorobo-ceph
+RUN chmod 0755 /usr/local/bin/odorobo-ceph
+
+ENTRYPOINT ["/usr/local/bin/odorobo-ceph"]