Skip to content

About

Collection of scripts to build and provision debian images for docker and libvirt

Resources

Readme
GPL-3.0 license
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository
BranchesTags

Latest commit

 

History

23 Commits

Folders and files

Repository files navigation

Debian Image Builder

This repository provides a collection of scripts as proof of concept for scripted

  • Generation of debian base images for docker and libvirt
  • Generation/Cloning/Provisioning of VMs

They are meant to document the result of learning and experimenting with the kvm/libvirt stack. In combination with the collection of recipes/links in the appendix, they might provide a good introduction into the topics.

Table of Contents

Goals

  • Usage of commands/scripts instead of GUIs
  • Usage of debuerreotype to gain the advantage of reproducibility
  • Generation of as similiar base images for docker/VMs as possible
  • Generation of as minimal VM templates as possible for later provisioning with ansible
  • Easy creation of test environments for ansible role development
  • Foundation for automated build/deploy infrastructure for docker images and VMs
  • Minimization of dependencies and neccessary trust

Tools

  • debian: The universal operating system
    • debootstrap: Main bootstrap tool for debian rootfs builds
    • debuerreotype: Reproducible debian rootfs builds
      • Wrapper for debootstrap
      • Installs packages from arbritary snapshots in time
      • Creates disc space optimized minbase images (with docker in mind) used in the official debian docker hub repository.
      • Docker specific adjustments can be reverted (with VMs in mind)
    • apt-cacher-ng: Cache for apt / apt-get packages (saves time and energy)
  • docker (Docker, Inc.): Container backend
  • libvirt (RedHat): VM backend, integrates kvm and qemu
    • virsh: Main console CLI for libvirt VMs
  • virt-manager (RedHat): Set of helper tools for managing VM lifecycles
  • libguestfs-tools (RedHat): Set of helper tools for handling VM filesystems
  • ansible (RedHat): Provisions any machine

Features

Ansible:

The provided ansible playbook is an example for both basic debian configuration and flavour. It includes tasks for

  • system
    • environment (set stage info)
    • host (set hostname and /etc/hosts)
    • network (configure interfaces)
    • packages (configure apt proxy, update/upgrade/reboot, install/remove packages)
    • locales (set timezone/LANG)
    • users (add/configure admin users, add/remove ssh keys)
    • terminal (add terminfos)
    • shell (add zsh, add grml/local config)
  • sshd
    • configuration (sane default)
    • motd (stats)

Security:

The provided scripts bootstrap everything with debootstrap and a keyring as root of trust. This keyring is generated and verified by

  • A pinned sign key for the keyring package
  • A pinned signed package dsc file for the keyring package
  • The sha256sum for the keyring package included in the dsc file
  • Additional checks during the make process of the keyring package

Other considerations:

  • Debuerreotype generates the widely used official docker hub debian images. They are not official debian releases, but debuerreotype is maintained by two debian maintainers and they are reproducible.
  • The reproducibility can be checked via shasum for each build in ./scripts/debuerreotype/builds/build.sha256.log (buildpath, shasum for included/excluded packages, shasums for normal/slim build)
  • The debuerreotype build folder is mounted into the debuerreotype container (defaults to the repository folder)
  • The debuerreotype image will add capacities SYS_ADMIN and drop SETFCAP in order to be able to fuse mount.
  • The initial libvirt VM password is passed via parameter
  • The provided sshd/ssh keys are obviously not secure and should not be used beyond testing

Todo:

  • Add swap partition to libvirt storages
  • Add lvg pool capability
  • Add libvirt pool name as subfolder in ./scripts/libvirt/image/NAME (or omit the whole intermediate image step)
  • Build and clone libvirt VM templates
  • Better handling of different libvirt networks
  • Bootstrap a builder VM and try to replace/trigger bash scripts with an ansible role
  • Try to use debuerreotype scripts without docker environment and examples/debian.sh (better parametrization, omit unneccessary builds)
  • Remove verification of sha512sum of keyring package (now that the dsc file and sign key is pinned)
  • Add generation script for ssh/sshd keys

Files

.
├── docs/                              - rendered documentation
│   ├── build.sh                       - script to render README.rst as HTML
│   ├── styles.css                     - stylesheet for documentation
│   └── README.html                    - rendered README.rst
│
├── tests/                             - test scripts
│   ├── SUITE-docker.sh                - docker test (build/import/run)
│   ├── SUITE-libvirt.sh               - libvirt test (build/connect)
│   └── SUITE-ansible.sh               - ansible test (build/provision/connect)
│
├── scripts/                           - script collection
│   ├── apt-cacher-ng/                 - apt cacher ng scripts
│   │   ├── cache/                     - cache folder mounted in the container
│   │   ├── log/                       - log folder mounted in the container
│   │   │
│   │   ├── Dockerfile                 - Dockerfile for apt-cacher-ng service
│   │   ├── entrypoint.sh              - docker entrypoint
│   │   │
│   │   ├── build.sh                   - build image
│   │   ├── start.sh                   - start service
│   │   ├── stop.sh                    - stop service
│   │   └── clean.sh                   - clean artifacts
│   │
│   ├── debootstrap/                   - debootstrap scripts
│   │   ├── upstream/                  - debootstrap subrepo (upstream)
│   │   ├── cache/                     - cache folder mounted in the chroot
│   │   ├── keyrings/                  - downloaded keyrings
│   │   │   ├── *.json                 - responses from package api
│   │   │   ├── *.sha512               - checksums for keyring packages
│   │   │   ├── *.gpg                  - sign keys for dsc files
│   │   │   └── *.dsc                  - signed keyring package descriptions
│   │   │
│   │   ├── builds/SUITE/rootfs        - generated rootfs
│   │   │
│   │   ├── build.sh                   - build debootstrap rootfs
│   │   └── clean.sh                   - clean artifacts
│   │
│   ├── debuerreotype/                 - debuerrotype scripts
│   │   ├── upstream/                  - debuerreotype subrepo (upstream)
│   │   ├── patches/                   - patches for debuerreotype scripts
│   │   │
│   │   ├── builds/
│   │   │   ├── BACKEND/SERIAL/ARCH/SUITE/
│   │   │   │   ├── slim/rootfs.tar.gz - slim version of rootfs
│   │   │   │   └── rootfs.tar.gz      - rootfs
│   │   │   └── build.sha256.log       - log of reproducible build shasums
│   │   │
│   │   ├── bootstrap.sh               - bootstrap a debuerrotype docker image
│   │   ├── build.sh                   - build debian rootfs
│   │   └── clean.sh                   - clean artifacts
│   │
│   ├── docker/                        - docker scripts
│   │   ├── build.sh                   - build debian docker image
│   │   └── clean.sh                   - clean artifacts
│   │
│   ├── libvirt/                       - libvirt scripts
│   │   ├── image/                     - image directory
│   │   ├── interface/                 - files for interface configuration
│   │   │   ├── dhcp                   - dhcp configuration
│   │   │   └── static                 - static ip configuration
│   │   ├── pool/NAME/                 - pool directory
│   │   ├── ssh/                       - ssh files
│   │   │   ├── libvirtlocal(.pub)     - ssh client test key
│   │   │   └── ssh_host_*_key(.pub)   - ssh server test keys
│   │   ├── xml/                       - generated xml definitions
│   │   │
│   │   ├── create_domain.sh           - create domain
│   │   ├── create_image.sh            - create image
│   │   ├── create_network.sh          - create network
│   │   ├── create_pool.sh             - create pool
│   │   ├── create_volume.sh           - create volume
│   │   │
│   │   ├── build.sh                   - build libvirt VM
│   │   └── clean.sh                   - clean artifacts
│   │
│   └── ansible/                       - ansible playbook
│       ├── group_vars/                - group variables
│       │   └── all.yml                - group variables for all hosts
│       ├── roles/                     - roles
│       │   ├── system/                - basic debian system provisioning
│       │   └── sshd/                  - sshd service configuration
│       │
│       ├── ansible.cfg                - configuration
│       └── debian.yml                 - playbook
│
├── README.rst                         - this readme
├── LICENCE.txt                        - the license for this repository
│
├── .project.sh                        - base script sourced by all other scripts
│
├── build.sh                           - main build script (libvirt + ansible)
└── clean.sh                           - clean artifacts

Installation

Requirements

Install requirements

  • Debian

    # Debian
sudo apt-get install \
    debootstap jetring gnupg2 \
    docker.io \
    libvirt-daemon-system qemu-system \
    virt-manager virt-viewer virtinst \
    libguestfs-tools \
    sipcalc \
    ansible

  • Archlinux

    sudo pacman -S \
    debootstap jetring gnupg \
    docker \
    libvirt qemu-base iptables-nft dnsmasq dmidecode openbsd-netcat \
    virt-manager virt-viewer \
    libguestfs guestfs-tools \
    sipcalc \
    ansible

Checkout this repo and subrepos

git clone <URL>
git submodule update --init --recursive

Add the hosts to /etc/hosts, e.g. for the test scripts

# debian image builder
192.168.100.101             bullseye-libvirt.test
192.168.100.111             bullseye-ansible.test
192.168.100.102             bookworm-libvirt.test
192.168.100.112             bookworm-ansible.test
192.168.100.103             trixie-libvirt.test
192.168.100.113             trixie-ansible.test

Add the ssh key (no passphrase) to your ssh agent

cp ./libvirt/ssh/libvirtlocal* ~/.ssh/
ssh-add ~/.ssh/libvirtlocal

If you use a firewall, adjust the rules for the vm network, e.g. nftables

chain input {
    [...]

    # libvirt
    iifname "virbr-*" ip saddr 192.168.0.0/16 ip daddr 192.168.100.1 \
        accept comment "allow libvirt host communication"
    iifname "virbr-*" ip saddr 192.168.0.0/16 ip daddr 192.168.101.1 \
        accept comment "allow libvirt host communication"
    iifname "virbr-*" ip saddr 10.0.0.0/8 ip daddr 10.63.215.1 \
        accept comment "allow libvirt host communication"
    iifname "virbr-*" ip6 saddr fe80::/10 ip6 daddr fe80::/10 \
        accept comment "allow libvirt host communication"
    iifname "virbr-*" ip protocol udp ip saddr 0.0.0.0 ip daddr 255.255.255.255 \
        accept comment "allow libvirt dns broadcast"

    [...]
    # log
}
chain forward {
    [...]

    # docker
    ip saddr 172.17.0.0/12 ip daddr 172.17.0.0/12 \
        accept comment "allow docker subnet commmunication"
    oifname enp14s0 ip saddr 172.17.0.0/12 \
        accept comment "allow docker internet commmunication out"
    iifname enp14s0 ip daddr 172.17.0.0/12 \
        accept comment "allow docker internet commmunication in"

    # libvirt
    ip saddr 192.168.100.0/24 ip daddr 192.168.0.0/16 \
    accept comment "allow libvirt subnet communication"
    iifname "virbr-*" ip saddr 192.168.0.0/16 \
        accept comment "allow libvirt communication out"
    oifname "virbr-*" ip daddr 192.168.0.0/16 \
        accept comment "allow libvirt communication in"
    iifname "virbr-*" ip saddr 10.0.0.0/8 \
        accept comment "allow libvirt communication out"
    oifname "virbr-*" ip daddr 10.0.0.0/8 \
        accept comment "allow libvirt communication in"

    [...]
    # log
}

Note: Docker might need to be configured to use the currently experimental native ``nftables`` firewall backend, see e.g. here

Build HTML documentation including the synopsis of the scripts (--help)

./docs/build.sh
xdg-open docs/README.html

Usage

docker

Build and run a trixie docker image debian:trixie-slim via test script

./tests/trixie-docker.sh

Build and run a custom trixie docker image my-trixie

./scripts/docker/build.sh trixie my-trixie
docker run --rm -it my-trixie bash

libvirt

Note: By default a libvirt network "debian" in range 192.168.100.0/24 and interface "virt-debian" are created.

Build and run a minimal trixie libvirt VM trixie-libvirt.test on 192.168.100.102 via test script

./tests/trixie-libvirt.sh

Build and run a custom minimal trixie libvirt VM my-trixie-libvirt.test on dhcp:

  1. Run the build script

    ./scripts/libvirt/build.sh --domain-name my-trixie-libvirt.test trixie

  2. Fetch the IP of the VM

    virsh net-dhcp-leases debian

  3. Connect to the VM

    ssh  root@<IP>

  4. Delete the VM

    virsh destroy my-trixie-libvirt.test
virsh undefine my-trixie-libvirt.test
rm ./scripts/libvirt/image/my-trixie-libvirt.test.lvm.ext4.qcow2
rm ./scripts/libvirt/pool/develop/my-trixie-libvirt.test.lvm.ext4.qcow2

ansible

Note: By default a libvirt network "debian" in range 192.168.100.0/24 and interface "virt-debian" are created.

Build and run an ansible provisioned trixie libvirt VM trixie-ansible.test on 192.168.100.112 via test script

./examples/trixie-ansible.sh

Build and run a custom ansible provisioned trixie libvirt VM my-trixie-ansible.test on 192.168.100.50:

  1. Add the host to /etc/hosts

    # debian image builder
192.168.100.50          my-trixie-ansible.test

  2. Run the build script

    ./build.sh --suite trixie my-trixie-ansible.test 192.168.100.50

  3. Connect to the VM

    ssh admin@my-trixie-ansible.test
ssh root@my-trixie-ansible.test

  4. Delete the VM

    virsh destroy my-trixie-ansible.test
virsh undefine my-trixie-ansible.test
rm ./scripts/libvirt/image/my-trixie-ansible.test.lvm.ext4.qcow2
rm ./scripts/libvirt/pool/develop/my-trixie-ansible.test.lvm.ext4.qcow2

Manually run the ansible playbook

ansible-playbook \
    --timeout 100 \
    --inventory my-trixie-ansible.test, \
    --user root \
    --tags setup \
    scripts/ansible/debian.yml

Manually run a system upgrade via the ansible playbook

ansible-playbook \
    --timeout 100 \
    --inventory my-trixie-ansible.test, \
    --user root \
    --tags upgrade \
    scripts/ansible/debian.yml

clean

Clean artifacts (custom domains need to be undefined manually)

# all (calls all scripts below)
./clean.sh

# libvirt
./scripts/libvirt/clean.sh
# docker
./scripts/docker/clean.sh
# debuerreotype
./scripts/debuerreotype/clean.sh
# debootstrap
./scripts/debootstrap/clean.sh

Scripts

Update

To update this repo (e.g. after new suite 'trixie' was released):

  1. Adjust variables and usage in .project.sh:

    # change to current date
debianTimestamp="${DEBIAN_TIMESTAMP:-2026-10-03T00:00:00Z}"

# change to current stable suite
debianDefault="trixie"

# add new suite/version
declare -A debianVersions=(
    [...]
    [trixie]=13
)

  2. Update debootstrap

    1. Update keyring and pin files (.dsc, .sha512, .gpg)

      1. Change into the keyring dir

        cd scripts/debootstrap/keyrings

      2. Check keyring version for new suite

        suite=trixie

src="https://sources.debian.org/api/src/debian-archive-keyring/"
wget -O debian-archive-keyring.$suite.sources.json "$src?suite=$suite"
cat debian-archive-keyring.$suite.sources.json
    # [...]
    #   "version": "2025.1",
    # [...]

version="2025.1"

      3. Download keyring package tarball and dsc (from debian)

        ftp='https://ftp.debian.org/debian/pool/main/d/debian-archive-keyring/'
wget $ftp/debian-archive-keyring_$version.tar.xz
wget $ftp/debian-archive-keyring_$version.dsc

      1. Verify gpg signature of dsc with last pinned key (in clean gnupghome)

        export GNUPGHOME=/tmp/dib-$suite
mkdir $GNUPGHOME; chmod go-rwx $GNUPGHOME
gpg --import debian-archive-keyring_*.gpg
gpg --verify debian-archive-keyring_$version.dsc
    # [...]
    # gpg: Good signature from [...]
    # [...]

      2. Verify checksum of tarball in dsc file

        echo $(grep -zoP "(?<=Sha256:\n ).*?(?= )" \
                 debian-archive-keyring_$version.dsc) \
     debian-archive-keyring_$version.tar.xz | sha256sum -c

      3. Generate new pinned sha512 file

        sha512sum debian-archive-keyring_$version.tar.xz \
    > debian-archive-keyring_$version.tar.xz.sha512

      4. Refresh, reverify and export updated gpg key

        gpg --verify debian-archive-keyring_$version.dsc
    # [...]
    # Primary key fingerprint:
    #   CA61 9D65 A72A 7BAD FC96  D280 1964 18AA EB74 C8A1
    # [...]
gpg --refresh CA619D65A72A7BADFC96D280196418AAEB74C8A1
gpg --verify debian-archive-keyring_$version.dsc
gpg --export --armor CA619D65A72A7BADFC96D280196418AAEB74C8A1 \
    > debian-archive-keyring_$version.gpg

      5. Add pinned files (dsc, sha512, gpg)

        git add *.dsc *.sha512 *.gpg

    2. Update debootstrap upstream

      1. Check upstream repo:
        • Check release notes
        • Check release key

      1. Change into debootstrap upstream dir

        cd scripts/debootstrap/upstream

      2. Verify newest tag signature

        tag="1.0.145"

git pull
export GNUPGHOME=/tmp/dib-$suite
mkdir $GNUPGHOME; chmod go-rwx $GNUPGHOME
gpg --import ../keyrings/debootstrap-bluca.gpg
gpg verify-tag $tag
    # [...]
    # gpg: Good signature from [...]
    # [...]

      3. Update to newest tag

        git checkout $tag

    3. Build the debootstrap rootfs

      1. Change into debootstrap dir

        cd scripts/debootstrap

      2. Build old and new suite

        rm -rf build/bookworm
sudo ./build.sh bookworm
sudo ./build.sh trixie

    4. Commit the changes

  3. Update debuerrotype

    1. Update debuerreotype upstream

      1. Check upstream repo:
        • check release notes
        • check diff

      1. Update to newest tag:

        tag="0.17"

cd scripts/debuerreotype/upstream
git pull
git checkout $tag

    2. Run clean scripts (apt-cacher-ng, debuerreotype)

      ./scripts/apt-cacher-ng/clean.sh
./scripts/debuerreotype/clean.sh

    3. Bootstrap the docker image

      ./scripts/debuerreotype/bootstrap.sh

    4. Build old and new docker image

      ./scripts/debuerreotype/build.sh bookworm
./scripts/debuerreotype/build.sh trixie

    5. Commit the changes

  4. Add new tests

    1. Copy and adjust test scripts

      cd tests
cp bookworm-docker.sh trixie-docker.sh
    # suite="trixie"
    # image="debian:trixie-slim"
cp bookworm-libvirt.sh trixie-libvirt.sh
    # suite=trixie
    # domain=trixie-ansible.test
    # ip="192.168.100.103/24"
cp bookworm-ansible.sh trixie-ansible.sh
    # domain=trixie-ansible.test
    # suite=trixie
    # ip="192.168.100.113/24"

    2. Add test domains in scripts/libvirt/clean.sh

      domains=(
    [...]
    trixie-ansible.test
    trixie-libvirt.test
)

  5. Test docker

    1. Run old and new test script

      sudo ./tests/bookworm-docker.sh
sudo ./tests/trixie-docker.sh

    2. Add fixes if neccessary and commit the changes

  6. Test libvirt

    1. Run clean script

      sudo ./scripts/libvirt/clean.sh

    2. Run old and new test script

      sudo ./tests/bookworm-libvirt.sh
sudo ./tests/trixie-libvirt.sh

    3. Add fixes if neccessary and commit the changes

  7. Test ansible

    1. Run old and new test script

      sudo ./tests/bookworm-ansible.sh
sudo ./tests/trixie-ansible.sh

    2. Add fixes if neccessary and commit the changes

  8. Update examples in README.rst

License

debian-image-builder -- Build/Provision debian images for docker/libvirt
Copyright (C) 2023 Alexander Blum <workispleasure@pleasureiswork.net>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Appendix

debootstrap

Description:

  • generates a basic debian rootfs
  • written in bash
  • has minimal dependencies (wget)

Usage

Generate base rootfs with default host keyring

debootstrap bullseye rootfs

Generate minbase rootfs with custom keyring and cache folder

debootstrap \
    --cache-dir=cache \
    --keyring=debian-keyring.gpg --force-check-gpg \
    --variant=minbase \
    bullseye rootfs

Keyring

Generate keyring

  • Via host package (/usr/share/keyrings/debian-keyring.gpg)

    apt-get install debian-keyring

  • Via rsync (no checksums/signature, but up-to-date)

    rsync -az --progress keyring.debian.org::keyrings/keyrings/ .

  • Via debian archive (choose package for suite)

    mirror=http://deb.debian.org/
keyring=debian-keyring_2021.07.26

# Fetch package
wget -q $mirror/debian/pool/main/d/debian-keyring/$keyring.tar.gz

# Compare with prepared pinned checksum
sha512sum --quiet --strict --check $keyring.tar.gz.sha512

# Generate keyring and verify signature
tar -xf $keyring.tar.gz
cd ${keyring//_/-}
make --silent -j1

Links

debuerreotype

Description:

  • provides a set of scripts to generate and optimize reproducible rootfs builds, described in its README
  • provides generic example build scripts for debian, ubuntu and others
  • provides a Dockerfile and docker run script to execute the build scripts within a clean build environment
  • written in bash
  • depends on
    • debootstrap in an executable path
    • dpkg in an executable path (so it needs debian as environment)

Usage

Note: For a list of all arguments/flags, execute the scripts with the ``--help`` flag.

Minimal example run on host to build a current snapshot trixie

debuerreotype-init /tmp/rootfs trixie now
debuerreotype-debian-sources-list /tmp/rootfs trixie
debuerreotype-tar /tmp/rootfs rootfs.tar.gz

Minimal example run on host to build a certain snapshot of trixie

debuerreotype-init /tmp/rootfs trixie 2023-03-01T00:00:00Z
debuerreotype-debian-sources-list /tmp/rootfs trixie
debuerreotype-tar /tmp/rootfs rootfs.tar.gz

Minimal example run on host to build a non-snapshot trixie

debuerreotype-init --non-debian /tmp/rootfs trixie http://debian.org/debian
debuerreotype-debian-sources-list /tmp/rootfs trixie
debuerreotype-tar /tmp/rootfs rootfs.tar.gz

Minimal snapshot example run in docker to build a specific snaphot of trixie

./docker-run.sh sh -euxc "
    debuerreotype-init /tmp/rootfs trixie 2023-03-01T00:00:00Z;
    debuerreotype-debian-sources-list /tmp/rootfs trixie;
    debuerreotype-tar /tmp/rootfs rootfs.tar.xz
"

Execution of debian example file in docker

./docker-run.sh sh -euxc "
    ./debuerreotype/examples/debian.sh builds trixie 2023-03-01T00:00:00Z
"

The gist of the debian example file

debuerreotype-init [...] "$rootfsDir" "$suite" "@$epoch"
debuerreotype-debian-sources-list [...] --snapshot "$rootfsDir" "$suite"
debuerreotype-minimizing-config "$rootfsDir"
debuerreotype-apt-get "$rootfsDir" update -qq
debuerreotype-apt-get "$rootfsDir" full-upgrade -yqq
debuerreotype-slimify "$rootfsDir"-slim
debuerreotype-debian-sources-list [...] "$rootfs" "$suite"
debuerreotype-tar [...] "$rootfs" "$targetBase.tar.xz"

Cache

Use apt-cacher-ng

# using docker image for builds on host/docker
proxyIp="$(docker exec $cacheImageName sh -c "hostname --ip-address")"
export http_proxy="http://$proxIp:3142"

# using local service for builds on host/docker
export http_proxy="http://127.0.0.1:3142"

Export

Export into docker

xz -cd rootfs.tar.gz | docker import - debian:trixie-slim

Export into qcow2

virt-make-fs --format=qcow2 --size=1G --partition=mbr --type=ext4 \
    rootfs.tar image.qcow2

Checksums

Verify checksums

  • folder -> tar

    echo "sha256:$(tar -cC rootfs . | sha256sum - | cut -d' ' -f1)"

  • tar

    echo "sha256:$(sha256sum rootfs.tar | cut -d' ' -f1)"

  • tar.gz -> tar

    echo "sha256:$(xz -cd rootfs.tar.gz | sha256sum - | cut -d' ' -f1)"

  • docker first layer = tar

    docker image inspect --format "{{index (.RootFS.Layers) 0}}" \
    debian:trixie-slim

  • vm filesystem -> tar

    mkdir rootfs
touch --no-dereference --date="@$(date -r image.qcow2 +%s)" rootfs
virt-tar-out -a image.qcow2 / rootfs.indeterministic.tar
tar --extract --file rootfs.indeterministic.tar --directory rootfs
tar --create --file rootfs.deterministic.tar --auto-compress \
    --directory rootfs --exclude "lost+found" --numeric-owner \
    --transform 's,^./,,' --sort name .
echo "sha256:$(sha256sum rootfs.deterministic.tar | cut -d' ' -f1)"

Diff tarballs

diffoscope source.tar target.tar

Differences

  • debuerrotype specific
    • hostname: empty or set to debuerreotype (random on docker start)
    • resolve.conf: 1.1.1.1 / 1.0.0.1 (host settings on docker start)
    • during bootstrap: apt with Acquire::Check-Valid-Until=false
  • docker specific
    • minimize config
      • [+] /sbin/initctl -> exit 0 (only for upstart)
      • [+] /usr/sbin/policy-rc.d -> exit 101
      • [+] /etc/dpkg/dpkg.cfg.d/docker-apt-speedup
      • [+] /etc/apt/apt.conf.d/docker-autoremove-suggests
      • [+] /etc/apt/apt.conf.d/docker-clean
      • [+] /etc/apt/apt.conf.d/docker-gzip-indexes
      • [+] /etc/apt/apt.conf.d/docker-no-languages
    • slimify
      • [+] /etc/dpkg/dpkg.cfg.d/docker
      • [-] /usr/share/man/man[0-9]
      • [-] /var/log/dpkg.log
      • [-] /var/log/bootstrap.log
      • [-] /var/log/alternatives.log
      • [-] /var/cache/ldconfig/aux-cache
      • [-] also see files .slimify-excludes|includes
  • revert the changes
    • set hostname
    • set resolve.conf
    • delete
      • /usr/sbin/policy-rc.d
      • /etc/dpkg/dpkg.cfg.d/docker*
      • /etc/apt/apt.conf.d/docker*
    • update (to regenerate man files)
  • sizes
    • docker image: 81M (tar.xz)
    • libvirt qcow2: 985M

Links

libvirt

Description:

  • is a toolkit to manage virtualization platforms
  • supports KVM, Hypervisor.framework, QEMU, Xen, Virtuozzo, VMWare ESX, LXC, BHyve and more
  • written in C, bindings in many languages (python, C#, ...)

Services:

  • libvirtd.service
  • virtlogd.service

Paths:

  • config
    • /etc/libvirt
    • /var/lib/libvirt
  • network
    • /etc/libvirt/<hypervisor>/networks/
  • storage
    • /var/lib/libvirt/images/ (system)
    • ~/VirtualMachines (session)
  • hooks
    • /etc/libvirt/hooks
  • logs
    • /var/log/libvirt/qemu
    • /var/log/libvirt/libvirtd.log

Logging:

vi /etc/libvirt/libvirtd.conf
    log_level = 1
    log_outputs="1:file:/var/log/libvirt/libvirtd.log"
service libvirtd restart

Connection

# virsh @ local system
virsh -c qemu:///system

# virsh @ local session
virsh -c qemu:///session

# virsh @ remote system
virsh -c qemu+ssh://<user>@<host>/system

# virsh @ remote system with debugging
[LIBVIRT_DEBUG=1] virsh -c qemu+ssh://<user>@<host>/system

# virt-viewer @ remote system
virt-viewer  --connect qemu+ssh://<user>@<host>/system domain

# virt-viewer @ remote system
virt-manager --connect qemu+ssh://<user>@<host>/system domain

Enable VNC:

virsh shutdown DOMAIN
virsh edit DOMAIN
    <devices>
      [...]
      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0|127.0.0.1'>
        <listen type='address' address='0.0.0.0|127.0.0.1'/>
      </graphics>
      [...]
    </devices>

Enable Console:

virsh shutdown DOMAIN
virsh edit DOMAIN
    <devices>
      [...]
      <console type='pty'>
        <target type='serial' port='0'/>
      </console>
      [...]
    </devices>

virsh

# host
nodeinfo

# info
list [--all]
dominfo DOMAIN
console DOMAIN [--force]
    Strg+5 exit
    Ctrl+] host

# management
define XMLFILE
create XMLFILE
migrate DOMAIN URI
save DOMAIN FILE
restore FILE

# lifecycle
start DOMAIN
autostart [--disable] DOMAIN
reboot DOMAIN
shutdown DOMAIN
suspend DOMAIN
resume DOMAIN
destroy DOMAIN
undefine DOMAIN --remove-all-storage

# configuration
dumpxml DOMAIN
edit DOMAIN
setmem DOMAIN SIZE
setcpus DOMAIN AMOUNT
attach-disk DOMAIN SOURCE TARGET [--persistent]
detach-disk DOMAIN TARGET [--persistent] [--live]

# volumes
pool-list [--all]
pool-refresh POOL
pool-create-as DOMAIN TYPE [SRCHOST] [SRCPATH] [SRCDEV] [SRCNAME] [TARGET]
pool-define-as DOMAIN TYPE [SRCHOST] [SRCPATH] [SRCDEV] [SRCNAME] [TARGET]
pool-build POOL
pool-start POOL
pool-autostart POOL
pool-undefine POOL
vol-list POOL
vol-create-as POOL FILE SIZE
vol-delete IMAGE [--pool POOL]
vol-wipe IMAGE

# snapshots
snapshot-list DOMAIN
snapshot-info DOMAIN [--snapshotname SNAPSHOT]
snapshot-create-as DOMAIN [--name NAME] [--description DESCRIPTION]
                          [--disk-only] [--atomic]
snapshot-revert DOMAIN [--snapshotname SNAPSHOT] [--running]
snapshot-delete DOMAIN [--snapshotname SNAPSHOT]

# network
net-list [--all]
net-dumpxml NETWORK
net-define XMLFILE
net-create XMLFILE
net-undefine NETWORK
net-start NETWORK
net-autostart NETWORK
net-destroy NETWORK
net-dhcp-leases NETWORK
domifaddr --source agent DOMAIN (qemo-guest-agent!)

# persist trasient objects created with `*-create`
OBJECTTYPE-dumpxml OBJECTNAME > OBJECTNAME.xml
OBJECTTYPE-define OBJECTNAME.xml

Network

Network bridge with NAT:

# copy xml from default net
virsh net-dumpxml default > br1.xml
# or create new xml
vi br1.xml
    <network>
      <name>br1</name>
      <forward mode='nat'>
        <nat>
          <port start='1024' end='65535'/>
        </nat>
      </forward>
      <bridge name='br1' stp='on' delay='0'/>
      <ip address='192.168.100.1' netmask='255.255.255.0'>
        <dhcp>
          <range start='192.168.100.10' end='192.168.100.100'/>
        </dhcp>
      </ip>
    </network>

# add bridge
virsh net-define br1.xml
virsh net-start br1
virsh net-autostart br1
# check bridge is added
virsh net-list --all
bridge link
ip addr show dev br1
# attach guest
virsh attach-interface --domain DOMAIN --type bridge \
    --source br1 --model virtio --config --live
# check attachment
virsh domiflist DOMAIN

# detach guest
virsh detach-interface --domain DOMAIN --type bridge --mac MAC --config
# check detachment
virsh domiflist DOMAIN
# remove bridge
virsh net-destroy br1
virsh net-undefine br1
# check bridge is removed
virsh net-list --all
bridge link
ip addr show dev br1

Network bridge without NAT:

# add bridge
ip link add name br1 type bridge
ip link set dev br1 up
ip link set eth0 up
ip link set eth0 masterbr1
# check bridge
bridge link

# remove bridge
ip link set eth0 nomaster
ip link set eth0 down
ip link delete br1 type bridge
# check bridge
bridge link

Pool

Create a new pool:

virsh pool-define-as POOL type [source-host] [source-path] [source-dev]
                               [source-name] [<target>] [--source-format format]
virsh pool-define-as POOL dir - - - - /home/username/.local/libvirt/images
virsh pool-define-as POOL fs - -  /dev/vg0/images - mntpoint
virsh pool-build     POOL  # warning: destructive operation
virsh pool-start     POOL
virsh pool-autostart POOL

Links

virt-manager

Description:

  • virt-manager is a desktop user interface for managing virtual machines through libvirt.
  • virt-viewer is a lightweight UI interface for interacting with the graphical display of virtualized guest OS. It can display VNC or SPICE, and uses libvirt to lookup the graphical connection details.
  • virt-install is a command line tool which provides an easy way to provision operating systems into virtual machines.
  • virt-clone is a command line tool for cloning existing inactive guests. It copies the disk images, and defines a config with new name, UUID and MAC address pointing to the copied disks.
  • virt-xml is a command line tool for easily editing libvirt domain XML using virt-install’s command line options.
  • virt-bootstrap is a command line tool providing an easy way to setup the root file system for libvirt-based containers.

virt-manager

virt-manager --connect qemu:///system
virt-manager --connect qemu+ssh://<user>@<host>/system domain

virt-viewer

virt-viewer  --connect qemu:///system
virt-viewer  --connect qemu+ssh://<user>@<host>/system domain

virt-install

# notation
--option
    example1
    example2

# minimum
--name
--memory
--disk|filesystem|nodisks

# host
--connect URI
    qemu:///system
    qemu:///session

# domain
--name DOMAIN
    test
--cpu CPU
    host
--virt-type TYPE
    kvm
--memory/--ram SIZE
    4096
--vcpus AMOUNT[,OPTIONS]
    2
    2,maxvcpus=4
--os-type TYPE
    Linux
--os-variant VARIANT
    debian11
    # fetch via
    # $osinfo-query os --fields=short-id \
    #     vendor="Debian Project" codename="trixie"
--initrd-inject=FILE
    preseed.cfg
--extra-args=KERNELOPTIONS
    auto=true hostname=HOSTNAME domain=DOMAIN console=tty0 \
        console=ttyS0,115200n8 serial
    console=ttyS0,115200n8 serial
--hvm

# boot
--boot DEVICES
    cdrom,hd
    uefi,bootmenu.enable=on,bios.useserial=on

# inject
--initrd-inject=FILE
    ks.cfg
+ --extra-args="ks=file:/ks.cfg ..."

# storage
--location URL
    https://d-i.debian.org/daily-images/amd64/
    /home/debian-11.0.0-amd64-DVD-1.iso
--disk FILE[,OPTIONS]
    image.qcow2,device=disk,bus=virtio,format=qcow2
    image.img,device=disk,bus=virtio,size=10,format=qcow2
    size=16,path=image.img,bus=virtio,cache=none
    path=image.img,size=20
    pool=testing,size=4
    size=8
--disk PATH
+ --import

# network
--network NETWORK[,OPTIONS]
    network=default,model=virtio
    bridge=br0,mac=MAC,model=virtio
    bridge=br0
    user

# periphery
--controller CONTROLLER[,OPTIONS]
    usb,model=none

# connections
--graphics GRAPHICS[,OPTIONS]
    none
    vnc,listen=0.0.0.0
--console CONSOLE[,OPTIONS]
    pty,target_type=serial
    pty,target_type=serial -x 'console=ttyS0,115200n8 serial'
--noautoconsole

virt-clone

virsh shutdown SOURCEDOMAIN
sudo virt-clone \
    --connect=qemu:///system \
    --original SOURCEDOMAIN \
    --name TARGETDOMAIN \
    --file /var/lib/libvirt/images/IMAGE.qcow2
ls /var/lib/libvirt/images
virsh list --all
vi /etc/libvirt/qemu/TARGETDOMAIN.xml
    <channel type='unix'>
        - <source mode='bind'
        -         path='.../domain-SOURCEDOMAIN/org.qemu.guest_agent.0'/>
        + <source mode='bind'
        +         path='.../domain-TARGETDOMAIN/org.qemu.guest_agent.0'/>
virsh start TARGETDOMAIN --console
> uuidgen eth0
> vim /etc/sysconfig/network-scripts/ifcfg-eth0
      - UUID=<OLDUUID>
      + UUID=<NEWUID>
> service network restart
> systemctl restart network.service

Links

libguestfs

Description:

  • is a set of tools for accessing and modifying virtual machine (VM) disk images
  • used for viewing and editing files inside guests, scripting changes to VMs, monitoring disk used/free statistics, creating guests, P2V, V2V, performing backups, cloning VMs, building VMs, formatting disks, resizing disks, etc
  • is available through a scriptable shell called guestfish, or an interactive rescue shell virt-rescue
  • has a 250 page manual
  • written in C, bindings in many languages (python, C#, ...)

Commands

guestfs(3) — main API documentation
guestfish(1) — interactive shell
guestmount(1) — mount guest filesystem in host
guestunmount(1) — unmount guest filesystem
virt-alignment-scan(1) — check alignment of virtual machine partitions
virt-builder(1) — quick image builder
virt-builder-repository(1) — create virt-builder repositories
virt-cat(1) — display a file
virt-copy-in(1) — copy files and directories into a VM
virt-copy-out(1) — copy files and directories out of a VM
virt-customize(1) — customize virtual machines
virt-df(1) — free space
virt-dib(1) — safe diskimage-builder
virt-diff(1) — differences
virt-edit(1) — edit a file
virt-filesystems(1) — display information about filesystems, devices, LVM
virt-format(1) — erase and make blank disks
virt-get-kernel(1) — get kernel from disk
virt-inspector(1) — inspect VM images
virt-list-filesystems(1) — list filesystems
virt-list-partitions(1) — list partitions
virt-log(1) — display log files
virt-ls(1) — list files
virt-make-fs(1) — make a filesystem
virt-p2v(1) — convert physical machine to run on KVM
virt-p2v-make-disk(1) — make P2V ISO
virt-p2v-make-kickstart(1) — make P2V kickstart
virt-rescue(1) — rescue shell
virt-resize(1) — resize virtual machines
virt-sparsify(1) — make virtual machines sparse (thin-provisioned)
virt-sysprep(1) — unconfigure a virtual machine before cloning
virt-tail(1) — follow log file
virt-tar(1) — archive and upload files
virt-tar-in(1) — archive and upload files
virt-tar-out(1) — archive and download files
virt-v2v(1) — convert guest to run on KVM

Usage

# filesystem operations
virt-make-fs --format=FORMAT --type=TYPE [--site=[+]SIZE] FOLDER IMAGE
virt-ls -l [-d DOMAIN | -a IMAGE] DIRECTORY
virt-cat [-d DOMAIN | -a IMAGE] FILE
virt-edit [-d DOMAIN | -a IMAGE] FILE
virt-tar-out [-d DOMAIN | -a IMAGE] GUESTPATH TARFILE
guestmount [-d DOMAIN | -a IMAGE] -i MOUNTPOINT [--ro|live]

# information gathering
virt-df [-d DOMAIN | -a IMAGE]
virt-log [-d DOMAIN | -a IMAGE]
virt-filesystems -l [-d DOMAIN | -a IMAGE]
                 [-h] [--partitions] [--all] [--long] [--uuid]

# image provisioning
virt-customize [-d DOMAIN | -a IMAGE]
               --hostname HOSTNAME
               --timezone TIMEZONE
               --root-password USER:PASSWORD
               --ssh-inject USER:file:PATH
               --delete FILE
               --link TARGET:LINK
               --write PATH:CONTENT
               --run-command COMMAND
virt-sysprep [same parameters as virt-customize]
             --operation OPERATION

Recipes

Create partitioned image and import rootfs via guestfish

guestfish <<EOF
disk-create rootfs.qcow2 qcow2 "$((2 * $(stat -c%s rootfs.tar)))"
add-drive rootfs.qcow2
run
part-disk /dev/sda gpt
mkfs ext4 /dev/sda
mount /dev/sda /
tar-in rootfs.tar /
umount-all
exit
EOF

Chroot into guest filesystem

virsh shutdown DOMAIN
guestmount -d DOMAIN -i /mnt
chroot /mnt
guestunmount /mnt

Links

About

Collection of scripts to build and provision debian images for docker and libvirt

Resources

Readme
GPL-3.0 license
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages