e2e: fast VM bootstrap using cached preprovisioned VM images ('boxes').

test/e2e/README.md

@@ -97,6 +97,70 @@ Before running E2E tests ensure that you have all the required components locall
     policies.test-suite balloons test10-health-checking : PASS
     ```
 
+## Caching a provisioned VM
+
+Most of the time a test run spends before the first test case goes into
+provisioning the VM: installing Kubernetes, the container runtime, the CNI
+plugin and Helm, and initializing a single-node cluster with `kubeadm`. The
+result only depends on the versions installed, so it can be exported with
+`vagrant package` and reused.
+
+Set `e2e_vm_cache` to opt in:
+
+```shell
+e2e_vm_cache=yes ./run_tests.sh policies.test-suite
+```
+
+| value | meaning |
+| --- | --- |
+| `no` | do not use or create cached boxes (the default) |
+| `yes` | use a cached box if there is one, otherwise provision and keep the result |
+| `refresh` | ignore any cached box, provision from scratch, then replace it |
+| `cleanup` | remove all cached boxes, printing each of them, and exit without running tests |
+| `nuke`, `drop` | synonyms of `cleanup` |
+
+The boxes live in `$CACHE_DIR/boxes`, next to the tarballs the framework
+already caches, and are named after everything which shapes the guest: the
+topology, the distro, the Kubernetes, runtime, CNI and Helm versions, and a
+hash of the files which provisioning uses, listed in `BOX_RECIPE_FILES` in
+`lib/vm.bash`. Editing any of them therefore invalidates the boxes instead of a
+later run reusing an image which predates the change.
+
+Worth knowing:
+
+- The topology is part of the name. The hostname of the VM is derived from it,
+  and `kubeadm` bakes the hostname into the name of the node, into the
+  certificates and into etcd, so a box only serves the topology it was made
+  for.
+
+- A box is only used when the VM is created for the first time. An output
+  directory which already has a `Vagrantfile` keeps the VM and the box it was
+  created from, so start from a clean output directory to benefit from the
+  cache.
+
+- Expect a couple of gigabytes per box, and note that vagrant unpacks a box
+  into `~/.vagrant.d/boxes` on first use, so the disk cost is roughly twice the
+  file size. Nothing prunes them automatically, so clean up when done:
+
+  ```shell
+  e2e_vm_cache=cleanup ./run_tests.sh policies.test-suite
+  ```
+
+  This removes both halves of every cached box, the file and the copy vagrant
+  unpacked, and leaves the downloaded distro images alone.
+
+- Boxes expire after 30 days (`BOX_CACHE_DECAY`). They contain a cluster whose
+  certificates the `kubeadm` defaults give a year to live, so they are not
+  meant to be kept around indefinitely.
+
+- `vagrant package` needs a recent enough `vagrant-qemu`. With an older one the
+  framework says so and provisions from scratch.
+
+- A VM which has just booted from a box has a cluster which is still starting
+  up, so the framework waits for the node and for the cluster DNS before
+  running any test. Raise `CLUSTER_READY_TIMEOUT` (300 seconds by default) if a
+  large topology needs longer.
+
 ## Writing tests
 
 A test case is a `code.var.sh` file in a

test/e2e/files/Vagrantfile.in

@@ -26,6 +26,7 @@ CRIO_SRC = ""
 CONTAINERD_RELEASE = ""
 CONTAINERD_SRC = ""
 CACHE_DIR="#{ENV['CACHE_DIR']}"
+NO_PROVISION = "#{ENV['E2E_NO_PROVISION']}"
 
 if CRI_RUNTIME == "containerd"
   CONTAINERD_RELEASE = "#{ENV['containerd_release']}"
@@ -104,31 +105,38 @@ Vagrant.configure("2") do |config|
     qemu.disk_resize = "#{ENV['VM_DISK_SIZE']}"
   end
 
-  config.vm.provision :ansible do |ansible|
-    ansible.playbook = "#{ENV['nri_resource_policy_src']}/test/e2e/playbook/provision.yaml"
-    ansible.extra_vars = {
-      network: "10.217.0.0/16",
-      hostname: HOSTNAME,
-      dns_nameserver: "#{ENV['DNS_NAMESERVER']}",
-      dns_search_domain: "#{ENV['DNS_SEARCH_DOMAIN']}",
-      https_proxy: "#{ENV['HTTPS_PROXY']}",
-      http_proxy: "#{ENV['HTTP_PROXY']}",
-      no_proxy: "#{ENV['NO_PROXY']}",
-      kernel_getsource: KERNEL_GETSOURCE,
-      kernel_config: KERNEL_CONFIG,
-      k8s_release: K8S_RELEASE,
-      k8s_version: K8S_VERSION,
-      helm_release: HELM_RELEASE,
-      cri_runtime: CRI_RUNTIME,
-      containerd_release: CONTAINERD_RELEASE,
-      containerd_src: CONTAINERD_SRC,
-      crio_release: CRIO_RELEASE,
-      crio_src: CRIO_SRC,
-      cache_dir: CACHE_DIR,
-      cni_plugin: CNI_PLUGIN,
-      cni_release: CNI_RELEASE,
-      nri_resource_policy_src: NRI_RESOURCE_POLICY_SRC,
-      outdir: OUTPUT_DIR,
-    }
+  # A VM created from an already provisioned box must never be provisioned
+  # again: the playbook has already run into that box, and kubeadm init
+  # cannot run a second time. Skipping it on the vagrant up which creates
+  # the VM is not enough, as vagrant then keeps the machine flagged as not
+  # provisioned and runs the provisioner on the next up.
+  if NO_PROVISION != "1"
+    config.vm.provision :ansible do |ansible|
+      ansible.playbook = "#{ENV['nri_resource_policy_src']}/test/e2e/playbook/provision.yaml"
+      ansible.extra_vars = {
+        network: "10.217.0.0/16",
+        hostname: HOSTNAME,
+        dns_nameserver: "#{ENV['DNS_NAMESERVER']}",
+        dns_search_domain: "#{ENV['DNS_SEARCH_DOMAIN']}",
+        https_proxy: "#{ENV['HTTPS_PROXY']}",
+        http_proxy: "#{ENV['HTTP_PROXY']}",
+        no_proxy: "#{ENV['NO_PROXY']}",
+        kernel_getsource: KERNEL_GETSOURCE,
+        kernel_config: KERNEL_CONFIG,
+        k8s_release: K8S_RELEASE,
+        k8s_version: K8S_VERSION,
+        helm_release: HELM_RELEASE,
+        cri_runtime: CRI_RUNTIME,
+        containerd_release: CONTAINERD_RELEASE,
+        containerd_src: CONTAINERD_SRC,
+        crio_release: CRIO_RELEASE,
+        crio_src: CRIO_SRC,
+        cache_dir: CACHE_DIR,
+        cni_plugin: CNI_PLUGIN,
+        cni_release: CNI_RELEASE,
+        nri_resource_policy_src: NRI_RESOURCE_POLICY_SRC,
+        outdir: OUTPUT_DIR,
+      }
+    end
   end
 end

test/e2e/files/env.in

@@ -13,3 +13,4 @@ DNS_SEARCH_DOMAIN=""
 
 SSH_PORT=
 CACHE_DIR=
+E2E_NO_PROVISION=