Skip to content

Virtual Machine

Virtual machines are supported via the KVM hypervisor using the QEMU emulator and the Libvirt virtualisation API.

Virtual machine architecture: sherpad drives libvirtd, which runs one QEMU process per VM on KVM; each virtual device is backed by a host resource

How it Works

When a lab is created, Sherpa performs the following for each VM node:

  1. Create the boot disk in the storage pool: a QCOW2 overlay on the base image by default, or a full copy when boot_disk.clone_method = "copy"
  2. Resize the boot disk if boot_disk.size is specified in the manifest
  3. Generate a libvirt domain XML definition from the node's parameters
  4. Apply zero-touch provisioning configuration (cloud-init ISO, ZTP scripts, etc.)
  5. Define and start the domain via the Libvirt API

Each VM runs as a persistent libvirt domain under KVM/QEMU.

Resource Allocation

Every node model has default resource parameters that can be overridden in the manifest. Typical defaults:

Resource Range Examples
CPU 1–4 vCPUs Cisco ASAv: 1, Arista vEOS: 2, Cisco Cat9000v: 4
RAM 2048–20480 MiB Cisco ASAv: 2048, Cisco CSR1000v: 3072, Cisco XRv9000: 20480
Machine Type pc, q35 variants Most use pc, some require specific q35 versions
BIOS SeaBIOS or UEFI Most use SeaBIOS; XRv9000, Nexus 9300v use UEFI

CPU mode defaults to host-model, which auto-detects the host CPU capabilities. Nested virtualisation is supported via the VMX CPU flag when required.

Performance Tuning

A node's cpu, numa, boot_disk and interfaces settings can tune it further: - give it the host CPU exactly (host-passthrough) and a CPU topology; - pin its vCPUs, and QEMU's own threads, to host CPUs; - hold its memory on one NUMA node, or prefer one NUMA node; - set its disk cache and I/O mode, and give its disks an I/O thread; - add virtio-net queues to its interfaces.

Sherpa refuses a node whose pinned CPUs overlap another node's, in any lab on the server. A node held strictly on one NUMA node never uses another node's memory, so the server also refuses it when that NUMA node does not have room for it. A node that only prefers a NUMA node (numa.memory_mode = "preferred") uses other nodes' memory when its node is full, and is only checked against the whole server.

Memory

A virtual machine takes the memory it is configured with as the guest touches it, not when it starts. The server therefore checks memory before it starts virtual machines and unikernels: their configured memory, plus that of every running virtual machine and unikernel and of the nodes other requests are starting, must fit in the server's memory less a host reserve. See [host_memory] for the reserve, the overcommit percentage (overcommit_percent) and the message a refusal gives.

Guest memory can also be backed by 2 MiB huge pages through the host's transparent hugepages. QEMU already asks for them, so this needs no manifest setting; the host's /sys/kernel/mm/transparent_hugepage/enabled must be madvise or always. The installer sets madvise unless it is already always.

Disk Management

Base images are imported into /opt/sherpa/images/. When a lab starts, each VM's boot disk is created in the Sherpa storage pool (sherpa-pool, at /opt/sherpa/libvirt/images/) as a QCOW2 overlay on the base image, or as a full copy when boot_disk.clone_method = "copy". See Boot Disk Overlays. The original base image is never modified. ZTP configuration disks, such as cloud-init ISOs, are created in the same pool.

Supported disk formats: qcow2, raw, iso, img.

Disk drivers vary by model:

Driver Usage
virtio Default for most models
sata Used by some models that lack virtio support
ide Legacy models

Boot disk resizing increases the volume capacity via the Libvirt storage API. The new size must be larger than the current allocation.

Networking

VM network interfaces are attached as libvirt network devices. The interface driver is configurable per model:

Driver Usage
virtio Default for most models
e1000 Models that require Intel NIC emulation
vmxnet3 VMware-compatible models

Each VM is connected to the management network on its first interface. Additional interfaces are attached for data plane links as defined in the manifest. Each end of a point-to-point link is a tap device joined to its peer by an eBPF redirect program.

Console Access

Each VM provides two console access methods:

Serial Console

The serial console of each VM is a UNIX socket that libvirt binds under /opt/sherpa/run/console/, in a folder only the sherpa account, the kvm group and root can reach. Connect with sherpa console <node> from any machine with the CLI. The connection goes through sherpad and only the lab's owner can open it. See Serial Console and sherpa console.

VNC

Each VM has a VNC graphics adapter. QEMU listens for it on a UNIX socket under /opt/sherpa/run/console/, next to the serial console. Connect a VNC viewer with sherpa vnc <node> from any machine with the CLI. The connection goes through sherpad and only the lab's owner can open it. See VNC and sherpa vnc.

Lifecycle

Operation Description
Create Domain XML is defined and the VM is started
Stop VM is shut down via Libvirt
Resume Stopped VM is restarted
Redeploy VM is destroyed, disks are deleted, and the node is recreated with fresh configuration
Destroy Domain is undefined and all associated disks are removed

Power behaviour:

  • On poweroff — VM is destroyed
  • On reboot — VM is restarted
  • On crash — VM is destroyed

Additional Features

  • Watchdog only when asked — Sherpa adds a hardware watchdog only when the node sets vm.watchdog. (q35 VMs always have the chipset's iTCO watchdog, which libvirt adds; it does nothing unless the guest starts it.) Cisco IOS images arm one early in boot and feed it only after a long init phase, so under nested virtualization a reset watchdog resets them in a loop
  • QEMU Guest Agent — virtio channel for host-guest communication
  • NVRAM — UEFI VMs store NVRAM at /var/lib/libvirt/qemu/nvram/
  • Windows support — Hyper-V enlightenments (relaxed, vapic, spinlocks, vpindex, runtime, synic, stimer, frequencies, tlbflush, ipi) and the Hyper-V clock are enabled for Windows VMs