Skip to content

Manifest

A Manifest describes the lab configuration parameters. Manifests are defined in the toml format in a file named manifest.toml in the current directory.

Example Manifest

# manifest.toml
name = "sexy-salamander"

[[nodes]]
name = "dev01"
model = "sonic_linux"

[[nodes]]
name = "dev02"
model = "arista_eos"
kind = "container"

[[links]]
src = "dev01::gig4"
dst = "dev02::gig4"

Required Parameters

name (string)

The name of the lab: 1 to 63 letters, digits, - or _, starting with a letter or digit, as a node's name is. sherpa validate, sherpa up, sherpa redeploy and the server refuse any other name.

To avoid naming collisions, the lab name is hashed with the users username to create a lab-id. Resources are suffixed with the lab-id to create unique resource names.

name = "sexy-salamander"

nodes (list<node_object>)

The nodes that will be managed as part of this lab.

nodes = []

Required Node Parameters

name (string)

The name of the device: 1 to 63 letters, digits, - or _, starting with a letter or digit. It becomes the node's host name, so it cannot contain dots, spaces or other characters.

name = "dev01"
model (string)

The model of the device. Each model has a page under Nodes. Some models take several kinds, see kind.

model = "sonic_linux"

Optional Node Parameters

A node's settings are grouped. A few are top-level keys of the node; the rest sit in a table named after their group, such as [nodes.cpu]. Every setting is optional: one left out keeps the value of the node's image. A key Sherpa does not know is refused.

[[nodes]]
name = "dev01"
model = "ubuntu_linux"
memory = 4096

[nodes.cpu]
count = 4

[nodes.interfaces]
data_count = 4

[nodes.ready_check]
skip = true

TOML also takes a group's keys inline or as dotted keys, so cpu = { count = 4 } and cpu.count = 4 are the same as the [nodes.cpu] table above. Each group may appear once per node.

Group Keys
cpu count, mode, topology, pinning, emulator_pinning
numa node, memory_mode
boot_disk size, clone_method, cache, io, io_thread_cpus
interfaces data_count, queues
ztp method, config
ready_check port, skip
vm watchdog
container user, privileged, shm_size, commands, volumes, platform
unikernel cmdline
Which node kinds take which settings

A node's kind is its kind, or else its model's default kind. Sherpa refuses a setting the node's kind does not take, and a vm, container or unikernel group on a node of another kind. The message names the node, the key and the kind, for example Manifest - node: 'web01' sets `container`, which does not apply to virtual_machine nodes; 'ubuntu_linux' is a virtual_machine. On a model that takes several kinds, the message names the kind the setting needs: Manifest - node: 'r1' sets `container`, which does not apply to virtual_machine nodes; 'arista_eos' is a virtual_machine unless the node sets `kind = "container"`.

Setting Virtual machine Container Unikernel
memory, cpu.count yes yes (see limits) yes
cpu.mode, cpu.topology, cpu.pinning, cpu.emulator_pinning yes
numa yes
boot_disk.size yes yes
boot_disk.clone_method, boot_disk.cache, boot_disk.io, boot_disk.io_thread_cpus yes
interfaces.data_count yes yes yes
interfaces.queues yes
ztp, ssh yes yes
ready_check.port yes yes yes
ready_check.skip yes yes yes
environment_variables yes yes
text_files, startup_scripts, systemd_units some models, see each
vm yes
container yes
unikernel yes

memory, cpu.count, boot_disk.size, interfaces.data_count and ready_check.port must be at least 1. sherpa validate, sherpa up, sherpa redeploy and the server refuse 0, naming the node and the setting.

Container CPU and memory limits

On a container node, memory and cpu.count are Docker limits:

  • memory is a hard memory limit in MB, with no swap beyond it. A container that uses more is stopped by the kernel's OOM killer, so leave it room.
  • cpu.count is a CPU quota: the container may use that many CPUs' worth of time, on any of the server's CPUs. It does not pin the container to CPUs, and it cannot be more than the server has.

A container that sets neither runs without limits. The image's CPU and memory values do not apply to containers.

A juniper_evolved container is the exception: memory and cpu.count size the router virtual machine inside the container, defaulting to the image's. Its container always gets a memory limit of that memory plus 2048 MB, for the container's own processes, and no CPU quota. See Juniper cJunosEvolved.

Within a kind, a setting may still apply only to some models or ZTP methods; each section below says so.

environment_variables (list<string>)

A list of environment variables to pass to a node. Each entry is KEY=VALUE, where KEY is letters, digits and _ and does not start with a digit. sherpa validate, sherpa up, sherpa redeploy and the server refuse any other entry, naming the node.

environment_variables = [
  "SOME_VAR1=hello_jimmy",
  "MUSICAL_GENIUS='xyz-to-da-1-2-3'",
  # Load variable from your client environment with a dollar sign ($) prefixed variable name.
  "GIT_AUTHOR_NAME=$GIT_AUTHOR_NAME",
]

Not for secrets

Do not put passwords, API keys or tokens in environment_variables, text_files, startup_scripts or systemd_units. Their contents are sent to the Sherpa server, which keeps them with the lab: in the node's generated configuration and its first-boot media. Sherpa server administrators can read them there. Values loaded from your client environment with $ are kept the same way. The server's database records only their paths, and KEY=$VAR as written, not their contents or values. For containers, environment variables also appear in docker inspect on the server.

Use provisioners for secrets. On virtual machines the CLI copies them to the node itself, so the server never sees them; containers get them as files the server keeps in memory only.

Where the variables are set depends on the node:

  • Linux and BSD VMs using cloud-init, and Flatcar: the variables are written to ~/.sherpa_env in the home directory of the login user, which is your account on the node. The file is readable only by that user, and nothing is written system-wide. A line that sources it is added to ~/.bash_profile, or to ~/.profile when there is no ~/.bash_profile, so the variables are set in login shells, including the scripts of an ssh provisioner.
  • Windows VMs: the variables are set machine-wide. A value is set as written, %, " and ' included: %NAME% is not expanded.
  • Containers: the variables are set in the container environment.

Note

On VMs, the variables are not set in non-login sessions, such as ssh <node> <command>, cron jobs and systemd services, or in startup_scripts, which run as root. If your dotfiles replace ~/.bash_profile or ~/.profile, keep this line in them:

if [ -r "$HOME/.sherpa_env" ]; then . "$HOME/.sherpa_env"; fi
image (string)

Custom image name to use instead of the default for this model.

image = "custom-ubuntu"

For a container node, image is the image repository without a tag: the node's version is the tag. It replaces the repo the version was pulled from, for every container model, including those with a built-in repo. sherpa up and sherpa redeploy check that Docker has <image>:<version> and fail if it does not, before they create anything.

kind (string)

The node's kind: virtual_machine, container or unikernel. It must be a kind the node's model takes. Without it, the node takes its model's default kind.

[[nodes]]
name = "r1"
model = "frr_linux"
kind = "container"

Most models take one kind, which is their default, so kind is not needed on them. Three models take several:

Model Kinds Default
arista_eos virtual_machine (vEOS-lab), container (cEOS-lab) virtual_machine
juniper_evolved virtual_machine (vJunosEvolved), container (cJunosEvolved) virtual_machine
generic virtual_machine, container, unikernel none

A generic node must set kind; without it the node is refused: Manifest - node: 'g1' sets no `kind`, which 'generic' needs; set it to one of: virtual_machine, container, unikernel. A kind the model does not take is refused, naming the kinds it takes: Manifest - node: 'fw1' sets `kind = "container"`, but 'cisco_asa' comes only as: virtual_machine. The node's version and image are the model's images of that kind, and the kind settings groups it may set are its kind's.

memory (int)

Configure the amount of RAM in MegaBytes (MB).

memory = 4096

The server refuses to start virtual machines, unikernels and containers with a memory limit whose memory does not fit beside the memory already promised on the server. See [host_memory].

On a container, memory is a hard Docker limit; a container without it is not limited. See Container CPU and memory limits.

provisioners (list<object>)

Provisioners get secrets onto the node, run by the sherpa CLI. They are never saved with the lab. There are two types: ssh for Linux and BSD virtual machines, and container for containers. See Provisioners for how they work and when they run.

[[nodes.provisioners]]
type = "ssh"
env_files = ["secrets/dev.env"]
scripts = ["secrets/setup.sh"]

[[nodes.provisioners.files]]
src = "~/.config/gh/hosts.yml"
dst = "~/.config/gh/hosts.yml"
mode = 600

Provisioners run only from the sherpa CLI. A manifest with provisioners is refused from the web UI and the REST API.

ssh provisioner properties, for Linux and BSD virtual machines:

Property Type Description
type string ssh
files list<object> Files copied to the node over SFTP
env_files list<string> Env files copied to ~/.env.d/, which login shells load. A value of exactly $NAME or ${NAME} is replaced with that variable from your environment
scripts list<string> Scripts run on the node, through their shebang line

The SSH provisioner runs as the node's login user, in this order: files, env_files, then scripts. Client paths expand ~, and relative paths start at the manifest's folder.

files properties:

Property Type Description
src string Path on the client
dst string Path on the node: absolute, or starting with ~/ for the login user's home. Letters, digits and . _ - + @ only
mode int File permissions (e.g., 600). Defaults to 600

A file is written with its mode from the start and replaces any file at dst in one step, so it is never more readable than mode allows, and a failed copy leaves the old file as it was. Missing folders on the way to dst are created readable only by the login user.

Env files hold KEY=value lines and are copied as ~/.env.d/<file name>, so two env files on a node cannot share a name. Scripts get the variables from environment_variables and ~/.env.d/.

container provisioner, for containers: secret files, mounted read-only at /run/secrets/<name> before the container starts.

[[nodes]]
name = "db01"
model = "mongo_db"
environment_variables = [
  "MONGO_INITDB_ROOT_USERNAME=admin",
  "MONGO_INITDB_ROOT_PASSWORD_FILE=/run/secrets/mongo_password",
]

[[nodes.provisioners]]
type = "container"

[[nodes.provisioners.files]]
src = "secrets/mongo-password"
name = "mongo_password"
Property Type Description
type string container
files list<object> Secret files

files properties:

Property Type Description
src string Path on the client
name string File name in /run/secrets. Letters, digits and . _ - + @ only
mode int File permissions (e.g., 400). Defaults to 444, so a container that does not run as root can read it

Many images read a secret from a file named in a *_FILE environment variable, as above.

startup_scripts (list<string>)

A list of paths to scripts on the client that will run on node startup. A path may start with ~; a relative path starts at the manifest's folder.

startup_scripts = [
  "/path/to/script1.sh",
  "/path/to/script2.sh",
]

Taken by: virtual machines using cloud-init ZTP: Linux, BSD and Windows (cloudbase-init). Flatcar, network devices and containers do not take it.

Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node with its own ztp.config, naming the node, the setting and the model, for example Manifest - node: 'r1' sets text_files, which arista_eos does not use as a virtual_machine.

Like environment_variables, startup scripts are stored on the Sherpa server and are not for secrets.

Startup scripts run as root. Each script is run directly, so start it with a shebang line, such as #!/bin/bash, to choose its interpreter. A script without one runs with the node's /bin/sh. Alpine, FreeBSD and OpenBSD do not ship Bash, so use #!/bin/sh there.

systemd_units (list<object>)

Systemd units to deploy to a Flatcar Linux node. The sherpa CLI reads each unit file on your machine and sends its contents. A relative path starts at the manifest's folder.

Taken by: Flatcar Linux, using ignition ZTP. No other node takes it.

Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node with its own ztp.config, naming the node, the setting and the model, for example Manifest - node: 'r1' sets systemd_units, which ubuntu_linux does not use.

[[nodes.systemd_units]]
name = "myservice.service"
source = "units/myservice.service"
enabled = true
Property Type Description
name string Unit file name, such as app.service. It cannot contain / or start with -
source string Path of the unit file on the client
enabled bool Whether to enable the unit
text_files (list<object>)

A list of text files to pass to a node.

[[nodes.text_files]]
src = "~/.claude.json"
dst = "/home/sherpa/.claude.json"
user = "sherpa"
group = "sherpa"
permissions = 600

Taken by: virtual machines using cloud-init ZTP (Linux, BSD and Windows through cloudbase-init) and Flatcar Linux using ignition. Network devices and containers do not take it.

Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node with its own ztp.config, naming the node, the setting and the model, for example Manifest - node: 'r1' sets text_files, which arista_eos does not use as a virtual_machine.

Note

The dst should be a fully qualified path.

Warning

Text files are stored on the Sherpa server and are not for secrets. See environment_variables.

Note

Text files are written late in the first boot, after cloud-init has created users, so user and group can name your account. They must name a user and group that exist on the node, otherwise cloud-init does not write the file.

Windows nodes

Windows has no file modes, so user, group and permissions are not applied as on Linux. A file whose permissions give others no access (such as 600 or 640) can be read only by Administrators and SYSTEM. Any other file takes the access of the folder it is written to, which on most folders lets every local user read it. The dst cannot contain " or %.

Property Type Description
src string Source file path on the client
dst string Destination path on the node
user string File owner
group string File group
permissions int File permissions as octal digits (e.g., 600), up to 777. sherpa validate and sherpa up refuse other values. Write the digits as a plain number: 0644 is not valid TOML, and TOML's octal literal 0o644 is the number 420, which Sherpa reads as mode 420
version (string)

Specific image version to use instead of the default.

version = "22.04"
cpu (object)

The node's virtual CPUs: how many, and for a virtual machine how they are presented and which host CPUs they run on.

[nodes.cpu]
count = 4
mode = "host-passthrough"
topology = { sockets = 1, cores = 2, threads = 2 }
pinning = [9, 25, 10, 26]
emulator_pinning = [8, 24]
Key Type Description
count int The number of vCPUs, in place of the image's. On a container, a CPU quota (see limits)
mode string host-model (the default) or host-passthrough, which gives the guest the host CPU exactly. host-passthrough gives the best performance for nested virtual machines
topology object The CPU layout the guest sees: sockets, cores and threads. Their product must equal count
pinning list<int> The host CPU for each vCPU, in vCPU order: one per vCPU, none repeated
emulator_pinning list<int> Host CPUs for QEMU's own threads, everything in the virtual machine's QEMU process except its vCPUs

mode, topology, pinning and emulator_pinning apply only to virtual machines. See Tuning a virtual machine for the rules on pinned host CPUs.

numa (object)

The host NUMA node a virtual machine's memory is placed on.

[nodes.numa]
node = 1
memory_mode = "preferred"
Key Type Description
node int The host NUMA node the guest's memory is placed on. How strictly is set by memory_mode
memory_mode string strict (the default) or preferred. strict keeps the memory on node even when that node is full. preferred uses node first and other nodes when it is full. Needs node

This group only applies to virtual machine nodes.

boot_disk (object)

The disk the node boots from: its size, and for a virtual machine how it is created and how QEMU does its I/O.

[nodes.boot_disk]
size = 100
clone_method = "copy"
cache = "none"
io = "native"
io_thread_cpus = [8, 24]
Key Type Description
size int The boot disk's size in GigaBytes (GB). If omitted, the disk is the same size as the base image. The size can equal or exceed the base image, but cannot shrink it
clone_method string How the boot disk is created from the base image: overlay or copy, see below
cache string The host cache mode for the disks: none, writeback or writethrough
io string How QEMU submits disk I/O: native, io_uring or threads. native needs cache = "none"
io_thread_cpus list<int> Give the disks their own I/O thread, pinned to these host CPUs. Needs virtio disks

clone_method values:

Value Description
overlay Default. Creates a small disk that stores only the changes from the base image. Fast to create and uses little space.
copy Creates an independent full copy of the base image.

Use copy when you need a disk that does not depend on the base image, or when the base image cannot be used for overlays. See Storage. clone_method cannot be set at the lab level.

size applies to virtual machines and to unikernels that boot from a disk (disk_boot, such as Nanos). clone_method, cache, io and io_thread_cpus apply only to virtual machines.

interfaces (object)

The node's data interfaces.

[nodes.interfaces]
data_count = 4
queues = 4
Key Type Description
data_count int The number of data interfaces, in place of the model's default
queues int virtio-net queues on each interface, from 1 to cpu.count. Needs virtio interfaces; the management interface gets them only if it is virtio too

data_count covers data interfaces only. The management interface and any interfaces reserved by the model are not included. Requesting more interfaces than the model supports fails manifest validation.

queues applies only to virtual machines, and needs cpu.count to be set.

ztp (object)

The node's zero touch provisioning.

[nodes.ztp]
method = "disabled"
config = "configs/dev01.txt"
Key Type Description
method string The node's ZTP setting, in place of its image's: disabled, or one of cloud-init, cdrom, disk, http, ignition, nocloud, tftp, usb and volume
config string Path to a custom ZTP configuration file on the client. It may start with ~; a relative path starts at the manifest's folder

method must be disabled or a method the node's model takes; sherpa validate reports any other. Most models take one method, so the key is mostly used to turn ZTP off for a node. Without it, the node uses its image's setting.

Only nodes whose ZTP setting is tftp, nocloud, cdrom, disk, usb or volume take a config, and a hashicorp_vault node, where it is Vault's own configuration file. Sherpa refuses it on any other node, including one whose ZTP is disabled.

Nodes that join a Talos cluster cannot set method = "disabled" or a config: Sherpa generates their machine configs.

This group applies to virtual machine and container nodes.

ready_check (object)

How Sherpa decides the node is ready after sherpa up, sherpa redeploy and sherpa resume.

[nodes.ready_check]
port = 80
skip = false
Key Type Description
port int Port to probe on the node's management IP
skip bool Skip the readiness check

port:

  • Virtual machines: replaces the model's readiness port. Most models are checked on SSH port 22; talos_linux is checked on the Talos API port 50000. On port 22 an SSH server must answer: the port must be open and complete an SSH handshake (no login is attempted). Any other port only needs to accept a TCP connection.
  • Unikernels: Sherpa waits for a TCP connection to this port in addition to the libvirt domain reaching the running state.
  • Containers: a container must first keep running for 5 seconds after it starts, as one without a port does. Sherpa then waits for this port to answer on its management IP, checked as for a virtual machine: on port 22 an SSH server must answer, any other port only needs to accept a TCP connection. Until then the node is starting, and sherpa up shows Node <name> - Waiting for port <port>. A container that exits while Sherpa waits is a failed node; one whose port does not answer in time is reported not ready, as a virtual machine is, and stays starting. Without a port, a container is ready once it keeps running, except a sonic_linux container, which is checked on SSH port 22.

The server records each node's ready check (the port, or the model's port, and skip) when sherpa up creates the node, and again when sherpa redeploy rebuilds it from the changed manifest. sherpa resume and the server's scanner use that record: the scanner probes a node left starting on its port, a virtual machine's too, and records it running only once the port answers. A node with no port, or with skip = true, is running once it runs.

With skip = true the node is still created and started; Sherpa just doesn't wait for it to be ready, and ignores port. A container is not waited for to keep running, but one that is not running right after it starts is still reported as a failed node, with its exit code and last log lines, and sherpa up exits with an error.

vm (object)

Settings only a virtual machine takes.

[nodes.vm]
watchdog = "reset"
Key Type Description
watchdog string Give the virtual machine a hardware watchdog, and choose what happens when it expires. Without it, Sherpa adds none

watchdog actions:

Action What happens when the watchdog expires
reset The VM is reset, as with the reset button
poweroff The VM is powered off at once
shutdown The guest is asked to shut down, as with the power button
pause The VM is paused
dump The VM's memory is dumped, and it keeps running
inject-nmi The guest receives a non-maskable interrupt
none Nothing; the device is present but has no effect

The watchdog is an emulated Intel 6300ESB timer. It does nothing until the guest starts it, and the guest must then keep resetting it: if the guest hangs and stops, the action is taken. It helps only guests that run a watchdog driver and something that feeds it, for example Linux with the i6300esb module and systemd's RuntimeWatchdogSec.

VMs with the q35 machine type also have the chipset's own iTCO watchdog, which libvirt adds by itself and gives the same action, reset when watchdog is not set. It also does nothing until the guest starts it.

Warning

Images that start the watchdog early in boot and feed it only once booted can reset in a loop when the boot takes longer than the watchdog's timeout. Cisco IOS images do this, especially under nested virtualization, so do not give them a reset watchdog.

container (object)

Settings only a container takes.

[nodes.container]
user = "root"
privileged = true
shm_size = 67108864
commands = ["sleep", "infinity"]
volumes = [
  { src = "/host/path", dst = "/container/path" },
]
platform = "bx"
Key Type Description
user string The user the container runs as, in place of the image's
privileged bool Run the container in privileged mode
shm_size int Shared memory (/dev/shm) size in bytes
commands list<string> The container's command, in place of the image's
volumes list<object> Bind mounts of server paths into the container: src, the path on the Sherpa server, and dst, the path in the container
platform string The PTX router a juniper_evolved container emulates, see below

Every container's hostname is its node name, whatever its model, and it keeps it across a restart, down/resume and redeploy.

A privileged container, or one that can mount server paths, is root on the Sherpa server, so only admins may set privileged or volumes; up and redeploy refuse a manifest from another user that does.

Some models set their own command, user, privileges or mounts to start their software and mount their ZTP config; see the model's page under Nodes.

platform is bt (PTX10001-36MR, 12 data interfaces) or bx (PTX10002-36QDD, up to 36). Without it, the node is bt. Only a juniper_evolved container takes platform. See Juniper cJunosEvolved.

unikernel (object)

Settings only a unikernel takes.

[nodes.unikernel]
cmdline = "/usr/bin/nginx"
Key Type Description
cmdline string Kernel command line arguments, passed to QEMU via the libvirt <cmdline> XML element. Overrides any auto-injected command line

cmdline applies only to unikernels using the direct_kernel boot mode.

Tuning a virtual machine

A virtual machine's performance settings are in its cpu, numa, boot_disk and interfaces groups: how its CPU is presented, which host CPUs it runs on, which NUMA node holds its memory, and how its disks and interfaces work.

[[nodes]]
name = "dev"
model = "ubuntu_linux"
memory = 49152

[nodes.cpu]
count = 14
mode = "host-passthrough"
topology = { sockets = 1, cores = 7, threads = 2 }
pinning = [9, 25, 10, 26, 11, 27, 12, 28, 13, 29, 14, 30, 15, 31]
emulator_pinning = [8, 24]

[nodes.numa]
node = 1
memory_mode = "preferred"

[nodes.boot_disk]
cache = "none"
io = "io_uring"
io_thread_cpus = [8, 24]

[nodes.interfaces]
queues = 4

Host CPUs and NUMA nodes are numbered from 0, as on the host (lscpu, numactl -H).

Rules for pinned host CPUs:

  • cpu.topology, cpu.pinning and interfaces.queues need cpu.count to be set.
  • A CPU in cpu.pinning belongs to that node alone. No other node can pin a vCPU to it, in any lab on the server, and no node can run its QEMU or I/O threads on it, the same node included.
  • cpu.emulator_pinning and boot_disk.io_thread_cpus may share CPUs with each other and with the same settings on other nodes.
  • The checks run at sherpa up and sherpa redeploy. Every CPU and the NUMA node must exist on the server.
  • A node holds its pinned CPUs from the moment its up or redeploy is accepted until its lab is destroyed, also while the lab is stopped. During a redeploy that changes the pins, the node holds both the old and the new CPUs; a redeploy that fails before the new pins are in place keeps the old ones.
  • CPUs another node holds are refused with their own error, and the command exits with status 75, like a memory refusal (CPU pins held).
CPU pins held

When a node pins a host CPU that a node of another lab holds, or a node another up or redeploy is starting right now, the server refuses the command before it creates or tears down anything. Of two ups started at once that ask for the same CPUs, the first to arrive builds its lab and the other is refused. The error names the CPUs and what each node pins to them, for example:

Lab creation failed: Host CPU 3, which node 'n1' would pin to its vCPUs, is pinned to the
vCPUs of node 'n1' of lab 'pins-a' (5c725af9). Try again once that node is destroyed or
redeployed without them (code: -32006)

The holding node and lab are named to admins and to that lab's owner. Everyone else reads "a node of another lab" instead. sherpa up and sherpa redeploy exit with status 75 (exit status); run the command again once the CPUs are free, or pin other CPUs.

Two nodes that already share a pinned CPU are not refused: sherpa resume and sherpa redeploy start such a node and show a warning that names the CPUs, such as Warning: Host CPU 3 of node 'n1' is also pinned by a node of another lab, against the CPU pinning rules. Starting the node anyway; redeploy one of the two on other host CPUs to fix it. sherpad also logs each pair of nodes that share a CPU every time it starts. A redeploy may keep the CPU it already shares, but cannot take more.

The memory of a node with numa.node and numa.memory_mode = "strict" (or left out) must fit on that NUMA node beside the other virtual machines held strictly there, or sherpa up, sherpa redeploy and sherpa resume refuse it (see [host_memory]). Virtual machines without numa.node may still fill that node, and the guest held there cannot take memory elsewhere: when the node runs out, the host's OOM killer stops a virtual machine. Leave room on the node, pin every large virtual machine on the server, or use numa.memory_mode = "preferred".

With numa.memory_mode = "preferred" the memory only counts against the whole server, not against the NUMA node. When the node is full the guest takes memory from another node, which is slower to reach, instead of being stopped.

Pinning keeps a virtual machine's threads on those CPUs. It does not reserve the CPUs: the host can still run other work on them.

Optional Parameters

bridges (list<bridge_object>)

Bridges define shared layer 2 network segments that connect multiple node interfaces.

Unlike links which create point-to-point connections between two nodes, bridges allow three or more interfaces to share a common network segment.

Each bridge link is defined with the node name and interface separated by a double colon (::)

[[bridges]]
name = "shared"
links = ["dev01::eth2", "dev02::eth2", "dev03::eth1"]
Property Type Description
name string A user-friendly name for the bridge
links list<string> A list of <device>::<interface> mappings to connect to the bridge

For more information on bridge types see the private-bridge architecture documentation.

description (string)

A short description of the lab's purpose. Optional.

name = "bgp-practice"
description = "Practice BGP peering and route policies between two routers."

Multiline strings and Unicode are supported, up to 1000 characters. Line breaks and tabs are the only control characters allowed; sherpa validate and sherpa up refuse others.

The description is shown wherever the lab is: sherpa inspect, the web UI's labs list and lab page, the admin labs page, and the API's lab details and lab list.

Links define the connectivity relationship between nodes in the topology.

Each device is defined with its name and interface separated by a double colon (::)

[[links]]
src = "dev01::eth0"
dst = "dev02::eth1"
Property Type Required Description
src string yes Source <device>::<interface> mapping
dst string yes Destination <device>::<interface> mapping
impairment object no Network impairment configuration

Apply network impairment to simulate real-world link conditions. The values apply to both directions. To impair each direction differently, or to change it on a running lab, use sherpa link impair.

[[links]]
src = "dev01::eth1"
dst = "dev02::eth1"

[links.impairment]
delay = 50
jitter = 10
loss_percent = 1.0
Property Type Description
delay int One-way delay in milliseconds, up to 274877 (about 4.5 minutes)
jitter int Delay jitter in milliseconds, up to 274877
loss_percent float Packet loss percentage (0.0 - 100.0)
reorder_percent float Packet reordering percentage (0.0 - 100.0). Reordered packets skip the delay, so set a delay too
corrupt_percent float Bit-flip corruption percentage (0.0 - 100.0)

sherpa validate and sherpa up refuse a value outside these ranges, naming the link.

Every link is a P2P eBPF link.

ready_timeout (int)

Global timeout in seconds for node readiness checks. Overrides the default timeout for all nodes in the lab. It applies to sherpa up, sherpa redeploy and sherpa resume. It must be at least 1: sherpa validate, sherpa up, sherpa redeploy and the server refuse 0.

ready_timeout = 300

provision_concurrency (int)

How many nodes the sherpa CLI runs ssh provisioners on at the same time. Defaults to 8, and must be from 1 to 255. Use 1 to provision one node at a time. Only the CLI reads it: it is not sent to the server.

provision_concurrency = 4

tailscale (object)

Connect the lab's management networks to the owner's Tailscale tailnet through a dedicated gateway. Nodes do not need their own Tailscale agent.

[tailscale]
enabled = true
auth_key_env = "SHERPA_TAILSCALE_KEY"
Property Type Default Description
enabled bool false Create a Tailscale gateway for the lab
auth_key_env string unset Client environment variable containing the auth key; required when enabled

Set the named variable in the client shell before running sherpa up and sherpa resume. Store only the variable name in the manifest, never the key. The key is required for creation and for each whole-lab resume, because sherpa down logs the gateway out. Omitting the section or setting enabled = false requires no credential and creates no gateway.

Enrollment requires server TLS and WSS with certificate validation enabled. Route approval is separate from enrollment; pending approval produces a warning without failing lab creation. See Tailscale setup and route approval.

talos_clusters (list<talos_cluster_object>)

Kubernetes clusters built from the lab's Talos Linux nodes. Each cluster lists its member nodes by name. Sherpa generates each cluster's secrets and machine configs, bootstraps it, and writes .sherpa/k8s/kubeconfig-<cluster> and .sherpa/talos/talosconfig-<cluster> in your working directory.

[[talos_clusters]]
name = "k8s"
controllers = ["cp01", "cp02", "cp03"]
workers = ["worker01"]

[[nodes]]
name = "cp01"
model = "talos_linux"

[[nodes]]
name = "cp02"
model = "talos_linux"

[[nodes]]
name = "cp03"
model = "talos_linux"

[[nodes]]
name = "worker01"
model = "talos_linux"
Property Type Description
name string Cluster name, also the Kubernetes cluster name. 1–63 lowercase letters, digits or -
controllers list<string> Nodes that run etcd and the Kubernetes control plane. Sherpa bootstraps the cluster on the first one
workers list<string> Nodes that run workloads only. Optional; without workers, workloads run on the controllers
cni string flannel (the default) or cilium. See Cilium
oidc string A keycloak_server node whose sherpa realm users log in to the cluster. Optional. See Keycloak login
vault string A hashicorp_vault node, without ztp.config, the cluster's workloads read secrets from. Optional. See Vault secrets

Members must be talos_linux nodes defined in nodes, and a node can be listed only once across all clusters. A talos_linux node that is not listed is a standalone Talos node.

Each cluster needs one or three controllers. Two are refused: etcd needs a majority of its members to stay up, so two controllers survive no failure and are less available than one. A lab can have several clusters; at most two of them can have three controllers, and at most two can use Cilium.