Manifest¶
A Manifest describes the lab configuration parameters.
Manifests are defined in the toml format in a file named manifest.toml
in the current directory.
Example Manifest¶
# manifest.toml
name = "sexy-salamander"
[[nodes]]
name = "dev01"
model = "sonic_linux"
[[nodes]]
name = "dev02"
model = "arista_eos"
kind = "container"
[[links]]
src = "dev01::gig4"
dst = "dev02::gig4"
Required Parameters¶
name (string)¶
The name of the lab: 1 to 63 letters, digits, - or _, starting with a letter or digit, as a
node's name is. sherpa validate, sherpa up, sherpa redeploy and the server refuse any
other name.
To avoid naming collisions, the lab name is hashed with the users username
to create a lab-id. Resources are suffixed with the lab-id to create unique resource names.
name = "sexy-salamander"
nodes (list<node_object>)¶
The nodes that will be managed as part of this lab.
nodes = []
Required Node Parameters¶
name (string)¶
The name of the device: 1 to 63 letters, digits, - or _, starting with a letter or digit.
It becomes the node's host name, so it cannot contain dots, spaces or other characters.
name = "dev01"
model (string)¶
The model of the device. Each model has a page under Nodes. Some models take several kinds,
see kind.
model = "sonic_linux"
Optional Node Parameters¶
A node's settings are grouped. A few are top-level keys of the node; the rest sit in a table
named after their group, such as [nodes.cpu]. Every setting is optional: one left out keeps
the value of the node's image. A key Sherpa does not know is refused.
[[nodes]]
name = "dev01"
model = "ubuntu_linux"
memory = 4096
[nodes.cpu]
count = 4
[nodes.interfaces]
data_count = 4
[nodes.ready_check]
skip = true
TOML also takes a group's keys inline or as dotted keys, so cpu = { count = 4 } and
cpu.count = 4 are the same as the [nodes.cpu] table above. Each group may appear once per
node.
| Group | Keys |
|---|---|
cpu |
count, mode, topology, pinning, emulator_pinning |
numa |
node, memory_mode |
boot_disk |
size, clone_method, cache, io, io_thread_cpus |
interfaces |
data_count, queues |
ztp |
method, config |
ready_check |
port, skip |
vm |
watchdog |
container |
user, privileged, shm_size, commands, volumes, platform |
unikernel |
cmdline |
Which node kinds take which settings¶
A node's kind is its kind, or else its model's default kind. Sherpa refuses
a setting the node's kind does not take, and a vm, container or unikernel group on a node
of another kind. The message names the node, the key and the kind, for example
Manifest - node: 'web01' sets `container`, which does not apply to virtual_machine nodes; 'ubuntu_linux' is a virtual_machine.
On a model that takes several kinds, the message names the kind the setting needs:
Manifest - node: 'r1' sets `container`, which does not apply to virtual_machine nodes; 'arista_eos' is a virtual_machine unless the node sets `kind = "container"`.
| Setting | Virtual machine | Container | Unikernel |
|---|---|---|---|
memory, cpu.count |
yes | yes (see limits) | yes |
cpu.mode, cpu.topology, cpu.pinning, cpu.emulator_pinning |
yes | ||
numa |
yes | ||
boot_disk.size |
yes | yes | |
boot_disk.clone_method, boot_disk.cache, boot_disk.io, boot_disk.io_thread_cpus |
yes | ||
interfaces.data_count |
yes | yes | yes |
interfaces.queues |
yes | ||
ztp, ssh |
yes | yes | |
ready_check.port |
yes | yes | yes |
ready_check.skip |
yes | yes | yes |
environment_variables |
yes | yes | |
text_files, startup_scripts, systemd_units |
some models, see each | ||
vm |
yes | ||
container |
yes | ||
unikernel |
yes |
memory, cpu.count, boot_disk.size, interfaces.data_count and ready_check.port must be
at least 1. sherpa validate, sherpa up, sherpa redeploy and the server refuse 0, naming
the node and the setting.
Container CPU and memory limits¶
On a container node, memory and cpu.count are Docker limits:
memoryis a hard memory limit in MB, with no swap beyond it. A container that uses more is stopped by the kernel's OOM killer, so leave it room.cpu.countis a CPU quota: the container may use that many CPUs' worth of time, on any of the server's CPUs. It does not pin the container to CPUs, and it cannot be more than the server has.
A container that sets neither runs without limits. The image's CPU and memory values do not apply to containers.
A juniper_evolved container is the exception: memory and cpu.count size the router
virtual machine inside the container, defaulting to the image's. Its container always gets a
memory limit of that memory plus 2048 MB, for the container's own processes, and no CPU quota.
See Juniper cJunosEvolved.
Within a kind, a setting may still apply only to some models or ZTP methods; each section below says so.
environment_variables (list<string>)¶
A list of environment variables to pass to a node. Each entry is KEY=VALUE, where KEY is
letters, digits and _ and does not start with a digit. sherpa validate, sherpa up,
sherpa redeploy and the server refuse any other entry, naming the node.
environment_variables = [
"SOME_VAR1=hello_jimmy",
"MUSICAL_GENIUS='xyz-to-da-1-2-3'",
# Load variable from your client environment with a dollar sign ($) prefixed variable name.
"GIT_AUTHOR_NAME=$GIT_AUTHOR_NAME",
]
Not for secrets
Do not put passwords, API keys or tokens in environment_variables, text_files,
startup_scripts or systemd_units. Their contents are sent to the Sherpa server, which
keeps them with the lab: in the node's generated configuration and its first-boot media. Sherpa server administrators can read them there. Values loaded from your
client environment with $ are kept the same way. The server's
database records only their paths, and
KEY=$VAR as written, not their contents or values. For containers, environment variables
also appear in docker inspect on the server.
Use provisioners for secrets. On virtual machines the CLI
copies them to the node itself, so the server never sees them; containers get them as files
the server keeps in memory only.
Where the variables are set depends on the node:
- Linux and BSD VMs using
cloud-init, and Flatcar: the variables are written to~/.sherpa_envin the home directory of the login user, which is your account on the node. The file is readable only by that user, and nothing is written system-wide. A line that sources it is added to~/.bash_profile, or to~/.profilewhen there is no~/.bash_profile, so the variables are set in login shells, including the scripts of ansshprovisioner. - Windows VMs: the variables are set machine-wide. A value is set as written,
%,"and'included:%NAME%is not expanded. - Containers: the variables are set in the container environment.
Note
On VMs, the variables are not set in non-login sessions, such as ssh <node> <command>,
cron jobs and systemd services, or in startup_scripts, which run as root. If your
dotfiles replace ~/.bash_profile or ~/.profile, keep this line in them:
if [ -r "$HOME/.sherpa_env" ]; then . "$HOME/.sherpa_env"; fi
image (string)¶
Custom image name to use instead of the default for this model.
image = "custom-ubuntu"
For a container node, image is the image repository without a tag: the node's version is the
tag. It replaces the repo the version was pulled from, for every container model, including those
with a built-in repo. sherpa up and sherpa redeploy check that Docker has <image>:<version>
and fail if it does not, before they create anything.
kind (string)¶
The node's kind: virtual_machine, container or unikernel. It must be a kind the node's
model takes. Without it, the node takes its model's default kind.
[[nodes]]
name = "r1"
model = "frr_linux"
kind = "container"
Most models take one kind, which is their default, so kind is not needed on them. Three
models take several:
| Model | Kinds | Default |
|---|---|---|
arista_eos |
virtual_machine (vEOS-lab), container (cEOS-lab) |
virtual_machine |
juniper_evolved |
virtual_machine (vJunosEvolved), container (cJunosEvolved) |
virtual_machine |
generic |
virtual_machine, container, unikernel |
none |
A generic node must set kind; without it the node is refused:
Manifest - node: 'g1' sets no `kind`, which 'generic' needs; set it to one of: virtual_machine, container, unikernel.
A kind the model does not take is refused, naming the kinds it takes:
Manifest - node: 'fw1' sets `kind = "container"`, but 'cisco_asa' comes only as: virtual_machine.
The node's version and image are the model's images of that kind, and the
kind settings groups it may set are its kind's.
memory (int)¶
Configure the amount of RAM in MegaBytes (MB).
memory = 4096
The server refuses to start virtual machines, unikernels and containers with a memory limit
whose memory does not fit beside the memory already promised on the server. See
[host_memory].
On a container, memory is a hard Docker limit; a container without it is not limited. See
Container CPU and memory limits.
provisioners (list<object>)¶
Provisioners get secrets onto the node, run by the sherpa CLI. They are never saved with the
lab. There are two types: ssh for Linux and BSD virtual machines, and container for
containers. See Provisioners for how they work and when
they run.
[[nodes.provisioners]]
type = "ssh"
env_files = ["secrets/dev.env"]
scripts = ["secrets/setup.sh"]
[[nodes.provisioners.files]]
src = "~/.config/gh/hosts.yml"
dst = "~/.config/gh/hosts.yml"
mode = 600
Provisioners run only from the sherpa CLI. A manifest with provisioners is refused from the
web UI and the REST API.
ssh provisioner properties, for Linux and BSD virtual
machines:
| Property | Type | Description |
|---|---|---|
type |
string | ssh |
files |
list<object> | Files copied to the node over SFTP |
env_files |
list<string> | Env files copied to ~/.env.d/, which login shells load. A value of exactly $NAME or ${NAME} is replaced with that variable from your environment |
scripts |
list<string> | Scripts run on the node, through their shebang line |
The SSH provisioner runs as the node's login user, in this order: files, env_files, then
scripts. Client paths expand ~, and relative paths start at the manifest's folder.
files properties:
| Property | Type | Description |
|---|---|---|
src |
string | Path on the client |
dst |
string | Path on the node: absolute, or starting with ~/ for the login user's home. Letters, digits and . _ - + @ only |
mode |
int | File permissions (e.g., 600). Defaults to 600 |
A file is written with its mode from the start and replaces any file at dst in one step, so
it is never more readable than mode allows, and a failed copy leaves the old file as it was.
Missing folders on the way to dst are created readable only by the login user.
Env files hold KEY=value lines and are copied as ~/.env.d/<file name>, so two env files on a
node cannot share a name. Scripts get the variables from environment_variables and
~/.env.d/.
container provisioner, for containers: secret
files, mounted read-only at /run/secrets/<name> before the container starts.
[[nodes]]
name = "db01"
model = "mongo_db"
environment_variables = [
"MONGO_INITDB_ROOT_USERNAME=admin",
"MONGO_INITDB_ROOT_PASSWORD_FILE=/run/secrets/mongo_password",
]
[[nodes.provisioners]]
type = "container"
[[nodes.provisioners.files]]
src = "secrets/mongo-password"
name = "mongo_password"
| Property | Type | Description |
|---|---|---|
type |
string | container |
files |
list<object> | Secret files |
files properties:
| Property | Type | Description |
|---|---|---|
src |
string | Path on the client |
name |
string | File name in /run/secrets. Letters, digits and . _ - + @ only |
mode |
int | File permissions (e.g., 400). Defaults to 444, so a container that does not run as root can read it |
Many images read a secret from a file named in a *_FILE environment variable, as above.
startup_scripts (list<string>)¶
A list of paths to scripts on the client that will run on node startup. A path may start
with ~; a relative path starts at the manifest's folder.
startup_scripts = [
"/path/to/script1.sh",
"/path/to/script2.sh",
]
Taken by: virtual machines using cloud-init ZTP: Linux, BSD and Windows (cloudbase-init).
Flatcar, network devices and containers do not take it.
Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and
sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node
with its own ztp.config, naming the node, the setting and the model, for example
Manifest - node: 'r1' sets text_files, which arista_eos does not use as a virtual_machine.
Like environment_variables, startup scripts are
stored on the Sherpa server and are not for secrets.
Startup scripts run as root. Each script is run directly, so start it with a shebang line,
such as #!/bin/bash, to choose its interpreter. A script without one runs with the node's
/bin/sh. Alpine, FreeBSD and OpenBSD do not ship Bash, so use #!/bin/sh there.
systemd_units (list<object>)¶
Systemd units to deploy to a Flatcar Linux node. The sherpa CLI reads each unit file on your
machine and sends its contents. A relative path starts at the manifest's folder.
Taken by: Flatcar Linux, using ignition ZTP. No other node takes it.
Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and
sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node
with its own ztp.config, naming the node, the setting and the model, for example
Manifest - node: 'r1' sets systemd_units, which ubuntu_linux does not use.
[[nodes.systemd_units]]
name = "myservice.service"
source = "units/myservice.service"
enabled = true
| Property | Type | Description |
|---|---|---|
name |
string | Unit file name, such as app.service. It cannot contain / or start with - |
source |
string | Path of the unit file on the client |
enabled |
bool | Whether to enable the unit |
text_files (list<object>)¶
A list of text files to pass to a node.
[[nodes.text_files]]
src = "~/.claude.json"
dst = "/home/sherpa/.claude.json"
user = "sherpa"
group = "sherpa"
permissions = 600
Taken by: virtual machines using cloud-init ZTP (Linux, BSD and Windows through
cloudbase-init) and Flatcar Linux using ignition. Network devices and containers do not take
it.
Taken only where a node's first-boot config carries it. sherpa validate, sherpa up and
sherpa redeploy refuse it on any other node, on a node whose ZTP is disabled, and on a node
with its own ztp.config, naming the node, the setting and the model, for example
Manifest - node: 'r1' sets text_files, which arista_eos does not use as a virtual_machine.
Note
The dst should be a fully qualified path.
Warning
Text files are stored on the Sherpa server and are not for secrets. See
environment_variables.
Note
Text files are written late in the first boot, after cloud-init has created
users, so user and group can name your account. They must name a user
and group that exist on the node, otherwise cloud-init does not write the file.
Windows nodes
Windows has no file modes, so user, group and permissions are not applied as on
Linux. A file whose permissions give others no access (such as 600 or 640) can be
read only by Administrators and SYSTEM. Any other file takes the access of the folder it
is written to, which on most folders lets every local user read it. The dst cannot
contain " or %.
| Property | Type | Description |
|---|---|---|
src |
string | Source file path on the client |
dst |
string | Destination path on the node |
user |
string | File owner |
group |
string | File group |
permissions |
int | File permissions as octal digits (e.g., 600), up to 777. sherpa validate and sherpa up refuse other values. Write the digits as a plain number: 0644 is not valid TOML, and TOML's octal literal 0o644 is the number 420, which Sherpa reads as mode 420 |
version (string)¶
Specific image version to use instead of the default.
version = "22.04"
cpu (object)¶
The node's virtual CPUs: how many, and for a virtual machine how they are presented and which host CPUs they run on.
[nodes.cpu]
count = 4
mode = "host-passthrough"
topology = { sockets = 1, cores = 2, threads = 2 }
pinning = [9, 25, 10, 26]
emulator_pinning = [8, 24]
| Key | Type | Description |
|---|---|---|
count |
int | The number of vCPUs, in place of the image's. On a container, a CPU quota (see limits) |
mode |
string | host-model (the default) or host-passthrough, which gives the guest the host CPU exactly. host-passthrough gives the best performance for nested virtual machines |
topology |
object | The CPU layout the guest sees: sockets, cores and threads. Their product must equal count |
pinning |
list<int> | The host CPU for each vCPU, in vCPU order: one per vCPU, none repeated |
emulator_pinning |
list<int> | Host CPUs for QEMU's own threads, everything in the virtual machine's QEMU process except its vCPUs |
mode, topology, pinning and emulator_pinning apply only to virtual machines. See
Tuning a virtual machine for the rules on pinned host CPUs.
numa (object)¶
The host NUMA node a virtual machine's memory is placed on.
[nodes.numa]
node = 1
memory_mode = "preferred"
| Key | Type | Description |
|---|---|---|
node |
int | The host NUMA node the guest's memory is placed on. How strictly is set by memory_mode |
memory_mode |
string | strict (the default) or preferred. strict keeps the memory on node even when that node is full. preferred uses node first and other nodes when it is full. Needs node |
This group only applies to virtual machine nodes.
boot_disk (object)¶
The disk the node boots from: its size, and for a virtual machine how it is created and how QEMU does its I/O.
[nodes.boot_disk]
size = 100
clone_method = "copy"
cache = "none"
io = "native"
io_thread_cpus = [8, 24]
| Key | Type | Description |
|---|---|---|
size |
int | The boot disk's size in GigaBytes (GB). If omitted, the disk is the same size as the base image. The size can equal or exceed the base image, but cannot shrink it |
clone_method |
string | How the boot disk is created from the base image: overlay or copy, see below |
cache |
string | The host cache mode for the disks: none, writeback or writethrough |
io |
string | How QEMU submits disk I/O: native, io_uring or threads. native needs cache = "none" |
io_thread_cpus |
list<int> | Give the disks their own I/O thread, pinned to these host CPUs. Needs virtio disks |
clone_method values:
| Value | Description |
|---|---|
overlay |
Default. Creates a small disk that stores only the changes from the base image. Fast to create and uses little space. |
copy |
Creates an independent full copy of the base image. |
Use copy when you need a disk that does not depend on the base image, or when the
base image cannot be used for overlays. See Storage.
clone_method cannot be set at the lab level.
size applies to virtual machines and to unikernels that boot from a disk (disk_boot, such
as Nanos). clone_method, cache, io and io_thread_cpus apply only to virtual machines.
interfaces (object)¶
The node's data interfaces.
[nodes.interfaces]
data_count = 4
queues = 4
| Key | Type | Description |
|---|---|---|
data_count |
int | The number of data interfaces, in place of the model's default |
queues |
int | virtio-net queues on each interface, from 1 to cpu.count. Needs virtio interfaces; the management interface gets them only if it is virtio too |
data_count covers data interfaces only. The management interface and any interfaces reserved
by the model are not included. Requesting more interfaces than the model supports fails
manifest validation.
queues applies only to virtual machines, and needs cpu.count to be set.
ztp (object)¶
The node's zero touch provisioning.
[nodes.ztp]
method = "disabled"
config = "configs/dev01.txt"
| Key | Type | Description |
|---|---|---|
method |
string | The node's ZTP setting, in place of its image's: disabled, or one of cloud-init, cdrom, disk, http, ignition, nocloud, tftp, usb and volume |
config |
string | Path to a custom ZTP configuration file on the client. It may start with ~; a relative path starts at the manifest's folder |
method must be disabled or a method the node's model takes; sherpa validate reports any
other. Most models take one method, so the key is mostly used to turn ZTP off for a node.
Without it, the node uses its image's setting.
Only nodes whose ZTP setting is tftp, nocloud, cdrom, disk, usb or volume take a
config, and a hashicorp_vault node, where it is Vault's own configuration file. Sherpa
refuses it on any other node, including one whose ZTP is disabled.
Nodes that join a Talos cluster cannot set
method = "disabled" or a config: Sherpa generates their machine configs.
This group applies to virtual machine and container nodes.
ready_check (object)¶
How Sherpa decides the node is ready after sherpa up, sherpa redeploy and sherpa resume.
[nodes.ready_check]
port = 80
skip = false
| Key | Type | Description |
|---|---|---|
port |
int | Port to probe on the node's management IP |
skip |
bool | Skip the readiness check |
port:
- Virtual machines: replaces the model's readiness port. Most models are checked on SSH
port
22;talos_linuxis checked on the Talos API port50000. On port22an SSH server must answer: the port must be open and complete an SSH handshake (no login is attempted). Any other port only needs to accept a TCP connection. - Unikernels: Sherpa waits for a TCP connection to this port in addition to the libvirt domain reaching the running state.
- Containers: a container must first keep running for 5 seconds after it starts, as one
without a
portdoes. Sherpa then waits for this port to answer on its management IP, checked as for a virtual machine: on port22an SSH server must answer, any other port only needs to accept a TCP connection. Until then the node isstarting, andsherpa upshowsNode <name> - Waiting for port <port>. A container that exits while Sherpa waits is a failed node; one whose port does not answer in time is reported not ready, as a virtual machine is, and staysstarting. Without aport, a container is ready once it keeps running, except asonic_linuxcontainer, which is checked on SSH port22.
The server records each node's ready check (the port, or the model's port, and
skip) when sherpa up creates the node, and again when sherpa
redeploy rebuilds it from the changed manifest.
sherpa resume and the server's scanner use that record:
the scanner probes a node left starting on its port, a virtual machine's too, and records it
running only once the port answers. A node with no port, or with skip = true, is running
once it runs.
With skip = true the node is still created and started; Sherpa just doesn't wait for it to be
ready, and ignores port. A container is not waited for to keep running, but one that is not running right after
it starts is still reported as a failed node, with its exit code and last log lines, and
sherpa up exits with an error.
vm (object)¶
Settings only a virtual machine takes.
[nodes.vm]
watchdog = "reset"
| Key | Type | Description |
|---|---|---|
watchdog |
string | Give the virtual machine a hardware watchdog, and choose what happens when it expires. Without it, Sherpa adds none |
watchdog actions:
| Action | What happens when the watchdog expires |
|---|---|
reset |
The VM is reset, as with the reset button |
poweroff |
The VM is powered off at once |
shutdown |
The guest is asked to shut down, as with the power button |
pause |
The VM is paused |
dump |
The VM's memory is dumped, and it keeps running |
inject-nmi |
The guest receives a non-maskable interrupt |
none |
Nothing; the device is present but has no effect |
The watchdog is an emulated Intel 6300ESB timer. It does nothing until the guest starts it, and
the guest must then keep resetting it: if the guest hangs and stops, the action is taken. It
helps only guests that run a watchdog driver and something that feeds it, for example Linux with
the i6300esb module and systemd's RuntimeWatchdogSec.
VMs with the q35 machine type also have the chipset's own iTCO watchdog, which libvirt adds by
itself and gives the same action, reset when watchdog is not set. It also does nothing until
the guest starts it.
Warning
Images that start the watchdog early in boot and feed it only once booted can reset in a
loop when the boot takes longer than the watchdog's timeout. Cisco IOS images do this,
especially under nested virtualization, so do not give them a reset watchdog.
container (object)¶
Settings only a container takes.
[nodes.container]
user = "root"
privileged = true
shm_size = 67108864
commands = ["sleep", "infinity"]
volumes = [
{ src = "/host/path", dst = "/container/path" },
]
platform = "bx"
| Key | Type | Description |
|---|---|---|
user |
string | The user the container runs as, in place of the image's |
privileged |
bool | Run the container in privileged mode |
shm_size |
int | Shared memory (/dev/shm) size in bytes |
commands |
list<string> | The container's command, in place of the image's |
volumes |
list<object> | Bind mounts of server paths into the container: src, the path on the Sherpa server, and dst, the path in the container |
platform |
string | The PTX router a juniper_evolved container emulates, see below |
Every container's hostname is its node name, whatever its model, and it keeps it across a
restart, down/resume and redeploy.
A privileged container, or one that can mount server paths, is root on the Sherpa server, so
only admins may set privileged or volumes; up and redeploy refuse a manifest from
another user that does.
Some models set their own command, user, privileges or mounts to start their software and mount their ZTP config; see the model's page under Nodes.
platform is bt (PTX10001-36MR, 12 data interfaces) or bx (PTX10002-36QDD, up to 36).
Without it, the node is bt. Only a juniper_evolved container takes platform. See
Juniper cJunosEvolved.
unikernel (object)¶
Settings only a unikernel takes.
[nodes.unikernel]
cmdline = "/usr/bin/nginx"
| Key | Type | Description |
|---|---|---|
cmdline |
string | Kernel command line arguments, passed to QEMU via the libvirt <cmdline> XML element. Overrides any auto-injected command line |
cmdline applies only to unikernels using the direct_kernel boot mode.
Tuning a virtual machine¶
A virtual machine's performance settings are in its cpu, numa, boot_disk and interfaces
groups: how its CPU is presented, which host CPUs it runs on, which NUMA node holds its memory,
and how its disks and interfaces work.
[[nodes]]
name = "dev"
model = "ubuntu_linux"
memory = 49152
[nodes.cpu]
count = 14
mode = "host-passthrough"
topology = { sockets = 1, cores = 7, threads = 2 }
pinning = [9, 25, 10, 26, 11, 27, 12, 28, 13, 29, 14, 30, 15, 31]
emulator_pinning = [8, 24]
[nodes.numa]
node = 1
memory_mode = "preferred"
[nodes.boot_disk]
cache = "none"
io = "io_uring"
io_thread_cpus = [8, 24]
[nodes.interfaces]
queues = 4
Host CPUs and NUMA nodes are numbered from 0, as on the host (lscpu, numactl -H).
Rules for pinned host CPUs:
cpu.topology,cpu.pinningandinterfaces.queuesneedcpu.countto be set.- A CPU in
cpu.pinningbelongs to that node alone. No other node can pin a vCPU to it, in any lab on the server, and no node can run its QEMU or I/O threads on it, the same node included. cpu.emulator_pinningandboot_disk.io_thread_cpusmay share CPUs with each other and with the same settings on other nodes.- The checks run at
sherpa upandsherpa redeploy. Every CPU and the NUMA node must exist on the server. - A node holds its pinned CPUs from the moment its
uporredeployis accepted until its lab is destroyed, also while the lab is stopped. During aredeploythat changes the pins, the node holds both the old and the new CPUs; a redeploy that fails before the new pins are in place keeps the old ones. - CPUs another node holds are refused with their own error, and the command exits with
status
75, like a memory refusal (CPU pins held).
CPU pins held¶
When a node pins a host CPU that a node of another lab holds, or a node another up or
redeploy is starting right now, the server refuses the command before it creates or tears
down anything. Of two ups started at once that ask for the same CPUs, the first to arrive
builds its lab and the other is refused. The error names the CPUs and what each node pins to
them, for example:
Lab creation failed: Host CPU 3, which node 'n1' would pin to its vCPUs, is pinned to the
vCPUs of node 'n1' of lab 'pins-a' (5c725af9). Try again once that node is destroyed or
redeployed without them (code: -32006)
The holding node and lab are named to admins and to that lab's owner. Everyone else reads "a
node of another lab" instead. sherpa up and sherpa redeploy exit with status 75
(exit status); run the command again once the CPUs are
free, or pin other CPUs.
Two nodes that already share a pinned CPU are not refused: sherpa resume and sherpa redeploy start such a node and show a warning that names
the CPUs, such as Warning: Host CPU 3 of node 'n1' is also pinned by a node of another lab,
against the CPU pinning rules. Starting the node anyway; redeploy one of the two on other host
CPUs to fix it. sherpad also logs each pair of nodes that share a CPU every time it starts.
A redeploy may keep the CPU it already shares, but cannot take more.
The memory of a node with numa.node and numa.memory_mode = "strict" (or left out) must fit
on that NUMA node beside the other virtual machines held strictly there, or sherpa up,
sherpa redeploy and sherpa resume refuse it (see
[host_memory]). Virtual machines without numa.node
may still fill that node, and the guest held there cannot take memory elsewhere: when the node
runs out, the host's OOM killer stops a virtual machine. Leave room on the node, pin every
large virtual machine on the server, or use numa.memory_mode = "preferred".
With numa.memory_mode = "preferred" the memory only counts against the whole server, not
against the NUMA node. When the node is full the guest takes memory from another node, which is
slower to reach, instead of being stopped.
Pinning keeps a virtual machine's threads on those CPUs. It does not reserve the CPUs: the host can still run other work on them.
Optional Parameters¶
bridges (list<bridge_object>)¶
Bridges define shared layer 2 network segments that connect multiple node interfaces.
Unlike links which create point-to-point connections between two nodes, bridges allow three or more interfaces to share a common network segment.
Each bridge link is defined with the node name and interface separated by a double colon (::)
[[bridges]]
name = "shared"
links = ["dev01::eth2", "dev02::eth2", "dev03::eth1"]
| Property | Type | Description |
|---|---|---|
name |
string | A user-friendly name for the bridge |
links |
list<string> | A list of <device>::<interface> mappings to connect to the bridge |
For more information on bridge types see the private-bridge architecture documentation.
description (string)¶
A short description of the lab's purpose. Optional.
name = "bgp-practice"
description = "Practice BGP peering and route policies between two routers."
Multiline strings and Unicode are supported, up to 1000 characters. Line breaks and tabs are
the only control characters allowed; sherpa validate and sherpa up refuse others.
The description is shown wherever the lab is: sherpa inspect, the web UI's labs list and lab
page, the admin labs page, and the API's lab details and lab list.
links (list<link_object>)¶
Links define the connectivity relationship between nodes in the topology.
Each device is defined with its name and interface separated by a double colon (::)
[[links]]
src = "dev01::eth0"
dst = "dev02::eth1"
Link Properties¶
| Property | Type | Required | Description |
|---|---|---|---|
src |
string | yes | Source <device>::<interface> mapping |
dst |
string | yes | Destination <device>::<interface> mapping |
impairment |
object | no | Network impairment configuration |
Link Impairment¶
Apply network impairment to simulate real-world link conditions. The values apply to both
directions. To impair each direction differently, or to change it on a running lab, use
sherpa link impair.
[[links]]
src = "dev01::eth1"
dst = "dev02::eth1"
[links.impairment]
delay = 50
jitter = 10
loss_percent = 1.0
| Property | Type | Description |
|---|---|---|
delay |
int | One-way delay in milliseconds, up to 274877 (about 4.5 minutes) |
jitter |
int | Delay jitter in milliseconds, up to 274877 |
loss_percent |
float | Packet loss percentage (0.0 - 100.0) |
reorder_percent |
float | Packet reordering percentage (0.0 - 100.0). Reordered packets skip the delay, so set a delay too |
corrupt_percent |
float | Bit-flip corruption percentage (0.0 - 100.0) |
sherpa validate and sherpa up refuse a value outside these ranges, naming the link.
Every link is a P2P eBPF link.
ready_timeout (int)¶
Global timeout in seconds for node readiness checks. Overrides the default timeout for all nodes in the lab.
It applies to sherpa up, sherpa redeploy and sherpa resume. It must be at least 1:
sherpa validate, sherpa up, sherpa redeploy and the server refuse 0.
ready_timeout = 300
provision_concurrency (int)¶
How many nodes the sherpa CLI runs ssh provisioners on at
the same time. Defaults to 8, and must be from 1 to 255. Use 1 to provision one node at a
time. Only the CLI reads it: it is not sent to the server.
provision_concurrency = 4
tailscale (object)¶
Connect the lab's management networks to the owner's Tailscale tailnet through a dedicated gateway. Nodes do not need their own Tailscale agent.
[tailscale]
enabled = true
auth_key_env = "SHERPA_TAILSCALE_KEY"
| Property | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Create a Tailscale gateway for the lab |
auth_key_env |
string | unset | Client environment variable containing the auth key; required when enabled |
Set the named variable in the client shell before running sherpa up and
sherpa resume. Store only the variable name in the manifest, never the key. The
key is required for creation and for each whole-lab resume, because sherpa down
logs the gateway out. Omitting the section
or setting enabled = false requires no credential and creates no gateway.
Enrollment requires server TLS and WSS with certificate validation enabled. Route approval is separate from enrollment; pending approval produces a warning without failing lab creation. See Tailscale setup and route approval.
talos_clusters (list<talos_cluster_object>)¶
Kubernetes clusters built from the lab's Talos Linux nodes. Each
cluster lists its member nodes by name. Sherpa generates each cluster's secrets and machine
configs, bootstraps it, and writes .sherpa/k8s/kubeconfig-<cluster> and
.sherpa/talos/talosconfig-<cluster> in your working directory.
[[talos_clusters]]
name = "k8s"
controllers = ["cp01", "cp02", "cp03"]
workers = ["worker01"]
[[nodes]]
name = "cp01"
model = "talos_linux"
[[nodes]]
name = "cp02"
model = "talos_linux"
[[nodes]]
name = "cp03"
model = "talos_linux"
[[nodes]]
name = "worker01"
model = "talos_linux"
| Property | Type | Description |
|---|---|---|
name |
string | Cluster name, also the Kubernetes cluster name. 1–63 lowercase letters, digits or - |
controllers |
list<string> | Nodes that run etcd and the Kubernetes control plane. Sherpa bootstraps the cluster on the first one |
workers |
list<string> | Nodes that run workloads only. Optional; without workers, workloads run on the controllers |
cni |
string | flannel (the default) or cilium. See Cilium |
oidc |
string | A keycloak_server node whose sherpa realm users log in to the cluster. Optional. See Keycloak login |
vault |
string | A hashicorp_vault node, without ztp.config, the cluster's workloads read secrets from. Optional. See Vault secrets |
Members must be talos_linux nodes defined in nodes, and a node can
be listed only once across all clusters. A talos_linux node that is not listed is a standalone
Talos node.
Each cluster needs one or three controllers. Two are refused: etcd needs a majority of its members to stay up, so two controllers survive no failure and are less available than one. A lab can have several clusters; at most two of them can have three controllers, and at most two can use Cilium.