Talos Linux¶
Talos Linux is an immutable, API-managed operating system for Kubernetes. It has no SSH,
shell or console login: it is managed through the Talos API on TCP port 50000 with
talosctl, and Kubernetes
with kubectl.
Sherpa can run Talos nodes in two ways:
- Sherpa-built clusters. Declare a cluster in the manifest. Sherpa generates the cluster
secrets and machine configs, bootstraps Kubernetes, and gives you a
kubeconfigandtalosconfig. - Your own configs. Give each node a machine config you generated with
talosctl, or none, so the node waits in maintenance mode.
Base Image¶
Sherpa uses the NoCloud image from the Talos Image Factory. Pull the vanilla image (no system extensions):
sherpa server image pull \
--model talos_linux \
--version 1.14.4 \
--default
Note
Sherpa-built clusters require Talos 1.14.x. Talos changes its configuration format between minor versions, so Sherpa refuses other versions for cluster nodes. Nodes with your own configs can use any version.
For a custom schematic with extra system extensions, generate one at factory.talos.dev and pass its NoCloud image URL:
sherpa server image pull \
--model talos_linux \
--version 1.14.4 \
--url https://factory.talos.dev/image/<schematic-id>/v1.14.4/nocloud-amd64.qcow2 \
--default
Or import a downloaded image:
sherpa server image import \
--src /some/path/<image_name>.qcow2 \
--model talos_linux \
--version 1.14.4 \
--default
Default Node Parameters¶
| Property | Value |
|---|---|
| Tested Version | 1.14.4 |
| Kind | Virtual Machine |
| CPU | 2 |
| RAM | 4GB |
| Boot Disk | 20GB |
| CDROM Driver | SATA |
| DISK | virtioa.qcow2 |
| DISK Driver | virtio |
| ZTP | nocloud |
| Management Interface | eth0 (static address) |
| Data Interfaces | eth1 |
| Interface Driver | virtio |
| Readiness | Talos API, TCP port 50000 (cluster workers: Ready Kubernetes node) |
The image is about 4 GB; Talos needs at least 10 GB, so nodes get a 20 GB boot disk unless
boot_disk.size is set.
Sherpa-Built Clusters¶
Declare clusters with talos_clusters. Each cluster lists its
controllers (the control plane) and workers by node name:
name = "k8s-lab"
[[talos_clusters]]
name = "k8s"
controllers = ["cp01"]
workers = ["worker01", "worker02"]
[[nodes]]
name = "cp01"
model = "talos_linux"
[[nodes]]
name = "worker01"
model = "talos_linux"
[[nodes]]
name = "worker02"
model = "talos_linux"
sherpa up then:
- generates the cluster's certificate authorities, keys and tokens, and a machine config for each node, delivered on the node's NoCloud CD-ROM;
- waits for each controller's Talos API;
- bootstraps etcd on the first controller and waits for every node to be a Ready Kubernetes node. Workers are only checked here: a worker's Talos API starts once it can reach the cluster, which with three controllers is through the shared address that comes up after the bootstrap;
- writes
.sherpa/k8s/kubeconfig-<cluster>and.sherpa/talos/talosconfig-<cluster>in your working directory, readable only by you.
export KUBECONFIG=$PWD/.sherpa/k8s/kubeconfig-k8s TALOSCONFIG=$PWD/.sherpa/talos/talosconfig-k8s
kubectl get nodes
talosctl -n <node-ip> version
sherpa download writes the same files again.
Control plane endpoint¶
A cluster has one or three controllers. Two are refused: etcd needs a majority of its members to stay up, so two controllers survive no failure and are less available than one.
- One controller: the Kubernetes API is on that node's management address.
- Three controllers: Sherpa reserves a shared address (a Talos
VIP) from the lab's
management network:
.11for the first such cluster,.12for the second. The controllers elect which of them holds it. A lab can have up to two such clusters. See IP addressing.
If a cluster has no workers, workloads are allowed on its controllers.
Cilium¶
Set cni = "cilium" on a cluster to run Cilium instead of Flannel:
[[talos_clusters]]
name = "k8s"
controllers = ["cp01", "cp02", "cp03"]
workers = ["worker01", "worker02"]
cni = "cilium"
Talos installs Cilium while the cluster bootstraps, so the nodes are Ready when sherpa up
finishes. The cluster gets:
- Cilium 1.20.4 as the CNI, replacing kube-proxy.
- Load balancer addresses on the lab's management network, announced by Cilium. Sherpa
reserves four for each Cilium cluster:
.13–.16for the first,.17–.20for the second. A lab can have up to two Cilium clusters. See IP addressing. - A Gateway named
sherpa, in thedefaultnamespace, at the first address. It accepts HTTP on port 80 for HTTPRoutes from any namespace. The Gateway API CRDs (v1.6.3, standard channel) are installed. - Hubble UI, Cilium's view of traffic between pods, at the second address.
- The other two addresses for your own
LoadBalancerservices.
sherpa up and sherpa download print the Gateway and Hubble UI addresses:
✅ Cluster k8s credentials: /home/user/k8s/.sherpa/k8s/kubeconfig-k8s and /home/user/k8s/.sherpa/talos/talosconfig-k8s
export KUBECONFIG=/home/user/k8s/.sherpa/k8s/kubeconfig-k8s TALOSCONFIG=/home/user/k8s/.sherpa/talos/talosconfig-k8s
Gateway: http://172.31.1.13
Hubble UI: http://172.31.1.14
Route a service through the Gateway:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: my-app
namespace: my-namespace
spec:
parentRefs:
- name: sherpa
namespace: default
rules:
- backendRefs:
- name: my-app
port: 80
The Cilium settings are fixed. Cilium's images are pulled from quay.io when the cluster
starts, so nodes need internet access, as they do for the Kubernetes images.
See the Kubernetes example for a cluster to try it on. Its repository adds an HTTPS listener to the Gateway, with a certificate from Vault through cert-manager, and puts a Keycloak login in front of Hubble UI with the Gateway's external auth. Hubble UI's own address is open to the management network until you close it, as the repository does with a Cilium network policy.
Keycloak login¶
Set oidc to a Keycloak node in the lab, and users of that
node's sherpa realm can log in to the cluster:
[[talos_clusters]]
name = "k8s"
controllers = ["cp01"]
workers = ["worker01"]
oidc = "kc01"
[[nodes]]
name = "kc01"
model = "keycloak_server"
- The cluster accepts the realm's tokens. A user's Kubernetes name is
oidc:<username>, and their Keycloak groups becomeoidc:<group>. - Members of the realm's
cluster-adminsgroup are cluster admins (cluster-admin), and members ofcluster-viewerscan read most resources (view). Bind other groups to your own roles in the same way, for exampleoidc:developers. .sherpa/k8s/kubeconfig-<cluster>gets a second context,oidc@<cluster>, which logs in through Keycloak. The admin certificate context,admin@<cluster>, stays the default.
The oidc@<cluster> context uses the kubelogin plugin,
kubectl oidc-login. Install it, for example with kubectl krew install oidc-login, then:
kubectl --context oidc@k8s get nodes
The first command opens a browser at Keycloak's login page. Sign in with a user from
.sherpa/keycloak/keycloak-<node>.toml; kubelogin keeps the token until it expires. See the
Keycloak page for the realm's users.
Keycloak can start after the cluster: the Kubernetes API keeps retrying the realm until it answers.
The realm can log users in to your apps too. The Kubernetes example's repository adds a Keycloak client for Hubble UI through the admin API and puts the login in front of it.
Vault secrets¶
Set vault to a Hashicorp Vault node in the lab, and the
cluster's workloads can read secrets from it:
[[talos_clusters]]
name = "k8s"
controllers = ["cp01"]
workers = ["worker01"]
vault = "vault01"
[[nodes]]
name = "vault01"
model = "hashicorp_vault"
Sherpa:
- installs the Vault Secrets Operator
1.6.2 in the
vault-secrets-operator-systemnamespace, connected to the Vault node over HTTPS. It is installed about two minutes after the cluster starts, so it may still be installing whensherpa upfinishes; - sets up Vault for the cluster: a KV secrets engine at
secret/, and a login for the cluster's service accounts. Workloads can read the secrets undersecret/<cluster>/and no others.
Write a secret with the vault CLI, using the root token from .sherpa/vault/vault-<node>.toml:
vault kv put secret/k8s/demo password=s3cret
Then sync it into a Kubernetes Secret, in any namespace:
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: demo
namespace: my-namespace
spec:
type: kv-v2
mount: secret
path: k8s/demo
destination:
name: demo
create: true
refreshAfter: 30s
The operator logs in with the namespace's default service account. The Vault node must run
the config Sherpa generates, without ztp.config. The operator's chart and images are
downloaded when it is installed, so nodes need internet access.
Vault can issue the cluster's certificates as well. The Kubernetes example's repository makes Vault's PKI engine an intermediate CA under the lab CA, and gives the Gateway an HTTPS certificate from it through cert-manager, which logs in to Vault with the cluster's auth method.
Cluster defaults¶
| Setting | Value |
|---|---|
| Kubernetes version | 1.37.1 (the Talos 1.14 default) |
| CNI | Flannel, or Cilium |
| Pod network | 10.244.0.0/16 |
| Service network | 10.96.0.0/12 |
| Pod Security | baseline enforced |
The lab's management network must not overlap the pod or service networks.
Stopping nodes¶
Talos ignores the ACPI power button. sherpa down shuts cluster nodes down through the Talos
API instead, as talosctl shutdown does: each node is cordoned and drained, then powers off.
Workers stop before controllers. A node still running after 60 seconds is powered off.
Talos nodes that are not in a Sherpa-built cluster are powered off directly, because Sherpa has
no credentials for their Talos API. Shut them down with talosctl shutdown first if they must
stop cleanly.
Redeploy¶
- A worker can be redeployed: it is rebuilt from the cluster's saved secrets and rejoins.
- A controller cannot: its etcd member would have to be removed first. Destroy and recreate the lab instead.
- A redeploy keeps the node's cluster and role. The manifest must give the node the cluster
and role it has in the lab: one that moves it to another cluster, changes its role, or takes
it out of its cluster or into one is refused before the node is touched, naming the node, its
cluster and role in the lab, and the manifest's. To change a cluster, destroy the lab and run
sherpa upagain. - A cluster's Vault node can be redeployed. It starts empty: Sherpa sets it up for the cluster again, but you must write its secrets again.
- A cluster's Keycloak node can be redeployed. Sherpa makes the
sherparealm again, with the same users and passwords.
Your Own Configs¶
Generate machine configs with talosctl:
talosctl gen config my-cluster https://<controlplane-ip>:6443
Reference each node's config with ztp.config. It is
delivered byte for byte as the node's NoCloud user-data, alongside Sherpa's static
management network settings. Drop the UnattendedInstallConfig document: the NoCloud image is
already installed.
A node without a ztp.config boots into maintenance mode with its management address and
waits to be configured:
talosctl apply-config --insecure -n <node-ip> --file controlplane.yaml
name = "talos-byo"
[[nodes]]
name = "cp01"
model = "talos_linux"
[nodes.ztp]
config = "controlplane.yaml"
[[nodes]]
name = "worker01"
model = "talos_linux"
[nodes.ztp]
config = "worker.yaml"
[[nodes]]
name = "spare01"
model = "talos_linux"
With your own configs you run talosctl bootstrap and fetch the kubeconfig yourself.
Node Access¶
There is no SSH or console access. sherpa ssh and sherpa console do not apply to Talos
nodes, and they are left out of the lab's SSH config. Use talosctl and kubectl.
Notes¶
- Nodes pull Kubernetes images from
ghcr.ioandregistry.k8s.iothrough the lab's NAT, so the Sherpa server needs internet access. - Sherpa-built clusters do not use the Talos discovery service.
- A node is reported ready when its Talos API answers. Maintenance mode also answers, so for
your own configs ready means reachable, not configured. Override with
ready_check.port. - Approximate time from power-on to a Ready cluster: about 2 minutes.