Kubernetes¶
Sherpa builds and bootstraps a Kubernetes cluster on Talos Linux nodes, with Cilium as its network. This example has three controllers, which share a virtual IP for the Kubernetes API, and two workers. Cilium replaces kube-proxy, gives the cluster a Gateway on the lab's management network, and shows the traffic between pods in Hubble. Users log in to the cluster through Keycloak, and workloads read their secrets from Vault. A util node has the tools to work with all of it.
The lab is also a repository, bwks/sherpa-k8s-lab, with this manifest and tools script, its Kubernetes manifests, and a walkthrough to follow end to end. The walkthrough goes further than this page: it gives the Gateway an HTTPS listener with a certificate from Vault, and puts a Keycloak login in front of Hubble UI.
| Role | Nodes | Model |
|---|---|---|
| Controller | cp01, cp02, cp03 |
Talos Linux |
| Worker | wrk01, wrk02 |
Talos Linux |
| Identity | kc01 |
Keycloak |
| Secrets | vault01 |
Hashicorp Vault |
| Tools | util |
Ubuntu |
- Controllers run the Kubernetes control plane and etcd. Three controllers let the cluster survive the loss of one, because etcd keeps a majority with two of three members. A cluster has either one or three controllers.
- Workers run your workloads. With no workers, workloads would be allowed on the controllers instead.
- Virtual IP. The controllers share an address on the management network,
.11, which follows whichever controller holds it. The Kubernetes API stays on that address if a controller fails. - Gateway. A Gateway API Gateway named
sherpaat.13. Route HTTP to your services with HTTPRoutes. - Hubble UI at
.14: a live map of which pods talk to which, and what policy drops. - Keycloak login. The cluster accepts users of
kc01'ssherparealm; itscluster-adminsgroup are cluster admins. See Keycloak login. - Vault secrets. The Vault Secrets Operator syncs secrets from
vault01into Kubernetes Secrets. See Vault secrets. - Util node. An Ubuntu VM with
kubectl,talosctl, thekubectl oidc-loginplugin, thevaultCLI and theciliumCLI, installed by a startup script on its first boot. It trusts the lab CA, so the tools reach Keycloak and Vault over HTTPS without extra settings.
There are no data plane links: the nodes talk to each other, and Kubernetes pods talk across
nodes, over the management network. The addresses are in the lab's management network, so
.13 is 172.31.1.13 in a lab on 172.31.1.0/24. sherpa up prints them.
Running it takes about 25 GB of memory on the Sherpa server: each Talos node has 4 GB, Keycloak and the util node 2 GB each, and Vault 1 GB.
Manifest¶
name = "sherpa-kubernetes"
[[talos_clusters]]
name = "k8s"
controllers = ["cp01", "cp02", "cp03"]
workers = ["wrk01", "wrk02"]
cni = "cilium"
oidc = "kc01"
vault = "vault01"
[[nodes]]
name = "cp01"
model = "talos_linux"
[[nodes]]
name = "cp02"
model = "talos_linux"
[[nodes]]
name = "cp03"
model = "talos_linux"
[[nodes]]
name = "wrk01"
model = "talos_linux"
[[nodes]]
name = "wrk02"
model = "talos_linux"
[[nodes]]
name = "kc01"
model = "keycloak_server"
[[nodes]]
name = "vault01"
model = "hashicorp_vault"
[[nodes]]
name = "util"
model = "ubuntu_linux"
memory = 2048
startup_scripts = ["install-tools.sh"]
[nodes.boot_disk]
size = 10
Save install-tools.sh next to the manifest. On the
util node's first boot it installs:
| Tool | Version |
|---|---|
kubectl |
1.37.1, the cluster's Kubernetes version |
talosctl |
1.14.4 |
kubectl oidc-login (kubelogin) |
1.36.4 |
vault |
1.21.4 |
cilium |
0.20.2 |
Each download is pinned and checked against its published SHA-256. The util node gets more memory and disk than an Ubuntu node's defaults (1 GB and the image's 2.3 GB), which the tools need.
Access¶
After sherpa up, the .sherpa folder in your working directory has the cluster's
k8s/kubeconfig-k8s and talos/talosconfig-k8s, the lab CA in ca/lab-ca.crt, Vault's root
token in vault/vault-vault01.toml, and the Keycloak realm's users in
keycloak/keycloak-kc01.toml. The output ends with:
✅ Lab CA certificate: /home/user/k8s/.sherpa/ca/lab-ca.crt
✅ Vault vault01 credentials: /home/user/k8s/.sherpa/vault/vault-vault01.toml
export VAULT_ADDR=https://172.31.1.27:8200 VAULT_CACERT=/home/user/k8s/.sherpa/ca/lab-ca.crt
✅ Keycloak kc01 users: /home/user/k8s/.sherpa/keycloak/keycloak-kc01.toml
Realm issuer: https://172.31.1.26:8443/realms/sherpa
✅ Cluster k8s credentials: /home/user/k8s/.sherpa/k8s/kubeconfig-k8s and /home/user/k8s/.sherpa/talos/talosconfig-k8s
export KUBECONFIG=/home/user/k8s/.sherpa/k8s/kubeconfig-k8s TALOSCONFIG=/home/user/k8s/.sherpa/talos/talosconfig-k8s
Gateway: http://172.31.1.13
Hubble UI: http://172.31.1.14
Copy the credential files to the util node and log in to it. Its SSH host name is
util.<lab-id>, as in .sherpa/ssh/config:
scp -F .sherpa/ssh/config .sherpa/k8s/kubeconfig-k8s .sherpa/talos/talosconfig-k8s .sherpa/vault/vault-vault01.toml .sherpa/keycloak/keycloak-kc01.toml util.<lab-id>:
sherpa ssh util
Use kubectl for Kubernetes and talosctl for the Talos nodes:
export KUBECONFIG=$HOME/kubeconfig-k8s TALOSCONFIG=$HOME/talosconfig-k8s
kubectl get nodes
talosctl -n <node-ip> version
cilium status
The tools work from your workstation too, if you install them there. Talos nodes have no SSH
or shell, so sherpa ssh does not apply to them. If you lose the config files,
sherpa download writes them again.
Try It¶
Run the steps below on the util node. They use the addresses of a lab on 172.31.1.0/24; use
the ones sherpa up printed. The repository's walkthrough
has the same steps, with the Kubernetes manifests as files, and two more:
- HTTPS on the Gateway:
cert-manager, with an issuer backed by Vault's PKI engine on
vault01, gives thesherpaGateway an HTTPS listener. The certificate chains to the lab CA, which the util node trusts. - A Keycloak login in front of Hubble UI:
Hubble UI moves behind the Gateway, which sends every request to oauth2-proxy through the
Gateway API
ExternalAuthfilter, and oauth2-proxy logs the user in withkc01'ssherparealm.
Publish an app through the Gateway¶
Run two replicas of an echo server and route the Gateway to them:
kubectl create namespace demo
kubectl -n demo create deployment echo --image=registry.k8s.io/echoserver:1.10 --replicas=2
kubectl -n demo expose deployment echo --port=80 --target-port=8080
kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo
namespace: demo
spec:
parentRefs:
- name: sherpa
namespace: default
rules:
- backendRefs:
- name: echo
port: 80
EOF
Each request answers from one of the two pods:
curl -s http://172.31.1.13/ | grep Hostname
Open the Hubble UI at http://172.31.1.14 and pick the demo namespace to watch the
requests arrive.
Block it with a network policy¶
Deny all traffic into the demo namespace:
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: demo
spec:
podSelector: {}
policyTypes: [Ingress]
EOF
curl now times out, and Hubble shows the requests dropped by policy. Allow traffic from the
Gateway only, with a Cilium policy:
kubectl apply -f - <<'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-gateway
namespace: demo
spec:
endpointSelector:
matchLabels:
app: echo
ingress:
- fromEntities: [ingress]
EOF
The Gateway reaches the pods again, while other pods in the cluster still cannot.
Log in through Keycloak¶
The kubeconfig's oidc@k8s context logs in through Keycloak in a browser. On a workstation
with the kubelogin plugin (for example
kubectl krew install oidc-login):
kubectl --context oidc@k8s auth whoami
A browser opens at Keycloak. Sign in as yourself, with the password from
.sherpa/keycloak/keycloak-kc01.toml,
and kubectl shows you as oidc:<username> in the group oidc:cluster-admins.
The util node has no browser. Add a context there that asks for the username and password instead:
kubectl config set-credentials oidc-password@k8s \
--exec-api-version=client.authentication.k8s.io/v1 \
--exec-command=kubectl \
--exec-arg=oidc-login --exec-arg=get-token \
--exec-arg=--oidc-issuer-url=https://172.31.1.26:8443/realms/sherpa \
--exec-arg=--oidc-client-id=kubernetes \
--exec-arg=--grant-type=password \
--exec-interactive-mode=IfAvailable
kubectl config set-context oidc-password@k8s --cluster=k8s --user=oidc-password@k8s
kubectl --context oidc-password@k8s auth whoami
To give someone read-only access, add a user in Keycloak's admin console
(https://172.31.1.26:8443/admin/, as sherpa) to the sherpa realm and the
cluster-viewers group. They can read most resources, but not change them or read Secrets.
Read a secret from Vault¶
Point the vault CLI at the node and write a secret under the cluster's path. The util node
trusts the lab CA; on a workstation, also set VAULT_CACERT=$PWD/.sherpa/ca/lab-ca.crt.
export VAULT_ADDR=https://172.31.1.27:8200
export VAULT_TOKEN=<root_token from .sherpa/vault/vault-vault01.toml>
vault kv put secret/k8s/demo password=s3cret
Sync it into the demo namespace:
kubectl apply -f - <<'EOF'
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: demo
namespace: demo
spec:
type: kv-v2
mount: secret
path: k8s/demo
destination:
name: demo
create: true
refreshAfter: 30s
EOF
kubectl -n demo get secret demo -o jsonpath='{.data.password}' | base64 -d
Change the secret with vault kv put and the Kubernetes Secret follows within 30 seconds.
Secrets outside secret/k8s/ are refused to the cluster.