K8TRE Installation
This guide provides instruction to quickly get K8TRE up and running on a local k8s cluster. It is structured with various entry points depending on your purpose, target machine setup and familiarity with technologies such as Kubernetes.
Before we can get cracking with K8TRE, we first need a Kubernetes (K8s) cluster available on your local host machine along with a single linux-based virtual machine to run the k8s cluster on.
Notes
All code blocks below coloured blue should be run on the host machine, while grey block commands should be run inside the Ubuntu VM
Prepare VM node
There are multiple ways to stand up and access a linux-based VM whether on a local machine or via some remote cloud resource. This guide assumes you have a local machine (Laptop?) without any existing framework for managing and deploying VMs. Although, if you already have an available framework setup (e.g. VirtualBox, VMware, etc) feel free to skip to the next section.
Update
This deployment guide has been updated to new versions of K3s (v1.35.3), Cilium (1.19.3) & ArgoCD (3.3.8) along with additional configuration to handle a larger k8tre cluster deployment footprint using native virtualisation on MacOS.
To spin up a local Ubuntu-based VM follow the instructions below based on your local machine OS:
UTM provides a framework for the creation of performant Linux-based VMs that directly uses Apple's virtualization framework. This is in contrast to Multipass which emulates hardware using QEMU.
-
Install UTM by downloading the installer here.
-
Download the Ubuntu 26.04 (Racoon) Server ISO image here
-
Open the UTM app and select the + button to start the creation of the VM. Select Virtualize > Linux.
-
Set appropriate hardware allocations for memory and CPU Cores depending on your host machine. For deploying the full KARECTL stack we recommend a minimum of 16GB RAM and 8 CPU Cores.
-
In the next pane, tick the 'Use Apple Virtualization' box and select Boot from ISO Image. Click Browse... and select the Ubuntu Server ISO image
-
Set the Storage size on the next pane to a minimum of 50GiB. Click continue until you reach the summary pane.
-
Click Save and then select the 'play' button to run the VM.
-
On booting, follow the Ubuntu installation steps and keep the defaults as suggested. As part of this, ensure the OpenSSH Server is installed.
-
Login into the VM with the username and password you specified in the Ubuntu installation steps.
-
Confirm the VM's IP address:
- Minimise the UTM VM window, open Terminal and ssh into the VM:
Option 1: Multipass
Multipass provides a simple approach to quickly spin up Ubuntu-based VMs on demand. It automatically downloads Ubuntu cloud images and handles all configuration.
-
Install Multipass by downloading the installer here or using Windows Package Manager in PowerShell:
Note
Multipass on Windows requires either Hyper-V (Windows Pro/Enterprise/Education) or VirtualBox (works on Windows Home).
- Hyper-V: Enable it in Windows Features -> Turn Windows features on or off -> Check "Hyper-V"
- VirtualBox: If on Windows Home or prefer VirtualBox, install VirtualBox first, then run:
multipass set local.driver=virtualbox
-
Check multipass is installed and accessible from PowerShell or Command Prompt:
-
Create a VM with Ubuntu 24.04 (this automatically downloads the Ubuntu image):
VirtualBox Network Configuration
When using VirtualBox as the driver, the VM often fails to obtain an IPv4 address. If
multipass info k8tre-vmshows no IPv4 address, you need to specify the network adapter during VM creation.First, delete the VM if already created:
To find your available network adapters, run:
Then recreate with the
--networkflag, specifying your network adapter (usually "Wi-Fi"): -
Check the VM is up and running:
You should see output similar to: - Open a terminal to the VM:
Option 2: VMware Workstation
If you prefer VMware Workstation, follow these steps:
-
Download and install VMware Workstation Pro:
- Visit Broadcom VMware Downloads
- Download VMware Workstation Pro for Windows
- Install and obtain free personal use license key
-
Download Ubuntu 24.04 Server ISO:
- Visit Ubuntu Downloads
- Download Ubuntu 24.04 LTS Server ISO
-
Create new VM in VMware:
- Open VMware Workstation
- File -> New Virtual Machine -> Typical
- Select "Installer disc image file (iso)" and browse to downloaded Ubuntu ISO
- VM Name:
k8tre-vm - Disk size: 12 GB (single file)
- Customize Hardware:
- Memory: 8 GB
- Processors: 2 CPUs
- Network Adapter: NAT or Bridged
-
Install Ubuntu:
- Start the VM and follow Ubuntu installation wizard
- Use default options, create user (e.g.,
ubuntu) - Wait for installation to complete and reboot
-
Install VMware Tools (for better performance):
-
Access VM terminal:
- Open the VM console in VMware Workstation, or
- SSH to the VM using the IP shown in VMware (run
ip ain VM to find IP)
Configure Ubuntu VM
Prepare the VM to run KARECTL on K3s by installing the following packages and applying configuration settings below:
sudo apt-get install -y nfs-common
sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/limits.conf > /dev/null <<EOF
[Manager]
DefaultLimitNOFILE=1048576
DefaultLimitNPROC=32768
DefaultLimitINOTIFY=524288
EOF
sudo tee /etc/sysctl.d/99-inotify.conf <<EOF
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=2048
fs.inotify.max_queued_events=65536
EOF
sudo sysctl --system
sudo systemctl daemon-reexec
# Ensure iscsid is available and enabled for Longhorn volume management support. If not, run sudo apt-get install -y open-iscsi && sudo systemctl enable --now iscsid
sudo systemctl status iscsid
Kubernetes Cluster (K3s)
At this point, we're ready to install K3s, a lightweight Kubernetes cluster distribution. You should have a target VM running Ubuntu 24.04 on your local host machine (or maybe remotely) accessible via your chosen command line tool.
1. Install K3s on the VM
Execute the following commands in terminal to download and install K3s onto the VM:
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml << EOF
node-name: k8tre-vm
tls-san:
- k8tre-dev
cluster-init: true
flannel-backend: none
disable-network-policy: true
disable:
- traefik
- servicelb
EOF
curl -sfSL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.3+k3s1 sh -
2. Cluster Access
Ensure the logged in user (e.g. ubuntu) can access the cluster by setting the user's kube/config:
mkdir -p ~/.kube
sudo cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc; source ~/.bashrc
3. Install Gateway API CRDs
K8TRE uses the Kubernetes Gateway API for ingress routing. Install the Gateway API CRDs before configuring the cluster networking:
kubectl apply --force-conflicts --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/experimental-install.yaml
4. Cluster Networking
K8TRE requires the target k8s cluster supports the Cilium container network interface (CNI) that provides modern support for network routing (e.g. cilium gateway) and access control management capabilities. The following commands install the cilium CLI on the ubuntu VM:
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
Then install cilium into the k3s cluster with Gateway API support and Hubble observability:
CILIUM_VERSION=1.19.3
K3S_POD_CIDR=10.42.0.0/16
cilium install --version $CILIUM_VERSION \
--set ipam.operator.clusterPoolIPv4PodCIDRList="$K3S_POD_CIDR" \
--set cni.chainingMode=portmap \
--set kubeProxyReplacement=true \
--set gatewayAPI.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true
In addition, install the portmap cilium CNI plugin for hostport support:
sudo mkdir -p /opt/cni/bin/
curl -sfSL https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-${CLI_ARCH}-v1.9.1.tgz | sudo tar -zxvf - -C /opt/cni/bin/ ./portmap
To ensure that Cilium is ready and configured in the cluster run:
Cluster DNS
Verify that cluster DNS resolution is working correctly:
If not, your VM may be operating within a private network on your host machine. The ubuntu VM resolver config which the default coreDNS service uses to forward DNS traffic may be unreachable (e.g. 127.0.0.53) from the coreDNS pod. To correct this run:
This will show what the underlying DNS server (NAT gateway) is being used for outbound traffic e.g.:Link 2 (enp0s1)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.64.1
DNS Servers: 192.168.64.1 fe80::a88f:d9ff:fe95:5e64
ArgoCD
K8TRE follows a declarative approach to deploy all agnostic and application-level components into a target cluster from a source git repository. To manage and automate this process, K8TRE relies on ArgoCD. If ArgoCD and GitOps model is unfamiliar, we first recommand gaining a brief understanding of what Argo is and why it is central to K8TRE here.
1. Install to Cluster
ARGOCD_VERSION=v3.3.8
kubectl create namespace argocd
kubectl apply --force-conflicts --server-side -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/$ARGOCD_VERSION/manifests/install.yaml
sleep 10
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s
Then install the argocd CLI tool:
sudo curl -sfSL https://github.com/argoproj/argo-cd/releases/download/$ARGOCD_VERSION/argocd-linux-${CLI_ARCH} -o /usr/local/bin/argocd
sudo chmod a+x /usr/local/bin/argocd
2. Portal Access
Expose ArgoCD (running in the k8s cluster) and provide access via the ArgoCD CLI tool and web-based management UI:
Before attempting to login to ArgoCD via CLI or web portal, first extract the initial password set up for the admin user:ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
echo $ARGOCD_PASSWORD
To access the web UI from your host web browser use the same URL and credentials. Note, for multipass VMs, replace localhost with the allocated IP address (run multipass info k8tre-vm to view the IP):
3. Set Cluster Labels
This command sets required labels on the target cluster which ArgoCD uses to ensure the correct agnostic/application configurations are applied for the given k8s distribution, in this example, definitions for k3s.
Note
The external-domain label defines the base domain used for all K8TRE services (e.g., keycloak.dev.k8tre.org, jupyter.dev.k8tre.org). Change k8tre.org to your own domain name.
Note
Specify a IP range for the local load balancer (metallb-ip-range) that is accessible from the bridge network your VM is bound. To check the network in use, follow the steps for your host OS below:
To identity the IP range of the VM network run:
This will return the network gateway IP e.g.:
Using 172.26.64.1 (and assuming a 255.255.255.0 net mask) i.e. 172.26.64.0-172.26.64.255 a example subnet for metallb-ip-range could be 172.26.64.240-172.26.64.250
argocd cluster set in-cluster \
--label environment=dev \
--label secret-store=kubernetes \
--label vendor=k3s \
--label external-domain=dev.k8tre.org \
--label external-dns=k3s \
--label storage-class=k3s \
--label metallb-ip-range=<e.g. 172.26.64.240-172.26.64.250>
4. Enable Kustomize Helm
This configuration update to ArgoCD allows kustomize (which argoCD uses under the hood) to render helm charts inside a build. Several K8TRE components (e.g. external-dns) install resources using Helm charts.
cat << EOF > argocd-cm-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
data:
kustomize.buildOptions: "--enable-helm --load-restrictor LoadRestrictionsNone"
EOF
kubectl apply -f argocd-cm-patch.yaml
kubectl rollout restart deployment argocd-repo-server -n argocd
5. Install Custom ArgoCD Plugin
K8TRE uses a custom ArgoCD plugin to enable environment variable substitution in Kustomize builds. This allows dynamic configuration across different environments (dev, stg, prd).
First, create the plugin configuration:
cat << EOF > cmp-plugin.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cmp-plugin
namespace: argocd
data:
plugin.yaml: |
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: kustomize-with-envsubst
spec:
version: v1.0
generate:
command: [sh, -c]
args:
- |
# Environment variables ENVIRONMENT, DOMAIN, and METALLB_IP_RANGE are passed from ApplicationSet via plugin.env
# ArgoCD makes them available as \$ARGOCD_ENV_<NAME> in the plugin container
# Replace patterns: \${VAR}, .ENVIRONMENT., .DOMAIN, and standalone ENVIRONMENT/DOMAIN
kustomize build --enable-helm --load-restrictor LoadRestrictionsNone . | \\
sed "s|\\\${ENVIRONMENT}|\${ARGOCD_ENV_ENVIRONMENT}|g; s|\\\${DOMAIN}|\${ARGOCD_ENV_DOMAIN}|g; s|\\\${METALLB_IP_RANGE}|\${ARGOCD_ENV_METALLB_IP_RANGE}|g; s|\\.ENVIRONMENT\\.|.\${ARGOCD_ENV_ENVIRONMENT}.|g; s|\\.DOMAIN|.\${ARGOCD_ENV_DOMAIN}|g; s|^ENVIRONMENT$|\${ARGOCD_ENV_ENVIRONMENT}|g; s|^DOMAIN$|\${ARGOCD_ENV_DOMAIN}|g"
EOF
kubectl apply -f cmp-plugin.yaml
Then patch the ArgoCD repo-server to add the plugin sidecar:
cat << EOF > add-cmp-sidecar.yaml
- op: add
path: /spec/template/spec/containers/-
value:
name: cmp-kustomize-envsubst
command:
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:$ARGOCD_VERSION
securityContext:
runAsNonRoot: true
runAsUser: 999
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
- mountPath: /home/argocd/cmp-server/plugins
name: plugins
- mountPath: /tmp
name: tmp
- mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: plugin.yaml
name: cmp-plugin
- op: add
path: /spec/template/spec/volumes/-
value:
name: cmp-plugin
configMap:
name: cmp-plugin
EOF
kubectl patch deployment argocd-repo-server -n argocd --type=json --patch-file add-cmp-sidecar.yaml
kubectl rollout status deployment argocd-repo-server -n argocd
6. Add Cilium Network Policy
The command below adds a networking policy that allows specific argocd services egress:
kubectl apply -f - << 'EOF'
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-argocd-egress
namespace: argocd
spec:
endpointSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- argocd-repo-server
- argocd-dex-server
- argocd-notifications-controller
egress:
- toEntities:
- cluster
- host
- toCIDR:
- 0.0.0.0/0
toPorts:
- ports:
- port: "443"
protocol: TCP
- port: "80"
protocol: TCP
EOF
K8TRE
You should now have a minimal VM and k3s cluster configuration (i.e. cilium, ArgoCD) to install K8TRE. To do this, we need to first configure ArgoCD to listen to the K8TRE repository that contains the agnostic/application definitions which ArgoCD will use to reconcile and deploy all specified resources in to the cluster. It is recommended that you point ArgoCD K8TRE repository for the purposes of this quickstart guide. However, if you plan to make changes to the vanilla K8TRE deployment, we recommend you fork it into your own git organisation and then configure ArgoCD to read from that particular repository and target revision. The K8TRE repository follows a common ArgoCD App-of-Apps pattern which is important to understand when looking to extend the base K8TRE configuration, see here for more details.
1. Add Target Repository
Add the target K8TRE repository to argoCD:
GITHUB_ORG=k8tre
GITHUB_REPOSITORY=k8tre
GITHUB_REVISION=main
argocd repo add https://github.com/$GITHUB_ORG/$GITHUB_REPOSITORY.git
2. Clone Target Repository
On the VM, clone the target K8TRE argo AoA repository:
Modify the default K8TRE root app-of-apps manifest so that it reads from the correct git repository and branch:sed -i -e "s%/k8tre/k8tre%/${GITHUB_ORG}/${GITHUB_REPOSITORY}%" -e "s%main%${GITHUB_REVISION}%" app_of_apps/root-app-of-apps.yaml
3. Apply K8TRE App Of Apps Manifest
This command applies the K8TRE AoA definition into the cluster which ArgoCD will begin to reconcile resources into the k3s cluster based on the agnostic/app definitions specified in the target repository.
The health status of K8TRE applications can be viewed via the ArgoCD web portal (i.e. http://Accessing K8TRE from your host machine
You have two options to access K8TRE from your host machine- custom DNS forwarding, or running a remote desktop alongside K8TRE.
Host DNS Configuration
For your host machine to resolve K8TRE service domains (e.g., portal.dev.k8tre.org), configure split DNS forwarding to route environment-specific domains to kare-dns.
Get the kare-dns LoadBalancer IP:
On your host machine, create a persistent DNS configuration matching the environment and domain from ArgoCD cluster labels:
sudo mkdir -p /etc/systemd/resolved.conf.d/
sudo tee /etc/systemd/resolved.conf.d/k8tre.conf << EOF
[Resolve]
DNS=<kare-dns-EXTERNAL-IP>
Domains=~<environment>.<domain>
EOF
For example, with environment=dev and external-domain=dev.k8tre.org:
sudo tee /etc/systemd/resolved.conf.d/k8tre.conf << EOF
[Resolve]
DNS=192.168.64.240
Domains=~dev.k8tre.org
EOF
The ~ prefix enables split DNS only dev.k8tre.org queries forward to kare-dns, all others use upstream DNS.
Restart systemd-resolved to apply:
Verify the configuration by running the following command. You should see a DNS resolution entry for the config defined above.
Only queries for dev.k8tre.org go to kare-dns
Set primary and secondary DNS servers Primary: kare-dns (172.26.71.210) Secondary: Google DNS (8.8.8.8) for other domains Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Default Switch)" -ServerAddresses ("172.26.64.212", "8.8.8.8")
Flush DNS cache
Test split DNS - should resolve via kare-dnsResult: 172.26.71.212
Test regular DNS - should work via Google DNS:
Create a remote desktop
You can deploy a remote Linux desktop container that require no local configuration. Note that this container runs on the same K3S cluster, but is outside K8TRE. It uses NoVNC to provide the desktop.
Be aware this container has no authentication, anyone with access to the port can access the desktop.
KAREDNS_COREDNS_IP=$(kubectl get svc kare-dns-coredns -n kare-dns -ojsonpath='{.spec.clusterIP}')
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: k8tre-access
spec:
containers:
- name: mate
image: docker.io/jlesage/firefox:latest
ports:
- containerPort: 5800
dnsPolicy: None
dnsConfig:
nameservers:
- ${KAREDNS_COREDNS_IP}
EOF
Setup a port-forward for port 5800:
and go to http://localhost:58004. K8TRE Secrets Management
K8TRE components can utilise its default secrets management service based on the K8s operator External Secrets Operator. This provides TRE operators with an abstraction layer to provision secrets from a range of commonly used key management solutions (e.g. Azure key Vault, AWS Secrets Manager). Out-of-the-box components in K8TRE require certain secrets to be defined and accessible in the cluster. For the purposes of this guide, we also need to create these secrets. In the K8TRE repository, we include a helper script to generate keys/secrets needed by core services running in the K8TRE deployment now running on your local machine. To generate the secrets ensure you are still in the k8tre/ repo you cloned earlier and first install uv by running the following:
Then execute the following command to create the secrets in the cluster:You should see a confirmation output describing the secrets and keys generated.
All services shown in the ArgoCD UI portal should, after a few minutes of reconciliation, all be in a healthy (green) state.
Next Steps
Now you have K8TRE running, why not try to access the default portal and open a K8TRE workspace: