Kata Containers Isolation Tier

Run agents in an environment under Kata Containers — each agent boots in a lightweight QEMU/KVM virtual machine with its own guest kernel, so there is no shared kernel with the host at all. This is the strongest isolation tier, intended for the most untrusted or multi-tenant agent workloads.

Read the gVisor isolation tier guide first if you haven't — Kata uses the same environment-level mechanism and the same dedicated-node model; this page covers what is Kata-specific.

Hard requirement: KVM (nested virtualization)

Kata boots a real VM per agent, so the node must expose /dev/kvm:

• ✅ Bare-metal Linux with VT-x (Intel) or AMD-V
• ✅ A nested-virtualization-capable cloud VM — for example GCP Intel N2 with --enable-nested-virtualization, or AWS EKS bare-metal (*.metal) instances
• ❌ macOS / Colima — no nested virtualization; use gVisor locally instead
• ❌ GCP E2, AMD N2D, ARM T2A machine types, and Container-Optimized OS (read-only filesystem)
• ❌ k3d / kind node containers — even on a KVM-capable host, the Kata guest-agent handshake cannot complete inside a container-based node; Kata needs a real Linux node

Verify on the node:

ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # expect: Y (Intel)

How It Works

Identical to gVisor, with one naming detail: the isolation tier is called kata, but the RuntimeClass (and containerd handler) it maps to is kata-qemu — that is the handler kata-deploy registers.

Environment (isolation tier: kata)
  └─ deploy / promote → runtimeClassName: kata-qemu
       └─ RuntimeClass "kata-qemu" scheduling → pod on the Kata node, QEMU/KVM VM + guest kernel

Like gVisor, Kata uses a dedicated node labeled kata=true and tainted kata=true:NoSchedule, so existing runc (and gVisor) agents are never touched — no downtime.

Requirements

Requirement	Details
A new (empty) Linux node	Joined to the same Kubernetes cluster; dedicated to Kata agents
Virtualization	/dev/kvm present and functional (see warning above)
Architecture	x86_64
OS	Ubuntu (or similar) with systemd + containerd — not Container-Optimized OS
Access	kubectl access to the cluster (the installer runs from your machine, not on the node)

Set Up a Kata Node

Step 1: Add a new node to the cluster

Provision a KVM-capable Linux node matching the requirements above and join it to your cluster using your platform's standard mechanism. Leave it empty.

Taint the node as early as possible

If your platform lets you set taints at node-pool creation, apply kata=true:NoSchedule there. An untainted node starts accepting regular platform pods immediately, and you will have to drain them all off before installing the runtime (the containerd reconfiguration restarts every pod on the node). If the node has already picked up workloads, kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data it first.

Step 2: Run the installer from a machine with kubectl access

Unlike the gVisor installer (which runs on the node), the Kata installer runs from any machine with kubectl access and does everything through the Kubernetes API:

curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/setup/install-kata.sh \
  -o install-kata.sh

KATA_NODES="<node-name>" bash install-kata.sh

What it does:

1. Pre-flights /dev/kvm on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
2. Labels the node(s) kata=true.
3. Applies the upstream kata-deploy DaemonSet, with a nodeSelector and toleration injected before apply — so it only ever lands on the labeled node(s); your other nodes are never reconfigured.
4. Auto-wires k3s nodes: k3s bundles its own containerd (configured only via config.toml.tmpl, read at startup), which upstream kata-deploy does not handle — the installer detects k3s nodes and patches the template + restarts the k3s agent for you.
5. Applies the kata-qemu RuntimeClass and taints the node(s) kata=true:NoSchedule.
6. Patches the Fluent Bit log DaemonSet to tolerate the taint, so agent logs are collected.

Verify

kubectl get runtimeclass kata-qemu
kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl get node <node-name> --show-labels   # kata=true

RKE2 nodes (bundled containerd)

Upstream kata-deploy assumes a standalone containerd at /etc/containerd/config.toml, which RKE2 nodes do not have. Two extra steps are needed on each RKE2 Kata node:

1. Before running the installer, create an empty stub so the kata-deploy pod gets past its pre-flight check (it crash-loops with cp: can't stat '/etc/containerd/config.toml' otherwise):

sudo mkdir -p /etc/containerd && sudo touch /etc/containerd/config.toml

2. After kata-deploy has installed the binaries into /opt/kata (the node gets the katacontainers.io/kata-runtime=true label on success), wire the runtime through RKE2's containerd template — kata-deploy wrote its config to the stub, which RKE2 never reads:

sudo mkdir -p /usr/local/bin
sudo ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-qemu-v2
sudo tee /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl > /dev/null <<'EOF'
{{ template "base" . }}

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
  runtime_type = "io.containerd.kata-qemu.v2"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
    ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
EOF
sudo systemctl restart rke2-agent

Rewrite, never append

The template must contain {{ template "base" . }} exactly once. If the file already exists (e.g. it also carries a gVisor or build-related block), merge the runtime sections into one file and rewrite it wholesale; appending a duplicate template line produces an invalid containerd config and the node's kubelet will not come back after the restart — recovery then requires SSH/console access, kubectl cannot reach the node.

Local development

There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable Linux host running the repository's dev k3d cluster:

make setup-kata

This joins the host itself to the cluster as a real (non-Docker) k3s agent node and then runs install-kata.sh against it.

Create a Kata Environment

Isolation tier is set at environment creation time. Open the Create Environment drawer in the console (Environments → Create Environment), select Sandboxing T3 (Kata Containers) from the Isolation Tier dropdown, and run the generated command — it includes ISOLATION_TIER=kata automatically.

If you script environment creation directly, set the variable yourself:

curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment.sh \
  -o add-environment.sh

ISOLATION_TIER=kata \
  ENV_NAME=secure-prod \
  DISPLAY_NAME="Secure Prod" \
  AGENT_MANAGER_TOKEN=<token> \
  bash add-environment.sh

The tier appears as a Sandboxing T3 shield chip in the environment list, and as a shield icon next to the environment name on the agent's Deploy and Overview pages — hover over it to confirm Sandboxing Tier 3 — Kata Containers. Then deploy or promote an agent to the environment as usual.

Verify VM Isolation

A Kata agent runs a different kernel from the node — that's the definitive check:

kubectl get pod <agent-pod> -n <namespace> -o jsonpath='{.spec.runtimeClassName}'   # kata-qemu

kubectl exec <agent-pod> -n <namespace> -- uname -r
# Differs from the node's own `uname -r`

kubectl exec <agent-pod> -n <namespace> -- grep hypervisor /proc/cpuinfo
# hypervisor flag present

If the pod reports the same kernel as the node, it is not running under Kata (the RuntimeClass wasn't applied or the pod fell back to runc).

Troubleshooting

Pod stuck in ContainerCreating

Almost always missing nested virtualization. Confirm on the node itself:

ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # expect: Y
kubectl describe pod <pod> -n <ns>            # look for kvm / RuntimeClass errors

No /dev/kvm → move to a bare-metal or nested-virt-enabled host (e.g. GCP Intel N2 with --enable-nested-virtualization).

no runtime handler kata-qemu errors

The RuntimeClass exists but containerd on the node has no kata-qemu runtime — the install didn't complete or containerd wasn't reloaded:

kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl -n kube-system logs -l name=kata-deploy

# on the node — confirm the runtime is in containerd's config:
sudo grep -n kata-qemu /var/lib/rancher/k3s/agent/etc/containerd/config.toml   # k3s nodes
grep -n kata-qemu /etc/containerd/config.toml                                  # other nodes

On k3s nodes, re-running install-kata.sh re-applies the containerd template wiring.

Pod runs but no logs / metrics / traces / try-it

This is cross-node networking or log-collection scheduling, not Kata itself — the failure modes and fixes are identical to gVisor's. Work through the gVisor troubleshooting section, substituting runtimeClassName: kata-qemu in the test pod.

Performance notes

• Booting a VM adds startup latency compared to runc/gVisor; Agent Manager's warm pools absorb most of it.
• kubectl port-forward should be avoided for Kata pods — reach agents through their Service/gateway endpoint.
• Usage metrics are attributed to the sandbox VM as a whole (reported under a sandbox container label), including the guest kernel's footprint.

Next Steps

• Lighter-weight syscall isolation without KVM: gVisor isolation tier.
• Environment management — pipelines, promotion, and environment settings.