# Kata Containers Isolation Tier

Run agents in an environment under [Kata Containers](https://katacontainers.io/) — each agent boots in a lightweight QEMU/KVM virtual machine with its **own guest kernel**, so there is no shared kernel with the host at all. This is the strongest isolation tier, intended for the most untrusted or multi-tenant agent workloads.

Read the [gVisor isolation tier guide](/agent-manager/docs/next/administration/isolation-tiers/gvisor/.md) first if you haven't — Kata uses the same environment-level mechanism and the same dedicated-node model; this page covers what is Kata-specific.

Hard requirement: KVM (nested virtualization)

Kata boots a real VM per agent, so the node **must expose `/dev/kvm`**:

* ✅ Bare-metal Linux with VT-x (Intel) or AMD-V
* ✅ A nested-virtualization-capable cloud VM — for example GCP **Intel N2** with `--enable-nested-virtualization`, or AWS EKS bare-metal (`*.metal`) instances
* ❌ macOS / Colima — no nested virtualization; use gVisor locally instead
* ❌ GCP E2, AMD N2D, ARM T2A machine types, and Container-Optimized OS (read-only filesystem)
* ❌ k3d / kind **node containers** — even on a KVM-capable host, the Kata guest-agent handshake cannot complete inside a container-based node; Kata needs a real Linux node

Verify on the node:

```
ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # expect: Y (Intel)
```

## How It Works[​](#how-it-works "Direct link to How It Works")

Identical to gVisor, with one naming detail: the isolation tier is called `kata`, but the RuntimeClass (and containerd handler) it maps to is **`kata-qemu`** — that is the handler kata-deploy registers.

```
Environment (isolation tier: kata)
  └─ deploy / promote → runtimeClassName: kata-qemu
       └─ RuntimeClass "kata-qemu" scheduling → pod on the Kata node, QEMU/KVM VM + guest kernel
```

Like gVisor, Kata uses a **dedicated node** labeled `kata=true` and tainted `kata=true:NoSchedule`, so existing runc (and gVisor) agents are never touched — no downtime.

## Requirements[​](#requirements "Direct link to Requirements")

| Requirement              | Details                                                                               |
| ------------------------ | ------------------------------------------------------------------------------------- |
| A new (empty) Linux node | Joined to the same Kubernetes cluster; dedicated to Kata agents                       |
| Virtualization           | `/dev/kvm` present and functional (see warning above)                                 |
| Architecture             | x86\_64                                                                               |
| OS                       | Ubuntu (or similar) with systemd + containerd — **not** Container-Optimized OS        |
| Access                   | kubectl access to the cluster (the installer runs from your machine, not on the node) |

## Set Up a Kata Node[​](#set-up-a-kata-node "Direct link to Set Up a Kata Node")

### Step 1: Add a new node to the cluster[​](#step-1-add-a-new-node-to-the-cluster "Direct link to Step 1: Add a new node to the cluster")

Provision a KVM-capable Linux node matching the requirements above and join it to your cluster using your platform's standard mechanism. Leave it empty.

Taint the node as early as possible

If your platform lets you set taints at node-pool creation, apply `kata=true:NoSchedule` there. An untainted node starts accepting regular platform pods immediately, and you will have to drain them all off before installing the runtime (the containerd reconfiguration restarts every pod on the node). If the node has already picked up workloads, `kubectl cordon` + `kubectl drain --ignore-daemonsets --delete-emptydir-data` it first.

### Step 2: Run the installer from a machine with kubectl access[​](#step-2-run-the-installer-from-a-machine-with-kubectl-access "Direct link to Step 2: Run the installer from a machine with kubectl access")

Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with kubectl access** and does everything through the Kubernetes API:

```
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/setup/install-kata.sh \
  -o install-kata.sh

KATA_NODES="<node-name>" bash install-kata.sh
```

What it does:

1. **Pre-flights `/dev/kvm`** on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
2. Labels the node(s) `kata=true`.
3. Applies the upstream **kata-deploy** DaemonSet, with a nodeSelector and toleration injected **before** apply — so it only ever lands on the labeled node(s); your other nodes are never reconfigured.
4. **Auto-wires k3s nodes**: k3s bundles its own containerd (configured only via `config.toml.tmpl`, read at startup), which upstream kata-deploy does not handle — the installer detects k3s nodes and patches the template + restarts the k3s agent for you.
5. Applies the `kata-qemu` RuntimeClass and taints the node(s) `kata=true:NoSchedule`.
6. Patches the Fluent Bit log DaemonSet to tolerate the taint, so agent logs are collected.

Verify

```
kubectl get runtimeclass kata-qemu
kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl get node <node-name> --show-labels   # kata=true
```

RKE2 nodes (bundled containerd)

Upstream kata-deploy assumes a standalone containerd at `/etc/containerd/config.toml`, which RKE2 nodes do not have. Two extra steps are needed on each RKE2 Kata node:

**1. Before running the installer**, create an empty stub so the kata-deploy pod gets past its pre-flight check (it crash-loops with `cp: can't stat '/etc/containerd/config.toml'` otherwise):

```
sudo mkdir -p /etc/containerd && sudo touch /etc/containerd/config.toml
```

**2. After kata-deploy has installed the binaries into `/opt/kata`** (the node gets the `katacontainers.io/kata-runtime=true` label on success), wire the runtime through RKE2's containerd template — kata-deploy wrote its config to the stub, which RKE2 never reads:

```
sudo mkdir -p /usr/local/bin
sudo ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-qemu-v2
sudo tee /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl > /dev/null <<'EOF'
{{ template "base" . }}

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
  runtime_type = "io.containerd.kata-qemu.v2"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
    ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
EOF
sudo systemctl restart rke2-agent
```

Rewrite, never append

The template must contain `{{ template "base" . }}` exactly once. If the file already exists (e.g. it also carries a gVisor or build-related block), merge the runtime sections into one file and rewrite it wholesale; appending a duplicate template line produces an invalid containerd config and the node's kubelet will not come back after the restart — recovery then requires SSH/console access, `kubectl` cannot reach the node.

### Local development[​](#local-development "Direct link to Local development")

There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable **Linux** host running the repository's dev k3d cluster:

```
make setup-kata
```

This joins the host itself to the cluster as a real (non-Docker) k3s agent node and then runs `install-kata.sh` against it.

## Create a Kata Environment[​](#create-a-kata-environment "Direct link to Create a Kata Environment")

Isolation tier is set at environment creation time. Open the **Create Environment** drawer in the console (Environments → Create Environment), select **Sandboxing T3 (Kata Containers)** from the **Isolation Tier** dropdown, and run the generated command — it includes `ISOLATION_TIER=kata` automatically.

If you script environment creation directly, set the variable yourself:

```
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment.sh \
  -o add-environment.sh

ISOLATION_TIER=kata \
  ENV_NAME=secure-prod \
  DISPLAY_NAME="Secure Prod" \
  AGENT_MANAGER_TOKEN=<token> \
  bash add-environment.sh
```

The tier appears as a **Sandboxing T3** shield chip in the environment list, and as a shield icon next to the environment name on the agent's Deploy and Overview pages — hover over it to confirm *Sandboxing Tier 3 — Kata Containers*. Then deploy or promote an agent to the environment as usual.

## Verify VM Isolation[​](#verify-vm-isolation "Direct link to Verify VM Isolation")

A Kata agent runs a **different kernel** from the node — that's the definitive check:

```
kubectl get pod <agent-pod> -n <namespace> -o jsonpath='{.spec.runtimeClassName}'   # kata-qemu

kubectl exec <agent-pod> -n <namespace> -- uname -r
# Differs from the node's own `uname -r`

kubectl exec <agent-pod> -n <namespace> -- grep hypervisor /proc/cpuinfo
# hypervisor flag present
```

If the pod reports the same kernel as the node, it is **not** running under Kata (the RuntimeClass wasn't applied or the pod fell back to runc).

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Pod stuck in ContainerCreating

Almost always missing nested virtualization. Confirm on the node itself:

```
ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # expect: Y
kubectl describe pod <pod> -n <ns>            # look for kvm / RuntimeClass errors
```

No `/dev/kvm` → move to a bare-metal or nested-virt-enabled host (e.g. GCP Intel N2 with `--enable-nested-virtualization`).

`no runtime handler kata-qemu` errors

The RuntimeClass exists but containerd on the node has no `kata-qemu` runtime — the install didn't complete or containerd wasn't reloaded:

```
kubectl -n kube-system rollout status daemonset/kata-deploy
kubectl -n kube-system logs -l name=kata-deploy

# on the node — confirm the runtime is in containerd's config:
sudo grep -n kata-qemu /var/lib/rancher/k3s/agent/etc/containerd/config.toml   # k3s nodes
grep -n kata-qemu /etc/containerd/config.toml                                  # other nodes
```

On k3s nodes, re-running `install-kata.sh` re-applies the containerd template wiring.

Pod runs but no logs / metrics / traces / try-it

This is cross-node networking or log-collection scheduling, not Kata itself — the failure modes and fixes are identical to gVisor's. Work through the [gVisor troubleshooting section](/agent-manager/docs/next/administration/isolation-tiers/gvisor/.md#troubleshooting), substituting `runtimeClassName: kata-qemu` in the test pod.

Performance notes

* Booting a VM adds startup latency compared to runc/gVisor; Agent Manager's warm pools absorb most of it.
* `kubectl port-forward` should be avoided for Kata pods — reach agents through their Service/gateway endpoint.
* Usage metrics are attributed to the sandbox VM as a whole (reported under a `sandbox` container label), including the guest kernel's footprint.

## Next Steps[​](#next-steps "Direct link to Next Steps")

* Lighter-weight syscall isolation without KVM: [gVisor isolation tier](/agent-manager/docs/next/administration/isolation-tiers/gvisor/.md).
* [Environment management](/agent-manager/docs/next/administration/environment-management/.md) — pipelines, promotion, and environment settings.