# On Your Environment

Install the Agent Manager on an existing Kubernetes cluster — AWS EKS, Google GKE, Azure AKS, or any distribution with LoadBalancer support.

Just want it running locally?

The [Quick Start Guide](/agent-manager/docs/v1.0.0-alpha1/getting-started/quick-start/.md) installs everything in a single command using a dev container with k3d. Use this page when you need to install on a managed cluster.

Stronger agent isolation tiers

Agents run sandboxed under the standard **runc** runtime by default. Agent Manager also supports stronger per-environment isolation tiers — **gVisor** (userspace kernel) and **Kata Containers** (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the [gVisor](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/kata/.md) setup guides.

## What You Will Get[​](#what-you-will-get "Direct link to What You Will Get")

Agent Manager is a two-layer system installed in two phases:

* **Phase 1 — OpenChoreo (base layer):** [OpenChoreo](https://openchoreo.dev) is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a **Control Plane** for API and configuration, a **Data Plane** for running workloads and gateways, a **Workflow Plane** for builds and CI pipelines, and an **Observability Plane** for traces, logs, and metrics via OpenSearch.

* **Phase 2 — Agent Manager :** The AI agent management platform installed on top of OpenChoreo. It includes the **Console** (web UI), **AMP API** (backend), **AI Gateway**, **PostgreSQL** (database), **Secrets Extension** (OpenBao for runtime secret injection), **Agent Manager Observer** (traces, logs, and metrics), and **Evaluation Engine** (automated agent evaluations).

This guide installs both layers on your existing Kubernetes cluster, production-first: a real domain, trusted TLS certificates issued automatically by cert-manager, and every management surface served through the three OpenChoreo plane gateways — the only LoadBalancers the platform needs.

No domain yet?

The main flow assumes you control a DNS zone. If you are evaluating on a cluster without one, follow the [nip.io appendix](#appendix-installing-without-a-domain-nipio) instead — same steps, with hostnames derived from LoadBalancer IPs and self-signed certificates.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### Cluster Requirements[​](#cluster-requirements "Direct link to Cluster Requirements")

| Requirement               | Minimum                                                                                                                                                                                                                                                                                                |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Kubernetes version        | 1.32+                                                                                                                                                                                                                                                                                                  |
| Nodes                     | 3                                                                                                                                                                                                                                                                                                      |
| CPU per node              | 4 cores                                                                                                                                                                                                                                                                                                |
| RAM per node              | 8 GB                                                                                                                                                                                                                                                                                                   |
| LoadBalancer support      | Required                                                                                                                                                                                                                                                                                               |
| Default StorageClass      | Required                                                                                                                                                                                                                                                                                               |
| Node kernel (build nodes) | Linux **6.3+** for user-namespaced agent builds (idmapped-mount support). On older kernels (Ubuntu 22.04's 5.15, Amazon Linux 2's 5.10) set `buildWorkflows.userNamespaces=false` on the Platform Resources chart — see [Troubleshooting](#build-pods-fail-with-mount_attr_idmap--idmap-mounts-errors) |

### Supported Providers[​](#supported-providers "Direct link to Supported Providers")

* **Amazon Web Services** (EKS)
* **Google Cloud Platform** (GKE)
* **Microsoft Azure** (AKS)
* Any Kubernetes distribution with LoadBalancer support

### Required Tools[​](#required-tools "Direct link to Required Tools")

| Tool                                               | Version | Purpose                                  |
| -------------------------------------------------- | ------- | ---------------------------------------- |
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+  | Kubernetes CLI                           |
| [Helm](https://helm.sh/docs/intro/install/)        | v3.12+  | Kubernetes package manager               |
| curl / dig                                         | —       | DNS resolution of LoadBalancer hostnames |

```
kubectl version --client && helm version
```

### Permissions[​](#permissions "Direct link to Permissions")

You need sufficient privileges to:

* Create namespaces, deploy Helm charts
* Create LoadBalancer services (only the OpenChoreo plane gateways need them — the Agent Manager services are ClusterIP and are served through those gateways)
* Manage cert-manager Issuers and Certificates
* Create CRDs and ClusterRoles

***

## Plan Your Deployment[​](#plan-your-deployment "Direct link to Plan Your Deployment")

Three decisions shape the installation. Make them now — everything below is driven by the variables you export here.

### 1. Hostnames[​](#1-hostnames "Direct link to 1. Hostnames")

Pick a base domain you control (a delegated subdomain like `amp.yourdomain.com` works well). Every management surface is a hostname on one of the three plane gateway LoadBalancers — after the platform installs, you publish DNS records against just those three targets:

| Hostname          | Serves                                                               | DNS record →                   |
| ----------------- | -------------------------------------------------------------------- | ------------------------------ |
| `console.<base>`  | Agent Manager Console                                                | control-plane gateway LB       |
| `api-amp.<base>`  | Agent Manager API (browser, `amctl`, MCP)                            | control-plane gateway LB       |
| `thunder.<base>`  | Thunder OAuth login (+ per-environment `<org>-<env>.thunder.<base>`) | control-plane gateway LB       |
| `cp.<base>`       | Gateway control plane (external AI gateways, optional)               | control-plane gateway LB       |
| `api.<base>`      | OpenChoreo API                                                       | control-plane gateway LB       |
| `traces.<base>`   | Agent Manager Observer                                               | observability-plane gateway LB |
| `*.agents.<base>` | Deployed-agent invocation endpoints (wildcard)                       | data-plane gateway LB          |

Concretely: `console`/`api-amp`/`thunder`/`cp`/`api` all point at the control-plane gateway (five records, or a single `*.<base>` wildcard), `traces` points at the observability-plane gateway, and `*.agents.<base>` plus the `agents.<base>` apex point at the data-plane gateway. [Step 10](#step-10-publish-dns-records) lists the exact records once the LoadBalancer addresses exist.

```
export VERSION="1.0.0-alpha1"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"

# Your base domain — everything below derives from it
export BASE_DOMAIN="amp.yourdomain.com"

export CONSOLE_PUBLIC_HOST="console.${BASE_DOMAIN}"
export API_PUBLIC_HOST="api-amp.${BASE_DOMAIN}"
export THUNDER_PUBLIC_HOST="thunder.${BASE_DOMAIN}"
export CP_GW_PUBLIC_HOST="cp.${BASE_DOMAIN}"
export OBS_API_PUBLIC_HOST="traces.${BASE_DOMAIN}"
export AGENTS_DOMAIN="agents.${BASE_DOMAIN}"

export CONSOLE_PUBLIC_URL="https://${CONSOLE_PUBLIC_HOST}"
export API_PUBLIC_URL="https://${API_PUBLIC_HOST}"
export THUNDER_PUBLIC_URL="https://${THUNDER_PUBLIC_HOST}"
export OBS_API_PUBLIC_URL="https://${OBS_API_PUBLIC_HOST}"

# In-cluster URL backend services use for Thunder JWKS/token calls
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"

# Trace-export endpoint shown to agent developers (see the OTLP note below)
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"
```

Why `api-amp` and not `api`?

The OpenChoreo API claims `api.<base>` on the same gateway, and the certificates below are single-level wildcards (`*.<base>`), so every hostname must sit directly under the base domain.

### 2. TLS certificates[​](#2-tls-certificates "Direct link to 2. TLS certificates")

The main flow uses **cert-manager with Let's Encrypt (DNS-01)**: publicly trusted certificates, automatic renewal, and wildcard support — you only need API credentials for the DNS provider hosting your zone. Step 3 also shows a corporate-CA alternative. Certificate issuance needs only the DNS zone, so it works before the DNS records for the gateways exist.

### 3. Dependencies[​](#3-dependencies "Direct link to 3. Dependencies")

The default install runs PostgreSQL, OpenBao, and OpenSearch in-cluster with development settings. Each install step below carries a **Production** callout with the values to swap in a managed database, a hardened secrets backend, and persistent observability storage. Decide up front which you will use — moving PostgreSQL data or Thunder's identity database later is disruptive.

OTLP trace ingest

`INSTRUMENTATION_URL` is the trace-export endpoint shown to agent developers; the traffic rides the data-plane gateway's `/otel` route. Agents running inside the cluster can use the default route hostname `http://default-default.gateway.localhost:19080/otel` (resolvable in-cluster). For agents running outside the cluster, install the API Platform Gateway extension with a public hostname whose DNS points at the data-plane gateway LoadBalancer — `--set gateway.vhost=https://otel.${BASE_DOMAIN}` and `--set gateway.hostname=otel.${BASE_DOMAIN}` — and set `INSTRUMENTATION_URL=https://otel.${BASE_DOMAIN}/otel`.

External AI gateways

AI gateways running outside this cluster connect through the control-plane gateway at `https://${CP_GW_PUBLIC_HOST}`. Phase 2 passes this hostname to the Agent Manager install; nothing else is needed beyond the DNS record. See [Register an AI Gateway](/agent-manager/docs/v1.0.0-alpha1/administration/register-ai-gateway/.md).

***

## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform")

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

* **Control Plane** — API server and configuration management for the platform
* **Data Plane** — runs deployed workloads and API gateways
* **Workflow Plane** — builds and CI pipelines for agent deployments
* **Observability Plane** — trace, log, and metrics collection via OpenSearch

This phase also installs Thunder (the identity provider) as a prerequisite, since the Control Plane and Observability Plane require Thunder's OIDC endpoints for JWT validation. **Estimated time: \~20-30 minutes** (varies by cluster and network).

### Step 1: Install Cluster Prerequisites[​](#step-1-install-cluster-prerequisites "Direct link to Step 1: Install Cluster Prerequisites")

**Gateway API CRDs (v1.4.1)** — standard Kubernetes resources for managing network gateways and routing:

```
kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

info

The `--force-conflicts` flag is needed if your cluster already has Gateway API CRDs managed by another controller (e.g., Traefik on k3s/Rancher Desktop).

**cert-manager (v1.19.2)** — automates TLS certificate issuance and renewal:

```
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --set startupapicheck.timeout=5m \
  --wait --timeout 360s
```

**External Secrets Operator (v1.3.2)** — syncs secrets from external stores (like OpenBao) into Kubernetes:

```
helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s
```

**kgateway (v2.2.1)** — the network gateway for OpenChoreo planes:

```
helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
```

Rancher Desktop / k3s users

k3s ships with Traefik which binds to host ports 80/443 and conflicts with OpenChoreo's kgateway. Remove Traefik before proceeding:

```
helm uninstall traefik -n kube-system
helm uninstall traefik-crd -n kube-system
```

After removing Traefik, re-apply the Gateway API CRDs (Traefik's CRD chart may have removed them):

```
kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

### Step 2: Setup Secrets Store (OpenBao)[​](#step-2-setup-secrets-store-openbao "Direct link to Step 2: Setup Secrets Store (OpenBao)")

OpenBao provides secret management for the Workflow Plane and deployed agents:

```
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-openbao.yaml \
  --timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
```

Configure the External Secrets ClusterSecretStore:

```
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF
```

Production

The values file above runs OpenBao in **dev mode** (in-memory backend, auto-unseal, well-known root token) — secrets are lost on pod restart. For production, install OpenBao without the dev-mode values: set `server.dev.enabled=false`, enable `server.dataStorage` persistence (or an external storage backend), configure [auto-unseal](https://openbao.org/docs/concepts/seal/), and create the Kubernetes auth role (`openchoreo-secret-writer-role`) plus the KV v2 mount at `secret/` that the ClusterSecretStore below expects.

### Step 3: Setup TLS Issuer[​](#step-3-setup-tls-issuer "Direct link to Step 3: Setup TLS Issuer")

Every gateway certificate below references one ClusterIssuer named `openchoreo-ca`. Create it from your chosen certificate authority — the rest of the guide is identical regardless of which tab you pick.

* Let's Encrypt (DNS-01)
* Corporate CA
* Self-signed (evaluation only)

DNS-01 validation issues publicly trusted certificates (including the wildcards this guide needs) by writing a TXT record to your DNS zone — the CA never connects to your cluster. Create a credentials Secret for your DNS provider, then the issuer. The example below uses AWS Route 53; cert-manager supports [Cloudflare, Google Cloud DNS, Azure DNS, and others](https://cert-manager.io/docs/configuration/acme/dns01/) with the same structure.

```
kubectl create secret generic route53-credentials \
  --namespace cert-manager \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>'

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyID: <AWS_ACCESS_KEY_ID>
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key
        selector:
          dnsZones:
            - "${BASE_DOMAIN}"
EOF
```

tip

Test with the Let's Encrypt staging server (`https://acme-staging-v02.api.letsencrypt.org/directory`) first if you expect to iterate — the production server rate-limits duplicate certificates to 5 per week.

If your organization runs an internal PKI, store its intermediate CA certificate and key as a Secret and create a CA issuer. Clients inside your network trust the certificates via your existing root-CA distribution.

```
kubectl create secret tls corporate-ca-secret \
  --namespace cert-manager \
  --cert=/path/to/intermediate-ca.crt \
  --key=/path/to/intermediate-ca.key

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  ca:
    secretName: corporate-ca-secret
EOF
```

For evaluation without a domain, the [nip.io appendix](#appendix-installing-without-a-domain-nipio) creates a self-signed chain under the same `openchoreo-ca` name. Browsers will warn on every hostname and backend components need TLS-verification overrides — do not use this beyond evaluation.

### Step 4: Install Thunder Extension (Identity Provider)[​](#step-4-install-thunder-extension-identity-provider "Direct link to Step 4: Install Thunder Extension (Identity Provider)")

Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder. The browser reaches it at `https://${THUNDER_PUBLIC_HOST}` through the control-plane gateway.

```
helm install amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.gateClient.scheme=https \
  --set thunder.configuration.gateClient.port=443 \
  --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \
  --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/instance=amp-thunder-extension \
  -n ${THUNDER_NS} --timeout=300s
```

warning

Thunder persists its configuration (including the issuer URL and the console client's OAuth redirect URIs) in a database on first boot. If you need to change `THUNDER_PUBLIC_URL` or `CONSOLE_PUBLIC_URL` after installation, you must **uninstall the chart, delete its PVC, and reinstall** — a `helm upgrade` alone will not change the issuer in issued tokens or the registered redirect URIs. This is why the hostnames are fixed in [Plan Your Deployment](#plan-your-deployment) before anything installs.

Verify

```
kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"${THUNDER_PUBLIC_URL}"  (must match your THUNDER_PUBLIC_URL value)
```

### Step 5: Install OpenChoreo Control Plane[​](#step-5-install-openchoreo-control-plane "Direct link to Step 5: Install OpenChoreo Control Plane")

Because the hostnames are fixed up front, the Control Plane installs once with its final configuration. Create the gateway's wildcard certificate first (issuance needs only the DNS zone, not the DNS records):

```
kubectl create namespace openchoreo-control-plane --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cp-gateway-tls
  namespace: openchoreo-control-plane
spec:
  secretName: cp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/cp-gateway-tls \
  -n openchoreo-control-plane --timeout=300s
```

Install the Control Plane with hostnames, TLS, and Thunder OIDC:

```
helm upgrade --install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.1.1 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --values - <<EOF
openchoreoApi:
  config:
    server:
      publicUrl: "https://api.${BASE_DOMAIN}"
  http:
    hostnames:
      - "api.${BASE_DOMAIN}"
backstage:
  enabled: false
  baseUrl: ""
  http:
    hostnames:
      - ""
security:
  oidc:
    issuer: "${THUNDER_PUBLIC_URL}"
    wellKnownEndpoint: "${THUNDER_INTERNAL_URL}/.well-known/openid-configuration"
    jwksUrl: "${THUNDER_INTERNAL_URL}/oauth2/jwks"
    authorizationUrl: "${THUNDER_PUBLIC_URL}/oauth2/authorize"
    tokenUrl: "${THUNDER_INTERNAL_URL}/oauth2/token"
gateway:
  tls:
    enabled: true
    hostname: "*.${BASE_DOMAIN}"
    certificateRefs:
      - name: cp-gateway-tls
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=300s
```

Webhook race condition

If the install fails with `no endpoints available for service "controller-manager-webhook-service"`, this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the `helm upgrade --install` command.

```
kubectl wait --for=condition=Available deployment --all \
  -n openchoreo-control-plane --timeout=300s
```

EKS Users

EKS LoadBalancers return a hostname instead of an IP — resolve it with `dig` when you publish DNS records later. For internet-facing access, annotate the gateway Service with `service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing`.

#### Patch the service-account entitlement claim (required with Thunder ≥ 0.45)[​](#patch-the-service-account-entitlement-claim-required-with-thunder--045 "Direct link to Patch the service-account entitlement claim (required with Thunder ≥ 0.45)")

Thunder 0.45+ puts the client name in the `client_id` claim (its `sub` is an opaque UUID), but OpenChoreo 1.1.1 extracts service-account entitlements from `sub`, and the chart does not expose the setting. Without this patch, every service-to-service call is **silently unauthorized** — the API returns `200` with empty lists and `403`s while all pods look healthy, and the gateway bootstrap later fails with `Environment 'default' not found`.

```
patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s

for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
  claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
  if [ "$claim" = "sub" ]; then
    kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge       -p '{"spec":{"entitlement":{"claim":"client_id"}}}'
  fi
done
```

note

Re-apply this patch after any `helm upgrade` of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's `observer-auth-config` in Step 8.

What the configuration does

* Backstage disabled (AMP provides its own console)
* OIDC issuer set to `THUNDER_PUBLIC_URL` (matches the `iss` claim in Thunder-issued JWTs)
* OIDC JWKS URL points to Thunder's in-cluster service (avoids external DNS dependency, and no TLS-verification overrides are needed since the in-cluster call is plain HTTP)
* OpenChoreo API at `api.${BASE_DOMAIN}`
* TLS enabled on the gateway with the wildcard certificate — Thunder, the Console, the API, and the gateway control plane all serve HTTPS under it

### Step 6: Setup Data Plane[​](#step-6-setup-data-plane "Direct link to Step 6: Setup Data Plane")

Copy the cluster-gateway CA certificate:

```
kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the deployed-agents wildcard certificate, then install the Data Plane with TLS in one pass:

```
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dp-gateway-tls
  namespace: openchoreo-data-plane
spec:
  secretName: dp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${AGENTS_DOMAIN}"
    - "${AGENTS_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/dp-gateway-tls \
  -n openchoreo-data-plane --timeout=300s

helm install openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.1.1 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${AGENTS_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=dp-gateway-tls" \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-dp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s
```

Register the Data Plane:

```
CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "${AGENTS_DOMAIN}"
          listenerName: http
          port: 80
        https:
          host: "${AGENTS_DOMAIN}"
          listenerName: https
          port: 443
  secretStoreRef:
    name: default
EOF
```

### Step 7: Setup Workflow Plane[​](#step-7-setup-workflow-plane "Direct link to Step 7: Setup Workflow Plane")

Copy the cluster-gateway CA certificate:

```
kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
```

Container Registry

The Workflow Plane needs a container registry to store built agent images. The registry endpoint is configured in **Phase 2 Step 3 (Platform Resources)** via the `global.registry.endpoint` or `global.baseDomain` Helm values. For local development, deploy an in-cluster `docker-registry` in this namespace — see the [k3d guide](/agent-manager/docs/v1.0.0-alpha1/getting-started/on-k3d/.md) for an example.

Install the Workflow Plane:

```
helm install openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.1.1 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --timeout 600s

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s
```

Register the Workflow Plane:

```
BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
  secretStoreRef:
    name: default
EOF
```

### Step 8: Setup Observability Plane[​](#step-8-setup-observability-plane "Direct link to Step 8: Setup Observability Plane")

Copy the cluster-gateway CA certificate:

```
kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the ExternalSecrets for OpenSearch and Observer credentials:

```
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF
```

Wait for the ExternalSecrets to sync:

```
kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s
```

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/oc-collector-configmap.yaml \
  -n openchoreo-observability-plane
```

Create the gateway certificate, then install the Observability Plane:

```
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: obs-gateway-tls
  namespace: openchoreo-observability-plane
spec:
  secretName: obs-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/obs-gateway-tls \
  -n openchoreo-observability-plane --timeout=300s

helm install openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.1.1 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${BASE_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=obs-gateway-tls" \
  --set clusterAgent.tls.generateCerts=true \
  --set observer.controlPlaneApiUrl="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --set observer.extraEnv.AUTH_SERVER_BASE_URL="${THUNDER_PUBLIC_URL}" \
  --set security.oidc.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set security.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-op.yaml \
  --timeout 25m

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=900s

for sts in $(kubectl get statefulset -n openchoreo-observability-plane -o name 2>/dev/null); do
  kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s
done
```

With Thunder ≥ 0.45, apply the same entitlement-claim patch from Step 5 to the observer (without it, build-log queries return `403 Access denied`):

```
patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s
```

Install observability modules (logs, metrics, tracing):

```
# Logs module
helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials" \
  --timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.6.1 \
  --timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m
```

Production

OpenSearch stores all traces, logs, and metrics. The default modules run it with ephemeral storage — set persistence before real use (`--set opensearch.persistence.enabled=true --set opensearch.persistence.size=100Gi` on the logs module that deploys OpenSearch, sized to your retention needs), or point the observability stack at a managed OpenSearch domain if your platform team operates one.

Register the Observability Plane and link it to other planes:

```
OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo-observability-plane.svc.cluster.local:8080
EOF

# Link Data Plane to Observability
kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability
kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
```

### Step 9: Verify OpenChoreo Installation[​](#step-9-verify-openchoreo-installation "Direct link to Step 9: Verify OpenChoreo Installation")

Before proceeding to Phase 2, confirm all planes are running:

```
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default
```

All pods should be in `Running` or `Completed` state.

### Step 10: Publish DNS Records[​](#step-10-publish-dns-records "Direct link to Step 10: Publish DNS Records")

The three plane gateways have LoadBalancers now — read their addresses and create the DNS records planned in [Plan Your Deployment](#plan-your-deployment):

```
for ns in openchoreo-control-plane openchoreo-observability-plane openchoreo-data-plane; do
  printf '%-34s %s\n' "$ns:" \
    "$(kubectl get svc gateway-default -n $ns \
        -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"
done
```

| Record                                                                                 | Type      | Points to                           |
| -------------------------------------------------------------------------------------- | --------- | ----------------------------------- |
| `console`, `api-amp`, `thunder`, `cp`, `api` `.${BASE_DOMAIN}` (or `*.${BASE_DOMAIN}`) | A / CNAME | control-plane gateway address       |
| `traces.${BASE_DOMAIN}`                                                                | A / CNAME | observability-plane gateway address |
| `*.agents.${BASE_DOMAIN}` and `agents.${BASE_DOMAIN}`                                  | A / CNAME | data-plane gateway address          |

Use A records for IPs (GKE, AKS) and CNAME records for hostnames (EKS). Verify resolution before continuing:

```
dig +short thunder.${BASE_DOMAIN} traces.${BASE_DOMAIN} test.${AGENTS_DOMAIN}
```

***

## Phase 2: Agent Manager Installation[​](#phase-2-agent-manager-installation "Direct link to Phase 2: Agent Manager Installation")

With OpenChoreo and Thunder running, you can now install the Agent Manager components — the API, console, and extensions that provide the AI agent management capabilities.

Production — managed PostgreSQL

The Agent Manager chart below deploys an in-cluster PostgreSQL with a default password by default. For production, use a managed database (RDS, Cloud SQL, Azure Database) and add these values to the `wso2-agent-manager` install in Step 2:

```
  --set postgresql.enabled=false \
  --set postgresql.external.host="your-db.example.com" \
  --set postgresql.external.port=5432 \
  --set postgresql.external.database=agentmanager \
  --set postgresql.external.username=agentmanager \
  --set postgresql.external.existingSecret=amp-db-credentials \
  --set postgresql.external.existingSecretPasswordKey=password
```

Create the `amp-db-credentials` Secret in `${AMP_NS}` beforehand. Sizing, backups, and HA then come from your database service.

<!-- -->

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

1. **Agent Manager Core :** Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
2. **Extensions :** Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.

Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables `THUNDER_PUBLIC_URL`, `THUNDER_INTERNAL_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`, `CONSOLE_PUBLIC_HOST`, `API_PUBLIC_HOST`, `OBS_API_PUBLIC_HOST`, and `INSTRUMENTATION_URL` must be set from the Configuration Variables section. The `*_PUBLIC_HOST` variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

***

### Core Components[​](#core-components "Direct link to Core Components")

Install these in order — each depends on the one before it.

#### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator")

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

```
helm install gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.6.0 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartVersion=1.1.1 \
  --timeout 600s
```

Wait for the operator to be ready:

```
kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s
```

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

```
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF
```

info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

#### Step 2: Agent Manager (API + Console + PostgreSQL)[​](#step-2-agent-manager-api--console--postgresql "Direct link to Step 2: Agent Manager (API + Console + PostgreSQL)")

The core platform: a Go API server, a React web console, and a PostgreSQL database.

```
helm install amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --timeout 1800s
```

Wait for all components:

```
# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s
```

Verify

```
kubectl get pods -n ${AMP_NS}
# Expected: amp-postgresql-0 (Running), amp-api-xxx (Running), amp-console-xxx (Running)
```

#### Step 3: Agent Sandbox Module[​](#step-3-agent-sandbox-module "Direct link to Step 3: Agent Sandbox Module")

Agents run as **sandboxed pods** managed by the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io/) controller (`SandboxTemplate` / `SandboxWarmPool` resources) instead of plain Deployments. This module is **required** — without it, agent deployments cannot be rendered.

```
helm upgrade --install agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6
```

Wait for the controller:

```
kubectl wait -n agent-sandbox-system \
  --for=condition=available \
  --timeout=180s \
  deployment/agent-sandbox-controller
```

Verify

```
kubectl get crd \
  sandboxtemplates.extensions.agents.x-k8s.io \
  sandboxwarmpools.extensions.agents.x-k8s.io \
  sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources
```

Stronger isolation tiers

With this module, agents run sandboxed under the standard **runc** runtime. Optionally, individual environments can run agents under **gVisor** or **Kata Containers** for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the [gVisor](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/kata/.md) isolation tier guides.

#### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources")

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the **container registry endpoint** used by build workflows to push agent images.

```
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --timeout 1800s
```

<!-- -->

##### Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at `host.k3d.internal:10082`. For other environments, override the registry settings:

```bash
# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
```

| Value                                        | Default                   | Description                                                 |
| -------------------------------------------- | ------------------------- | ----------------------------------------------------------- |
| `global.registry.endpoint`                   | `host.k3d.internal:10082` | Registry endpoint for pushing images                        |
| `global.baseDomain`                          | `""`                      | When set, registry endpoint becomes `registry.<baseDomain>` |
| `global.defaultResources.registry.tlsVerify` | `false`                   | Enable TLS verification for registry connections            |

***

### Extensions[​](#extensions "Direct link to Extensions")

These can be installed in any order after Core is ready.

#### Step 5: Observability Extension (Agent Manager Observer)[​](#step-5-observability-extension-agent-manager-observer "Direct link to Step 5: Observability Extension (Agent Manager Observer)")

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

```
helm install amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
```

#### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension")

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

```
helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s
```

info

The default `publisher.apiKey` must match `publisherApiKey.value` in the Agent Manager chart. Both default to `amp-internal-api-key`.

#### Step 7: API Platform Gateway Extension[​](#step-7-api-platform-gateway-extension "Direct link to Step 7: API Platform Gateway Extension")

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

```
helm install api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s
```

After the gateway is running, apply the OTEL trace collection RestApi:

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml
```

Verify

```
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"
```

### Wire the Remaining Public Endpoints[​](#wire-the-remaining-public-endpoints "Direct link to Wire the Remaining Public Endpoints")

Two settings are not covered by the shared installation steps above — apply them once all Phase 2 charts are installed.

**External AI gateway endpoint** — register the `cp.` hostname on the control-plane gateway route and let the console render the right setup commands:

```
helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --reuse-values \
  --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \
  --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}"
```

**Deployed-agent endpoints** — point the default Environment's gateway at the public agents domain (the Environment's setting wholly replaces the data plane's, so without this override agent invoke URLs are built against a placeholder host) and route the workload publisher's token calls in-cluster:

```
helm upgrade amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set environment.gateway.http.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.http.port=443 \
  --set environment.gateway.https.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.https.port=443
```

***

## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform")

Run a full status check to confirm everything is running:

```
# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'
```

### Via the Plane Gateways[​](#via-the-plane-gateways "Direct link to Via the Plane Gateways")

Everything is hostname-routed through the three plane gateway LoadBalancers, over TLS:

| Service                    | URL                              |
| -------------------------- | -------------------------------- |
| **Agent Manager Console**  | `https://${CONSOLE_PUBLIC_HOST}` |
| **Agent Manager API**      | `https://${API_PUBLIC_HOST}`     |
| **Thunder (OAuth login)**  | `https://${THUNDER_PUBLIC_HOST}` |
| **Agent Manager Observer** | `https://${OBS_API_PUBLIC_HOST}` |
| **OpenChoreo API**         | `https://api.${BASE_DOMAIN}`     |
| **Gateway control plane**  | `https://${CP_GW_PUBLIC_HOST}`   |

Open the Console, log in, then create → build → deploy an agent; its invoke URL is published under `https://<org>-<project>.${AGENTS_DOMAIN}`.

Port-forwarding remains available as a debugging fallback for any individual service (all Agent Manager services are ClusterIP), e.g. `kubectl port-forward -n wso2-amp svc/amp-console 3000:3000`.

**Default credentials:** `admin` / `admin`

***

## Appendix: Installing Without a Domain (nip.io)[​](#appendix-installing-without-a-domain-nipio "Direct link to Appendix: Installing Without a Domain (nip.io)")

For evaluation on a cluster where you control no DNS zone, [nip.io](https://nip.io) turns LoadBalancer IPs into resolvable hostnames. The flow is the same as above with four differences.

**1. Self-signed issuer.** In Step 3, create a self-signed chain under the same `openchoreo-ca` name instead of an ACME or corporate issuer:

```
kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: openchoreo-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: openchoreo-ca
  secretName: openchoreo-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  ca:
    secretName: openchoreo-ca-secret
EOF
```

Browsers will warn on every hostname (import `openchoreo-ca-secret`'s `ca.crt` into your trust store to avoid this), and the Control Plane install in Step 5 needs one extra override because it cannot verify Thunder's JWKS against an untrusted chain: add `--set-string openchoreoApi.config.security.authentication.jwt.jwks.skip_tls_verify=true` (and `--set-string security.oidc.jwksUrlTlsInsecureSkipVerify=true` on the Observability Plane install in Step 8).

**2. Hostnames come from the first LoadBalancer.** The control-plane gateway must exist before you can derive a base domain, but Thunder needs the final hostnames at install time. So: run Step 5's namespace creation, then install *only* the kgateway-provisioned gateway by running the Control Plane install once with placeholder hostnames (`api.placeholder.tld`, issuer `https://thunder.placeholder.tld`, `gateway.tls.enabled=false`), wait for the LoadBalancer, and derive the base domain:

```
CP_LB_IP=$(kubectl get svc gateway-default -n openchoreo-control-plane \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export BASE_DOMAIN="${CP_LB_IP//./-}.nip.io"
```

Now export the [Plan Your Deployment](#plan-your-deployment) variables from this `BASE_DOMAIN`, continue with Thunder (Step 4), and rerun the Step 5 install command — with the real values — as `helm upgrade`.

**3. One caveat on the shared base domain.** All hostnames (`console.`, `traces.`, `*.agents.` …) resolve to the control-plane gateway's IP, but `traces.` and `*.agents.` must reach the *other* two gateways. Point them correctly by using per-gateway nip.io domains instead: derive `TRACES`/`AGENTS` hostnames from the observability and data-plane LoadBalancer IPs after those planes install, e.g. `export OBS_API_PUBLIC_HOST="traces.$(kubectl get svc gateway-default -n openchoreo-observability-plane -o jsonpath='{.status.loadBalancer.ingress[0].ip}' | tr . -).nip.io"` and the equivalent `AGENTS_DOMAIN` from the data-plane gateway — then use those values in the observability/data-plane certificates and in Phase 2.

**4. Skip Step 10** (DNS records) — nip.io resolution is automatic.

The Thunder reinstall warning applies with full force here: if you tear down and recreate the cluster, the LoadBalancer IP — and therefore every hostname — changes, and Thunder must be reinstalled with the new values.

***

## Cloud Provider Notes[​](#cloud-provider-notes "Direct link to Cloud Provider Notes")

AWS EKS

* LoadBalancers return a **hostname** instead of an IP — use `dig` to resolve
* For internet-facing access, annotate LoadBalancer services:
  <!-- -->
  ```
  service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
  ```
* Ensure security groups allow HTTP/HTTPS traffic

Google Cloud Platform (GKE)

* LoadBalancers return IPs directly — no special handling needed
* Ensure firewall rules allow HTTP/HTTPS traffic to LoadBalancers

Microsoft Azure (AKS)

* LoadBalancers return IPs directly — no special handling needed
* Ensure Network Security Groups allow HTTP/HTTPS traffic

Rancher Desktop / k3s

* Remove Traefik before installation (see [Step 1](#step-1-install-cluster-prerequisites))
* Single-node clusters work for development but may run low on resources with all observability modules
* LoadBalancer IPs are assigned via the built-in k3s servicelb
* **cgroup `pids` controller issue** — see [Build workflow fails with cgroup pids error](#build-workflow-fails-with-cgroup-pids-error) in Troubleshooting

***

## Cleanup[​](#cleanup "Direct link to Cleanup")

Remove all Agent Manager and OpenChoreo resources:

```
# 1. Delete plane registrations
kubectl delete clusterdataplane default -n default
kubectl delete clusterworkflowplane default -n default
kubectl delete observabilityplane default -n default

# 2. Uninstall all Helm releases
helm uninstall amp -n wso2-amp
helm uninstall api-platform-default-default -n openchoreo-data-plane
helm uninstall amp-thunder-extension -n amp-thunder
helm uninstall amp-observability-traces -n openchoreo-observability-plane
helm uninstall amp-evaluation-extension -n openchoreo-workflow-plane
helm uninstall amp-platform-resources -n default
helm uninstall gateway-operator -n openchoreo-data-plane
helm uninstall openchoreo-observability-plane -n openchoreo-observability-plane
helm uninstall openchoreo-workflow-plane -n openchoreo-workflow-plane
helm uninstall openchoreo-data-plane -n openchoreo-data-plane
helm uninstall openchoreo-control-plane -n openchoreo-control-plane
helm uninstall openbao -n openbao
helm uninstall external-secrets -n external-secrets
helm uninstall cert-manager -n cert-manager

# 3. Delete namespaces
kubectl delete namespace wso2-amp amp-thunder \
  openchoreo-observability-plane openchoreo-workflow-plane \
  openchoreo-data-plane openchoreo-control-plane \
  openbao external-secrets cert-manager
```

***

## Production Considerations[​](#production-considerations "Direct link to Production Considerations")

The main flow above already covers real domains, trusted wildcard TLS, and gateway-served endpoints. Beyond it, plan for:

1. **Managed dependencies** — apply the **Production** callouts inline above: external PostgreSQL, OpenBao without dev mode, OpenSearch persistence
2. **Identity provider** — the seeded `admin`/`admin` user is for bootstrap only; connect your organization's IdP or manage users in Thunder before exposing the platform
3. **High availability** — deploy multiple replicas across availability zones and adjust resource requests/limits to workload
4. **Security hardening** — network policies, scoped RBAC, pod security standards
5. **Certificate operations** — with Let's Encrypt, renewals are automatic; monitor cert-manager and keep the DNS-provider credentials Secret rotated

***

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

LoadBalancer not getting external IP

```
kubectl describe svc <service-name> -n <namespace>
```

For EKS, ensure the AWS Load Balancer Controller is installed and the service has the correct annotations.

On k3s/Rancher Desktop, check if another service (like Traefik) is already using the required ports:

```
kubectl get svc -A --field-selector spec.type=LoadBalancer
```

Certificate not being issued

```
kubectl describe certificate <cert-name> -n <namespace>
kubectl get clusterissuers
kubectl get certificaterequests -n <namespace>
```

Plane registration issues

```
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
```

Agent Manager API returns 401 for environment/gateway calls

This typically means the OpenChoreo Control Plane's OIDC issuer does not match the `iss` claim in Thunder-issued JWTs. Verify:

```
# Check what issuer Thunder puts in tokens
kubectl exec -n amp-thunder deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'

# Check what the Control Plane expects
kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
  | grep issuer
```

Both must match exactly. If they don't, update the Control Plane's `security.oidc.issuer` to match Thunder's issuer.

Console shows "refused to connect" on login

The console redirects the browser to Thunder for OAuth login at `THUNDER_PUBLIC_URL`. Check that the `thunder.${BASE_DOMAIN}` DNS record points at the control-plane gateway LoadBalancer and that the Thunder HTTPRoute exists:

```
dig +short thunder.${BASE_DOMAIN}
kubectl get httproute -n openchoreo-control-plane amp-thunder-extension
curl -s -o /dev/null -w '%{http_code}\n' https://thunder.${BASE_DOMAIN}/.well-known/openid-configuration
```

If you need to change Thunder's public URL after installation, you must uninstall, delete the PVC, and reinstall:

```
helm uninstall amp-thunder-extension -n amp-thunder
kubectl delete pvc -n amp-thunder --all
# Then reinstall with the new THUNDER_PUBLIC_URL
```

OpenSearch connectivity issues

```
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
```

Build pods fail with MOUNT\_ATTR\_IDMAP / idmap mounts errors

If every agent build fails at pod-sandbox creation with events like:

```
failed to create containerd container: snapshotter "overlayfs" doesn't support
idmap mounts on this host, configure `slow_chown` to allow a slower and expensive fallback
```

or

```
OCI runtime create failed: ... failed to set MOUNT_ATTR_IDMAP on
/var/lib/kubelet/pods/.../volumes/kubernetes.io~projected/...: invalid argument
(maybe the filesystem used doesn't support idmap mounts on this kernel?)
```

the build pods are requesting a Linux **user namespace** (`hostUsers: false` — the build's root is unprivileged on the node), which requires idmapped-mount support for every filesystem the pod mounts. tmpfs (used by projected service-account token volumes) only gained that support in **Linux 6.3** — see the [Kubernetes user namespaces documentation](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/). Common node images below that line: Ubuntu 22.04 (kernel 5.15) and Amazon Linux 2 (kernel 5.10). Older Kubernetes versions silently ignored `hostUsers`, so the same manifests may have "worked" on an older cluster without ever using the feature.

Two fixes:

1. **Preferred:** use build nodes with kernel ≥ 6.3 (Ubuntu 24.04, Amazon Linux 2023, recent COS releases).
2. **Otherwise:** run builds without user namespaces by setting the Platform Resources chart value:

```
helm upgrade amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set buildWorkflows.userNamespaces=false
```

Note that containerd's `slow_chown` option alone is **not** sufficient — it only covers the overlayfs snapshotter, and the build then fails on the tmpfs/projected-volume mount instead.

Agent deploys fail with "Secret does not exist" after an OpenBao restart

This guide installs OpenBao in **dev mode** (in-memory storage). Any restart of the `openbao-0` pod — including node drains and reboots — wipes every secret it holds. Already-running agents keep working (their secrets were injected at startup), but any **new** deployment or environment promotion fails with ExternalSecrets stuck in `SecretSyncedError` / `Secret does not exist`.

Fix: re-save the affected agent's environment variables in the console (agent configuration → environment variables → re-enter and save). The values are rewritten to OpenBao and the ExternalSecret syncs within its refresh interval. For production, configure OpenBao with persistent storage.

Build workflow fails with cgroup pids error (Rancher Desktop)

If the build workflow fails with:

```
Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available
Error: exit status 126
```

This happens on Rancher Desktop because the underlying Lima VM (Alpine Linux) does not delegate the `pids` cgroup controller to containers. The Podman containers inside the build workflow cannot create the required cgroup namespace.

**Fix:** Patch the ClusterWorkflowTemplates that use Podman to inject a `containers.conf` that disables cgroup management.

note

The patch commands below require **python3** to be installed on your machine.

Run these commands to patch each template:

```
# Patch gcp-buildpacks-build (build-image step)
kubectl get clusterworkflowtemplate gcp-buildpacks-build -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# Patch publish-image
kubectl get clusterworkflowtemplate publish-image -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# Patch amp-generate-workload
kubectl get clusterworkflowtemplate amp-generate-workload -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = fix + script
json.dump(data, sys.stdout)
" | kubectl apply -f -
```

After patching, re-trigger the build workflow. These patches are applied in-cluster and will be overwritten if the Helm chart (`amp-platform-resources`) is reinstalled.

note

This issue affects Rancher Desktop specifically because it runs k3s inside a Lima VM with Alpine Linux, which uses OpenRC instead of systemd. The `pids` cgroup controller is not delegated to containers by default. Other Kubernetes distributions (EKS, GKE, AKS) are not affected.